![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4184342%2Fceaa6dcb05d8c8d0043c822944b14426%2F000.png?generation=1591460200706162&alt=media\")
.\nBut ,when I use your 512x512 tfrecords not external data,it can work.The top scoring public notebook here uses the 30k training and an additional 30k external images. I created TFRecords here with all these 30K train, 30k external, and 10k test including meta data:
\n\n feature = {\n 'image': _bytes_feature,\n 'image_name': _bytes_feature,\n 'patient_id': _int64_feature,\n 'sex': _int64_feature,\n 'age_approx': _int64_feature,\n 'anatom_site_general_challenge': _int64_feature,\n 'source': _int64_feature,\n 'target': _int64_feature\n }\nThank you Alex Shonenkov for posting these external jpeg images and publishing your great starter notebook here. (Alex describes his dataset here and creates his Jpeg dataset here)
\n\nFor those curious, I posted the code to generate the TFRecords here
\n\nhttps://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images
\n\nhttps://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg
\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579
", "messages": [ { "id": 874540, "postDate": "2020-06-05T04:56:26.673Z", "content": "The top scoring public notebook here uses the 30k training and an additional 30k external images. I created TFRecords here with all these 30K train, 30k external, and 10k test including meta data:
\n\n feature = {\n 'image': _bytes_feature,\n 'image_name': _bytes_feature,\n 'patient_id': _int64_feature,\n 'sex': _int64_feature,\n 'age_approx': _int64_feature,\n 'anatom_site_general_challenge': _int64_feature,\n 'source': _int64_feature,\n 'target': _int64_feature\n }\nThank you Alex Shonenkov for posting these external jpeg images and publishing your great starter notebook here. (Alex describes his dataset here and creates his Jpeg dataset here)
\n\nFor those curious, I posted the code to generate the TFRecords here
\n\nhttps://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images
\n\nhttps://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg
\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579
", "rawMarkdown": "### TFRecords External Data plus Train plus Test with Meta Data\nThe top scoring public notebook [here][1] uses the 30k training and an additional 30k external images. I created TFRecords [here][3] with all these 30K train, 30k external, and 10k test including meta data:\n\n feature = {\n 'image': _bytes_feature,\n 'image_name': _bytes_feature,\n 'patient_id': _int64_feature,\n 'sex': _int64_feature,\n 'age_approx': _int64_feature,\n 'anatom_site_general_challenge': _int64_feature,\n 'source': _int64_feature,\n 'target': _int64_feature\n }\n\nThank you [Alex Shonenkov][2] for posting these external jpeg images and publishing your great starter notebook [here][7]. (Alex describes his dataset [here][5] and creates his Jpeg dataset [here][6])\n\nFor those curious, I posted the code to generate the TFRecords [here][4]\n\n### TFRecord Kaggle Dataset here:\nhttps://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images\n\n### JPEG Kaggle Dataset here:\nhttps://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg\n\n### More TFRecord 768x768, 512x512, 384x384, and 256x256\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579\n\n[1]: https://www.kaggle.com/shonenkov/inference-single-model-melanoma-starter\n[2]: https://www.kaggle.com/shonenkov\n[3]: https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images\n[4]: https://www.kaggle.com/cdeotte/how-to-create-tfrecords\n[5]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155859\n[6]: https://www.kaggle.com/shonenkov/merge-external-data\n[7]: https://www.kaggle.com/shonenkov/inference-single-model-melanoma-starter", "votes": 55 }, { "id": 874737, "postDate": "2020-06-05T09:07:34.210Z", "content": "Did you apply any specific rules for external data? for example ISIC dataset has categories such as carcinoma and lesions (which are by definition malignant). However, these are not covered in this competition training set. I have been using external data too, but so far dropped such categories. Maybe will need to rethink this...
", "rawMarkdown": "Did you apply any specific rules for external data? for example ISIC dataset has categories such as carcinoma and lesions (which are by definition malignant). However, these are not covered in this competition training set. I have been using external data too, but so far dropped such categories. Maybe will need to rethink this...", "votes": 3, "replies": [ { "id": 874750, "postDate": "2020-06-05T09:17:27.580Z", "content": "my understanding is that the goal here is to detect melanoma and therefore carcinoma is also 0 here. Melanoma is by far more serious than carcinoma as it can easily spread to other organs. Also your understanding of lesions is incorrect as lesion is any damage and it can be both caused by mechanical or chemical agents, disease, including neoplasm which can be both malignant and benign.
", "rawMarkdown": "my understanding is that the goal here is to detect melanoma and therefore carcinoma is also 0 here. Melanoma is by far more serious than carcinoma as it can easily spread to other organs. Also your understanding of lesions is incorrect as lesion is any damage and it can be both caused by mechanical or chemical agents, disease, including neoplasm which can be both malignant and benign. ", "votes": 1 }, { "id": 874754, "postDate": "2020-06-05T09:19:23.987Z", "content": "I agree with you completely :) But the question is that carcinomas/sarcomas/malignant lesions bring out-of-distribution images which may not necessarily be a good thing for the competition target metric
", "rawMarkdown": "I agree with you completely :) But the question is that carcinomas/sarcomas/malignant lesions bring out-of-distribution images which may not necessarily be a good thing for the competition target metric", "votes": 1 }, { "id": 874756, "postDate": "2020-06-05T09:21:21.417Z", "content": "Another thing I noticed is, that although ISIC contains thousands of melanoma cases, but most of them appear different from our training set melanomas - mostly due to different microscopic zoom (my guess). So it is still also questionable if to use all ISIC melanomas too :)
", "rawMarkdown": "Another thing I noticed is, that although ISIC contains thousands of melanoma cases, but most of them appear different from our training set melanomas - mostly due to different microscopic zoom (my guess). So it is still also questionable if to use all ISIC melanomas too :)" }, { "id": 874764, "postDate": "2020-06-05T09:29:29.213Z", "content": "that's another question my friend and it's yet to be answered :). My gut feeling is that it should help if used wisely. After all, external data is much closer to the official data compared to imagenet so it must be useful in some way.
", "rawMarkdown": "that's another question my friend and it's yet to be answered :). My gut feeling is that it should help if used wisely. After all, external data is much closer to the official data compared to imagenet so it must be useful in some way. " }, { "id": 875438, "postDate": "2020-06-05T19:25:52.570Z", "content": "Great questions @raddar . I just used Alex Shonenkov's external dataset that he describes here and creates here because I saw how well his notebook using them performed here. We need to ask those questions to Alex.
", "rawMarkdown": "Great questions @raddar . I just used Alex Shonenkov's external dataset that he describes [here][1] and creates [here][2] because I saw how well his notebook using them performed [here][3]. We need to ask those questions to Alex.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155859\n[2]: https://www.kaggle.com/shonenkov/merge-external-data\n[3]: https://www.kaggle.com/shonenkov/inference-single-model-melanoma-starter" } ] }, { "id": 874543, "postDate": "2020-06-05T05:06:13.357Z", "content": "This external data is helpful. My current LB 0.945 is an ensemble of models using these TFRecords and two other models using two different sized image datasets from my other TFRecords here. So my ensemble uses images of three different sizes.
", "rawMarkdown": "This external data is helpful. My current LB 0.945 is an ensemble of models using these TFRecords and two other models using two different sized image datasets from my other TFRecords [here][1]. So my ensemble uses images of three different sizes.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579", "votes": 4, "replies": [ { "id": 874605, "postDate": "2020-06-05T06:05:45.877Z", "content": "thx @cdeotte have you noted the difference external data makes :)
", "rawMarkdown": "thx @cdeotte have you noted the difference external data makes :)" }, { "id": 874608, "postDate": "2020-06-05T06:11:31.797Z", "content": "It helped add variety to my ensemble. As for a single model, it doesn't seem better than my other single models so far, which all score around 0.925. But i don't trust my CV LB yet, so I'm not sure.
", "rawMarkdown": "It helped add variety to my ensemble. As for a single model, it doesn't seem better than my other single models so far, which all score around 0.925. But i don't trust my CV LB yet, so I'm not sure." } ] }, { "id": 1027957, "postDate": "2020-09-26T13:42:29.730Z", "content": "Good job, continue like this ! BRAVO
", "rawMarkdown": "Good job, continue like this ! BRAVO ", "votes": 1 }, { "id": 936924, "postDate": "2020-07-20T15:43:38.073Z", "content": "Hello guys. This is my first competition. I wanted to ask if we can optimize AUC during training and if it would be a good idea. I have seen implementations where you can pass AUC as a metric to optimize but there have been some discussions where optimizing AUC is not very successful.\n@cdeotte said that increasing cross-entropy score doesn't guarantee that your model will perform well on AUC score too. So how should one go about it?
", "rawMarkdown": "Hello guys. This is my first competition. I wanted to ask if we can optimize AUC during training and if it would be a good idea. I have seen implementations where you can pass AUC as a metric to optimize but there have been some discussions where optimizing AUC is not very successful.\n@cdeotte said that increasing cross-entropy score doesn't guarantee that your model will perform well on AUC score too. So how should one go about it?", "votes": 1, "replies": [ { "id": 936930, "postDate": "2020-07-20T15:57:23.477Z", "content": "This is a current hot research topic that has not been solved yet. You can not use AUC directly as a loss function since it is not differentiable. So many researchers have proposed new loss functions that encourage maximizing AUC. What works and what doesn't work is still being debated.
", "rawMarkdown": "This is a current hot research topic that has not been solved yet. You can not use AUC directly as a loss function since it is not differentiable. So many researchers have proposed new loss functions that encourage maximizing AUC. What works and what doesn't work is still being debated.", "votes": 3 } ] }, { "id": 898608, "postDate": "2020-06-23T16:17:16.957Z", "content": "Good day. Excuse me stupid question - I am not experienced with tfr format and when I've tried to use 512X512 package in this kernel - https://www.kaggle.com/ragnar123/efficientnet-x-384 it gives me error. Are trf format interchangeable, or needs to be adjusted to model?
", "rawMarkdown": "Good day. Excuse me stupid question - I am not experienced with tfr format and when I've tried to use 512X512 package in this kernel - https://www.kaggle.com/ragnar123/efficientnet-x-384 it gives me error. Are trf format interchangeable, or needs to be adjusted to model?", "votes": 1, "replies": [ { "id": 914671, "postDate": "2020-07-04T06:16:30.040Z", "content": "I have two types of TFRecords with slightly different fields. (6 fields are same 1 field is different). I have my 768x768, 512x512, 384x384, 256x256 ones. And they all use the same read function below. This is the function in Ragnar's notebook that you link.
\n\ndef read_labeled_tfrecord(example):\nLABELED_TFREC_FORMAT = {\n 'image': tf.io.FixedLenFeature([], tf.string),\n 'image_name': tf.io.FixedLenFeature([], tf.string),\n 'patient_id': tf.io.FixedLenFeature([], tf.int64),\n 'sex': tf.io.FixedLenFeature([], tf.int64),\n 'age_approx': tf.io.FixedLenFeature([], tf.int64),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'diagnosis': tf.io.FixedLenFeature([], tf.int64),\n 'target': tf.io.FixedLenFeature([], tf.int64)\n}\nIf you would like to use my other 512x512 image that include external data, then change this function to the following:
\n\ndef read_labeled_tfrecord(example):\nLABELED_TFREC_FORMAT = {\n 'image': tf.io.FixedLenFeature([], tf.string),\n 'image_name': tf.io.FixedLenFeature([], tf.string),\n 'patient_id': tf.io.FixedLenFeature([], tf.int64),\n 'sex': tf.io.FixedLenFeature([], tf.int64),\n 'age_approx': tf.io.FixedLenFeature([], tf.int64),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'source': tf.io.FixedLenFeature([], tf.int64),\n 'target': tf.io.FixedLenFeature([], tf.int64)\n}\nThe main difference is that the former has a field named diagnosis and the latter has a field source. You can remove fields from either function if you don't plan to use them. But you cannot add a field that does not exist. So the former dataset cannot have source and the later dataset cannot have diagnosis.
I started working on this competition yesterday. These comments are really interesting to start with. Great.
", "rawMarkdown": "I started working on this competition yesterday. These comments are really interesting to start with. Great.", "votes": 1, "replies": [ { "id": 896223, "postDate": "2020-06-22T01:52:02.853Z", "content": "Thanks
", "rawMarkdown": "Thanks" } ] }, { "id": 885216, "postDate": "2020-06-14T02:55:03.970Z", "content": "UPDATE: I confirm that this dataset when used in an ensemble with my other datasets here can score at least LB 0.949!
", "rawMarkdown": "UPDATE: I confirm that this dataset when used in an ensemble with my other datasets [here][1] can score at least LB 0.949!\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579", "votes": 1, "replies": [ { "id": 885258, "postDate": "2020-06-14T04:13:48.073Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 878558, "postDate": "2020-06-08T16:22:38.390Z", "content": "Thank you for the dataset, Chris!\nI have a question, can you help me?\nI trained my model on this dataset (image+meta multiinput), it has awesome CV, but it performs worse on the LB then my model trained only on competiton data. Augmentations, architecture etc. are the same.\nAt first I thought that maybe patient_id causes leakage, and it just causes more problems on the bigger dataset. But I retrained my model without patient_idand the CV-LB scores are almost the same. Any thoughts why this is happening?
I have seen before when training NN and target class has 1% positive and metric is AUC. It appears that optimizing cross entropy in this case can have large swings in AUC. Because both AUC 0.90 and 0.95 can have the same cross entropy score! So you see that increasing cross entropy score doesn't guarantee that your model is AUC 0.95 versus AUC 0.90.
\n\nI'm still working on trying to stabilize and optimize AUC.
", "rawMarkdown": "I have seen before when training NN and target class has 1% positive and metric is AUC. It appears that optimizing cross entropy in this case can have large swings in AUC. Because both AUC 0.90 and 0.95 can have the same cross entropy score! So you see that increasing cross entropy score doesn't guarantee that your model is AUC 0.95 versus AUC 0.90.\n\nI'm still working on trying to stabilize and optimize AUC.", "votes": 3 }, { "id": 878724, "postDate": "2020-06-08T19:11:45.923Z", "content": "Thank you!\nI thought it was some kind of leakage, but didn't think of the obvious.
", "rawMarkdown": "Thank you!\nI thought it was some kind of leakage, but didn't think of the obvious." } ] }, { "id": 2666138, "postDate": "2024-02-24T07:05:27.743Z", "content": "Is there a reason whhy the testing data does not contain target value?
", "rawMarkdown": "Is there a reason whhy the testing data does not contain target value?" }, { "id": 876534, "postDate": "2020-06-06T18:48:08.300Z", "content": "UPDATE: version 1 of these TFRecords had some incorrect image_name which created errors when submitting to LB. I just uploaded version 2 today, and i just made a submission using the new version 2. I confirm that the new version 2 has correct image_name and scores well on LB. Thanks @chihantsai for bringing this to my attention.
Seems something wrong on https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images ,\nI got same image_name on test.
", "rawMarkdown": "Seems something wrong on https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images ,\nI got same image_name on test.", "replies": [ { "id": 876301, "postDate": "2020-06-06T15:58:52.483Z", "content": "Can you explain the problem in more detail?
", "rawMarkdown": "Can you explain the problem in more detail?" }, { "id": 876334, "postDate": "2020-06-06T16:18:44.987Z", "content": "I tried use tfecords-70k in this ,\nI got \n.\nBut ,when I use your 512x512 tfrecords not external data,it can work.
Ah, yes there is a bug in my Kaggle notebook TFRecord generation code, it writes the wrong file names to the test TFRecords. I will fix the notebook now. Thanks.
", "rawMarkdown": "Ah, yes there is a bug in my Kaggle notebook TFRecord generation code, it writes the wrong file names to the test TFRecords. I will fix the notebook now. Thanks.", "votes": 1 }, { "id": 876370, "postDate": "2020-06-06T16:37:44.303Z", "content": "Training seems has same problem.
", "rawMarkdown": "Training seems has same problem.", "votes": 1 }, { "id": 876380, "postDate": "2020-06-06T16:43:04.467Z", "content": "Yes, thanks i saw that. I'm fixing both and rerunning notebook now. Then i will update Kaggle dataset.
", "rawMarkdown": "Yes, thanks i saw that. I'm fixing both and rerunning notebook now. Then i will update Kaggle dataset.", "votes": 2 }, { "id": 876439, "postDate": "2020-06-06T17:24:15.030Z", "content": "The new corrected Kaggle dataset has been uploaded. Let me know if you see any other problems.
", "rawMarkdown": "The new corrected Kaggle dataset has been uploaded. Let me know if you see any other problems.", "votes": 2 } ] }, { "id": 895803, "postDate": "2020-06-21T16:08:38.827Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 896109, "postDate": "2020-06-21T21:16:42.743Z", "content": "Thanks for the info. I did not remove them. The code to create the TFRecords is here. I use all of Alex's jpegs.
\n\nWith so many images, a few duplicates won't cause a problem. Furthermore, if you apply data augmentation during training, then the duplicate images will not be duplicate.
\n\nI can confirm that using the dataset as I have published it can score over 0.950+ CV LB
", "rawMarkdown": "Thanks for the info. I did not remove them. The code to create the TFRecords is [here][1]. I use all of Alex's jpegs. \n\nWith so many images, a few duplicates won't cause a problem. Furthermore, if you apply data augmentation during training, then the duplicate images will not be duplicate.\n\nI can confirm that using the dataset as I have published it can score over 0.950+ CV LB\n\n[1]: https://www.kaggle.com/cdeotte/how-to-create-tfrecords" } ] }, { "id": 894067, "postDate": "2020-06-20T06:34:29.137Z", "content": "@cdeotte \nThank for great datasets!!
", "rawMarkdown": "@cdeotte \nThank for great datasets!! ", "votes": 1 }, { "id": 874542, "postDate": "2020-06-05T05:03:14.170Z", "content": "Excellent again. \nthank you both.
", "rawMarkdown": "Excellent again. \nthank you both.", "votes": 1 } ], "comments": [ { "id": 874737, "author_name": "raddar", "author_url": "", "post_date": "2020-06-05T09:07:34.210000", "content": "Did you apply any specific rules for external data? for example ISIC dataset has categories such as carcinoma and lesions (which are by definition malignant). However, these are not covered in this competition training set. I have been using external data too, but so far dropped such categories. Maybe will need to rethink this...
", "votes": 3, "replies": [ { "id": 874750, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-06-05T09:17:27.580000", "content": "my understanding is that the goal here is to detect melanoma and therefore carcinoma is also 0 here. Melanoma is by far more serious than carcinoma as it can easily spread to other organs. Also your understanding of lesions is incorrect as lesion is any damage and it can be both caused by mechanical or chemical agents, disease, including neoplasm which can be both malignant and benign.
", "votes": 1, "replies": [] }, { "id": 874754, "author_name": "raddar", "author_url": "", "post_date": "2020-06-05T09:19:23.987000", "content": "I agree with you completely :) But the question is that carcinomas/sarcomas/malignant lesions bring out-of-distribution images which may not necessarily be a good thing for the competition target metric
", "votes": 1, "replies": [] }, { "id": 874756, "author_name": "raddar", "author_url": "", "post_date": "2020-06-05T09:21:21.417000", "content": "Another thing I noticed is, that although ISIC contains thousands of melanoma cases, but most of them appear different from our training set melanomas - mostly due to different microscopic zoom (my guess). So it is still also questionable if to use all ISIC melanomas too :)
", "votes": 0, "replies": [] }, { "id": 874764, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-06-05T09:29:29.213000", "content": "that's another question my friend and it's yet to be answered :). My gut feeling is that it should help if used wisely. After all, external data is much closer to the official data compared to imagenet so it must be useful in some way.
", "votes": 0, "replies": [] }, { "id": 875438, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-05T19:25:52.570000", "content": "Great questions @raddar . I just used Alex Shonenkov's external dataset that he describes here and creates here because I saw how well his notebook using them performed here. We need to ask those questions to Alex.
", "votes": 0, "replies": [] } ] }, { "id": 874543, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-05T05:06:13.357000", "content": "This external data is helpful. My current LB 0.945 is an ensemble of models using these TFRecords and two other models using two different sized image datasets from my other TFRecords here. So my ensemble uses images of three different sizes.
", "votes": 4, "replies": [ { "id": 874605, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-06-05T06:05:45.877000", "content": "thx @cdeotte have you noted the difference external data makes :)
", "votes": 0, "replies": [] }, { "id": 874608, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-05T06:11:31.797000", "content": "It helped add variety to my ensemble. As for a single model, it doesn't seem better than my other single models so far, which all score around 0.925. But i don't trust my CV LB yet, so I'm not sure.
", "votes": 0, "replies": [] } ] }, { "id": 1027957, "author_name": "Naim Mhedhbi", "author_url": "", "post_date": "2020-09-26T13:42:29.730000", "content": "Good job, continue like this ! BRAVO
", "votes": 1, "replies": [] }, { "id": 936924, "author_name": "Raman Dutt", "author_url": "", "post_date": "2020-07-20T15:43:38.073000", "content": "Hello guys. This is my first competition. I wanted to ask if we can optimize AUC during training and if it would be a good idea. I have seen implementations where you can pass AUC as a metric to optimize but there have been some discussions where optimizing AUC is not very successful.\n@cdeotte said that increasing cross-entropy score doesn't guarantee that your model will perform well on AUC score too. So how should one go about it?
", "votes": 1, "replies": [ { "id": 936930, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-20T15:57:23.477000", "content": "This is a current hot research topic that has not been solved yet. You can not use AUC directly as a loss function since it is not differentiable. So many researchers have proposed new loss functions that encourage maximizing AUC. What works and what doesn't work is still being debated.
", "votes": 3, "replies": [] } ] }, { "id": 898608, "author_name": "Serge", "author_url": "", "post_date": "2020-06-23T16:17:16.957000", "content": "Good day. Excuse me stupid question - I am not experienced with tfr format and when I've tried to use 512X512 package in this kernel - https://www.kaggle.com/ragnar123/efficientnet-x-384 it gives me error. Are trf format interchangeable, or needs to be adjusted to model?
", "votes": 1, "replies": [ { "id": 914671, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-04T06:16:30.040000", "content": "I have two types of TFRecords with slightly different fields. (6 fields are same 1 field is different). I have my 768x768, 512x512, 384x384, 256x256 ones. And they all use the same read function below. This is the function in Ragnar's notebook that you link.
\n\ndef read_labeled_tfrecord(example):\nLABELED_TFREC_FORMAT = {\n 'image': tf.io.FixedLenFeature([], tf.string),\n 'image_name': tf.io.FixedLenFeature([], tf.string),\n 'patient_id': tf.io.FixedLenFeature([], tf.int64),\n 'sex': tf.io.FixedLenFeature([], tf.int64),\n 'age_approx': tf.io.FixedLenFeature([], tf.int64),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'diagnosis': tf.io.FixedLenFeature([], tf.int64),\n 'target': tf.io.FixedLenFeature([], tf.int64)\n}\nIf you would like to use my other 512x512 image that include external data, then change this function to the following:
\n\ndef read_labeled_tfrecord(example):\nLABELED_TFREC_FORMAT = {\n 'image': tf.io.FixedLenFeature([], tf.string),\n 'image_name': tf.io.FixedLenFeature([], tf.string),\n 'patient_id': tf.io.FixedLenFeature([], tf.int64),\n 'sex': tf.io.FixedLenFeature([], tf.int64),\n 'age_approx': tf.io.FixedLenFeature([], tf.int64),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'source': tf.io.FixedLenFeature([], tf.int64),\n 'target': tf.io.FixedLenFeature([], tf.int64)\n}\nThe main difference is that the former has a field named diagnosis and the latter has a field source. You can remove fields from either function if you don't plan to use them. But you cannot add a field that does not exist. So the former dataset cannot have source and the later dataset cannot have diagnosis.
I started working on this competition yesterday. These comments are really interesting to start with. Great.
", "votes": 1, "replies": [ { "id": 896223, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-22T01:52:02.853000", "content": "Thanks
", "votes": 0, "replies": [] } ] }, { "id": 885216, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-14T02:55:03.970000", "content": "UPDATE: I confirm that this dataset when used in an ensemble with my other datasets here can score at least LB 0.949!
", "votes": 1, "replies": [ { "id": 885258, "author_name": "", "author_url": "", "post_date": "2020-06-14T04:13:48.073000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 878558, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2020-06-08T16:22:38.390000", "content": "Thank you for the dataset, Chris!\nI have a question, can you help me?\nI trained my model on this dataset (image+meta multiinput), it has awesome CV, but it performs worse on the LB then my model trained only on competiton data. Augmentations, architecture etc. are the same.\nAt first I thought that maybe patient_id causes leakage, and it just causes more problems on the bigger dataset. But I retrained my model without patient_idand the CV-LB scores are almost the same. Any thoughts why this is happening?
I have seen before when training NN and target class has 1% positive and metric is AUC. It appears that optimizing cross entropy in this case can have large swings in AUC. Because both AUC 0.90 and 0.95 can have the same cross entropy score! So you see that increasing cross entropy score doesn't guarantee that your model is AUC 0.95 versus AUC 0.90.
\n\nI'm still working on trying to stabilize and optimize AUC.
", "votes": 3, "replies": [] }, { "id": 878724, "author_name": "Stanislav Blinov", "author_url": "", "post_date": "2020-06-08T19:11:45.923000", "content": "Thank you!\nI thought it was some kind of leakage, but didn't think of the obvious.
", "votes": 0, "replies": [] } ] }, { "id": 2666138, "author_name": "Raj123465", "author_url": "", "post_date": "2024-02-24T07:05:27.743000", "content": "Is there a reason whhy the testing data does not contain target value?
", "votes": 0, "replies": [] }, { "id": 876534, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-06T18:48:08.300000", "content": "UPDATE: version 1 of these TFRecords had some incorrect image_name which created errors when submitting to LB. I just uploaded version 2 today, and i just made a submission using the new version 2. I confirm that the new version 2 has correct image_name and scores well on LB. Thanks @chihantsai for bringing this to my attention.
Seems something wrong on https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images ,\nI got same image_name on test.
", "votes": 0, "replies": [ { "id": 876301, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-06T15:58:52.483000", "content": "Can you explain the problem in more detail?
", "votes": 0, "replies": [] }, { "id": 876334, "author_name": "fate", "author_url": "", "post_date": "2020-06-06T16:18:44.987000", "content": "I tried use tfecords-70k in this ,\nI got \n.\nBut ,when I use your 512x512 tfrecords not external data,it can work.
Ah, yes there is a bug in my Kaggle notebook TFRecord generation code, it writes the wrong file names to the test TFRecords. I will fix the notebook now. Thanks.
", "votes": 1, "replies": [] }, { "id": 876370, "author_name": "fate", "author_url": "", "post_date": "2020-06-06T16:37:44.303000", "content": "Training seems has same problem.
", "votes": 1, "replies": [] }, { "id": 876380, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-06T16:43:04.467000", "content": "Yes, thanks i saw that. I'm fixing both and rerunning notebook now. Then i will update Kaggle dataset.
", "votes": 2, "replies": [] }, { "id": 876439, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-06T17:24:15.030000", "content": "The new corrected Kaggle dataset has been uploaded. Let me know if you see any other problems.
", "votes": 2, "replies": [] } ] }, { "id": 895803, "author_name": "", "author_url": "", "post_date": "2020-06-21T16:08:38.827000", "content": "", "votes": 1, "replies": [ { "id": 896109, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-06-21T21:16:42.743000", "content": "Thanks for the info. I did not remove them. The code to create the TFRecords is here. I use all of Alex's jpegs.
\n\nWith so many images, a few duplicates won't cause a problem. Furthermore, if you apply data augmentation during training, then the duplicate images will not be duplicate.
\n\nI can confirm that using the dataset as I have published it can score over 0.950+ CV LB
", "votes": 0, "replies": [] } ] }, { "id": 894067, "author_name": "Nishi yasu", "author_url": "", "post_date": "2020-06-20T06:34:29.137000", "content": "@cdeotte \nThank for great datasets!!
", "votes": 1, "replies": [] }, { "id": 874542, "author_name": "Bruce Young", "author_url": "", "post_date": "2020-06-05T05:03:14.170000", "content": "Excellent again. \nthank you both.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "874540": "### TFRecords External Data plus Train plus Test with Meta Data\nThe top scoring public notebook [here][1] uses the 30k training and an additional 30k external images. I created TFRecords [here][3] with all these 30K train, 30k external, and 10k test including meta data:\n\n feature = {\n 'image': _bytes_feature,\n 'image_name': _bytes_feature,\n 'patient_id': _int64_feature,\n 'sex': _int64_feature,\n 'age_approx': _int64_feature,\n 'anatom_site_general_challenge': _int64_feature,\n 'source': _int64_feature,\n 'target': _int64_feature\n }\n\nThank you [Alex Shonenkov][2] for posting these external jpeg images and publishing your great starter notebook [here][7]. (Alex describes his dataset [here][5] and creates his Jpeg dataset [here][6])\n\nFor those curious, I posted the code to generate the TFRecords [here][4]\n\n### TFRecord Kaggle Dataset here:\nhttps://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images\n\n### JPEG Kaggle Dataset here:\nhttps://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg\n\n### More TFRecord 768x768, 512x512, 384x384, and 256x256\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579\n\n[1]: https://www.kaggle.com/shonenkov/inference-single-model-melanoma-starter\n[2]: https://www.kaggle.com/shonenkov\n[3]: https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images\n[4]: https://www.kaggle.com/cdeotte/how-to-create-tfrecords\n[5]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155859\n[6]: https://www.kaggle.com/shonenkov/merge-external-data\n[7]: https://www.kaggle.com/shonenkov/inference-single-model-melanoma-starter", "874737": "Did you apply any specific rules for external data? for example ISIC dataset has categories such as carcinoma and lesions (which are by definition malignant). However, these are not covered in this competition training set. I have been using external data too, but so far dropped such categories. Maybe will need to rethink this...", "874543": "This external data is helpful. My current LB 0.945 is an ensemble of models using these TFRecords and two other models using two different sized image datasets from my other TFRecords [here][1]. So my ensemble uses images of three different sizes.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579", "1027957": "Good job, continue like this ! BRAVO ", "936924": "Hello guys. This is my first competition. I wanted to ask if we can optimize AUC during training and if it would be a good idea. I have seen implementations where you can pass AUC as a metric to optimize but there have been some discussions where optimizing AUC is not very successful.\n@cdeotte said that increasing cross-entropy score doesn't guarantee that your model will perform well on AUC score too. So how should one go about it?", "898608": "Good day. Excuse me stupid question - I am not experienced with tfr format and when I've tried to use 512X512 package in this kernel - https://www.kaggle.com/ragnar123/efficientnet-x-384 it gives me error. Are trf format interchangeable, or needs to be adjusted to model?", "895572": "I started working on this competition yesterday. These comments are really interesting to start with. Great.", "885216": "UPDATE: I confirm that this dataset when used in an ensemble with my other datasets [here][1] can score at least LB 0.949!\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/155579", "878558": "Thank you for the dataset, Chris!\nI have a question, can you help me?\nI trained my model on this dataset (image+meta multiinput), it has awesome CV, but it performs worse on the LB then my model trained only on competiton data. Augmentations, architecture etc. are the same.\nAt first I thought that maybe `patient_id` causes leakage, and it just causes more problems on the bigger dataset. But I retrained my model without `patient_id `and the CV-LB scores are almost the same. Any thoughts why this is happening?", "2666138": "Is there a reason whhy the testing data does not contain target value?", "876534": "UPDATE: version 1 of these TFRecords had some incorrect `image_name` which created errors when submitting to LB. I just uploaded version 2 today, and i just made a submission using the new version 2. I confirm that the new version 2 has correct `image_name` and scores well on LB. Thanks @chihantsai for bringing this to my attention.", "876188": "Seems something wrong on https://www.kaggle.com/cdeotte/512x512-melanoma-tfrecords-70k-images ,\nI got same image_name on test.", "895803": "", "894067": "@cdeotte \nThank for great datasets!! ", "874542": "Excellent again. \nthank you both." } }