![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F00448b6d6fcd8733d9cc028d847645b3%2FScreen%20Shot%202020-07-22%20at%207.55.03%20PM.png?generation=1595472921909492&alt=media\")
Just when you thought it wasn't possible for me to publish any more datasets, here are more Kaggle datasets with brand new, never seen before malignant melanoma images!.
\n\nThe training data only has 584 malignant images out of 33126. This isn't many examples for our models to learn what malignant looks like. So below are TFRecords containing 4000 high quality malignant examples.
\n\nTo teach your model more about malignant images, you can add more TFRecords that only contain malignant images to your training data. Do not include them in your validation data. In order to compare whether these extra images help, you must always use the same validation data of just 2020 comp data but you can add more data to your training data:
\n\nGCS_PATH = KaggleDatasets().get_gcs_path('melanoma-256x256')\nGCS_PATH2 = KaggleDatasets().get_gcs_path('malignant-v2-256x256')\nfiles_train = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\nMALIGNANT = [GCS_PATH2 + '/train%.2i*.tfrec'%x for x in IDX]\nfiles_train += tf.io.gfile.glob(MALIGNANT)\nnp.random.shuffle(files_train)\nwhere IDX is a list of numbers identifying which malignant TFRecords you wish to include. (Experiment by including some or all, you can even include some multiple times like IDX = [0,0,1,1] + MORE).
These datasets contain 60 TFRecords containing 4000 malignant images (and one JPEG folder of the 580 new never seen before malignant images). The TFRecords coincide with my triple stratified TFRecords, so you can use them together and still be tripled stratified!\n* 1024x1024 TFRecords JPEGs with target and meta (980MB)\n* 768x768 TFRecords JPEGs with target and meta (580MB)\n* 512x512 TFRecords JPEGs with targets and meta (290MB)\n* 384x384 TFRecords JPEGs with targets and meta (178MB)\n* 256x256 TFRecords JPEGs with targets and meta (90MB)\n* 192x192 TFRecords JPEGs with targets and meta (55MB)\n* 128x128 TFRecords JPEGs with targets and meta (30MB)
\n\nThe first 15 TFRecords contain the malignant images from this years 2020 comp. There are 584 malignant images. Note that these TFRecords coincide with my other 15 training TFRecords. So the malignant images that are in my train TFRecord00, 01, 02 etc are the same malignant that are in my malignant TFRecord00, 01, 02 respectively.\n
The next 15 TFRecords contain 580 never seen before malignant images!. These images have been downloaded from ISIC's online gallery here. The online gallery has a total of 2285 malignant images. But only 580 are not contained in 2020, 2019, 2018, nor 2017 data. So only 580 are new to us.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F2e6f62dd3044a2c3d60b9665911cb954%2FScreen%20Shot%202020-07-22%20at%207.56.02%20PM.png?generation=1595472980100684&alt=media\")
The next 15 even numbered TFRecords contain the malignant images from 2018 2017 comp data. There are 1614 malignant images (1664 with 13 dups removed and 37 weird removed). Note that these TFRecords coincide with my other 15 training 2018 2017 TFRecords. So the malignant images that are in my 2018 2017 TFRecord00, 02, 04, etc are the same malignant that are in my malignant TFRecord30, 32, 34, respectively. (You need to add 30 to each of the 0, 2, ..., 26, 28 even numbers in my other 2018 2017 train data for the numbers to match).\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F3449dd036fb547bd86db87960f38c6fc%2FScreen%20Shot%202020-07-22%20at%207.57.17%20PM.png?generation=1595473052692047&alt=media\")
The next 15 odd numbered TFRecords contain the malignant images from 2019 new portion comp data. According to RAPIDS t-SNE discussion, there are 1185 good new portion 2019 malignant images among the total 2858 new portion 2019 malignant. Note that these TFRecords coincide with my other 15 training new portion 2019 TFRecords. So the malignant images that are in my 2019 TFRecord01, 03, 05 etc are the same malignant that are in my malignant TFRecord31, 33, 35 respectively minus filtered. (You need to add 30 to each of the 1, 3, ..., 27, 29 odd numbers in my other 2019 train data for the numbers to match).\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F7fb0c9482e2a543c989a1310093ebc3a%2FScreen%20Shot%202020-07-22%20at%207.56.50%20PM.png?generation=1595473025198630&alt=media\")
There is a starter notebook here showing how to upsample using the new Malignant datasets. The starter notebook also shows how to perform coarse dropout data augmentation. Enjoy!
\n\nTo download full datasets containing both the benign and malignant data resized to 1024x1024, 768x768, 512x512, 384x384, 256x256, 192x192, 128x128\n* The 2020 comp data is here\n* The 2019, 2018, 2017 comp data is here
", "messages": [ { "id": 940572, "postDate": "2020-07-23T03:08:48.777Z", "content": "Just when you thought it wasn't possible for me to publish any more datasets, here are more Kaggle datasets with brand new, never seen before malignant melanoma images!.
\n\nThe training data only has 584 malignant images out of 33126. This isn't many examples for our models to learn what malignant looks like. So below are TFRecords containing 4000 high quality malignant examples.
\n\nTo teach your model more about malignant images, you can add more TFRecords that only contain malignant images to your training data. Do not include them in your validation data. In order to compare whether these extra images help, you must always use the same validation data of just 2020 comp data but you can add more data to your training data:
\n\nGCS_PATH = KaggleDatasets().get_gcs_path('melanoma-256x256')\nGCS_PATH2 = KaggleDatasets().get_gcs_path('malignant-v2-256x256')\nfiles_train = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\nMALIGNANT = [GCS_PATH2 + '/train%.2i*.tfrec'%x for x in IDX]\nfiles_train += tf.io.gfile.glob(MALIGNANT)\nnp.random.shuffle(files_train)\nwhere IDX is a list of numbers identifying which malignant TFRecords you wish to include. (Experiment by including some or all, you can even include some multiple times like IDX = [0,0,1,1] + MORE).
These datasets contain 60 TFRecords containing 4000 malignant images (and one JPEG folder of the 580 new never seen before malignant images). The TFRecords coincide with my triple stratified TFRecords, so you can use them together and still be tripled stratified!\n* 1024x1024 TFRecords JPEGs with target and meta (980MB)\n* 768x768 TFRecords JPEGs with target and meta (580MB)\n* 512x512 TFRecords JPEGs with targets and meta (290MB)\n* 384x384 TFRecords JPEGs with targets and meta (178MB)\n* 256x256 TFRecords JPEGs with targets and meta (90MB)\n* 192x192 TFRecords JPEGs with targets and meta (55MB)\n* 128x128 TFRecords JPEGs with targets and meta (30MB)
\n\nThe first 15 TFRecords contain the malignant images from this years 2020 comp. There are 584 malignant images. Note that these TFRecords coincide with my other 15 training TFRecords. So the malignant images that are in my train TFRecord00, 01, 02 etc are the same malignant that are in my malignant TFRecord00, 01, 02 respectively.\n
The next 15 TFRecords contain 580 never seen before malignant images!. These images have been downloaded from ISIC's online gallery here. The online gallery has a total of 2285 malignant images. But only 580 are not contained in 2020, 2019, 2018, nor 2017 data. So only 580 are new to us.\n
The next 15 even numbered TFRecords contain the malignant images from 2018 2017 comp data. There are 1614 malignant images (1664 with 13 dups removed and 37 weird removed). Note that these TFRecords coincide with my other 15 training 2018 2017 TFRecords. So the malignant images that are in my 2018 2017 TFRecord00, 02, 04, etc are the same malignant that are in my malignant TFRecord30, 32, 34, respectively. (You need to add 30 to each of the 0, 2, ..., 26, 28 even numbers in my other 2018 2017 train data for the numbers to match).\n
The next 15 odd numbered TFRecords contain the malignant images from 2019 new portion comp data. According to RAPIDS t-SNE discussion, there are 1185 good new portion 2019 malignant images among the total 2858 new portion 2019 malignant. Note that these TFRecords coincide with my other 15 training new portion 2019 TFRecords. So the malignant images that are in my 2019 TFRecord01, 03, 05 etc are the same malignant that are in my malignant TFRecord31, 33, 35 respectively minus filtered. (You need to add 30 to each of the 1, 3, ..., 27, 29 odd numbers in my other 2019 train data for the numbers to match).\n
There is a starter notebook here showing how to upsample using the new Malignant datasets. The starter notebook also shows how to perform coarse dropout data augmentation. Enjoy!
\n\nTo download full datasets containing both the benign and malignant data resized to 1024x1024, 768x768, 512x512, 384x384, 256x256, 192x192, 128x128\n* The 2020 comp data is here\n* The 2019, 2018, 2017 comp data is here
", "rawMarkdown": "# More Kaggle Datasets 😄 4000 Malignant Images!\nJust when you thought it wasn't possible for me to publish any more datasets, here are more Kaggle datasets with **brand new, never seen before malignant melanoma images!**. \n\nThe training data only has 584 malignant images out of 33126. This isn't many examples for our models to learn what malignant looks like. So below are TFRecords containing 4000 high quality malignant examples.\n\n# How To Upsample Malignant Images\nTo teach your model more about malignant images, you can add more TFRecords that only contain malignant images to your training data. Do not include them in your validation data. In order to compare whether these extra images help, you must always use the same validation data of just 2020 comp data but you can add more data to your training data:\n\n GCS_PATH = KaggleDatasets().get_gcs_path('melanoma-256x256')\n GCS_PATH2 = KaggleDatasets().get_gcs_path('malignant-v2-256x256')\n files_train = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\n MALIGNANT = [GCS_PATH2 + '/train%.2i*.tfrec'%x for x in IDX]\n files_train += tf.io.gfile.glob(MALIGNANT)\n np.random.shuffle(files_train)\n\nwhere `IDX` is a list of numbers identifying which malignant TFRecords you wish to include. (Experiment by including some or all, you can even include some multiple times like `IDX = [0,0,1,1] + MORE`).\n\n# Download Links\nThese datasets contain 60 TFRecords containing 4000 malignant images (and one JPEG folder of the 580 new never seen before malignant images). The TFRecords coincide with my triple stratified TFRecords, so you can use them together and still be tripled stratified!\n* [1024x1024 TFRecords JPEGs with target and meta][7] (980MB)\n* [768x768 TFRecords JPEGs with target and meta][6] (580MB)\n* [512x512 TFRecords JPEGs with targets and meta][5] (290MB)\n* [384x384 TFRecords JPEGs with targets and meta][4] (178MB)\n* [256x256 TFRecords JPEGs with targets and meta][3] (90MB)\n* [192x192 TFRecords JPEGs with targets and meta][2] (55MB)\n* [128x128 TFRecords JPEGs with targets and meta][1] (30MB)\n\n# TFRecords 0-14\nThe first 15 TFRecords contain the malignant images from this years 2020 comp. There are 584 malignant images. Note that these TFRecords coincide with my other 15 training TFRecords. So the malignant images that are in my train TFRecord00, 01, 02 etc are the same malignant that are in my malignant TFRecord00, 01, 02 respectively.\n\n\n\n# TFRecords 15-29\nThe next 15 TFRecords contain 580 **never seen before malignant images!**. These images have been downloaded from ISIC's online gallery [here][8]. The online gallery has a total of 2285 malignant images. But only 580 are not contained in 2020, 2019, 2018, nor 2017 data. So only 580 are new to us.\n\n\n\n# TFRecords even numbered 30, 32, ..., 56, 58\nThe next 15 **even numbered** TFRecords contain the malignant images from 2018 2017 comp data. There are 1614 malignant images (1664 with 13 dups removed and 37 weird removed). Note that these TFRecords coincide with my other 15 training 2018 2017 TFRecords. So the malignant images that are in my 2018 2017 TFRecord00, 02, 04, etc are the same malignant that are in my malignant TFRecord30, 32, 34, respectively. (You need to add 30 to each of the 0, 2, ..., 26, 28 even numbers in my other 2018 2017 train data for the numbers to match).\n\n\n\n# TFRecords odd numbered 31, 33, ..., 57, 59\nThe next 15 **odd numbered** TFRecords contain the malignant images from 2019 new portion comp data. According to [RAPIDS t-SNE discussion][11], there are 1185 good new portion 2019 malignant images among the total 2858 new portion 2019 malignant. Note that these TFRecords coincide with my other 15 training new portion 2019 TFRecords. So the malignant images that are in my 2019 TFRecord01, 03, 05 etc are the same malignant that are in my malignant TFRecord31, 33, 35 respectively minus filtered. (You need to add 30 to each of the 1, 3, ..., 27, 29 odd numbers in my other 2019 train data for the numbers to match).\n\n\n# Starter Notebook\nThere is a starter notebook [here][12] showing how to upsample using the new Malignant datasets. The starter notebook also shows how to perform coarse dropout data augmentation. Enjoy!\n\n# Full Training Datasets\nTo download full datasets containing both the benign and malignant data resized to 1024x1024, 768x768, 512x512, 384x384, 256x256, 192x192, 128x128\n* The 2020 comp data is [here][9]\n* The 2019, 2018, 2017 comp data is [here][10]\n\n\n[1]: https://www.kaggle.com/cdeotte/malignant-v2-128x128\n[2]: https://www.kaggle.com/cdeotte/malignant-v2-192x192\n[3]: https://www.kaggle.com/cdeotte/malignant-v2-256x256\n[4]: https://www.kaggle.com/cdeotte/malignant-v2-384x384\n[5]: https://www.kaggle.com/cdeotte/malignant-v2-512x512\n[6]: https://www.kaggle.com/cdeotte/malignant-v2-768x768\n[7]: https://www.kaggle.com/cdeotte/malignant-v2-1024x1024\n[8]: https://www.isic-archive.com/#!/topWithHeader/onlyHeaderTop/gallery\n[9]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164092\n[10]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910\n[11]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168028\n[12]: https://www.kaggle.com/cdeotte/tfrecord-experiments-upsample-and-coarse-dropout", "votes": 185 }, { "id": 944193, "postDate": "2020-07-24T22:59:36.237Z", "content": "UPDATE: I posted a starter notebook here showing how to use the new upsample Malignant Kaggle datasets. Enjoy!
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F1072df5bac18d75b8ec85b6971a68d07%2FScreen%20Shot%202020-07-24%20at%203.57.47%20PM.png?generation=1595631480936155&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F81fe43d79bb72f5e7734eb75d20c715c%2FScreen%20Shot%202020-07-24%20at%203.58.22%20PM.png?generation=1595631512825268&alt=media\")
UPDATE: I ran a careful experiment to see if upsample improves AUC for kNN. Using only 2020 data, it appears that upsample can indeed increase AUC for kNN by at least 0.003!. The plot is below and the experiment is described here
\nUPDATE2: I ran another experiment to see if upsample improves AUC for CNN. Using only 2020 data, it appears that upsample can indeed increase AUC for CNN. I will post the CNN experiment code after the comp finishes.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F4caa9d5cc1e1da34fc4a135fe14b5ff5%2Faucc.png?generation=1596847727243572&alt=media\")
Just to make sure this wasn't luck, i ran the experiment 100 more times with 100 different seeds. The average AUC increase is 0.00207 with STD 0.00122. We are 99.99999% confident that upsample increases AUC!
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fe5a3fac04e3f1e844d0a3fa19efeb76e%2Fhist.png?generation=1596895739271319&alt=media\")
P-values and CI's are not the same thing though :)
", "rawMarkdown": "P-values and CI's are not the same thing though :)", "votes": 2 }, { "id": 962957, "postDate": "2020-08-08T15:12:41.443Z", "content": "There was typo in my post. STD = 0.0012 not 0.0017 (as we can see in the plot).
\nFor a 95% one-sided hypothesis test, you need z score > 1.65 and for a 95% two-sided hypothesis test you need z score > 1.96. This is a one sided hypothesis test and we have z score = (0.00207 / 0.00122) * sqrt(100) = 17.0, therefore the result is conclusive.
UPDATE: We need to use a t test instead of a z test. The t test statistic is 35.09 using online calculator here so p<0.0001 and we are 99.99999% confident
", "rawMarkdown": "There was typo in my post. STD = 0.0012 not 0.0017 (as we can see in the plot). \n\nFor a 95% one-sided hypothesis test, you need `z score > 1.65` and for a 95% two-sided hypothesis test you need `z score > 1.96`. This is a one sided hypothesis test and we have `z score = (0.00207 / 0.00122) * sqrt(100) = 17.0`, therefore the result is conclusive. \n\nUPDATE: We need to use a t test instead of a z test. The t test statistic is 35.09 using online calculator [here][1] so p<0.0001 and we are 99.99999% confident\n\n[1]: https://www.usablestats.com/calcs/1samplet&summary=1", "votes": 2 }, { "id": 963038, "postDate": "2020-08-08T16:08:23.567Z", "content": "@cdeotte why don't you say upsampling helps KNN? Your experiment may mislead people into thinking that upsampling helps CNN. It may be true, but your experiment is not confirming it by any mean.
", "rawMarkdown": "@cdeotte why don't you say upsampling helps KNN? Your experiment may mislead people into thinking that upsampling helps CNN. It may be true, but your experiment is not confirming it by any mean.", "votes": 1 }, { "id": 964521, "postDate": "2020-08-10T01:07:08.840Z", "content": "UPDATE: I originally computed my test statistic wrong in my paired t-test and have updated it above. (I forgot to multiply by the square root of n). Using the correct test statistic (from online calculator here), we reject the null hypothesis with p = < .00001.
We are 99.99999% confident that upsampling increases AUC for kNN in this comp!
\n\nwhy don't you say upsampling helps KNN? Your experiment may mislead people into thinking that upsampling helps CNN
\n
Good point @cpmpml .CNN and kNN may react differently to upsample. I have conducted another paired t-test for CNN, and found that upsample increases AUC for CNN with p<0.00001. We are 99.99999% confident that upsampling increases AUC for CNN.
\nI will update my other posts to say this and I will post the code for the CNN experiments after the comp finishes.
", "rawMarkdown": "> why don't you say upsampling helps KNN? Your experiment may mislead people into thinking that upsampling helps CNN\n\nGood point @cpmpml .CNN and kNN may react differently to upsample. I have conducted another paired t-test for CNN, and found that upsample increases AUC for CNN with p<0.00001. We are 99.99999% confident that upsampling increases AUC for CNN. \n\nI will update my other posts to say this and I will post the code for the CNN experiments after the comp finishes.", "votes": 1 }, { "id": 965236, "postDate": "2020-08-10T13:40:09.237Z", "content": "Thank you for your rigorous explanation! I'm glad this discussion turned out to be productive (at least for me).
\n\nI'm removing my first comment as the numbers are different now.
", "rawMarkdown": "Thank you for your rigorous explanation! I'm glad this discussion turned out to be productive (at least for me). \n\nI'm removing my first comment as the numbers are different now.", "votes": 1 } ] }, { "id": 940612, "postDate": "2020-07-23T03:50:02.920Z", "content": "Great thanks Chris. \nTwo obvious questions...\nHave you seen improvements using this new data?\nIs this external data ok to use under the competition rules?
\n\nSomeone put grease on the competition ladder so I’m slipping down it fast.
\n\nThanks for your continuing hard work and sharing.
", "rawMarkdown": "Great thanks Chris. \nTwo obvious questions...\nHave you seen improvements using this new data?\nIs this external data ok to use under the competition rules?\n\nSomeone put grease on the competition ladder so I’m slipping down it fast.\n\nThanks for your continuing hard work and sharing.", "votes": 3, "replies": [ { "id": 940618, "postDate": "2020-07-23T03:53:12.823Z", "content": "\n\n\nSomeone put grease on the competition ladder so I’m slipping down it fast.
\n
Bruce, if I am honest, I am pretty sure I am overfitting the Public Leaderboard and so are the others that have recently surpassed you on the competition ladder.
\n\nI have been following your position since the start, and I believe you have nothing to worry about, I think with the Private Leaderboard, you will have a strong position unless you are overfitting too :)
", "rawMarkdown": ">Someone put grease on the competition ladder so I’m slipping down it fast.\n\n\nBruce, if I am honest, I am pretty sure I am overfitting the Public Leaderboard and so are the others that have recently surpassed you on the competition ladder.\n\nI have been following your position since the start, and I believe you have nothing to worry about, I think with the Private Leaderboard, you will have a strong position unless you are overfitting too :)", "votes": 2 }, { "id": 940633, "postDate": "2020-07-23T04:05:36.530Z", "content": "The changes on the leaderboard are strange. Some people with very few submissions also got 0.96+.
", "rawMarkdown": "The changes on the leaderboard are strange. Some people with very few submissions also got 0.96+.", "votes": 6 }, { "id": 940648, "postDate": "2020-07-23T04:19:02.167Z", "content": "\n\n\nIs this external data ok to use under the competition rules?
\n
Yes. We are already using 75% of this data. These are the malignant images from 2020, 2019, 2018, 2017 comp data. Most notebooks use these already. But now with these records, we can be more flexible. For example, we can use 2020 benign and malignant. And then 2019 2018 2017 malignant only. Or we can use 2020, 2019, 2018, 2017 plus another dose of malignant from 2020, 2019, 2018, 2017. Since we are using data augmentation, the second copy of malignant will look different to our models and help train them.
\n\nThere are only 580 new images in this dataset never seen before. This data is from the competition host's website and is ok to use. The host has confirmed it's use in multiple other discussions, example here.
", "rawMarkdown": "> Is this external data ok to use under the competition rules?\n\nYes. We are already using 75% of this data. These are the malignant images from 2020, 2019, 2018, 2017 comp data. Most notebooks use these already. But now with these records, we can be more flexible. For example, we can use 2020 benign and malignant. And then 2019 2018 2017 malignant only. Or we can use 2020, 2019, 2018, 2017 plus another dose of malignant from 2020, 2019, 2018, 2017. Since we are using data augmentation, the second copy of malignant will look different to our models and help train them. \n\nThere are only 580 new images in this dataset never seen before. This data is from the competition host's website and is ok to use. The host has confirmed it's use in multiple other discussions, example [here][1].\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154296#865350", "votes": 1 }, { "id": 940655, "postDate": "2020-07-23T04:23:22.683Z", "content": "\n\n\nSomeone put grease on the competition ladder so I’m slipping down it fast.
\n
Two image Kaggle comps just ended, so I suspect more Kagglers will be joining soon and the leaderboard will become crowded soon. But we shouldn't worry too much about public LB. I suspect there will be big surprises on the private leaderboard so everyone should focus on maximizing their local CV which is 33,000 images as opposed to maximizing public LB which is 3,000 images.
", "rawMarkdown": "> Someone put grease on the competition ladder so I’m slipping down it fast.\n\nTwo image Kaggle comps just ended, so I suspect more Kagglers will be joining soon and the leaderboard will become crowded soon. But we shouldn't worry too much about public LB. I suspect there will be big surprises on the private leaderboard so everyone should focus on maximizing their local CV which is 33,000 images as opposed to maximizing public LB which is 3,000 images.", "votes": 11, "replies": [ { "id": 941449, "postDate": "2020-07-23T08:13:11.903Z", "content": "As Chris said above that other competitions just got over and some of the top performers there will now start focusing on the SIIM-ISIC Melanoma competition!
\n\nEspecially poteman who just won (#1 final Private LB) the PANDA challenge a few hours ago today (23-Jul) and is already at SIIM-ISIC LB #12 (0.9627)!\nCongrats @poteman !
\n\nAnother participant SeuTao came 6th in PANDA, now just started SIIM Melanoma competition and with just 2 submissions already reached #195 (0.9549)!
\n\nSimilarly Sanchit Singh came 9th in PANDA, already reached #206 (0.9543),\nNirjhar Roy came 17th in PANDA, already reached #136 (0.9565), etc
", "rawMarkdown": "As Chris said above that other competitions just got over and some of the top performers there will now start focusing on the SIIM-ISIC Melanoma competition!\n\nEspecially **poteman** who just [won (#1 final Private LB)](https://www.kaggle.com/c/prostate-cancer-grade-assessment/leaderboard) the PANDA challenge a few hours ago today (23-Jul) and is already at SIIM-ISIC LB **#12 (0.9627)**!\nCongrats @poteman !\n\nAnother participant SeuTao came 6th in PANDA, now just started SIIM Melanoma competition and with just 2 submissions already reached #195 (0.9549)!\n\nSimilarly Sanchit Singh came 9th in PANDA, already reached #206 (0.9543),\nNirjhar Roy came 17th in PANDA, already reached #136 (0.9565), etc" } ] }, { "id": 941236, "postDate": "2020-07-23T06:25:14.547Z", "content": "@zzy990106 yesterday someone holding top rank in public LB posted his kernel which was giving LB score of 95.65 and helped many cheaters to come up on top.\nThankfully, It was deleted after 1-2 hours.
", "rawMarkdown": "@zzy990106 yesterday someone holding top rank in public LB posted his kernel which was giving LB score of 95.65 and helped many cheaters to come up on top.\nThankfully, It was deleted after 1-2 hours.", "votes": 2 }, { "id": 941356, "postDate": "2020-07-23T07:23:20.120Z", "rawMarkdown": "", "isDeleted": true }, { "id": 941400, "postDate": "2020-07-23T07:54:41.830Z", "content": "It's unfair. Many teams are still using this kernel's output and it becomes 'Private Code Sharing'.
", "rawMarkdown": "It's unfair. Many teams are still using this kernel's output and it becomes 'Private Code Sharing'.", "votes": 2 }, { "id": 941785, "postDate": "2020-07-23T12:13:37.487Z", "rawMarkdown": "", "isDeleted": true }, { "id": 942416, "postDate": "2020-07-23T18:21:50.940Z", "rawMarkdown": "", "isDeleted": true }, { "id": 956911, "postDate": "2020-08-03T22:37:32.810Z", "content": "What did they do in the notebook that \"greased\" the leaderboard? I would think that if you are publically sharing a notebook and people use what they see in that notebook, that's not cheating. Unless what they were doing in the notebook was some sort of banned technique.
", "rawMarkdown": "What did they do in the notebook that \"greased\" the leaderboard? I would think that if you are publically sharing a notebook and people use what they see in that notebook, that's not cheating. Unless what they were doing in the notebook was some sort of banned technique." } ] }, { "id": 974344, "postDate": "2020-08-17T23:09:33.047Z", "content": "Ohh I wished I would have seen them before… Thanks!
", "rawMarkdown": "Ohh I wished I would have seen them before... Thanks! ", "votes": 1 }, { "id": 963105, "postDate": "2020-08-08T17:07:23.073Z", "content": "Thank you so much for all the works! But I found myself really confused by the TTA, is it normal for the auc on the oof to be higher without TTA? It's really strange and the difference can up to 7 percent for one fold (0.93 for auc without TTA, and 0.86 with TTA).
", "rawMarkdown": "Thank you so much for all the works! But I found myself really confused by the TTA, is it normal for the auc on the oof to be higher without TTA? It's really strange and the difference can up to 7 percent for one fold (0.93 for auc without TTA, and 0.86 with TTA).", "votes": 1, "replies": [ { "id": 964519, "postDate": "2020-08-10T00:59:54.770Z", "content": "You are right, TTA will (nearly) always increase AUC.
\nIn my popular notebook here \"with TTA\" and \"without TTA\" is misleading. The AUC for \"without TTA\" is the maximum AUC achieved during the epoch and the AUC for \"with TTA\" is TTA applied to the minimum val loss during epoch. So you see \"with TTA\" and \"without TTA\" are not being applied to the same model.
\nFor a true comparison, change the following code in cell 13:
\n sv = tf.keras.callbacks.ModelCheckpoint(\n 'fold-%i.h5'%fold, monitor='val_loss', verbose=0, save_best_only=True,\n save_weights_only=True, mode='min', save_freq='epoch')\nReplace monitor='val_loss' with monitor='val_auc' and change mode='min' to mode='max'. Then \"with TTA\" will (nearly) always be greater than \"without TTA\". Because both will be applied to the same model.
Appreciate!
", "rawMarkdown": "Appreciate!", "votes": 1 }, { "id": 968620, "postDate": "2020-08-13T06:28:36.217Z", "content": "I had a model that TTA decreases AUC. the drop is about 0.02, not as big as yours. still wondering why it happened for this particular model but not others.
", "rawMarkdown": "I had a model that TTA decreases AUC. the drop is about 0.02, not as big as yours. still wondering why it happened for this particular model but not others. ", "votes": 1 }, { "id": 968627, "postDate": "2020-08-13T06:33:00.517Z", "content": "Try my suggestion above. If you use the same save model weights, I rarely see without TTA beat with TTA. (As you see in my comment above, my public notebook doesn't use the same model weights for with and without comparison).
", "rawMarkdown": "Try my suggestion above. If you use the same save model weights, I rarely see without TTA beat with TTA. (As you see in my comment above, my public notebook doesn't use the same model weights for with and without comparison)." } ] }, { "id": 960021, "postDate": "2020-08-06T04:57:47.990Z", "content": "@cdeotte Thank you so much for so many dataset contributions in this competition. May I ask: how do you \"center crop\" each original Kaggle image? I found in some images the interested region is very small, while in some other they are nearly as large as the image itself.
", "rawMarkdown": "@cdeotte Thank you so much for so many dataset contributions in this competition. May I ask: how do you \"center crop\" each original Kaggle image? I found in some images the interested region is very small, while in some other they are nearly as large as the image itself.", "votes": 1, "replies": [ { "id": 960025, "postDate": "2020-08-06T05:03:17.917Z", "content": "Here's the code
\n\n img = cv2.imread(PATH+files[k])\n w = img.shape[1]; h = img.shape[0]; s = min(w,h)\n w2 = (w-s)//2; h2 = (h-s)//2\n img = img[h2:h-h2,w2:w-w2,:]\n img = cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA)\nI find the largest possible centered square and then resize that to the desired resolution.
", "rawMarkdown": "Here's the code\n\n img = cv2.imread(PATH+files[k])\n w = img.shape[1]; h = img.shape[0]; s = min(w,h)\n w2 = (w-s)//2; h2 = (h-s)//2\n img = img[h2:h-h2,w2:w-w2,:]\n img = cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA)\n\nI find the largest possible centered square and then resize that to the desired resolution.", "votes": 6 }, { "id": 960066, "postDate": "2020-08-06T05:47:52.183Z", "content": "@cdeotte is there any good way to do a smart center crop similar to this technique except using opencv mask? or there's too much variation between images to get a good smart crop prior to resizing?
", "rawMarkdown": "@cdeotte is there any good way to do a smart center crop similar to [this technique](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171745) except using opencv mask? or there's too much variation between images to get a good smart crop prior to resizing?" }, { "id": 960086, "postDate": "2020-08-06T06:03:00.730Z", "content": "I think most skin lesions are in the middle of the image because the purpose of the photo is the lesion, so the photographer centered the lesion.
\n\nI don't think there are many (if any) images where the lesion if off center enough to justify cropping somewhere else than center.
", "rawMarkdown": "I think most skin lesions are in the middle of the image because the purpose of the photo is the lesion, so the photographer centered the lesion.\n\nI don't think there are many (if any) images where the lesion if off center enough to justify cropping somewhere else than center." } ] }, { "id": 952309, "postDate": "2020-07-30T19:58:24.807Z", "content": "This is a life saver! I was planning on using augmented malignant images to counter the high sampling bias but this will just do the job for me and my model. Thanks!
", "rawMarkdown": "This is a life saver! I was planning on using augmented malignant images to counter the high sampling bias but this will just do the job for me and my model. Thanks!", "votes": 1, "replies": [ { "id": 952323, "postDate": "2020-07-30T20:14:46.537Z", "content": "Wonderful. If you add these to your training pipeline they will get data augmentation and these malignant images will be unique every epoch.
", "rawMarkdown": "Wonderful. If you add these to your training pipeline they will get data augmentation and these malignant images will be unique every epoch.", "votes": 1 } ] }, { "id": 951901, "postDate": "2020-07-30T13:42:40.343Z", "content": "Is there a quick way to convert the TfRecords back to JPG to easily use them in PyTorch?
", "rawMarkdown": "Is there a quick way to convert the TfRecords back to JPG to easily use them in PyTorch?", "votes": 1, "replies": [ { "id": 951911, "postDate": "2020-07-30T13:49:21.710Z", "content": "All my datasets have a corresponding JPEG dataset for PyTorch, 2020 here and 2019 2018 2017 here. (And the ISIC-archive malignant JPEGs are in a folder in this discussion post's TFRecord dataset).
\n\nIf you want to upsample using JPEG in PyTorch, then you just need to have your dataloader output malignant images more frequently. (For example, make a list of all malignant from data and output them with twice probability).
\n\nAlso if you download my malignant TFRecord dataset (from this discussion post), i have 3 CSV files which list all the names of the malignant images. The first and third CSV filenames come from my JPEG datasets for 2020 comp data and 2019 2018 2017 comp data respectively. The second CSV are the ISIC-archive scrapped JPEGs. There is a folder included in my malignant TFRecords dataset that contains the JPEGs for these ISIC-archive scrapped images since they do not appear in my 2020 nor 2019 2018 2017 datasets.
", "rawMarkdown": "All my datasets have a corresponding JPEG dataset for PyTorch, 2020 [here][1] and 2019 2018 2017 [here][2]. (And the ISIC-archive malignant JPEGs are in a folder in this discussion post's TFRecord dataset).\n\nIf you want to upsample using JPEG in PyTorch, then you just need to have your dataloader output malignant images more frequently. (For example, make a list of all malignant from data and output them with twice probability).\n\nAlso if you download my malignant TFRecord dataset (from this discussion post), i have 3 CSV files which list all the names of the malignant images. The first and third CSV filenames come from my JPEG datasets for 2020 comp data and 2019 2018 2017 comp data respectively. The second CSV are the ISIC-archive scrapped JPEGs. There is a folder included in my malignant TFRecords dataset that contains the JPEGs for these ISIC-archive scrapped images since they do not appear in my 2020 nor 2019 2018 2017 datasets.\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164092\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910", "votes": 1 }, { "id": 952385, "postDate": "2020-07-30T21:38:52.737Z", "content": "Thank you for the answer Chris! It makes sense, and I was able to start using the data on my model.
", "rawMarkdown": "Thank you for the answer Chris! It makes sense, and I was able to start using the data on my model.", "votes": 1 } ] }, { "id": 949792, "postDate": "2020-07-29T00:04:14.977Z", "content": "Thanks a lot for sharing these datasets!
", "rawMarkdown": "Thanks a lot for sharing these datasets!", "votes": 1 }, { "id": 945924, "postDate": "2020-07-26T08:12:34.713Z", "content": "Nice article
", "rawMarkdown": "Nice article", "votes": 1 }, { "id": 945860, "postDate": "2020-07-26T07:27:01.193Z", "content": "Nice post. It helps me a lot !
", "rawMarkdown": "Nice post. It helps me a lot !", "votes": 1 }, { "id": 944957, "postDate": "2020-07-25T13:27:47.073Z", "content": "Hi @cdeotte thanks once again for this awesome work, I had upsampling of malignant images on my todo list, and you just saved me a lot of time.\nI was going to add these datasets to my kernels and I notice that you have v1 and v2 datasets, what is the difference between them?
", "rawMarkdown": "Hi @cdeotte thanks once again for this awesome work, I had upsampling of malignant images on my todo list, and you just saved me a lot of time.\nI was going to add these datasets to my kernels and I notice that you have v1 and v2 datasets, what is the difference between them?", "votes": 1, "replies": [ { "id": 944989, "postDate": "2020-07-25T13:39:29.270Z", "content": "Version 2 has everything that version 1 has and more. So use version 2 which are the links in this discussion post.
\n\nVersion 1 is an old dataset i was experimenting with weeks ago. Version 1 is only the malignant from 2019 (both new portion and 2018 2017 mixed together). Version 1 does not have duplicates removed nor is version 1 triple stratified. Version 1 coincides with my version 1 of my other TFRecords. But right now all my TFRecords (this year 2020 comp data and 2019 comp data are all version 2 with triple stratified, leak-free, with duplicates removed).
", "rawMarkdown": "Version 2 has everything that version 1 has and more. So use version 2 which are the links in this discussion post.\n\nVersion 1 is an old dataset i was experimenting with weeks ago. Version 1 is **only** the malignant from 2019 (both new portion and 2018 2017 mixed together). Version 1 does not have duplicates removed nor is version 1 triple stratified. Version 1 coincides with my version 1 of my other TFRecords. But right now all my TFRecords (this year 2020 comp data and 2019 comp data are all version 2 with triple stratified, leak-free, with duplicates removed).", "votes": 1 } ] }, { "id": 944955, "postDate": "2020-07-25T13:26:25.973Z", "content": "Hey @cdeotte , Great post!
\n\nIt is hard for me to compile all at once. \nTo sum up you are using:\n- ISIC 2017-2019, \n - SIIM data,\n - ISIC archive data,\nThose datasets are having non empty intersection -> this is what is hard to graps.
\n\nThere are 584 malignant in the 2020 dataset (are they present in any other dataset? )\nThere are 2000< malignant images in the ISIC archive (they are also available in the 2017-2019 data)
\n\nQuestion: \nIs the ISIC archive superset of the ISIC 2017-2019? Are there any images in the 2017-2019 datasets that are not present in the ISIC ARCHIVE?
", "rawMarkdown": "Hey @cdeotte , Great post! \n\nIt is hard for me to compile all at once. \nTo sum up you are using:\n- ISIC 2017-2019, \n - SIIM data,\n - ISIC archive data,\nThose datasets are having non empty intersection -> this is what is hard to graps. \n\nThere are 584 malignant in the 2020 dataset (are they present in any other dataset? )\nThere are 2000< malignant images in the ISIC archive (they are also available in the 2017-2019 data) \n\nQuestion: \nIs the ISIC archive superset of the ISIC 2017-2019? Are there any images in the 2017-2019 datasets that are not present in the ISIC ARCHIVE? \n\n\n", "votes": 1, "replies": [ { "id": 945058, "postDate": "2020-07-25T14:39:50.400Z", "content": "I have published 3 Kaggle datasets (and each dataset has 7 sizes and TFRecord and JPEG versions).\n* This year's 2020 comp data called 2020 data\n* Last year's 2019 comp data (which includes 2018 2017 data) called 2019 data\n* This discussion post Malignant dataset called Malignant data
After removing duplicates, this year's 2020 comp data has 32542 benign and 584 malignant. Those 584 malignant are in both my 2020 data and Malignant data TFRecords 0-14.
After removing duplicates, last year's 2019 comp data has 20809 benign and 4522 malignant. We can further break this dataset into \"new 2019 portion\" which has 9556 benign and 2858 malignant. And \"old portion\" (which is 2018 2017 data) which has 11253 benign and 1664 malignant. These 1664 are in both my 2019 data and Malignant data TFRecords 30,32,...,56,58. And of the \"new portion\" 2858, half are good ones (1185 determined by RAPIDS TSNE) are in both my 2019 data and Malignant data TFRecords 31,33,...,57,59.
Lastly the ISIC archive has 2285 malignant images. Of these 2285, only 580 are not in 2020, 2019, 2018, 2017. I downloaded them and put them in my Malignant data TFRecords 15-29. The remaining 1705 malignant on the ISIC archive are mostly the \"old portion\" 2018 2017 comp data. The \"new portion\" from 2019 are from BCN20000 dataset which is not in the ISIC archive (explained here)
Great, thank you a lot, got much better understanding!
", "rawMarkdown": "Great, thank you a lot, got much better understanding! \n\n", "votes": 1 }, { "id": 949709, "postDate": "2020-07-28T20:34:58.287Z", "content": "Hi @cdeotte thanks for your awesome work, I am comfused about the number of 2018 2017 data. It should be 1664 cause 2019 comp data has 4522 malignant. \n>>TFRecords` even numbered 30, 32, …, 56, 58 The next 15 even numbered TFRecords contain the malignant images from 2018 2017 comp data. There are 1627 malignant images.
\n\nThat means you discard 37 samples or it is just a mistake of number?
", "rawMarkdown": "Hi @cdeotte thanks for your awesome work, I am comfused about the number of 2018 2017 data. It should be 1664 cause 2019 comp data has 4522 malignant. \n>>TFRecords` even numbered 30, 32, …, 56, 58 The next 15 even numbered TFRecords contain the malignant images from 2018 2017 comp data. There are 1627 malignant images. \n\nThat means you discard 37 samples or it is just a mistake of number?" }, { "id": 949730, "postDate": "2020-07-28T21:25:32.593Z", "content": "@hujingyuan Great question and great attention to detail. I discarded 37 malignant images from 2018 2017 that looked weird and I also removed 13 duplicates. So TFRecords 30,32,..., 56,68 have 1614 malignant images. (which is 1664 minus 37 minus 13. Some of my previous posts had wrong numbers).
I plotted all of 2019 malignant (both new portion and 2018 2017 portion) using t-SNE. There was a huge island of malignant images all by themselves and I removed them. (They do not appear in the plot below). There were 1710 bad (weird) malignant images. 37 were from 2018 2017 and 1673 were from new portion 2019.
\n\nIn the plot below left image is this year's 2020 comp data with orange benign and blue malignant. In the plot below right, all orange is 2020 comp data (both malignant and benign) and the blue data is the good malignant from 2019 data.
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F4e63cca72b49eb873a77a657d4c96e00%2FScreen%20Shot%202020-07-28%20at%202.13.46%20PM.png?generation=1595971007173137&alt=media\")
In the plot below left image is this year's 2020 comp data with orange benign and blue malignant. In the plot below right, all orange is 2020 comp data (both malignant and benign) and the blue data is the new 580 malignant that I scrapped from the ISIC-archive website. (We can see that they are high quality in distribution).\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F25025522df2f9141a50ef273df9b9391%2FScreen%20Shot%202020-07-28%20at%202.13.57%20PM.png?generation=1595971428975768&alt=media\")
I explain RAPIDS cuML t-SNE here and here
", "rawMarkdown": "@hujingyuan Great question and great attention to detail. I discarded 37 malignant images from 2018 2017 that looked weird and I also removed 13 duplicates. So TFRecords `30,32,..., 56,68` have 1614 malignant images. (which is 1664 minus 37 minus 13. Some of my previous posts had wrong numbers).\n\nI plotted all of 2019 malignant (both new portion and 2018 2017 portion) using t-SNE. There was a huge island of malignant images all by themselves and I removed them. (They do not appear in the plot below). There were 1710 bad (weird) malignant images. 37 were from 2018 2017 and 1673 were from new portion 2019.\n\nIn the plot below left image is this year's 2020 comp data with orange benign and blue malignant. In the plot below right, all orange is 2020 comp data (both malignant and benign) and the blue data is the good malignant from 2019 data.\n\n\n\nIn the plot below left image is this year's 2020 comp data with orange benign and blue malignant. In the plot below right, all orange is 2020 comp data (both malignant and benign) and the blue data is the new 580 malignant that I scrapped from the ISIC-archive website. (We can see that they are high quality in distribution).\n\n\nI explain RAPIDS cuML t-SNE [here][1] and [here][2]\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167551\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168028", "votes": 2 }, { "id": 949849, "postDate": "2020-07-29T02:34:42.333Z", "content": "@cdeotte I got it, thank you again for the explanation and the great work.👍
", "rawMarkdown": "@cdeotte I got it, thank you again for the explanation and the great work.👍 ", "votes": 1 } ] }, { "id": 942917, "postDate": "2020-07-24T04:16:41.623Z", "content": "train_malig_1.csv is the 584 2020 malignants\ntrain_malig_2.csv is the 580 new malignants
\n\nWhat is train_malig_3.csv? I would have thought it was the 1185 good new portion 2019 malignants, plus the 2017, 2018 1672 malignants, but that sums to 2857 but the CSV only says 2812
", "rawMarkdown": "train_malig_1.csv is the 584 2020 malignants\ntrain_malig_2.csv is the 580 new malignants \n\nWhat is train_malig_3.csv? I would have thought it was the 1185 good new portion 2019 malignants, plus the 2017, 2018 1672 malignants, but that sums to 2857 but the CSV only says 2812", "votes": 1, "replies": [ { "id": 942956, "postDate": "2020-07-24T04:38:31.553Z", "content": "train3.csv is both the malignant good portion of 2019 which is 1185 images and the malignant of 2018 & 2017 which is 1627 images (for total of 2812). (You can distinguish the good new portion of 2019 as width==1024 and height==1024 while the 2018 + 2017 does not).
The good new portion 2019 are in TFRecords odd numbered 31, 33, ..., 57, 59. And 2018 + 2017 malignant are in TFRecords even numbered 30, 32, ..., 56, 58.
(In your post you wrote 1672 when it is 1627)
", "rawMarkdown": "`train3.csv` is both the malignant good portion of 2019 which is 1185 images and the malignant of 2018 & 2017 which is 1627 images (for total of 2812). (You can distinguish the good new portion of 2019 as `width==1024 and height==1024` while the 2018 + 2017 does not).\n\nThe good new portion 2019 are in TFRecords odd numbered `31, 33, ..., 57, 59`. And 2018 + 2017 malignant are in TFRecords even numbered `30, 32, ..., 56, 58`.\n\n(In your post you wrote 1672 when it is 1627)" }, { "id": 943036, "postDate": "2020-07-24T05:34:59.933Z", "content": "Thanks Chris, I am just trying to reconcile the numbers you have above.
\n\nIn TFRecords even numbered 30, 32, …, 56, 58 you wrote \"There are 1672 malignant images.\"\nIn TFRecords odd numbered 31, 33, …, 57, 59 you wrote \"there are 1185 good new portion 2019 malignant images among the total 2858 new portion 2019 malignant\"
\n\nThanks again, this is very helpful.
", "rawMarkdown": "Thanks Chris, I am just trying to reconcile the numbers you have above. \n\nIn *TFRecords even numbered 30, 32, …, 56, 58* you wrote \"There are 1672 malignant images.\"\nIn *TFRecords odd numbered 31, 33, …, 57, 59* you wrote \"there are 1185 good new portion 2019 malignant images among the total 2858 new portion 2019 malignant\"\n\nThanks again, this is very helpful. " }, { "id": 943047, "postDate": "2020-07-24T05:47:37.203Z", "content": "@brianfeeny Once again, you wrote the wrong number. It is 1627 not 1672
", "rawMarkdown": "@brianfeeny Once again, you wrote the wrong number. It is 1627 not 1672" }, { "id": 943052, "postDate": "2020-07-24T05:51:42.377Z", "content": "Chris, what I am trying to say is that YOU wrote 1672 in your post in this thread. I was quoting what you wrote. Do you see?
", "rawMarkdown": "Chris, what I am trying to say is that YOU wrote 1672 in your post in this thread. I was quoting what you wrote. Do you see?", "votes": 1 }, { "id": 943061, "postDate": "2020-07-24T06:01:26.067Z", "content": "Ah, yes my mistake. Thanks, i fixed it.
", "rawMarkdown": "Ah, yes my mistake. Thanks, i fixed it.", "votes": 1 }, { "id": 955791, "postDate": "2020-08-03T00:39:37.050Z", "content": "@cdeotte so are there or are there not duplicates in train_malig_3.csv?\nlen(train_malig_3.csv) = 2812
It's supposed to equal 1614 (1664 with 13 dups removed and 37 weird removed (2017, 2018)) + 1185 (new malignant 2019) = 2799 .
\n\nSo it's supposed to equal 2799 yet it equals 2812.
\n\nYet the length of that CSV equals 2812, which is 13 more. Does that mean the 13 duplicates are in the CSV and if so are they somehow marked?
", "rawMarkdown": "@cdeotte so are there or are there not duplicates in `train_malig_3.csv`?\n`len(train_malig_3.csv)` = 2812 \n\nIt's supposed to equal 1614 (1664 with 13 dups removed and 37 weird removed (2017, 2018)) + 1185 (new malignant 2019) = **2799** . \n\nSo it's supposed to equal 2799 yet it equals 2812.\n\nYet the length of that CSV equals 2812, which is 13 more. Does that mean the 13 duplicates are in the CSV and if so are they somehow marked?\n", "votes": 1 }, { "id": 955801, "postDate": "2020-08-03T00:57:45.897Z", "content": "Yes 13 rows in the CSV are duplicates marked with tfrecord = -1 and they are not included in any of the TFRecords.
If I want to only add new Malignant,I should let IDX=list(range(15,29)).\nRight?
", "rawMarkdown": "If I want to only add new Malignant,I should let IDX=list(range(15,29)).\nRight?", "votes": 1, "replies": [ { "id": 942487, "postDate": "2020-07-23T19:09:58.457Z", "content": "Yes exactly. And if you want to add a double dose, you can do IDX=list(range(15,29)) + list(range(15,29)). Note that we are using data augmentation, so each copy will look different to our model. So you can also consider adding more copies of existing malignant.
make correction: IDX=list(range(15,30))
", "rawMarkdown": "make correction: IDX=list(range(15,30))" }, { "id": 943641, "postDate": "2020-07-24T13:38:35.710Z", "content": "Yes, good catch. It needs to be (15,30).
@cdeotte is this data can be used with neural network with meta data??
", "rawMarkdown": "@cdeotte is this data can be used with neural network with meta data??", "votes": 1, "replies": [ { "id": 942111, "postDate": "2020-07-23T15:31:20.543Z", "content": "Yes. All these TFRecords have the same meta features as my other TFRecords, i.e. age, gender, site, etc. But note that data from 2019, 2018, 2017 does not have patient_id. There is also CSV files in the dataset listing the images and their meta features.
@cdeotte But i see that this data didnt onehot encoded the variables.So is it good to feed it into neural network??
", "rawMarkdown": "@cdeotte But i see that this data didnt onehot encoded the variables.So is it good to feed it into neural network??" }, { "id": 942148, "postDate": "2020-07-23T15:57:57.183Z", "content": "You have three options. You can one hot encode, or use embeddings, or use numeric.
\n\ndef read_labeled_tfrecord(example):\ntfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'target' : tf.io.FixedLenFeature([], tf.int64)\n} \nexample = tf.io.parse_single_example(example, tfrec_format)\nOHE = tf.one_hot(example['anatom_site_general_challenge']+1, 7)\nreturn (example['image'],OHE), example['target']\ndef read_labeled_tfrecord(example):\ntfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'target' : tf.io.FixedLenFeature([], tf.int64)\n} \nexample = tf.io.parse_single_example(example, tfrec_format)\nCAT = example['anatom_site_general_challenge']+1\nreturn (example['image'],CAT), example['target']\nand then in your NN
\n\ndef build_model():\n inp1 = tf.keras.layers.Input(shape=(*IMAGE_SIZE,3))\n inp2 = tf.keras.layers.Input(shape=(1,))\n x2 = tf.keras.Embedding(7, 3, input_length=1)(inp2)\n x2 = tf.keras.Reshape(target_shape=(3, ))(x2)\n # BUILD MODEL HERE\n x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n model = tf.keras.models.Model(inputs=[inp1,inp2], outputs=x)\n return model\nNUM = (example['anatom_site_general_challenge']+1)/3.0 - 1.5\nreturn (example['image'],NUM), example['target']\nand then in your NN
\n\ndef build_model():\n inp1 = tf.keras.layers.Input(shape=(*IMAGE_SIZE,3))\n inp2 = tf.keras.layers.Input(shape=(1,))\n # BUILD MODEL HERE\n x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n model = tf.keras.models.Model(inputs=[inp1,inp2], outputs=x)\n return model\n@cdeotte Has this increased your cv score and lb score??\nAnd btw have you tried pseudo labelling?? Coz for me its not working 😑
", "rawMarkdown": "@cdeotte Has this increased your cv score and lb score??\nAnd btw have you tried pseudo labelling?? Coz for me its not working 😑 ", "votes": 1 }, { "id": 942232, "postDate": "2020-07-23T16:45:19.870Z", "content": "I have not added meta features to my CNN models yet.
", "rawMarkdown": "I have not added meta features to my CNN models yet. " }, { "id": 943675, "postDate": "2020-07-24T14:10:02.543Z", "content": "@cdeotte Btw you said that by using triple stratifed data there wont be any leaks in training.And also you said that we have some stable CV and LB score by using only competition KFold splits data as validation.But in my case my CV LB has a std of like from +/- 0.015 to +/- 0.095. Is there any problem in my strategies??
", "rawMarkdown": "@cdeotte Btw you said that by using triple stratifed data there wont be any leaks in training.And also you said that we have some stable CV and LB score by using only competition KFold splits data as validation.But in my case my CV LB has a std of like from +/- 0.015 to +/- 0.095. Is there any problem in my strategies??" }, { "id": 943708, "postDate": "2020-07-24T14:27:17.860Z", "content": "Here are the important points\n* you want LB to increase whenever your CV increases\n* you want the same validation data for all of your experiments\n* you want validation score to have low standard deviation when repeating same experiment
\n\nLet me explain the 3rd point. If you run the same experiment over and over, you will get a validation score each time. You want these scores to have low standard deviation. Because if your standard deviation is 0.01. Then you can only know that an experiment is better than previous experiment if the new score is 2 times greater than the standard deviation, i.e. 0.02.
\n\n(When standard deviation is large, it requires that you discover a larger magic to pass evaluation. When standard deviation is small, little magics can be discovered).
\n\nThere are 2 ways to decrease standard deviation. \n* adjust your model, learning schedule, augmentation, regularization, loss, etc\n* for each experiment run the same fold 5 times and take the average as your validation score.
\n\nIdeally, you would like to do the former because then you only need to run 1 fold. But it is harder to find a model with low standard deviation val score. But it is possible. My offline model has low standard deviation.
", "rawMarkdown": "Here are the important points\n* you want LB to increase whenever your CV increases\n* you want the same validation data for **all** of your experiments\n* you want validation score to have low standard deviation when repeating same experiment\n\nLet me explain the 3rd point. If you run the same experiment over and over, you will get a validation score each time. You want these scores to have low standard deviation. Because if your standard deviation is 0.01. Then you can only know that an experiment is better than previous experiment if the new score is 2 times greater than the standard deviation, i.e. 0.02. \n\n(When standard deviation is large, it requires that you discover a larger magic to pass evaluation. When standard deviation is small, little magics can be discovered).\n\nThere are 2 ways to decrease standard deviation. \n* adjust your model, learning schedule, augmentation, regularization, loss, etc\n* for each experiment run the same fold 5 times and take the average as your validation score.\n\nIdeally, you would like to do the former because then you only need to run 1 fold. But it is harder to find a model with low standard deviation val score. But it is possible. My offline model has low standard deviation.\n", "votes": 1 }, { "id": 945180, "postDate": "2020-07-25T16:13:39.977Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 941931, "postDate": "2020-07-23T13:55:45.073Z", "content": "Thank you very much for sharing. I have a question: you say there are 4k new images but on the dataset page it says that the jpeg folder only contains around 600 files, did I miss something?
", "rawMarkdown": "Thank you very much for sharing. I have a question: you say there are 4k new images but on the dataset page it says that the jpeg folder only contains around 600 files, did I miss something?", "votes": 1, "replies": [ { "id": 941945, "postDate": "2020-07-23T14:03:46.753Z", "content": "The dataset contains 4k malignant images. Only 580 are new meaning that those 580 are not contained in previously published 2020 data, nor 2019 data, nor 2018, nor 2017. The other 3420 images are the malignant images from 2020, 2019, 2018, 2017.
\n\nIt is helpful to have the 2020, 2019, 2018, 2017 malignant in their own TFRecords, so if we want three times as many malignant as the 2020 comp data, then we can add the full 2020 comp data, and then add 2 copies of the 2020 malignant data. (We can also experiment with adding one or more copies of 2019, 2018, 2017, or the 580 new malignant images).
", "rawMarkdown": "The dataset contains 4k malignant images. Only 580 are **new** meaning that those 580 are not contained in previously published 2020 data, nor 2019 data, nor 2018, nor 2017. The other 3420 images are the malignant images from 2020, 2019, 2018, 2017.\n\nIt is helpful to have the 2020, 2019, 2018, 2017 malignant in their own TFRecords, so if we want three times as many malignant as the 2020 comp data, then we can add the **full** 2020 comp data, and then add 2 copies of the 2020 malignant data. (We can also experiment with adding one or more copies of 2019, 2018, 2017, or the 580 new malignant images).", "votes": 1 }, { "id": 941960, "postDate": "2020-07-23T14:15:04.407Z", "content": "Oh okay, got it. Thank you again!
", "rawMarkdown": "Oh okay, got it. Thank you again!", "votes": 1 } ] }, { "id": 941599, "postDate": "2020-07-23T10:13:50.210Z", "content": "Thanks for sharing @cdeotte. The two competitions panda and alaska have just finished, LB will definitely become crowded soon :))
", "rawMarkdown": "Thanks for sharing @cdeotte. The two competitions panda and alaska have just finished, LB will definitely become crowded soon :))", "votes": 1 }, { "id": 941242, "postDate": "2020-07-23T06:33:22.930Z", "content": "wow，amazing，you can always surprise people.👍
", "rawMarkdown": "wow，amazing，you can always surprise people.👍 ", "votes": 1 }, { "id": 955589, "postDate": "2020-08-02T18:14:56.360Z", "content": "Playing a bit with only adding target=1 of old data to my training data and am observing very weird behavior that CV increases and LB decreases quite significantly. Anyone experiencing something similar?
", "rawMarkdown": "Playing a bit with only adding target=1 of old data to my training data and am observing very weird behavior that CV increases and LB decreases quite significantly. Anyone experiencing something similar?", "votes": 2, "replies": [ { "id": 955617, "postDate": "2020-08-02T18:30:08.593Z", "content": "Are you only adding the target=1 old data to the training folds and not the validation folds? In that case, it is weird that your CV increases and LB decreases.
Yes, only training data. CV is going up quite significantly, and LB down also significantly. Trying to figure out what's going on...\nMaybe duplicates...need to check that, but I doubt that can be the issue. At least then LB should not get worse.
", "rawMarkdown": "Yes, only training data. CV is going up quite significantly, and LB down also significantly. Trying to figure out what's going on...\nMaybe duplicates...need to check that, but I doubt that can be the issue. At least then LB should not get worse." }, { "id": 955665, "postDate": "2020-08-02T19:57:18.923Z", "content": "If you are using my data, there are no duplicates. (All have been removed). The only thing you need to be careful about is when adding more 2020 comp data malignant. (You need to add the correct extra 2020 malignant to the correct train folds). When using target=1 old data there will never be leaks using 2020 validation data.
I am not removing dups currently, just move them to same folds.
", "rawMarkdown": "I am not removing dups currently, just move them to same folds." }, { "id": 956625, "postDate": "2020-08-03T16:29:41.310Z", "content": "@philippsinger Exactly the same here. I've got 0.957 up from 0.947 with single fold after adding positives to training set only. But LB is... <0.9
", "rawMarkdown": "@philippsinger Exactly the same here. I've got 0.957 up from 0.947 with single fold after adding positives to training set only. But LB is... <0.9", "votes": 1 }, { "id": 956684, "postDate": "2020-08-03T17:28:50.903Z", "content": "Yep, also <0.9 here
", "rawMarkdown": "Yep, also <0.9 here", "votes": 2 }, { "id": 957042, "postDate": "2020-08-04T02:41:24.310Z", "content": "@cateek Your CV is great. Any tips?
", "rawMarkdown": "@cateek Your CV is great. Any tips? " }, { "id": 957979, "postDate": "2020-08-04T17:03:31.863Z", "content": "@virajbagal It's nothing fancy, mixnet_xl with some variations, @cdeotte 's dataset, 384 resolution, some standard augments. No TTA, no pseudo.
", "rawMarkdown": "@virajbagal It's nothing fancy, mixnet_xl with some variations, @cdeotte 's dataset, 384 resolution, some standard augments. No TTA, no pseudo.", "votes": 1 }, { "id": 958066, "postDate": "2020-08-04T18:35:06.583Z", "content": "@cateek So you are saying your own CV is .957 yet your LB is <.9? I would try to tighten that up, that is such a large spread.
", "rawMarkdown": "@cateek So you are saying your own CV is .957 yet your LB is <.9? I would try to tighten that up, that is such a large spread." }, { "id": 958104, "postDate": "2020-08-04T19:12:53.350Z", "content": "@brianfeeny Exactly. Several models with local score ranging from 0.91 to 0.95 with different LB scores and no visible correlation
", "rawMarkdown": "@brianfeeny Exactly. Several models with local score ranging from 0.91 to 0.95 with different LB scores and no visible correlation\n" }, { "id": 958687, "postDate": "2020-08-05T05:40:07.847Z", "content": "@cateek Is ur CV stable (around 0.95) with change of seed?
", "rawMarkdown": "@cateek Is ur CV stable (around 0.95) with change of seed?" }, { "id": 958801, "postDate": "2020-08-05T06:35:25.437Z", "content": "@virajbagal Haven't tested that
", "rawMarkdown": "@virajbagal Haven't tested that " }, { "id": 959082, "postDate": "2020-08-05T10:22:25.050Z", "content": "I'm also getting worse LB scores when I append the malignant data to my train set. I think simply appending malignant images is not a good idea, because it will change the training distribution. Shouldn't you add the extra data first, then do the data stratification afterwards? That way you make sure you have similar distributions in your train and val sets...
", "rawMarkdown": "I'm also getting worse LB scores when I append the malignant data to my train set. I think simply appending malignant images is not a good idea, because it will change the training distribution. Shouldn't you add the extra data first, then do the data stratification afterwards? That way you make sure you have similar distributions in your train and val sets...", "votes": 1 }, { "id": 959283, "postDate": "2020-08-05T13:21:07.277Z", "content": "Yes it changes the training distribution, that is the whole point. You should not add any extra data to the validation.
", "rawMarkdown": "Yes it changes the training distribution, that is the whole point. You should not add any extra data to the validation." }, { "id": 960047, "postDate": "2020-08-06T05:26:48.813Z", "content": "It probably partly learns to recognize the dataset source (ISIC18, ISIC19 or ISIC20) instead of malign/benign if you only add the positive ones.
", "rawMarkdown": "It probably partly learns to recognize the dataset source (ISIC18, ISIC19 or ISIC20) instead of malign/benign if you only add the positive ones. " }, { "id": 960256, "postDate": "2020-08-06T09:15:20.743Z", "content": "@group16 do you recommend adding both benign and malignant when upsampling?
", "rawMarkdown": "@group16 do you recommend adding both benign and malignant when upsampling?" }, { "id": 960268, "postDate": "2020-08-06T09:23:00.563Z", "content": "I would recommend that especially for the external data sources that are very different from the data provided here in Kaggle. One way to determine this is by studying Chris his excellent t-SNE plots. The 2019 ISIC data is vastly different from this dataset, so I guess (I am of course not entirely certain) that by only adding the positive samples to your dataset, you confuse your model (as it will learn to detect ISIC 2019 data which corresponds to the positive label in that case)
", "rawMarkdown": "I would recommend that especially for the external data sources that are very different from the data provided here in Kaggle. One way to determine this is by studying Chris his excellent t-SNE plots. The 2019 ISIC data is vastly different from this dataset, so I guess (I am of course not entirely certain) that by only adding the positive samples to your dataset, you confuse your model (as it will learn to detect ISIC 2019 data which corresponds to the positive label in that case)", "votes": 1 }, { "id": 960277, "postDate": "2020-08-06T09:27:26.627Z", "content": "@group16 well if you add all of the old train data, then you are back to square 1, highly unbalanced data. Do you think a combination of adding the old data, both benign and malignant, and then upsampling the malignant is the way?
", "rawMarkdown": "@group16 well if you add all of the old train data, then you are back to square 1, highly unbalanced data. Do you think a combination of adding the old data, both benign and malignant, and then upsampling the malignant is the way?" }, { "id": 960292, "postDate": "2020-08-06T09:35:23.977Z", "content": "@group16 Possible, but why is CV going up?
", "rawMarkdown": "@group16 Possible, but why is CV going up?" }, { "id": 960300, "postDate": "2020-08-06T09:43:31.907Z", "content": "@philippsinger That is indeed a good question and a rather weird observation. Surely you validate only on ISIC20 images? And potential duplicates are already removed? If some of the positive ISIC2019 images also occur in the ISIC20 dataset (there are a few duplicates), then that can increase the CV significantly as there are not a lot of positives...
\n\n@brianfeeny You are completely correct, the data will be unbalanced again. There are other ways to combat this:\n* Make sure the proportion malignant/benign is greater in the external data you add (this reduces the imbalance).\n* I never like not utilizing information (called undersampling, which is what not including benign external images does). I don't like over-sampling either, as it will introduce many biases (e.g. SMOTE assumes linear relations between your data points) and will only generate from the already learned train distribution, which your network already handles. As such, I prefer using different objectives that better handle imbalanced data. Focal loss, label smoothing or giving more weight to positive samples are all much more (theoretically) satisfying solutions than under- or oversampling.
", "rawMarkdown": "@philippsinger That is indeed a good question and a rather weird observation. Surely you validate only on ISIC20 images? And potential duplicates are already removed? If some of the positive ISIC2019 images also occur in the ISIC20 dataset (there are a few duplicates), then that can increase the CV significantly as there are not a lot of positives...\n\n@brianfeeny You are completely correct, the data will be unbalanced again. There are other ways to combat this:\n* Make sure the proportion malignant/benign is greater in the external data you add (this reduces the imbalance).\n* I never like **not** utilizing information (called undersampling, which is what not including benign external images does). I don't like over-sampling either, as it will introduce many biases (e.g. SMOTE assumes linear relations between your data points) and will only generate from the already learned train distribution, which your network already handles. As such, I prefer using different objectives that better handle imbalanced data. **Focal loss**, **label smoothing** or **giving more weight to positive samples** are all much more (theoretically) satisfying solutions than under- or oversampling.", "votes": 2 }, { "id": 960397, "postDate": "2020-08-06T11:07:51.983Z", "content": "@group16 thanks for the tips. I tried FocalLoss, but it just did so poorly, I must have tried 3 different implementations of it. I just swapped out BCE for FocalLoss and AUC went into the toilet. I would revisit it, but not sure what I could be doing wrong. I also tried to do class weights. I was using sklearns Compute Class Weight. The weights it suggested were rather extreme, but I tried this back when just using the default hugely unbalanced data. Perhaps I should forgo the sklearn class and just try to give malignant like 2x or 3x.
", "rawMarkdown": "@group16 thanks for the tips. I tried FocalLoss, but it just did so poorly, I must have tried 3 different implementations of it. I just swapped out BCE for FocalLoss and AUC went into the toilet. I would revisit it, but not sure what I could be doing wrong. I also tried to do class weights. I was using sklearns [Compute Class Weight](https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_class_weight.html). The weights it suggested were rather extreme, but I tried this back when just using the default hugely unbalanced data. Perhaps I should forgo the sklearn class and just try to give malignant like 2x or 3x.", "votes": 1 }, { "id": 960564, "postDate": "2020-08-06T14:06:34.440Z", "content": "\n\n\nwell if you add all of the old train data, then you are back to square 1, highly unbalanced data.
\n
No, the data from 2019 has something like 20% malignant whereas this 2020 comp data is 2% malignant. So if you add all of 2019 external both benign and malignant, then your overall malignant is 11%. Then if you additionally add 1 copy of 2019 malignant, you're approximately at 20%, with 2 copies 30%, with 3 copies 40% etc.
\n\nSo the original data is \"highly unbalanced\" at 2%, but 11%, 20%, 30%, and 40% are not \"highly unbalanced\".
", "rawMarkdown": "> well if you add all of the old train data, then you are back to square 1, highly unbalanced data.\n \nNo, the data from 2019 has something like 20% malignant whereas this 2020 comp data is 2% malignant. So if you add all of 2019 external both benign and malignant, then your overall malignant is 11%. Then if you additionally add 1 copy of 2019 malignant, you're approximately at 20%, with 2 copies 30%, with 3 copies 40% etc.\n\nSo the original data is \"highly unbalanced\" at 2%, but 11%, 20%, 30%, and 40% are not \"highly unbalanced\".", "votes": 1 }, { "id": 960618, "postDate": "2020-08-06T14:41:02.780Z", "content": "How can training the unbalanced data with different levels of balance (by adjust sample rate of each class in data generator) affect AUC performance since AUC only deals with ranking over a population?
", "rawMarkdown": "How can training the unbalanced data with different levels of balance (by adjust sample rate of each class in data generator) affect AUC performance since AUC only deals with ranking over a population? ", "votes": 1 }, { "id": 960623, "postDate": "2020-08-06T14:44:40.020Z", "rawMarkdown": "", "isDeleted": true }, { "id": 960665, "postDate": "2020-08-06T15:26:01.583Z", "content": "@philippsinger I got the same when adding 2019 malignant data to training folds but not validations folds. CV went up to 0.923 (256x256 effnet-b0) but LB went down to 0.8840.
\n\nIt is a bit reassuring to see it is not (solely) due to my messing up...
", "rawMarkdown": "@philippsinger I got the same when adding 2019 malignant data to training folds but not validations folds. CV went up to 0.923 (256x256 effnet-b0) but LB went down to 0.8840.\n\nIt is a bit reassuring to see it is not (solely) due to my messing up...", "votes": 1 }, { "id": 961226, "postDate": "2020-08-07T02:50:22.390Z", "content": "If people are having trouble making upsampling work with my malignant dataset work, I suggest adding 2018 2017 benign simultaneously with adding extra malignant from 2019, 2018, 2017 or ISIC-archive. Otherwise your model doesn't learn malignant it learns to identify external versus non-external.
", "rawMarkdown": "If people are having trouble making upsampling work with my malignant dataset work, I suggest adding 2018 2017 benign simultaneously with adding extra malignant from 2019, 2018, 2017 or ISIC-archive. Otherwise your model doesn't learn malignant it learns to identify external versus non-external.", "votes": 2 } ] }, { "id": 946280, "postDate": "2020-07-26T13:42:31.490Z", "content": "Nice one @cdeotte, cleared many of my preprocessing issues.!
", "rawMarkdown": "Nice one @cdeotte, cleared many of my preprocessing issues.! ", "votes": 2 }, { "id": 942657, "postDate": "2020-07-23T23:00:05.030Z", "content": "Did you get all of the malignant images or are there more?\nI am considering on working further on this topic after the contest ends.\nEspecially I am wondering why there are no black skin samples in our dataset. Did you find any?
", "rawMarkdown": "Did you get all of the malignant images or are there more?\nI am considering on working further on this topic after the contest ends.\nEspecially I am wondering why there are no black skin samples in our dataset. Did you find any?", "votes": 2, "replies": [ { "id": 942662, "postDate": "2020-07-23T23:07:24.410Z", "content": "I got all malignant images from ISIC-archive (their website database), this year's 2020 comp, and previous year's 2019, 2018, 2017 comps.
\n\nNote that there are some 2019 malignant that I do not include in this dataset. But all the 2019 malignant are of course in my 2019 dataset.
", "rawMarkdown": "I got all malignant images from ISIC-archive (their website database), this year's 2020 comp, and previous year's 2019, 2018, 2017 comps.\n\nNote that there are some 2019 malignant that I do not include in this dataset. But all the 2019 malignant are of course in my 2019 dataset.", "votes": 1 }, { "id": 943172, "postDate": "2020-07-24T07:22:43.140Z", "content": "If your old, have white skin and blue eyes you can almost count on at least one bad mole in your lifetime. I also looked at a bunch of images and saw no black skin samples. I had plans to add \"skin color\" as a meta data feature by looking (automating the look), as references I saw indicated my old white skin is 33 times more likely to have melanoma than black skin.
\n\nStill have it on my to do to visually look at more images - I am not getting much improvement in results with the current meta data. I added some average color stuff, but the black ring of the microscope seems to make simple color data of low value.
", "rawMarkdown": "If your old, have white skin and blue eyes you can almost count on at least one bad mole in your lifetime. I also looked at a bunch of images and saw no black skin samples. I had plans to add \"skin color\" as a meta data feature by looking (automating the look), as references I saw indicated my old white skin is 33 times more likely to have melanoma than black skin. \n\nStill have it on my to do to visually look at more images - I am not getting much improvement in results with the current meta data. I added some average color stuff, but the black ring of the microscope seems to make simple color data of low value." } ] }, { "id": 952441, "postDate": "2020-07-30T23:09:53.280Z", "content": "@cdeotte Thanks for sharing this dataset ! Just to be clarified, there are 580 new malignant data. So by 4000 you mean 7 set of the same 580 data in different sizes. Is that right ?
", "rawMarkdown": "@cdeotte Thanks for sharing this dataset ! Just to be clarified, there are 580 new malignant data. So by 4000 you mean 7 set of the same 580 data in different sizes. Is that right ?", "replies": [ { "id": 952454, "postDate": "2020-07-30T23:48:13.850Z", "content": "By 4000, i mean 3420 repeat and 580 new. The repeat are the malignant from 2020 2019 2018 2017 comp data. The new are malignant scrapped from ISIC-archive website.
\n\nAll the new are in TFRecords numbered 15-29. And they are in the included folder named JPEG as jpegs. The list of new is in the CSV train_malig_2.csv.
thankyou for your great work!
\n\nMay I ask that why the data dont have patient_id?
", "rawMarkdown": "thankyou for your great work!\n\nMay I ask that why the data dont have patient_id?", "replies": [ { "id": 944563, "postDate": "2020-07-25T07:30:55.693Z", "content": "All old data before the year 2020 doesn't have patient_id.
Thanks @synked
", "rawMarkdown": "Thanks @synked" }, { "id": 944608, "postDate": "2020-07-25T07:54:58.560Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 944622, "postDate": "2020-07-25T08:05:39.503Z", "content": "okey, thanks
", "rawMarkdown": "okey, thanks" }, { "id": 944624, "postDate": "2020-07-25T08:06:12.860Z", "rawMarkdown": "", "isDeleted": true }, { "id": 944625, "postDate": "2020-07-25T08:08:10.457Z", "content": "me too! @x2t2cx
", "rawMarkdown": "me too! @x2t2cx " } ] }, { "id": 943183, "postDate": "2020-07-24T07:30:09.350Z", "content": "I have joined my first Kaggle competition.I am facing lots of difficulties in understanding the dataset and the folds. Which dataset to use? How to execute the folds? What to do with the tfrecord,metadata,external data posted by @cdeotte ?Basically I am stuck upto the basics of solving a problem. Can anyone help me in moving forward in these issue? What should I do ? Thanks for stopping by.
", "rawMarkdown": "I have joined my first Kaggle competition.I am facing lots of difficulties in understanding the dataset and the folds. Which dataset to use? How to execute the folds? What to do with the tfrecord,metadata,external data posted by @cdeotte ?Basically I am stuck upto the basics of solving a problem. Can anyone help me in moving forward in these issue? What should I do ? Thanks for stopping by.", "replies": [ { "id": 943193, "postDate": "2020-07-24T07:39:26.997Z", "content": "Rony - Did you check the 0.9454 score public kernel Triple Stratified KFold with TFRecords
\n\nBy the way, if this is your first competition, you can use the official Topic New to Machine Learning or Kaggle?
", "rawMarkdown": "Rony - Did you check the 0.9454 score public kernel [Triple Stratified KFold with TFRecords](https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords)\n\nBy the way, if this is your first competition, you can use the official Topic [New to Machine Learning or Kaggle?](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/154297)", "votes": 1 } ] }, { "id": 942349, "postDate": "2020-07-23T17:51:15.687Z", "content": "I get an error adding tfrecords, I dont understand - any clues?
\n\nGCS_PATH3 = KaggleDatasets().get_gcs_path('malignant-v2-256x256')\nMALIGNANT = [GCS_PATH3 + '/train%.2i*.tfrec'%x for x in IDX]\nprint(MALIGNANT)\nfiles_train += tf.io.gfile.glob(MALIGNANT)
\n\n--->files_train += tf.io.gfile.glob(MALIGNANT)\n np.random.shuffle(files_train)
\n\nUFuncTypeError: ufunc 'add' did not contain a loop with signature matching types (dtype(' dtype('
", "rawMarkdown": "I get an error adding tfrecords, I dont understand - any clues?\n\nGCS_PATH3 = KaggleDatasets().get_gcs_path('malignant-v2-256x256')\nMALIGNANT = [GCS_PATH3 + '/train%.2i*.tfrec'%x for x in IDX]\nprint(MALIGNANT)\nfiles_train += tf.io.gfile.glob(MALIGNANT)\n\n--->files_train += tf.io.gfile.glob(MALIGNANT)\n np.random.shuffle(files_train)\n\nUFuncTypeError: ufunc 'add' did not contain a loop with signature matching types (dtype('", "replies": [ { "id": 942395, "postDate": "2020-07-23T18:10:29.600Z", "content": "I assume that the preceeding code is
\n\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))\nYou have turned files_train into NumPy array. Change the above line to
files_train = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\nRemove the conversion to NumPy will fix the problem. We don't need it. We shuffle everything afterward anyways with
\n\nnp.random.shuffle(files_train)\nyou are right - thx
", "rawMarkdown": "you are right - thx", "votes": 1 } ] }, { "id": 959116, "postDate": "2020-08-05T11:13:46.863Z", "rawMarkdown": "", "isDeleted": true }, { "id": 957422, "postDate": "2020-08-04T09:41:18.427Z", "rawMarkdown": "", "votes": 3, "isDeleted": true, "replies": [ { "id": 957742, "postDate": "2020-08-04T14:27:06.530Z", "content": "There are 3 sources. (1) The 2020 malignant come from this year's Kaggle comp data (2) The 2019 (2018 2017) malignant come from 2019 comp data here. (3) The 580 unique malignant come from the ISIC-archive here.
\n\nTo download images from the archive, you select malignant in the filter on the left, then click Select All on the Page for Download, and then click Download as ZIP. Then go to each of the 29 pages of malignant images and download the 29 ZIP files. You will get 2285 malignant images from ISIC-archive, then use RAPIDS cuML kNN (notebook here) to remove the images that already exist in 2020, 2019, 2018, 2017 data. You will be left with 580 new malignant from ISIC-archive. (Lastly I center square crop resize them).
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F0200ddaf056aa8555b197bf63bc5581d%2FScreen%20Shot%202020-08-04%20at%207.23.03%20AM.png?generation=1596550998217598&alt=media\")
I'm not sure I understand your question. We cannot know for sure whether something will help private or not. But if an ensemble helps both CV and LB, then it will most likely help private too.
\n\n(To verify CV, you must produce OOF predictions for both the sub and tab using the same KFold, then ensemble the OOF and check new val score).
Thanks for the links and datasets. I wasn't aware of these images. I'll try them out.
", "rawMarkdown": "Thanks for the links and datasets. I wasn't aware of these images. I'll try them out.", "votes": 1 }, { "id": 944263, "postDate": "2020-07-25T01:19:18.483Z", "content": "@synked I'm looking at your datasets. Can you tell me where these datasets came from?
", "rawMarkdown": "@synked I'm looking at your datasets. Can you tell me where these datasets came from?" }, { "id": 944414, "postDate": "2020-07-25T05:20:31.980Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 991211, "postDate": "2020-08-30T08:13:04.780Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing", "votes": 3 }, { "id": 973018, "postDate": "2020-08-17T03:42:46.860Z", "content": "Thanks
", "rawMarkdown": "Thanks", "votes": 1 }, { "id": 963847, "postDate": "2020-08-09T11:02:06.517Z", "content": "Awesome! Thanks for sharing!
", "rawMarkdown": "Awesome! Thanks for sharing!", "votes": 1 }, { "id": 943312, "postDate": "2020-07-24T09:32:04.580Z", "content": "Thanks for sharing @cdeotte
", "rawMarkdown": "Thanks for sharing @cdeotte ", "votes": 1 }, { "id": 942681, "postDate": "2020-07-23T23:44:14.980Z", "content": "Thanks for sharing @cdeotte
", "rawMarkdown": "Thanks for sharing @cdeotte ", "votes": 1 }, { "id": 941495, "postDate": "2020-07-23T08:47:42.793Z", "content": "thanks @cdeotte
", "rawMarkdown": "thanks @cdeotte ", "votes": 1 }, { "id": 940735, "postDate": "2020-07-23T05:30:57.433Z", "content": "Thanks @cdeotte
", "rawMarkdown": "Thanks @cdeotte ", "votes": 1 } ], "comments": [ { "id": 944193, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-24T22:59:36.237000", "content": "UPDATE: I posted a starter notebook here showing how to use the new upsample Malignant Kaggle datasets. Enjoy!
\n\n
UPDATE: I ran a careful experiment to see if upsample improves AUC for kNN. Using only 2020 data, it appears that upsample can indeed increase AUC for kNN by at least 0.003!. The plot is below and the experiment is described here
\nUPDATE2: I ran another experiment to see if upsample improves AUC for CNN. Using only 2020 data, it appears that upsample can indeed increase AUC for CNN. I will post the CNN experiment code after the comp finishes.
\n
Just to make sure this wasn't luck, i ran the experiment 100 more times with 100 different seeds. The average AUC increase is 0.00207 with STD 0.00122. We are 99.99999% confident that upsample increases AUC!
\n
P-values and CI's are not the same thing though :)
", "votes": 2, "replies": [] }, { "id": 962957, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-08T15:12:41.443000", "content": "There was typo in my post. STD = 0.0012 not 0.0017 (as we can see in the plot).
\nFor a 95% one-sided hypothesis test, you need z score > 1.65 and for a 95% two-sided hypothesis test you need z score > 1.96. This is a one sided hypothesis test and we have z score = (0.00207 / 0.00122) * sqrt(100) = 17.0, therefore the result is conclusive.
UPDATE: We need to use a t test instead of a z test. The t test statistic is 35.09 using online calculator here so p<0.0001 and we are 99.99999% confident
", "votes": 2, "replies": [] }, { "id": 963038, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-08T16:08:23.567000", "content": "@cdeotte why don't you say upsampling helps KNN? Your experiment may mislead people into thinking that upsampling helps CNN. It may be true, but your experiment is not confirming it by any mean.
", "votes": 1, "replies": [] }, { "id": 964521, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-10T01:07:08.840000", "content": "UPDATE: I originally computed my test statistic wrong in my paired t-test and have updated it above. (I forgot to multiply by the square root of n). Using the correct test statistic (from online calculator here), we reject the null hypothesis with p = < .00001.
We are 99.99999% confident that upsampling increases AUC for kNN in this comp!
\n\nwhy don't you say upsampling helps KNN? Your experiment may mislead people into thinking that upsampling helps CNN
\n
Good point @cpmpml .CNN and kNN may react differently to upsample. I have conducted another paired t-test for CNN, and found that upsample increases AUC for CNN with p<0.00001. We are 99.99999% confident that upsampling increases AUC for CNN.
\nI will update my other posts to say this and I will post the code for the CNN experiments after the comp finishes.
", "votes": 1, "replies": [] }, { "id": 965236, "author_name": "Tolga", "author_url": "", "post_date": "2020-08-10T13:40:09.237000", "content": "Thank you for your rigorous explanation! I'm glad this discussion turned out to be productive (at least for me).
\n\nI'm removing my first comment as the numbers are different now.
", "votes": 1, "replies": [] } ] }, { "id": 940612, "author_name": "Bruce Young", "author_url": "", "post_date": "2020-07-23T03:50:02.920000", "content": "Great thanks Chris. \nTwo obvious questions...\nHave you seen improvements using this new data?\nIs this external data ok to use under the competition rules?
\n\nSomeone put grease on the competition ladder so I’m slipping down it fast.
\n\nThanks for your continuing hard work and sharing.
", "votes": 3, "replies": [ { "id": 940618, "author_name": "Aman Arora", "author_url": "", "post_date": "2020-07-23T03:53:12.823000", "content": "\n\n\nSomeone put grease on the competition ladder so I’m slipping down it fast.
\n
Bruce, if I am honest, I am pretty sure I am overfitting the Public Leaderboard and so are the others that have recently surpassed you on the competition ladder.
\n\nI have been following your position since the start, and I believe you have nothing to worry about, I think with the Private Leaderboard, you will have a strong position unless you are overfitting too :)
", "votes": 2, "replies": [] }, { "id": 940633, "author_name": "Leon", "author_url": "", "post_date": "2020-07-23T04:05:36.530000", "content": "The changes on the leaderboard are strange. Some people with very few submissions also got 0.96+.
", "votes": 6, "replies": [] }, { "id": 940648, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T04:19:02.167000", "content": "\n\n\nIs this external data ok to use under the competition rules?
\n
Yes. We are already using 75% of this data. These are the malignant images from 2020, 2019, 2018, 2017 comp data. Most notebooks use these already. But now with these records, we can be more flexible. For example, we can use 2020 benign and malignant. And then 2019 2018 2017 malignant only. Or we can use 2020, 2019, 2018, 2017 plus another dose of malignant from 2020, 2019, 2018, 2017. Since we are using data augmentation, the second copy of malignant will look different to our models and help train them.
\n\nThere are only 580 new images in this dataset never seen before. This data is from the competition host's website and is ok to use. The host has confirmed it's use in multiple other discussions, example here.
", "votes": 1, "replies": [] }, { "id": 940655, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T04:23:22.683000", "content": "\n\n\nSomeone put grease on the competition ladder so I’m slipping down it fast.
\n
Two image Kaggle comps just ended, so I suspect more Kagglers will be joining soon and the leaderboard will become crowded soon. But we shouldn't worry too much about public LB. I suspect there will be big surprises on the private leaderboard so everyone should focus on maximizing their local CV which is 33,000 images as opposed to maximizing public LB which is 3,000 images.
", "votes": 11, "replies": [ { "id": 941449, "author_name": "Sirish Somanchi", "author_url": "", "post_date": "2020-07-23T08:13:11.903000", "content": "As Chris said above that other competitions just got over and some of the top performers there will now start focusing on the SIIM-ISIC Melanoma competition!
\n\nEspecially poteman who just won (#1 final Private LB) the PANDA challenge a few hours ago today (23-Jul) and is already at SIIM-ISIC LB #12 (0.9627)!\nCongrats @poteman !
\n\nAnother participant SeuTao came 6th in PANDA, now just started SIIM Melanoma competition and with just 2 submissions already reached #195 (0.9549)!
\n\nSimilarly Sanchit Singh came 9th in PANDA, already reached #206 (0.9543),\nNirjhar Roy came 17th in PANDA, already reached #136 (0.9565), etc
", "votes": 0, "replies": [] } ] }, { "id": 941236, "author_name": "Prateek Mishra", "author_url": "", "post_date": "2020-07-23T06:25:14.547000", "content": "@zzy990106 yesterday someone holding top rank in public LB posted his kernel which was giving LB score of 95.65 and helped many cheaters to come up on top.\nThankfully, It was deleted after 1-2 hours.
", "votes": 2, "replies": [] }, { "id": 941356, "author_name": "", "author_url": "", "post_date": "2020-07-23T07:23:20.120000", "content": "", "votes": 0, "replies": [] }, { "id": 941400, "author_name": "Leon", "author_url": "", "post_date": "2020-07-23T07:54:41.830000", "content": "It's unfair. Many teams are still using this kernel's output and it becomes 'Private Code Sharing'.
", "votes": 2, "replies": [] }, { "id": 941785, "author_name": "", "author_url": "", "post_date": "2020-07-23T12:13:37.487000", "content": "", "votes": 0, "replies": [] }, { "id": 942416, "author_name": "", "author_url": "", "post_date": "2020-07-23T18:21:50.940000", "content": "", "votes": 0, "replies": [] }, { "id": 956911, "author_name": "Signal", "author_url": "", "post_date": "2020-08-03T22:37:32.810000", "content": "What did they do in the notebook that \"greased\" the leaderboard? I would think that if you are publically sharing a notebook and people use what they see in that notebook, that's not cheating. Unless what they were doing in the notebook was some sort of banned technique.
", "votes": 0, "replies": [] } ] }, { "id": 974344, "author_name": "Bea", "author_url": "", "post_date": "2020-08-17T23:09:33.047000", "content": "Ohh I wished I would have seen them before… Thanks!
", "votes": 1, "replies": [] }, { "id": 963105, "author_name": "ElaborateJia", "author_url": "", "post_date": "2020-08-08T17:07:23.073000", "content": "Thank you so much for all the works! But I found myself really confused by the TTA, is it normal for the auc on the oof to be higher without TTA? It's really strange and the difference can up to 7 percent for one fold (0.93 for auc without TTA, and 0.86 with TTA).
", "votes": 1, "replies": [ { "id": 964519, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-10T00:59:54.770000", "content": "You are right, TTA will (nearly) always increase AUC.
\nIn my popular notebook here \"with TTA\" and \"without TTA\" is misleading. The AUC for \"without TTA\" is the maximum AUC achieved during the epoch and the AUC for \"with TTA\" is TTA applied to the minimum val loss during epoch. So you see \"with TTA\" and \"without TTA\" are not being applied to the same model.
\nFor a true comparison, change the following code in cell 13:
\n sv = tf.keras.callbacks.ModelCheckpoint(\n 'fold-%i.h5'%fold, monitor='val_loss', verbose=0, save_best_only=True,\n save_weights_only=True, mode='min', save_freq='epoch')\nReplace monitor='val_loss' with monitor='val_auc' and change mode='min' to mode='max'. Then \"with TTA\" will (nearly) always be greater than \"without TTA\". Because both will be applied to the same model.
Appreciate!
", "votes": 1, "replies": [] }, { "id": 968620, "author_name": "yimacs", "author_url": "", "post_date": "2020-08-13T06:28:36.217000", "content": "I had a model that TTA decreases AUC. the drop is about 0.02, not as big as yours. still wondering why it happened for this particular model but not others.
", "votes": 1, "replies": [] }, { "id": 968627, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-13T06:33:00.517000", "content": "Try my suggestion above. If you use the same save model weights, I rarely see without TTA beat with TTA. (As you see in my comment above, my public notebook doesn't use the same model weights for with and without comparison).
", "votes": 0, "replies": [] } ] }, { "id": 960021, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-08-06T04:57:47.990000", "content": "@cdeotte Thank you so much for so many dataset contributions in this competition. May I ask: how do you \"center crop\" each original Kaggle image? I found in some images the interested region is very small, while in some other they are nearly as large as the image itself.
", "votes": 1, "replies": [ { "id": 960025, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-06T05:03:17.917000", "content": "Here's the code
\n\n img = cv2.imread(PATH+files[k])\n w = img.shape[1]; h = img.shape[0]; s = min(w,h)\n w2 = (w-s)//2; h2 = (h-s)//2\n img = img[h2:h-h2,w2:w-w2,:]\n img = cv2.resize(img,(DIM,DIM),interpolation = cv2.INTER_AREA)\nI find the largest possible centered square and then resize that to the desired resolution.
", "votes": 6, "replies": [] }, { "id": 960066, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-06T05:47:52.183000", "content": "@cdeotte is there any good way to do a smart center crop similar to this technique except using opencv mask? or there's too much variation between images to get a good smart crop prior to resizing?
", "votes": 0, "replies": [] }, { "id": 960086, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-06T06:03:00.730000", "content": "I think most skin lesions are in the middle of the image because the purpose of the photo is the lesion, so the photographer centered the lesion.
\n\nI don't think there are many (if any) images where the lesion if off center enough to justify cropping somewhere else than center.
", "votes": 0, "replies": [] } ] }, { "id": 952309, "author_name": "Aman Sharma", "author_url": "", "post_date": "2020-07-30T19:58:24.807000", "content": "This is a life saver! I was planning on using augmented malignant images to counter the high sampling bias but this will just do the job for me and my model. Thanks!
", "votes": 1, "replies": [ { "id": 952323, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-30T20:14:46.537000", "content": "Wonderful. If you add these to your training pipeline they will get data augmentation and these malignant images will be unique every epoch.
", "votes": 1, "replies": [] } ] }, { "id": 951901, "author_name": "Gianluca Rossi", "author_url": "", "post_date": "2020-07-30T13:42:40.343000", "content": "Is there a quick way to convert the TfRecords back to JPG to easily use them in PyTorch?
", "votes": 1, "replies": [ { "id": 951911, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-30T13:49:21.710000", "content": "All my datasets have a corresponding JPEG dataset for PyTorch, 2020 here and 2019 2018 2017 here. (And the ISIC-archive malignant JPEGs are in a folder in this discussion post's TFRecord dataset).
\n\nIf you want to upsample using JPEG in PyTorch, then you just need to have your dataloader output malignant images more frequently. (For example, make a list of all malignant from data and output them with twice probability).
\n\nAlso if you download my malignant TFRecord dataset (from this discussion post), i have 3 CSV files which list all the names of the malignant images. The first and third CSV filenames come from my JPEG datasets for 2020 comp data and 2019 2018 2017 comp data respectively. The second CSV are the ISIC-archive scrapped JPEGs. There is a folder included in my malignant TFRecords dataset that contains the JPEGs for these ISIC-archive scrapped images since they do not appear in my 2020 nor 2019 2018 2017 datasets.
", "votes": 1, "replies": [] }, { "id": 952385, "author_name": "Gianluca Rossi", "author_url": "", "post_date": "2020-07-30T21:38:52.737000", "content": "Thank you for the answer Chris! It makes sense, and I was able to start using the data on my model.
", "votes": 1, "replies": [] } ] }, { "id": 949792, "author_name": "Salman Ibne Eunus", "author_url": "", "post_date": "2020-07-29T00:04:14.977000", "content": "Thanks a lot for sharing these datasets!
", "votes": 1, "replies": [] }, { "id": 945924, "author_name": "Rajesh Pachiyappan", "author_url": "", "post_date": "2020-07-26T08:12:34.713000", "content": "Nice article
", "votes": 1, "replies": [] }, { "id": 945860, "author_name": "Saran Pannasuriyaporn", "author_url": "", "post_date": "2020-07-26T07:27:01.193000", "content": "Nice post. It helps me a lot !
", "votes": 1, "replies": [] }, { "id": 944957, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2020-07-25T13:27:47.073000", "content": "Hi @cdeotte thanks once again for this awesome work, I had upsampling of malignant images on my todo list, and you just saved me a lot of time.\nI was going to add these datasets to my kernels and I notice that you have v1 and v2 datasets, what is the difference between them?
", "votes": 1, "replies": [ { "id": 944989, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-25T13:39:29.270000", "content": "Version 2 has everything that version 1 has and more. So use version 2 which are the links in this discussion post.
\n\nVersion 1 is an old dataset i was experimenting with weeks ago. Version 1 is only the malignant from 2019 (both new portion and 2018 2017 mixed together). Version 1 does not have duplicates removed nor is version 1 triple stratified. Version 1 coincides with my version 1 of my other TFRecords. But right now all my TFRecords (this year 2020 comp data and 2019 comp data are all version 2 with triple stratified, leak-free, with duplicates removed).
", "votes": 1, "replies": [] } ] }, { "id": 944955, "author_name": "Janek Idziak", "author_url": "", "post_date": "2020-07-25T13:26:25.973000", "content": "Hey @cdeotte , Great post!
\n\nIt is hard for me to compile all at once. \nTo sum up you are using:\n- ISIC 2017-2019, \n - SIIM data,\n - ISIC archive data,\nThose datasets are having non empty intersection -> this is what is hard to graps.
\n\nThere are 584 malignant in the 2020 dataset (are they present in any other dataset? )\nThere are 2000< malignant images in the ISIC archive (they are also available in the 2017-2019 data)
\n\nQuestion: \nIs the ISIC archive superset of the ISIC 2017-2019? Are there any images in the 2017-2019 datasets that are not present in the ISIC ARCHIVE?
", "votes": 1, "replies": [ { "id": 945058, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-25T14:39:50.400000", "content": "I have published 3 Kaggle datasets (and each dataset has 7 sizes and TFRecord and JPEG versions).\n* This year's 2020 comp data called 2020 data\n* Last year's 2019 comp data (which includes 2018 2017 data) called 2019 data\n* This discussion post Malignant dataset called Malignant data
After removing duplicates, this year's 2020 comp data has 32542 benign and 584 malignant. Those 584 malignant are in both my 2020 data and Malignant data TFRecords 0-14.
After removing duplicates, last year's 2019 comp data has 20809 benign and 4522 malignant. We can further break this dataset into \"new 2019 portion\" which has 9556 benign and 2858 malignant. And \"old portion\" (which is 2018 2017 data) which has 11253 benign and 1664 malignant. These 1664 are in both my 2019 data and Malignant data TFRecords 30,32,...,56,58. And of the \"new portion\" 2858, half are good ones (1185 determined by RAPIDS TSNE) are in both my 2019 data and Malignant data TFRecords 31,33,...,57,59.
Lastly the ISIC archive has 2285 malignant images. Of these 2285, only 580 are not in 2020, 2019, 2018, 2017. I downloaded them and put them in my Malignant data TFRecords 15-29. The remaining 1705 malignant on the ISIC archive are mostly the \"old portion\" 2018 2017 comp data. The \"new portion\" from 2019 are from BCN20000 dataset which is not in the ISIC archive (explained here)
Great, thank you a lot, got much better understanding!
", "votes": 1, "replies": [] }, { "id": 949709, "author_name": "HuJingyuan", "author_url": "", "post_date": "2020-07-28T20:34:58.287000", "content": "Hi @cdeotte thanks for your awesome work, I am comfused about the number of 2018 2017 data. It should be 1664 cause 2019 comp data has 4522 malignant. \n>>TFRecords` even numbered 30, 32, …, 56, 58 The next 15 even numbered TFRecords contain the malignant images from 2018 2017 comp data. There are 1627 malignant images.
\n\nThat means you discard 37 samples or it is just a mistake of number?
", "votes": 0, "replies": [] }, { "id": 949730, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-28T21:25:32.593000", "content": "@hujingyuan Great question and great attention to detail. I discarded 37 malignant images from 2018 2017 that looked weird and I also removed 13 duplicates. So TFRecords 30,32,..., 56,68 have 1614 malignant images. (which is 1664 minus 37 minus 13. Some of my previous posts had wrong numbers).
I plotted all of 2019 malignant (both new portion and 2018 2017 portion) using t-SNE. There was a huge island of malignant images all by themselves and I removed them. (They do not appear in the plot below). There were 1710 bad (weird) malignant images. 37 were from 2018 2017 and 1673 were from new portion 2019.
\n\nIn the plot below left image is this year's 2020 comp data with orange benign and blue malignant. In the plot below right, all orange is 2020 comp data (both malignant and benign) and the blue data is the good malignant from 2019 data.
\n\n
In the plot below left image is this year's 2020 comp data with orange benign and blue malignant. In the plot below right, all orange is 2020 comp data (both malignant and benign) and the blue data is the new 580 malignant that I scrapped from the ISIC-archive website. (We can see that they are high quality in distribution).\n
I explain RAPIDS cuML t-SNE here and here
", "votes": 2, "replies": [] }, { "id": 949849, "author_name": "HuJingyuan", "author_url": "", "post_date": "2020-07-29T02:34:42.333000", "content": "@cdeotte I got it, thank you again for the explanation and the great work.👍
", "votes": 1, "replies": [] } ] }, { "id": 942917, "author_name": "Signal", "author_url": "", "post_date": "2020-07-24T04:16:41.623000", "content": "train_malig_1.csv is the 584 2020 malignants\ntrain_malig_2.csv is the 580 new malignants
\n\nWhat is train_malig_3.csv? I would have thought it was the 1185 good new portion 2019 malignants, plus the 2017, 2018 1672 malignants, but that sums to 2857 but the CSV only says 2812
", "votes": 1, "replies": [ { "id": 942956, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-24T04:38:31.553000", "content": "train3.csv is both the malignant good portion of 2019 which is 1185 images and the malignant of 2018 & 2017 which is 1627 images (for total of 2812). (You can distinguish the good new portion of 2019 as width==1024 and height==1024 while the 2018 + 2017 does not).
The good new portion 2019 are in TFRecords odd numbered 31, 33, ..., 57, 59. And 2018 + 2017 malignant are in TFRecords even numbered 30, 32, ..., 56, 58.
(In your post you wrote 1672 when it is 1627)
", "votes": 0, "replies": [] }, { "id": 943036, "author_name": "Signal", "author_url": "", "post_date": "2020-07-24T05:34:59.933000", "content": "Thanks Chris, I am just trying to reconcile the numbers you have above.
\n\nIn TFRecords even numbered 30, 32, …, 56, 58 you wrote \"There are 1672 malignant images.\"\nIn TFRecords odd numbered 31, 33, …, 57, 59 you wrote \"there are 1185 good new portion 2019 malignant images among the total 2858 new portion 2019 malignant\"
\n\nThanks again, this is very helpful.
", "votes": 0, "replies": [] }, { "id": 943047, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-24T05:47:37.203000", "content": "@brianfeeny Once again, you wrote the wrong number. It is 1627 not 1672
", "votes": 0, "replies": [] }, { "id": 943052, "author_name": "Signal", "author_url": "", "post_date": "2020-07-24T05:51:42.377000", "content": "Chris, what I am trying to say is that YOU wrote 1672 in your post in this thread. I was quoting what you wrote. Do you see?
", "votes": 1, "replies": [] }, { "id": 943061, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-24T06:01:26.067000", "content": "Ah, yes my mistake. Thanks, i fixed it.
", "votes": 1, "replies": [] }, { "id": 955791, "author_name": "Signal", "author_url": "", "post_date": "2020-08-03T00:39:37.050000", "content": "@cdeotte so are there or are there not duplicates in train_malig_3.csv?\nlen(train_malig_3.csv) = 2812
It's supposed to equal 1614 (1664 with 13 dups removed and 37 weird removed (2017, 2018)) + 1185 (new malignant 2019) = 2799 .
\n\nSo it's supposed to equal 2799 yet it equals 2812.
\n\nYet the length of that CSV equals 2812, which is 13 more. Does that mean the 13 duplicates are in the CSV and if so are they somehow marked?
", "votes": 1, "replies": [] }, { "id": 955801, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-03T00:57:45.897000", "content": "Yes 13 rows in the CSV are duplicates marked with tfrecord = -1 and they are not included in any of the TFRecords.
If I want to only add new Malignant,I should let IDX=list(range(15,29)).\nRight?
", "votes": 1, "replies": [ { "id": 942487, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T19:09:58.457000", "content": "Yes exactly. And if you want to add a double dose, you can do IDX=list(range(15,29)) + list(range(15,29)). Note that we are using data augmentation, so each copy will look different to our model. So you can also consider adding more copies of existing malignant.
make correction: IDX=list(range(15,30))
", "votes": 0, "replies": [] }, { "id": 943641, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-24T13:38:35.710000", "content": "Yes, good catch. It needs to be (15,30).
@cdeotte is this data can be used with neural network with meta data??
", "votes": 1, "replies": [ { "id": 942111, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T15:31:20.543000", "content": "Yes. All these TFRecords have the same meta features as my other TFRecords, i.e. age, gender, site, etc. But note that data from 2019, 2018, 2017 does not have patient_id. There is also CSV files in the dataset listing the images and their meta features.
@cdeotte But i see that this data didnt onehot encoded the variables.So is it good to feed it into neural network??
", "votes": 0, "replies": [] }, { "id": 942148, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T15:57:57.183000", "content": "You have three options. You can one hot encode, or use embeddings, or use numeric.
\n\ndef read_labeled_tfrecord(example):\ntfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'target' : tf.io.FixedLenFeature([], tf.int64)\n} \nexample = tf.io.parse_single_example(example, tfrec_format)\nOHE = tf.one_hot(example['anatom_site_general_challenge']+1, 7)\nreturn (example['image'],OHE), example['target']\ndef read_labeled_tfrecord(example):\ntfrec_format = {\n 'image' : tf.io.FixedLenFeature([], tf.string),\n 'anatom_site_general_challenge': tf.io.FixedLenFeature([], tf.int64),\n 'target' : tf.io.FixedLenFeature([], tf.int64)\n} \nexample = tf.io.parse_single_example(example, tfrec_format)\nCAT = example['anatom_site_general_challenge']+1\nreturn (example['image'],CAT), example['target']\nand then in your NN
\n\ndef build_model():\n inp1 = tf.keras.layers.Input(shape=(*IMAGE_SIZE,3))\n inp2 = tf.keras.layers.Input(shape=(1,))\n x2 = tf.keras.Embedding(7, 3, input_length=1)(inp2)\n x2 = tf.keras.Reshape(target_shape=(3, ))(x2)\n # BUILD MODEL HERE\n x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n model = tf.keras.models.Model(inputs=[inp1,inp2], outputs=x)\n return model\nNUM = (example['anatom_site_general_challenge']+1)/3.0 - 1.5\nreturn (example['image'],NUM), example['target']\nand then in your NN
\n\ndef build_model():\n inp1 = tf.keras.layers.Input(shape=(*IMAGE_SIZE,3))\n inp2 = tf.keras.layers.Input(shape=(1,))\n # BUILD MODEL HERE\n x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n model = tf.keras.models.Model(inputs=[inp1,inp2], outputs=x)\n return model\n@cdeotte Has this increased your cv score and lb score??\nAnd btw have you tried pseudo labelling?? Coz for me its not working 😑
", "votes": 1, "replies": [] }, { "id": 942232, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T16:45:19.870000", "content": "I have not added meta features to my CNN models yet.
", "votes": 0, "replies": [] }, { "id": 943675, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-24T14:10:02.543000", "content": "@cdeotte Btw you said that by using triple stratifed data there wont be any leaks in training.And also you said that we have some stable CV and LB score by using only competition KFold splits data as validation.But in my case my CV LB has a std of like from +/- 0.015 to +/- 0.095. Is there any problem in my strategies??
", "votes": 0, "replies": [] }, { "id": 943708, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-24T14:27:17.860000", "content": "Here are the important points\n* you want LB to increase whenever your CV increases\n* you want the same validation data for all of your experiments\n* you want validation score to have low standard deviation when repeating same experiment
\n\nLet me explain the 3rd point. If you run the same experiment over and over, you will get a validation score each time. You want these scores to have low standard deviation. Because if your standard deviation is 0.01. Then you can only know that an experiment is better than previous experiment if the new score is 2 times greater than the standard deviation, i.e. 0.02.
\n\n(When standard deviation is large, it requires that you discover a larger magic to pass evaluation. When standard deviation is small, little magics can be discovered).
\n\nThere are 2 ways to decrease standard deviation. \n* adjust your model, learning schedule, augmentation, regularization, loss, etc\n* for each experiment run the same fold 5 times and take the average as your validation score.
\n\nIdeally, you would like to do the former because then you only need to run 1 fold. But it is harder to find a model with low standard deviation val score. But it is possible. My offline model has low standard deviation.
", "votes": 1, "replies": [] }, { "id": 945180, "author_name": "", "author_url": "", "post_date": "2020-07-25T16:13:39.977000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 941931, "author_name": "Pasquale", "author_url": "", "post_date": "2020-07-23T13:55:45.073000", "content": "Thank you very much for sharing. I have a question: you say there are 4k new images but on the dataset page it says that the jpeg folder only contains around 600 files, did I miss something?
", "votes": 1, "replies": [ { "id": 941945, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T14:03:46.753000", "content": "The dataset contains 4k malignant images. Only 580 are new meaning that those 580 are not contained in previously published 2020 data, nor 2019 data, nor 2018, nor 2017. The other 3420 images are the malignant images from 2020, 2019, 2018, 2017.
\n\nIt is helpful to have the 2020, 2019, 2018, 2017 malignant in their own TFRecords, so if we want three times as many malignant as the 2020 comp data, then we can add the full 2020 comp data, and then add 2 copies of the 2020 malignant data. (We can also experiment with adding one or more copies of 2019, 2018, 2017, or the 580 new malignant images).
", "votes": 1, "replies": [] }, { "id": 941960, "author_name": "Pasquale", "author_url": "", "post_date": "2020-07-23T14:15:04.407000", "content": "Oh okay, got it. Thank you again!
", "votes": 1, "replies": [] } ] }, { "id": 941599, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-07-23T10:13:50.210000", "content": "Thanks for sharing @cdeotte. The two competitions panda and alaska have just finished, LB will definitely become crowded soon :))
", "votes": 1, "replies": [] }, { "id": 941242, "author_name": "lixiangyang", "author_url": "", "post_date": "2020-07-23T06:33:22.930000", "content": "wow，amazing，you can always surprise people.👍
", "votes": 1, "replies": [] }, { "id": 955589, "author_name": "Psi", "author_url": "", "post_date": "2020-08-02T18:14:56.360000", "content": "Playing a bit with only adding target=1 of old data to my training data and am observing very weird behavior that CV increases and LB decreases quite significantly. Anyone experiencing something similar?
", "votes": 2, "replies": [ { "id": 955617, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-02T18:30:08.593000", "content": "Are you only adding the target=1 old data to the training folds and not the validation folds? In that case, it is weird that your CV increases and LB decreases.
Yes, only training data. CV is going up quite significantly, and LB down also significantly. Trying to figure out what's going on...\nMaybe duplicates...need to check that, but I doubt that can be the issue. At least then LB should not get worse.
", "votes": 0, "replies": [] }, { "id": 955665, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-02T19:57:18.923000", "content": "If you are using my data, there are no duplicates. (All have been removed). The only thing you need to be careful about is when adding more 2020 comp data malignant. (You need to add the correct extra 2020 malignant to the correct train folds). When using target=1 old data there will never be leaks using 2020 validation data.
I am not removing dups currently, just move them to same folds.
", "votes": 0, "replies": [] }, { "id": 956625, "author_name": "Eek The Cat", "author_url": "", "post_date": "2020-08-03T16:29:41.310000", "content": "@philippsinger Exactly the same here. I've got 0.957 up from 0.947 with single fold after adding positives to training set only. But LB is... <0.9
", "votes": 1, "replies": [] }, { "id": 956684, "author_name": "Psi", "author_url": "", "post_date": "2020-08-03T17:28:50.903000", "content": "Yep, also <0.9 here
", "votes": 2, "replies": [] }, { "id": 957042, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-08-04T02:41:24.310000", "content": "@cateek Your CV is great. Any tips?
", "votes": 0, "replies": [] }, { "id": 957979, "author_name": "Eek The Cat", "author_url": "", "post_date": "2020-08-04T17:03:31.863000", "content": "@virajbagal It's nothing fancy, mixnet_xl with some variations, @cdeotte 's dataset, 384 resolution, some standard augments. No TTA, no pseudo.
", "votes": 1, "replies": [] }, { "id": 958066, "author_name": "Signal", "author_url": "", "post_date": "2020-08-04T18:35:06.583000", "content": "@cateek So you are saying your own CV is .957 yet your LB is <.9? I would try to tighten that up, that is such a large spread.
", "votes": 0, "replies": [] }, { "id": 958104, "author_name": "Eek The Cat", "author_url": "", "post_date": "2020-08-04T19:12:53.350000", "content": "@brianfeeny Exactly. Several models with local score ranging from 0.91 to 0.95 with different LB scores and no visible correlation
", "votes": 0, "replies": [] }, { "id": 958687, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2020-08-05T05:40:07.847000", "content": "@cateek Is ur CV stable (around 0.95) with change of seed?
", "votes": 0, "replies": [] }, { "id": 958801, "author_name": "Eek The Cat", "author_url": "", "post_date": "2020-08-05T06:35:25.437000", "content": "@virajbagal Haven't tested that
", "votes": 0, "replies": [] }, { "id": 959082, "author_name": "Richard Shaw", "author_url": "", "post_date": "2020-08-05T10:22:25.050000", "content": "I'm also getting worse LB scores when I append the malignant data to my train set. I think simply appending malignant images is not a good idea, because it will change the training distribution. Shouldn't you add the extra data first, then do the data stratification afterwards? That way you make sure you have similar distributions in your train and val sets...
", "votes": 1, "replies": [] }, { "id": 959283, "author_name": "Signal", "author_url": "", "post_date": "2020-08-05T13:21:07.277000", "content": "Yes it changes the training distribution, that is the whole point. You should not add any extra data to the validation.
", "votes": 0, "replies": [] }, { "id": 960047, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-08-06T05:26:48.813000", "content": "It probably partly learns to recognize the dataset source (ISIC18, ISIC19 or ISIC20) instead of malign/benign if you only add the positive ones.
", "votes": 0, "replies": [] }, { "id": 960256, "author_name": "Signal", "author_url": "", "post_date": "2020-08-06T09:15:20.743000", "content": "@group16 do you recommend adding both benign and malignant when upsampling?
", "votes": 0, "replies": [] }, { "id": 960268, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-08-06T09:23:00.563000", "content": "I would recommend that especially for the external data sources that are very different from the data provided here in Kaggle. One way to determine this is by studying Chris his excellent t-SNE plots. The 2019 ISIC data is vastly different from this dataset, so I guess (I am of course not entirely certain) that by only adding the positive samples to your dataset, you confuse your model (as it will learn to detect ISIC 2019 data which corresponds to the positive label in that case)
", "votes": 1, "replies": [] }, { "id": 960277, "author_name": "Signal", "author_url": "", "post_date": "2020-08-06T09:27:26.627000", "content": "@group16 well if you add all of the old train data, then you are back to square 1, highly unbalanced data. Do you think a combination of adding the old data, both benign and malignant, and then upsampling the malignant is the way?
", "votes": 0, "replies": [] }, { "id": 960292, "author_name": "Psi", "author_url": "", "post_date": "2020-08-06T09:35:23.977000", "content": "@group16 Possible, but why is CV going up?
", "votes": 0, "replies": [] }, { "id": 960300, "author_name": "Gilles Vandewiele", "author_url": "", "post_date": "2020-08-06T09:43:31.907000", "content": "@philippsinger That is indeed a good question and a rather weird observation. Surely you validate only on ISIC20 images? And potential duplicates are already removed? If some of the positive ISIC2019 images also occur in the ISIC20 dataset (there are a few duplicates), then that can increase the CV significantly as there are not a lot of positives...
\n\n@brianfeeny You are completely correct, the data will be unbalanced again. There are other ways to combat this:\n* Make sure the proportion malignant/benign is greater in the external data you add (this reduces the imbalance).\n* I never like not utilizing information (called undersampling, which is what not including benign external images does). I don't like over-sampling either, as it will introduce many biases (e.g. SMOTE assumes linear relations between your data points) and will only generate from the already learned train distribution, which your network already handles. As such, I prefer using different objectives that better handle imbalanced data. Focal loss, label smoothing or giving more weight to positive samples are all much more (theoretically) satisfying solutions than under- or oversampling.
", "votes": 2, "replies": [] }, { "id": 960397, "author_name": "Signal", "author_url": "", "post_date": "2020-08-06T11:07:51.983000", "content": "@group16 thanks for the tips. I tried FocalLoss, but it just did so poorly, I must have tried 3 different implementations of it. I just swapped out BCE for FocalLoss and AUC went into the toilet. I would revisit it, but not sure what I could be doing wrong. I also tried to do class weights. I was using sklearns Compute Class Weight. The weights it suggested were rather extreme, but I tried this back when just using the default hugely unbalanced data. Perhaps I should forgo the sklearn class and just try to give malignant like 2x or 3x.
", "votes": 1, "replies": [] }, { "id": 960564, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-06T14:06:34.440000", "content": "\n\n\nwell if you add all of the old train data, then you are back to square 1, highly unbalanced data.
\n
No, the data from 2019 has something like 20% malignant whereas this 2020 comp data is 2% malignant. So if you add all of 2019 external both benign and malignant, then your overall malignant is 11%. Then if you additionally add 1 copy of 2019 malignant, you're approximately at 20%, with 2 copies 30%, with 3 copies 40% etc.
\n\nSo the original data is \"highly unbalanced\" at 2%, but 11%, 20%, 30%, and 40% are not \"highly unbalanced\".
", "votes": 1, "replies": [] }, { "id": 960618, "author_name": "Kha Vo", "author_url": "", "post_date": "2020-08-06T14:41:02.780000", "content": "How can training the unbalanced data with different levels of balance (by adjust sample rate of each class in data generator) affect AUC performance since AUC only deals with ranking over a population?
", "votes": 1, "replies": [] }, { "id": 960623, "author_name": "", "author_url": "", "post_date": "2020-08-06T14:44:40.020000", "content": "", "votes": 0, "replies": [] }, { "id": 960665, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-06T15:26:01.583000", "content": "@philippsinger I got the same when adding 2019 malignant data to training folds but not validations folds. CV went up to 0.923 (256x256 effnet-b0) but LB went down to 0.8840.
\n\nIt is a bit reassuring to see it is not (solely) due to my messing up...
", "votes": 1, "replies": [] }, { "id": 961226, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-07T02:50:22.390000", "content": "If people are having trouble making upsampling work with my malignant dataset work, I suggest adding 2018 2017 benign simultaneously with adding extra malignant from 2019, 2018, 2017 or ISIC-archive. Otherwise your model doesn't learn malignant it learns to identify external versus non-external.
", "votes": 2, "replies": [] } ] }, { "id": 946280, "author_name": "Saish Reddy Komalla", "author_url": "", "post_date": "2020-07-26T13:42:31.490000", "content": "Nice one @cdeotte, cleared many of my preprocessing issues.!
", "votes": 2, "replies": [] }, { "id": 942657, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2020-07-23T23:00:05.030000", "content": "Did you get all of the malignant images or are there more?\nI am considering on working further on this topic after the contest ends.\nEspecially I am wondering why there are no black skin samples in our dataset. Did you find any?
", "votes": 2, "replies": [ { "id": 942662, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T23:07:24.410000", "content": "I got all malignant images from ISIC-archive (their website database), this year's 2020 comp, and previous year's 2019, 2018, 2017 comps.
\n\nNote that there are some 2019 malignant that I do not include in this dataset. But all the 2019 malignant are of course in my 2019 dataset.
", "votes": 1, "replies": [] }, { "id": 943172, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2020-07-24T07:22:43.140000", "content": "If your old, have white skin and blue eyes you can almost count on at least one bad mole in your lifetime. I also looked at a bunch of images and saw no black skin samples. I had plans to add \"skin color\" as a meta data feature by looking (automating the look), as references I saw indicated my old white skin is 33 times more likely to have melanoma than black skin.
\n\nStill have it on my to do to visually look at more images - I am not getting much improvement in results with the current meta data. I added some average color stuff, but the black ring of the microscope seems to make simple color data of low value.
", "votes": 0, "replies": [] } ] }, { "id": 952441, "author_name": "Payam Khorramshahi", "author_url": "", "post_date": "2020-07-30T23:09:53.280000", "content": "@cdeotte Thanks for sharing this dataset ! Just to be clarified, there are 580 new malignant data. So by 4000 you mean 7 set of the same 580 data in different sizes. Is that right ?
", "votes": 0, "replies": [ { "id": 952454, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-30T23:48:13.850000", "content": "By 4000, i mean 3420 repeat and 580 new. The repeat are the malignant from 2020 2019 2018 2017 comp data. The new are malignant scrapped from ISIC-archive website.
\n\nAll the new are in TFRecords numbered 15-29. And they are in the included folder named JPEG as jpegs. The list of new is in the CSV train_malig_2.csv.
thankyou for your great work!
\n\nMay I ask that why the data dont have patient_id?
", "votes": 0, "replies": [ { "id": 944563, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-25T07:30:55.693000", "content": "All old data before the year 2020 doesn't have patient_id.
Thanks @synked
", "votes": 0, "replies": [] }, { "id": 944608, "author_name": "", "author_url": "", "post_date": "2020-07-25T07:54:58.560000", "content": "", "votes": 1, "replies": [] }, { "id": 944622, "author_name": "yangDDD", "author_url": "", "post_date": "2020-07-25T08:05:39.503000", "content": "okey, thanks
", "votes": 0, "replies": [] }, { "id": 944624, "author_name": "", "author_url": "", "post_date": "2020-07-25T08:06:12.860000", "content": "", "votes": 0, "replies": [] }, { "id": 944625, "author_name": "yangDDD", "author_url": "", "post_date": "2020-07-25T08:08:10.457000", "content": "me too! @x2t2cx
", "votes": 0, "replies": [] } ] }, { "id": 943183, "author_name": "Rony", "author_url": "", "post_date": "2020-07-24T07:30:09.350000", "content": "I have joined my first Kaggle competition.I am facing lots of difficulties in understanding the dataset and the folds. Which dataset to use? How to execute the folds? What to do with the tfrecord,metadata,external data posted by @cdeotte ?Basically I am stuck upto the basics of solving a problem. Can anyone help me in moving forward in these issue? What should I do ? Thanks for stopping by.
", "votes": 0, "replies": [ { "id": 943193, "author_name": "Sirish Somanchi", "author_url": "", "post_date": "2020-07-24T07:39:26.997000", "content": "Rony - Did you check the 0.9454 score public kernel Triple Stratified KFold with TFRecords
\n\nBy the way, if this is your first competition, you can use the official Topic New to Machine Learning or Kaggle?
", "votes": 1, "replies": [] } ] }, { "id": 942349, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2020-07-23T17:51:15.687000", "content": "I get an error adding tfrecords, I dont understand - any clues?
\n\nGCS_PATH3 = KaggleDatasets().get_gcs_path('malignant-v2-256x256')\nMALIGNANT = [GCS_PATH3 + '/train%.2i*.tfrec'%x for x in IDX]\nprint(MALIGNANT)\nfiles_train += tf.io.gfile.glob(MALIGNANT)
\n\n--->files_train += tf.io.gfile.glob(MALIGNANT)\n np.random.shuffle(files_train)
\n\nUFuncTypeError: ufunc 'add' did not contain a loop with signature matching types (dtype(' dtype('
", "votes": 0, "replies": [ { "id": 942395, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-23T18:10:29.600000", "content": "I assume that the preceeding code is
\n\nfiles_train = np.sort(np.array(tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')))\nYou have turned files_train into NumPy array. Change the above line to
files_train = tf.io.gfile.glob(GCS_PATH + '/train*.tfrec')\nRemove the conversion to NumPy will fix the problem. We don't need it. We shuffle everything afterward anyways with
\n\nnp.random.shuffle(files_train)\nyou are right - thx
", "votes": 1, "replies": [] } ] }, { "id": 959116, "author_name": "", "author_url": "", "post_date": "2020-08-05T11:13:46.863000", "content": "", "votes": 0, "replies": [] }, { "id": 957422, "author_name": "", "author_url": "", "post_date": "2020-08-04T09:41:18.427000", "content": "", "votes": 3, "replies": [ { "id": 957742, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-04T14:27:06.530000", "content": "There are 3 sources. (1) The 2020 malignant come from this year's Kaggle comp data (2) The 2019 (2018 2017) malignant come from 2019 comp data here. (3) The 580 unique malignant come from the ISIC-archive here.
\n\nTo download images from the archive, you select malignant in the filter on the left, then click Select All on the Page for Download, and then click Download as ZIP. Then go to each of the 29 pages of malignant images and download the 29 ZIP files. You will get 2285 malignant images from ISIC-archive, then use RAPIDS cuML kNN (notebook here) to remove the images that already exist in 2020, 2019, 2018, 2017 data. You will be left with 580 new malignant from ISIC-archive. (Lastly I center square crop resize them).