![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Fb9fd40b45aa428e23a9a470322aa8c7b%2Fcrop.png?generation=1597700942674639&alt=media\")
Thank you Kaggle, SIIM, and ISIC for an exciting competition. Thank you Kagglers for wonderful shared content and great discussions!
\nEarly on I decided to build a large ensemble instead of optimizing a single model. The AUC metric seemed very unstable with this unbalanced dataset and using ensembles, heavy TTA, and crop augmentation helped stabilize it.
\nMy main (first) submission was an ensemble that maximized CV where all models used the same triple stratified leak-free CV with seed = 42. (How to CV explained here). Next I believed public notebooks could add diversity too, so my final 3 submissions were:
Sub_1 - 9 models of mine ensemble - CV 0.9505 LB 0.9578 - Private 0.9418Sub_2 - 5 mine plus 10 public single models - CV ??? LB 0.9662 - Private 0.9425Sub_3 = 0.75 * Sub_1 + 0.25 * Sub_2 - CV ??? LB 0.9603 - Private 0.9429Crop Augmentation was key to prevent overfitting during training when using external data, upsampling, and large EfficientNet backbones. Crop augmentation also helped stabilize AUC particularly when used in TTA.
\nPreviously we saw that using different image sizes adds diversity (explained here). What many people don't realize is that you can use TFRecord sized 512x512 but train on random 256x256 crops (different each epoch). Then training goes fast because your EfficientNet only processes size 256x256 but you are getting features from 512x512 resolution. This allows us to train quickly on large sizes such as 1024x1024 and 768x768 (using crops of 512 and 384 respectively).
\nIn the picture below, we read the top row from TFRecords, then random crop, then train with the bottom row.
\n
The following 8 models have ensemble CV 0.9500, Public LB 0.9577, and Private LB 0.9420 . Then ensembling meta data with geometric mean: image_ensemble**0.9 * tabular_model**0.1, increases CV to 0.9505, Public LB to 0.9578, and Private LB to 0.9418.
| CV | \nLB | \nread size | \ncrop size | \neffNet | \next data | \nupsample | \n
|---|---|---|---|---|---|---|
| 0.936 | \n0.956 | \n512 | \n384 | \nB5 | \n2018 | \n1,1,1,1 | \n
| 0.935 | \n0.937 | \n768 | \n512 | \nB6 | \n2019 2018 | \n3,3,0,0 | \n
| 0.935 | \n0.949 | \n768 | \n512 | \nB7 | \n2018 | \n1,1,1,1 | \n
| 0.933 | \n0.950 | \n1024 | \n512 | \nB6 | \n2018 | \n2,2,2,2 | \n
| 0.927 | \n0.942 | \n768 | \n384 | \nB4 | \n2018 | \n0,0,0,0 | \n
| 0.920 | \n0.941 | \n512 | \n384 | \nB5 | \n2019 2018 | \n10,0,0,0 | \n
| 0.916 | \n0.946 | \n384 | \n384 | \nB345 | \nno | \n0,0,0,0 | \n
| 0.910 | \n0.950 | \n384 | \n384 | \nB6 | \n2018 | \n0,0,0,0 | \n
The above models use a variety of different augmentation, losses, optimizers, and learning rate schedules. External data explained here. Upsample explained here.
\nIf you download my popular notebook here to your local machine or cloud provider, then you can run the code quickly using multiple GPUs by adding the following one line of code
\nDEVICE = \"GPU\"\nstrategy = tf.distribute.MirroredStrategy()\nMost of my models including my most accurate single model with CV 0.936 and LB 0.956 were trained using four Nvidia V100 GPUs. Thank you Nvidia for the use of GPUs!
\nIn the past 2 months, I trained 50+ diverse models. How do we ensemble 50+ models? Train all models using the same triple stratified folds seed = 42 from my notebook here. Then to create an ensemble, start with the model that has largest CV and repeatedly try adding one model to increase CV. Whichever one additional model increases the CV the most (and at least 0.0003), keep that model and then iterate through all models again. Repeat this process until CV score stops increasing (by at least 0.0003).
# START ENSEMBLE USING MODEL WITH LARGEST CV\n Repeat until CV does not increase by 0.0003+ :\n # TRY ADDING EVERY MODEL ONE AT A TIME AND REMEMBER \n # HOW MUCH EACH INCREASES THE ENSEMBLE CV SCORE\n for k in range( len(models) ):\n for w in [0.01, 0.02, ..., 0.98, 0.99]:\n # TRY ADDING MODEL k WITH WEIGHT w TO ENSEMBLE\n trial = w * model[k,] + (1-w) * ensemble\n auc_trial = roc_auc_score(true, trial)\n # ADD ONE NEW MODEL TO ENSEMBLE THAT INCREASED CV THE MOST\n # CHECK NEW CV SCORE. IF IT INCREASED REPEAT LOOP\nI posted my solution code here showing how to ensemble OOF files using forward selection. The notebook uses 39 of my Melanoma models' OOF files. Forward selection chooses 8 of them and achieves CV 0.950, Public LB 0.958, Private LB 0.942.
", "messages": [ { "id": 2905912, "postDate": "2024-07-05T08:25:10.963Z", "content": "Does the data include HAM10000 dataset? or is it mutually exclusive?
", "rawMarkdown": "Does the data include HAM10000 dataset? or is it mutually exclusive?", "votes": 1 }, { "id": 974529, "postDate": "2020-08-18T01:14:57.153Z", "content": "Thank you Kaggle, SIIM, and ISIC for an exciting competition. Thank you Kagglers for wonderful shared content and great discussions!
\nEarly on I decided to build a large ensemble instead of optimizing a single model. The AUC metric seemed very unstable with this unbalanced dataset and using ensembles, heavy TTA, and crop augmentation helped stabilize it.
\nMy main (first) submission was an ensemble that maximized CV where all models used the same triple stratified leak-free CV with seed = 42. (How to CV explained here). Next I believed public notebooks could add diversity too, so my final 3 submissions were:
Sub_1 - 9 models of mine ensemble - CV 0.9505 LB 0.9578 - Private 0.9418Sub_2 - 5 mine plus 10 public single models - CV ??? LB 0.9662 - Private 0.9425Sub_3 = 0.75 * Sub_1 + 0.25 * Sub_2 - CV ??? LB 0.9603 - Private 0.9429Crop Augmentation was key to prevent overfitting during training when using external data, upsampling, and large EfficientNet backbones. Crop augmentation also helped stabilize AUC particularly when used in TTA.
\nPreviously we saw that using different image sizes adds diversity (explained here). What many people don't realize is that you can use TFRecord sized 512x512 but train on random 256x256 crops (different each epoch). Then training goes fast because your EfficientNet only processes size 256x256 but you are getting features from 512x512 resolution. This allows us to train quickly on large sizes such as 1024x1024 and 768x768 (using crops of 512 and 384 respectively).
\nIn the picture below, we read the top row from TFRecords, then random crop, then train with the bottom row.
\n
The following 8 models have ensemble CV 0.9500, Public LB 0.9577, and Private LB 0.9420 . Then ensembling meta data with geometric mean: image_ensemble**0.9 * tabular_model**0.1, increases CV to 0.9505, Public LB to 0.9578, and Private LB to 0.9418.
| CV | \nLB | \nread size | \ncrop size | \neffNet | \next data | \nupsample | \n
|---|---|---|---|---|---|---|
| 0.936 | \n0.956 | \n512 | \n384 | \nB5 | \n2018 | \n1,1,1,1 | \n
| 0.935 | \n0.937 | \n768 | \n512 | \nB6 | \n2019 2018 | \n3,3,0,0 | \n
| 0.935 | \n0.949 | \n768 | \n512 | \nB7 | \n2018 | \n1,1,1,1 | \n
| 0.933 | \n0.950 | \n1024 | \n512 | \nB6 | \n2018 | \n2,2,2,2 | \n
| 0.927 | \n0.942 | \n768 | \n384 | \nB4 | \n2018 | \n0,0,0,0 | \n
| 0.920 | \n0.941 | \n512 | \n384 | \nB5 | \n2019 2018 | \n10,0,0,0 | \n
| 0.916 | \n0.946 | \n384 | \n384 | \nB345 | \nno | \n0,0,0,0 | \n
| 0.910 | \n0.950 | \n384 | \n384 | \nB6 | \n2018 | \n0,0,0,0 | \n
The above models use a variety of different augmentation, losses, optimizers, and learning rate schedules. External data explained here. Upsample explained here.
\nIf you download my popular notebook here to your local machine or cloud provider, then you can run the code quickly using multiple GPUs by adding the following one line of code
\nDEVICE = \"GPU\"\nstrategy = tf.distribute.MirroredStrategy()\nMost of my models including my most accurate single model with CV 0.936 and LB 0.956 were trained using four Nvidia V100 GPUs. Thank you Nvidia for the use of GPUs!
\nIn the past 2 months, I trained 50+ diverse models. How do we ensemble 50+ models? Train all models using the same triple stratified folds seed = 42 from my notebook here. Then to create an ensemble, start with the model that has largest CV and repeatedly try adding one model to increase CV. Whichever one additional model increases the CV the most (and at least 0.0003), keep that model and then iterate through all models again. Repeat this process until CV score stops increasing (by at least 0.0003).
# START ENSEMBLE USING MODEL WITH LARGEST CV\n Repeat until CV does not increase by 0.0003+ :\n # TRY ADDING EVERY MODEL ONE AT A TIME AND REMEMBER \n # HOW MUCH EACH INCREASES THE ENSEMBLE CV SCORE\n for k in range( len(models) ):\n for w in [0.01, 0.02, ..., 0.98, 0.99]:\n # TRY ADDING MODEL k WITH WEIGHT w TO ENSEMBLE\n trial = w * model[k,] + (1-w) * ensemble\n auc_trial = roc_auc_score(true, trial)\n # ADD ONE NEW MODEL TO ENSEMBLE THAT INCREASED CV THE MOST\n # CHECK NEW CV SCORE. IF IT INCREASED REPEAT LOOP\nI posted my solution code here showing how to ensemble OOF files using forward selection. The notebook uses 39 of my Melanoma models' OOF files. Forward selection chooses 8 of them and achieves CV 0.950, Public LB 0.958, Private LB 0.942.
", "rawMarkdown": "# Melanoma Model Ensemble!\nThank you Kaggle, SIIM, and ISIC for an exciting competition. Thank you Kagglers for wonderful shared content and great discussions! \n\nEarly on I decided to build a large ensemble instead of optimizing a single model. The AUC metric seemed very unstable with this unbalanced dataset and using ensembles, heavy TTA, and crop augmentation helped stabilize it.\n\n# My Final 3 Submissions\nMy main (first) submission was an ensemble that maximized CV where all models used the same [triple stratified leak-free CV][3] with `seed = 42`. (How to CV explained [here][5]). Next I believed public notebooks could add diversity too, so my final 3 submissions were:\n\n* `Sub_1` - 9 models of mine ensemble - CV 0.9505 LB 0.9578 - Private 0.9418\n* `Sub_2` - 5 mine plus 10 public single models - CV ??? LB 0.9662 - Private 0.9425\n* `Sub_3 = 0.75 * Sub_1 + 0.25 * Sub_2` - CV ??? LB 0.9603 - Private 0.9429\n\n# Crop Augmentation\nCrop Augmentation was key to prevent overfitting during training when using external data, upsampling, and large EfficientNet backbones. Crop augmentation also helped stabilize AUC particularly when used in TTA.\n\nPreviously we saw that using different image sizes adds diversity (explained [here][2]). What many people don't realize is that you can use TFRecord sized 512x512 but train on random 256x256 crops (different each epoch). Then training goes fast because your EfficientNet only processes size 256x256 but you are getting features from 512x512 resolution. This allows us to train quickly on large sizes such as 1024x1024 and 768x768 (using crops of 512 and 384 respectively).\n\nIn the picture below, we read the top row from TFRecords, then random crop, then train with the bottom row.\n\n\n\n# My Final Models\n\nThe following 8 models have ensemble CV 0.9500, Public LB 0.9577, and Private LB 0.9420 . Then ensembling meta data with geometric mean: `image_ensemble**0.9 * tabular_model**0.1`, increases CV to 0.9505, Public LB to 0.9578, and Private LB to 0.9418.\n\n \n| CV | LB | read size | crop size | effNet | ext data | upsample |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0.936 | 0.956 | 512 | 384 | B5 | 2018 | 1,1,1,1 |\n| 0.935 | 0.937 | 768 | 512 | B6 | 2019 2018 | 3,3,0,0 |\n| 0.935 | 0.949 | 768 | 512 | B7 | 2018 | 1,1,1,1 |\n| 0.933 | 0.950 | 1024 | 512 | B6 | 2018 | 2,2,2,2 |\n| 0.927 | 0.942 | 768 | 384 | B4 | 2018 | 0,0,0,0 |\n| 0.920 | 0.941 | 512 | 384 | B5 | 2019 2018 | 10,0,0,0 |\n| 0.916 | 0.946 | 384 | 384 | B345 | no | 0,0,0,0 |\n| 0.910 | 0.950 | 384 | 384 | B6 | 2018 | 0,0,0,0 |\n \nThe above models use a variety of different augmentation, losses, optimizers, and learning rate schedules. External data explained [here][6]. Upsample explained [here][7].\n\n# My Training\n\nIf you download my popular notebook [here][1] to your local machine or cloud provider, then you can run the code quickly using multiple GPUs by adding the following one line of code\n\n DEVICE = \"GPU\"\n strategy = tf.distribute.MirroredStrategy()\n\nMost of my models including my most accurate single model with CV 0.936 and LB 0.956 were trained using four Nvidia V100 GPUs. Thank you Nvidia for the use of GPUs!\n \n# Ensemble Pseudo Code\n \nIn the past 2 months, I trained 50+ diverse models. How do we ensemble 50+ models? Train all models using the same triple stratified folds `seed = 42` from my notebook [here][1]. Then to create an ensemble, start with the model that has largest CV and repeatedly try adding one model to increase CV. Whichever one additional model increases the CV the most (and at least 0.0003), keep that model and then iterate through all models again. Repeat this process until CV score stops increasing (by at least 0.0003).\n \n # START ENSEMBLE USING MODEL WITH LARGEST CV\n Repeat until CV does not increase by 0.0003+ :\n # TRY ADDING EVERY MODEL ONE AT A TIME AND REMEMBER \n # HOW MUCH EACH INCREASES THE ENSEMBLE CV SCORE\n for k in range( len(models) ):\n for w in [0.01, 0.02, ..., 0.98, 0.99]:\n # TRY ADDING MODEL k WITH WEIGHT w TO ENSEMBLE\n trial = w * model[k,] + (1-w) * ensemble\n auc_trial = roc_auc_score(true, trial)\n # ADD ONE NEW MODEL TO ENSEMBLE THAT INCREASED CV THE MOST\n # CHECK NEW CV SCORE. IF IT INCREASED REPEAT LOOP\n\n# Ensemble Starter Notebook\nI posted my solution code [here][4] showing how to ensemble OOF files using forward selection. The notebook uses 39 of my Melanoma models' OOF files. Forward selection chooses 8 of them and achieves CV 0.950, Public LB 0.958, Private LB 0.942.\n\n[1]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165526\n[4]: https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private\n[5]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614\n[6]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910\n[7]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169139\n", "votes": 194 }, { "id": 974870, "postDate": "2020-08-18T04:35:13.570Z", "content": "You were the MVP of this competition @cdeotte . Thank you for all the work that you did, preparing the data and providing useful insight throughout the competition.
", "rawMarkdown": "You were the MVP of this competition @cdeotte . Thank you for all the work that you did, preparing the data and providing useful insight throughout the competition. ", "votes": 7, "replies": [ { "id": 974923, "postDate": "2020-08-18T04:58:46.733Z", "content": "Thanks Kaz. Great job with your team's strong finish. I look forward to reading about your model. What was your CV score?
", "rawMarkdown": "Thanks Kaz. Great job with your team's strong finish. I look forward to reading about your model. What was your CV score?", "votes": 1 } ] }, { "id": 974613, "postDate": "2020-08-18T02:00:56.807Z", "content": "Great work @cdeotte - your idea of cropping larger images to the smaller image size is really smart! As always you were a huge inspiration to many (myself included) and we appreciate all you shared and your enthusiasm for data science!
", "rawMarkdown": "Great work @cdeotte - your idea of cropping larger images to the smaller image size is really smart! As always you were a huge inspiration to many (myself included) and we appreciate all you shared and your enthusiasm for data science!", "votes": 3, "replies": [ { "id": 974619, "postDate": "2020-08-18T02:02:25.063Z", "content": "Thanks Rob. Congrats to you and your team's great finish.
", "rawMarkdown": "Thanks Rob. Congrats to you and your team's great finish. ", "votes": 1 } ] }, { "id": 974648, "postDate": "2020-08-18T02:15:46.030Z", "content": "Thanks @cdeotte for all your great contributions.
\nEven though most of my experiments was done on Pytorch. I was solely using your Jpegs datasets and almost all sizes (unless 1024 and 128 :p)
\nSo, you was an inspiration and helped (almost) all competitors in one or another way
@cdeotte
\nOne of the reasons kaggler doesn't use TF because they need to write much stuff from scratch using unfriendly TF syntax. But not only you use it but also you make it more beautiful for this competition. Congratulation and many thanks to you for your wonderful contribution. Those Triple One GM status really suits you, Perfect. Hoping for the last one 😉
I saw the podcast of yours with data science chai on YouTube, oh mine, I thought you're a very moody person, very deep voice, maybe hard attitude … but no, I found you are really cool and open-minded. However, thanks again for everything. And be like this forever, not only awesome in work but also kind as a human.
", "rawMarkdown": "@cdeotte \nOne of the reasons kaggler doesn't use `TF` because they need to write much stuff from scratch using unfriendly `TF` syntax. But not only you use it but also you make it more beautiful for this competition. Congratulation and many thanks to you for your wonderful contribution. Those **Triple One GM** status really suits you, Perfect. Hoping for the last one 😉\n\nI saw the podcast of yours with data science chai on YouTube, oh mine, I thought you're a very moody person, very deep voice, maybe hard attitude ... but no, I found you are really cool and open-minded. However, thanks again for everything. And be like this forever, not only awesome in work but also kind as a human. ", "votes": 2, "replies": [ { "id": 974928, "postDate": "2020-08-18T05:00:15.393Z", "content": "Thank you @ipythonx for your kind words.
", "rawMarkdown": "Thank you @ipythonx for your kind words. ", "votes": 2 } ] }, { "id": 974597, "postDate": "2020-08-18T01:52:56.760Z", "content": "Your public kernels were tooooo good (not only score but overall pipeline) many people just blended all possible ways and with 3 submissions got more luck. In my opinion you deserve better place for sure and if there would be special medal for mentoring and sharing it would be your gold medal.
", "rawMarkdown": "Your public kernels were tooooo good (not only score but overall pipeline) many people just blended all possible ways and with 3 submissions got more luck. In my opinion you deserve better place for sure and if there would be special medal for mentoring and sharing it would be your gold medal.", "votes": 2, "replies": [ { "id": 974630, "postDate": "2020-08-18T02:06:03.910Z", "content": "Thanks Konstantin.
", "rawMarkdown": "Thanks Konstantin.", "votes": 1 } ] }, { "id": 991359, "postDate": "2020-08-30T10:51:44.900Z", "content": "Thanks for sharing, This is helpful for me.
", "rawMarkdown": "Thanks for sharing, This is helpful for me.", "votes": 1 }, { "id": 987858, "postDate": "2020-08-27T15:21:00.130Z", "content": "Thanks for sharing your approach. Very helpful.
", "rawMarkdown": "Thanks for sharing your approach. Very helpful.", "votes": 1 }, { "id": 983756, "postDate": "2020-08-24T15:13:41.017Z", "content": "Thanks for sharing your approach and all the baseline kernels you created for reference.
", "rawMarkdown": "Thanks for sharing your approach and all the baseline kernels you created for reference.", "votes": 1 }, { "id": 983426, "postDate": "2020-08-24T09:48:29.173Z", "content": "@cdeotte Congratulations Chris and thanks alot for anchoring noobs like me throughout the competition.
", "rawMarkdown": "@cdeotte Congratulations Chris and thanks alot for anchoring noobs like me throughout the competition.", "votes": 1, "replies": [ { "id": 987003, "postDate": "2020-08-26T22:45:17.570Z", "content": "Thanks @ankitsajwan . Congratulations on achieving solo silver. Well done.
", "rawMarkdown": "Thanks @ankitsajwan . Congratulations on achieving solo silver. Well done.", "votes": 1 } ] }, { "id": 982816, "postDate": "2020-08-23T17:23:01.660Z", "content": "Thanks for sharing your approach. Very helpful.
", "rawMarkdown": "Thanks for sharing your approach. Very helpful.", "votes": 1 }, { "id": 980679, "postDate": "2020-08-21T19:18:09.670Z", "content": "Most awesome! I like your tackling the problem from a few different angles. Also, just watched your interview on YouTube. Very inspirational!
", "rawMarkdown": "Most awesome! I like your tackling the problem from a few different angles. Also, just watched your interview on YouTube. Very inspirational!", "votes": 1, "replies": [ { "id": 980723, "postDate": "2020-08-21T19:55:34.010Z", "content": "Thank you Baron
", "rawMarkdown": "Thank you Baron" } ] }, { "id": 980082, "postDate": "2020-08-21T10:19:25.940Z", "content": "Interesting!
", "rawMarkdown": "Interesting!", "votes": 1 }, { "id": 977954, "postDate": "2020-08-19T20:14:10.577Z", "content": "Quick question about Crop Augmentation @cdeotte. What is the size of the images used during the validation/test step? Let's assume at training time we are using 512x512 images randomly cropped to 384x384. Are you using for validation and test images the original images of size 512x512, 512x512 randomly cropped to 384x384, or the original image at size 384x384?
", "rawMarkdown": "Quick question about Crop Augmentation @cdeotte. What is the size of the images used during the validation/test step? Let's assume at training time we are using 512x512 images randomly cropped to 384x384. Are you using for validation and test images the original images of size 512x512, 512x512 randomly cropped to 384x384, or the original image at size 384x384?", "votes": 1, "replies": [ { "id": 978046, "postDate": "2020-08-19T22:14:11.397Z", "content": "Great question. After training, all OOF predictions use TTA = 21. So the validation OOF predictions are the average of 21 random crops. (Also the test set predictions are the average of 21 random crops).
During training, TF computes the validation loss and validation AUC using a single 384 center crop. (Assuming we're training with 512 and random cropping 384). This is the validation loss used for early stopping or model checkpoint.
", "rawMarkdown": "Great question. After training, all OOF predictions use `TTA = 21`. So the validation OOF predictions are the average of 21 random crops. (Also the test set predictions are the average of 21 random crops).\n\nDuring training, TF computes the validation loss and validation AUC using a single 384 center crop. (Assuming we're training with 512 and random cropping 384). This is the validation loss used for early stopping or model checkpoint.", "votes": 2, "replies": [ { "id": 978113, "postDate": "2020-08-20T00:48:31.407Z", "content": "Got it! Thanks for sharing this trick @cdeotte!
", "rawMarkdown": "Got it! Thanks for sharing this trick @cdeotte!", "votes": 1 } ] }, { "id": 978059, "postDate": "2020-08-19T22:43:33.690Z", "content": "In my popular public notebook here, just change prepare_image() function to the following.
IMG_SIZES = [512]*FOLDS\nCROP = 384\n\ndef prepare_image(img, augment=True, dim=256): \n img = tf.image.decode_jpeg(img, channels=3)\n img = tf.cast(img, tf.float32) / 255.0\n\n if augment:\n if dim!=CROP: img = tf.image.random_crop(img, [CROP, CROP, 3])\n img = transform(img,DIM=CROP)\n img = tf.image.random_flip_left_right(img)\n img = tf.image.random_saturation(img, 0.7, 1.3)\n img = tf.image.random_contrast(img, 0.8, 1.2)\n img = tf.image.random_brightness(img, 0.1)\n elif dim!=CROP:\n img = tf.image.central_crop(img,CROP/dim)\n\n img = tf.reshape(img, [CROP,CROP, 3])\n\n return img\nAnd then when you build the model, use
\nmodel = build_model(dim=CROP, ef=EFF_NETS[fold])\nThanks, you let me quickly learn how to use Tensorflow
", "rawMarkdown": "Thanks, you let me quickly learn how to use Tensorflow", "votes": 1 }, { "id": 977147, "postDate": "2020-08-19T10:18:17.320Z", "content": "Congrats for your rank!
", "rawMarkdown": "Congrats for your rank!", "votes": 1, "replies": [ { "id": 978283, "postDate": "2020-08-20T04:42:13.450Z", "content": "Thanks XY=ABCD. Congrats for your bronze medal
", "rawMarkdown": "Thanks XY=ABCD. Congrats for your bronze medal" } ] }, { "id": 976900, "postDate": "2020-08-19T06:59:57.480Z", "content": "The best part, in my opinion is, how easy-to-use & understand you code is !
", "rawMarkdown": "The best part, in my opinion is, how easy-to-use & understand you code is !", "votes": 1 }, { "id": 976299, "postDate": "2020-08-18T19:20:00.993Z", "content": "UPDATE: I posted a starter notebook here showing how to ensemble OOF files using forward selection. I use my 39 of my Melanoma models' OOF files. Forward selection chooses 8 of them and achieves CV 0.950, Public LB 0.958, Private LB 0.942.
", "rawMarkdown": "UPDATE: I posted a starter notebook [here][1] showing how to ensemble OOF files using forward selection. I use my 39 of my Melanoma models' OOF files. Forward selection chooses 8 of them and achieves CV 0.950, Public LB 0.958, Private LB 0.942.\n\n[1]: https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private", "votes": 1 }, { "id": 976046, "postDate": "2020-08-18T15:51:53.257Z", "content": "Thanks mate ! Your CV and public notebook were really nifty !
", "rawMarkdown": "Thanks mate ! Your CV and public notebook were really nifty ! ", "votes": 1 }, { "id": 975813, "postDate": "2020-08-18T13:30:24.823Z", "content": "Thanks Chris! I guess >90% people used your baseline model to experiment for this competition🙌
", "rawMarkdown": "Thanks Chris! I guess >90% people used your baseline model to experiment for this competition🙌", "votes": 1 }, { "id": 975806, "postDate": "2020-08-18T13:27:02.233Z", "content": "thank for your dataset help me have first medal👍
", "rawMarkdown": "thank for your dataset help me have first medal👍", "votes": 1 }, { "id": 975325, "postDate": "2020-08-18T08:45:17.243Z", "content": "Congrats Chris
", "rawMarkdown": "Congrats Chris", "votes": 1 }, { "id": 975311, "postDate": "2020-08-18T08:38:05.150Z", "content": "Thanks Chris - it was so kind of you to share all your hard work and I'm sure a large percentage of those in the medal positions got there with your hard work as solid foundation - that was the case for me so thank you!
", "rawMarkdown": "Thanks Chris - it was so kind of you to share all your hard work and I'm sure a large percentage of those in the medal positions got there with your hard work as solid foundation - that was the case for me so thank you!", "votes": 1, "replies": [ { "id": 983077, "postDate": "2020-08-24T03:12:24.670Z", "content": "Thanks Mark. Congrats on finishing 49th solo silver. That's fantastic.
", "rawMarkdown": "Thanks Mark. Congrats on finishing 49th solo silver. That's fantastic." } ] }, { "id": 975291, "postDate": "2020-08-18T08:27:28.673Z", "content": "Thanks for sharing. Thanks to your kernel and dataset, I was able to participate this competition well. I experiment a lot based on your triple stratified CV. Here are some experiment results.
\n| cv | \nimage size | \nmodel | \next | \nmethod | \n
|---|---|---|---|---|
| 0.944 | \n512 | \nb6 | \n2018 | \nmultitask learning - target(2), diagnosis(3), site(6) | \n
| 0.942 | \n512 | \nb6 | \n2018 | \naugmentation - mixup | \n
Thanks again.
", "rawMarkdown": "Thanks for sharing. Thanks to your kernel and dataset, I was able to participate this competition well. I experiment a lot based on your triple stratified CV. Here are some experiment results.\n\n| cv | image size | model | ext | method |\n| --- | --- | --- | --- | --- |\n| 0.944 | 512 | b6 | 2018 | multitask learning - target(2), diagnosis(3), site(6) |\n| 0.942 | 512 | b6 | 2018 | augmentation - mixup |\n\nThanks again.\n", "votes": 1, "replies": [ { "id": 976032, "postDate": "2020-08-18T15:42:29.197Z", "content": "Those are strong models Sogna. Great CV. Congrats on solo silver finish!
", "rawMarkdown": "Those are strong models Sogna. Great CV. Congrats on solo silver finish!", "votes": 1 } ] }, { "id": 975209, "postDate": "2020-08-18T07:41:23.523Z", "content": "Thanks, @cdeotte for your constant guidance throughout this competition.
\nAnd yes, we have also used Random Cropping + More TTA and it did the trick for us also.
\nOnce again, Thank you, Chris. 😀✌🏻
", "rawMarkdown": "Thanks, @cdeotte for your constant guidance throughout this competition. \n\nAnd yes, we have also used **Random Cropping** + **More TTA** and it did the trick for us also. \n\nOnce again, Thank you, Chris. 😀✌🏻", "votes": 1 }, { "id": 975102, "postDate": "2020-08-18T06:39:06.330Z", "content": "Congrats Chris :D Really hope your kernels would be on pytorch someday :(
", "rawMarkdown": "Congrats Chris :D Really hope your kernels would be on pytorch someday :( ", "votes": 1 }, { "id": 975079, "postDate": "2020-08-18T06:16:41.507Z", "content": "Congrats Chris, You are like the organizer of this competition along with Kaggle. Learnt a lot from your discussions, notebooks, dataset.
\nBut I don't get the sub_2. what do you mean by the mixture of public and mine? Public ensembles of the ensemble or just a model?
I never trust public ensemble notebooks in competitions. I mixed some public notebook single models (that i read the code and trust) with my single models by blending public submission.csv files with my submission.csv files.
When blending with public notebooks, you need to be general like weighting everything equally, or weight strong LB as 2 and weak LB as 1. You can't fine tune the weights like you can if you use OOF. Otherwise you will overfit.
", "rawMarkdown": "I never trust public ensemble notebooks in competitions. I mixed some public notebook single models (that i read the code and trust) with my single models by blending public `submission.csv` files with my `submission.csv` files.\n\nWhen blending with public notebooks, you need to be general like weighting everything equally, or weight strong LB as 2 and weak LB as 1. You can't fine tune the weights like you can if you use OOF. Otherwise you will overfit.", "votes": 1 }, { "id": 976110, "postDate": "2020-08-18T16:34:20.610Z", "content": "Sorry for asking that question specifically.
\nI did submissions as you did, I choose the best CV models for my sub_1 and sub_2. But in the third submission, I ensembled sub_1 with a public notebook which is an ensembles of models and that sub_3 gave me the highest private LB.
\nEven after spending months on this competition, I am not proud of myself. I never knew that this sub_3 is going to give me the silver medal. This is not the way for someone to get there first medal.
", "rawMarkdown": "Sorry for asking that question specifically.\n\nI did submissions as you did, I choose the best CV models for my sub_1 and sub_2. But in the third submission, I ensembled sub_1 with a public notebook which is an ensembles of models and that sub_3 gave me the highest private LB. \n\nEven after spending months on this competition, I am not proud of myself. I never knew that this sub_3 is going to give me the silver medal. This is not the way for someone to get there first medal. ", "votes": 1 }, { "id": 977809, "postDate": "2020-08-19T17:59:48.077Z", "content": "Don't feel bad. It takes skill to read the dozens of public notebooks and decide which ones can be trusted. For example, my best sub in Melanoma comp was also using my work plus some public notebooks that I trusted.
\nIn most comps, I use my second final submission as a mix of my work with public notebooks that I trust.
", "rawMarkdown": "Don't feel bad. It takes skill to read the dozens of public notebooks and decide which ones can be trusted. For example, my best sub in Melanoma comp was also using my work plus some public notebooks that I trusted. \n\nIn most comps, I use my second final submission as a mix of my work with public notebooks that I trust.", "votes": 1 }, { "id": 977818, "postDate": "2020-08-19T18:08:29.440Z", "content": "Thank you @cdeotte, you are a 4x GM and more than that a great human being.
", "rawMarkdown": "Thank you @cdeotte, you are a 4x GM and more than that a great human being.", "votes": 1 } ] }, { "id": 975069, "postDate": "2020-08-18T06:12:54.047Z", "content": "Thanks Chris for making this competition so accessible. I like your 'Forward Selection' Ensembling method, I look forward to trying it.
\nA question about your cropping augmentation: You did the cropping at train time (for example) by cropping 512 squares out of 1024 images (i.e., you didn't create a new dataset)? That is a cool idea, I trained 1024 models but it took a minimum of 15 hours (5 notebook runs, one for each fold), I could of got this down to ~5 hours!
\nTrust your CV indeed! I admit I was surprised to jump ~900 places. I had a leak in my CV and only managed to correct it in the last few weeks of the competition - therefore I had very little time for experimentation!
", "rawMarkdown": "Thanks Chris for making this competition so accessible. I like your 'Forward Selection' Ensembling method, I look forward to trying it.\n\nA question about your cropping augmentation: You did the cropping at *train time* (for example) by cropping 512 squares out of 1024 images (i.e., you didn't create a new dataset)? That is a cool idea, I trained 1024 models but it took a minimum of 15 hours (5 notebook runs, one for each fold), I could of got this down to ~5 hours!\n\nTrust your CV indeed! I admit I was surprised to jump ~900 places. I had a leak in my CV and only managed to correct it in the last few weeks of the competition - therefore I had very little time for experimentation!", "votes": 1, "replies": [ { "id": 975090, "postDate": "2020-08-18T06:24:52.060Z", "content": "Yes at train time. I introduced a new variable CROP = ??? in addition to IMAGE_SIZE = ???. The varible IMAGE_SIZE becomes dim in the code below. (This is my popular notebook updated)
if augment:\n if dim!=CROP: img = tf.image.random_crop(img, [CROP, CROP, 3])\n img = transform(img,DIM=CROP)\n img = tf.image.random_flip_left_right(img)\n #img = tf.image.random_hue(img, 0.01)\n img = tf.image.random_saturation(img, 0.7, 1.3)\n img = tf.image.random_contrast(img, 0.8, 1.2)\n img = tf.image.random_brightness(img, 0.1)\nelif dim!=CROP:\n img = tf.image.central_crop(img,CROP/dim)\nElegant, thanks.
", "rawMarkdown": "Elegant, thanks.", "votes": 1 } ] }, { "id": 974829, "postDate": "2020-08-18T04:16:40.907Z", "content": "Congratulations and thanks a lot @cdeotte for your contribution in the competition. Your datasets and notebooks helped a lot of kagglers to get good score overall.
", "rawMarkdown": "Congratulations and thanks a lot @cdeotte for your contribution in the competition. Your datasets and notebooks helped a lot of kagglers to get good score overall.", "votes": 1 }, { "id": 974802, "postDate": "2020-08-18T03:59:57.653Z", "content": "Thank you Chris, You did really amazing contribution to this competition! Our result is largely thanks to you.
\n (I got extra boost to work after watching you at Chai Time DS and Accelerator power hour)
Thanks Janek. 28th postion! Your team did great. Congratulations.
", "rawMarkdown": "Thanks Janek. 28th postion! Your team did great. Congratulations." } ] }, { "id": 974739, "postDate": "2020-08-18T03:17:55.043Z", "content": "Congrats @cdeotte. You did a perfect job in this competition. What is your best single model? And what do you think is the key of this competition besides ensemble and random crop?
", "rawMarkdown": "Congrats @cdeotte. You did a perfect job in this competition. What is your best single model? And what do you think is the key of this competition besides ensemble and random crop?", "votes": 1, "replies": [ { "id": 974751, "postDate": "2020-08-18T03:27:32.903Z", "content": "My best single model is the top row of the table above. It was 512x512 then random crop 384x384, EfficientNetB5 with 2018 external data and all my extra malignant data. It got triple stratified CV 0.936 and LB 0.956
\nAll my models were pretty simple, just Adam optimizer, Ramp up then decay learn schedule, Binary cross entropy loss. And just global average pooling on EffNet backbones. Then for my different models, I just varied the image size, backbone size, external data, and upsample. (So I used same basic model for all).
\nI think it I spent time to make my single model template stronger. Then all my models would be stronger and the ensemble would score higher. I'm curious to see what the top teams did.
", "rawMarkdown": "My best single model is the top row of the table above. It was 512x512 then random crop 384x384, EfficientNetB5 with 2018 external data and all my extra malignant data. It got triple stratified CV 0.936 and LB 0.956\n\nAll my models were pretty simple, just Adam optimizer, Ramp up then decay learn schedule, Binary cross entropy loss. And just global average pooling on EffNet backbones. Then for my different models, I just varied the image size, backbone size, external data, and upsample. (So I used same basic model for all).\n\nI think it I spent time to make my single model template stronger. Then all my models would be stronger and the ensemble would score higher. I'm curious to see what the top teams did.", "votes": 1 } ] }, { "id": 974714, "postDate": "2020-08-18T03:01:42.627Z", "content": "oh snap! we use the same trick. great job Chris! I wish I had more time to explore this idea with other resolutions and random cropping.
", "rawMarkdown": "oh snap! we use the same trick. great job Chris! I wish I had more time to explore this idea with other resolutions and random cropping.", "votes": 1, "replies": [ { "id": 974729, "postDate": "2020-08-18T03:11:04.457Z", "content": "Awesome. It is a great trick. We get all the information in the higher resolution images but we get to train at the speed of the smaller images. Congrats on solo gold Tim.
", "rawMarkdown": "Awesome. It is a great trick. We get all the information in the higher resolution images but we get to train at the speed of the smaller images. Congrats on solo gold Tim.", "votes": 1 } ] }, { "id": 974700, "postDate": "2020-08-18T02:40:44.770Z", "content": "Congrat and thanks for sharing Chris!
\nI notice that using external data 2019 and 2019 in your datasets help alot the model generalized better in private LB.
Congrats chris and thanks for sharing all
", "rawMarkdown": "Congrats chris and thanks for sharing all", "votes": 1 }, { "id": 974666, "postDate": "2020-08-18T02:24:00.003Z", "content": "Congrats and Thank you Chris for your great help thru your published great notebooks and discussions.
", "rawMarkdown": "Congrats and Thank you Chris for your great help thru your published great notebooks and discussions.", "votes": 1 }, { "id": 974621, "postDate": "2020-08-18T02:02:40.127Z", "content": "Thanks for your datasets @cdeotte :) The CV was well built and I was able to jump right away thanks to them and then rush to a silver medal in the last 2 or 3 weeks of the competition.
\nSo it's in part thanks to you ;D
", "rawMarkdown": "Thanks for your datasets @cdeotte :) The CV was well built and I was able to jump right away thanks to them and then rush to a silver medal in the last 2 or 3 weeks of the competition.\n\nSo it's in part thanks to you ;D", "votes": 1, "replies": [ { "id": 974624, "postDate": "2020-08-18T02:03:48.513Z", "content": "Fantastic. Great solo silver. Congrats!
", "rawMarkdown": "Fantastic. Great solo silver. Congrats!" } ] }, { "id": 974611, "postDate": "2020-08-18T02:00:22.467Z", "content": "Cropping this way has been an idea too - but still too busy with the more basic stuff …
\nWanted to have cv oof scores greater than 0.93 - but did not manage most of the time - except using a label smoothing value of 0.01 which seemed not to be a good idea.
\nStill wondering why you did not score better. Maybe metadata ensembling was the problem - which is unexpected…
\nThe second place did not use metadata.. ? !
\nGrats and thanks again.
Thanks Roman. Random cropping worked very well. It allowed to train at small resolution speed but gave the models all the information from high resolution images.
\nI'm happy with my silver finish. To go higher, I think I needed to focus on a optimizing my single model design. If I started with stronger single models, my ensemble LB would have been much higher.
", "rawMarkdown": "Thanks Roman. Random cropping worked very well. It allowed to train at small resolution speed but gave the models all the information from high resolution images.\n\nI'm happy with my silver finish. To go higher, I think I needed to focus on a optimizing my single model design. If I started with stronger single models, my ensemble LB would have been much higher. ", "votes": 1 }, { "id": 975992, "postDate": "2020-08-18T15:11:35.110Z", "content": "Well, Chris, I tried something in this competition and I am not sure whether this is a good idea - so I want to ask you about your opinion:
\nI calculated the oof cv scores and after all folds are done I tried to weight the results to get the best overall oof score and use that weights on the test predictions. Does this make sense?
calculated_weights_preds_test[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTSMIX[fold]
\ninstead of
\nnormally_weighted_preds[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTS[fold]
\n…. I tried late submissions on the results, and the newly weighted results did awful - hmm…
", "rawMarkdown": "Well, Chris, I tried something in this competition and I am not sure whether this is a good idea - so I want to ask you about your opinion:\nI calculated the oof cv scores and after all folds are done I tried to weight the results to get the best overall oof score and use that weights on the test predictions. Does this make sense?\n\ncalculated_weights_preds_test[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTSMIX[fold]\n\ninstead of\n\nnormally_weighted_preds[:,0] += np.mean(pred.reshape((ct_test,TTA),order='F'),axis=1) * WGTS[fold]\n\n.... I tried late submissions on the results, and the newly weighted results did awful - hmm...\n" }, { "id": 976038, "postDate": "2020-08-18T15:46:44.193Z", "content": "I see what you're trying to do. That's an interesting idea.
\nUnfortunately, that won't work because the OOF are not an ensemble. In the OOF, each training image only has 1 prediction. Whereas in the test dataset, each image has 5 different predictions. So the test predictions are an ensemble, but the OOF predictions are just a \"merging\". As such you can't use \"merging\" weights to decide ensemble weights.
", "rawMarkdown": "I see what you're trying to do. That's an interesting idea.\n\nUnfortunately, that won't work because the OOF are not an ensemble. In the OOF, each training image only has 1 prediction. Whereas in the test dataset, each image has 5 different predictions. So the test predictions are an ensemble, but the OOF predictions are just a \"merging\". As such you can't use \"merging\" weights to decide ensemble weights." }, { "id": 976039, "postDate": "2020-08-18T15:46:44.320Z", "content": "I see what you're trying to do. That's an interesting idea.
\nUnfortunately, that won't work because the OOF are not an ensemble. In the OOF, each training image only has 1 prediction. Whereas in the test dataset, each image has 5 different predictions. So the test predictions are an ensemble, but the OOF predictions are just a \"merging\". As such you can't use \"merging\" weights to decide ensemble weights.
", "rawMarkdown": "I see what you're trying to do. That's an interesting idea.\n\nUnfortunately, that won't work because the OOF are not an ensemble. In the OOF, each training image only has 1 prediction. Whereas in the test dataset, each image has 5 different predictions. So the test predictions are an ensemble, but the OOF predictions are just a \"merging\". As such you can't use \"merging\" weights to decide ensemble weights." } ] }, { "id": 974592, "postDate": "2020-08-18T01:49:43.200Z", "content": "Congratualation @cdeotte even with the shakedown it is still a good position, I also had a brute force script to find the best models to ensemble.
\nI am a little curious about your meta-model, did it had anything special? Also, did you saw consistent improvements by loading larget image sizes and cropping to smaller ones instead of just loading the smaller files? I was trying this by the end but did not have much time to get reliable results.
If you load 512x512 and random crop to 256x256. That is not the same as loading 256x256. In the former, the images contain 4x more pixel information. Note that I am not resizing. My model is slowing viewing the entire 512x512 over time.
\nSo loading 512x512 and cropping to 256x256 is the same as training on 512x512. And training on 512x512 always did better than using the smaller image sizes.
\nIf you wish to train on 256x256, then you can load 256x256 and random crop to 128x128. Then your model will train very fast cause it processes 128x128 but it is training on 256x256
", "rawMarkdown": "If you load 512x512 and random crop to 256x256. That is not the same as loading 256x256. In the former, the images contain 4x more pixel information. Note that I am not resizing. My model is slowing viewing the entire 512x512 over time.\n\nSo loading 512x512 and cropping to 256x256 is the same as training on 512x512. And training on 512x512 always did better than using the smaller image sizes.\n\nIf you wish to train on 256x256, then you can load 256x256 and random crop to 128x128. Then your model will train very fast cause it processes 128x128 but it is training on 256x256", "votes": 3 }, { "id": 974741, "postDate": "2020-08-18T03:19:53.663Z", "content": "There was nothing special about my meta model. By itself, it had CV 0.760 and public LB 0.780, so I trusted that it wasn't overfitting train. It increased my CV 0.0005 which could have helped private LB a little but it turned out not to help private LB.
", "rawMarkdown": "There was nothing special about my meta model. By itself, it had CV 0.760 and public LB 0.780, so I trusted that it wasn't overfitting train. It increased my CV 0.0005 which could have helped private LB a little but it turned out not to help private LB.", "votes": 1 }, { "id": 975585, "postDate": "2020-08-18T11:16:41.690Z", "content": "I got it Chris, sounds really interesting, thanks!
", "rawMarkdown": "I got it Chris, sounds really interesting, thanks!" } ] }, { "id": 974562, "postDate": "2020-08-18T01:32:10.817Z", "content": "Hi, thanks for the explanation.
\nOne question, on the ensemble part, when you are calculating the auc_trial which dataset do you use as the validation one? Do you iterate over your folds or did you previously built a validation dataset?
", "rawMarkdown": "Hi, thanks for the explanation. \n\nOne question, on the ensemble part, when you are calculating the auc_trial which dataset do you use as the validation one? Do you iterate over your folds or did you previously built a validation dataset?", "votes": 1, "replies": [ { "id": 974580, "postDate": "2020-08-18T01:42:15.103Z", "content": "When you use 5 KFold, each model makes predictions on every training image. If you look at the output files of my public notebook here, you will see a file named oof.csv. Now imagine that you have two oof files named oof1.csv and oof2.csv. To compute the ensemble score between two models, you do this
df1 = pd.read_csv('oof1.csv')\ndf2 = pd.read_csv('oof2.csv')\nauc = roc_auc_score(df1.target, 0.5 * df1.pred + 0.5 * df2.pred)\nNote that df1.target and df2.target are the same thing. If you wonder what df1.target is, it is the same as
train = pd.read_csv('train.csv')\nprint( train.target.values )\nBut they are in a different order as df1.target (which are ordered by how the TFRecords get read).
Thanks for sharing this insightful nugget of knowledge!
", "rawMarkdown": "Thanks for sharing this insightful nugget of knowledge!", "votes": 1 } ] }, { "id": 974653, "postDate": "2020-08-18T02:18:14.373Z", "content": "Oh nice! I have heard about Out-of-Fold validation but I have never try it. Gonna used in the next one.
\nThanks a lot! I admire your detailed and thoughtful responses.
", "rawMarkdown": "Oh nice! I have heard about Out-of-Fold validation but I have never try it. Gonna used in the next one.\n\nThanks a lot! I admire your detailed and thoughtful responses.", "votes": 1 } ] }, { "id": 974560, "postDate": "2020-08-18T01:31:24.347Z", "content": "Chris, thank you! As happy as I am for the winners, I am also gutted and was secretly rooting for you to get 'Gold'. You're contributions have really helped a lot.
\nAlso, thanks to the splits you provided - it was easier to trust CV and final solution included an ensemble of your public TPU notebook. :)
For what you have shared in this competition, I can say this for you - \"Some men just want to watch the world learn.\"
\nHere is my ensemble based on your ideas (2020+2018+2017 data).
\nSr. No. -Size -Model -OOF_AUC -Public LB -Private LB -Weight
\n1 -192 -B6 -0.906 -0.9323 -0.9218 -2
\n2 -256 -B7 -0.908 -0.9339 -0.9246 -4
\n3 -384 -B6 -0.914 -0.9434 -0.9255 -5
\n4 -512 -B6 -0.909 -0.9499 -0.9205 -4
\n5 -768 -B6 -0.918 -0.9493 -0.9285 -5
Image Ensemble - Random searched the weights for max OOF_AUC.
\nOOF_AUC - 0.9394 Pub. LB - 0.9528 Pri. LB - 0.9370
I trained Gradient Boosted Trees (5 fold CV) with 3 features only (age, sex, location)
\nOOF_AUC - 0.68 Pub. LB - 0.6777 Pri. LB - 0.6639
Final ensemble (0.87 * image ensemble + 0.13 * Gradient Boosted Tree Model) [Experimented with weights]
\nOOF_AUC - 0.9382 Pub. LB - 0.9548 and 87th position on Private LB.
Thanks a ton!!! 😊
\nTrust the CV.
", "rawMarkdown": "For what you have shared in this competition, I can say this for you - **\"Some men just want to watch the world learn.\"** \n\n Here is my ensemble based on your ideas (2020+2018+2017 data).\n**Sr. No. \t-Size \t-Model\t-OOF_AUC\t-Public LB\t-Private LB\t-Weight**\n1\t -192\t -B6\t -0.906\t -0.9323\t -0.9218\t -2\n2\t -256\t -B7\t -0.908\t -0.9339\t -0.9246\t -4\n3\t -384\t -B6\t -0.914\t -0.9434\t -0.9255\t -5\n4\t -512\t -B6\t -0.909\t -0.9499\t -0.9205\t -4\n5\t -768\t -B6\t -0.918\t -0.9493\t -0.9285\t -5\n\nImage Ensemble - Random searched the weights for max OOF_AUC. \nOOF_AUC - 0.9394 Pub. LB - 0.9528 Pri. LB - 0.9370\n\nI trained Gradient Boosted Trees (5 fold CV) with 3 features only (age, sex, location) \nOOF_AUC - 0.68 Pub. LB - 0.6777 Pri. LB - 0.6639\n\nFinal ensemble (0.87 * image ensemble + 0.13 * Gradient Boosted Tree Model) [Experimented with weights]\nOOF_AUC - 0.9382 Pub. LB - 0.9548 and 87th position on Private LB.\n\nThanks a ton!!! 😊\n\n**Trust the CV.**", "votes": 2, "replies": [ { "id": 974900, "postDate": "2020-08-18T04:46:53.500Z", "content": "This is a great job Manjesh. You did perfect. You built diverse models and maximized OOF CV. Then you trained meta model and ensembled it all together.
", "rawMarkdown": "This is a great job Manjesh. You did perfect. You built diverse models and maximized OOF CV. Then you trained meta model and ensembled it all together.", "votes": 1 } ] }, { "id": 974832, "postDate": "2020-08-18T04:18:14.413Z", "content": "Thanks a lot @cdeotte !! Our current position is mostly due to your extensive work during the competition. Also, feels a bit awkward that your position has come just a few places after us, but seriously, we learnt a whole lot from you since this is our very first competition.
", "rawMarkdown": "Thanks a lot @cdeotte !! Our current position is mostly due to your extensive work during the competition. Also, feels a bit awkward that your position has come just a few places after us, but seriously, we learnt a whole lot from you since this is our very first competition. ", "votes": 2, "replies": [ { "id": 974837, "postDate": "2020-08-18T04:21:48.763Z", "content": "@swadeshjana @ritachetadas @nitimohan Your team did fantastic. 62nd out of 3400 teams. Great job.
", "rawMarkdown": "@swadeshjana @ritachetadas @nitimohan Your team did fantastic. 62nd out of 3400 teams. Great job.", "votes": 1 } ] }, { "id": 974636, "postDate": "2020-08-18T02:08:31.793Z", "content": "Great work, @cdeotte. And thanks for your contribution of this competition. Your public kernel and dataset are impressive and helped a lot of the Kagglers.
\nYes, The public LB is very unstable. so what we can do is just do our best to improve our cv.
Wow, i just read that your ensemble CV is 0.960. That's incredible Gary and team. Congrats on building a great model, I look forward to reading your teams' full writeup.
", "rawMarkdown": "Wow, i just read that your ensemble CV is 0.960. That's incredible Gary and team. Congrats on building a great model, I look forward to reading your teams' full writeup.", "votes": 1 } ] }, { "id": 974633, "postDate": "2020-08-18T02:06:49.240Z", "content": "Thanks a lot Chris for raising the standards for everyone across the board - your discussions, datasets, kernels helped everyone perform better and hopefully makes the world a bit safer by detecting melanoma sooner.
", "rawMarkdown": "Thanks a lot Chris for raising the standards for everyone across the board - your discussions, datasets, kernels helped everyone perform better and hopefully makes the world a bit safer by detecting melanoma sooner. ", "votes": 2 }, { "id": 974566, "postDate": "2020-08-18T01:33:41.687Z", "content": "With me, my private increase 627 ranks :)
\nMeta data is really useful. Can you try with this : image_ensemble0.5 + tabular_model(~0.69)0.5 ? (Use your late submit)
", "rawMarkdown": "With me, my private increase 627 ranks :)\n\nMeta data is really useful. Can you try with this : image_ensemble*0.5 + tabular_model(~0.69)*0.5 ? (Use your late submit)\n", "votes": 2, "replies": [ { "id": 974608, "postDate": "2020-08-18T02:00:03.517Z", "content": "If I use my private LB 0.942 model and do 0.5 * image + 0.5 * meta it becomes private 0.943. That is powerful.
Thanks a lot Chris for all your contributions towards this competition. It has helped so many, not only to get a good score but also to learn several interesting things. You are my role model!
", "rawMarkdown": "Thanks a lot Chris for all your contributions towards this competition. It has helped so many, not only to get a good score but also to learn several interesting things. You are my role model!", "votes": 2 }, { "id": 974548, "postDate": "2020-08-18T01:25:36.507Z", "content": "Thanks Chris,
\nIn my eyes u r still the best !! Not just in Melanoma. but in other competitions as well.
\nYour skills & contributions in dataset preparation & answering diligently to our questions is much appreciated !
cheers
\nsid
You have done a lot of work. Thank you! That‘s amazing
", "rawMarkdown": "You have done a lot of work. Thank you! That‘s amazing", "votes": 2 }, { "id": 979217, "postDate": "2020-08-20T17:41:55.183Z", "content": "UPDATE: I posted another discussion here explaining How To CV and How To Ensemble
", "rawMarkdown": "UPDATE: I posted another discussion [here][1] explaining How To CV and How To Ensemble\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614" }, { "id": 974599, "postDate": "2020-08-18T01:54:20.870Z", "content": "thank you so much @cdeotte i am a kaggle novice and i have learned a lot from you!
\ni read all articles that you wrote, and i got a good result
Thanks Statking. I saw your writeup. You did fantastic! Congratulations on your excellent finish.
", "rawMarkdown": "Thanks Statking. I saw your writeup. You did fantastic! Congratulations on your excellent finish." } ] }, { "id": 974583, "postDate": "2020-08-18T01:44:24.133Z", "content": "Thanks for your notebooks and discussions throughout this competition, Chris. This was the first time I've played around with image data let alone first CV competition. I definitely learned a lot and had fun experimenting with the different things that you had described but in pytorch. Thanks!!
", "rawMarkdown": "Thanks for your notebooks and discussions throughout this competition, Chris. This was the first time I've played around with image data let alone first CV competition. I definitely learned a lot and had fun experimenting with the different things that you had described but in pytorch. Thanks!!" }, { "id": 986403, "postDate": "2020-08-26T13:05:11.290Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 974704, "postDate": "2020-08-18T02:45:36.500Z", "rawMarkdown": "", "votes": 4, "isDeleted": true }, { "id": 991209, "postDate": "2020-08-30T08:12:51.570Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing", "votes": 3 }, { "id": 974533, "postDate": "2020-08-18T01:16:34.167Z", "content": "Thanks again for your contribution :)
", "rawMarkdown": "Thanks again for your contribution :)", "votes": 3 }, { "id": 987215, "postDate": "2020-08-27T05:03:23.143Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing ", "votes": 1 }, { "id": 986393, "postDate": "2020-08-26T12:54:35.747Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing", "votes": 1 }, { "id": 980352, "postDate": "2020-08-21T14:29:11.220Z", "content": "Thanks! Very helpful!!
", "rawMarkdown": "Thanks! Very helpful!!", "votes": 1 }, { "id": 979645, "postDate": "2020-08-21T03:03:29.097Z", "content": "Thanks! Very helpful!!
", "rawMarkdown": "Thanks! Very helpful!!", "votes": 1 }, { "id": 979368, "postDate": "2020-08-20T19:29:21.253Z", "content": "Thanks! Very helpful!!
", "rawMarkdown": "Thanks! Very helpful!! ", "votes": 1 }, { "id": 974547, "postDate": "2020-08-18T01:24:11.160Z", "content": "Thanks a lot Chris!
", "rawMarkdown": "Thanks a lot Chris!", "votes": 1 }, { "id": 2896920, "postDate": "2024-06-30T07:21:53.780Z", "content": "Thanks for the contribution .
", "rawMarkdown": "Thanks for the contribution ." } ], "comments": [ { "id": 2905912, "author_name": "Ajay Asaithambi", "author_url": "", "post_date": "2024-07-05T08:25:10.963000", "content": "Does the data include HAM10000 dataset? or is it mutually exclusive?
", "votes": 1, "replies": [] }, { "id": 974870, "author_name": "Μαριος Μιχαηλιδης KazAnova", "author_url": "", "post_date": "2020-08-18T04:35:13.570000", "content": "You were the MVP of this competition @cdeotte . Thank you for all the work that you did, preparing the data and providing useful insight throughout the competition.
", "votes": 7, "replies": [ { "id": 974923, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T04:58:46.733000", "content": "Thanks Kaz. Great job with your team's strong finish. I look forward to reading about your model. What was your CV score?
", "votes": 1, "replies": [] } ] }, { "id": 974613, "author_name": "Rob Mulla", "author_url": "", "post_date": "2020-08-18T02:00:56.807000", "content": "Great work @cdeotte - your idea of cropping larger images to the smaller image size is really smart! As always you were a huge inspiration to many (myself included) and we appreciate all you shared and your enthusiasm for data science!
", "votes": 3, "replies": [ { "id": 974619, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T02:02:25.063000", "content": "Thanks Rob. Congrats to you and your team's great finish.
", "votes": 1, "replies": [] } ] }, { "id": 974648, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-18T02:15:46.030000", "content": "Thanks @cdeotte for all your great contributions.
\nEven though most of my experiments was done on Pytorch. I was solely using your Jpegs datasets and almost all sizes (unless 1024 and 128 :p)
\nSo, you was an inspiration and helped (almost) all competitors in one or another way
@cdeotte
\nOne of the reasons kaggler doesn't use TF because they need to write much stuff from scratch using unfriendly TF syntax. But not only you use it but also you make it more beautiful for this competition. Congratulation and many thanks to you for your wonderful contribution. Those Triple One GM status really suits you, Perfect. Hoping for the last one 😉
I saw the podcast of yours with data science chai on YouTube, oh mine, I thought you're a very moody person, very deep voice, maybe hard attitude … but no, I found you are really cool and open-minded. However, thanks again for everything. And be like this forever, not only awesome in work but also kind as a human.
", "votes": 2, "replies": [ { "id": 974928, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T05:00:15.393000", "content": "Thank you @ipythonx for your kind words.
", "votes": 2, "replies": [] } ] }, { "id": 974597, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2020-08-18T01:52:56.760000", "content": "Your public kernels were tooooo good (not only score but overall pipeline) many people just blended all possible ways and with 3 submissions got more luck. In my opinion you deserve better place for sure and if there would be special medal for mentoring and sharing it would be your gold medal.
", "votes": 2, "replies": [ { "id": 974630, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T02:06:03.910000", "content": "Thanks Konstantin.
", "votes": 1, "replies": [] } ] }, { "id": 991359, "author_name": "Dewei Chen", "author_url": "", "post_date": "2020-08-30T10:51:44.900000", "content": "Thanks for sharing, This is helpful for me.
", "votes": 1, "replies": [] }, { "id": 987858, "author_name": "Bivek Subedi", "author_url": "", "post_date": "2020-08-27T15:21:00.130000", "content": "Thanks for sharing your approach. Very helpful.
", "votes": 1, "replies": [] }, { "id": 983756, "author_name": "Snehal Lokesh", "author_url": "", "post_date": "2020-08-24T15:13:41.017000", "content": "Thanks for sharing your approach and all the baseline kernels you created for reference.
", "votes": 1, "replies": [] }, { "id": 983426, "author_name": "sajwankit", "author_url": "", "post_date": "2020-08-24T09:48:29.173000", "content": "@cdeotte Congratulations Chris and thanks alot for anchoring noobs like me throughout the competition.
", "votes": 1, "replies": [ { "id": 987003, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-26T22:45:17.570000", "content": "Thanks @ankitsajwan . Congratulations on achieving solo silver. Well done.
", "votes": 1, "replies": [] } ] }, { "id": 982816, "author_name": "Atri Saxena", "author_url": "", "post_date": "2020-08-23T17:23:01.660000", "content": "Thanks for sharing your approach. Very helpful.
", "votes": 1, "replies": [] }, { "id": 980679, "author_name": "Baron Smith", "author_url": "", "post_date": "2020-08-21T19:18:09.670000", "content": "Most awesome! I like your tackling the problem from a few different angles. Also, just watched your interview on YouTube. Very inspirational!
", "votes": 1, "replies": [ { "id": 980723, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-21T19:55:34.010000", "content": "Thank you Baron
", "votes": 0, "replies": [] } ] }, { "id": 980082, "author_name": "Tiago Nunes", "author_url": "", "post_date": "2020-08-21T10:19:25.940000", "content": "Interesting!
", "votes": 1, "replies": [] }, { "id": 977954, "author_name": "Gianluca Rossi", "author_url": "", "post_date": "2020-08-19T20:14:10.577000", "content": "Quick question about Crop Augmentation @cdeotte. What is the size of the images used during the validation/test step? Let's assume at training time we are using 512x512 images randomly cropped to 384x384. Are you using for validation and test images the original images of size 512x512, 512x512 randomly cropped to 384x384, or the original image at size 384x384?
", "votes": 1, "replies": [ { "id": 978046, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T22:14:11.397000", "content": "Great question. After training, all OOF predictions use TTA = 21. So the validation OOF predictions are the average of 21 random crops. (Also the test set predictions are the average of 21 random crops).
During training, TF computes the validation loss and validation AUC using a single 384 center crop. (Assuming we're training with 512 and random cropping 384). This is the validation loss used for early stopping or model checkpoint.
", "votes": 2, "replies": [ { "id": 978113, "author_name": "Gianluca Rossi", "author_url": "", "post_date": "2020-08-20T00:48:31.407000", "content": "Got it! Thanks for sharing this trick @cdeotte!
", "votes": 1, "replies": [] } ] }, { "id": 978059, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T22:43:33.690000", "content": "In my popular public notebook here, just change prepare_image() function to the following.
IMG_SIZES = [512]*FOLDS\nCROP = 384\n\ndef prepare_image(img, augment=True, dim=256): \n img = tf.image.decode_jpeg(img, channels=3)\n img = tf.cast(img, tf.float32) / 255.0\n\n if augment:\n if dim!=CROP: img = tf.image.random_crop(img, [CROP, CROP, 3])\n img = transform(img,DIM=CROP)\n img = tf.image.random_flip_left_right(img)\n img = tf.image.random_saturation(img, 0.7, 1.3)\n img = tf.image.random_contrast(img, 0.8, 1.2)\n img = tf.image.random_brightness(img, 0.1)\n elif dim!=CROP:\n img = tf.image.central_crop(img,CROP/dim)\n\n img = tf.reshape(img, [CROP,CROP, 3])\n\n return img\nAnd then when you build the model, use
\nmodel = build_model(dim=CROP, ef=EFF_NETS[fold])\nThanks, you let me quickly learn how to use Tensorflow
", "votes": 1, "replies": [] }, { "id": 977147, "author_name": "XY=ABCD", "author_url": "", "post_date": "2020-08-19T10:18:17.320000", "content": "Congrats for your rank!
", "votes": 1, "replies": [ { "id": 978283, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-20T04:42:13.450000", "content": "Thanks XY=ABCD. Congrats for your bronze medal
", "votes": 0, "replies": [] } ] }, { "id": 976900, "author_name": "Nitesh Chaudhry", "author_url": "", "post_date": "2020-08-19T06:59:57.480000", "content": "The best part, in my opinion is, how easy-to-use & understand you code is !
", "votes": 1, "replies": [] }, { "id": 976299, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T19:20:00.993000", "content": "UPDATE: I posted a starter notebook here showing how to ensemble OOF files using forward selection. I use my 39 of my Melanoma models' OOF files. Forward selection chooses 8 of them and achieves CV 0.950, Public LB 0.958, Private LB 0.942.
", "votes": 1, "replies": [] }, { "id": 976046, "author_name": "RealSid", "author_url": "", "post_date": "2020-08-18T15:51:53.257000", "content": "Thanks mate ! Your CV and public notebook were really nifty !
", "votes": 1, "replies": [] }, { "id": 975813, "author_name": "Salaryman", "author_url": "", "post_date": "2020-08-18T13:30:24.823000", "content": "Thanks Chris! I guess >90% people used your baseline model to experiment for this competition🙌
", "votes": 1, "replies": [] }, { "id": 975806, "author_name": "Eevee", "author_url": "", "post_date": "2020-08-18T13:27:02.233000", "content": "thank for your dataset help me have first medal👍
", "votes": 1, "replies": [] }, { "id": 975325, "author_name": "Mayank Agrawal", "author_url": "", "post_date": "2020-08-18T08:45:17.243000", "content": "Congrats Chris
", "votes": 1, "replies": [] }, { "id": 975311, "author_name": "Mark P", "author_url": "", "post_date": "2020-08-18T08:38:05.150000", "content": "Thanks Chris - it was so kind of you to share all your hard work and I'm sure a large percentage of those in the medal positions got there with your hard work as solid foundation - that was the case for me so thank you!
", "votes": 1, "replies": [ { "id": 983077, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-24T03:12:24.670000", "content": "Thanks Mark. Congrats on finishing 49th solo silver. That's fantastic.
", "votes": 0, "replies": [] } ] }, { "id": 975291, "author_name": "Wonho Song", "author_url": "", "post_date": "2020-08-18T08:27:28.673000", "content": "Thanks for sharing. Thanks to your kernel and dataset, I was able to participate this competition well. I experiment a lot based on your triple stratified CV. Here are some experiment results.
\n| cv | \nimage size | \nmodel | \next | \nmethod | \n
|---|---|---|---|---|
| 0.944 | \n512 | \nb6 | \n2018 | \nmultitask learning - target(2), diagnosis(3), site(6) | \n
| 0.942 | \n512 | \nb6 | \n2018 | \naugmentation - mixup | \n
Thanks again.
", "votes": 1, "replies": [ { "id": 976032, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T15:42:29.197000", "content": "Those are strong models Sogna. Great CV. Congrats on solo silver finish!
", "votes": 1, "replies": [] } ] }, { "id": 975209, "author_name": "Meet Ranoliya", "author_url": "", "post_date": "2020-08-18T07:41:23.523000", "content": "Thanks, @cdeotte for your constant guidance throughout this competition.
\nAnd yes, we have also used Random Cropping + More TTA and it did the trick for us also.
\nOnce again, Thank you, Chris. 😀✌🏻
", "votes": 1, "replies": [] }, { "id": 975102, "author_name": "Haider Ali Shuvo", "author_url": "", "post_date": "2020-08-18T06:39:06.330000", "content": "Congrats Chris :D Really hope your kernels would be on pytorch someday :(
", "votes": 1, "replies": [] }, { "id": 975079, "author_name": "Aptha K S", "author_url": "", "post_date": "2020-08-18T06:16:41.507000", "content": "Congrats Chris, You are like the organizer of this competition along with Kaggle. Learnt a lot from your discussions, notebooks, dataset.
\nBut I don't get the sub_2. what do you mean by the mixture of public and mine? Public ensembles of the ensemble or just a model?
I never trust public ensemble notebooks in competitions. I mixed some public notebook single models (that i read the code and trust) with my single models by blending public submission.csv files with my submission.csv files.
When blending with public notebooks, you need to be general like weighting everything equally, or weight strong LB as 2 and weak LB as 1. You can't fine tune the weights like you can if you use OOF. Otherwise you will overfit.
", "votes": 1, "replies": [] }, { "id": 976110, "author_name": "Aptha K S", "author_url": "", "post_date": "2020-08-18T16:34:20.610000", "content": "Sorry for asking that question specifically.
\nI did submissions as you did, I choose the best CV models for my sub_1 and sub_2. But in the third submission, I ensembled sub_1 with a public notebook which is an ensembles of models and that sub_3 gave me the highest private LB.
\nEven after spending months on this competition, I am not proud of myself. I never knew that this sub_3 is going to give me the silver medal. This is not the way for someone to get there first medal.
", "votes": 1, "replies": [] }, { "id": 977809, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T17:59:48.077000", "content": "Don't feel bad. It takes skill to read the dozens of public notebooks and decide which ones can be trusted. For example, my best sub in Melanoma comp was also using my work plus some public notebooks that I trusted.
\nIn most comps, I use my second final submission as a mix of my work with public notebooks that I trust.
", "votes": 1, "replies": [] }, { "id": 977818, "author_name": "Aptha K S", "author_url": "", "post_date": "2020-08-19T18:08:29.440000", "content": "Thank you @cdeotte, you are a 4x GM and more than that a great human being.
", "votes": 1, "replies": [] } ] }, { "id": 975069, "author_name": "FChmiel", "author_url": "", "post_date": "2020-08-18T06:12:54.047000", "content": "Thanks Chris for making this competition so accessible. I like your 'Forward Selection' Ensembling method, I look forward to trying it.
\nA question about your cropping augmentation: You did the cropping at train time (for example) by cropping 512 squares out of 1024 images (i.e., you didn't create a new dataset)? That is a cool idea, I trained 1024 models but it took a minimum of 15 hours (5 notebook runs, one for each fold), I could of got this down to ~5 hours!
\nTrust your CV indeed! I admit I was surprised to jump ~900 places. I had a leak in my CV and only managed to correct it in the last few weeks of the competition - therefore I had very little time for experimentation!
", "votes": 1, "replies": [ { "id": 975090, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T06:24:52.060000", "content": "Yes at train time. I introduced a new variable CROP = ??? in addition to IMAGE_SIZE = ???. The varible IMAGE_SIZE becomes dim in the code below. (This is my popular notebook updated)
if augment:\n if dim!=CROP: img = tf.image.random_crop(img, [CROP, CROP, 3])\n img = transform(img,DIM=CROP)\n img = tf.image.random_flip_left_right(img)\n #img = tf.image.random_hue(img, 0.01)\n img = tf.image.random_saturation(img, 0.7, 1.3)\n img = tf.image.random_contrast(img, 0.8, 1.2)\n img = tf.image.random_brightness(img, 0.1)\nelif dim!=CROP:\n img = tf.image.central_crop(img,CROP/dim)\nElegant, thanks.
", "votes": 1, "replies": [] } ] }, { "id": 974829, "author_name": "Karan", "author_url": "", "post_date": "2020-08-18T04:16:40.907000", "content": "Congratulations and thanks a lot @cdeotte for your contribution in the competition. Your datasets and notebooks helped a lot of kagglers to get good score overall.
", "votes": 1, "replies": [] }, { "id": 974802, "author_name": "", "author_url": "", "post_date": "2020-08-18T03:59:57.653000", "content": "", "votes": 1, "replies": [ { "id": 974807, "author_name": "", "author_url": "", "post_date": "2020-08-18T04:01:35.990000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 974739, "author_name": "", "author_url": "", "post_date": "2020-08-18T03:17:55.043000", "content": "", "votes": 1, "replies": [ { "id": 974751, "author_name": "", "author_url": "", "post_date": "2020-08-18T03:27:32.903000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 974714, "author_name": "", "author_url": "", "post_date": "2020-08-18T03:01:42.627000", "content": "", "votes": 1, "replies": [ { "id": 974729, "author_name": "", "author_url": "", "post_date": "2020-08-18T03:11:04.457000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 974700, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:40:44.770000", "content": "", "votes": 1, "replies": [] }, { "id": 974673, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:27:16.680000", "content": "", "votes": 1, "replies": [] }, { "id": 974666, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:24:00.003000", "content": "", "votes": 1, "replies": [] }, { "id": 974621, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:02:40.127000", "content": "", "votes": 1, "replies": [ { "id": 974624, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:03:48.513000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 974611, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:00:22.467000", "content": "", "votes": 1, "replies": [ { "id": 974736, "author_name": "", "author_url": "", "post_date": "2020-08-18T03:16:16.480000", "content": "", "votes": 1, "replies": [] }, { "id": 975992, "author_name": "", "author_url": "", "post_date": "2020-08-18T15:11:35.110000", "content": "", "votes": 0, "replies": [] }, { "id": 976038, "author_name": "", "author_url": "", "post_date": "2020-08-18T15:46:44.193000", "content": "", "votes": 0, "replies": [] }, { "id": 976039, "author_name": "", "author_url": "", "post_date": "2020-08-18T15:46:44.320000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 974592, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:49:43.200000", "content": "", "votes": 1, "replies": [ { "id": 974602, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:56:09.420000", "content": "", "votes": 3, "replies": [] }, { "id": 974741, "author_name": "", "author_url": "", "post_date": "2020-08-18T03:19:53.663000", "content": "", "votes": 1, "replies": [] }, { "id": 975585, "author_name": "", "author_url": "", "post_date": "2020-08-18T11:16:41.690000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 974562, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:32:10.817000", "content": "", "votes": 1, "replies": [ { "id": 974580, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:42:15.103000", "content": "", "votes": 3, "replies": [ { "id": 974591, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:49:11.150000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 974653, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:18:14.373000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 974560, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:31:24.347000", "content": "", "votes": 1, "replies": [] }, { "id": 974869, "author_name": "", "author_url": "", "post_date": "2020-08-18T04:34:18.540000", "content": "", "votes": 2, "replies": [ { "id": 974900, "author_name": "", "author_url": "", "post_date": "2020-08-18T04:46:53.500000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 974832, "author_name": "", "author_url": "", "post_date": "2020-08-18T04:18:14.413000", "content": "", "votes": 2, "replies": [ { "id": 974837, "author_name": "", "author_url": "", "post_date": "2020-08-18T04:21:48.763000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 974636, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:08:31.793000", "content": "", "votes": 2, "replies": [ { "id": 974732, "author_name": "", "author_url": "", "post_date": "2020-08-18T03:12:52.657000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 974633, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:06:49.240000", "content": "", "votes": 2, "replies": [] }, { "id": 974566, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:33:41.687000", "content": "", "votes": 2, "replies": [ { "id": 974608, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:00:03.517000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 974550, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:26:05.317000", "content": "", "votes": 2, "replies": [] }, { "id": 974548, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:25:36.507000", "content": "", "votes": 2, "replies": [] }, { "id": 974543, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:22:57.557000", "content": "", "votes": 2, "replies": [] }, { "id": 979217, "author_name": "", "author_url": "", "post_date": "2020-08-20T17:41:55.183000", "content": "", "votes": 0, "replies": [] }, { "id": 974599, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:54:20.870000", "content": "", "votes": 0, "replies": [ { "id": 974614, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:00:58.947000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 974583, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:44:24.133000", "content": "", "votes": 0, "replies": [] }, { "id": 986403, "author_name": "", "author_url": "", "post_date": "2020-08-26T13:05:11.290000", "content": "", "votes": 1, "replies": [] }, { "id": 974704, "author_name": "", "author_url": "", "post_date": "2020-08-18T02:45:36.500000", "content": "", "votes": 4, "replies": [] }, { "id": 991209, "author_name": "", "author_url": "", "post_date": "2020-08-30T08:12:51.570000", "content": "", "votes": 3, "replies": [] }, { "id": 974533, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:16:34.167000", "content": "", "votes": 3, "replies": [] }, { "id": 987215, "author_name": "", "author_url": "", "post_date": "2020-08-27T05:03:23.143000", "content": "", "votes": 1, "replies": [] }, { "id": 986393, "author_name": "", "author_url": "", "post_date": "2020-08-26T12:54:35.747000", "content": "", "votes": 1, "replies": [] }, { "id": 980352, "author_name": "", "author_url": "", "post_date": "2020-08-21T14:29:11.220000", "content": "", "votes": 1, "replies": [] }, { "id": 979645, "author_name": "", "author_url": "", "post_date": "2020-08-21T03:03:29.097000", "content": "", "votes": 1, "replies": [] }, { "id": 979368, "author_name": "", "author_url": "", "post_date": "2020-08-20T19:29:21.253000", "content": "", "votes": 1, "replies": [] }, { "id": 974547, "author_name": "", "author_url": "", "post_date": "2020-08-18T01:24:11.160000", "content": "", "votes": 1, "replies": [] }, { "id": 2896920, "author_name": "", "author_url": "", "post_date": "2024-06-30T07:21:53.780000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2905912": "Does the data include HAM10000 dataset? or is it mutually exclusive?", "974529": "# Melanoma Model Ensemble!\nThank you Kaggle, SIIM, and ISIC for an exciting competition. Thank you Kagglers for wonderful shared content and great discussions! \n\nEarly on I decided to build a large ensemble instead of optimizing a single model. The AUC metric seemed very unstable with this unbalanced dataset and using ensembles, heavy TTA, and crop augmentation helped stabilize it.\n\n# My Final 3 Submissions\nMy main (first) submission was an ensemble that maximized CV where all models used the same [triple stratified leak-free CV][3] with `seed = 42`. (How to CV explained [here][5]). Next I believed public notebooks could add diversity too, so my final 3 submissions were:\n\n* `Sub_1` - 9 models of mine ensemble - CV 0.9505 LB 0.9578 - Private 0.9418\n* `Sub_2` - 5 mine plus 10 public single models - CV ??? LB 0.9662 - Private 0.9425\n* `Sub_3 = 0.75 * Sub_1 + 0.25 * Sub_2` - CV ??? LB 0.9603 - Private 0.9429\n\n# Crop Augmentation\nCrop Augmentation was key to prevent overfitting during training when using external data, upsampling, and large EfficientNet backbones. Crop augmentation also helped stabilize AUC particularly when used in TTA.\n\nPreviously we saw that using different image sizes adds diversity (explained [here][2]). What many people don't realize is that you can use TFRecord sized 512x512 but train on random 256x256 crops (different each epoch). Then training goes fast because your EfficientNet only processes size 256x256 but you are getting features from 512x512 resolution. This allows us to train quickly on large sizes such as 1024x1024 and 768x768 (using crops of 512 and 384 respectively).\n\nIn the picture below, we read the top row from TFRecords, then random crop, then train with the bottom row.\n\n\n\n# My Final Models\n\nThe following 8 models have ensemble CV 0.9500, Public LB 0.9577, and Private LB 0.9420 . Then ensembling meta data with geometric mean: `image_ensemble**0.9 * tabular_model**0.1`, increases CV to 0.9505, Public LB to 0.9578, and Private LB to 0.9418.\n\n \n| CV | LB | read size | crop size | effNet | ext data | upsample |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0.936 | 0.956 | 512 | 384 | B5 | 2018 | 1,1,1,1 |\n| 0.935 | 0.937 | 768 | 512 | B6 | 2019 2018 | 3,3,0,0 |\n| 0.935 | 0.949 | 768 | 512 | B7 | 2018 | 1,1,1,1 |\n| 0.933 | 0.950 | 1024 | 512 | B6 | 2018 | 2,2,2,2 |\n| 0.927 | 0.942 | 768 | 384 | B4 | 2018 | 0,0,0,0 |\n| 0.920 | 0.941 | 512 | 384 | B5 | 2019 2018 | 10,0,0,0 |\n| 0.916 | 0.946 | 384 | 384 | B345 | no | 0,0,0,0 |\n| 0.910 | 0.950 | 384 | 384 | B6 | 2018 | 0,0,0,0 |\n \nThe above models use a variety of different augmentation, losses, optimizers, and learning rate schedules. External data explained [here][6]. Upsample explained [here][7].\n\n# My Training\n\nIf you download my popular notebook [here][1] to your local machine or cloud provider, then you can run the code quickly using multiple GPUs by adding the following one line of code\n\n DEVICE = \"GPU\"\n strategy = tf.distribute.MirroredStrategy()\n\nMost of my models including my most accurate single model with CV 0.936 and LB 0.956 were trained using four Nvidia V100 GPUs. Thank you Nvidia for the use of GPUs!\n \n# Ensemble Pseudo Code\n \nIn the past 2 months, I trained 50+ diverse models. How do we ensemble 50+ models? Train all models using the same triple stratified folds `seed = 42` from my notebook [here][1]. Then to create an ensemble, start with the model that has largest CV and repeatedly try adding one model to increase CV. Whichever one additional model increases the CV the most (and at least 0.0003), keep that model and then iterate through all models again. Repeat this process until CV score stops increasing (by at least 0.0003).\n \n # START ENSEMBLE USING MODEL WITH LARGEST CV\n Repeat until CV does not increase by 0.0003+ :\n # TRY ADDING EVERY MODEL ONE AT A TIME AND REMEMBER \n # HOW MUCH EACH INCREASES THE ENSEMBLE CV SCORE\n for k in range( len(models) ):\n for w in [0.01, 0.02, ..., 0.98, 0.99]:\n # TRY ADDING MODEL k WITH WEIGHT w TO ENSEMBLE\n trial = w * model[k,] + (1-w) * ensemble\n auc_trial = roc_auc_score(true, trial)\n # ADD ONE NEW MODEL TO ENSEMBLE THAT INCREASED CV THE MOST\n # CHECK NEW CV SCORE. IF IT INCREASED REPEAT LOOP\n\n# Ensemble Starter Notebook\nI posted my solution code [here][4] showing how to ensemble OOF files using forward selection. The notebook uses 39 of my Melanoma models' OOF files. Forward selection chooses 8 of them and achieves CV 0.950, Public LB 0.958, Private LB 0.942.\n\n[1]: https://www.kaggle.com/cdeotte/triple-stratified-kfold-with-tfrecords\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/165526\n[4]: https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private\n[5]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614\n[6]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/164910\n[7]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/169139\n", "974870": "You were the MVP of this competition @cdeotte . Thank you for all the work that you did, preparing the data and providing useful insight throughout the competition. ", "974613": "Great work @cdeotte - your idea of cropping larger images to the smaller image size is really smart! As always you were a huge inspiration to many (myself included) and we appreciate all you shared and your enthusiasm for data science!", "974648": "Thanks @cdeotte for all your great contributions. \n\nEven though most of my experiments was done on Pytorch. I was solely using your Jpegs datasets and almost all sizes (unless 1024 and 128 :p)\nSo, you was an inspiration and helped (almost) all competitors in one or another way ", "974896": "@cdeotte \nOne of the reasons kaggler doesn't use `TF` because they need to write much stuff from scratch using unfriendly `TF` syntax. But not only you use it but also you make it more beautiful for this competition. Congratulation and many thanks to you for your wonderful contribution. Those **Triple One GM** status really suits you, Perfect. Hoping for the last one 😉\n\nI saw the podcast of yours with data science chai on YouTube, oh mine, I thought you're a very moody person, very deep voice, maybe hard attitude ... but no, I found you are really cool and open-minded. However, thanks again for everything. And be like this forever, not only awesome in work but also kind as a human. ", "974597": "Your public kernels were tooooo good (not only score but overall pipeline) many people just blended all possible ways and with 3 submissions got more luck. In my opinion you deserve better place for sure and if there would be special medal for mentoring and sharing it would be your gold medal.", "991359": "Thanks for sharing, This is helpful for me.", "987858": "Thanks for sharing your approach. Very helpful.", "983756": "Thanks for sharing your approach and all the baseline kernels you created for reference.", "983426": "@cdeotte Congratulations Chris and thanks alot for anchoring noobs like me throughout the competition.", "982816": "Thanks for sharing your approach. Very helpful.", "980679": "Most awesome! I like your tackling the problem from a few different angles. Also, just watched your interview on YouTube. Very inspirational!", "980082": "Interesting!", "977954": "Quick question about Crop Augmentation @cdeotte. What is the size of the images used during the validation/test step? Let's assume at training time we are using 512x512 images randomly cropped to 384x384. Are you using for validation and test images the original images of size 512x512, 512x512 randomly cropped to 384x384, or the original image at size 384x384?", "977445": "Thanks, you let me quickly learn how to use Tensorflow", "977147": "Congrats for your rank!", "976900": "The best part, in my opinion is, how easy-to-use & understand you code is !", "976299": "UPDATE: I posted a starter notebook [here][1] showing how to ensemble OOF files using forward selection. I use my 39 of my Melanoma models' OOF files. Forward selection chooses 8 of them and achieves CV 0.950, Public LB 0.958, Private LB 0.942.\n\n[1]: https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private", "976046": "Thanks mate ! Your CV and public notebook were really nifty ! ", "975813": "Thanks Chris! I guess >90% people used your baseline model to experiment for this competition🙌", "975806": "thank for your dataset help me have first medal👍", "975325": "Congrats Chris", "975311": "Thanks Chris - it was so kind of you to share all your hard work and I'm sure a large percentage of those in the medal positions got there with your hard work as solid foundation - that was the case for me so thank you!", "975291": "Thanks for sharing. Thanks to your kernel and dataset, I was able to participate this competition well. I experiment a lot based on your triple stratified CV. Here are some experiment results.\n\n| cv | image size | model | ext | method |\n| --- | --- | --- | --- | --- |\n| 0.944 | 512 | b6 | 2018 | multitask learning - target(2), diagnosis(3), site(6) |\n| 0.942 | 512 | b6 | 2018 | augmentation - mixup |\n\nThanks again.\n", "975209": "Thanks, @cdeotte for your constant guidance throughout this competition. \n\nAnd yes, we have also used **Random Cropping** + **More TTA** and it did the trick for us also. \n\nOnce again, Thank you, Chris. 😀✌🏻", "975102": "Congrats Chris :D Really hope your kernels would be on pytorch someday :( ", "975079": "Congrats Chris, You are like the organizer of this competition along with Kaggle. Learnt a lot from your discussions, notebooks, dataset.\nBut I don't get the sub_2. what do you mean by the mixture of public and mine? Public ensembles of the ensemble or just a model?", "975069": "Thanks Chris for making this competition so accessible. I like your 'Forward Selection' Ensembling method, I look forward to trying it.\n\nA question about your cropping augmentation: You did the cropping at *train time* (for example) by cropping 512 squares out of 1024 images (i.e., you didn't create a new dataset)? That is a cool idea, I trained 1024 models but it took a minimum of 15 hours (5 notebook runs, one for each fold), I could of got this down to ~5 hours!\n\nTrust your CV indeed! I admit I was surprised to jump ~900 places. I had a leak in my CV and only managed to correct it in the last few weeks of the competition - therefore I had very little time for experimentation!", "974829": "Congratulations and thanks a lot @cdeotte for your contribution in the competition. Your datasets and notebooks helped a lot of kagglers to get good score overall.", "974802": "Thank you Chris, You did really amazing contribution to this competition! Our result is largely thanks to you.\n (I got extra boost to work after watching you at Chai Time DS and Accelerator power hour) \n", "974739": "Congrats @cdeotte. You did a perfect job in this competition. What is your best single model? And what do you think is the key of this competition besides ensemble and random crop?", "974714": "oh snap! we use the same trick. great job Chris! I wish I had more time to explore this idea with other resolutions and random cropping.", "974700": "Congrat and thanks for sharing Chris!\nI notice that using external data 2019 and 2019 in your datasets help alot the model generalized better in private LB.", "974673": "Congrats chris and thanks for sharing all", "974666": "Congrats and Thank you Chris for your great help thru your published great notebooks and discussions.", "974621": "Thanks for your datasets @cdeotte :) The CV was well built and I was able to jump right away thanks to them and then rush to a silver medal in the last 2 or 3 weeks of the competition.\n\nSo it's in part thanks to you ;D", "974611": "Cropping this way has been an idea too - but still too busy with the more basic stuff ...\nWanted to have cv oof scores greater than 0.93 - but did not manage most of the time - except using a label smoothing value of 0.01 which seemed not to be a good idea.\nStill wondering why you did not score better. Maybe metadata ensembling was the problem - which is unexpected...\nThe second place did not use metadata.. ? !\nGrats and thanks again.", "974592": "Congratualation @cdeotte even with the shakedown it is still a good position, I also had a brute force script to find the best models to ensemble.\nI am a little curious about your meta-model, did it had anything special? Also, did you saw consistent improvements by loading larget image sizes and cropping to smaller ones instead of just loading the smaller files? I was trying this by the end but did not have much time to get reliable results.", "974562": "Hi, thanks for the explanation. \n\nOne question, on the ensemble part, when you are calculating the auc_trial which dataset do you use as the validation one? Do you iterate over your folds or did you previously built a validation dataset?", "974560": "Chris, thank you! As happy as I am for the winners, I am also gutted and was secretly rooting for you to get 'Gold'. You're contributions have really helped a lot. \nAlso, thanks to the splits you provided - it was easier to trust CV and final solution included an ensemble of your public TPU notebook. :)", "974869": "For what you have shared in this competition, I can say this for you - **\"Some men just want to watch the world learn.\"** \n\n Here is my ensemble based on your ideas (2020+2018+2017 data).\n**Sr. No. \t-Size \t-Model\t-OOF_AUC\t-Public LB\t-Private LB\t-Weight**\n1\t -192\t -B6\t -0.906\t -0.9323\t -0.9218\t -2\n2\t -256\t -B7\t -0.908\t -0.9339\t -0.9246\t -4\n3\t -384\t -B6\t -0.914\t -0.9434\t -0.9255\t -5\n4\t -512\t -B6\t -0.909\t -0.9499\t -0.9205\t -4\n5\t -768\t -B6\t -0.918\t -0.9493\t -0.9285\t -5\n\nImage Ensemble - Random searched the weights for max OOF_AUC. \nOOF_AUC - 0.9394 Pub. LB - 0.9528 Pri. LB - 0.9370\n\nI trained Gradient Boosted Trees (5 fold CV) with 3 features only (age, sex, location) \nOOF_AUC - 0.68 Pub. LB - 0.6777 Pri. LB - 0.6639\n\nFinal ensemble (0.87 * image ensemble + 0.13 * Gradient Boosted Tree Model) [Experimented with weights]\nOOF_AUC - 0.9382 Pub. LB - 0.9548 and 87th position on Private LB.\n\nThanks a ton!!! 😊\n\n**Trust the CV.**", "974832": "Thanks a lot @cdeotte !! Our current position is mostly due to your extensive work during the competition. Also, feels a bit awkward that your position has come just a few places after us, but seriously, we learnt a whole lot from you since this is our very first competition. ", "974636": "Great work, @cdeotte. And thanks for your contribution of this competition. Your public kernel and dataset are impressive and helped a lot of the Kagglers. \nYes, The public LB is very unstable. so what we can do is just do our best to improve our cv.", "974633": "Thanks a lot Chris for raising the standards for everyone across the board - your discussions, datasets, kernels helped everyone perform better and hopefully makes the world a bit safer by detecting melanoma sooner. ", "974566": "With me, my private increase 627 ranks :)\n\nMeta data is really useful. Can you try with this : image_ensemble*0.5 + tabular_model(~0.69)*0.5 ? (Use your late submit)\n", "974550": "Thanks a lot Chris for all your contributions towards this competition. It has helped so many, not only to get a good score but also to learn several interesting things. You are my role model!", "974548": "Thanks Chris,\nIn my eyes u r still the best !! Not just in Melanoma. but in other competitions as well. \nYour skills & contributions in dataset preparation & answering diligently to our questions is much appreciated !\n\ncheers\nsid", "974543": "You have done a lot of work. Thank you! That‘s amazing", "979217": "UPDATE: I posted another discussion [here][1] explaining How To CV and How To Ensemble\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614", "974599": "thank you so much @cdeotte i am a kaggle novice and i have learned a lot from you!\ni read all articles that you wrote, and i got a good result", "974583": "Thanks for your notebooks and discussions throughout this competition, Chris. This was the first time I've played around with image data let alone first CV competition. I definitely learned a lot and had fun experimenting with the different things that you had described but in pytorch. Thanks!!", "986403": "", "974704": "", "991209": "thanks for sharing", "974533": "Thanks again for your contribution :)", "987215": "Thanks for sharing ", "986393": "Thanks for sharing", "980352": "Thanks! Very helpful!!", "979645": "Thanks! Very helpful!!", "979368": "Thanks! Very helpful!! ", "974547": "Thanks a lot Chris!", "2896920": "Thanks for the contribution ." } }