\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1120704%2Fae8a6c8597a1f1ab4649fc72627c9378%2Fmeta_NN.png?generation=1597734243315450&alt=media\")
Congratulations to all winners, especially my teammates Qishen @haqishen and Gary @garybios who are both computer vision competition veterans. I learned a lot from you. So happy to see you both on top of another LB after the deepfake drama.
\nBig shoutout to my colleague Chris @cdeotte who I feel like served as an unofficial host of this competition with all your datasets, notebooks and tutorials. The competition wouldn't be this popular with all your contribution. We used your resized jpg images and triple stratified leak-free folds. Thanks Chris.
\nThe test set is very small, with very small proportion of positive samples. So public LB has huge variance. Even the 2020 train data is not big enough for validation purpose, due to its small positive samples.
\nTo have stable validation, we used 2018+2019+2020's data for both train and validation. We track two cv scores, cv_all and cv_2020. The former is much more stable than the latter.
Second key to LB survival is ensemble. Our single model's LB-cv correlation is essential 0, but the bigger the ensemble, the more stable the LB. In the last few days, our ensemble's LB was steadily increasing as we added better models.
\nOur final ensemble 1 optimizes cv_all and final ensemble 2 optimizes cv_2020
Ensemble 1: cv_all=0.9845, cv_2020=0.9600, public=0.9442, private=0.9490 (1st place)
\nEnsemble 2: cv_2020=0.9638, public=0.9494, private=0.9481 (3rd place)
Our best single model has cv_2020=0.9481
All scores above are 5 fold, TTAx8.
\nTPU with TF seems to be dominating the public notebooks in this competition. It seems to be faster than torch/GPU on EfficientNets. However, our experiments showed that its high LB scores are due to better luck with the public LB. Our torch/GPU models have much better CV, thanks to the better PyTorch ecosystem and flexibility for faster experiments.
\nOur ensembles consists of EfficientNet B3-B7, se_resnext101, resnest101. There are models with or without meta data. Input size ranges from 384 to 896. (All input are from Chris's resized jpgs. For example, for 896 input we read 1024 jpgs and resize to 896.)
\nIn some (not all) of our models, we used 14 meta data from this and this public notebooks, as illustrated below.
\n
We find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01.
\n2020 data's diagnosis sets and 2019 data's diagnosis sets are different. We manually map 2020's to 2019's. There are 9 labels.
\n2020 seborrheic keratosis -> BKL\n2020 lichenoid keratosis -> BKL\n2020 solar lentigo -> BKL\n2020 lentigo NOS -> BKL\n2020 cafe-au-lait macule -> unknown\n2020 atypical melanocytic proliferation -> unknown\n2020 nevus -> NV\n2020 melanoma -> MEL\nFor prediction, we simply take the MEL class's softmax probability.
transforms_train = A.Compose([\n A.Transpose(p=0.5),\n A.VerticalFlip(p=0.5),\n A.HorizontalFlip(p=0.5),\n A.RandomBrightness(limit=0.2, p=0.75),\n A.RandomContrast(limit=0.2, p=0.75),\n A.OneOf([\n A.MotionBlur(blur_limit=5),\n A.MedianBlur(blur_limit=5),\n A.GaussianBlur(blur_limit=5),\n A.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n\n A.OneOf([\n A.OpticalDistortion(distort_limit=1.0),\n A.GridDistortion(num_steps=5, distort_limit=1.),\n A.ElasticTransform(alpha=3),\n ], p=0.7),\n\n A.CLAHE(clip_limit=4.0, p=0.7),\n A.HueSaturationValue(hue_shift_limit=10, sat_shift_limit=20, val_shift_limit=10, p=0.5),\n A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n A.Resize(image_size, image_size),\n A.Cutout(max_h_size=int(image_size * 0.375), max_w_size=int(image_size * 0.375), num_holes=1, p=0.7), \n A.Normalize()\n])\n\ntransforms_val = A.Compose([\n A.Resize(image_size, image_size),\n A.Normalize()\n])\nWhen ensembling different folds, or different models, we first rank all the probabilities of each model/fold, to ensure they are evenly distributed. In pandas, it can be done by df['pred'] = df['pred'].rank(pct=True)
https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution
", "messages": [ { "id": 975018, "postDate": "2020-08-18T05:46:18.767Z", "content": "Congratulations to all winners, especially my teammates Qishen @haqishen and Gary @garybios who are both computer vision competition veterans. I learned a lot from you. So happy to see you both on top of another LB after the deepfake drama.
\nBig shoutout to my colleague Chris @cdeotte who I feel like served as an unofficial host of this competition with all your datasets, notebooks and tutorials. The competition wouldn't be this popular with all your contribution. We used your resized jpg images and triple stratified leak-free folds. Thanks Chris.
\nThe test set is very small, with very small proportion of positive samples. So public LB has huge variance. Even the 2020 train data is not big enough for validation purpose, due to its small positive samples.
\nTo have stable validation, we used 2018+2019+2020's data for both train and validation. We track two cv scores, cv_all and cv_2020. The former is much more stable than the latter.
Second key to LB survival is ensemble. Our single model's LB-cv correlation is essential 0, but the bigger the ensemble, the more stable the LB. In the last few days, our ensemble's LB was steadily increasing as we added better models.
\nOur final ensemble 1 optimizes cv_all and final ensemble 2 optimizes cv_2020
Ensemble 1: cv_all=0.9845, cv_2020=0.9600, public=0.9442, private=0.9490 (1st place)
\nEnsemble 2: cv_2020=0.9638, public=0.9494, private=0.9481 (3rd place)
Our best single model has cv_2020=0.9481
All scores above are 5 fold, TTAx8.
\nTPU with TF seems to be dominating the public notebooks in this competition. It seems to be faster than torch/GPU on EfficientNets. However, our experiments showed that its high LB scores are due to better luck with the public LB. Our torch/GPU models have much better CV, thanks to the better PyTorch ecosystem and flexibility for faster experiments.
\nOur ensembles consists of EfficientNet B3-B7, se_resnext101, resnest101. There are models with or without meta data. Input size ranges from 384 to 896. (All input are from Chris's resized jpgs. For example, for 896 input we read 1024 jpgs and resize to 896.)
\nIn some (not all) of our models, we used 14 meta data from this and this public notebooks, as illustrated below.
\n
We find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01.
\n2020 data's diagnosis sets and 2019 data's diagnosis sets are different. We manually map 2020's to 2019's. There are 9 labels.
\n2020 seborrheic keratosis -> BKL\n2020 lichenoid keratosis -> BKL\n2020 solar lentigo -> BKL\n2020 lentigo NOS -> BKL\n2020 cafe-au-lait macule -> unknown\n2020 atypical melanocytic proliferation -> unknown\n2020 nevus -> NV\n2020 melanoma -> MEL\nFor prediction, we simply take the MEL class's softmax probability.
transforms_train = A.Compose([\n A.Transpose(p=0.5),\n A.VerticalFlip(p=0.5),\n A.HorizontalFlip(p=0.5),\n A.RandomBrightness(limit=0.2, p=0.75),\n A.RandomContrast(limit=0.2, p=0.75),\n A.OneOf([\n A.MotionBlur(blur_limit=5),\n A.MedianBlur(blur_limit=5),\n A.GaussianBlur(blur_limit=5),\n A.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n\n A.OneOf([\n A.OpticalDistortion(distort_limit=1.0),\n A.GridDistortion(num_steps=5, distort_limit=1.),\n A.ElasticTransform(alpha=3),\n ], p=0.7),\n\n A.CLAHE(clip_limit=4.0, p=0.7),\n A.HueSaturationValue(hue_shift_limit=10, sat_shift_limit=20, val_shift_limit=10, p=0.5),\n A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n A.Resize(image_size, image_size),\n A.Cutout(max_h_size=int(image_size * 0.375), max_w_size=int(image_size * 0.375), num_holes=1, p=0.7), \n A.Normalize()\n])\n\ntransforms_val = A.Compose([\n A.Resize(image_size, image_size),\n A.Normalize()\n])\nWhen ensembling different folds, or different models, we first rank all the probabilities of each model/fold, to ensure they are evenly distributed. In pandas, it can be done by df['pred'] = df['pred'].rank(pct=True)
https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution
", "rawMarkdown": "Congratulations to all winners, especially my teammates Qishen @haqishen and Gary @garybios who are both computer vision competition veterans. I learned a lot from you. So happy to see you both on top of another LB after the deepfake drama.\n\nBig shoutout to my colleague Chris @cdeotte who I feel like served as an unofficial host of this competition with all your datasets, notebooks and tutorials. The competition wouldn't be this popular with all your contribution. We used your resized jpg images and triple stratified leak-free folds. Thanks Chris.\n\n## How we survived the shake\nThe test set is very small, with very small proportion of positive samples. So public LB has huge variance. Even the 2020 train data is not big enough for validation purpose, due to its small positive samples. \n\nTo have stable validation, we used 2018+2019+2020's data for both train and **validation**. We track two cv scores, `cv_all` and `cv_2020`. The former is much more stable than the latter.\n\nSecond key to LB survival is ensemble. Our single model's LB-cv correlation is essential 0, but the bigger the ensemble, the more stable the LB. In the last few days, our ensemble's LB was steadily increasing as we added better models.\n\nOur final ensemble 1 optimizes `cv_all` and final ensemble 2 optimizes `cv_2020`\n\nEnsemble 1: `cv_all=0.9845, cv_2020=0.9600, public=0.9442, private=0.9490` (1st place)\nEnsemble 2: `cv_2020=0.9638, public=0.9494, private=0.9481` (3rd place)\n\nOur best single model has `cv_2020=0.9481`\n\nAll scores above are 5 fold, TTAx8.\n\n## TPU vs GPU, TF vs torch\nTPU with TF seems to be dominating the public notebooks in this competition. It seems to be faster than torch/GPU on EfficientNets. However, our experiments showed that its high LB scores are due to better luck with the public LB. Our torch/GPU models have much better CV, thanks to the better PyTorch ecosystem and flexibility for faster experiments.\n\n## Models\nOur ensembles consists of EfficientNet B3-B7, se_resnext101, resnest101. There are models with or without meta data. Input size ranges from 384 to 896. (All input are from Chris's resized jpgs. For example, for 896 input we read 1024 jpgs and resize to 896.) \n\n## Meta data\nIn some (not all) of our models, we used 14 meta data from [this](https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet) and [this](https://www.kaggle.com/awsaf49/xgboost-tabular-data-ml-cv-85-lb-787#Image-Size) public notebooks, as illustrated below.\n\n\n## Targets\nWe find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01. \n\n2020 data's diagnosis sets and 2019 data's diagnosis sets are different. We manually map 2020's to 2019's. There are 9 labels.\n```\n2020 seborrheic keratosis -> BKL\n2020 lichenoid keratosis -> BKL\n2020 solar lentigo -> BKL\n2020 lentigo NOS -> BKL\n2020 cafe-au-lait macule -> unknown\n2020 atypical melanocytic proliferation -> unknown\n2020 nevus -> NV\n2020 melanoma -> MEL\n```\nFor prediction, we simply take the `MEL` class's softmax probability.\n\n## Augmentations\n```\ntransforms_train = A.Compose([\n A.Transpose(p=0.5),\n A.VerticalFlip(p=0.5),\n A.HorizontalFlip(p=0.5),\n A.RandomBrightness(limit=0.2, p=0.75),\n A.RandomContrast(limit=0.2, p=0.75),\n A.OneOf([\n A.MotionBlur(blur_limit=5),\n A.MedianBlur(blur_limit=5),\n A.GaussianBlur(blur_limit=5),\n A.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n\n A.OneOf([\n A.OpticalDistortion(distort_limit=1.0),\n A.GridDistortion(num_steps=5, distort_limit=1.),\n A.ElasticTransform(alpha=3),\n ], p=0.7),\n\n A.CLAHE(clip_limit=4.0, p=0.7),\n A.HueSaturationValue(hue_shift_limit=10, sat_shift_limit=20, val_shift_limit=10, p=0.5),\n A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n A.Resize(image_size, image_size),\n A.Cutout(max_h_size=int(image_size * 0.375), max_w_size=int(image_size * 0.375), num_holes=1, p=0.7), \n A.Normalize()\n])\n\ntransforms_val = A.Compose([\n A.Resize(image_size, image_size),\n A.Normalize()\n])\n```\n\n## Post processing\nWhen ensembling different folds, or different models, we first rank all the probabilities of each model/fold, to ensure they are evenly distributed. In pandas, it can be done by `df['pred'] = df['pred'].rank(pct=True)`\n\n## Code\nhttps://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution", "votes": 383 }, { "id": 975841, "postDate": "2020-08-18T13:46:00.657Z", "content": "Congrats @haqishen @boliu0 @garybios and thanks for sharing. Looks like your validation discipline and labeling strategy made the difference. I'm glad to see that pytorch prevailed.
", "rawMarkdown": "Congrats @haqishen @boliu0 @garybios and thanks for sharing. Looks like your validation discipline and labeling strategy made the difference. I'm glad to see that pytorch prevailed.", "votes": 17, "replies": [ { "id": 976620, "postDate": "2020-08-19T01:43:29.457Z", "content": "pytorch + GPUs ;)
", "rawMarkdown": "pytorch + GPUs ;)", "votes": 7 } ] }, { "id": 975329, "postDate": "2020-08-18T08:49:12.200Z", "content": "Congratulations! Very proud to see a NVIDIA colleague win!
\nYou made my first regret: I didn't think of using diagnosis as target. I'm tempted to retrain my bes model with it… I did use all 2019 targets, and it helped me.
\nYour idea of using a global CV is also very interesting. Whatever the mean, having a reliable CV is the key. you found it. Again, congrats.
\nYou did not say much about your models. You just used models as is with the right number of classes?
", "rawMarkdown": "Congratulations! Very proud to see a NVIDIA colleague win! \n\nYou made my first regret: I didn't think of using diagnosis as target. I'm tempted to retrain my bes model with it... I did use all 2019 targets, and it helped me.\n\nYour idea of using a global CV is also very interesting. Whatever the mean, having a reliable CV is the key. you found it. Again, congrats.\n\nYou did not say much about your models. You just used models as is with the right number of classes?", "votes": 9, "replies": [ { "id": 975342, "postDate": "2020-08-18T08:55:31.613Z", "content": "@cpmpml hi, uncle.
\nyes, we didn't do a lot of work on the modification of model, but simply replaced the FC head and added dropout
Thanks. Yeah the model is pretty vanilla. We have posted our code here: https://www.kaggle.com/haqishen/1st-place-soluiton-code-small-ver#Define-Dataset
", "rawMarkdown": "Thanks. Yeah the model is pretty vanilla. We have posted our code here: https://www.kaggle.com/haqishen/1st-place-soluiton-code-small-ver#Define-Dataset", "votes": 1 } ] }, { "id": 975338, "postDate": "2020-08-18T08:54:49.363Z", "content": "One thing to add regarding diagnosis.
\nWe found out the following thing:
\ndiagnosis=nevus from train (keep them in val)diagnosis=unknown Anyone figured this out? Maybe what it means? I was thinking that those labeled explicitly as nevus are more correct? Is there another bias? Unfortunately did not have too much time to follow up on it, but I expect that having the extra targets helps also with that a bit (we partly also did it, but did not see that much of improvements as you did).
", "rawMarkdown": "One thing to add regarding diagnosis.\n\nWe found out the following thing:\n\n- Only training and evaluating on 2020 data\n- Removing all `diagnosis=nevus` from train (keep them in val)\n- Score drops a lot!\n- Remove similar amount of `diagnosis=unknown` \n- Score does not change (only tiny random range).\n\nAnyone figured this out? Maybe what it means? I was thinking that those labeled explicitly as nevus are more correct? Is there another bias? Unfortunately did not have too much time to follow up on it, but I expect that having the extra targets helps also with that a bit (we partly also did it, but did not see that much of improvements as you did).", "votes": 6, "replies": [ { "id": 977573, "postDate": "2020-08-19T15:08:31.543Z", "content": "I did a similar experiment :
\nAnyone noticed this?
", "rawMarkdown": "I did a similar experiment :\n- train a first model\n- look at your oof errors\n- if you get rid of all benign lesions with error >0.2 and and all malignant with error > 0.95 then I got a +0.002~+0.003 improvement in CV (not removing anything from the validation set only in the training set)\n- now what happens if you try to train only on \"hard examples\" above ? My validation scores where around AUC=0.35. So the model can't learn anything, it learns something that is actually misleading (auc << 0.5) . I could not tell whether they were just indistinguishile examples or if they could be some mislabelled examples.\n\nAnyone noticed this?", "votes": 1 }, { "id": 977968, "postDate": "2020-08-19T20:29:16.663Z", "content": "Yes, we also did that, we also fittet on 2020 and predicted 2019 and then removed hard samples, this improves CV quite a lot, but is a form of leak as you use models trained on the data to remove other data, similar as in pseudo tagging.
", "rawMarkdown": "Yes, we also did that, we also fittet on 2020 and predicted 2019 and then removed hard samples, this improves CV quite a lot, but is a form of leak as you use models trained on the data to remove other data, similar as in pseudo tagging.", "votes": 2 }, { "id": 978100, "postDate": "2020-08-20T00:13:13.053Z", "content": "Don’t know how you got a downvote for this answer ^^
\nIf I just remove training examples and don’t change validation I don’t see where the leak comes into play. It felt like an inverse focal loss, you focus only on easy examples and don’t take hard examples in your loss.
\nIt’s an unusual approach and I feel it can work only if there is something wrong with labels.
Great work =))
", "rawMarkdown": "Great work =))", "votes": 1 }, { "id": 975165, "postDate": "2020-08-18T07:19:55.047Z", "content": "Happy that Pytorch won, congrats!
\nOnly figured out on the last day that taking CV across all data is better.
", "rawMarkdown": "Happy that Pytorch won, congrats!\n\nOnly figured out on the last day that taking CV across all data is better.", "votes": 3, "replies": [ { "id": 975188, "postDate": "2020-08-18T07:30:29.187Z", "content": "@philippsinger I am sorry that you don't have enough time to do the new experiment.
\nyes, In the early stage of our experiment, our same pipeline and parameters can obtained different CV_2020, so we also doubt the stability of CV_2020. There are more pos samples in 2018/19/20. We believe that this CV_all will be more stable and guarantee the generalization.
Yeah, it makes sense. Overall our CV / LB correlation was quite good, but it got a bit messy when reaching higher CV scores. We were a bit too cautious with some image distribution differences between the datasets, but putting it all together here also makes sense. Good job!
", "rawMarkdown": "Yeah, it makes sense. Overall our CV / LB correlation was quite good, but it got a bit messy when reaching higher CV scores. We were a bit too cautious with some image distribution differences between the datasets, but putting it all together here also makes sense. Good job!", "votes": 1 }, { "id": 975249, "postDate": "2020-08-18T08:05:40.970Z", "content": "In fact I imply that days ago but seems no one trust me ;)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F448347%2Fc77d614cf0133841f857b0aa5fdfd819%2Fimage%20(8).png?generation=1597737914311803&alt=media\")
Do you automatically win another $10,000? Your model uses contextual information. You add the feature 0 * patient_id and your model does not use contextual information. So you win both additional $5,000 prizes.
\n\nSpecial Prizes: Awarded to the top scoring models using or not using patient-level contextual information.
\n
With Context - $5,000 (Top-scoring model making use of patient-level contextual information)\nWithout Context - $5,000 (Top-scoring model without using any patient-level contextual information)\nWell, seems we can only automatically get half of it ;)
\n(Still feel confuse on what is patient-level contextual information)
Training with diagnosis then using MEL class softmax is a great idea. Congratulations on an amazing model and 1st place!
Hi Chris, based on your abundant experience, does it mean more classes (using diagnosis as @boliu0 did) may help model learn better than binary class (using target as we normally did) on such highly-imbalanced data?
Congratulations on your 1st ! and thank you for the write up!
\nI was surprised that you used 2020+2019+2018 for validation. A lot of kernels and my experiments immediately overfit in this case, achieving 0.99CV.
\nI have questions about some of the fundamentals of making training stable. I took a lot of time here and didn't get to the more substantive considerations.
Do you remove the questionable augmentations one by one from the base transform with all the augmentations, or do you use something that's almost decided from the beginning and rarely changes?
\nOf course why didn't you use the method of putting in more small holes like dropout, or overlapping images like mixup?
You need to adjust the lr when you change (models, datasets, lossfunction) small to large. I'm having a hard time finding just the right lr, scheduler, epoch for every competition, could you please tell me the procedure for adjusting lr?
\nAssuming we use cosine annealing, how do we determine the max lr, train epoch?
Did you use spearman corr for selecting models or try stacking ?
\nIn the case of ensembling different models, we can just rank ensemble, but I didn't know that we can also use rank averaging the same model's different fold.
\nIn general, the outputs of the same model's different fold have similar distribution, we often average the outputs of the model as it is. However, in this task, the test output distribution for each fold may be very different. I did not know how to deal with this problem. Thanks!
I'm sorry for many questions, poor english. I would really appreciate it if you could respond to the questions. Once again, congratulations on your 1st !!!
", "rawMarkdown": "Congratulations on your 1st ! and thank you for the write up!\n\nI was surprised that you used 2020+2019+2018 for validation. A lot of kernels and my experiments immediately overfit in this case, achieving 0.99CV.\nI have questions about some of the fundamentals of making training stable. I took a lot of time here and didn't get to the more substantive considerations.\n\n- How do you coordinate augmentation?\n\nDo you remove the questionable augmentations one by one from the base transform with all the augmentations, or do you use something that's almost decided from the beginning and rarely changes?\nOf course why didn't you use the method of putting in more small holes like dropout, or overlapping images like mixup?\n\n- What did you use for optimizer, schedular? Did you learn the larger model from the beginning?\n\nYou need to adjust the lr when you change (models, datasets, lossfunction) small to large. I'm having a hard time finding just the right lr, scheduler, epoch for every competition, could you please tell me the procedure for adjusting lr?\nAssuming we use cosine annealing, how do we determine the max lr, train epoch?\n\n- Is ensemble optimization a baysian weight optimization using oof?\n\nDid you use spearman corr for selecting models or try stacking ?\n\n---\n\nIn the case of ensembling different models, we can just rank ensemble, but I didn't know that we can also use rank averaging the same model's different fold.\nIn general, the outputs of the same model's different fold have similar distribution, we often average the outputs of the model as it is. However, in this task, the test output distribution for each fold may be very different. I did not know how to deal with this problem. Thanks!\n\nI'm sorry for many questions, poor english. I would really appreciate it if you could respond to the questions. Once again, congratulations on your 1st !!!", "votes": 4, "replies": [ { "id": 975300, "postDate": "2020-08-18T08:31:09.330Z", "content": "Thanks!
\n\n\nI was surprised that you used 2020+2019+2018 for validation. A lot of kernels and my experiments immediately overfit in this case, achieving 0.99CV.
\n
You may have a leak on your data split. We cannot get over cv0.98 when validating on 2020+2019+2018 for a single model.
\n\n\nHow do you coordinate augmentation?
\n
We tune the parameter one by one to find the best combinations. Then we fix it till the end of the competition.
\n\n\nWhat did you use for optimizer, schedular? Did you learn the larger model from the beginning?
\n
We tried some different optimizer and schedular but see no much difference. We use warmup + cosine w/ Adam till the end of the competition.
\n\n\nIs ensemble optimization a baysian weight optimization using oof?
\n
No. we just take a simple avg of those oof or test files after ranking seperately. No level 2 model.
", "rawMarkdown": "Thanks!\n\n> I was surprised that you used 2020+2019+2018 for validation. A lot of kernels and my experiments immediately overfit in this case, achieving 0.99CV.\n\nYou may have a leak on your data split. We cannot get over cv0.98 when validating on 2020+2019+2018 for a single model.\n\n> How do you coordinate augmentation?\n\nWe tune the parameter one by one to find the best combinations. Then we fix it till the end of the competition.\n\n> What did you use for optimizer, schedular? Did you learn the larger model from the beginning?\n\nWe tried some different optimizer and schedular but see no much difference. We use warmup + cosine w/ Adam till the end of the competition.\n\n> Is ensemble optimization a baysian weight optimization using oof?\n\nNo. we just take a simple avg of those oof or test files after ranking seperately. No level 2 model.\n\n", "votes": 4 } ] }, { "id": 977420, "postDate": "2020-08-19T13:15:55.873Z", "content": "\n\nHow we survived the shake
\n
But you did not. You gained 886 places. This can be called a definition of the shakeup.
\nSurviving is having 1st place on Public and staying 1st on Private.
This is just a small remark. Otherwise - great job and congratulations with a 1st place.
", "rawMarkdown": "> How we survived the shake\n\nBut you did not. You gained 886 places. This can be called a definition of the shakeup.\nSurviving is having 1st place on Public and staying 1st on Private.\n\nThis is just a small remark. Otherwise - great job and congratulations with a 1st place.", "votes": 2, "replies": [ { "id": 977440, "postDate": "2020-08-19T13:31:21.167Z", "content": "It depends on your definition of survive. 😏 If we emerged as a survivor after a major event (whether it's shakeup or shakedown), I think we can say we survived.
\nAnyways, thanks for your notebook showing how to use meta data.
", "rawMarkdown": "It depends on your definition of survive. 😏 If we emerged as a survivor after a major event (whether it's shakeup or shakedown), I think we can say we survived.\n\nAnyways, thanks for your notebook showing how to use meta data.", "votes": 1 }, { "id": 977652, "postDate": "2020-08-19T16:09:16.927Z", "content": "Actually I'd like to say that we've got a shake down -- from CV 0.9638 to pvt LB 0.9490
\nMany people got pvt LB score 0.94xx while their CV score is 0.93xx or something. They are called shake up IMO ;)
If what you want is a competition that public LB tracks well with private LB, here you are:
\nhttps://www.kaggle.com/c/landmark-recognition-2020
Hi, congratulations with the first place!
", "rawMarkdown": "Hi, congratulations with the first place!", "votes": 3 }, { "id": 1018407, "postDate": "2020-09-19T16:59:48.300Z", "content": "Hi guys! may I ask where is the imbalance in dataset being handled?
", "rawMarkdown": "Hi guys! may I ask where is the imbalance in dataset being handled?", "votes": 1, "replies": [ { "id": 1018825, "postDate": "2020-09-20T02:47:06.680Z", "content": "No, they don't use any weighted loss. I checked their code which uses simple unweighted loss https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution/blob/master/train.py#L270
", "rawMarkdown": "No, they don't use any weighted loss. I checked their code which uses simple unweighted loss https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution/blob/master/train.py#L270" } ] }, { "id": 992039, "postDate": "2020-08-30T20:56:22.463Z", "content": "congrats for that
", "rawMarkdown": "congrats for that", "votes": 1 }, { "id": 977635, "postDate": "2020-08-19T15:54:01.753Z", "content": "Congrats! Do you have any particular reason choosing Swish (https://arxiv.org/abs/1710.05941) rather than ReLU as activation?
", "rawMarkdown": "Congrats! Do you have any particular reason choosing Swish (https://arxiv.org/abs/1710.05941) rather than ReLU as activation?", "votes": 1, "replies": [ { "id": 977959, "postDate": "2020-08-19T20:17:19.447Z", "content": "Here it probably doesn't matter. It might be better than Relu depending on the data and network.
", "rawMarkdown": "Here it probably doesn't matter. It might be better than Relu depending on the data and network.", "votes": 1 } ] }, { "id": 977569, "postDate": "2020-08-19T15:06:33.960Z", "content": "Congrats @boliu0, @haqishen, and @garybios for a well deserved win. Thanks for sharing your solution.
\nI like the idea of using 2 types of CV and I wish I thought about using diagnosis as targets. Brilliant idea.
", "rawMarkdown": "Congrats @boliu0, @haqishen, and @garybios for a well deserved win. Thanks for sharing your solution.\n\nI like the idea of using 2 types of CV and I wish I thought about using diagnosis as targets. Brilliant idea.", "votes": 1 }, { "id": 977388, "postDate": "2020-08-19T12:58:24.463Z", "content": "excuse me, what is cv and lb? I'm really confused.
", "rawMarkdown": "excuse me, what is cv and lb? I'm really confused.", "votes": 1, "replies": [ { "id": 977443, "postDate": "2020-08-19T13:32:29.490Z", "content": "cv means cross-validation. People use \"cv\" to refer to cross validation score
\nlb means leaderboard. People use \"lb\" to refer to leaderboard score
", "rawMarkdown": "cv means cross-validation. People use \"cv\" to refer to cross validation score\n\nlb means leaderboard. People use \"lb\" to refer to leaderboard score", "votes": 2 }, { "id": 977506, "postDate": "2020-08-19T14:21:04.107Z", "content": "Thank you so much for replying to my beginner *ss. Still, I have a few other questions if you don't mind.
\nThe way I understand it is that you used different versions of the Data as cv right? didn't you fear there would be an overlap between training data and cv data? isn't it bad practise to do so?
\nAnd one last thing: how did you choose the neural architectures? and do you believe there are better-engineered architectures out there that would be more efficient and/or yield better results?
", "rawMarkdown": "Thank you so much for replying to my beginner *ss. Still, I have a few other questions if you don't mind.\n\nThe way I understand it is that you used different versions of the Data as cv right? didn't you fear there would be an overlap between training data and cv data? isn't it bad practise to do so?\n\nAnd one last thing: how did you choose the neural architectures? and do you believe there are better-engineered architectures out there that would be more efficient and/or yield better results?" }, { "id": 977548, "postDate": "2020-08-19T14:52:17.147Z", "content": "Check out Chris's post about cross validation: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614
\nIn short, there's never overlap between training and cv data
\nCurrently, best computer vision architectures are different version of Efficient Nets. Generally you don't need to worry about changing it, especially for beginners
", "rawMarkdown": "Check out Chris's post about cross validation: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614\n\nIn short, there's never overlap between training and cv data\n\nCurrently, best computer vision architectures are different version of Efficient Nets. Generally you don't need to worry about changing it, especially for beginners ", "votes": 2 }, { "id": 977604, "postDate": "2020-08-19T15:36:24.053Z", "content": "Thank you so much for your insight. and btw congrats for being 1st.
", "rawMarkdown": "Thank you so much for your insight. and btw congrats for being 1st.", "votes": 1 }, { "id": 977605, "postDate": "2020-08-19T15:36:24.180Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 976458, "postDate": "2020-08-18T21:47:08.563Z", "content": "Hi, congratulations with the first place! I scrolled through the comments but can't find the answer, have you used any patient level information? That is, other patient's images influencing the prediction.
", "rawMarkdown": "Hi, congratulations with the first place! I scrolled through the comments but can't find the answer, have you used any patient level information? That is, other patient's images influencing the prediction.", "votes": 1, "replies": [ { "id": 976466, "postDate": "2020-08-18T22:00:43.057Z", "content": "Thanks. No we didn't. We treat each data sample independently.
", "rawMarkdown": "Thanks. No we didn't. We treat each data sample independently.", "votes": 2 }, { "id": 976468, "postDate": "2020-08-18T22:03:06.777Z", "content": "Do you think there is no information in it, or very little additional information? How have you arrived to that conclusion? Thanks!
", "rawMarkdown": "Do you think there is no information in it, or very little additional information? How have you arrived to that conclusion? Thanks!" }, { "id": 976497, "postDate": "2020-08-18T22:35:42.790Z", "content": "Personally I don't think there's much (if any) additional information beyond single images and meta data. It's just my hunch, not conclusion based on experiments.
", "rawMarkdown": "Personally I don't think there's much (if any) additional information beyond single images and meta data. It's just my hunch, not conclusion based on experiments.", "votes": 1 }, { "id": 976633, "postDate": "2020-08-19T02:02:00.677Z", "content": "What is your meta feature n_images? Isn't that based on patient_id?
you're right. It is based on patient_id information. (Although dropping it doesn't hurt our score.)
Congrats to you and your team on 1st place.
", "rawMarkdown": "Congrats to you and your team on 1st place.", "votes": 1 }, { "id": 975662, "postDate": "2020-08-18T12:07:27.437Z", "content": "Big congrats to @boliu0, @haqishen, and @garybios!
\nMay I ask if you have done some ablation experiments to verify the following blurring & distortion augmentations are effective? I can understand distortion is at least non-harmful since the melanoma is in irregular shapes. But I don't understand why the blurring methods are working here. Could you explain for me? Thank you in advance!
\n A.OneOf([\n A.MotionBlur(blur_limit=5),\n A.MedianBlur(blur_limit=5),\n A.GaussianBlur(blur_limit=5),\n A.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n\n A.OneOf([\n A.OpticalDistortion(distort_limit=1.0),\n A.GridDistortion(num_steps=5, distort_limit=1.),\n A.ElasticTransform(alpha=3),\n ], p=0.7),\nAlso, very interested in the post-processing method df['pred'] = df['pred'].rank(pct=True). I think all my submissions are probs of melanoma classes when compare to your \"ranks percentile\". Could you explain a little bit?
It's hard to do ablation on augmentations -- the cv score variance is too large to draw meaningful conclusions when comparing two runs that only diff by an augmentation parameter.
\nWhen we tune the augmentations, we only check 1 fold's score without TTA, to have a general idea. However, even 5 fold cv with TTAx8 have noise (i.e. same code with different seeds lead to somewhat different scores).
\nSo the augmentation hyper parameters we used are not optimal, but should be good enough.
", "rawMarkdown": "It's hard to do ablation on augmentations -- the cv score variance is too large to draw meaningful conclusions when comparing two runs that only diff by an augmentation parameter.\n\nWhen we tune the augmentations, we only check 1 fold's score without TTA, to have a general idea. However, even 5 fold cv with TTAx8 have noise (i.e. same code with different seeds lead to somewhat different scores).\n\nSo the augmentation hyper parameters we used are not optimal, but should be good enough.", "votes": 2 }, { "id": 975962, "postDate": "2020-08-18T14:48:30.143Z", "content": "Say your model 1's probability distribution has mean p=0.05; model 2's probability distribution has mean p=0.5. If you simply take average without ranking first, then model 2 would dominate model 1.
\nBy using rank, you can force them to make equal contributions.
", "rawMarkdown": "Say your model 1's probability distribution has mean p=0.05; model 2's probability distribution has mean p=0.5. If you simply take average without ranking first, then model 2 would dominate model 1. \n\nBy using rank, you can force them to make equal contributions.", "votes": 1 }, { "id": 976061, "postDate": "2020-08-18T15:59:24.997Z", "content": "Hi @boliu0, thank you for your explanation. I have some follow up questions hoping to clarify.
\n\n\nIt's hard to do ablation on augmentations -- the cv score variance is too large to draw meaningful conclusions when comparing two runs that only diff by an augmentation parameter.
\n
I think that's exactly the pitfall I've been stepping in. Each run I just change one type of augmentations and see the score difference, but as you said, the comparison may not be valid at all since the score variance is not neglectable.
\n\n\nWhen we tune the augmentations, we only check 1 fold's score without TTA, to have a general idea.
\n
So when you do the tuning work, do you modify a bunch of parameters/augmentation method together? Something like, Combo 1 gives 0.95, while Combo 2 gives me 0.98, then I'll stay with Combo2. Is there some systematic way to explore relatively optimal augmentation hyperparameters?
\n\n\nBy using rank, you can force them to make equal contributions.
\n
I'm a little bit lost here. Please correct me if my understanding is wrong. We predict each image in the test set with 2 models, one prediction has mean p = 0.05, the other prediction has mean p =0.5. If I just do simple averaging, the output p = 0.275. Ok, model 2 dominates model 1.
If we rank predictions firstly, I guess all pred_probs values are actually converted to percentile rank values. Therefore, the absolute difference between the original pred_probs has no influence anymore. which forces 2 models to contribute equally. Is this what you're saying?
So this trick is designed for the AUC metric of his competition specifically?
Thank you very much!
", "rawMarkdown": "Hi @boliu0, thank you for your explanation. I have some follow up questions hoping to clarify. \n\n> It's hard to do ablation on augmentations -- the cv score variance is too large to draw meaningful conclusions when comparing two runs that only diff by an augmentation parameter.\n\nI think that's exactly the pitfall I've been stepping in. Each run I just change one type of augmentations and see the score difference, but as you said, the comparison may not be valid at all since the score variance is not neglectable. \n\n> When we tune the augmentations, we only check 1 fold's score without TTA, to have a general idea.\n\nSo when you do the tuning work, do you modify a bunch of parameters/augmentation method together? Something like, Combo 1 gives 0.95, while Combo 2 gives me 0.98, then I'll stay with Combo2. Is there some systematic way to explore relatively optimal augmentation hyperparameters?\n\n> By using rank, you can force them to make equal contributions.\n\nI'm a little bit lost here. Please correct me if my understanding is wrong. We predict each image in the test set with 2 models, one prediction has `mean p = 0.05`, the other prediction has `mean p =0.5`. If I just do simple averaging, the output `p = 0.275`. Ok, model 2 dominates model 1. \n\nIf we rank predictions firstly, I guess all `pred_probs` values are actually converted to `percentile rank values`. Therefore, the absolute difference between the original `pred_probs` has no influence anymore. which forces 2 models to contribute equally. Is this what you're saying?\n\nSo this trick is designed for the `AUC` metric of his competition specifically?\n\nThank you very much!\n", "votes": 1 }, { "id": 976216, "postDate": "2020-08-18T18:12:53.597Z", "content": "\n\ndo you modify a bunch of parameters/augmentation method together?
\n
Usually make one change at a time
\n\n\nIs there some systematic way to explore relatively optimal augmentation hyperparameters?
\n
Use smaller model and smaller image for faster iteration
\n\n\nIf we rank predictions firstly, I guess all pred_probs values are actually converted to percentile rank values. Therefore, the absolute difference between the original pred_probs has no influence anymore. which forces 2 models to contribute equally. Is this what you're saying?
\n
Yes precisely
\n\n\nSo this trick is designed for the AUC metric of his competition specifically?
\n
It applies to any metric where only the relative order of predictions (not absolute values) matter
", "rawMarkdown": "> do you modify a bunch of parameters/augmentation method together?\n\nUsually make one change at a time\n\n> Is there some systematic way to explore relatively optimal augmentation hyperparameters?\n\nUse smaller model and smaller image for faster iteration\n\n> If we rank predictions firstly, I guess all pred_probs values are actually converted to percentile rank values. Therefore, the absolute difference between the original pred_probs has no influence anymore. which forces 2 models to contribute equally. Is this what you're saying?\n\nYes precisely\n\n> So this trick is designed for the AUC metric of his competition specifically?\n\nIt applies to any metric where only the relative order of predictions (not absolute values) matter", "votes": 1 }, { "id": 976660, "postDate": "2020-08-19T02:45:12.880Z", "content": "Hi @boliu0, great thanks for your patient explanations! Learn quite a lot! Once again, big congrats!
", "rawMarkdown": "Hi @boliu0, great thanks for your patient explanations! Learn quite a lot! Once again, big congrats!", "votes": 1 } ] }, { "id": 975660, "postDate": "2020-08-18T12:06:22.880Z", "content": "Congratulations to your team. You have unlocked quite a few secrets in the data here which have had many of us scratching our heads over the months.
\nWell done and very well deserved with your disciplined approach!
", "rawMarkdown": "Congratulations to your team. You have unlocked quite a few secrets in the data here which have had many of us scratching our heads over the months.\n\nWell done and very well deserved with your disciplined approach!", "votes": 1 }, { "id": 975649, "postDate": "2020-08-18T11:59:07.833Z", "content": "Congrats on the win! I guess building a reliable CV is key in this comp. Feel bad that I'm still nowhere good enough and feel good that there's still more to learn!!!
", "rawMarkdown": "Congrats on the win! I guess building a reliable CV is key in this comp. Feel bad that I'm still nowhere good enough and feel good that there's still more to learn!!!", "votes": 1 }, { "id": 975435, "postDate": "2020-08-18T09:49:30.567Z", "content": "Congrats on result @boliu0 and team and thanks for writeup solution!
", "rawMarkdown": "Congrats on result @boliu0 and team and thanks for writeup solution! ", "votes": 1 }, { "id": 975285, "postDate": "2020-08-18T08:25:07.167Z", "content": "Great writeup!
\nI have a question regarding targets:
\n\n\nTargets
\n
\n We find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01. 2020 data's diagnosis sets and 2019 data's diagnosis sets are different. We manually map 2020's to 2019's. There are 9 labels.2020 seborrheic keratosis -> BKL
\n
\n 2020 lichenoid keratosis -> BKL
\n 2020 solar lentigo -> BKL
\n 2020 lentigo NOS -> BKL
\n 2020 cafe-au-lait macule -> unknown
\n 2020 atypical melanocytic proliferation -> unknown
\n 2020 nevus -> NV
\n 2020 melanoma -> MEL
\n For prediction, we simply take the MEL class's softmax probability.
You mapped unknown as its own class. There were no unknown in 2018-19, so we thought unknown was actually \"anything except melanoma\", and we did:
diagnosis_dict = { \n 'unknown': -100,\n 'atypical melanocytic proliferation': -100,\n 'cafe-au-lait macule': -100,\n\n 'NV': 0,\n 'nevus': 0,\n 'lentigo NOS': 0,\n\n 'MEL': 1,\n 'melanoma': 1,\n\n 'BCC': 2,\n\n 'BKL': 3,\n 'solar lentigo': 3,\n 'seborrheic keratosis': 3, \n 'lichen planus-like keratosis': 3,\n 'lichenoid keratosis': 3,\n\n 'SCC': 4,\n\n 'VASC': 5,\n 'DF': 6,\n 'AK': 7,\n}\nThen our idea for the loss is \"when not unknown, use CrossEntropy as usual, but for unknown only add loss if we are predicting melanoma and label is unknown:
diagnosis_loss = self.diagnosis_loss(diag_pred,diag_targ)\n# diagnosis unknown set to -100 DOES give us some information: it is NOT melanoma, so we attribute accordinly:\ndiag_targ_unknown = torch.full_like(diag_targ,-100)\ndiag_targ_unknown[diag_targ==-100] = 1\ndiagnosis_loss += torch.clamp_min(self.diagnosis_unknown_loss(-diag_pred,diag_targ_unknown)-math.log(7.),0.)\ndiagnosis_loss = reduce_loss(diagnosis_loss, self.label_loss.reduction )\n
\n(that log(7) is an approximation of calculating -Log(1-Softmax(diag_pred)) in slightly different way more numerically stable).
Did you try other ways of attributing unknown?
Thanks! We've tried making all targets other than mel or nervous or BKL to unknown (4 classes in total in this case)
\nBut the result is quite similar, or a little bit worse compared to the 9 classes version.
", "rawMarkdown": "Thanks! We've tried making all targets other than mel or nervous or BKL to unknown (4 classes in total in this case)\n\nBut the result is quite similar, or a little bit worse compared to the 9 classes version.", "votes": 3 } ] }, { "id": 975223, "postDate": "2020-08-18T07:51:11.263Z", "content": "Great, Congratulations to you and your team, well done 😍
", "rawMarkdown": "Great, Congratulations to you and your team, well done 😍", "votes": 1 }, { "id": 975101, "postDate": "2020-08-18T06:38:38.687Z", "content": "\n\nTPU with TF seems to be dominating the public notebooks in this competition. It seems to be faster than a torch/GPU on EfficientNets. However, our experiments showed that its high LB scores are due to better luck with the public LB. Our torch/GPU models have a much better CV
\n
At first, big congrats. However, would you please elaborate on this? What kind of experiment have you conducted? I like to know your experimental comparison results between TF and Torch.
There are different components in the pipeline where pytorch is easier to use or more flexible than TF. For example, in TF, it's hard to achieve augmentations similar to ours.
", "rawMarkdown": "There are different components in the pipeline where pytorch is easier to use or more flexible than TF. For example, in TF, it's hard to achieve augmentations similar to ours.", "votes": 3 } ] }, { "id": 975081, "postDate": "2020-08-18T06:17:11.937Z", "content": "Brilliant! Amazing work and great write up Bo, Gary and Qishen. I wish I had more time to try training and validating on all three datasets. Also interesting approach to keeping multiple years of CV. I might have tried that as well if I had started with competition earlier.
", "rawMarkdown": "Brilliant! Amazing work and great write up Bo, Gary and Qishen. I wish I had more time to try training and validating on all three datasets. Also interesting approach to keeping multiple years of CV. I might have tried that as well if I had started with competition earlier.", "votes": 1 }, { "id": 984019, "postDate": "2020-08-24T19:15:43.390Z", "content": "I'm a newbie to Kaggle, and I love how you said \"simply\" in your description above :) Congrats on your success!
", "rawMarkdown": "I'm a newbie to Kaggle, and I love how you said \"simply\" in your description above :) Congrats on your success!", "votes": 2 }, { "id": 980809, "postDate": "2020-08-21T22:12:09.510Z", "content": "Thanks for sharing! I stumbled over this topic by a paper, that compares the classification accuracy of dermatologists and a ResNet-50 model (https://www.sciencedirect.com/science/article/pii/S0959804919302217)l. Pretty amazing to see the progress!
", "rawMarkdown": "Thanks for sharing! I stumbled over this topic by a paper, that compares the classification accuracy of dermatologists and a ResNet-50 model (https://www.sciencedirect.com/science/article/pii/S0959804919302217)l. Pretty amazing to see the progress!", "votes": 2 }, { "id": 977828, "postDate": "2020-08-19T18:18:45.847Z", "content": "Thanks @boliu0 for sharing this valuable insight and well structured Notebook.
\nI was trying to stick to Keras but it looks like Pytorch gives more flexibility (and better results overall).
\nAgain great job!! 👍
That's great, so PyTorch won again! :)
\nCongratulations to you and the team, who have after the deepfake \"fraud\", proved themselves again, that too with flying green colors of shake up. 😄
", "rawMarkdown": "That's great, so PyTorch won again! :)\n\nCongratulations to you and the team, who have after the deepfake \"fraud\", proved themselves again, that too with flying green colors of shake up. 😄", "votes": 2, "replies": [ { "id": 975082, "postDate": "2020-08-18T06:17:26.623Z", "content": "thanks, @sarques, we are so excited that we can win the first place again after Deepfake, which gave me a renewed confidence in Kaggle.
", "rawMarkdown": "thanks, @sarques, we are so excited that we can win the first place again after Deepfake, which gave me a renewed confidence in Kaggle.", "votes": 4 }, { "id": 975086, "postDate": "2020-08-18T06:21:35.670Z", "content": "Yes, it was truly deserved! What happened there has challenged a lot of people in many ways. After that happening, we were trying to read those legal terms a lot in this competition too, that was a mishap on their side, no need to let yourself down! We admire you, You are our Grand Master after all! :)
", "rawMarkdown": "Yes, it was truly deserved! What happened there has challenged a lot of people in many ways. After that happening, we were trying to read those legal terms a lot in this competition too, that was a mishap on their side, no need to let yourself down! We admire you, You are our Grand Master after all! :) " } ] }, { "id": 975033, "postDate": "2020-08-18T05:55:02.273Z", "content": "thanks. no tabular data at all?
", "rawMarkdown": "thanks. no tabular data at all? ", "votes": 2, "replies": [ { "id": 975067, "postDate": "2020-08-18T06:12:02.740Z", "content": "We did use meta (i.e. tabular) data in some of our models, similar to this public notebook. Let me add more information on that and update the original post.
", "rawMarkdown": "We did use meta (i.e. tabular) data in some of our models, similar to [this](https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet) public notebook. Let me add more information on that and update the original post. " }, { "id": 975267, "postDate": "2020-08-18T08:12:17.630Z", "content": "thanks. i think meta data helps (a lot) … and it helps in an interesting way.
\nbasically it remove bias as the two type of data are different. in my early experiments, they are quite uncorrelated. meta data also provides a good prior
\ni have a feeling that it is something like this: Stein's paradox
\nhttps://en.wikipedia.org/wiki/Stein%27s_example
\" Suppose we are to estimate three unrelated parameters, such as the US wheat yield for 1993, the number of spectators at the Wimbledon tennis tournament in 2001, and the weight of a randomly chosen candy bar from the supermarket. Suppose we have independent Gaussian measurements of each of these quantities. Stein's example now tells us that we can get a better estimate (on average) for the vector of three parameters by simultaneously using the three unrelated measurements.\"
", "rawMarkdown": "thanks. i think meta data helps (a lot) ... and it helps in an interesting way.\n\nbasically it remove bias as the two type of data are different. in my early experiments, they are quite uncorrelated. meta data also provides a good prior\n\ni have a feeling that it is something like this: Stein's paradox\nhttps://en.wikipedia.org/wiki/Stein%27s_example\n\n\" Suppose we are to estimate three unrelated parameters, such as the US wheat yield for 1993, the number of spectators at the Wimbledon tennis tournament in 2001, and the weight of a randomly chosen candy bar from the supermarket. Suppose we have independent Gaussian measurements of each of these quantities. Stein's example now tells us that we can get a better estimate (on average) for the vector of three parameters by simultaneously using the three unrelated measurements.\"\n\n", "votes": 3 }, { "id": 975817, "postDate": "2020-08-18T13:33:45.480Z", "content": "Note that models using meta data tend to overfit more easily than models with images only, because augmentations do not apply to meta data
", "rawMarkdown": "Note that models using meta data tend to overfit more easily than models with images only, because augmentations do not apply to meta data", "votes": 2 } ] }, { "id": 975032, "postDate": "2020-08-18T05:54:12.027Z", "content": "Congrats for your result and thank you for the write-up. That is an interesting set of augmentations.
\n\n\nWe find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01.
\n
Interesting , we also tried that and saw no much difference versus the binary target. I am wondering if there are other factors to that need to be in place for it to work.
", "rawMarkdown": "Congrats for your result and thank you for the write-up. That is an interesting set of augmentations. \n\n> We find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01.\n\nInteresting , we also tried that and saw no much difference versus the binary target. I am wondering if there are other factors to that need to be in place for it to work. ", "votes": 2, "replies": [ { "id": 975040, "postDate": "2020-08-18T05:57:57.407Z", "content": "Thanks, It need a few epochs longer than BCE and to optimize the learning rate again. Maybe some augmentation parameters work as well but it's hard to tell.
", "rawMarkdown": "Thanks, It need a few epochs longer than BCE and to optimize the learning rate again. Maybe some augmentation parameters work as well but it's hard to tell.", "votes": 4 }, { "id": 975282, "postDate": "2020-08-18T08:21:09.440Z", "content": "\"we also tried that and saw no much difference versus the binary target.\"
\nthe following may gives more clue:
\nif multi-target helps, in theory if we do clustering (unlabelled targets) loss, maybe results will even be better. This reminds me of fb swapping assignment view (SWav) paper, which basically learn sub-clusters and sampe-to-cluster constrastive loss
", "rawMarkdown": "\"we also tried that and saw no much difference versus the binary target.\"\n\nthe following may gives more clue:\n- visualize UMAP or other embedding maps for models of binary and multi targets \n- swap some multi targets (add noise) and see if how fast results degrade\n\nif multi-target helps, in theory if we do clustering (unlabelled targets) loss, maybe results will even be better. This reminds me of fb swapping assignment view (SWav) paper, which basically learn sub-clusters and sampe-to-cluster constrastive loss", "votes": 4 } ] }, { "id": 975028, "postDate": "2020-08-18T05:52:09.683Z", "content": "Congrats @boliu0, @haqishen, and @garybios.
\nAnd thanks for sharing!
A big Congratulations.
", "rawMarkdown": "A big **Congratulations**. " }, { "id": 975316, "postDate": "2020-08-18T08:39:19.280Z", "content": "What computing resources did you use?
", "rawMarkdown": "What computing resources did you use?", "votes": 1 }, { "id": 975485, "postDate": "2020-08-18T10:11:50.143Z", "content": "Now that's a surprise!!! Winner uses PyTorch! :)
\nI wasn't able to create bigger score with TPU than GPU so I decided something is wrong with my approach and I will not be able to fix it in time so I resigned.
\nThank you for these details, this is very interesting.
\nI am happy to see diagnosis was usable. I was trying to build another model to predict nevus only then to use hidden layer as features.
", "rawMarkdown": "Now that's a surprise!!! Winner uses PyTorch! :)\n\nI wasn't able to create bigger score with TPU than GPU so I decided something is wrong with my approach and I will not be able to fix it in time so I resigned. \n\nThank you for these details, this is very interesting.\n\nI am happy to see diagnosis was usable. I was trying to build another model to predict nevus only then to use hidden layer as features.\n" }, { "id": 982640, "postDate": "2020-08-23T14:43:30.717Z", "content": "@boliu0 Can you ps explain this?
\n\n\n2020 seborrheic keratosis -> BKL
\n
\n 2020 lichenoid keratosis -> BKL
\n 2020 solar lentigo -> BKL
\n 2020 lentigo NOS -> BKL
\n 2020 cafe-au-lait macule -> unknown
\n 2020 atypical melanocytic proliferation -> unknown
\n 2020 nevus -> NV
\n 2020 melanoma -> MEL
The 2020 dataset has the following 5 unique values, but you wrote 8 above:
\n\n\nunknown
\n
\n melanoma
\n nevus
\n seborrheic keratosis
\n lentigo NOS
The 2019 dataset has the following 5 unique values:
\n\n\nBCC
\n
\n BKL
\n AK
\n NV
\n MEL
Please explain the mapping. And for the BKL 4 to 1 mapping, did u guys find those in papers?
", "rawMarkdown": "@boliu0 Can you ps explain this? \n\n\n> 2020 seborrheic keratosis -> BKL\n2020 lichenoid keratosis -> BKL\n2020 solar lentigo -> BKL\n2020 lentigo NOS -> BKL\n2020 cafe-au-lait macule -> unknown\n2020 atypical melanocytic proliferation -> unknown\n2020 nevus -> NV\n2020 melanoma -> MEL\n\n\nThe 2020 dataset has the following 5 unique values, but you wrote 8 above:\n\n> unknown\nmelanoma\nnevus\nseborrheic keratosis\nlentigo NOS\n\nThe 2019 dataset has the following 5 unique values:\n\n\n> BCC\nBKL \nAK\nNV\nMEL\n\nPlease explain the mapping. And for the BKL 4 to 1 mapping, did u guys find those in papers? \n", "votes": -1, "replies": [ { "id": 982694, "postDate": "2020-08-23T15:34:07.617Z", "content": "The mapping was based on the diagnosis description on last year competition's website: https://challenge2019.isic-archive.com/
", "rawMarkdown": "The mapping was based on the diagnosis description on last year competition's website: https://challenge2019.isic-archive.com/" } ] }, { "id": 977341, "postDate": "2020-08-19T12:31:09.767Z", "content": "……………………
", "rawMarkdown": "........................", "votes": -5 }, { "id": 975274, "postDate": "2020-08-18T08:17:56.733Z", "content": "Congratulations, interesting that you guys used heavy augmentation and for sure that PyTorch made it to the top.
", "rawMarkdown": "Congratulations, interesting that you guys used heavy augmentation and for sure that PyTorch made it to the top.", "votes": -1 }, { "id": 975041, "postDate": "2020-08-18T05:59:18.530Z", "content": "Very clean write up, and easy to digest. Congrats on trusting your cvs and the well deserved shake up.
", "rawMarkdown": "Very clean write up, and easy to digest. Congrats on trusting your cvs and the well deserved shake up.", "votes": -1 }, { "id": 2917844, "postDate": "2024-07-11T20:27:21.647Z", "content": "Hey @boliu0 , @haqishen , and @garybios -- in case you guys are up for it, we are running a new Melanoma detection challenge now:
\nhttps://www.kaggle.com/competitions/isic-2024-challenge/overview
\nWould love to see you participate :)
", "rawMarkdown": "Hey @boliu0 , @haqishen , and @garybios -- in case you guys are up for it, we are running a new Melanoma detection challenge now:\n\nhttps://www.kaggle.com/competitions/isic-2024-challenge/overview\n\nWould love to see you participate :)" }, { "id": 2916893, "postDate": "2024-07-11T10:31:47.343Z", "content": "Hello Please i would like to talk to you in private
\nThanks
Congratulations team and thanks for sharing the solution 😀
", "rawMarkdown": "Congratulations team and thanks for sharing the solution 😀" }, { "id": 2899051, "postDate": "2024-07-01T13:16:31.650Z", "content": "Nice work!
", "rawMarkdown": "Nice work!" }, { "id": 2624862, "postDate": "2024-01-29T03:32:33.627Z", "content": "Great work!
", "rawMarkdown": "Great work!" }, { "id": 2566580, "postDate": "2023-12-19T02:15:57.623Z", "content": "Congratulations!
\ni have a question thar about you take 'use_meta'.
\nfrom you commit code only have 4 sort of the \"use_meta\" was used ,while 14.
@haqishen , thanks for sharing . I wanted to know how to come up with such augmentations ! There are lot of augmentations you applied .
", "rawMarkdown": "@haqishen , thanks for sharing . I wanted to know how to come up with such augmentations ! There are lot of augmentations you applied . " }, { "id": 1265127, "postDate": "2021-04-06T16:31:19.183Z", "content": "Hi, the ensemble.py output a probability represented in a floating number between 0-1. How did you select the threshold between benign and malignant? did you simply use 0.5 as cut off?
\nThanks and congrats!
", "rawMarkdown": "Hi, the ensemble.py output a probability represented in a floating number between 0-1. How did you select the threshold between benign and malignant? did you simply use 0.5 as cut off?\n\nThanks and congrats!" }, { "id": 1004460, "postDate": "2020-09-09T18:18:44.200Z", "content": "I also want to ask, why do not need to use weights in nn.CrossEntropyLoss to account for imbalance numbers of labels.
", "rawMarkdown": "I also want to ask, why do not need to use weights in nn.CrossEntropyLoss to account for imbalance numbers of labels.", "replies": [ { "id": 1159277, "postDate": "2021-01-19T06:57:12.490Z", "content": "Hello @datduong2,
\nAccording to me, they didn't used weights in the loss because same kind of imbalance is there in the private dataset. So, it's better not to use weights and let the model learn the data distribution in the training data which is same in the private set.
", "rawMarkdown": "Hello @datduong2,\n\nAccording to me, they didn't used weights in the loss because same kind of imbalance is there in the private dataset. So, it's better not to use weights and let the model learn the data distribution in the training data which is same in the private set.", "votes": 1 } ] }, { "id": 1004458, "postDate": "2020-09-09T18:18:29.867Z", "content": "I also want to ask, since you are predicting 9 labels, why do not need to use weights in nn.CrossEntropyLoss to account for imbalance numbers of labels?
", "rawMarkdown": "I also want to ask, since you are predicting 9 labels, why do not need to use weights in nn.CrossEntropyLoss to account for imbalance numbers of labels?" }, { "id": 991370, "postDate": "2020-08-30T11:06:53.033Z", "content": "Thanks for sharing, Nvidia wins in final. Congratulations!
", "rawMarkdown": "Thanks for sharing, Nvidia wins in final. Congratulations! " }, { "id": 988848, "postDate": "2020-08-28T10:26:21.040Z", "content": "Congratulations!!!
", "rawMarkdown": "Congratulations!!!" }, { "id": 987915, "postDate": "2020-08-27T16:04:46.580Z", "content": "I'm quite new to Kaggle. Thanks for sharing and congratulations for the 1st place.
", "rawMarkdown": "I'm quite new to Kaggle. Thanks for sharing and congratulations for the 1st place." }, { "id": 987754, "postDate": "2020-08-27T13:45:03.220Z", "content": "hi, congrats good to go…
", "rawMarkdown": "hi, congrats good to go...\n" }, { "id": 986832, "postDate": "2020-08-26T20:23:04.587Z", "content": "Well done! Congrats!
", "rawMarkdown": "Well done! Congrats!" }, { "id": 985881, "postDate": "2020-08-26T04:48:04.963Z", "content": "congratulatios! @haqishen @boliu0 @garybios Thanks for sharing. perfect use of pytorch
", "rawMarkdown": "congratulatios! @haqishen @boliu0 @garybios Thanks for sharing. perfect use of pytorch" }, { "id": 985454, "postDate": "2020-08-25T18:28:08.630Z", "content": "What is LB? (As in public LB).
\nHow did you come up with this model? (I want to know the thought process)
\nAs you can see I guess I am the last person in this world starting with Kaggle. (Beginner here)
Nice path for newbies
", "rawMarkdown": "Nice path for newbies" }, { "id": 982836, "postDate": "2020-08-23T17:43:50.110Z", "content": "Hi, the notebook is failing with the following error (see attachments) . any suggestions ?
", "rawMarkdown": "Hi, the notebook is failing with the following error (see attachments) . any suggestions ? " }, { "id": 982434, "postDate": "2020-08-23T11:14:16.383Z", "content": "Congratulations with the 1st place!
", "rawMarkdown": "Congratulations with the 1st place!" }, { "id": 980469, "postDate": "2020-08-21T16:06:49.210Z", "content": "Congratulations, and thanks for sharing!
", "rawMarkdown": "Congratulations, and thanks for sharing!" }, { "id": 980453, "postDate": "2020-08-21T15:53:53.360Z", "content": "Congratulations on the first place!
", "rawMarkdown": "Congratulations on the first place!" }, { "id": 979616, "postDate": "2020-08-21T02:01:48.193Z", "content": "Congrats!
", "rawMarkdown": "Congrats! " }, { "id": 979492, "postDate": "2020-08-20T22:11:08.107Z", "content": "Congrats on the win!
", "rawMarkdown": "Congrats on the win!" }, { "id": 977792, "postDate": "2020-08-19T17:53:10.647Z", "content": "@boliu0 @haqishen @garybios Congratulations! So you guys used new 2019 data despite it having different distribution than the 2020 test data?
", "rawMarkdown": "@boliu0 @haqishen @garybios Congratulations! So you guys used new 2019 data despite it having different distribution than the 2020 test data?", "replies": [ { "id": 977951, "postDate": "2020-08-19T20:13:15.783Z", "content": "yeah we used 2019 data that Chris shared. We used it because of its different distribution, e.g. higher positive sample proportion
", "rawMarkdown": "yeah we used 2019 data that Chris shared. We used it *because of* its different distribution, e.g. higher positive sample proportion ", "votes": 1 } ] }, { "id": 977442, "postDate": "2020-08-19T13:32:01.953Z", "content": "Intuitively, how does training with cross-entropy loss and then classify using MEL's softmax improve the score?
", "rawMarkdown": "Intuitively, how does training with cross-entropy loss and then classify using MEL's softmax improve the score?", "replies": [ { "id": 977448, "postDate": "2020-08-19T13:38:03.547Z", "content": "malignant vs benign information is a subset of diagnosis information
\nBy using diagnosis as target, you're giving the model more information. It can help the model learn representation better.
\nImagine you have a bunch of images of dog, cat, wolf, bird, fox …
\nMethod 1: You label them as dog vs not dog
\nMethod 2: You label them as dog, cat, wolf, bird, fox separately
In method 2, the model could learn decision boundary of dog vs wolf better, because now it knows what's what.
", "rawMarkdown": "malignant vs benign information is a subset of diagnosis information\n\nBy using diagnosis as target, you're giving the model more information. It can help the model learn representation better.\n\nImagine you have a bunch of images of dog, cat, wolf, bird, fox ...\nMethod 1: You label them as dog vs not dog\nMethod 2: You label them as dog, cat, wolf, bird, fox separately\n\nIn method 2, the model could learn decision boundary of dog vs wolf better, because now it knows what's what.", "votes": 8 }, { "id": 977463, "postDate": "2020-08-19T13:46:08.363Z", "content": "Cheers, that makes sense! Are there anymore more case study of this particular technique in literature or you guys just found this to be useful for this particular competition?
", "rawMarkdown": "Cheers, that makes sense! Are there anymore more case study of this particular technique in literature or you guys just found this to be useful for this particular competition?" }, { "id": 977551, "postDate": "2020-08-19T14:54:48.990Z", "content": "I'm not aware papers on this trick. (There are too many AI papers these days to keep up with)
\nAnd it's a common trick in ML, probably not significant enough for a paper.
", "rawMarkdown": "I'm not aware papers on this trick. (There are too many AI papers these days to keep up with)\n\nAnd it's a common trick in ML, probably not significant enough for a paper." } ] }, { "id": 976810, "postDate": "2020-08-19T05:35:37.450Z", "content": "Awesome Idea
", "rawMarkdown": "Awesome Idea" }, { "id": 976343, "postDate": "2020-08-18T19:53:22.183Z", "content": "Great! This is an amazing work! Congrats!
", "rawMarkdown": "Great! This is an amazing work! Congrats!" }, { "id": 976301, "postDate": "2020-08-18T19:20:39.500Z", "content": "Our single model's LB-cv correlation is essential 0, but the bigger the ensemble, the more stable the LB. .
Sorry, I am a newbie here, I didn't get the intuition of this line?
", "rawMarkdown": "`Our single model's LB-cv correlation is essential 0, but the bigger the ensemble, the more stable the LB. .`\n\nSorry, I am a newbie here, I didn't get the intuition of this line? ", "replies": [ { "id": 976315, "postDate": "2020-08-18T19:29:52.243Z", "content": "Think of any given single model's LB score as a normal random variable with big standard deviation. Now, if you take the average of 5 such i.i.d. normal random variables, the standard deviation will be smaller, according to law of large number.
\nSimilarly, ensemble model's variation (i.e. \"shake\") is generally smaller than single model. The variation is averaged out.
", "rawMarkdown": "Think of any given single model's LB score as a normal random variable with big standard deviation. Now, if you take the average of 5 such i.i.d. normal random variables, the standard deviation will be smaller, according to law of large number.\n\nSimilarly, ensemble model's variation (i.e. \"shake\") is generally smaller than single model. The variation is averaged out.", "votes": 3 }, { "id": 976341, "postDate": "2020-08-18T19:49:39.310Z", "content": "Thanks, I get it now,
", "rawMarkdown": "Thanks, I get it now, " } ] }, { "id": 975822, "postDate": "2020-08-18T13:37:47.030Z", "content": "Did you trained each model( EfficientNet B3-B7, se_resnext101, resnest101) on all (384 to 896) resolution images, or you use different models of the same type for diff resolutions (for example one EffNet-B3 for 384, another EffNet-B3 for 512 etc)?
\nThanks for grate work btw🎉
", "rawMarkdown": "Did you trained each model( EfficientNet B3-B7, se_resnext101, resnest101) on all (384 to 896) resolution images, or you use different models of the same type for diff resolutions (for example one EffNet-B3 for 384, another EffNet-B3 for 512 etc)?\n\nThanks for grate work btw🎉", "replies": [ { "id": 975975, "postDate": "2020-08-18T14:56:41.253Z", "content": "Not all combinations. That would be too many 😄
\nFor example, 896 can only be used on the smaller B3
", "rawMarkdown": "Not all combinations. That would be too many 😄\n\nFor example, 896 can only be used on the smaller B3", "votes": 1 } ] }, { "id": 975748, "postDate": "2020-08-18T12:58:48.863Z", "content": "Congrat! Thanks for sharing your great efforts!
", "rawMarkdown": "Congrat! Thanks for sharing your great efforts!" }, { "id": 975379, "postDate": "2020-08-18T09:12:53.840Z", "content": "Huge congratz! And thanks for the writeup!
\nI have a quick question. You mentioned that you used \"diagnosis as targets with cross entropy loss instead of binary target with BCE loss.\" Do you mean you made multi-class prediction using diagnosis feature as target with cross entropy loss (instead of using the target feature with BNC loss)?
\nThanks and congratulations again!
yeah he changed the target for models to diagnosis and made it a multi classification task
", "rawMarkdown": "yeah he changed the target for models to diagnosis and made it a multi classification task", "votes": 1 } ] }, { "id": 975255, "postDate": "2020-08-18T08:07:09.853Z", "content": "Congrats! Thanks for sharing ur great knowledge.
\nI have one question. Could you please tell me which method you used when ensemble or stacking oof?
just a simple average of ranked probabilities
", "rawMarkdown": "just a simple average of ranked probabilities", "votes": 2 }, { "id": 975640, "postDate": "2020-08-18T11:48:49.807Z", "content": "Can I ask this part?
\n\n\njust a simple average of ranked probabilities
\n
Is it similar to this?
\nsubmission = rankdata(model1_result) * a + rankdata(model2_result) * b
Thanks in advance! @boliu0
", "rawMarkdown": "Can I ask this part?\n> just a simple average of ranked probabilities\n\nIs it similar to this?\n`submission = rankdata(model1_result) * a + rankdata(model2_result) * b`\n\n\nThanks in advance! @boliu0 " }, { "id": 975644, "postDate": "2020-08-18T11:52:45.760Z", "content": "just remove * a or * b then you get the thing.
It's actually replace a and b with 1/2 in your example. I don't know what will happen if you submit probabilities larger than 1.
I think what he meant is weighing the prediction of different models predictions differently as in a+b=1
", "rawMarkdown": "I think what he meant is weighing the prediction of different models predictions differently as in a+b=1" } ] }, { "id": 975243, "postDate": "2020-08-18T08:02:26.387Z", "content": "Congratulations! Question about the augmentation: flip X/Y I understand, but Transpose and no \"Rotate 180\"?
", "rawMarkdown": "Congratulations! Question about the augmentation: flip X/Y I understand, but Transpose and no \"Rotate 180\"?", "replies": [ { "id": 975838, "postDate": "2020-08-18T13:43:48.913Z", "content": "For any 2-dimensional image, rotate90 or rotate180 can be achieved using flip and transpose operations only.
\nFor example, transpose+vertical flip equals to rotate90
", "rawMarkdown": "For any 2-dimensional image, rotate90 or rotate180 can be achieved using flip and transpose operations only. \n\nFor example, transpose+vertical flip equals to rotate90", "votes": 1 }, { "id": 976047, "postDate": "2020-08-18T15:52:28.093Z", "content": "Pardon me not being fluent in Python. I did not realize the Compose was called more than once. Then I understand.
", "rawMarkdown": "Pardon me not being fluent in Python. I did not realize the Compose was called more than once. Then I understand." } ] }, { "id": 975211, "postDate": "2020-08-18T07:43:15.377Z", "content": "Congratulations - I'm so pleased to see the metadata used in this way - really good work. I tried to concat it to the end of one of my effnet models (much more simply than you did though) and it made an increase but in the end my highest scoring one was just an ensemble of effnets and a tabular model from @demirbkr .
\nHow big a difference did the metadata make to your score on the private LB?
", "rawMarkdown": "Congratulations - I'm so pleased to see the metadata used in this way - really good work. I tried to concat it to the end of one of my effnet models (much more simply than you did though) and it made an increase but in the end my highest scoring one was just an ensemble of effnets and a tabular model from @demirbkr .\n\nHow big a difference did the metadata make to your score on the private LB?", "replies": [ { "id": 975225, "postDate": "2020-08-18T07:53:15.623Z", "content": "We didn't trust LB so we didn't submit every single model.
\nOn cv, adding meta helped cv_all but hurt cv_2020. Overall, combining models with meta and models without meta seems like good diversity for ensemble.
\n\nAll scores above are 5 fold, TTAx8.
\n
All TTAs are horizontal flips?
", "rawMarkdown": ">All scores above are 5 fold, TTAx8.\n\n\n\nAll TTAs are horizontal flips?", "replies": [ { "id": 975177, "postDate": "2020-08-18T07:25:40.297Z", "content": "yes, Transpose/VerticalFlip/HorizontalFlip
", "rawMarkdown": "yes, Transpose/VerticalFlip/HorizontalFlip", "votes": 1 } ] }, { "id": 975116, "postDate": "2020-08-18T06:49:04.550Z", "content": "Congrats , Very well written :D
", "rawMarkdown": "Congrats , Very well written :D " }, { "id": 1108591, "postDate": "2020-12-10T20:11:27.323Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 977386, "postDate": "2020-08-19T12:56:44.783Z", "rawMarkdown": "", "isDeleted": true }, { "id": 976752, "postDate": "2020-08-19T04:31:43.750Z", "rawMarkdown": "", "isDeleted": true }, { "id": 975864, "postDate": "2020-08-18T13:55:14.457Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2920763, "postDate": "2024-07-13T19:26:06.507Z", "content": "Thanks. Really appreciate the share.
", "rawMarkdown": "Thanks. Really appreciate the share. " }, { "id": 991728, "postDate": "2020-08-30T16:13:13.550Z", "content": "This is awesome, thanks for sharing
", "rawMarkdown": "This is awesome, thanks for sharing" }, { "id": 989842, "postDate": "2020-08-29T06:34:49.267Z", "content": "Good work. Thanks for sharing.
", "rawMarkdown": "Good work. Thanks for sharing." }, { "id": 988671, "postDate": "2020-08-28T07:35:17.073Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 988555, "postDate": "2020-08-28T05:55:27.373Z", "content": "Thanks for sharing and Congrats!
", "rawMarkdown": "Thanks for sharing and Congrats!\n" }, { "id": 987857, "postDate": "2020-08-27T15:20:11.650Z", "content": "Thanks for sharing and Congrats!
", "rawMarkdown": "Thanks for sharing and Congrats!" }, { "id": 987834, "postDate": "2020-08-27T14:58:14.083Z", "content": "It was really helpful! Thanks.
", "rawMarkdown": "It was really helpful! Thanks." }, { "id": 985640, "postDate": "2020-08-25T22:29:11.500Z", "content": "Congrats and Thanks for Sharing man!!!
", "rawMarkdown": "Congrats and Thanks for Sharing man!!!" }, { "id": 981500, "postDate": "2020-08-22T13:42:14.537Z", "content": "Thanks for sharing and Congrats!
", "rawMarkdown": "Thanks for sharing and Congrats!" }, { "id": 978599, "postDate": "2020-08-20T09:23:02.910Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 975038, "postDate": "2020-08-18T05:57:24Z", "content": "Great writeup - Thanks for sharing.
", "rawMarkdown": "Great writeup - Thanks for sharing." } ], "comments": [ { "id": 975841, "author_name": "Giba", "author_url": "", "post_date": "2020-08-18T13:46:00.657000", "content": "Congrats @haqishen @boliu0 @garybios and thanks for sharing. Looks like your validation discipline and labeling strategy made the difference. I'm glad to see that pytorch prevailed.
", "votes": 17, "replies": [ { "id": 976620, "author_name": "Dieter", "author_url": "", "post_date": "2020-08-19T01:43:29.457000", "content": "pytorch + GPUs ;)
", "votes": 7, "replies": [] } ] }, { "id": 975329, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-18T08:49:12.200000", "content": "Congratulations! Very proud to see a NVIDIA colleague win!
\nYou made my first regret: I didn't think of using diagnosis as target. I'm tempted to retrain my bes model with it… I did use all 2019 targets, and it helped me.
\nYour idea of using a global CV is also very interesting. Whatever the mean, having a reliable CV is the key. you found it. Again, congrats.
\nYou did not say much about your models. You just used models as is with the right number of classes?
", "votes": 9, "replies": [ { "id": 975342, "author_name": "Gary", "author_url": "", "post_date": "2020-08-18T08:55:31.613000", "content": "@cpmpml hi, uncle.
\nyes, we didn't do a lot of work on the modification of model, but simply replaced the FC head and added dropout
Thanks. Yeah the model is pretty vanilla. We have posted our code here: https://www.kaggle.com/haqishen/1st-place-soluiton-code-small-ver#Define-Dataset
", "votes": 1, "replies": [] } ] }, { "id": 975338, "author_name": "Psi", "author_url": "", "post_date": "2020-08-18T08:54:49.363000", "content": "One thing to add regarding diagnosis.
\nWe found out the following thing:
\ndiagnosis=nevus from train (keep them in val)diagnosis=unknown Anyone figured this out? Maybe what it means? I was thinking that those labeled explicitly as nevus are more correct? Is there another bias? Unfortunately did not have too much time to follow up on it, but I expect that having the extra targets helps also with that a bit (we partly also did it, but did not see that much of improvements as you did).
", "votes": 6, "replies": [ { "id": 977573, "author_name": "Optimo", "author_url": "", "post_date": "2020-08-19T15:08:31.543000", "content": "I did a similar experiment :
\nAnyone noticed this?
", "votes": 1, "replies": [] }, { "id": 977968, "author_name": "Psi", "author_url": "", "post_date": "2020-08-19T20:29:16.663000", "content": "Yes, we also did that, we also fittet on 2020 and predicted 2019 and then removed hard samples, this improves CV quite a lot, but is a form of leak as you use models trained on the data to remove other data, similar as in pseudo tagging.
", "votes": 2, "replies": [] }, { "id": 978100, "author_name": "Optimo", "author_url": "", "post_date": "2020-08-20T00:13:13.053000", "content": "Don’t know how you got a downvote for this answer ^^
\nIf I just remove training examples and don’t change validation I don’t see where the leak comes into play. It felt like an inverse focal loss, you focus only on easy examples and don’t take hard examples in your loss.
\nIt’s an unusual approach and I feel it can work only if there is something wrong with labels.
Great work =))
", "votes": 1, "replies": [] }, { "id": 975165, "author_name": "Psi", "author_url": "", "post_date": "2020-08-18T07:19:55.047000", "content": "Happy that Pytorch won, congrats!
\nOnly figured out on the last day that taking CV across all data is better.
", "votes": 3, "replies": [ { "id": 975188, "author_name": "Gary", "author_url": "", "post_date": "2020-08-18T07:30:29.187000", "content": "@philippsinger I am sorry that you don't have enough time to do the new experiment.
\nyes, In the early stage of our experiment, our same pipeline and parameters can obtained different CV_2020, so we also doubt the stability of CV_2020. There are more pos samples in 2018/19/20. We believe that this CV_all will be more stable and guarantee the generalization.
Yeah, it makes sense. Overall our CV / LB correlation was quite good, but it got a bit messy when reaching higher CV scores. We were a bit too cautious with some image distribution differences between the datasets, but putting it all together here also makes sense. Good job!
", "votes": 1, "replies": [] }, { "id": 975249, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-08-18T08:05:40.970000", "content": "In fact I imply that days ago but seems no one trust me ;)
\n
Do you automatically win another $10,000? Your model uses contextual information. You add the feature 0 * patient_id and your model does not use contextual information. So you win both additional $5,000 prizes.
\n\nSpecial Prizes: Awarded to the top scoring models using or not using patient-level contextual information.
\n
With Context - $5,000 (Top-scoring model making use of patient-level contextual information)\nWithout Context - $5,000 (Top-scoring model without using any patient-level contextual information)\nWell, seems we can only automatically get half of it ;)
\n(Still feel confuse on what is patient-level contextual information)
Training with diagnosis then using MEL class softmax is a great idea. Congratulations on an amazing model and 1st place!
Hi Chris, based on your abundant experience, does it mean more classes (using diagnosis as @boliu0 did) may help model learn better than binary class (using target as we normally did) on such highly-imbalanced data?
Congratulations on your 1st ! and thank you for the write up!
\nI was surprised that you used 2020+2019+2018 for validation. A lot of kernels and my experiments immediately overfit in this case, achieving 0.99CV.
\nI have questions about some of the fundamentals of making training stable. I took a lot of time here and didn't get to the more substantive considerations.
Do you remove the questionable augmentations one by one from the base transform with all the augmentations, or do you use something that's almost decided from the beginning and rarely changes?
\nOf course why didn't you use the method of putting in more small holes like dropout, or overlapping images like mixup?
You need to adjust the lr when you change (models, datasets, lossfunction) small to large. I'm having a hard time finding just the right lr, scheduler, epoch for every competition, could you please tell me the procedure for adjusting lr?
\nAssuming we use cosine annealing, how do we determine the max lr, train epoch?
Did you use spearman corr for selecting models or try stacking ?
\nIn the case of ensembling different models, we can just rank ensemble, but I didn't know that we can also use rank averaging the same model's different fold.
\nIn general, the outputs of the same model's different fold have similar distribution, we often average the outputs of the model as it is. However, in this task, the test output distribution for each fold may be very different. I did not know how to deal with this problem. Thanks!
I'm sorry for many questions, poor english. I would really appreciate it if you could respond to the questions. Once again, congratulations on your 1st !!!
", "votes": 4, "replies": [ { "id": 975300, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-08-18T08:31:09.330000", "content": "Thanks!
\n\n\nI was surprised that you used 2020+2019+2018 for validation. A lot of kernels and my experiments immediately overfit in this case, achieving 0.99CV.
\n
You may have a leak on your data split. We cannot get over cv0.98 when validating on 2020+2019+2018 for a single model.
\n\n\nHow do you coordinate augmentation?
\n
We tune the parameter one by one to find the best combinations. Then we fix it till the end of the competition.
\n\n\nWhat did you use for optimizer, schedular? Did you learn the larger model from the beginning?
\n
We tried some different optimizer and schedular but see no much difference. We use warmup + cosine w/ Adam till the end of the competition.
\n\n\nIs ensemble optimization a baysian weight optimization using oof?
\n
No. we just take a simple avg of those oof or test files after ranking seperately. No level 2 model.
", "votes": 4, "replies": [] } ] }, { "id": 977420, "author_name": "Roman", "author_url": "", "post_date": "2020-08-19T13:15:55.873000", "content": "\n\nHow we survived the shake
\n
But you did not. You gained 886 places. This can be called a definition of the shakeup.
\nSurviving is having 1st place on Public and staying 1st on Private.
This is just a small remark. Otherwise - great job and congratulations with a 1st place.
", "votes": 2, "replies": [ { "id": 977440, "author_name": "Bo", "author_url": "", "post_date": "2020-08-19T13:31:21.167000", "content": "It depends on your definition of survive. 😏 If we emerged as a survivor after a major event (whether it's shakeup or shakedown), I think we can say we survived.
\nAnyways, thanks for your notebook showing how to use meta data.
", "votes": 1, "replies": [] }, { "id": 977652, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-08-19T16:09:16.927000", "content": "Actually I'd like to say that we've got a shake down -- from CV 0.9638 to pvt LB 0.9490
\nMany people got pvt LB score 0.94xx while their CV score is 0.93xx or something. They are called shake up IMO ;)
If what you want is a competition that public LB tracks well with private LB, here you are:
\nhttps://www.kaggle.com/c/landmark-recognition-2020
Hi, congratulations with the first place!
", "votes": 3, "replies": [] }, { "id": 1018407, "author_name": "Aniruddh Rajagopal", "author_url": "", "post_date": "2020-09-19T16:59:48.300000", "content": "Hi guys! may I ask where is the imbalance in dataset being handled?
", "votes": 1, "replies": [ { "id": 1018825, "author_name": "Dat Duong2", "author_url": "", "post_date": "2020-09-20T02:47:06.680000", "content": "No, they don't use any weighted loss. I checked their code which uses simple unweighted loss https://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution/blob/master/train.py#L270
", "votes": 0, "replies": [] } ] }, { "id": 992039, "author_name": "Elvin Aghammadzada", "author_url": "", "post_date": "2020-08-30T20:56:22.463000", "content": "congrats for that
", "votes": 1, "replies": [] }, { "id": 977635, "author_name": "Sandy Khosasi", "author_url": "", "post_date": "2020-08-19T15:54:01.753000", "content": "Congrats! Do you have any particular reason choosing Swish (https://arxiv.org/abs/1710.05941) rather than ReLU as activation?
", "votes": 1, "replies": [ { "id": 977959, "author_name": "Bo", "author_url": "", "post_date": "2020-08-19T20:17:19.447000", "content": "Here it probably doesn't matter. It might be better than Relu depending on the data and network.
", "votes": 1, "replies": [] } ] }, { "id": 977569, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2020-08-19T15:06:33.960000", "content": "Congrats @boliu0, @haqishen, and @garybios for a well deserved win. Thanks for sharing your solution.
\nI like the idea of using 2 types of CV and I wish I thought about using diagnosis as targets. Brilliant idea.
", "votes": 1, "replies": [] }, { "id": 977388, "author_name": "DarkCube", "author_url": "", "post_date": "2020-08-19T12:58:24.463000", "content": "excuse me, what is cv and lb? I'm really confused.
", "votes": 1, "replies": [ { "id": 977443, "author_name": "Bo", "author_url": "", "post_date": "2020-08-19T13:32:29.490000", "content": "cv means cross-validation. People use \"cv\" to refer to cross validation score
\nlb means leaderboard. People use \"lb\" to refer to leaderboard score
", "votes": 2, "replies": [] }, { "id": 977506, "author_name": "DarkCube", "author_url": "", "post_date": "2020-08-19T14:21:04.107000", "content": "Thank you so much for replying to my beginner *ss. Still, I have a few other questions if you don't mind.
\nThe way I understand it is that you used different versions of the Data as cv right? didn't you fear there would be an overlap between training data and cv data? isn't it bad practise to do so?
\nAnd one last thing: how did you choose the neural architectures? and do you believe there are better-engineered architectures out there that would be more efficient and/or yield better results?
", "votes": 0, "replies": [] }, { "id": 977548, "author_name": "Bo", "author_url": "", "post_date": "2020-08-19T14:52:17.147000", "content": "Check out Chris's post about cross validation: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/175614
\nIn short, there's never overlap between training and cv data
\nCurrently, best computer vision architectures are different version of Efficient Nets. Generally you don't need to worry about changing it, especially for beginners
", "votes": 2, "replies": [] }, { "id": 977604, "author_name": "DarkCube", "author_url": "", "post_date": "2020-08-19T15:36:24.053000", "content": "Thank you so much for your insight. and btw congrats for being 1st.
", "votes": 1, "replies": [] }, { "id": 977605, "author_name": "", "author_url": "", "post_date": "2020-08-19T15:36:24.180000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 976458, "author_name": "nosound", "author_url": "", "post_date": "2020-08-18T21:47:08.563000", "content": "Hi, congratulations with the first place! I scrolled through the comments but can't find the answer, have you used any patient level information? That is, other patient's images influencing the prediction.
", "votes": 1, "replies": [ { "id": 976466, "author_name": "Bo", "author_url": "", "post_date": "2020-08-18T22:00:43.057000", "content": "Thanks. No we didn't. We treat each data sample independently.
", "votes": 2, "replies": [] }, { "id": 976468, "author_name": "nosound", "author_url": "", "post_date": "2020-08-18T22:03:06.777000", "content": "Do you think there is no information in it, or very little additional information? How have you arrived to that conclusion? Thanks!
", "votes": 0, "replies": [] }, { "id": 976497, "author_name": "Bo", "author_url": "", "post_date": "2020-08-18T22:35:42.790000", "content": "Personally I don't think there's much (if any) additional information beyond single images and meta data. It's just my hunch, not conclusion based on experiments.
", "votes": 1, "replies": [] }, { "id": 976633, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T02:02:00.677000", "content": "What is your meta feature n_images? Isn't that based on patient_id?
you're right. It is based on patient_id information. (Although dropping it doesn't hurt our score.)
Congrats to you and your team on 1st place.
", "votes": 1, "replies": [] }, { "id": 975662, "author_name": "", "author_url": "", "post_date": "2020-08-18T12:07:27.437000", "content": "Big congrats to @boliu0, @haqishen, and @garybios!
\nMay I ask if you have done some ablation experiments to verify the following blurring & distortion augmentations are effective? I can understand distortion is at least non-harmful since the melanoma is in irregular shapes. But I don't understand why the blurring methods are working here. Could you explain for me? Thank you in advance!
\n A.OneOf([\n A.MotionBlur(blur_limit=5),\n A.MedianBlur(blur_limit=5),\n A.GaussianBlur(blur_limit=5),\n A.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n\n A.OneOf([\n A.OpticalDistortion(distort_limit=1.0),\n A.GridDistortion(num_steps=5, distort_limit=1.),\n A.ElasticTransform(alpha=3),\n ], p=0.7),\nAlso, very interested in the post-processing method df['pred'] = df['pred'].rank(pct=True). I think all my submissions are probs of melanoma classes when compare to your \"ranks percentile\". Could you explain a little bit?
It's hard to do ablation on augmentations -- the cv score variance is too large to draw meaningful conclusions when comparing two runs that only diff by an augmentation parameter.
\nWhen we tune the augmentations, we only check 1 fold's score without TTA, to have a general idea. However, even 5 fold cv with TTAx8 have noise (i.e. same code with different seeds lead to somewhat different scores).
\nSo the augmentation hyper parameters we used are not optimal, but should be good enough.
", "votes": 2, "replies": [] }, { "id": 975962, "author_name": "Bo", "author_url": "", "post_date": "2020-08-18T14:48:30.143000", "content": "Say your model 1's probability distribution has mean p=0.05; model 2's probability distribution has mean p=0.5. If you simply take average without ranking first, then model 2 would dominate model 1.
\nBy using rank, you can force them to make equal contributions.
", "votes": 1, "replies": [] }, { "id": 976061, "author_name": "", "author_url": "", "post_date": "2020-08-18T15:59:24.997000", "content": "Hi @boliu0, thank you for your explanation. I have some follow up questions hoping to clarify.
\n\n\nIt's hard to do ablation on augmentations -- the cv score variance is too large to draw meaningful conclusions when comparing two runs that only diff by an augmentation parameter.
\n
I think that's exactly the pitfall I've been stepping in. Each run I just change one type of augmentations and see the score difference, but as you said, the comparison may not be valid at all since the score variance is not neglectable.
\n\n\nWhen we tune the augmentations, we only check 1 fold's score without TTA, to have a general idea.
\n
So when you do the tuning work, do you modify a bunch of parameters/augmentation method together? Something like, Combo 1 gives 0.95, while Combo 2 gives me 0.98, then I'll stay with Combo2. Is there some systematic way to explore relatively optimal augmentation hyperparameters?
\n\n\nBy using rank, you can force them to make equal contributions.
\n
I'm a little bit lost here. Please correct me if my understanding is wrong. We predict each image in the test set with 2 models, one prediction has mean p = 0.05, the other prediction has mean p =0.5. If I just do simple averaging, the output p = 0.275. Ok, model 2 dominates model 1.
If we rank predictions firstly, I guess all pred_probs values are actually converted to percentile rank values. Therefore, the absolute difference between the original pred_probs has no influence anymore. which forces 2 models to contribute equally. Is this what you're saying?
So this trick is designed for the AUC metric of his competition specifically?
Thank you very much!
", "votes": 1, "replies": [] }, { "id": 976216, "author_name": "Bo", "author_url": "", "post_date": "2020-08-18T18:12:53.597000", "content": "\n\ndo you modify a bunch of parameters/augmentation method together?
\n
Usually make one change at a time
\n\n\nIs there some systematic way to explore relatively optimal augmentation hyperparameters?
\n
Use smaller model and smaller image for faster iteration
\n\n\nIf we rank predictions firstly, I guess all pred_probs values are actually converted to percentile rank values. Therefore, the absolute difference between the original pred_probs has no influence anymore. which forces 2 models to contribute equally. Is this what you're saying?
\n
Yes precisely
\n\n\nSo this trick is designed for the AUC metric of his competition specifically?
\n
It applies to any metric where only the relative order of predictions (not absolute values) matter
", "votes": 1, "replies": [] }, { "id": 976660, "author_name": "", "author_url": "", "post_date": "2020-08-19T02:45:12.880000", "content": "Hi @boliu0, great thanks for your patient explanations! Learn quite a lot! Once again, big congrats!
", "votes": 1, "replies": [] } ] }, { "id": 975660, "author_name": "jsyphil", "author_url": "", "post_date": "2020-08-18T12:06:22.880000", "content": "Congratulations to your team. You have unlocked quite a few secrets in the data here which have had many of us scratching our heads over the months.
\nWell done and very well deserved with your disciplined approach!
", "votes": 1, "replies": [] }, { "id": 975649, "author_name": "brachester", "author_url": "", "post_date": "2020-08-18T11:59:07.833000", "content": "Congrats on the win! I guess building a reliable CV is key in this comp. Feel bad that I'm still nowhere good enough and feel good that there's still more to learn!!!
", "votes": 1, "replies": [] }, { "id": 975435, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-08-18T09:49:30.567000", "content": "Congrats on result @boliu0 and team and thanks for writeup solution!
", "votes": 1, "replies": [] }, { "id": 975285, "author_name": "Andrés Miguel Torrubia Sáez", "author_url": "", "post_date": "2020-08-18T08:25:07.167000", "content": "Great writeup!
\nI have a question regarding targets:
\n\n\nTargets
\n
\n We find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01. 2020 data's diagnosis sets and 2019 data's diagnosis sets are different. We manually map 2020's to 2019's. There are 9 labels.2020 seborrheic keratosis -> BKL
\n
\n 2020 lichenoid keratosis -> BKL
\n 2020 solar lentigo -> BKL
\n 2020 lentigo NOS -> BKL
\n 2020 cafe-au-lait macule -> unknown
\n 2020 atypical melanocytic proliferation -> unknown
\n 2020 nevus -> NV
\n 2020 melanoma -> MEL
\n For prediction, we simply take the MEL class's softmax probability.
You mapped unknown as its own class. There were no unknown in 2018-19, so we thought unknown was actually \"anything except melanoma\", and we did:
diagnosis_dict = { \n 'unknown': -100,\n 'atypical melanocytic proliferation': -100,\n 'cafe-au-lait macule': -100,\n\n 'NV': 0,\n 'nevus': 0,\n 'lentigo NOS': 0,\n\n 'MEL': 1,\n 'melanoma': 1,\n\n 'BCC': 2,\n\n 'BKL': 3,\n 'solar lentigo': 3,\n 'seborrheic keratosis': 3, \n 'lichen planus-like keratosis': 3,\n 'lichenoid keratosis': 3,\n\n 'SCC': 4,\n\n 'VASC': 5,\n 'DF': 6,\n 'AK': 7,\n}\nThen our idea for the loss is \"when not unknown, use CrossEntropy as usual, but for unknown only add loss if we are predicting melanoma and label is unknown:
diagnosis_loss = self.diagnosis_loss(diag_pred,diag_targ)\n# diagnosis unknown set to -100 DOES give us some information: it is NOT melanoma, so we attribute accordinly:\ndiag_targ_unknown = torch.full_like(diag_targ,-100)\ndiag_targ_unknown[diag_targ==-100] = 1\ndiagnosis_loss += torch.clamp_min(self.diagnosis_unknown_loss(-diag_pred,diag_targ_unknown)-math.log(7.),0.)\ndiagnosis_loss = reduce_loss(diagnosis_loss, self.label_loss.reduction )\n
\n(that log(7) is an approximation of calculating -Log(1-Softmax(diag_pred)) in slightly different way more numerically stable).
Did you try other ways of attributing unknown?
Thanks! We've tried making all targets other than mel or nervous or BKL to unknown (4 classes in total in this case)
\nBut the result is quite similar, or a little bit worse compared to the 9 classes version.
", "votes": 3, "replies": [] } ] }, { "id": 975223, "author_name": "Abdur Rahim", "author_url": "", "post_date": "2020-08-18T07:51:11.263000", "content": "Great, Congratulations to you and your team, well done 😍
", "votes": 1, "replies": [] }, { "id": 975101, "author_name": "Innat", "author_url": "", "post_date": "2020-08-18T06:38:38.687000", "content": "\n\nTPU with TF seems to be dominating the public notebooks in this competition. It seems to be faster than a torch/GPU on EfficientNets. However, our experiments showed that its high LB scores are due to better luck with the public LB. Our torch/GPU models have a much better CV
\n
At first, big congrats. However, would you please elaborate on this? What kind of experiment have you conducted? I like to know your experimental comparison results between TF and Torch.
There are different components in the pipeline where pytorch is easier to use or more flexible than TF. For example, in TF, it's hard to achieve augmentations similar to ours.
", "votes": 3, "replies": [] } ] }, { "id": 975081, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-18T06:17:11.937000", "content": "Brilliant! Amazing work and great write up Bo, Gary and Qishen. I wish I had more time to try training and validating on all three datasets. Also interesting approach to keeping multiple years of CV. I might have tried that as well if I had started with competition earlier.
", "votes": 1, "replies": [] }, { "id": 984019, "author_name": "Baron Smith", "author_url": "", "post_date": "2020-08-24T19:15:43.390000", "content": "I'm a newbie to Kaggle, and I love how you said \"simply\" in your description above :) Congrats on your success!
", "votes": 2, "replies": [] }, { "id": 980809, "author_name": "Stefan Preusler", "author_url": "", "post_date": "2020-08-21T22:12:09.510000", "content": "Thanks for sharing! I stumbled over this topic by a paper, that compares the classification accuracy of dermatologists and a ResNet-50 model (https://www.sciencedirect.com/science/article/pii/S0959804919302217)l. Pretty amazing to see the progress!
", "votes": 2, "replies": [] }, { "id": 977828, "author_name": "Jeremy Berros", "author_url": "", "post_date": "2020-08-19T18:18:45.847000", "content": "Thanks @boliu0 for sharing this valuable insight and well structured Notebook.
\nI was trying to stick to Keras but it looks like Pytorch gives more flexibility (and better results overall).
\nAgain great job!! 👍
That's great, so PyTorch won again! :)
\nCongratulations to you and the team, who have after the deepfake \"fraud\", proved themselves again, that too with flying green colors of shake up. 😄
", "votes": 2, "replies": [ { "id": 975082, "author_name": "Gary", "author_url": "", "post_date": "2020-08-18T06:17:26.623000", "content": "thanks, @sarques, we are so excited that we can win the first place again after Deepfake, which gave me a renewed confidence in Kaggle.
", "votes": 4, "replies": [] }, { "id": 975086, "author_name": "Gajendra Saraswat", "author_url": "", "post_date": "2020-08-18T06:21:35.670000", "content": "Yes, it was truly deserved! What happened there has challenged a lot of people in many ways. After that happening, we were trying to read those legal terms a lot in this competition too, that was a mishap on their side, no need to let yourself down! We admire you, You are our Grand Master after all! :)
", "votes": 0, "replies": [] } ] }, { "id": 975033, "author_name": "", "author_url": "", "post_date": "2020-08-18T05:55:02.273000", "content": "", "votes": 2, "replies": [ { "id": 975067, "author_name": "", "author_url": "", "post_date": "2020-08-18T06:12:02.740000", "content": "", "votes": 0, "replies": [] }, { "id": 975267, "author_name": "", "author_url": "", "post_date": "2020-08-18T08:12:17.630000", "content": "", "votes": 3, "replies": [] }, { "id": 975817, "author_name": "", "author_url": "", "post_date": "2020-08-18T13:33:45.480000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 975032, "author_name": "", "author_url": "", "post_date": "2020-08-18T05:54:12.027000", "content": "", "votes": 2, "replies": [ { "id": 975040, "author_name": "", "author_url": "", "post_date": "2020-08-18T05:57:57.407000", "content": "", "votes": 4, "replies": [] }, { "id": 975282, "author_name": "", "author_url": "", "post_date": "2020-08-18T08:21:09.440000", "content": "", "votes": 4, "replies": [] } ] }, { "id": 975028, "author_name": "", "author_url": "", "post_date": "2020-08-18T05:52:09.683000", "content": "", "votes": 2, "replies": [] }, { "id": 975157, "author_name": "", "author_url": "", "post_date": "2020-08-18T07:14:34.270000", "content": "", "votes": 0, "replies": [] }, { "id": 975316, "author_name": "", "author_url": "", "post_date": "2020-08-18T08:39:19.280000", "content": "", "votes": 1, "replies": [] }, { "id": 975485, "author_name": "", "author_url": "", "post_date": "2020-08-18T10:11:50.143000", "content": "", "votes": 0, "replies": [] }, { "id": 982640, "author_name": "", "author_url": "", "post_date": "2020-08-23T14:43:30.717000", "content": "", "votes": -1, "replies": [ { "id": 982694, "author_name": "", "author_url": "", "post_date": "2020-08-23T15:34:07.617000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 977341, "author_name": "", "author_url": "", "post_date": "2020-08-19T12:31:09.767000", "content": "", "votes": -5, "replies": [] }, { "id": 975274, "author_name": "", "author_url": "", "post_date": "2020-08-18T08:17:56.733000", "content": "", "votes": -1, "replies": [] }, { "id": 975041, "author_name": "", "author_url": "", "post_date": "2020-08-18T05:59:18.530000", "content": "", "votes": -1, "replies": [] }, { "id": 2917844, "author_name": "", "author_url": "", "post_date": "2024-07-11T20:27:21.647000", "content": "", "votes": 0, "replies": [] }, { "id": 2916893, "author_name": "", "author_url": "", "post_date": "2024-07-11T10:31:47.343000", "content": "", "votes": 0, "replies": [] }, { "id": 2916393, "author_name": "", "author_url": "", "post_date": "2024-07-11T01:20:25.917000", "content": "", "votes": 0, "replies": [] }, { "id": 2899051, "author_name": "", "author_url": "", "post_date": "2024-07-01T13:16:31.650000", "content": "", "votes": 0, "replies": [] }, { "id": 2624862, "author_name": "", "author_url": "", "post_date": "2024-01-29T03:32:33.627000", "content": "", "votes": 0, "replies": [] }, { "id": 2566580, "author_name": "", "author_url": "", "post_date": "2023-12-19T02:15:57.623000", "content": "", "votes": 0, "replies": [] }, { "id": 1312606, "author_name": "", "author_url": "", "post_date": "2021-05-18T06:25:53.737000", "content": "", "votes": 0, "replies": [] }, { "id": 1265127, "author_name": "", "author_url": "", "post_date": "2021-04-06T16:31:19.183000", "content": "", "votes": 0, "replies": [] }, { "id": 1004460, "author_name": "", "author_url": "", "post_date": "2020-09-09T18:18:44.200000", "content": "", "votes": 0, "replies": [ { "id": 1159277, "author_name": "", "author_url": "", "post_date": "2021-01-19T06:57:12.490000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1004458, "author_name": "", "author_url": "", "post_date": "2020-09-09T18:18:29.867000", "content": "", "votes": 0, "replies": [] }, { "id": 991370, "author_name": "", "author_url": "", "post_date": "2020-08-30T11:06:53.033000", "content": "", "votes": 0, "replies": [] }, { "id": 988848, "author_name": "", "author_url": "", "post_date": "2020-08-28T10:26:21.040000", "content": "", "votes": 0, "replies": [] }, { "id": 987915, "author_name": "", "author_url": "", "post_date": "2020-08-27T16:04:46.580000", "content": "", "votes": 0, "replies": [] }, { "id": 987754, "author_name": "", "author_url": "", "post_date": "2020-08-27T13:45:03.220000", "content": "", "votes": 0, "replies": [] }, { "id": 986832, "author_name": "", "author_url": "", "post_date": "2020-08-26T20:23:04.587000", "content": "", "votes": 0, "replies": [] }, { "id": 985881, "author_name": "", "author_url": "", "post_date": "2020-08-26T04:48:04.963000", "content": "", "votes": 0, "replies": [] }, { "id": 985454, "author_name": "", "author_url": "", "post_date": "2020-08-25T18:28:08.630000", "content": "", "votes": 0, "replies": [] }, { "id": 984044, "author_name": "", "author_url": "", "post_date": "2020-08-24T19:30:51.893000", "content": "", "votes": 0, "replies": [] }, { "id": 982836, "author_name": "", "author_url": "", "post_date": "2020-08-23T17:43:50.110000", "content": "", "votes": 0, "replies": [] }, { "id": 982434, "author_name": "", "author_url": "", "post_date": "2020-08-23T11:14:16.383000", "content": "", "votes": 0, "replies": [] }, { "id": 980469, "author_name": "", "author_url": "", "post_date": "2020-08-21T16:06:49.210000", "content": "", "votes": 0, "replies": [] }, { "id": 980453, "author_name": "", "author_url": "", "post_date": "2020-08-21T15:53:53.360000", "content": "", "votes": 0, "replies": [] }, { "id": 979616, "author_name": "", "author_url": "", "post_date": "2020-08-21T02:01:48.193000", "content": "", "votes": 0, "replies": [] }, { "id": 979492, "author_name": "", "author_url": "", "post_date": "2020-08-20T22:11:08.107000", "content": "", "votes": 0, "replies": [] }, { "id": 977792, "author_name": "", "author_url": "", "post_date": "2020-08-19T17:53:10.647000", "content": "", "votes": 0, "replies": [ { "id": 977951, "author_name": "", "author_url": "", "post_date": "2020-08-19T20:13:15.783000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 977442, "author_name": "", "author_url": "", "post_date": "2020-08-19T13:32:01.953000", "content": "", "votes": 0, "replies": [ { "id": 977448, "author_name": "", "author_url": "", "post_date": "2020-08-19T13:38:03.547000", "content": "", "votes": 8, "replies": [] }, { "id": 977463, "author_name": "", "author_url": "", "post_date": "2020-08-19T13:46:08.363000", "content": "", "votes": 0, "replies": [] }, { "id": 977551, "author_name": "", "author_url": "", "post_date": "2020-08-19T14:54:48.990000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 976810, "author_name": "", "author_url": "", "post_date": "2020-08-19T05:35:37.450000", "content": "", "votes": 0, "replies": [] }, { "id": 976343, "author_name": "", "author_url": "", "post_date": "2020-08-18T19:53:22.183000", "content": "", "votes": 0, "replies": [] }, { "id": 976301, "author_name": "", "author_url": "", "post_date": "2020-08-18T19:20:39.500000", "content": "", "votes": 0, "replies": [ { "id": 976315, "author_name": "", "author_url": "", "post_date": "2020-08-18T19:29:52.243000", "content": "", "votes": 3, "replies": [] }, { "id": 976341, "author_name": "", "author_url": "", "post_date": "2020-08-18T19:49:39.310000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 975822, "author_name": "", "author_url": "", "post_date": "2020-08-18T13:37:47.030000", "content": "", "votes": 0, "replies": [ { "id": 975975, "author_name": "", "author_url": "", "post_date": "2020-08-18T14:56:41.253000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 975748, "author_name": "", "author_url": "", "post_date": "2020-08-18T12:58:48.863000", "content": "", "votes": 0, "replies": [] }, { "id": 975379, "author_name": "", "author_url": "", "post_date": "2020-08-18T09:12:53.840000", "content": "", "votes": 0, "replies": [ { "id": 977788, "author_name": "", "author_url": "", "post_date": "2020-08-19T17:51:26.680000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 975255, "author_name": "", "author_url": "", "post_date": "2020-08-18T08:07:09.853000", "content": "", "votes": 0, "replies": [ { "id": 975275, "author_name": "", "author_url": "", "post_date": "2020-08-18T08:18:16.700000", "content": "", "votes": 2, "replies": [] }, { "id": 975640, "author_name": "", "author_url": "", "post_date": "2020-08-18T11:48:49.807000", "content": "", "votes": 0, "replies": [] }, { "id": 975644, "author_name": "", "author_url": "", "post_date": "2020-08-18T11:52:45.760000", "content": "", "votes": 0, "replies": [] }, { "id": 975839, "author_name": "", "author_url": "", "post_date": "2020-08-18T13:45:03.453000", "content": "", "votes": 0, "replies": [] }, { "id": 977527, "author_name": "", "author_url": "", "post_date": "2020-08-19T14:34:55.077000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 975243, "author_name": "", "author_url": "", "post_date": "2020-08-18T08:02:26.387000", "content": "", "votes": 0, "replies": [ { "id": 975838, "author_name": "", "author_url": "", "post_date": "2020-08-18T13:43:48.913000", "content": "", "votes": 1, "replies": [] }, { "id": 976047, "author_name": "", "author_url": "", "post_date": "2020-08-18T15:52:28.093000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 975211, "author_name": "", "author_url": "", "post_date": "2020-08-18T07:43:15.377000", "content": "", "votes": 0, "replies": [ { "id": 975225, "author_name": "", "author_url": "", "post_date": "2020-08-18T07:53:15.623000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 975175, "author_name": "", "author_url": "", "post_date": "2020-08-18T07:23:04.977000", "content": "", "votes": 0, "replies": [ { "id": 975177, "author_name": "", "author_url": "", "post_date": "2020-08-18T07:25:40.297000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 975116, "author_name": "", "author_url": "", "post_date": "2020-08-18T06:49:04.550000", "content": "", "votes": 0, "replies": [] }, { "id": 1108591, "author_name": "", "author_url": "", "post_date": "2020-12-10T20:11:27.323000", "content": "", "votes": 2, "replies": [] }, { "id": 977386, "author_name": "", "author_url": "", "post_date": "2020-08-19T12:56:44.783000", "content": "", "votes": 0, "replies": [] }, { "id": 976752, "author_name": "", "author_url": "", "post_date": "2020-08-19T04:31:43.750000", "content": "", "votes": 0, "replies": [] }, { "id": 975864, "author_name": "", "author_url": "", "post_date": "2020-08-18T13:55:14.457000", "content": "", "votes": 0, "replies": [] }, { "id": 2920763, "author_name": "", "author_url": "", "post_date": "2024-07-13T19:26:06.507000", "content": "", "votes": 0, "replies": [] }, { "id": 991728, "author_name": "", "author_url": "", "post_date": "2020-08-30T16:13:13.550000", "content": "", "votes": 0, "replies": [] }, { "id": 989842, "author_name": "", "author_url": "", "post_date": "2020-08-29T06:34:49.267000", "content": "", "votes": 0, "replies": [] }, { "id": 988671, "author_name": "", "author_url": "", "post_date": "2020-08-28T07:35:17.073000", "content": "", "votes": 0, "replies": [] }, { "id": 988555, "author_name": "", "author_url": "", "post_date": "2020-08-28T05:55:27.373000", "content": "", "votes": 0, "replies": [] }, { "id": 987857, "author_name": "", "author_url": "", "post_date": "2020-08-27T15:20:11.650000", "content": "", "votes": 0, "replies": [] }, { "id": 987834, "author_name": "", "author_url": "", "post_date": "2020-08-27T14:58:14.083000", "content": "", "votes": 0, "replies": [] }, { "id": 985640, "author_name": "", "author_url": "", "post_date": "2020-08-25T22:29:11.500000", "content": "", "votes": 0, "replies": [] }, { "id": 981500, "author_name": "", "author_url": "", "post_date": "2020-08-22T13:42:14.537000", "content": "", "votes": 0, "replies": [] }, { "id": 978599, "author_name": "", "author_url": "", "post_date": "2020-08-20T09:23:02.910000", "content": "", "votes": 0, "replies": [] }, { "id": 975038, "author_name": "", "author_url": "", "post_date": "2020-08-18T05:57:24", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "975018": "Congratulations to all winners, especially my teammates Qishen @haqishen and Gary @garybios who are both computer vision competition veterans. I learned a lot from you. So happy to see you both on top of another LB after the deepfake drama.\n\nBig shoutout to my colleague Chris @cdeotte who I feel like served as an unofficial host of this competition with all your datasets, notebooks and tutorials. The competition wouldn't be this popular with all your contribution. We used your resized jpg images and triple stratified leak-free folds. Thanks Chris.\n\n## How we survived the shake\nThe test set is very small, with very small proportion of positive samples. So public LB has huge variance. Even the 2020 train data is not big enough for validation purpose, due to its small positive samples. \n\nTo have stable validation, we used 2018+2019+2020's data for both train and **validation**. We track two cv scores, `cv_all` and `cv_2020`. The former is much more stable than the latter.\n\nSecond key to LB survival is ensemble. Our single model's LB-cv correlation is essential 0, but the bigger the ensemble, the more stable the LB. In the last few days, our ensemble's LB was steadily increasing as we added better models.\n\nOur final ensemble 1 optimizes `cv_all` and final ensemble 2 optimizes `cv_2020`\n\nEnsemble 1: `cv_all=0.9845, cv_2020=0.9600, public=0.9442, private=0.9490` (1st place)\nEnsemble 2: `cv_2020=0.9638, public=0.9494, private=0.9481` (3rd place)\n\nOur best single model has `cv_2020=0.9481`\n\nAll scores above are 5 fold, TTAx8.\n\n## TPU vs GPU, TF vs torch\nTPU with TF seems to be dominating the public notebooks in this competition. It seems to be faster than torch/GPU on EfficientNets. However, our experiments showed that its high LB scores are due to better luck with the public LB. Our torch/GPU models have much better CV, thanks to the better PyTorch ecosystem and flexibility for faster experiments.\n\n## Models\nOur ensembles consists of EfficientNet B3-B7, se_resnext101, resnest101. There are models with or without meta data. Input size ranges from 384 to 896. (All input are from Chris's resized jpgs. For example, for 896 input we read 1024 jpgs and resize to 896.) \n\n## Meta data\nIn some (not all) of our models, we used 14 meta data from [this](https://www.kaggle.com/nroman/melanoma-pytorch-starter-efficientnet) and [this](https://www.kaggle.com/awsaf49/xgboost-tabular-data-ml-cv-85-lb-787#Image-Size) public notebooks, as illustrated below.\n\n\n## Targets\nWe find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01. \n\n2020 data's diagnosis sets and 2019 data's diagnosis sets are different. We manually map 2020's to 2019's. There are 9 labels.\n```\n2020 seborrheic keratosis -> BKL\n2020 lichenoid keratosis -> BKL\n2020 solar lentigo -> BKL\n2020 lentigo NOS -> BKL\n2020 cafe-au-lait macule -> unknown\n2020 atypical melanocytic proliferation -> unknown\n2020 nevus -> NV\n2020 melanoma -> MEL\n```\nFor prediction, we simply take the `MEL` class's softmax probability.\n\n## Augmentations\n```\ntransforms_train = A.Compose([\n A.Transpose(p=0.5),\n A.VerticalFlip(p=0.5),\n A.HorizontalFlip(p=0.5),\n A.RandomBrightness(limit=0.2, p=0.75),\n A.RandomContrast(limit=0.2, p=0.75),\n A.OneOf([\n A.MotionBlur(blur_limit=5),\n A.MedianBlur(blur_limit=5),\n A.GaussianBlur(blur_limit=5),\n A.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n\n A.OneOf([\n A.OpticalDistortion(distort_limit=1.0),\n A.GridDistortion(num_steps=5, distort_limit=1.),\n A.ElasticTransform(alpha=3),\n ], p=0.7),\n\n A.CLAHE(clip_limit=4.0, p=0.7),\n A.HueSaturationValue(hue_shift_limit=10, sat_shift_limit=20, val_shift_limit=10, p=0.5),\n A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),\n A.Resize(image_size, image_size),\n A.Cutout(max_h_size=int(image_size * 0.375), max_w_size=int(image_size * 0.375), num_holes=1, p=0.7), \n A.Normalize()\n])\n\ntransforms_val = A.Compose([\n A.Resize(image_size, image_size),\n A.Normalize()\n])\n```\n\n## Post processing\nWhen ensembling different folds, or different models, we first rank all the probabilities of each model/fold, to ensure they are evenly distributed. In pandas, it can be done by `df['pred'] = df['pred'].rank(pct=True)`\n\n## Code\nhttps://github.com/haqishen/SIIM-ISIC-Melanoma-Classification-1st-Place-Solution", "975841": "Congrats @haqishen @boliu0 @garybios and thanks for sharing. Looks like your validation discipline and labeling strategy made the difference. I'm glad to see that pytorch prevailed.", "975329": "Congratulations! Very proud to see a NVIDIA colleague win! \n\nYou made my first regret: I didn't think of using diagnosis as target. I'm tempted to retrain my bes model with it... I did use all 2019 targets, and it helped me.\n\nYour idea of using a global CV is also very interesting. Whatever the mean, having a reliable CV is the key. you found it. Again, congrats.\n\nYou did not say much about your models. You just used models as is with the right number of classes?", "975338": "One thing to add regarding diagnosis.\n\nWe found out the following thing:\n\n- Only training and evaluating on 2020 data\n- Removing all `diagnosis=nevus` from train (keep them in val)\n- Score drops a lot!\n- Remove similar amount of `diagnosis=unknown` \n- Score does not change (only tiny random range).\n\nAnyone figured this out? Maybe what it means? I was thinking that those labeled explicitly as nevus are more correct? Is there another bias? Unfortunately did not have too much time to follow up on it, but I expect that having the extra targets helps also with that a bit (we partly also did it, but did not see that much of improvements as you did).", "1495858": "Great work =))", "975165": "Happy that Pytorch won, congrats!\n\nOnly figured out on the last day that taking CV across all data is better.", "976608": "Do you automatically win another $10,000? Your model uses contextual information. You add the feature `0 * patient_id` and your model does not use contextual information. So you win both additional $5,000 prizes.\n\n> Special Prizes: Awarded to the top scoring models using or not using patient-level contextual information.\n\n With Context - $5,000 (Top-scoring model making use of patient-level contextual information)\n Without Context - $5,000 (Top-scoring model without using any patient-level contextual information)\n", "976570": "Training with `diagnosis` then using `MEL` class softmax is a great idea. Congratulations on an amazing model and 1st place!", "975263": "Congratulations on your 1st ! and thank you for the write up!\n\nI was surprised that you used 2020+2019+2018 for validation. A lot of kernels and my experiments immediately overfit in this case, achieving 0.99CV.\nI have questions about some of the fundamentals of making training stable. I took a lot of time here and didn't get to the more substantive considerations.\n\n- How do you coordinate augmentation?\n\nDo you remove the questionable augmentations one by one from the base transform with all the augmentations, or do you use something that's almost decided from the beginning and rarely changes?\nOf course why didn't you use the method of putting in more small holes like dropout, or overlapping images like mixup?\n\n- What did you use for optimizer, schedular? Did you learn the larger model from the beginning?\n\nYou need to adjust the lr when you change (models, datasets, lossfunction) small to large. I'm having a hard time finding just the right lr, scheduler, epoch for every competition, could you please tell me the procedure for adjusting lr?\nAssuming we use cosine annealing, how do we determine the max lr, train epoch?\n\n- Is ensemble optimization a baysian weight optimization using oof?\n\nDid you use spearman corr for selecting models or try stacking ?\n\n---\n\nIn the case of ensembling different models, we can just rank ensemble, but I didn't know that we can also use rank averaging the same model's different fold.\nIn general, the outputs of the same model's different fold have similar distribution, we often average the outputs of the model as it is. However, in this task, the test output distribution for each fold may be very different. I did not know how to deal with this problem. Thanks!\n\nI'm sorry for many questions, poor english. I would really appreciate it if you could respond to the questions. Once again, congratulations on your 1st !!!", "977420": "> How we survived the shake\n\nBut you did not. You gained 886 places. This can be called a definition of the shakeup.\nSurviving is having 1st place on Public and staying 1st on Private.\n\nThis is just a small remark. Otherwise - great job and congratulations with a 1st place.", "978766": "Hi, congratulations with the first place!", "1018407": "Hi guys! may I ask where is the imbalance in dataset being handled?", "992039": "congrats for that", "977635": "Congrats! Do you have any particular reason choosing Swish (https://arxiv.org/abs/1710.05941) rather than ReLU as activation?", "977569": "Congrats @boliu0, @haqishen, and @garybios for a well deserved win. Thanks for sharing your solution.\n\nI like the idea of using 2 types of CV and I wish I thought about using diagnosis as targets. Brilliant idea.", "977388": "excuse me, what is cv and lb? I'm really confused.", "976458": "Hi, congratulations with the first place! I scrolled through the comments but can't find the answer, have you used any patient level information? That is, other patient's images influencing the prediction.", "976307": "Congrats to you and your team on 1st place.", "975662": "Big congrats to @boliu0, @haqishen, and @garybios!\n\nMay I ask if you have done some ablation experiments to verify the following blurring & distortion augmentations are effective? I can understand distortion is at least non-harmful since the melanoma is in irregular shapes. But I don't understand why the blurring methods are working here. Could you explain for me? Thank you in advance!\n\n```\n A.OneOf([\n A.MotionBlur(blur_limit=5),\n A.MedianBlur(blur_limit=5),\n A.GaussianBlur(blur_limit=5),\n A.GaussNoise(var_limit=(5.0, 30.0)),\n ], p=0.7),\n\n A.OneOf([\n A.OpticalDistortion(distort_limit=1.0),\n A.GridDistortion(num_steps=5, distort_limit=1.),\n A.ElasticTransform(alpha=3),\n ], p=0.7),\n```", "975660": "Congratulations to your team. You have unlocked quite a few secrets in the data here which have had many of us scratching our heads over the months.\n\nWell done and very well deserved with your disciplined approach!", "975649": "Congrats on the win! I guess building a reliable CV is key in this comp. Feel bad that I'm still nowhere good enough and feel good that there's still more to learn!!!", "975435": "Congrats on result @boliu0 and team and thanks for writeup solution! ", "975285": "Great writeup!\n\nI have a question regarding targets:\n\n> Targets\nWe find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01. 2020 data's diagnosis sets and 2019 data's diagnosis sets are different. We manually map 2020's to 2019's. There are 9 labels.\n\n> 2020 seborrheic keratosis -> BKL\n> 2020 lichenoid keratosis -> BKL\n> 2020 solar lentigo -> BKL\n> 2020 lentigo NOS -> BKL\n> 2020 cafe-au-lait macule -> unknown\n> 2020 atypical melanocytic proliferation -> unknown\n> 2020 nevus -> NV\n> 2020 melanoma -> MEL\n> For prediction, we simply take the MEL class's softmax probability.\n\nYou mapped `unknown` as its own class. There were no `unknown` in 2018-19, so we thought `unknown` was actually \"anything except melanoma\", and we did:\n\n```\ndiagnosis_dict = { \n 'unknown': -100,\n 'atypical melanocytic proliferation': -100,\n 'cafe-au-lait macule': -100,\n \n 'NV': 0,\n 'nevus': 0,\n 'lentigo NOS': 0,\n \n 'MEL': 1,\n 'melanoma': 1,\n \n 'BCC': 2,\n \n 'BKL': 3,\n 'solar lentigo': 3,\n 'seborrheic keratosis': 3, \n 'lichen planus-like keratosis': 3,\n 'lichenoid keratosis': 3,\n \n 'SCC': 4,\n \n 'VASC': 5,\n 'DF': 6,\n 'AK': 7,\n}\n```\n\nThen our idea for the loss is \"when not unknown, use CrossEntropy as usual, but for unknown only add loss if we are predicting melanoma and label is `unknown`:\n\n```\ndiagnosis_loss = self.diagnosis_loss(diag_pred,diag_targ)\n# diagnosis unknown set to -100 DOES give us some information: it is NOT melanoma, so we attribute accordinly:\ndiag_targ_unknown = torch.full_like(diag_targ,-100)\ndiag_targ_unknown[diag_targ==-100] = 1\ndiagnosis_loss += torch.clamp_min(self.diagnosis_unknown_loss(-diag_pred,diag_targ_unknown)-math.log(7.),0.)\ndiagnosis_loss = reduce_loss(diagnosis_loss, self.label_loss.reduction )\n``` \n(that `log(7)` is an approximation of calculating -Log(1-Softmax(diag_pred)) in slightly different way more numerically stable).\n\nDid you try other ways of attributing `unknown`?", "975223": "Great, Congratulations to you and your team, well done 😍", "975101": "> TPU with TF seems to be dominating the public notebooks in this competition. It seems to be faster than a torch/GPU on EfficientNets. However, our experiments showed that its high LB scores are due to better luck with the public LB. Our torch/GPU models have a much better CV\n\nAt first, big congrats. However, would you please elaborate on this? What kind of experiment have you conducted? I like to know your experimental comparison results between `TF` and `Torch`. ", "975081": "Brilliant! Amazing work and great write up Bo, Gary and Qishen. I wish I had more time to try training and validating on all three datasets. Also interesting approach to keeping multiple years of CV. I might have tried that as well if I had started with competition earlier.", "984019": "I'm a newbie to Kaggle, and I love how you said \"simply\" in your description above :) Congrats on your success!", "980809": "Thanks for sharing! I stumbled over this topic by a paper, that compares the classification accuracy of dermatologists and a ResNet-50 model (https://www.sciencedirect.com/science/article/pii/S0959804919302217)l. Pretty amazing to see the progress!", "977828": "Thanks @boliu0 for sharing this valuable insight and well structured Notebook.\nI was trying to stick to Keras but it looks like Pytorch gives more flexibility (and better results overall).\nAgain great job!! 👍", "975035": "That's great, so PyTorch won again! :)\n\nCongratulations to you and the team, who have after the deepfake \"fraud\", proved themselves again, that too with flying green colors of shake up. 😄", "975033": "thanks. no tabular data at all? ", "975032": "Congrats for your result and thank you for the write-up. That is an interesting set of augmentations. \n\n> We find that using diagnosis as targets with cross entropy loss instead of binary target with BCE loss can boost score by ~0.01.\n\nInteresting , we also tried that and saw no much difference versus the binary target. I am wondering if there are other factors to that need to be in place for it to work. ", "975028": "Congrats @boliu0, @haqishen, and @garybios.\nAnd thanks for sharing! ", "975157": "A big **Congratulations**. ", "975316": "What computing resources did you use?", "975485": "Now that's a surprise!!! Winner uses PyTorch! :)\n\nI wasn't able to create bigger score with TPU than GPU so I decided something is wrong with my approach and I will not be able to fix it in time so I resigned. \n\nThank you for these details, this is very interesting.\n\nI am happy to see diagnosis was usable. I was trying to build another model to predict nevus only then to use hidden layer as features.\n", "982640": "@boliu0 Can you ps explain this? \n\n\n> 2020 seborrheic keratosis -> BKL\n2020 lichenoid keratosis -> BKL\n2020 solar lentigo -> BKL\n2020 lentigo NOS -> BKL\n2020 cafe-au-lait macule -> unknown\n2020 atypical melanocytic proliferation -> unknown\n2020 nevus -> NV\n2020 melanoma -> MEL\n\n\nThe 2020 dataset has the following 5 unique values, but you wrote 8 above:\n\n> unknown\nmelanoma\nnevus\nseborrheic keratosis\nlentigo NOS\n\nThe 2019 dataset has the following 5 unique values:\n\n\n> BCC\nBKL \nAK\nNV\nMEL\n\nPlease explain the mapping. And for the BKL 4 to 1 mapping, did u guys find those in papers? \n", "977341": "........................", "975274": "Congratulations, interesting that you guys used heavy augmentation and for sure that PyTorch made it to the top.", "975041": "Very clean write up, and easy to digest. Congrats on trusting your cvs and the well deserved shake up.", "2917844": "Hey @boliu0 , @haqishen , and @garybios -- in case you guys are up for it, we are running a new Melanoma detection challenge now:\n\nhttps://www.kaggle.com/competitions/isic-2024-challenge/overview\n\nWould love to see you participate :)", "2916893": "Hello Please i would like to talk to you in private \nThanks ", "2916393": "Congratulations team and thanks for sharing the solution 😀", "2899051": "Nice work!", "2624862": "Great work!", "2566580": "Congratulations! \ni have a question thar about you take 'use_meta'.\nfrom you commit code only have 4 sort of the \"use_meta\" was used ,while 14.", "1312606": "@haqishen , thanks for sharing . I wanted to know how to come up with such augmentations ! There are lot of augmentations you applied . ", "1265127": "Hi, the ensemble.py output a probability represented in a floating number between 0-1. How did you select the threshold between benign and malignant? did you simply use 0.5 as cut off?\n\nThanks and congrats!", "1004460": "I also want to ask, why do not need to use weights in nn.CrossEntropyLoss to account for imbalance numbers of labels.", "1004458": "I also want to ask, since you are predicting 9 labels, why do not need to use weights in nn.CrossEntropyLoss to account for imbalance numbers of labels?", "991370": "Thanks for sharing, Nvidia wins in final. Congratulations! ", "988848": "Congratulations!!!", "987915": "I'm quite new to Kaggle. Thanks for sharing and congratulations for the 1st place.", "987754": "hi, congrats good to go...\n", "986832": "Well done! Congrats!", "985881": "congratulatios! @haqishen @boliu0 @garybios Thanks for sharing. perfect use of pytorch", "985454": "What is LB? (As in public LB).\nHow did you come up with this model? (I want to know the thought process)\nAs you can see I guess I am the last person in this world starting with Kaggle. (Beginner here)", "984044": "Nice path for newbies", "982836": "Hi, the notebook is failing with the following error (see attachments) . any suggestions ? ", "982434": "Congratulations with the 1st place!", "980469": "Congratulations, and thanks for sharing!", "980453": "Congratulations on the first place!", "979616": "Congrats! ", "979492": "Congrats on the win!", "977792": "@boliu0 @haqishen @garybios Congratulations! So you guys used new 2019 data despite it having different distribution than the 2020 test data?", "977442": "Intuitively, how does training with cross-entropy loss and then classify using MEL's softmax improve the score?", "976810": "Awesome Idea", "976343": "Great! This is an amazing work! Congrats!", "976301": "`Our single model's LB-cv correlation is essential 0, but the bigger the ensemble, the more stable the LB. .`\n\nSorry, I am a newbie here, I didn't get the intuition of this line? ", "975822": "Did you trained each model( EfficientNet B3-B7, se_resnext101, resnest101) on all (384 to 896) resolution images, or you use different models of the same type for diff resolutions (for example one EffNet-B3 for 384, another EffNet-B3 for 512 etc)?\n\nThanks for grate work btw🎉", "975748": "Congrat! Thanks for sharing your great efforts!", "975379": "Huge congratz! And thanks for the writeup!\nI have a quick question. You mentioned that you used \"*diagnosis as targets with cross entropy loss instead of binary target with BCE loss.*\" Do you mean you made multi-class prediction using **diagnosis** feature as target with cross entropy loss (instead of using the **target** feature with BNC loss)?\nThanks and congratulations again!", "975255": "Congrats! Thanks for sharing ur great knowledge. \nI have one question. Could you please tell me which method you used when ensemble or stacking oof?", "975243": "Congratulations! Question about the augmentation: flip X/Y I understand, but Transpose and no \"Rotate 180\"?", "975211": "Congratulations - I'm so pleased to see the metadata used in this way - really good work. I tried to concat it to the end of one of my effnet models (much more simply than you did though) and it made an increase but in the end my highest scoring one was just an ensemble of effnets and a tabular model from @demirbkr .\n\nHow big a difference did the metadata make to your score on the private LB?", "975175": ">All scores above are 5 fold, TTAx8.\n\n\n\nAll TTAs are horizontal flips?", "975116": "Congrats , Very well written :D ", "1108591": "", "977386": "", "976752": "", "975864": "", "2920763": "Thanks. Really appreciate the share. ", "991728": "This is awesome, thanks for sharing", "989842": "Good work. Thanks for sharing.", "988671": "Thanks for sharing.", "988555": "Thanks for sharing and Congrats!\n", "987857": "Thanks for sharing and Congrats!", "987834": "It was really helpful! Thanks.", "985640": "Congrats and Thanks for Sharing man!!!", "981500": "Thanks for sharing and Congrats!", "978599": "Thank you!", "975038": "Great writeup - Thanks for sharing." } }