![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1224218%2F5cfec77ce60a48096ed84861c07ec52f%2FUntitled.png?generation=1567905802716797&alt=media\")
Congrats everyone with excellent results and ending of this exciting competition!
\n\nData\nWe used full-size images from 2015 train dataset to pretrain our models. Models trained on old data only gave us ~0.73-0.75 on public LB.\nCurrent competition train.\nOld competition test.
\n\nModels\nOverall we tried different architectures including se-resnext50, se-resnext101, densenet, but they were slow to train and performed poorly. The only NN architecture that performed well was efficient-net, so our final solution contains multiple b3, b4, and b5 networks.\nB3 image size: 300\nB4 image size: 460\nB5 image size: 456
\n\nPreprocessing\nI cropped black background on images and then resized to the desired image size before training my models. @aispiriants didn't perform any preprocessing except resizing images. I've also tried Ben's color preprocessing, but it didn't boost score, so I stop trying in the middle of the competition.
\n\nAugmentations\nWe used a lot of augmentations, at least more than I ever used before :) \nAll from the wonderful albumentations library: Blur, Flip, RandomBrightnessContrast, ShiftScaleRotate, ElasticTransform, Transpose, GridDistortion, HueSaturationValue, CLAHE, CoarseDropout.
\n\nTraining\nOverall training/submit flow is pretty simple:\nPretrain on old data (~80 epochs on b3, ~15 epochs on b5)\nTrain on current competition data (~50 epochs on b3, ~15 epochs on b5)\nPick top-performing epochs from the training process and blend them by a simple mean. We also used flips as TTA.
\n\nSecret ingredient :)\nWe pseudo-labeled current test data and then fine-tune networks again using train+pseudolabeled test. It gave us a huge boost and a lot of motivation, so we repeated a pseudo-labeling round once more that boosted a little bit more. We also pseudo-labeled previous competition test data to increase our training dataset size and fine-tuned some models on it too.
", "messages": [ { "id": 620807, "postDate": "2019-09-08T01:20:16.203Z", "content": "Congrats everyone with excellent results and ending of this exciting competition!
\n\nData\nWe used full-size images from 2015 train dataset to pretrain our models. Models trained on old data only gave us ~0.73-0.75 on public LB.\nCurrent competition train.\nOld competition test.
\n\nModels\nOverall we tried different architectures including se-resnext50, se-resnext101, densenet, but they were slow to train and performed poorly. The only NN architecture that performed well was efficient-net, so our final solution contains multiple b3, b4, and b5 networks.\nB3 image size: 300\nB4 image size: 460\nB5 image size: 456
\n\nPreprocessing\nI cropped black background on images and then resized to the desired image size before training my models. @aispiriants didn't perform any preprocessing except resizing images. I've also tried Ben's color preprocessing, but it didn't boost score, so I stop trying in the middle of the competition.
\n\nAugmentations\nWe used a lot of augmentations, at least more than I ever used before :) \nAll from the wonderful albumentations library: Blur, Flip, RandomBrightnessContrast, ShiftScaleRotate, ElasticTransform, Transpose, GridDistortion, HueSaturationValue, CLAHE, CoarseDropout.
\n\nTraining\nOverall training/submit flow is pretty simple:\nPretrain on old data (~80 epochs on b3, ~15 epochs on b5)\nTrain on current competition data (~50 epochs on b3, ~15 epochs on b5)\nPick top-performing epochs from the training process and blend them by a simple mean. We also used flips as TTA.
\n\nSecret ingredient :)\nWe pseudo-labeled current test data and then fine-tune networks again using train+pseudolabeled test. It gave us a huge boost and a lot of motivation, so we repeated a pseudo-labeling round once more that boosted a little bit more. We also pseudo-labeled previous competition test data to increase our training dataset size and fine-tuned some models on it too.
", "rawMarkdown": "Congrats everyone with excellent results and ending of this exciting competition!\n\n**Data**\nWe used full-size images from 2015 train dataset to pretrain our models. Models trained on old data only gave us ~0.73-0.75 on public LB.\nCurrent competition train.\nOld competition test.\n\n**Models**\nOverall we tried different architectures including se-resnext50, se-resnext101, densenet, but they were slow to train and performed poorly. The only NN architecture that performed well was efficient-net, so our final solution contains multiple b3, b4, and b5 networks.\nB3 image size: 300\nB4 image size: 460\nB5 image size: 456\n\n**Preprocessing**\nI cropped black background on images and then resized to the desired image size before training my models. @aispiriants didn't perform any preprocessing except resizing images. I've also tried Ben's color preprocessing, but it didn't boost score, so I stop trying in the middle of the competition.\n\n**Augmentations**\nWe used a lot of augmentations, at least more than I ever used before :) \nAll from the wonderful albumentations library: Blur, Flip, RandomBrightnessContrast, ShiftScaleRotate, ElasticTransform, Transpose, GridDistortion, HueSaturationValue, CLAHE, CoarseDropout.\n\n**Training**\nOverall training/submit flow is pretty simple:\nPretrain on old data (~80 epochs on b3, ~15 epochs on b5)\nTrain on current competition data (~50 epochs on b3, ~15 epochs on b5)\nPick top-performing epochs from the training process and blend them by a simple mean. We also used flips as TTA.\n\n**Secret ingredient :)**\nWe pseudo-labeled current test data and then fine-tune networks again using train+pseudolabeled test. It gave us a huge boost and a lot of motivation, so we repeated a pseudo-labeling round once more that boosted a little bit more. We also pseudo-labeled previous competition test data to increase our training dataset size and fine-tuned some models on it too.", "votes": 144 }, { "id": 620812, "postDate": "2019-09-08T01:25:03.223Z", "content": "We also had a submit that scores for the first or second place but didn't select it for the final result 😞
Anyway, it will be a good lesson to pay more attention to submit selection in future :)
", "rawMarkdown": "We also had a submit that scores for the first or second place but didn't select it for the final result 😞 \n\nAnyway, it will be a good lesson to pay more attention to submit selection in future :)", "votes": 7, "replies": [ { "id": 621195, "postDate": "2019-09-08T09:56:34.820Z", "content": "Ohh ! but anyways a great score. Many congrats :)
", "rawMarkdown": "Ohh ! but anyways a great score. Many congrats :)", "votes": 2 }, { "id": 621387, "postDate": "2019-09-08T13:27:36.213Z", "content": "@firenero, there is a reason you did not choose that submission.
\n\nThis competition is challenging indeed. I only chose my current position model because I was convinced it is my best even though it scored 0.796 on the public LB (i.e. private LB = 0.914) whilst my best public LB scoring model was 0.803 (i.e. private LB=0.910).
", "rawMarkdown": "@firenero, there is a reason you did not choose that submission. \n\nThis competition is challenging indeed. I only chose my current position model because I was convinced it is my best even though it scored 0.796 on the public LB (i.e. private LB = 0.914) whilst my best public LB scoring model was 0.803 (i.e. private LB=0.910)." } ] }, { "id": 620842, "postDate": "2019-09-08T02:16:33.450Z", "content": "Congratulations! You deserved it!
", "rawMarkdown": "Congratulations! You deserved it!", "votes": 3 }, { "id": 620816, "postDate": "2019-09-08T01:32:10.497Z", "content": "We pseudo-labeled current test data and then fine-tune networks again using train+pseudolabeled test. It gave us a huge boost and a lot of motivation, so we repeated a pseudo-labeling round once more that boosted a little bit more. We also pseudo-labeled previous competition test data to increase our training dataset size and fine-tuned some models on it too.
This was also part of our solution =) Great work Congratulations
", "rawMarkdown": "`We pseudo-labeled current test data and then fine-tune networks again using train+pseudolabeled test. It gave us a huge boost and a lot of motivation, so we repeated a pseudo-labeling round once more that boosted a little bit more. We also pseudo-labeled previous competition test data to increase our training dataset size and fine-tuned some models on it too.`\n\nThis was also part of our solution =) Great work Congratulations ", "votes": 3, "replies": [ { "id": 620821, "postDate": "2019-09-08T01:45:49.687Z", "content": "I feel like pseudo-labeling increased the robustness of our models a lot. Did you observe the same?
", "rawMarkdown": "I feel like pseudo-labeling increased the robustness of our models a lot. Did you observe the same?", "votes": 1 }, { "id": 620824, "postDate": "2019-09-08T01:50:13.133Z", "content": "I did try pseudo-labeling, but it didn't help me tbh.
", "rawMarkdown": "I did try pseudo-labeling, but it didn't help me tbh.", "votes": 2 }, { "id": 620827, "postDate": "2019-09-08T01:54:10.293Z", "content": "Yesss! In fact pseudo labeling was last thing on my list to experiment. And yesterday before going to bed I tried this and immediately got boost =)
", "rawMarkdown": "Yesss! In fact pseudo labeling was last thing on my list to experiment. And yesterday before going to bed I tried this and immediately got boost =) ", "votes": 4 }, { "id": 620839, "postDate": "2019-09-08T02:13:54.873Z", "content": "also, Looks like almost everyone was using EfficientNets, waiting for a top solution which didn't use it :p
", "rawMarkdown": "also, Looks like almost everyone was using EfficientNets, waiting for a top solution which didn't use it :p", "votes": 2 }, { "id": 620850, "postDate": "2019-09-08T02:22:18.810Z", "content": "to be honest, I have no idea why traditional networks like ResNets didn't work, but I'm eager to know why 😃
", "rawMarkdown": "to be honest, I have no idea why *traditional* networks like ResNets didn't work, but I'm eager to know why 😃 ", "votes": 4 }, { "id": 620860, "postDate": "2019-09-08T02:29:49.493Z", "content": "For Resnets always showing good training loss but horrible validation loss. And efficient nets were always showing stable loss for both training and validation. After some comparison I think it has to do with activation function by default efficient uses Swish. If you substitute ReLU in Resnet with Swish you will start observing good results =) At least this what i observed in quick experiments. Maybe somebody else have other insights =)
I think this may also have to do with the optimizer used. I tried using SGD and achieved almost perfect match with training and validation loss. I did not have time to submit it however.
", "rawMarkdown": "I think this may also have to do with the optimizer used. I tried using SGD and achieved almost perfect match with training and validation loss. I did not have time to submit it however. ", "votes": 3 }, { "id": 620870, "postDate": "2019-09-08T02:41:37.383Z", "content": "@drhabib Interesting info, thank you. In fact, your posts help us a lot at the beginning of the competition, especially about pretraining on old data :)\nCongrats on getting master btw :)
", "rawMarkdown": "@drhabib Interesting info, thank you. In fact, your posts help us a lot at the beginning of the competition, especially about pretraining on old data :)\nCongrats on getting master btw :)", "votes": 3 }, { "id": 620876, "postDate": "2019-09-08T02:47:05.413Z", "content": "Thank youuu and congrats on wining and getting one step closer to become Grandmaster=))
", "rawMarkdown": "Thank youuu and congrats on wining and getting one step closer to become Grandmaster=)) ", "votes": 3 }, { "id": 620884, "postDate": "2019-09-08T03:21:50.593Z", "content": "This pseudo-labelling is also behind my score boost. Congrat @firenero Mykhailo !!
", "rawMarkdown": "This pseudo-labelling is also behind my score boost. Congrat @firenero Mykhailo !!", "votes": 3 }, { "id": 621113, "postDate": "2019-09-08T07:57:40.130Z", "content": "This is so interesting that all top teams used pseudo tagging because we did not. Did a bunch of experiments and it didn't help, will elaborate in the solution post I am about to make. So doing so well without pseudo is nice for us ;)
", "rawMarkdown": "This is so interesting that all top teams used pseudo tagging because we did not. Did a bunch of experiments and it didn't help, will elaborate in the solution post I am about to make. So doing so well without pseudo is nice for us ;)", "votes": 7 }, { "id": 621127, "postDate": "2019-09-08T08:18:36.377Z", "content": "congrats for your top rank .. i too faced regret after seeing I dint select best solution:).. it all about learning
\n\n1) Could you point me to the various augs you did some are commmon ones but some i never used.\nHow you used them ,any kernel on git hub you can share ?\n2) Task for psuedo labelling, iheard lot but dint get a way to do it better way
", "rawMarkdown": "congrats for your top rank .. i too faced regret after seeing I dint select best solution:).. it all about learning\n\n1) Could you point me to the various augs you did some are commmon ones but some i never used.\nHow you used them ,any kernel on git hub you can share ?\n2) Task for psuedo labelling, iheard lot but dint get a way to do it better way", "votes": 1 } ] }, { "id": 621208, "postDate": "2019-09-08T10:29:46.593Z", "content": "@drhabib @firenero @ratthachat \nYou've mentioned that pseudo-labeling gave you a huge boost. Could you please specify the scores with and without pseudo-labeling?
", "rawMarkdown": "@drhabib @firenero @ratthachat \nYou've mentioned that pseudo-labeling gave you a huge boost. Could you please specify the scores with and without pseudo-labeling?", "votes": 4, "replies": [ { "id": 621356, "postDate": "2019-09-08T13:11:57.800Z", "content": "Hi @alexx, I just update my thread if you want a bit more details. But to summarize, public boost from 82x to 850 (ensemble) but private boost only from 925-6 to 929
", "rawMarkdown": "Hi @alexx, I just update my thread if you want a bit more details. But to summarize, public boost from 82x to 850 (ensemble) but private boost only from 925-6 to 929", "votes": 2 } ] }, { "id": 623665, "postDate": "2019-09-11T07:30:08.960Z", "content": "@firenero You should update the heading to '2nd place solution' :-) Congrats on the podium and thanks for sharing your solution!
", "rawMarkdown": "@firenero You should update the heading to '2nd place solution' :-) Congrats on the podium and thanks for sharing your solution!", "votes": 1, "replies": [ { "id": 624220, "postDate": "2019-09-11T20:14:19.327Z", "content": "haha, can't say I'm disappointed with this change :)
", "rawMarkdown": "haha, can't say I'm disappointed with this change :)", "votes": 1 } ] }, { "id": 621526, "postDate": "2019-09-08T16:13:31.530Z", "content": "Hi @firenero , Thank you for Sharing. What was the size of the training set from the 2015 and 2019 dataset? What machine configuration you used to train models?
", "rawMarkdown": "Hi @firenero , Thank you for Sharing. What was the size of the training set from the 2015 and 2019 dataset? What machine configuration you used to train models?", "votes": 1, "replies": [ { "id": 621720, "postDate": "2019-09-08T21:13:56.473Z", "content": "I don't remember a number of images in train sets, sorry. I think you could check them in the old competition data.
\n\nI have i9-7900X CPU and 2x1080TI.
", "rawMarkdown": "I don't remember a number of images in train sets, sorry. I think you could check them in the old competition data.\n\nI have i9-7900X CPU and 2x1080TI. ", "votes": 1 }, { "id": 621769, "postDate": "2019-09-08T23:14:24.320Z", "content": "Thanks @firenero . There were many dark and dirty images. First place solution also did not clean images for 2015 data or Fixing duplicate images in 2019 training set? Have you performed any training data cleanup?
", "rawMarkdown": "Thanks @firenero . There were many dark and dirty images. First place solution also did not clean images for 2015 data or Fixing duplicate images in 2019 training set? Have you performed any training data cleanup?" }, { "id": 621805, "postDate": "2019-09-09T01:01:44.573Z", "content": "No, just trained on all data.
", "rawMarkdown": "No, just trained on all data." } ] }, { "id": 621376, "postDate": "2019-09-08T13:21:23.840Z", "content": "Congrats @firenero, @aispiriants and thanks for sharing your solution overview.
", "rawMarkdown": "Congrats @firenero, @aispiriants and thanks for sharing your solution overview.", "votes": 1 }, { "id": 621350, "postDate": "2019-09-08T13:06:59.243Z", "content": "Congrats! Thank you for sharing a great solution....
", "rawMarkdown": "Congrats! Thank you for sharing a great solution....", "votes": 1 }, { "id": 621271, "postDate": "2019-09-08T11:46:08.257Z", "content": "Congrats! Neat solution!\nAbout: \"Pick top-performing epochs from the training process and blend them by a simple mean. \", what was your criteria for top-performance? Minimum val_loss, or maximum Kappa? And what loss function did you use?
", "rawMarkdown": "Congrats! Neat solution!\nAbout: \"Pick top-performing epochs from the training process and blend them by a simple mean. \", what was your criteria for top-performance? Minimum val_loss, or maximum Kappa? And what loss function did you use?", "votes": 1, "replies": [ { "id": 621344, "postDate": "2019-09-08T12:59:24.830Z", "content": "Loss function is MSE.
\n\nWe tried both ways of selection best model with close results but overall we prefer higher QWK values
", "rawMarkdown": "Loss function is MSE.\n\nWe tried both ways of selection best model with close results but overall we prefer higher QWK values", "votes": 1 } ] }, { "id": 621263, "postDate": "2019-09-08T11:36:27.517Z", "content": "Thanks,Great job,Your solution enlightened me.
", "rawMarkdown": "Thanks,Great job,Your solution enlightened me.", "votes": 1 }, { "id": 621225, "postDate": "2019-09-08T10:44:58.800Z", "content": "Congratulations and thanks for sharing :)
", "rawMarkdown": "Congratulations and thanks for sharing :)", "votes": 1 }, { "id": 621202, "postDate": "2019-09-08T10:08:24.320Z", "content": "congrats and gread job!
", "rawMarkdown": "congrats and gread job!", "votes": 1 }, { "id": 621184, "postDate": "2019-09-08T09:39:18.383Z", "content": "Congratulation and thanks for sharing. Did you use any ensemble technique ?
", "rawMarkdown": "Congratulation and thanks for sharing. Did you use any ensemble technique ?", "votes": 1, "replies": [ { "id": 621346, "postDate": "2019-09-08T13:01:55.663Z", "content": "We blended all our models and TTA using the simple mean. Didn't try anything fancy in this competition because complex ensembling techniques have never worked in previous comps for me :)
", "rawMarkdown": "We blended all our models and TTA using the simple mean. Didn't try anything fancy in this competition because complex ensembling techniques have never worked in previous comps for me :) ", "votes": 1 } ] }, { "id": 621177, "postDate": "2019-09-08T09:28:00.230Z", "content": "Congratulations ! This is great job. Can you answer 1 Q, would be help for me. How did you do pseudolabelling? \n@firenero
", "rawMarkdown": "Congratulations ! This is great job. Can you answer 1 Q, would be help for me. How did you do pseudolabelling? \n@firenero ", "votes": 1, "replies": [ { "id": 621367, "postDate": "2019-09-08T13:17:16.640Z", "content": "Congratulations and thanks for sharing. It's really inspiring to use pseudo label method.
", "rawMarkdown": "Congratulations and thanks for sharing. It's really inspiring to use pseudo label method.", "votes": 1 }, { "id": 621072, "postDate": "2019-09-08T07:23:11.610Z", "content": "Congrats! How did you choose confident predictions for pseudo-labelling? For example, in classification tasks we can choose samples by probability, but what about regression?
", "rawMarkdown": "Congrats! How did you choose confident predictions for pseudo-labelling? For example, in classification tasks we can choose samples by probability, but what about regression?", "votes": 1, "replies": [ { "id": 621353, "postDate": "2019-09-08T13:09:24.743Z", "content": "We didn't :)\nAll pseudo-labeled test was used without selecting confident or unconfident predictions.
", "rawMarkdown": "We didn't :)\nAll pseudo-labeled test was used without selecting confident or unconfident predictions.", "votes": 2 }, { "id": 621394, "postDate": "2019-09-08T13:32:13.073Z", "content": "Would appreciate if any eg you can provide in..\nThanks in advance..
", "rawMarkdown": "Would appreciate if any eg you can provide in..\nThanks in advance.." } ] }, { "id": 620950, "postDate": "2019-09-08T05:04:40.523Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 620918, "postDate": "2019-09-08T04:11:57.340Z", "content": "Congrats !! @firenero, Thank you for sharing 👍 , I also want to know why ResNets didnt work.
", "rawMarkdown": "Congrats !! @firenero, Thank you for sharing 👍 , I also want to know why ResNets didnt work.", "votes": 1 }, { "id": 620904, "postDate": "2019-09-08T04:01:50.143Z", "content": "Congratulations\nGreat work\nAnd Thanks for sharing your approach and insights.!! @firenero
", "rawMarkdown": "Congratulations\nGreat work\nAnd Thanks for sharing your approach and insights.!! @firenero ", "votes": 1 }, { "id": 620892, "postDate": "2019-09-08T03:41:32.567Z", "content": "Congrats and thank you for sharing !\nI have a question : what were the learning rate and the optimizer you used?\nFor me, all my models need between 15 and 20 epochs to converge and I used Adam(lr=1e-4) with ReduceLROnPlateau (wait=4) as scheduler
", "rawMarkdown": "Congrats and thank you for sharing !\nI have a question : what were the learning rate and the optimizer you used?\nFor me, all my models need between 15 and 20 epochs to converge and I used Adam(lr=1e-4) with ReduceLROnPlateau (wait=4) as scheduler ", "votes": 1 }, { "id": 620817, "postDate": "2019-09-08T01:34:14.667Z", "content": "Congrats on the prize and thanks for sharing your solution!
", "rawMarkdown": "Congrats on the prize and thanks for sharing your solution!", "votes": 1 }, { "id": 622135, "postDate": "2019-09-09T10:12:05.980Z", "content": "Congrats and thanks for sharing!
\n\nI have a question relating with the validation test strategy used for selecting the best models. Did you use a subset of the training set, do you use CV, an external dataset or the public PB?
\n\nThanks!
", "rawMarkdown": "Congrats and thanks for sharing!\n\nI have a question relating with the validation test strategy used for selecting the best models. Did you use a subset of the training set, do you use CV, an external dataset or the public PB?\n\nThanks!", "votes": 2, "replies": [ { "id": 622299, "postDate": "2019-09-09T13:33:26.793Z", "content": "For pretrain, we validated on all current train.\nFor current train, we validated on 5% of the current train and public LB.
", "rawMarkdown": "For pretrain, we validated on all current train.\nFor current train, we validated on 5% of the current train and public LB.", "votes": 3 } ] }, { "id": 621444, "postDate": "2019-09-08T14:18:13.440Z", "content": "Congratulations!\nIt seems that many people benefit from pseudo labeling, I learned a lot from you guys, thanks for sharing!
", "rawMarkdown": "Congratulations!\nIt seems that many people benefit from pseudo labeling, I learned a lot from you guys, thanks for sharing!", "votes": 2 }, { "id": 620907, "postDate": "2019-09-08T04:04:39.397Z", "content": "Congrats @firenero , Thanks for sharing your approach
", "rawMarkdown": "Congrats @firenero , Thanks for sharing your approach", "votes": 2 }, { "id": 620867, "postDate": "2019-09-08T02:39:22.843Z", "content": "Contrats! Thanks for sharing this very nice solution.
", "rawMarkdown": "Contrats! Thanks for sharing this very nice solution.", "votes": 2 }, { "id": 620849, "postDate": "2019-09-08T02:21:53.927Z", "content": "Congrats to you and your team!!!! Glad to know that Psuedolabeling worked out really well for you.\nI'm curious though about your models....are those classification or regression models?
", "rawMarkdown": "Congrats to you and your team!!!! Glad to know that Psuedolabeling worked out really well for you.\nI'm curious though about your models....are those classification or regression models?", "votes": 2, "replies": [ { "id": 620851, "postDate": "2019-09-08T02:23:01.083Z", "content": "All models are regression
", "rawMarkdown": "All models are regression", "votes": 1 } ] }, { "id": 620833, "postDate": "2019-09-08T02:05:47.020Z", "content": "Congrats, 4th is such a great achievement! I've also considered pseudo-labelling, but didn't really try it because I thought the public test set isn't large enough... Also heavy augmentations seems critical in this competition.
", "rawMarkdown": "Congrats, 4th is such a great achievement! I've also considered pseudo-labelling, but didn't really try it because I thought the public test set isn't large enough... Also heavy augmentations seems critical in this competition.", "votes": 2 }, { "id": 620825, "postDate": "2019-09-08T01:50:35.760Z", "content": "Congratulations :)
", "rawMarkdown": "Congratulations :)", "votes": 2 }, { "id": 621480, "postDate": "2019-09-08T15:15:28.387Z", "content": "Congrats!!! That secret weapon is so amazing, always to be thankful to learn new practical stuff from all you brilliant kagglers :)
", "rawMarkdown": "Congrats!!! That secret weapon is so amazing, always to be thankful to learn new practical stuff from all you brilliant kagglers :)" }, { "id": 623679, "postDate": "2019-09-11T07:46:39.100Z", "content": "Congrats with second place! It seems we have one more shakeup in this competition :)
", "rawMarkdown": "Congrats with second place! It seems we have one more shakeup in this competition :)", "votes": -1 }, { "id": 627944, "postDate": "2019-09-16T16:14:28.770Z", "content": "Congratulations @firenero and @aispiriants your your 2nd place ! And thanks a lot for sharing.
\n\nRegarding the training with 2015 data. Did you use all the data for the 80 epochs? Or was these epoch the same size than 2019 data?
\n\nHow did you deal with the unbalanced dataset?
\n\nDid you use any framework or tool over Pytorch? fastai, Ignite, another solution ... or your own framework/pipeline
\n\nRegarding fine tuning, when and how many epochs did you train the model head only / the whole net?
\n\nDid you change the efficientnet head?
", "rawMarkdown": "Congratulations @firenero and @aispiriants your your 2nd place ! And thanks a lot for sharing.\n\nRegarding the training with 2015 data. Did you use all the data for the 80 epochs? Or was these epoch the same size than 2019 data?\n\nHow did you deal with the unbalanced dataset?\n\nDid you use any framework or tool over Pytorch? fastai, Ignite, another solution ... or your own framework/pipeline\n\nRegarding fine tuning, when and how many epochs did you train the model head only / the whole net?\n\nDid you change the efficientnet head?\n" }, { "id": 625016, "postDate": "2019-09-12T16:10:40.493Z", "content": "Thank you for sharing. It was a great help!
", "rawMarkdown": "Thank you for sharing. It was a great help!" }, { "id": 624924, "postDate": "2019-09-12T14:26:30.787Z", "content": "Exiting!
", "rawMarkdown": "Exiting! " }, { "id": 624709, "postDate": "2019-09-12T09:56:22.923Z", "content": "cool
", "rawMarkdown": "cool\n" }, { "id": 624302, "postDate": "2019-09-12T00:27:11.277Z", "content": "Congratulations and thanks for sharing :)
", "rawMarkdown": "Congratulations and thanks for sharing :)" }, { "id": 623767, "postDate": "2019-09-11T09:31:09.520Z", "content": "Congratulations with such a high result!\n1. How did you choose such image sizes (300 for B3, 460 for B4, B5 for 456)? In general, why did you decide to make them different - to build an uncorrelated ensemble?\n2. As far as I understand, you dealt with this task as with regression. Did you choose the thresholds for the rounding to 5 classes, if yes, then how? If not (just 0.5 1.5 2.5 3.5), then why :)
", "rawMarkdown": "Congratulations with such a high result!\n1. How did you choose such image sizes (300 for B3, 460 for B4, B5 for 456)? In general, why did you decide to make them different - to build an uncorrelated ensemble?\n2. As far as I understand, you dealt with this task as with regression. Did you choose the thresholds for the rounding to 5 classes, if yes, then how? If not (just 0.5 1.5 2.5 3.5), then why :)" }, { "id": 623311, "postDate": "2019-09-10T18:21:48.353Z", "content": "👍
", "rawMarkdown": "👍 " }, { "id": 622946, "postDate": "2019-09-10T09:43:44.447Z", "content": "Congratulations! 👌
", "rawMarkdown": "Congratulations! 👌 " }, { "id": 622886, "postDate": "2019-09-10T08:02:14.150Z", "content": "Hi Mykhailo, congratulations again.
\n\nWhat about performance scores of the individual models previous to the ensembling?
\n\nDo you have information about which one is the best performing model previous to the ensembling?
\n\nThanks!
", "rawMarkdown": "Hi Mykhailo, congratulations again. \n\nWhat about performance scores of the individual models previous to the ensembling? \n\nDo you have information about which one is the best performing model previous to the ensembling? \n\nThanks!", "replies": [ { "id": 623060, "postDate": "2019-09-10T12:42:29.600Z", "content": "Best single model was b5 - scored 0.930 on private lb
", "rawMarkdown": "Best single model was b5 - scored 0.930 on private lb", "votes": 3 } ] }, { "id": 622611, "postDate": "2019-09-09T22:03:30.863Z", "content": "good job
", "rawMarkdown": "good job" }, { "id": 622263, "postDate": "2019-09-09T12:53:30.310Z", "content": "Congrats! Thanks for sharing, very cool!
", "rawMarkdown": "Congrats! Thanks for sharing, very cool!" }, { "id": 621817, "postDate": "2019-09-09T01:32:58.777Z", "content": "Thank you for sharing and congratulations!
", "rawMarkdown": "Thank you for sharing and congratulations!" }, { "id": 621490, "postDate": "2019-09-08T15:24:11.110Z", "content": "Congrats for achieving prize and thanks for sharing your solution!\nIt's very interesting you trained models about 50 to 80 epochs because my model normally give best score around 10 epochs. Did you use the pre-trained model with ImageNet or not pre-trained one??
", "rawMarkdown": "Congrats for achieving prize and thanks for sharing your solution!\nIt's very interesting you trained models about 50 to 80 epochs because my model normally give best score around 10 epochs. Did you use the pre-trained model with ImageNet or not pre-trained one??", "replies": [ { "id": 621534, "postDate": "2019-09-08T16:23:24.583Z", "content": "yeap why not overfitting with 20+ epoch solution? Very curious about the lr and lr schedule plan.
", "rawMarkdown": "yeap why not overfitting with 20+ epoch solution? Very curious about the lr and lr schedule plan." }, { "id": 621717, "postDate": "2019-09-08T21:11:38.870Z", "content": "Pretrained on imagenet. \nLR was 3e-4 with the following reducing to 1e-4 and 1e-5 when loss stopped decreasing for 4-5 epochs. I've used AdamW optimizer and my teammate used Adam (both with amsgrad). \nProbably I just didn't know that my models are overfitting :)
\n\nIn fact, I think that hard augmentations allowed to train much longer. When I tried training without so much augs, models stopped learning in like 10 epochs as you guys saying.
", "rawMarkdown": "Pretrained on imagenet. \nLR was 3e-4 with the following reducing to 1e-4 and 1e-5 when loss stopped decreasing for 4-5 epochs. I've used AdamW optimizer and my teammate used Adam (both with amsgrad). \nProbably I just didn't know that my models are overfitting :)\n\nIn fact, I think that hard augmentations allowed to train much longer. When I tried training without so much augs, models stopped learning in like 10 epochs as you guys saying.", "votes": 3 }, { "id": 622737, "postDate": "2019-09-10T03:06:41.367Z", "content": "Mykhailo - Did you retrain the entire models, or just a top layer? What output layer(s) did you use? Thanks!
", "rawMarkdown": "Mykhailo - Did you retrain the entire models, or just a top layer? What output layer(s) did you use? Thanks!" }, { "id": 623715, "postDate": "2019-09-11T08:33:26.797Z", "content": "@firenero \nCongrats!\n Did you retrain the entire models, or just a top layer? What output layer(s) did you use? How to make parameter groups?
", "rawMarkdown": "@firenero \nCongrats!\n Did you retrain the entire models, or just a top layer? What output layer(s) did you use? How to make parameter groups?" } ] }, { "id": 620923, "postDate": "2019-09-08T04:18:20.790Z", "content": "Congrats and thanks for sharing.\nCould you give some more details about how you performed pseudolabeling? Did you label just public test data? Well done!
", "rawMarkdown": "Congrats and thanks for sharing.\nCould you give some more details about how you performed pseudolabeling? Did you label just public test data? Well done!", "replies": [ { "id": 621391, "postDate": "2019-09-08T13:29:46.570Z", "content": "https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/107926#621367\nHere I've described how we did pseudo-label.\nWe labeled public test and public test from 2015 competition.
", "rawMarkdown": "https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/107926#621367\nHere I've described how we did pseudo-label.\nWe labeled public test and public test from 2015 competition. ", "votes": 1 }, { "id": 621393, "postDate": "2019-09-08T13:31:13.413Z", "content": "Thanks.. how about the transformations hsv shift,elastic etc. They are very time consuming...
", "rawMarkdown": "Thanks.. how about the transformations hsv shift,elastic etc. They are very time consuming..." }, { "id": 621718, "postDate": "2019-09-08T21:12:12.380Z", "content": "I just let my CPU do its job 😄
", "rawMarkdown": "I just let my CPU do its job 😄 " }, { "id": 622997, "postDate": "2019-09-10T10:55:47.743Z", "content": "@firenero So you didn't use any probability threshold to label the test data, did you? As I understand, you labeled all the public test dataset, right? Thanks
", "rawMarkdown": "@firenero So you didn't use any probability threshold to label the test data, did you? As I understand, you labeled all the public test dataset, right? Thanks" } ] }, { "id": 625827, "postDate": "2019-09-13T13:58:43.060Z", "rawMarkdown": "", "isDeleted": true }, { "id": 625017, "postDate": "2019-09-12T16:11:16.460Z", "rawMarkdown": "", "isDeleted": true }, { "id": 623706, "postDate": "2019-09-11T08:25:06.320Z", "rawMarkdown": "", "isDeleted": true }, { "id": 621065, "postDate": "2019-09-08T07:21:23.243Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 621257, "postDate": "2019-09-08T11:34:51.220Z", "content": "Impressive! Thank you for sharing
", "rawMarkdown": "Impressive! Thank you for sharing", "votes": 1 }, { "id": 620814, "postDate": "2019-09-08T01:30:56.760Z", "content": "Congrats and thanks for sharing!
", "rawMarkdown": "Congrats and thanks for sharing!", "votes": 2 }, { "id": 620835, "postDate": "2019-09-08T02:10:09.367Z", "content": "congrats and thanks for sharing 👍
", "rawMarkdown": "congrats and thanks for sharing 👍 ", "votes": 1 }, { "id": 624805, "postDate": "2019-09-12T11:28:11.047Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 623099, "postDate": "2019-09-10T13:33:22.633Z", "content": "Congratulations! Thank you for sharing.
", "rawMarkdown": "Congratulations! Thank you for sharing." }, { "id": 622329, "postDate": "2019-09-09T14:03:56.847Z", "content": "Thank you for sharing.
", "rawMarkdown": "Thank you for sharing." }, { "id": 621841, "postDate": "2019-09-09T02:24:15.900Z", "content": "Thank you, your sharing helped me
", "rawMarkdown": "Thank you, your sharing helped me" } ], "comments": [ { "id": 620812, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T01:25:03.223000", "content": "We also had a submit that scores for the first or second place but didn't select it for the final result 😞
Anyway, it will be a good lesson to pay more attention to submit selection in future :)
", "votes": 7, "replies": [ { "id": 621195, "author_name": "Filemon", "author_url": "", "post_date": "2019-09-08T09:56:34.820000", "content": "Ohh ! but anyways a great score. Many congrats :)
", "votes": 2, "replies": [] }, { "id": 621387, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-08T13:27:36.213000", "content": "@firenero, there is a reason you did not choose that submission.
\n\nThis competition is challenging indeed. I only chose my current position model because I was convinced it is my best even though it scored 0.796 on the public LB (i.e. private LB = 0.914) whilst my best public LB scoring model was 0.803 (i.e. private LB=0.910).
", "votes": 0, "replies": [] } ] }, { "id": 620842, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-09-08T02:16:33.450000", "content": "Congratulations! You deserved it!
", "votes": 3, "replies": [] }, { "id": 620816, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-08T01:32:10.497000", "content": "We pseudo-labeled current test data and then fine-tune networks again using train+pseudolabeled test. It gave us a huge boost and a lot of motivation, so we repeated a pseudo-labeling round once more that boosted a little bit more. We also pseudo-labeled previous competition test data to increase our training dataset size and fine-tuned some models on it too.
This was also part of our solution =) Great work Congratulations
", "votes": 3, "replies": [ { "id": 620821, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T01:45:49.687000", "content": "I feel like pseudo-labeling increased the robustness of our models a lot. Did you observe the same?
", "votes": 1, "replies": [] }, { "id": 620824, "author_name": "Rishabh Agrahari", "author_url": "", "post_date": "2019-09-08T01:50:13.133000", "content": "I did try pseudo-labeling, but it didn't help me tbh.
", "votes": 2, "replies": [] }, { "id": 620827, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-08T01:54:10.293000", "content": "Yesss! In fact pseudo labeling was last thing on my list to experiment. And yesterday before going to bed I tried this and immediately got boost =)
", "votes": 4, "replies": [] }, { "id": 620839, "author_name": "Rishabh Agrahari", "author_url": "", "post_date": "2019-09-08T02:13:54.873000", "content": "also, Looks like almost everyone was using EfficientNets, waiting for a top solution which didn't use it :p
", "votes": 2, "replies": [] }, { "id": 620850, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T02:22:18.810000", "content": "to be honest, I have no idea why traditional networks like ResNets didn't work, but I'm eager to know why 😃
", "votes": 4, "replies": [] }, { "id": 620860, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-08T02:29:49.493000", "content": "For Resnets always showing good training loss but horrible validation loss. And efficient nets were always showing stable loss for both training and validation. After some comparison I think it has to do with activation function by default efficient uses Swish. If you substitute ReLU in Resnet with Swish you will start observing good results =) At least this what i observed in quick experiments. Maybe somebody else have other insights =)
I think this may also have to do with the optimizer used. I tried using SGD and achieved almost perfect match with training and validation loss. I did not have time to submit it however.
", "votes": 3, "replies": [] }, { "id": 620870, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T02:41:37.383000", "content": "@drhabib Interesting info, thank you. In fact, your posts help us a lot at the beginning of the competition, especially about pretraining on old data :)\nCongrats on getting master btw :)
", "votes": 3, "replies": [] }, { "id": 620876, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-08T02:47:05.413000", "content": "Thank youuu and congrats on wining and getting one step closer to become Grandmaster=))
", "votes": 3, "replies": [] }, { "id": 620884, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T03:21:50.593000", "content": "This pseudo-labelling is also behind my score boost. Congrat @firenero Mykhailo !!
", "votes": 3, "replies": [] }, { "id": 621113, "author_name": "Psi", "author_url": "", "post_date": "2019-09-08T07:57:40.130000", "content": "This is so interesting that all top teams used pseudo tagging because we did not. Did a bunch of experiments and it didn't help, will elaborate in the solution post I am about to make. So doing so well without pseudo is nice for us ;)
", "votes": 7, "replies": [] }, { "id": 621127, "author_name": "Jaideep", "author_url": "", "post_date": "2019-09-08T08:18:36.377000", "content": "congrats for your top rank .. i too faced regret after seeing I dint select best solution:).. it all about learning
\n\n1) Could you point me to the various augs you did some are commmon ones but some i never used.\nHow you used them ,any kernel on git hub you can share ?\n2) Task for psuedo labelling, iheard lot but dint get a way to do it better way
", "votes": 1, "replies": [] } ] }, { "id": 621208, "author_name": "alexx", "author_url": "", "post_date": "2019-09-08T10:29:46.593000", "content": "@drhabib @firenero @ratthachat \nYou've mentioned that pseudo-labeling gave you a huge boost. Could you please specify the scores with and without pseudo-labeling?
", "votes": 4, "replies": [ { "id": 621356, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T13:11:57.800000", "content": "Hi @alexx, I just update my thread if you want a bit more details. But to summarize, public boost from 82x to 850 (ensemble) but private boost only from 925-6 to 929
", "votes": 2, "replies": [] } ] }, { "id": 623665, "author_name": "Vopani", "author_url": "", "post_date": "2019-09-11T07:30:08.960000", "content": "@firenero You should update the heading to '2nd place solution' :-) Congrats on the podium and thanks for sharing your solution!
", "votes": 1, "replies": [ { "id": 624220, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-11T20:14:19.327000", "content": "haha, can't say I'm disappointed with this change :)
", "votes": 1, "replies": [] } ] }, { "id": 621526, "author_name": "Himanshu Gamit", "author_url": "", "post_date": "2019-09-08T16:13:31.530000", "content": "Hi @firenero , Thank you for Sharing. What was the size of the training set from the 2015 and 2019 dataset? What machine configuration you used to train models?
", "votes": 1, "replies": [ { "id": 621720, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T21:13:56.473000", "content": "I don't remember a number of images in train sets, sorry. I think you could check them in the old competition data.
\n\nI have i9-7900X CPU and 2x1080TI.
", "votes": 1, "replies": [] }, { "id": 621769, "author_name": "Himanshu Gamit", "author_url": "", "post_date": "2019-09-08T23:14:24.320000", "content": "Thanks @firenero . There were many dark and dirty images. First place solution also did not clean images for 2015 data or Fixing duplicate images in 2019 training set? Have you performed any training data cleanup?
", "votes": 0, "replies": [] }, { "id": 621805, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-09T01:01:44.573000", "content": "No, just trained on all data.
", "votes": 0, "replies": [] } ] }, { "id": 621376, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-08T13:21:23.840000", "content": "Congrats @firenero, @aispiriants and thanks for sharing your solution overview.
", "votes": 1, "replies": [] }, { "id": 621350, "author_name": "KhanhVD", "author_url": "", "post_date": "2019-09-08T13:06:59.243000", "content": "Congrats! Thank you for sharing a great solution....
", "votes": 1, "replies": [] }, { "id": 621271, "author_name": "FelipeKitamura, MD, PhD", "author_url": "", "post_date": "2019-09-08T11:46:08.257000", "content": "Congrats! Neat solution!\nAbout: \"Pick top-performing epochs from the training process and blend them by a simple mean. \", what was your criteria for top-performance? Minimum val_loss, or maximum Kappa? And what loss function did you use?
", "votes": 1, "replies": [ { "id": 621344, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T12:59:24.830000", "content": "Loss function is MSE.
\n\nWe tried both ways of selection best model with close results but overall we prefer higher QWK values
", "votes": 1, "replies": [] } ] }, { "id": 621263, "author_name": "ynhuhu", "author_url": "", "post_date": "2019-09-08T11:36:27.517000", "content": "Thanks,Great job,Your solution enlightened me.
", "votes": 1, "replies": [] }, { "id": 621225, "author_name": "Noah Weber", "author_url": "", "post_date": "2019-09-08T10:44:58.800000", "content": "Congratulations and thanks for sharing :)
", "votes": 1, "replies": [] }, { "id": 621202, "author_name": "keleas", "author_url": "", "post_date": "2019-09-08T10:08:24.320000", "content": "congrats and gread job!
", "votes": 1, "replies": [] }, { "id": 621184, "author_name": "CC Joshua ", "author_url": "", "post_date": "2019-09-08T09:39:18.383000", "content": "Congratulation and thanks for sharing. Did you use any ensemble technique ?
", "votes": 1, "replies": [ { "id": 621346, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T13:01:55.663000", "content": "We blended all our models and TTA using the simple mean. Didn't try anything fancy in this competition because complex ensembling techniques have never worked in previous comps for me :)
", "votes": 1, "replies": [] } ] }, { "id": 621177, "author_name": "Anurag Trivedi", "author_url": "", "post_date": "2019-09-08T09:28:00.230000", "content": "Congratulations ! This is great job. Can you answer 1 Q, would be help for me. How did you do pseudolabelling? \n@firenero
", "votes": 1, "replies": [ { "id": 621367, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T13:17:16.640000", "content": "Congratulations and thanks for sharing. It's really inspiring to use pseudo label method.
", "votes": 1, "replies": [] }, { "id": 621072, "author_name": "Vassily Baranov", "author_url": "", "post_date": "2019-09-08T07:23:11.610000", "content": "Congrats! How did you choose confident predictions for pseudo-labelling? For example, in classification tasks we can choose samples by probability, but what about regression?
", "votes": 1, "replies": [ { "id": 621353, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T13:09:24.743000", "content": "We didn't :)\nAll pseudo-labeled test was used without selecting confident or unconfident predictions.
", "votes": 2, "replies": [] }, { "id": 621394, "author_name": "Jaideep", "author_url": "", "post_date": "2019-09-08T13:32:13.073000", "content": "Would appreciate if any eg you can provide in..\nThanks in advance..
", "votes": 0, "replies": [] } ] }, { "id": 620950, "author_name": "simon", "author_url": "", "post_date": "2019-09-08T05:04:40.523000", "content": "Congratulations!
", "votes": 1, "replies": [] }, { "id": 620918, "author_name": "Rohit Modi", "author_url": "", "post_date": "2019-09-08T04:11:57.340000", "content": "Congrats !! @firenero, Thank you for sharing 👍 , I also want to know why ResNets didnt work.
", "votes": 1, "replies": [] }, { "id": 620904, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-09-08T04:01:50.143000", "content": "Congratulations\nGreat work\nAnd Thanks for sharing your approach and insights.!! @firenero
", "votes": 1, "replies": [] }, { "id": 620892, "author_name": "Firas Baba", "author_url": "", "post_date": "2019-09-08T03:41:32.567000", "content": "Congrats and thank you for sharing !\nI have a question : what were the learning rate and the optimizer you used?\nFor me, all my models need between 15 and 20 epochs to converge and I used Adam(lr=1e-4) with ReduceLROnPlateau (wait=4) as scheduler
", "votes": 1, "replies": [] }, { "id": 620817, "author_name": "ilovescience", "author_url": "", "post_date": "2019-09-08T01:34:14.667000", "content": "Congrats on the prize and thanks for sharing your solution!
", "votes": 1, "replies": [] }, { "id": 622135, "author_name": "jordi.delatorre", "author_url": "", "post_date": "2019-09-09T10:12:05.980000", "content": "Congrats and thanks for sharing!
\n\nI have a question relating with the validation test strategy used for selecting the best models. Did you use a subset of the training set, do you use CV, an external dataset or the public PB?
\n\nThanks!
", "votes": 2, "replies": [ { "id": 622299, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-09T13:33:26.793000", "content": "For pretrain, we validated on all current train.\nFor current train, we validated on 5% of the current train and public LB.
", "votes": 3, "replies": [] } ] }, { "id": 621444, "author_name": "Qishen Ha", "author_url": "", "post_date": "2019-09-08T14:18:13.440000", "content": "Congratulations!\nIt seems that many people benefit from pseudo labeling, I learned a lot from you guys, thanks for sharing!
", "votes": 2, "replies": [] }, { "id": 620907, "author_name": "Kranthi Kumar", "author_url": "", "post_date": "2019-09-08T04:04:39.397000", "content": "Congrats @firenero , Thanks for sharing your approach
", "votes": 2, "replies": [] }, { "id": 620867, "author_name": "Yuanhao", "author_url": "", "post_date": "2019-09-08T02:39:22.843000", "content": "Contrats! Thanks for sharing this very nice solution.
", "votes": 2, "replies": [] }, { "id": 620849, "author_name": "Bibek", "author_url": "", "post_date": "2019-09-08T02:21:53.927000", "content": "Congrats to you and your team!!!! Glad to know that Psuedolabeling worked out really well for you.\nI'm curious though about your models....are those classification or regression models?
", "votes": 2, "replies": [ { "id": 620851, "author_name": "Mykhailo Matviiv", "author_url": "", "post_date": "2019-09-08T02:23:01.083000", "content": "All models are regression
", "votes": 1, "replies": [] } ] }, { "id": 620833, "author_name": "Homoalways", "author_url": "", "post_date": "2019-09-08T02:05:47.020000", "content": "Congrats, 4th is such a great achievement! I've also considered pseudo-labelling, but didn't really try it because I thought the public test set isn't large enough... Also heavy augmentations seems critical in this competition.
", "votes": 2, "replies": [] }, { "id": 620825, "author_name": "Rishabh Agrahari", "author_url": "", "post_date": "2019-09-08T01:50:35.760000", "content": "Congratulations :)
", "votes": 2, "replies": [] }, { "id": 621480, "author_name": "jayjhlin", "author_url": "", "post_date": "2019-09-08T15:15:28.387000", "content": "Congrats!!! That secret weapon is so amazing, always to be thankful to learn new practical stuff from all you brilliant kagglers :)
", "votes": 0, "replies": [] }, { "id": 623679, "author_name": "", "author_url": "", "post_date": "2019-09-11T07:46:39.100000", "content": "", "votes": -1, "replies": [] }, { "id": 627944, "author_name": "", "author_url": "", "post_date": "2019-09-16T16:14:28.770000", "content": "", "votes": 0, "replies": [] }, { "id": 625016, "author_name": "", "author_url": "", "post_date": "2019-09-12T16:10:40.493000", "content": "", "votes": 0, "replies": [] }, { "id": 624924, "author_name": "", "author_url": "", "post_date": "2019-09-12T14:26:30.787000", "content": "", "votes": 0, "replies": [] }, { "id": 624709, "author_name": "", "author_url": "", "post_date": "2019-09-12T09:56:22.923000", "content": "", "votes": 0, "replies": [] }, { "id": 624302, "author_name": "", "author_url": "", "post_date": "2019-09-12T00:27:11.277000", "content": "", "votes": 0, "replies": [] }, { "id": 623767, "author_name": "", "author_url": "", "post_date": "2019-09-11T09:31:09.520000", "content": "", "votes": 0, "replies": [] }, { "id": 623311, "author_name": "", "author_url": "", "post_date": "2019-09-10T18:21:48.353000", "content": "", "votes": 0, "replies": [] }, { "id": 622946, "author_name": "", "author_url": "", "post_date": "2019-09-10T09:43:44.447000", "content": "", "votes": 0, "replies": [] }, { "id": 622886, "author_name": "", "author_url": "", "post_date": "2019-09-10T08:02:14.150000", "content": "", "votes": 0, "replies": [ { "id": 623060, "author_name": "", "author_url": "", "post_date": "2019-09-10T12:42:29.600000", "content": "", "votes": 3, "replies": [] } ] }, { "id": 622611, "author_name": "", "author_url": "", "post_date": "2019-09-09T22:03:30.863000", "content": "", "votes": 0, "replies": [] }, { "id": 622263, "author_name": "", "author_url": "", "post_date": "2019-09-09T12:53:30.310000", "content": "", "votes": 0, "replies": [] }, { "id": 621817, "author_name": "", "author_url": "", "post_date": "2019-09-09T01:32:58.777000", "content": "", "votes": 0, "replies": [] }, { "id": 621490, "author_name": "", "author_url": "", "post_date": "2019-09-08T15:24:11.110000", "content": "", "votes": 0, "replies": [ { "id": 621534, "author_name": "", "author_url": "", "post_date": "2019-09-08T16:23:24.583000", "content": "", "votes": 0, "replies": [] }, { "id": 621717, "author_name": "", "author_url": "", "post_date": "2019-09-08T21:11:38.870000", "content": "", "votes": 3, "replies": [] }, { "id": 622737, "author_name": "", "author_url": "", "post_date": "2019-09-10T03:06:41.367000", "content": "", "votes": 0, "replies": [] }, { "id": 623715, "author_name": "", "author_url": "", "post_date": "2019-09-11T08:33:26.797000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 620923, "author_name": "", "author_url": "", "post_date": "2019-09-08T04:18:20.790000", "content": "", "votes": 0, "replies": [ { "id": 621391, "author_name": "", "author_url": "", "post_date": "2019-09-08T13:29:46.570000", "content": "", "votes": 1, "replies": [] }, { "id": 621393, "author_name": "", "author_url": "", "post_date": "2019-09-08T13:31:13.413000", "content": "", "votes": 0, "replies": [] }, { "id": 621718, "author_name": "", "author_url": "", "post_date": "2019-09-08T21:12:12.380000", "content": "", "votes": 0, "replies": [] }, { "id": 622997, "author_name": "", "author_url": "", "post_date": "2019-09-10T10:55:47.743000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 625827, "author_name": "", "author_url": "", "post_date": "2019-09-13T13:58:43.060000", "content": "", "votes": 0, "replies": [] }, { "id": 625017, "author_name": "", "author_url": "", "post_date": "2019-09-12T16:11:16.460000", "content": "", "votes": 0, "replies": [] }, { "id": 623706, "author_name": "", "author_url": "", "post_date": "2019-09-11T08:25:06.320000", "content": "", "votes": 0, "replies": [] }, { "id": 621065, "author_name": "", "author_url": "", "post_date": "2019-09-08T07:21:23.243000", "content": "", "votes": 1, "replies": [] }, { "id": 621257, "author_name": "", "author_url": "", "post_date": "2019-09-08T11:34:51.220000", "content": "", "votes": 1, "replies": [] }, { "id": 620814, "author_name": "", "author_url": "", "post_date": "2019-09-08T01:30:56.760000", "content": "", "votes": 2, "replies": [] }, { "id": 620835, "author_name": "", "author_url": "", "post_date": "2019-09-08T02:10:09.367000", "content": "", "votes": 1, "replies": [] }, { "id": 624805, "author_name": "", "author_url": "", "post_date": "2019-09-12T11:28:11.047000", "content": "", "votes": 0, "replies": [] }, { "id": 623099, "author_name": "", "author_url": "", "post_date": "2019-09-10T13:33:22.633000", "content": "", "votes": 0, "replies": [] }, { "id": 622329, "author_name": "", "author_url": "", "post_date": "2019-09-09T14:03:56.847000", "content": "", "votes": 0, "replies": [] }, { "id": 621841, "author_name": "", "author_url": "", "post_date": "2019-09-09T02:24:15.900000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "620807": "Congrats everyone with excellent results and ending of this exciting competition!\n\n**Data**\nWe used full-size images from 2015 train dataset to pretrain our models. Models trained on old data only gave us ~0.73-0.75 on public LB.\nCurrent competition train.\nOld competition test.\n\n**Models**\nOverall we tried different architectures including se-resnext50, se-resnext101, densenet, but they were slow to train and performed poorly. The only NN architecture that performed well was efficient-net, so our final solution contains multiple b3, b4, and b5 networks.\nB3 image size: 300\nB4 image size: 460\nB5 image size: 456\n\n**Preprocessing**\nI cropped black background on images and then resized to the desired image size before training my models. @aispiriants didn't perform any preprocessing except resizing images. I've also tried Ben's color preprocessing, but it didn't boost score, so I stop trying in the middle of the competition.\n\n**Augmentations**\nWe used a lot of augmentations, at least more than I ever used before :) \nAll from the wonderful albumentations library: Blur, Flip, RandomBrightnessContrast, ShiftScaleRotate, ElasticTransform, Transpose, GridDistortion, HueSaturationValue, CLAHE, CoarseDropout.\n\n**Training**\nOverall training/submit flow is pretty simple:\nPretrain on old data (~80 epochs on b3, ~15 epochs on b5)\nTrain on current competition data (~50 epochs on b3, ~15 epochs on b5)\nPick top-performing epochs from the training process and blend them by a simple mean. We also used flips as TTA.\n\n**Secret ingredient :)**\nWe pseudo-labeled current test data and then fine-tune networks again using train+pseudolabeled test. It gave us a huge boost and a lot of motivation, so we repeated a pseudo-labeling round once more that boosted a little bit more. We also pseudo-labeled previous competition test data to increase our training dataset size and fine-tuned some models on it too.", "620812": "We also had a submit that scores for the first or second place but didn't select it for the final result 😞 \n\nAnyway, it will be a good lesson to pay more attention to submit selection in future :)", "620842": "Congratulations! You deserved it!", "620816": "`We pseudo-labeled current test data and then fine-tune networks again using train+pseudolabeled test. It gave us a huge boost and a lot of motivation, so we repeated a pseudo-labeling round once more that boosted a little bit more. We also pseudo-labeled previous competition test data to increase our training dataset size and fine-tuned some models on it too.`\n\nThis was also part of our solution =) Great work Congratulations ", "621208": "@drhabib @firenero @ratthachat \nYou've mentioned that pseudo-labeling gave you a huge boost. Could you please specify the scores with and without pseudo-labeling?", "623665": "@firenero You should update the heading to '2nd place solution' :-) Congrats on the podium and thanks for sharing your solution!", "621526": "Hi @firenero , Thank you for Sharing. What was the size of the training set from the 2015 and 2019 dataset? What machine configuration you used to train models?", "621376": "Congrats @firenero, @aispiriants and thanks for sharing your solution overview.", "621350": "Congrats! Thank you for sharing a great solution....", "621271": "Congrats! Neat solution!\nAbout: \"Pick top-performing epochs from the training process and blend them by a simple mean. \", what was your criteria for top-performance? Minimum val_loss, or maximum Kappa? And what loss function did you use?", "621263": "Thanks,Great job,Your solution enlightened me.", "621225": "Congratulations and thanks for sharing :)", "621202": "congrats and gread job!", "621184": "Congratulation and thanks for sharing. Did you use any ensemble technique ?", "621177": "Congratulations ! This is great job. Can you answer 1 Q, would be help for me. How did you do pseudolabelling? \n@firenero ", "621154": "Congratulations and thanks for sharing. It's really inspiring to use pseudo label method.", "621072": "Congrats! How did you choose confident predictions for pseudo-labelling? For example, in classification tasks we can choose samples by probability, but what about regression?", "620950": "Congratulations!", "620918": "Congrats !! @firenero, Thank you for sharing 👍 , I also want to know why ResNets didnt work.", "620904": "Congratulations\nGreat work\nAnd Thanks for sharing your approach and insights.!! @firenero ", "620892": "Congrats and thank you for sharing !\nI have a question : what were the learning rate and the optimizer you used?\nFor me, all my models need between 15 and 20 epochs to converge and I used Adam(lr=1e-4) with ReduceLROnPlateau (wait=4) as scheduler ", "620817": "Congrats on the prize and thanks for sharing your solution!", "622135": "Congrats and thanks for sharing!\n\nI have a question relating with the validation test strategy used for selecting the best models. Did you use a subset of the training set, do you use CV, an external dataset or the public PB?\n\nThanks!", "621444": "Congratulations!\nIt seems that many people benefit from pseudo labeling, I learned a lot from you guys, thanks for sharing!", "620907": "Congrats @firenero , Thanks for sharing your approach", "620867": "Contrats! Thanks for sharing this very nice solution.", "620849": "Congrats to you and your team!!!! Glad to know that Psuedolabeling worked out really well for you.\nI'm curious though about your models....are those classification or regression models?", "620833": "Congrats, 4th is such a great achievement! I've also considered pseudo-labelling, but didn't really try it because I thought the public test set isn't large enough... Also heavy augmentations seems critical in this competition.", "620825": "Congratulations :)", "621480": "Congrats!!! That secret weapon is so amazing, always to be thankful to learn new practical stuff from all you brilliant kagglers :)", "623679": "Congrats with second place! It seems we have one more shakeup in this competition :)", "627944": "Congratulations @firenero and @aispiriants your your 2nd place ! And thanks a lot for sharing.\n\nRegarding the training with 2015 data. Did you use all the data for the 80 epochs? Or was these epoch the same size than 2019 data?\n\nHow did you deal with the unbalanced dataset?\n\nDid you use any framework or tool over Pytorch? fastai, Ignite, another solution ... or your own framework/pipeline\n\nRegarding fine tuning, when and how many epochs did you train the model head only / the whole net?\n\nDid you change the efficientnet head?\n", "625016": "Thank you for sharing. It was a great help!", "624924": "Exiting! ", "624709": "cool\n", "624302": "Congratulations and thanks for sharing :)", "623767": "Congratulations with such a high result!\n1. How did you choose such image sizes (300 for B3, 460 for B4, B5 for 456)? In general, why did you decide to make them different - to build an uncorrelated ensemble?\n2. As far as I understand, you dealt with this task as with regression. Did you choose the thresholds for the rounding to 5 classes, if yes, then how? If not (just 0.5 1.5 2.5 3.5), then why :)", "623311": "👍 ", "622946": "Congratulations! 👌 ", "622886": "Hi Mykhailo, congratulations again. \n\nWhat about performance scores of the individual models previous to the ensembling? \n\nDo you have information about which one is the best performing model previous to the ensembling? \n\nThanks!", "622611": "good job", "622263": "Congrats! Thanks for sharing, very cool!", "621817": "Thank you for sharing and congratulations!", "621490": "Congrats for achieving prize and thanks for sharing your solution!\nIt's very interesting you trained models about 50 to 80 epochs because my model normally give best score around 10 epochs. Did you use the pre-trained model with ImageNet or not pre-trained one??", "620923": "Congrats and thanks for sharing.\nCould you give some more details about how you performed pseudolabeling? Did you label just public test data? Well done!", "625827": "", "625017": "", "623706": "", "621065": "", "621257": "Impressive! Thank you for sharing", "620814": "Congrats and thanks for sharing!", "620835": "congrats and thanks for sharing 👍 ", "624805": "Thanks for sharing!", "623099": "Congratulations! Thank you for sharing.", "622329": "Thank you for sharing.", "621841": "Thank you, your sharing helped me" } }