Please, share you experience. We had a few submissions which could bring us to bronze, but we made wrong assumption which submission is better on private LB... and lost medals.
\n\nAnd you?
", "messages": [ { "id": "620813", "postDate": "09/08/2019 01:27:12", "content": "Please, share you experience. We had a few submissions which could bring us to bronze, but we made wrong assumption which submission is better on private LB... and lost medals.
\n\nAnd you?
", "rawMarkdown": "Please, share you experience. We had a few submissions which could bring us to bronze, but we made wrong assumption which submission is better on private LB... and lost medals. \n\nAnd you?", "votes": null }, { "id": "620830", "postDate": "09/08/2019 01:58:30", "content": "We had a few submissions which would have gotten us silver 🤕
\n\nThe public leaderboard deceived us as we thought we were getting poor scores due to overfitting when in fact, those models that performed ~0.76 - 0.78 on public LB had private LB scores of above 0.920.
\n\nI think manual selection of best models now would be nice as it would increase the overall scores on the entire leaderboard. It would be nice if people wrote about what worked and what didn't so I'll be waiting on that!
", "rawMarkdown": "We had a few submissions which would have gotten us silver 🤕 \n\nThe public leaderboard deceived us as we thought we were getting poor scores due to overfitting when in fact, those models that performed ~0.76 - 0.78 on public LB had private LB scores of above 0.920. \n\nI think manual selection of best models now would be nice as it would increase the overall scores on the entire leaderboard. It would be nice if people wrote about what worked and what didn't so I'll be waiting on that!", "votes": null }, { "id": "620991", "postDate": "09/08/2019 06:11:58", "content": "I am still in bronze zone but interesting experience to share. I selected EfficientNet different ensembles for final selections and my real best was single model just forked from https://www.kaggle.com/drhabib/starter-kernel-for-0-79 with image size 256 and fine-tuned. I didn't select it and even didn't use it in ensemble because its public LB was rather low (0.783). And in fact this single model worked better than all my ensembles.
", "rawMarkdown": "I am still in bronze zone but interesting experience to share. I selected EfficientNet different ensembles for final selections and my real best was single model just forked from https://www.kaggle.com/drhabib/starter-kernel-for-0-79 with image size 256 and fine-tuned. I didn't select it and even didn't use it in ensemble because its public LB was rather low (0.783). And in fact this single model worked better than all my ensembles.", "votes": null }, { "id": "621027", "postDate": "09/08/2019 06:47:47", "content": "I dropped from gold/near gold (top 20) to ~1800-1900th place. In other words, I dropped ~1800 positions. I was a bit in shock this morning! Anyone else had a similar drop? I had an older submissions with a score of 0.933 (which included many of the models of this poor submission). Thus I hypothesize that I must have included a model which made really awful predictions. Perhaps I should have clipped the outputs of my regression models, but it had no effect on the public leader board.
", "rawMarkdown": "I dropped from gold/near gold (top 20) to ~1800-1900th place. In other words, I dropped ~1800 positions. I was a bit in shock this morning! Anyone else had a similar drop? I had an older submissions with a score of 0.933 (which included many of the models of this poor submission). Thus I hypothesize that I must have included a model which made really awful predictions. Perhaps I should have clipped the outputs of my regression models, but it had no effect on the public leader board.", "votes": null }, { "id": "621048", "postDate": "09/08/2019 07:01:48", "content": "Wold have been silver - but dropped out of the medal completely :)
", "rawMarkdown": "Wold have been silver - but dropped out of the medal completely :)", "votes": null }, { "id": "621100", "postDate": "09/08/2019 07:45:57", "content": "@akensert You were indeed in bad luck! We almost got the same experience as you but we just a bit more lucky ... See you in the next round Akensert!
\n\nEDIT just to clarify, we did select our most promising 2 subs, but this selection process is really difficult!
", "rawMarkdown": "akensert You were indeed in bad luck! We almost got the same experience as you but we just a bit more lucky ... See you in the next round Akensert!\n\n**EDIT** just to clarify, we did select our most promising 2 subs, but this selection process is really difficult!", "votes": null }, { "id": "621135", "postDate": "09/08/2019 08:31:51", "content": "@ratthachat Yeah, I noticed your drop too. Unfortunate! Indeed, see you in the next round :)
", "rawMarkdown": "ratthachat Yeah, I noticed your drop too. Unfortunate! Indeed, see you in the next round :)", "votes": null }, { "id": "621159", "postDate": "09/08/2019 09:05:11", "content": "Down by 297 and lost a bronze medal, but an important thing I want to share is I did partial submission(i.e turned of GPU and submitted just the CSV file ) during experimenting and only ran inference kernel for the 2 selected model. At that time it made easy to check results within 1s but I am regretting now because there may be some models which performed better than the selected model. I have learnt my lesson.
\n\nThis is my first deep learning competition and I am proud of myself. Next stop Severstal: Steel Defect Detection
", "rawMarkdown": "Down by 297 and lost a bronze medal, but an important thing I want to share is I did partial submission(i.e turned of GPU and submitted just the CSV file ) during experimenting and only ran inference kernel for the 2 selected model. At that time it made easy to check results within 1s but I am regretting now because there may be some models which performed better than the selected model. I have learnt my lesson.\n\nThis is my first deep learning competition and I am proud of myself. Next stop Severstal: Steel Defect Detection", "votes": null }, { "id": "621207", "postDate": "09/08/2019 10:28:49", "content": "Very weird competition! Had over 0.150 score difference between public and private score. The public leaderboard data seemed to come from an entirely different distribution compared to the rest.
", "rawMarkdown": "Very weird competition! Had over 0.150 score difference between public and private score. The public leaderboard data seemed to come from an entirely different distribution compared to the rest.", "votes": null }, { "id": "621282", "postDate": "09/08/2019 11:58:14", "content": "that's a drop from top 10 to 18xx. MY GOD.
", "rawMarkdown": "that's a drop from top 10 to 18xx. MY GOD.", "votes": null }, { "id": "621467", "postDate": "09/08/2019 14:45:23", "content": "I know you since our LB rank is quite similar during the last month.\nSorry to hear that... You should have ranked near my position on the private LB as well...
", "rawMarkdown": "I know you since our LB rank is quite similar during the last month.\nSorry to hear that... You should have ranked near my position on the private LB as well...", "votes": null }, { "id": "621518", "postDate": "09/08/2019 16:02:06", "content": "@haqishen Congrats to your gold! :) And thank you. It is how it is. Although, I feel this drop between public and private LB was a real hard slap in the face. I invested quite a lot of time and effort into this competition (like many of us of course). I truly understand that it's likely to overfit with close to 200 submissions. But from a potential gold placement to 1855th, that's rough.
\n\n@ratthachat Most promising meaning highest public LB score? I selected my highest scoring one, and also included my biggest ensemble for it's \"robustness\".
", "rawMarkdown": "haqishen Congrats to your gold! :) And thank you. It is how it is. Although, I feel this drop between public and private LB was a real hard slap in the face. I invested quite a lot of time and effort into this competition (like many of us of course). I truly understand that it's likely to overfit with close to 200 submissions. But from a potential gold placement to 1855th, that's rough. \n\n@ratthachat Most promising meaning highest public LB score? I selected my highest scoring one, and also included my biggest ensemble for it's \"robustness\".", "votes": null }, { "id": "621552", "postDate": "09/08/2019 16:44:19", "content": "Yeah, I understand your feeling...\nAs you are experienced enough for getting gold medals, I think the most important thing is that what we can learn from the competition. If you finally find out what made this shake (but not just hypothesis), it will be an unique experience which will help you avoid it in the future.\nFurther more, I would like to learn from your 0.933 solution. Would you please write a quick summary on what you done? Thank you ;)
", "rawMarkdown": "Yeah, I understand your feeling...\nAs you are experienced enough for getting gold medals, I think the most important thing is that what we can learn from the competition. If you finally find out what made this shake (but not just hypothesis), it will be an unique experience which will help you avoid it in the future.\nFurther more, I would like to learn from your 0.933 solution. Would you please write a quick summary on what you done? Thank you ;)", "votes": null }, { "id": "621564", "postDate": "09/08/2019 17:03:06", "content": "Same with me. Never KO again...
", "rawMarkdown": "Same with me. Never KO again...", "votes": null }, { "id": "621602", "postDate": "09/08/2019 17:47:55", "content": "@haqishen Very true. Firstly, I actually peaked around the beginning of August (for single models). What later boosted my score was ensembling. In short, my main focus was to preprocess the images to avoid learning spurious features (shapes of the black area etc.) I ended up with: crop_image_from_gray --> circle_crop (using the biggest radius instead of smallest (as in the public kernel), thus needing to pad the image)--> randomly cropped it further --> randomly created black top/bottom borders. I then augmented it pretty heavily (brightness/color-balance, rotation/flips, zoom, blur, gaussian noise, and sometimes random deletion (black boxes)). Finally cv2.addWeighted or clahe. I saved the best performing models throughout time, and [weight-]averaged them. The architectures I most commonly used were EfficientNetB5 and Xception, with varying input dims (256x256x3 and 299x299x3 seemed to suffice for EfficientNetB5 and Xception respectively). I started out with TTA but towards the end I removed it and included more models instead (seemed more promising). DenseNet, ResNeXt, ResNets, Inception, InceptionResNet, NASNetLarge, EfficientNetB6/7 didn't seem do it for me at the time (although most of them I didn't give a fair try). And yes, also, I always started the training with the inclusion of a balanced subset of the external dataset, and later fine-tuned it to the competition data (sometimes balanced undersampling, sometimes not), with AdamOptimizer(lr=1e-4) during external training and lr=1e-5 during fine-tuning. There is so much to cover, but this is what came to mind!
\n\nNote that the above accounts for both my 0.933 and 0.877(best public LB) submissions (which still boggles my mind). The difference is that a few additional models were included for the 0.877 submission (with low weights for the weighted-average..). I have yet to understand this.
\n\nAnd btw, thank you for sharing your solution too! (I just read your post)
", "rawMarkdown": "haqishen Very true. Firstly, I actually peaked around the beginning of August (for single models). What later boosted my score was ensembling. **In short**, my main focus was to preprocess the images to avoid learning spurious features (shapes of the black area etc.) I ended up with: crop_image_from_gray --> circle_crop (using the biggest radius instead of smallest (as in the public kernel), thus needing to pad the image)--> randomly cropped it further --> randomly created black top/bottom borders. I then augmented it pretty heavily (brightness/color-balance, rotation/flips, zoom, blur, gaussian noise, and sometimes random deletion (black boxes)). Finally cv2.addWeighted or clahe. I saved the best performing models throughout time, and [weight-]averaged them. The architectures I most commonly used were EfficientNetB5 and Xception, with varying input dims (256x256x3 and 299x299x3 seemed to suffice for EfficientNetB5 and Xception respectively). I started out with TTA but towards the end I removed it and included more models instead (seemed more promising). DenseNet, ResNeXt, ResNets, Inception, InceptionResNet, NASNetLarge, EfficientNetB6/7 didn't seem do it for me at the time (although most of them I didn't give a fair try). And yes, also, I always started the training with the inclusion of a balanced subset of the external dataset, and later fine-tuned it to the competition data (sometimes balanced undersampling, sometimes not), with AdamOptimizer(lr=1e-4) during external training and lr=1e-5 during fine-tuning. There is so much to cover, but this is what came to mind!\n\n**Note** that the above accounts for both my 0.933 and 0.877(best public LB) submissions (which still boggles my mind). The difference is that a few additional models were included for the 0.877 submission (with low weights for the weighted-average..). I have yet to understand this.\n\nAnd btw, thank you for sharing your solution too! (I just read your post)", "votes": null }, { "id": "621615", "postDate": "09/08/2019 18:08:15", "content": "It is difficult to predict the private leader-board from the public. My top submission on the public leader-board would have not qualify for any medal, luckily to edge my bets, I decide to select my best public leader-board and the most different model from the best, even if it had 0.01 lower score. This second model was ensembling more different augmentation strategies.
", "rawMarkdown": "It is difficult to predict the private leader-board from the public. My top submission on the public leader-board would have not qualify for any medal, luckily to edge my bets, I decide to select my best public leader-board and the most different model from the best, even if it had 0.01 lower score. This second model was ensembling more different augmentation strategies.", "votes": null }, { "id": "621637", "postDate": "09/08/2019 19:24:30", "content": "You should've used trimmed mean, so your bad model could not spoil the whole ensemble
", "rawMarkdown": "You should've used trimmed mean, so your bad model could not spoil the whole ensemble", "votes": null }, { "id": "621770", "postDate": "09/08/2019 23:20:53", "content": "@akensert It's not highest .855, one is .854 with many ensembles (with clipped) (so ensembles alone are not enough), this will fall like 1000+ too. Another is ensembles of .847 but different base models. And this last one luckily saves us from tragedy. Again, sorry to hear your story but since we are almost faced the same situation, I really feel you. Hope us to see the new challenge again soon!
", "rawMarkdown": "akensert It's not highest .855, one is .854 with many ensembles (with clipped) (so ensembles alone are not enough), this will fall like 1000+ too. Another is ensembles of .847 but different base models. And this last one luckily saves us from tragedy. Again, sorry to hear your story but since we are almost faced the same situation, I really feel you. Hope us to see the new challenge again soon!", "votes": null }, { "id": "621846", "postDate": "09/09/2019 02:37:42", "content": "Thank you for sharing!\nYou made an effort to eliminate the effect of meta data (black area), which turned out to be a quite important trick in this game.\nSo you're not using pseudo label to get 0.933 as well. After reading high score teams' post, I now highly suspect that 0.936 private LB is came from eliminating effect of meta data + carefully pseudo labelling ;)\nHope to see you in future competitions and have good luck next time!
\n\nEDIT : 1st place have post his solution which totally out of my imagination
", "rawMarkdown": "Thank you for sharing!\nYou made an effort to eliminate the effect of meta data (black area), which turned out to be a quite important trick in this game.\nSo you're not using pseudo label to get 0.933 as well. After reading high score teams' post, I now highly suspect that 0.936 private LB is came from eliminating effect of meta data + carefully pseudo labelling ;)\nHope to see you in future competitions and have good luck next time!\n\n**EDIT** : 1st place have post his solution which totally out of my imagination", "votes": null }, { "id": "622022", "postDate": "09/09/2019 07:13:46", "content": "@spsancti Yes indeed @ratthachat Interesting, thank you very much for sharing this. @haqishen No problem! Gonna check that one out. I didn't expect people to pseudo-label actually :)
", "rawMarkdown": "spsancti Yes indeed @ratthachat Interesting, thank you very much for sharing this. @haqishen No problem! Gonna check that one out. I didn't expect people to pseudo-label actually :)", "votes": null } ], "comments": [ { "id": 620830, "author_name": "sterls", "author_url": "", "post_date": "09/08/2019 01:58:30", "content": "We had a few submissions which would have gotten us silver 🤕
\n\nThe public leaderboard deceived us as we thought we were getting poor scores due to overfitting when in fact, those models that performed ~0.76 - 0.78 on public LB had private LB scores of above 0.920.
\n\nI think manual selection of best models now would be nice as it would increase the overall scores on the entire leaderboard. It would be nice if people wrote about what worked and what didn't so I'll be waiting on that!
", "votes": null, "replies": [] }, { "id": 620991, "author_name": "demonplus", "author_url": "", "post_date": "09/08/2019 06:11:58", "content": "I am still in bronze zone but interesting experience to share. I selected EfficientNet different ensembles for final selections and my real best was single model just forked from https://www.kaggle.com/drhabib/starter-kernel-for-0-79 with image size 256 and fine-tuned. I didn't select it and even didn't use it in ensemble because its public LB was rather low (0.783). And in fact this single model worked better than all my ensembles.
", "votes": null, "replies": [] }, { "id": 621027, "author_name": "akensert", "author_url": "", "post_date": "09/08/2019 06:47:47", "content": "I dropped from gold/near gold (top 20) to ~1800-1900th place. In other words, I dropped ~1800 positions. I was a bit in shock this morning! Anyone else had a similar drop? I had an older submissions with a score of 0.933 (which included many of the models of this poor submission). Thus I hypothesize that I must have included a model which made really awful predictions. Perhaps I should have clipped the outputs of my regression models, but it had no effect on the public leader board.
", "votes": null, "replies": [ { "id": 621100, "author_name": "ratthachat", "author_url": "", "post_date": "09/08/2019 07:45:57", "content": "@akensert You were indeed in bad luck! We almost got the same experience as you but we just a bit more lucky ... See you in the next round Akensert!
\n\nEDIT just to clarify, we did select our most promising 2 subs, but this selection process is really difficult!
", "votes": null, "replies": [] }, { "id": 621135, "author_name": "akensert", "author_url": "", "post_date": "09/08/2019 08:31:51", "content": "@ratthachat Yeah, I noticed your drop too. Unfortunate! Indeed, see you in the next round :)
", "votes": null, "replies": [] }, { "id": 621282, "author_name": "moewie94", "author_url": "", "post_date": "09/08/2019 11:58:14", "content": "that's a drop from top 10 to 18xx. MY GOD.
", "votes": null, "replies": [] }, { "id": 621467, "author_name": "haqishen", "author_url": "", "post_date": "09/08/2019 14:45:23", "content": "I know you since our LB rank is quite similar during the last month.\nSorry to hear that... You should have ranked near my position on the private LB as well...
", "votes": null, "replies": [] }, { "id": 621518, "author_name": "akensert", "author_url": "", "post_date": "09/08/2019 16:02:06", "content": "@haqishen Congrats to your gold! :) And thank you. It is how it is. Although, I feel this drop between public and private LB was a real hard slap in the face. I invested quite a lot of time and effort into this competition (like many of us of course). I truly understand that it's likely to overfit with close to 200 submissions. But from a potential gold placement to 1855th, that's rough.
\n\n@ratthachat Most promising meaning highest public LB score? I selected my highest scoring one, and also included my biggest ensemble for it's \"robustness\".
", "votes": null, "replies": [] }, { "id": 621552, "author_name": "haqishen", "author_url": "", "post_date": "09/08/2019 16:44:19", "content": "Yeah, I understand your feeling...\nAs you are experienced enough for getting gold medals, I think the most important thing is that what we can learn from the competition. If you finally find out what made this shake (but not just hypothesis), it will be an unique experience which will help you avoid it in the future.\nFurther more, I would like to learn from your 0.933 solution. Would you please write a quick summary on what you done? Thank you ;)
", "votes": null, "replies": [] }, { "id": 621602, "author_name": "akensert", "author_url": "", "post_date": "09/08/2019 17:47:55", "content": "@haqishen Very true. Firstly, I actually peaked around the beginning of August (for single models). What later boosted my score was ensembling. In short, my main focus was to preprocess the images to avoid learning spurious features (shapes of the black area etc.) I ended up with: crop_image_from_gray --> circle_crop (using the biggest radius instead of smallest (as in the public kernel), thus needing to pad the image)--> randomly cropped it further --> randomly created black top/bottom borders. I then augmented it pretty heavily (brightness/color-balance, rotation/flips, zoom, blur, gaussian noise, and sometimes random deletion (black boxes)). Finally cv2.addWeighted or clahe. I saved the best performing models throughout time, and [weight-]averaged them. The architectures I most commonly used were EfficientNetB5 and Xception, with varying input dims (256x256x3 and 299x299x3 seemed to suffice for EfficientNetB5 and Xception respectively). I started out with TTA but towards the end I removed it and included more models instead (seemed more promising). DenseNet, ResNeXt, ResNets, Inception, InceptionResNet, NASNetLarge, EfficientNetB6/7 didn't seem do it for me at the time (although most of them I didn't give a fair try). And yes, also, I always started the training with the inclusion of a balanced subset of the external dataset, and later fine-tuned it to the competition data (sometimes balanced undersampling, sometimes not), with AdamOptimizer(lr=1e-4) during external training and lr=1e-5 during fine-tuning. There is so much to cover, but this is what came to mind!
\n\nNote that the above accounts for both my 0.933 and 0.877(best public LB) submissions (which still boggles my mind). The difference is that a few additional models were included for the 0.877 submission (with low weights for the weighted-average..). I have yet to understand this.
\n\nAnd btw, thank you for sharing your solution too! (I just read your post)
", "votes": null, "replies": [] }, { "id": 621637, "author_name": "spsancti", "author_url": "", "post_date": "09/08/2019 19:24:30", "content": "You should've used trimmed mean, so your bad model could not spoil the whole ensemble
", "votes": null, "replies": [] }, { "id": 621770, "author_name": "ratthachat", "author_url": "", "post_date": "09/08/2019 23:20:53", "content": "@akensert It's not highest .855, one is .854 with many ensembles (with clipped) (so ensembles alone are not enough), this will fall like 1000+ too. Another is ensembles of .847 but different base models. And this last one luckily saves us from tragedy. Again, sorry to hear your story but since we are almost faced the same situation, I really feel you. Hope us to see the new challenge again soon!
", "votes": null, "replies": [] }, { "id": 621846, "author_name": "haqishen", "author_url": "", "post_date": "09/09/2019 02:37:42", "content": "Thank you for sharing!\nYou made an effort to eliminate the effect of meta data (black area), which turned out to be a quite important trick in this game.\nSo you're not using pseudo label to get 0.933 as well. After reading high score teams' post, I now highly suspect that 0.936 private LB is came from eliminating effect of meta data + carefully pseudo labelling ;)\nHope to see you in future competitions and have good luck next time!
\n\nEDIT : 1st place have post his solution which totally out of my imagination
", "votes": null, "replies": [] }, { "id": 622022, "author_name": "akensert", "author_url": "", "post_date": "09/09/2019 07:13:46", "content": "@spsancti Yes indeed @ratthachat Interesting, thank you very much for sharing this. @haqishen No problem! Gonna check that one out. I didn't expect people to pseudo-label actually :)
", "votes": null, "replies": [] } ] }, { "id": 621048, "author_name": "gpamoukoff", "author_url": "", "post_date": "09/08/2019 07:01:48", "content": "Wold have been silver - but dropped out of the medal completely :)
", "votes": null, "replies": [] }, { "id": 621159, "author_name": "apthagowda", "author_url": "", "post_date": "09/08/2019 09:05:11", "content": "Down by 297 and lost a bronze medal, but an important thing I want to share is I did partial submission(i.e turned of GPU and submitted just the CSV file ) during experimenting and only ran inference kernel for the 2 selected model. At that time it made easy to check results within 1s but I am regretting now because there may be some models which performed better than the selected model. I have learnt my lesson.
\n\nThis is my first deep learning competition and I am proud of myself. Next stop Severstal: Steel Defect Detection
", "votes": null, "replies": [] }, { "id": 621207, "author_name": "carlolepelaars", "author_url": "", "post_date": "09/08/2019 10:28:49", "content": "Very weird competition! Had over 0.150 score difference between public and private score. The public leaderboard data seemed to come from an entirely different distribution compared to the rest.
", "votes": null, "replies": [ { "id": 621564, "author_name": "gpamoukoff", "author_url": "", "post_date": "09/08/2019 17:03:06", "content": "Same with me. Never KO again...
", "votes": null, "replies": [] } ] }, { "id": 621615, "author_name": "agentili", "author_url": "", "post_date": "09/08/2019 18:08:15", "content": "It is difficult to predict the private leader-board from the public. My top submission on the public leader-board would have not qualify for any medal, luckily to edge my bets, I decide to select my best public leader-board and the most different model from the best, even if it had 0.01 lower score. This second model was ensembling more different augmentation strategies.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "620813": "Please, share you experience. We had a few submissions which could bring us to bronze, but we made wrong assumption which submission is better on private LB... and lost medals. \n\nAnd you?", "620830": "We had a few submissions which would have gotten us silver 🤕 \n\nThe public leaderboard deceived us as we thought we were getting poor scores due to overfitting when in fact, those models that performed ~0.76 - 0.78 on public LB had private LB scores of above 0.920. \n\nI think manual selection of best models now would be nice as it would increase the overall scores on the entire leaderboard. It would be nice if people wrote about what worked and what didn't so I'll be waiting on that!", "620991": "I am still in bronze zone but interesting experience to share. I selected EfficientNet different ensembles for final selections and my real best was single model just forked from https://www.kaggle.com/drhabib/starter-kernel-for-0-79 with image size 256 and fine-tuned. I didn't select it and even didn't use it in ensemble because its public LB was rather low (0.783). And in fact this single model worked better than all my ensembles.", "621027": "I dropped from gold/near gold (top 20) to ~1800-1900th place. In other words, I dropped ~1800 positions. I was a bit in shock this morning! Anyone else had a similar drop? I had an older submissions with a score of 0.933 (which included many of the models of this poor submission). Thus I hypothesize that I must have included a model which made really awful predictions. Perhaps I should have clipped the outputs of my regression models, but it had no effect on the public leader board.", "621048": "Wold have been silver - but dropped out of the medal completely :)", "621100": "akensert You were indeed in bad luck! We almost got the same experience as you but we just a bit more lucky ... See you in the next round Akensert!\n\n**EDIT** just to clarify, we did select our most promising 2 subs, but this selection process is really difficult!", "621135": "ratthachat Yeah, I noticed your drop too. Unfortunate! Indeed, see you in the next round :)", "621159": "Down by 297 and lost a bronze medal, but an important thing I want to share is I did partial submission(i.e turned of GPU and submitted just the CSV file ) during experimenting and only ran inference kernel for the 2 selected model. At that time it made easy to check results within 1s but I am regretting now because there may be some models which performed better than the selected model. I have learnt my lesson.\n\nThis is my first deep learning competition and I am proud of myself. Next stop Severstal: Steel Defect Detection", "621207": "Very weird competition! Had over 0.150 score difference between public and private score. The public leaderboard data seemed to come from an entirely different distribution compared to the rest.", "621282": "that's a drop from top 10 to 18xx. MY GOD.", "621467": "I know you since our LB rank is quite similar during the last month.\nSorry to hear that... You should have ranked near my position on the private LB as well...", "621518": "haqishen Congrats to your gold! :) And thank you. It is how it is. Although, I feel this drop between public and private LB was a real hard slap in the face. I invested quite a lot of time and effort into this competition (like many of us of course). I truly understand that it's likely to overfit with close to 200 submissions. But from a potential gold placement to 1855th, that's rough. \n\n@ratthachat Most promising meaning highest public LB score? I selected my highest scoring one, and also included my biggest ensemble for it's \"robustness\".", "621552": "Yeah, I understand your feeling...\nAs you are experienced enough for getting gold medals, I think the most important thing is that what we can learn from the competition. If you finally find out what made this shake (but not just hypothesis), it will be an unique experience which will help you avoid it in the future.\nFurther more, I would like to learn from your 0.933 solution. Would you please write a quick summary on what you done? Thank you ;)", "621564": "Same with me. Never KO again...", "621602": "haqishen Very true. Firstly, I actually peaked around the beginning of August (for single models). What later boosted my score was ensembling. **In short**, my main focus was to preprocess the images to avoid learning spurious features (shapes of the black area etc.) I ended up with: crop_image_from_gray --> circle_crop (using the biggest radius instead of smallest (as in the public kernel), thus needing to pad the image)--> randomly cropped it further --> randomly created black top/bottom borders. I then augmented it pretty heavily (brightness/color-balance, rotation/flips, zoom, blur, gaussian noise, and sometimes random deletion (black boxes)). Finally cv2.addWeighted or clahe. I saved the best performing models throughout time, and [weight-]averaged them. The architectures I most commonly used were EfficientNetB5 and Xception, with varying input dims (256x256x3 and 299x299x3 seemed to suffice for EfficientNetB5 and Xception respectively). I started out with TTA but towards the end I removed it and included more models instead (seemed more promising). DenseNet, ResNeXt, ResNets, Inception, InceptionResNet, NASNetLarge, EfficientNetB6/7 didn't seem do it for me at the time (although most of them I didn't give a fair try). And yes, also, I always started the training with the inclusion of a balanced subset of the external dataset, and later fine-tuned it to the competition data (sometimes balanced undersampling, sometimes not), with AdamOptimizer(lr=1e-4) during external training and lr=1e-5 during fine-tuning. There is so much to cover, but this is what came to mind!\n\n**Note** that the above accounts for both my 0.933 and 0.877(best public LB) submissions (which still boggles my mind). The difference is that a few additional models were included for the 0.877 submission (with low weights for the weighted-average..). I have yet to understand this.\n\nAnd btw, thank you for sharing your solution too! (I just read your post)", "621615": "It is difficult to predict the private leader-board from the public. My top submission on the public leader-board would have not qualify for any medal, luckily to edge my bets, I decide to select my best public leader-board and the most different model from the best, even if it had 0.01 lower score. This second model was ensembling more different augmentation strategies.", "621637": "You should've used trimmed mean, so your bad model could not spoil the whole ensemble", "621770": "akensert It's not highest .855, one is .854 with many ensembles (with clipped) (so ensembles alone are not enough), this will fall like 1000+ too. Another is ensembles of .847 but different base models. And this last one luckily saves us from tragedy. Again, sorry to hear your story but since we are almost faced the same situation, I really feel you. Hope us to see the new challenge again soon!", "621846": "Thank you for sharing!\nYou made an effort to eliminate the effect of meta data (black area), which turned out to be a quite important trick in this game.\nSo you're not using pseudo label to get 0.933 as well. After reading high score teams' post, I now highly suspect that 0.936 private LB is came from eliminating effect of meta data + carefully pseudo labelling ;)\nHope to see you in future competitions and have good luck next time!\n\n**EDIT** : 1st place have post his solution which totally out of my imagination", "622022": "spsancti Yes indeed @ratthachat Interesting, thank you very much for sharing this. @haqishen No problem! Gonna check that one out. I didn't expect people to pseudo-label actually :)" }, "source": "meta" }