![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F479538%2F0ccf9f1fc86e9347f05cfa6abff82efb%2Ffig1.PNG?generation=1568006573020268&alt=media\")
(I wrote this thread before the deadline end just not to lose inspiration to write due to potential shake up :)
\n\nAPTOS 2019, really, had been a magical competition for me. Never in my dream that I would be able to compete head-to-head with top kagglers until the end (at least according to the public LB); we have at least Data Science Bowl champion, Jigsaw champion, Whale champion, Quora champion and many gold/silver/bronze-awards computer vision experts as well as many new rising stars; all of them I have learned so much from their sharing in the past as well as new things I have learned from many kernel / discussion contributors here in APTOS too! (I tried to vote your topics/kernels as much as I could)
\n\nI have to thanks to all my four teammates, and I am honor to be team up with them. I will write in details how each one of them help me improve my own solution below. (hopefully not too much shaken up)
\n\nI also thank every participant in this competition to have loved all my kernels & discussions. As you can see that in other competitions, I have completely normal performances :) , so I will treat my performance here as approximately 63% lucky factor (1 - 1/e).
\n\nBefore going to my solution, I would like to share some secret : “before” this competition, my philosophy in competition “in the past” is to find some secret magic, i.e. finding secret architectures / secret loss functions / secret training process / hyperparameters etc. …. However, as I failed all of these in previous competitions, (and having read/talked with others) I have changed my philosophy to “finding basics” and “sharing to community” (as I thought that I have no chance anyway :),
\n\nand when I gave up to find magic, ironically, that was, how real “magic” experience happen to me …
\n\n—————\nThis is a night-time in Thailand, and I am going to sleep with my 3-year old daughter. Deadline is at 7.00AM of my tomorrow morning, so when I wake up I will see the results, and I will add my personal write-up here :)
\n\nNo matter how much shake-up will happen, this is not the end of our journey … It’s just a new beginning to the new challenge.
\n\nGood Luck to You All!
\n\nupdated the result is out, we didn't choose robust enough choice :)) \nCongratulations to all winners!!!\nWhat I wrote above didn't change. I recieve overwhelming learning and friendship experience and great teamwork from these 2-3 months. Thank you all my friends.
\n\nMy personal best model regarding stage1 got satisfiable results of Pub850/Priv929 and quite consistent CV.
\n\nI use very similar model to Gary's (8th place) with 2-heads (regression/classification) and also pseudo labelling. Preprocessing / augmentations are exactly the same as my kernels. By using iterative pseudo labels very similar to this post by @bibek777
\n\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/106177#latest-610648
\n\nBy iterating 3 times and ensemble them I was able to boost from Public 0.82x --> 0.850
\n\nI could continue but I was also afraid to overfit ( but I did not see overfitting sign by my CV and my Grad-CAM visualization)
\n\nIt turns out that this model is not overfit but got a very little boost in private (926 --> 929), this is understandable since public is not like private at all :) Since this is my personal model, we didn't select it and rather try more promising combination to others
\n\n(I wrote this thread before the deadline end just not to lose inspiration to write due to potential shake up :)
\n\nAPTOS 2019, really, had been a magical competition for me. Never in my dream that I would be able to compete head-to-head with top kagglers until the end (at least according to the public LB); we have at least Data Science Bowl champion, Jigsaw champion, Whale champion, Quora champion and many gold/silver/bronze-awards computer vision experts as well as many new rising stars; all of them I have learned so much from their sharing in the past as well as new things I have learned from many kernel / discussion contributors here in APTOS too! (I tried to vote your topics/kernels as much as I could)
\n\nI have to thanks to all my four teammates, and I am honor to be team up with them. I will write in details how each one of them help me improve my own solution below. (hopefully not too much shaken up)
\n\nI also thank every participant in this competition to have loved all my kernels & discussions. As you can see that in other competitions, I have completely normal performances :) , so I will treat my performance here as approximately 63% lucky factor (1 - 1/e).
\n\nBefore going to my solution, I would like to share some secret : “before” this competition, my philosophy in competition “in the past” is to find some secret magic, i.e. finding secret architectures / secret loss functions / secret training process / hyperparameters etc. …. However, as I failed all of these in previous competitions, (and having read/talked with others) I have changed my philosophy to “finding basics” and “sharing to community” (as I thought that I have no chance anyway :),
\n\nand when I gave up to find magic, ironically, that was, how real “magic” experience happen to me …
\n\n—————\nThis is a night-time in Thailand, and I am going to sleep with my 3-year old daughter. Deadline is at 7.00AM of my tomorrow morning, so when I wake up I will see the results, and I will add my personal write-up here :)
\n\nNo matter how much shake-up will happen, this is not the end of our journey … It’s just a new beginning to the new challenge.
\n\nGood Luck to You All!
\n\nupdated the result is out, we didn't choose robust enough choice :)) \nCongratulations to all winners!!!\nWhat I wrote above didn't change. I recieve overwhelming learning and friendship experience and great teamwork from these 2-3 months. Thank you all my friends.
\n\nMy personal best model regarding stage1 got satisfiable results of Pub850/Priv929 and quite consistent CV.
\n\nI use very similar model to Gary's (8th place) with 2-heads (regression/classification) and also pseudo labelling. Preprocessing / augmentations are exactly the same as my kernels. By using iterative pseudo labels very similar to this post by @bibek777
\n\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/106177#latest-610648
\n\nBy iterating 3 times and ensemble them I was able to boost from Public 0.82x --> 0.850
\n\nI could continue but I was also afraid to overfit ( but I did not see overfitting sign by my CV and my Grad-CAM visualization)
\n\nIt turns out that this model is not overfit but got a very little boost in private (926 --> 929), this is understandable since public is not like private at all :) Since this is my personal model, we didn't select it and rather try more promising combination to others
\n\nHi @ratthachat \nThank you for your great contribution to Kaggle community.\nI have learned a lot from you and sometimes discussed over the same topic.
\n\nSame as your opinion, I thought that the grading distribution of private test data would be like that of train.\nBut we could not bet on only train-like distribution. \nIn the end, we made 2 types of models, one was trained with APTOS train distribution, the other was with much higher severity distribution (I have posted our model pipeline below, the upper model was the former).\nIf we could bet on only... ok. That's enough, I will stop talking about “woulda, coulda shoulda”.
\n\nNow competition is over, please have a rest at ease.\nSee in another competition.\n
Hi @maxwell110 , I also learned so much from you since FreeSound to Aptos, thanks so much for your kernel/discussion contributions to our community!
\n\nYour powerpoint skill is still top-notch as usual! I will rest at ease as you suggested at the moment :D
", "rawMarkdown": "Hi @maxwell110 , I also learned so much from you since FreeSound to Aptos, thanks so much for your kernel/discussion contributions to our community!\n\nYour powerpoint skill is still top-notch as usual! I will rest at ease as you suggested at the moment :D", "votes": 2 }, { "id": 622191, "postDate": "2019-09-09T11:21:32.680Z", "content": "@maxwell110 your banner is awsome, very good work!
", "rawMarkdown": "@maxwell110 your banner is awsome, very good work!", "votes": 2 }, { "id": 622236, "postDate": "2019-09-09T12:15:37.017Z", "content": "@maxwell110, congrats and thanks for sharing your pipeline.
", "rawMarkdown": "@maxwell110, congrats and thanks for sharing your pipeline.", "votes": 2 } ] }, { "id": 620433, "postDate": "2019-09-07T13:57:19.037Z", "content": "Neuron Engineer! Your kernels were big part of my solutions =) Thank you very much for doing awesome work! =)
", "rawMarkdown": "Neuron Engineer! Your kernels were big part of my solutions =) Thank you very much for doing awesome work! =) ", "votes": 6, "replies": [ { "id": 620699, "postDate": "2019-09-07T21:26:40.717Z", "content": "I wake up a bit early just to reply this thread before knowing the results. I learn a lot from you too, DrHB!! See you in the next comp. :D
\n\nEdit. the result is out and you got gold!! Congratulation! Well deserve @drhabib
", "rawMarkdown": "I wake up a bit early just to reply this thread before knowing the results. I learn a lot from you too, DrHB!! See you in the next comp. :D\n\n**Edit**. the result is out and you got gold!! Congratulation! Well deserve @drhabib", "votes": 2 } ] }, { "id": 620783, "postDate": "2019-09-08T00:51:40.870Z", "content": "@ratthachat Thanks for all your fantastic contributions throughout this competition, you've helped me and many others out considerably. Hard luck with the final drop but like you say, the real gain here is the amount we learn and the people we work with. See you in the next competition!
", "rawMarkdown": "@ratthachat Thanks for all your fantastic contributions throughout this competition, you've helped me and many others out considerably. Hard luck with the final drop but like you say, the real gain here is the amount we learn and the people we work with. See you in the next competition!", "votes": 3, "replies": [ { "id": 620797, "postDate": "2019-09-08T01:08:32.440Z", "content": "Hi Tom!! I was going to ask Rishabh @rishabhiitbhu to say hi to you! It was really nice to know you and learn from your idea. Lets connect!
", "rawMarkdown": "Hi Tom!! I was going to ask Rishabh @rishabhiitbhu to say hi to you! It was really nice to know you and learn from your idea. Lets connect!", "votes": 1 } ] }, { "id": 620757, "postDate": "2019-09-08T00:16:21.590Z", "content": "Thanks @ratthachat , I have learned a lot from you, this is my first competition medal, and what you shared helped me a lot, I was rooting for you to get gold, but for us all, knowledge is always the biggest prize.
", "rawMarkdown": "Thanks @ratthachat , I have learned a lot from you, this is my first competition medal, and what you shared helped me a lot, I was rooting for you to get gold, but for us all, knowledge is always the biggest prize.", "votes": 3, "replies": [ { "id": 620796, "postDate": "2019-09-08T01:05:21.403Z", "content": "@dimitreoliveira Dimitre, your kernel is one of earliest keras kernel here, and I also used many parts of your codes, thank you and see you in the next comp. :)
", "rawMarkdown": "@dimitreoliveira Dimitre, your kernel is one of earliest keras kernel here, and I also used many parts of your codes, thank you and see you in the next comp. :)", "votes": 2 }, { "id": 620823, "postDate": "2019-09-08T01:47:11.363Z", "content": "Thanks @ratthachat , I'm really glad to hear that I also could help, until the next one 👍
", "rawMarkdown": "Thanks @ratthachat , I'm really glad to hear that I also could help, until the next one 👍 ", "votes": 1 } ] }, { "id": 620549, "postDate": "2019-09-07T16:52:58.957Z", "content": "@ratthachat, whatever happens congratulations. You did great in this comeptition.
", "rawMarkdown": " @ratthachat, whatever happens congratulations. You did great in this comeptition.", "votes": 3, "replies": [ { "id": 620706, "postDate": "2019-09-07T21:34:06.860Z", "content": "YaGana, your consistency is one of my inspiration :)
", "rawMarkdown": "YaGana, your consistency is one of my inspiration :)", "votes": 1 }, { "id": 620718, "postDate": "2019-09-07T22:28:50.393Z", "content": "Thanks for your kind words @ratthachat and good luck.
\n\nParticipating in this competition is hard for me as I am travelling abroad for a couple of months now with hardly a consistent internet connection. How I miss my broadband/ fibre optic internet service :(
", "rawMarkdown": "Thanks for your kind words @ratthachat and good luck. \n\nParticipating in this competition is hard for me as I am travelling abroad for a couple of months now with hardly a consistent internet connection. How I miss my broadband/ fibre optic internet service :(", "votes": 1 } ] }, { "id": 620448, "postDate": "2019-09-07T14:12:12.303Z", "content": "Oh man, can't express in words how important you have been for this competition. Your kernels and insights in the discussion threads... I've learnt a lot from you. People like you make this community better every day. Sincere thanks.
", "rawMarkdown": "Oh man, can't express in words how important you have been for this competition. Your kernels and insights in the discussion threads... I've learnt a lot from you. People like you make this community better every day. Sincere thanks.", "votes": 3, "replies": [ { "id": 620703, "postDate": "2019-09-07T21:28:59.713Z", "content": "Hi Rishabh, thank you for your words! I also learned a lot from you, really! (this includes other competitions where your kernels are also excellent :)
", "rawMarkdown": "Hi Rishabh, thank you for your words! I also learned a lot from you, really! (this includes other competitions where your kernels are also excellent :)", "votes": 1 }, { "id": 620857, "postDate": "2019-09-08T02:28:30.063Z", "content": "Thanks, exciting times ahead :)
", "rawMarkdown": "Thanks, exciting times ahead :)", "votes": 1 } ] }, { "id": 621287, "postDate": "2019-09-08T12:03:43.637Z", "content": "@ratthachat Thanks for your wonderful kernels and for your story
", "rawMarkdown": " @ratthachat Thanks for your wonderful kernels and for your story", "votes": 1, "replies": [ { "id": 621365, "postDate": "2019-09-08T13:17:03.073Z", "content": "@demonplus Thanks also for making this competition wonderful Anna, see you in the next round! :)
", "rawMarkdown": "@demonplus Thanks also for making this competition wonderful Anna, see you in the next round! :)", "votes": 1 } ] }, { "id": 621203, "postDate": "2019-09-08T10:10:58.397Z", "content": "Thank you for your kernels @ratthachat learnt a lot from them, although I am nowhere close to others in this thread but I do hope to improve in other competitions.
", "rawMarkdown": "Thank you for your kernels @ratthachat learnt a lot from them, although I am nowhere close to others in this thread but I do hope to improve in other competitions.", "votes": 1, "replies": [ { "id": 621373, "postDate": "2019-09-08T13:19:22.770Z", "content": "Hey @imnitishng you will surely improve, just please continue , you will surprise how much you grow ;)
", "rawMarkdown": "Hey @imnitishng you will surely improve, just please continue , you will surprise how much you grow ;)", "votes": 1 } ] }, { "id": 621146, "postDate": "2019-09-08T08:49:40.850Z", "content": "I think your inspiration will raise this competition to a higher level. Thank you and your teammates!!
", "rawMarkdown": "\nI think your inspiration will raise this competition to a higher level. Thank you and your teammates!!", "votes": 1, "replies": [ { "id": 621370, "postDate": "2019-09-08T13:17:54.080Z", "content": "Really appreciate to your kind words Takayoshi @spidermandance !
", "rawMarkdown": "Really appreciate to your kind words Takayoshi @spidermandance !", "votes": 1 } ] }, { "id": 620799, "postDate": "2019-09-08T01:09:30.030Z", "content": "Thank you so much for your great contribution.\nYour kernels and discussions helped me a lot in understanding and tackling the problems.
", "rawMarkdown": "Thank you so much for your great contribution.\nYour kernels and discussions helped me a lot in understanding and tackling the problems.", "votes": 1, "replies": [ { "id": 621114, "postDate": "2019-09-08T07:57:45.577Z", "content": "@ttya16 Thanks so much! I have to keep learning too, lets do it together !
", "rawMarkdown": "@ttya16 Thanks so much! I have to keep learning too, lets do it together !", "votes": 1 } ] }, { "id": 620743, "postDate": "2019-09-07T23:50:11.057Z", "content": "I have learnt a lot from you. Thanks for sharing your knowledge @ratthachat . Good Luck
", "rawMarkdown": "I have learnt a lot from you. Thanks for sharing your knowledge @ratthachat . Good Luck ", "votes": 1, "replies": [ { "id": 620801, "postDate": "2019-09-08T01:11:34.430Z", "content": "Thanks Viraj @virajbagal ! Let us continue to learn together.
", "rawMarkdown": "Thanks Viraj @virajbagal ! Let us continue to learn together. ", "votes": 1 } ] }, { "id": 620713, "postDate": "2019-09-07T21:43:03.747Z", "content": "Good luck to you. And good night. Your kernel is amazing!
", "rawMarkdown": "Good luck to you. And good night. Your kernel is amazing!", "votes": 1, "replies": [ { "id": 620800, "postDate": "2019-09-08T01:10:38.740Z", "content": "Thanks Hilal @abualabed ! See you in the next comp!
", "rawMarkdown": "Thanks Hilal @abualabed ! See you in the next comp!", "votes": 1 } ] }, { "id": 620689, "postDate": "2019-09-07T21:09:59.210Z", "content": "Inspiring and insightful story @ratthachat \nThank you for all what you have shared and for sharing this experience as well, which I understand as the importance of focusing on the fundamentals instead of spending most of the time searching for something different.\nI bet your model will be robust to the possible shake up, since you have a solid preprocessing.. let's see. Best wishes ;)
", "rawMarkdown": "Inspiring and insightful story @ratthachat \nThank you for all what you have shared and for sharing this experience as well, which I understand as the importance of focusing on the fundamentals instead of spending most of the time searching for something different.\nI bet your model will be robust to the possible shake up, since you have a solid preprocessing.. let's see. Best wishes ;)", "votes": 1, "replies": [ { "id": 620711, "postDate": "2019-09-07T21:41:16.007Z", "content": "Carlos, thanks for your encouragement and wish. Really appreaciate it! I didn't dare saying anything before the shake up :D I wish the best of luck for you too!
", "rawMarkdown": "Carlos, thanks for your encouragement and wish. Really appreaciate it! I didn't dare saying anything before the shake up :D I wish the best of luck for you too!", "votes": 1 } ] }, { "id": 620593, "postDate": "2019-09-07T18:08:26.813Z", "content": "Congrats! Great job, keep it up.
", "rawMarkdown": "Congrats! Great job, keep it up.", "votes": 1 }, { "id": 620568, "postDate": "2019-09-07T17:22:23.307Z", "content": "I have learnt a lot from you @ratthachat . Your kernels were the starting point for my solution. My best wishes for best result for you :).
", "rawMarkdown": "I have learnt a lot from you @ratthachat . Your kernels were the starting point for my solution. My best wishes for best result for you :). ", "votes": 1, "replies": [ { "id": 620707, "postDate": "2019-09-07T21:36:17.563Z", "content": "Thanks Manoj!! I wish the best for you too.
", "rawMarkdown": "Thanks Manoj!! I wish the best for you too.", "votes": 1 } ] }, { "id": 620538, "postDate": "2019-09-07T16:19:07.037Z", "content": "Thanks @ratthachat ! Your work in this competition was excellent.
\n\nReally loved your kernels. Good luck! 😊
", "rawMarkdown": "Thanks @ratthachat ! Your work in this competition was excellent.\n\nReally loved your kernels. Good luck! 😊 ", "votes": 1, "replies": [ { "id": 620704, "postDate": "2019-09-07T21:32:03.340Z", "content": "Hi Federico, let us cont. to learn together in the next comp.!! It's truly nice to connect with you :)
", "rawMarkdown": "Hi Federico, let us cont. to learn together in the next comp.!! It's truly nice to connect with you :)", "votes": 1 } ] }, { "id": 620826, "postDate": "2019-09-08T01:51:45.847Z", "content": "Thanks a lot @ratthachat , your preprocessing visualization kernel taught me many new methods and helped me understand what each one did. This is my first medal and your kernel definitely helped. Its too bad that the shakeup was pretty intense for people at the top of the LB but your resources will continue to be invaluable :)
", "rawMarkdown": "Thanks a lot @ratthachat , your preprocessing visualization kernel taught me many new methods and helped me understand what each one did. This is my first medal and your kernel definitely helped. Its too bad that the shakeup was pretty intense for people at the top of the LB but your resources will continue to be invaluable :)", "votes": 2, "replies": [ { "id": 621110, "postDate": "2019-09-08T07:56:37.533Z", "content": "Hi @sidhanthholalkere ! Thank you so much for your encouragement! I hope we both can continue to contribute to our community better and better.
", "rawMarkdown": "Hi @sidhanthholalkere ! Thank you so much for your encouragement! I hope we both can continue to contribute to our community better and better.", "votes": 1 } ] }, { "id": 620780, "postDate": "2019-09-08T00:48:11.537Z", "content": "Thank you for all your contributions to the community and congratulations on your great result!\nI learned a lot from you.
\n\nOne surprise is that you have 3 year old daughter. I'd love to hear your time management tips. As I know it's really hard to focus and spend time on something intense like kaggle. (I'm a parent also).
", "rawMarkdown": "Thank you for all your contributions to the community and congratulations on your great result!\nI learned a lot from you.\n\nOne surprise is that you have 3 year old daughter. I'd love to hear your time management tips. As I know it's really hard to focus and spend time on something intense like kaggle. (I'm a parent also).", "votes": 2, "replies": [ { "id": 620798, "postDate": "2019-09-08T01:09:07.110Z", "content": "I will email to say hi to you @higepon :)
", "rawMarkdown": "I will email to say hi to you @higepon :)", "votes": 3 }, { "id": 620815, "postDate": "2019-09-08T01:31:58.920Z", "content": "Wut? Cool :)
", "rawMarkdown": "Wut? Cool :)", "votes": 1 } ] }, { "id": 620542, "postDate": "2019-09-07T16:25:11.290Z", "content": "Learned a lot through this competition, also your kernels and work in the discussions is amazing and is extremely helpful, as i have learned a lot through them @ratthachat @drhabib.
", "rawMarkdown": "Learned a lot through this competition, also your kernels and work in the discussions is amazing and is extremely helpful, as i have learned a lot through them @ratthachat @drhabib.", "votes": 2, "replies": [ { "id": 621120, "postDate": "2019-09-08T08:08:52.083Z", "content": "@modojj Thanks so much for your kind words! Let us continue our journey into the next challenge ;)
", "rawMarkdown": "@modojj Thanks so much for your kind words! Let us continue our journey into the next challenge ;)", "votes": 1 } ] }, { "id": 620439, "postDate": "2019-09-07T14:03:16.067Z", "content": "I have learned a lot from your kernel, wish you good luck!
", "rawMarkdown": "I have learned a lot from your kernel, wish you good luck!", "votes": 2, "replies": [ { "id": 620702, "postDate": "2019-09-07T21:27:34.633Z", "content": "Thanks so much donglee, let us continue to learn together :)
", "rawMarkdown": "Thanks so much donglee, let us continue to learn together :)", "votes": 2 } ] }, { "id": 620436, "postDate": "2019-09-07T13:59:56.457Z", "content": "Good Luck.. @ratthachat
", "rawMarkdown": "Good Luck.. @ratthachat ", "votes": 1 }, { "id": 621695, "postDate": "2019-09-08T20:46:11.440Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 621834, "postDate": "2019-09-09T02:11:19.097Z", "content": "@thestonemx Oscar, I have one kernels for preprocessing training data, another one for external data, another one on test data, and another one on creating pseudo labelling. Yet another one on “real training” ... So my kernels are really mess. But I share main idea here.
\n\nYou can see the big picture from the post of Bibek I mentioned above. Straightforwardly, you just use you best model to predict “test data”, and then assign the labels and use it like a normal training dataset. Using this, you should get a good result in a very straightforward way.
\n\nIn my case, a little trick, I found to be better than the above method is to use “2 best models” to predict together, and assign “pseudo soft labels” instead of “pseudo hard labels” e.g. if one model predicts ‘1’ and other predicts ‘2’, I will assign soft label as ‘2.5’. These two models are very consistent and always predict on+/- 1 range, so there is no case like (0+4)/2 = 2.
", "rawMarkdown": "@thestonemx Oscar, I have one kernels for preprocessing training data, another one for external data, another one on test data, and another one on creating pseudo labelling. Yet another one on “real training” ... So my kernels are really mess. But I share main idea here. \n\nYou can see the big picture from the post of Bibek I mentioned above. Straightforwardly, you just use you best model to predict “test data”, and then assign the labels and use it like a normal training dataset. Using this, you should get a good result in a very straightforward way.\n\nIn my case, a little trick, I found to be better than the above method is to use “2 best models” to predict together, and assign “pseudo soft labels” instead of “pseudo hard labels” e.g. if one model predicts ‘1’ and other predicts ‘2’, I will assign soft label as ‘2.5’. These two models are very consistent and always predict on+/- 1 range, so there is no case like (0+4)/2 = 2.", "votes": 2 } ] }, { "id": 621211, "postDate": "2019-09-08T10:32:26.550Z", "content": "Thanks for the contributions:)
", "rawMarkdown": "Thanks for the contributions:)", "votes": 1 } ], "comments": [ { "id": 621934, "author_name": "Maxwell", "author_url": "", "post_date": "2019-09-09T05:26:46.890000", "content": "Hi @ratthachat \nThank you for your great contribution to Kaggle community.\nI have learned a lot from you and sometimes discussed over the same topic.
\n\nSame as your opinion, I thought that the grading distribution of private test data would be like that of train.\nBut we could not bet on only train-like distribution. \nIn the end, we made 2 types of models, one was trained with APTOS train distribution, the other was with much higher severity distribution (I have posted our model pipeline below, the upper model was the former).\nIf we could bet on only... ok. That's enough, I will stop talking about “woulda, coulda shoulda”.
\n\nNow competition is over, please have a rest at ease.\nSee in another competition.\n
Hi @maxwell110 , I also learned so much from you since FreeSound to Aptos, thanks so much for your kernel/discussion contributions to our community!
\n\nYour powerpoint skill is still top-notch as usual! I will rest at ease as you suggested at the moment :D
", "votes": 2, "replies": [] }, { "id": 622191, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2019-09-09T11:21:32.680000", "content": "@maxwell110 your banner is awsome, very good work!
", "votes": 2, "replies": [] }, { "id": 622236, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-09T12:15:37.017000", "content": "@maxwell110, congrats and thanks for sharing your pipeline.
", "votes": 2, "replies": [] } ] }, { "id": 620433, "author_name": "DrHB", "author_url": "", "post_date": "2019-09-07T13:57:19.037000", "content": "Neuron Engineer! Your kernels were big part of my solutions =) Thank you very much for doing awesome work! =)
", "votes": 6, "replies": [ { "id": 620699, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-07T21:26:40.717000", "content": "I wake up a bit early just to reply this thread before knowing the results. I learn a lot from you too, DrHB!! See you in the next comp. :D
\n\nEdit. the result is out and you got gold!! Congratulation! Well deserve @drhabib
", "votes": 2, "replies": [] } ] }, { "id": 620783, "author_name": "Tom Aindow", "author_url": "", "post_date": "2019-09-08T00:51:40.870000", "content": "@ratthachat Thanks for all your fantastic contributions throughout this competition, you've helped me and many others out considerably. Hard luck with the final drop but like you say, the real gain here is the amount we learn and the people we work with. See you in the next competition!
", "votes": 3, "replies": [ { "id": 620797, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T01:08:32.440000", "content": "Hi Tom!! I was going to ask Rishabh @rishabhiitbhu to say hi to you! It was really nice to know you and learn from your idea. Lets connect!
", "votes": 1, "replies": [] } ] }, { "id": 620757, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2019-09-08T00:16:21.590000", "content": "Thanks @ratthachat , I have learned a lot from you, this is my first competition medal, and what you shared helped me a lot, I was rooting for you to get gold, but for us all, knowledge is always the biggest prize.
", "votes": 3, "replies": [ { "id": 620796, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T01:05:21.403000", "content": "@dimitreoliveira Dimitre, your kernel is one of earliest keras kernel here, and I also used many parts of your codes, thank you and see you in the next comp. :)
", "votes": 2, "replies": [] }, { "id": 620823, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2019-09-08T01:47:11.363000", "content": "Thanks @ratthachat , I'm really glad to hear that I also could help, until the next one 👍
", "votes": 1, "replies": [] } ] }, { "id": 620549, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-07T16:52:58.957000", "content": "@ratthachat, whatever happens congratulations. You did great in this comeptition.
", "votes": 3, "replies": [ { "id": 620706, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-07T21:34:06.860000", "content": "YaGana, your consistency is one of my inspiration :)
", "votes": 1, "replies": [] }, { "id": 620718, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-07T22:28:50.393000", "content": "Thanks for your kind words @ratthachat and good luck.
\n\nParticipating in this competition is hard for me as I am travelling abroad for a couple of months now with hardly a consistent internet connection. How I miss my broadband/ fibre optic internet service :(
", "votes": 1, "replies": [] } ] }, { "id": 620448, "author_name": "Rishabh Agrahari", "author_url": "", "post_date": "2019-09-07T14:12:12.303000", "content": "Oh man, can't express in words how important you have been for this competition. Your kernels and insights in the discussion threads... I've learnt a lot from you. People like you make this community better every day. Sincere thanks.
", "votes": 3, "replies": [ { "id": 620703, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-07T21:28:59.713000", "content": "Hi Rishabh, thank you for your words! I also learned a lot from you, really! (this includes other competitions where your kernels are also excellent :)
", "votes": 1, "replies": [] }, { "id": 620857, "author_name": "Rishabh Agrahari", "author_url": "", "post_date": "2019-09-08T02:28:30.063000", "content": "Thanks, exciting times ahead :)
", "votes": 1, "replies": [] } ] }, { "id": 621287, "author_name": "Anna Novikova", "author_url": "", "post_date": "2019-09-08T12:03:43.637000", "content": "@ratthachat Thanks for your wonderful kernels and for your story
", "votes": 1, "replies": [ { "id": 621365, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T13:17:03.073000", "content": "@demonplus Thanks also for making this competition wonderful Anna, see you in the next round! :)
", "votes": 1, "replies": [] } ] }, { "id": 621203, "author_name": "Nitish Gupta", "author_url": "", "post_date": "2019-09-08T10:10:58.397000", "content": "Thank you for your kernels @ratthachat learnt a lot from them, although I am nowhere close to others in this thread but I do hope to improve in other competitions.
", "votes": 1, "replies": [ { "id": 621373, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T13:19:22.770000", "content": "Hey @imnitishng you will surely improve, just please continue , you will surprise how much you grow ;)
", "votes": 1, "replies": [] } ] }, { "id": 621146, "author_name": "Takayoshi Makabe", "author_url": "", "post_date": "2019-09-08T08:49:40.850000", "content": "I think your inspiration will raise this competition to a higher level. Thank you and your teammates!!
", "votes": 1, "replies": [ { "id": 621370, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T13:17:54.080000", "content": "Really appreciate to your kind words Takayoshi @spidermandance !
", "votes": 1, "replies": [] } ] }, { "id": 620799, "author_name": "ttya16", "author_url": "", "post_date": "2019-09-08T01:09:30.030000", "content": "Thank you so much for your great contribution.\nYour kernels and discussions helped me a lot in understanding and tackling the problems.
", "votes": 1, "replies": [ { "id": 621114, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T07:57:45.577000", "content": "@ttya16 Thanks so much! I have to keep learning too, lets do it together !
", "votes": 1, "replies": [] } ] }, { "id": 620743, "author_name": "Viraj Bagal", "author_url": "", "post_date": "2019-09-07T23:50:11.057000", "content": "I have learnt a lot from you. Thanks for sharing your knowledge @ratthachat . Good Luck
", "votes": 1, "replies": [ { "id": 620801, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T01:11:34.430000", "content": "Thanks Viraj @virajbagal ! Let us continue to learn together.
", "votes": 1, "replies": [] } ] }, { "id": 620713, "author_name": "Hilal Shaath", "author_url": "", "post_date": "2019-09-07T21:43:03.747000", "content": "Good luck to you. And good night. Your kernel is amazing!
", "votes": 1, "replies": [ { "id": 620800, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T01:10:38.740000", "content": "Thanks Hilal @abualabed ! See you in the next comp!
", "votes": 1, "replies": [] } ] }, { "id": 620689, "author_name": "Carlos Prades K.", "author_url": "", "post_date": "2019-09-07T21:09:59.210000", "content": "Inspiring and insightful story @ratthachat \nThank you for all what you have shared and for sharing this experience as well, which I understand as the importance of focusing on the fundamentals instead of spending most of the time searching for something different.\nI bet your model will be robust to the possible shake up, since you have a solid preprocessing.. let's see. Best wishes ;)
", "votes": 1, "replies": [ { "id": 620711, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-07T21:41:16.007000", "content": "Carlos, thanks for your encouragement and wish. Really appreaciate it! I didn't dare saying anything before the shake up :D I wish the best of luck for you too!
", "votes": 1, "replies": [] } ] }, { "id": 620593, "author_name": "Rashidul H", "author_url": "", "post_date": "2019-09-07T18:08:26.813000", "content": "Congrats! Great job, keep it up.
", "votes": 1, "replies": [] }, { "id": 620568, "author_name": "Manoj Prabhakar", "author_url": "", "post_date": "2019-09-07T17:22:23.307000", "content": "I have learnt a lot from you @ratthachat . Your kernels were the starting point for my solution. My best wishes for best result for you :).
", "votes": 1, "replies": [ { "id": 620707, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-07T21:36:17.563000", "content": "Thanks Manoj!! I wish the best for you too.
", "votes": 1, "replies": [] } ] }, { "id": 620538, "author_name": "Federico Raimondi Cominesi", "author_url": "", "post_date": "2019-09-07T16:19:07.037000", "content": "Thanks @ratthachat ! Your work in this competition was excellent.
\n\nReally loved your kernels. Good luck! 😊
", "votes": 1, "replies": [ { "id": 620704, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-07T21:32:03.340000", "content": "Hi Federico, let us cont. to learn together in the next comp.!! It's truly nice to connect with you :)
", "votes": 1, "replies": [] } ] }, { "id": 620826, "author_name": "sh", "author_url": "", "post_date": "2019-09-08T01:51:45.847000", "content": "Thanks a lot @ratthachat , your preprocessing visualization kernel taught me many new methods and helped me understand what each one did. This is my first medal and your kernel definitely helped. Its too bad that the shakeup was pretty intense for people at the top of the LB but your resources will continue to be invaluable :)
", "votes": 2, "replies": [ { "id": 621110, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T07:56:37.533000", "content": "Hi @sidhanthholalkere ! Thank you so much for your encouragement! I hope we both can continue to contribute to our community better and better.
", "votes": 1, "replies": [] } ] }, { "id": 620780, "author_name": "higepon", "author_url": "", "post_date": "2019-09-08T00:48:11.537000", "content": "Thank you for all your contributions to the community and congratulations on your great result!\nI learned a lot from you.
\n\nOne surprise is that you have 3 year old daughter. I'd love to hear your time management tips. As I know it's really hard to focus and spend time on something intense like kaggle. (I'm a parent also).
", "votes": 2, "replies": [ { "id": 620798, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T01:09:07.110000", "content": "I will email to say hi to you @higepon :)
", "votes": 3, "replies": [] }, { "id": 620815, "author_name": "higepon", "author_url": "", "post_date": "2019-09-08T01:31:58.920000", "content": "Wut? Cool :)
", "votes": 1, "replies": [] } ] }, { "id": 620542, "author_name": "Rohit Modi", "author_url": "", "post_date": "2019-09-07T16:25:11.290000", "content": "Learned a lot through this competition, also your kernels and work in the discussions is amazing and is extremely helpful, as i have learned a lot through them @ratthachat @drhabib.
", "votes": 2, "replies": [ { "id": 621120, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T08:08:52.083000", "content": "@modojj Thanks so much for your kind words! Let us continue our journey into the next challenge ;)
", "votes": 1, "replies": [] } ] }, { "id": 620439, "author_name": "donglee", "author_url": "", "post_date": "2019-09-07T14:03:16.067000", "content": "I have learned a lot from your kernel, wish you good luck!
", "votes": 2, "replies": [ { "id": 620702, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-07T21:27:34.633000", "content": "Thanks so much donglee, let us continue to learn together :)
", "votes": 2, "replies": [] } ] }, { "id": 620436, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-09-07T13:59:56.457000", "content": "Good Luck.. @ratthachat
", "votes": 1, "replies": [] }, { "id": 621695, "author_name": "", "author_url": "", "post_date": "2019-09-08T20:46:11.440000", "content": "", "votes": 1, "replies": [ { "id": 621834, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-09T02:11:19.097000", "content": "@thestonemx Oscar, I have one kernels for preprocessing training data, another one for external data, another one on test data, and another one on creating pseudo labelling. Yet another one on “real training” ... So my kernels are really mess. But I share main idea here.
\n\nYou can see the big picture from the post of Bibek I mentioned above. Straightforwardly, you just use you best model to predict “test data”, and then assign the labels and use it like a normal training dataset. Using this, you should get a good result in a very straightforward way.
\n\nIn my case, a little trick, I found to be better than the above method is to use “2 best models” to predict together, and assign “pseudo soft labels” instead of “pseudo hard labels” e.g. if one model predicts ‘1’ and other predicts ‘2’, I will assign soft label as ‘2.5’. These two models are very consistent and always predict on+/- 1 range, so there is no case like (0+4)/2 = 2.
", "votes": 2, "replies": [] } ] }, { "id": 621211, "author_name": "Noah Weber", "author_url": "", "post_date": "2019-09-08T10:32:26.550000", "content": "Thanks for the contributions:)
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "620430": "(I wrote this thread before the deadline end just not to lose inspiration to write due to potential shake up :) \n\nAPTOS 2019, really, had been a magical competition for me. Never in my dream that I would be able to compete head-to-head with top kagglers until the end (at least according to the public LB); we have at least Data Science Bowl champion, Jigsaw champion, Whale champion, Quora champion and many gold/silver/bronze-awards computer vision experts as well as many new rising stars; all of them I have learned so much from their sharing in the past as well as new things I have learned from many kernel / discussion contributors here in APTOS too! (I tried to vote your topics/kernels as much as I could)\n\nI have to thanks to all my four teammates, and I am honor to be team up with them. I will write in details how each one of them help me improve my own solution below. (hopefully not too much shaken up)\n\nI also thank every participant in this competition to have loved all my kernels & discussions. As you can see that in other competitions, I have completely normal performances :) , so I will treat my performance here as approximately 63% lucky factor (1 - 1/e).\n\nBefore going to my solution, I would like to share some secret : “before” this competition, my philosophy in competition “in the past” is to find some secret magic, i.e. finding secret architectures / secret loss functions / secret training process / hyperparameters etc. …. However, as I **failed** all of these in previous competitions, (and having read/talked with others) I have changed my philosophy to “finding basics” and “sharing to community” (as I thought that I have no chance anyway :), \n\nand when I gave up to find magic, ironically, that was, how real “magic” experience happen to me …\n\n—————\nThis is a night-time in Thailand, and I am going to sleep with my 3-year old daughter. Deadline is at 7.00AM of my tomorrow morning, so when I wake up I will see the results, and I will add my personal write-up here :) \n\nNo matter how much shake-up will happen, this is not the end of our journey … It’s just a new beginning to the new challenge.\n\nGood Luck to You All!\n\n-----\n**updated** the result is out, we didn't choose robust enough choice :)) \nCongratulations to all winners!!!\nWhat I wrote above didn't change. I recieve overwhelming learning and friendship experience and great teamwork from these 2-3 months. Thank you all my friends.\n\nMy personal best model regarding stage1 got satisfiable results of Pub850/Priv929 and quite consistent CV.\n\n### Update: it is quite surprising (or not) that many top solutions use similar methods as my models esp. the pseudo labelling which seems to be everyone key ingredient. Beside this trick, I almost share all of my ideas with kernels / discussions ... So maybe I will just briefly mention my solution here: \n\nI use very similar model to Gary's (8th place) with 2-heads (regression/classification) and also pseudo labelling. Preprocessing / augmentations are exactly the same as my kernels. By using iterative pseudo labels very similar to this post by @bibek777 \n\nhttps://www.kaggle.com/c/aptos2019-blindness-detection/discussion/106177#latest-610648\n\nBy iterating 3 times and ensemble them I was able to boost from Public 0.82x --> 0.850\n\nI could continue but I was also afraid to overfit ( but I did not see overfitting sign by my CV and my Grad-CAM visualization)\n\nIt turns out that this model is not overfit but got a very little boost in private (926 --> 929), this is understandable since public is not like private at all :) Since this is my personal model, we didn't select it and rather try more promising combination to others\n\n## Finally and most importantly, congrat to Aravind to have many high-quality model reaching 0.93x!!", "621934": "Hi @ratthachat \nThank you for your great contribution to Kaggle community.\nI have learned a lot from you and sometimes discussed over the same topic.\n\nSame as your opinion, I thought that the grading distribution of private test data would be like that of train.\nBut we could not bet on only train-like distribution. \nIn the end, we made 2 types of models, one was trained with APTOS train distribution, the other was with much higher severity distribution (I have posted our model pipeline below, the upper model was the former).\nIf we could bet on only... ok. That's enough, I will stop talking about “woulda, coulda shoulda”.\n \n \nNow competition is over, please have a rest at ease.\nSee in another competition.\n\n", "620433": "Neuron Engineer! Your kernels were big part of my solutions =) Thank you very much for doing awesome work! =) ", "620783": "@ratthachat Thanks for all your fantastic contributions throughout this competition, you've helped me and many others out considerably. Hard luck with the final drop but like you say, the real gain here is the amount we learn and the people we work with. See you in the next competition!", "620757": "Thanks @ratthachat , I have learned a lot from you, this is my first competition medal, and what you shared helped me a lot, I was rooting for you to get gold, but for us all, knowledge is always the biggest prize.", "620549": " @ratthachat, whatever happens congratulations. You did great in this comeptition.", "620448": "Oh man, can't express in words how important you have been for this competition. Your kernels and insights in the discussion threads... I've learnt a lot from you. People like you make this community better every day. Sincere thanks.", "621287": " @ratthachat Thanks for your wonderful kernels and for your story", "621203": "Thank you for your kernels @ratthachat learnt a lot from them, although I am nowhere close to others in this thread but I do hope to improve in other competitions.", "621146": "\nI think your inspiration will raise this competition to a higher level. Thank you and your teammates!!", "620799": "Thank you so much for your great contribution.\nYour kernels and discussions helped me a lot in understanding and tackling the problems.", "620743": "I have learnt a lot from you. Thanks for sharing your knowledge @ratthachat . Good Luck ", "620713": "Good luck to you. And good night. Your kernel is amazing!", "620689": "Inspiring and insightful story @ratthachat \nThank you for all what you have shared and for sharing this experience as well, which I understand as the importance of focusing on the fundamentals instead of spending most of the time searching for something different.\nI bet your model will be robust to the possible shake up, since you have a solid preprocessing.. let's see. Best wishes ;)", "620593": "Congrats! Great job, keep it up.", "620568": "I have learnt a lot from you @ratthachat . Your kernels were the starting point for my solution. My best wishes for best result for you :). ", "620538": "Thanks @ratthachat ! Your work in this competition was excellent.\n\nReally loved your kernels. Good luck! 😊 ", "620826": "Thanks a lot @ratthachat , your preprocessing visualization kernel taught me many new methods and helped me understand what each one did. This is my first medal and your kernel definitely helped. Its too bad that the shakeup was pretty intense for people at the top of the LB but your resources will continue to be invaluable :)", "620780": "Thank you for all your contributions to the community and congratulations on your great result!\nI learned a lot from you.\n\nOne surprise is that you have 3 year old daughter. I'd love to hear your time management tips. As I know it's really hard to focus and spend time on something intense like kaggle. (I'm a parent also).", "620542": "Learned a lot through this competition, also your kernels and work in the discussions is amazing and is extremely helpful, as i have learned a lot through them @ratthachat @drhabib.", "620439": "I have learned a lot from your kernel, wish you good luck!", "620436": "Good Luck.. @ratthachat ", "621695": "", "621211": "Thanks for the contributions:)" } }