{
  "id": 107960,
  "title": "5th place solution",
  "url": "/competitions/aptos2019-blindness-detection/discussion/107960",
  "author_name": "Gary",
  "post_date": "2019-09-08T06:46:34.089000",
  "votes": 69,
  "comment_count": 24,
  "views": 0,
  "content": "<p>Congrats to everyone,  Here is my solution, not complicated.</p>\n\n<h2>models</h2>\n\n<ul>\n<li>efficientnet-b0</li>\n<li>efficientnet-b1</li>\n<li>efficientnet-b2</li>\n<li>efficientnet-b3\nI just combined the classification and regression task together, multi loss maybe not easy to ovetfitting. But the test phrase just with the prediction of regression, code like these\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1270655%2F600b35c1bac596b5636e9d40c8b2c8cd%2Fcode.jpg?generation=1567923622631376&amp;alt=media\" alt=\"\"></li>\n</ul>\n\n<h2>Data preprocessing and Augmentations</h2>\n\n<ul>\n<li>No more special preprocessing method, I just cropped black background on images.</li>\n<li>image_size: 384x384</li>\n<li>Just Fliplr, Flipud, Randomrotate(0, 360), zoom(1.0, 1.35) and self designed perspective transformation method.</li>\n</ul>\n\n<h2>Training strategy</h2>\n\n<ul>\n<li>5 fold CV</li>\n<li>according to the better public LB, I did not pretrained on 2015-data and then finetune on 2019-data, I just combined 2015-data and 2019-data together and then train the model, valid on 2015 priavte test data and 2019 valid data.</li>\n<li>lr_schedule: adam optimizer, lrs=[5e-4, 1e-4, 1e-5, 1e-6], and the position of lr decays is [10, 16, 22], train 25 epochs.</li>\n<li>2*TTA</li>\n<li>threshold: [0.5, 1.5, 2.5, 3.5]</li>\n</ul>\n\n<h2>Pseudo labels</h2>\n\n<ul>\n<li>I think the pseudo labels is the most import key to win in this competition. What I thinking about is, because we don't know the distribution of the 2019 private test set, it might be similar with 2015data, or 2019 training data or 2019 public data. So I can combine three different distribution data together to train the model,  and the model can recognition three different distribution data.</li>\n<li>I don't use all the 2019 public data as the pseudo labels samples, I chose the higer confidence samples, the codes like these\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1270655%2F8579cf548aeb7a64be574dc56236bf9e%2Fpseudo_labels.jpg?generation=1567924747948513&amp;alt=media\" alt=\"\"></li>\n<li>pseudo labels helpd me improve the local mse loss and qwk.</li>\n</ul>\n\n<h2>Conclusion</h2>\n\n<ul>\n<li>the trainning data is strange and most kagglers met some strange problems in this competition, like the LB is unstable and can not find the relation between the local cv and public LB, and...</li>\n<li>But the final results show that we can still trust our local CV.</li>\n</ul>",
  "messages": [
    {
      "id": 621025,
      "postDate": "2019-09-08T06:46:34.090Z",
      "content": "<p>Congrats to everyone,  Here is my solution, not complicated.</p>\n\n<h2>models</h2>\n\n<ul>\n<li>efficientnet-b0</li>\n<li>efficientnet-b1</li>\n<li>efficientnet-b2</li>\n<li>efficientnet-b3\nI just combined the classification and regression task together, multi loss maybe not easy to ovetfitting. But the test phrase just with the prediction of regression, code like these\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1270655%2F600b35c1bac596b5636e9d40c8b2c8cd%2Fcode.jpg?generation=1567923622631376&amp;alt=media\" alt=\"\"></li>\n</ul>\n\n<h2>Data preprocessing and Augmentations</h2>\n\n<ul>\n<li>No more special preprocessing method, I just cropped black background on images.</li>\n<li>image_size: 384x384</li>\n<li>Just Fliplr, Flipud, Randomrotate(0, 360), zoom(1.0, 1.35) and self designed perspective transformation method.</li>\n</ul>\n\n<h2>Training strategy</h2>\n\n<ul>\n<li>5 fold CV</li>\n<li>according to the better public LB, I did not pretrained on 2015-data and then finetune on 2019-data, I just combined 2015-data and 2019-data together and then train the model, valid on 2015 priavte test data and 2019 valid data.</li>\n<li>lr_schedule: adam optimizer, lrs=[5e-4, 1e-4, 1e-5, 1e-6], and the position of lr decays is [10, 16, 22], train 25 epochs.</li>\n<li>2*TTA</li>\n<li>threshold: [0.5, 1.5, 2.5, 3.5]</li>\n</ul>\n\n<h2>Pseudo labels</h2>\n\n<ul>\n<li>I think the pseudo labels is the most import key to win in this competition. What I thinking about is, because we don't know the distribution of the 2019 private test set, it might be similar with 2015data, or 2019 training data or 2019 public data. So I can combine three different distribution data together to train the model,  and the model can recognition three different distribution data.</li>\n<li>I don't use all the 2019 public data as the pseudo labels samples, I chose the higer confidence samples, the codes like these\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1270655%2F8579cf548aeb7a64be574dc56236bf9e%2Fpseudo_labels.jpg?generation=1567924747948513&amp;alt=media\" alt=\"\"></li>\n<li>pseudo labels helpd me improve the local mse loss and qwk.</li>\n</ul>\n\n<h2>Conclusion</h2>\n\n<ul>\n<li>the trainning data is strange and most kagglers met some strange problems in this competition, like the LB is unstable and can not find the relation between the local cv and public LB, and...</li>\n<li>But the final results show that we can still trust our local CV.</li>\n</ul>",
      "rawMarkdown": "Congrats to everyone,  Here is my solution, not complicated.\n## models\n- efficientnet-b0\n- efficientnet-b1\n- efficientnet-b2\n- efficientnet-b3\nI just combined the classification and regression task together, multi loss maybe not easy to ovetfitting. But the test phrase just with the prediction of regression, code like these\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1270655%2F600b35c1bac596b5636e9d40c8b2c8cd%2Fcode.jpg?generation=1567923622631376&amp;alt=media)\n\n## Data preprocessing and Augmentations\n- No more special preprocessing method, I just cropped black background on images.\n- image_size: 384x384\n- Just Fliplr, Flipud, Randomrotate(0, 360), zoom(1.0, 1.35) and self designed perspective transformation method.\n\n## Training strategy\n- 5 fold CV\n- according to the better public LB, I did not pretrained on 2015-data and then finetune on 2019-data, I just combined 2015-data and 2019-data together and then train the model, valid on 2015 priavte test data and 2019 valid data.\n- lr_schedule: adam optimizer, lrs=[5e-4, 1e-4, 1e-5, 1e-6], and the position of lr decays is [10, 16, 22], train 25 epochs.\n- 2*TTA\n- threshold: [0.5, 1.5, 2.5, 3.5]\n \n## Pseudo labels\n- I think the pseudo labels is the most import key to win in this competition. What I thinking about is, because we don't know the distribution of the 2019 private test set, it might be similar with 2015data, or 2019 training data or 2019 public data. So I can combine three different distribution data together to train the model,  and the model can recognition three different distribution data.\n- I don't use all the 2019 public data as the pseudo labels samples, I chose the higer confidence samples, the codes like these\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1270655%2F8579cf548aeb7a64be574dc56236bf9e%2Fpseudo_labels.jpg?generation=1567924747948513&amp;alt=media)\n- pseudo labels helpd me improve the local mse loss and qwk.\n\n## Conclusion\n- the trainning data is strange and most kagglers met some strange problems in this competition, like the LB is unstable and can not find the relation between the local cv and public LB, and...\n- But the final results show that we can still trust our local CV.",
      "votes": 69
    },
    {
      "id": 734839,
      "postDate": "2020-02-02T04:25:31.513Z",
      "content": "<p>Thanks for explaining your model. I new to deep learning. What do you mean by combining Regression and Classification tasks? You mean you have continuous and discrete scores, and then you used some kind of function to combine them?</p>",
      "rawMarkdown": "Thanks for explaining your model. I new to deep learning. What do you mean by combining Regression and Classification tasks? You mean you have continuous and discrete scores, and then you used some kind of function to combine them?"
    },
    {
      "id": 624391,
      "postDate": "2019-09-12T03:10:24.163Z",
      "content": "<p>Congratulations on gold. I have a very basic question. Pseudo labels you decided to do because training and test qwk scores were very different. Basically when we  decide to go for Pseudo  labeling. </p>",
      "rawMarkdown": "Congratulations on gold. I have a very basic question. Pseudo labels you decided to do because training and test qwk scores were very different. Basically when we  decide to go for Pseudo  labeling. "
    },
    {
      "id": 623782,
      "postDate": "2019-09-11T09:49:13.263Z",
      "content": "<p>Congratulations with your solo gold! Your idea to use multi-headed model for this task is impressive for me, took a note for myself for the future :)</p>",
      "rawMarkdown": "Congratulations with your solo gold! Your idea to use multi-headed model for this task is impressive for me, took a note for myself for the future :)"
    },
    {
      "id": 623270,
      "postDate": "2019-09-10T16:51:10.777Z",
      "content": "<p>Congratz on your gold and really appreciate your insight!</p>\n\n<p>Just wondering, when you do regression and classification together, what is your loss function? still MSE loss?\nAre you average the regression and classification result, or you have better ways to find out their weight.</p>\n\n<p>I have similar ideas during competition, but I failed... :(</p>\n\n<p>I grab a resnet model, and implemented as retina net (some middle layers to output result, I actually picked 5, just like the retina net paper)</p>\n\n<p>Then some conv -&gt; flatten -&gt; pooling -&gt; output, then calculate each output with the ground truth(MSE), then average prediction between them.</p>\n\n<p>What I meant to do here is regression model, but single model gives 5 outputs from different layers, then average them all. </p>\n\n<p>It worked, but when I pretrain it with 2015 data, most of the case the loss just went 0 after couple epochs. </p>",
      "rawMarkdown": "Congratz on your gold and really appreciate your insight!\n\nJust wondering, when you do regression and classification together, what is your loss function? still MSE loss?\nAre you average the regression and classification result, or you have better ways to find out their weight.\n\nI have similar ideas during competition, but I failed... :(\n\nI grab a resnet model, and implemented as retina net (some middle layers to output result, I actually picked 5, just like the retina net paper)\n\nThen some conv -&gt; flatten -&gt; pooling -&gt; output, then calculate each output with the ground truth(MSE), then average prediction between them.\n\nWhat I meant to do here is regression model, but single model gives 5 outputs from different layers, then average them all. \n\nIt worked, but when I pretrain it with 2015 data, most of the case the loss just went 0 after couple epochs. "
    },
    {
      "id": 621569,
      "postDate": "2019-09-08T17:08:35.923Z",
      "content": "<p>Congrats! </p>",
      "rawMarkdown": "Congrats! "
    },
    {
      "id": 621499,
      "postDate": "2019-09-08T15:40:31.810Z",
      "content": "<p>Thanks for sharing your solution! I've also tried multi-task learning but that didn't work for me.... I felt it was difficult to balance 2 losses(BCE and RMSE?), can you please let me know your thoughts about how to manage 2 different scale loss?? </p>",
      "rawMarkdown": "Thanks for sharing your solution! I've also tried multi-task learning but that didn't work for me.... I felt it was difficult to balance 2 losses(BCE and RMSE?), can you please let me know your thoughts about how to manage 2 different scale loss?? "
    },
    {
      "id": 621449,
      "postDate": "2019-09-08T14:21:30.307Z",
      "content": "<p>Congratulations!\nFinally it turns out that you trusted the truth and not getting overfitting 😆 </p>",
      "rawMarkdown": "Congratulations!\nFinally it turns out that you trusted the truth and not getting overfitting 😆 ",
      "replies": [
        {
          "id": 621461,
          "postDate": "2019-09-08T14:39:10.230Z",
          "content": "<p>Hah..I just a lucky man😏 </p>",
          "rawMarkdown": "Hah..I just a lucky man😏 "
        }
      ]
    },
    {
      "id": 621396,
      "postDate": "2019-09-08T13:33:35.920Z",
      "content": "<p>Congrats <a href=\"/garybios\">@garybios</a> on a solo gold  and thanks for sharing your solution overview.</p>",
      "rawMarkdown": "Congrats @garybios on a solo gold  and thanks for sharing your solution overview."
    },
    {
      "id": 621380,
      "postDate": "2019-09-08T13:24:43.610Z",
      "content": "<p>Congratulations! </p>",
      "rawMarkdown": "Congratulations! "
    },
    {
      "id": 621170,
      "postDate": "2019-09-08T09:20:44.953Z",
      "content": "<p>Congrats!</p>",
      "rawMarkdown": "Congrats!"
    },
    {
      "id": 621117,
      "postDate": "2019-09-08T08:06:42.757Z",
      "content": "<p>Thanks for the great tip! I'm new to deep learning, may I ask some novice questions? What do u meant by pseudo labelling? on which data? and how did u combine the distribution of multiple datasets? Thank you for sharing!</p>",
      "rawMarkdown": "Thanks for the great tip! I'm new to deep learning, may I ask some novice questions? What do u meant by pseudo labelling? on which data? and how did u combine the distribution of multiple datasets? Thank you for sharing!",
      "replies": [
        {
          "id": 621133,
          "postDate": "2019-09-08T08:31:18.803Z",
          "content": "<p>on 2019 public test-set, you can google \"what is pseudo label\".</p>",
          "rawMarkdown": "on 2019 public test-set, you can google \"what is pseudo label\"."
        },
        {
          "id": 621141,
          "postDate": "2019-09-08T08:43:04.100Z",
          "content": "<p>wow, I always thought it's for external unlabelled data, never thought I could use it on test data! I'll try it next time. thanks!</p>",
          "rawMarkdown": "wow, I always thought it's for external unlabelled data, never thought I could use it on test data! I'll try it next time. thanks!"
        },
        {
          "id": 621143,
          "postDate": "2019-09-08T08:47:43.040Z",
          "content": "<p>public test-set is also a unlabelled data for us.</p>",
          "rawMarkdown": "public test-set is also a unlabelled data for us."
        }
      ]
    },
    {
      "id": 621087,
      "postDate": "2019-09-08T07:30:35.960Z",
      "content": "<p>Congratulations, nice solution</p>",
      "rawMarkdown": "Congratulations, nice solution"
    },
    {
      "id": 621081,
      "postDate": "2019-09-08T07:26:49.050Z",
      "content": "<p>Hey Gary! Congrat to your solo gold! Huge acheivement!!</p>",
      "rawMarkdown": "Hey Gary! Congrat to your solo gold! Huge acheivement!!",
      "replies": [
        {
          "id": 621086,
          "postDate": "2019-09-08T07:29:13.280Z",
          "content": "<p>Thank you, you also did a great contribution for this competition.</p>",
          "rawMarkdown": "Thank you, you also did a great contribution for this competition."
        }
      ]
    },
    {
      "id": 621050,
      "postDate": "2019-09-08T07:02:37.943Z",
      "content": "<p>Congrats and thanks for sharing :)\nWhat image size did you use for your efficient models?</p>",
      "rawMarkdown": "Congrats and thanks for sharing :)\nWhat image size did you use for your efficient models?",
      "replies": [
        {
          "id": 621075,
          "postDate": "2019-09-08T07:24:09.937Z",
          "content": "<p>sorry, I forget about this, it is 384x384.</p>",
          "rawMarkdown": "sorry, I forget about this, it is 384x384.",
          "votes": 1
        }
      ]
    },
    {
      "id": 621047,
      "postDate": "2019-09-08T07:01:04.093Z",
      "content": "<p>congratulations!! gary\nwonderful solution, I also  combined the classification and regression to train, and test only use regression, however I don't use Pseudo labels to further improve</p>",
      "rawMarkdown": "congratulations!! gary\nwonderful solution, I also  combined the classification and regression to train, and test only use regression, however I don't use Pseudo labels to further improve",
      "replies": [
        {
          "id": 621071,
          "postDate": "2019-09-08T07:23:08.177Z",
          "content": "<p>thank you, you can try pseudo labels next time.</p>",
          "rawMarkdown": "thank you, you can try pseudo labels next time."
        }
      ]
    },
    {
      "id": 621952,
      "postDate": "2019-09-09T05:47:58.643Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 621037,
      "postDate": "2019-09-08T06:57:23.510Z",
      "content": "<p>Thank you for sharing this. </p>",
      "rawMarkdown": "Thank you for sharing this. ",
      "votes": 1
    }
  ],
  "comments": [
    {
      "id": 734839,
      "author_name": "palakurthi",
      "author_url": "",
      "post_date": "2020-02-02T04:25:31.513000",
      "content": "<p>Thanks for explaining your model. I new to deep learning. What do you mean by combining Regression and Classification tasks? You mean you have continuous and discrete scores, and then you used some kind of function to combine them?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 624391,
      "author_name": "NeerajSharma",
      "author_url": "",
      "post_date": "2019-09-12T03:10:24.163000",
      "content": "<p>Congratulations on gold. I have a very basic question. Pseudo labels you decided to do because training and test qwk scores were very different. Basically when we  decide to go for Pseudo  labeling. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 623782,
      "author_name": "Evgeny Kovalev",
      "author_url": "",
      "post_date": "2019-09-11T09:49:13.263000",
      "content": "<p>Congratulations with your solo gold! Your idea to use multi-headed model for this task is impressive for me, took a note for myself for the future :)</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 623270,
      "author_name": "Hao He",
      "author_url": "",
      "post_date": "2019-09-10T16:51:10.777000",
      "content": "<p>Congratz on your gold and really appreciate your insight!</p>\n\n<p>Just wondering, when you do regression and classification together, what is your loss function? still MSE loss?\nAre you average the regression and classification result, or you have better ways to find out their weight.</p>\n\n<p>I have similar ideas during competition, but I failed... :(</p>\n\n<p>I grab a resnet model, and implemented as retina net (some middle layers to output result, I actually picked 5, just like the retina net paper)</p>\n\n<p>Then some conv -&gt; flatten -&gt; pooling -&gt; output, then calculate each output with the ground truth(MSE), then average prediction between them.</p>\n\n<p>What I meant to do here is regression model, but single model gives 5 outputs from different layers, then average them all. </p>\n\n<p>It worked, but when I pretrain it with 2015 data, most of the case the loss just went 0 after couple epochs. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621569,
      "author_name": "Xuan Cao",
      "author_url": "",
      "post_date": "2019-09-08T17:08:35.923000",
      "content": "<p>Congrats! </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621499,
      "author_name": "Camaro",
      "author_url": "",
      "post_date": "2019-09-08T15:40:31.810000",
      "content": "<p>Thanks for sharing your solution! I've also tried multi-task learning but that didn't work for me.... I felt it was difficult to balance 2 losses(BCE and RMSE?), can you please let me know your thoughts about how to manage 2 different scale loss?? </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621449,
      "author_name": "Qishen Ha",
      "author_url": "",
      "post_date": "2019-09-08T14:21:30.307000",
      "content": "<p>Congratulations!\nFinally it turns out that you trusted the truth and not getting overfitting 😆 </p>",
      "votes": 0,
      "replies": [
        {
          "id": 621461,
          "author_name": "Gary",
          "author_url": "",
          "post_date": "2019-09-08T14:39:10.230000",
          "content": "<p>Hah..I just a lucky man😏 </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 621396,
      "author_name": "YaGana Sheriff-Hussaini",
      "author_url": "",
      "post_date": "2019-09-08T13:33:35.920000",
      "content": "<p>Congrats <a href=\"/garybios\">@garybios</a> on a solo gold  and thanks for sharing your solution overview.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621380,
      "author_name": "seefun",
      "author_url": "",
      "post_date": "2019-09-08T13:24:43.610000",
      "content": "<p>Congratulations! </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621170,
      "author_name": "Yuanhao",
      "author_url": "",
      "post_date": "2019-09-08T09:20:44.953000",
      "content": "<p>Congrats!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621117,
      "author_name": "Sirong Huang",
      "author_url": "",
      "post_date": "2019-09-08T08:06:42.757000",
      "content": "<p>Thanks for the great tip! I'm new to deep learning, may I ask some novice questions? What do u meant by pseudo labelling? on which data? and how did u combine the distribution of multiple datasets? Thank you for sharing!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 621133,
          "author_name": "Gary",
          "author_url": "",
          "post_date": "2019-09-08T08:31:18.803000",
          "content": "<p>on 2019 public test-set, you can google \"what is pseudo label\".</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 621141,
          "author_name": "Sirong Huang",
          "author_url": "",
          "post_date": "2019-09-08T08:43:04.100000",
          "content": "<p>wow, I always thought it's for external unlabelled data, never thought I could use it on test data! I'll try it next time. thanks!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 621143,
          "author_name": "Gary",
          "author_url": "",
          "post_date": "2019-09-08T08:47:43.040000",
          "content": "<p>public test-set is also a unlabelled data for us.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 621087,
      "author_name": "Anna Novikova",
      "author_url": "",
      "post_date": "2019-09-08T07:30:35.960000",
      "content": "<p>Congratulations, nice solution</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621081,
      "author_name": "Neuron Engineer",
      "author_url": "",
      "post_date": "2019-09-08T07:26:49.050000",
      "content": "<p>Hey Gary! Congrat to your solo gold! Huge acheivement!!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 621086,
          "author_name": "Gary",
          "author_url": "",
          "post_date": "2019-09-08T07:29:13.280000",
          "content": "<p>Thank you, you also did a great contribution for this competition.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 621050,
      "author_name": "Rishabh Agrahari",
      "author_url": "",
      "post_date": "2019-09-08T07:02:37.943000",
      "content": "<p>Congrats and thanks for sharing :)\nWhat image size did you use for your efficient models?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 621075,
          "author_name": "Gary",
          "author_url": "",
          "post_date": "2019-09-08T07:24:09.937000",
          "content": "<p>sorry, I forget about this, it is 384x384.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 621047,
      "author_name": "kevin",
      "author_url": "",
      "post_date": "2019-09-08T07:01:04.093000",
      "content": "<p>congratulations!! gary\nwonderful solution, I also  combined the classification and regression to train, and test only use regression, however I don't use Pseudo labels to further improve</p>",
      "votes": 0,
      "replies": [
        {
          "id": 621071,
          "author_name": "Gary",
          "author_url": "",
          "post_date": "2019-09-08T07:23:08.177000",
          "content": "<p>thank you, you can try pseudo labels next time.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 621952,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-09-09T05:47:58.643000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 621037,
      "author_name": "Anand Selvadurai",
      "author_url": "",
      "post_date": "2019-09-08T06:57:23.510000",
      "content": "<p>Thank you for sharing this. </p>",
      "votes": 1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "621025": "Congrats to everyone,  Here is my solution, not complicated.\n## models\n- efficientnet-b0\n- efficientnet-b1\n- efficientnet-b2\n- efficientnet-b3\nI just combined the classification and regression task together, multi loss maybe not easy to ovetfitting. But the test phrase just with the prediction of regression, code like these\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1270655%2F600b35c1bac596b5636e9d40c8b2c8cd%2Fcode.jpg?generation=1567923622631376&amp;alt=media)\n\n## Data preprocessing and Augmentations\n- No more special preprocessing method, I just cropped black background on images.\n- image_size: 384x384\n- Just Fliplr, Flipud, Randomrotate(0, 360), zoom(1.0, 1.35) and self designed perspective transformation method.\n\n## Training strategy\n- 5 fold CV\n- according to the better public LB, I did not pretrained on 2015-data and then finetune on 2019-data, I just combined 2015-data and 2019-data together and then train the model, valid on 2015 priavte test data and 2019 valid data.\n- lr_schedule: adam optimizer, lrs=[5e-4, 1e-4, 1e-5, 1e-6], and the position of lr decays is [10, 16, 22], train 25 epochs.\n- 2*TTA\n- threshold: [0.5, 1.5, 2.5, 3.5]\n \n## Pseudo labels\n- I think the pseudo labels is the most import key to win in this competition. What I thinking about is, because we don't know the distribution of the 2019 private test set, it might be similar with 2015data, or 2019 training data or 2019 public data. So I can combine three different distribution data together to train the model,  and the model can recognition three different distribution data.\n- I don't use all the 2019 public data as the pseudo labels samples, I chose the higer confidence samples, the codes like these\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1270655%2F8579cf548aeb7a64be574dc56236bf9e%2Fpseudo_labels.jpg?generation=1567924747948513&amp;alt=media)\n- pseudo labels helpd me improve the local mse loss and qwk.\n\n## Conclusion\n- the trainning data is strange and most kagglers met some strange problems in this competition, like the LB is unstable and can not find the relation between the local cv and public LB, and...\n- But the final results show that we can still trust our local CV.",
    "734839": "Thanks for explaining your model. I new to deep learning. What do you mean by combining Regression and Classification tasks? You mean you have continuous and discrete scores, and then you used some kind of function to combine them?",
    "624391": "Congratulations on gold. I have a very basic question. Pseudo labels you decided to do because training and test qwk scores were very different. Basically when we  decide to go for Pseudo  labeling. ",
    "623782": "Congratulations with your solo gold! Your idea to use multi-headed model for this task is impressive for me, took a note for myself for the future :)",
    "623270": "Congratz on your gold and really appreciate your insight!\n\nJust wondering, when you do regression and classification together, what is your loss function? still MSE loss?\nAre you average the regression and classification result, or you have better ways to find out their weight.\n\nI have similar ideas during competition, but I failed... :(\n\nI grab a resnet model, and implemented as retina net (some middle layers to output result, I actually picked 5, just like the retina net paper)\n\nThen some conv -&gt; flatten -&gt; pooling -&gt; output, then calculate each output with the ground truth(MSE), then average prediction between them.\n\nWhat I meant to do here is regression model, but single model gives 5 outputs from different layers, then average them all. \n\nIt worked, but when I pretrain it with 2015 data, most of the case the loss just went 0 after couple epochs. ",
    "621569": "Congrats! ",
    "621499": "Thanks for sharing your solution! I've also tried multi-task learning but that didn't work for me.... I felt it was difficult to balance 2 losses(BCE and RMSE?), can you please let me know your thoughts about how to manage 2 different scale loss?? ",
    "621449": "Congratulations!\nFinally it turns out that you trusted the truth and not getting overfitting 😆 ",
    "621396": "Congrats @garybios on a solo gold  and thanks for sharing your solution overview.",
    "621380": "Congratulations! ",
    "621170": "Congrats!",
    "621117": "Thanks for the great tip! I'm new to deep learning, may I ask some novice questions? What do u meant by pseudo labelling? on which data? and how did u combine the distribution of multiple datasets? Thank you for sharing!",
    "621087": "Congratulations, nice solution",
    "621081": "Hey Gary! Congrat to your solo gold! Huge acheivement!!",
    "621050": "Congrats and thanks for sharing :)\nWhat image size did you use for your efficient models?",
    "621047": "congratulations!! gary\nwonderful solution, I also  combined the classification and regression to train, and test only use regression, however I don't use Pseudo labels to further improve",
    "621952": "",
    "621037": "Thank you for sharing this. "
  }
}