{ "id": 107944, "title": "13th place solution", "url": "/competitions/aptos2019-blindness-detection/discussion/107944", "author_name": "Camaro", "post_date": "2019-09-08T04:02:04.617000", "votes": 62, "comment_count": 22, "views": 0, "content": "

Congrats all the prize winner and everyone finished in gold zone! And I really want to say thank you to my great teammate @mhiro2 and @yasufuminakama, I'm pretty confident that I couldn't get gold medal if I'm alone...

\n\n

This is my first time to write solution, but I'll try to open up what we did as much as possible.

\n\n

OVERVIEW

\n\n\n\n

CHALENGE

\n\n\n\n

DETAIL

\n\n

preprocess

\n\n

The most important part of this competition is preprocess. As everyone realized, this competition had a severe leakage with image size and shape. At first I tried many kinds of preprocess (ex. zoom, circle crop or corner cut etc.) to eliminate\nthis leakage, but almost all failed. When I believed train images and test images looked very similar, that makes LB score significantly dropped. Only rotate worked good for me, but it was very unstable.(ex I used 120 degree. If I changed to 90 or 180, LB score significantly dropped.) \nAnd the most interesting image shape was (480,640), which consist of over 70% of test set. That makes me come up with below hypothesis: \nAfter admin made the dataset, they realized there was strong correlation between image size/shape and diagnosis score. They want to prevent all kagglers from submitting LGBM model which use only image meta features. But they also want to the model which can detect retinopathy, so they decided to crop only some part of public test set....\nThat means the image size and shape information is still useful, so I stopped focusing on preprocess. I'm now confident that this thoughts was somehow right after checking private score because it's incredibly higher than previous competition. And you can see that the evidence of this in this great kernel's private score.\nWhat we used are below. I believe this difference between preprocess might diversify our models.

\n\n

```python

\n\n

for SeResNext

\n\n

Compose([\n BensCrop(img_size), #from 2015 solution\n RandomHorizontalFlip(),\n RandomVerticalFlip(),\n RandomRotation((-120, 120)),\n ),\n ])

\n\n

for EfficientNet B5

\n\n

Compose([\n Resize(img_size),\n HorizontalFlip(),\n VerticalFlip(),\n Rotate(),\n RandomContrast(0.5),\n IAAAdditiveGaussianNoise(p=0.25),\n ])\n```

\n\n

dataset

\n\n

The 2nd important part of our result was how to split and select dataset. I tried many combination, and finally I found below setting works well for me.\n1. split current train dataset to 5 folds by stratified k fold.\n2. train on 4/5 train set and previous competition train set\n3. validate 1/5 train set

\n\n

When I pre-trained on 2015 dataset and fine-tuned on 2019 dataset, my single model(SeResNext)'s score was LB0.811, but after I changed to above method, my score was boost to LB0.827.

\n\n

model and training method

\n\n

We used 2 kinds of model, SeResNext101 and EfficientNet B5. Each models used a differnt training method.

\n\n

For SeResNext101\n- image size:320\n- loss:RMSE\n- train on 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5 of 2019 dataset)

\n\n

For EfficientNet B5\n- image size:256 -> 320\n- loss:MSE\n- 2 step training\n 1. pre-train on 4/5folds of 2015 dataset (validate with 2019 dataset) (img_size=256)\n 2. fine-tune on 1/5fold of 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5fold)(img_size=320)

\n\n

Before starting ensemble, SeResNext scored LB0.827 and EfficientNet scored LB0.830.

\n\n

pseudo labeling and ensemble

\n\n

We used 2 kinds of pseudo labeling.\n1. make soft label for test set and re-train from scratch(ImageNet pre-trained model).\n2. make hard label for test set and fine-tune from trained model\nBoth method boost LB score a lot, so we decided used both method to diversify our models. Sometimes we also use external data (Messidor and IEEE) to give some fresh information to our model, but I'm not sure it's necessary or not.\nBy combining pseudo labeling and ensemble, our both single models reach LB0.846. Below is the result of what we did.(I know it's not beautiful, but we had no time to go back and start from 1st stage model...)

\n\n

\"\"

\n\n

(I deleted the row which didn't contribute to our final model)

\n\n

Finally we submitted below 2 models.\n- No.2+No.5+No.18+No.19(public0.930)\n- No.16(public0.930)

\n\n

UPDATED

\n\n

Here is my single model's scores that shows how pseudo labeling was effective on private dataset.\nmodel: SeResNext101\n- before applied pseudo labeling: private LB 0.922\n- pseudo labeling with only SeResNext101's labels: private LB 0.925\n- pseudo labeling with both SeResNext101 and EfficientNet's labels: private LB 0.931\nInterestingly last model is better than our final submission and enough to win gold medal :)

\n\n

Thank you for reading, please feel free to ask me if you have a question!

", "messages": [ { "id": 620905, "postDate": "2019-09-08T04:02:04.617Z", "content": "

Congrats all the prize winner and everyone finished in gold zone! And I really want to say thank you to my great teammate @mhiro2 and @yasufuminakama, I'm pretty confident that I couldn't get gold medal if I'm alone...

\n\n

This is my first time to write solution, but I'll try to open up what we did as much as possible.

\n\n

OVERVIEW

\n\n\n\n

CHALENGE

\n\n\n\n

DETAIL

\n\n

preprocess

\n\n

The most important part of this competition is preprocess. As everyone realized, this competition had a severe leakage with image size and shape. At first I tried many kinds of preprocess (ex. zoom, circle crop or corner cut etc.) to eliminate\nthis leakage, but almost all failed. When I believed train images and test images looked very similar, that makes LB score significantly dropped. Only rotate worked good for me, but it was very unstable.(ex I used 120 degree. If I changed to 90 or 180, LB score significantly dropped.) \nAnd the most interesting image shape was (480,640), which consist of over 70% of test set. That makes me come up with below hypothesis: \nAfter admin made the dataset, they realized there was strong correlation between image size/shape and diagnosis score. They want to prevent all kagglers from submitting LGBM model which use only image meta features. But they also want to the model which can detect retinopathy, so they decided to crop only some part of public test set....\nThat means the image size and shape information is still useful, so I stopped focusing on preprocess. I'm now confident that this thoughts was somehow right after checking private score because it's incredibly higher than previous competition. And you can see that the evidence of this in this great kernel's private score.\nWhat we used are below. I believe this difference between preprocess might diversify our models.

\n\n

```python

\n\n

for SeResNext

\n\n

Compose([\n BensCrop(img_size), #from 2015 solution\n RandomHorizontalFlip(),\n RandomVerticalFlip(),\n RandomRotation((-120, 120)),\n ),\n ])

\n\n

for EfficientNet B5

\n\n

Compose([\n Resize(img_size),\n HorizontalFlip(),\n VerticalFlip(),\n Rotate(),\n RandomContrast(0.5),\n IAAAdditiveGaussianNoise(p=0.25),\n ])\n```

\n\n

dataset

\n\n

The 2nd important part of our result was how to split and select dataset. I tried many combination, and finally I found below setting works well for me.\n1. split current train dataset to 5 folds by stratified k fold.\n2. train on 4/5 train set and previous competition train set\n3. validate 1/5 train set

\n\n

When I pre-trained on 2015 dataset and fine-tuned on 2019 dataset, my single model(SeResNext)'s score was LB0.811, but after I changed to above method, my score was boost to LB0.827.

\n\n

model and training method

\n\n

We used 2 kinds of model, SeResNext101 and EfficientNet B5. Each models used a differnt training method.

\n\n

For SeResNext101\n- image size:320\n- loss:RMSE\n- train on 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5 of 2019 dataset)

\n\n

For EfficientNet B5\n- image size:256 -> 320\n- loss:MSE\n- 2 step training\n 1. pre-train on 4/5folds of 2015 dataset (validate with 2019 dataset) (img_size=256)\n 2. fine-tune on 1/5fold of 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5fold)(img_size=320)

\n\n

Before starting ensemble, SeResNext scored LB0.827 and EfficientNet scored LB0.830.

\n\n

pseudo labeling and ensemble

\n\n

We used 2 kinds of pseudo labeling.\n1. make soft label for test set and re-train from scratch(ImageNet pre-trained model).\n2. make hard label for test set and fine-tune from trained model\nBoth method boost LB score a lot, so we decided used both method to diversify our models. Sometimes we also use external data (Messidor and IEEE) to give some fresh information to our model, but I'm not sure it's necessary or not.\nBy combining pseudo labeling and ensemble, our both single models reach LB0.846. Below is the result of what we did.(I know it's not beautiful, but we had no time to go back and start from 1st stage model...)

\n\n

\"\"

\n\n

(I deleted the row which didn't contribute to our final model)

\n\n

Finally we submitted below 2 models.\n- No.2+No.5+No.18+No.19(public0.930)\n- No.16(public0.930)

\n\n

UPDATED

\n\n

Here is my single model's scores that shows how pseudo labeling was effective on private dataset.\nmodel: SeResNext101\n- before applied pseudo labeling: private LB 0.922\n- pseudo labeling with only SeResNext101's labels: private LB 0.925\n- pseudo labeling with both SeResNext101 and EfficientNet's labels: private LB 0.931\nInterestingly last model is better than our final submission and enough to win gold medal :)

\n\n

Thank you for reading, please feel free to ask me if you have a question!

", "rawMarkdown": "Congrats all the prize winner and everyone finished in gold zone! And I really want to say thank you to my great teammate @mhiro2 and @yasufuminakama, I'm pretty confident that I couldn't get gold medal if I'm alone...\n\nThis is my first time to write solution, but I'll try to open up what we did as much as possible. \n\n## OVERVIEW\n- ensemble 4 models, 2 SeResNext101 and 2 EfficientNet B5\n- 2 kinds of pseudo labeling\n- use old competition data\n\n## CHALENGE\n- severe leakage with image size\n- difference between train / test distribution\n- unstable metric, QWK\n\n## DETAIL\n\n### preprocess\nThe most important part of this competition is preprocess. As everyone realized, this competition had a severe leakage with image size and shape. At first I tried many kinds of preprocess (ex. zoom, circle crop or corner cut etc.) to eliminate\nthis leakage, but almost all failed. When I believed train images and test images looked very similar, that makes LB score significantly dropped. Only rotate worked good for me, but it was very unstable.(ex I used 120 degree. If I changed to 90 or 180, LB score significantly dropped.) \nAnd the most interesting image shape was (480,640), which consist of over 70% of test set. That makes me come up with below hypothesis: \nAfter admin made the dataset, they realized there was strong correlation between image size/shape and diagnosis score. They want to prevent all kagglers from submitting LGBM model which use only image meta features. But they also want to the model which can detect retinopathy, so they decided to crop only some part of public test set....\nThat means the image size and shape information is still useful, so I stopped focusing on preprocess. I'm now confident that this thoughts was somehow right after checking private score because it's incredibly higher than previous competition. And you can see that the evidence of this in [this great kernel](https://www.kaggle.com/taindow/be-careful-what-you-train-on)'s private score.\nWhat we used are below. I believe this difference between preprocess might diversify our models.\n\n```python\n#for SeResNext\nCompose([\n BensCrop(img_size), #from 2015 solution\n RandomHorizontalFlip(),\n RandomVerticalFlip(),\n RandomRotation((-120, 120)),\n ),\n ]) \n#for EfficientNet B5\nCompose([\n Resize(img_size),\n HorizontalFlip(),\n VerticalFlip(),\n Rotate(),\n RandomContrast(0.5),\n IAAAdditiveGaussianNoise(p=0.25),\n ])\n```\n\n\n\n### dataset\nThe 2nd important part of our result was how to split and select dataset. I tried many combination, and finally I found below setting works well for me.\n1. split current train dataset to 5 folds by stratified k fold.\n2. train on 4/5 train set and previous competition train set\n3. validate 1/5 train set\n\nWhen I pre-trained on 2015 dataset and fine-tuned on 2019 dataset, my single model(SeResNext)'s score was LB0.811, but after I changed to above method, my score was boost to LB0.827.\n\n\n### model and training method\nWe used 2 kinds of model, SeResNext101 and EfficientNet B5. Each models used a differnt training method.\n\nFor SeResNext101\n- image size:320\n- loss:RMSE\n- train on 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5 of 2019 dataset) \n\nFor EfficientNet B5\n- image size:256 -> 320\n- loss:MSE\n- 2 step training\n 1. pre-train on 4/5folds of 2015 dataset (validate with 2019 dataset) (img_size=256)\n 2. fine-tune on 1/5fold of 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5fold)(img_size=320)\n\n\nBefore starting ensemble, SeResNext scored LB0.827 and EfficientNet scored LB0.830.\n\n\n### pseudo labeling and ensemble\n\nWe used 2 kinds of pseudo labeling.\n1. make soft label for test set and re-train from scratch(ImageNet pre-trained model).\n2. make hard label for test set and fine-tune from trained model\nBoth method boost LB score a lot, so we decided used both method to diversify our models. Sometimes we also use external data (Messidor and IEEE) to give some fresh information to our model, but I'm not sure it's necessary or not.\nBy combining pseudo labeling and ensemble, our both single models reach LB0.846. Below is the result of what we did.(I know it's not beautiful, but we had no time to go back and start from 1st stage model...)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1554318%2F741b1967136cf06b94afaae75fde562f%2F2019-09-07%2011.29.02.png?generation=1567913393476257&alt=media)\n\n(I deleted the row which didn't contribute to our final model)\n\nFinally we submitted below 2 models.\n- No.2+No.5+No.18+No.19(public0.930)\n- No.16(public0.930)\n\n### UPDATED\nHere is my single model's scores that shows how pseudo labeling was effective on private dataset.\nmodel: SeResNext101\n- before applied pseudo labeling: **private LB 0.922**\n- pseudo labeling with only SeResNext101's labels: **private LB 0.925**\n- pseudo labeling with both SeResNext101 and EfficientNet's labels: **private LB 0.931**\nInterestingly last model is better than our final submission and enough to win gold medal :)\n\n\nThank you for reading, please feel free to ask me if you have a question!", "votes": 62 }, { "id": 620917, "postDate": "2019-09-08T04:11:24.470Z", "content": "

Congratulations\nGreat work\nAnd Thanks for sharing your approach and insights.!! @bamps53

", "rawMarkdown": "Congratulations\nGreat work\nAnd Thanks for sharing your approach and insights.!! @bamps53 ", "votes": 1 }, { "id": 620924, "postDate": "2019-09-08T04:21:01.783Z", "content": "

Hey Camaro @bamps53 , @mhiro2 , congratulation!! You deserve this gold.

", "rawMarkdown": "Hey Camaro @bamps53 , @mhiro2 , congratulation!! You deserve this gold.", "votes": 2, "replies": [ { "id": 620930, "postDate": "2019-09-08T04:34:39.637Z", "content": "

Thanks @ratthachat, your excellent and insightful kernels inspired me a lot!!! I learned from you many things...I really admire you!

", "rawMarkdown": "Thanks @ratthachat, your excellent and insightful kernels inspired me a lot!!! I learned from you many things...I really admire you!", "votes": 4 } ] }, { "id": 620908, "postDate": "2019-09-08T04:05:26.037Z", "content": "

Congrats @bamps53 , Thanks for sharing your approach...

", "rawMarkdown": "Congrats @bamps53 , Thanks for sharing your approach...", "votes": 2 }, { "id": 623566, "postDate": "2019-09-11T04:58:40.727Z", "content": "

Great work

", "rawMarkdown": "Great work" }, { "id": 623484, "postDate": "2019-09-11T02:39:44.130Z", "content": "

Beautiful!

", "rawMarkdown": "Beautiful!" }, { "id": 623346, "postDate": "2019-09-10T19:26:54.203Z", "content": "

Congratulations and thanks a lot for sharing your fantastic solution.

\n\n

One question What is exactly your approach for fine-tuning of final 2019-data models after pre-train on 2015-data (few epochs, low learning rates, only unfreeze few layers)?

", "rawMarkdown": "Congratulations and thanks a lot for sharing your fantastic solution. \n\nOne question What is exactly your approach for fine-tuning of final 2019-data models after pre-train on 2015-data (few epochs, low learning rates, only unfreeze few layers)?", "replies": [ { "id": 624383, "postDate": "2019-09-12T03:02:57.620Z", "content": "

@payert For my model(SeResNext), I've never tuned only with 2019 model. I always used 2015 + 2019 train set at the same time. For my teammates model(EfficientNet), they used different strategy but they still doesn't use only 2019 train set (they blend 1/5 of 2015 train set at 2nd stage).

", "rawMarkdown": "@payert For my model(SeResNext), I've never tuned only with 2019 model. I always used 2015 + 2019 train set at the same time. For my teammates model(EfficientNet), they used different strategy but they still doesn't use only 2019 train set (they blend 1/5 of 2015 train set at 2nd stage)." } ] }, { "id": 621491, "postDate": "2019-09-08T15:25:07.530Z", "content": "

Congrates !

\n\n

I agree with you that private test set image sizes does affect the model.
\nI did probing on private test set by following some sizes used in testset. image sizes, \n640 x 480 is about 30-40%, \n2416 x 1736, 819 x 614 , 2048 x 1536 is ranging about 10-20%.
\n2588 x 1958, 1050 x 1050, 2896 x 1944 is ranging 0-10%

\n\n

the retina image appear differently in different sizes (some zoomed with 4 sides cropped, some with top bottom crop) , if training & validation data can pre-process according the distribution on private testset, it should able to predict the private test set well.

", "rawMarkdown": "Congrates !\n\nI agree with you that private test set image sizes does affect the model. \nI did probing on private test set by following some sizes used in testset. image sizes, \n640 x 480 is about 30-40%, \n2416 x 1736, 819 x 614 , 2048 x 1536 is ranging about 10-20%. \n2588 x 1958, 1050 x 1050, 2896 x 1944 is ranging 0-10%\n\nthe retina image appear differently in different sizes (some zoomed with 4 sides cropped, some with top bottom crop) , if training & validation data can pre-process according the distribution on private testset, it should able to predict the private test set well. \n\n\n" }, { "id": 621431, "postDate": "2019-09-08T14:10:24.947Z", "content": "

Congrats for the result!\nYour pseudo label experiments are very interesting.

", "rawMarkdown": "Congrats for the result!\nYour pseudo label experiments are very interesting." }, { "id": 621414, "postDate": "2019-09-08T13:45:46.790Z", "content": "

Congrats @bamps53 , @mhiro2 and thanks for sharing your solution overview.

", "rawMarkdown": "Congrats @bamps53 , @mhiro2 and thanks for sharing your solution overview." }, { "id": 621352, "postDate": "2019-09-08T13:08:26.030Z", "content": "

Congrats @bamps53 , Thanks for sharing your great solution

", "rawMarkdown": "Congrats @bamps53 , Thanks for sharing your great solution " }, { "id": 621284, "postDate": "2019-09-08T12:01:34.060Z", "content": "

Congratulations\nGreat work

", "rawMarkdown": "Congratulations\nGreat work" }, { "id": 621222, "postDate": "2019-09-08T10:44:37.423Z", "content": "

Congratulations and thanks for sharing :)

", "rawMarkdown": "Congratulations and thanks for sharing :)" }, { "id": 620926, "postDate": "2019-09-08T04:23:37.110Z", "content": "

Congrats and thanks for sharing! Would you mind let me know how much boost is brought by pseudo labeling on private LB for a single model?

", "rawMarkdown": "Congrats and thanks for sharing! Would you mind let me know how much boost is brought by pseudo labeling on private LB for a single model? ", "replies": [ { "id": 620938, "postDate": "2019-09-08T04:44:32.547Z", "content": "

Honestly I'm not sure for now because I used a script which can get only public score....\nBut I'm also interested in, so I'll make late submission and check it.\nPlease give me for a while, hopefully I'll update tomorrow!!

", "rawMarkdown": "Honestly I'm not sure for now because I used a script which can get only public score....\nBut I'm also interested in, so I'll make late submission and check it.\nPlease give me for a while, hopefully I'll update tomorrow!!", "votes": 1 }, { "id": 620942, "postDate": "2019-09-08T04:55:50.457Z", "content": "

Thanks! \nHaha, I think most people use the 'fast-submit' way until the last weeks, which is kinda of bad to check the model performance afterwards.

", "rawMarkdown": "Thanks! \nHaha, I think most people use the 'fast-submit' way until the last weeks, which is kinda of bad to check the model performance afterwards. " }, { "id": 621525, "postDate": "2019-09-08T16:11:26.660Z", "content": "

Haha yeah, I have no references for our single models on private LB :)

", "rawMarkdown": "Haha yeah, I have no references for our single models on private LB :)" }, { "id": 622244, "postDate": "2019-09-09T12:28:25.660Z", "content": "

I updated my single model's score. It boosted from private LB 0.922 to 0.931!!

", "rawMarkdown": "I updated my single model's score. It boosted from private LB 0.922 to 0.931!!" } ] }, { "id": 620925, "postDate": "2019-09-08T04:22:55.910Z", "content": "

i am rookie so i want to know what‘mean soft label and hard label?can you tell me? thank you

", "rawMarkdown": "i am rookie so i want to know what‘mean soft label and hard label?can you tell me? thank you ", "replies": [ { "id": 620933, "postDate": "2019-09-08T04:37:34.407Z", "content": "

I meant that soft label is model output before using optimizer, which has continuous values. And hard label is after using optimizer, which has discrete class labels.

", "rawMarkdown": "I meant that soft label is model output before using optimizer, which has continuous values. And hard label is after using optimizer, which has discrete class labels.", "votes": 3 } ] }, { "id": 646939, "postDate": "2019-10-11T22:44:12.200Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 620917, "author_name": "Ailurophile", "author_url": "", "post_date": "2019-09-08T04:11:24.470000", "content": "

Congratulations\nGreat work\nAnd Thanks for sharing your approach and insights.!! @bamps53

", "votes": 1, "replies": [] }, { "id": 620924, "author_name": "Neuron Engineer", "author_url": "", "post_date": "2019-09-08T04:21:01.783000", "content": "

Hey Camaro @bamps53 , @mhiro2 , congratulation!! You deserve this gold.

", "votes": 2, "replies": [ { "id": 620930, "author_name": "Camaro", "author_url": "", "post_date": "2019-09-08T04:34:39.637000", "content": "

Thanks @ratthachat, your excellent and insightful kernels inspired me a lot!!! I learned from you many things...I really admire you!

", "votes": 4, "replies": [] } ] }, { "id": 620908, "author_name": "Kranthi Kumar", "author_url": "", "post_date": "2019-09-08T04:05:26.037000", "content": "

Congrats @bamps53 , Thanks for sharing your approach...

", "votes": 2, "replies": [] }, { "id": 623566, "author_name": "Gurjyot Singh", "author_url": "", "post_date": "2019-09-11T04:58:40.727000", "content": "

Great work

", "votes": 0, "replies": [] }, { "id": 623484, "author_name": "Hilal Shaath", "author_url": "", "post_date": "2019-09-11T02:39:44.130000", "content": "

Beautiful!

", "votes": 0, "replies": [] }, { "id": 623346, "author_name": "TPayer", "author_url": "", "post_date": "2019-09-10T19:26:54.203000", "content": "

Congratulations and thanks a lot for sharing your fantastic solution.

\n\n

One question What is exactly your approach for fine-tuning of final 2019-data models after pre-train on 2015-data (few epochs, low learning rates, only unfreeze few layers)?

", "votes": 0, "replies": [ { "id": 624383, "author_name": "Camaro", "author_url": "", "post_date": "2019-09-12T03:02:57.620000", "content": "

@payert For my model(SeResNext), I've never tuned only with 2019 model. I always used 2015 + 2019 train set at the same time. For my teammates model(EfficientNet), they used different strategy but they still doesn't use only 2019 train set (they blend 1/5 of 2015 train set at 2nd stage).

", "votes": 0, "replies": [] } ] }, { "id": 621491, "author_name": "CC Joshua ", "author_url": "", "post_date": "2019-09-08T15:25:07.530000", "content": "

Congrates !

\n\n

I agree with you that private test set image sizes does affect the model.
\nI did probing on private test set by following some sizes used in testset. image sizes, \n640 x 480 is about 30-40%, \n2416 x 1736, 819 x 614 , 2048 x 1536 is ranging about 10-20%.
\n2588 x 1958, 1050 x 1050, 2896 x 1944 is ranging 0-10%

\n\n

the retina image appear differently in different sizes (some zoomed with 4 sides cropped, some with top bottom crop) , if training & validation data can pre-process according the distribution on private testset, it should able to predict the private test set well.

", "votes": 0, "replies": [] }, { "id": 621431, "author_name": "Qishen Ha", "author_url": "", "post_date": "2019-09-08T14:10:24.947000", "content": "

Congrats for the result!\nYour pseudo label experiments are very interesting.

", "votes": 0, "replies": [] }, { "id": 621414, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-09-08T13:45:46.790000", "content": "

Congrats @bamps53 , @mhiro2 and thanks for sharing your solution overview.

", "votes": 0, "replies": [] }, { "id": 621352, "author_name": "KhanhVD", "author_url": "", "post_date": "2019-09-08T13:08:26.030000", "content": "

Congrats @bamps53 , Thanks for sharing your great solution

", "votes": 0, "replies": [] }, { "id": 621284, "author_name": "JM100", "author_url": "", "post_date": "2019-09-08T12:01:34.060000", "content": "

Congratulations\nGreat work

", "votes": 0, "replies": [] }, { "id": 621222, "author_name": "Noah Weber", "author_url": "", "post_date": "2019-09-08T10:44:37.423000", "content": "

Congratulations and thanks for sharing :)

", "votes": 0, "replies": [] }, { "id": 620926, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-09-08T04:23:37.110000", "content": "

Congrats and thanks for sharing! Would you mind let me know how much boost is brought by pseudo labeling on private LB for a single model?

", "votes": 0, "replies": [ { "id": 620938, "author_name": "Camaro", "author_url": "", "post_date": "2019-09-08T04:44:32.547000", "content": "

Honestly I'm not sure for now because I used a script which can get only public score....\nBut I'm also interested in, so I'll make late submission and check it.\nPlease give me for a while, hopefully I'll update tomorrow!!

", "votes": 1, "replies": [] }, { "id": 620942, "author_name": "Xuan Cao", "author_url": "", "post_date": "2019-09-08T04:55:50.457000", "content": "

Thanks! \nHaha, I think most people use the 'fast-submit' way until the last weeks, which is kinda of bad to check the model performance afterwards.

", "votes": 0, "replies": [] }, { "id": 621525, "author_name": "Psi", "author_url": "", "post_date": "2019-09-08T16:11:26.660000", "content": "

Haha yeah, I have no references for our single models on private LB :)

", "votes": 0, "replies": [] }, { "id": 622244, "author_name": "Camaro", "author_url": "", "post_date": "2019-09-09T12:28:25.660000", "content": "

I updated my single model's score. It boosted from private LB 0.922 to 0.931!!

", "votes": 0, "replies": [] } ] }, { "id": 620925, "author_name": "HeVi27", "author_url": "", "post_date": "2019-09-08T04:22:55.910000", "content": "

i am rookie so i want to know what‘mean soft label and hard label?can you tell me? thank you

", "votes": 0, "replies": [ { "id": 620933, "author_name": "Camaro", "author_url": "", "post_date": "2019-09-08T04:37:34.407000", "content": "

I meant that soft label is model output before using optimizer, which has continuous values. And hard label is after using optimizer, which has discrete class labels.

", "votes": 3, "replies": [] } ] }, { "id": 646939, "author_name": "", "author_url": "", "post_date": "2019-10-11T22:44:12.200000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "620905": "Congrats all the prize winner and everyone finished in gold zone! And I really want to say thank you to my great teammate @mhiro2 and @yasufuminakama, I'm pretty confident that I couldn't get gold medal if I'm alone...\n\nThis is my first time to write solution, but I'll try to open up what we did as much as possible. \n\n## OVERVIEW\n- ensemble 4 models, 2 SeResNext101 and 2 EfficientNet B5\n- 2 kinds of pseudo labeling\n- use old competition data\n\n## CHALENGE\n- severe leakage with image size\n- difference between train / test distribution\n- unstable metric, QWK\n\n## DETAIL\n\n### preprocess\nThe most important part of this competition is preprocess. As everyone realized, this competition had a severe leakage with image size and shape. At first I tried many kinds of preprocess (ex. zoom, circle crop or corner cut etc.) to eliminate\nthis leakage, but almost all failed. When I believed train images and test images looked very similar, that makes LB score significantly dropped. Only rotate worked good for me, but it was very unstable.(ex I used 120 degree. If I changed to 90 or 180, LB score significantly dropped.) \nAnd the most interesting image shape was (480,640), which consist of over 70% of test set. That makes me come up with below hypothesis: \nAfter admin made the dataset, they realized there was strong correlation between image size/shape and diagnosis score. They want to prevent all kagglers from submitting LGBM model which use only image meta features. But they also want to the model which can detect retinopathy, so they decided to crop only some part of public test set....\nThat means the image size and shape information is still useful, so I stopped focusing on preprocess. I'm now confident that this thoughts was somehow right after checking private score because it's incredibly higher than previous competition. And you can see that the evidence of this in [this great kernel](https://www.kaggle.com/taindow/be-careful-what-you-train-on)'s private score.\nWhat we used are below. I believe this difference between preprocess might diversify our models.\n\n```python\n#for SeResNext\nCompose([\n BensCrop(img_size), #from 2015 solution\n RandomHorizontalFlip(),\n RandomVerticalFlip(),\n RandomRotation((-120, 120)),\n ),\n ]) \n#for EfficientNet B5\nCompose([\n Resize(img_size),\n HorizontalFlip(),\n VerticalFlip(),\n Rotate(),\n RandomContrast(0.5),\n IAAAdditiveGaussianNoise(p=0.25),\n ])\n```\n\n\n\n### dataset\nThe 2nd important part of our result was how to split and select dataset. I tried many combination, and finally I found below setting works well for me.\n1. split current train dataset to 5 folds by stratified k fold.\n2. train on 4/5 train set and previous competition train set\n3. validate 1/5 train set\n\nWhen I pre-trained on 2015 dataset and fine-tuned on 2019 dataset, my single model(SeResNext)'s score was LB0.811, but after I changed to above method, my score was boost to LB0.827.\n\n\n### model and training method\nWe used 2 kinds of model, SeResNext101 and EfficientNet B5. Each models used a differnt training method.\n\nFor SeResNext101\n- image size:320\n- loss:RMSE\n- train on 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5 of 2019 dataset) \n\nFor EfficientNet B5\n- image size:256 -> 320\n- loss:MSE\n- 2 step training\n 1. pre-train on 4/5folds of 2015 dataset (validate with 2019 dataset) (img_size=256)\n 2. fine-tune on 1/5fold of 2015 dataset + 4/5folds of 2019 dataset (validate with 1/5fold)(img_size=320)\n\n\nBefore starting ensemble, SeResNext scored LB0.827 and EfficientNet scored LB0.830.\n\n\n### pseudo labeling and ensemble\n\nWe used 2 kinds of pseudo labeling.\n1. make soft label for test set and re-train from scratch(ImageNet pre-trained model).\n2. make hard label for test set and fine-tune from trained model\nBoth method boost LB score a lot, so we decided used both method to diversify our models. Sometimes we also use external data (Messidor and IEEE) to give some fresh information to our model, but I'm not sure it's necessary or not.\nBy combining pseudo labeling and ensemble, our both single models reach LB0.846. Below is the result of what we did.(I know it's not beautiful, but we had no time to go back and start from 1st stage model...)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1554318%2F741b1967136cf06b94afaae75fde562f%2F2019-09-07%2011.29.02.png?generation=1567913393476257&alt=media)\n\n(I deleted the row which didn't contribute to our final model)\n\nFinally we submitted below 2 models.\n- No.2+No.5+No.18+No.19(public0.930)\n- No.16(public0.930)\n\n### UPDATED\nHere is my single model's scores that shows how pseudo labeling was effective on private dataset.\nmodel: SeResNext101\n- before applied pseudo labeling: **private LB 0.922**\n- pseudo labeling with only SeResNext101's labels: **private LB 0.925**\n- pseudo labeling with both SeResNext101 and EfficientNet's labels: **private LB 0.931**\nInterestingly last model is better than our final submission and enough to win gold medal :)\n\n\nThank you for reading, please feel free to ask me if you have a question!", "620917": "Congratulations\nGreat work\nAnd Thanks for sharing your approach and insights.!! @bamps53 ", "620924": "Hey Camaro @bamps53 , @mhiro2 , congratulation!! You deserve this gold.", "620908": "Congrats @bamps53 , Thanks for sharing your approach...", "623566": "Great work", "623484": "Beautiful!", "623346": "Congratulations and thanks a lot for sharing your fantastic solution. \n\nOne question What is exactly your approach for fine-tuning of final 2019-data models after pre-train on 2015-data (few epochs, low learning rates, only unfreeze few layers)?", "621491": "Congrates !\n\nI agree with you that private test set image sizes does affect the model. \nI did probing on private test set by following some sizes used in testset. image sizes, \n640 x 480 is about 30-40%, \n2416 x 1736, 819 x 614 , 2048 x 1536 is ranging about 10-20%. \n2588 x 1958, 1050 x 1050, 2896 x 1944 is ranging 0-10%\n\nthe retina image appear differently in different sizes (some zoomed with 4 sides cropped, some with top bottom crop) , if training & validation data can pre-process according the distribution on private testset, it should able to predict the private test set well. \n\n\n", "621431": "Congrats for the result!\nYour pseudo label experiments are very interesting.", "621414": "Congrats @bamps53 , @mhiro2 and thanks for sharing your solution overview.", "621352": "Congrats @bamps53 , Thanks for sharing your great solution ", "621284": "Congratulations\nGreat work", "621222": "Congratulations and thanks for sharing :)", "620926": "Congrats and thanks for sharing! Would you mind let me know how much boost is brought by pseudo labeling on private LB for a single model? ", "620925": "i am rookie so i want to know what‘mean soft label and hard label?can you tell me? thank you ", "646939": "" } }