Hi everyone thanks for this competition. This is my first jouney of computer vision.\nI tried every solution from discussions but my best LB score is only 0.787。。。\nMy methods will be:\n- efficientnetb5 with size 456\n- use data of Resized 2015 & 2019 @benjaminwarner (resized and crop)\n- train 2015 data and validate 2019 train-data for 4 epochs then finetune valid part of 2019 data for 7 epochs\n- lr2e-4 no schedule with Adamwarmup 0.05 and no wd\n- output layer with avg and maxpool and dropout0.3\n- optimized kappa, \n- seed42, \n- 0 label sample weights 0.125, 1 label 0.5, 2 label 0.25, 3 & 4 label 1\n- autoaugment and cutout and rotate flip\nWith this methods to get LB 0.787, I can have CV0.9091 at 2019 train-data validation and CV0.9586 for finetuning
\n\nAnd then I tried other methods, none of them works:\n- train 2015 data and validate 2019 train-data for 3 epochs (because 0.9091 was observed at epoch 3)\n- change sample weights to 100% fit the train-data 2019 distribution\n- try preprocessing of crop_image_from_gray and circle_crop\n- try combination of preprocessing from @https://www.kaggle.com/haydenmuscat/full-eye-cropping-and-a-true-colour-ben-s-algo\n(totally download preprocess and upload dataset cost me most of the time, especially 2015 raw data are compressed in a messy way)\n- classification\n- multilabel (mse+bce in classification)
\n\nI remember someone on top rank said: for train 2015 and valid 2019 , CV can reach 0.93. \nIf you do it in this way, what is your CV score?\nIf anyone has advice, please don't hesitate to write your comments.\nThank you all.\n(Raw 2015 dataset without any preprocessing: \nhttps://www.kaggle.com/httpwwwfszyc/rsra15train1\nhttps://www.kaggle.com/httpwwwfszyc/rsra15train2\nhttps://www.kaggle.com/httpwwwfszyc/rsra15\nhttps://www.kaggle.com/httpwwwfszyc/rsra15test1\nhttps://www.kaggle.com/httpwwwfszyc/rsra15test2)
", "messages": [ { "id": "620371", "postDate": "09/07/2019 11:53:16", "content": "Hi everyone thanks for this competition. This is my first jouney of computer vision.\nI tried every solution from discussions but my best LB score is only 0.787。。。\nMy methods will be:\n- efficientnetb5 with size 456\n- use data of Resized 2015 & 2019 @benjaminwarner (resized and crop)\n- train 2015 data and validate 2019 train-data for 4 epochs then finetune valid part of 2019 data for 7 epochs\n- lr2e-4 no schedule with Adamwarmup 0.05 and no wd\n- output layer with avg and maxpool and dropout0.3\n- optimized kappa, \n- seed42, \n- 0 label sample weights 0.125, 1 label 0.5, 2 label 0.25, 3 & 4 label 1\n- autoaugment and cutout and rotate flip\nWith this methods to get LB 0.787, I can have CV0.9091 at 2019 train-data validation and CV0.9586 for finetuning
\n\nAnd then I tried other methods, none of them works:\n- train 2015 data and validate 2019 train-data for 3 epochs (because 0.9091 was observed at epoch 3)\n- change sample weights to 100% fit the train-data 2019 distribution\n- try preprocessing of crop_image_from_gray and circle_crop\n- try combination of preprocessing from @https://www.kaggle.com/haydenmuscat/full-eye-cropping-and-a-true-colour-ben-s-algo\n(totally download preprocess and upload dataset cost me most of the time, especially 2015 raw data are compressed in a messy way)\n- classification\n- multilabel (mse+bce in classification)
\n\nI remember someone on top rank said: for train 2015 and valid 2019 , CV can reach 0.93. \nIf you do it in this way, what is your CV score?\nIf anyone has advice, please don't hesitate to write your comments.\nThank you all.\n(Raw 2015 dataset without any preprocessing: \nhttps://www.kaggle.com/httpwwwfszyc/rsra15train1\nhttps://www.kaggle.com/httpwwwfszyc/rsra15train2\nhttps://www.kaggle.com/httpwwwfszyc/rsra15\nhttps://www.kaggle.com/httpwwwfszyc/rsra15test1\nhttps://www.kaggle.com/httpwwwfszyc/rsra15test2)
", "rawMarkdown": "Hi everyone thanks for this competition. This is my first jouney of computer vision.\nI tried every solution from discussions but my best LB score is only 0.787。。。\nMy methods will be:\n- efficientnetb5 with size 456\n- use data of Resized 2015 & 2019 @benjaminwarner (resized and crop)\n- train 2015 data and validate 2019 train-data for 4 epochs then finetune valid part of 2019 data for 7 epochs\n- lr2e-4 no schedule with Adamwarmup 0.05 and no wd\n- output layer with avg and maxpool and dropout0.3\n- optimized kappa, \n- seed42, \n- 0 label sample weights 0.125, 1 label 0.5, 2 label 0.25, 3 & 4 label 1\n- autoaugment and cutout and rotate flip\nWith this methods to get LB 0.787, I can have CV0.9091 at 2019 train-data validation and CV0.9586 for finetuning\n\nAnd then I tried other methods, none of them works:\n- train 2015 data and validate 2019 train-data for 3 epochs (because 0.9091 was observed at epoch 3)\n- change sample weights to 100% fit the train-data 2019 distribution\n- try preprocessing of crop_image_from_gray and circle_crop\n- try combination of preprocessing from @https://www.kaggle.com/haydenmuscat/full-eye-cropping-and-a-true-colour-ben-s-algo\n(totally download preprocess and upload dataset cost me most of the time, especially 2015 raw data are compressed in a messy way)\n- classification\n- multilabel (mse+bce in classification)\n\nI remember someone on top rank said: for train 2015 and valid 2019 , CV can reach 0.93. \nIf you do it in this way, what is your CV score?\nIf anyone has advice, please don't hesitate to write your comments.\nThank you all.\n(Raw 2015 dataset without any preprocessing: \nhttps://www.kaggle.com/httpwwwfszyc/rsra15train1\nhttps://www.kaggle.com/httpwwwfszyc/rsra15train2\nhttps://www.kaggle.com/httpwwwfszyc/rsra15\nhttps://www.kaggle.com/httpwwwfszyc/rsra15test1\nhttps://www.kaggle.com/httpwwwfszyc/rsra15test2)", "votes": null }, { "id": "620407", "postDate": "09/07/2019 13:12:28", "content": "Just directly do ensemble, at least for me I can't even tell if our method to use 2015 data and 2019 data is correct or not. But ensemble will make your solution more stable. In my experiments, I don't think many of my models that could get score higher than 0.81 will be better than some of my models only get around 0.79. QWK is not stable, public test set is very weird at least very weird distribution, two submissions that with very different distribution could get same public score. So somehow forget public lb unless you can explain it.
", "rawMarkdown": "Just directly do ensemble, at least for me I can't even tell if our method to use 2015 data and 2019 data is correct or not. But ensemble will make your solution more stable. In my experiments, I don't think many of my models that could get score higher than 0.81 will be better than some of my models only get around 0.79. QWK is not stable, public test set is very weird at least very weird distribution, two submissions that with very different distribution could get same public score. So somehow forget public lb unless you can explain it.", "votes": null }, { "id": "620411", "postDate": "09/07/2019 13:19:29", "content": "Thanks for your advice, well i tried ensembling once but the CV get worse... maybe because the qwk is not stable. I will try ensemble other models
", "rawMarkdown": "Thanks for your advice, well i tried ensembling once but the CV get worse... maybe because the qwk is not stable. I will try ensemble other models", "votes": null }, { "id": "620462", "postDate": "09/07/2019 14:25:59", "content": "Just make full use of all your models, and good luck!
", "rawMarkdown": "Just make full use of all your models, and good luck!", "votes": null }, { "id": "620520", "postDate": "09/07/2019 16:01:16", "content": "I used efficient-net b5 with image size 300. first I trained on previous data for 15 epochs and used current data as the validation set. Then I trained the b5 model on the current dataset. I trained the model as a regression task. I used OptimizedRounder technique by Abhishek. I got .806 LB score. I trained b4 using the same setting and use ensembling. I was able to increase score a little bit. I have removed the black area around images while training models. I chose the best model by monitoring validation loss. My CV score was .92.
", "rawMarkdown": "I used efficient-net b5 with image size 300. first I trained on previous data for 15 epochs and used current data as the validation set. Then I trained the b5 model on the current dataset. I trained the model as a regression task. I used [OptimizedRounder](https://www.kaggle.com/abhishek/optimizer-for-quadratic-weighted-kappa) technique by Abhishek. I got .806 LB score. I trained b4 using the same setting and use ensembling. I was able to increase score a little bit. I have removed the black area around images while training models. I chose the best model by monitoring validation loss. My CV score was .92.", "votes": null }, { "id": "620522", "postDate": "09/07/2019 16:03:40", "content": "oh this is good. so how is your CV score on current data when training on 15 data? and is your \"remove black area\" means your image is circle shape?
", "rawMarkdown": "oh this is good. so how is your CV score on current data when training on 15 data? and is your \"remove black area\" means your image is circle shape?", "votes": null }, { "id": "620532", "postDate": "09/07/2019 16:11:23", "content": "My CV score was around .65 when training on 15 data using 19 data as validation. but matric was validation loss. while training on 19 data my matric was cohen kappa score and CV score was .92.
", "rawMarkdown": "My CV score was around .65 when training on 15 data using 19 data as validation. but matric was validation loss. while training on 19 data my matric was cohen kappa score and CV score was .92.", "votes": null }, { "id": "620539", "postDate": "09/07/2019 16:19:27", "content": "matric was validation loss? You mean accuracy?
", "rawMarkdown": "matric was validation loss? You mean accuracy?", "votes": null }, { "id": "620541", "postDate": "09/07/2019 16:22:35", "content": "yes, accuracy.
", "rawMarkdown": "yes, accuracy.", "votes": null }, { "id": "620671", "postDate": "09/07/2019 20:12:04", "content": "@httpwwwfszyc Have you tried ensembling different models?
", "rawMarkdown": "httpwwwfszyc Have you tried ensembling different models?", "votes": null }, { "id": "620672", "postDate": "09/07/2019 20:13:24", "content": "try my best b5 model and a b2 model, but CV and LB both decrease
", "rawMarkdown": "try my best b5 model and a b2 model, but CV and LB both decrease", "votes": null }, { "id": "620675", "postDate": "09/07/2019 20:18:38", "content": "I can say (this was said already in many threads) that just by using TTA in my models and then blending them helped us increase our score. I guess we need to wait and see what really worked :)
", "rawMarkdown": "I can say (this was said already in many threads) that just by using TTA in my models and then blending them helped us increase our score. I guess we need to wait and see what really worked :)", "votes": null }, { "id": "620676", "postDate": "09/07/2019 20:22:41", "content": "yeap.
", "rawMarkdown": "yeap.", "votes": null } ], "comments": [ { "id": 620407, "author_name": "jionie", "author_url": "", "post_date": "09/07/2019 13:12:28", "content": "Just directly do ensemble, at least for me I can't even tell if our method to use 2015 data and 2019 data is correct or not. But ensemble will make your solution more stable. In my experiments, I don't think many of my models that could get score higher than 0.81 will be better than some of my models only get around 0.79. QWK is not stable, public test set is very weird at least very weird distribution, two submissions that with very different distribution could get same public score. So somehow forget public lb unless you can explain it.
", "votes": null, "replies": [ { "id": 620411, "author_name": "httpwwwfszyc", "author_url": "", "post_date": "09/07/2019 13:19:29", "content": "Thanks for your advice, well i tried ensembling once but the CV get worse... maybe because the qwk is not stable. I will try ensemble other models
", "votes": null, "replies": [] }, { "id": 620462, "author_name": "jionie", "author_url": "", "post_date": "09/07/2019 14:25:59", "content": "Just make full use of all your models, and good luck!
", "votes": null, "replies": [] } ] }, { "id": 620520, "author_name": "neeraj17", "author_url": "", "post_date": "09/07/2019 16:01:16", "content": "I used efficient-net b5 with image size 300. first I trained on previous data for 15 epochs and used current data as the validation set. Then I trained the b5 model on the current dataset. I trained the model as a regression task. I used OptimizedRounder technique by Abhishek. I got .806 LB score. I trained b4 using the same setting and use ensembling. I was able to increase score a little bit. I have removed the black area around images while training models. I chose the best model by monitoring validation loss. My CV score was .92.
", "votes": null, "replies": [ { "id": 620522, "author_name": "httpwwwfszyc", "author_url": "", "post_date": "09/07/2019 16:03:40", "content": "oh this is good. so how is your CV score on current data when training on 15 data? and is your \"remove black area\" means your image is circle shape?
", "votes": null, "replies": [] }, { "id": 620532, "author_name": "neeraj17", "author_url": "", "post_date": "09/07/2019 16:11:23", "content": "My CV score was around .65 when training on 15 data using 19 data as validation. but matric was validation loss. while training on 19 data my matric was cohen kappa score and CV score was .92.
", "votes": null, "replies": [] }, { "id": 620539, "author_name": "httpwwwfszyc", "author_url": "", "post_date": "09/07/2019 16:19:27", "content": "matric was validation loss? You mean accuracy?
", "votes": null, "replies": [] }, { "id": 620541, "author_name": "neeraj17", "author_url": "", "post_date": "09/07/2019 16:22:35", "content": "yes, accuracy.
", "votes": null, "replies": [] } ] }, { "id": 620671, "author_name": "raimonds1993", "author_url": "", "post_date": "09/07/2019 20:12:04", "content": "@httpwwwfszyc Have you tried ensembling different models?
", "votes": null, "replies": [ { "id": 620672, "author_name": "httpwwwfszyc", "author_url": "", "post_date": "09/07/2019 20:13:24", "content": "try my best b5 model and a b2 model, but CV and LB both decrease
", "votes": null, "replies": [] }, { "id": 620675, "author_name": "raimonds1993", "author_url": "", "post_date": "09/07/2019 20:18:38", "content": "I can say (this was said already in many threads) that just by using TTA in my models and then blending them helped us increase our score. I guess we need to wait and see what really worked :)
", "votes": null, "replies": [] }, { "id": 620676, "author_name": "httpwwwfszyc", "author_url": "", "post_date": "09/07/2019 20:22:41", "content": "yeap.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "620371": "Hi everyone thanks for this competition. This is my first jouney of computer vision.\nI tried every solution from discussions but my best LB score is only 0.787。。。\nMy methods will be:\n- efficientnetb5 with size 456\n- use data of Resized 2015 & 2019 @benjaminwarner (resized and crop)\n- train 2015 data and validate 2019 train-data for 4 epochs then finetune valid part of 2019 data for 7 epochs\n- lr2e-4 no schedule with Adamwarmup 0.05 and no wd\n- output layer with avg and maxpool and dropout0.3\n- optimized kappa, \n- seed42, \n- 0 label sample weights 0.125, 1 label 0.5, 2 label 0.25, 3 & 4 label 1\n- autoaugment and cutout and rotate flip\nWith this methods to get LB 0.787, I can have CV0.9091 at 2019 train-data validation and CV0.9586 for finetuning\n\nAnd then I tried other methods, none of them works:\n- train 2015 data and validate 2019 train-data for 3 epochs (because 0.9091 was observed at epoch 3)\n- change sample weights to 100% fit the train-data 2019 distribution\n- try preprocessing of crop_image_from_gray and circle_crop\n- try combination of preprocessing from @https://www.kaggle.com/haydenmuscat/full-eye-cropping-and-a-true-colour-ben-s-algo\n(totally download preprocess and upload dataset cost me most of the time, especially 2015 raw data are compressed in a messy way)\n- classification\n- multilabel (mse+bce in classification)\n\nI remember someone on top rank said: for train 2015 and valid 2019 , CV can reach 0.93. \nIf you do it in this way, what is your CV score?\nIf anyone has advice, please don't hesitate to write your comments.\nThank you all.\n(Raw 2015 dataset without any preprocessing: \nhttps://www.kaggle.com/httpwwwfszyc/rsra15train1\nhttps://www.kaggle.com/httpwwwfszyc/rsra15train2\nhttps://www.kaggle.com/httpwwwfszyc/rsra15\nhttps://www.kaggle.com/httpwwwfszyc/rsra15test1\nhttps://www.kaggle.com/httpwwwfszyc/rsra15test2)", "620407": "Just directly do ensemble, at least for me I can't even tell if our method to use 2015 data and 2019 data is correct or not. But ensemble will make your solution more stable. In my experiments, I don't think many of my models that could get score higher than 0.81 will be better than some of my models only get around 0.79. QWK is not stable, public test set is very weird at least very weird distribution, two submissions that with very different distribution could get same public score. So somehow forget public lb unless you can explain it.", "620411": "Thanks for your advice, well i tried ensembling once but the CV get worse... maybe because the qwk is not stable. I will try ensemble other models", "620462": "Just make full use of all your models, and good luck!", "620520": "I used efficient-net b5 with image size 300. first I trained on previous data for 15 epochs and used current data as the validation set. Then I trained the b5 model on the current dataset. I trained the model as a regression task. I used [OptimizedRounder](https://www.kaggle.com/abhishek/optimizer-for-quadratic-weighted-kappa) technique by Abhishek. I got .806 LB score. I trained b4 using the same setting and use ensembling. I was able to increase score a little bit. I have removed the black area around images while training models. I chose the best model by monitoring validation loss. My CV score was .92.", "620522": "oh this is good. so how is your CV score on current data when training on 15 data? and is your \"remove black area\" means your image is circle shape?", "620532": "My CV score was around .65 when training on 15 data using 19 data as validation. but matric was validation loss. while training on 19 data my matric was cohen kappa score and CV score was .92.", "620539": "matric was validation loss? You mean accuracy?", "620541": "yes, accuracy.", "620671": "httpwwwfszyc Have you tried ensembling different models?", "620672": "try my best b5 model and a b2 model, but CV and LB both decrease", "620675": "I can say (this was said already in many threads) that just by using TTA in my models and then blending them helped us increase our score. I guess we need to wait and see what really worked :)", "620676": "yeap." }, "source": "meta" }