{ "id": 205913, "title": "Weird improvement trick [Can someone explain the reason?]", "url": "/competitions/cassava-leaf-disease-classification/discussion/205913", "author_name": "", "post_date": "2020-12-22T13:09:18.036640800Z", "votes": 7, "comment_count": 10, "views": 0, "content": "

I tried training ResNet-18 on stratified 5 folds with an image size of 448 x 448. When I calculate the OOF score using all 5 models with the image size 448 x 448 (same as used in training) it gives an OOF score of 0.885, now when I calculate the OOF score using image size 512 x 512, surprisingly it improves to 0.887.

\n

This also improves my public LB score. I came to know this by mistake and don't know the possible reason behind this.

", "messages": [ { "id": "1122469", "postDate": "12/22/2020 13:09:18", "content": "

I tried training ResNet-18 on stratified 5 folds with an image size of 448 x 448. When I calculate the OOF score using all 5 models with the image size 448 x 448 (same as used in training) it gives an OOF score of 0.885, now when I calculate the OOF score using image size 512 x 512, surprisingly it improves to 0.887.

\n

This also improves my public LB score. I came to know this by mistake and don't know the possible reason behind this.

", "rawMarkdown": "I tried training ResNet-18 on stratified 5 folds with an image size of `448 x 448`. When I calculate the OOF score using all 5 models with the image size `448 x 448` (same as used in training) it gives an OOF score of 0.885, now when I calculate the OOF score using image size `512 x 512`, surprisingly it improves to 0.887.\n\nThis also improves my public LB score. I came to know this by mistake and don't know the possible reason behind this.", "votes": null }, { "id": "1122546", "postDate": "12/22/2020 14:15:55", "content": "

Why do you think that's weird? By being able to look at more of the picture at once or the same area on higher resolution, you are using slightly more information. Thanks to the pretaining on ImageNet and the decent amount of training data we have here, it is not so surprising that the model can cope with that extra information.

", "rawMarkdown": "Why do you think that's weird? By being able to look at more of the picture at once or the same area on higher resolution, you are using slightly more information. Thanks to the pretaining on ImageNet and the decent amount of training data we have here, it is not so surprising that the model can cope with that extra information.", "votes": null }, { "id": "1122617", "postDate": "12/22/2020 15:12:51", "content": "

you should read this paper: https://arxiv.org/pdf/1906.06423.pdf
\nIf you are using RandomResized crop as an augmentation its probably why! It explains how you can test on higher image resolution

", "rawMarkdown": "you should read this paper: https://arxiv.org/pdf/1906.06423.pdf\nIf you are using RandomResized crop as an augmentation its probably why! It explains how you can test on higher image resolution", "votes": null }, { "id": "1122656", "postDate": "12/22/2020 15:37:14", "content": "

@yannmajewski Thanks for pointing it out. Yes I'm using random resized crop.

", "rawMarkdown": "yannmajewski Thanks for pointing it out. Yes I'm using random resized crop.", "votes": null }, { "id": "1122664", "postDate": "12/22/2020 15:41:48", "content": "

This might be why! Have a good read :)

", "rawMarkdown": "This might be why! Have a good read :)", "votes": null }, { "id": "1123213", "postDate": "12/23/2020 03:28:31", "content": "

For this dataset, the difference between 0.885 and 0.887 does not seem to be statistically significant. Most probably, it is just noise.

", "rawMarkdown": "For this dataset, the difference between 0.885 and 0.887 does not seem to be statistically significant. Most probably, it is just noise.", "votes": null }, { "id": "1124835", "postDate": "12/24/2020 08:16:10", "content": "

I would agree with @graf10a . If you have time - try other resolutions near (but below) 448x448 to see what you get.

\n

And if you are relying on the random seed to keep things consistent across experiments, I'm not sure it would work in cases where augmentations depend on the image size. The way a random sized crop is generated might look different with the same seed but different image size.

", "rawMarkdown": "I would agree with @graf10a . If you have time - try other resolutions near (but below) 448x448 to see what you get.\n\nAnd if you are relying on the random seed to keep things consistent across experiments, I'm not sure it would work in cases where augmentations depend on the image size. The way a random sized crop is generated might look different with the same seed but different image size.", "votes": null }, { "id": "1126291", "postDate": "12/25/2020 13:46:48", "content": "

@graf10a Yes good point but this improved the scores consistently. For example 0.892 to 0.896.

", "rawMarkdown": "graf10a Yes good point but this improved the scores consistently. For example 0.892 to 0.896.", "votes": null }, { "id": "1126491", "postDate": "12/25/2020 16:45:01", "content": "

@alexandersoare Good idea. I tried increasing image size while inferencing on the same model (trained on 448x448) to 600x600 and it didn't boost my score further.

", "rawMarkdown": "alexandersoare Good idea. I tried increasing image size while inferencing on the same model (trained on 448x448) to 600x600 and it didn't boost my score further.", "votes": null }, { "id": "1126512", "postDate": "12/25/2020 17:06:28", "content": "

Ahh I just realised you weren't talking about retraining on 512x512. Yep I've tried this trick before. Never got anything I could be sure of though. PS check discord

", "rawMarkdown": "Ahh I just realised you weren't talking about retraining on 512x512. Yep I've tried this trick before. Never got anything I could be sure of though. PS check discord", "votes": null }, { "id": "1128269", "postDate": "12/27/2020 09:33:20", "content": "

A really nice qoute --- \" A single image explains more than a thousand words. \"
\nSo if you are changing the size of a image , then you are getting more information from it and the having more better resolution the mode learns more from that image and hence performs much better.

\n

Adding other kind of augmentations to it , increase the learning of your model. And TTA plays a cherry on the top kind of feel.
\nSo just play with these concepts , try more experiments and try to improve your CV score .
\nThere are high changes of shakeup in the competition and hence just pay more attention to your CV score and not much on LB.

", "rawMarkdown": "A really nice qoute --- \" A single image explains more than a thousand words. \"\nSo if you are changing the size of a image , then you are getting more information from it and the having more better resolution the mode learns more from that image and hence performs much better.\n\nAdding other kind of augmentations to it , increase the learning of your model. And TTA plays a cherry on the top kind of feel.\nSo just play with these concepts , try more experiments and try to improve your CV score .\nThere are high changes of shakeup in the competition and hence just pay more attention to your CV score and not much on LB.", "votes": null } ], "comments": [ { "id": 1122546, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "12/22/2020 14:15:55", "content": "

Why do you think that's weird? By being able to look at more of the picture at once or the same area on higher resolution, you are using slightly more information. Thanks to the pretaining on ImageNet and the decent amount of training data we have here, it is not so surprising that the model can cope with that extra information.

", "votes": null, "replies": [] }, { "id": 1122617, "author_name": "yannmajewski", "author_url": "", "post_date": "12/22/2020 15:12:51", "content": "

you should read this paper: https://arxiv.org/pdf/1906.06423.pdf
\nIf you are using RandomResized crop as an augmentation its probably why! It explains how you can test on higher image resolution

", "votes": null, "replies": [ { "id": 1122656, "author_name": "kaushal2896", "author_url": "", "post_date": "12/22/2020 15:37:14", "content": "

@yannmajewski Thanks for pointing it out. Yes I'm using random resized crop.

", "votes": null, "replies": [] }, { "id": 1122664, "author_name": "yannmajewski", "author_url": "", "post_date": "12/22/2020 15:41:48", "content": "

This might be why! Have a good read :)

", "votes": null, "replies": [] } ] }, { "id": 1123213, "author_name": "graf10a", "author_url": "", "post_date": "12/23/2020 03:28:31", "content": "

For this dataset, the difference between 0.885 and 0.887 does not seem to be statistically significant. Most probably, it is just noise.

", "votes": null, "replies": [ { "id": 1126291, "author_name": "kaushal2896", "author_url": "", "post_date": "12/25/2020 13:46:48", "content": "

@graf10a Yes good point but this improved the scores consistently. For example 0.892 to 0.896.

", "votes": null, "replies": [] } ] }, { "id": 1124835, "author_name": "alexandersoare", "author_url": "", "post_date": "12/24/2020 08:16:10", "content": "

I would agree with @graf10a . If you have time - try other resolutions near (but below) 448x448 to see what you get.

\n

And if you are relying on the random seed to keep things consistent across experiments, I'm not sure it would work in cases where augmentations depend on the image size. The way a random sized crop is generated might look different with the same seed but different image size.

", "votes": null, "replies": [ { "id": 1126491, "author_name": "kaushal2896", "author_url": "", "post_date": "12/25/2020 16:45:01", "content": "

@alexandersoare Good idea. I tried increasing image size while inferencing on the same model (trained on 448x448) to 600x600 and it didn't boost my score further.

", "votes": null, "replies": [] }, { "id": 1126512, "author_name": "alexandersoare", "author_url": "", "post_date": "12/25/2020 17:06:28", "content": "

Ahh I just realised you weren't talking about retraining on 512x512. Yep I've tried this trick before. Never got anything I could be sure of though. PS check discord

", "votes": null, "replies": [] } ] }, { "id": 1128269, "author_name": "prashantarorat", "author_url": "", "post_date": "12/27/2020 09:33:20", "content": "

A really nice qoute --- \" A single image explains more than a thousand words. \"
\nSo if you are changing the size of a image , then you are getting more information from it and the having more better resolution the mode learns more from that image and hence performs much better.

\n

Adding other kind of augmentations to it , increase the learning of your model. And TTA plays a cherry on the top kind of feel.
\nSo just play with these concepts , try more experiments and try to improve your CV score .
\nThere are high changes of shakeup in the competition and hence just pay more attention to your CV score and not much on LB.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1122469": "I tried training ResNet-18 on stratified 5 folds with an image size of `448 x 448`. When I calculate the OOF score using all 5 models with the image size `448 x 448` (same as used in training) it gives an OOF score of 0.885, now when I calculate the OOF score using image size `512 x 512`, surprisingly it improves to 0.887.\n\nThis also improves my public LB score. I came to know this by mistake and don't know the possible reason behind this.", "1122546": "Why do you think that's weird? By being able to look at more of the picture at once or the same area on higher resolution, you are using slightly more information. Thanks to the pretaining on ImageNet and the decent amount of training data we have here, it is not so surprising that the model can cope with that extra information.", "1122617": "you should read this paper: https://arxiv.org/pdf/1906.06423.pdf\nIf you are using RandomResized crop as an augmentation its probably why! It explains how you can test on higher image resolution", "1122656": "yannmajewski Thanks for pointing it out. Yes I'm using random resized crop.", "1122664": "This might be why! Have a good read :)", "1123213": "For this dataset, the difference between 0.885 and 0.887 does not seem to be statistically significant. Most probably, it is just noise.", "1124835": "I would agree with @graf10a . If you have time - try other resolutions near (but below) 448x448 to see what you get.\n\nAnd if you are relying on the random seed to keep things consistent across experiments, I'm not sure it would work in cases where augmentations depend on the image size. The way a random sized crop is generated might look different with the same seed but different image size.", "1126291": "graf10a Yes good point but this improved the scores consistently. For example 0.892 to 0.896.", "1126491": "alexandersoare Good idea. I tried increasing image size while inferencing on the same model (trained on 448x448) to 600x600 and it didn't boost my score further.", "1126512": "Ahh I just realised you weren't talking about retraining on 512x512. Yep I've tried this trick before. Never got anything I could be sure of though. PS check discord", "1128269": "A really nice qoute --- \" A single image explains more than a thousand words. \"\nSo if you are changing the size of a image , then you are getting more information from it and the having more better resolution the mode learns more from that image and hence performs much better.\n\nAdding other kind of augmentations to it , increase the learning of your model. And TTA plays a cherry on the top kind of feel.\nSo just play with these concepts , try more experiments and try to improve your CV score .\nThere are high changes of shakeup in the competition and hence just pay more attention to your CV score and not much on LB." }, "source": "meta" }