![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4367831%2Fe230cf84b270c5c0e19e3bede722e06d%2FSelection_085.jpg?generation=1611069573949012&alt=media\")
At this post, I read that multiple people get 0.900+ CV accuracy with their model, and far better LB than mine. I understand that the dataset is noisy, but others could get a much better score than I, with the effnet or resnext architecture.
\nHere is the training cooperation table. I made the validation process from k = 5 fold 0. I generated a .csv file, where I wrote what is fold 0, so the validation set is the same. I tested two architectures resnet50_32x4d and effnet-b4, and several methods, but somehow couldn't pass the 0.898 validation accuracy. I tried:
`cbb', 'cbsd' ,'cgm', 'healthy' classes to get the same distribution, or by 2x to reduce the class imbalance
The basic approach is. In the last case (column), I used a scheduler.
Image size 512\nOptimizer Ranger (Lookahed + RAdam)\nDataset 2020\nFold 0/5\nAugmentation random scale, random rotate, random contrast, random hsv\n random flip transpose, do_random_cutout\nsmooth 0.05\nI tried with light augmentation based on this topic, so I reduced the augmentation (other configure parts same with basic approach).
Augmentation random flip transpose,\nThe random flip transpose:
\ndef do_random_flip_transpose(image):\n if np.random.rand() < 0.5:\n image = cv2.flip(image, 0)\n if np.random.rand() < 0.5:\n image = cv2.flip(image, 1)\n if np.random.rand() < 0.5:\n image = image.transpose(1, 0, 2)\n return np.ascontiguousarray(image)\nI started this competition 16 days ago and spent nearly \"full-time\" daily without any considerable achievement. I ran out of ideas, so I need external advice.
\nThanks for the help!
", "messages": [ { "id": "1159968", "postDate": "01/19/2021 15:25:31", "content": "At this post, I read that multiple people get 0.900+ CV accuracy with their model, and far better LB than mine. I understand that the dataset is noisy, but others could get a much better score than I, with the effnet or resnext architecture.
\nHere is the training cooperation table. I made the validation process from k = 5 fold 0. I generated a .csv file, where I wrote what is fold 0, so the validation set is the same. I tested two architectures resnet50_32x4d and effnet-b4, and several methods, but somehow couldn't pass the 0.898 validation accuracy. I tried:
`cbb', 'cbsd' ,'cgm', 'healthy' classes to get the same distribution, or by 2x to reduce the class imbalance
The basic approach is. In the last case (column), I used a scheduler.
Image size 512\nOptimizer Ranger (Lookahed + RAdam)\nDataset 2020\nFold 0/5\nAugmentation random scale, random rotate, random contrast, random hsv\n random flip transpose, do_random_cutout\nsmooth 0.05\nI tried with light augmentation based on this topic, so I reduced the augmentation (other configure parts same with basic approach).
Augmentation random flip transpose,\nThe random flip transpose:
\ndef do_random_flip_transpose(image):\n if np.random.rand() < 0.5:\n image = cv2.flip(image, 0)\n if np.random.rand() < 0.5:\n image = cv2.flip(image, 1)\n if np.random.rand() < 0.5:\n image = image.transpose(1, 0, 2)\n return np.ascontiguousarray(image)\nI started this competition 16 days ago and spent nearly \"full-time\" daily without any considerable achievement. I ran out of ideas, so I need external advice.
\nThanks for the help!
", "rawMarkdown": "At this [post](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/203111), I read that multiple people get 0.900+ CV accuracy with their model, and far better LB than mine. I understand that the dataset is noisy, but others could get a much better score than I, with the effnet or resnext architecture.\n\nHere is the training cooperation table. I made the validation process from `k = 5` fold 0. I generated a .csv file, where I wrote what is fold 0, so the validation set is the same. I tested two architectures `resnet50_32x4d` and `effnet-b4`, and several methods, but somehow couldn't pass the 0.898 validation accuracy. I tried:\n1. [Bi-Tempered Logistic Loss](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/202017) loss\n2. [Focal Cosine Loss](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/203271)\n3. [RandAugment](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/208887#1139841)\n4. Up-sample the ``cbb', 'cbsd' ,'cgm', 'healthy'` classes to get the same distribution, or by 2x to reduce the class imbalance\n5. Combine the 2020 dataset with 2019 (merged column in the image)\n6. Snapmix\n7. Learning scheduler (Effnet-b4-scheduler column in the image)\nNeither one approach improved my LB or CV.\n\n\n\nThe `basic` approach is. In the last case (column), I used a scheduler.\n\n```bash\nImage size 512\nOptimizer Ranger (Lookahed + RAdam)\nDataset 2020\nFold 0/5\nAugmentation random scale, random rotate, random contrast, random hsv\n random flip transpose, do_random_cutout\nsmooth 0.05\n```\n\nI tried with light augmentation based on this [topic](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/206489), so I reduced the augmentation (other configure parts same with `basic` approach).\n\n```bash\nAugmentation\trandom flip transpose,\n```\n\nThe random flip transpose:\n\n```python\ndef do_random_flip_transpose(image):\n if np.random.rand() < 0.5:\n image = cv2.flip(image, 0)\n if np.random.rand() < 0.5:\n image = cv2.flip(image, 1)\n if np.random.rand() < 0.5:\n image = image.transpose(1, 0, 2)\n return np.ascontiguousarray(image)\n```\n\n### Questions:\n\n1. What am I doing wrong?\n2. For the same validation fold, for CV 0.898, I got LB between [0.883, 0.896]. Should I entirely pay attention to the LB score?\n3. What else can I try?\n\nI started this competition 16 days ago and spent nearly \"full-time\" daily without any considerable achievement. I ran out of ideas, so I need external advice.\n\nThanks for the help!", "votes": null }, { "id": "1160162", "postDate": "01/19/2021 17:47:54", "content": "Light Augs didn't help for me and made my model overfitting, thus poor CV and LB. I use heavy augmentations.
\nMy CV/LB relation is pretty much stable . The gap is between [- 0.002; +0.002]. But my CV is higher than 0.898 though.
\nAdding 2019 dataset may help too.
\nThat being said, beware that while doing stuffs that basically everyone else knows/is doing, may help to climb on the LB, you'll also need some clever tricks and out of the box thinking to remain at the top of Private LB, like almost all kaggle competitions.
", "rawMarkdown": "Light Augs didn't help for me and made my model overfitting, thus poor CV and LB. I use heavy augmentations. \n\nMy CV/LB relation is pretty much stable . The gap is between [- 0.002; +0.002]. But my CV is higher than 0.898 though.\n\nAdding 2019 dataset may help too.\n\nThat being said, beware that while doing stuffs that basically everyone else knows/is doing, may help to climb on the LB, you'll also need some clever tricks and out of the box thinking to remain at the top of Private LB, like almost all kaggle competitions.", "votes": null }, { "id": "1160368", "postDate": "01/19/2021 21:24:46", "content": "thanks @thakurudit for the response. I forgot to write that I tried snapmix, and to combine 2020 with the 2019 dataset, so I update the post
", "rawMarkdown": "thanks @thakurudit for the response. I forgot to write that I tried snapmix, and to combine 2020 with the 2019 dataset, so I update the post", "votes": null }, { "id": "1160385", "postDate": "01/19/2021 21:39:32", "content": "Thanks for your response! I forgot to write, but I tried to merge the 2020 and 2019 dataset as well. @serigne do you use the simple k-fold method or something else for the cross-validation?
\nDo you have any thought why the CV and LB score is changing for the same validation set?
Start with B0ns(default) and adam,as a baseline, and then start testing around with common setups before using larger models. E.g. I got ValAcc 0.896/Lb 0.893 with singel fold B0ns. Start with heavy aug and reduce while testing instead of vice versa.
", "rawMarkdown": "Start with B0ns(default) and adam,as a baseline, and then start testing around with common setups before using larger models. E.g. I got ValAcc 0.896/Lb 0.893 with singel fold B0ns. Start with heavy aug and reduce while testing instead of vice versa.", "votes": null }, { "id": "1160608", "postDate": "01/20/2021 03:28:26", "content": "Did you use valid time augmentation for CV score?
", "rawMarkdown": "Did you use valid time augmentation for CV score?", "votes": null }, { "id": "1160618", "postDate": "01/20/2021 03:41:18", "content": "One question, @serigne Do you seed everything in your notebook or do you use a different seed to get different validation splits each time? If so, do you mind sharing your seed number?
", "rawMarkdown": "One question, @serigne Do you seed everything in your notebook or do you use a different seed to get different validation splits each time? If so, do you mind sharing your seed number?", "votes": null }, { "id": "1160733", "postDate": "01/20/2021 05:10:32", "content": "\n\n\nLight Augs didn't help for me and made my model overfitting
\n
same thing for me, I don't how how people are achieving high Lb but unstable Cv relations with light augs.
", "rawMarkdown": "serigne \n> Light Augs didn't help for me and made my model overfitting\n\nsame thing for me, I don't how how people are achieving high Lb but unstable Cv relations with light augs.", "votes": null }, { "id": "1161393", "postDate": "01/20/2021 14:44:22", "content": "@junyingsg
\nYeah I seed everything to make my experiments reproducible and see where improvements come from.
I don't tweak the seed number and use just 42.
", "rawMarkdown": "junyingsg \nYeah I seed everything to make my experiments reproducible and see where improvements come from. \n\nI don't tweak the seed number and use just 42.", "votes": null }, { "id": "1161480", "postDate": "01/20/2021 15:34:32", "content": "Thanks a lot!
", "rawMarkdown": "Thanks a lot!", "votes": null }, { "id": "1164365", "postDate": "01/22/2021 10:23:29", "content": "Another thing you can do is use 768x768 image sizes. I did that and it helped me jump 100 places on the lb with correlated cv.
", "rawMarkdown": "Another thing you can do is use 768x768 image sizes. I did that and it helped me jump 100 places on the lb with correlated cv.", "votes": null }, { "id": "1170000", "postDate": "01/25/2021 21:59:37", "content": "Try with other augmentations, other libs.
\nI know that people got +0.001/0.002+ with 2019, not sure why not significant but still some boost
Thanks @woshifym for your suggestion, I will try!
", "rawMarkdown": "Thanks @woshifym for your suggestion, I will try!", "votes": null }, { "id": "1180632", "postDate": "02/01/2021 11:38:52", "content": "Whats the point in doing this except pushing your cv?
", "rawMarkdown": "Whats the point in doing this except pushing your cv?", "votes": null }, { "id": "1180690", "postDate": "02/01/2021 12:18:56", "content": "The CV increase is the only benefit
", "rawMarkdown": "The CV increase is the only benefit", "votes": null } ], "comments": [ { "id": 1160162, "author_name": "thakurudit", "author_url": "", "post_date": "01/19/2021 17:47:54", "content": "thanks @thakurudit for the response. I forgot to write that I tried snapmix, and to combine 2020 with the 2019 dataset, so I update the post
", "votes": null, "replies": [] }, { "id": 1164365, "author_name": "thakurudit", "author_url": "", "post_date": "01/22/2021 10:23:29", "content": "Another thing you can do is use 768x768 image sizes. I did that and it helped me jump 100 places on the lb with correlated cv.
", "votes": null, "replies": [] } ] }, { "id": 1160205, "author_name": "serigne", "author_url": "", "post_date": "01/19/2021 18:25:33", "content": "Light Augs didn't help for me and made my model overfitting, thus poor CV and LB. I use heavy augmentations.
\nMy CV/LB relation is pretty much stable . The gap is between [- 0.002; +0.002]. But my CV is higher than 0.898 though.
\nAdding 2019 dataset may help too.
\nThat being said, beware that while doing stuffs that basically everyone else knows/is doing, may help to climb on the LB, you'll also need some clever tricks and out of the box thinking to remain at the top of Private LB, like almost all kaggle competitions.
", "votes": null, "replies": [ { "id": 1160385, "author_name": "bessenyeiszilrd", "author_url": "", "post_date": "01/19/2021 21:39:32", "content": "Thanks for your response! I forgot to write, but I tried to merge the 2020 and 2019 dataset as well. @serigne do you use the simple k-fold method or something else for the cross-validation?
\nDo you have any thought why the CV and LB score is changing for the same validation set?
One question, @serigne Do you seed everything in your notebook or do you use a different seed to get different validation splits each time? If so, do you mind sharing your seed number?
", "votes": null, "replies": [] }, { "id": 1160733, "author_name": "thakurudit", "author_url": "", "post_date": "01/20/2021 05:10:32", "content": "\n\n\nLight Augs didn't help for me and made my model overfitting
\n
same thing for me, I don't how how people are achieving high Lb but unstable Cv relations with light augs.
", "votes": null, "replies": [] }, { "id": 1161393, "author_name": "serigne", "author_url": "", "post_date": "01/20/2021 14:44:22", "content": "@junyingsg
\nYeah I seed everything to make my experiments reproducible and see where improvements come from.
I don't tweak the seed number and use just 42.
", "votes": null, "replies": [] }, { "id": 1161480, "author_name": "junyingsg", "author_url": "", "post_date": "01/20/2021 15:34:32", "content": "Thanks a lot!
", "votes": null, "replies": [] } ] }, { "id": 1160392, "author_name": "kirderf", "author_url": "", "post_date": "01/19/2021 21:47:26", "content": "Start with B0ns(default) and adam,as a baseline, and then start testing around with common setups before using larger models. E.g. I got ValAcc 0.896/Lb 0.893 with singel fold B0ns. Start with heavy aug and reduce while testing instead of vice versa.
", "votes": null, "replies": [] }, { "id": 1160608, "author_name": "woshifym", "author_url": "", "post_date": "01/20/2021 03:28:26", "content": "Did you use valid time augmentation for CV score?
", "votes": null, "replies": [ { "id": 1180597, "author_name": "bessenyeiszilrd", "author_url": "", "post_date": "02/01/2021 10:58:49", "content": "Thanks @woshifym for your suggestion, I will try!
", "votes": null, "replies": [] }, { "id": 1180632, "author_name": "alexanderriedel", "author_url": "", "post_date": "02/01/2021 11:38:52", "content": "Whats the point in doing this except pushing your cv?
", "votes": null, "replies": [] }, { "id": 1180690, "author_name": "bessenyeiszilrd", "author_url": "", "post_date": "02/01/2021 12:18:56", "content": "The CV increase is the only benefit
", "votes": null, "replies": [] } ] }, { "id": 1170000, "author_name": "muhakabartay", "author_url": "", "post_date": "01/25/2021 21:59:37", "content": "Try with other augmentations, other libs.
\nI know that people got +0.001/0.002+ with 2019, not sure why not significant but still some boost