Congratulations to all the winners and to everyone who took part in this. It was a great learning experience for me and I'll share what finally worked for me. There is still so much to learn by going through the solution overviews and more discussions/kernels. The best performance I was able to get was public LB: 0.9570, private LB: 0.9403.
\nMy best model is an ensemble (just the mean of the predictions) of 4 models. I am using @cdeotte's Triple Stratified Split. 3 models were trained on data from 2017 + 2018 + 2020. One of the models was trained on 2017 + 2018 + 2019 + archives + 2020.
\n3 models were trained on 512x512 images and 1 model on 384x384.
\n2 models use the following augmentations (using kornia):
- name: Rescale\n params:\n value: 255.\n- name: RandomAffine\n params:\n degrees: 180\n translate:\n - 0.02\n - 0.02\n- name: RandomHorizontalFlip\n params:\n p: 0.5\n- name: ColorJitter\n params:\n saturation:\n - 0.7\n - 1.3\n contrast:\n - 0.8\n - 1.2\n brightness: 0.1\n- name: Normalize\n params:\n mean: imagenet\n std: imagenet\nThe meaning of different params can be found from kornia's documentation.
The other 2 models additionally use Cutout:
- name: RandomErasing\n params:\n p: 0.5\n ratio:\n - 0.3\n - 3.3\n scale:\n - 0.02\n - 0.1\nTo iterate faster, instead of upsampling the minority class, I downsample the majority class per epoch. However, to avoid wasting data, I sample different instances from the majority class per epoch.
\nThe same network architecture is used for all the 4 models - EfficientNet-B5 features followed by a Linear layer.
\nOptimizer: AdamW\nweight decay: 0.1\nepochs: 20\nLR scheduler: 1cycle with initial LR = 5e-6, max LR = 2e-4 (LR range test)\nCongratulations to all the winners and to everyone who took part in this. It was a great learning experience for me and I'll share what finally worked for me. There is still so much to learn by going through the solution overviews and more discussions/kernels. The best performance I was able to get was public LB: 0.9570, private LB: 0.9403.
\nMy best model is an ensemble (just the mean of the predictions) of 4 models. I am using @cdeotte's Triple Stratified Split. 3 models were trained on data from 2017 + 2018 + 2020. One of the models was trained on 2017 + 2018 + 2019 + archives + 2020.
\n3 models were trained on 512x512 images and 1 model on 384x384.
\n2 models use the following augmentations (using kornia):
- name: Rescale\n params:\n value: 255.\n- name: RandomAffine\n params:\n degrees: 180\n translate:\n - 0.02\n - 0.02\n- name: RandomHorizontalFlip\n params:\n p: 0.5\n- name: ColorJitter\n params:\n saturation:\n - 0.7\n - 1.3\n contrast:\n - 0.8\n - 1.2\n brightness: 0.1\n- name: Normalize\n params:\n mean: imagenet\n std: imagenet\nThe meaning of different params can be found from kornia's documentation.
The other 2 models additionally use Cutout:
- name: RandomErasing\n params:\n p: 0.5\n ratio:\n - 0.3\n - 3.3\n scale:\n - 0.02\n - 0.1\nTo iterate faster, instead of upsampling the minority class, I downsample the majority class per epoch. However, to avoid wasting data, I sample different instances from the majority class per epoch.
\nThe same network architecture is used for all the 4 models - EfficientNet-B5 features followed by a Linear layer.
\nOptimizer: AdamW\nweight decay: 0.1\nepochs: 20\nLR scheduler: 1cycle with initial LR = 5e-6, max LR = 2e-4 (LR range test)\nGreat job Aman ! Could you please elaborate what you mean when you say custom head wrt CNN ? (NooB question alert 🙈)
", "rawMarkdown": "Great job Aman ! Could you please elaborate what you mean when you say custom head wrt CNN ? (NooB question alert 🙈)", "votes": null }, { "id": "976167", "postDate": "08/18/2020 17:23:28", "content": "Thanks @realsid. Don't worry, it's very important to ask questions! :)
\nSo, for my best models, I only added a single linear layer on top of the features from the backbone (to avoid overfitting). But in general, and also looking at the solutions of the winners, it is better to add more layers on top of the convolutional features. Is that clearer?
", "rawMarkdown": "Thanks @realsid. Don't worry, it's very important to ask questions! :) \n\nSo, for my best models, I only added a single linear layer on top of the features from the backbone (to avoid overfitting). But in general, and also looking at the solutions of the winners, it is better to add more layers on top of the convolutional features. Is that clearer?", "votes": null }, { "id": "976549", "postDate": "08/19/2020 00:10:52", "content": "\n\nTo iterate faster, instead of upsampling the minority class, I downsample the majority class per epoch. However, to avoid wasting data, I sample different instances from the majority class per epoch.
\n
Nice trick Aman. I'm gonna try this in my next comp. Congrats on your strong solo silver finish.
", "rawMarkdown": "> To iterate faster, instead of upsampling the minority class, I downsample the majority class per epoch. However, to avoid wasting data, I sample different instances from the majority class per epoch.\n\nNice trick Aman. I'm gonna try this in my next comp. Congrats on your strong solo silver finish.", "votes": null }, { "id": "976720", "postDate": "08/19/2020 04:06:16", "content": "Definitely, I see what you mean by adding Linear layer/Dropout etc. Even I wanted to do something like this, but lacked the time ! Thank you for answering !
", "rawMarkdown": "Definitely, I see what you mean by adding Linear layer/Dropout etc. Even I wanted to do something like this, but lacked the time ! Thank you for answering !", "votes": null }, { "id": "976976", "postDate": "08/19/2020 07:57:26", "content": "Yes, the trick is to start on the competition early enough. And I missed that too. All the best!
", "rawMarkdown": "Yes, the trick is to start on the competition early enough. And I missed that too. All the best!", "votes": null }, { "id": "976977", "postDate": "08/19/2020 07:58:24", "content": "Thanks, Chris. Happy to know that I could contribute an idea to your playbook! :)
", "rawMarkdown": "Thanks, Chris. Happy to know that I could contribute an idea to your playbook! :)", "votes": null }, { "id": "982781", "postDate": "08/23/2020 16:55:53", "content": "Thank you for this.
", "rawMarkdown": "Thank you for this.", "votes": null } ], "comments": [ { "id": 976116, "author_name": "realsid", "author_url": "", "post_date": "08/18/2020 16:37:47", "content": "Great job Aman ! Could you please elaborate what you mean when you say custom head wrt CNN ? (NooB question alert 🙈)
", "votes": null, "replies": [ { "id": 976167, "author_name": "themlenthusiast", "author_url": "", "post_date": "08/18/2020 17:23:28", "content": "Thanks @realsid. Don't worry, it's very important to ask questions! :)
\nSo, for my best models, I only added a single linear layer on top of the features from the backbone (to avoid overfitting). But in general, and also looking at the solutions of the winners, it is better to add more layers on top of the convolutional features. Is that clearer?
", "votes": null, "replies": [] }, { "id": 976720, "author_name": "realsid", "author_url": "", "post_date": "08/19/2020 04:06:16", "content": "Definitely, I see what you mean by adding Linear layer/Dropout etc. Even I wanted to do something like this, but lacked the time ! Thank you for answering !
", "votes": null, "replies": [] }, { "id": 976976, "author_name": "themlenthusiast", "author_url": "", "post_date": "08/19/2020 07:57:26", "content": "Yes, the trick is to start on the competition early enough. And I missed that too. All the best!
", "votes": null, "replies": [] } ] }, { "id": 976549, "author_name": "cdeotte", "author_url": "", "post_date": "08/19/2020 00:10:52", "content": "\n\nTo iterate faster, instead of upsampling the minority class, I downsample the majority class per epoch. However, to avoid wasting data, I sample different instances from the majority class per epoch.
\n
Nice trick Aman. I'm gonna try this in my next comp. Congrats on your strong solo silver finish.
", "votes": null, "replies": [ { "id": 976977, "author_name": "themlenthusiast", "author_url": "", "post_date": "08/19/2020 07:58:24", "content": "Thanks, Chris. Happy to know that I could contribute an idea to your playbook! :)
", "votes": null, "replies": [] } ] }, { "id": 982781, "author_name": "kiran109", "author_url": "", "post_date": "08/23/2020 16:55:53", "content": "Thank you for this.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "975488": "Congratulations to all the winners and to everyone who took part in this. It was a great learning experience for me and I'll share what finally worked for me. There is still so much to learn by going through the solution overviews and more discussions/kernels. The best performance I was able to get was public LB: 0.9570, private LB: 0.9403.\n\n## Data\nMy best model is an ensemble (just the mean of the predictions) of 4 models. I am using @cdeotte's Triple Stratified Split. 3 models were trained on data from 2017 + 2018 + 2020. One of the models was trained on 2017 + 2018 + 2019 + archives + 2020.\n\n## Image sizes\n3 models were trained on 512x512 images and 1 model on 384x384.\n\n## Augmentations\n2 models use the following augmentations (using `kornia`):\n```\n- name: Rescale\n params:\n value: 255.\n- name: RandomAffine\n params:\n degrees: 180\n translate:\n - 0.02\n - 0.02\n- name: RandomHorizontalFlip\n params:\n p: 0.5\n- name: ColorJitter\n params:\n saturation:\n - 0.7\n - 1.3\n contrast:\n - 0.8\n - 1.2\n brightness: 0.1\n- name: Normalize\n params:\n mean: imagenet\n std: imagenet\n```\nThe meaning of different params can be found from `kornia`'s [documentation](https://github.com/kornia/kornia/blob/master/kornia/augmentation/augmentation.py).\n\nThe other 2 models additionally use `Cutout`:\n```\n- name: RandomErasing\n params:\n p: 0.5\n ratio:\n - 0.3\n - 3.3\n scale:\n - 0.02\n - 0.1\n```\n\n## Sampling\nTo iterate faster, instead of upsampling the minority class, I downsample the majority class per epoch. However, to avoid wasting data, I sample different instances from the majority class per epoch.\n\n## Network\nThe same network architecture is used for all the 4 models - EfficientNet-B5 features followed by a Linear layer.\n\n## Optimization\n```\nOptimizer: AdamW\nweight decay: 0.1\nepochs: 20\nLR scheduler: 1cycle with initial LR = 5e-6, max LR = 2e-4 (LR range test)\n```\n\n## Things I wanted to try but couldn't\n- Train on images of different sizes for the 3 best configs on 512x512 and ensemble\n- Incorporate meta-data by stacking it with the features from the deep model\n- Spend time improving my ensembles\n- More augmentations\n- Custom head", "976116": "Great job Aman ! Could you please elaborate what you mean when you say custom head wrt CNN ? (NooB question alert 🙈)", "976167": "Thanks @realsid. Don't worry, it's very important to ask questions! :) \n\nSo, for my best models, I only added a single linear layer on top of the features from the backbone (to avoid overfitting). But in general, and also looking at the solutions of the winners, it is better to add more layers on top of the convolutional features. Is that clearer?", "976549": "> To iterate faster, instead of upsampling the minority class, I downsample the majority class per epoch. However, to avoid wasting data, I sample different instances from the majority class per epoch.\n\nNice trick Aman. I'm gonna try this in my next comp. Congrats on your strong solo silver finish.", "976720": "Definitely, I see what you mean by adding Linear layer/Dropout etc. Even I wanted to do something like this, but lacked the time ! Thank you for answering !", "976976": "Yes, the trick is to start on the competition early enough. And I missed that too. All the best!", "976977": "Thanks, Chris. Happy to know that I could contribute an idea to your playbook! :)", "982781": "Thank you for this." }, "source": "meta" }