{ "id": 311211, "title": "7 More Computer Vision Tricks to Improve Score", "url": "/competitions/happy-whale-and-dolphin/discussion/311211", "author_name": "Sanyam Bhutani", "post_date": "2022-03-05T14:55:29.844000", "votes": 117, "comment_count": 38, "views": 0, "content": "

Hey Everybody!

\n

Since I posted the previous collection of tricks/tips, I have been on a quest to add more suggestions and I have been able to formulate a few more suggestions. These are tricks from papers and by kagglers that might help your score in this competition. Here's the list of 7 more tricks:

\n

1. Test Time Augmentation (TTA):

\n

When training our models, we apply a few transformations to our training set: Image Augmentation, normalization, etc

\n

So now our model is used to learn from the \"transformed\" dataset. Ex Phalanx has created this wonderful dataset here that crops into images. Now our model is used to detect whales in cropped images. So, we would like to have a similar test dataset as well.

\n

Applying Augmentation techniques while inferencing is known as TTA. One of my most frequent mistakes is forgetting to normalise images when inferencing-same applies to any image augmentations too.

\n

2. Sequential Unfreezing while Transfer Learning:

\n

I learned this trick during the fastai course. When we are performing transfer learning, our model has already captured a lot of information.

\n

The initial layers (Layers close to inputs) retain more info about the structure of objects, etc and the latter layers (close to output) learn more about the dataset. We can envision our model to be grouped in layers like so:

\n

Input(Group) -> HiddenSetEarly -> HiddenSetLater -> Output(Group)

\n

When performing transfer learning, its usually a good idea to just train the last few layers and then unfreeze the earlier layers sequentially

\n

3. Differential Learning Rates:

\n

Continuing with the previous point, another trick I learned via fastai:

\n

The initial few layers need little to no re-training, so applying differential learning rates to a different group of CNN layers is a great idea:

\n

Ex:
\nOutput(Group): Lr = 10e-3
\nHiddenSetLater: LR = 0.5 * 10e-4
\nInput(Group): Lr = 10e-5

\n

This would make minimal changes to the initial layers and more changes to the head (output) layers making our model converge a bit faster

\n

4. PyTorch: use LazyLayers

\n

Note: I learned this trick thanks to Datasaurus, please see his post here

\n

TL;DR when you perform transfer learning using a backbone, you have to re-write the last few layers with the dimensions in mind. For lazy folks like me, PyTorch has a LazyLayer (pun intended) to make it easier:

\n

self.fc = nn.LazyLinear(self.cfg.target_size)

\n

Note: This would otherwise be self.fc = nn.Linear(self.n_features, self.cfg.target_size)

\n

Once again, thanks Datasaurus for sharing this in his original post

\n

5. Label Smoothing:

\n

We have seen a lot of discussions in this competition about the funny images that exist in the dataset. This is not uncommon, ImageNet and many datasets themselves have many \"mislabeled\" images. The trick to helping here is using Label Smoothing.

\n

This is a good writeup about how it works. TL;DR: Adding noise to all labels helps our model generalize better

\n

6. Use GeM Pooling + ArcFace:

\n

Debarshi has a fantastic starter pack here that uses GeM pooling + ArcFace.

\n

I had covered this in a live stream, the short version: If you are working with a large number of labels and images in an imbalanced dataset:

\n

Replacing your Max/AvgPooling layer with a GeM pooling layer and ArcFace loss is effective for such cases. I plan to cover this in more detail soon.

\n

7. PsuedoLabelling:

\n

PsuedoLabelling involves a form of semi-supervised learning. Chris Deotte teaches this in his fantastic kernel here

\n

If you want to utilise more labels in training: You grab the most confident predictions from your model and then add them to your training dataset and build a new model.

\n

Like always, for anyone that has read this far, I have 1 bonus tip this time:

\n

Bonus Tip: Use SSDs for image datasets

\n

Again, I respectfully understand many kagglers use cloud hardware and don't have to worry about this step. However, if you've invested in local setups, I was for a long time not worrying about where I store my datasets. This would especially create IO bottlenecks for image datasets.

\n

Always, consider storing your datasets on an M.2 Drive or SSD when you are reading them into a model so that you don't see a bottleneck.

\n

I hope these tips are helpful!
\nSee you on the LB! :)

", "messages": [ { "id": 1712996, "postDate": "2022-03-05T14:55:29.843Z", "content": "

Hey Everybody!

\n

Since I posted the previous collection of tricks/tips, I have been on a quest to add more suggestions and I have been able to formulate a few more suggestions. These are tricks from papers and by kagglers that might help your score in this competition. Here's the list of 7 more tricks:

\n

1. Test Time Augmentation (TTA):

\n

When training our models, we apply a few transformations to our training set: Image Augmentation, normalization, etc

\n

So now our model is used to learn from the \"transformed\" dataset. Ex Phalanx has created this wonderful dataset here that crops into images. Now our model is used to detect whales in cropped images. So, we would like to have a similar test dataset as well.

\n

Applying Augmentation techniques while inferencing is known as TTA. One of my most frequent mistakes is forgetting to normalise images when inferencing-same applies to any image augmentations too.

\n

2. Sequential Unfreezing while Transfer Learning:

\n

I learned this trick during the fastai course. When we are performing transfer learning, our model has already captured a lot of information.

\n

The initial layers (Layers close to inputs) retain more info about the structure of objects, etc and the latter layers (close to output) learn more about the dataset. We can envision our model to be grouped in layers like so:

\n

Input(Group) -> HiddenSetEarly -> HiddenSetLater -> Output(Group)

\n

When performing transfer learning, its usually a good idea to just train the last few layers and then unfreeze the earlier layers sequentially

\n

3. Differential Learning Rates:

\n

Continuing with the previous point, another trick I learned via fastai:

\n

The initial few layers need little to no re-training, so applying differential learning rates to a different group of CNN layers is a great idea:

\n

Ex:
\nOutput(Group): Lr = 10e-3
\nHiddenSetLater: LR = 0.5 * 10e-4
\nInput(Group): Lr = 10e-5

\n

This would make minimal changes to the initial layers and more changes to the head (output) layers making our model converge a bit faster

\n

4. PyTorch: use LazyLayers

\n

Note: I learned this trick thanks to Datasaurus, please see his post here

\n

TL;DR when you perform transfer learning using a backbone, you have to re-write the last few layers with the dimensions in mind. For lazy folks like me, PyTorch has a LazyLayer (pun intended) to make it easier:

\n

self.fc = nn.LazyLinear(self.cfg.target_size)

\n

Note: This would otherwise be self.fc = nn.Linear(self.n_features, self.cfg.target_size)

\n

Once again, thanks Datasaurus for sharing this in his original post

\n

5. Label Smoothing:

\n

We have seen a lot of discussions in this competition about the funny images that exist in the dataset. This is not uncommon, ImageNet and many datasets themselves have many \"mislabeled\" images. The trick to helping here is using Label Smoothing.

\n

This is a good writeup about how it works. TL;DR: Adding noise to all labels helps our model generalize better

\n

6. Use GeM Pooling + ArcFace:

\n

Debarshi has a fantastic starter pack here that uses GeM pooling + ArcFace.

\n

I had covered this in a live stream, the short version: If you are working with a large number of labels and images in an imbalanced dataset:

\n

Replacing your Max/AvgPooling layer with a GeM pooling layer and ArcFace loss is effective for such cases. I plan to cover this in more detail soon.

\n

7. PsuedoLabelling:

\n

PsuedoLabelling involves a form of semi-supervised learning. Chris Deotte teaches this in his fantastic kernel here

\n

If you want to utilise more labels in training: You grab the most confident predictions from your model and then add them to your training dataset and build a new model.

\n

Like always, for anyone that has read this far, I have 1 bonus tip this time:

\n

Bonus Tip: Use SSDs for image datasets

\n

Again, I respectfully understand many kagglers use cloud hardware and don't have to worry about this step. However, if you've invested in local setups, I was for a long time not worrying about where I store my datasets. This would especially create IO bottlenecks for image datasets.

\n

Always, consider storing your datasets on an M.2 Drive or SSD when you are reading them into a model so that you don't see a bottleneck.

\n

I hope these tips are helpful!
\nSee you on the LB! :)

", "rawMarkdown": "Hey Everybody! \n\nSince I posted the previous collection of tricks/tips, I have been on a quest to add more suggestions and I have been able to formulate a few more suggestions. These are tricks from papers and by kagglers that might help your score in this competition. Here's the list of 7 more tricks:\n\n### 1. Test Time Augmentation (TTA):\n\nWhen training our models, we apply a few transformations to our training set: Image Augmentation, normalization, etc \n\nSo now our model is used to learn from the \"transformed\" dataset. Ex Phalanx has created this wonderful dataset [here](https://www.kaggle.com/phalanx/whale2-cropped-dataset) that crops into images. Now our model is used to detect whales in cropped images. So, we would like to have a similar test dataset as well. \n\nApplying Augmentation techniques while inferencing is known as TTA. One of my most frequent mistakes is forgetting to normalise images when inferencing-same applies to any image augmentations too. \n\n### 2. Sequential Unfreezing while Transfer Learning:\n\nI learned this trick during the fastai course. When we are performing transfer learning, our model has already captured a lot of information. \n\nThe initial layers (Layers close to inputs) retain more info about the structure of objects, etc and the latter layers (close to output) learn more about the dataset. We can envision our model to be grouped in layers like so:\n\n`Input(Group) -> HiddenSetEarly -> HiddenSetLater -> Output(Group)`\n\nWhen performing transfer learning, its usually a good idea to just train the last few layers and then unfreeze the earlier layers **sequentially**\n\n### 3. Differential Learning Rates: \n\nContinuing with the previous point, another trick I learned via fastai:\n\nThe initial few layers need little to no re-training, so applying differential learning rates to a different group of CNN layers is a great idea:\n\nEx: \nOutput(Group): Lr = 10e-3\nHiddenSetLater: LR = 0.5 * 10e-4\nInput(Group): Lr = 10e-5\n\nThis would make minimal changes to the initial layers and more changes to the head (output) layers making our model converge a bit faster\n\n### 4. PyTorch: use LazyLayers\n\nNote: I learned this trick thanks to Datasaurus, please see his post [here](https://www.kaggle.com/c/petfinder-pawpularity-score/discussion/279460) \n\nTL;DR when you perform transfer learning using a backbone, you have to re-write the last few layers with the dimensions in mind. For lazy folks like me, PyTorch has a LazyLayer (pun intended) to make it easier:\n\n`self.fc = nn.LazyLinear(self.cfg.target_size)`\n\nNote: This would otherwise be `self.fc = nn.Linear(self.n_features, self.cfg.target_size)`\n\nOnce again, thanks Datasaurus for sharing this in his original post\n\n### 5. Label Smoothing:\n\nWe have seen a lot of discussions in this competition about the funny images that exist in the dataset. This is not uncommon, ImageNet and many datasets themselves have many \"mislabeled\" images. The trick to helping here is using Label Smoothing. \n\n[This](https://paperswithcode.com/method/label-smoothing) is a good writeup about how it works. TL;DR: Adding noise to all labels helps our model generalize better\n\n### 6. Use GeM Pooling + ArcFace:\n\nDebarshi has a fantastic starter pack [here](https://www.kaggle.com/debarshichanda/pytorch-arcface-gem-pooling-starter) that uses GeM pooling + ArcFace. \n\nI had covered this in a live stream, the short version: If you are working with a large number of labels and images in an imbalanced dataset:\n\nReplacing your Max/AvgPooling layer with a GeM pooling layer and ArcFace loss is effective for such cases. I plan to cover this in more detail soon.\n\n### 7. PsuedoLabelling:\n\nPsuedoLabelling involves a form of semi-supervised learning. Chris Deotte teaches this in his fantastic kernel [here](https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969)\n\nIf you want to utilise more labels in training: You grab the most confident predictions from your model and then add them to your training dataset and build a new model. \n\nLike always, for anyone that has read this far, I have 1 bonus tip this time:\n\n### Bonus Tip: Use SSDs for image datasets\n\nAgain, I respectfully understand many kagglers use cloud hardware and don't have to worry about this step. However, if you've invested in local setups, I was for a long time not worrying about where I store my datasets. This would especially create IO bottlenecks for image datasets. \n\nAlways, consider storing your datasets on an M.2 Drive or SSD when you are reading them into a model so that you don't see a bottleneck. \n\nI hope these tips are helpful! \nSee you on the LB! :)", "votes": 117 }, { "id": 1714410, "postDate": "2022-03-07T01:20:37.237Z", "content": "

Great pointers. Thanks Sanyam!

", "rawMarkdown": "Great pointers. Thanks Sanyam!", "votes": 3, "replies": [ { "id": 1714468, "postDate": "2022-03-07T03:25:08.093Z", "content": "

Thanks so much, Chris! And thank you for the awesome PsuedoLabelling Notebook 🙏

", "rawMarkdown": "Thanks so much, Chris! And thank you for the awesome PsuedoLabelling Notebook 🙏", "votes": 3 } ] }, { "id": 1713789, "postDate": "2022-03-06T11:47:22.773Z", "content": "

Does anyone have success with GeM in this task? In my case it works worse than average

", "rawMarkdown": "Does anyone have success with GeM in this task? In my case it works worse than average", "votes": 3, "replies": [ { "id": 1714467, "postDate": "2022-03-07T03:24:39.650Z", "content": "

I'm still working on it but I read some discussions mention having success with it

", "rawMarkdown": "I'm still working on it but I read some discussions mention having success with it", "votes": 1 }, { "id": 1725028, "postDate": "2022-03-16T18:40:15.877Z", "content": "

GeM is a slightly better for me. But I haven't yet explored this thoroughly (So i might be wrong)

", "rawMarkdown": "GeM is a slightly better for me. But I haven't yet explored this thoroughly (So i might be wrong)" } ] }, { "id": 1713655, "postDate": "2022-03-06T08:47:01.390Z", "content": "

This is very useful. thanks for sharing!

", "rawMarkdown": "This is very useful. thanks for sharing!", "votes": 4 }, { "id": 1723767, "postDate": "2022-03-15T17:39:47.603Z", "content": "

Thanks for sharing!
\nI'm also lazy, but instead of nn.LazyLinear, I use nn.Identity on the head layer(s), like so

\n
def skip_head(m):\n    for attr in ['classifier', 'global_pool', 'head', 'fc']:\n        if hasattr(m, attr): setattr(m, attr, nn.Identity())\n    return m\n\nbody = skip_head(model)\n
\n

Then I add my custom head. But for the latter, you need to infer in_features from the model. If you have no BN, just a simple output layer, nn.LazyLinear does that for you.

", "rawMarkdown": "Thanks for sharing! \nI'm also lazy, but instead of `nn.LazyLinear`, I use `nn.Identity` on the head layer(s), like so\n\n```\ndef skip_head(m):\n for attr in ['classifier', 'global_pool', 'head', 'fc']:\n if hasattr(m, attr): setattr(m, attr, nn.Identity())\n return m\n\nbody = skip_head(model)\n```\nThen I add my custom head. But for the latter, you need to infer `in_features` from the model. If you have no BN, just a simple output layer, `nn.LazyLinear` does that for you.", "votes": 1, "replies": [ { "id": 1726015, "postDate": "2022-03-17T15:25:00.443Z", "content": "

Thanks so much for sharing! I will definitely use this in my experiments!

", "rawMarkdown": "Thanks so much for sharing! I will definitely use this in my experiments! " } ] }, { "id": 1722856, "postDate": "2022-03-14T23:27:43.617Z", "content": "

This is very useful. thanks for sharing!

", "rawMarkdown": "This is very useful. thanks for sharing!", "votes": 2 }, { "id": 1719960, "postDate": "2022-03-12T10:06:42.593Z", "content": "

Thanks for sharing mate, its worthy checking

", "rawMarkdown": "Thanks for sharing mate, its worthy checking" }, { "id": 1719343, "postDate": "2022-03-11T16:46:32.313Z", "content": "

how can I increase the contrast in images in the preprocessing phase ?

", "rawMarkdown": "how can I increase the contrast in images in the preprocessing phase ?" }, { "id": 1718928, "postDate": "2022-03-11T09:56:21.657Z", "content": "

Seems useful.Thx

", "rawMarkdown": "Seems useful.Thx" }, { "id": 1718272, "postDate": "2022-03-10T17:21:19.387Z", "content": "

Thanks Sanyam. Will use some of the techniques mentioned. 👍

", "rawMarkdown": "Thanks Sanyam. Will use some of the techniques mentioned. 👍" }, { "id": 1717541, "postDate": "2022-03-10T02:55:16.023Z", "content": "

Nice Tricks, I will try it out.

", "rawMarkdown": "Nice Tricks, I will try it out." }, { "id": 1716832, "postDate": "2022-03-09T12:51:32.700Z", "content": "

Thank you, I learnt a lot from your post, but I'm still a little confused about normalising images when inferencing-same applies to any image augmentations too. I do not understand how the topic of image normalization is specifically related to image augmentations. What should we pay attention to about image normalization when we are using TTA?

", "rawMarkdown": "Thank you, I learnt a lot from your post, but I'm still a little confused about normalising images when inferencing-same applies to any image augmentations too. I do not understand how the topic of image normalization is specifically related to image augmentations. What should we pay attention to about image normalization when we are using TTA?" }, { "id": 1716822, "postDate": "2022-03-09T12:34:25.800Z", "content": "

Wow! Thank you Sanyam! This helps alot !! Great work!

", "rawMarkdown": "Wow! Thank you Sanyam! This helps alot !! Great work!" }, { "id": 1716087, "postDate": "2022-03-08T16:27:17.790Z", "content": "

This is very useful. Thanks for sharing, Sanyam! 👍

", "rawMarkdown": "This is very useful. Thanks for sharing, Sanyam! 👍" }, { "id": 1715497, "postDate": "2022-03-08T04:24:08.383Z", "content": "

Thank you Sanyam for sharing it with the community ^_^

", "rawMarkdown": "Thank you Sanyam for sharing it with the community ^_^" }, { "id": 1714555, "postDate": "2022-03-07T05:44:17.783Z", "content": "

This is very useful. Thanks for sharing!

", "rawMarkdown": "This is very useful. Thanks for sharing!" }, { "id": 1713769, "postDate": "2022-03-06T11:19:48.550Z", "content": "

very useful help in implementation

", "rawMarkdown": "very useful help in implementation" }, { "id": 1713756, "postDate": "2022-03-06T10:50:34.100Z", "content": "

Nice tricks. 👍

", "rawMarkdown": "Nice tricks. 👍" }, { "id": 1760895, "postDate": "2022-04-19T15:19:25.153Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1754742, "postDate": "2022-04-14T00:29:14.723Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!" }, { "id": 1726157, "postDate": "2022-03-17T17:34:36.063Z", "content": "

Thanks it's very useful!!

", "rawMarkdown": "Thanks it's very useful!!" }, { "id": 1722246, "postDate": "2022-03-14T10:59:49.957Z", "content": "

Thanks for pointing this out!

", "rawMarkdown": "Thanks for pointing this out!" }, { "id": 1719987, "postDate": "2022-03-12T10:46:37.173Z", "content": "

Thanks for sharing.

", "rawMarkdown": "Thanks for sharing." }, { "id": 1719894, "postDate": "2022-03-12T08:27:14.323Z", "content": "

Great pointers. Thanks Sanyam Bhutani

", "rawMarkdown": "Great pointers. Thanks Sanyam Bhutani" }, { "id": 1719773, "postDate": "2022-03-12T05:27:37.890Z", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing" }, { "id": 1719761, "postDate": "2022-03-12T04:59:08.713Z", "content": "

Thanks for sharing 👍

", "rawMarkdown": "Thanks for sharing 👍" }, { "id": 1719404, "postDate": "2022-03-11T18:10:40.067Z", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing" }, { "id": 1718997, "postDate": "2022-03-11T11:52:14.207Z", "content": "

very useful . thanks for sharing

", "rawMarkdown": "very useful . thanks for sharing" }, { "id": 1718619, "postDate": "2022-03-11T02:37:53.217Z", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing" }, { "id": 1715621, "postDate": "2022-03-08T07:36:08.093Z", "content": "

Thanks Sanyam!

", "rawMarkdown": "Thanks Sanyam!" }, { "id": 1715567, "postDate": "2022-03-08T06:45:30.207Z", "content": "

Great post!
\nThanks for sharing.

", "rawMarkdown": "Great post!\nThanks for sharing." }, { "id": 1715491, "postDate": "2022-03-08T04:21:47.527Z", "content": "

Thanks for sharing.👍

", "rawMarkdown": "Thanks for sharing.👍" }, { "id": 1715289, "postDate": "2022-03-07T20:19:04.827Z", "content": "

Very informative, thanks for sharing.

", "rawMarkdown": "Very informative, thanks for sharing." }, { "id": 1714475, "postDate": "2022-03-07T03:39:25.667Z", "content": "

thanks for sharing

", "rawMarkdown": "thanks for sharing" }, { "id": 1714008, "postDate": "2022-03-06T14:47:54.867Z", "content": "

Thanks for sharing!

", "rawMarkdown": "Thanks for sharing!" } ], "comments": [ { "id": 1714410, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-03-07T01:20:37.237000", "content": "

Great pointers. Thanks Sanyam!

", "votes": 3, "replies": [ { "id": 1714468, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2022-03-07T03:25:08.093000", "content": "

Thanks so much, Chris! And thank you for the awesome PsuedoLabelling Notebook 🙏

", "votes": 3, "replies": [] } ] }, { "id": 1713789, "author_name": "Aleksey Alekseev", "author_url": "", "post_date": "2022-03-06T11:47:22.773000", "content": "

Does anyone have success with GeM in this task? In my case it works worse than average

", "votes": 3, "replies": [ { "id": 1714467, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2022-03-07T03:24:39.650000", "content": "

I'm still working on it but I read some discussions mention having success with it

", "votes": 1, "replies": [] }, { "id": 1725028, "author_name": "Ayushman Buragohain", "author_url": "", "post_date": "2022-03-16T18:40:15.877000", "content": "

GeM is a slightly better for me. But I haven't yet explored this thoroughly (So i might be wrong)

", "votes": 0, "replies": [] } ] }, { "id": 1713655, "author_name": "viraj kadam", "author_url": "", "post_date": "2022-03-06T08:47:01.390000", "content": "

This is very useful. thanks for sharing!

", "votes": 4, "replies": [] }, { "id": 1723767, "author_name": "Marius Wanko", "author_url": "", "post_date": "2022-03-15T17:39:47.603000", "content": "

Thanks for sharing!
\nI'm also lazy, but instead of nn.LazyLinear, I use nn.Identity on the head layer(s), like so

\n
def skip_head(m):\n    for attr in ['classifier', 'global_pool', 'head', 'fc']:\n        if hasattr(m, attr): setattr(m, attr, nn.Identity())\n    return m\n\nbody = skip_head(model)\n
\n

Then I add my custom head. But for the latter, you need to infer in_features from the model. If you have no BN, just a simple output layer, nn.LazyLinear does that for you.

", "votes": 1, "replies": [ { "id": 1726015, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2022-03-17T15:25:00.443000", "content": "

Thanks so much for sharing! I will definitely use this in my experiments!

", "votes": 0, "replies": [] } ] }, { "id": 1722856, "author_name": "Omar Khaled", "author_url": "", "post_date": "2022-03-14T23:27:43.617000", "content": "

This is very useful. thanks for sharing!

", "votes": 2, "replies": [] }, { "id": 1719960, "author_name": "enock nkuya", "author_url": "", "post_date": "2022-03-12T10:06:42.593000", "content": "

Thanks for sharing mate, its worthy checking

", "votes": 0, "replies": [] }, { "id": 1719343, "author_name": "Amin Fellah", "author_url": "", "post_date": "2022-03-11T16:46:32.313000", "content": "

how can I increase the contrast in images in the preprocessing phase ?

", "votes": 0, "replies": [] }, { "id": 1718928, "author_name": "Zakhar Dmitriev", "author_url": "", "post_date": "2022-03-11T09:56:21.657000", "content": "

Seems useful.Thx

", "votes": 0, "replies": [] }, { "id": 1718272, "author_name": "Samyuktha Mobile", "author_url": "", "post_date": "2022-03-10T17:21:19.387000", "content": "

Thanks Sanyam. Will use some of the techniques mentioned. 👍

", "votes": 0, "replies": [] }, { "id": 1717541, "author_name": "Rafael Rodrigues", "author_url": "", "post_date": "2022-03-10T02:55:16.023000", "content": "

Nice Tricks, I will try it out.

", "votes": 0, "replies": [] }, { "id": 1716832, "author_name": "Trunway", "author_url": "", "post_date": "2022-03-09T12:51:32.700000", "content": "

Thank you, I learnt a lot from your post, but I'm still a little confused about normalising images when inferencing-same applies to any image augmentations too. I do not understand how the topic of image normalization is specifically related to image augmentations. What should we pay attention to about image normalization when we are using TTA?

", "votes": 0, "replies": [] }, { "id": 1716822, "author_name": "PEvagelidakis", "author_url": "", "post_date": "2022-03-09T12:34:25.800000", "content": "

Wow! Thank you Sanyam! This helps alot !! Great work!

", "votes": 0, "replies": [] }, { "id": 1716087, "author_name": "Olga Vainer", "author_url": "", "post_date": "2022-03-08T16:27:17.790000", "content": "

This is very useful. Thanks for sharing, Sanyam! 👍

", "votes": 0, "replies": [] }, { "id": 1715497, "author_name": "SJ", "author_url": "", "post_date": "2022-03-08T04:24:08.383000", "content": "

Thank you Sanyam for sharing it with the community ^_^

", "votes": 0, "replies": [] }, { "id": 1714555, "author_name": "Artem Burenok", "author_url": "", "post_date": "2022-03-07T05:44:17.783000", "content": "

This is very useful. Thanks for sharing!

", "votes": 0, "replies": [] }, { "id": 1713769, "author_name": "gaurav", "author_url": "", "post_date": "2022-03-06T11:19:48.550000", "content": "

very useful help in implementation

", "votes": 0, "replies": [] }, { "id": 1713756, "author_name": "Ishan Mehta115", "author_url": "", "post_date": "2022-03-06T10:50:34.100000", "content": "

Nice tricks. 👍

", "votes": 0, "replies": [] }, { "id": 1760895, "author_name": "", "author_url": "", "post_date": "2022-04-19T15:19:25.153000", "content": "", "votes": 0, "replies": [] }, { "id": 1754742, "author_name": "", "author_url": "", "post_date": "2022-04-14T00:29:14.723000", "content": "

Thanks for sharing!

", "votes": 0, "replies": [] }, { "id": 1726157, "author_name": "Mirette Moawad", "author_url": "", "post_date": "2022-03-17T17:34:36.063000", "content": "

Thanks it's very useful!!

", "votes": 0, "replies": [] }, { "id": 1722246, "author_name": "LG", "author_url": "", "post_date": "2022-03-14T10:59:49.957000", "content": "

Thanks for pointing this out!

", "votes": 0, "replies": [] }, { "id": 1719987, "author_name": "SB Wani", "author_url": "", "post_date": "2022-03-12T10:46:37.173000", "content": "

Thanks for sharing.

", "votes": 0, "replies": [] }, { "id": 1719894, "author_name": "Usama Ahmed", "author_url": "", "post_date": "2022-03-12T08:27:14.323000", "content": "

Great pointers. Thanks Sanyam Bhutani

", "votes": 0, "replies": [] }, { "id": 1719773, "author_name": "Chanukya Vardhan", "author_url": "", "post_date": "2022-03-12T05:27:37.890000", "content": "

Thanks for sharing

", "votes": 0, "replies": [] }, { "id": 1719761, "author_name": "shaho h.panahi", "author_url": "", "post_date": "2022-03-12T04:59:08.713000", "content": "

Thanks for sharing 👍

", "votes": 0, "replies": [] }, { "id": 1719404, "author_name": "sarthak1620", "author_url": "", "post_date": "2022-03-11T18:10:40.067000", "content": "

Thanks for sharing

", "votes": 0, "replies": [] }, { "id": 1718997, "author_name": "surya prakash", "author_url": "", "post_date": "2022-03-11T11:52:14.207000", "content": "

very useful . thanks for sharing

", "votes": 0, "replies": [] }, { "id": 1718619, "author_name": "Usama Ansari", "author_url": "", "post_date": "2022-03-11T02:37:53.217000", "content": "

Thanks for sharing

", "votes": 0, "replies": [] }, { "id": 1715621, "author_name": "Marc Junior Nkengue", "author_url": "", "post_date": "2022-03-08T07:36:08.093000", "content": "

Thanks Sanyam!

", "votes": 0, "replies": [] }, { "id": 1715567, "author_name": "", "author_url": "", "post_date": "2022-03-08T06:45:30.207000", "content": "", "votes": 0, "replies": [] }, { "id": 1715491, "author_name": "", "author_url": "", "post_date": "2022-03-08T04:21:47.527000", "content": "", "votes": 0, "replies": [] }, { "id": 1715289, "author_name": "", "author_url": "", "post_date": "2022-03-07T20:19:04.827000", "content": "", "votes": 0, "replies": [] }, { "id": 1714475, "author_name": "", "author_url": "", "post_date": "2022-03-07T03:39:25.667000", "content": "", "votes": 0, "replies": [] }, { "id": 1714008, "author_name": "", "author_url": "", "post_date": "2022-03-06T14:47:54.867000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1712996": "Hey Everybody! \n\nSince I posted the previous collection of tricks/tips, I have been on a quest to add more suggestions and I have been able to formulate a few more suggestions. These are tricks from papers and by kagglers that might help your score in this competition. Here's the list of 7 more tricks:\n\n### 1. Test Time Augmentation (TTA):\n\nWhen training our models, we apply a few transformations to our training set: Image Augmentation, normalization, etc \n\nSo now our model is used to learn from the \"transformed\" dataset. Ex Phalanx has created this wonderful dataset [here](https://www.kaggle.com/phalanx/whale2-cropped-dataset) that crops into images. Now our model is used to detect whales in cropped images. So, we would like to have a similar test dataset as well. \n\nApplying Augmentation techniques while inferencing is known as TTA. One of my most frequent mistakes is forgetting to normalise images when inferencing-same applies to any image augmentations too. \n\n### 2. Sequential Unfreezing while Transfer Learning:\n\nI learned this trick during the fastai course. When we are performing transfer learning, our model has already captured a lot of information. \n\nThe initial layers (Layers close to inputs) retain more info about the structure of objects, etc and the latter layers (close to output) learn more about the dataset. We can envision our model to be grouped in layers like so:\n\n`Input(Group) -> HiddenSetEarly -> HiddenSetLater -> Output(Group)`\n\nWhen performing transfer learning, its usually a good idea to just train the last few layers and then unfreeze the earlier layers **sequentially**\n\n### 3. Differential Learning Rates: \n\nContinuing with the previous point, another trick I learned via fastai:\n\nThe initial few layers need little to no re-training, so applying differential learning rates to a different group of CNN layers is a great idea:\n\nEx: \nOutput(Group): Lr = 10e-3\nHiddenSetLater: LR = 0.5 * 10e-4\nInput(Group): Lr = 10e-5\n\nThis would make minimal changes to the initial layers and more changes to the head (output) layers making our model converge a bit faster\n\n### 4. PyTorch: use LazyLayers\n\nNote: I learned this trick thanks to Datasaurus, please see his post [here](https://www.kaggle.com/c/petfinder-pawpularity-score/discussion/279460) \n\nTL;DR when you perform transfer learning using a backbone, you have to re-write the last few layers with the dimensions in mind. For lazy folks like me, PyTorch has a LazyLayer (pun intended) to make it easier:\n\n`self.fc = nn.LazyLinear(self.cfg.target_size)`\n\nNote: This would otherwise be `self.fc = nn.Linear(self.n_features, self.cfg.target_size)`\n\nOnce again, thanks Datasaurus for sharing this in his original post\n\n### 5. Label Smoothing:\n\nWe have seen a lot of discussions in this competition about the funny images that exist in the dataset. This is not uncommon, ImageNet and many datasets themselves have many \"mislabeled\" images. The trick to helping here is using Label Smoothing. \n\n[This](https://paperswithcode.com/method/label-smoothing) is a good writeup about how it works. TL;DR: Adding noise to all labels helps our model generalize better\n\n### 6. Use GeM Pooling + ArcFace:\n\nDebarshi has a fantastic starter pack [here](https://www.kaggle.com/debarshichanda/pytorch-arcface-gem-pooling-starter) that uses GeM pooling + ArcFace. \n\nI had covered this in a live stream, the short version: If you are working with a large number of labels and images in an imbalanced dataset:\n\nReplacing your Max/AvgPooling layer with a GeM pooling layer and ArcFace loss is effective for such cases. I plan to cover this in more detail soon.\n\n### 7. PsuedoLabelling:\n\nPsuedoLabelling involves a form of semi-supervised learning. Chris Deotte teaches this in his fantastic kernel [here](https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969)\n\nIf you want to utilise more labels in training: You grab the most confident predictions from your model and then add them to your training dataset and build a new model. \n\nLike always, for anyone that has read this far, I have 1 bonus tip this time:\n\n### Bonus Tip: Use SSDs for image datasets\n\nAgain, I respectfully understand many kagglers use cloud hardware and don't have to worry about this step. However, if you've invested in local setups, I was for a long time not worrying about where I store my datasets. This would especially create IO bottlenecks for image datasets. \n\nAlways, consider storing your datasets on an M.2 Drive or SSD when you are reading them into a model so that you don't see a bottleneck. \n\nI hope these tips are helpful! \nSee you on the LB! :)", "1714410": "Great pointers. Thanks Sanyam!", "1713789": "Does anyone have success with GeM in this task? In my case it works worse than average", "1713655": "This is very useful. thanks for sharing!", "1723767": "Thanks for sharing! \nI'm also lazy, but instead of `nn.LazyLinear`, I use `nn.Identity` on the head layer(s), like so\n\n```\ndef skip_head(m):\n for attr in ['classifier', 'global_pool', 'head', 'fc']:\n if hasattr(m, attr): setattr(m, attr, nn.Identity())\n return m\n\nbody = skip_head(model)\n```\nThen I add my custom head. But for the latter, you need to infer `in_features` from the model. If you have no BN, just a simple output layer, `nn.LazyLinear` does that for you.", "1722856": "This is very useful. thanks for sharing!", "1719960": "Thanks for sharing mate, its worthy checking", "1719343": "how can I increase the contrast in images in the preprocessing phase ?", "1718928": "Seems useful.Thx", "1718272": "Thanks Sanyam. Will use some of the techniques mentioned. 👍", "1717541": "Nice Tricks, I will try it out.", "1716832": "Thank you, I learnt a lot from your post, but I'm still a little confused about normalising images when inferencing-same applies to any image augmentations too. I do not understand how the topic of image normalization is specifically related to image augmentations. What should we pay attention to about image normalization when we are using TTA?", "1716822": "Wow! Thank you Sanyam! This helps alot !! Great work!", "1716087": "This is very useful. Thanks for sharing, Sanyam! 👍", "1715497": "Thank you Sanyam for sharing it with the community ^_^", "1714555": "This is very useful. Thanks for sharing!", "1713769": "very useful help in implementation", "1713756": "Nice tricks. 👍", "1760895": "", "1754742": "Thanks for sharing!", "1726157": "Thanks it's very useful!!", "1722246": "Thanks for pointing this out!", "1719987": "Thanks for sharing.", "1719894": "Great pointers. Thanks Sanyam Bhutani", "1719773": "Thanks for sharing", "1719761": "Thanks for sharing 👍", "1719404": "Thanks for sharing", "1718997": "very useful . thanks for sharing", "1718619": "Thanks for sharing", "1715621": "Thanks Sanyam!", "1715567": "Great post!\nThanks for sharing.", "1715491": "Thanks for sharing.👍", "1715289": "Very informative, thanks for sharing.", "1714475": "thanks for sharing", "1714008": "Thanks for sharing!" } }