Hello!
\nI wanted to create a post sharing some general tips & suggestions that might help improve training speeds & accuracy, I have learned these via courses, reading top writeups, or papers, and plan to try all of these in this comp.
\nWithout further ado:
\nThe first two tips are focused on allowing faster prototyping-the more ideas you can try, the more chances you have to score better. To iterate faster, we need to start small to keep our training times small:
\nAyush has kindly created a Datasets thread here that points to all the datasets shared. Starting with a small dataset size allows you to iterate faster.
\nSince you'll be working with smaller GPU memory usage, you can increase batch_size and also iterate faster. Once you're confident in your ideas and see consistent score improvements, you can scale to larger image sizes.
Continuing with the previous line, you should start with just a small number of classes or examples and validate your training models there.
\nEx: Train on 10 classes, check if it improves CV -> Submit
\nScale idea to 20 classes, check CV, and submit again
If all goes well, train on the complete dataset.
\nWho doesn't want up to 50% faster training?
\nNVIDIA GPUs have Tensor-Cores which offer huge speedups when using \"Half-Precision\" Tensors. I have written a more detailed blog here, the short version is to try using fp_16 training to observe speedups on any GPU (and TPU!)
Kaggle offers 20 hours of TPUs every week. TPUs have 8 cores, which allow your batch_sizes to be scaled by a factor of 8. This allows for much faster training and faster iteration.
\nNote: I have recently discovered Hugging Face Accelerate which claims to give you easy workflow on TPUs with PyTorch too
\nThis idea IIRC was introduced in the Efficientnet papers and also taught in the fastai courses.
\nChris Deotte has a fantastic post talking about CNN Input image sizes. This blog teaches you how progressive resizing works in fastai. TL;DR:
\nThis process allows much faster convergence and better performance
\nI believe this concept was introduced in the MobileNet paper first and I saw it resurface in a recent discussion related to ConvNext architectures. Depthwise Convolutions have fewer filters and hence train faster.
\nSee here for some tips on making it work in PyTorch
\nChanging your learning_rate during the training of your model:
A slow lr takes too long and fast lr might not help your model converge, using this logic, we should use dynamic learning rates.
\nThere are many schedulers that allow this: I would recommend using fastai and its fine_tune() or fit_one_cycle() function. See here for more details.
This one is in-line with the previous one:
\nFrom the paper, \"Bag of Tricks\", one of the ticks highlights using LR warmup:
\nWhen you start training a model, it has more \"randomness\" as it's just starting to learn features, hence starting with a smaller learning_rate first allows it to pick details, and later you can increase it to the expected schedule or value after the \"warmup\" epochs are done and your model has learned some details.
NNs benefit from more data. A slight change in an image can really help a model improve its understanding of features inside of an image.
\nUsing correct image augmentations can really help your model. I had posted a nb sharing fastai image augmentations tutorial, I will be sharing an updated one for this competition soon if it's helpful.
\nChris Deotte in his recent CTDS interview shared some secrets. Qishen Ha, whose team had won the TF GBR competition also shared some tips of making these work
\nTL;DR of both: Try a lot of experiments and try as many augmentations. Start with augmentations off and then add them one by one to see if your training improves.
\nAlso, visualise results as you train models to make sure they're learning about the whales and not backgrounds!
\nI have two more bonus suggestions for anyone that has read this far :)
\nTimm and Tfimm, the latter being a TF-port of the former is a fantastic resource! Ross, posts almost all the cutting edge model weights along with extremely optimised training methods. I would highly recommend also spending time digging into their source code but at the least using the library is a solid suggestion for anyone working on CV problems
\nI understand many people are using Kaggle kernels and Colab for training. However, if you've invested in local hardware, Ross had taught in a thread on Twitter that the NGC Containers for PyTorch are very optimised and offer speedups
\nI hope you find these helpful and also find some training or score boosts! :)
\nHappy Kaggling!
", "messages": [ { "id": 1706561, "postDate": "2022-02-27T15:19:39.993Z", "content": "Hello!
\nI wanted to create a post sharing some general tips & suggestions that might help improve training speeds & accuracy, I have learned these via courses, reading top writeups, or papers, and plan to try all of these in this comp.
\nWithout further ado:
\nThe first two tips are focused on allowing faster prototyping-the more ideas you can try, the more chances you have to score better. To iterate faster, we need to start small to keep our training times small:
\nAyush has kindly created a Datasets thread here that points to all the datasets shared. Starting with a small dataset size allows you to iterate faster.
\nSince you'll be working with smaller GPU memory usage, you can increase batch_size and also iterate faster. Once you're confident in your ideas and see consistent score improvements, you can scale to larger image sizes.
Continuing with the previous line, you should start with just a small number of classes or examples and validate your training models there.
\nEx: Train on 10 classes, check if it improves CV -> Submit
\nScale idea to 20 classes, check CV, and submit again
If all goes well, train on the complete dataset.
\nWho doesn't want up to 50% faster training?
\nNVIDIA GPUs have Tensor-Cores which offer huge speedups when using \"Half-Precision\" Tensors. I have written a more detailed blog here, the short version is to try using fp_16 training to observe speedups on any GPU (and TPU!)
Kaggle offers 20 hours of TPUs every week. TPUs have 8 cores, which allow your batch_sizes to be scaled by a factor of 8. This allows for much faster training and faster iteration.
\nNote: I have recently discovered Hugging Face Accelerate which claims to give you easy workflow on TPUs with PyTorch too
\nThis idea IIRC was introduced in the Efficientnet papers and also taught in the fastai courses.
\nChris Deotte has a fantastic post talking about CNN Input image sizes. This blog teaches you how progressive resizing works in fastai. TL;DR:
\nThis process allows much faster convergence and better performance
\nI believe this concept was introduced in the MobileNet paper first and I saw it resurface in a recent discussion related to ConvNext architectures. Depthwise Convolutions have fewer filters and hence train faster.
\nSee here for some tips on making it work in PyTorch
\nChanging your learning_rate during the training of your model:
A slow lr takes too long and fast lr might not help your model converge, using this logic, we should use dynamic learning rates.
\nThere are many schedulers that allow this: I would recommend using fastai and its fine_tune() or fit_one_cycle() function. See here for more details.
This one is in-line with the previous one:
\nFrom the paper, \"Bag of Tricks\", one of the ticks highlights using LR warmup:
\nWhen you start training a model, it has more \"randomness\" as it's just starting to learn features, hence starting with a smaller learning_rate first allows it to pick details, and later you can increase it to the expected schedule or value after the \"warmup\" epochs are done and your model has learned some details.
NNs benefit from more data. A slight change in an image can really help a model improve its understanding of features inside of an image.
\nUsing correct image augmentations can really help your model. I had posted a nb sharing fastai image augmentations tutorial, I will be sharing an updated one for this competition soon if it's helpful.
\nChris Deotte in his recent CTDS interview shared some secrets. Qishen Ha, whose team had won the TF GBR competition also shared some tips of making these work
\nTL;DR of both: Try a lot of experiments and try as many augmentations. Start with augmentations off and then add them one by one to see if your training improves.
\nAlso, visualise results as you train models to make sure they're learning about the whales and not backgrounds!
\nI have two more bonus suggestions for anyone that has read this far :)
\nTimm and Tfimm, the latter being a TF-port of the former is a fantastic resource! Ross, posts almost all the cutting edge model weights along with extremely optimised training methods. I would highly recommend also spending time digging into their source code but at the least using the library is a solid suggestion for anyone working on CV problems
\nI understand many people are using Kaggle kernels and Colab for training. However, if you've invested in local hardware, Ross had taught in a thread on Twitter that the NGC Containers for PyTorch are very optimised and offer speedups
\nI hope you find these helpful and also find some training or score boosts! :)
\nHappy Kaggling!
", "rawMarkdown": "Hello! \n\nI wanted to create a post sharing some general tips & suggestions that might help improve training speeds & accuracy, I have learned these via courses, reading top writeups, or papers, and plan to try all of these in this comp.\n\nWithout further ado:\n\n### 1. Start with Smaller Resolution:\n\nThe first two tips are focused on allowing faster prototyping-the more ideas you can try, the more chances you have to score better. To iterate faster, we need to start small to keep our training times small:\n\nAyush has kindly created a Datasets thread [here](https://www.kaggle.com/c/happy-whale-and-dolphin/discussion/309691) that points to all the datasets shared. Starting with a small dataset size allows you to iterate faster. \n\nSince you'll be working with smaller GPU memory usage, you can increase `batch_size` and also iterate faster. Once you're confident in your ideas and see consistent score improvements, you can scale to larger image sizes.\n\n### 2. Start with subsets of Data:\n\nContinuing with the previous line, you should start with just a small number of classes or examples and validate your training models there. \n\nEx: Train on 10 classes, check if it improves CV -> Submit\nScale idea to 20 classes, check CV, and submit again \n\nIf all goes well, train on the complete dataset.\n\n### 3. Use FP16 or Half-Precision Training:\n\nWho doesn't want up to 50% faster training? \n\nNVIDIA GPUs have Tensor-Cores which offer huge speedups when using \"Half-Precision\" Tensors. I have written a more detailed blog [here](https://hackernoon.com/rtx-2080ti-vs-gtx-1080ti-fastai-mixed-precision-training-comparisons-on-cifar-100-761d8f615d7f), the short version is to try using `fp_16` training to observe speedups on any GPU (and TPU!)\n\n### 4. Use TPUs:\n\nKaggle offers 20 hours of TPUs every week. TPUs have 8 cores, which allow your batch_sizes to be scaled by a factor of 8. This allows for much faster training and faster iteration. \n\nNote: I have recently discovered Hugging Face Accelerate which claims to give you easy workflow on TPUs with PyTorch too\n\n### 5. Progressive Resizing:\n\nThis idea IIRC was introduced in the Efficientnet papers and also taught in the fastai courses. \n\nChris Deotte has a fantastic [post](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147) talking about CNN Input image sizes. [This](https://medium.com/analytics-vidhya/novel-techniques-to-win-an-image-classification-hackathon-part-2-e33bf0ad5fe6) blog teaches you how progressive resizing works in fastai. TL;DR:\n\n- Train model on size: small\n- Save weights and re-train model on larger image size\n- Save weights again and re-train on final image sizes\n\nThis process allows much faster convergence and better performance \n\n### 6. Experiment: Depthwise Convs instead of Regular Convs:\n\nI believe this [concept](https://paperswithcode.com/method/depthwise-convolution) was introduced in the MobileNet paper first and I saw it resurface in a recent discussion related to ConvNext architectures. Depthwise Convolutions have fewer filters and hence train faster. \n\n[See here](https://discuss.pytorch.org/t/how-to-modify-a-conv2d-to-depthwise-separable-convolution/15843/4) for some tips on making it work in PyTorch\n\n### 7. LR Scheduler:\n\nChanging your `learning_rate` during the training of your model:\n\nA slow lr takes too long and fast lr might not help your model converge, using this logic, we should use dynamic learning rates.\n\nThere are many schedulers that allow this: I would recommend using `fastai` and its `fine_tune()` or `fit_one_cycle()` function. See [here](https://forums.fast.ai/t/fine-tune-vs-fit-one-cycle/66029) for more details. \n\n### 8. LR Warmup:\n\nThis one is in-line with the previous one:\n\nFrom the paper, [\"Bag of Tricks\"](https://arxiv.org/pdf/1812.01187.pdf), one of the ticks highlights using LR warmup:\n\nWhen you start training a model, it has more \"randomness\" as it's just starting to learn features, hence starting with a smaller `learning_rate` first allows it to pick details, and later you can increase it to the expected schedule or value after the \"warmup\" epochs are done and your model has learned some details.\n\n### 9. Image Augmentations:\n\nNNs benefit from more data. A slight change in an image can really help a model improve its understanding of features inside of an image. \n\nUsing correct image augmentations can really help your model. I had posted a nb sharing fastai image augmentations tutorial, I will be sharing an updated one for this competition soon if it's helpful.\n\nChris Deotte in his recent [CTDS interview](https://www.youtube.com/watch?v=XXmujwhjyIo) shared some secrets. Qishen Ha, whose team had won the TF GBR competition also shared [some tips](https://www.youtube.com/watch?v=8e-EIilkvL0) of making these work\n\nTL;DR of both: Try a lot of experiments and try as many augmentations. Start with augmentations off and then add them one by one to see if your training improves. \n\nAlso, visualise results as you train models to make sure they're learning about the whales and not backgrounds!\n\nI have two more bonus suggestions for anyone that has read this far :) \n\n### Bonus Tip #1: Use Timm or Tfimm:\n\n[Timm](https://github.com/rwightman/pytorch-image-models) and [Tfimm](https://github.com/martinsbruveris/tensorflow-image-models), the latter being a TF-port of the former is a fantastic resource! Ross, posts almost all the cutting edge model weights along with *extremely* optimised training methods. I would highly recommend also spending time digging into their source code but at the least using the library is a solid suggestion for anyone working on CV problems\n\n### Bonus Tip #2: Use NGC Containers for Local training: \n\nI understand many people are using Kaggle kernels and Colab for training. However, if you've invested in local hardware, [Ross](https://twitter.com/wightmanr) had taught in a thread on Twitter that the [NGC Containers](https://catalog.ngc.nvidia.com) for PyTorch are very optimised and offer speedups\n\nI hope you find these helpful and also find some training or score boosts! :)\n\nHappy Kaggling!", "votes": 168 }, { "id": 2185594, "postDate": "2023-03-17T07:18:03.950Z", "content": "Your observation is excellent @init27
", "rawMarkdown": "Your observation is excellent @init27 ", "votes": 2 }, { "id": 1709782, "postDate": "2022-03-02T13:39:22.903Z", "content": "One more idea about Image Augmentations is that , you can also use Image Augmentations during inference. Its very helpful for Objcet Detection tasks. I have found that by using few variants of validation image and then combining all the predicted bouding boxes, I could extract much better results from the same model.
I think you are referring to TTA(Test Time Augmentation). In my experiments TTA doesn't work every time, if your data is too noisy.
", "rawMarkdown": "I think you are referring to TTA(Test Time Augmentation). In my experiments TTA doesn't work every time, if your data is too noisy.", "votes": 1 } ] }, { "id": 1707312, "postDate": "2022-02-28T10:05:52.470Z", "content": "Nice work! I wish there more posts that share a lot of tricks in a single thread.
", "rawMarkdown": "Nice work! I wish there more posts that share a lot of tricks in a single thread.", "votes": 1, "replies": [ { "id": 1711858, "postDate": "2022-03-04T11:30:58.810Z", "content": "Thank you, CroDoc! π
\nI've found many posts scattered across different competitions, I'll try my best to collect ideas like so or maybe even livestream with some chai if time permits.
\nThese tricks are however really simple compared to the incredible ones shared by Kagglers in competitions
", "rawMarkdown": "Thank you, CroDoc! π\n\nI've found many posts scattered across different competitions, I'll try my best to collect ideas like so or maybe even livestream with some chai if time permits. \n\nThese tricks are however really simple compared to the incredible ones shared by Kagglers in competitions", "votes": 1 }, { "id": 1711921, "postDate": "2022-03-04T13:22:30.093Z", "content": "Ask @cdeotte to be on Chai again and share the best general tricks for computer vision π€
", "rawMarkdown": "Ask @cdeotte to be on Chai again and share the best general tricks for computer vision π€" } ] }, { "id": 1711488, "postDate": "2022-03-04T03:40:26.910Z", "content": "Great points @init27 as always!
\nbut I am quite skeptical of your 2nd point, that is use less classes then use more. I don't think it is a good idea, I think it will confuse the model when we will introduce more number of classes later.
\nAlthough I am not sure, can you a bit describe it more. Thanks
Thank you!
\nActually, maybe I didn't I explain it well-I'm not suggesting doing transfer learning with the weights.
\nI was envisioning this:
\nLet's say I want to try model X + Y tricks/augmentations/preprocessing: start on just 10 classes and submit to LB.
\nThen train the same pipeline (no transfer learning, train again) on 15 classes, resubmit-is there an improvement? Yes, then that means the idea works and ideally, this would really speed up the iteration speed compared to training on a large number of classes.
\nThis is again a small suggestion to change your iteration speed from training on complete dataset to the speed required to train on subsets.
", "rawMarkdown": "Thank you!\n\nActually, maybe I didn't I explain it well-I'm not suggesting doing transfer learning with the weights. \n\nI was envisioning this:\n\nLet's say I want to try model X + Y tricks/augmentations/preprocessing: start on just 10 classes and submit to LB. \n\nThen train the same pipeline (no transfer learning, train again) on 15 classes, resubmit-is there an improvement? Yes, then that means the idea works and ideally, this would really speed up the iteration speed compared to training on a large number of classes.\n\nThis is again a small suggestion to change your iteration speed from training on complete dataset to the speed required to train on subsets. ", "votes": 3 }, { "id": 1712515, "postDate": "2022-03-05T02:37:25.763Z", "content": "Ahh now I get it, thanks for the explanation, I think I should apply this.
", "rawMarkdown": "Ahh now I get it, thanks for the explanation, I think I should apply this." } ] }, { "id": 3213583, "postDate": "2025-05-30T07:52:40.070Z", "content": "This is packed with practical gold. I love the tips on progressive resizing and FP16. It is super helpful for speeding up experimentation without sacrificing performance. Thanks for sharing this!
", "rawMarkdown": "This is packed with practical gold. I love the tips on progressive resizing and FP16. It is super helpful for speeding up experimentation without sacrificing performance. Thanks for sharing this! " }, { "id": 3088500, "postDate": "2025-01-04T18:27:31.850Z", "content": "How many data is augmented? I think if we augmented all, it could memorize instead of learning pattern, right?
", "rawMarkdown": "How many data is augmented? I think if we augmented all, it could memorize instead of learning pattern, right?" }, { "id": 2512647, "postDate": "2023-11-04T18:08:02.290Z", "content": "Useful Info. Thanks for sharing these highly informative points.
", "rawMarkdown": "Useful Info. Thanks for sharing these highly informative points." }, { "id": 1943550, "postDate": "2022-09-17T15:35:57.097Z", "content": "Thanks for sharing. It is very helpful !
", "rawMarkdown": "Thanks for sharing. It is very helpful !" }, { "id": 1936530, "postDate": "2022-09-12T20:12:18.600Z", "content": "Informative, I will apply in future.
", "rawMarkdown": "Informative, I will apply in future." }, { "id": 1869398, "postDate": "2022-07-24T18:00:30.867Z", "content": "Really useful tips, thanks for sharing !
", "rawMarkdown": "Really useful tips, thanks for sharing !" }, { "id": 1725404, "postDate": "2022-03-17T04:25:26.800Z", "content": "Thank you thank you so much for sharing it. Really Helpful!!!!
", "rawMarkdown": "Thank you thank you so much for sharing it. Really Helpful!!!!" }, { "id": 1710695, "postDate": "2022-03-03T08:44:13.147Z", "content": "Interesting post. Thanks for sharing..:)
", "rawMarkdown": "Interesting post. Thanks for sharing..:)" }, { "id": 1710015, "postDate": "2022-03-02T17:42:22.677Z", "content": "Thanks for sharing these tricks and tips , it will really help us @init27
", "rawMarkdown": "Thanks for sharing these tricks and tips , it will really help us @init27 " }, { "id": 1709627, "postDate": "2022-03-02T11:28:31.367Z", "content": "Interesting post. Thanks for sharing..:)
", "rawMarkdown": "Interesting post. Thanks for sharing..:)" }, { "id": 1709023, "postDate": "2022-03-01T22:22:09.113Z", "content": "Amazing tips!
", "rawMarkdown": "Amazing tips!" }, { "id": 1708830, "postDate": "2022-03-01T18:25:55.707Z", "content": "Thanks for sharing these tricks. @init27 π
", "rawMarkdown": "Thanks for sharing these tricks. @init27 π" }, { "id": 1714033, "postDate": "2022-03-06T15:10:39.490Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1710836, "postDate": "2022-03-03T11:38:40.353Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3505915, "postDate": "2026-07-30T17:21:36.143Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" }, { "id": 1720037, "postDate": "2022-03-12T12:02:25.700Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" }, { "id": 1714303, "postDate": "2022-03-06T19:09:36.567Z", "content": "Thanks for sharing
", "rawMarkdown": "Thanks for sharing" }, { "id": 1713712, "postDate": "2022-03-06T09:58:12.683Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 1712898, "postDate": "2022-03-05T13:12:05.007Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing. " }, { "id": 1709597, "postDate": "2022-03-02T10:57:35.670Z", "content": "Thanks for sharing! π―
", "rawMarkdown": "Thanks for sharing! π―" }, { "id": 1709047, "postDate": "2022-03-01T22:56:43.927Z", "content": "I learned a lot. Thank you!
", "rawMarkdown": "I learned a lot. Thank you!" }, { "id": 1707530, "postDate": "2022-02-28T14:47:26.450Z", "content": "Thanks for these tricks.
", "rawMarkdown": "Thanks for these tricks." } ], "comments": [ { "id": 2185594, "author_name": "Yeakub Sadlil", "author_url": "", "post_date": "2023-03-17T07:18:03.950000", "content": "Your observation is excellent @init27
", "votes": 2, "replies": [] }, { "id": 1709782, "author_name": "Subhendu", "author_url": "", "post_date": "2022-03-02T13:39:22.903000", "content": "One more idea about Image Augmentations is that , you can also use Image Augmentations during inference. Its very helpful for Objcet Detection tasks. I have found that by using few variants of validation image and then combining all the predicted bouding boxes, I could extract much better results from the same model.
I think you are referring to TTA(Test Time Augmentation). In my experiments TTA doesn't work every time, if your data is too noisy.
", "votes": 1, "replies": [] } ] }, { "id": 1707312, "author_name": "CroDoc", "author_url": "", "post_date": "2022-02-28T10:05:52.470000", "content": "Nice work! I wish there more posts that share a lot of tricks in a single thread.
", "votes": 1, "replies": [ { "id": 1711858, "author_name": "Sanyam Bhutani", "author_url": "", "post_date": "2022-03-04T11:30:58.810000", "content": "Thank you, CroDoc! π
\nI've found many posts scattered across different competitions, I'll try my best to collect ideas like so or maybe even livestream with some chai if time permits.
\nThese tricks are however really simple compared to the incredible ones shared by Kagglers in competitions
", "votes": 1, "replies": [] }, { "id": 1711921, "author_name": "CroDoc", "author_url": "", "post_date": "2022-03-04T13:22:30.093000", "content": "Ask @cdeotte to be on Chai again and share the best general tricks for computer vision π€
", "votes": 0, "replies": [] } ] }, { "id": 1711488, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-03-04T03:40:26.910000", "content": "Great points @init27 as always!
\nbut I am quite skeptical of your 2nd point, that is use less classes then use more. I don't think it is a good idea, I think it will confuse the model when we will introduce more number of classes later.
\nAlthough I am not sure, can you a bit describe it more. Thanks
Thank you!
\nActually, maybe I didn't I explain it well-I'm not suggesting doing transfer learning with the weights.
\nI was envisioning this:
\nLet's say I want to try model X + Y tricks/augmentations/preprocessing: start on just 10 classes and submit to LB.
\nThen train the same pipeline (no transfer learning, train again) on 15 classes, resubmit-is there an improvement? Yes, then that means the idea works and ideally, this would really speed up the iteration speed compared to training on a large number of classes.
\nThis is again a small suggestion to change your iteration speed from training on complete dataset to the speed required to train on subsets.
", "votes": 3, "replies": [] }, { "id": 1712515, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-03-05T02:37:25.763000", "content": "Ahh now I get it, thanks for the explanation, I think I should apply this.
", "votes": 0, "replies": [] } ] }, { "id": 3213583, "author_name": "Cedric Ogiesoba-Eguakun", "author_url": "", "post_date": "2025-05-30T07:52:40.070000", "content": "This is packed with practical gold. I love the tips on progressive resizing and FP16. It is super helpful for speeding up experimentation without sacrificing performance. Thanks for sharing this!
", "votes": 0, "replies": [] }, { "id": 3088500, "author_name": "Thomas Artemius", "author_url": "", "post_date": "2025-01-04T18:27:31.850000", "content": "How many data is augmented? I think if we augmented all, it could memorize instead of learning pattern, right?
", "votes": 0, "replies": [] }, { "id": 2512647, "author_name": "Nithiyashree Venkatachalam Kannan", "author_url": "", "post_date": "2023-11-04T18:08:02.290000", "content": "Useful Info. Thanks for sharing these highly informative points.
", "votes": 0, "replies": [] }, { "id": 1943550, "author_name": "HanHungHsun", "author_url": "", "post_date": "2022-09-17T15:35:57.097000", "content": "Thanks for sharing. It is very helpful !
", "votes": 0, "replies": [] }, { "id": 1936530, "author_name": "Ahsan Zafar Mehmood", "author_url": "", "post_date": "2022-09-12T20:12:18.600000", "content": "Informative, I will apply in future.
", "votes": 0, "replies": [] }, { "id": 1869398, "author_name": "Mehdi Elion", "author_url": "", "post_date": "2022-07-24T18:00:30.867000", "content": "Really useful tips, thanks for sharing !
", "votes": 0, "replies": [] }, { "id": 1725404, "author_name": "Dev Khant", "author_url": "", "post_date": "2022-03-17T04:25:26.800000", "content": "Thank you thank you so much for sharing it. Really Helpful!!!!
", "votes": 0, "replies": [] }, { "id": 1710695, "author_name": "Nouran Elsayed(target)", "author_url": "", "post_date": "2022-03-03T08:44:13.147000", "content": "Interesting post. Thanks for sharing..:)
", "votes": 0, "replies": [] }, { "id": 1710015, "author_name": "Fozan", "author_url": "", "post_date": "2022-03-02T17:42:22.677000", "content": "Thanks for sharing these tricks and tips , it will really help us @init27
", "votes": 0, "replies": [] }, { "id": 1709627, "author_name": "Aadil Hussain", "author_url": "", "post_date": "2022-03-02T11:28:31.367000", "content": "Interesting post. Thanks for sharing..:)
", "votes": 0, "replies": [] }, { "id": 1709023, "author_name": "Junming Gao", "author_url": "", "post_date": "2022-03-01T22:22:09.113000", "content": "Amazing tips!
", "votes": 0, "replies": [] }, { "id": 1708830, "author_name": "Ishan Mehta115", "author_url": "", "post_date": "2022-03-01T18:25:55.707000", "content": "Thanks for sharing these tricks. @init27 π
", "votes": 0, "replies": [] }, { "id": 1714033, "author_name": "", "author_url": "", "post_date": "2022-03-06T15:10:39.490000", "content": "", "votes": 0, "replies": [] }, { "id": 1710836, "author_name": "", "author_url": "", "post_date": "2022-03-03T11:38:40.353000", "content": "", "votes": 0, "replies": [] }, { "id": 3505915, "author_name": "bluebblll", "author_url": "", "post_date": "2026-07-30T17:21:36.143000", "content": "Thanks for sharing
", "votes": 0, "replies": [] }, { "id": 1720037, "author_name": "cwh094", "author_url": "", "post_date": "2022-03-12T12:02:25.700000", "content": "Thanks for sharing
", "votes": 0, "replies": [] }, { "id": 1714303, "author_name": "younis shaik", "author_url": "", "post_date": "2022-03-06T19:09:36.567000", "content": "Thanks for sharing
", "votes": 0, "replies": [] }, { "id": 1713712, "author_name": "BoxuanZhang", "author_url": "", "post_date": "2022-03-06T09:58:12.683000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] }, { "id": 1712898, "author_name": "Δ°lker Kara", "author_url": "", "post_date": "2022-03-05T13:12:05.007000", "content": "Thanks for sharing.
", "votes": 0, "replies": [] }, { "id": 1709597, "author_name": "MαΊ‘nh Δα»", "author_url": "", "post_date": "2022-03-02T10:57:35.670000", "content": "Thanks for sharing! π―
", "votes": 0, "replies": [] }, { "id": 1709047, "author_name": "Moyashii", "author_url": "", "post_date": "2022-03-01T22:56:43.927000", "content": "I learned a lot. Thank you!
", "votes": 0, "replies": [] }, { "id": 1707530, "author_name": "Artem Burenok", "author_url": "", "post_date": "2022-02-28T14:47:26.450000", "content": "Thanks for these tricks.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1706561": "Hello! \n\nI wanted to create a post sharing some general tips & suggestions that might help improve training speeds & accuracy, I have learned these via courses, reading top writeups, or papers, and plan to try all of these in this comp.\n\nWithout further ado:\n\n### 1. Start with Smaller Resolution:\n\nThe first two tips are focused on allowing faster prototyping-the more ideas you can try, the more chances you have to score better. To iterate faster, we need to start small to keep our training times small:\n\nAyush has kindly created a Datasets thread [here](https://www.kaggle.com/c/happy-whale-and-dolphin/discussion/309691) that points to all the datasets shared. Starting with a small dataset size allows you to iterate faster. \n\nSince you'll be working with smaller GPU memory usage, you can increase `batch_size` and also iterate faster. Once you're confident in your ideas and see consistent score improvements, you can scale to larger image sizes.\n\n### 2. Start with subsets of Data:\n\nContinuing with the previous line, you should start with just a small number of classes or examples and validate your training models there. \n\nEx: Train on 10 classes, check if it improves CV -> Submit\nScale idea to 20 classes, check CV, and submit again \n\nIf all goes well, train on the complete dataset.\n\n### 3. Use FP16 or Half-Precision Training:\n\nWho doesn't want up to 50% faster training? \n\nNVIDIA GPUs have Tensor-Cores which offer huge speedups when using \"Half-Precision\" Tensors. I have written a more detailed blog [here](https://hackernoon.com/rtx-2080ti-vs-gtx-1080ti-fastai-mixed-precision-training-comparisons-on-cifar-100-761d8f615d7f), the short version is to try using `fp_16` training to observe speedups on any GPU (and TPU!)\n\n### 4. Use TPUs:\n\nKaggle offers 20 hours of TPUs every week. TPUs have 8 cores, which allow your batch_sizes to be scaled by a factor of 8. This allows for much faster training and faster iteration. \n\nNote: I have recently discovered Hugging Face Accelerate which claims to give you easy workflow on TPUs with PyTorch too\n\n### 5. Progressive Resizing:\n\nThis idea IIRC was introduced in the Efficientnet papers and also taught in the fastai courses. \n\nChris Deotte has a fantastic [post](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/160147) talking about CNN Input image sizes. [This](https://medium.com/analytics-vidhya/novel-techniques-to-win-an-image-classification-hackathon-part-2-e33bf0ad5fe6) blog teaches you how progressive resizing works in fastai. TL;DR:\n\n- Train model on size: small\n- Save weights and re-train model on larger image size\n- Save weights again and re-train on final image sizes\n\nThis process allows much faster convergence and better performance \n\n### 6. Experiment: Depthwise Convs instead of Regular Convs:\n\nI believe this [concept](https://paperswithcode.com/method/depthwise-convolution) was introduced in the MobileNet paper first and I saw it resurface in a recent discussion related to ConvNext architectures. Depthwise Convolutions have fewer filters and hence train faster. \n\n[See here](https://discuss.pytorch.org/t/how-to-modify-a-conv2d-to-depthwise-separable-convolution/15843/4) for some tips on making it work in PyTorch\n\n### 7. LR Scheduler:\n\nChanging your `learning_rate` during the training of your model:\n\nA slow lr takes too long and fast lr might not help your model converge, using this logic, we should use dynamic learning rates.\n\nThere are many schedulers that allow this: I would recommend using `fastai` and its `fine_tune()` or `fit_one_cycle()` function. See [here](https://forums.fast.ai/t/fine-tune-vs-fit-one-cycle/66029) for more details. \n\n### 8. LR Warmup:\n\nThis one is in-line with the previous one:\n\nFrom the paper, [\"Bag of Tricks\"](https://arxiv.org/pdf/1812.01187.pdf), one of the ticks highlights using LR warmup:\n\nWhen you start training a model, it has more \"randomness\" as it's just starting to learn features, hence starting with a smaller `learning_rate` first allows it to pick details, and later you can increase it to the expected schedule or value after the \"warmup\" epochs are done and your model has learned some details.\n\n### 9. Image Augmentations:\n\nNNs benefit from more data. A slight change in an image can really help a model improve its understanding of features inside of an image. \n\nUsing correct image augmentations can really help your model. I had posted a nb sharing fastai image augmentations tutorial, I will be sharing an updated one for this competition soon if it's helpful.\n\nChris Deotte in his recent [CTDS interview](https://www.youtube.com/watch?v=XXmujwhjyIo) shared some secrets. Qishen Ha, whose team had won the TF GBR competition also shared [some tips](https://www.youtube.com/watch?v=8e-EIilkvL0) of making these work\n\nTL;DR of both: Try a lot of experiments and try as many augmentations. Start with augmentations off and then add them one by one to see if your training improves. \n\nAlso, visualise results as you train models to make sure they're learning about the whales and not backgrounds!\n\nI have two more bonus suggestions for anyone that has read this far :) \n\n### Bonus Tip #1: Use Timm or Tfimm:\n\n[Timm](https://github.com/rwightman/pytorch-image-models) and [Tfimm](https://github.com/martinsbruveris/tensorflow-image-models), the latter being a TF-port of the former is a fantastic resource! Ross, posts almost all the cutting edge model weights along with *extremely* optimised training methods. I would highly recommend also spending time digging into their source code but at the least using the library is a solid suggestion for anyone working on CV problems\n\n### Bonus Tip #2: Use NGC Containers for Local training: \n\nI understand many people are using Kaggle kernels and Colab for training. However, if you've invested in local hardware, [Ross](https://twitter.com/wightmanr) had taught in a thread on Twitter that the [NGC Containers](https://catalog.ngc.nvidia.com) for PyTorch are very optimised and offer speedups\n\nI hope you find these helpful and also find some training or score boosts! :)\n\nHappy Kaggling!", "2185594": "Your observation is excellent @init27 ", "1709782": "One more idea about **Image Augmentations** is that , you can also use `Image Augmentations` during inference. Its very helpful for Objcet Detection tasks. I have found that by using few variants of validation image and then combining all the predicted bouding boxes, I could extract much better results from the same model. ", "1707312": "Nice work! I wish there more posts that share a lot of tricks in a single thread.", "1711488": "Great points @init27 as always!\nbut I am quite skeptical of your 2nd point, that is use less classes then use more. I don't think it is a good idea, I think it will confuse the model when we will introduce more number of classes later. \nAlthough I am not sure, can you a bit describe it more. Thanks", "3213583": "This is packed with practical gold. I love the tips on progressive resizing and FP16. It is super helpful for speeding up experimentation without sacrificing performance. Thanks for sharing this! ", "3088500": "How many data is augmented? I think if we augmented all, it could memorize instead of learning pattern, right?", "2512647": "Useful Info. Thanks for sharing these highly informative points.", "1943550": "Thanks for sharing. It is very helpful !", "1936530": "Informative, I will apply in future.", "1869398": "Really useful tips, thanks for sharing !", "1725404": "Thank you thank you so much for sharing it. Really Helpful!!!!", "1710695": "Interesting post. Thanks for sharing..:)", "1710015": "Thanks for sharing these tricks and tips , it will really help us @init27 ", "1709627": "Interesting post. Thanks for sharing..:)", "1709023": "Amazing tips!", "1708830": "Thanks for sharing these tricks. @init27 π", "1714033": "", "1710836": "", "3505915": "Thanks for sharing", "1720037": "Thanks for sharing", "1714303": "Thanks for sharing", "1713712": "Thanks for sharing.", "1712898": "Thanks for sharing. ", "1709597": "Thanks for sharing! π―", "1709047": "I learned a lot. Thank you!", "1707530": "Thanks for these tricks." } }