There are many misconceptions that you must either go one way or the other when choosing to use PyTorch or fastai, and fastai has too much magic to perfectly reproduce what you've done in PyTorch inside the framework. In my kernel here I did my best to show an example where that isn't true.
\nSome key factors people tend to glaze over (IMO):
\nCosineAnnealingWarmRestarts is equivalent to running fit_flat_cos with a start_pct of 0 (just cosine annealing schedule)I'd like to use this discussion as a Q/A from Kagglers for fastai-related implementations. What are you having trouble with? What could there be more examples of? Did this example notebook help clear your thoughts on a few things, or did it just enact more confusion?
\nThanks :)
", "messages": [ { "id": "1093718", "postDate": "11/28/2020 00:36:46", "content": "There are many misconceptions that you must either go one way or the other when choosing to use PyTorch or fastai, and fastai has too much magic to perfectly reproduce what you've done in PyTorch inside the framework. In my kernel here I did my best to show an example where that isn't true.
\nSome key factors people tend to glaze over (IMO):
\nCosineAnnealingWarmRestarts is equivalent to running fit_flat_cos with a start_pct of 0 (just cosine annealing schedule)I'd like to use this discussion as a Q/A from Kagglers for fastai-related implementations. What are you having trouble with? What could there be more examples of? Did this example notebook help clear your thoughts on a few things, or did it just enact more confusion?
\nThanks :)
", "rawMarkdown": "There are many misconceptions that you must either go one way or the other when choosing to use PyTorch or fastai, and fastai has too much magic to perfectly reproduce what you've done in PyTorch inside the framework. In my kernel [here](https://www.kaggle.com/muellerzr/recreating-abhishek-s-tez-with-fastai/) I did my best to show an example where that *isn't* true.\n\nSome key factors people tend to glaze over (IMO):\n* Make sure your augmentations are the same. Every library does things a bit differently. For example fastai does it's Hue and Saturation on RGB, while Albumentations does it on HSV. As a result the two have very different logics\n* Make sure your schedulers are the same. In the example here torch's `CosineAnnealingWarmRestarts` is equivalent to running `fit_flat_cos` with a `start_pct` of 0 (just cosine annealing schedule)\n\nI'd like to use this discussion as a Q/A from Kagglers for fastai-related implementations. What are you having trouble with? What could there be more examples of? Did this example notebook help clear your thoughts on a few things, or did it just enact more confusion?\n\nThanks :)", "votes": null }, { "id": "1093725", "postDate": "11/28/2020 00:59:13", "content": "I think another major difference between other approaches to TTA and the fastai's version is that fastai interpolates the x number of predictions with a prediction on the center crop while other TTA approaches don't include it at all (they only the x number of predictions on random augs of data)…
\nSetting beta=0 (I did this in my kernel) will change it so be similar behavior as most kaggle kernels (but it still runs the center crop).
", "rawMarkdown": "I think another major difference between other approaches to TTA and the fastai's version is that fastai interpolates the x number of predictions with a prediction on the center crop while other TTA approaches don't include it at all (they only the x number of predictions on random augs of data)... \n\nSetting beta=0 (I did this in my [kernel](https://www.kaggle.com/tanlikesmath/cassava-classification-eda-fastai-starter)) will change it so be similar behavior as most kaggle kernels (but it still runs the center crop).", "votes": null }, { "id": "1094215", "postDate": "11/28/2020 12:17:41", "content": "@muellerzr I know this is beat to death but I'd like to see the learning rate be more science than art (or magic ;)). Might be fun and useful to build this unless I'm missing it and it's been built but I haven't seen it incorporated in any notebooks.
\nAnd, where is there more information on the wwf library you created? I was on your page but maybe I'm missing what it does? Is it pulling in all the functions to run timm?
", "rawMarkdown": "muellerzr I know this is beat to death but I'd like to see the learning rate be more science than art (or magic ;)). Might be fun and useful to build this unless I'm missing it and it's been built but I haven't seen it incorporated in any notebooks. \n\nAnd, where is there more information on the wwf library you created? I was on your page but maybe I'm missing what it does? Is it pulling in all the functions to run timm?", "votes": null }, { "id": "1094216", "postDate": "11/28/2020 12:19:08", "content": "Really good notebook btw. I was just about to build one from scratch and saw yours.. Always great to learn a new thing or two! And love that you and Zach used Timm.
", "rawMarkdown": "Really good notebook btw. I was just about to build one from scratch and saw yours.. Always great to learn a new thing or two! And love that you and Zach used Timm.", "votes": null }, { "id": "1094390", "postDate": "11/28/2020 15:22:03", "content": "Well, it is science! It's a published paper: https://arxiv.org/abs/1506.01186. The science is picking the steepest slope found. We can algorithmically get this number back, which there are a number of different methods to do so. fastai has suggestions=True, and we have another example by Andrew Chang which goes into even more detail: https://forums.fast.ai/t/automated-learning-rate-suggester/44199
", "rawMarkdown": "Well, it is science! It's a published paper: https://arxiv.org/abs/1506.01186. The science is picking the steepest slope found. We can algorithmically get this number back, which there are a number of different methods to do so. fastai has suggestions=True, and we have another example by Andrew Chang which goes into even more detail: https://forums.fast.ai/t/automated-learning-rate-suggester/44199", "votes": null }, { "id": "1094391", "postDate": "11/28/2020 15:23:32", "content": "walkwithfastai.com I wrote the timm example to show how fastai is working internally with its cnn_learner and adapting it for timm. The same code covered in the timm article is the ones being exported to wwf.timm through the power of nbdev
Awesome. I will give that a try too.. I was adding the functions. But great that this was added. Appreciate the work.
\nI had come across Andrew Chang's code. I'll put that into a version of my notebook :)
", "rawMarkdown": "Awesome. I will give that a try too.. I was adding the functions. But great that this was added. Appreciate the work. \n\nI had come across Andrew Chang's code. I'll put that into a version of my notebook :)", "votes": null }, { "id": "1136881", "postDate": "01/03/2021 14:07:23", "content": "Did anybody get TPUs to work with fastai? If so, do you upgrade to fastai2 on TPU-environment or downgrade to fastai1 on GPU-env. Or can I even use a learn.export()ed & load_learner across both versions?
", "rawMarkdown": "Did anybody get TPUs to work with fastai? If so, do you upgrade to fastai2 on TPU-environment or downgrade to fastai1 on GPU-env. Or can I even use a learn.export()ed & load_learner across both versions?", "votes": null }, { "id": "1137358", "postDate": "01/03/2021 21:35:29", "content": "fastai doesn't currently have TPU support (but it is in somewhat active development).
", "rawMarkdown": "fastai doesn't currently have TPU support (but it is in somewhat active development).", "votes": null }, { "id": "1137382", "postDate": "01/03/2021 22:03:49", "content": "Just found this: https://www.kaggle.com/johnyquest/tpu-fastai-notebook
\nBut it didn't improve speed (maybe because BS is small in the example.).
I think another major difference between other approaches to TTA and the fastai's version is that fastai interpolates the x number of predictions with a prediction on the center crop while other TTA approaches don't include it at all (they only the x number of predictions on random augs of data)…
\nSetting beta=0 (I did this in my kernel) will change it so be similar behavior as most kaggle kernels (but it still runs the center crop).
", "votes": null, "replies": [ { "id": 1094216, "author_name": "crained", "author_url": "", "post_date": "11/28/2020 12:19:08", "content": "Really good notebook btw. I was just about to build one from scratch and saw yours.. Always great to learn a new thing or two! And love that you and Zach used Timm.
", "votes": null, "replies": [] } ] }, { "id": 1094215, "author_name": "crained", "author_url": "", "post_date": "11/28/2020 12:17:41", "content": "@muellerzr I know this is beat to death but I'd like to see the learning rate be more science than art (or magic ;)). Might be fun and useful to build this unless I'm missing it and it's been built but I haven't seen it incorporated in any notebooks.
\nAnd, where is there more information on the wwf library you created? I was on your page but maybe I'm missing what it does? Is it pulling in all the functions to run timm?
", "votes": null, "replies": [ { "id": 1094390, "author_name": "muellerzr", "author_url": "", "post_date": "11/28/2020 15:22:03", "content": "Well, it is science! It's a published paper: https://arxiv.org/abs/1506.01186. The science is picking the steepest slope found. We can algorithmically get this number back, which there are a number of different methods to do so. fastai has suggestions=True, and we have another example by Andrew Chang which goes into even more detail: https://forums.fast.ai/t/automated-learning-rate-suggester/44199
", "votes": null, "replies": [] }, { "id": 1094391, "author_name": "muellerzr", "author_url": "", "post_date": "11/28/2020 15:23:32", "content": "walkwithfastai.com I wrote the timm example to show how fastai is working internally with its cnn_learner and adapting it for timm. The same code covered in the timm article is the ones being exported to wwf.timm through the power of nbdev
Awesome. I will give that a try too.. I was adding the functions. But great that this was added. Appreciate the work.
\nI had come across Andrew Chang's code. I'll put that into a version of my notebook :)
", "votes": null, "replies": [] } ] }, { "id": 1136881, "author_name": "joatom", "author_url": "", "post_date": "01/03/2021 14:07:23", "content": "Did anybody get TPUs to work with fastai? If so, do you upgrade to fastai2 on TPU-environment or downgrade to fastai1 on GPU-env. Or can I even use a learn.export()ed & load_learner across both versions?
", "votes": null, "replies": [ { "id": 1137358, "author_name": "tanlikesmath", "author_url": "", "post_date": "01/03/2021 21:35:29", "content": "fastai doesn't currently have TPU support (but it is in somewhat active development).
", "votes": null, "replies": [] }, { "id": 1137382, "author_name": "joatom", "author_url": "", "post_date": "01/03/2021 22:03:49", "content": "Just found this: https://www.kaggle.com/johnyquest/tpu-fastai-notebook
\nBut it didn't improve speed (maybe because BS is small in the example.).