https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/nfnet.py
\nNFNet models were recently released in pytorch-image-models. These are only the model definitions and no imagenet pretrained weights are available at the moment, but potentially something to try in the final days of this competition.
\nHow to use timm on Kaggle:
\nAdd this dataset: https://www.kaggle.com/stanleyjzheng/timm-nfnet
import sys; sys.path.insert(0,'../input/timm-nfnet')\nimport timm\n\nmodel = timm.create_model('nfnet_f1', pretrained=False)\nI'm currently comparing NFNet-F0 to Effnet-B0 on VBD chest xray - I will update this thread with my results in a few hours. NFNet is not in the PyPi package yet, so I had to install from source. Will have results out on speed/vRAM usage soon.
\nEdit: added dataset for kaggle installation
", "messages": [ { "id": 1200425, "postDate": "2021-02-14T16:43:27.193Z", "content": "https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/nfnet.py
\nNFNet models were recently released in pytorch-image-models. These are only the model definitions and no imagenet pretrained weights are available at the moment, but potentially something to try in the final days of this competition.
\nHow to use timm on Kaggle:
\nAdd this dataset: https://www.kaggle.com/stanleyjzheng/timm-nfnet
import sys; sys.path.insert(0,'../input/timm-nfnet')\nimport timm\n\nmodel = timm.create_model('nfnet_f1', pretrained=False)\nI'm currently comparing NFNet-F0 to Effnet-B0 on VBD chest xray - I will update this thread with my results in a few hours. NFNet is not in the PyPi package yet, so I had to install from source. Will have results out on speed/vRAM usage soon.
\nEdit: added dataset for kaggle installation
", "rawMarkdown": "https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/nfnet.py\n\nNFNet models were recently released in pytorch-image-models. These are only the model definitions and no imagenet pretrained weights are available at the moment, but potentially something to try in the final days of this competition. \n\nHow to use timm on Kaggle: \nAdd this dataset: https://www.kaggle.com/stanleyjzheng/timm-nfnet\n\n```python\nimport sys; sys.path.insert(0,'../input/timm-nfnet')\nimport timm\n\nmodel = timm.create_model('nfnet_f1', pretrained=False)\n```\n\nI'm currently comparing NFNet-F0 to Effnet-B0 on VBD chest xray - I will update this thread with my results in a few hours. NFNet is not in the PyPi package yet, so I had to install from source. Will have results out on speed/vRAM usage soon.\n\nEdit: added dataset for kaggle installation", "votes": 34 }, { "id": 1200709, "postDate": "2021-02-14T22:49:14.847Z", "content": "Thanks for sharing Stanley!
\nif anyone is curious about this method, Yannic Kilcher just released a great video going over the paper:
\nhttps://www.youtube.com/watch?v=rNkHjZtH0RQ
Speed results on RTX3090 using AMP.
\nEffnet b0 512x512 (5.3m params), batch size 16 (7.1gb vRAM used): avg 0.2004s/step
\nEfficientnet b3 512x512 (12m params), batch size 16 (11.2gb vRAM used): avg 0.3372s/step
\nNFNet F0 512x512 (71.5m params), batch size 16 (12.3gb vRAM used): avg 0.6365s/step
\nNFNet F1 512x512, (132.6m params), batch size 16 (22.4gb vRAM used): avg 1.0725s/step
Update:
\nAll NFNet above use GeLU.
\nNFNet F0 SiLU 512x512 (71.5m params), batch size 16 (10.7gb vRAM used): avg 0.5501s/step
So SiLU has lower vRAM usage and training latency.
", "rawMarkdown": "Speed results on RTX3090 using AMP.\nEffnet b0 512x512 (5.3m params), batch size 16 (7.1gb vRAM used): avg 0.2004s/step\nEfficientnet b3 512x512 (12m params), batch size 16 (11.2gb vRAM used): avg 0.3372s/step\nNFNet F0 512x512 (71.5m params), batch size 16 (12.3gb vRAM used): avg 0.6365s/step\nNFNet F1 512x512, (132.6m params), batch size 16 (22.4gb vRAM used): avg 1.0725s/step\n\nUpdate:\nAll NFNet above use GeLU.\nNFNet F0 SiLU 512x512 (71.5m params), batch size 16 (10.7gb vRAM used): avg 0.5501s/step\n\nSo SiLU has lower vRAM usage and training latency.\n", "votes": 3, "replies": [ { "id": 1200497, "postDate": "2021-02-14T18:07:34.017Z", "content": "What data did you use to test these, As you said there wasn't imagenet weights in them?
", "rawMarkdown": "What data did you use to test these, As you said there wasn't imagenet weights in them?" }, { "id": 1200501, "postDate": "2021-02-14T18:11:34.943Z", "content": "This is for the vbd chest x-ray competition, which is my main focus at the moment. Dataset shouldn't matter for speed though.
\nThere are no imagenet weights yet so this is a random weights initialization
It's weird that in paper they claimed NFNet is faster than EffNet. 😳
I think it's down to JAX. Gelu GPU on pytorch is apparently slower, and I'm training at 512x512 on both models, while the comparison in the paper is 192x192 vs 224x224 for effnetb0 vs nfnetb0 if I remember correctly.
\nI also just found out there is no AGC implementation in the pytorch version, which causes loss to explode at high LR. This may also affect speed.
", "rawMarkdown": "I think it's down to JAX. Gelu GPU on pytorch is apparently slower, and I'm training at 512x512 on both models, while the comparison in the paper is 192x192 vs 224x224 for effnetb0 vs nfnetb0 if I remember correctly.\n\nI also just found out there is no AGC implementation in the pytorch version, which causes loss to explode at high LR. This may also affect speed." }, { "id": 1200597, "postDate": "2021-02-14T19:13:56.217Z", "content": "btw did you use the official weight for NfNet?
\n\nGelu GPU on pytorch is apparently slower
\n
I lose the reference but Mish it's better than Gelu for computer vision tasks. Maybe mish could improve the speed.
", "rawMarkdown": "> Gelu GPU on pytorch is apparently slower\n \nI lose the reference but **Mish** it's better than **Gelu** for computer vision tasks. Maybe mish could improve the speed." }, { "id": 1200605, "postDate": "2021-02-14T19:28:16.183Z", "content": "The F0 model also has more parameters than b0 and b3 combined.
\nAGC will probably be implemented at the backward stage for PyTorch? So you will most likely have to implement it yourself.
", "rawMarkdown": "The F0 model also has more parameters than b0 and b3 combined.\n\nAGC will probably be implemented at the backward stage for PyTorch? So you will most likely have to implement it yourself.", "votes": 1 }, { "id": 1200634, "postDate": "2021-02-14T20:02:01.830Z", "content": "\n\nAGC will probably be implemented at the backward stage for PyTorch? So you will most likely have to implement it yourself.
\n
That's a great point, I somehow completely overlooked that, thanks. Will give it a shot.
", "rawMarkdown": "> AGC will probably be implemented at the backward stage for PyTorch? So you will most likely have to implement it yourself.\n\nThat's a great point, I somehow completely overlooked that, thanks. Will give it a shot." }, { "id": 1200654, "postDate": "2021-02-14T20:25:34.970Z", "content": "@stanleyjzheng are you also using effnetb0 without pretrained weight?
Yes, correct, though it shouldn't matter speed-wise
", "rawMarkdown": "Yes, correct, though it shouldn't matter speed-wise", "votes": 1 }, { "id": 1201774, "postDate": "2021-02-15T16:40:45.813Z", "content": "@stanleyjzheng My experiments seem to line up with yours. I cant seem to use high lr without a Nan loss returning after some time. Looks like AGC is really important to train with out BN
", "rawMarkdown": "@stanleyjzheng My experiments seem to line up with yours. I cant seem to use high lr without a Nan loss returning after some time. Looks like AGC is really important to train with out BN", "votes": 1 }, { "id": 1208102, "postDate": "2021-02-18T06:50:55.870Z", "content": "goodness, I need 2x3090 for NFNet man, it is way too slow.
", "rawMarkdown": "goodness, I need 2x3090 for NFNet man, it is way too slow." } ] }, { "id": 1202011, "postDate": "2021-02-15T19:49:35.603Z", "content": "Tried F3s yesterday. Trained on 448 images. It's about 12 minutes/epoch on RTX3090*2 with DDP, AMP. Slow, indeed.
", "rawMarkdown": "Tried F3s yesterday. Trained on 448 images. It's about 12 minutes/epoch on RTX3090*2 with DDP, AMP. Slow, indeed.", "votes": 4, "replies": [ { "id": 1202109, "postDate": "2021-02-15T21:05:42.123Z", "content": "Im thinking that this model doesnt scale well with higher resolutions in comparison to effnets. Or maybe the implementation isnt right/optimized.
", "rawMarkdown": "Im thinking that this model doesnt scale well with higher resolutions in comparison to effnets. Or maybe the implementation isnt right/optimized." } ] }, { "id": 1207721, "postDate": "2021-02-18T01:50:48.510Z", "content": "You can use pretrained weights now from this great dataset Stanley made!
\nhttps://www.kaggle.com/stanleyjzheng/nfnet-pretrained
\nEasy implementation! Give it a try. Thanks @stanleyjzheng
The pre-trained weights are now available from the DeepMind github, as well.
", "rawMarkdown": "The pre-trained weights are now [available](https://github.com/deepmind/deepmind-research/tree/master/nfnets) from the DeepMind github, as well.", "votes": 1, "replies": [ { "id": 1207299, "postDate": "2021-02-17T19:37:32.740Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1205626, "postDate": "2021-02-16T20:30:07.127Z", "content": "AGC just added in pytorch-image-models (updated my dataset too)
\nIt's in timm/utils/agc.py.
Example:
\nfrom utils import adaptive_clip_grad\n\nloss.backward()\nadaptive_clip_grad(model.parameters(), clip_factor=0.01, eps=1e-3, norm_type=2.0)\noptimizer.step()\nIs something like this available for keras(TF2)?
\nOr is this another one of those \"learn pytorch\" scenarios where keras won't get anything till its too late.
This is a brand new model released 3 days ago, so unfortuantely no TF implementation yet, a bit of unfortunate timing.
", "rawMarkdown": "This is a brand new model released 3 days ago, so unfortuantely no TF implementation yet, a bit of unfortunate timing. ", "votes": 1 }, { "id": 1200436, "postDate": "2021-02-14T16:51:58.840Z", "content": "yeah I know it released like 3 days ago, the team which released this was deepmind, if I am not wrong.
\nEven google guys are not using TF2 for research, I think they are using JAX.
Hi, I was also looking for keras implementation and I found this could be interesting I still didn't put it into test. https://libraries.io/pypi/nfnets-keras. I will update my reply once I do.
", "rawMarkdown": "Hi, I was also looking for keras implementation and I found this could be interesting I still didn't put it into test. https://libraries.io/pypi/nfnets-keras. I will update my reply once I do. ", "votes": 1 }, { "id": 1201494, "postDate": "2021-02-15T12:38:26.037Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1201838, "postDate": "2021-02-15T17:24:22.337Z", "content": "Woah, looks legit, will try to implement this and tell the results if it really works. Gonna check the code as well, I have got the gist of the paper, let's see.
", "rawMarkdown": "Woah, looks legit, will try to implement this and tell the results if it really works. Gonna check the code as well, I have got the gist of the paper, let's see." }, { "id": 1204624, "postDate": "2021-02-16T09:19:05.627Z", "content": "I tried it. But, it is not working, there are some issues. I was able to resolve the first issue which was a simple change. Variant error fixed.
\nBut, after this, there are more issues. The one I am stuck at is:
\nTypeError: Failed to convert object of type <class 'tuple'> to Tensor. Contents: (None, 1, 1, 1). Consider casting elements to a supported type.
The error is in line:
\nnfnets_keras/nfnet_layers.py:58 call *\n r = tf.random.uniform(shape = [batch_size, 1, 1, 1], dtype = x.dtype)\nIt's a very interesting paper. Talks a lot about Batch normalization.
\nhttps://www.linkedin.com/posts/amritpal-singh-001_paperabrsummary-computervision-google-activity-6767776421348741120-SFos/
\nWould love to hear your opinions.
Thanks for sharing Stanley!
\nif anyone is curious about this method, Yannic Kilcher just released a great video going over the paper:
\nhttps://www.youtube.com/watch?v=rNkHjZtH0RQ
Speed results on RTX3090 using AMP.
\nEffnet b0 512x512 (5.3m params), batch size 16 (7.1gb vRAM used): avg 0.2004s/step
\nEfficientnet b3 512x512 (12m params), batch size 16 (11.2gb vRAM used): avg 0.3372s/step
\nNFNet F0 512x512 (71.5m params), batch size 16 (12.3gb vRAM used): avg 0.6365s/step
\nNFNet F1 512x512, (132.6m params), batch size 16 (22.4gb vRAM used): avg 1.0725s/step
Update:
\nAll NFNet above use GeLU.
\nNFNet F0 SiLU 512x512 (71.5m params), batch size 16 (10.7gb vRAM used): avg 0.5501s/step
So SiLU has lower vRAM usage and training latency.
", "votes": 3, "replies": [ { "id": 1200497, "author_name": "Mohneesh_Sreegirisetty", "author_url": "", "post_date": "2021-02-14T18:07:34.017000", "content": "What data did you use to test these, As you said there wasn't imagenet weights in them?
", "votes": 0, "replies": [] }, { "id": 1200501, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2021-02-14T18:11:34.943000", "content": "This is for the vbd chest x-ray competition, which is my main focus at the moment. Dataset shouldn't matter for speed though.
\nThere are no imagenet weights yet so this is a random weights initialization
It's weird that in paper they claimed NFNet is faster than EffNet. 😳
I think it's down to JAX. Gelu GPU on pytorch is apparently slower, and I'm training at 512x512 on both models, while the comparison in the paper is 192x192 vs 224x224 for effnetb0 vs nfnetb0 if I remember correctly.
\nI also just found out there is no AGC implementation in the pytorch version, which causes loss to explode at high LR. This may also affect speed.
", "votes": 0, "replies": [] }, { "id": 1200597, "author_name": "Awsaf", "author_url": "", "post_date": "2021-02-14T19:13:56.217000", "content": "btw did you use the official weight for NfNet?
\n\nGelu GPU on pytorch is apparently slower
\n
I lose the reference but Mish it's better than Gelu for computer vision tasks. Maybe mish could improve the speed.
", "votes": 0, "replies": [] }, { "id": 1200605, "author_name": "JunYong Tong", "author_url": "", "post_date": "2021-02-14T19:28:16.183000", "content": "The F0 model also has more parameters than b0 and b3 combined.
\nAGC will probably be implemented at the backward stage for PyTorch? So you will most likely have to implement it yourself.
", "votes": 1, "replies": [] }, { "id": 1200634, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2021-02-14T20:02:01.830000", "content": "\n\nAGC will probably be implemented at the backward stage for PyTorch? So you will most likely have to implement it yourself.
\n
That's a great point, I somehow completely overlooked that, thanks. Will give it a shot.
", "votes": 0, "replies": [] }, { "id": 1200654, "author_name": "Awsaf", "author_url": "", "post_date": "2021-02-14T20:25:34.970000", "content": "@stanleyjzheng are you also using effnetb0 without pretrained weight?
Yes, correct, though it shouldn't matter speed-wise
", "votes": 1, "replies": [] }, { "id": 1201774, "author_name": "Yann Majewski", "author_url": "", "post_date": "2021-02-15T16:40:45.813000", "content": "@stanleyjzheng My experiments seem to line up with yours. I cant seem to use high lr without a Nan loss returning after some time. Looks like AGC is really important to train with out BN
", "votes": 1, "replies": [] }, { "id": 1208102, "author_name": "gao-hongnan", "author_url": "", "post_date": "2021-02-18T06:50:55.870000", "content": "goodness, I need 2x3090 for NFNet man, it is way too slow.
", "votes": 0, "replies": [] } ] }, { "id": 1202011, "author_name": "sin", "author_url": "", "post_date": "2021-02-15T19:49:35.603000", "content": "Tried F3s yesterday. Trained on 448 images. It's about 12 minutes/epoch on RTX3090*2 with DDP, AMP. Slow, indeed.
", "votes": 4, "replies": [ { "id": 1202109, "author_name": "Yann Majewski", "author_url": "", "post_date": "2021-02-15T21:05:42.123000", "content": "Im thinking that this model doesnt scale well with higher resolutions in comparison to effnets. Or maybe the implementation isnt right/optimized.
", "votes": 0, "replies": [] } ] }, { "id": 1207721, "author_name": "Andy Jian Zhou", "author_url": "", "post_date": "2021-02-18T01:50:48.510000", "content": "You can use pretrained weights now from this great dataset Stanley made!
\nhttps://www.kaggle.com/stanleyjzheng/nfnet-pretrained
\nEasy implementation! Give it a try. Thanks @stanleyjzheng
The pre-trained weights are now available from the DeepMind github, as well.
", "votes": 1, "replies": [ { "id": 1207299, "author_name": "", "author_url": "", "post_date": "2021-02-17T19:37:32.740000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1205626, "author_name": "Stanley Zheng", "author_url": "", "post_date": "2021-02-16T20:30:07.127000", "content": "AGC just added in pytorch-image-models (updated my dataset too)
\nIt's in timm/utils/agc.py.
Example:
\nfrom utils import adaptive_clip_grad\n\nloss.backward()\nadaptive_clip_grad(model.parameters(), clip_factor=0.01, eps=1e-3, norm_type=2.0)\noptimizer.step()\nIs something like this available for keras(TF2)?
\nOr is this another one of those \"learn pytorch\" scenarios where keras won't get anything till its too late.
This is a brand new model released 3 days ago, so unfortuantely no TF implementation yet, a bit of unfortunate timing.
", "votes": 1, "replies": [] }, { "id": 1200436, "author_name": "Mohneesh_Sreegirisetty", "author_url": "", "post_date": "2021-02-14T16:51:58.840000", "content": "yeah I know it released like 3 days ago, the team which released this was deepmind, if I am not wrong.
\nEven google guys are not using TF2 for research, I think they are using JAX.
Hi, I was also looking for keras implementation and I found this could be interesting I still didn't put it into test. https://libraries.io/pypi/nfnets-keras. I will update my reply once I do.
", "votes": 1, "replies": [] }, { "id": 1201494, "author_name": "", "author_url": "", "post_date": "2021-02-15T12:38:26.037000", "content": "", "votes": 0, "replies": [] }, { "id": 1201838, "author_name": "Mohneesh_Sreegirisetty", "author_url": "", "post_date": "2021-02-15T17:24:22.337000", "content": "Woah, looks legit, will try to implement this and tell the results if it really works. Gonna check the code as well, I have got the gist of the paper, let's see.
", "votes": 0, "replies": [] }, { "id": 1204624, "author_name": "Prateek Jain", "author_url": "", "post_date": "2021-02-16T09:19:05.627000", "content": "I tried it. But, it is not working, there are some issues. I was able to resolve the first issue which was a simple change. Variant error fixed.
\nBut, after this, there are more issues. The one I am stuck at is:
\nTypeError: Failed to convert object of type <class 'tuple'> to Tensor. Contents: (None, 1, 1, 1). Consider casting elements to a supported type.
The error is in line:
\nnfnets_keras/nfnet_layers.py:58 call *\n r = tf.random.uniform(shape = [batch_size, 1, 1, 1], dtype = x.dtype)\nIt's a very interesting paper. Talks a lot about Batch normalization.
\nhttps://www.linkedin.com/posts/amritpal-singh-001_paperabrsummary-computervision-google-activity-6767776421348741120-SFos/
\nWould love to hear your opinions.