The pretrained weights for the new State of the Art NFNet F* models were released on DeepMind's GitHub repo today in Haiku (Jax) format. I've converted them to PyTorch for ease of use.
\nDataset link: www.kaggle.com/stanleyjzheng/nfnet-pretrained
\nYou will also need to compile TIMM (pytorch-image-models) from scratch. Here is a snippet:
\nAdd this TIMM dataset
\nimport sys; sys.path.insert(0,'../input/timm-nfnet')\nimport torch\nimport timm\n\nmodel = timm.create_model('nfnet_f0', pretrained=False)\nmodel.load_state_dict(torch.load('../input/nfnet-pretrained/NFNet-f0.pt'))\nmodel.head.fc = nn.Linear(3072, num_classes)\nConverting them to Tensorflow is relatively trivial as well: https://dm-haiku.readthedocs.io/en/latest/notebooks/jax2tf.html
\nTIMM also has adaptive gradient clipping (AGC) to be used with NFNets:
\nfrom timm.utils.agc import adaptive_clip_grad\n\nloss.backward()\nadaptive_clip_grad(model.parameters(), clip_factor=0.01, eps=1e-3, norm_type=2.0)\noptimizer.step()\nEdit: Thank you all so much for datasets expert! 4x expert now!!!
", "messages": [ { "id": "1207383", "postDate": "02/17/2021 20:19:29", "content": "The pretrained weights for the new State of the Art NFNet F* models were released on DeepMind's GitHub repo today in Haiku (Jax) format. I've converted them to PyTorch for ease of use.
\nDataset link: www.kaggle.com/stanleyjzheng/nfnet-pretrained
\nYou will also need to compile TIMM (pytorch-image-models) from scratch. Here is a snippet:
\nAdd this TIMM dataset
\nimport sys; sys.path.insert(0,'../input/timm-nfnet')\nimport torch\nimport timm\n\nmodel = timm.create_model('nfnet_f0', pretrained=False)\nmodel.load_state_dict(torch.load('../input/nfnet-pretrained/NFNet-f0.pt'))\nmodel.head.fc = nn.Linear(3072, num_classes)\nConverting them to Tensorflow is relatively trivial as well: https://dm-haiku.readthedocs.io/en/latest/notebooks/jax2tf.html
\nTIMM also has adaptive gradient clipping (AGC) to be used with NFNets:
\nfrom timm.utils.agc import adaptive_clip_grad\n\nloss.backward()\nadaptive_clip_grad(model.parameters(), clip_factor=0.01, eps=1e-3, norm_type=2.0)\noptimizer.step()\nEdit: Thank you all so much for datasets expert! 4x expert now!!!
", "rawMarkdown": "The pretrained weights for the new State of the Art NFNet F* models were released on DeepMind's [GitHub repo](https://github.com/deepmind/deepmind-research/tree/master/nfnets) today in Haiku (Jax) format. I've converted them to PyTorch for ease of use.\n\nDataset link: www.kaggle.com/stanleyjzheng/nfnet-pretrained\n\nYou will also need to compile TIMM (pytorch-image-models) from scratch. Here is a snippet:\n\nAdd this [TIMM dataset](https://www.kaggle.com/stanleyjzheng/timm-nfnet)\n\n```python\nimport sys; sys.path.insert(0,'../input/timm-nfnet')\nimport torch\nimport timm\n\nmodel = timm.create_model('nfnet_f0', pretrained=False)\nmodel.load_state_dict(torch.load('../input/nfnet-pretrained/NFNet-f0.pt'))\nmodel.head.fc = nn.Linear(3072, num_classes)\n```\n\nConverting them to Tensorflow is relatively trivial as well: https://dm-haiku.readthedocs.io/en/latest/notebooks/jax2tf.html \n\nTIMM also has adaptive gradient clipping (AGC) to be used with NFNets:\n```python\nfrom timm.utils.agc import adaptive_clip_grad\n\nloss.backward()\nadaptive_clip_grad(model.parameters(), clip_factor=0.01, eps=1e-3, norm_type=2.0)\noptimizer.step()\n```\n\nEdit: Thank you all so much for datasets expert! 4x expert now!!!", "votes": null }, { "id": "1207386", "postDate": "02/17/2021 20:21:52", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "1207403", "postDate": "02/17/2021 20:33:51", "content": "How did you convert weights to PyTorch?
\nCan you add the other models too?
I used their utils.py, all the other models are uploading, but 5gb takes a while to upload.
", "rawMarkdown": "I used their utils.py, all the other models are uploading, but 5gb takes a while to upload.", "votes": null }, { "id": "1207407", "postDate": "02/17/2021 20:36:16", "content": "That's great!
\nThank you
My pleasure :)
", "rawMarkdown": "My pleasure :)", "votes": null }, { "id": "1207521", "postDate": "02/17/2021 22:24:46", "content": "Thanks Stanley. I didn't read this too carefully, but is AGC required for NFNets?
", "rawMarkdown": "Thanks Stanley. I didn't read this too carefully, but is AGC required for NFNets?", "votes": null }, { "id": "1207526", "postDate": "02/17/2021 22:30:57", "content": "No problem. Yes - it should fix the gradient explosion problems we had earlier.
", "rawMarkdown": "No problem. Yes - it should fix the gradient explosion problems we had earlier.", "votes": null }, { "id": "1207685", "postDate": "02/18/2021 01:06:32", "content": "Thanks Stanley. Will try it out. I am curious how these compare to the other models on this dataset. Looks like F0 has 68M params!
", "rawMarkdown": "Thanks Stanley. Will try it out. I am curious how these compare to the other models on this dataset. Looks like F0 has 68M params!", "votes": null }, { "id": "1207988", "postDate": "02/18/2021 05:28:12", "content": "Really nice work @stanleyjzheng, thanks a lot man!
", "rawMarkdown": "Really nice work @stanleyjzheng, thanks a lot man!", "votes": null }, { "id": "1207992", "postDate": "02/18/2021 05:31:44", "content": "@reighns Thanks!
\nYour upvote pushed my dataset to bronze, thanks for 4x expert haha :)
", "rawMarkdown": "reighns Thanks!\n\nYour upvote pushed my dataset to bronze, thanks for 4x expert haha :)", "votes": null }, { "id": "1207995", "postDate": "02/18/2021 05:34:01", "content": "LOL hahahahha, nice nice, anyways I am unsure what adaptive gradient clipping does. Would you pointing me to the right resource on it. So in the training pipeline, I just need to add the snippet you showed above to make it work right? Can I use PyTorch's AMP as well?
", "rawMarkdown": "LOL hahahahha, nice nice, anyways I am unsure what adaptive gradient clipping does. Would you pointing me to the right resource on it. So in the training pipeline, I just need to add the snippet you showed above to make it work right? Can I use PyTorch's AMP as well?", "votes": null }, { "id": "1208004", "postDate": "02/18/2021 05:37:59", "content": "Yep, definitely! Just between your optimizer and your loss, you need to add that adaptive clip grad line. For example, mine with AMP is
\nscaler.scale(loss).backward()\nadaptive_clip_grad(model.parameters(), clip_factor=0.01, eps=1e-3, norm_type=2.0)\nscaler.step(optimizer)\nscaler.update()\nIf you're interested in the technical details, the paper is great.
\nDiscussion here (it's the same as regular pytorch gradient clipping): https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191
I tried exactly this and my train loss goes to Nan in second epoch and I didn't have a chance to debug this yet.
", "rawMarkdown": "I tried exactly this and my train loss goes to Nan in second epoch and I didn't have a chance to debug this yet.", "votes": null }, { "id": "1208008", "postDate": "02/18/2021 05:43:18", "content": "Interesting - try lower LR maybe? I'm not in this competition but I'm in VBD and it works great. LR > 5e-4 gives me NaN loss. Sorry, I have no clue
", "rawMarkdown": "Interesting - try lower LR maybe? I'm not in this competition but I'm in VBD and it works great. LR > 5e-4 gives me NaN loss. Sorry, I have no clue", "votes": null }, { "id": "1208024", "postDate": "02/18/2021 05:54:56", "content": "no worries. Most likely an implementation error on my part. I dont have enough time to use it for this competition but I will definitely use it for others. Thanks!
", "rawMarkdown": "no worries. Most likely an implementation error on my part. I dont have enough time to use it for this competition but I will definitely use it for others. Thanks!", "votes": null }, { "id": "1208112", "postDate": "02/18/2021 07:00:49", "content": "@trushk Even I am getting nans
\nI was using lr 1e-4 to 1e-6 with cosine scheduling
Thanks a bunch! I am training with it now. I noticed that Roff (Timm) will prompt a line: No pretrained weights exist for this model. Using random initialization. But I think it is ok since we overrode it with our pretrained weights here.
", "rawMarkdown": "Thanks a bunch! I am training with it now. I noticed that Roff (Timm) will prompt a line: No pretrained weights exist for this model. Using random initialization. But I think it is ok since we overrode it with our pretrained weights here.", "votes": null }, { "id": "1209512", "postDate": "02/18/2021 23:55:02", "content": "Thank you Stanley for sharing this.
", "rawMarkdown": "Thank you Stanley for sharing this.", "votes": null }, { "id": "1209974", "postDate": "02/19/2021 06:16:44", "content": "It's a very interesting paper. Talks a lot about Batch normalization.
\nhttps://www.linkedin.com/posts/amritpal-singh-001_paperabrsummary-computervision-google-activity-6767776421348741120-SFos/
\nWould love to hear your opinions.
Thank you for sharing.
", "rawMarkdown": "Thank you for sharing.", "votes": null }, { "id": "1211531", "postDate": "02/20/2021 10:03:38", "content": "Thank you for sharing such insightful content @stanleyjzheng
", "rawMarkdown": "Thank you for sharing such insightful content @stanleyjzheng", "votes": null }, { "id": "1216171", "postDate": "02/24/2021 06:17:44", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "1219208", "postDate": "02/26/2021 14:51:13", "content": "nice work!. Thank you.
", "rawMarkdown": "nice work!. Thank you.", "votes": null }, { "id": "1225166", "postDate": "03/03/2021 11:32:07", "content": "I'm not in this competition either, using it elsewhere, but what optimizer do You guys use? As to my first experiments, Adam is likely to produce the instability and NaNs. The original paper suggests Nesterov' momentum SGD with momentum=0.9, weight_decay=2e-5. It seems to be much more stable when in use with NFNets and AGC. Also, try to include a \"warm up\" phase (increasing lr gradually during first, say, 5 epochs). This setup works fine for me, i'm training nf_net_f3 with lr=0.1*BATCH_SIZE/256 (also as paper suggests). Will be glad if You also share Your experiences in this
", "rawMarkdown": "I'm not in this competition either, using it elsewhere, but what optimizer do You guys use? As to my first experiments, Adam is likely to produce the instability and NaNs. The original paper suggests Nesterov' momentum SGD with momentum=0.9, weight_decay=2e-5. It seems to be much more stable when in use with NFNets and AGC. Also, try to include a \"warm up\" phase (increasing lr gradually during first, say, 5 epochs). This setup works fine for me, i'm training nf_net_f3 with lr=0.1*BATCH_SIZE/256 (also as paper suggests). Will be glad if You also share Your experiences in this", "votes": null }, { "id": "1239102", "postDate": "03/15/2021 13:10:30", "content": "The paper states, that AGC is only necessary for large batchsizes and heavy data augmentations
\n\n", "rawMarkdown": "The paper states, that AGC is only necessary for large batchsizes and heavy data augmentations\n\n> Using AGC, we can train NF-ResNets stably with larger\nbatch sizes (up to 4096), as well as with very strong data\naugmentations like RandAugment (Cubuk et al., 2020) for\nwhich NF-ResNets without AGC fail to train [...] As anticipated, the benefits of using AGC are smaller when the batch size is small.", "votes": null }, { "id": "1621224", "postDate": "12/17/2021 14:43:47", "content": "Using AGC, we can train NF-ResNets stably with larger
\n
\n batch sizes (up to 4096), as well as with very strong data
\n augmentations like RandAugment (Cubuk et al., 2020) for
\n which NF-ResNets without AGC fail to train […] As anticipated, the benefits of using AGC are smaller when the batch size is small.
great. Nice work~!!
\ngod mind ✨✨✨
Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 1207403, "author_name": "debarshichanda", "author_url": "", "post_date": "02/17/2021 20:33:51", "content": "How did you convert weights to PyTorch?
\nCan you add the other models too?
I used their utils.py, all the other models are uploading, but 5gb takes a while to upload.
", "votes": null, "replies": [] }, { "id": 1207407, "author_name": "debarshichanda", "author_url": "", "post_date": "02/17/2021 20:36:16", "content": "That's great!
\nThank you
My pleasure :)
", "votes": null, "replies": [] } ] }, { "id": 1207521, "author_name": "underwearfitting", "author_url": "", "post_date": "02/17/2021 22:24:46", "content": "Thanks Stanley. I didn't read this too carefully, but is AGC required for NFNets?
", "votes": null, "replies": [ { "id": 1207526, "author_name": "stanleyjzheng", "author_url": "", "post_date": "02/17/2021 22:30:57", "content": "No problem. Yes - it should fix the gradient explosion problems we had earlier.
", "votes": null, "replies": [] }, { "id": 1239102, "author_name": "aliabdin1", "author_url": "", "post_date": "03/15/2021 13:10:30", "content": "The paper states, that AGC is only necessary for large batchsizes and heavy data augmentations
\n\n", "votes": null, "replies": [] } ] }, { "id": 1207685, "author_name": "trushk", "author_url": "", "post_date": "02/18/2021 01:06:32", "content": "Using AGC, we can train NF-ResNets stably with larger
\n
\n batch sizes (up to 4096), as well as with very strong data
\n augmentations like RandAugment (Cubuk et al., 2020) for
\n which NF-ResNets without AGC fail to train […] As anticipated, the benefits of using AGC are smaller when the batch size is small.
Thanks Stanley. Will try it out. I am curious how these compare to the other models on this dataset. Looks like F0 has 68M params!
", "votes": null, "replies": [] }, { "id": 1207988, "author_name": "reighns", "author_url": "", "post_date": "02/18/2021 05:28:12", "content": "Really nice work @stanleyjzheng, thanks a lot man!
", "votes": null, "replies": [ { "id": 1207992, "author_name": "stanleyjzheng", "author_url": "", "post_date": "02/18/2021 05:31:44", "content": "@reighns Thanks!
\nYour upvote pushed my dataset to bronze, thanks for 4x expert haha :)
", "votes": null, "replies": [] }, { "id": 1207995, "author_name": "reighns", "author_url": "", "post_date": "02/18/2021 05:34:01", "content": "LOL hahahahha, nice nice, anyways I am unsure what adaptive gradient clipping does. Would you pointing me to the right resource on it. So in the training pipeline, I just need to add the snippet you showed above to make it work right? Can I use PyTorch's AMP as well?
", "votes": null, "replies": [] }, { "id": 1208004, "author_name": "stanleyjzheng", "author_url": "", "post_date": "02/18/2021 05:37:59", "content": "Yep, definitely! Just between your optimizer and your loss, you need to add that adaptive clip grad line. For example, mine with AMP is
\nscaler.scale(loss).backward()\nadaptive_clip_grad(model.parameters(), clip_factor=0.01, eps=1e-3, norm_type=2.0)\nscaler.step(optimizer)\nscaler.update()\nIf you're interested in the technical details, the paper is great.
\nDiscussion here (it's the same as regular pytorch gradient clipping): https://discuss.pytorch.org/t/proper-way-to-do-gradient-clipping/191
I tried exactly this and my train loss goes to Nan in second epoch and I didn't have a chance to debug this yet.
", "votes": null, "replies": [] }, { "id": 1208008, "author_name": "stanleyjzheng", "author_url": "", "post_date": "02/18/2021 05:43:18", "content": "Interesting - try lower LR maybe? I'm not in this competition but I'm in VBD and it works great. LR > 5e-4 gives me NaN loss. Sorry, I have no clue
", "votes": null, "replies": [] }, { "id": 1208024, "author_name": "trushk", "author_url": "", "post_date": "02/18/2021 05:54:56", "content": "no worries. Most likely an implementation error on my part. I dont have enough time to use it for this competition but I will definitely use it for others. Thanks!
", "votes": null, "replies": [] }, { "id": 1208112, "author_name": "debarshichanda", "author_url": "", "post_date": "02/18/2021 07:00:49", "content": "@trushk Even I am getting nans
\nI was using lr 1e-4 to 1e-6 with cosine scheduling
Thanks a bunch! I am training with it now. I noticed that Roff (Timm) will prompt a line: No pretrained weights exist for this model. Using random initialization. But I think it is ok since we overrode it with our pretrained weights here.
", "votes": null, "replies": [] }, { "id": 1225166, "author_name": "snufkin77", "author_url": "", "post_date": "03/03/2021 11:32:07", "content": "I'm not in this competition either, using it elsewhere, but what optimizer do You guys use? As to my first experiments, Adam is likely to produce the instability and NaNs. The original paper suggests Nesterov' momentum SGD with momentum=0.9, weight_decay=2e-5. It seems to be much more stable when in use with NFNets and AGC. Also, try to include a \"warm up\" phase (increasing lr gradually during first, say, 5 epochs). This setup works fine for me, i'm training nf_net_f3 with lr=0.1*BATCH_SIZE/256 (also as paper suggests). Will be glad if You also share Your experiences in this
", "votes": null, "replies": [] } ] }, { "id": 1209512, "author_name": "cdeotte", "author_url": "", "post_date": "02/18/2021 23:55:02", "content": "Thank you Stanley for sharing this.
", "votes": null, "replies": [] }, { "id": 1209974, "author_name": "amritpal333", "author_url": "", "post_date": "02/19/2021 06:16:44", "content": "It's a very interesting paper. Talks a lot about Batch normalization.
\nhttps://www.linkedin.com/posts/amritpal-singh-001_paperabrsummary-computervision-google-activity-6767776421348741120-SFos/
\nWould love to hear your opinions.
Thank you for sharing.
", "votes": null, "replies": [] }, { "id": 1211531, "author_name": "alifrahman", "author_url": "", "post_date": "02/20/2021 10:03:38", "content": "Thank you for sharing such insightful content @stanleyjzheng
", "votes": null, "replies": [] }, { "id": 1216171, "author_name": "himanshu1999", "author_url": "", "post_date": "02/24/2021 06:17:44", "content": "Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 1219208, "author_name": "mssjss", "author_url": "", "post_date": "02/26/2021 14:51:13", "content": "nice work!. Thank you.
", "votes": null, "replies": [] }, { "id": 1621224, "author_name": "kimalpha", "author_url": "", "post_date": "12/17/2021 14:43:47", "content": "great. Nice work~!!
\ngod mind ✨✨✨