\n
![\"\"](\"https://raw.githubusercontent.com/facebookresearch/deit/main/.github/deit.png\")
New paper by Facebook AI Research (FAIR) showing great results for Vision Transformer by only training on 1.2M ImageNet images.
\nSome benchmark on ImageNet:
\n
You can easily install it through torch hub:
\nimport torch\n# check you have the right version of timm\nimport timm\nassert timm.__version__ == \"0.3.2\"\n\n# now load it with torchhub\nmodel = torch.hub.load(\n 'facebookresearch/deit:main', 'deit_base_patch16_224', \n pretrained=True\n)\nI'm cautiously optimistic about this, As @serigne mentioned in another post:
\n\n\nBe Carefull when they say they beat EfficientNet . In general, they don't really respect the scaling and training processes in the original EfficientNet paper when doing the comparison.
\nAt Facebook, they have already claimed beating Effnet With the Regnet Paper. But all those who retrained the models (like Ross Wightman with timm) show that the latter is way far from beating Effnet.
\nAnd if you use pretrained regnet for fine-tuning , you'll see it performs in general poorly than efficientnet, particularly with higher resulutions than 224x224
\n
But it's worth trying out since the code and trained weights are all available!
\nHere's an image retrieved from the blog post:
\n![\"\"](\"https://scontent.fymy1-2.fna.fbcdn.net/v/t39.2365-6/131568808_300918484690143_5468311438414224765_n.png?_nc_cat=101&ccb=2&_nc_sid=ad8a9d&_nc_ohc=0Bm6ISoMSa4AX8pW3qU&_nc_ht=scontent.fymy1-2.fna&oh=08d7f1568484e12bc4c2a24e824ae0e0&oe=600A4F7A\")
New paper by Facebook AI Research (FAIR) showing great results for Vision Transformer by only training on 1.2M ImageNet images.
\nSome benchmark on ImageNet:
\n
You can easily install it through torch hub:
\nimport torch\n# check you have the right version of timm\nimport timm\nassert timm.__version__ == \"0.3.2\"\n\n# now load it with torchhub\nmodel = torch.hub.load(\n 'facebookresearch/deit:main', 'deit_base_patch16_224', \n pretrained=True\n)\nI'm cautiously optimistic about this, As @serigne mentioned in another post:
\n\n\nBe Carefull when they say they beat EfficientNet . In general, they don't really respect the scaling and training processes in the original EfficientNet paper when doing the comparison.
\nAt Facebook, they have already claimed beating Effnet With the Regnet Paper. But all those who retrained the models (like Ross Wightman with timm) show that the latter is way far from beating Effnet.
\nAnd if you use pretrained regnet for fine-tuning , you'll see it performs in general poorly than efficientnet, particularly with higher resulutions than 224x224
\n
But it's worth trying out since the code and trained weights are all available!
\nHere's an image retrieved from the blog post:
\n
i will try this.
\ni tried Vit before, but couldn't make it work.
\nnote that efficientnet has inbuilt drop connect. this is one of the reasons why its results is more robust. (or easier for poelp to make it work)
", "rawMarkdown": "i will try this.\n\ni tried Vit before, but couldn't make it work. \n\nnote that efficientnet has inbuilt drop connect. this is one of the reasons why its results is more robust. (or easier for poelp to make it work)", "votes": 4, "replies": [ { "id": 1125686, "postDate": "2020-12-25T00:44:28.147Z", "content": "Funny anecdote about the drop connect: for this competition I was using the qubvel/callidor implementation (the one on keras-applications) but realized that the drop connect rate was incorrect. So I switched to the official tf.keras implementation which has the correct rate, but the performance decrease drastically for larger Effnets (even though I tried various values of drop_connect_rate)
Tried it. Cannot break 0.92 :/
", "rawMarkdown": "Tried it. Cannot break 0.92 :/", "votes": 1 }, { "id": 1127658, "postDate": "2020-12-26T18:05:48.443Z", "content": "I think self-Attention will replace CNN just like it replaced RNN. But for now, it's not mature enough to outperfom efficiently approaches like Efficientnet.
\nBeside of that , I don't like Facebook's guys attitude who copied and paste huge part of @rwightman timm , without crediting him properly and while putting all the copied code in a more restrictive license
", "rawMarkdown": "I think self-Attention will replace CNN just like it replaced RNN. But for now, it's not mature enough to outperfom efficiently approaches like Efficientnet.\n\nBeside of that , I don't like Facebook's guys attitude who copied and paste huge part of @rwightman timm , without [crediting him properly](https://twitter.com/wightmanr/status/1342195464880320512) and while putting all the copied code in a [more restrictive license](https://twitter.com/wightmanr/status/1342143859380219906)", "votes": 2 }, { "id": 1127969, "postDate": "2020-12-27T04:33:43.757Z", "content": "emmmm，maybe they just used the tricks partial to their model in the evaluation.
\nAnyway,it is recognized that efficientnet have a good performance on the majority of task related to image.
Here is an easy wrapper for checking out this model in PyTorch
\nhttps://github.com/lucidrains/vit-pytorch
Yeah, I just saw this on @paperswithcode's Twitter Feed. First Transformers replaced RNN's now they are coming after CNN's too 😄. Also, was surprised to see the code in Pytorch most transformer paper nowadays use JAX based ecosystems such as google/Trax or google/Flax. Excited to see more transformers in Kaggle competitions.
", "rawMarkdown": "Yeah, I just saw this on [@paperswithcode](https://twitter.com/paperswithcode/status/1342100539761434624?s=20)'s Twitter Feed. First Transformers replaced RNN's now they are coming after CNN's too 😄. Also, was surprised to see the code in Pytorch most transformer paper nowadays use JAX based ecosystems such as [google/Trax](https://github.com/google/trax) or [google/Flax](https://github.com/google/flax). Excited to see more transformers in Kaggle competitions.", "replies": [ { "id": 1127662, "postDate": "2020-12-26T18:11:26.260Z", "content": "JAX is from Google. The authors of this paper work at Facebook. So it's easier/suitable for them to copy the already implemented ViT in Pytorch by @rwightman as basis of their work^^
", "rawMarkdown": "JAX is from Google. The authors of this paper work at Facebook. So it's easier/suitable for them to copy the already implemented ViT in Pytorch by @rwightman as basis of their work^^" } ] }, { "id": 1125344, "postDate": "2020-12-24T16:34:23.753Z", "content": "@hengck23 Thoughts on this?
", "rawMarkdown": "@hengck23 Thoughts on this?" } ], "comments": [ { "id": 1125677, "author_name": "hengck23", "author_url": "", "post_date": "2020-12-25T00:15:37.783000", "content": "i will try this.
\ni tried Vit before, but couldn't make it work.
\nnote that efficientnet has inbuilt drop connect. this is one of the reasons why its results is more robust. (or easier for poelp to make it work)
", "votes": 4, "replies": [ { "id": 1125686, "author_name": "xhlulu", "author_url": "", "post_date": "2020-12-25T00:44:28.147000", "content": "Funny anecdote about the drop connect: for this competition I was using the qubvel/callidor implementation (the one on keras-applications) but realized that the drop connect rate was incorrect. So I switched to the official tf.keras implementation which has the correct rate, but the performance decrease drastically for larger Effnets (even though I tried various values of drop_connect_rate)
Tried it. Cannot break 0.92 :/
", "votes": 1, "replies": [] }, { "id": 1127658, "author_name": "Serigne ", "author_url": "", "post_date": "2020-12-26T18:05:48.443000", "content": "I think self-Attention will replace CNN just like it replaced RNN. But for now, it's not mature enough to outperfom efficiently approaches like Efficientnet.
\nBeside of that , I don't like Facebook's guys attitude who copied and paste huge part of @rwightman timm , without crediting him properly and while putting all the copied code in a more restrictive license
", "votes": 2, "replies": [] }, { "id": 1127969, "author_name": "Xu Zhiyuan", "author_url": "", "post_date": "2020-12-27T04:33:43.757000", "content": "emmmm，maybe they just used the tricks partial to their model in the evaluation.
\nAnyway,it is recognized that efficientnet have a good performance on the majority of task related to image.
Here is an easy wrapper for checking out this model in PyTorch
\nhttps://github.com/lucidrains/vit-pytorch
Yeah, I just saw this on @paperswithcode's Twitter Feed. First Transformers replaced RNN's now they are coming after CNN's too 😄. Also, was surprised to see the code in Pytorch most transformer paper nowadays use JAX based ecosystems such as google/Trax or google/Flax. Excited to see more transformers in Kaggle competitions.
", "votes": 0, "replies": [ { "id": 1127662, "author_name": "Serigne ", "author_url": "", "post_date": "2020-12-26T18:11:26.260000", "content": "JAX is from Google. The authors of this paper work at Facebook. So it's easier/suitable for them to copy the already implemented ViT in Pytorch by @rwightman as basis of their work^^
", "votes": 0, "replies": [] } ] }, { "id": 1125344, "author_name": "xhlulu", "author_url": "", "post_date": "2020-12-24T16:34:23.753000", "content": "@hengck23 Thoughts on this?
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1125339": "New paper by Facebook AI Research (FAIR) showing great results for Vision Transformer by only training on 1.2M ImageNet images.\n\n* [Blog post](https://ai.facebook.com/blog/data-efficient-image-transformers-a-promising-new-technique-for-image-classification/)\n* [Arxiv paper](https://arxiv.org/abs/2012.12877)\n* [Code](https://github.com/facebookresearch/deit)\n\nSome benchmark on ImageNet:\n\n\nYou can easily install it through torch hub:\n```\nimport torch\n# check you have the right version of timm\nimport timm\nassert timm.__version__ == \"0.3.2\"\n\n# now load it with torchhub\nmodel = torch.hub.load(\n 'facebookresearch/deit:main', 'deit_base_patch16_224', \n pretrained=True\n)\n```\n\nI'm cautiously optimistic about this, As @serigne mentioned in [another post](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/206345#1124900):\n> Be Carefull when they say they beat EfficientNet . In general, they don't really respect the scaling and training processes in the original EfficientNet paper when doing the comparison. \n> \n> At Facebook, they have already claimed beating Effnet With the Regnet Paper. But all those who retrained the models (like Ross Wightman with timm) show that the latter is way far from beating Effnet. \n> \n> And if you use pretrained regnet for fine-tuning , you'll see it performs in general poorly than efficientnet, particularly with higher resulutions than 224x224\n\nBut it's worth trying out since the code and trained weights are all available!\n\nHere's an image retrieved from the blog post:\n\n\n", "1125677": "i will try this.\n\ni tried Vit before, but couldn't make it work. \n\nnote that efficientnet has inbuilt drop connect. this is one of the reasons why its results is more robust. (or easier for poelp to make it work)", "1126038": "Tried it. Cannot break 0.92 :/", "1127658": "I think self-Attention will replace CNN just like it replaced RNN. But for now, it's not mature enough to outperfom efficiently approaches like Efficientnet.\n\nBeside of that , I don't like Facebook's guys attitude who copied and paste huge part of @rwightman timm , without [crediting him properly](https://twitter.com/wightmanr/status/1342195464880320512) and while putting all the copied code in a [more restrictive license](https://twitter.com/wightmanr/status/1342143859380219906)", "1127969": "emmmm，maybe they just used the tricks partial to their model in the evaluation.\nAnyway,it is recognized that efficientnet have a good performance on the majority of task related to image.", "1126351": "Here is an easy wrapper for checking out this model in PyTorch\nhttps://github.com/lucidrains/vit-pytorch", "1125879": "Yeah, I just saw this on [@paperswithcode](https://twitter.com/paperswithcode/status/1342100539761434624?s=20)'s Twitter Feed. First Transformers replaced RNN's now they are coming after CNN's too 😄. Also, was surprised to see the code in Pytorch most transformer paper nowadays use JAX based ecosystems such as [google/Trax](https://github.com/google/trax) or [google/Flax](https://github.com/google/flax). Excited to see more transformers in Kaggle competitions.", "1125344": "@hengck23 Thoughts on this?" } }