![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3492127%2F3b99cd8b398b3d4349539ae307f7547d%2Fdd1.png?generation=1611251617068659&alt=media\")
DeiT is DeiT: Data-efficient Image Transformers like ViT.
Now, we can use 384x384 image size for DeiT.
https://github.com/facebookresearch/deit
\nThank you!
", "messages": [ { "id": "1163530", "postDate": "01/21/2021 18:08:50", "content": "DeiT is DeiT: Data-efficient Image Transformers like ViT.
Now, we can use 384x384 image size for DeiT.
https://github.com/facebookresearch/deit
\nThank you!
", "rawMarkdown": "# Introduction\nDeiT is `DeiT: Data-efficient Image Transformers` like `ViT`.\n\n\n# New pretrained Weights\nNow, we can use `384x384` image size for DeiT.\n\n\n\n- DeiT-base 384\n- DeiT-base distilled 384 (1000 epochs)\n\nhttps://github.com/facebookresearch/deit\n\n# End\n- Official inference for DeiT(Colab)\n\nhttps://colab.research.google.com/github/facebookresearch/deit/blob/colab/notebooks/deit_inference.ipynb\n\nThank you!", "votes": null }, { "id": "1163816", "postDate": "01/22/2021 00:36:30", "content": "Thanks a lot for sharing ! I read up on DeiT on the last discussion and was wondering exactly where the 384 weights were! Definitely gonna use them since it should be an improvement on ViT models. Thanks again for sharing.
", "rawMarkdown": "Thanks a lot for sharing ! I read up on DeiT on the last discussion and was wondering exactly where the 384 weights were! Definitely gonna use them since it should be an improvement on ViT models. Thanks again for sharing.", "votes": null }, { "id": "1164430", "postDate": "01/22/2021 11:18:00", "content": "Hi @heroseo you have made some significant contributions in this competition , I just wanted to ask if you have tried DIET model yourself and if yes , can you share your cv/lb
\nThanks
I found new pretrained weights for DeiT yesterday. So, it needs experimentation using 384x384.
In case of 224x224, My cv and lb were not good. (public lb is lower than Resnext50_32x4d.)
What's the difference between deit without distill and vit?
", "rawMarkdown": "What's the difference between deit without distill and vit?", "votes": null }, { "id": "1164546", "postDate": "01/22/2021 12:46:21", "content": "Simply explain it, distill uses teacher models. @kewang777
", "rawMarkdown": "Simply explain it, distill uses teacher models. @kewang777", "votes": null }, { "id": "1165164", "postDate": "01/22/2021 18:37:03", "content": "My last submission made this morning was with deit 384 base (not distilled). CV 0.891, LB: 0.896, single model. I plan to train distilled version tonight, I can update if you want.
", "rawMarkdown": "My last submission made this morning was with deit 384 base (not distilled). CV 0.891, LB: 0.896, single model. I plan to train distilled version tonight, I can update if you want.", "votes": null }, { "id": "1165212", "postDate": "01/22/2021 19:20:04", "content": "@piantic thanks for being so kind and replying all my queries
\nAlso @claverru thanks for sharing the update , I am looking forward to experiment with DeiT as well
Thanks for sharing it. How can I use DeiT for 512*512 image size. Is there any way?
", "rawMarkdown": "Thanks for sharing it. How can I use DeiT for 512*512 image size. Is there any way?", "votes": null }, { "id": "1166770", "postDate": "01/23/2021 20:15:12", "content": "The Vision Transformer is conceptually different from architectures like resnet.
\nCurrently, it is not possible to use 512 image size for DeiT or ViT.
@piantic Thanks for the quick informatic replay.
", "rawMarkdown": "piantic Thanks for the quick informatic replay.", "votes": null }, { "id": "1166884", "postDate": "01/23/2021 21:44:04", "content": "I think it is actually possible (I have not tried though).
\nhttps://github.com/facebookresearch/deit/blob/main/main.py
\nLines 264-282 both included.
\nIn fact in the paper they mention something about pretraining with 224 and finetuning the same model with 384.
", "rawMarkdown": "I think it is actually possible (I have not tried though). \n\n[https://github.com/facebookresearch/deit/blob/main/main.py](https://github.com/facebookresearch/deit/blob/main/main.py)\n\nLines 264-282 both included.\n\nIn fact in the paper they mention something about pretraining with 224 and finetuning the same model with 384.", "votes": null }, { "id": "1166886", "postDate": "01/23/2021 21:45:58", "content": "I'm looking forward how to properly train the distilled version. I've read something about self-distilled finetuning, though I don't know if this would be possible/appropiate for our task.
", "rawMarkdown": "I'm looking forward how to properly train the distilled version. I've read something about self-distilled finetuning, though I don't know if this would be possible/appropiate for our task.", "votes": null }, { "id": "1166894", "postDate": "01/23/2021 22:03:19", "content": "Even if you modify the code, we can't use pretrained weights for 512 image size.
\n", "rawMarkdown": "Even if you modify the code, we can't use pretrained weights for 512 image size.\n\n@claverru", "votes": null }, { "id": "1184552", "postDate": "02/03/2021 15:18:51", "content": "Have you tried the DeiT ? How much LB you can get, thanks!
", "rawMarkdown": "Have you tried the DeiT ? How much LB you can get, thanks!", "votes": null }, { "id": "1194419", "postDate": "02/10/2021 07:33:17", "content": "@claverru i face weight format incompatibility issue… did face any thing
", "rawMarkdown": "claverru i face weight format incompatibility issue... did face any thing", "votes": null }, { "id": "1194567", "postDate": "02/10/2021 08:43:21", "content": "@claverru Is deit without distill the same with vit? I read the code but didn't find any difference….
", "rawMarkdown": "claverru Is deit without distill the same with vit? I read the code but didn't find any difference....", "votes": null }, { "id": "1194664", "postDate": "02/10/2021 09:50:03", "content": "Could you tell me how to load deit model without network for inference?
\nThe code
\ntorch.hub.load('facebookresearch/deit:main')
\nseems does not work
", "rawMarkdown": "Could you tell me how to load deit model without network for inference?\n\nThe code \n\ntorch.hub.load('facebookresearch/deit:main') \n\nseems does not work", "votes": null }, { "id": "1194995", "postDate": "02/10/2021 13:54:53", "content": "https://www.kaggle.com/piantic/how-to-finetuning-models-pytorch-xla-tpu
\nI updated DeiT using timm not torch.hub.
please see my notebook-v9. :)
\n@cuteffff
Thanks! @piantic
", "rawMarkdown": "Thanks! @piantic", "votes": null }, { "id": "1195846", "postDate": "02/11/2021 05:25:13", "content": "Thank you very much!
", "rawMarkdown": "Thank you very much!", "votes": null }, { "id": "1210242", "postDate": "02/19/2021 09:34:02", "content": "I was waiting until the end of the competition to say that yes, you can. I may release a public notebook with the explanation. In fact, one of my best submissions has a DeiT finetuned on 480 image size. Even more, I'm pretty sure you can do it two very different ways.
\n\nEdit: I've done it in one way, but I strongly believe there is another way.
", "rawMarkdown": "I was waiting until the end of the competition to say that yes, you can. I may release a public notebook with the explanation. In fact, one of my best submissions has a DeiT finetuned on 480 image size. Even more, I'm pretty sure you can do it two very different ways.\n\n@piantic \n\nEdit: I've done it in one way, but I strongly believe there is another way.", "votes": null }, { "id": "1210267", "postDate": "02/19/2021 09:57:49", "content": "If you share your solutions, everyone will learn something new. :)
\n", "rawMarkdown": "If you share your solutions, everyone will learn something new. :)\n\n@claverru", "votes": null } ], "comments": [ { "id": 1163816, "author_name": "capiru", "author_url": "", "post_date": "01/22/2021 00:36:30", "content": "Thanks a lot for sharing ! I read up on DeiT on the last discussion and was wondering exactly where the 384 weights were! Definitely gonna use them since it should be an improvement on ViT models. Thanks again for sharing.
", "votes": null, "replies": [] }, { "id": 1164430, "author_name": "tanulsingh077", "author_url": "", "post_date": "01/22/2021 11:18:00", "content": "Hi @heroseo you have made some significant contributions in this competition , I just wanted to ask if you have tried DIET model yourself and if yes , can you share your cv/lb
\nThanks
I found new pretrained weights for DeiT yesterday. So, it needs experimentation using 384x384.
In case of 224x224, My cv and lb were not good. (public lb is lower than Resnext50_32x4d.)
My last submission made this morning was with deit 384 base (not distilled). CV 0.891, LB: 0.896, single model. I plan to train distilled version tonight, I can update if you want.
", "votes": null, "replies": [] }, { "id": 1165212, "author_name": "tanulsingh077", "author_url": "", "post_date": "01/22/2021 19:20:04", "content": "@piantic thanks for being so kind and replying all my queries
\nAlso @claverru thanks for sharing the update , I am looking forward to experiment with DeiT as well
I'm looking forward how to properly train the distilled version. I've read something about self-distilled finetuning, though I don't know if this would be possible/appropiate for our task.
", "votes": null, "replies": [] }, { "id": 1194419, "author_name": "jaideepvalani", "author_url": "", "post_date": "02/10/2021 07:33:17", "content": "@claverru i face weight format incompatibility issue… did face any thing
", "votes": null, "replies": [] }, { "id": 1194567, "author_name": "kewang777", "author_url": "", "post_date": "02/10/2021 08:43:21", "content": "@claverru Is deit without distill the same with vit? I read the code but didn't find any difference….
", "votes": null, "replies": [] }, { "id": 1194664, "author_name": "cuteffff", "author_url": "", "post_date": "02/10/2021 09:50:03", "content": "Could you tell me how to load deit model without network for inference?
\nThe code
\ntorch.hub.load('facebookresearch/deit:main')
\nseems does not work
", "votes": null, "replies": [] }, { "id": 1194995, "author_name": "piantic", "author_url": "", "post_date": "02/10/2021 13:54:53", "content": "https://www.kaggle.com/piantic/how-to-finetuning-models-pytorch-xla-tpu
\nI updated DeiT using timm not torch.hub.
please see my notebook-v9. :)
\n@cuteffff
Thanks! @piantic
", "votes": null, "replies": [] } ] }, { "id": 1164543, "author_name": "kewang777", "author_url": "", "post_date": "01/22/2021 12:43:08", "content": "What's the difference between deit without distill and vit?
", "votes": null, "replies": [ { "id": 1164546, "author_name": "piantic", "author_url": "", "post_date": "01/22/2021 12:46:21", "content": "Simply explain it, distill uses teacher models. @kewang777
", "votes": null, "replies": [] } ] }, { "id": 1166750, "author_name": "durbin164", "author_url": "", "post_date": "01/23/2021 19:58:17", "content": "Thanks for sharing it. How can I use DeiT for 512*512 image size. Is there any way?
", "votes": null, "replies": [ { "id": 1166770, "author_name": "piantic", "author_url": "", "post_date": "01/23/2021 20:15:12", "content": "The Vision Transformer is conceptually different from architectures like resnet.
\nCurrently, it is not possible to use 512 image size for DeiT or ViT.
@piantic Thanks for the quick informatic replay.
", "votes": null, "replies": [] }, { "id": 1166884, "author_name": "claverru", "author_url": "", "post_date": "01/23/2021 21:44:04", "content": "I think it is actually possible (I have not tried though).
\nhttps://github.com/facebookresearch/deit/blob/main/main.py
\nLines 264-282 both included.
\nIn fact in the paper they mention something about pretraining with 224 and finetuning the same model with 384.
", "votes": null, "replies": [] }, { "id": 1166894, "author_name": "piantic", "author_url": "", "post_date": "01/23/2021 22:03:19", "content": "Even if you modify the code, we can't use pretrained weights for 512 image size.
\n", "votes": null, "replies": [] }, { "id": 1210242, "author_name": "claverru", "author_url": "", "post_date": "02/19/2021 09:34:02", "content": "I was waiting until the end of the competition to say that yes, you can. I may release a public notebook with the explanation. In fact, one of my best submissions has a DeiT finetuned on 480 image size. Even more, I'm pretty sure you can do it two very different ways.
\n\nEdit: I've done it in one way, but I strongly believe there is another way.
", "votes": null, "replies": [] }, { "id": 1210267, "author_name": "piantic", "author_url": "", "post_date": "02/19/2021 09:57:49", "content": "If you share your solutions, everyone will learn something new. :)
\n", "votes": null, "replies": [] } ] }, { "id": 1184552, "author_name": "clw5180", "author_url": "", "post_date": "02/03/2021 15:18:51", "content": "Have you tried the DeiT ? How much LB you can get, thanks!
", "votes": null, "replies": [] }, { "id": 1195846, "author_name": "", "author_url": "", "post_date": "02/11/2021 05:25:13", "content": "Thank you very much!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1163530": "# Introduction\nDeiT is `DeiT: Data-efficient Image Transformers` like `ViT`.\n\n\n# New pretrained Weights\nNow, we can use `384x384` image size for DeiT.\n\n\n\n- DeiT-base 384\n- DeiT-base distilled 384 (1000 epochs)\n\nhttps://github.com/facebookresearch/deit\n\n# End\n- Official inference for DeiT(Colab)\n\nhttps://colab.research.google.com/github/facebookresearch/deit/blob/colab/notebooks/deit_inference.ipynb\n\nThank you!", "1163816": "Thanks a lot for sharing ! I read up on DeiT on the last discussion and was wondering exactly where the 384 weights were! Definitely gonna use them since it should be an improvement on ViT models. Thanks again for sharing.", "1164430": "Hi @heroseo you have made some significant contributions in this competition , I just wanted to ask if you have tried DIET model yourself and if yes , can you share your cv/lb\nThanks", "1164444": "I found new pretrained weights for DeiT yesterday. So, it needs experimentation using `384x384`.\n\nIn case of `224x224`, My cv and lb were not good. (public lb is lower than [Resnext50_32x4d](https://www.kaggle.com/piantic/no-tta-cassava-resnext50-32x4d-inference-lb0-903).)\n\n@tanulsingh077", "1164543": "What's the difference between deit without distill and vit?", "1164546": "Simply explain it, distill uses teacher models. @kewang777", "1165164": "My last submission made this morning was with deit 384 base (not distilled). CV 0.891, LB: 0.896, single model. I plan to train distilled version tonight, I can update if you want.", "1165212": "piantic thanks for being so kind and replying all my queries \nAlso @claverru thanks for sharing the update , I am looking forward to experiment with DeiT as well", "1166750": "Thanks for sharing it. How can I use DeiT for 512*512 image size. Is there any way?", "1166770": "The Vision Transformer is conceptually different from architectures like resnet.\nCurrently, it is not possible to use 512 image size for DeiT or ViT.\n\n@durbin164", "1166774": "piantic Thanks for the quick informatic replay.", "1166884": "I think it is actually possible (I have not tried though). \n\n[https://github.com/facebookresearch/deit/blob/main/main.py](https://github.com/facebookresearch/deit/blob/main/main.py)\n\nLines 264-282 both included.\n\nIn fact in the paper they mention something about pretraining with 224 and finetuning the same model with 384.", "1166886": "I'm looking forward how to properly train the distilled version. I've read something about self-distilled finetuning, though I don't know if this would be possible/appropiate for our task.", "1166894": "Even if you modify the code, we can't use pretrained weights for 512 image size.\n\n@claverru", "1184552": "Have you tried the DeiT ? How much LB you can get, thanks!", "1194419": "claverru i face weight format incompatibility issue... did face any thing", "1194567": "claverru Is deit without distill the same with vit? I read the code but didn't find any difference....", "1194664": "Could you tell me how to load deit model without network for inference?\n\nThe code \n\ntorch.hub.load('facebookresearch/deit:main') \n\nseems does not work", "1194995": "https://www.kaggle.com/piantic/how-to-finetuning-models-pytorch-xla-tpu\n\nI updated `DeiT` using `timm` not torch.hub.\n\nplease see my notebook-v9. :)\n@cuteffff", "1195084": "Thanks! @piantic", "1195846": "Thank you very much!", "1210242": "I was waiting until the end of the competition to say that yes, you can. I may release a public notebook with the explanation. In fact, one of my best submissions has a DeiT finetuned on 480 image size. Even more, I'm pretty sure you can do it two very different ways.\n\n@piantic \n\nEdit: I've done it in one way, but I strongly believe there is another way.", "1210267": "If you share your solutions, everyone will learn something new. :)\n\n@claverru" }, "source": "meta" }