Hi all,
\nI found this paper very interesting: https://arxiv.org/pdf/2012.12877.pdf
\nHowever, I noticed there were not any implementations that easily allowed usage of pretrained models. So, I went ahead and made one!
https://github.com/Graeme22/DistillableVisionTransformer
\nI hope this is useful to someone. I haven't tested it for the competition yet but I believe it will make life much easier for those using transformers. Enjoy!
", "messages": [ { "id": 1136486, "postDate": "2021-01-03T06:00:36.547Z", "content": "Hi all,
\nI found this paper very interesting: https://arxiv.org/pdf/2012.12877.pdf
\nHowever, I noticed there were not any implementations that easily allowed usage of pretrained models. So, I went ahead and made one!
https://github.com/Graeme22/DistillableVisionTransformer
\nI hope this is useful to someone. I haven't tested it for the competition yet but I believe it will make life much easier for those using transformers. Enjoy!
", "rawMarkdown": "Hi all,\n\nI found this paper very interesting: https://arxiv.org/pdf/2012.12877.pdf\nHowever, I noticed there were not any implementations that easily allowed usage of pretrained models. So, I went ahead and made one!\n\nhttps://github.com/Graeme22/DistillableVisionTransformer\n\nI hope this is useful to someone. I haven't tested it for the competition yet but I believe it will make life much easier for those using transformers. Enjoy!", "votes": 4 }, { "id": 1148709, "postDate": "2021-01-11T10:31:48.257Z", "content": "Sorry, didn't work.
\nteacher = ResNext(LB = 0.9), student = ViT_B-16(LB = 0.895)。
\nBut the distillation model LB = 0.888.
\nSomewhat doubt the paper ------- only tricks😊
Hi, sorry it didn't provide an improvement for you. As I mentioned, I haven't tested it for this competition, but some things I would try would be larger batch size on TPUs, maybe start the student from scratch instead of pretrained, or mess with the loss hyperparameters. One of the things that is quite important for training transformers is a high volume of data, so perhaps some self-supervised technique on the old competition data could be helpful. Hope that gives you some ideas, good luck!
", "rawMarkdown": "Hi, sorry it didn't provide an improvement for you. As I mentioned, I haven't tested it for this competition, but some things I would try would be larger batch size on TPUs, maybe start the student from scratch instead of pretrained, or mess with the loss hyperparameters. One of the things that is quite important for training transformers is a high volume of data, so perhaps some self-supervised technique on the old competition data could be helpful. Hope that gives you some ideas, good luck!" }, { "id": 1149840, "postDate": "2021-01-12T07:33:54.313Z", "content": "Totally agree with you about 'start the student from scratch'. It turns out to be a fancy solution.
\nFacebook guys save some time, but it is still a lot(300e).
\nThank you anyway, the code is great!
Sorry, didn't work.
\nteacher = ResNext(LB = 0.9), student = ViT_B-16(LB = 0.895)。
\nBut the distillation model LB = 0.888.
\nSomewhat doubt the paper ------- only tricks😊
Hi, sorry it didn't provide an improvement for you. As I mentioned, I haven't tested it for this competition, but some things I would try would be larger batch size on TPUs, maybe start the student from scratch instead of pretrained, or mess with the loss hyperparameters. One of the things that is quite important for training transformers is a high volume of data, so perhaps some self-supervised technique on the old competition data could be helpful. Hope that gives you some ideas, good luck!
", "votes": 0, "replies": [] }, { "id": 1149840, "author_name": "wangbingo", "author_url": "", "post_date": "2021-01-12T07:33:54.313000", "content": "Totally agree with you about 'start the student from scratch'. It turns out to be a fancy solution.
\nFacebook guys save some time, but it is still a lot(300e).
\nThank you anyway, the code is great!