![\"Comparison](\"https://raw.githubusercontent.com/sail-sg/volo/main/figures/compare.png\")
A new paper on Vision Transfomer was submitted to arXiv three days ago. It's called VOLO: Vision Outlooker.
\nLi Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, Shuicheng Yan
\nVOLO-D5, which is the largest in the VOLO models, achieved 87.1% top-1 accuracy on ImageNet-1K classification without using any extra training data.
\n
To our delight, the implementation and pretrained weights has already been published by the authors [GitHub] .
\nI've uploaded them on Kaggle Datasets. Now you can use them from Kaggle Notebooks:
\nhttps://www.kaggle.com/ttahara/volo-package
I've published a new baseline using VOLO-D1 as an example of use it. The experimental setup is roughly the same as the my previous resnet18d baseline.
\n\nThis baseline got 0.727 with image_size=256x256.
\nI'm thinking that it might achieve better score and could be useful for ensemble.
| fold ＼ model | \nResNet18D(320x320) | \nVOLO-D1(256x256) | \n
|---|---|---|
| 0 | \n0.8358 | \n0.8501 | \n
| 1 | \n0.8147 | \n0.8266 | \n
| 2 | \n0.8242 | \n0.8328 | \n
| 3 | \n0.8250 | \n0.8354 | \n
| 4 | \n0.8276 | \n0.8368 | \n
| OOF | \n0.8248 | \n0.8362 | \n
| Public LB | \n0.726 | \n0.727 | \n
\nI published the notebooks before updating competition data for rapid reporting.
\nI will re-run these notebooks after update ;)
A new paper on Vision Transfomer was submitted to arXiv three days ago. It's called VOLO: Vision Outlooker.
\nLi Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, Shuicheng Yan
\nVOLO-D5, which is the largest in the VOLO models, achieved 87.1% top-1 accuracy on ImageNet-1K classification without using any extra training data.
\n
To our delight, the implementation and pretrained weights has already been published by the authors [GitHub] .
\nI've uploaded them on Kaggle Datasets. Now you can use them from Kaggle Notebooks:
\nhttps://www.kaggle.com/ttahara/volo-package
I've published a new baseline using VOLO-D1 as an example of use it. The experimental setup is roughly the same as the my previous resnet18d baseline.
\n\nThis baseline got 0.727 with image_size=256x256.
\nI'm thinking that it might achieve better score and could be useful for ensemble.
| fold ＼ model | \nResNet18D(320x320) | \nVOLO-D1(256x256) | \n
|---|---|---|
| 0 | \n0.8358 | \n0.8501 | \n
| 1 | \n0.8147 | \n0.8266 | \n
| 2 | \n0.8242 | \n0.8328 | \n
| 3 | \n0.8250 | \n0.8354 | \n
| 4 | \n0.8276 | \n0.8368 | \n
| OOF | \n0.8248 | \n0.8362 | \n
| Public LB | \n0.726 | \n0.727 | \n
\nI published the notebooks before updating competition data for rapid reporting.
\nI will re-run these notebooks after update ;)
model:volo_d1-384
\nimage_size:640
\n1fold:cv 99.01 lb 97.7
Thank you for sharing.
\nThe result is worse than I expected 🤔
If you don't mind, can you tell me about your experimental setup?
", "rawMarkdown": "Thank you for sharing. \nThe result is worse than I expected 🤔\n\nIf you don't mind, can you tell me about your experimental setup?", "votes": 1 }, { "id": 1372747, "postDate": "2021-07-02T01:48:52.380Z", "content": "@ttahara
\nlr=1e-4
\nepochs=25
\ncutmix+mixup 1:1
\ndouble rtx3090 1fold 50000 + s
\nObviously, GPU is a little hard to run. It needs TPU!
@zhangeng
\nThanks. Sorry for late reply.
\nYes, ViTs including this model require more and more GPUs or TPUs 😓
Here are a few my thoughts for training VOLO.
\n640x640 is so large … I think it is good to start from smaller size (384x384 or 512x512).
\nIn paper, authors recommend us to set learning rate 1.6e-3 ×batch_size / 1024 for VOLO-D1 and weight decay rate 5.0e-2 for all VOLO models. (see Table 3)
In addition to normal data augmentations, authors used Stochastic Depth and Token Labeling objective with MixToken.
\nI don't know much about the latter, and it seems a bit difficult to use this because we need to prepare token labels.
\nOn the other hand, the former can be easily used by setting a hyperparameter drop_path_rate.
import timm\nfrom volo.models import volo_d1 # register model to timm\nfrom volo.utils import load_pretrained_weights\n\npretrained_model_path = \"../input/volo-package/d1_384_85.2.pth.tar\"\n\nmodel = timm.create_model(\n \"volo_d1\", img_size=384,\n mix_token=False, return_dense=False, drop_path_rate=0.1)\n\nload_pretrained_weights(model, pretrained_model_path, strict=False)\nI learned two things
\n1.lr=> learning rate 1.6e-3 ×batch_size / 1024
\n2.setting a hyperparameter drop_path_rate.
\nThank you for your help. After the data is updated, I must try again to change the hyperparameter in these two places
Update: I ran training & inference notebooks again for the updated competition dataset.
", "rawMarkdown": "**Update**: I ran training & inference notebooks again for the updated competition dataset.", "votes": 2 }, { "id": 1370815, "postDate": "2021-06-30T12:25:12.037Z", "content": "I am trying to implement VOLO in tensorflow. Any Help would be appreciated. You can Dm if you want to help (sorry if this is Off topic).
", "rawMarkdown": "I am trying to implement VOLO in tensorflow. Any Help would be appreciated. You can Dm if you want to help (sorry if this is Off topic).", "votes": 2, "replies": [ { "id": 1371433, "postDate": "2021-07-01T02:19:04.107Z", "content": "hello!Even if the tensorflow Volo model is implemented, how to transfer the weight is still a problem!
", "rawMarkdown": "hello!Even if the tensorflow Volo model is implemented, how to transfer the weight is still a problem!" }, { "id": 1371453, "postDate": "2021-07-01T02:49:43.260Z", "content": "@zhangeng that surely is a problem but we can retrain the model on ImageNet maybe
", "rawMarkdown": "@zhangeng that surely is a problem but we can retrain the model on ImageNet maybe", "votes": 1 } ] }, { "id": 1408284, "postDate": "2021-08-02T12:00:52.420Z", "content": "When I apply your code:
\nmodel_name = base_name.split(\"-\")[0]\nassert timm.is_model(model_name), \"you can use only models in timm.\"\nI found that there is no volo_d1 in my timm library. How can I add volo to the timm library?
", "rawMarkdown": "When I apply your code:\n```python\nmodel_name = base_name.split(\"-\")[0]\nassert timm.is_model(model_name), \"you can use only models in timm.\"\n```\n I found that there is no volo_d1 in my timm library. How can I add volo to the timm library?" }, { "id": 1371700, "postDate": "2021-07-01T07:24:58.717Z", "content": "interesting
", "rawMarkdown": "interesting" }, { "id": 1367594, "postDate": "2021-06-27T22:35:47.410Z", "content": "thx - VERY interesting!!
\nDid you do any testing with droprate? 0.5 seems rather high….
", "rawMarkdown": "thx - VERY interesting!!\n\nDid you do any testing with droprate? 0.5 seems rather high....", "replies": [ { "id": 1369810, "postDate": "2021-06-29T15:52:03.503Z", "content": "Thanks.
\nWhich drop_rate are you referring to?
", "rawMarkdown": "Thanks.\n\nWhich drop_rate are you referring to?", "votes": 2 }, { "id": 1370521, "postDate": "2021-06-30T08:08:10.603Z", "content": "sorry for being not exact. I meant
\n nn.ReLU(), nn.Dropout(0.5),]) in the head of the volo model.
\nEfficientNets are trained with 0.2-0.4 dropout in the head as proposed by the paper. So 0.5 seems quite a heavy regularisation.
OK, I see.
\n# # prepare head classifier\nif dims_head[0] is None:\n dims_head[0] = in_features\n\nlayers_list = []\nfor i in range(len(dims_head) - 2):\n in_dim, out_dim = dims_head[i: i + 2]\n layers_list.extend([\n nn.Linear(in_dim, out_dim),\n nn.ReLU(), nn.Dropout(0.5),])\nlayers_list.append(\n nn.Linear(dims_head[-2], dims_head[-1]))\nself.head = nn.Sequential(*layers_list)\nI write the code for customizing head classifier.
\nBut I used a linear layer as head in this notebook (by setting dims_head = [384, 1] )
Is there any tensorflow implementation out There ?
", "rawMarkdown": "Is there any tensorflow implementation out There ?", "replies": [ { "id": 1367138, "postDate": "2021-06-27T13:01:07.797Z", "content": "As far as I know, any tensorflow implementation doesn't exist. It's only been three days since the paper was published.
\nI used the official pytorch implementation by authors.
", "rawMarkdown": "As far as I know, any tensorflow implementation doesn't exist. It's only been three days since the paper was published.\n\nI used the official pytorch implementation by authors.", "votes": 1 } ] }, { "id": 1369049, "postDate": "2021-06-29T05:35:28.740Z", "content": "cool! thanks for sharing!
", "rawMarkdown": "cool! thanks for sharing!" } ], "comments": [ { "id": 1371431, "author_name": "zhangeng", "author_url": "", "post_date": "2021-07-01T02:17:09.180000", "content": "model:volo_d1-384
\nimage_size:640
\n1fold:cv 99.01 lb 97.7
Thank you for sharing.
\nThe result is worse than I expected 🤔
If you don't mind, can you tell me about your experimental setup?
", "votes": 1, "replies": [] }, { "id": 1372747, "author_name": "zhangeng", "author_url": "", "post_date": "2021-07-02T01:48:52.380000", "content": "@ttahara
\nlr=1e-4
\nepochs=25
\ncutmix+mixup 1:1
\ndouble rtx3090 1fold 50000 + s
\nObviously, GPU is a little hard to run. It needs TPU!
@zhangeng
\nThanks. Sorry for late reply.
\nYes, ViTs including this model require more and more GPUs or TPUs 😓
Here are a few my thoughts for training VOLO.
\n640x640 is so large … I think it is good to start from smaller size (384x384 or 512x512).
\nIn paper, authors recommend us to set learning rate 1.6e-3 ×batch_size / 1024 for VOLO-D1 and weight decay rate 5.0e-2 for all VOLO models. (see Table 3)
In addition to normal data augmentations, authors used Stochastic Depth and Token Labeling objective with MixToken.
\nI don't know much about the latter, and it seems a bit difficult to use this because we need to prepare token labels.
\nOn the other hand, the former can be easily used by setting a hyperparameter drop_path_rate.
import timm\nfrom volo.models import volo_d1 # register model to timm\nfrom volo.utils import load_pretrained_weights\n\npretrained_model_path = \"../input/volo-package/d1_384_85.2.pth.tar\"\n\nmodel = timm.create_model(\n \"volo_d1\", img_size=384,\n mix_token=False, return_dense=False, drop_path_rate=0.1)\n\nload_pretrained_weights(model, pretrained_model_path, strict=False)\nI learned two things
\n1.lr=> learning rate 1.6e-3 ×batch_size / 1024
\n2.setting a hyperparameter drop_path_rate.
\nThank you for your help. After the data is updated, I must try again to change the hyperparameter in these two places
Update: I ran training & inference notebooks again for the updated competition dataset.
", "votes": 2, "replies": [] }, { "id": 1370815, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-06-30T12:25:12.037000", "content": "I am trying to implement VOLO in tensorflow. Any Help would be appreciated. You can Dm if you want to help (sorry if this is Off topic).
", "votes": 2, "replies": [ { "id": 1371433, "author_name": "zhangeng", "author_url": "", "post_date": "2021-07-01T02:19:04.107000", "content": "hello!Even if the tensorflow Volo model is implemented, how to transfer the weight is still a problem!
", "votes": 0, "replies": [] }, { "id": 1371453, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-07-01T02:49:43.260000", "content": "@zhangeng that surely is a problem but we can retrain the model on ImageNet maybe
", "votes": 1, "replies": [] } ] }, { "id": 1408284, "author_name": "Gainover", "author_url": "", "post_date": "2021-08-02T12:00:52.420000", "content": "When I apply your code:
\nmodel_name = base_name.split(\"-\")[0]\nassert timm.is_model(model_name), \"you can use only models in timm.\"\nI found that there is no volo_d1 in my timm library. How can I add volo to the timm library?
", "votes": 0, "replies": [] }, { "id": 1371700, "author_name": "karthikeyan", "author_url": "", "post_date": "2021-07-01T07:24:58.717000", "content": "interesting
", "votes": 0, "replies": [] }, { "id": 1367594, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2021-06-27T22:35:47.410000", "content": "thx - VERY interesting!!
\nDid you do any testing with droprate? 0.5 seems rather high….
", "votes": 0, "replies": [ { "id": 1369810, "author_name": "Tawara", "author_url": "", "post_date": "2021-06-29T15:52:03.503000", "content": "Thanks.
\nWhich drop_rate are you referring to?
", "votes": 2, "replies": [] }, { "id": 1370521, "author_name": "Roman Weilguny", "author_url": "", "post_date": "2021-06-30T08:08:10.603000", "content": "sorry for being not exact. I meant
\n nn.ReLU(), nn.Dropout(0.5),]) in the head of the volo model.
\nEfficientNets are trained with 0.2-0.4 dropout in the head as proposed by the paper. So 0.5 seems quite a heavy regularisation.
OK, I see.
\n# # prepare head classifier\nif dims_head[0] is None:\n dims_head[0] = in_features\n\nlayers_list = []\nfor i in range(len(dims_head) - 2):\n in_dim, out_dim = dims_head[i: i + 2]\n layers_list.extend([\n nn.Linear(in_dim, out_dim),\n nn.ReLU(), nn.Dropout(0.5),])\nlayers_list.append(\n nn.Linear(dims_head[-2], dims_head[-1]))\nself.head = nn.Sequential(*layers_list)\nI write the code for customizing head classifier.
\nBut I used a linear layer as head in this notebook (by setting dims_head = [384, 1] )
Is there any tensorflow implementation out There ?
", "votes": 0, "replies": [ { "id": 1367138, "author_name": "Tawara", "author_url": "", "post_date": "2021-06-27T13:01:07.797000", "content": "As far as I know, any tensorflow implementation doesn't exist. It's only been three days since the paper was published.
\nI used the official pytorch implementation by authors.
", "votes": 1, "replies": [] } ] }, { "id": 1369049, "author_name": "Tian", "author_url": "", "post_date": "2021-06-29T05:35:28.740000", "content": "cool! thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1366604": "A new paper on Vision Transfomer was submitted to arXiv three days ago. It's called _VOLO: Vision Outlooker_.\n\n##### VOLO: Vision Outlooker for Visual Recognition [[arXiv:2106.13112](https://arxiv.org/abs/2106.13112)] \n Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, Shuicheng Yan\n\nVOLO-D5, which is the largest in the VOLO models, achieved 87.1% top-1 accuracy on ImageNet-1K classification **without using any extra training data**.\n\n\n\nTo our delight, the implementation and pretrained weights has already been published by the authors [[GitHub](https://github.com/sail-sg/volo)] .\nI've uploaded them **on Kaggle Datasets**. Now you can use them from Kaggle Notebooks:\nhttps://www.kaggle.com/ttahara/volo-package\n\nI've published a new baseline using _VOLO-D1_ as an example of use it. The experimental setup is roughly the same as the [my previous resnet18d baseline](https://www.kaggle.com/ttahara/rerun-seti-e-t-resnet18d-baseline).\n\n* Training([fold0](https://www.kaggle.com/ttahara/rerun-seti-e-t-volo-d1-baseline-training?scriptVersionId=68249988), [fold1](https://www.kaggle.com/ttahara/rerun-seti-e-t-volo-d1-baseline-training?scriptVersionId=68250008), [fold2](https://www.kaggle.com/ttahara/rerun-seti-e-t-volo-d1-baseline-training?scriptVersionId=68281727), [fold3](https://www.kaggle.com/ttahara/rerun-seti-e-t-volo-d1-baseline-training?scriptVersionId=68281737), [fold4](https://www.kaggle.com/ttahara/rerun-seti-e-t-volo-d1-baseline-training?scriptVersionId=68335611))\n* [Inference Notebook](https://www.kaggle.com/ttahara/rerun-seti-e-t-volo-d1-baseline-inference?scriptVersionId=68361639)\n\nThis baseline got ~~0.974~~ 0.727 **with image_size=256x256**.\nI'm thinking that it might achieve better score and could be useful for ensemble.\n\n| fold ＼ model | ResNet18D(**320x320**) | _VOLO-D1_(**256x256**)|\n|:-------------:|:-----------------------:|:----------------------:|\n| 0 | ~~0.9827~~ 0.8358 | ~~0.9817~~ 0.8501 |\n| 1 | ~~0.9817~~ 0.8147 | ~~0.9833~~ 0.8266 |\n| 2 | ~~0.9843~~ 0.8242 | ~~0.9835~~ 0.8328 |\n| 3 | ~~0.9828~~ 0.8250 | ~~0.9843~~ 0.8354 |\n| 4 | ~~0.9842~~ 0.8276 | ~~0.9835~~ 0.8368 |\n| OOF | ~~0.9830~~ 0.8248 | ~~0.9831~~ 0.8362 |\n| Public LB | ~~0.974~~ 0.726 | ~~0.974~~ 0.727 |\n\n