TL;DR: A new paper by Wortsman et al. proposes a simple approach (coined model soup) that often improves accuracy by averaging the weights of models fine-tuned independently, with no additional training and no additional cost at inference time. This approach is often more effective than the conventional recipe for maximizing accuracy and selecting the best model which involves: 1) training models with various hyperparameters and 2) picking the best model on the held-out validation set when fine-tuning.
\nResults: A fine-tuned ViT-G model attains 90.94% top-1 accuracy on ImageNet (a new state-of-the-art).
", "messages": [ { "id": "1719071", "postDate": "03/11/2022 12:55:58", "content": "TL;DR: A new paper by Wortsman et al. proposes a simple approach (coined model soup) that often improves accuracy by averaging the weights of models fine-tuned independently, with no additional training and no additional cost at inference time. This approach is often more effective than the conventional recipe for maximizing accuracy and selecting the best model which involves: 1) training models with various hyperparameters and 2) picking the best model on the held-out validation set when fine-tuning.
\nResults: A fine-tuned ViT-G model attains 90.94% top-1 accuracy on ImageNet (a new state-of-the-art).
", "rawMarkdown": "## Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time\n\n[Paper.](https://arxiv.org/pdf/2203.05482v1.pdf)\n[References.](https://www.linkedin.com/posts/papers-with-code_machinelearning-deeplearning-activity-6908011230762778625-lBLG)\n\n**TL;DR:** A new paper by Wortsman et al. proposes a simple approach (coined model soup) that often improves accuracy by averaging the weights of models fine-tuned independently, with no additional training and no additional cost at inference time. This approach is often more effective than the conventional recipe for maximizing accuracy and selecting the best model which involves: 1) training models with various hyperparameters and 2) picking the best model on the held-out validation set when fine-tuning. \n\nResults: **A fine-tuned ViT-G model attains 90.94% top-1 accuracy on ImageNet (a new state-of-the-art).**", "votes": null }, { "id": "1719221", "postDate": "03/11/2022 15:15:41", "content": "Have you tried it?
", "rawMarkdown": "Have you tried it?", "votes": null }, { "id": "1719237", "postDate": "03/11/2022 15:32:21", "content": "@ipythonx Thanks for sharing. How often we overlook the power of simplistic approaches in our obsession with something 'advanced'.
", "rawMarkdown": "ipythonx Thanks for sharing. How often we overlook the power of simplistic approaches in our obsession with something 'advanced'.", "votes": null }, { "id": "1719247", "postDate": "03/11/2022 15:39:45", "content": "Have you tried it?
", "rawMarkdown": "Have you tried it?", "votes": null }, { "id": "1719333", "postDate": "03/11/2022 16:31:36", "content": "paper with no code.
", "rawMarkdown": "paper with no code.", "votes": null }, { "id": "1720156", "postDate": "03/12/2022 14:15:45", "content": "To my understanding this does not really have any affect on kaggle competitions since we don't care about inference time and it does not beat ensembles.
", "rawMarkdown": "To my understanding this does not really have any affect on kaggle competitions since we don't care about inference time and it does not beat ensembles.", "votes": null }, { "id": "1720368", "postDate": "03/12/2022 18:15:28", "content": "\n\non kaggle competitions since we don't care about inference time
\n
that maybe holds here but it's not true for code competitions
", "rawMarkdown": "> on kaggle competitions since we don't care about inference time\n\nthat maybe holds here but it's not true for code competitions", "votes": null } ], "comments": [ { "id": 1719221, "author_name": "kwentar", "author_url": "", "post_date": "03/11/2022 15:15:41", "content": "Have you tried it?
", "votes": null, "replies": [] }, { "id": 1719237, "author_name": "gianetan", "author_url": "", "post_date": "03/11/2022 15:32:21", "content": "@ipythonx Thanks for sharing. How often we overlook the power of simplistic approaches in our obsession with something 'advanced'.
", "votes": null, "replies": [] }, { "id": 1719247, "author_name": "yongdawoon", "author_url": "", "post_date": "03/11/2022 15:39:45", "content": "Have you tried it?
", "votes": null, "replies": [] }, { "id": 1719333, "author_name": "dragonzhang", "author_url": "", "post_date": "03/11/2022 16:31:36", "content": "paper with no code.
", "votes": null, "replies": [] }, { "id": 1720156, "author_name": "tvaranka", "author_url": "", "post_date": "03/12/2022 14:15:45", "content": "To my understanding this does not really have any affect on kaggle competitions since we don't care about inference time and it does not beat ensembles.
", "votes": null, "replies": [ { "id": 1720368, "author_name": "imeintanis", "author_url": "", "post_date": "03/12/2022 18:15:28", "content": "\n\non kaggle competitions since we don't care about inference time
\n
that maybe holds here but it's not true for code competitions
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1719071": "## Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time\n\n[Paper.](https://arxiv.org/pdf/2203.05482v1.pdf)\n[References.](https://www.linkedin.com/posts/papers-with-code_machinelearning-deeplearning-activity-6908011230762778625-lBLG)\n\n**TL;DR:** A new paper by Wortsman et al. proposes a simple approach (coined model soup) that often improves accuracy by averaging the weights of models fine-tuned independently, with no additional training and no additional cost at inference time. This approach is often more effective than the conventional recipe for maximizing accuracy and selecting the best model which involves: 1) training models with various hyperparameters and 2) picking the best model on the held-out validation set when fine-tuning. \n\nResults: **A fine-tuned ViT-G model attains 90.94% top-1 accuracy on ImageNet (a new state-of-the-art).**", "1719221": "Have you tried it?", "1719237": "ipythonx Thanks for sharing. How often we overlook the power of simplistic approaches in our obsession with something 'advanced'.", "1719247": "Have you tried it?", "1719333": "paper with no code.", "1720156": "To my understanding this does not really have any affect on kaggle competitions since we don't care about inference time and it does not beat ensembles.", "1720368": "> on kaggle competitions since we don't care about inference time\n\nthat maybe holds here but it's not true for code competitions" }, "source": "meta" }