I rerun @kaerunantoka ' s training notebook in my local machine with larger batch size, which is set to 128. In order to keep optimization iteration as the same roughly, I increase the training epoch from 23 to 200 (and lower lr a little).
\nThe iteration step of public notebook is (18452 / 5 * 4) / 16 * 23 = 17079, gaining 0.71 in the public leaderboard. And mine is (18452 / 5 * 4) / 128 * 200 = 18565, gaining 0.68 instead.
\nWhat are the possible reasons behind it?
", "messages": [ { "id": 1784223, "postDate": "2022-05-11T03:24:01.153Z", "content": "I faced a similar problem. Perhaps the cause is the optimal threshold.
\nThe optimal threshold varies widely by epoch, iteration, and lr. Even by 0.01 up or down, the score can vary greatly.
\nUnfortunately, the only way to find the optimal threshold is through trial and trial.
The paper [1] discusses the problem of learning difficulty at large batch sizes.
\nThis paper focuses on the problem that existing learning algorithms do not scale to large batch sizes, and proposes a learning algorithm called LARS that adapts to large batch sizes.
Another problem with some learning algorithms is that the optimal learning rate shifts when the batch size is changed [2]. For example, YOLOv5 addresses this problem by adopting a policy of setting the learning rate proportional to the batch size.
\n[1] LARS: https://arxiv.org/abs/1708.03888
\n[2] https://arxiv.org/abs/1706.02677
There are other learning algorithm addressing large batch size:
\npytorch lightning has its own implementation of LARS.
\nI haven't use this yet, but this might solve the problem with large batch size.
\nhttps://lightning-flash.readthedocs.io/en/0.5.0/api/generated/flash.core.optimizers.LARS.html
I rerun @kaerunantoka ' s training notebook in my local machine with larger batch size, which is set to 128. In order to keep optimization iteration as the same roughly, I increase the training epoch from 23 to 200 (and lower lr a little).
\nThe iteration step of public notebook is (18452 / 5 * 4) / 16 * 23 = 17079, gaining 0.71 in the public leaderboard. And mine is (18452 / 5 * 4) / 128 * 200 = 18565, gaining 0.68 instead.
\nWhat are the possible reasons behind it?
", "rawMarkdown": "I rerun @kaerunantoka ' s [training notebook](https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0) in my local machine with larger batch size, which is set to 128. In order to keep optimization iteration as the same roughly, I increase the training epoch from 23 to 200 (and lower lr a little). \n\nThe iteration step of public notebook is (18452 / 5 * 4) / 16 * 23 = 17079, gaining **0.71** in the public leaderboard. And mine is (18452 / 5 * 4) / 128 * 200 = 18565, gaining **0.68** instead.\n\nWhat are the possible reasons behind it?", "votes": 1 } ], "comments": [ { "id": 1784223, "author_name": "shinmura0", "author_url": "", "post_date": "2022-05-11T03:24:01.153000", "content": "I faced a similar problem. Perhaps the cause is the optimal threshold.
\nThe optimal threshold varies widely by epoch, iteration, and lr. Even by 0.01 up or down, the score can vary greatly.
\nUnfortunately, the only way to find the optimal threshold is through trial and trial.
The paper [1] discusses the problem of learning difficulty at large batch sizes.
\nThis paper focuses on the problem that existing learning algorithms do not scale to large batch sizes, and proposes a learning algorithm called LARS that adapts to large batch sizes.
Another problem with some learning algorithms is that the optimal learning rate shifts when the batch size is changed [2]. For example, YOLOv5 addresses this problem by adopting a policy of setting the learning rate proportional to the batch size.
\n[1] LARS: https://arxiv.org/abs/1708.03888
\n[2] https://arxiv.org/abs/1706.02677
There are other learning algorithm addressing large batch size:
\npytorch lightning has its own implementation of LARS.
\nI haven't use this yet, but this might solve the problem with large batch size.
\nhttps://lightning-flash.readthedocs.io/en/0.5.0/api/generated/flash.core.optimizers.LARS.html