Sometimes my model firstly converges to around 100, then diverges, what could be the problem? The loss oscillates around 180
", "messages": [ { "id": 3185239, "postDate": "2025-04-23T03:30:57.790Z", "content": "Sometimes my model firstly converges to around 100, then diverges, what could be the problem? The loss oscillates around 180
", "rawMarkdown": "Sometimes my model firstly converges to around 100, then diverges, what could be the problem? The loss oscillates around 180", "votes": 9 }, { "id": 3186957, "postDate": "2025-04-25T10:45:52.657Z", "content": "The model diverging problem is resolved by using larger batch size.
", "rawMarkdown": "The model diverging problem is resolved by using larger batch size. ", "votes": 5, "replies": [ { "id": 3186985, "postDate": "2025-04-25T11:53:46.747Z", "content": "I thought to myself, maybe I will write to try larger batch/smaller dim (both known to cause diverging especially in transformer), but then said, you surely already know this so is reduntant to write, lol.
", "rawMarkdown": "I thought to myself, maybe I will write to try larger batch/smaller dim (both known to cause diverging especially in transformer), but then said, you surely already know this so is reduntant to write, lol.", "votes": 1 } ] }, { "id": 3186315, "postDate": "2025-04-24T13:35:33.950Z", "content": "Are you using TPUs ? If so, this might be related (otherwise, probably not).
", "rawMarkdown": "Are you using TPUs ? If so, [this might be related](https://www.kaggle.com/code/gguillard/a-very-strange-tpu-easter-egg) (otherwise, probably not).", "votes": 1 }, { "id": 3185269, "postDate": "2025-04-23T04:32:17.480Z", "content": "how about changing the batch size and lr to avoid local minimum?
", "rawMarkdown": "how about changing the batch size and lr to avoid local minimum?", "votes": 1, "replies": [ { "id": 3185323, "postDate": "2025-04-23T06:54:59.143Z", "content": "sometimes it converges to 60, sometimes it fails. No idea what the difference is.
", "rawMarkdown": "sometimes it converges to 60, sometimes it fails. No idea what the difference is.", "votes": 1, "replies": [ { "id": 3185825, "postDate": "2025-04-23T22:05:25.007Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3186647, "postDate": "2025-04-25T02:00:21.550Z", "content": "Did you train from scratch or do fine-tuning?
\nI’ve noticed that when I do fine-tuning, I often encounter early convergence, and the loss doesn’t go down.
The same discovery
", "rawMarkdown": "The same discovery", "votes": 1 } ] } ] }, { "id": 3187922, "postDate": "2025-04-26T17:49:42.163Z", "content": "Hey bro, I've seen you at the top so many times! How do you manage to do that? I've tried but haven't even come close. It's truly a work of art. If you don't mind sharing your approach after the competition ends, I’d love to learn from it.
", "rawMarkdown": "Hey bro, I've seen you at the top so many times! How do you manage to do that? I've tried but haven't even come close. It's truly a work of art. If you don't mind sharing your approach after the competition ends, I’d love to learn from it." } ], "comments": [ { "id": 3186957, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-04-25T10:45:52.657000", "content": "The model diverging problem is resolved by using larger batch size.
", "votes": 5, "replies": [ { "id": 3186985, "author_name": "greySnow", "author_url": "", "post_date": "2025-04-25T11:53:46.747000", "content": "I thought to myself, maybe I will write to try larger batch/smaller dim (both known to cause diverging especially in transformer), but then said, you surely already know this so is reduntant to write, lol.
", "votes": 1, "replies": [] } ] }, { "id": 3186315, "author_name": "gguillard", "author_url": "", "post_date": "2025-04-24T13:35:33.950000", "content": "Are you using TPUs ? If so, this might be related (otherwise, probably not).
", "votes": 1, "replies": [] }, { "id": 3185269, "author_name": "Anh Bui", "author_url": "", "post_date": "2025-04-23T04:32:17.480000", "content": "how about changing the batch size and lr to avoid local minimum?
", "votes": 1, "replies": [ { "id": 3185323, "author_name": "yuanzhe zhou", "author_url": "", "post_date": "2025-04-23T06:54:59.143000", "content": "sometimes it converges to 60, sometimes it fails. No idea what the difference is.
", "votes": 1, "replies": [ { "id": 3185825, "author_name": "", "author_url": "", "post_date": "2025-04-23T22:05:25.007000", "content": "", "votes": 0, "replies": [] }, { "id": 3186647, "author_name": "atom1231", "author_url": "", "post_date": "2025-04-25T02:00:21.550000", "content": "Did you train from scratch or do fine-tuning?
\nI’ve noticed that when I do fine-tuning, I often encounter early convergence, and the loss doesn’t go down.
The same discovery
", "votes": 1, "replies": [] } ] } ] }, { "id": 3187922, "author_name": "mohanapavan bezawada", "author_url": "", "post_date": "2025-04-26T17:49:42.163000", "content": "Hey bro, I've seen you at the top so many times! How do you manage to do that? I've tried but haven't even come close. It's truly a work of art. If you don't mind sharing your approach after the competition ends, I’d love to learn from it.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3185239": "Sometimes my model firstly converges to around 100, then diverges, what could be the problem? The loss oscillates around 180", "3186957": "The model diverging problem is resolved by using larger batch size. ", "3186315": "Are you using TPUs ? If so, [this might be related](https://www.kaggle.com/code/gguillard/a-very-strange-tpu-easter-egg) (otherwise, probably not).", "3185269": "how about changing the batch size and lr to avoid local minimum?", "3187922": "Hey bro, I've seen you at the top so many times! How do you manage to do that? I've tried but haven't even come close. It's truly a work of art. If you don't mind sharing your approach after the competition ends, I’d love to learn from it." } }