![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13889710%2Fcef7b4f07c348c98ee7a6164f5870c2a%2FScreenshot%202023-10-04%20at%207.28.22%20AM.png?generation=1696375826209112&alt=media\")
When I was training the model last night, I encounter a bug never seen before:
\n
There are other weirds signs as well in wandb:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13889710%2F17ec39e19c5c99ff0f8a218476140786%2FScreenshot%202023-10-04%20at%207.31.55%20AM.png?generation=1696375966626611&alt=media\")
The global step and epoch decreases which normally only increases.
\nMy assumption:
\nThe padding is set to True in the Data Collator. The sequence length of each batch thus varies, resulting in different cuda memory consumption. Considering that I have added several length audios, this may be the source of the problem.
Few problems I can't explain:
\n1) If there is the batch-size reduced to zero, it should be caused by setting auto_find_batch_size=True. In that case, how can few thousands steps be taken before crashing.
\n2) why does the global_step/epoch decrease?
Since truncating my audio to a suitable size is lots of work, I plea for more insight before attempting to debug.🫡
", "messages": [ { "id": "2466542", "postDate": "10/03/2023 23:58:25", "content": "When I was training the model last night, I encounter a bug never seen before:
\n
There are other weirds signs as well in wandb:
\n
The global step and epoch decreases which normally only increases.
\nMy assumption:
\nThe padding is set to True in the Data Collator. The sequence length of each batch thus varies, resulting in different cuda memory consumption. Considering that I have added several length audios, this may be the source of the problem.
Few problems I can't explain:
\n1) If there is the batch-size reduced to zero, it should be caused by setting auto_find_batch_size=True. In that case, how can few thousands steps be taken before crashing.
\n2) why does the global_step/epoch decrease?
Since truncating my audio to a suitable size is lots of work, I plea for more insight before attempting to debug.🫡
", "rawMarkdown": "When I was training the model last night, I encounter a bug never seen before:\n\n\n\nThere are other weirds signs as well in wandb:\n\n\n\nThe global step and epoch decreases which normally only increases. \n\nMy assumption: \nThe `padding` is set to `True` in the Data Collator. The sequence length of each batch thus varies, resulting in different cuda memory consumption. Considering that I have added several length audios, this may be the source of the problem.\n\nFew problems I can't explain:\n1) If there is the batch-size reduced to zero, it should be caused by setting `auto_find_batch_size=True`. In that case, how can few thousands steps be taken before crashing.\n2) why does the global_step/epoch decrease?\n\nSince truncating my audio to a suitable size is lots of work, I plea for more insight before attempting to debug.🫡", "votes": null }, { "id": "2466543", "postDate": "10/04/2023 00:00:17", "content": "I'm really kind of new in this field, forgive if I am wrong in some of my understanding😵💫
", "rawMarkdown": "I'm really kind of new in this field, forgive if I am wrong in some of my understanding😵💫", "votes": null }, { "id": "2466639", "postDate": "10/04/2023 02:26:06", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F13889710%2Fdd2273251e52eedbf1b99d1e263fca78%2FScreenshot%202023-10-04%20at%2010.21.53%20AM.png?generation=1696386227439904&alt=media\")
This is not even a singular case. In a previous successful session, there is also this weird decrease in the global step and percentage of epoch ran. It never happens again afterwards
", "rawMarkdown": "\n\nThis is not even a singular case. In a previous **successful** session, there is also this weird decrease in the global step and percentage of epoch ran. It never happens again afterwards", "votes": null }, { "id": "2467488", "postDate": "10/04/2023 15:38:34", "content": "Since the error is from accelerater pack, there is one function in train that needs accelerater:
\nauto_find_batch_size
\nThat means your GPU was out of memory on every nonzero size.
\nmaybe you can set fp16=True, or disable adam scheduler.
I'm really kind of new in this field, forgive if I am wrong in some of my understanding😵💫
", "votes": null, "replies": [] }, { "id": 2466639, "author_name": "renyiwei", "author_url": "", "post_date": "10/04/2023 02:26:06", "content": "
This is not even a singular case. In a previous successful session, there is also this weird decrease in the global step and percentage of epoch ran. It never happens again afterwards
", "votes": null, "replies": [] }, { "id": 2467488, "author_name": "hongori", "author_url": "", "post_date": "10/04/2023 15:38:34", "content": "Since the error is from accelerater pack, there is one function in train that needs accelerater:
\nauto_find_batch_size
\nThat means your GPU was out of memory on every nonzero size.
\nmaybe you can set fp16=True, or disable adam scheduler.