I am stuck with a problem in my submission, as I have a multi-model pipeline that is long, and it seems to fail after two hours. In the logs, I only see that it says Notebook Threw Exception. Is there no way to see in more detail where the error might be? Since it is very difficult to debug seeing how long it takes and on top of that, one of the two submissions we have in the day is spent if there is an error. I have tested the inference pipeline on the whole training set and it works.
", "messages": [ { "id": "2950279", "postDate": "08/07/2024 12:43:09", "content": "I am stuck with a problem in my submission, as I have a multi-model pipeline that is long, and it seems to fail after two hours. In the logs, I only see that it says Notebook Threw Exception. Is there no way to see in more detail where the error might be? Since it is very difficult to debug seeing how long it takes and on top of that, one of the two submissions we have in the day is spent if there is an error. I have tested the inference pipeline on the whole training set and it works.
", "rawMarkdown": "I am stuck with a problem in my submission, as I have a multi-model pipeline that is long, and it seems to fail after two hours. In the logs, I only see that it says Notebook Threw Exception. Is there no way to see in more detail where the error might be? Since it is very difficult to debug seeing how long it takes and on top of that, one of the two submissions we have in the day is spent if there is an error. I have tested the inference pipeline on the whole training set and it works.", "votes": null }, { "id": "2950322", "postDate": "08/07/2024 13:45:12", "content": "I've been stuck a while with a not described OOM error. I've only catch it when I ran it on a subset of train big enough to cause it. Check if there is a batch too big, not always notify it.
\nEDIT: Too fast answer, so no, Is not OOM sorry.
", "rawMarkdown": "I've been stuck a while with a not described OOM error. I've only catch it when I ran it on a subset of train big enough to cause it. Check if there is a batch too big, not always notify it.\n\nEDIT: Too fast answer, so no, Is not OOM sorry.", "votes": null }, { "id": "2950326", "postDate": "08/07/2024 13:53:14", "content": "Thanks! I'll try to use autocast. As my batches has variable size (depending on series length), it could cause it. But the thing is, why doesn't kaggle show more explicit logs? Do they think we can extract test set information from there?
", "rawMarkdown": "Thanks! I'll try to use autocast. As my batches has variable size (depending on series length), it could cause it. But the thing is, why doesn't kaggle show more explicit logs? Do they think we can extract test set information from there?", "votes": null }, { "id": "2951010", "postDate": "08/08/2024 06:43:38", "content": "Hey @itsnotedu ,
\nCheck out the tiny dataset created by me which can be drop in replacement of original dataset for debugging this type of errors quickly.
\nDiscussion Link : [NEW DATASET] Tiny Debug Dataset for RSNA 2024 Lumbar Spine Degenerative Classification
\nDataset Link : RSNA_LSDC_2024_submission_debug_dataset
\nNotebook Link : RSNA_LSDC_2024_submission_debug
Thanks for the idea! I made other dataset with more cases, like hundreds of images for a single series in order to force an OOM error. Now the submission works!
", "rawMarkdown": "Thanks for the idea! I made other dataset with more cases, like hundreds of images for a single series in order to force an OOM error. Now the submission works!", "votes": null } ], "comments": [ { "id": 2950322, "author_name": "sacuscreed", "author_url": "", "post_date": "08/07/2024 13:45:12", "content": "I've been stuck a while with a not described OOM error. I've only catch it when I ran it on a subset of train big enough to cause it. Check if there is a batch too big, not always notify it.
\nEDIT: Too fast answer, so no, Is not OOM sorry.
", "votes": null, "replies": [ { "id": 2950326, "author_name": "itsnotedu", "author_url": "", "post_date": "08/07/2024 13:53:14", "content": "Thanks! I'll try to use autocast. As my batches has variable size (depending on series length), it could cause it. But the thing is, why doesn't kaggle show more explicit logs? Do they think we can extract test set information from there?
", "votes": null, "replies": [] } ] }, { "id": 2951010, "author_name": "rohitchaudhari25", "author_url": "", "post_date": "08/08/2024 06:43:38", "content": "Hey @itsnotedu ,
\nCheck out the tiny dataset created by me which can be drop in replacement of original dataset for debugging this type of errors quickly.
\nDiscussion Link : [NEW DATASET] Tiny Debug Dataset for RSNA 2024 Lumbar Spine Degenerative Classification
\nDataset Link : RSNA_LSDC_2024_submission_debug_dataset
\nNotebook Link : RSNA_LSDC_2024_submission_debug
Thanks for the idea! I made other dataset with more cases, like hundreds of images for a single series in order to force an OOM error. Now the submission works!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2950279": "I am stuck with a problem in my submission, as I have a multi-model pipeline that is long, and it seems to fail after two hours. In the logs, I only see that it says Notebook Threw Exception. Is there no way to see in more detail where the error might be? Since it is very difficult to debug seeing how long it takes and on top of that, one of the two submissions we have in the day is spent if there is an error. I have tested the inference pipeline on the whole training set and it works.", "2950322": "I've been stuck a while with a not described OOM error. I've only catch it when I ran it on a subset of train big enough to cause it. Check if there is a batch too big, not always notify it.\n\nEDIT: Too fast answer, so no, Is not OOM sorry.", "2950326": "Thanks! I'll try to use autocast. As my batches has variable size (depending on series length), it could cause it. But the thing is, why doesn't kaggle show more explicit logs? Do they think we can extract test set information from there?", "2951010": "Hey @itsnotedu ,\n\nCheck out the tiny dataset created by me which can be drop in replacement of original dataset for debugging this type of errors quickly.\n\n**Discussion Link :** [[NEW DATASET] Tiny Debug Dataset for RSNA 2024 Lumbar Spine Degenerative Classification](https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/523247)\n**Dataset Link :** [RSNA_LSDC_2024_submission_debug_dataset](https://www.kaggle.com/datasets/rohitchaudhari25/rsna-lsdc-2024-submission-debug-dataset)\n**Notebook Link :** [RSNA_LSDC_2024_submission_debug](https://www.kaggle.com/code/rohitchaudhari25/rsna-lsdc-2024-submission-debug/notebook)", "2953891": "Thanks for the idea! I made other dataset with more cases, like hundreds of images for a single series in order to force an OOM error. Now the submission works!" }, "source": "meta" }