I used GPU P100 and submitted but its been more then 1 hour and submission isn't completed yet. I do know that we are expected to have around 120 volumes but if i am not wrong for the training they gave us 700+ volumes and on training my 1 epoch took arounf 16 min So, by that it shouldn't take this much time. Can Some one explain what's can be the problem as this is my first time entering in the featured competition.
", "messages": [ { "id": 3396107, "postDate": "2026-01-24T10:37:54.117Z", "content": "I used GPU P100 and submitted but its been more then 1 hour and submission isn't completed yet. I do know that we are expected to have around 120 volumes but if i am not wrong for the training they gave us 700+ volumes and on training my 1 epoch took arounf 16 min So, by that it shouldn't take this much time. Can Some one explain what's can be the problem as this is my first time entering in the featured competition.
", "rawMarkdown": "I used GPU P100 and submitted but its been more then 1 hour and submission isn't completed yet. I do know that we are expected to have around 120 volumes but if i am not wrong for the training they gave us 700+ volumes and on training my 1 epoch took arounf 16 min So, by that it shouldn't take this much time. Can Some one explain what's can be the problem as this is my first time entering in the featured competition.", "votes": 2 }, { "id": 3396121, "postDate": "2026-01-24T11:28:33.033Z", "content": "Metric calculation takes about four hours.\nhttps://www.kaggle.com/competitions/vesuvius-challenge-surface-detection/discussion/621455
", "rawMarkdown": "Metric calculation takes about four hours.\nhttps://www.kaggle.com/competitions/vesuvius-challenge-surface-detection/discussion/621455", "votes": 1, "replies": [ { "id": 3396135, "postDate": "2026-01-24T12:07:46Z", "content": "thankyou for the details its helps a lot. if you don't mind can you clear a few more doubts of mine as well.
\nBASE_DIR = Path(\"/kaggle/input/vesuvius-challenge-surface-detection\")
\nTEST_IMG_DIR = BASE_DIR / \"test_images\"
\nWORKDIR = Path(\"/kaggle/working\")
\nOUT_DIR = WORKDIR / \"predictions\"
\nOUT_DIR.mkdir(parents=True, exist_ok=True)
\nit s\nhow len = 1 and run inference on that which is i think is correct considering originally the folder had only 1 test set image.\nSo, if can and just verify that i am doing things correct and my model is running on there hidden set. IT's just confusing me.
You are doing it right. The test folder will contain the full test set (public + private) when Kaggle reruns your notebook (you will see it run twice)
", "rawMarkdown": "You are doing it right. The test folder will contain the full test set (public + private) when Kaggle reruns your notebook (you will see it run twice)", "votes": 1 } ] }, { "id": 3400033, "postDate": "2026-01-31T18:43:27.890Z", "content": "I tried a simple inference pipeline and submission. Inference notebook finished within 5-7 minutes. But, metric calculation ran more than 9h and exited.
\nCouldn't see what is error as well / why submission failed when notebook was successful.
", "rawMarkdown": "I tried a simple inference pipeline and submission. Inference notebook finished within 5-7 minutes. But, metric calculation ran more than 9h and exited. \n\nCouldn't see what is error as well / why submission failed when notebook was successful.", "replies": [ { "id": 3400057, "postDate": "2026-01-31T20:41:27.043Z", "content": "Are you using the native nnU-Net pipeline with mirroring enabled? Double check you’re not also stacking extra TTA on top of that (extra flips/aug loops), because that can blow up runtime fast. Also watch what models you’re running — fold-all models (or lots of folds) will take way longer than split-fold setups (ex: 5 split folds vs just 2 fold-all models, etc.).
\nMy first few submissions timed out too — I had mirroring + extra TTA going and it was just too slow. Once I disabled the extra stuff and dialed it in, my local/in-notebook inference finishes fast. For example my log is like:
\n225.8s ✅ Saved mask: predictions/1407735.tif\nand then it zips and writes submission.zip right after.
\nThe hidden test set run still takes a while — for me it’s a little over 8 hours to finish.
\nAlso, if you’re running 2 fold-all models, you’ll pretty much need the T4 setup and run them in parallel — on a P100 it’s likely to time out if you run them sequentially.
", "rawMarkdown": "Are you using the native nnU-Net pipeline with mirroring enabled? Double check you’re not also stacking extra TTA on top of that (extra flips/aug loops), because that can blow up runtime fast. Also watch what models you’re running — fold-all models (or lots of folds) will take way longer than split-fold setups (ex: 5 split folds vs just 2 fold-all models, etc.).\n\nMy first few submissions timed out too — I had mirroring + extra TTA going and it was just too slow. Once I disabled the extra stuff and dialed it in, my local/in-notebook inference finishes fast. For example my log is like:\n\n225.8s ✅ Saved mask: predictions/1407735.tif\nand then it zips and writes submission.zip right after.\n\nThe hidden test set run still takes a while — for me it’s a little over 8 hours to finish.\n\nAlso, if you’re running 2 fold-all models, you’ll pretty much need the T4 setup and run them in parallel — on a P100 it’s likely to time out if you run them sequentially." } ] } ] }, { "id": 3399184, "postDate": "2026-01-30T11:01:04.293Z", "content": "Do you know which GPU will be available for the testing phase?
", "rawMarkdown": "Do you know which GPU will be available for the testing phase?\n", "replies": [ { "id": 3399190, "postDate": "2026-01-30T11:10:39.687Z", "content": "For mine i was able to select it when i saved for the version. I think GPU saved on that time will be used.
", "rawMarkdown": "For mine i was able to select it when i saved for the version. I think GPU saved on that time will be used.", "replies": [ { "id": 3399364, "postDate": "2026-01-30T18:09:10.930Z", "content": "Thanks! I'm a bit confused how this is supposed to work. I have created the notebook that runs inferences to all .tif files in the input. Do you know if this is correct? I just noticed that it's doing inference for the training cases: \nDoing case: /kaggle/input/vesuvius-challenge-surface-detection/train_images/2761996776.tif
Metric calculation takes about four hours.\nhttps://www.kaggle.com/competitions/vesuvius-challenge-surface-detection/discussion/621455
", "votes": 1, "replies": [ { "id": 3396135, "author_name": "Kaustuk000", "author_url": "", "post_date": "2026-01-24T12:07:46", "content": "thankyou for the details its helps a lot. if you don't mind can you clear a few more doubts of mine as well.
\nBASE_DIR = Path(\"/kaggle/input/vesuvius-challenge-surface-detection\")
\nTEST_IMG_DIR = BASE_DIR / \"test_images\"
\nWORKDIR = Path(\"/kaggle/working\")
\nOUT_DIR = WORKDIR / \"predictions\"
\nOUT_DIR.mkdir(parents=True, exist_ok=True)
\nit s\nhow len = 1 and run inference on that which is i think is correct considering originally the folder had only 1 test set image.\nSo, if can and just verify that i am doing things correct and my model is running on there hidden set. IT's just confusing me.
You are doing it right. The test folder will contain the full test set (public + private) when Kaggle reruns your notebook (you will see it run twice)
", "votes": 1, "replies": [] } ] }, { "id": 3400033, "author_name": "Maapu", "author_url": "", "post_date": "2026-01-31T18:43:27.890000", "content": "I tried a simple inference pipeline and submission. Inference notebook finished within 5-7 minutes. But, metric calculation ran more than 9h and exited.
\nCouldn't see what is error as well / why submission failed when notebook was successful.
", "votes": 0, "replies": [ { "id": 3400057, "author_name": "TWEAK", "author_url": "", "post_date": "2026-01-31T20:41:27.043000", "content": "Are you using the native nnU-Net pipeline with mirroring enabled? Double check you’re not also stacking extra TTA on top of that (extra flips/aug loops), because that can blow up runtime fast. Also watch what models you’re running — fold-all models (or lots of folds) will take way longer than split-fold setups (ex: 5 split folds vs just 2 fold-all models, etc.).
\nMy first few submissions timed out too — I had mirroring + extra TTA going and it was just too slow. Once I disabled the extra stuff and dialed it in, my local/in-notebook inference finishes fast. For example my log is like:
\n225.8s ✅ Saved mask: predictions/1407735.tif\nand then it zips and writes submission.zip right after.
\nThe hidden test set run still takes a while — for me it’s a little over 8 hours to finish.
\nAlso, if you’re running 2 fold-all models, you’ll pretty much need the T4 setup and run them in parallel — on a P100 it’s likely to time out if you run them sequentially.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3399184, "author_name": "Andre Filipe Ferreira", "author_url": "", "post_date": "2026-01-30T11:01:04.293000", "content": "Do you know which GPU will be available for the testing phase?
", "votes": 0, "replies": [ { "id": 3399190, "author_name": "Kaustuk000", "author_url": "", "post_date": "2026-01-30T11:10:39.687000", "content": "For mine i was able to select it when i saved for the version. I think GPU saved on that time will be used.
", "votes": 0, "replies": [ { "id": 3399364, "author_name": "Andre Filipe Ferreira", "author_url": "", "post_date": "2026-01-30T18:09:10.930000", "content": "Thanks! I'm a bit confused how this is supposed to work. I have created the notebook that runs inferences to all .tif files in the input. Do you know if this is correct? I just noticed that it's doing inference for the training cases: \nDoing case: /kaggle/input/vesuvius-challenge-surface-detection/train_images/2761996776.tif