Copied from https://www.kaggle.com/code/alincijov/training-hubmap-lb-0-75-swin-transformer-v1
\nSuccessfully trained and validated.
\nNow when I use it for submission, I get \"Notebook Exceeded Allowed Compute\" error for dozens of times.
\nI tried to reduce the memory usage using as many tips from kaggle discussion and info searched online as possible, but still this error appears.
Here's some details:
\nInput data contains only 2 trained models, each 160MB ish.
\nSet batch_size to 1 and num_workers to 0.
\nSet image numpy variables return to the same name as pre-modify(disruptive modify?)
\nDelete unnecessary variables and used gc_collect()
\nUsed torch.no_grad():
I tested using all the 351 images in a train folder, and it works fine in several minutes, though used up to 11.1GB RAM and 1.1GB of GPU memory.
\nSomehow when I submit the code, it works fine at first, and then when it passed to Kaggle internal computation stuff, it ends up with the error code.
\nCould someone comment on my code…?
\nhttps://www.kaggle.com/code/matakaggle/notebook61b356a23f
Any tips will be greatly appreciated!!
", "messages": [ { "id": "1938203", "postDate": "09/14/2022 03:21:10", "content": "Copied from https://www.kaggle.com/code/alincijov/training-hubmap-lb-0-75-swin-transformer-v1
\nSuccessfully trained and validated.
\nNow when I use it for submission, I get \"Notebook Exceeded Allowed Compute\" error for dozens of times.
\nI tried to reduce the memory usage using as many tips from kaggle discussion and info searched online as possible, but still this error appears.
Here's some details:
\nInput data contains only 2 trained models, each 160MB ish.
\nSet batch_size to 1 and num_workers to 0.
\nSet image numpy variables return to the same name as pre-modify(disruptive modify?)
\nDelete unnecessary variables and used gc_collect()
\nUsed torch.no_grad():
I tested using all the 351 images in a train folder, and it works fine in several minutes, though used up to 11.1GB RAM and 1.1GB of GPU memory.
\nSomehow when I submit the code, it works fine at first, and then when it passed to Kaggle internal computation stuff, it ends up with the error code.
\nCould someone comment on my code…?
\nhttps://www.kaggle.com/code/matakaggle/notebook61b356a23f
Any tips will be greatly appreciated!!
", "rawMarkdown": "Copied from https://www.kaggle.com/code/alincijov/training-hubmap-lb-0-75-swin-transformer-v1\nSuccessfully trained and validated.\nNow when I use it for submission, I get \"Notebook Exceeded Allowed Compute\" error for dozens of times.\nI tried to reduce the memory usage using as many tips from kaggle discussion and info searched online as possible, but still this error appears.\n\nHere's some details:\nInput data contains only 2 trained models, each 160MB ish.\nSet batch_size to 1 and num_workers to 0.\nSet image numpy variables return to the same name as pre-modify(disruptive modify?)\nDelete unnecessary variables and used gc_collect()\nUsed torch.no_grad():\n\nI tested using all the 351 images in a train folder, and it works fine in several minutes, though used up to 11.1GB RAM and 1.1GB of GPU memory.\n\nSomehow when I submit the code, it works fine at first, and then when it passed to Kaggle internal computation stuff, it ends up with the error code.\n\nCould someone comment on my code...?\nhttps://www.kaggle.com/code/matakaggle/notebook61b356a23f\n\nAny tips will be greatly appreciated!!", "votes": null }, { "id": "1938662", "postDate": "09/14/2022 09:19:29", "content": "Today one of our submissions also fell with a Notebook Timeout.
\nThe only change was replacing simple decoder with convolutional one.
\nI don't know if this is due to a lot of calculations or something happened in the system.
", "rawMarkdown": "Today one of our submissions also fell with a Notebook Timeout.\n\nThe only change was replacing simple decoder with convolutional one.\n\nI don't know if this is due to a lot of calculations or something happened in the system.", "votes": null }, { "id": "1938679", "postDate": "09/14/2022 09:36:23", "content": "Notebook Exceeded Allowed Compute error generally occurs because of ram shortage. And here is what I got when I ran your notebook: check. As you see, your code infers a single image twice and writes both of the results in a submission.csv. There could be a chance that your code would write each result more than one time, so it could cause unexpected error such as an unexpected loop to spend ram more than needed. I would suggest you to run the whole code through validation image files and check if the result.csv from inferred validation images is printed accordingly.
\nPS I experienced the same error when my submission.csv had multiple results on one image (careful when you combine tiled images and make the inference). Hopefully your case's dealt well and soon.
", "rawMarkdown": "Notebook Exceeded Allowed Compute error generally occurs because of ram shortage. And here is what I got when I ran your notebook: [check](https://raw.githubusercontent.com/cheul0518/Competitions/main/HuBMAP_HPA/11.png). As you see, your code infers a single image twice and writes both of the results in a submission.csv. There could be a chance that your code would write each result more than one time, so it could cause unexpected error such as an unexpected loop to spend ram more than needed. I would suggest you to run the whole code through validation image files and check if the result.csv from inferred validation images is printed accordingly.\n\nPS I experienced the same error when my submission.csv had multiple results on one image (careful when you combine tiled images and make the inference). Hopefully your case's dealt well and soon.", "votes": null }, { "id": "1939018", "postDate": "09/14/2022 13:40:28", "content": "Try using batch size while prediction
", "rawMarkdown": "Try using batch size while prediction", "votes": null }, { "id": "1939366", "postDate": "09/14/2022 16:47:31", "content": "Thank you so much! I modified my code again to prevent double reading the images, and first committed, and then from the viewers page, pressed submit and then it worked fine!!
", "rawMarkdown": "Thank you so much! I modified my code again to prevent double reading the images, and first committed, and then from the viewers page, pressed submit and then it worked fine!!", "votes": null } ], "comments": [ { "id": 1938662, "author_name": "celidos", "author_url": "", "post_date": "09/14/2022 09:19:29", "content": "Today one of our submissions also fell with a Notebook Timeout.
\nThe only change was replacing simple decoder with convolutional one.
\nI don't know if this is due to a lot of calculations or something happened in the system.
", "votes": null, "replies": [] }, { "id": 1938679, "author_name": "cheulkay", "author_url": "", "post_date": "09/14/2022 09:36:23", "content": "Notebook Exceeded Allowed Compute error generally occurs because of ram shortage. And here is what I got when I ran your notebook: check. As you see, your code infers a single image twice and writes both of the results in a submission.csv. There could be a chance that your code would write each result more than one time, so it could cause unexpected error such as an unexpected loop to spend ram more than needed. I would suggest you to run the whole code through validation image files and check if the result.csv from inferred validation images is printed accordingly.
\nPS I experienced the same error when my submission.csv had multiple results on one image (careful when you combine tiled images and make the inference). Hopefully your case's dealt well and soon.
", "votes": null, "replies": [] }, { "id": 1939018, "author_name": "alokjain0", "author_url": "", "post_date": "09/14/2022 13:40:28", "content": "Try using batch size while prediction
", "votes": null, "replies": [] }, { "id": 1939366, "author_name": "matakaggle", "author_url": "", "post_date": "09/14/2022 16:47:31", "content": "Thank you so much! I modified my code again to prevent double reading the images, and first committed, and then from the viewers page, pressed submit and then it worked fine!!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1938203": "Copied from https://www.kaggle.com/code/alincijov/training-hubmap-lb-0-75-swin-transformer-v1\nSuccessfully trained and validated.\nNow when I use it for submission, I get \"Notebook Exceeded Allowed Compute\" error for dozens of times.\nI tried to reduce the memory usage using as many tips from kaggle discussion and info searched online as possible, but still this error appears.\n\nHere's some details:\nInput data contains only 2 trained models, each 160MB ish.\nSet batch_size to 1 and num_workers to 0.\nSet image numpy variables return to the same name as pre-modify(disruptive modify?)\nDelete unnecessary variables and used gc_collect()\nUsed torch.no_grad():\n\nI tested using all the 351 images in a train folder, and it works fine in several minutes, though used up to 11.1GB RAM and 1.1GB of GPU memory.\n\nSomehow when I submit the code, it works fine at first, and then when it passed to Kaggle internal computation stuff, it ends up with the error code.\n\nCould someone comment on my code...?\nhttps://www.kaggle.com/code/matakaggle/notebook61b356a23f\n\nAny tips will be greatly appreciated!!", "1938662": "Today one of our submissions also fell with a Notebook Timeout.\n\nThe only change was replacing simple decoder with convolutional one.\n\nI don't know if this is due to a lot of calculations or something happened in the system.", "1938679": "Notebook Exceeded Allowed Compute error generally occurs because of ram shortage. And here is what I got when I ran your notebook: [check](https://raw.githubusercontent.com/cheul0518/Competitions/main/HuBMAP_HPA/11.png). As you see, your code infers a single image twice and writes both of the results in a submission.csv. There could be a chance that your code would write each result more than one time, so it could cause unexpected error such as an unexpected loop to spend ram more than needed. I would suggest you to run the whole code through validation image files and check if the result.csv from inferred validation images is printed accordingly.\n\nPS I experienced the same error when my submission.csv had multiple results on one image (careful when you combine tiled images and make the inference). Hopefully your case's dealt well and soon.", "1939018": "Try using batch size while prediction", "1939366": "Thank you so much! I modified my code again to prevent double reading the images, and first committed, and then from the viewers page, pressed submit and then it worked fine!!" }, "source": "meta" }