I am using trained model my submission is taking to long so i set number of EPOCHS=0 so it just have to load the model and then do inference on test dataset.
\nBut after submitting and waiting for 9 hours it also failed ;-(
\nWhen i save the file it run code in just 2-3 mins.
I am using trained model my submission is taking to long so i set number of EPOCHS=0 so it just have to load the model and then do inference on test dataset.
\nBut after submitting and waiting for 9 hours it also failed ;-(
\nWhen i save the file it run code in just 2-3 mins.
After failing it gave Notebook Timeout but EPOCHS are 0 and when i run all the cell of notebook it take just 2 mins
", "rawMarkdown": "After failing it gave **Notebook Timeout** but EPOCHS are 0 and when i run all the cell of notebook it take just 2 mins ", "votes": 1, "replies": [ { "id": 1102693, "postDate": "2020-12-05T07:44:38.967Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1102743, "postDate": "2020-12-05T08:53:01.723Z", "content": "How can i solve that. For testset i am using 5 as batch size
", "rawMarkdown": "How can i solve that. For testset i am using 5 as batch size" }, { "id": 1102746, "postDate": "2020-12-05T08:55:44.833Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1102896, "postDate": "2020-12-05T13:06:22.830Z", "content": "When you run your notebook, you just have one test set image (obviously, it's very quick to run that through your model). When you submit to the leaderboard, it is then run on around 15,000 images.
\nI don't think your GPU quota should matter when you submit to the leaderboard. It's more likely that it took longer than you expected and exceeded the time limit, or that e.g. a out-of-memory error stopped the processing without stopping the notebook (in that case it looks like it kept running).
\nIf you want to test how long it takes, time how long the prediction cell of the notebook (obviously, some of the initial set-up also takes time) takes when you run a large number of training set images through the model (you can probably extrapolate from 500 to 1000).
", "rawMarkdown": "When you run your notebook, you just have one test set image (obviously, it's very quick to run that through your model). When you submit to the leaderboard, it is then run on around 15,000 images.\n\nI don't think your GPU quota should matter when you submit to the leaderboard. It's more likely that it took longer than you expected and exceeded the time limit, or that e.g. a out-of-memory error stopped the processing without stopping the notebook (in that case it looks like it kept running).\n\nIf you want to test how long it takes, time how long the prediction cell of the notebook (obviously, some of the initial set-up also takes time) takes when you run a large number of training set images through the model (you can probably extrapolate from 500 to 1000).", "votes": 1 }, { "id": 1103430, "postDate": "2020-12-05T22:37:55.427Z", "content": "I had the same problem (out of memory). I solved it by performing prediction on small batches and reuse the memory. You can point your test data to your training image directory to validate the issue.
", "rawMarkdown": "I had the same problem (out of memory). I solved it by performing prediction on small batches and reuse the memory. You can point your test data to your training image directory to validate the issue." }, { "id": 1195270, "postDate": "2021-02-10T16:40:45.033Z", "content": "This worked for me too… Great. Thanks.
", "rawMarkdown": "This worked for me too... Great. Thanks." } ] }, { "id": 1102782, "postDate": "2020-12-05T09:55:23.187Z", "content": "It is said that the model will be evaluated on GPU accelerator. What accelerator are you using?
", "rawMarkdown": "It is said that the model will be evaluated on GPU accelerator. What accelerator are you using?", "replies": [ { "id": 1102827, "postDate": "2020-12-05T11:06:12.937Z", "content": "Yes it was running on GPU. But i think my GPU quota was exhausted that is why it failed.
", "rawMarkdown": "Yes it was running on GPU. But i think my GPU quota was exhausted that is why it failed. ", "votes": 1 }, { "id": 1102829, "postDate": "2020-12-05T11:07:09.017Z", "content": "Thankyou for reply
", "rawMarkdown": "Thankyou for reply " } ] }, { "id": 1102761, "postDate": "2020-12-05T09:21:30.627Z", "content": "Yes submission evaluation is too slow. I have submitted with a batch size of 32, it took 1 hour to run.
\nHow big is your model in terms of size? and are you resizing the input at test time?
", "rawMarkdown": "Yes submission evaluation is too slow. I have submitted with a batch size of 32, it took 1 hour to run.\n\nHow big is your model in terms of size? and are you resizing the input at test time?\n\n", "replies": [ { "id": 1102828, "postDate": "2020-12-05T11:06:57.580Z", "content": "Thankyou for reply.
\nI think my GPU quota was exhausted that is why it failed.
After failing it gave Notebook Timeout but EPOCHS are 0 and when i run all the cell of notebook it take just 2 mins
", "votes": 1, "replies": [ { "id": 1102693, "author_name": "", "author_url": "", "post_date": "2020-12-05T07:44:38.967000", "content": "", "votes": 1, "replies": [] }, { "id": 1102743, "author_name": "rajan", "author_url": "", "post_date": "2020-12-05T08:53:01.723000", "content": "How can i solve that. For testset i am using 5 as batch size
", "votes": 0, "replies": [] }, { "id": 1102746, "author_name": "", "author_url": "", "post_date": "2020-12-05T08:55:44.833000", "content": "", "votes": 0, "replies": [] }, { "id": 1102896, "author_name": "Björn", "author_url": "", "post_date": "2020-12-05T13:06:22.830000", "content": "When you run your notebook, you just have one test set image (obviously, it's very quick to run that through your model). When you submit to the leaderboard, it is then run on around 15,000 images.
\nI don't think your GPU quota should matter when you submit to the leaderboard. It's more likely that it took longer than you expected and exceeded the time limit, or that e.g. a out-of-memory error stopped the processing without stopping the notebook (in that case it looks like it kept running).
\nIf you want to test how long it takes, time how long the prediction cell of the notebook (obviously, some of the initial set-up also takes time) takes when you run a large number of training set images through the model (you can probably extrapolate from 500 to 1000).
", "votes": 1, "replies": [] }, { "id": 1103430, "author_name": "luqing Wang", "author_url": "", "post_date": "2020-12-05T22:37:55.427000", "content": "I had the same problem (out of memory). I solved it by performing prediction on small batches and reuse the memory. You can point your test data to your training image directory to validate the issue.
", "votes": 0, "replies": [] }, { "id": 1195270, "author_name": "Vinodh Arumugam", "author_url": "", "post_date": "2021-02-10T16:40:45.033000", "content": "This worked for me too… Great. Thanks.
", "votes": 0, "replies": [] } ] }, { "id": 1102782, "author_name": "Shanmukh", "author_url": "", "post_date": "2020-12-05T09:55:23.187000", "content": "It is said that the model will be evaluated on GPU accelerator. What accelerator are you using?
", "votes": 0, "replies": [ { "id": 1102827, "author_name": "rajan", "author_url": "", "post_date": "2020-12-05T11:06:12.937000", "content": "Yes it was running on GPU. But i think my GPU quota was exhausted that is why it failed.
", "votes": 1, "replies": [] }, { "id": 1102829, "author_name": "rajan", "author_url": "", "post_date": "2020-12-05T11:07:09.017000", "content": "Thankyou for reply
", "votes": 0, "replies": [] } ] }, { "id": 1102761, "author_name": "Harveen Singh Chadha", "author_url": "", "post_date": "2020-12-05T09:21:30.627000", "content": "Yes submission evaluation is too slow. I have submitted with a batch size of 32, it took 1 hour to run.
\nHow big is your model in terms of size? and are you resizing the input at test time?
", "votes": 0, "replies": [ { "id": 1102828, "author_name": "rajan", "author_url": "", "post_date": "2020-12-05T11:06:57.580000", "content": "Thankyou for reply.
\nI think my GPU quota was exhausted that is why it failed.