I have been with stuck with the notebook timeout error for about 2 weeks. I have referred all possible discussion posts about notebook timeout error and related errors but nothing seemed to solve the issue. I have tried all possible types of optimization spanning from multiprocessing to using scripts. Instead of custom architecture, I used a pre-trained densenet121 with FASTAI as the framework. I feel like I have hit a brick wall and I have no idea as to how to proceed. Any help would be of immense help.
", "messages": [ { "id": "750729", "postDate": "02/19/2020 16:38:04", "content": "I have been with stuck with the notebook timeout error for about 2 weeks. I have referred all possible discussion posts about notebook timeout error and related errors but nothing seemed to solve the issue. I have tried all possible types of optimization spanning from multiprocessing to using scripts. Instead of custom architecture, I used a pre-trained densenet121 with FASTAI as the framework. I feel like I have hit a brick wall and I have no idea as to how to proceed. Any help would be of immense help.
", "rawMarkdown": "I have been with stuck with the notebook timeout error for about 2 weeks. I have referred all possible discussion posts about notebook timeout error and related errors but nothing seemed to solve the issue. I have tried all possible types of optimization spanning from multiprocessing to using scripts. Instead of custom architecture, I used a pre-trained densenet121 with FASTAI as the framework. I feel like I have hit a brick wall and I have no idea as to how to proceed. Any help would be of immense help.", "votes": null }, { "id": "750750", "postDate": "02/19/2020 16:57:46", "content": "Does your submission do training or just inference? First off, you should train models offline or in another notebook, then load model weights and only do inference in your submission. Next, you should use your model to infer all 200,000 training images. Time how long that takes. If you can get that time below 2 hours, then switch to test image inference and submit.
", "rawMarkdown": "Does your submission do training or just inference? First off, you should train models offline or in another notebook, then load model weights and only do inference in your submission. Next, you should use your model to infer all 200,000 training images. Time how long that takes. If you can get that time below 2 hours, then switch to test image inference and submit.", "votes": null }, { "id": "750762", "postDate": "02/19/2020 17:05:05", "content": "I use a seperate inference notebook
", "rawMarkdown": "I use a seperate inference notebook", "votes": null }, { "id": "750787", "postDate": "02/19/2020 17:32:22", "content": "Increase the batch size (inference).
", "rawMarkdown": "Increase the batch size (inference).", "votes": null }, { "id": "750836", "postDate": "02/19/2020 18:21:19", "content": "I have tried it with sizes ranging from 96 to 256
", "rawMarkdown": "I have tried it with sizes ranging from 96 to 256", "votes": null }, { "id": "750888", "postDate": "02/19/2020 19:33:03", "content": "https://www.kaggle.com/venky2506/fastai-inference-128x128-v1?scriptVersionId=28922224\nA version of my inference notebook
", "rawMarkdown": "https://www.kaggle.com/venky2506/fastai-inference-128x128-v1?scriptVersionId=28922224\nA version of my inference notebook", "votes": null }, { "id": "750913", "postDate": "02/19/2020 20:16:43", "content": "Are you using cpu for inference?
", "rawMarkdown": "Are you using cpu for inference?", "votes": null }, { "id": "751317", "postDate": "02/20/2020 05:23:48", "content": "I tried both
", "rawMarkdown": "I tried both", "votes": null }, { "id": "751349", "postDate": "02/20/2020 06:30:54", "content": "Can you reduce the number of epochs, as well as reduce number of instances (maybe 25000) and check. Just to confirm, if its due to the time taken or some other issue...
", "rawMarkdown": "Can you reduce the number of epochs, as well as reduce number of instances (maybe 25000) and check. Just to confirm, if its due to the time taken or some other issue...", "votes": null }, { "id": "751369", "postDate": "02/20/2020 06:53:32", "content": "What do you mean by no of instances?
", "rawMarkdown": "What do you mean by no of instances?", "votes": null }, { "id": "751379", "postDate": "02/20/2020 07:07:31", "content": "I don't think num_epochs affect the submission time as I have used a separate training and inference notebook
", "rawMarkdown": "I don't think num_epochs affect the submission time as I have used a separate training and inference notebook", "votes": null }, { "id": "751413", "postDate": "02/20/2020 07:21:41", "content": "When you infer all 200,000 training images, how long does it take?
", "rawMarkdown": "When you infer all 200,000 training images, how long does it take?", "votes": null }, { "id": "751450", "postDate": "02/20/2020 07:47:17", "content": "Actually, I haven't measured that
", "rawMarkdown": "Actually, I haven't measured that", "votes": null } ], "comments": [ { "id": 750750, "author_name": "cdeotte", "author_url": "", "post_date": "02/19/2020 16:57:46", "content": "Does your submission do training or just inference? First off, you should train models offline or in another notebook, then load model weights and only do inference in your submission. Next, you should use your model to infer all 200,000 training images. Time how long that takes. If you can get that time below 2 hours, then switch to test image inference and submit.
", "votes": null, "replies": [] }, { "id": 750762, "author_name": "venky2506", "author_url": "", "post_date": "02/19/2020 17:05:05", "content": "I use a seperate inference notebook
", "votes": null, "replies": [] }, { "id": 750787, "author_name": "pestipeti", "author_url": "", "post_date": "02/19/2020 17:32:22", "content": "Increase the batch size (inference).
", "votes": null, "replies": [] }, { "id": 750836, "author_name": "venky2506", "author_url": "", "post_date": "02/19/2020 18:21:19", "content": "I have tried it with sizes ranging from 96 to 256
", "votes": null, "replies": [] }, { "id": 750888, "author_name": "venky2506", "author_url": "", "post_date": "02/19/2020 19:33:03", "content": "https://www.kaggle.com/venky2506/fastai-inference-128x128-v1?scriptVersionId=28922224\nA version of my inference notebook
", "votes": null, "replies": [] }, { "id": 750913, "author_name": "max6296", "author_url": "", "post_date": "02/19/2020 20:16:43", "content": "Are you using cpu for inference?
", "votes": null, "replies": [] }, { "id": 751317, "author_name": "venky2506", "author_url": "", "post_date": "02/20/2020 05:23:48", "content": "I tried both
", "votes": null, "replies": [] }, { "id": 751349, "author_name": "anirbank", "author_url": "", "post_date": "02/20/2020 06:30:54", "content": "Can you reduce the number of epochs, as well as reduce number of instances (maybe 25000) and check. Just to confirm, if its due to the time taken or some other issue...
", "votes": null, "replies": [] }, { "id": 751369, "author_name": "venky2506", "author_url": "", "post_date": "02/20/2020 06:53:32", "content": "What do you mean by no of instances?
", "votes": null, "replies": [] }, { "id": 751379, "author_name": "venky2506", "author_url": "", "post_date": "02/20/2020 07:07:31", "content": "I don't think num_epochs affect the submission time as I have used a separate training and inference notebook
", "votes": null, "replies": [] }, { "id": 751413, "author_name": "cdeotte", "author_url": "", "post_date": "02/20/2020 07:21:41", "content": "When you infer all 200,000 training images, how long does it take?
", "votes": null, "replies": [] }, { "id": 751450, "author_name": "venky2506", "author_url": "", "post_date": "02/20/2020 07:47:17", "content": "Actually, I haven't measured that
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "750729": "I have been with stuck with the notebook timeout error for about 2 weeks. I have referred all possible discussion posts about notebook timeout error and related errors but nothing seemed to solve the issue. I have tried all possible types of optimization spanning from multiprocessing to using scripts. Instead of custom architecture, I used a pre-trained densenet121 with FASTAI as the framework. I feel like I have hit a brick wall and I have no idea as to how to proceed. Any help would be of immense help.", "750750": "Does your submission do training or just inference? First off, you should train models offline or in another notebook, then load model weights and only do inference in your submission. Next, you should use your model to infer all 200,000 training images. Time how long that takes. If you can get that time below 2 hours, then switch to test image inference and submit.", "750762": "I use a seperate inference notebook", "750787": "Increase the batch size (inference).", "750836": "I have tried it with sizes ranging from 96 to 256", "750888": "https://www.kaggle.com/venky2506/fastai-inference-128x128-v1?scriptVersionId=28922224\nA version of my inference notebook", "750913": "Are you using cpu for inference?", "751317": "I tried both", "751349": "Can you reduce the number of epochs, as well as reduce number of instances (maybe 25000) and check. Just to confirm, if its due to the time taken or some other issue...", "751369": "What do you mean by no of instances?", "751379": "I don't think num_epochs affect the submission time as I have used a separate training and inference notebook", "751413": "When you infer all 200,000 training images, how long does it take?", "751450": "Actually, I haven't measured that" }, "source": "meta" }