Hi,
\nI'm wondering: What is the most efficient way to submit (an inference kernel)? For me it is almost impossible to submit 5 submissions a day (not that I want to, I'm just curious). Here is my approach so far:
\nThus, in total I need between 3 and 6 hours to submit my notebook. I always use GPU and I'm using models such as EfficientNet-B0-B4 or ResNet18,34,50.
\nRecently I tried to submit a model with TTA prediction but I got an 'Notebook Timeout' Error when trying to submit it.
\nAm I doing something wrong? Thanks for your help.
", "messages": [ { "id": "1132365", "postDate": "12/30/2020 10:50:07", "content": "Hi,
\nI'm wondering: What is the most efficient way to submit (an inference kernel)? For me it is almost impossible to submit 5 submissions a day (not that I want to, I'm just curious). Here is my approach so far:
\nThus, in total I need between 3 and 6 hours to submit my notebook. I always use GPU and I'm using models such as EfficientNet-B0-B4 or ResNet18,34,50.
\nRecently I tried to submit a model with TTA prediction but I got an 'Notebook Timeout' Error when trying to submit it.
\nAm I doing something wrong? Thanks for your help.
", "rawMarkdown": "Hi,\n\nI'm wondering: What is the most efficient way to submit (an inference kernel)? For me it is almost impossible to submit 5 submissions a day (not that I want to, I'm just curious). Here is my approach so far:\n\n1. Create a new Notebook and upload my local Notebook file as well as other important inputs such as the model and other libraries. --> This is relatively fast (~10 min)\n2. Save & Run All (Commit) --> Even using GPU this step takes between 1-2 hours\n3. Submitting to Competition --> This step takes even longer than the second step. (~2-4 hours)\n\nThus, in total I need between 3 and 6 hours to submit my notebook. I always use GPU and I'm using models such as EfficientNet-B0-B4 or ResNet18,34,50.\n\nRecently I tried to submit a model with TTA prediction but I got an 'Notebook Timeout' Error when trying to submit it. \n\nAm I doing something wrong? Thanks for your help.", "votes": null }, { "id": "1132414", "postDate": "12/30/2020 11:47:17", "content": "Are you using ensemble models ? For me it takes about 2 mins to load, 10-20 mins for commit and save, and about 20 mins for submission on single model, no augmentations on test set as well, just straight inference. I have tried with EfficientNet B7 and DenseNet201.
", "rawMarkdown": "Are you using ensemble models ? For me it takes about 2 mins to load, 10-20 mins for commit and save, and about 20 mins for submission on single model, no augmentations on test set as well, just straight inference. I have tried with EfficientNet B7 and DenseNet201.", "votes": null }, { "id": "1132439", "postDate": "12/30/2020 12:11:23", "content": "No I'm just using a single model. But maybe it's because I'm also augmenting the test images. I'll try to submit without augmentation. Thanks for the pointer and thanks for letting me know your running times
", "rawMarkdown": "No I'm just using a single model. But maybe it's because I'm also augmenting the test images. I'll try to submit without augmentation. Thanks for the pointer and thanks for letting me know your running times", "votes": null }, { "id": "1133386", "postDate": "12/31/2020 07:09:19", "content": "Hello!
\nI need almost the same time to submit as you.
\nHowever, I can run the next model during \"step 3\" in your approach. Probably, you do not need to wait to finish the score calculation.
You are right! And also you can submit multiple submission. Thanks for sharing!
", "rawMarkdown": "You are right! And also you can submit multiple submission. Thanks for sharing!", "votes": null }, { "id": "1133862", "postDate": "12/31/2020 15:36:24", "content": "After the comments here in this discussion, I realized that I was also augmenting my test images. On my local machine it takes 4 times as long to predict test images with augmentation instead of test images without augmentation. (which is reasonable!)
", "rawMarkdown": "After the comments here in this discussion, I realized that I was also augmenting my test images. On my local machine it takes 4 times as long to predict test images with augmentation instead of test images without augmentation. (which is reasonable!)", "votes": null }, { "id": "1146746", "postDate": "01/10/2021 02:32:51", "content": "You can completely skip step two with a simple trick, namely checking if the sample_submission is the public one. I made a quick template: https://www.kaggle.com/marcelbischoff/ranzcr-quick-submission-template
\nIf there is not enough time you want to do two notebooks. One for training, then export the weights into a dataset and then another notebook which loads the weights and does only inference.
", "rawMarkdown": "You can completely skip step two with a simple trick, namely checking if the sample_submission is the public one. I made a quick template: https://www.kaggle.com/marcelbischoff/ranzcr-quick-submission-template\n\nIf there is not enough time you want to do two notebooks. One for training, then export the weights into a dataset and then another notebook which loads the weights and does only inference.", "votes": null }, { "id": "1149320", "postDate": "01/11/2021 19:01:05", "content": "Smart! Thanks for sharing!
", "rawMarkdown": "Smart! Thanks for sharing!", "votes": null } ], "comments": [ { "id": 1132414, "author_name": "user123454321", "author_url": "", "post_date": "12/30/2020 11:47:17", "content": "Are you using ensemble models ? For me it takes about 2 mins to load, 10-20 mins for commit and save, and about 20 mins for submission on single model, no augmentations on test set as well, just straight inference. I have tried with EfficientNet B7 and DenseNet201.
", "votes": null, "replies": [ { "id": 1132439, "author_name": "oberfink", "author_url": "", "post_date": "12/30/2020 12:11:23", "content": "No I'm just using a single model. But maybe it's because I'm also augmenting the test images. I'll try to submit without augmentation. Thanks for the pointer and thanks for letting me know your running times
", "votes": null, "replies": [] } ] }, { "id": 1133386, "author_name": "sakuraandblackcat", "author_url": "", "post_date": "12/31/2020 07:09:19", "content": "Hello!
\nI need almost the same time to submit as you.
\nHowever, I can run the next model during \"step 3\" in your approach. Probably, you do not need to wait to finish the score calculation.
You are right! And also you can submit multiple submission. Thanks for sharing!
", "votes": null, "replies": [] } ] }, { "id": 1133862, "author_name": "oberfink", "author_url": "", "post_date": "12/31/2020 15:36:24", "content": "After the comments here in this discussion, I realized that I was also augmenting my test images. On my local machine it takes 4 times as long to predict test images with augmentation instead of test images without augmentation. (which is reasonable!)
", "votes": null, "replies": [] }, { "id": 1146746, "author_name": "marcelbischoff", "author_url": "", "post_date": "01/10/2021 02:32:51", "content": "You can completely skip step two with a simple trick, namely checking if the sample_submission is the public one. I made a quick template: https://www.kaggle.com/marcelbischoff/ranzcr-quick-submission-template
\nIf there is not enough time you want to do two notebooks. One for training, then export the weights into a dataset and then another notebook which loads the weights and does only inference.
", "votes": null, "replies": [ { "id": 1149320, "author_name": "oberfink", "author_url": "", "post_date": "01/11/2021 19:01:05", "content": "Smart! Thanks for sharing!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1132365": "Hi,\n\nI'm wondering: What is the most efficient way to submit (an inference kernel)? For me it is almost impossible to submit 5 submissions a day (not that I want to, I'm just curious). Here is my approach so far:\n\n1. Create a new Notebook and upload my local Notebook file as well as other important inputs such as the model and other libraries. --> This is relatively fast (~10 min)\n2. Save & Run All (Commit) --> Even using GPU this step takes between 1-2 hours\n3. Submitting to Competition --> This step takes even longer than the second step. (~2-4 hours)\n\nThus, in total I need between 3 and 6 hours to submit my notebook. I always use GPU and I'm using models such as EfficientNet-B0-B4 or ResNet18,34,50.\n\nRecently I tried to submit a model with TTA prediction but I got an 'Notebook Timeout' Error when trying to submit it. \n\nAm I doing something wrong? Thanks for your help.", "1132414": "Are you using ensemble models ? For me it takes about 2 mins to load, 10-20 mins for commit and save, and about 20 mins for submission on single model, no augmentations on test set as well, just straight inference. I have tried with EfficientNet B7 and DenseNet201.", "1132439": "No I'm just using a single model. But maybe it's because I'm also augmenting the test images. I'll try to submit without augmentation. Thanks for the pointer and thanks for letting me know your running times", "1133386": "Hello!\nI need almost the same time to submit as you. \nHowever, I can run the next model during \"step 3\" in your approach. Probably, you do not need to wait to finish the score calculation.", "1133586": "You are right! And also you can submit multiple submission. Thanks for sharing!", "1133862": "After the comments here in this discussion, I realized that I was also augmenting my test images. On my local machine it takes 4 times as long to predict test images with augmentation instead of test images without augmentation. (which is reasonable!)", "1146746": "You can completely skip step two with a simple trick, namely checking if the sample_submission is the public one. I made a quick template: https://www.kaggle.com/marcelbischoff/ranzcr-quick-submission-template\n\nIf there is not enough time you want to do two notebooks. One for training, then export the weights into a dataset and then another notebook which loads the weights and does only inference.", "1149320": "Smart! Thanks for sharing!" }, "source": "meta" }