Hello, guys,
\nIt has been really a tough day for me, as I tried 3 or four time to do the inference on Kaggle kernels(notebook and script). However, all these tries give me the following errors.
\nRuntimeError: DataLoader worker (pid 293) is killed by signal: Killed.
I thought it may be the limit of running time of a Kaggle kernel. So I have a question about our submission, should I inference on my local machine and upload my submission file as an input to a kernel? It may not be an elegant way, but it should work.
", "messages": [ { "id": "1002922", "postDate": "09/08/2020 14:28:14", "content": "Hello, guys,
\nIt has been really a tough day for me, as I tried 3 or four time to do the inference on Kaggle kernels(notebook and script). However, all these tries give me the following errors.
\nRuntimeError: DataLoader worker (pid 293) is killed by signal: Killed.
I thought it may be the limit of running time of a Kaggle kernel. So I have a question about our submission, should I inference on my local machine and upload my submission file as an input to a kernel? It may not be an elegant way, but it should work.
", "rawMarkdown": "Hello, guys,\n\nIt has been really a tough day for me, as I tried 3 or four time to do the inference on Kaggle kernels(notebook and script). However, all these tries give me the following errors. \n`RuntimeError: DataLoader worker (pid 293) is killed by signal: Killed.`\n\nI thought it may be the limit of running time of a Kaggle kernel. So I have a question about our submission, should I inference on my local machine and upload my submission file as an input to a kernel? It may not be an elegant way, but it should work.", "votes": null }, { "id": "1002935", "postDate": "09/08/2020 14:40:26", "content": "Hi @usherbob It is better to run training and inference separately. And Inference no way exceeds kaggle limit of 9 hour. Just to give you the math.
\nFor model:Resnet50
\nNumber of sample(timestep) in test = 71122
\nIf your batch size = 32
\nthen your iterations = 2223
\nRunning on = 1 GPU
\nRunning time for this iteration = 5500 sec
\nEven if you go down on your batch size to 8 and run 8891 iterations. Your run time will be around 7000 sec.
\nThink you need to recheck what part of inference is taking time and see if it is really needed. Normally infer should not take more time than i mentioned.
\n\nshould I inference on my local machine and upload my submission file as an input to a kernel?
\n
Yes, and it's much faster. For me, it only takes less than half an hour by setting num_workers=40.
", "rawMarkdown": "> should I inference on my local machine and upload my submission file as an input to a kernel? \n\nYes, and it's much faster. For me, it only takes less than half an hour by setting num_workers=40.", "votes": null }, { "id": "1006042", "postDate": "09/11/2020 02:32:23", "content": "Hi @deepakrajpurushothaman, the inference did not take 9 hours or longer. It's just that the notebook kernel reminds us to continue editing or leaving when the notebook runs 40 minutes or longer. If I did not respond to the reminding immediately, it'll kill the kernel and the inference dies.
", "rawMarkdown": "Hi @deepakrajpurushothaman, the inference did not take 9 hours or longer. It's just that the notebook kernel reminds us to continue editing or leaving when the notebook runs 40 minutes or longer. If I did not respond to the reminding immediately, it'll kill the kernel and the inference dies.", "votes": null }, { "id": "1006052", "postDate": "09/11/2020 02:50:59", "content": "@usherbob that is sad. Maybe you are facing something i cant imagine.
\nCase1: This is what I normally do, once i am sure all of my code runs properly I click 'commit and run all' and close the window and return to it after runtime.
\nCase2: This is special case. if i want to use editor after 'commit and run'. I will be editing and 40 min idle reminder wont pop up.
\nCase3: If I a situation where you 'commit and run' and keep editor idle for 40 min or longer. I recommend you to press the button 'cancel' and not 'continue editing'. If you press 'continue editing' after keeping notebook idle for 40 min It will kill the existing running kernel.
\nThis is my experience. FYI I am on kaggle from last 3 weeks. I hope I have got it right.
Thanks for your suggestions, I'll have a try.
", "rawMarkdown": "Thanks for your suggestions, I'll have a try.", "votes": null } ], "comments": [ { "id": 1002935, "author_name": "deepakrajpurushothaman", "author_url": "", "post_date": "09/08/2020 14:40:26", "content": "Hi @usherbob It is better to run training and inference separately. And Inference no way exceeds kaggle limit of 9 hour. Just to give you the math.
\nFor model:Resnet50
\nNumber of sample(timestep) in test = 71122
\nIf your batch size = 32
\nthen your iterations = 2223
\nRunning on = 1 GPU
\nRunning time for this iteration = 5500 sec
\nEven if you go down on your batch size to 8 and run 8891 iterations. Your run time will be around 7000 sec.
\nThink you need to recheck what part of inference is taking time and see if it is really needed. Normally infer should not take more time than i mentioned.
Hi @deepakrajpurushothaman, the inference did not take 9 hours or longer. It's just that the notebook kernel reminds us to continue editing or leaving when the notebook runs 40 minutes or longer. If I did not respond to the reminding immediately, it'll kill the kernel and the inference dies.
", "votes": null, "replies": [] }, { "id": 1006052, "author_name": "deepakrajpurushothaman", "author_url": "", "post_date": "09/11/2020 02:50:59", "content": "@usherbob that is sad. Maybe you are facing something i cant imagine.
\nCase1: This is what I normally do, once i am sure all of my code runs properly I click 'commit and run all' and close the window and return to it after runtime.
\nCase2: This is special case. if i want to use editor after 'commit and run'. I will be editing and 40 min idle reminder wont pop up.
\nCase3: If I a situation where you 'commit and run' and keep editor idle for 40 min or longer. I recommend you to press the button 'cancel' and not 'continue editing'. If you press 'continue editing' after keeping notebook idle for 40 min It will kill the existing running kernel.
\nThis is my experience. FYI I am on kaggle from last 3 weeks. I hope I have got it right.
Thanks for your suggestions, I'll have a try.
", "votes": null, "replies": [] } ] }, { "id": 1002960, "author_name": "zzy990106", "author_url": "", "post_date": "09/08/2020 14:58:40", "content": "\n\nshould I inference on my local machine and upload my submission file as an input to a kernel?
\n
Yes, and it's much faster. For me, it only takes less than half an hour by setting num_workers=40.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1002922": "Hello, guys,\n\nIt has been really a tough day for me, as I tried 3 or four time to do the inference on Kaggle kernels(notebook and script). However, all these tries give me the following errors. \n`RuntimeError: DataLoader worker (pid 293) is killed by signal: Killed.`\n\nI thought it may be the limit of running time of a Kaggle kernel. So I have a question about our submission, should I inference on my local machine and upload my submission file as an input to a kernel? It may not be an elegant way, but it should work.", "1002935": "Hi @usherbob It is better to run training and inference separately. And Inference no way exceeds kaggle limit of 9 hour. Just to give you the math. \nFor model:Resnet50\nNumber of sample(timestep) in test = 71122\nIf your batch size = 32 \nthen your iterations = 2223\nRunning on = 1 GPU\nRunning time for this iteration = 5500 sec \nEven if you go down on your batch size to 8 and run 8891 iterations. Your run time will be around 7000 sec. \nThink you need to recheck what part of inference is taking time and see if it is really needed. Normally infer should not take more time than i mentioned.", "1002960": "> should I inference on my local machine and upload my submission file as an input to a kernel? \n\nYes, and it's much faster. For me, it only takes less than half an hour by setting num_workers=40.", "1006042": "Hi @deepakrajpurushothaman, the inference did not take 9 hours or longer. It's just that the notebook kernel reminds us to continue editing or leaving when the notebook runs 40 minutes or longer. If I did not respond to the reminding immediately, it'll kill the kernel and the inference dies.", "1006052": "usherbob that is sad. Maybe you are facing something i cant imagine. \nCase1: This is what I normally do, once i am sure all of my code runs properly I click 'commit and run all' and close the window and return to it after runtime. \nCase2: This is special case. if i want to use editor after 'commit and run'. I will be editing and 40 min idle reminder wont pop up. \nCase3: If I a situation where you 'commit and run' and keep editor idle for 40 min or longer. I recommend you to press the button 'cancel' and not 'continue editing'. If you press 'continue editing' after keeping notebook idle for 40 min It will kill the existing running kernel. \nThis is my experience. FYI I am on kaggle from last 3 weeks. I hope I have got it right.", "1009493": "Thanks for your suggestions, I'll have a try." }, "source": "meta" }