My kernel executes in under 20 mins, yet it's been pending in submission mode for over 6h now - despite throwing no errors about input, issues with internet or whatever. Anybody else encountered that problem?
", "messages": [ { "id": 1045607, "postDate": "2020-10-10T20:11:41.567Z", "content": "My kernel executes in under 20 mins, yet it's been pending in submission mode for over 6h now - despite throwing no errors about input, issues with internet or whatever. Anybody else encountered that problem?
", "rawMarkdown": "My kernel executes in under 20 mins, yet it's been pending in submission mode for over 6h now - despite throwing no errors about input, issues with internet or whatever. Anybody else encountered that problem?", "votes": 19 }, { "id": 1046136, "postDate": "2020-10-11T11:29:35.603Z", "content": "Only reason could be if we are using some grouping feature usually it will run very fast in example . But on every batch of huge private test grouping and sorting might take long time and memory . Let me test without that and confirm
", "rawMarkdown": "Only reason could be if we are using some grouping feature usually it will run very fast in example . But on every batch of huge private test grouping and sorting might take long time and memory . Let me test without that and confirm ", "votes": 3, "replies": [ { "id": 1047500, "postDate": "2020-10-12T17:02:47.653Z", "content": "Yep, as pandas have quadratic complexity if someone let's say keep on appending to an existing dataframe.
", "rawMarkdown": "Yep, as pandas have quadratic complexity if someone let's say keep on appending to an existing dataframe." }, { "id": 1047678, "postDate": "2020-10-12T20:54:17.450Z", "content": "FWIW I did what @phoenix9032 suggested and it helped.
", "rawMarkdown": "FWIW I did what @phoenix9032 suggested and it helped." } ] }, { "id": 1047460, "postDate": "2020-10-12T16:15:02.767Z", "content": "Remember that the full hidden test set is several orders of magnitude larger than the example test set that you processed in 20 minutes.
", "rawMarkdown": "Remember that the full hidden test set is several orders of magnitude larger than the example test set that you processed in 20 minutes.", "votes": 1, "replies": [ { "id": 1047677, "postDate": "2020-10-12T20:53:52.710Z", "content": "Yes, I know that :-) Turns out the problem was elsewhere: pd.merge is really really slow for large data - removing one line involving joining the test set with a result of a groupby solved the problem.
", "rawMarkdown": "Yes, I know that :-) Turns out the problem was elsewhere: pd.merge is really really slow for large data - removing one line involving joining the test set with a result of a groupby solved the problem.", "votes": 4 }, { "id": 1049470, "postDate": "2020-10-14T12:51:15.047Z", "content": "@sohier, is there any option to increase test batch size during inference? I mean to increase size of the test when iterating over the loop env.iter_test().
", "rawMarkdown": "@sohier, is there any option to increase test batch size during inference? I mean to increase size of the test when iterating over the loop env.iter_test().", "votes": 2 }, { "id": 1100009, "postDate": "2020-12-02T18:55:00.460Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1046009, "postDate": "2020-10-11T08:45:04.077Z", "content": "Have you considered splitting the training and submission part? I have only made one \"dummy\" sub, that worked <10 mins
", "rawMarkdown": "Have you considered splitting the training and submission part? I have only made one \"dummy\" sub, that worked <10 mins", "votes": 1, "replies": [ { "id": 1046130, "postDate": "2020-10-11T11:19:50.023Z", "content": "Not sure Aditya . Considering my training is only for 1 minute and my commit finishes in around 5 minute .. the kernel is running for past 1 hour now .
", "rawMarkdown": "Not sure Aditya . Considering my training is only for 1 minute and my commit finishes in around 5 minute .. the kernel is running for past 1 hour now . " } ] }, { "id": 1050275, "postDate": "2020-10-15T08:05:30.580Z", "content": "It seems that running on GPU is taking way more time than making the submission on CPU. Anyone else faced this?
", "rawMarkdown": "It seems that running on GPU is taking **way** more time than making the submission on CPU. Anyone else faced this?", "votes": 2, "replies": [ { "id": 1074384, "postDate": "2020-11-10T15:38:24.510Z", "content": "Yes, I used Rapids and it's running for past 2.5 hrs (with pandas it took ~2 hrs). Unfortunately Idk why.
\nI'm going to try to split inference and training.
\nIs there a way to find exact time each notebook submission (i.e. inference on unknown test set) takes ?
Thank you SO MUCH. This was my problem, I was using GPU and it took 4 times more than using CPU.
", "rawMarkdown": "Thank you SO MUCH. This was my problem, I was using GPU and it took 4 times more than using CPU." } ] }, { "id": 1050449, "postDate": "2020-10-15T12:30:32.767Z", "content": "I'm facing the same problem.
\nI believe that I am not optimizing enough.
Did you figure out this problem?
", "rawMarkdown": "Did you figure out this problem?" }, { "id": 1046617, "postDate": "2020-10-11T21:00:04.117Z", "content": "Yes I am facing same issue as well
", "rawMarkdown": "Yes I am facing same issue as well" }, { "id": 1045982, "postDate": "2020-10-11T07:58:12.817Z", "content": "Yes it took around 3 hrs for me too, Don't know why it takes to long time for submission
", "rawMarkdown": "Yes it took around 3 hrs for me too, Don't know why it takes to long time for submission" }, { "id": 1045685, "postDate": "2020-10-10T22:33:20.960Z", "content": "Not 6 , but 3 hours for a 30 mins kernel today . I think it varies on the load or something , not sure .
", "rawMarkdown": "Not 6 , but 3 hours for a 30 mins kernel today . I think it varies on the load or something , not sure .", "replies": [ { "id": 1045910, "postDate": "2020-10-11T06:29:06.373Z", "content": "Did it eventually finish successfully? Mine just died after 8h with \"submission error\" message, no additional detail…
", "rawMarkdown": "Did it eventually finish successfully? Mine just died after 8h with \"submission error\" message, no additional detail...", "votes": 3 }, { "id": 1045912, "postDate": "2020-10-11T06:32:36.023Z", "content": "Submitted 2 . One success and one failure :)
", "rawMarkdown": "Submitted 2 . One success and one failure :) ", "votes": 1 } ] } ], "comments": [ { "id": 1046136, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-10-11T11:29:35.603000", "content": "Only reason could be if we are using some grouping feature usually it will run very fast in example . But on every batch of huge private test grouping and sorting might take long time and memory . Let me test without that and confirm
", "votes": 3, "replies": [ { "id": 1047500, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-12T17:02:47.653000", "content": "Yep, as pandas have quadratic complexity if someone let's say keep on appending to an existing dataframe.
", "votes": 0, "replies": [] }, { "id": 1047678, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2020-10-12T20:54:17.450000", "content": "FWIW I did what @phoenix9032 suggested and it helped.
", "votes": 0, "replies": [] } ] }, { "id": 1047460, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-10-12T16:15:02.767000", "content": "Remember that the full hidden test set is several orders of magnitude larger than the example test set that you processed in 20 minutes.
", "votes": 1, "replies": [ { "id": 1047677, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2020-10-12T20:53:52.710000", "content": "Yes, I know that :-) Turns out the problem was elsewhere: pd.merge is really really slow for large data - removing one line involving joining the test set with a result of a groupby solved the problem.
", "votes": 4, "replies": [] }, { "id": 1049470, "author_name": "Gleb Anferov", "author_url": "", "post_date": "2020-10-14T12:51:15.047000", "content": "@sohier, is there any option to increase test batch size during inference? I mean to increase size of the test when iterating over the loop env.iter_test().
", "votes": 2, "replies": [] }, { "id": 1100009, "author_name": "", "author_url": "", "post_date": "2020-12-02T18:55:00.460000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1046009, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-11T08:45:04.077000", "content": "Have you considered splitting the training and submission part? I have only made one \"dummy\" sub, that worked <10 mins
", "votes": 1, "replies": [ { "id": 1046130, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-10-11T11:19:50.023000", "content": "Not sure Aditya . Considering my training is only for 1 minute and my commit finishes in around 5 minute .. the kernel is running for past 1 hour now .
", "votes": 0, "replies": [] } ] }, { "id": 1050275, "author_name": "Aditya Soni", "author_url": "", "post_date": "2020-10-15T08:05:30.580000", "content": "It seems that running on GPU is taking way more time than making the submission on CPU. Anyone else faced this?
", "votes": 2, "replies": [ { "id": 1074384, "author_name": "RB", "author_url": "", "post_date": "2020-11-10T15:38:24.510000", "content": "Yes, I used Rapids and it's running for past 2.5 hrs (with pandas it took ~2 hrs). Unfortunately Idk why.
\nI'm going to try to split inference and training.
\nIs there a way to find exact time each notebook submission (i.e. inference on unknown test set) takes ?
Thank you SO MUCH. This was my problem, I was using GPU and it took 4 times more than using CPU.
", "votes": 0, "replies": [] } ] }, { "id": 1050449, "author_name": "pea98258", "author_url": "", "post_date": "2020-10-15T12:30:32.767000", "content": "I'm facing the same problem.
\nI believe that I am not optimizing enough.
Did you figure out this problem?
", "votes": 0, "replies": [] }, { "id": 1046617, "author_name": "Kamal", "author_url": "", "post_date": "2020-10-11T21:00:04.117000", "content": "Yes I am facing same issue as well
", "votes": 0, "replies": [] }, { "id": 1045982, "author_name": "V.Prasanna Kumar", "author_url": "", "post_date": "2020-10-11T07:58:12.817000", "content": "Yes it took around 3 hrs for me too, Don't know why it takes to long time for submission
", "votes": 0, "replies": [] }, { "id": 1045685, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-10-10T22:33:20.960000", "content": "Not 6 , but 3 hours for a 30 mins kernel today . I think it varies on the load or something , not sure .
", "votes": 0, "replies": [ { "id": 1045910, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2020-10-11T06:29:06.373000", "content": "Did it eventually finish successfully? Mine just died after 8h with \"submission error\" message, no additional detail…
", "votes": 3, "replies": [] }, { "id": 1045912, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2020-10-11T06:32:36.023000", "content": "Submitted 2 . One success and one failure :)
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1045607": "My kernel executes in under 20 mins, yet it's been pending in submission mode for over 6h now - despite throwing no errors about input, issues with internet or whatever. Anybody else encountered that problem?", "1046136": "Only reason could be if we are using some grouping feature usually it will run very fast in example . But on every batch of huge private test grouping and sorting might take long time and memory . Let me test without that and confirm ", "1047460": "Remember that the full hidden test set is several orders of magnitude larger than the example test set that you processed in 20 minutes.", "1046009": "Have you considered splitting the training and submission part? I have only made one \"dummy\" sub, that worked <10 mins", "1050275": "It seems that running on GPU is taking **way** more time than making the submission on CPU. Anyone else faced this?", "1050449": "I'm facing the same problem.\nI believe that I am not optimizing enough.", "1049157": "Did you figure out this problem?", "1046617": "Yes I am facing same issue as well", "1045982": "Yes it took around 3 hrs for me too, Don't know why it takes to long time for submission", "1045685": "Not 6 , but 3 hours for a 30 mins kernel today . I think it varies on the load or something , not sure ." } }