This is a Kernels-Only (Code) Competition. As such, the competition deadline and stage 2 close procedure will differ from our standard competitions. Here is a summary of the notable differences. Subscribe to the competition's forum to stay up to date on the latest important updates.
\n\nThis is among the first competitions being run in this unique format, so we ask for your patience during this time as we fine tune the process and engineering required to make more of these competitions a reality for our community.
", "messages": [ { "id": "452545", "postDate": "01/08/2019 21:49:05", "content": "This is a Kernels-Only (Code) Competition. As such, the competition deadline and stage 2 close procedure will differ from our standard competitions. Here is a summary of the notable differences. Subscribe to the competition's forum to stay up to date on the latest important updates.
\n\nThis is among the first competitions being run in this unique format, so we ask for your patience during this time as we fine tune the process and engineering required to make more of these competitions a reality for our community.
", "rawMarkdown": "This is a Kernels-Only (Code) Competition. As such, the competition deadline and stage 2 close procedure will differ from our standard competitions. Here is a summary of the notable differences. Subscribe to the competition's forum to stay up to date on the latest important updates.\n\n* **February 5 (11:59PM UTC) is the final deadline** for teams to both: (A) make any new kernel submissions & (B) select your 2 final submissions.\n* You **must manually select your final 2 submissions**, by navigating to [\"My Submissions\"](https://www.kaggle.com/c/quora-insincere-questions-classification/submissions) and choosing the “Use for Final Score” checkboxes corresponding with the selected submissions. These submissions' corresponding kernels will be the only models run on the unseen, held out private test set. It is your best score on this test set that will determine your final score and ranking in the competition. Unlike other competitions, our system will not auto-select submissions for you, and those who do not make selections will no longer be reflected as a participant on the final leaderboard.\n* **Kernels which error for any reason or exceed the stated restrictions when run on the private test set will be discarded without exception**. It is your responsibility to ensure your model is coded with this in mind. It is recommended that, due to runtime variability and private test set composition, you give your runtime adequate buffer to anticipate this. Please refer to the [Data page](https://www.kaggle.com/c/quora-insincere-questions-classification/data) for details on the stage 2 test set composition. Also refer to the [Kernels FAQ Page](https://www.kaggle.com/c/quora-insincere-questions-classification#Kernels-FAQ) for details of the quotas available to you in this competition, which are equivalent to the constraints applied during stage 2.\n* At the submission deadline (February 5th, 11:59PM UTC), submissions will be disabled and the competition timeline will be extended by 3 weeks, during which time Kernels will be rescored. The rescore process re-runs your selected kernels in bulk, behind the scenes, against the private test set. We will provide an update when this process is complete and the private leaderboard is final.\n\nThis is among the first competitions being run in this unique format, so we ask for your patience during this time as we fine tune the process and engineering required to make more of these competitions a reality for our community.", "votes": null }, { "id": "452666", "postDate": "01/09/2019 02:37:12", "content": "Thank you for this information!
\n\nMy only concern is the third point: yesterday-today we (kagglers) experienced a lot of problems with kernel execution due two kernel-only competitions ending. So it could be possible that some kernels won't run successfully due to lack of resourses of Kaggle...
", "rawMarkdown": "Thank you for this information!\n\nMy only concern is the third point: yesterday-today we (kagglers) experienced a lot of problems with kernel execution due two kernel-only competitions ending. So it could be possible that some kernels won't run successfully due to lack of resourses of Kaggle...", "votes": null }, { "id": "452714", "postDate": "01/09/2019 04:26:16", "content": "Thanks for summarizing this @juliaelliott;
\n\nHere are some extra cautions I suggest who are new to kernels' competition
\n\nAvoid hard coded variables and shapes in your kernel
\n\nLog your runtime. Don't rely on Kaggle runtime numbers
\n\nKeep your eyes open on any kernel/docker image changes
\n\nPeace.
", "rawMarkdown": "Thanks for summarizing this @juliaelliott;\n\nHere are some extra cautions I suggest who are new to kernels' competition\n\n1. **Avoid hard coded variables and shapes in your kernel**\n - The final private train and test although will have same column names but different set of rows and size of dataset. Make sure you don't set any hard changes w.r.t to length of the dataframe or any shape in your neural network\n\n2. **Log your runtime. Don't rely on Kaggle runtime numbers**\n - We see inconsistency in scores and kernel runtime depending on surge of usage. It is wise to log your own runtime and compare its scalability with huge private set. In other words if you are close to 6999 seconds. You might want to see things tuned there!\n\n3. **Keep your eyes open on any kernel/docker image changes**\n- Lately few on the discussion have reported huge runtimes with kernel updates. Make sure you make version checks on libraries you are using to avoid last minute confusions. \n\nPeace.", "votes": null }, { "id": "452896", "postDate": "01/09/2019 10:26:39", "content": "thanks - tip 2 is a really important point and something I hadn't thought of
", "rawMarkdown": "thanks - tip 2 is a really important point and something I hadn't thought of", "votes": null }, { "id": "452987", "postDate": "01/09/2019 12:59:41", "content": "\"if you are close to 6999 seconds\" you mean to 7200 seconds (3600s x 2h)
", "rawMarkdown": "\"if you are close to 6999 seconds\" you mean to 7200 seconds (3600s x 2h)", "votes": null }, { "id": "453022", "postDate": "01/09/2019 14:29:08", "content": "How to see the kernel/docker image changes?
", "rawMarkdown": "How to see the kernel/docker image changes?", "votes": null }, { "id": "453032", "postDate": "01/09/2019 14:51:59", "content": "You can see the latest discussions here - https://www.kaggle.com/discussion\nFor more specific information you can hover to Github public docker image - https://github.com/Kaggle/docker-python
", "rawMarkdown": "You can see the latest discussions here - https://www.kaggle.com/discussion\nFor more specific information you can hover to Github public docker image - https://github.com/Kaggle/docker-python", "votes": null }, { "id": "453069", "postDate": "01/09/2019 16:14:27", "content": "\"Kernels which error for any reason or exceed the stated restrictions when run on the private test set will be discarded without exception.\"
\n\nIn my experience, there is already up to 10% (or more) variability in kernel run-times for identical models, which is a lot. Considering that we are trying to squeeze in as much as we can within the 2 hour limit, this adds another layer of unnecessary randomness. In competitions like this, randomness is unfairness. Wouldn't it be fairer (and simpler) if any submission that clears the first stage (i.e. completes under 7200s) be automatically be considered as valid for the second round even if it fails to make the 7200s limit?
", "rawMarkdown": "\"Kernels which error for any reason or exceed the stated restrictions when run on the private test set will be discarded without exception.\"\n\n\nIn my experience, there is already up to 10% (or more) variability in kernel run-times for identical models, which is a lot. Considering that we are trying to squeeze in as much as we can within the 2 hour limit, this adds another layer of unnecessary randomness. In competitions like this, randomness is unfairness. Wouldn't it be fairer (and simpler) if any submission that clears the first stage (i.e. completes under 7200s) be automatically be considered as valid for the second round even if it fails to make the 7200s limit?", "votes": null }, { "id": "453191", "postDate": "01/09/2019 20:40:26", "content": "Fair point. I think it makes sense.
", "rawMarkdown": "Fair point. I think it makes sense.", "votes": null }, { "id": "453314", "postDate": "01/10/2019 02:08:05", "content": "What is the stage 2 public time ?
", "rawMarkdown": "What is the stage 2 public time ?", "votes": null }, { "id": "455098", "postDate": "01/13/2019 01:24:24", "content": "With your approach one can check the length of the test.csv in his kernel and run a different model for the second stage
", "rawMarkdown": "With your approach one can check the length of the test.csv in his kernel and run a different model for the second stage", "votes": null }, { "id": "455144", "postDate": "01/13/2019 04:17:21", "content": "I think it is really important to measure the run time during prediction on stage 1 test set. Make an assumption that the run time of previous stage 1 test set is multiply to 6.7 during stage 2 and add 8-10% buffer time for stage 2. We never know the complexity of stage 2 test set.
", "rawMarkdown": "I think it is really important to measure the run time during prediction on stage 1 test set. Make an assumption that the run time of previous stage 1 test set is multiply to 6.7 during stage 2 and add 8-10% buffer time for stage 2. We never know the complexity of stage 2 test set.", "votes": null }, { "id": "455239", "postDate": "01/13/2019 11:12:51", "content": "Yeah, I suppose this can be gamed. No way around it, I guess
", "rawMarkdown": "Yeah, I suppose this can be gamed. No way around it, I guess", "votes": null }, { "id": "457061", "postDate": "01/16/2019 22:28:40", "content": "hi, I didn't see any information about final private train data.
", "rawMarkdown": "hi, I didn't see any information about final private train data.", "votes": null }, { "id": "463614", "postDate": "01/30/2019 09:41:25", "content": "@juliaelliott Can you please make a clarification about hard-coded dictionaries and lists of values in kernels? Are they allowed or not? That is very imortant to us. Thanks
", "rawMarkdown": "juliaelliott Can you please make a clarification about hard-coded dictionaries and lists of values in kernels? Are they allowed or not? That is very imortant to us. Thanks", "votes": null }, { "id": "463723", "postDate": "01/30/2019 13:35:06", "content": "@juliaelliott: Please respond! :-)
", "rawMarkdown": "juliaelliott: Please respond! :-)", "votes": null }, { "id": "464425", "postDate": "01/31/2019 20:58:08", "content": "Will we be able to see the output files and logs generated when our kernels are run for stage 2? I hope so...
\n\n(This was not the case in the Mercari competition: you can only see logs/submissions for public LB runs, with just a score for the private LB... If a kernel fails to run in stage 2 — as happened to 1/3 of the teams in Mercari — it would be really nice to see the actual logs/outputs to quickly find out why, to learn from it.)
", "rawMarkdown": "Will we be able to see the output files and logs generated when our kernels are run for stage 2? I hope so...\n\n(This was not the case in the [Mercari][1] competition: you can only see logs/submissions for public LB runs, with just a score for the private LB... If a kernel fails to run in stage 2 — as happened to 1/3 of the teams in Mercari — it would be really nice to see the actual logs/outputs to quickly find out why, to learn from it.)\n\n [1]: https://www.kaggle.com/c/mercari-price-suggestion-challenge", "votes": null }, { "id": "464501", "postDate": "02/01/2019 01:38:19", "content": "thanks!
", "rawMarkdown": "thanks!", "votes": null }, { "id": "464550", "postDate": "02/01/2019 04:22:08", "content": "Hi, where can we find submission daily limit?
", "rawMarkdown": "Hi, where can we find submission daily limit?", "votes": null }, { "id": "464782", "postDate": "02/01/2019 13:59:28", "content": "Hi, I ran the same kernel 5 times, each time is different. And the runtime in my log is also different from the one displayed by the kernel. How does the rescore phase determine whether it is timed out?
", "rawMarkdown": "Hi, I ran the same kernel 5 times, each time is different. And the runtime in my log is also different from the one displayed by the kernel. How does the rescore phase determine whether it is timed out?", "votes": null }, { "id": "465872", "postDate": "02/04/2019 08:07:38", "content": "In the Rules: You may submit a maximum of 5 entries per day.
", "rawMarkdown": "In the Rules: You may submit a maximum of 5 entries per day.", "votes": null }, { "id": "466186", "postDate": "02/04/2019 20:34:46", "content": "Hi,
\n\nWould it be possible to run each kernel several (e.g. 3) times in case they fail/timeover? As far as I know, when we benchmark the speed of programs we tend to take the minimum runtime out of all the measurements since slower speeds are likely to be side effects of other processes in the system. In the same vein, slow kernels might just be caused by random fluctuations like being batched with heavier kernels. So it would seem fairer to try several runs.\nIf you don't want to reward people pushing the limits, you could run all kernels multiple times and take the best submission out of the runs, so people who timeover in any of those runs lose a chance to get a better submission. Considering how volatile the scores on the leaderboard seem to be even with the same kernel, I think many of us would appreciate it.
\n\nHoping the best for this competition!
", "rawMarkdown": "Hi, \n\nWould it be possible to run each kernel several (e.g. 3) times in case they fail/timeover? As far as I know, when we benchmark the speed of programs we tend to take the minimum runtime out of all the measurements since slower speeds are likely to be side effects of other processes in the system. In the same vein, slow kernels might just be caused by random fluctuations like being batched with heavier kernels. So it would seem fairer to try several runs.\nIf you don't want to reward people pushing the limits, you could run all kernels multiple times and take the best submission out of the runs, so people who timeover in any of those runs lose a chance to get a better submission. Considering how volatile the scores on the leaderboard seem to be even with the same kernel, I think many of us would appreciate it.\n\nHoping the best for this competition!", "votes": null }, { "id": "466242", "postDate": "02/04/2019 23:57:35", "content": "\n\n\nKernels which error for any reason or exceed the stated restrictions when run on the private test set will be discarded without exception.
\n
I was ok with this until a few hours ago...
\n\nI've run my solution over 40 times without problems, but when I ran it this evening, for the first time there was a CUDA error in importing keras:
\n\n2019-02-04 20:33:47.437361: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error\n2019-02-04 20:33:47.437459: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] retrieving CUDA diagnostic information for host: ddaf5ed64f42\n2019-02-04 20:33:47.437478: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:170] hostname: ddaf5ed64f42\n2019-02-04 20:33:47.437647: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:194] libcuda reported version is: 396.44.0\n2019-02-04 20:33:47.437696: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:198] kernel reported version is: 396.44.0\n2019-02-04 20:33:47.437724: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:305] kernel version seems to match DSO: 396.44.0\nThen every call to model.fit() failed (errors attached).
\n\nRunning the exact same script again, it succeeded.
\n\nI'm sure this error is rare, but with over 4000 teams, hence over 8000 submissions to generate, it becomes likely to happen to some of them.
\n\nI hope Kaggle can see sense here and do some simple statistical analysis of failed submissions, even just a quick grep of the output logs for the above (or similar, known but random) errors?
\n\n(This problem is worse if we cannot see logs/output of our stage 2 runs - @juliaelliott could you please respond to my earlier question, even just a yes/no answer? Will we get to see logs?)
", "rawMarkdown": "> Kernels which error for **any reason** or exceed the stated restrictions when run on the private test set will be discarded without exception.\n\nI was ok with this until a few hours ago...\n\nI've run my solution over 40 times without problems, but when I ran it this evening, for the first time there was a CUDA error in importing keras:\n\n\n 2019-02-04 20:33:47.437361: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error\n 2019-02-04 20:33:47.437459: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] retrieving CUDA diagnostic information for host: ddaf5ed64f42\n 2019-02-04 20:33:47.437478: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:170] hostname: ddaf5ed64f42\n 2019-02-04 20:33:47.437647: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:194] libcuda reported version is: 396.44.0\n 2019-02-04 20:33:47.437696: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:198] kernel reported version is: 396.44.0\n 2019-02-04 20:33:47.437724: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:305] kernel version seems to match DSO: 396.44.0\n\n\nThen every call to model.fit() failed (errors attached).\n\nRunning the ***exact same script*** again, it succeeded.\n\nI'm sure this error is rare, but with over 4000 teams, hence over 8000 submissions to generate, it becomes likely to happen to some of them.\n\nI hope Kaggle can see sense here and do some simple statistical analysis of failed submissions, even just a quick grep of the output logs for the above (or similar, known but random) errors?\n\n(This problem is worse if we cannot see logs/output of our stage 2 runs - @juliaelliott could you please respond to my earlier question, even just a yes/no answer? Will we get to see logs?)", "votes": null }, { "id": "466703", "postDate": "02/05/2019 20:55:40", "content": "Hi it was interesting!
", "rawMarkdown": "Hi it was interesting!", "votes": null }, { "id": "466777", "postDate": "02/05/2019 23:01:17", "content": "FYI: We've updated the timeline, per my original forum post, to 3 weeks out. Submissions will continue to be accepted until February 5, 2019 at 11:59pm UTC.
\n\nDuring the 3 weeks remaining, we will be running your selected submissions against the private test set, but no submissions will be accepted. Stay tuned for the final leaderboard and rankings reveal!
", "rawMarkdown": "FYI: We've updated the timeline, per my original forum post, to 3 weeks out. Submissions will continue to be accepted until February 5, 2019 at 11:59pm UTC. \n\nDuring the 3 weeks remaining, we will be running your selected submissions against the private test set, but no submissions will be accepted. Stay tuned for the final leaderboard and rankings reveal!", "votes": null }, { "id": "466806", "postDate": "02/06/2019 00:31:55", "content": "Thanks! This was a great first experience with Kaggle competitions and I'm feeling hugely motivated to participate in others.
", "rawMarkdown": "Thanks! This was a great first experience with Kaggle competitions and I'm feeling hugely motivated to participate in others.", "votes": null }, { "id": "466872", "postDate": "02/06/2019 05:34:54", "content": "Why does it take so long to run the selected submissions?
", "rawMarkdown": "Why does it take so long to run the selected submissions?", "votes": null }, { "id": "467190", "postDate": "02/06/2019 15:59:44", "content": "will our model selected re-run and public LB refresh?
", "rawMarkdown": "will our model selected re-run and public LB refresh?", "votes": null }, { "id": "468354", "postDate": "02/08/2019 18:36:51", "content": "Important PSA: Be aware that if you delete the kernel that either of your selected submissions are based on, this will disqualify it from being run on the private test set, and therefore that kernel's resulting submission will no longer be eligible for stage 2 scoring or leaderboard ranking.
", "rawMarkdown": "Important PSA: Be aware that if you delete the kernel that either of your selected submissions are based on, this will disqualify it from being run on the private test set, and therefore that kernel's resulting submission will no longer be eligible for stage 2 scoring or leaderboard ranking.", "votes": null }, { "id": "470682", "postDate": "02/13/2019 11:22:24", "content": "What docker-image-version is used during evaluation-phase? \nIt seems like the R-docker-image got changed lately. (click me)
", "rawMarkdown": "What docker-image-version is used during evaluation-phase? \nIt seems like the R-docker-image got changed lately. (click me)", "votes": null }, { "id": "471501", "postDate": "02/14/2019 14:52:42", "content": "Second round can also be a bit slower - since test set is several times larger than 1st rounds test set: 376k vs 56k. Depending what one does, that might increase running time quite a bit.
\n\nEven in simple case: running n epochs, each epoch can take lot longer.
", "rawMarkdown": "Second round can also be a bit slower - since test set is several times larger than 1st rounds test set: 376k vs 56k. Depending what one does, that might increase running time quite a bit. \n\nEven in simple case: running n epochs, each epoch can take lot longer.", "votes": null } ], "comments": [ { "id": 452666, "author_name": "artgor", "author_url": "", "post_date": "01/09/2019 02:37:12", "content": "Thank you for this information!
\n\nMy only concern is the third point: yesterday-today we (kagglers) experienced a lot of problems with kernel execution due two kernel-only competitions ending. So it could be possible that some kernels won't run successfully due to lack of resourses of Kaggle...
", "votes": null, "replies": [] }, { "id": 452714, "author_name": "shaz13", "author_url": "", "post_date": "01/09/2019 04:26:16", "content": "Thanks for summarizing this @juliaelliott;
\n\nHere are some extra cautions I suggest who are new to kernels' competition
\n\nAvoid hard coded variables and shapes in your kernel
\n\nLog your runtime. Don't rely on Kaggle runtime numbers
\n\nKeep your eyes open on any kernel/docker image changes
\n\nPeace.
", "votes": null, "replies": [ { "id": 452896, "author_name": "hamishdickson", "author_url": "", "post_date": "01/09/2019 10:26:39", "content": "thanks - tip 2 is a really important point and something I hadn't thought of
", "votes": null, "replies": [] }, { "id": 452987, "author_name": "manuelsh", "author_url": "", "post_date": "01/09/2019 12:59:41", "content": "\"if you are close to 6999 seconds\" you mean to 7200 seconds (3600s x 2h)
", "votes": null, "replies": [] }, { "id": 453022, "author_name": "canming", "author_url": "", "post_date": "01/09/2019 14:29:08", "content": "How to see the kernel/docker image changes?
", "votes": null, "replies": [] }, { "id": 453032, "author_name": "shaz13", "author_url": "", "post_date": "01/09/2019 14:51:59", "content": "You can see the latest discussions here - https://www.kaggle.com/discussion\nFor more specific information you can hover to Github public docker image - https://github.com/Kaggle/docker-python
", "votes": null, "replies": [] }, { "id": 457061, "author_name": "konohayui", "author_url": "", "post_date": "01/16/2019 22:28:40", "content": "hi, I didn't see any information about final private train data.
", "votes": null, "replies": [] } ] }, { "id": 453069, "author_name": "kagsen", "author_url": "", "post_date": "01/09/2019 16:14:27", "content": "\"Kernels which error for any reason or exceed the stated restrictions when run on the private test set will be discarded without exception.\"
\n\nIn my experience, there is already up to 10% (or more) variability in kernel run-times for identical models, which is a lot. Considering that we are trying to squeeze in as much as we can within the 2 hour limit, this adds another layer of unnecessary randomness. In competitions like this, randomness is unfairness. Wouldn't it be fairer (and simpler) if any submission that clears the first stage (i.e. completes under 7200s) be automatically be considered as valid for the second round even if it fails to make the 7200s limit?
", "votes": null, "replies": [ { "id": 453191, "author_name": "manuelsh", "author_url": "", "post_date": "01/09/2019 20:40:26", "content": "Fair point. I think it makes sense.
", "votes": null, "replies": [] }, { "id": 455098, "author_name": "kostyaatarik", "author_url": "", "post_date": "01/13/2019 01:24:24", "content": "With your approach one can check the length of the test.csv in his kernel and run a different model for the second stage
", "votes": null, "replies": [] }, { "id": 455239, "author_name": "kagsen", "author_url": "", "post_date": "01/13/2019 11:12:51", "content": "Yeah, I suppose this can be gamed. No way around it, I guess
", "votes": null, "replies": [] }, { "id": 471501, "author_name": "jannen", "author_url": "", "post_date": "02/14/2019 14:52:42", "content": "Second round can also be a bit slower - since test set is several times larger than 1st rounds test set: 376k vs 56k. Depending what one does, that might increase running time quite a bit.
\n\nEven in simple case: running n epochs, each epoch can take lot longer.
", "votes": null, "replies": [] } ] }, { "id": 453314, "author_name": "salonsai", "author_url": "", "post_date": "01/10/2019 02:08:05", "content": "What is the stage 2 public time ?
", "votes": null, "replies": [] }, { "id": 455144, "author_name": "", "author_url": "", "post_date": "01/13/2019 04:17:21", "content": "I think it is really important to measure the run time during prediction on stage 1 test set. Make an assumption that the run time of previous stage 1 test set is multiply to 6.7 during stage 2 and add 8-10% buffer time for stage 2. We never know the complexity of stage 2 test set.
", "votes": null, "replies": [] }, { "id": 463614, "author_name": "wispzero", "author_url": "", "post_date": "01/30/2019 09:41:25", "content": "@juliaelliott Can you please make a clarification about hard-coded dictionaries and lists of values in kernels? Are they allowed or not? That is very imortant to us. Thanks
", "votes": null, "replies": [ { "id": 463723, "author_name": "", "author_url": "", "post_date": "01/30/2019 13:35:06", "content": "@juliaelliott: Please respond! :-)
", "votes": null, "replies": [] } ] }, { "id": 464425, "author_name": "jtrotman", "author_url": "", "post_date": "01/31/2019 20:58:08", "content": "Will we be able to see the output files and logs generated when our kernels are run for stage 2? I hope so...
\n\n(This was not the case in the Mercari competition: you can only see logs/submissions for public LB runs, with just a score for the private LB... If a kernel fails to run in stage 2 — as happened to 1/3 of the teams in Mercari — it would be really nice to see the actual logs/outputs to quickly find out why, to learn from it.)
", "votes": null, "replies": [] }, { "id": 464501, "author_name": "lsjsj92", "author_url": "", "post_date": "02/01/2019 01:38:19", "content": "thanks!
", "votes": null, "replies": [] }, { "id": 464550, "author_name": "niuddd", "author_url": "", "post_date": "02/01/2019 04:22:08", "content": "Hi, where can we find submission daily limit?
", "votes": null, "replies": [ { "id": 465872, "author_name": "bnunez", "author_url": "", "post_date": "02/04/2019 08:07:38", "content": "In the Rules: You may submit a maximum of 5 entries per day.
", "votes": null, "replies": [] } ] }, { "id": 464782, "author_name": "wochidadonggua", "author_url": "", "post_date": "02/01/2019 13:59:28", "content": "Hi, I ran the same kernel 5 times, each time is different. And the runtime in my log is also different from the one displayed by the kernel. How does the rescore phase determine whether it is timed out?
", "votes": null, "replies": [] }, { "id": 466186, "author_name": "keitakurita", "author_url": "", "post_date": "02/04/2019 20:34:46", "content": "Hi,
\n\nWould it be possible to run each kernel several (e.g. 3) times in case they fail/timeover? As far as I know, when we benchmark the speed of programs we tend to take the minimum runtime out of all the measurements since slower speeds are likely to be side effects of other processes in the system. In the same vein, slow kernels might just be caused by random fluctuations like being batched with heavier kernels. So it would seem fairer to try several runs.\nIf you don't want to reward people pushing the limits, you could run all kernels multiple times and take the best submission out of the runs, so people who timeover in any of those runs lose a chance to get a better submission. Considering how volatile the scores on the leaderboard seem to be even with the same kernel, I think many of us would appreciate it.
\n\nHoping the best for this competition!
", "votes": null, "replies": [] }, { "id": 466242, "author_name": "jtrotman", "author_url": "", "post_date": "02/04/2019 23:57:35", "content": "\n\n\nKernels which error for any reason or exceed the stated restrictions when run on the private test set will be discarded without exception.
\n
I was ok with this until a few hours ago...
\n\nI've run my solution over 40 times without problems, but when I ran it this evening, for the first time there was a CUDA error in importing keras:
\n\n2019-02-04 20:33:47.437361: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error\n2019-02-04 20:33:47.437459: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] retrieving CUDA diagnostic information for host: ddaf5ed64f42\n2019-02-04 20:33:47.437478: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:170] hostname: ddaf5ed64f42\n2019-02-04 20:33:47.437647: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:194] libcuda reported version is: 396.44.0\n2019-02-04 20:33:47.437696: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:198] kernel reported version is: 396.44.0\n2019-02-04 20:33:47.437724: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:305] kernel version seems to match DSO: 396.44.0\nThen every call to model.fit() failed (errors attached).
\n\nRunning the exact same script again, it succeeded.
\n\nI'm sure this error is rare, but with over 4000 teams, hence over 8000 submissions to generate, it becomes likely to happen to some of them.
\n\nI hope Kaggle can see sense here and do some simple statistical analysis of failed submissions, even just a quick grep of the output logs for the above (or similar, known but random) errors?
\n\n(This problem is worse if we cannot see logs/output of our stage 2 runs - @juliaelliott could you please respond to my earlier question, even just a yes/no answer? Will we get to see logs?)
", "votes": null, "replies": [] }, { "id": 466703, "author_name": "sedrak", "author_url": "", "post_date": "02/05/2019 20:55:40", "content": "Hi it was interesting!
", "votes": null, "replies": [] }, { "id": 466777, "author_name": "juliaelliott", "author_url": "", "post_date": "02/05/2019 23:01:17", "content": "FYI: We've updated the timeline, per my original forum post, to 3 weeks out. Submissions will continue to be accepted until February 5, 2019 at 11:59pm UTC.
\n\nDuring the 3 weeks remaining, we will be running your selected submissions against the private test set, but no submissions will be accepted. Stay tuned for the final leaderboard and rankings reveal!
", "votes": null, "replies": [ { "id": 466872, "author_name": "radustoicescu", "author_url": "", "post_date": "02/06/2019 05:34:54", "content": "Why does it take so long to run the selected submissions?
", "votes": null, "replies": [] } ] }, { "id": 466806, "author_name": "rtroper4", "author_url": "", "post_date": "02/06/2019 00:31:55", "content": "Thanks! This was a great first experience with Kaggle competitions and I'm feeling hugely motivated to participate in others.
", "votes": null, "replies": [] }, { "id": 467190, "author_name": "supportvectordevin", "author_url": "", "post_date": "02/06/2019 15:59:44", "content": "will our model selected re-run and public LB refresh?
", "votes": null, "replies": [] }, { "id": 468354, "author_name": "juliaelliott", "author_url": "", "post_date": "02/08/2019 18:36:51", "content": "Important PSA: Be aware that if you delete the kernel that either of your selected submissions are based on, this will disqualify it from being run on the private test set, and therefore that kernel's resulting submission will no longer be eligible for stage 2 scoring or leaderboard ranking.
", "votes": null, "replies": [] }, { "id": 470682, "author_name": "springmanndaniel", "author_url": "", "post_date": "02/13/2019 11:22:24", "content": "What docker-image-version is used during evaluation-phase? \nIt seems like the R-docker-image got changed lately. (click me)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "452545": "This is a Kernels-Only (Code) Competition. As such, the competition deadline and stage 2 close procedure will differ from our standard competitions. Here is a summary of the notable differences. Subscribe to the competition's forum to stay up to date on the latest important updates.\n\n* **February 5 (11:59PM UTC) is the final deadline** for teams to both: (A) make any new kernel submissions & (B) select your 2 final submissions.\n* You **must manually select your final 2 submissions**, by navigating to [\"My Submissions\"](https://www.kaggle.com/c/quora-insincere-questions-classification/submissions) and choosing the “Use for Final Score” checkboxes corresponding with the selected submissions. These submissions' corresponding kernels will be the only models run on the unseen, held out private test set. It is your best score on this test set that will determine your final score and ranking in the competition. Unlike other competitions, our system will not auto-select submissions for you, and those who do not make selections will no longer be reflected as a participant on the final leaderboard.\n* **Kernels which error for any reason or exceed the stated restrictions when run on the private test set will be discarded without exception**. It is your responsibility to ensure your model is coded with this in mind. It is recommended that, due to runtime variability and private test set composition, you give your runtime adequate buffer to anticipate this. Please refer to the [Data page](https://www.kaggle.com/c/quora-insincere-questions-classification/data) for details on the stage 2 test set composition. Also refer to the [Kernels FAQ Page](https://www.kaggle.com/c/quora-insincere-questions-classification#Kernels-FAQ) for details of the quotas available to you in this competition, which are equivalent to the constraints applied during stage 2.\n* At the submission deadline (February 5th, 11:59PM UTC), submissions will be disabled and the competition timeline will be extended by 3 weeks, during which time Kernels will be rescored. The rescore process re-runs your selected kernels in bulk, behind the scenes, against the private test set. We will provide an update when this process is complete and the private leaderboard is final.\n\nThis is among the first competitions being run in this unique format, so we ask for your patience during this time as we fine tune the process and engineering required to make more of these competitions a reality for our community.", "452666": "Thank you for this information!\n\nMy only concern is the third point: yesterday-today we (kagglers) experienced a lot of problems with kernel execution due two kernel-only competitions ending. So it could be possible that some kernels won't run successfully due to lack of resourses of Kaggle...", "452714": "Thanks for summarizing this @juliaelliott;\n\nHere are some extra cautions I suggest who are new to kernels' competition\n\n1. **Avoid hard coded variables and shapes in your kernel**\n - The final private train and test although will have same column names but different set of rows and size of dataset. Make sure you don't set any hard changes w.r.t to length of the dataframe or any shape in your neural network\n\n2. **Log your runtime. Don't rely on Kaggle runtime numbers**\n - We see inconsistency in scores and kernel runtime depending on surge of usage. It is wise to log your own runtime and compare its scalability with huge private set. In other words if you are close to 6999 seconds. You might want to see things tuned there!\n\n3. **Keep your eyes open on any kernel/docker image changes**\n- Lately few on the discussion have reported huge runtimes with kernel updates. Make sure you make version checks on libraries you are using to avoid last minute confusions. \n\nPeace.", "452896": "thanks - tip 2 is a really important point and something I hadn't thought of", "452987": "\"if you are close to 6999 seconds\" you mean to 7200 seconds (3600s x 2h)", "453022": "How to see the kernel/docker image changes?", "453032": "You can see the latest discussions here - https://www.kaggle.com/discussion\nFor more specific information you can hover to Github public docker image - https://github.com/Kaggle/docker-python", "453069": "\"Kernels which error for any reason or exceed the stated restrictions when run on the private test set will be discarded without exception.\"\n\n\nIn my experience, there is already up to 10% (or more) variability in kernel run-times for identical models, which is a lot. Considering that we are trying to squeeze in as much as we can within the 2 hour limit, this adds another layer of unnecessary randomness. In competitions like this, randomness is unfairness. Wouldn't it be fairer (and simpler) if any submission that clears the first stage (i.e. completes under 7200s) be automatically be considered as valid for the second round even if it fails to make the 7200s limit?", "453191": "Fair point. I think it makes sense.", "453314": "What is the stage 2 public time ?", "455098": "With your approach one can check the length of the test.csv in his kernel and run a different model for the second stage", "455144": "I think it is really important to measure the run time during prediction on stage 1 test set. Make an assumption that the run time of previous stage 1 test set is multiply to 6.7 during stage 2 and add 8-10% buffer time for stage 2. We never know the complexity of stage 2 test set.", "455239": "Yeah, I suppose this can be gamed. No way around it, I guess", "457061": "hi, I didn't see any information about final private train data.", "463614": "juliaelliott Can you please make a clarification about hard-coded dictionaries and lists of values in kernels? Are they allowed or not? That is very imortant to us. Thanks", "463723": "juliaelliott: Please respond! :-)", "464425": "Will we be able to see the output files and logs generated when our kernels are run for stage 2? I hope so...\n\n(This was not the case in the [Mercari][1] competition: you can only see logs/submissions for public LB runs, with just a score for the private LB... If a kernel fails to run in stage 2 — as happened to 1/3 of the teams in Mercari — it would be really nice to see the actual logs/outputs to quickly find out why, to learn from it.)\n\n [1]: https://www.kaggle.com/c/mercari-price-suggestion-challenge", "464501": "thanks!", "464550": "Hi, where can we find submission daily limit?", "464782": "Hi, I ran the same kernel 5 times, each time is different. And the runtime in my log is also different from the one displayed by the kernel. How does the rescore phase determine whether it is timed out?", "465872": "In the Rules: You may submit a maximum of 5 entries per day.", "466186": "Hi, \n\nWould it be possible to run each kernel several (e.g. 3) times in case they fail/timeover? As far as I know, when we benchmark the speed of programs we tend to take the minimum runtime out of all the measurements since slower speeds are likely to be side effects of other processes in the system. In the same vein, slow kernels might just be caused by random fluctuations like being batched with heavier kernels. So it would seem fairer to try several runs.\nIf you don't want to reward people pushing the limits, you could run all kernels multiple times and take the best submission out of the runs, so people who timeover in any of those runs lose a chance to get a better submission. Considering how volatile the scores on the leaderboard seem to be even with the same kernel, I think many of us would appreciate it.\n\nHoping the best for this competition!", "466242": "> Kernels which error for **any reason** or exceed the stated restrictions when run on the private test set will be discarded without exception.\n\nI was ok with this until a few hours ago...\n\nI've run my solution over 40 times without problems, but when I ran it this evening, for the first time there was a CUDA error in importing keras:\n\n\n 2019-02-04 20:33:47.437361: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error\n 2019-02-04 20:33:47.437459: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] retrieving CUDA diagnostic information for host: ddaf5ed64f42\n 2019-02-04 20:33:47.437478: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:170] hostname: ddaf5ed64f42\n 2019-02-04 20:33:47.437647: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:194] libcuda reported version is: 396.44.0\n 2019-02-04 20:33:47.437696: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:198] kernel reported version is: 396.44.0\n 2019-02-04 20:33:47.437724: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:305] kernel version seems to match DSO: 396.44.0\n\n\nThen every call to model.fit() failed (errors attached).\n\nRunning the ***exact same script*** again, it succeeded.\n\nI'm sure this error is rare, but with over 4000 teams, hence over 8000 submissions to generate, it becomes likely to happen to some of them.\n\nI hope Kaggle can see sense here and do some simple statistical analysis of failed submissions, even just a quick grep of the output logs for the above (or similar, known but random) errors?\n\n(This problem is worse if we cannot see logs/output of our stage 2 runs - @juliaelliott could you please respond to my earlier question, even just a yes/no answer? Will we get to see logs?)", "466703": "Hi it was interesting!", "466777": "FYI: We've updated the timeline, per my original forum post, to 3 weeks out. Submissions will continue to be accepted until February 5, 2019 at 11:59pm UTC. \n\nDuring the 3 weeks remaining, we will be running your selected submissions against the private test set, but no submissions will be accepted. Stay tuned for the final leaderboard and rankings reveal!", "466806": "Thanks! This was a great first experience with Kaggle competitions and I'm feeling hugely motivated to participate in others.", "466872": "Why does it take so long to run the selected submissions?", "467190": "will our model selected re-run and public LB refresh?", "468354": "Important PSA: Be aware that if you delete the kernel that either of your selected submissions are based on, this will disqualify it from being run on the private test set, and therefore that kernel's resulting submission will no longer be eligible for stage 2 scoring or leaderboard ranking.", "470682": "What docker-image-version is used during evaluation-phase? \nIt seems like the R-docker-image got changed lately. (click me)", "471501": "Second round can also be a bit slower - since test set is several times larger than 1st rounds test set: 376k vs 56k. Depending what one does, that might increase running time quite a bit. \n\nEven in simple case: running n epochs, each epoch can take lot longer." }, "source": "meta" }