How do I know for sure that my code will be able to run fast enough when the competition ends? When we submit it to the public leaderboard does it compute all the 5k test images but only grade based on 18%?
", "messages": [ { "id": "1292457", "postDate": "05/04/2021 01:15:46", "content": "How do I know for sure that my code will be able to run fast enough when the competition ends? When we submit it to the public leaderboard does it compute all the 5k test images but only grade based on 18%?
", "rawMarkdown": "How do I know for sure that my code will be able to run fast enough when the competition ends? When we submit it to the public leaderboard does it compute all the 5k test images but only grade based on 18%?", "votes": null }, { "id": "1292957", "postDate": "05/04/2021 12:52:49", "content": "Your submission is run on the full private test dataset right after you submit. There's no additional runs after end of a competition. If your submission was scored successfully on public LB, it will be scored successfully on private LB.
", "rawMarkdown": "Your submission is run on the full private test dataset right after you submit. There's no additional runs after end of a competition. If your submission was scored successfully on public LB, it will be scored successfully on private LB.", "votes": null }, { "id": "1293003", "postDate": "05/04/2021 13:24:21", "content": "Much appreciated @atamazian this is my first competition so I am super green! =)
", "rawMarkdown": "Much appreciated @atamazian this is my first competition so I am super green! =)", "votes": null }, { "id": "1293037", "postDate": "05/04/2021 13:50:19", "content": "Is that true? I'm also really concerned about the time limit, since I plan to ensemble multiple trained models.
", "rawMarkdown": "Is that true? I'm also really concerned about the time limit, since I plan to ensemble multiple trained models.", "votes": null }, { "id": "1293059", "postDate": "05/04/2021 14:08:22", "content": "I think it is. First your kernel is run on all private test dataset, and then resulting csv is splitted according to public/private ratio. Both parts are ranked - first for the public LB, and second for the private LB. Private LB is hidden until a competition end.
", "rawMarkdown": "I think it is. First your kernel is run on all private test dataset, and then resulting csv is splitted according to public/private ratio. Both parts are ranked - first for the public LB, and second for the private LB. Private LB is hidden until a competition end.", "votes": null }, { "id": "1293060", "postDate": "05/04/2021 14:09:26", "content": "As for multiple models, I also use many models. I don't think it's possible to get good result here with only only one model, even if it's 5-folded.
", "rawMarkdown": "As for multiple models, I also use many models. I don't think it's possible to get good result here with only only one model, even if it's 5-folded.", "votes": null }, { "id": "1293088", "postDate": "05/04/2021 14:45:06", "content": "Thanks for your patient reply, I now understand the scoring mechanism.
", "rawMarkdown": "Thanks for your patient reply, I now understand the scoring mechanism.", "votes": null }, { "id": "1293098", "postDate": "05/04/2021 14:52:47", "content": "Here's the official page regarding that subject https://www.kaggle.com/code-competition-debugging
", "rawMarkdown": "Here's the official page regarding that subject https://www.kaggle.com/code-competition-debugging", "votes": null }, { "id": "1295523", "postDate": "05/06/2021 14:16:46", "content": "Thank you @atamazian I had the exact same question!
", "rawMarkdown": "Thank you @atamazian I had the exact same question!", "votes": null }, { "id": "1298134", "postDate": "05/08/2021 15:19:51", "content": "I can second that.
\nIn the Cassava Lead Disease I had a model that took between 8-9 hours to submit. The competition specified that the notebook time had to be <9 hours.
\nI was worried as the public leaderboard was being tested on 69% of the data, and there was no way that my notebook would stay <9 hours after being ran on the hidden test set.
\nNevertheless, I selected this model to be used for final scoring to test the limits and it all worked fine.
", "rawMarkdown": "I can second that. \n\nIn the Cassava Lead Disease I had a model that took between 8-9 hours to submit. The competition specified that the notebook time had to be <9 hours.\n\n I was worried as the public leaderboard was being tested on 69% of the data, and there was no way that my notebook would stay <9 hours after being ran on the hidden test set.\n\nNevertheless, I selected this model to be used for final scoring to test the limits and it all worked fine.", "votes": null }, { "id": "1306806", "postDate": "05/14/2021 05:24:02", "content": "Now I have some doubts about the scoring. Actually, the scoring issue just happended in another competition: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238075, where submissions with correct public LB scores failed in the private LB, and got zero.
", "rawMarkdown": "Now I have some doubts about the scoring. Actually, the scoring issue just happended in another competition: [https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238075](url), where submissions with correct public LB scores failed in the private LB, and got zero.", "votes": null }, { "id": "1306817", "postDate": "05/14/2021 05:30:29", "content": "Moreover, I don't see any evidence of running the whole test set while submitting in this meterial you provided: https://www.kaggle.com/code-competition-debugging. So is there any official statement on how the scoring system works? Running on the whole test set or part of them?
", "rawMarkdown": "Moreover, I don't see any evidence of running the whole test set while submitting in this meterial you provided: [https://www.kaggle.com/code-competition-debugging](url). So is there any official statement on how the scoring system works? Running on the whole test set or part of them?", "votes": null }, { "id": "1307740", "postDate": "05/14/2021 16:29:34", "content": "@ytepzhi the default for code competitions is for every submission to be scored against both the public and private test sets. The only exceptions I can think of off hand would be competitions where the data has to be updated at some point after launch. For example, the Jane Street Market Prediction was set up with a private test set to be gathered after all notebooks were submitted. However, this is an unusual and clearly labeled arrangement.
", "rawMarkdown": "ytepzhi the default for code competitions is for every submission to be scored against both the public and private test sets. The only exceptions I can think of off hand would be competitions where the data has to be updated at some point after launch. For example, the [Jane Street Market Prediction](https://www.kaggle.com/c/jane-street-market-prediction/overview/timeline) was set up with a private test set to be gathered after all notebooks were submitted. However, this is an unusual and clearly labeled arrangement.", "votes": null }, { "id": "1307751", "postDate": "05/14/2021 16:37:04", "content": "Thanks for your generous reply.
", "rawMarkdown": "Thanks for your generous reply.", "votes": null } ], "comments": [ { "id": 1292957, "author_name": "atamazian", "author_url": "", "post_date": "05/04/2021 12:52:49", "content": "Your submission is run on the full private test dataset right after you submit. There's no additional runs after end of a competition. If your submission was scored successfully on public LB, it will be scored successfully on private LB.
", "votes": null, "replies": [ { "id": 1293037, "author_name": "ytepzhi", "author_url": "", "post_date": "05/04/2021 13:50:19", "content": "Is that true? I'm also really concerned about the time limit, since I plan to ensemble multiple trained models.
", "votes": null, "replies": [] }, { "id": 1293059, "author_name": "atamazian", "author_url": "", "post_date": "05/04/2021 14:08:22", "content": "I think it is. First your kernel is run on all private test dataset, and then resulting csv is splitted according to public/private ratio. Both parts are ranked - first for the public LB, and second for the private LB. Private LB is hidden until a competition end.
", "votes": null, "replies": [] }, { "id": 1293060, "author_name": "atamazian", "author_url": "", "post_date": "05/04/2021 14:09:26", "content": "As for multiple models, I also use many models. I don't think it's possible to get good result here with only only one model, even if it's 5-folded.
", "votes": null, "replies": [] }, { "id": 1293088, "author_name": "ytepzhi", "author_url": "", "post_date": "05/04/2021 14:45:06", "content": "Thanks for your patient reply, I now understand the scoring mechanism.
", "votes": null, "replies": [] }, { "id": 1293098, "author_name": "atamazian", "author_url": "", "post_date": "05/04/2021 14:52:47", "content": "Here's the official page regarding that subject https://www.kaggle.com/code-competition-debugging
", "votes": null, "replies": [] }, { "id": 1298134, "author_name": "brendanartley", "author_url": "", "post_date": "05/08/2021 15:19:51", "content": "I can second that.
\nIn the Cassava Lead Disease I had a model that took between 8-9 hours to submit. The competition specified that the notebook time had to be <9 hours.
\nI was worried as the public leaderboard was being tested on 69% of the data, and there was no way that my notebook would stay <9 hours after being ran on the hidden test set.
\nNevertheless, I selected this model to be used for final scoring to test the limits and it all worked fine.
", "votes": null, "replies": [] }, { "id": 1306806, "author_name": "ytepzhi", "author_url": "", "post_date": "05/14/2021 05:24:02", "content": "Now I have some doubts about the scoring. Actually, the scoring issue just happended in another competition: https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238075, where submissions with correct public LB scores failed in the private LB, and got zero.
", "votes": null, "replies": [] }, { "id": 1306817, "author_name": "ytepzhi", "author_url": "", "post_date": "05/14/2021 05:30:29", "content": "Moreover, I don't see any evidence of running the whole test set while submitting in this meterial you provided: https://www.kaggle.com/code-competition-debugging. So is there any official statement on how the scoring system works? Running on the whole test set or part of them?
", "votes": null, "replies": [] }, { "id": 1307740, "author_name": "sohier", "author_url": "", "post_date": "05/14/2021 16:29:34", "content": "@ytepzhi the default for code competitions is for every submission to be scored against both the public and private test sets. The only exceptions I can think of off hand would be competitions where the data has to be updated at some point after launch. For example, the Jane Street Market Prediction was set up with a private test set to be gathered after all notebooks were submitted. However, this is an unusual and clearly labeled arrangement.
", "votes": null, "replies": [] }, { "id": 1307751, "author_name": "ytepzhi", "author_url": "", "post_date": "05/14/2021 16:37:04", "content": "Thanks for your generous reply.
", "votes": null, "replies": [] } ] }, { "id": 1293003, "author_name": "coldfir3", "author_url": "", "post_date": "05/04/2021 13:24:21", "content": "Much appreciated @atamazian this is my first competition so I am super green! =)
", "votes": null, "replies": [] }, { "id": 1295523, "author_name": "mylonsong", "author_url": "", "post_date": "05/06/2021 14:16:46", "content": "Thank you @atamazian I had the exact same question!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1292457": "How do I know for sure that my code will be able to run fast enough when the competition ends? When we submit it to the public leaderboard does it compute all the 5k test images but only grade based on 18%?", "1292957": "Your submission is run on the full private test dataset right after you submit. There's no additional runs after end of a competition. If your submission was scored successfully on public LB, it will be scored successfully on private LB.", "1293003": "Much appreciated @atamazian this is my first competition so I am super green! =)", "1293037": "Is that true? I'm also really concerned about the time limit, since I plan to ensemble multiple trained models.", "1293059": "I think it is. First your kernel is run on all private test dataset, and then resulting csv is splitted according to public/private ratio. Both parts are ranked - first for the public LB, and second for the private LB. Private LB is hidden until a competition end.", "1293060": "As for multiple models, I also use many models. I don't think it's possible to get good result here with only only one model, even if it's 5-folded.", "1293088": "Thanks for your patient reply, I now understand the scoring mechanism.", "1293098": "Here's the official page regarding that subject https://www.kaggle.com/code-competition-debugging", "1295523": "Thank you @atamazian I had the exact same question!", "1298134": "I can second that. \n\nIn the Cassava Lead Disease I had a model that took between 8-9 hours to submit. The competition specified that the notebook time had to be <9 hours.\n\n I was worried as the public leaderboard was being tested on 69% of the data, and there was no way that my notebook would stay <9 hours after being ran on the hidden test set.\n\nNevertheless, I selected this model to be used for final scoring to test the limits and it all worked fine.", "1306806": "Now I have some doubts about the scoring. Actually, the scoring issue just happended in another competition: [https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/238075](url), where submissions with correct public LB scores failed in the private LB, and got zero.", "1306817": "Moreover, I don't see any evidence of running the whole test set while submitting in this meterial you provided: [https://www.kaggle.com/code-competition-debugging](url). So is there any official statement on how the scoring system works? Running on the whole test set or part of them?", "1307740": "ytepzhi the default for code competitions is for every submission to be scored against both the public and private test sets. The only exceptions I can think of off hand would be competitions where the data has to be updated at some point after launch. For example, the [Jane Street Market Prediction](https://www.kaggle.com/c/jane-street-market-prediction/overview/timeline) was set up with a private test set to be gathered after all notebooks were submitted. However, this is an unusual and clearly labeled arrangement.", "1307751": "Thanks for your generous reply." }, "source": "meta" }