![\"fork-of-sub-no-code](\"https://i.postimg.cc/cLbNKPgR/What.png\")
\n
![\"\"](\"https://i.postimg.cc/26pRGvx3/Seriously.png\")
I have found the following two notebooks in this competition which seem to be copying rle masks from a csv file then saving it as the submission.csv. My understanding was that we have to run our models in a Kaggle Notebook without an internet connection and within the 40 hour per week GPU/TPU limit in order to prevent us from querying the public test data's correct answers. If we are allowed to submit csv files and get them scored, then what's to stop someone from submitting manually created submissions and finding the true masks for the test set?
\nI'm aware the public score isn't the same as the private score. However, with the correct masks for the public test data, you can fine-tune your models for better predictions. The author of fork-of-sub-no-code is currently 33rd in the LB.
@addisonhoward can you please clarify whether this is allowed?
\nNotebooks in question:
\nhttps://www.kaggle.com/graafffff/fork-of-sub-no-code/
\nhttps://www.kaggle.com/finlay/csv-write-for-fast-public-score/
\n
I have found the following two notebooks in this competition which seem to be copying rle masks from a csv file then saving it as the submission.csv. My understanding was that we have to run our models in a Kaggle Notebook without an internet connection and within the 40 hour per week GPU/TPU limit in order to prevent us from querying the public test data's correct answers. If we are allowed to submit csv files and get them scored, then what's to stop someone from submitting manually created submissions and finding the true masks for the test set?
\nI'm aware the public score isn't the same as the private score. However, with the correct masks for the public test data, you can fine-tune your models for better predictions. The author of fork-of-sub-no-code is currently 33rd in the LB.
@addisonhoward can you please clarify whether this is allowed?
\nNotebooks in question:
\nhttps://www.kaggle.com/graafffff/fork-of-sub-no-code/
\nhttps://www.kaggle.com/finlay/csv-write-for-fast-public-score/
\n
\n\nHowever, with the correct masks for the public test data, you can fine-tune your models for better predictions.
\n
This sounds a very reasonable thing to try to me!
", "rawMarkdown": "> However, with the correct masks for the public test data, you can fine-tune your models for better predictions. \n\nThis sounds a very reasonable thing to try to me!", "votes": null }, { "id": "1292393", "postDate": "05/03/2021 22:45:51", "content": "You’re reading that out of context, or maybe my writing isn’t clear. The problem is how the true masks are obtained. If we are allowed to submit any csv including ones not generated by a model in a Kaggle notebook, then that’s a problem.
", "rawMarkdown": "You’re reading that out of context, or maybe my writing isn’t clear. The problem is how the true masks are obtained. If we are allowed to submit any csv including ones not generated by a model in a Kaggle notebook, then that’s a problem.", "votes": null }, { "id": "1292567", "postDate": "05/04/2021 05:28:22", "content": "For this competition (unlike others) pretty much anything goes wrt public test. Hand Labeling of public test and not required to be considered external data or made public.see post Although at least one public test was, which presumably some could use instead of predictions to increase public LB scores.
\nWhether or not this translates into anything useful in private test remains to be seen. The new private test is supposed to be unseen by anyone, not available anywhere even on the Portal, although the methodology for annotations are meant to be the same for train, public and private test per host. How fibrous crescent glomeruli get handled could be deal breakers. Questions remain over labels, noise. So we'll see.
\nBut any submissions for the Final LB that are only using csvs and not doing inference on the actual private test will not have good results. It can perhaps be used as a way to not consume much GPU time when generating a submission file, so inference only runs on true private test with submit. Same as checking length of sample submission and bypass inference just output the sample.
", "rawMarkdown": "For this competition (unlike others) pretty much anything goes wrt public test. Hand Labeling of public test and not required to be considered external data or made public.[see post](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616#1254955) Although at least one public test was, which presumably some could use instead of predictions to increase public LB scores.\n\nWhether or not this translates into anything useful in private test remains to be seen. The new private test is supposed to be unseen by anyone, not available anywhere even on the Portal, although the methodology for annotations are meant to be the same for train, public and private test [per host](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/234083#1281244). How fibrous crescent glomeruli get handled could be deal breakers. Questions remain over labels, noise. So we'll see. \n\nBut any submissions for the Final LB that are only using csvs and not doing inference on the actual private test will not have good results. It can perhaps be used as a way to not consume much GPU time when generating a submission file, so inference only runs on true private test with submit. Same as checking length of sample submission and bypass inference just output the sample.", "votes": null }, { "id": "1292731", "postDate": "05/04/2021 08:41:22", "content": "They re-submit csv just with public masks so such submission should get 0 for private test set.
", "rawMarkdown": "They re-submit csv just with public masks so such submission should get 0 for private test set.", "votes": null }, { "id": "1293682", "postDate": "05/05/2021 05:12:21", "content": "I think the purpose of this is just to see the PB score faster (use your own GPU to generate csv offline). But everyone needs to use the kernel to make online predictions at the time of final submission.
", "rawMarkdown": "I think the purpose of this is just to see the PB score faster (use your own GPU to generate csv offline). But everyone needs to use the kernel to make online predictions at the time of final submission.", "votes": null }, { "id": "1294725", "postDate": "05/05/2021 21:00:14", "content": "It is allowed in this competition, but wont be useful in the private set though. This is the main reason the leaderboard, in this competition, doenst mean much. ( Once anyone can hand label and submit)
", "rawMarkdown": "It is allowed in this competition, but wont be useful in the private set though. This is the main reason the leaderboard, in this competition, doenst mean much. ( Once anyone can hand label and submit)", "votes": null }, { "id": "1295888", "postDate": "05/06/2021 19:19:41", "content": "Kaggle Admin here. Will confirm rpsantosa_kaggle's comment above as official.
", "rawMarkdown": "Kaggle Admin here. Will confirm rpsantosa_kaggle's comment above as official.", "votes": null } ], "comments": [ { "id": 1292379, "author_name": "jamesphoward", "author_url": "", "post_date": "05/03/2021 21:56:46", "content": "\n\nHowever, with the correct masks for the public test data, you can fine-tune your models for better predictions.
\n
This sounds a very reasonable thing to try to me!
", "votes": null, "replies": [ { "id": 1292393, "author_name": "erikdali", "author_url": "", "post_date": "05/03/2021 22:45:51", "content": "You’re reading that out of context, or maybe my writing isn’t clear. The problem is how the true masks are obtained. If we are allowed to submit any csv including ones not generated by a model in a Kaggle notebook, then that’s a problem.
", "votes": null, "replies": [] }, { "id": 1294725, "author_name": "rpsantosakaggle", "author_url": "", "post_date": "05/05/2021 21:00:14", "content": "It is allowed in this competition, but wont be useful in the private set though. This is the main reason the leaderboard, in this competition, doenst mean much. ( Once anyone can hand label and submit)
", "votes": null, "replies": [] }, { "id": 1295888, "author_name": "addisonhoward", "author_url": "", "post_date": "05/06/2021 19:19:41", "content": "Kaggle Admin here. Will confirm rpsantosa_kaggle's comment above as official.
", "votes": null, "replies": [] } ] }, { "id": 1292567, "author_name": "something4kag", "author_url": "", "post_date": "05/04/2021 05:28:22", "content": "For this competition (unlike others) pretty much anything goes wrt public test. Hand Labeling of public test and not required to be considered external data or made public.see post Although at least one public test was, which presumably some could use instead of predictions to increase public LB scores.
\nWhether or not this translates into anything useful in private test remains to be seen. The new private test is supposed to be unseen by anyone, not available anywhere even on the Portal, although the methodology for annotations are meant to be the same for train, public and private test per host. How fibrous crescent glomeruli get handled could be deal breakers. Questions remain over labels, noise. So we'll see.
\nBut any submissions for the Final LB that are only using csvs and not doing inference on the actual private test will not have good results. It can perhaps be used as a way to not consume much GPU time when generating a submission file, so inference only runs on true private test with submit. Same as checking length of sample submission and bypass inference just output the sample.
", "votes": null, "replies": [] }, { "id": 1292731, "author_name": "hypocrites", "author_url": "", "post_date": "05/04/2021 08:41:22", "content": "They re-submit csv just with public masks so such submission should get 0 for private test set.
", "votes": null, "replies": [] }, { "id": 1293682, "author_name": "rongyumeng", "author_url": "", "post_date": "05/05/2021 05:12:21", "content": "I think the purpose of this is just to see the PB score faster (use your own GPU to generate csv offline). But everyone needs to use the kernel to make online predictions at the time of final submission.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1292376": "I have found the following two notebooks in this competition which seem to be copying rle masks from a csv file then saving it as the submission.csv. My understanding was that we have to run our models in a Kaggle Notebook without an internet connection and within the 40 hour per week GPU/TPU limit in order to prevent us from querying the public test data's correct answers. If we are allowed to submit csv files and get them scored, then what's to stop someone from submitting manually created submissions and finding the true masks for the test set?\n\nI'm aware the public score isn't the same as the private score. However, with the correct masks for the public test data, you can fine-tune your models for better predictions. The author of `fork-of-sub-no-code` is currently 33rd in the LB.\n\n@addisonhoward can you please clarify whether this is allowed?\n\nNotebooks in question:\nhttps://www.kaggle.com/graafffff/fork-of-sub-no-code/\nhttps://www.kaggle.com/finlay/csv-write-for-fast-public-score/\n\n\n", "1292379": "> However, with the correct masks for the public test data, you can fine-tune your models for better predictions. \n\nThis sounds a very reasonable thing to try to me!", "1292393": "You’re reading that out of context, or maybe my writing isn’t clear. The problem is how the true masks are obtained. If we are allowed to submit any csv including ones not generated by a model in a Kaggle notebook, then that’s a problem.", "1292567": "For this competition (unlike others) pretty much anything goes wrt public test. Hand Labeling of public test and not required to be considered external data or made public.[see post](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/227616#1254955) Although at least one public test was, which presumably some could use instead of predictions to increase public LB scores.\n\nWhether or not this translates into anything useful in private test remains to be seen. The new private test is supposed to be unseen by anyone, not available anywhere even on the Portal, although the methodology for annotations are meant to be the same for train, public and private test [per host](https://www.kaggle.com/c/hubmap-kidney-segmentation/discussion/234083#1281244). How fibrous crescent glomeruli get handled could be deal breakers. Questions remain over labels, noise. So we'll see. \n\nBut any submissions for the Final LB that are only using csvs and not doing inference on the actual private test will not have good results. It can perhaps be used as a way to not consume much GPU time when generating a submission file, so inference only runs on true private test with submit. Same as checking length of sample submission and bypass inference just output the sample.", "1292731": "They re-submit csv just with public masks so such submission should get 0 for private test set.", "1293682": "I think the purpose of this is just to see the PB score faster (use your own GPU to generate csv offline). But everyone needs to use the kernel to make online predictions at the time of final submission.", "1294725": "It is allowed in this competition, but wont be useful in the private set though. This is the main reason the leaderboard, in this competition, doenst mean much. ( Once anyone can hand label and submit)", "1295888": "Kaggle Admin here. Will confirm rpsantosa_kaggle's comment above as official." }, "source": "meta" }