EDIT: Everything is fine, just keep in mind that your kernel runs only once on both public + private datasets and then the score is computed only for the public part.
\n\nI came through a strange issue recently. My current pipeline is not to have full inference code in the Kaggle kernel, but rather to train & predict locally and load the submission file through a private dataset.
\n\nIn order to handle the private dataset case, I check how many lines are in a sample_submission.csv file, and if the number matches the number found in my submission file, I submit my submission, otherwise I submit the sample submission. Theoretically, this should give approximately zero score for the private dataset, and some non-zero (assuming that my model does its job) score for the public dataset.
However, my kernel gives me EXACTLY 0 score. The resulting submission I see after doing a commit looks exactly as a submission I've generated locally. My submission has a quite significant overlap with other public submission files. But the score is ZERO.
\n\nThe only explanation I see here is that we actually have an additional public test set that is hidden from us, but in fact, used internally to compute the leaderboard score, so the real public test set (346 examples) is being simply ignored.
\n\nI'd appreciate it if anyone could shed some light on this problem. @philculliton could you provide some feedback on this?
\n\nMy problem seems to overlap with this topic, but the topic doesn't seem to provide an answer.
", "messages": [ { "id": 696954, "postDate": "2019-12-17T09:36:28.583Z", "content": "EDIT: Everything is fine, just keep in mind that your kernel runs only once on both public + private datasets and then the score is computed only for the public part.
\n\nI came through a strange issue recently. My current pipeline is not to have full inference code in the Kaggle kernel, but rather to train & predict locally and load the submission file through a private dataset.
\n\nIn order to handle the private dataset case, I check how many lines are in a sample_submission.csv file, and if the number matches the number found in my submission file, I submit my submission, otherwise I submit the sample submission. Theoretically, this should give approximately zero score for the private dataset, and some non-zero (assuming that my model does its job) score for the public dataset.
However, my kernel gives me EXACTLY 0 score. The resulting submission I see after doing a commit looks exactly as a submission I've generated locally. My submission has a quite significant overlap with other public submission files. But the score is ZERO.
\n\nThe only explanation I see here is that we actually have an additional public test set that is hidden from us, but in fact, used internally to compute the leaderboard score, so the real public test set (346 examples) is being simply ignored.
\n\nI'd appreciate it if anyone could shed some light on this problem. @philculliton could you provide some feedback on this?
\n\nMy problem seems to overlap with this topic, but the topic doesn't seem to provide an answer.
", "rawMarkdown": "**EDIT**: Everything is fine, just keep in mind that your kernel runs only once on both public + private datasets and then the score is computed only for the public part.\n\n\n------\n\n\nI came through a strange issue recently. My current pipeline is not to have full inference code in the Kaggle kernel, but rather to train & predict locally and load the submission file through a private dataset. \n\nIn order to handle the private dataset case, I check how many lines are in a `sample_submission.csv` file, and if the number matches the number found in my submission file, I submit my submission, otherwise I submit the sample submission. Theoretically, this should give approximately zero score for the private dataset, and some non-zero (assuming that my model does its job) score for the public dataset.\n\nHowever, my kernel gives me EXACTLY 0 score. The resulting submission I see after doing a commit looks exactly as a submission I've generated locally. My submission has a quite significant overlap with other public submission files. But the score is ZERO. \n\nThe only explanation I see here is that we actually have an additional public test set that is hidden from us, but in fact, used internally to compute the leaderboard score, so the real public test set (346 examples) is being simply ignored.\n\nI'd appreciate it if anyone could shed some light on this problem. @philculliton could you provide some feedback on this?\n\nMy problem seems to overlap with [this topic](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/119280), but the topic doesn't seem to provide an answer.\n", "votes": 10 }, { "id": 697026, "postDate": "2019-12-17T11:39:55.303Z", "content": "Indeed, it's not only the metric which is obscure but the submission process as well.
\n\nAs Phil helped me a bit with debugging, I'll clarify on the issue that I encountered (though it's mentioned in the same post).
\n\nI used a similar hack to save time (not working for me anymore, Notebook Threw Exception). The problem with my code was that it worked fine with public test data (346-long) but failed on the private part (which is from 3k to 3500-long), and I actually figure out why.
Turned out that after an exception thrown for the private test part, the submission \"backed off\" to sample submission, therefore Zeros. When I executed the buggy code without hacks saving time, I got Submission CSV Not Found.
Today, I also got Submission CSV Not Found. And when I just sent all blank answer (without any prediction from model), I got Notebook Threw Exception. Commit is working, but not submission.
Are you doing like this? If so, your submission always use sample submission.
\n\n\nif len(submission_df) == 346 * 2:\n # submit your submission.csv\nelse:\n # submit sample_submission.csv\n
I tried same process and found that the if len(submission_df) == 346 * 2 statement is satified only in COMMIT and not in SUBMISSION.\nI guess the submission process include both public and private test set, then compute public LB by using public test set only.
You are right. The proper way of doing this is to fill the sample submission file with the predictions made locally, not to replace the file.
", "rawMarkdown": "You are right. The proper way of doing this is to fill the sample submission file with the predictions made locally, not to replace the file.", "votes": 1 }, { "id": 697055, "postDate": "2019-12-17T12:23:10.433Z", "content": "Yes, at SUBMISSION stage len(submission_df) is now about 7k-long so you always submit a sample submission file.
Indeed, it's not only the metric which is obscure but the submission process as well.
\n\nAs Phil helped me a bit with debugging, I'll clarify on the issue that I encountered (though it's mentioned in the same post).
\n\nI used a similar hack to save time (not working for me anymore, Notebook Threw Exception). The problem with my code was that it worked fine with public test data (346-long) but failed on the private part (which is from 3k to 3500-long), and I actually figure out why.
Turned out that after an exception thrown for the private test part, the submission \"backed off\" to sample submission, therefore Zeros. When I executed the buggy code without hacks saving time, I got Submission CSV Not Found.
Today, I also got Submission CSV Not Found. And when I just sent all blank answer (without any prediction from model), I got Notebook Threw Exception. Commit is working, but not submission.
Are you doing like this? If so, your submission always use sample submission.
\n\n\nif len(submission_df) == 346 * 2:\n # submit your submission.csv\nelse:\n # submit sample_submission.csv\n
I tried same process and found that the if len(submission_df) == 346 * 2 statement is satified only in COMMIT and not in SUBMISSION.\nI guess the submission process include both public and private test set, then compute public LB by using public test set only.
You are right. The proper way of doing this is to fill the sample submission file with the predictions made locally, not to replace the file.
", "votes": 1, "replies": [] }, { "id": 697055, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-12-17T12:23:10.433000", "content": "Yes, at SUBMISSION stage len(submission_df) is now about 7k-long so you always submit a sample submission file.