This competition allows \"offline model training.\" Is there anything preventing a cynical competitor from going through 692 test questions and answering them manually, then uploading the results to a kernel and submitting them?
", "messages": [ { "id": "673333", "postDate": "11/14/2019 21:07:31", "content": "This competition allows \"offline model training.\" Is there anything preventing a cynical competitor from going through 692 test questions and answering them manually, then uploading the results to a kernel and submitting them?
", "rawMarkdown": "This competition allows \"offline model training.\" Is there anything preventing a cynical competitor from going through 692 test questions and answering them manually, then uploading the results to a kernel and submitting them?", "votes": null }, { "id": "673359", "postDate": "11/14/2019 22:17:20", "content": "The leaderboard is calculated from a hidden test set, so you'll get a submission error if you try. At least by the end you'd probably gain some random trivia knowledge :)
", "rawMarkdown": "The leaderboard is calculated from a hidden test set, so you'll get a submission error if you try. At least by the end you'd probably gain some random trivia knowledge :)", "votes": null }, { "id": "673796", "postDate": "11/15/2019 13:57:13", "content": "The test set contains 692 questions and the wikipedia articles containing the answers. What's hidden about it?
\n\nAre you saying your kernel gets scored on a different set of questions after you submit? If so, how are those different questions fed into your kernel? I thought all you had to do was fill in submission.csv with 692 predicted answers.
", "rawMarkdown": "The test set contains 692 questions and the wikipedia articles containing the answers. What's hidden about it?\n\nAre you saying your kernel gets scored on a different set of questions after you submit? If so, how are those different questions fed into your kernel? I thought all you had to do was fill in submission.csv with 692 predicted answers.", "votes": null }, { "id": "673930", "postDate": "11/15/2019 17:05:14", "content": "Yes, the kernel is run against a different test set for the LB. The filenames, etc, will be the same, so the kernel should run fine unless you hard coded iteration/example counts, but the contents of the test .jsonl file will be different.
@binnersley you believe the scoring program provides a different 'simplified-nq-test.jsonl' input file than the one we can download, and this creates the LB and PLB scores. If so, any kernel that does not read/loop through 'simplified-nq-test.jsonl' should fail with a scoring error. Is this correct?
", "rawMarkdown": "binnersley you believe the scoring program provides a different 'simplified-nq-test.jsonl' input file than the one we can download, and this creates the LB and PLB scores. If so, any kernel that does not read/loop through 'simplified-nq-test.jsonl' should fail with a scoring error. Is this correct?", "votes": null }, { "id": "674028", "postDate": "11/15/2019 20:14:12", "content": "Yes, I believe this is correct.
", "rawMarkdown": "Yes, I believe this is correct.", "votes": null }, { "id": "674359", "postDate": "11/16/2019 10:27:59", "content": "@alonbochman but that solution will fail on private test set.
", "rawMarkdown": "alonbochman but that solution will fail on private test set.", "votes": null }, { "id": "675127", "postDate": "11/17/2019 16:28:06", "content": "If you want you can get LB = 1 for public, but it isn't relevant for the private LB. \nThis is the one of the major reasons this is a kernel competition - You can't see the data related to the private LB at all.
", "rawMarkdown": "If you want you can get LB = 1 for public, but it isn't relevant for the private LB. \nThis is the one of the major reasons this is a kernel competition - You can't see the data related to the private LB at all.", "votes": null } ], "comments": [ { "id": 673359, "author_name": "binnersley", "author_url": "", "post_date": "11/14/2019 22:17:20", "content": "The leaderboard is calculated from a hidden test set, so you'll get a submission error if you try. At least by the end you'd probably gain some random trivia knowledge :)
", "votes": null, "replies": [] }, { "id": 673796, "author_name": "alonbochman", "author_url": "", "post_date": "11/15/2019 13:57:13", "content": "The test set contains 692 questions and the wikipedia articles containing the answers. What's hidden about it?
\n\nAre you saying your kernel gets scored on a different set of questions after you submit? If so, how are those different questions fed into your kernel? I thought all you had to do was fill in submission.csv with 692 predicted answers.
", "votes": null, "replies": [ { "id": 673930, "author_name": "binnersley", "author_url": "", "post_date": "11/15/2019 17:05:14", "content": "Yes, the kernel is run against a different test set for the LB. The filenames, etc, will be the same, so the kernel should run fine unless you hard coded iteration/example counts, but the contents of the test .jsonl file will be different.
@binnersley you believe the scoring program provides a different 'simplified-nq-test.jsonl' input file than the one we can download, and this creates the LB and PLB scores. If so, any kernel that does not read/loop through 'simplified-nq-test.jsonl' should fail with a scoring error. Is this correct?
", "votes": null, "replies": [] }, { "id": 674028, "author_name": "binnersley", "author_url": "", "post_date": "11/15/2019 20:14:12", "content": "Yes, I believe this is correct.
", "votes": null, "replies": [] }, { "id": 675127, "author_name": "yuval6967", "author_url": "", "post_date": "11/17/2019 16:28:06", "content": "If you want you can get LB = 1 for public, but it isn't relevant for the private LB. \nThis is the one of the major reasons this is a kernel competition - You can't see the data related to the private LB at all.
", "votes": null, "replies": [] } ] }, { "id": 674359, "author_name": "axel81", "author_url": "", "post_date": "11/16/2019 10:27:59", "content": "@alonbochman but that solution will fail on private test set.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "673333": "This competition allows \"offline model training.\" Is there anything preventing a cynical competitor from going through 692 test questions and answering them manually, then uploading the results to a kernel and submitting them?", "673359": "The leaderboard is calculated from a hidden test set, so you'll get a submission error if you try. At least by the end you'd probably gain some random trivia knowledge :)", "673796": "The test set contains 692 questions and the wikipedia articles containing the answers. What's hidden about it?\n\nAre you saying your kernel gets scored on a different set of questions after you submit? If so, how are those different questions fed into your kernel? I thought all you had to do was fill in submission.csv with 692 predicted answers.", "673930": "Yes, the kernel is run against a different test set for the LB. The filenames, etc, will be the same, so the kernel should run fine unless you hard coded iteration/example counts, but the contents of the test `.jsonl` file will be different.", "674021": "binnersley you believe the scoring program provides a different 'simplified-nq-test.jsonl' input file than the one we can download, and this creates the LB and PLB scores. If so, any kernel that does not read/loop through 'simplified-nq-test.jsonl' should fail with a scoring error. Is this correct?", "674028": "Yes, I believe this is correct.", "674359": "alonbochman but that solution will fail on private test set.", "675127": "If you want you can get LB = 1 for public, but it isn't relevant for the private LB. \nThis is the one of the major reasons this is a kernel competition - You can't see the data related to the private LB at all." }, "source": "meta" }