{
  "id": 190791,
  "title": "will our submitted notebooks be re-run after the competition deadline?",
  "url": "/competitions/riiid-test-answer-prediction/discussion/190791",
  "author_name": "",
  "post_date": "2020-10-13T11:31:01.252769300Z",
  "votes": 7,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I have 3 questions about the submission and private test dataset.</p>\n<p>Q1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).</p>\n<p>Q2. Will our submissions be re-run after the deadline in order to compute the private LB score? Or it will just use the score for the remaining 80% private test dataset that is potentially already calculated when we submit?</p>\n<p>Q3. If the submission will be re-run after the deadline, would <code>questions.csv</code>, <code>lectures.csv</code> and the private test dataset remain the same as when we submit for the public LB score before the deadline?</p>\n<p>My notebook <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190667\" target=\"_blank\">Riiid - verifying private test dataset properties</a> that verifies the private test dataset properties could be trusted <strong>only if</strong> </p>\n<pre><code>- Q1 is True \n- the answer for Q2 is that the private LB score is already calculated, OR Q3 is True\n</code></pre>\n<p><a href=\"https://www.kaggle.com/sohier\" target=\"_blank\">@sohier</a> - Is it possible to answer these 3 questions? Thanks.</p>",
  "messages": [
    {
      "id": "1048308",
      "postDate": "10/13/2020 11:31:01",
      "content": "<p>I have 3 questions about the submission and private test dataset.</p>\n<p>Q1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).</p>\n<p>Q2. Will our submissions be re-run after the deadline in order to compute the private LB score? Or it will just use the score for the remaining 80% private test dataset that is potentially already calculated when we submit?</p>\n<p>Q3. If the submission will be re-run after the deadline, would <code>questions.csv</code>, <code>lectures.csv</code> and the private test dataset remain the same as when we submit for the public LB score before the deadline?</p>\n<p>My notebook <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190667\" target=\"_blank\">Riiid - verifying private test dataset properties</a> that verifies the private test dataset properties could be trusted <strong>only if</strong> </p>\n<pre><code>- Q1 is True \n- the answer for Q2 is that the private LB score is already calculated, OR Q3 is True\n</code></pre>\n<p><a href=\"https://www.kaggle.com/sohier\" target=\"_blank\">@sohier</a> - Is it possible to answer these 3 questions? Thanks.</p>",
      "rawMarkdown": "I have 3 questions about the submission and private test dataset.\n\nQ1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).\n\nQ2. Will our submissions be re-run after the deadline in order to compute the private LB score? Or it will just use the score for the remaining 80% private test dataset that is potentially already calculated when we submit?\n\nQ3. If the submission will be re-run after the deadline, would `questions.csv`, `lectures.csv` and the private test dataset remain the same as when we submit for the public LB score before the deadline?\n\nMy notebook [Riiid - verifying private test dataset properties](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190667) that verifies the private test dataset properties could be trusted **only if** \n\n    - Q1 is True \n    - the answer for Q2 is that the private LB score is already calculated, OR Q3 is True\n\n@sohier - Is it possible to answer these 3 questions? Thanks.",
      "votes": null
    },
    {
      "id": "1048329",
      "postDate": "10/13/2020 11:43:24",
      "content": "<p>In all kaggle comps, Public LB and Private LB score for each sub is calculated // IMO.</p>",
      "rawMarkdown": "In all kaggle comps, Public LB and Private LB score for each sub is calculated // IMO.",
      "votes": null
    },
    {
      "id": "1052861",
      "postDate": "10/18/2020 11:09:50",
      "content": "<p>I'm not so sure about your statements … In Facebook Deep Fake Detection and NFL Bowl 2020, submitted notebooks were re-run on a fresh new test set !</p>",
      "rawMarkdown": "I'm not so sure about your statements ... In Facebook Deep Fake Detection and NFL Bowl 2020, submitted notebooks were re-run on a fresh new test set !",
      "votes": null
    },
    {
      "id": "1052867",
      "postDate": "10/18/2020 11:20:01",
      "content": "<p>Well in that case, it's a different case as it's a two-tier comp.</p>",
      "rawMarkdown": "Well in that case, it's a different case as it's a two-tier comp.",
      "votes": null
    },
    {
      "id": "1055439",
      "postDate": "10/20/2020 19:38:54",
      "content": "<blockquote>\n  <p>Q1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).</p>\n</blockquote>\n<p><a href=\"https://www.kaggle.com/yihdarshieh\" target=\"_blank\">@yihdarshieh</a> See <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191696#1055225\" target=\"_blank\">https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191696#1055225</a></p>",
      "rawMarkdown": "> Q1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).\n\n@yihdarshieh See https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191696#1055225",
      "votes": null
    },
    {
      "id": "1059692",
      "postDate": "10/25/2020 11:01:53",
      "content": "<p><a href=\"https://www.kaggle.com/nyakaggle\" target=\"_blank\">@nyakaggle</a> , thanks! That's good news.</p>",
      "rawMarkdown": "nyakaggle , thanks! That's good news.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1048329,
      "author_name": "adityaecdrid",
      "author_url": "",
      "post_date": "10/13/2020 11:43:24",
      "content": "<p>In all kaggle comps, Public LB and Private LB score for each sub is calculated // IMO.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1052861,
          "author_name": "kneroma",
          "author_url": "",
          "post_date": "10/18/2020 11:09:50",
          "content": "<p>I'm not so sure about your statements … In Facebook Deep Fake Detection and NFL Bowl 2020, submitted notebooks were re-run on a fresh new test set !</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1052867,
          "author_name": "adityaecdrid",
          "author_url": "",
          "post_date": "10/18/2020 11:20:01",
          "content": "<p>Well in that case, it's a different case as it's a two-tier comp.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1055439,
      "author_name": "nyakaggle",
      "author_url": "",
      "post_date": "10/20/2020 19:38:54",
      "content": "<blockquote>\n  <p>Q1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).</p>\n</blockquote>\n<p><a href=\"https://www.kaggle.com/yihdarshieh\" target=\"_blank\">@yihdarshieh</a> See <a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191696#1055225\" target=\"_blank\">https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191696#1055225</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1059692,
          "author_name": "yihdarshieh",
          "author_url": "",
          "post_date": "10/25/2020 11:01:53",
          "content": "<p><a href=\"https://www.kaggle.com/nyakaggle\" target=\"_blank\">@nyakaggle</a> , thanks! That's good news.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1048308": "I have 3 questions about the submission and private test dataset.\n\nQ1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).\n\nQ2. Will our submissions be re-run after the deadline in order to compute the private LB score? Or it will just use the score for the remaining 80% private test dataset that is potentially already calculated when we submit?\n\nQ3. If the submission will be re-run after the deadline, would `questions.csv`, `lectures.csv` and the private test dataset remain the same as when we submit for the public LB score before the deadline?\n\nMy notebook [Riiid - verifying private test dataset properties](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190667) that verifies the private test dataset properties could be trusted **only if** \n\n    - Q1 is True \n    - the answer for Q2 is that the private LB score is already calculated, OR Q3 is True\n\n@sohier - Is it possible to answer these 3 questions? Thanks.",
    "1048329": "In all kaggle comps, Public LB and Private LB score for each sub is calculated // IMO.",
    "1052861": "I'm not so sure about your statements ... In Facebook Deep Fake Detection and NFL Bowl 2020, submitted notebooks were re-run on a fresh new test set !",
    "1052867": "Well in that case, it's a different case as it's a two-tier comp.",
    "1055439": "> Q1. When we submit a notebook for this competition, does it run on the whole private test dataset (and 20% of them are used to calculate the public LB score).\n\n@yihdarshieh See https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/191696#1055225",
    "1059692": "nyakaggle , thanks! That's good news."
  },
  "source": "meta"
}