{
  "id": 475128,
  "title": "Do we see all test data?",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/475128",
  "author_name": "",
  "post_date": "2024-02-07T07:44:24.430295700Z",
  "votes": 4,
  "comment_count": 3,
  "views": 0,
  "content": "<p>In the notebooks we predict on test data and the public leaderboard is calculated on 30% of the test data.<br>\nDoes this mean the test data we load in the notebook is 30% of the test data, or do we see all test data, but the evaluation is just on 30% of them?</p>",
  "messages": [
    {
      "id": "2640970",
      "postDate": "02/07/2024 07:44:24",
      "content": "<p>In the notebooks we predict on test data and the public leaderboard is calculated on 30% of the test data.<br>\nDoes this mean the test data we load in the notebook is 30% of the test data, or do we see all test data, but the evaluation is just on 30% of them?</p>",
      "rawMarkdown": "In the notebooks we predict on test data and the public leaderboard is calculated on 30% of the test data.\nDoes this mean the test data we load in the notebook is 30% of the test data, or do we see all test data, but the evaluation is just on 30% of them?",
      "votes": null
    },
    {
      "id": "2641006",
      "postDate": "02/07/2024 08:18:30",
      "content": "<p>You don't get to see all of the test data. The data that's loaded in the notebook is just a tiny sample so that your code will work. When you make a submission it will be replaced with the full test set.</p>",
      "rawMarkdown": "You don't get to see all of the test data. The data that's loaded in the notebook is just a tiny sample so that your code will work. When you make a submission it will be replaced with the full test set.",
      "votes": null
    },
    {
      "id": "2641336",
      "postDate": "02/07/2024 12:51:04",
      "content": "<p>Exactly as Branden said. The test files you see are just tiny sample for kagglers to prepare their own notebooks. Those case_ids are excluded then from the evaluation. Then your notebook will be evaluated on the whole test sample. Only predictions for public leaderboard are used to calculate the score in public leaderboard.</p>\n<p>This competition is code competition, because we wanted to leave the test sample hidden. This would simulate real-world scenario when you don't know what is going to happen in future. </p>",
      "rawMarkdown": "Exactly as Branden said. The test files you see are just tiny sample for kagglers to prepare their own notebooks. Those case_ids are excluded then from the evaluation. Then your notebook will be evaluated on the whole test sample. Only predictions for public leaderboard are used to calculate the score in public leaderboard.\n\nThis competition is code competition, because we wanted to leave the test sample hidden. This would simulate real-world scenario when you don't know what is going to happen in future.",
      "votes": null
    },
    {
      "id": "2642045",
      "postDate": "02/07/2024 21:29:07",
      "content": "<p>I think in context to some recent competitions which do not even have col all test data because they are rea-world such as trading or electricity… so</p>\n<ol>\n<li>when you run your notebook, the test data is just a fraction  of the test data</li>\n<li>when you submit to the competition and your notebook runs again, it sees all test data and performs prediction, but only 30% is immediately shown on LB, and the rest waits till the competition finishes</li>\n</ol>\n<p>this means that your notebook can run fine but can crash for some other reason on submission when the test data are much larger</p>",
      "rawMarkdown": "I think in context to some recent competitions which do not even have col all test data because they are rea-world such as trading or electricity... so\n1. when you run your notebook, the test data is just a fraction  of the test data\n2. when you submit to the competition and your notebook runs again, it sees all test data and performs prediction, but only 30% is immediately shown on LB, and the rest waits till the competition finishes\n\nthis means that your notebook can run fine but can crash for some other reason on submission when the test data are much larger",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2641006,
      "author_name": "brandenkmurray",
      "author_url": "",
      "post_date": "02/07/2024 08:18:30",
      "content": "<p>You don't get to see all of the test data. The data that's loaded in the notebook is just a tiny sample so that your code will work. When you make a submission it will be replaced with the full test set.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2641336,
      "author_name": "jetakow",
      "author_url": "",
      "post_date": "02/07/2024 12:51:04",
      "content": "<p>Exactly as Branden said. The test files you see are just tiny sample for kagglers to prepare their own notebooks. Those case_ids are excluded then from the evaluation. Then your notebook will be evaluated on the whole test sample. Only predictions for public leaderboard are used to calculate the score in public leaderboard.</p>\n<p>This competition is code competition, because we wanted to leave the test sample hidden. This would simulate real-world scenario when you don't know what is going to happen in future. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2642045,
      "author_name": "jirkaborovec",
      "author_url": "",
      "post_date": "02/07/2024 21:29:07",
      "content": "<p>I think in context to some recent competitions which do not even have col all test data because they are rea-world such as trading or electricity… so</p>\n<ol>\n<li>when you run your notebook, the test data is just a fraction  of the test data</li>\n<li>when you submit to the competition and your notebook runs again, it sees all test data and performs prediction, but only 30% is immediately shown on LB, and the rest waits till the competition finishes</li>\n</ol>\n<p>this means that your notebook can run fine but can crash for some other reason on submission when the test data are much larger</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2640970": "In the notebooks we predict on test data and the public leaderboard is calculated on 30% of the test data.\nDoes this mean the test data we load in the notebook is 30% of the test data, or do we see all test data, but the evaluation is just on 30% of them?",
    "2641006": "You don't get to see all of the test data. The data that's loaded in the notebook is just a tiny sample so that your code will work. When you make a submission it will be replaced with the full test set.",
    "2641336": "Exactly as Branden said. The test files you see are just tiny sample for kagglers to prepare their own notebooks. Those case_ids are excluded then from the evaluation. Then your notebook will be evaluated on the whole test sample. Only predictions for public leaderboard are used to calculate the score in public leaderboard.\n\nThis competition is code competition, because we wanted to leave the test sample hidden. This would simulate real-world scenario when you don't know what is going to happen in future.",
    "2642045": "I think in context to some recent competitions which do not even have col all test data because they are rea-world such as trading or electricity... so\n1. when you run your notebook, the test data is just a fraction  of the test data\n2. when you submit to the competition and your notebook runs again, it sees all test data and performs prediction, but only 30% is immediately shown on LB, and the rest waits till the competition finishes\n\nthis means that your notebook can run fine but can crash for some other reason on submission when the test data are much larger"
  },
  "source": "meta"
}