{
  "id": 416751,
  "title": "Does the private leaderboard use the latest version of trained model or the submitted version?",
  "url": "/competitions/predict-student-performance-from-game-play/discussion/416751",
  "author_name": "",
  "post_date": "2023-06-12T23:51:56.789088600Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello everyone, <br>\nThis is my first notebook competition, and I have a question regarding versions of input files.</p>\n<p>I have two notebooks: one for training models, and one for inference. Let’s say I submit my inference notebook, which uses the output of training notebook (called <code>train_v1</code>) as input. After that I modify and commit my training notebook, which outputs <code>train_v2</code>. At this time, when I open the inference notebook, it will automatically uses the latest version of input, i.e. <code>train_v2</code>.</p>\n<p>My understanding is that at the end of the competition, our selected submmited notebooks will be re-scored using a hidden dataset different than the public leaderboard.</p>\n<p>My question is, which version of input model will the private leaderboard use when scoring the notebook? Will it be the same version used in submission (<code>train_v1</code>), or will it use the latest version (<code>train_v2</code>)?</p>\n<p>If It uses the latest version then I’ll have to backup my models… because sadly sometimes the latest model performs worse than the old model. </p>",
  "messages": [
    {
      "id": "2299907",
      "postDate": "06/12/2023 23:51:56",
      "content": "<p>Hello everyone, <br>\nThis is my first notebook competition, and I have a question regarding versions of input files.</p>\n<p>I have two notebooks: one for training models, and one for inference. Let’s say I submit my inference notebook, which uses the output of training notebook (called <code>train_v1</code>) as input. After that I modify and commit my training notebook, which outputs <code>train_v2</code>. At this time, when I open the inference notebook, it will automatically uses the latest version of input, i.e. <code>train_v2</code>.</p>\n<p>My understanding is that at the end of the competition, our selected submmited notebooks will be re-scored using a hidden dataset different than the public leaderboard.</p>\n<p>My question is, which version of input model will the private leaderboard use when scoring the notebook? Will it be the same version used in submission (<code>train_v1</code>), or will it use the latest version (<code>train_v2</code>)?</p>\n<p>If It uses the latest version then I’ll have to backup my models… because sadly sometimes the latest model performs worse than the old model. </p>",
      "rawMarkdown": "Hello everyone, \nThis is my first notebook competition, and I have a question regarding versions of input files.\n\nI have two notebooks: one for training models, and one for inference. Let’s say I submit my inference notebook, which uses the output of training notebook (called `train_v1`) as input. After that I modify and commit my training notebook, which outputs `train_v2`. At this time, when I open the inference notebook, it will automatically uses the latest version of input, i.e. `train_v2`.\n\nMy understanding is that at the end of the competition, our selected submmited notebooks will be re-scored using a hidden dataset different than the public leaderboard.\n\nMy question is, which version of input model will the private leaderboard use when scoring the notebook? Will it be the same version used in submission (`train_v1`), or will it use the latest version (`train_v2`)?\n\nIf It uses the latest version then I’ll have to backup my models… because sadly sometimes the latest model performs worse than the old model.",
      "votes": null
    },
    {
      "id": "2300563",
      "postDate": "06/13/2023 08:55:18",
      "content": "<blockquote>\n  <p>My understanding is that at the end of the competition, our selected submitted notebooks will be re-scored using a hidden dataset different than the public leaderboard.</p>\n</blockquote>\n<p>According to my former code competition experience, that is not the case. At the end of the competition, private LB will NOT re-run your notebook, but immediately show your private LB score. So I think the hidden dataset was already scored the first time you submitted them, only the score is hidden.</p>\n<p>That's not the official explanation but only my understanding though.</p>",
      "rawMarkdown": ">My understanding is that at the end of the competition, our selected submitted notebooks will be re-scored using a hidden dataset different than the public leaderboard.\n\nAccording to my former code competition experience, that is not the case. At the end of the competition, private LB will NOT re-run your notebook, but immediately show your private LB score. So I think the hidden dataset was already scored the first time you submitted them, only the score is hidden.\n\nThat's not the official explanation but only my understanding though.",
      "votes": null
    },
    {
      "id": "2300583",
      "postDate": "06/13/2023 09:17:44",
      "content": "<p>Oh that makes sense.  Thanks for your explanation!<br>\n1700 teams * 3 submission are way too many notebooks to re-run.</p>",
      "rawMarkdown": "Oh that makes sense.  Thanks for your explanation!\n1700 teams * 3 submission are way too many notebooks to re-run.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2300563,
      "author_name": "ataraxian",
      "author_url": "",
      "post_date": "06/13/2023 08:55:18",
      "content": "<blockquote>\n  <p>My understanding is that at the end of the competition, our selected submitted notebooks will be re-scored using a hidden dataset different than the public leaderboard.</p>\n</blockquote>\n<p>According to my former code competition experience, that is not the case. At the end of the competition, private LB will NOT re-run your notebook, but immediately show your private LB score. So I think the hidden dataset was already scored the first time you submitted them, only the score is hidden.</p>\n<p>That's not the official explanation but only my understanding though.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2300583,
          "author_name": "ncchen",
          "author_url": "",
          "post_date": "06/13/2023 09:17:44",
          "content": "<p>Oh that makes sense.  Thanks for your explanation!<br>\n1700 teams * 3 submission are way too many notebooks to re-run.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2299907": "Hello everyone, \nThis is my first notebook competition, and I have a question regarding versions of input files.\n\nI have two notebooks: one for training models, and one for inference. Let’s say I submit my inference notebook, which uses the output of training notebook (called `train_v1`) as input. After that I modify and commit my training notebook, which outputs `train_v2`. At this time, when I open the inference notebook, it will automatically uses the latest version of input, i.e. `train_v2`.\n\nMy understanding is that at the end of the competition, our selected submmited notebooks will be re-scored using a hidden dataset different than the public leaderboard.\n\nMy question is, which version of input model will the private leaderboard use when scoring the notebook? Will it be the same version used in submission (`train_v1`), or will it use the latest version (`train_v2`)?\n\nIf It uses the latest version then I’ll have to backup my models… because sadly sometimes the latest model performs worse than the old model.",
    "2300563": ">My understanding is that at the end of the competition, our selected submitted notebooks will be re-scored using a hidden dataset different than the public leaderboard.\n\nAccording to my former code competition experience, that is not the case. At the end of the competition, private LB will NOT re-run your notebook, but immediately show your private LB score. So I think the hidden dataset was already scored the first time you submitted them, only the score is hidden.\n\nThat's not the official explanation but only my understanding though.",
    "2300583": "Oh that makes sense.  Thanks for your explanation!\n1700 teams * 3 submission are way too many notebooks to re-run."
  },
  "source": "meta"
}