{
  "id": 46427,
  "title": "Choosing best 2 submissions for final leaderboard?",
  "url": "/competitions/tensorflow-speech-recognition-challenge/discussion/46427",
  "author_name": "",
  "post_date": "2017-12-27T06:55:25.918582400Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>First timer Kaggle user here. When you submit a new prediction output, you're greeted by the following message:</p>\n\n<blockquote>\n  <p>You can select up to 2 submissions to be used to calculate your final leaderboard score. If 2 submissions are not selected, they will be chosen based on your best submission scores on the public leaderboard.</p>\n  \n  <p>Your final score may not be based on the same exact subset of data as the public leaderboard, but rather a different private data subset of your full submission — your public score is only a rough indication of what your final score is.</p>\n  \n  <p>You should thus choose submissions that will most likely be best overall, and not necessarily on the public subset.</p>\n</blockquote>\n\n<p>I'm not sure I understand the catch here. When would you not select your best two submissions on the public leaderboard as your final two submissions?</p>",
  "messages": [
    {
      "id": "262586",
      "postDate": "12/27/2017 06:55:25",
      "content": "<p>First timer Kaggle user here. When you submit a new prediction output, you're greeted by the following message:</p>\n\n<blockquote>\n  <p>You can select up to 2 submissions to be used to calculate your final leaderboard score. If 2 submissions are not selected, they will be chosen based on your best submission scores on the public leaderboard.</p>\n  \n  <p>Your final score may not be based on the same exact subset of data as the public leaderboard, but rather a different private data subset of your full submission — your public score is only a rough indication of what your final score is.</p>\n  \n  <p>You should thus choose submissions that will most likely be best overall, and not necessarily on the public subset.</p>\n</blockquote>\n\n<p>I'm not sure I understand the catch here. When would you not select your best two submissions on the public leaderboard as your final two submissions?</p>",
      "rawMarkdown": "First timer Kaggle user here. When you submit a new prediction output, you're greeted by the following message:\n\n&gt; You can select up to 2 submissions to be used to calculate your final leaderboard score. If 2 submissions are not selected, they will be chosen based on your best submission scores on the public leaderboard.\n&gt;\n&gt; Your final score may not be based on the same exact subset of data as the public leaderboard, but rather a different private data subset of your full submission — your public score is only a rough indication of what your final score is.\n&gt; \n&gt; You should thus choose submissions that will most likely be best overall, and not necessarily on the public subset.\n\nI'm not sure I understand the catch here. When would you not select your best two submissions on the public leaderboard as your final two submissions?",
      "votes": null
    },
    {
      "id": "262596",
      "postDate": "12/27/2017 08:03:58",
      "content": "<blockquote>\n  <p>This leaderboard is calculated with approximately 30% of the test data.</p>\n</blockquote>\n\n<p>So the score in public leaderboard may not completely equal in all test data.</p>",
      "rawMarkdown": "&gt; This leaderboard is calculated with approximately 30% of the test data.\n\nSo the score in public leaderboard may not completely equal in all test data.",
      "votes": null
    },
    {
      "id": "262617",
      "postDate": "12/27/2017 09:30:42",
      "content": "<p>I get that. My question is: what's the strategy for choosing which submissions to use as your final submission? Just choose the top performers on the public leaderboard and hope for the best??</p>",
      "rawMarkdown": "I get that. My question is: what's the strategy for choosing which submissions to use as your final submission? Just choose the top performers on the public leaderboard and hope for the best??",
      "votes": null
    },
    {
      "id": "262716",
      "postDate": "12/27/2017 17:44:49",
      "content": "<p>Hi formigone -</p>\n\n<p>A typical strategy is to choose the submission with the highest Public leaderboard score, and then the submission with your best local, cross-validated score.</p>",
      "rawMarkdown": "Hi formigone -\n\nA typical strategy is to choose the submission with the highest Public leaderboard score, and then the submission with your best local, cross-validated score.",
      "votes": null
    },
    {
      "id": "262952",
      "postDate": "12/28/2017 12:25:58",
      "content": "<p>What could be the case is that the final score has much more (percentage wise) unknowns in it then the current PB set has. So right now you could have a good PB score, but it could still be very wrong regarding the unknowns. So you could have a strategy in which you select the following two submissions:</p>\n\n<p>1) One based on your best PB score (so assuming the unknowns percentage won't change during the final) \n2) One that puts more emphasis on unknowns (so assuming the unknown/known balance will change during final). Right now it might be a lower PB score but that then will change.</p>",
      "rawMarkdown": "What could be the case is that the final score has much more (percentage wise) unknowns in it then the current PB set has. So right now you could have a good PB score, but it could still be very wrong regarding the unknowns. So you could have a strategy in which you select the following two submissions:\n\n1) One based on your best PB score (so assuming the unknowns percentage won't change during the final) \n2) One that puts more emphasis on unknowns (so assuming the unknown/known balance will change during final). Right now it might be a lower PB score but that then will change.",
      "votes": null
    },
    {
      "id": "262971",
      "postDate": "12/28/2017 13:41:47",
      "content": "<p>Well, 2) is something you would have to bet on, no one knows for sure. But 1) is unknown to us as well. So you might want just focus on optimizing your approach based on your local validation set up. </p>",
      "rawMarkdown": "Well, 2) is something you would have to bet on, no one knows for sure. But 1) is unknown to us as well. So you might want just focus on optimizing your approach based on your local validation set up.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 262596,
      "author_name": "whiteworld",
      "author_url": "",
      "post_date": "12/27/2017 08:03:58",
      "content": "<blockquote>\n  <p>This leaderboard is calculated with approximately 30% of the test data.</p>\n</blockquote>\n\n<p>So the score in public leaderboard may not completely equal in all test data.</p>",
      "votes": null,
      "replies": [
        {
          "id": 262617,
          "author_name": "formigone",
          "author_url": "",
          "post_date": "12/27/2017 09:30:42",
          "content": "<p>I get that. My question is: what's the strategy for choosing which submissions to use as your final submission? Just choose the top performers on the public leaderboard and hope for the best??</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 262716,
          "author_name": "inversion",
          "author_url": "",
          "post_date": "12/27/2017 17:44:49",
          "content": "<p>Hi formigone -</p>\n\n<p>A typical strategy is to choose the submission with the highest Public leaderboard score, and then the submission with your best local, cross-validated score.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 262952,
      "author_name": "peterdekkers101",
      "author_url": "",
      "post_date": "12/28/2017 12:25:58",
      "content": "<p>What could be the case is that the final score has much more (percentage wise) unknowns in it then the current PB set has. So right now you could have a good PB score, but it could still be very wrong regarding the unknowns. So you could have a strategy in which you select the following two submissions:</p>\n\n<p>1) One based on your best PB score (so assuming the unknowns percentage won't change during the final) \n2) One that puts more emphasis on unknowns (so assuming the unknown/known balance will change during final). Right now it might be a lower PB score but that then will change.</p>",
      "votes": null,
      "replies": [
        {
          "id": 262971,
          "author_name": "ryanzhang",
          "author_url": "",
          "post_date": "12/28/2017 13:41:47",
          "content": "<p>Well, 2) is something you would have to bet on, no one knows for sure. But 1) is unknown to us as well. So you might want just focus on optimizing your approach based on your local validation set up. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "262586": "First timer Kaggle user here. When you submit a new prediction output, you're greeted by the following message:\n\n&gt; You can select up to 2 submissions to be used to calculate your final leaderboard score. If 2 submissions are not selected, they will be chosen based on your best submission scores on the public leaderboard.\n&gt;\n&gt; Your final score may not be based on the same exact subset of data as the public leaderboard, but rather a different private data subset of your full submission — your public score is only a rough indication of what your final score is.\n&gt; \n&gt; You should thus choose submissions that will most likely be best overall, and not necessarily on the public subset.\n\nI'm not sure I understand the catch here. When would you not select your best two submissions on the public leaderboard as your final two submissions?",
    "262596": "&gt; This leaderboard is calculated with approximately 30% of the test data.\n\nSo the score in public leaderboard may not completely equal in all test data.",
    "262617": "I get that. My question is: what's the strategy for choosing which submissions to use as your final submission? Just choose the top performers on the public leaderboard and hope for the best??",
    "262716": "Hi formigone -\n\nA typical strategy is to choose the submission with the highest Public leaderboard score, and then the submission with your best local, cross-validated score.",
    "262952": "What could be the case is that the final score has much more (percentage wise) unknowns in it then the current PB set has. So right now you could have a good PB score, but it could still be very wrong regarding the unknowns. So you could have a strategy in which you select the following two submissions:\n\n1) One based on your best PB score (so assuming the unknowns percentage won't change during the final) \n2) One that puts more emphasis on unknowns (so assuming the unknown/known balance will change during final). Right now it might be a lower PB score but that then will change.",
    "262971": "Well, 2) is something you would have to bet on, no one knows for sure. But 1) is unknown to us as well. So you might want just focus on optimizing your approach based on your local validation set up."
  },
  "source": "meta"
}