{
  "id": 2843,
  "title": "Submitting Your Final Models",
  "url": "/competitions/predict-closed-questions-on-stack-overflow/discussion/2843",
  "author_name": "",
  "post_date": "2012-10-09T06:49:43.990Z",
  "votes": null,
  "comment_count": 15,
  "views": 7167,
  "content": "<p>As a reminder, the deadline to submit your final models is in less than 24 hours (at 11:59PM UTC on October 9).</p>\r\n<p>In order to be eligible for prize money, you must upload any code and external data necessary to train your models and make predictions on new test samples prior to this date. This should include a README with instructions on anything needed to run your\r\n training and prediction code, but does not need to include a thorough description of the methods you use (the prize winners will submit a more detailed description following\r\n<a href=\"https://www.kaggle.com/wiki/WinningModelDocumentationTemplate\">these guidelines</a>).</p>\r\n<p>From October 10-October 23, we will be collecting a new set of data to use for the final model evaluation. On October 24, we will release this new set in the same format as the pubic leaderboard set, along with a new training set containing data through\r\n October 9. You may optionally retrain your model on this updated training set, and need to submit your final predictions by Thursday, November 1.</p>",
  "messages": [
    {
      "id": "15329",
      "postDate": "10/09/2012 06:49:43",
      "content": "<p>As a reminder, the deadline to submit your final models is in less than 24 hours (at 11:59PM UTC on October 9).</p>\r\n<p>In order to be eligible for prize money, you must upload any code and external data necessary to train your models and make predictions on new test samples prior to this date. This should include a README with instructions on anything needed to run your\r\n training and prediction code, but does not need to include a thorough description of the methods you use (the prize winners will submit a more detailed description following\r\n<a href=\"https://www.kaggle.com/wiki/WinningModelDocumentationTemplate\">these guidelines</a>).</p>\r\n<p>From October 10-October 23, we will be collecting a new set of data to use for the final model evaluation. On October 24, we will release this new set in the same format as the pubic leaderboard set, along with a new training set containing data through\r\n October 9. You may optionally retrain your model on this updated training set, and need to submit your final predictions by Thursday, November 1.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15338",
      "postDate": "10/09/2012 11:36:42",
      "content": "<p>Hi,</p>\r\n<p>1. It seems the submissions page does not anymore allow selecting the best entry (see attachment). Does that mean the time for selecting the best entry has passed?</p>\r\n<p>2. You say &quot;<span>make predictions on <strong><em>new</em></strong> test samples</span>&quot;. Where these new test samples can be found from? Does data page contain a recently updated\r\n<span>public_leaderboard</span>?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15346",
      "postDate": "10/09/2012 14:12:06",
      "content": "<p>When uploading a submission I also added a model attachment but it doesn't show up on the &quot;My Submissions&quot; page.&nbsp;I can attach it from that page though and it seems to stick ... I hope that's the right way to go about it.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15349",
      "postDate": "10/09/2012 14:30:45",
      "content": "<p>Wow, I think I figured it out: using the same file name for attachments to different submissions is not a good idea.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15350",
      "postDate": "10/09/2012 14:34:04",
      "content": "<p>I wonder though which version of malacka.zip ended up being in effect ... Ben, can you give me the md5 checksum, please?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15353",
      "postDate": "10/09/2012 15:20:28",
      "content": "<p>[quote=Gábor Melis;15350]</p>\r\n<p>I wonder though which version of malacka.zip ended up being in effect ... Ben, can you give me the md5 checksum, please?</p>\r\n<p>[/quote]Checksum is&nbsp;BB-AA-C0-E2-E9-B7-C5-E1-CE-B7-0E-73-8C-EC-28-3E.</p>\r\n<p>Strange, submitting 2 files with the same name should have been fine. We'll look into it. (However, you can only attach one file per submission. Is it possible that this is the problem?)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15355",
      "postDate": "10/09/2012 15:57:22",
      "content": "<p>Ben, can you have a look into the UI issue I have mentioned?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15356",
      "postDate": "10/09/2012 16:10:03",
      "content": "<p>Thank you. That's the latest.</p>\r\n<p>No, I only attached one per submission.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15359",
      "postDate": "10/09/2012 17:40:12",
      "content": "<p>[quote=deekey;15338]</p>\r\n<p>Hi,</p>\r\n<p>1. It seems the submissions page does not anymore allow selecting the best entry (see attachment). Does that mean the time for selecting the best entry has passed?</p>\r\n<p>2. You say &quot;<span>make predictions on <strong><em>new</em></strong> test samples</span>&quot;. Where these new test samples can be found from? Does data page contain a recently updated\r\n<span>public_leaderboard</span>?</p>\r\n<p>[/quote]The final evaluation period comes in 2 weeks, along with the new test samples. See the\r\n<a href=\"https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow/details/timeline\">\r\ntimeline </a>for more information.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15403",
      "postDate": "10/10/2012 17:39:37",
      "content": "<p>I'm curious how the next phase works. Do Kaggle admins run each contestant's code on the new test samples? &nbsp;I'm guessing the answer is no, based on your post at the top. &nbsp;In that case, can contestants who did not submit code still make a submission to the\r\n private leaderboard, even if they are not eligible to win?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15405",
      "postDate": "10/10/2012 18:14:28",
      "content": "<p>[quote=Chris;15403]</p>\r\n<p>I'm curious how the next phase works. Do Kaggle admins run each contestant's code on the new test samples? &nbsp;I'm guessing the answer is no, based on your post at the top. &nbsp;In that case, can contestants who did not submit code still make a submission to the\r\n private leaderboard, even if they are not eligible to win?</p>\r\n<p>[/quote]No, only the top model(s) will be verified at the end. Anyone may make a final submission, but it will only be eligible for prize money if it was generated from the submitted model (and any great discrepancies in performance will be deleted without\r\n further investigation).</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15434",
      "postDate": "10/11/2012 17:04:43",
      "content": "<p>interesting ben.<br>\r\nSay I am not interested in prize money - can I just submit a new model to see if it works well?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15437",
      "postDate": "10/11/2012 17:08:47",
      "content": "<p>[quote=Black Magic;15434]</p>\r\n<p>interesting ben.<br>\r\nSay I am not interested in prize money - can I just submit a new model to see if it works well?</p>\r\n<p>[/quote]Not until the contest is over</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15441",
      "postDate": "10/11/2012 18:12:16",
      "content": "<p>if you are only testing the top 3 submissions, how would we ensure folks that are below 3 are not disadvantaged because someone who is 15th tries to become 4th</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15442",
      "postDate": "10/11/2012 18:15:22",
      "content": "<p>I find this bizarre. If somebody improves his score and does not want to share his code, say 3rd becomes 1st and then says I don't want the prize, do we take back his #1 position?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "15447",
      "postDate": "10/11/2012 19:41:47",
      "content": "<p>no response!<br>\r\nit is only fair that those who submitted models on time are considered for prize. However others must be free to improve their models and submit - in case they improve scores, their ranking stays but they do not get a prize.<br>\r\nI am sure most would agree that this is fair. Why hold on for a few more days to improve models?</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 15338,
      "author_name": "dmitrykan",
      "author_url": "",
      "post_date": "10/09/2012 11:36:42",
      "content": "<p>Hi,</p>\r\n<p>1. It seems the submissions page does not anymore allow selecting the best entry (see attachment). Does that mean the time for selecting the best entry has passed?</p>\r\n<p>2. You say &quot;<span>make predictions on <strong><em>new</em></strong> test samples</span>&quot;. Where these new test samples can be found from? Does data page contain a recently updated\r\n<span>public_leaderboard</span>?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15346,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "10/09/2012 14:12:06",
      "content": "<p>When uploading a submission I also added a model attachment but it doesn't show up on the &quot;My Submissions&quot; page.&nbsp;I can attach it from that page though and it seems to stick ... I hope that's the right way to go about it.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15349,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "10/09/2012 14:30:45",
      "content": "<p>Wow, I think I figured it out: using the same file name for attachments to different submissions is not a good idea.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15350,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "10/09/2012 14:34:04",
      "content": "<p>I wonder though which version of malacka.zip ended up being in effect ... Ben, can you give me the md5 checksum, please?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15353,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/09/2012 15:20:28",
      "content": "<p>[quote=Gábor Melis;15350]</p>\r\n<p>I wonder though which version of malacka.zip ended up being in effect ... Ben, can you give me the md5 checksum, please?</p>\r\n<p>[/quote]Checksum is&nbsp;BB-AA-C0-E2-E9-B7-C5-E1-CE-B7-0E-73-8C-EC-28-3E.</p>\r\n<p>Strange, submitting 2 files with the same name should have been fine. We'll look into it. (However, you can only attach one file per submission. Is it possible that this is the problem?)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15355,
      "author_name": "dmitrykan",
      "author_url": "",
      "post_date": "10/09/2012 15:57:22",
      "content": "<p>Ben, can you have a look into the UI issue I have mentioned?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15356,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "10/09/2012 16:10:03",
      "content": "<p>Thank you. That's the latest.</p>\r\n<p>No, I only attached one per submission.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15359,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/09/2012 17:40:12",
      "content": "<p>[quote=deekey;15338]</p>\r\n<p>Hi,</p>\r\n<p>1. It seems the submissions page does not anymore allow selecting the best entry (see attachment). Does that mean the time for selecting the best entry has passed?</p>\r\n<p>2. You say &quot;<span>make predictions on <strong><em>new</em></strong> test samples</span>&quot;. Where these new test samples can be found from? Does data page contain a recently updated\r\n<span>public_leaderboard</span>?</p>\r\n<p>[/quote]The final evaluation period comes in 2 weeks, along with the new test samples. See the\r\n<a href=\"https://www.kaggle.com/c/predict-closed-questions-on-stack-overflow/details/timeline\">\r\ntimeline </a>for more information.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15403,
      "author_name": "cdubois",
      "author_url": "",
      "post_date": "10/10/2012 17:39:37",
      "content": "<p>I'm curious how the next phase works. Do Kaggle admins run each contestant's code on the new test samples? &nbsp;I'm guessing the answer is no, based on your post at the top. &nbsp;In that case, can contestants who did not submit code still make a submission to the\r\n private leaderboard, even if they are not eligible to win?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15405,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/10/2012 18:14:28",
      "content": "<p>[quote=Chris;15403]</p>\r\n<p>I'm curious how the next phase works. Do Kaggle admins run each contestant's code on the new test samples? &nbsp;I'm guessing the answer is no, based on your post at the top. &nbsp;In that case, can contestants who did not submit code still make a submission to the\r\n private leaderboard, even if they are not eligible to win?</p>\r\n<p>[/quote]No, only the top model(s) will be verified at the end. Anyone may make a final submission, but it will only be eligible for prize money if it was generated from the submitted model (and any great discrepancies in performance will be deleted without\r\n further investigation).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15434,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "10/11/2012 17:04:43",
      "content": "<p>interesting ben.<br>\r\nSay I am not interested in prize money - can I just submit a new model to see if it works well?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15437,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "10/11/2012 17:08:47",
      "content": "<p>[quote=Black Magic;15434]</p>\r\n<p>interesting ben.<br>\r\nSay I am not interested in prize money - can I just submit a new model to see if it works well?</p>\r\n<p>[/quote]Not until the contest is over</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15441,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "10/11/2012 18:12:16",
      "content": "<p>if you are only testing the top 3 submissions, how would we ensure folks that are below 3 are not disadvantaged because someone who is 15th tries to become 4th</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15442,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "10/11/2012 18:15:22",
      "content": "<p>I find this bizarre. If somebody improves his score and does not want to share his code, say 3rd becomes 1st and then says I don't want the prize, do we take back his #1 position?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 15447,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "10/11/2012 19:41:47",
      "content": "<p>no response!<br>\r\nit is only fair that those who submitted models on time are considered for prize. However others must be free to improve their models and submit - in case they improve scores, their ranking stays but they do not get a prize.<br>\r\nI am sure most would agree that this is fair. Why hold on for a few more days to improve models?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "15329": "",
    "15338": "",
    "15346": "",
    "15349": "",
    "15350": "",
    "15353": "",
    "15355": "",
    "15356": "",
    "15359": "",
    "15403": "",
    "15405": "",
    "15434": "",
    "15437": "",
    "15441": "",
    "15442": "",
    "15447": ""
  },
  "source": "meta"
}