{
  "id": 19318,
  "title": "Final model specification",
  "url": "/competitions/second-annual-data-science-bowl/discussion/19318",
  "author_name": "",
  "post_date": "2016-03-04T13:44:06.743Z",
  "votes": null,
  "comment_count": 9,
  "views": 1129,
  "content": "<p>I have a few questions about the final model submission:</p>\n\n<ol>\n<li>Is there any reference about how the program should be written, like the command line arguments? Would it be simply like:</li>\n</ol>\n\n<p><code>&lt;main program&gt; -d &lt;path to test cases with folder like '500', '501'...&gt; -o &lt;outfile csv&gt;</code></p>\n\n<p>Or I can just assume the correct parameter will be set correctly in the code, and there is no command line parameters to be handled?</p>\n\n<ol start=\"2\">\n<li><p>Since the scoring will be automatically, what should be assumed in the machine? If I am developing using windows with cuda (using theano), will the machine running my code has similar setup (with a decent gpu so that the run time can be minimized)? How do I ensure all the pre-requisite (package dependency) are met? </p></li>\n<li><p>Is there time limit to how long the code can run for each test cases on average? If it runs like 5 minutes, is it acceptable?</p></li>\n<li><p>Sometimes, the softwares that the code using may crash (like python crash, or cuda related, probably bugs in software that I haven't written). Should I assume my code is responsible to handling these situations (It may not be possible in some cases)? Should I write my code that can be resumed and restarted, and does it needs to handle the crashing and restart the job automatically?</p></li>\n</ol>",
  "messages": [
    {
      "id": "110294",
      "postDate": "03/04/2016 13:44:06",
      "content": "<p>I have a few questions about the final model submission:</p>\n\n<ol>\n<li>Is there any reference about how the program should be written, like the command line arguments? Would it be simply like:</li>\n</ol>\n\n<p><code>&lt;main program&gt; -d &lt;path to test cases with folder like '500', '501'...&gt; -o &lt;outfile csv&gt;</code></p>\n\n<p>Or I can just assume the correct parameter will be set correctly in the code, and there is no command line parameters to be handled?</p>\n\n<ol start=\"2\">\n<li><p>Since the scoring will be automatically, what should be assumed in the machine? If I am developing using windows with cuda (using theano), will the machine running my code has similar setup (with a decent gpu so that the run time can be minimized)? How do I ensure all the pre-requisite (package dependency) are met? </p></li>\n<li><p>Is there time limit to how long the code can run for each test cases on average? If it runs like 5 minutes, is it acceptable?</p></li>\n<li><p>Sometimes, the softwares that the code using may crash (like python crash, or cuda related, probably bugs in software that I haven't written). Should I assume my code is responsible to handling these situations (It may not be possible in some cases)? Should I write my code that can be resumed and restarted, and does it needs to handle the crashing and restart the job automatically?</p></li>\n</ol>",
      "rawMarkdown": "I have a few questions about the final model submission:\r\n\r\n1. Is there any reference about how the program should be written, like the command line arguments? Would it be simply like:\r\n\r\n`<main program> -d <path to test cases with folder like '500', '501'...> -o <outfile csv>`\r\n\r\nOr I can just assume the correct parameter will be set correctly in the code, and there is no command line parameters to be handled?\r\n\r\n2. Since the scoring will be automatically, what should be assumed in the machine? If I am developing using windows with cuda (using theano), will the machine running my code has similar setup (with a decent gpu so that the run time can be minimized)? How do I ensure all the pre-requisite (package dependency) are met? \r\n\r\n3. Is there time limit to how long the code can run for each test cases on average? If it runs like 5 minutes, is it acceptable?\r\n\r\n4. Sometimes, the softwares that the code using may crash (like python crash, or cuda related, probably bugs in software that I haven't written). Should I assume my code is responsible to handling these situations (It may not be possible in some cases)? Should I write my code that can be resumed and restarted, and does it needs to handle the crashing and restart the job automatically?",
      "votes": null
    },
    {
      "id": "110297",
      "postDate": "03/04/2016 14:01:05",
      "content": "<p>It looks like some of the problems are answered in this:</p>\n\n<p><a href=\"https://www.kaggle.com/wiki/ModelSubmissionBestPractices\">https://www.kaggle.com/wiki/ModelSubmissionBestPractices</a></p>\n\n<p>Still, if 3 and 4 can be answered, it would be great.</p>",
      "rawMarkdown": "It looks like some of the problems are answered in this:\r\n\r\nhttps://www.kaggle.com/wiki/ModelSubmissionBestPractices\r\n\r\nStill, if 3 and 4 can be answered, it would be great.",
      "votes": null
    },
    {
      "id": "110302",
      "postDate": "03/04/2016 14:26:56",
      "content": "<p>There is no time limit restriction other than your code needs to finish (don't crank some parameter up such that it causes execution to take years).</p>\n\n<p>You should debug your code to reasonable levels. If it crashes every third time, something is wrong and should take a hard look at gracefully handling it. If it crashes every thousandth time, we're not going to hold you responsible.</p>\n\n<p>We are very reasonable in evaluating the merit of the science and not the perfection of the code.</p>",
      "rawMarkdown": "There is no time limit restriction other than your code needs to finish (don't crank some parameter up such that it causes execution to take years).\r\n\r\nYou should debug your code to reasonable levels. If it crashes every third time, something is wrong and should take a hard look at gracefully handling it. If it crashes every thousandth time, we're not going to hold you responsible.\r\n\r\nWe are very reasonable in evaluating the merit of the science and not the perfection of the code.",
      "votes": null
    },
    {
      "id": "110311",
      "postDate": "03/04/2016 15:14:38",
      "content": "<p>How can we make sure that the results are reproducible even if we set the constant seeds ?</p>\n\n<p><a href=\"https://www.kaggle.com/wiki/WinningModelDocumentationTemplate\">https://www.kaggle.com/wiki/WinningModelDocumentationTemplate</a>\nResults are exactly reproducible. Set seeds to any random number generators to constant values along with anything else necessary to exactly reproduce results, and make sure these are saved in your code.</p>",
      "rawMarkdown": "How can we make sure that the results are reproducible even if we set the constant seeds ?\r\n \r\n\r\n\r\nhttps://www.kaggle.com/wiki/WinningModelDocumentationTemplate\r\nResults are exactly reproducible. Set seeds to any random number generators to constant values along with anything else necessary to exactly reproduce results, and make sure these are saved in your code.",
      "votes": null
    },
    {
      "id": "110312",
      "postDate": "03/04/2016 15:25:37",
      "content": "<p>Setting seeds is usually sufficient.</p>",
      "rawMarkdown": "Setting seeds is usually sufficient.",
      "votes": null
    },
    {
      "id": "110334",
      "postDate": "03/04/2016 18:08:55",
      "content": "<p>Can you please clarify the following. I understand that in order to be eligible for prize we need to upload the model. But can we still make stage two submissions and be on the final leader board without uploading the model.</p>\n\n<p>Thanks.</p>",
      "rawMarkdown": "Can you please clarify the following. I understand that in order to be eligible for prize we need to upload the model. But can we still make stage two submissions and be on the final leader board without uploading the model.\r\n\r\nThanks.",
      "votes": null
    },
    {
      "id": "110348",
      "postDate": "03/04/2016 19:56:53",
      "content": "<p>Yes, you may still submit and be on the leaderboard if you do not upload a model.</p>",
      "rawMarkdown": "Yes, you may still submit and be on the leaderboard if you do not upload a model.",
      "votes": null
    },
    {
      "id": "110900",
      "postDate": "03/09/2016 12:41:21",
      "content": "<p>[quote=William Cukierski;110348]</p>\n\n<p>Yes, you may still submit and be on the leaderboard if you do not upload a model.</p>\n\n<p>[/quote]</p>\n\n<p>Will, would you mind clarifying this last point? I didn't submit a model as there was no chance I was going to be in the running for this competition, and frankly, I joined late so was still tweaking things. I'd still potentially like to submit for the final leaderboard if possible, but when I do I'm told the competition is no longer accepting new entrants. </p>",
      "rawMarkdown": "[quote=William Cukierski;110348]\r\n\r\nYes, you may still submit and be on the leaderboard if you do not upload a model.\r\n\r\n[/quote]\r\n\r\nWill, would you mind clarifying this last point? I didn't submit a model as there was no chance I was going to be in the running for this competition, and frankly, I joined late so was still tweaking things. I'd still potentially like to submit for the final leaderboard if possible, but when I do I'm told the competition is no longer accepting new entrants.",
      "votes": null
    },
    {
      "id": "110962",
      "postDate": "03/09/2016 22:54:32",
      "content": "<p>@Maineiac Apologies for the glitch. You're all set to submit now.</p>",
      "rawMarkdown": "Maineiac Apologies for the glitch. You're all set to submit now.",
      "votes": null
    },
    {
      "id": "110985",
      "postDate": "03/10/2016 05:13:44",
      "content": "<p>[quote=William Cukierski;110962]</p>\n\n<p>@Maineiac Apologies for the glitch. You're all set to submit now.</p>\n\n<p>[/quote]</p>\n\n<p>Awesome. Thanks, Will.</p>",
      "rawMarkdown": "[quote=William Cukierski;110962]\r\n\r\n@Maineiac Apologies for the glitch. You're all set to submit now.\r\n\r\n[/quote]\r\n\r\nAwesome. Thanks, Will.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 110297,
      "author_name": "rosstch",
      "author_url": "",
      "post_date": "03/04/2016 14:01:05",
      "content": "<p>It looks like some of the problems are answered in this:</p>\n\n<p><a href=\"https://www.kaggle.com/wiki/ModelSubmissionBestPractices\">https://www.kaggle.com/wiki/ModelSubmissionBestPractices</a></p>\n\n<p>Still, if 3 and 4 can be answered, it would be great.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 110302,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "03/04/2016 14:26:56",
      "content": "<p>There is no time limit restriction other than your code needs to finish (don't crank some parameter up such that it causes execution to take years).</p>\n\n<p>You should debug your code to reasonable levels. If it crashes every third time, something is wrong and should take a hard look at gracefully handling it. If it crashes every thousandth time, we're not going to hold you responsible.</p>\n\n<p>We are very reasonable in evaluating the merit of the science and not the perfection of the code.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 110311,
      "author_name": "data2015",
      "author_url": "",
      "post_date": "03/04/2016 15:14:38",
      "content": "<p>How can we make sure that the results are reproducible even if we set the constant seeds ?</p>\n\n<p><a href=\"https://www.kaggle.com/wiki/WinningModelDocumentationTemplate\">https://www.kaggle.com/wiki/WinningModelDocumentationTemplate</a>\nResults are exactly reproducible. Set seeds to any random number generators to constant values along with anything else necessary to exactly reproduce results, and make sure these are saved in your code.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 110312,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "03/04/2016 15:25:37",
      "content": "<p>Setting seeds is usually sufficient.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 110334,
      "author_name": "alexch",
      "author_url": "",
      "post_date": "03/04/2016 18:08:55",
      "content": "<p>Can you please clarify the following. I understand that in order to be eligible for prize we need to upload the model. But can we still make stage two submissions and be on the final leader board without uploading the model.</p>\n\n<p>Thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 110348,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "03/04/2016 19:56:53",
      "content": "<p>Yes, you may still submit and be on the leaderboard if you do not upload a model.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 110900,
      "author_name": "maineiac",
      "author_url": "",
      "post_date": "03/09/2016 12:41:21",
      "content": "<p>[quote=William Cukierski;110348]</p>\n\n<p>Yes, you may still submit and be on the leaderboard if you do not upload a model.</p>\n\n<p>[/quote]</p>\n\n<p>Will, would you mind clarifying this last point? I didn't submit a model as there was no chance I was going to be in the running for this competition, and frankly, I joined late so was still tweaking things. I'd still potentially like to submit for the final leaderboard if possible, but when I do I'm told the competition is no longer accepting new entrants. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 110962,
      "author_name": "wcukierski",
      "author_url": "",
      "post_date": "03/09/2016 22:54:32",
      "content": "<p>@Maineiac Apologies for the glitch. You're all set to submit now.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 110985,
      "author_name": "maineiac",
      "author_url": "",
      "post_date": "03/10/2016 05:13:44",
      "content": "<p>[quote=William Cukierski;110962]</p>\n\n<p>@Maineiac Apologies for the glitch. You're all set to submit now.</p>\n\n<p>[/quote]</p>\n\n<p>Awesome. Thanks, Will.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "110294": "I have a few questions about the final model submission:\r\n\r\n1. Is there any reference about how the program should be written, like the command line arguments? Would it be simply like:\r\n\r\n`<main program> -d <path to test cases with folder like '500', '501'...> -o <outfile csv>`\r\n\r\nOr I can just assume the correct parameter will be set correctly in the code, and there is no command line parameters to be handled?\r\n\r\n2. Since the scoring will be automatically, what should be assumed in the machine? If I am developing using windows with cuda (using theano), will the machine running my code has similar setup (with a decent gpu so that the run time can be minimized)? How do I ensure all the pre-requisite (package dependency) are met? \r\n\r\n3. Is there time limit to how long the code can run for each test cases on average? If it runs like 5 minutes, is it acceptable?\r\n\r\n4. Sometimes, the softwares that the code using may crash (like python crash, or cuda related, probably bugs in software that I haven't written). Should I assume my code is responsible to handling these situations (It may not be possible in some cases)? Should I write my code that can be resumed and restarted, and does it needs to handle the crashing and restart the job automatically?",
    "110297": "It looks like some of the problems are answered in this:\r\n\r\nhttps://www.kaggle.com/wiki/ModelSubmissionBestPractices\r\n\r\nStill, if 3 and 4 can be answered, it would be great.",
    "110302": "There is no time limit restriction other than your code needs to finish (don't crank some parameter up such that it causes execution to take years).\r\n\r\nYou should debug your code to reasonable levels. If it crashes every third time, something is wrong and should take a hard look at gracefully handling it. If it crashes every thousandth time, we're not going to hold you responsible.\r\n\r\nWe are very reasonable in evaluating the merit of the science and not the perfection of the code.",
    "110311": "How can we make sure that the results are reproducible even if we set the constant seeds ?\r\n \r\n\r\n\r\nhttps://www.kaggle.com/wiki/WinningModelDocumentationTemplate\r\nResults are exactly reproducible. Set seeds to any random number generators to constant values along with anything else necessary to exactly reproduce results, and make sure these are saved in your code.",
    "110312": "Setting seeds is usually sufficient.",
    "110334": "Can you please clarify the following. I understand that in order to be eligible for prize we need to upload the model. But can we still make stage two submissions and be on the final leader board without uploading the model.\r\n\r\nThanks.",
    "110348": "Yes, you may still submit and be on the leaderboard if you do not upload a model.",
    "110900": "[quote=William Cukierski;110348]\r\n\r\nYes, you may still submit and be on the leaderboard if you do not upload a model.\r\n\r\n[/quote]\r\n\r\nWill, would you mind clarifying this last point? I didn't submit a model as there was no chance I was going to be in the running for this competition, and frankly, I joined late so was still tweaking things. I'd still potentially like to submit for the final leaderboard if possible, but when I do I'm told the competition is no longer accepting new entrants.",
    "110962": "Maineiac Apologies for the glitch. You're all set to submit now.",
    "110985": "[quote=William Cukierski;110962]\r\n\r\n@Maineiac Apologies for the glitch. You're all set to submit now.\r\n\r\n[/quote]\r\n\r\nAwesome. Thanks, Will."
  },
  "source": "meta"
}