{
  "id": 3066,
  "title": "i am an idiot",
  "url": "/competitions/predict-closed-questions-on-stack-overflow/discussion/3066",
  "author_name": "",
  "post_date": "2012-11-04T00:35:55Z",
  "votes": null,
  "comment_count": 7,
  "views": 5587,
  "content": "<p>http://imgur.com/o7uSG</p>",
  "messages": [
    {
      "id": "16553",
      "postDate": "11/04/2012 00:35:55",
      "content": "<p>http://imgur.com/o7uSG</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16554",
      "postDate": "11/04/2012 01:18:53",
      "content": "<p>Ouch. Sorry to hear that. Why such a difference, though? Is the better one made from retrained model, and the worse one from old model?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16556",
      "postDate": "11/04/2012 02:25:00",
      "content": "<p>They were both re-trained models, but the later one included the feature &quot;was posted in october 2012&quot; (and &quot;august 2012&quot; as well) while the better one had everything july and after sharing the same feature, which is what I did for the public leaderboard\r\n part. This essentially gave me a different &quot;prior&quot; for each month.</p>\r\n<p>But I suspect something went drastically wrong since it's worse than the prior benchmark, and I wasn't keeping good enough notes while making my final submission. The two should have been roughly the same. I probably mixed up a model file somewhere; I'm\r\n retracing my steps. Sigh.</p>\r\n<p>edit: yeah, oops. &nbsp;I know what happened. &nbsp;I had cut out the first couple weeks of october as a validation set, then built the training set without that, and the 2012/10 feature thus never appeared in the training set, and so when it was encountered in a\r\n test example it totally failed to correct for that bias. &nbsp;ARGH. &nbsp;I have all sorts of tools that could have detected that, but I didn't spend enough time checking my solution.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16557",
      "postDate": "11/04/2012 02:37:22",
      "content": "<p>Hmm. I was under impression that changes like that (adding features, tweaking parameters, etc) are prohibited by rules at this stage. I mean, the model's code was supposed to be frozen when public leaderboard closed. Am I wrong?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16559",
      "postDate": "11/04/2012 06:27:21",
      "content": "<p>yes, model code cannot be changed.<br>\r\nMy estimate based on my analysis was that with the new dataset everyone's LogLoss will be approximately 2x of earlier. Going by final leaderboard, that is correct!</p>\r\n<p>More than anything else, it is known that in case of severely imbalanced datasets such as this one, most models will over-predict the majority class. A metric other than LogLoss would have been more appropriate - something based on multi-class precision\r\n and recall</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16571",
      "postDate": "11/04/2012 14:26:12",
      "content": "<p>I didn't re-train on new data. Ouch.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16572",
      "postDate": "11/04/2012 15:05:29",
      "content": "<p>[quote=jsn13;16557]</p>\r\n<p>Hmm. I was under impression that changes like that (adding features, tweaking parameters, etc) are prohibited by rules at this stage. I mean, the model's code was supposed to be frozen when public leaderboard closed. Am I wrong?</p>\r\n<p>[/quote]</p>\r\n<p>Yeah, well, it was a bit unclear. &nbsp;I mean, I bumped the cap on the yymm feature from July to October, and re-ran training, which added those features automatically. &nbsp;So maybe I wasn't supposed to do that? &nbsp;I'm not sure.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "16573",
      "postDate": "11/04/2012 15:09:43",
      "content": "<p>[quote=Foxtrot;16571]</p>\r\n<p>I didn't re-train on new data. Ouch.</p>\r\n<p>[/quote]</p>\r\n<p>Me neither - yup, wrong decision.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 16554,
      "author_name": "jsn1313",
      "author_url": "",
      "post_date": "11/04/2012 01:18:53",
      "content": "<p>Ouch. Sorry to hear that. Why such a difference, though? Is the better one made from retrained model, and the worse one from old model?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16556,
      "author_name": "andysloane",
      "author_url": "",
      "post_date": "11/04/2012 02:25:00",
      "content": "<p>They were both re-trained models, but the later one included the feature &quot;was posted in october 2012&quot; (and &quot;august 2012&quot; as well) while the better one had everything july and after sharing the same feature, which is what I did for the public leaderboard\r\n part. This essentially gave me a different &quot;prior&quot; for each month.</p>\r\n<p>But I suspect something went drastically wrong since it's worse than the prior benchmark, and I wasn't keeping good enough notes while making my final submission. The two should have been roughly the same. I probably mixed up a model file somewhere; I'm\r\n retracing my steps. Sigh.</p>\r\n<p>edit: yeah, oops. &nbsp;I know what happened. &nbsp;I had cut out the first couple weeks of october as a validation set, then built the training set without that, and the 2012/10 feature thus never appeared in the training set, and so when it was encountered in a\r\n test example it totally failed to correct for that bias. &nbsp;ARGH. &nbsp;I have all sorts of tools that could have detected that, but I didn't spend enough time checking my solution.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16557,
      "author_name": "jsn1313",
      "author_url": "",
      "post_date": "11/04/2012 02:37:22",
      "content": "<p>Hmm. I was under impression that changes like that (adding features, tweaking parameters, etc) are prohibited by rules at this stage. I mean, the model's code was supposed to be frozen when public leaderboard closed. Am I wrong?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16559,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "11/04/2012 06:27:21",
      "content": "<p>yes, model code cannot be changed.<br>\r\nMy estimate based on my analysis was that with the new dataset everyone's LogLoss will be approximately 2x of earlier. Going by final leaderboard, that is correct!</p>\r\n<p>More than anything else, it is known that in case of severely imbalanced datasets such as this one, most models will over-predict the majority class. A metric other than LogLoss would have been more appropriate - something based on multi-class precision\r\n and recall</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16571,
      "author_name": "zygmunt",
      "author_url": "",
      "post_date": "11/04/2012 14:26:12",
      "content": "<p>I didn't re-train on new data. Ouch.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16572,
      "author_name": "andysloane",
      "author_url": "",
      "post_date": "11/04/2012 15:05:29",
      "content": "<p>[quote=jsn13;16557]</p>\r\n<p>Hmm. I was under impression that changes like that (adding features, tweaking parameters, etc) are prohibited by rules at this stage. I mean, the model's code was supposed to be frozen when public leaderboard closed. Am I wrong?</p>\r\n<p>[/quote]</p>\r\n<p>Yeah, well, it was a bit unclear. &nbsp;I mean, I bumped the cap on the yymm feature from July to October, and re-ran training, which added those features automatically. &nbsp;So maybe I wasn't supposed to do that? &nbsp;I'm not sure.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 16573,
      "author_name": "ephesus",
      "author_url": "",
      "post_date": "11/04/2012 15:09:43",
      "content": "<p>[quote=Foxtrot;16571]</p>\r\n<p>I didn't re-train on new data. Ouch.</p>\r\n<p>[/quote]</p>\r\n<p>Me neither - yup, wrong decision.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "16553": "",
    "16554": "",
    "16556": "",
    "16557": "",
    "16559": "",
    "16571": "",
    "16572": "",
    "16573": ""
  },
  "source": "meta"
}