{
  "id": 14580,
  "title": "histCTR",
  "url": "/competitions/avito-context-ad-clicks/discussion/14580",
  "author_name": "",
  "post_date": "2015-06-06T09:38:50.690Z",
  "votes": 4,
  "comment_count": 3,
  "views": 1315,
  "content": "<p>I don't know how it is computed, but it clearly reminds me Russian government elections.</p>\n<p>I computed statistics on relations of histCTR and the probability of click.&nbsp;</p>\n<p>In the attached file histCTR is the x-axis and the actual probability of click is the y-axis.</p>\n<p>You can surely see these strange jumps around 0.05, 0.1 and 0.15 histCTR values.</p>\n<p>I still can't understand how to write my own script in the scripts section so I also upload my python scripts here (you can run them simply with</p>\n<p>&quot;python postprocessing.py&quot;)</p>",
  "messages": [
    {
      "id": "81102",
      "postDate": "06/06/2015 09:38:50",
      "content": "<p>I don't know how it is computed, but it clearly reminds me Russian government elections.</p>\n<p>I computed statistics on relations of histCTR and the probability of click.&nbsp;</p>\n<p>In the attached file histCTR is the x-axis and the actual probability of click is the y-axis.</p>\n<p>You can surely see these strange jumps around 0.05, 0.1 and 0.15 histCTR values.</p>\n<p>I still can't understand how to write my own script in the scripts section so I also upload my python scripts here (you can run them simply with</p>\n<p>&quot;python postprocessing.py&quot;)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81269",
      "postDate": "06/08/2015 20:38:12",
      "content": "<p>I made some additional experiments and I found more strange things. Now they are about the leaderboard evaluation metric.</p>\n<p>I validated simple submissions like histCTR, all zeros and so on along with different bounding values (which are the part of the metric, preventing it to be infinity) and I got very strange results.</p>\n<p>Boundary threshold = 0.001<br>HistCTR: 0.0385527977722<br>All zeros: 0.0432560260702<br>All 0.01: 0.0381632790961<br>All 0.1: 0.118803135624</p>\n<p>Boundary threshold = 0.001<br>HistCTR: 0.0385527977722<br>All zeros: 0.0432560260702<br>All 0.01: 0.0381632790961<br>All 0.1: 0.118803135624</p>\n<p>Boundary threshold = 1e-09<br>HistCTR: 0.0398330744313<br>All zeros: 0.126784941385<br>All 0.01: 0.0381632790961<br>All 0.1: 0.118803135624</p>\n<p>As we see, they are significantly better than corresponding leaderboard benchmarks. Moreover, validation results on a little bit postprocessed histCTR which get 0.05005 at the leaderboard (my current best result) are also significantly better (something around 0.036). So optimizing validation score helps to optimize leaderboard score but in a tricky way.</p>\n\n<p>Does anyone have the idea what is going on?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81401",
      "postDate": "06/09/2015 20:44:28",
      "content": "<p>I think we can infer&nbsp;that the number of clicks in the trainset is smaller than it is in the public testset</p>\n<p>The all zeros get worse with a smaller threshold because they are&nbsp;more different from 1 after applying the threshold (and large errors have large impact on the score); all 0.1 or 0.01 are already too far from 0 to be effected by the threshold</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "81423",
      "postDate": "06/09/2015 23:08:56",
      "content": "<p>Yes, I found at the data description page that a little bit special validation set is used at that contest, so my &quot;standard&quot; validation method is incorrect. I'm generating the right split to training and validation sets right now at my laptop (at least I hope so).</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 81269,
      "author_name": "agrin88",
      "author_url": "",
      "post_date": "06/08/2015 20:38:12",
      "content": "<p>I made some additional experiments and I found more strange things. Now they are about the leaderboard evaluation metric.</p>\n<p>I validated simple submissions like histCTR, all zeros and so on along with different bounding values (which are the part of the metric, preventing it to be infinity) and I got very strange results.</p>\n<p>Boundary threshold = 0.001<br>HistCTR: 0.0385527977722<br>All zeros: 0.0432560260702<br>All 0.01: 0.0381632790961<br>All 0.1: 0.118803135624</p>\n<p>Boundary threshold = 0.001<br>HistCTR: 0.0385527977722<br>All zeros: 0.0432560260702<br>All 0.01: 0.0381632790961<br>All 0.1: 0.118803135624</p>\n<p>Boundary threshold = 1e-09<br>HistCTR: 0.0398330744313<br>All zeros: 0.126784941385<br>All 0.01: 0.0381632790961<br>All 0.1: 0.118803135624</p>\n<p>As we see, they are significantly better than corresponding leaderboard benchmarks. Moreover, validation results on a little bit postprocessed histCTR which get 0.05005 at the leaderboard (my current best result) are also significantly better (something around 0.036). So optimizing validation score helps to optimize leaderboard score but in a tricky way.</p>\n\n<p>Does anyone have the idea what is going on?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81401,
      "author_name": "gertjac",
      "author_url": "",
      "post_date": "06/09/2015 20:44:28",
      "content": "<p>I think we can infer&nbsp;that the number of clicks in the trainset is smaller than it is in the public testset</p>\n<p>The all zeros get worse with a smaller threshold because they are&nbsp;more different from 1 after applying the threshold (and large errors have large impact on the score); all 0.1 or 0.01 are already too far from 0 to be effected by the threshold</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 81423,
      "author_name": "agrin88",
      "author_url": "",
      "post_date": "06/09/2015 23:08:56",
      "content": "<p>Yes, I found at the data description page that a little bit special validation set is used at that contest, so my &quot;standard&quot; validation method is incorrect. I'm generating the right split to training and validation sets right now at my laptop (at least I hope so).</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "81102": "",
    "81269": "",
    "81401": "",
    "81423": ""
  },
  "source": "meta"
}