{
  "id": 67194,
  "title": "Let's probe Public LB class weights ?",
  "url": "/competitions/PLAsTiCC-2018/discussion/67194",
  "author_name": "",
  "post_date": "2018-09-29T20:17:06.442765600Z",
  "votes": 51,
  "comment_count": 22,
  "views": 0,
  "content": "<p>Class / Public LB:</p>\n\n<ul>\n<li>99 / 30.701</li>\n<li>95 / 32.620</li>\n<li>92 / 32.620</li>\n<li>90 / 32.620</li>\n<li>88 / 32.620</li>\n<li>67 / 32.620</li>\n<li>65 / 32.620</li>\n<li>64 / 30.692</li>\n<li>62 / 32.620</li>\n<li>53 / 32.622</li>\n<li>52 / 32.620</li>\n<li>42 / 32.620</li>\n<li>16 / 32.620</li>\n<li>15 / 30.702</li>\n<li>6 / 32.620</li>\n</ul>",
  "messages": [
    {
      "id": "396007",
      "postDate": "09/29/2018 20:17:06",
      "content": "<p>Class / Public LB:</p>\n\n<ul>\n<li>99 / 30.701</li>\n<li>95 / 32.620</li>\n<li>92 / 32.620</li>\n<li>90 / 32.620</li>\n<li>88 / 32.620</li>\n<li>67 / 32.620</li>\n<li>65 / 32.620</li>\n<li>64 / 30.692</li>\n<li>62 / 32.620</li>\n<li>53 / 32.622</li>\n<li>52 / 32.620</li>\n<li>42 / 32.620</li>\n<li>16 / 32.620</li>\n<li>15 / 30.702</li>\n<li>6 / 32.620</li>\n</ul>",
      "rawMarkdown": "Class / Public LB:\n\n - 99 / 30.701\n - 95 / 32.620\n - 92 / 32.620\n - 90 / 32.620\n - 88 / 32.620\n - 67 / 32.620\n - 65 / 32.620\n - 64 / 30.692\n - 62 / 32.620\n - 53 / 32.622\n - 52 / 32.620\n - 42 / 32.620\n - 16 / 32.620\n - 15 / 30.702\n - 6 / 32.620",
      "votes": null
    },
    {
      "id": "396032",
      "postDate": "09/29/2018 22:05:18",
      "content": "<p>lol</p>\n\n<p>Kaggle staff: “We don’t want to post the weights because we think if we do disclose the weights it will encourage too much LB probing.”</p>\n\n<p>Kaggle users: “Let’s probe the LB to figure out the weights because they won’t tell us!”</p>",
      "rawMarkdown": "lol\n\nKaggle staff: “We don’t want to post the weights because we think if we do disclose the weights it will encourage too much LB probing.”\n\nKaggle users: “Let’s probe the LB to figure out the weights because they won’t tell us!”",
      "votes": null
    },
    {
      "id": "396077",
      "postDate": "09/30/2018 01:40:06",
      "content": "<p>They had to have known this would happen. They know who we are.</p>",
      "rawMarkdown": "They had to have known this would happen. They know who we are.",
      "votes": null
    },
    {
      "id": "396205",
      "postDate": "09/30/2018 08:19:11",
      "content": "<p><a href=\"https://www.kaggle.com/onodera/all-class-6\">https://www.kaggle.com/onodera/all-class-6</a></p>\n\n<ul>\n<li>6 /32.620</li>\n</ul>",
      "rawMarkdown": "https://www.kaggle.com/onodera/all-class-6\n\n* 6 /32.620",
      "votes": null
    },
    {
      "id": "396315",
      "postDate": "09/30/2018 12:56:29",
      "content": "<p><a href=\"https://www.kaggle.com/paulorzp/class-92-1\">https://www.kaggle.com/paulorzp/class-92-1</a></p>\n\n<p>92 / 32.620</p>\n\n<p>95 / 32.620</p>",
      "rawMarkdown": "https://www.kaggle.com/paulorzp/class-92-1\n\n92 / 32.620\n\n95 / 32.620",
      "votes": null
    },
    {
      "id": "396402",
      "postDate": "09/30/2018 17:20:30",
      "content": "<p>So far, looks like only class 99 is not perfectly balanced</p>",
      "rawMarkdown": "So far, looks like only class 99 is not perfectly balanced",
      "votes": null
    },
    {
      "id": "396518",
      "postDate": "09/30/2018 23:25:10",
      "content": "<p>my submission result:\nthe class 53 result looks a bit odd for me..\n<br> <br>\nclass 42 : 32.620 (corrected ! )<br>\nclass 52 : 32.620 (corrected ! ) <br>\nclass 53 : 32.622 (corrected ! )<br>\nclass 62 : 32.620 (corrected ! )<br>\nclass 64 : 30.692 (corrected ! ) <br></p>",
      "rawMarkdown": "my submission result:\nthe class 53 result looks a bit odd for me..\n<br> <br>\nclass 42 : 32.620 (corrected ! )<br>\nclass 52 : 32.620 (corrected ! ) <br>\nclass 53 : 32.622 (corrected ! )<br>\nclass 62 : 32.620 (corrected ! )<br>\nclass 64 : 30.692 (corrected ! ) <br>",
      "votes": null
    },
    {
      "id": "396558",
      "postDate": "10/01/2018 02:05:45",
      "content": "<p>Thanks for the collaboration <a href=\"/mamasinkgs\">@mamasinkgs</a>, <a href=\"/paulorzp\">@paulorzp</a> and <a href=\"/onodera\">@onodera</a> :-)</p>",
      "rawMarkdown": "Thanks for the collaboration @mamasinkgs, @paulorzp and @onodera :-)",
      "votes": null
    },
    {
      "id": "396559",
      "postDate": "10/01/2018 02:07:10",
      "content": "<p>Looks like 9 classes have exact the same weight: 6, 16, 62, 65, 67, 88, 90, 92 and 95</p>",
      "rawMarkdown": "Looks like 9 classes have exact the same weight: 6, 16, 62, 65, 67, 88, 90, 92 and 95",
      "votes": null
    },
    {
      "id": "396786",
      "postDate": "10/01/2018 11:32:32",
      "content": "<p>oh, my code contained an unbelievable easy bug... I will submit again :( <br>\nNote : already corrected</p>",
      "rawMarkdown": "oh, my code contained an unbelievable easy bug... I will submit again :( <br>\nNote : already corrected",
      "votes": null
    },
    {
      "id": "396830",
      "postDate": "10/01/2018 12:57:20",
      "content": "<p>Me too</p>",
      "rawMarkdown": "Me too",
      "votes": null
    },
    {
      "id": "397146",
      "postDate": "10/02/2018 01:22:40",
      "content": "<p>So working this out, most classes have a weight of 1.9188. Classes 99, 64 and 15 have double the standard weight which is 3.8376. This fits the measurements with a little bit of jitter at the 0.002 level.</p>",
      "rawMarkdown": "So working this out, most classes have a weight of 1.9188. Classes 99, 64 and 15 have double the standard weight which is 3.8376. This fits the measurements with a little bit of jitter at the 0.002 level.",
      "votes": null
    },
    {
      "id": "397153",
      "postDate": "10/02/2018 01:45:00",
      "content": "<p>To follow up on this, you can simply use weights of 1 for most classes and 2 for classes 99, 64 and 15 in the provided metric since the overall scaling cancels out. The strange final number comes from -log(1e-15) = 34.5388. If you remove one unit of weight, you get -log(1e-15) * 17 / 18 = 32.620. Removing two units gives -log(1e-15) * 16 / 18 = 30.701.</p>",
      "rawMarkdown": "To follow up on this, you can simply use weights of 1 for most classes and 2 for classes 99, 64 and 15 in the provided metric since the overall scaling cancels out. The strange final number comes from -log(1e-15) = 34.5388. If you remove one unit of weight, you get -log(1e-15) * 17 / 18 = 32.620. Removing two units gives -log(1e-15) * 16 / 18 = 30.701.",
      "votes": null
    },
    {
      "id": "407782",
      "postDate": "10/21/2018 18:39:47",
      "content": "<p>Awesome, this community is amazing!</p>",
      "rawMarkdown": "Awesome, this community is amazing!",
      "votes": null
    },
    {
      "id": "409633",
      "postDate": "10/24/2018 16:13:09",
      "content": "<p>I just submitted a constant prediction to independently confirm that weights are 1/18 for all class except for classes 15, 64, and 99 where is is 1/9.  If you minimize loss with constant probabilities per class, then the min loss is obtained when each class probability is equal to its weight.  The loss value then becomes:</p>\n\n<pre><code>-[12 log(1/18) + 6 log(1/9)] / 18 = [2 log(18) + log(9)] / 3 = 2.659\n</code></pre>\n\n<p>This is what I just scored.</p>\n\n<p>PS. For @Olivier, I managed to get in top 1000 ;)</p>\n\n<p>Edited to fix typos.  Thanks to Glimmung for pointing to the issue.</p>",
      "rawMarkdown": "I just submitted a constant prediction to independently confirm that weights are 1/18 for all class except for classes 15, 64, and 99 where is is 1/9.  If you minimize loss with constant probabilities per class, then the min loss is obtained when each class probability is equal to its weight.  The loss value then becomes:\n\n    -[12 log(1/18) + 6 log(1/9)] / 18 = [2 log(18) + log(9)] / 3 = 2.659\n\nThis is what I just scored.\n\nPS. For @Olivier, I managed to get in top 1000 ;)\n\nEdited to fix typos.  Thanks to Glimmung for pointing to the issue.",
      "votes": null
    },
    {
      "id": "409687",
      "postDate": "10/24/2018 17:41:06",
      "content": "<p>To be picky: the first is -47.9, the second is .443</p>\n\n<p>Perhaps 2/3 log(2) + 2 log(3) = 2.659 ?</p>",
      "rawMarkdown": "To be picky: the first is -47.9, the second is .443\n\nPerhaps 2/3 log(2) + 2 log(3) = 2.659 ?",
      "votes": null
    },
    {
      "id": "409695",
      "postDate": "10/24/2018 17:58:01",
      "content": "<blockquote>\n  <p>the first is -47.9, the second is .443</p>\n</blockquote>\n\n<p>??</p>\n\n<p>Are you using log base 10?</p>",
      "rawMarkdown": "&gt; the first is -47.9, the second is .443\n\n??\n\nAre you using log base 10?",
      "votes": null
    },
    {
      "id": "409705",
      "postDate": "10/24/2018 18:14:46",
      "content": "<p>Log2. -[12 log(1/18) + 6 log(1/9)] = 47.9 ??</p>",
      "rawMarkdown": "Log2. -[12 log(1/18) + 6 log(1/9)] = 47.9 ??",
      "votes": null
    },
    {
      "id": "409706",
      "postDate": "10/24/2018 18:17:36",
      "content": "<p>[2 log(18) + log(9)] / 18 = 0.443 ??</p>",
      "rawMarkdown": "[2 log(18) + log(9)] / 18 = 0.443 ??",
      "votes": null
    },
    {
      "id": "409707",
      "postDate": "10/24/2018 18:18:33",
      "content": "<p>Sorry, fixed  my typo.\n This is using natural logarithm.</p>",
      "rawMarkdown": "Sorry, fixed  my typo.\n This is using natural logarithm.",
      "votes": null
    },
    {
      "id": "414607",
      "postDate": "11/03/2018 06:31:11",
      "content": "<p>Hey <a href=\"/titericz\">@titericz</a>. This might be a dumb question. But what exactly do you mean by LB probing of class weights?</p>",
      "rawMarkdown": "Hey @titericz. This might be a dumb question. But what exactly do you mean by LB probing of class weights?",
      "votes": null
    },
    {
      "id": "414776",
      "postDate": "11/03/2018 15:30:44",
      "content": "<p>You submit a prediction where all objects are predicted to be of the same class and use the resulting LB score.  Explanations are provided in other responses in this topic, esp <a href=\"https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#397153\">https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#397153</a></p>",
      "rawMarkdown": "You submit a prediction where all objects are predicted to be of the same class and use the resulting LB score.  Explanations are provided in other responses in this topic, esp https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#397153",
      "votes": null
    },
    {
      "id": "414793",
      "postDate": "11/03/2018 16:04:36",
      "content": "<p>Oh nice. Thanks <a href=\"/cpmpml\">@cpmpml</a>.</p>",
      "rawMarkdown": "Oh nice. Thanks @cpmpml.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 396032,
      "author_name": "peterhurford",
      "author_url": "",
      "post_date": "09/29/2018 22:05:18",
      "content": "<p>lol</p>\n\n<p>Kaggle staff: “We don’t want to post the weights because we think if we do disclose the weights it will encourage too much LB probing.”</p>\n\n<p>Kaggle users: “Let’s probe the LB to figure out the weights because they won’t tell us!”</p>",
      "votes": null,
      "replies": [
        {
          "id": 396077,
          "author_name": "nnnnick",
          "author_url": "",
          "post_date": "09/30/2018 01:40:06",
          "content": "<p>They had to have known this would happen. They know who we are.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 396205,
      "author_name": "onodera",
      "author_url": "",
      "post_date": "09/30/2018 08:19:11",
      "content": "<p><a href=\"https://www.kaggle.com/onodera/all-class-6\">https://www.kaggle.com/onodera/all-class-6</a></p>\n\n<ul>\n<li>6 /32.620</li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 396315,
      "author_name": "paulorzp",
      "author_url": "",
      "post_date": "09/30/2018 12:56:29",
      "content": "<p><a href=\"https://www.kaggle.com/paulorzp/class-92-1\">https://www.kaggle.com/paulorzp/class-92-1</a></p>\n\n<p>92 / 32.620</p>\n\n<p>95 / 32.620</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 396402,
      "author_name": "titericz",
      "author_url": "",
      "post_date": "09/30/2018 17:20:30",
      "content": "<p>So far, looks like only class 99 is not perfectly balanced</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 396518,
      "author_name": "mamasinkgs",
      "author_url": "",
      "post_date": "09/30/2018 23:25:10",
      "content": "<p>my submission result:\nthe class 53 result looks a bit odd for me..\n<br> <br>\nclass 42 : 32.620 (corrected ! )<br>\nclass 52 : 32.620 (corrected ! ) <br>\nclass 53 : 32.622 (corrected ! )<br>\nclass 62 : 32.620 (corrected ! )<br>\nclass 64 : 30.692 (corrected ! ) <br></p>",
      "votes": null,
      "replies": [
        {
          "id": 396786,
          "author_name": "mamasinkgs",
          "author_url": "",
          "post_date": "10/01/2018 11:32:32",
          "content": "<p>oh, my code contained an unbelievable easy bug... I will submit again :( <br>\nNote : already corrected</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 396558,
      "author_name": "titericz",
      "author_url": "",
      "post_date": "10/01/2018 02:05:45",
      "content": "<p>Thanks for the collaboration <a href=\"/mamasinkgs\">@mamasinkgs</a>, <a href=\"/paulorzp\">@paulorzp</a> and <a href=\"/onodera\">@onodera</a> :-)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 396559,
      "author_name": "titericz",
      "author_url": "",
      "post_date": "10/01/2018 02:07:10",
      "content": "<p>Looks like 9 classes have exact the same weight: 6, 16, 62, 65, 67, 88, 90, 92 and 95</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 396830,
      "author_name": "pavansanagapati",
      "author_url": "",
      "post_date": "10/01/2018 12:57:20",
      "content": "<p>Me too</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 397146,
      "author_name": "kyleboone",
      "author_url": "",
      "post_date": "10/02/2018 01:22:40",
      "content": "<p>So working this out, most classes have a weight of 1.9188. Classes 99, 64 and 15 have double the standard weight which is 3.8376. This fits the measurements with a little bit of jitter at the 0.002 level.</p>",
      "votes": null,
      "replies": [
        {
          "id": 397153,
          "author_name": "kyleboone",
          "author_url": "",
          "post_date": "10/02/2018 01:45:00",
          "content": "<p>To follow up on this, you can simply use weights of 1 for most classes and 2 for classes 99, 64 and 15 in the provided metric since the overall scaling cancels out. The strange final number comes from -log(1e-15) = 34.5388. If you remove one unit of weight, you get -log(1e-15) * 17 / 18 = 32.620. Removing two units gives -log(1e-15) * 16 / 18 = 30.701.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 407782,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "10/21/2018 18:39:47",
      "content": "<p>Awesome, this community is amazing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 409633,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "10/24/2018 16:13:09",
      "content": "<p>I just submitted a constant prediction to independently confirm that weights are 1/18 for all class except for classes 15, 64, and 99 where is is 1/9.  If you minimize loss with constant probabilities per class, then the min loss is obtained when each class probability is equal to its weight.  The loss value then becomes:</p>\n\n<pre><code>-[12 log(1/18) + 6 log(1/9)] / 18 = [2 log(18) + log(9)] / 3 = 2.659\n</code></pre>\n\n<p>This is what I just scored.</p>\n\n<p>PS. For @Olivier, I managed to get in top 1000 ;)</p>\n\n<p>Edited to fix typos.  Thanks to Glimmung for pointing to the issue.</p>",
      "votes": null,
      "replies": [
        {
          "id": 409687,
          "author_name": "glimmung",
          "author_url": "",
          "post_date": "10/24/2018 17:41:06",
          "content": "<p>To be picky: the first is -47.9, the second is .443</p>\n\n<p>Perhaps 2/3 log(2) + 2 log(3) = 2.659 ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 409695,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "10/24/2018 17:58:01",
          "content": "<blockquote>\n  <p>the first is -47.9, the second is .443</p>\n</blockquote>\n\n<p>??</p>\n\n<p>Are you using log base 10?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 409705,
          "author_name": "glimmung",
          "author_url": "",
          "post_date": "10/24/2018 18:14:46",
          "content": "<p>Log2. -[12 log(1/18) + 6 log(1/9)] = 47.9 ??</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 409706,
          "author_name": "glimmung",
          "author_url": "",
          "post_date": "10/24/2018 18:17:36",
          "content": "<p>[2 log(18) + log(9)] / 18 = 0.443 ??</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 409707,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "10/24/2018 18:18:33",
          "content": "<p>Sorry, fixed  my typo.\n This is using natural logarithm.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 414607,
      "author_name": "ayerajath",
      "author_url": "",
      "post_date": "11/03/2018 06:31:11",
      "content": "<p>Hey <a href=\"/titericz\">@titericz</a>. This might be a dumb question. But what exactly do you mean by LB probing of class weights?</p>",
      "votes": null,
      "replies": [
        {
          "id": 414776,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "11/03/2018 15:30:44",
          "content": "<p>You submit a prediction where all objects are predicted to be of the same class and use the resulting LB score.  Explanations are provided in other responses in this topic, esp <a href=\"https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#397153\">https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#397153</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 414793,
          "author_name": "ayerajath",
          "author_url": "",
          "post_date": "11/03/2018 16:04:36",
          "content": "<p>Oh nice. Thanks <a href=\"/cpmpml\">@cpmpml</a>.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "396007": "Class / Public LB:\n\n - 99 / 30.701\n - 95 / 32.620\n - 92 / 32.620\n - 90 / 32.620\n - 88 / 32.620\n - 67 / 32.620\n - 65 / 32.620\n - 64 / 30.692\n - 62 / 32.620\n - 53 / 32.622\n - 52 / 32.620\n - 42 / 32.620\n - 16 / 32.620\n - 15 / 30.702\n - 6 / 32.620",
    "396032": "lol\n\nKaggle staff: “We don’t want to post the weights because we think if we do disclose the weights it will encourage too much LB probing.”\n\nKaggle users: “Let’s probe the LB to figure out the weights because they won’t tell us!”",
    "396077": "They had to have known this would happen. They know who we are.",
    "396205": "https://www.kaggle.com/onodera/all-class-6\n\n* 6 /32.620",
    "396315": "https://www.kaggle.com/paulorzp/class-92-1\n\n92 / 32.620\n\n95 / 32.620",
    "396402": "So far, looks like only class 99 is not perfectly balanced",
    "396518": "my submission result:\nthe class 53 result looks a bit odd for me..\n<br> <br>\nclass 42 : 32.620 (corrected ! )<br>\nclass 52 : 32.620 (corrected ! ) <br>\nclass 53 : 32.622 (corrected ! )<br>\nclass 62 : 32.620 (corrected ! )<br>\nclass 64 : 30.692 (corrected ! ) <br>",
    "396558": "Thanks for the collaboration @mamasinkgs, @paulorzp and @onodera :-)",
    "396559": "Looks like 9 classes have exact the same weight: 6, 16, 62, 65, 67, 88, 90, 92 and 95",
    "396786": "oh, my code contained an unbelievable easy bug... I will submit again :( <br>\nNote : already corrected",
    "396830": "Me too",
    "397146": "So working this out, most classes have a weight of 1.9188. Classes 99, 64 and 15 have double the standard weight which is 3.8376. This fits the measurements with a little bit of jitter at the 0.002 level.",
    "397153": "To follow up on this, you can simply use weights of 1 for most classes and 2 for classes 99, 64 and 15 in the provided metric since the overall scaling cancels out. The strange final number comes from -log(1e-15) = 34.5388. If you remove one unit of weight, you get -log(1e-15) * 17 / 18 = 32.620. Removing two units gives -log(1e-15) * 16 / 18 = 30.701.",
    "407782": "Awesome, this community is amazing!",
    "409633": "I just submitted a constant prediction to independently confirm that weights are 1/18 for all class except for classes 15, 64, and 99 where is is 1/9.  If you minimize loss with constant probabilities per class, then the min loss is obtained when each class probability is equal to its weight.  The loss value then becomes:\n\n    -[12 log(1/18) + 6 log(1/9)] / 18 = [2 log(18) + log(9)] / 3 = 2.659\n\nThis is what I just scored.\n\nPS. For @Olivier, I managed to get in top 1000 ;)\n\nEdited to fix typos.  Thanks to Glimmung for pointing to the issue.",
    "409687": "To be picky: the first is -47.9, the second is .443\n\nPerhaps 2/3 log(2) + 2 log(3) = 2.659 ?",
    "409695": "&gt; the first is -47.9, the second is .443\n\n??\n\nAre you using log base 10?",
    "409705": "Log2. -[12 log(1/18) + 6 log(1/9)] = 47.9 ??",
    "409706": "[2 log(18) + log(9)] / 18 = 0.443 ??",
    "409707": "Sorry, fixed  my typo.\n This is using natural logarithm.",
    "414607": "Hey @titericz. This might be a dumb question. But what exactly do you mean by LB probing of class weights?",
    "414776": "You submit a prediction where all objects are predicted to be of the same class and use the resulting LB score.  Explanations are provided in other responses in this topic, esp https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194#397153",
    "414793": "Oh nice. Thanks @cpmpml."
  },
  "source": "meta"
}