{
  "id": 71416,
  "title": "what's your estimate of shakeup?",
  "url": "/competitions/airbus-ship-detection/discussion/71416",
  "author_name": "Peiyuan Liao",
  "post_date": "2018-11-13T13:41:06.233000",
  "votes": 3,
  "comment_count": 19,
  "views": 0,
  "content": "<p>Seems that all of us are really close, and due to how private/public data is split, I wonder if anyone has done any analysis on the magnitude of the shakeup.</p>",
  "messages": [
    {
      "id": 420370,
      "postDate": "2018-11-13T14:05:23.450Z",
      "content": "<p>My estimation is those who used only public LB score as a performance metric may end up 30-50 positions lower on the private LB.</p>",
      "rawMarkdown": "My estimation is those who used only public LB score as a performance metric may end up 30-50 positions lower on the private LB.",
      "votes": 6,
      "replies": [
        {
          "id": 421404,
          "postDate": "2018-11-15T00:45:12.610Z",
          "content": "<p>Nostradamus! haha</p>\n\n<p>Congratulations :)</p>",
          "rawMarkdown": "Nostradamus! haha\n\nCongratulations :)"
        },
        {
          "id": 421582,
          "postDate": "2018-11-15T06:51:36.380Z",
          "content": "<p>Thanks! :)</p>",
          "rawMarkdown": "Thanks! :)"
        }
      ]
    },
    {
      "id": 421369,
      "postDate": "2018-11-15T00:10:28.610Z",
      "content": "<p>selecting right submission was most important, I missed submission of private score 0.850...</p>",
      "rawMarkdown": "selecting right submission was most important, I missed submission of private score 0.850...",
      "votes": 3,
      "replies": [
        {
          "id": 421373,
          "postDate": "2018-11-15T00:18:48.740Z",
          "content": "<p>True, I also missed my best submission. Really hard to choose subs because of inconsistencies while validating on public LB</p>",
          "rawMarkdown": "True, I also missed my best submission. Really hard to choose subs because of inconsistencies while validating on public LB",
          "votes": 1
        },
        {
          "id": 421374,
          "postDate": "2018-11-15T00:21:04.097Z",
          "content": "<p>I feel like this public leaderboard mislead us quite a lot</p>",
          "rawMarkdown": "I feel like this public leaderboard mislead us quite a lot",
          "votes": 1
        },
        {
          "id": 421390,
          "postDate": "2018-11-15T00:33:16.610Z",
          "content": "<p>I agree with you, I ended up submitted top 40th private score.</p>",
          "rawMarkdown": "I agree with you, I ended up submitted top 40th private score."
        },
        {
          "id": 421400,
          "postDate": "2018-11-15T00:39:28.793Z",
          "content": "<p>we had a top 30 submission......</p>",
          "rawMarkdown": "we had a top 30 submission......",
          "votes": 1
        },
        {
          "id": 421402,
          "postDate": "2018-11-15T00:44:23.727Z",
          "content": "<p>I got fooled as well, had a 0.851 sub haha</p>\n\n<p>But it is part of the game. The public was just 12% of the data and training images also had overlap, so care should be taken regarding that</p>",
          "rawMarkdown": "I got fooled as well, had a 0.851 sub haha\n\nBut it is part of the game. The public was just 12% of the data and training images also had overlap, so care should be taken regarding that",
          "votes": 1
        },
        {
          "id": 421403,
          "postDate": "2018-11-15T00:44:28.040Z",
          "content": "<p>again, what a shake up</p>",
          "rawMarkdown": "again, what a shake up"
        },
        {
          "id": 421405,
          "postDate": "2018-11-15T00:47:16.860Z",
          "content": "<p>Had a 0.848 submission, but the public score has baited a lot of participants to do a wrong submission!! </p>",
          "rawMarkdown": "Had a 0.848 submission, but the public score has baited a lot of participants to do a wrong submission!! ",
          "votes": 1
        },
        {
          "id": 421486,
          "postDate": "2018-11-15T04:04:16.893Z",
          "content": "<p>That‘s true！We are lucky to choose the sub only losing 3 place in LB. Actually pseudo labeling worked in this competition (single model 853 in private), but we didn't choose that.</p>",
          "rawMarkdown": "That‘s true！We are lucky to choose the sub only losing 3 place in LB. Actually pseudo labeling worked in this competition (single model 853 in private), but we didn't choose that.",
          "votes": 1
        },
        {
          "id": 421500,
          "postDate": "2018-11-15T04:30:00.173Z",
          "content": "<p>It's really frustrated that got worse private grade than LB grade ....\nI beat the 0.699 LB score that from lafoss kernel, but get the worse private score.\nThe grade suddenly drop a lot .... </p>",
          "rawMarkdown": "It's really frustrated that got worse private grade than LB grade ....\nI beat the 0.699 LB score that from lafoss kernel, but get the worse private score.\nThe grade suddenly drop a lot .... ",
          "votes": 2
        },
        {
          "id": 421506,
          "postDate": "2018-11-15T04:43:29.020Z",
          "content": "<p>I missed selecting my best model's submission as well because I thought it is over-fitting. Another reason I did not trust it is because the result was inconsistent.</p>\n\n<p>In this competition, I must admit I was discouraged by the leak earlier on so decided not to spend too much time on it. I did something similar to what @Lafoss did in <a href=\"https://www.kaggle.com/iafoss/fine-tuning-resnet34-on-ship-detection/notebook\">this notebook</a> before the leak. So I adapted it to the new data and submitted.</p>",
          "rawMarkdown": "I missed selecting my best model's submission as well because I thought it is over-fitting. Another reason I did not trust it is because the result was inconsistent.\n\nIn this competition, I must admit I was discouraged by the leak earlier on so decided not to spend too much time on it. I did something similar to what @Lafoss did in [this notebook][1] before the leak. So I adapted it to the new data and submitted.\n\n\n  [1]: https://www.kaggle.com/iafoss/fine-tuning-resnet34-on-ship-detection/notebook",
          "votes": 1
        }
      ]
    },
    {
      "id": 421361,
      "postDate": "2018-11-15T00:03:15.843Z",
      "content": "<p>what a shakeup</p>",
      "rawMarkdown": "what a shakeup",
      "votes": 3,
      "replies": [
        {
          "id": 421365,
          "postDate": "2018-11-15T00:06:04.507Z",
          "content": "<p>indeed</p>",
          "rawMarkdown": "indeed",
          "votes": 1
        }
      ]
    },
    {
      "id": 420353,
      "postDate": "2018-11-13T13:41:06.233Z",
      "content": "<p>Seems that all of us are really close, and due to how private/public data is split, I wonder if anyone has done any analysis on the magnitude of the shakeup.</p>",
      "rawMarkdown": "Seems that all of us are really close, and due to how private/public data is split, I wonder if anyone has done any analysis on the magnitude of the shakeup.",
      "votes": 3
    },
    {
      "id": 421463,
      "postDate": "2018-11-15T03:31:43.137Z",
      "content": "<p>Kind of survived from big shakeup, I believe it is always better to trust more on your local validation than public LB, especially when there is only <code>12%</code> of test data used.</p>\n\n<p>Though it is also very important to remove overlapping images from validation set. I used a <code>66/33</code> split (<code>3 folds</code>) and removed overlapping images for each fold (kudos to <a href=\"/iafoss\">@iafoss</a> 's <a href=\"https://www.kaggle.com/iafoss/list-of-overlapping-images-for-validation-set/notebook\">kernel</a>)</p>\n\n<p>It turns out that my 2nd best model (<code>0.84428</code> on private LB) is also the best local validated single model (<code>0.840686</code> on local) on only one fold.</p>\n\n<p>Interestingly model ensembling seemed not adding too much value and also had diverse results on LB.\nMy ensemble model of 3 best ones on public LB (<code>0.73028</code>) dropped to only <code>0.84147</code> on private one.</p>\n\n<p>Here are some more comparisons to show that public LB is mostly misleading:</p>\n\n<pre>Local        Public       Private\n0.799186     0.71529      0.83770\n0.801906     0.72464      0.84026\n0.822382     0.71658      0.84243\n0.827833     0.71731      0.84185\n0.840686     0.72397      0.84428\n</pre>\n\n<p>Also glad about not being beaten by the <code>neptune.ml</code> team, who is sharing competitive solutions publicly everywhere before the end of competition and made the life of other hard working kagglers much more difficult to get a medal.</p>\n\n<p>Btw, I used <code>UNet+ResNet34</code> with <code>hypercolumns</code>, <code>lovasz loss</code> and <code>deep supervision</code> modified from the great <a href=\"https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/69101\">fastai solution</a> on pytorch shared by <a href=\"/vishnus\">@vishnus</a> in the salt competition. Some changes were made to do inference by batches to avoid OOM on such a large number of images. Also applied progressive learning trick learnt from fast.ai course to train images from 256x256 to 384x384, 512x512 and finally 768x768, though the best model only trained on 384x384 resolution.</p>\n\n<p>Burned out lots of <code>K80</code>, <code>P100</code> and <code>V100</code> GPUs on Google Cloud as well...</p>",
      "rawMarkdown": "Kind of survived from big shakeup, I believe it is always better to trust more on your local validation than public LB, especially when there is only `12%` of test data used.\n\nThough it is also very important to remove overlapping images from validation set. I used a `66/33` split (`3 folds`) and removed overlapping images for each fold (kudos to @iafoss 's [kernel](https://www.kaggle.com/iafoss/list-of-overlapping-images-for-validation-set/notebook))\n\nIt turns out that my 2nd best model (`0.84428` on private LB) is also the best local validated single model (`0.840686` on local) on only one fold.\n\nInterestingly model ensembling seemed not adding too much value and also had diverse results on LB.\nMy ensemble model of 3 best ones on public LB (`0.73028`) dropped to only `0.84147` on private one.\n\nHere are some more comparisons to show that public LB is mostly misleading:\n\n<pre>Local        Public       Private\n0.799186     0.71529      0.83770\n0.801906     0.72464      0.84026\n0.822382     0.71658      0.84243\n0.827833     0.71731      0.84185\n0.840686     0.72397      0.84428\n</pre>\n\nAlso glad about not being beaten by the `neptune.ml` team, who is sharing competitive solutions publicly everywhere before the end of competition and made the life of other hard working kagglers much more difficult to get a medal.\n\nBtw, I used `UNet+ResNet34` with `hypercolumns`, `lovasz loss` and `deep supervision` modified from the great [fastai solution](https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/69101) on pytorch shared by @vishnus in the salt competition. Some changes were made to do inference by batches to avoid OOM on such a large number of images. Also applied progressive learning trick learnt from fast.ai course to train images from 256x256 to 384x384, 512x512 and finally 768x768, though the best model only trained on 384x384 resolution.\n\nBurned out lots of `K80`, `P100` and `V100` GPUs on Google Cloud as well...\n",
      "votes": 2
    },
    {
      "id": 420638,
      "postDate": "2018-11-13T23:33:19.720Z",
      "content": "<p>Everyone's score will go up for sure, probably in the 0.9+. </p>\n\n<p>Position-wise I expect a huge shake-up, I don't know where I will end up honestly, but the very top guys will remain top.</p>",
      "rawMarkdown": "Everyone's score will go up for sure, probably in the 0.9+. \n\nPosition-wise I expect a huge shake-up, I don't know where I will end up honestly, but the very top guys will remain top."
    },
    {
      "id": 420377,
      "postDate": "2018-11-13T14:16:23.343Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 420370,
      "author_name": "Dmytro Danevskyi",
      "author_url": "",
      "post_date": "2018-11-13T14:05:23.450000",
      "content": "<p>My estimation is those who used only public LB score as a performance metric may end up 30-50 positions lower on the private LB.</p>",
      "votes": 6,
      "replies": [
        {
          "id": 421404,
          "author_name": "Eduardo Rocha de Andrade",
          "author_url": "",
          "post_date": "2018-11-15T00:45:12.610000",
          "content": "<p>Nostradamus! haha</p>\n\n<p>Congratulations :)</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 421582,
          "author_name": "Dmytro Danevskyi",
          "author_url": "",
          "post_date": "2018-11-15T06:51:36.380000",
          "content": "<p>Thanks! :)</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 421369,
      "author_name": "Soonhwan Kwon",
      "author_url": "",
      "post_date": "2018-11-15T00:10:28.610000",
      "content": "<p>selecting right submission was most important, I missed submission of private score 0.850...</p>",
      "votes": 3,
      "replies": [
        {
          "id": 421373,
          "author_name": "Zito Relova",
          "author_url": "",
          "post_date": "2018-11-15T00:18:48.740000",
          "content": "<p>True, I also missed my best submission. Really hard to choose subs because of inconsistencies while validating on public LB</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 421374,
          "author_name": "MarcYueZhao",
          "author_url": "",
          "post_date": "2018-11-15T00:21:04.097000",
          "content": "<p>I feel like this public leaderboard mislead us quite a lot</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 421390,
          "author_name": "Soonhwan Kwon",
          "author_url": "",
          "post_date": "2018-11-15T00:33:16.610000",
          "content": "<p>I agree with you, I ended up submitted top 40th private score.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 421400,
          "author_name": "Peiyuan Liao",
          "author_url": "",
          "post_date": "2018-11-15T00:39:28.793000",
          "content": "<p>we had a top 30 submission......</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 421402,
          "author_name": "Eduardo Rocha de Andrade",
          "author_url": "",
          "post_date": "2018-11-15T00:44:23.727000",
          "content": "<p>I got fooled as well, had a 0.851 sub haha</p>\n\n<p>But it is part of the game. The public was just 12% of the data and training images also had overlap, so care should be taken regarding that</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 421403,
          "author_name": "Soonhwan Kwon",
          "author_url": "",
          "post_date": "2018-11-15T00:44:28.040000",
          "content": "<p>again, what a shake up</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 421405,
          "author_name": "PiGram",
          "author_url": "",
          "post_date": "2018-11-15T00:47:16.860000",
          "content": "<p>Had a 0.848 submission, but the public score has baited a lot of participants to do a wrong submission!! </p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 421486,
          "author_name": "SeuTao",
          "author_url": "",
          "post_date": "2018-11-15T04:04:16.893000",
          "content": "<p>That‘s true！We are lucky to choose the sub only losing 3 place in LB. Actually pseudo labeling worked in this competition (single model 853 in private), but we didn't choose that.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 421500,
          "author_name": "0xFunky",
          "author_url": "",
          "post_date": "2018-11-15T04:30:00.173000",
          "content": "<p>It's really frustrated that got worse private grade than LB grade ....\nI beat the 0.699 LB score that from lafoss kernel, but get the worse private score.\nThe grade suddenly drop a lot .... </p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 421506,
          "author_name": "YaGana Sheriff-Hussaini",
          "author_url": "",
          "post_date": "2018-11-15T04:43:29.020000",
          "content": "<p>I missed selecting my best model's submission as well because I thought it is over-fitting. Another reason I did not trust it is because the result was inconsistent.</p>\n\n<p>In this competition, I must admit I was discouraged by the leak earlier on so decided not to spend too much time on it. I did something similar to what @Lafoss did in <a href=\"https://www.kaggle.com/iafoss/fine-tuning-resnet34-on-ship-detection/notebook\">this notebook</a> before the leak. So I adapted it to the new data and submitted.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 421361,
      "author_name": "MarcYueZhao",
      "author_url": "",
      "post_date": "2018-11-15T00:03:15.843000",
      "content": "<p>what a shakeup</p>",
      "votes": 3,
      "replies": [
        {
          "id": 421365,
          "author_name": "Peiyuan Liao",
          "author_url": "",
          "post_date": "2018-11-15T00:06:04.507000",
          "content": "<p>indeed</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 421463,
      "author_name": "Mark Peng",
      "author_url": "",
      "post_date": "2018-11-15T03:31:43.137000",
      "content": "<p>Kind of survived from big shakeup, I believe it is always better to trust more on your local validation than public LB, especially when there is only <code>12%</code> of test data used.</p>\n\n<p>Though it is also very important to remove overlapping images from validation set. I used a <code>66/33</code> split (<code>3 folds</code>) and removed overlapping images for each fold (kudos to <a href=\"/iafoss\">@iafoss</a> 's <a href=\"https://www.kaggle.com/iafoss/list-of-overlapping-images-for-validation-set/notebook\">kernel</a>)</p>\n\n<p>It turns out that my 2nd best model (<code>0.84428</code> on private LB) is also the best local validated single model (<code>0.840686</code> on local) on only one fold.</p>\n\n<p>Interestingly model ensembling seemed not adding too much value and also had diverse results on LB.\nMy ensemble model of 3 best ones on public LB (<code>0.73028</code>) dropped to only <code>0.84147</code> on private one.</p>\n\n<p>Here are some more comparisons to show that public LB is mostly misleading:</p>\n\n<pre>Local        Public       Private\n0.799186     0.71529      0.83770\n0.801906     0.72464      0.84026\n0.822382     0.71658      0.84243\n0.827833     0.71731      0.84185\n0.840686     0.72397      0.84428\n</pre>\n\n<p>Also glad about not being beaten by the <code>neptune.ml</code> team, who is sharing competitive solutions publicly everywhere before the end of competition and made the life of other hard working kagglers much more difficult to get a medal.</p>\n\n<p>Btw, I used <code>UNet+ResNet34</code> with <code>hypercolumns</code>, <code>lovasz loss</code> and <code>deep supervision</code> modified from the great <a href=\"https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/69101\">fastai solution</a> on pytorch shared by <a href=\"/vishnus\">@vishnus</a> in the salt competition. Some changes were made to do inference by batches to avoid OOM on such a large number of images. Also applied progressive learning trick learnt from fast.ai course to train images from 256x256 to 384x384, 512x512 and finally 768x768, though the best model only trained on 384x384 resolution.</p>\n\n<p>Burned out lots of <code>K80</code>, <code>P100</code> and <code>V100</code> GPUs on Google Cloud as well...</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 420638,
      "author_name": "Khoi Nguyen",
      "author_url": "",
      "post_date": "2018-11-13T23:33:19.720000",
      "content": "<p>Everyone's score will go up for sure, probably in the 0.9+. </p>\n\n<p>Position-wise I expect a huge shake-up, I don't know where I will end up honestly, but the very top guys will remain top.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 420377,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-11-13T14:16:23.343000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "420370": "My estimation is those who used only public LB score as a performance metric may end up 30-50 positions lower on the private LB.",
    "421369": "selecting right submission was most important, I missed submission of private score 0.850...",
    "421361": "what a shakeup",
    "420353": "Seems that all of us are really close, and due to how private/public data is split, I wonder if anyone has done any analysis on the magnitude of the shakeup.",
    "421463": "Kind of survived from big shakeup, I believe it is always better to trust more on your local validation than public LB, especially when there is only `12%` of test data used.\n\nThough it is also very important to remove overlapping images from validation set. I used a `66/33` split (`3 folds`) and removed overlapping images for each fold (kudos to @iafoss 's [kernel](https://www.kaggle.com/iafoss/list-of-overlapping-images-for-validation-set/notebook))\n\nIt turns out that my 2nd best model (`0.84428` on private LB) is also the best local validated single model (`0.840686` on local) on only one fold.\n\nInterestingly model ensembling seemed not adding too much value and also had diverse results on LB.\nMy ensemble model of 3 best ones on public LB (`0.73028`) dropped to only `0.84147` on private one.\n\nHere are some more comparisons to show that public LB is mostly misleading:\n\n<pre>Local        Public       Private\n0.799186     0.71529      0.83770\n0.801906     0.72464      0.84026\n0.822382     0.71658      0.84243\n0.827833     0.71731      0.84185\n0.840686     0.72397      0.84428\n</pre>\n\nAlso glad about not being beaten by the `neptune.ml` team, who is sharing competitive solutions publicly everywhere before the end of competition and made the life of other hard working kagglers much more difficult to get a medal.\n\nBtw, I used `UNet+ResNet34` with `hypercolumns`, `lovasz loss` and `deep supervision` modified from the great [fastai solution](https://www.kaggle.com/c/tgs-salt-identification-challenge/discussion/69101) on pytorch shared by @vishnus in the salt competition. Some changes were made to do inference by batches to avoid OOM on such a large number of images. Also applied progressive learning trick learnt from fast.ai course to train images from 256x256 to 384x384, 512x512 and finally 768x768, though the best model only trained on 384x384 resolution.\n\nBurned out lots of `K80`, `P100` and `V100` GPUs on Google Cloud as well...\n",
    "420638": "Everyone's score will go up for sure, probably in the 0.9+. \n\nPosition-wise I expect a huge shake-up, I don't know where I will end up honestly, but the very top guys will remain top.",
    "420377": ""
  }
}