{
  "id": 230731,
  "title": "submission.csv size correlates to public LB score",
  "url": "/competitions/hubmap-kidney-segmentation/discussion/230731",
  "author_name": "",
  "post_date": "2021-04-05T12:16:47.769645600Z",
  "votes": 7,
  "comment_count": 11,
  "views": 0,
  "content": "<p>I found out a correlation between the size of my submission.csv and the public LB score. And it makes sense, as the file size is related somehow to the number and size of predicted glomeruli.</p>\n<p><img src=\"https://lh3.googleusercontent.com/4XjR6_NtH9TERvG0SKhtzUh2pqipyV16-hfbHWXytDtrvFanYEuJ2YR87sbd3-auo-Id7wh_APCYQVN1V2xdLcTmvLWbcfeAaH6oNkFYOrkrxMZ2mBQaTOAz7tMIwzSMew=w1280\" alt=\"Figure 1\"></p>\n<p>I wonder if this is something that happens to other kagglers. If I had to guess, I would say it does.</p>",
  "messages": [
    {
      "id": "1263436",
      "postDate": "04/05/2021 12:16:47",
      "content": "<p>I found out a correlation between the size of my submission.csv and the public LB score. And it makes sense, as the file size is related somehow to the number and size of predicted glomeruli.</p>\n<p><img src=\"https://lh3.googleusercontent.com/4XjR6_NtH9TERvG0SKhtzUh2pqipyV16-hfbHWXytDtrvFanYEuJ2YR87sbd3-auo-Id7wh_APCYQVN1V2xdLcTmvLWbcfeAaH6oNkFYOrkrxMZ2mBQaTOAz7tMIwzSMew=w1280\" alt=\"Figure 1\"></p>\n<p>I wonder if this is something that happens to other kagglers. If I had to guess, I would say it does.</p>",
      "rawMarkdown": "I found out a correlation between the size of my submission.csv and the public LB score. And it makes sense, as the file size is related somehow to the number and size of predicted glomeruli.\n\n![Figure 1](https://lh3.googleusercontent.com/4XjR6_NtH9TERvG0SKhtzUh2pqipyV16-hfbHWXytDtrvFanYEuJ2YR87sbd3-auo-Id7wh_APCYQVN1V2xdLcTmvLWbcfeAaH6oNkFYOrkrxMZ2mBQaTOAz7tMIwzSMew=w1280)\n\nI wonder if this is something that happens to other kagglers. If I had to guess, I would say it does.",
      "votes": null
    },
    {
      "id": "1263485",
      "postDate": "04/05/2021 13:02:31",
      "content": "<p>yes, it makes sense.<br>\nI think this indicates that most errors are false negative 🤔</p>",
      "rawMarkdown": "yes, it makes sense.\nI think this indicates that most errors are false negative 🤔",
      "votes": null
    },
    {
      "id": "1263837",
      "postDate": "04/05/2021 17:48:22",
      "content": "<table>\n<thead>\n<tr>\n<th>Sub</th>\n<th>Threshold</th>\n<th>File Size (MB)</th>\n<th>LB Score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>0.4</td>\n<td>5.23</td>\n<td>0.918</td>\n</tr>\n<tr>\n<td>2</td>\n<td>0.4</td>\n<td>5.26</td>\n<td>0.921</td>\n</tr>\n<tr>\n<td>3</td>\n<td>0.3</td>\n<td>5.33</td>\n<td>0.923</td>\n</tr>\n<tr>\n<td>8</td>\n<td>0.3</td>\n<td>5.34</td>\n<td>0.924</td>\n</tr>\n<tr>\n<td>4</td>\n<td>0.3</td>\n<td>5.37</td>\n<td>0.920</td>\n</tr>\n<tr>\n<td><strong>5</strong></td>\n<td><strong>0.3</strong></td>\n<td><strong>5.57</strong></td>\n<td><strong>0.925</strong></td>\n</tr>\n</tbody>\n</table>\n<p>I can see this thing happening in my last five submissions too😅.</p>\n<p><strong>Update</strong>: To test this further I reduced the threshold of my best submission. It increased file size. But the LB Score kept decreasing. So my learning is that I should stick to one threshold and if file size increases then its time for celebration😂🎉.</p>\n<table>\n<thead>\n<tr>\n<th>Sub</th>\n<th>Threshold</th>\n<th>File Size (MB)</th>\n<th>LB Score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>5</strong></td>\n<td><strong>0.3</strong></td>\n<td><strong>5.57</strong></td>\n<td><strong>0.925</strong></td>\n</tr>\n<tr>\n<td>6</td>\n<td>0.2</td>\n<td>5.77</td>\n<td>0.922</td>\n</tr>\n<tr>\n<td>7</td>\n<td>0.1</td>\n<td>6.29</td>\n<td>0.909</td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "Sub|Threshold| File Size (MB) | LB Score  |\n|---|---| --- | --- |\n1|0.4| 5.23 | 0.918 |\n2|0.4| 5.26 | 0.921 |\n3|0.3| 5.33 | 0.923 |\n8|0.3| 5.34 | 0.924 |\n4|0.3| 5.37 | 0.920 |\n**5**|**0.3**| **5.57** | **0.925** |\n\nI can see this thing happening in my last five submissions too😅.\n\n**Update**: To test this further I reduced the threshold of my best submission. It increased file size. But the LB Score kept decreasing. So my learning is that I should stick to one threshold and if file size increases then its time for celebration😂🎉.\n\nSub|Threshold| File Size (MB) | LB Score  |\n|---|---| --- | --- |\n**5**|**0.3**| **5.57** | **0.925** |\n6|0.2| 5.77 | 0.922 |\n7|0.1| 6.29 | 0.909 |",
      "votes": null
    },
    {
      "id": "1264021",
      "postDate": "04/05/2021 20:14:08",
      "content": "<p>Interesting observation. And I think it makes sense for the file size to increase and the LB score to decrease when the threshold is reduced. My rationale is that you are probably letting more spots make it to the final mask.</p>",
      "rawMarkdown": "Interesting observation. And I think it makes sense for the file size to increase and the LB score to decrease when the threshold is reduced. My rationale is that you are probably letting more spots make it to the final mask.",
      "votes": null
    },
    {
      "id": "1264060",
      "postDate": "04/05/2021 20:52:21",
      "content": "<p>Thanks for sharing! Do you see the same behaviour on CV? My CV degrades significantly when the threshold is below 0.4</p>",
      "rawMarkdown": "Thanks for sharing! Do you see the same behaviour on CV? My CV degrades significantly when the threshold is below 0.4",
      "votes": null
    },
    {
      "id": "1264068",
      "postDate": "04/05/2021 20:59:28",
      "content": "<p>If you compare it across the same model, threshold, and input image size then you can monitor this.</p>\n<p>I quickly checked my submissions and they varied across different models and input image sizes.</p>",
      "rawMarkdown": "If you compare it across the same model, threshold, and input image size then you can monitor this.\n\nI quickly checked my submissions and they varied across different models and input image sizes.",
      "votes": null
    },
    {
      "id": "1264891",
      "postDate": "04/06/2021 13:42:28",
      "content": "<p>strangely enough, this is true in my case as well. I have not played with the threshold to this date</p>",
      "rawMarkdown": "strangely enough, this is true in my case as well. I have not played with the threshold to this date",
      "votes": null
    },
    {
      "id": "1264948",
      "postDate": "04/06/2021 14:12:51",
      "content": "<p>This seems to be true for the region below 5MB. Above 5MB I see no clear pattern.</p>",
      "rawMarkdown": "This seems to be true for the region below 5MB. Above 5MB I see no clear pattern.",
      "votes": null
    },
    {
      "id": "1265188",
      "postDate": "04/06/2021 17:05:56",
      "content": "<p>… also, beware of the fact that the correlation is not linear. these results are all for threshold 0.4:</p>\n<p>LB -&gt; Size (MB):<br>\n0.929 -&gt; 5.4<br>\n0.930 -&gt; 5.42<br>\n0.931 -&gt; 5.41<br>\n0.932 -&gt; 5.45</p>",
      "rawMarkdown": "... also, beware of the fact that the correlation is not linear. these results are all for threshold 0.4:\n\nLB -> Size (MB):\n0.929 -> 5.4\n0.930 -> 5.42\n0.931 -> 5.41\n0.932 -> 5.45",
      "votes": null
    },
    {
      "id": "1266911",
      "postDate": "04/08/2021 07:03:16",
      "content": "<p><a href=\"https://www.kaggle.com/andras\" target=\"_blank\">@andras</a>, what training image size are you using?</p>",
      "rawMarkdown": "andras, what training image size are you using?",
      "votes": null
    },
    {
      "id": "1266930",
      "postDate": "04/08/2021 07:25:22",
      "content": "<p><a href=\"https://www.kaggle.com/hetulvpatel\" target=\"_blank\">@hetulvpatel</a> training image size are you using?</p>",
      "rawMarkdown": "hetulvpatel training image size are you using?",
      "votes": null
    },
    {
      "id": "1267031",
      "postDate": "04/08/2021 08:55:55",
      "content": "<p><a href=\"https://www.kaggle.com/projdev\" target=\"_blank\">@projdev</a> I use 256x256 size crops for training.</p>",
      "rawMarkdown": "projdev I use 256x256 size crops for training.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1263485,
      "author_name": "igoryakovenko",
      "author_url": "",
      "post_date": "04/05/2021 13:02:31",
      "content": "<p>yes, it makes sense.<br>\nI think this indicates that most errors are false negative 🤔</p>",
      "votes": null,
      "replies": [
        {
          "id": 1264948,
          "author_name": "felipekitamura",
          "author_url": "",
          "post_date": "04/06/2021 14:12:51",
          "content": "<p>This seems to be true for the region below 5MB. Above 5MB I see no clear pattern.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1263837,
      "author_name": "hetulvpatel",
      "author_url": "",
      "post_date": "04/05/2021 17:48:22",
      "content": "<table>\n<thead>\n<tr>\n<th>Sub</th>\n<th>Threshold</th>\n<th>File Size (MB)</th>\n<th>LB Score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1</td>\n<td>0.4</td>\n<td>5.23</td>\n<td>0.918</td>\n</tr>\n<tr>\n<td>2</td>\n<td>0.4</td>\n<td>5.26</td>\n<td>0.921</td>\n</tr>\n<tr>\n<td>3</td>\n<td>0.3</td>\n<td>5.33</td>\n<td>0.923</td>\n</tr>\n<tr>\n<td>8</td>\n<td>0.3</td>\n<td>5.34</td>\n<td>0.924</td>\n</tr>\n<tr>\n<td>4</td>\n<td>0.3</td>\n<td>5.37</td>\n<td>0.920</td>\n</tr>\n<tr>\n<td><strong>5</strong></td>\n<td><strong>0.3</strong></td>\n<td><strong>5.57</strong></td>\n<td><strong>0.925</strong></td>\n</tr>\n</tbody>\n</table>\n<p>I can see this thing happening in my last five submissions too😅.</p>\n<p><strong>Update</strong>: To test this further I reduced the threshold of my best submission. It increased file size. But the LB Score kept decreasing. So my learning is that I should stick to one threshold and if file size increases then its time for celebration😂🎉.</p>\n<table>\n<thead>\n<tr>\n<th>Sub</th>\n<th>Threshold</th>\n<th>File Size (MB)</th>\n<th>LB Score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>5</strong></td>\n<td><strong>0.3</strong></td>\n<td><strong>5.57</strong></td>\n<td><strong>0.925</strong></td>\n</tr>\n<tr>\n<td>6</td>\n<td>0.2</td>\n<td>5.77</td>\n<td>0.922</td>\n</tr>\n<tr>\n<td>7</td>\n<td>0.1</td>\n<td>6.29</td>\n<td>0.909</td>\n</tr>\n</tbody>\n</table>",
      "votes": null,
      "replies": [
        {
          "id": 1264021,
          "author_name": "felipekitamura",
          "author_url": "",
          "post_date": "04/05/2021 20:14:08",
          "content": "<p>Interesting observation. And I think it makes sense for the file size to increase and the LB score to decrease when the threshold is reduced. My rationale is that you are probably letting more spots make it to the final mask.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1264060,
          "author_name": "anjum48",
          "author_url": "",
          "post_date": "04/05/2021 20:52:21",
          "content": "<p>Thanks for sharing! Do you see the same behaviour on CV? My CV degrades significantly when the threshold is below 0.4</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1264068,
          "author_name": "tom88jerry",
          "author_url": "",
          "post_date": "04/05/2021 20:59:28",
          "content": "<p>If you compare it across the same model, threshold, and input image size then you can monitor this.</p>\n<p>I quickly checked my submissions and they varied across different models and input image sizes.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1266930,
          "author_name": "projdev",
          "author_url": "",
          "post_date": "04/08/2021 07:25:22",
          "content": "<p><a href=\"https://www.kaggle.com/hetulvpatel\" target=\"_blank\">@hetulvpatel</a> training image size are you using?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1267031,
          "author_name": "hetulvpatel",
          "author_url": "",
          "post_date": "04/08/2021 08:55:55",
          "content": "<p><a href=\"https://www.kaggle.com/projdev\" target=\"_blank\">@projdev</a> I use 256x256 size crops for training.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1264891,
      "author_name": "andrasferenczi",
      "author_url": "",
      "post_date": "04/06/2021 13:42:28",
      "content": "<p>strangely enough, this is true in my case as well. I have not played with the threshold to this date</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1265188,
      "author_name": "andrasferenczi",
      "author_url": "",
      "post_date": "04/06/2021 17:05:56",
      "content": "<p>… also, beware of the fact that the correlation is not linear. these results are all for threshold 0.4:</p>\n<p>LB -&gt; Size (MB):<br>\n0.929 -&gt; 5.4<br>\n0.930 -&gt; 5.42<br>\n0.931 -&gt; 5.41<br>\n0.932 -&gt; 5.45</p>",
      "votes": null,
      "replies": [
        {
          "id": 1266911,
          "author_name": "projdev",
          "author_url": "",
          "post_date": "04/08/2021 07:03:16",
          "content": "<p><a href=\"https://www.kaggle.com/andras\" target=\"_blank\">@andras</a>, what training image size are you using?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1263436": "I found out a correlation between the size of my submission.csv and the public LB score. And it makes sense, as the file size is related somehow to the number and size of predicted glomeruli.\n\n![Figure 1](https://lh3.googleusercontent.com/4XjR6_NtH9TERvG0SKhtzUh2pqipyV16-hfbHWXytDtrvFanYEuJ2YR87sbd3-auo-Id7wh_APCYQVN1V2xdLcTmvLWbcfeAaH6oNkFYOrkrxMZ2mBQaTOAz7tMIwzSMew=w1280)\n\nI wonder if this is something that happens to other kagglers. If I had to guess, I would say it does.",
    "1263485": "yes, it makes sense.\nI think this indicates that most errors are false negative 🤔",
    "1263837": "Sub|Threshold| File Size (MB) | LB Score  |\n|---|---| --- | --- |\n1|0.4| 5.23 | 0.918 |\n2|0.4| 5.26 | 0.921 |\n3|0.3| 5.33 | 0.923 |\n8|0.3| 5.34 | 0.924 |\n4|0.3| 5.37 | 0.920 |\n**5**|**0.3**| **5.57** | **0.925** |\n\nI can see this thing happening in my last five submissions too😅.\n\n**Update**: To test this further I reduced the threshold of my best submission. It increased file size. But the LB Score kept decreasing. So my learning is that I should stick to one threshold and if file size increases then its time for celebration😂🎉.\n\nSub|Threshold| File Size (MB) | LB Score  |\n|---|---| --- | --- |\n**5**|**0.3**| **5.57** | **0.925** |\n6|0.2| 5.77 | 0.922 |\n7|0.1| 6.29 | 0.909 |",
    "1264021": "Interesting observation. And I think it makes sense for the file size to increase and the LB score to decrease when the threshold is reduced. My rationale is that you are probably letting more spots make it to the final mask.",
    "1264060": "Thanks for sharing! Do you see the same behaviour on CV? My CV degrades significantly when the threshold is below 0.4",
    "1264068": "If you compare it across the same model, threshold, and input image size then you can monitor this.\n\nI quickly checked my submissions and they varied across different models and input image sizes.",
    "1264891": "strangely enough, this is true in my case as well. I have not played with the threshold to this date",
    "1264948": "This seems to be true for the region below 5MB. Above 5MB I see no clear pattern.",
    "1265188": "... also, beware of the fact that the correlation is not linear. these results are all for threshold 0.4:\n\nLB -> Size (MB):\n0.929 -> 5.4\n0.930 -> 5.42\n0.931 -> 5.41\n0.932 -> 5.45",
    "1266911": "andras, what training image size are you using?",
    "1266930": "hetulvpatel training image size are you using?",
    "1267031": "projdev I use 256x256 size crops for training."
  },
  "source": "meta"
}