{
  "id": 468161,
  "title": "Simple experiments to assess effect of folding",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/468161",
  "author_name": "",
  "post_date": "2024-01-15T15:33:47.516650100Z",
  "votes": 6,
  "comment_count": 6,
  "views": 0,
  "content": "<h2>Introduction</h2>\n<p>This post simply demonstrates how a number of folds in <code>resnet34d</code> base model's can affect the task of classifying harmful brain activity.</p>\n<h2>Experiment</h2>\n<p>Baseline : <a href=\"https://www.kaggle.com/ttahara\" target=\"_blank\">@ttahara</a>'s <a href=\"https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-inference\" target=\"_blank\">notebook</a> </p>\n<h2>Result</h2>\n<table>\n<thead>\n<tr>\n<th>FOLD</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>5</td>\n<td>0.49</td>\n</tr>\n<tr>\n<td>4</td>\n<td>0.49</td>\n</tr>\n<tr>\n<td>3</td>\n<td>0.50</td>\n</tr>\n<tr>\n<td>2</td>\n<td>0.53</td>\n</tr>\n<tr>\n<td>1</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td><br></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h2>Conclusion</h2>\n<p>This result implies that increasing the number of folds could slightly enhance the performance in 6 folds and 7 folds, or decrease it due to over-folding.</p>\n<p>Thanks to <a href=\"https://www.kaggle.com/ttahara\" target=\"_blank\">@ttahara</a> for the great sharing.</p>",
  "messages": [
    {
      "id": "2603129",
      "postDate": "01/15/2024 15:33:47",
      "content": "<h2>Introduction</h2>\n<p>This post simply demonstrates how a number of folds in <code>resnet34d</code> base model's can affect the task of classifying harmful brain activity.</p>\n<h2>Experiment</h2>\n<p>Baseline : <a href=\"https://www.kaggle.com/ttahara\" target=\"_blank\">@ttahara</a>'s <a href=\"https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-inference\" target=\"_blank\">notebook</a> </p>\n<h2>Result</h2>\n<table>\n<thead>\n<tr>\n<th>FOLD</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>5</td>\n<td>0.49</td>\n</tr>\n<tr>\n<td>4</td>\n<td>0.49</td>\n</tr>\n<tr>\n<td>3</td>\n<td>0.50</td>\n</tr>\n<tr>\n<td>2</td>\n<td>0.53</td>\n</tr>\n<tr>\n<td>1</td>\n<td>0.65</td>\n</tr>\n<tr>\n<td><br></td>\n<td></td>\n</tr>\n</tbody>\n</table>\n<h2>Conclusion</h2>\n<p>This result implies that increasing the number of folds could slightly enhance the performance in 6 folds and 7 folds, or decrease it due to over-folding.</p>\n<p>Thanks to <a href=\"https://www.kaggle.com/ttahara\" target=\"_blank\">@ttahara</a> for the great sharing.</p>",
      "rawMarkdown": "## Introduction\n\nThis post simply demonstrates how a number of folds in `resnet34d` base model's can affect the task of classifying harmful brain activity.\n\n## Experiment\nBaseline : @ttahara's [notebook](https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-inference) \n\n## Result\n\n| FOLD | LB |\n| ---- | -- |\n| 5    | 0.49 |\n| 4    | 0.49 |\n| 3    | 0.50 |\n| 2    | 0.53 |\n| 1    | 0.65 |\n<br>\n\n## Conclusion\nThis result implies that increasing the number of folds could slightly enhance the performance in 6 folds and 7 folds, or decrease it due to over-folding.\n\nThanks to @ttahara for the great sharing.",
      "votes": null
    },
    {
      "id": "2604995",
      "postDate": "01/16/2024 19:23:54",
      "content": "<p>You are simulating the ensemble effect. You should never tune the number of folds based on LB score.</p>",
      "rawMarkdown": "You are simulating the ensemble effect. You should never tune the number of folds based on LB score.",
      "votes": null
    },
    {
      "id": "2605013",
      "postDate": "01/16/2024 19:34:54",
      "content": "<p><a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> Since, we do not know the # of test samples. is it not good to choice the # folds based on local cv ~ lb score difference? </p>",
      "rawMarkdown": "gunesevitan Since, we do not know the # of test samples. is it not good to choice the # folds based on local cv ~ lb score difference?",
      "votes": null
    },
    {
      "id": "2605479",
      "postDate": "01/17/2024 05:08:38",
      "content": "<p>That is the generalization gap which isn't necessarily a bad thing. I also don't think you can narrow it by increasing/decreasing the number of folds.</p>",
      "rawMarkdown": "That is the generalization gap which isn't necessarily a bad thing. I also don't think you can narrow it by increasing/decreasing the number of folds.",
      "votes": null
    },
    {
      "id": "2605498",
      "postDate": "01/17/2024 05:27:40",
      "content": "<p>Understood <a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> Thanks for the explanation.</p>",
      "rawMarkdown": "Understood @gunesevitan Thanks for the explanation.",
      "votes": null
    },
    {
      "id": "2605528",
      "postDate": "01/17/2024 05:44:07",
      "content": "<p><a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> this is the reason mention i.e increasing to 10 folds, his cv and lb gap is reduced for this dataset - <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467915#2601716\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467915#2601716</a><br>\n<strong>5 Folds</strong> : CV: 0.82 LB: 0.67 Gap: 0.15<br>\n<strong>10 Folds</strong> : CV: 0.75 LB 0.73 Gap: 0.02</p>",
      "rawMarkdown": "gunesevitan this is the reason mention i.e increasing to 10 folds, his cv and lb gap is reduced for this dataset - https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467915#2601716\n**5 Folds** : CV: 0.82 LB: 0.67 Gap: 0.15\n**10 Folds** : CV: 0.75 LB 0.73 Gap: 0.02",
      "votes": null
    },
    {
      "id": "2605557",
      "postDate": "01/17/2024 05:55:10",
      "content": "<p>Yeah, but the gap itself is not a problem. It only shows that distributions of training and test are different. All you need is good correlation between cv and lb scores. </p>",
      "rawMarkdown": "Yeah, but the gap itself is not a problem. It only shows that distributions of training and test are different. All you need is good correlation between cv and lb scores.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2604995,
      "author_name": "gunesevitan",
      "author_url": "",
      "post_date": "01/16/2024 19:23:54",
      "content": "<p>You are simulating the ensemble effect. You should never tune the number of folds based on LB score.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2605013,
          "author_name": "seshurajup",
          "author_url": "",
          "post_date": "01/16/2024 19:34:54",
          "content": "<p><a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> Since, we do not know the # of test samples. is it not good to choice the # folds based on local cv ~ lb score difference? </p>",
          "votes": null,
          "replies": [
            {
              "id": 2605479,
              "author_name": "gunesevitan",
              "author_url": "",
              "post_date": "01/17/2024 05:08:38",
              "content": "<p>That is the generalization gap which isn't necessarily a bad thing. I also don't think you can narrow it by increasing/decreasing the number of folds.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2605498,
                  "author_name": "seshurajup",
                  "author_url": "",
                  "post_date": "01/17/2024 05:27:40",
                  "content": "<p>Understood <a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> Thanks for the explanation.</p>",
                  "votes": null,
                  "replies": []
                },
                {
                  "id": 2605528,
                  "author_name": "seshurajup",
                  "author_url": "",
                  "post_date": "01/17/2024 05:44:07",
                  "content": "<p><a href=\"https://www.kaggle.com/gunesevitan\" target=\"_blank\">@gunesevitan</a> this is the reason mention i.e increasing to 10 folds, his cv and lb gap is reduced for this dataset - <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467915#2601716\" target=\"_blank\">https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467915#2601716</a><br>\n<strong>5 Folds</strong> : CV: 0.82 LB: 0.67 Gap: 0.15<br>\n<strong>10 Folds</strong> : CV: 0.75 LB 0.73 Gap: 0.02</p>",
                  "votes": null,
                  "replies": [
                    {
                      "id": 2605557,
                      "author_name": "gunesevitan",
                      "author_url": "",
                      "post_date": "01/17/2024 05:55:10",
                      "content": "<p>Yeah, but the gap itself is not a problem. It only shows that distributions of training and test are different. All you need is good correlation between cv and lb scores. </p>",
                      "votes": null,
                      "replies": []
                    }
                  ]
                }
              ]
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2603129": "## Introduction\n\nThis post simply demonstrates how a number of folds in `resnet34d` base model's can affect the task of classifying harmful brain activity.\n\n## Experiment\nBaseline : @ttahara's [notebook](https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-inference) \n\n## Result\n\n| FOLD | LB |\n| ---- | -- |\n| 5    | 0.49 |\n| 4    | 0.49 |\n| 3    | 0.50 |\n| 2    | 0.53 |\n| 1    | 0.65 |\n<br>\n\n## Conclusion\nThis result implies that increasing the number of folds could slightly enhance the performance in 6 folds and 7 folds, or decrease it due to over-folding.\n\nThanks to @ttahara for the great sharing.",
    "2604995": "You are simulating the ensemble effect. You should never tune the number of folds based on LB score.",
    "2605013": "gunesevitan Since, we do not know the # of test samples. is it not good to choice the # folds based on local cv ~ lb score difference?",
    "2605479": "That is the generalization gap which isn't necessarily a bad thing. I also don't think you can narrow it by increasing/decreasing the number of folds.",
    "2605498": "Understood @gunesevitan Thanks for the explanation.",
    "2605528": "gunesevitan this is the reason mention i.e increasing to 10 folds, his cv and lb gap is reduced for this dataset - https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467915#2601716\n**5 Folds** : CV: 0.82 LB: 0.67 Gap: 0.15\n**10 Folds** : CV: 0.75 LB 0.73 Gap: 0.02",
    "2605557": "Yeah, but the gap itself is not a problem. It only shows that distributions of training and test are different. All you need is good correlation between cv and lb scores."
  },
  "source": "meta"
}