{
  "id": 493317,
  "title": "Model CV - LB Thread",
  "url": "/competitions/birdclef-2024/discussion/493317",
  "author_name": "",
  "post_date": "2024-04-12T23:38:28.256240800Z",
  "votes": 29,
  "comment_count": 16,
  "views": 0,
  "content": "<p>Hello,</p>\n<p>Creating this thread to share the CV - LB that people got.</p>\n<p>For the moment i am using:</p>\n<ul>\n<li>efficientNetB1 with attention layer </li>\n<li>5 sec crop (only the first and last 5 sec of a sample is used)</li>\n<li>BCE Loss</li>\n<li>CV split ~90/10 =&gt; AUC = 0.98 / CMAP = 0.89 (metric from 2023 competition)</li>\n<li>LB single fold : 0.59</li>\n</ul>",
  "messages": [
    {
      "id": "2749234",
      "postDate": "04/12/2024 23:38:28",
      "content": "<p>Hello,</p>\n<p>Creating this thread to share the CV - LB that people got.</p>\n<p>For the moment i am using:</p>\n<ul>\n<li>efficientNetB1 with attention layer </li>\n<li>5 sec crop (only the first and last 5 sec of a sample is used)</li>\n<li>BCE Loss</li>\n<li>CV split ~90/10 =&gt; AUC = 0.98 / CMAP = 0.89 (metric from 2023 competition)</li>\n<li>LB single fold : 0.59</li>\n</ul>",
      "rawMarkdown": "Hello,\n\nCreating this thread to share the CV - LB that people got.\n\nFor the moment i am using:\n- efficientNetB1 with attention layer \n- 5 sec crop (only the first and last 5 sec of a sample is used)\n- BCE Loss\n- CV split ~90/10 => AUC = 0.98 / CMAP = 0.89 (metric from 2023 competition)\n- LB single fold : 0.59",
      "votes": null
    },
    {
      "id": "2749244",
      "postDate": "04/12/2024 23:46:51",
      "content": "<p>I was just about to make a discussion about it ! </p>\n<table>\n<thead>\n<tr>\n<th>CV</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.974</td>\n<td>0.59</td>\n</tr>\n<tr>\n<td>0.984</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>0.983</td>\n<td>0.61</td>\n</tr>\n<tr>\n<td>0.979</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>0.980</td>\n<td>0.61</td>\n</tr>\n<tr>\n<td>0.982</td>\n<td>0.58</td>\n</tr>\n<tr>\n<td>0.982</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>0.985</td>\n<td>0.59</td>\n</tr>\n<tr>\n<td>0.978</td>\n<td>0.61</td>\n</tr>\n<tr>\n<td>0.980</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>0.978</td>\n<td>0.61</td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F1534110d356519b2ee77f37aa462a92f%2Foutput.png?generation=1712965586836286&amp;alt=media\"></p>\n<p>Those are my results..</p>",
      "rawMarkdown": "I was just about to make a discussion about it ! \n\n|   CV   |  LB  |\n|--------|------|\n| 0.974  | 0.59 |\n| 0.984  | 0.6  |\n| 0.983  | 0.61 |\n| 0.979  | 0.6  |\n| 0.980  | 0.61 |\n| 0.982  | 0.58 |\n| 0.982  | 0.6  |\n| 0.985  | 0.59 |\n| 0.978  | 0.61 |\n| 0.980  | 0.6  |\n| 0.978  | 0.61 |\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F1534110d356519b2ee77f37aa462a92f%2Foutput.png?generation=1712965586836286&alt=media)\n\n\nThose are my results..",
      "votes": null
    },
    {
      "id": "2749406",
      "postDate": "04/13/2024 03:40:09",
      "content": "<p>I have a few ideas, some obvious, but I feel we are still missing something significant to bridge the gap a little. </p>",
      "rawMarkdown": "I have a few ideas, some obvious, but I feel we are still missing something significant to bridge the gap a little.",
      "votes": null
    },
    {
      "id": "2749519",
      "postDate": "04/13/2024 05:22:55",
      "content": "<p>That's a really good scores just for the starting and ending 5 second samples, can I ask you whether you used secondary labels or ignored them or excluded those files ? I tried including all samples by taking mid 5 second the predictions were pretty bad,</p>",
      "rawMarkdown": "That's a really good scores just for the starting and ending 5 second samples, can I ask you whether you used secondary labels or ignored them or excluded those files ? I tried including all samples by taking mid 5 second the predictions were pretty bad,",
      "votes": null
    },
    {
      "id": "2749903",
      "postDate": "04/13/2024 10:49:06",
      "content": "<p>interesting i also do not see much correlation between my CV and LB ; need to try few more experience</p>",
      "rawMarkdown": "interesting i also do not see much correlation between my CV and LB ; need to try few more experience",
      "votes": null
    },
    {
      "id": "2750045",
      "postDate": "04/13/2024 12:36:52",
      "content": "<p>My <a href=\"https://www.kaggle.com/code/zijiangyang1116/birdclef-24-efficientnetb0-pytorch-inference\" target=\"_blank\">baseline</a>:</p>\n<ul>\n<li>EfficientNetB0</li>\n<li>center 5 sec crop of a sample (only train dataset of BirdCLEF'24)</li>\n<li>CrossEntropyLoss</li>\n<li>5-fold of KFold=&gt; AUC = 0.936 (metric from <a href=\"https://www.kaggle.com/code/metric/birdclef-roc-auc\" target=\"_blank\">Birdclef ROC AUC</a>)</li>\n<li>LB of single fold with best AUC: 0.61</li>\n</ul>",
      "rawMarkdown": "My [baseline](https://www.kaggle.com/code/zijiangyang1116/birdclef-24-efficientnetb0-pytorch-inference):\n\n* EfficientNetB0\n* center 5 sec crop of a sample (only train dataset of BirdCLEF'24)\n* CrossEntropyLoss\n* 5-fold of KFold=> AUC = 0.936 (metric from [Birdclef ROC AUC](https://www.kaggle.com/code/metric/birdclef-roc-auc))\n* LB of single fold with best AUC: 0.61",
      "votes": null
    },
    {
      "id": "2750150",
      "postDate": "04/13/2024 13:31:36",
      "content": "<p>hey, i ignore the columns secondary labels. I use all samples but only using primary_label information</p>",
      "rawMarkdown": "hey, i ignore the columns secondary labels. I use all samples but only using primary_label information",
      "votes": null
    },
    {
      "id": "2750215",
      "postDate": "04/13/2024 14:30:50",
      "content": "<p>My 5 Fold ROCAUC is ~0.97, but the highest LB i've been able to get was 0.64. This is where I get stuck every year and usually just give up because I'm not able to figure out something that correlates with LB performance.</p>",
      "rawMarkdown": "My 5 Fold ROCAUC is ~0.97, but the highest LB i've been able to get was 0.64. This is where I get stuck every year and usually just give up because I'm not able to figure out something that correlates with LB performance.",
      "votes": null
    },
    {
      "id": "2763178",
      "postDate": "04/20/2024 10:45:17",
      "content": "<p>5-fold split with results from the first fold:</p>\n<p>CV: 0.99 🫠<br>\nLB: 0.64</p>",
      "rawMarkdown": "5-fold split with results from the first fold:\n\nCV: 0.99 🫠\nLB: 0.64",
      "votes": null
    },
    {
      "id": "2763600",
      "postDate": "04/20/2024 16:06:49",
      "content": "<p>you should keep going! this year i am struggling too regarding stability</p>",
      "rawMarkdown": "you should keep going! this year i am struggling too regarding stability",
      "votes": null
    },
    {
      "id": "2763602",
      "postDate": "04/20/2024 16:08:12",
      "content": "<p>seems most people have issue with stability. I might start to try to do 5 fold as well. I wanted to limit myself to train/val split to do faster iteration but so far it was not productive. <br>\nMay I ask if you are using in data aug in your training?</p>",
      "rawMarkdown": "seems most people have issue with stability. I might start to try to do 5 fold as well. I wanted to limit myself to train/val split to do faster iteration but so far it was not productive. \nMay I ask if you are using in data aug in your training?",
      "votes": null
    },
    {
      "id": "2763625",
      "postDate": "04/20/2024 16:28:00",
      "content": "<p>yes, I use audio and some spec level augmentations (time and frequency masking).</p>",
      "rawMarkdown": "yes, I use audio and some spec level augmentations (time and frequency masking).",
      "votes": null
    },
    {
      "id": "2763633",
      "postDate": "04/20/2024 16:35:10",
      "content": "<p>Btw, my current setup is similar to a single train/val split. I am not training on the full folds. I also have a limited compute so I try things on a single fold which is the same thing as an 80-20 split.</p>",
      "rawMarkdown": "Btw, my current setup is similar to a single train/val split. I am not training on the full folds. I also have a limited compute so I try things on a single fold which is the same thing as an 80-20 split.",
      "votes": null
    },
    {
      "id": "2763686",
      "postDate": "04/20/2024 17:19:45",
      "content": "<p>oh got it i thought you were taking the best fold on LB! </p>",
      "rawMarkdown": "oh got it i thought you were taking the best fold on LB!",
      "votes": null
    },
    {
      "id": "2779840",
      "postDate": "04/27/2024 21:03:02",
      "content": "<p>The same case for me, the CV looks sooooo high and makes me spend many hours debugging if there's a leak… but finally can't find any bug.</p>",
      "rawMarkdown": "The same case for me, the CV looks sooooo high and makes me spend many hours debugging if there's a leak... but finally can't find any bug.",
      "votes": null
    },
    {
      "id": "2810563",
      "postDate": "05/13/2024 10:18:32",
      "content": "<p>Simple Efficientnet B0 (First 5 seconds training) (FocalBCE Loss) <br>\nCV: 0.97<br>\nLB: 0.66<br>\nInfer Time: 20 to 24 Min. </p>\n<p>Haven't tried ensemble yet.</p>",
      "rawMarkdown": "Simple Efficientnet B0 (First 5 seconds training) (FocalBCE Loss) \nCV: 0.97\nLB: 0.66\nInfer Time: 20 to 24 Min. \n\nHaven't tried ensemble yet.",
      "votes": null
    },
    {
      "id": "2811959",
      "postDate": "05/14/2024 03:02:21",
      "content": "<p>Hi Salman, do you clean the training data, or use them directly?</p>",
      "rawMarkdown": "Hi Salman, do you clean the training data, or use them directly?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2749244,
      "author_name": "janmpia",
      "author_url": "",
      "post_date": "04/12/2024 23:46:51",
      "content": "<p>I was just about to make a discussion about it ! </p>\n<table>\n<thead>\n<tr>\n<th>CV</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>0.974</td>\n<td>0.59</td>\n</tr>\n<tr>\n<td>0.984</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>0.983</td>\n<td>0.61</td>\n</tr>\n<tr>\n<td>0.979</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>0.980</td>\n<td>0.61</td>\n</tr>\n<tr>\n<td>0.982</td>\n<td>0.58</td>\n</tr>\n<tr>\n<td>0.982</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>0.985</td>\n<td>0.59</td>\n</tr>\n<tr>\n<td>0.978</td>\n<td>0.61</td>\n</tr>\n<tr>\n<td>0.980</td>\n<td>0.6</td>\n</tr>\n<tr>\n<td>0.978</td>\n<td>0.61</td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F1534110d356519b2ee77f37aa462a92f%2Foutput.png?generation=1712965586836286&amp;alt=media\"></p>\n<p>Those are my results..</p>",
      "votes": null,
      "replies": [
        {
          "id": 2749903,
          "author_name": "ludovick",
          "author_url": "",
          "post_date": "04/13/2024 10:49:06",
          "content": "<p>interesting i also do not see much correlation between my CV and LB ; need to try few more experience</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2749406,
      "author_name": "cody11null",
      "author_url": "",
      "post_date": "04/13/2024 03:40:09",
      "content": "<p>I have a few ideas, some obvious, but I feel we are still missing something significant to bridge the gap a little. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2749519,
      "author_name": "arunsensei",
      "author_url": "",
      "post_date": "04/13/2024 05:22:55",
      "content": "<p>That's a really good scores just for the starting and ending 5 second samples, can I ask you whether you used secondary labels or ignored them or excluded those files ? I tried including all samples by taking mid 5 second the predictions were pretty bad,</p>",
      "votes": null,
      "replies": [
        {
          "id": 2750150,
          "author_name": "ludovick",
          "author_url": "",
          "post_date": "04/13/2024 13:31:36",
          "content": "<p>hey, i ignore the columns secondary labels. I use all samples but only using primary_label information</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2750045,
      "author_name": "zijiangyang1116",
      "author_url": "",
      "post_date": "04/13/2024 12:36:52",
      "content": "<p>My <a href=\"https://www.kaggle.com/code/zijiangyang1116/birdclef-24-efficientnetb0-pytorch-inference\" target=\"_blank\">baseline</a>:</p>\n<ul>\n<li>EfficientNetB0</li>\n<li>center 5 sec crop of a sample (only train dataset of BirdCLEF'24)</li>\n<li>CrossEntropyLoss</li>\n<li>5-fold of KFold=&gt; AUC = 0.936 (metric from <a href=\"https://www.kaggle.com/code/metric/birdclef-roc-auc\" target=\"_blank\">Birdclef ROC AUC</a>)</li>\n<li>LB of single fold with best AUC: 0.61</li>\n</ul>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2750215,
      "author_name": "willrice",
      "author_url": "",
      "post_date": "04/13/2024 14:30:50",
      "content": "<p>My 5 Fold ROCAUC is ~0.97, but the highest LB i've been able to get was 0.64. This is where I get stuck every year and usually just give up because I'm not able to figure out something that correlates with LB performance.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2763600,
          "author_name": "ludovick",
          "author_url": "",
          "post_date": "04/20/2024 16:06:49",
          "content": "<p>you should keep going! this year i am struggling too regarding stability</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2763178,
      "author_name": "snnclsr",
      "author_url": "",
      "post_date": "04/20/2024 10:45:17",
      "content": "<p>5-fold split with results from the first fold:</p>\n<p>CV: 0.99 🫠<br>\nLB: 0.64</p>",
      "votes": null,
      "replies": [
        {
          "id": 2763602,
          "author_name": "ludovick",
          "author_url": "",
          "post_date": "04/20/2024 16:08:12",
          "content": "<p>seems most people have issue with stability. I might start to try to do 5 fold as well. I wanted to limit myself to train/val split to do faster iteration but so far it was not productive. <br>\nMay I ask if you are using in data aug in your training?</p>",
          "votes": null,
          "replies": [
            {
              "id": 2763625,
              "author_name": "snnclsr",
              "author_url": "",
              "post_date": "04/20/2024 16:28:00",
              "content": "<p>yes, I use audio and some spec level augmentations (time and frequency masking).</p>",
              "votes": null,
              "replies": []
            },
            {
              "id": 2763633,
              "author_name": "snnclsr",
              "author_url": "",
              "post_date": "04/20/2024 16:35:10",
              "content": "<p>Btw, my current setup is similar to a single train/val split. I am not training on the full folds. I also have a limited compute so I try things on a single fold which is the same thing as an 80-20 split.</p>",
              "votes": null,
              "replies": [
                {
                  "id": 2763686,
                  "author_name": "ludovick",
                  "author_url": "",
                  "post_date": "04/20/2024 17:19:45",
                  "content": "<p>oh got it i thought you were taking the best fold on LB! </p>",
                  "votes": null,
                  "replies": []
                }
              ]
            }
          ]
        }
      ]
    },
    {
      "id": 2779840,
      "author_name": "leonshangguan",
      "author_url": "",
      "post_date": "04/27/2024 21:03:02",
      "content": "<p>The same case for me, the CV looks sooooo high and makes me spend many hours debugging if there's a leak… but finally can't find any bug.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2810563,
      "author_name": "salmanahmedtamu",
      "author_url": "",
      "post_date": "05/13/2024 10:18:32",
      "content": "<p>Simple Efficientnet B0 (First 5 seconds training) (FocalBCE Loss) <br>\nCV: 0.97<br>\nLB: 0.66<br>\nInfer Time: 20 to 24 Min. </p>\n<p>Haven't tried ensemble yet.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2811959,
          "author_name": "tanxxx",
          "author_url": "",
          "post_date": "05/14/2024 03:02:21",
          "content": "<p>Hi Salman, do you clean the training data, or use them directly?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2749234": "Hello,\n\nCreating this thread to share the CV - LB that people got.\n\nFor the moment i am using:\n- efficientNetB1 with attention layer \n- 5 sec crop (only the first and last 5 sec of a sample is used)\n- BCE Loss\n- CV split ~90/10 => AUC = 0.98 / CMAP = 0.89 (metric from 2023 competition)\n- LB single fold : 0.59",
    "2749244": "I was just about to make a discussion about it ! \n\n|   CV   |  LB  |\n|--------|------|\n| 0.974  | 0.59 |\n| 0.984  | 0.6  |\n| 0.983  | 0.61 |\n| 0.979  | 0.6  |\n| 0.980  | 0.61 |\n| 0.982  | 0.58 |\n| 0.982  | 0.6  |\n| 0.985  | 0.59 |\n| 0.978  | 0.61 |\n| 0.980  | 0.6  |\n| 0.978  | 0.61 |\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F1534110d356519b2ee77f37aa462a92f%2Foutput.png?generation=1712965586836286&alt=media)\n\n\nThose are my results..",
    "2749406": "I have a few ideas, some obvious, but I feel we are still missing something significant to bridge the gap a little.",
    "2749519": "That's a really good scores just for the starting and ending 5 second samples, can I ask you whether you used secondary labels or ignored them or excluded those files ? I tried including all samples by taking mid 5 second the predictions were pretty bad,",
    "2749903": "interesting i also do not see much correlation between my CV and LB ; need to try few more experience",
    "2750045": "My [baseline](https://www.kaggle.com/code/zijiangyang1116/birdclef-24-efficientnetb0-pytorch-inference):\n\n* EfficientNetB0\n* center 5 sec crop of a sample (only train dataset of BirdCLEF'24)\n* CrossEntropyLoss\n* 5-fold of KFold=> AUC = 0.936 (metric from [Birdclef ROC AUC](https://www.kaggle.com/code/metric/birdclef-roc-auc))\n* LB of single fold with best AUC: 0.61",
    "2750150": "hey, i ignore the columns secondary labels. I use all samples but only using primary_label information",
    "2750215": "My 5 Fold ROCAUC is ~0.97, but the highest LB i've been able to get was 0.64. This is where I get stuck every year and usually just give up because I'm not able to figure out something that correlates with LB performance.",
    "2763178": "5-fold split with results from the first fold:\n\nCV: 0.99 🫠\nLB: 0.64",
    "2763600": "you should keep going! this year i am struggling too regarding stability",
    "2763602": "seems most people have issue with stability. I might start to try to do 5 fold as well. I wanted to limit myself to train/val split to do faster iteration but so far it was not productive. \nMay I ask if you are using in data aug in your training?",
    "2763625": "yes, I use audio and some spec level augmentations (time and frequency masking).",
    "2763633": "Btw, my current setup is similar to a single train/val split. I am not training on the full folds. I also have a limited compute so I try things on a single fold which is the same thing as an 80-20 split.",
    "2763686": "oh got it i thought you were taking the best fold on LB!",
    "2779840": "The same case for me, the CV looks sooooo high and makes me spend many hours debugging if there's a leak... but finally can't find any bug.",
    "2810563": "Simple Efficientnet B0 (First 5 seconds training) (FocalBCE Loss) \nCV: 0.97\nLB: 0.66\nInfer Time: 20 to 24 Min. \n\nHaven't tried ensemble yet.",
    "2811959": "Hi Salman, do you clean the training data, or use them directly?"
  },
  "source": "meta"
}