{ "id": 232651, "title": "CV vs Public LB", "url": "/competitions/birdclef-2021/discussion/232651", "author_name": "", "post_date": "2021-04-14T17:25:50.585036100Z", "votes": 14, "comment_count": 17, "views": 0, "content": "

Hi All,

\n

wanted to check how your CV compares with the public LB?
\nAre you seeing a good correlation? as we have limited submissions in this, would be helpful if you can share what you are seeing…

\n

I have not done many experiments and submission -- but for me the CV score and public LB has good correlation in the ones I have.

\n

Hopefully, this means we may trust our CV -- pls do share if anyone notices anything different in your experiments

\n

Thanks!

", "messages": [ { "id": "1273829", "postDate": "04/14/2021 17:25:50", "content": "

Hi All,

\n

wanted to check how your CV compares with the public LB?
\nAre you seeing a good correlation? as we have limited submissions in this, would be helpful if you can share what you are seeing…

\n

I have not done many experiments and submission -- but for me the CV score and public LB has good correlation in the ones I have.

\n

Hopefully, this means we may trust our CV -- pls do share if anyone notices anything different in your experiments

\n

Thanks!

", "rawMarkdown": "Hi All,\n\nwanted to check how your CV compares with the public LB?\nAre you seeing a good correlation? as we have limited submissions in this, would be helpful if you can share what you are seeing...\n\nI have not done many experiments and submission -- but for me the CV score and public LB has good correlation in the ones I have.\n\nHopefully, this means we may trust our CV -- pls do share if anyone notices anything different in your experiments \n\nThanks!", "votes": null }, { "id": "1278531", "postDate": "04/20/2021 02:40:54", "content": "

Here are the results of some experiments using @kneroma notebook https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference

\n

f1 score -> Public LB score

\n

0.67766 -> 0.59

\n

0.685153-> 0.61 (v1 of notebook above)

\n

0.692444-> 0.61

\n

0.695736-> 0.62

\n

0.702431-> 0.64

\n

the validation F1 scores in the notebook showcase a good indication of the Public LB scores.

", "rawMarkdown": "Here are the results of some experiments using @kneroma notebook [https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference](https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference)\n\n f1 score -> Public LB score\n\n0.67766 -> 0.59\n\n0.685153-> 0.61 (v1 of notebook above)\n\n0.692444-> 0.61\n\n0.695736-> 0.62\n\n 0.702431-> 0.64\n\n\nthe validation F1 scores in the notebook showcase a good indication of the Public LB scores.", "votes": null }, { "id": "1278577", "postDate": "04/20/2021 03:55:20", "content": "

Thanks for sharing @kmldas . I got the same feeling about the CV in here. More again, it seems like higher CVs lead to higher LB correlation. As an example, the difference between this CV and my current LB is only about 0.01.

", "rawMarkdown": "Thanks for sharing @kmldas . I got the same feeling about the CV in [here](https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference). More again, it seems like higher CVs lead to higher LB correlation. As an example, the difference between this CV and my current LB is only about **0.01**.", "votes": null }, { "id": "1298633", "postDate": "05/09/2021 04:49:02", "content": "

Here are the results of some experiments using @kneroma's notebook
\nversion f1 score/threshold -> Public LB score
\n7 0.6574/0.74 -> 0.58

\n

6 0.6536/0.67 -> 0.57

\n

5 0.654236/0.72 -> 0.60

\n

4 0.652847/0.7 -> 0.60

\n

3 0.655278/0.4 -> 0.56

\n

2 0.643472/0.8 -> 0.57

\n

1 0.535472/0.5 -> 0.58

", "rawMarkdown": "Here are the results of some experiments using @kneroma's [notebook](https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference)\nversion f1 score/threshold -> Public LB score \n7 0.6574/0.74 -> 0.58\n\n6 0.6536/0.67 -> 0.57\n\n5 0.654236/0.72 -> 0.60\n\n4 0.652847/0.7 -> 0.60\n\n3 0.655278/0.4 -> 0.56\n\n2 0.643472/0.8 -> 0.57\n\n1 0.535472/0.5 -> 0.58", "votes": null }, { "id": "1298771", "postDate": "05/09/2021 07:21:45", "content": "

Nice one. If you can train 2 different models and submit their average prediction, your LB could significantly improve. In fact, what is the score of the bag of your 5 cited models ?

", "rawMarkdown": "Nice one. If you can train 2 different models and submit their average prediction, your LB could significantly improve. In fact, what is the score of the bag of your 5 cited models ?", "votes": null }, { "id": "1298782", "postDate": "05/09/2021 07:37:18", "content": "

I haven't bag models, I just trained a singal model with your data fold0 as validation dataset, I want to find a good baseline and do next progress.
\nSomething different with your inference notebook is that my threshold is high, the 0.60 LB used sheshold 0.7 not 0.25 (I use resnest50 too). And I find that my CV f1 increase followed by threshold increase. And I always choose the best CV to submit.

", "rawMarkdown": "I haven't bag models, I just trained a singal model with your data fold0 as validation dataset, I want to find a good baseline and do next progress.\nSomething different with your inference notebook is that my threshold is high, the 0.60 LB used sheshold 0.7 not 0.25 (I use resnest50 too). And I find that my CV f1 increase followed by threshold increase. And I always choose the best CV to submit.", "votes": null }, { "id": "1298801", "postDate": "05/09/2021 08:17:06", "content": "

Ok nice strategy !

", "rawMarkdown": "Ok nice strategy !", "votes": null }, { "id": "1298880", "postDate": "05/09/2021 09:43:16", "content": "

Thanks. Additionally, have you tried efficientnet? If so, how about the performance?
\nAnother question, have you transform the image into such size (224 * 224) for resnest?

", "rawMarkdown": "Thanks. Additionally, have you tried efficientnet? If so, how about the performance?\nAnother question, have you transform the image into such size (224 * 224) for resnest?", "votes": null }, { "id": "1298909", "postDate": "05/09/2021 10:05:17", "content": "

Yes I tried. But they don't seem to outperform resnest50, they even seem worse. I can't explain why.

\n

No I don't resize the melspecs. If I were to resize the mels, I'd prefer to adjust the mels computer params in order to get bigger images instead.

", "rawMarkdown": "Yes I tried. But they don't seem to outperform resnest50, they even seem worse. I can't explain why.\n\nNo I don't resize the melspecs. If I were to resize the mels, I'd prefer to adjust the mels computer params in order to get bigger images instead.", "votes": null }, { "id": "1298956", "postDate": "05/09/2021 11:10:22", "content": "

Thanks, I also find that efficientnet perform worse.

", "rawMarkdown": "Thanks, I also find that efficientnet perform worse.", "votes": null }, { "id": "1299187", "postDate": "05/09/2021 14:27:01", "content": "

I find efficientnet family performs worse individually, couldn't make it work (yet).
\nWhen ensembled gave a nice boost to cv but not for LB :(

", "rawMarkdown": "I find efficientnet family performs worse individually, couldn't make it work (yet).\nWhen ensembled gave a nice boost to cv but not for LB :(", "votes": null }, { "id": "1300495", "postDate": "05/10/2021 14:00:30", "content": "

This is a good topic.Continue to pay attention…👀

", "rawMarkdown": "This is a good topic.Continue to pay attention...👀", "votes": null }, { "id": "1300499", "postDate": "05/10/2021 14:02:44", "content": "

How did you split the test set and training set? 5 folder?

", "rawMarkdown": "How did you split the test set and training set? 5 folder?", "votes": null }, { "id": "1300534", "postDate": "05/10/2021 14:42:49", "content": "

HI @majunfu pls have a look at @kneroma's notebook :https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-training-colab

\n

This is his training file and should help… its a good notebook to study as you try to understand how to deal with this competition

", "rawMarkdown": "HI @majunfu pls have a look at @kneroma's notebook :https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-training-colab\n\nThis is his training file and should help... its a good notebook to study as you try to understand how to deal with this competition", "votes": null }, { "id": "1300593", "postDate": "05/10/2021 15:21:10", "content": "

Thank you very much👍

", "rawMarkdown": "Thank you very much👍", "votes": null }, { "id": "1301112", "postDate": "05/11/2021 00:30:26", "content": "

Hello,did u train from this?

", "rawMarkdown": "Hello,did u train from this?", "votes": null }, { "id": "1307298", "postDate": "05/14/2021 11:13:07", "content": "

Had a hard time figuring out relationships between validation with training short clips and validation with training sound scape. I am able to train models with steadily growing training short clips validation f1 but val f1 with soundscape stuck around 0.64-0.65, and lb stuck at 0.59. Maybe this comes from domain shift?

", "rawMarkdown": "Had a hard time figuring out relationships between validation with training short clips and validation with training sound scape. I am able to train models with steadily growing training short clips validation f1 but val f1 with soundscape stuck around 0.64-0.65, and lb stuck at 0.59. Maybe this comes from domain shift?", "votes": null }, { "id": "1323956", "postDate": "05/26/2021 14:25:41", "content": "

I couldn't manage to get stable CV vs LB. I guess it is due to the small validation set and public LB size.

", "rawMarkdown": "I couldn't manage to get stable CV vs LB. I guess it is due to the small validation set and public LB size.", "votes": null } ], "comments": [ { "id": 1278531, "author_name": "kmldas", "author_url": "", "post_date": "04/20/2021 02:40:54", "content": "

Here are the results of some experiments using @kneroma notebook https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference

\n

f1 score -> Public LB score

\n

0.67766 -> 0.59

\n

0.685153-> 0.61 (v1 of notebook above)

\n

0.692444-> 0.61

\n

0.695736-> 0.62

\n

0.702431-> 0.64

\n

the validation F1 scores in the notebook showcase a good indication of the Public LB scores.

", "votes": null, "replies": [ { "id": 1278577, "author_name": "kneroma", "author_url": "", "post_date": "04/20/2021 03:55:20", "content": "

Thanks for sharing @kmldas . I got the same feeling about the CV in here. More again, it seems like higher CVs lead to higher LB correlation. As an example, the difference between this CV and my current LB is only about 0.01.

", "votes": null, "replies": [] }, { "id": 1300499, "author_name": "majunfu", "author_url": "", "post_date": "05/10/2021 14:02:44", "content": "

How did you split the test set and training set? 5 folder?

", "votes": null, "replies": [] }, { "id": 1300534, "author_name": "kmldas", "author_url": "", "post_date": "05/10/2021 14:42:49", "content": "

HI @majunfu pls have a look at @kneroma's notebook :https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-training-colab

\n

This is his training file and should help… its a good notebook to study as you try to understand how to deal with this competition

", "votes": null, "replies": [] }, { "id": 1300593, "author_name": "majunfu", "author_url": "", "post_date": "05/10/2021 15:21:10", "content": "

Thank you very much👍

", "votes": null, "replies": [] }, { "id": 1301112, "author_name": "zekunn", "author_url": "", "post_date": "05/11/2021 00:30:26", "content": "

Hello,did u train from this?

", "votes": null, "replies": [] } ] }, { "id": 1298633, "author_name": "whurobin", "author_url": "", "post_date": "05/09/2021 04:49:02", "content": "

Here are the results of some experiments using @kneroma's notebook
\nversion f1 score/threshold -> Public LB score
\n7 0.6574/0.74 -> 0.58

\n

6 0.6536/0.67 -> 0.57

\n

5 0.654236/0.72 -> 0.60

\n

4 0.652847/0.7 -> 0.60

\n

3 0.655278/0.4 -> 0.56

\n

2 0.643472/0.8 -> 0.57

\n

1 0.535472/0.5 -> 0.58

", "votes": null, "replies": [ { "id": 1298771, "author_name": "kneroma", "author_url": "", "post_date": "05/09/2021 07:21:45", "content": "

Nice one. If you can train 2 different models and submit their average prediction, your LB could significantly improve. In fact, what is the score of the bag of your 5 cited models ?

", "votes": null, "replies": [] }, { "id": 1298782, "author_name": "whurobin", "author_url": "", "post_date": "05/09/2021 07:37:18", "content": "

I haven't bag models, I just trained a singal model with your data fold0 as validation dataset, I want to find a good baseline and do next progress.
\nSomething different with your inference notebook is that my threshold is high, the 0.60 LB used sheshold 0.7 not 0.25 (I use resnest50 too). And I find that my CV f1 increase followed by threshold increase. And I always choose the best CV to submit.

", "votes": null, "replies": [] }, { "id": 1298801, "author_name": "kneroma", "author_url": "", "post_date": "05/09/2021 08:17:06", "content": "

Ok nice strategy !

", "votes": null, "replies": [] }, { "id": 1298880, "author_name": "whurobin", "author_url": "", "post_date": "05/09/2021 09:43:16", "content": "

Thanks. Additionally, have you tried efficientnet? If so, how about the performance?
\nAnother question, have you transform the image into such size (224 * 224) for resnest?

", "votes": null, "replies": [] }, { "id": 1298909, "author_name": "kneroma", "author_url": "", "post_date": "05/09/2021 10:05:17", "content": "

Yes I tried. But they don't seem to outperform resnest50, they even seem worse. I can't explain why.

\n

No I don't resize the melspecs. If I were to resize the mels, I'd prefer to adjust the mels computer params in order to get bigger images instead.

", "votes": null, "replies": [] }, { "id": 1298956, "author_name": "whurobin", "author_url": "", "post_date": "05/09/2021 11:10:22", "content": "

Thanks, I also find that efficientnet perform worse.

", "votes": null, "replies": [] }, { "id": 1299187, "author_name": "jy2tong", "author_url": "", "post_date": "05/09/2021 14:27:01", "content": "

I find efficientnet family performs worse individually, couldn't make it work (yet).
\nWhen ensembled gave a nice boost to cv but not for LB :(

", "votes": null, "replies": [] } ] }, { "id": 1300495, "author_name": "majunfu", "author_url": "", "post_date": "05/10/2021 14:00:30", "content": "

This is a good topic.Continue to pay attention…👀

", "votes": null, "replies": [] }, { "id": 1307298, "author_name": "tonychenxyz", "author_url": "", "post_date": "05/14/2021 11:13:07", "content": "

Had a hard time figuring out relationships between validation with training short clips and validation with training sound scape. I am able to train models with steadily growing training short clips validation f1 but val f1 with soundscape stuck around 0.64-0.65, and lb stuck at 0.59. Maybe this comes from domain shift?

", "votes": null, "replies": [] }, { "id": 1323956, "author_name": "aerdem4", "author_url": "", "post_date": "05/26/2021 14:25:41", "content": "

I couldn't manage to get stable CV vs LB. I guess it is due to the small validation set and public LB size.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1273829": "Hi All,\n\nwanted to check how your CV compares with the public LB?\nAre you seeing a good correlation? as we have limited submissions in this, would be helpful if you can share what you are seeing...\n\nI have not done many experiments and submission -- but for me the CV score and public LB has good correlation in the ones I have.\n\nHopefully, this means we may trust our CV -- pls do share if anyone notices anything different in your experiments \n\nThanks!", "1278531": "Here are the results of some experiments using @kneroma notebook [https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference](https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference)\n\n f1 score -> Public LB score\n\n0.67766 -> 0.59\n\n0.685153-> 0.61 (v1 of notebook above)\n\n0.692444-> 0.61\n\n0.695736-> 0.62\n\n 0.702431-> 0.64\n\n\nthe validation F1 scores in the notebook showcase a good indication of the Public LB scores.", "1278577": "Thanks for sharing @kmldas . I got the same feeling about the CV in [here](https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference). More again, it seems like higher CVs lead to higher LB correlation. As an example, the difference between this CV and my current LB is only about **0.01**.", "1298633": "Here are the results of some experiments using @kneroma's [notebook](https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-inference)\nversion f1 score/threshold -> Public LB score \n7 0.6574/0.74 -> 0.58\n\n6 0.6536/0.67 -> 0.57\n\n5 0.654236/0.72 -> 0.60\n\n4 0.652847/0.7 -> 0.60\n\n3 0.655278/0.4 -> 0.56\n\n2 0.643472/0.8 -> 0.57\n\n1 0.535472/0.5 -> 0.58", "1298771": "Nice one. If you can train 2 different models and submit their average prediction, your LB could significantly improve. In fact, what is the score of the bag of your 5 cited models ?", "1298782": "I haven't bag models, I just trained a singal model with your data fold0 as validation dataset, I want to find a good baseline and do next progress.\nSomething different with your inference notebook is that my threshold is high, the 0.60 LB used sheshold 0.7 not 0.25 (I use resnest50 too). And I find that my CV f1 increase followed by threshold increase. And I always choose the best CV to submit.", "1298801": "Ok nice strategy !", "1298880": "Thanks. Additionally, have you tried efficientnet? If so, how about the performance?\nAnother question, have you transform the image into such size (224 * 224) for resnest?", "1298909": "Yes I tried. But they don't seem to outperform resnest50, they even seem worse. I can't explain why.\n\nNo I don't resize the melspecs. If I were to resize the mels, I'd prefer to adjust the mels computer params in order to get bigger images instead.", "1298956": "Thanks, I also find that efficientnet perform worse.", "1299187": "I find efficientnet family performs worse individually, couldn't make it work (yet).\nWhen ensembled gave a nice boost to cv but not for LB :(", "1300495": "This is a good topic.Continue to pay attention...👀", "1300499": "How did you split the test set and training set? 5 folder?", "1300534": "HI @majunfu pls have a look at @kneroma's notebook :https://www.kaggle.com/kneroma/clean-fast-simple-bird-identifier-training-colab\n\nThis is his training file and should help... its a good notebook to study as you try to understand how to deal with this competition", "1300593": "Thank you very much👍", "1301112": "Hello,did u train from this?", "1307298": "Had a hard time figuring out relationships between validation with training short clips and validation with training sound scape. I am able to train models with steadily growing training short clips validation f1 but val f1 with soundscape stuck around 0.64-0.65, and lb stuck at 0.59. Maybe this comes from domain shift?", "1323956": "I couldn't manage to get stable CV vs LB. I guess it is due to the small validation set and public LB size." }, "source": "meta" }