{ "id": 395185, "title": "Single Model CV-LB Thread", "url": "/competitions/birdclef-2023/discussion/395185", "author_name": "", "post_date": "2023-03-16T07:06:55.617455Z", "votes": 20, "comment_count": 20, "views": 0, "content": "

Here are my results so far:

\n

Used Efficientnet series models , Single 80-20% train test stratified split based on this notebook (link)

\n

CV - LB

\n

0.810 - 0.77
\n0.831 - 0.78
\n0.836 - 0.78 (higher rank)
\n0.845 - 0.79
\n0.851 - 0.79 (higher rank)

", "messages": [ { "id": "2184150", "postDate": "03/16/2023 07:06:55", "content": "

Here are my results so far:

\n

Used Efficientnet series models , Single 80-20% train test stratified split based on this notebook (link)

\n

CV - LB

\n

0.810 - 0.77
\n0.831 - 0.78
\n0.836 - 0.78 (higher rank)
\n0.845 - 0.79
\n0.851 - 0.79 (higher rank)

", "rawMarkdown": "Here are my results so far:\n\nUsed Efficientnet series models , Single 80-20% train test stratified split based on this [notebook (link)](https://www.kaggle.com/code/nischaydnk/split-creating-melspecs-stage-1)\n\n\nCV - LB\n\n0.810 - 0.77\n0.831 - 0.78\n0.836 - 0.78 (higher rank)\n0.845 - 0.79 \n**0.851** - **0.79** (higher rank)", "votes": null }, { "id": "2184298", "postDate": "03/16/2023 09:31:36", "content": "

Same model, split and data @nischaydnk uses in his public notebook to get .77

\n

CV: .823 LB: .77 Change: Remove Augmentations Except CutMix
\nCV: .872 LB: .77 (higher rank) Change: Train more epochs with better learning rate scheduling and BCELoss

\n

Looks like not every way of improving CV will improve the LB

", "rawMarkdown": "Same model, split and data @nischaydnk uses in his public notebook to get .77\n\nCV: .823 LB: .77 Change: Remove Augmentations Except CutMix\nCV: .872 LB: .77 (higher rank) Change: Train more epochs with better learning rate scheduling and BCELoss\n\nLooks like not every way of improving CV will improve the LB", "votes": null }, { "id": "2184386", "postDate": "03/16/2023 10:38:50", "content": "

eca-nfnet-l0, 3s audio sample, CV 0.7859, LB 0.78

", "rawMarkdown": "eca-nfnet-l0, 3s audio sample, CV 0.7859, LB 0.78", "votes": null }, { "id": "2184435", "postDate": "03/16/2023 11:38:15", "content": "

Model: efficientnet_b3
\nCV Setup: Single fold 80%-20% split using primary label for stratification. For validation only validating on the first 5 seconds of each audio at 32k sample rate
\nCV with 5 padding rows: 0.8949
\nLB = 0.78

\n

Update:
\nSame split
\nCV: 0.9061
\nLB: 0.8

", "rawMarkdown": "Model: efficientnet_b3\nCV Setup: Single fold 80%-20% split using primary label for stratification. For validation only validating on the first 5 seconds of each audio at 32k sample rate\nCV with 5 padding rows: 0.8949\nLB = 0.78\n\nUpdate:\nSame split\nCV: 0.9061\nLB: 0.8", "votes": null }, { "id": "2184448", "postDate": "03/16/2023 11:45:47", "content": "

thanks for reporting, mhhh I think we have similar split. don't know why cv-lb gap is too large for you & @harshitsheoran 👀

", "rawMarkdown": "thanks for reporting, mhhh I think we have similar split. don't know why cv-lb gap is too large for you & @harshitsheoran 👀", "votes": null }, { "id": "2184461", "postDate": "03/16/2023 11:56:03", "content": "

What part of the validation audio are you using?, I am using the first 5 seconds because I think that the signal is clearer, this could lead to better CV. I have not tried a validation with the entire signal, maybe the entire signal transformed to a spectrogram will be more noisy therefore worst cv.

\n

Also are you using secondary labels as targets, or only primary label?

", "rawMarkdown": "What part of the validation audio are you using?, I am using the first 5 seconds because I think that the signal is clearer, this could lead to better CV. I have not tried a validation with the entire signal, maybe the entire signal transformed to a spectrogram will be more noisy therefore worst cv.\n\nAlso are you using secondary labels as targets, or only primary label?", "votes": null }, { "id": "2184473", "postDate": "03/16/2023 12:04:23", "content": "

Model: PANNs architecture(eca_nfnet_l0)
\nCV:0.91
\nLB:0.81

", "rawMarkdown": "Model: PANNs architecture(eca_nfnet_l0)\nCV:0.91\nLB:0.81", "votes": null }, { "id": "2184508", "postDate": "03/16/2023 12:21:09", "content": "

My current solution is not much different from the public baseline (link) I have shared. I am also validating on only first 5 seconds, might not be the best way to judge the model, and you are also right about using entire signal.
\nI'm not using secondary labels yet.

\n

There are ~200 test audios each with 120 samples of 5 seconds each, making it around 200*120 = 24,000 samples.

\n

Meanwhile in validation for first 5 seconds ---> ~3400 samples

\n

That makes me think maybe we can pad lesser sample (2,3) instead of padding = 5 could be a better approach to comment over CV keeping the ratio of padded samples / total samples balanced.

\n

Or we may use multiple samples for each audios for better judgement and also try out post processing at audio level.

", "rawMarkdown": "My current solution is not much different from the public [baseline (link)](https://www.kaggle.com/competitions/birdclef-2023/discussion/394162) I have shared. I am also validating on only first 5 seconds, might not be the best way to judge the model, and you are also right about using entire signal. \nI'm not using secondary labels yet. \n\nThere are ~200 test audios each with 120 samples of 5 seconds each, making it around 200*120 = 24,000 samples. \n\nMeanwhile in validation for first 5 seconds ---> ~3400 samples\n\nThat makes me think maybe we can pad lesser sample (2,3) instead of padding = 5 could be a better approach to comment over CV keeping the ratio of padded samples / total samples balanced. \n\nOr we may use multiple samples for each audios for better judgement and also try out post processing at audio level.", "votes": null }, { "id": "2184516", "postDate": "03/16/2023 12:27:41", "content": "

Nice score. Do you mind sharing your CV setup @myso1987 ??

", "rawMarkdown": "Nice score. Do you mind sharing your CV setup @myso1987 ??", "votes": null }, { "id": "2184520", "postDate": "03/16/2023 12:29:39", "content": "

I agree we should respect the ratio of padded rows, so we are actually doing the same, maybe a good idea could be to expand the validation to all the 5 seconds chunks that the validation has for a better CV.

\n

I can't explain cv gab for now 😄

", "rawMarkdown": "I agree we should respect the ratio of padded rows, so we are actually doing the same, maybe a good idea could be to expand the validation to all the 5 seconds chunks that the validation has for a better CV.\n\nI can't explain cv gab for now 😄", "votes": null }, { "id": "2186586", "postDate": "03/17/2023 23:18:29", "content": "

Model: tf_efficientnetv2_s
\nCV:0.679
\nLB:0.78

", "rawMarkdown": "Model: tf_efficientnetv2_s\nCV:0.679\nLB:0.78", "votes": null }, { "id": "2188773", "postDate": "03/20/2023 01:17:01", "content": "

Did you train on 3-second samples and predicted on 5 secs?
\nIf yes, then which 3 sec samples you choose to train?

", "rawMarkdown": "Did you train on 3-second samples and predicted on 5 secs?\nIf yes, then which 3 sec samples you choose to train?", "votes": null }, { "id": "2189806", "postDate": "03/20/2023 19:08:30", "content": "

Experiment 1:

\n

Model: eca_nfnet_l0
\nNo Mixup
\nSimple Aug: Horizontal and Vertical Flip on Spec.
\nNo upsampling or downsampling
\n20 Epochs.

\n

LB: 0.78
\nCMAP@5: 0-15 sec validation 0.88
\nCMAP@5: 0-5 sec validation 0.845
\nCMAP@5: 5-10 sec validation 0.81

", "rawMarkdown": "Experiment 1:\n\nModel: eca_nfnet_l0\nNo Mixup\nSimple Aug: Horizontal and Vertical Flip on Spec.\nNo upsampling or downsampling\n20 Epochs.\n\nLB: 0.78\nCMAP@5: 0-15 sec validation 0.88\nCMAP@5: 0-5 sec validation 0.845\nCMAP@5: 5-10 sec validation 0.81", "votes": null }, { "id": "2191650", "postDate": "03/22/2023 05:32:48", "content": "

Experiment 2

\n

CV 0.875
\nLB 0.8

\n

eca_nfnet_l0
\nno external data

", "rawMarkdown": "Experiment 2\n\nCV 0.875\nLB 0.8\n\neca_nfnet_l0\nno external data", "votes": null }, { "id": "2193118", "postDate": "03/23/2023 05:15:13", "content": "

Single model, simple split 80/20 (assuring all classes are in training set)

\n", "rawMarkdown": "Single model, simple split 80/20 (assuring all classes are in training set)\n\n- SED model, backbone tf_efficientnet_b0_ns\n- Audio augmentations, image augmentations, cutmix, mixup\n- CV: 0.78036 (padded cmap)\n- LB: 0.78", "votes": null }, { "id": "2193693", "postDate": "03/23/2023 12:27:34", "content": "", "rawMarkdown": "Baseline - Random split 80\\20\nefficientnet_b0\nCV: 0.767 -> LB: 0.76", "votes": null }, { "id": "2210939", "postDate": "04/05/2023 18:08:32", "content": "

efficientb0_ns, single fold, cv 0.87 lb 0.79. I took the cMAP method from your notebook, so it's a fair comparison to yours. Thanks for sharing it btw.

", "rawMarkdown": "efficientb0_ns, single fold, cv 0.87 lb 0.79. I took the cMAP method from your [notebook](https://www.kaggle.com/code/nischaydnk/birdclef-2023-pytorch-lightning-training-w-cmap), so it's a fair comparison to yours. Thanks for sharing it btw.", "votes": null }, { "id": "2210975", "postDate": "04/05/2023 18:29:42", "content": "

hey @nischaydnk have you tried eca_nfnet_l0 this model on your cv split ?

", "rawMarkdown": "hey @nischaydnk have you tried eca_nfnet_l0 this model on your cv split ?", "votes": null }, { "id": "2210977", "postDate": "04/05/2023 18:30:15", "content": "

hi @hinepo can you share some details about SED Model architecture ?

", "rawMarkdown": "hi @hinepo can you share some details about SED Model architecture ?", "votes": null }, { "id": "2211001", "postDate": "04/05/2023 18:43:27", "content": "

Hi @vpkprasanna

\n

I am using a model similar to this one, in cell 17

\n

For architecture details, maybe you should visit this link

", "rawMarkdown": "Hi @vpkprasanna \n\nI am using a model similar to [this one](https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0), in cell 17\n\nFor architecture details, maybe you should visit [this link](https://www.kaggle.com/competitions/rfcx-species-audio-detection/discussion/211007#1151299)", "votes": null }, { "id": "2237729", "postDate": "04/27/2023 23:16:04", "content": "

Model: eca_nfnet_l0

\n

Features: melspectrogram

\n

CV (hold-out): 0.86 (with secondary label)

\n

LB : 0.8 (Near 0.81)

\n

no pretrain, no external data

\n

CV is by far better without secondary label (cv: 0.908), but LB is somewhat better with secondary label.

", "rawMarkdown": "Model: eca_nfnet_l0\n\nFeatures: melspectrogram\n\nCV (hold-out): 0.86 (with secondary label)\n\nLB : 0.8 (Near 0.81)\n\nno pretrain, no external data\n\nCV is by far better without secondary label (cv: 0.908), but LB is somewhat better with secondary label.", "votes": null } ], "comments": [ { "id": 2184298, "author_name": "harshitsheoran", "author_url": "", "post_date": "03/16/2023 09:31:36", "content": "

Same model, split and data @nischaydnk uses in his public notebook to get .77

\n

CV: .823 LB: .77 Change: Remove Augmentations Except CutMix
\nCV: .872 LB: .77 (higher rank) Change: Train more epochs with better learning rate scheduling and BCELoss

\n

Looks like not every way of improving CV will improve the LB

", "votes": null, "replies": [] }, { "id": 2184386, "author_name": "aryankhatana", "author_url": "", "post_date": "03/16/2023 10:38:50", "content": "

eca-nfnet-l0, 3s audio sample, CV 0.7859, LB 0.78

", "votes": null, "replies": [ { "id": 2188773, "author_name": "salmanahmedtamu", "author_url": "", "post_date": "03/20/2023 01:17:01", "content": "

Did you train on 3-second samples and predicted on 5 secs?
\nIf yes, then which 3 sec samples you choose to train?

", "votes": null, "replies": [] } ] }, { "id": 2184435, "author_name": "ragnar123", "author_url": "", "post_date": "03/16/2023 11:38:15", "content": "

Model: efficientnet_b3
\nCV Setup: Single fold 80%-20% split using primary label for stratification. For validation only validating on the first 5 seconds of each audio at 32k sample rate
\nCV with 5 padding rows: 0.8949
\nLB = 0.78

\n

Update:
\nSame split
\nCV: 0.9061
\nLB: 0.8

", "votes": null, "replies": [ { "id": 2184448, "author_name": "nischaydnk", "author_url": "", "post_date": "03/16/2023 11:45:47", "content": "

thanks for reporting, mhhh I think we have similar split. don't know why cv-lb gap is too large for you & @harshitsheoran 👀

", "votes": null, "replies": [ { "id": 2184461, "author_name": "ragnar123", "author_url": "", "post_date": "03/16/2023 11:56:03", "content": "

What part of the validation audio are you using?, I am using the first 5 seconds because I think that the signal is clearer, this could lead to better CV. I have not tried a validation with the entire signal, maybe the entire signal transformed to a spectrogram will be more noisy therefore worst cv.

\n

Also are you using secondary labels as targets, or only primary label?

", "votes": null, "replies": [ { "id": 2184508, "author_name": "nischaydnk", "author_url": "", "post_date": "03/16/2023 12:21:09", "content": "

My current solution is not much different from the public baseline (link) I have shared. I am also validating on only first 5 seconds, might not be the best way to judge the model, and you are also right about using entire signal.
\nI'm not using secondary labels yet.

\n

There are ~200 test audios each with 120 samples of 5 seconds each, making it around 200*120 = 24,000 samples.

\n

Meanwhile in validation for first 5 seconds ---> ~3400 samples

\n

That makes me think maybe we can pad lesser sample (2,3) instead of padding = 5 could be a better approach to comment over CV keeping the ratio of padded samples / total samples balanced.

\n

Or we may use multiple samples for each audios for better judgement and also try out post processing at audio level.

", "votes": null, "replies": [ { "id": 2184520, "author_name": "ragnar123", "author_url": "", "post_date": "03/16/2023 12:29:39", "content": "

I agree we should respect the ratio of padded rows, so we are actually doing the same, maybe a good idea could be to expand the validation to all the 5 seconds chunks that the validation has for a better CV.

\n

I can't explain cv gab for now 😄

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2184473, "author_name": "myso1987", "author_url": "", "post_date": "03/16/2023 12:04:23", "content": "

Model: PANNs architecture(eca_nfnet_l0)
\nCV:0.91
\nLB:0.81

", "votes": null, "replies": [ { "id": 2184516, "author_name": "nischaydnk", "author_url": "", "post_date": "03/16/2023 12:27:41", "content": "

Nice score. Do you mind sharing your CV setup @myso1987 ??

", "votes": null, "replies": [] } ] }, { "id": 2186586, "author_name": "shigengtian", "author_url": "", "post_date": "03/17/2023 23:18:29", "content": "

Model: tf_efficientnetv2_s
\nCV:0.679
\nLB:0.78

", "votes": null, "replies": [] }, { "id": 2189806, "author_name": "salmanahmedtamu", "author_url": "", "post_date": "03/20/2023 19:08:30", "content": "

Experiment 1:

\n

Model: eca_nfnet_l0
\nNo Mixup
\nSimple Aug: Horizontal and Vertical Flip on Spec.
\nNo upsampling or downsampling
\n20 Epochs.

\n

LB: 0.78
\nCMAP@5: 0-15 sec validation 0.88
\nCMAP@5: 0-5 sec validation 0.845
\nCMAP@5: 5-10 sec validation 0.81

", "votes": null, "replies": [] }, { "id": 2191650, "author_name": "salmanahmedtamu", "author_url": "", "post_date": "03/22/2023 05:32:48", "content": "

Experiment 2

\n

CV 0.875
\nLB 0.8

\n

eca_nfnet_l0
\nno external data

", "votes": null, "replies": [] }, { "id": 2193118, "author_name": "hinepo", "author_url": "", "post_date": "03/23/2023 05:15:13", "content": "

Single model, simple split 80/20 (assuring all classes are in training set)

\n", "votes": null, "replies": [ { "id": 2210977, "author_name": "vpkprasanna", "author_url": "", "post_date": "04/05/2023 18:30:15", "content": "

hi @hinepo can you share some details about SED Model architecture ?

", "votes": null, "replies": [ { "id": 2211001, "author_name": "hinepo", "author_url": "", "post_date": "04/05/2023 18:43:27", "content": "

Hi @vpkprasanna

\n

I am using a model similar to this one, in cell 17

\n

For architecture details, maybe you should visit this link

", "votes": null, "replies": [] } ] } ] }, { "id": 2193693, "author_name": "paulojunqueira", "author_url": "", "post_date": "03/23/2023 12:27:34", "content": "", "votes": null, "replies": [] }, { "id": 2210939, "author_name": "leonshangguan", "author_url": "", "post_date": "04/05/2023 18:08:32", "content": "

efficientb0_ns, single fold, cv 0.87 lb 0.79. I took the cMAP method from your notebook, so it's a fair comparison to yours. Thanks for sharing it btw.

", "votes": null, "replies": [] }, { "id": 2210975, "author_name": "vpkprasanna", "author_url": "", "post_date": "04/05/2023 18:29:42", "content": "

hey @nischaydnk have you tried eca_nfnet_l0 this model on your cv split ?

", "votes": null, "replies": [] }, { "id": 2237729, "author_name": "moritake04", "author_url": "", "post_date": "04/27/2023 23:16:04", "content": "

Model: eca_nfnet_l0

\n

Features: melspectrogram

\n

CV (hold-out): 0.86 (with secondary label)

\n

LB : 0.8 (Near 0.81)

\n

no pretrain, no external data

\n

CV is by far better without secondary label (cv: 0.908), but LB is somewhat better with secondary label.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2184150": "Here are my results so far:\n\nUsed Efficientnet series models , Single 80-20% train test stratified split based on this [notebook (link)](https://www.kaggle.com/code/nischaydnk/split-creating-melspecs-stage-1)\n\n\nCV - LB\n\n0.810 - 0.77\n0.831 - 0.78\n0.836 - 0.78 (higher rank)\n0.845 - 0.79 \n**0.851** - **0.79** (higher rank)", "2184298": "Same model, split and data @nischaydnk uses in his public notebook to get .77\n\nCV: .823 LB: .77 Change: Remove Augmentations Except CutMix\nCV: .872 LB: .77 (higher rank) Change: Train more epochs with better learning rate scheduling and BCELoss\n\nLooks like not every way of improving CV will improve the LB", "2184386": "eca-nfnet-l0, 3s audio sample, CV 0.7859, LB 0.78", "2184435": "Model: efficientnet_b3\nCV Setup: Single fold 80%-20% split using primary label for stratification. For validation only validating on the first 5 seconds of each audio at 32k sample rate\nCV with 5 padding rows: 0.8949\nLB = 0.78\n\nUpdate:\nSame split\nCV: 0.9061\nLB: 0.8", "2184448": "thanks for reporting, mhhh I think we have similar split. don't know why cv-lb gap is too large for you & @harshitsheoran 👀", "2184461": "What part of the validation audio are you using?, I am using the first 5 seconds because I think that the signal is clearer, this could lead to better CV. I have not tried a validation with the entire signal, maybe the entire signal transformed to a spectrogram will be more noisy therefore worst cv.\n\nAlso are you using secondary labels as targets, or only primary label?", "2184473": "Model: PANNs architecture(eca_nfnet_l0)\nCV:0.91\nLB:0.81", "2184508": "My current solution is not much different from the public [baseline (link)](https://www.kaggle.com/competitions/birdclef-2023/discussion/394162) I have shared. I am also validating on only first 5 seconds, might not be the best way to judge the model, and you are also right about using entire signal. \nI'm not using secondary labels yet. \n\nThere are ~200 test audios each with 120 samples of 5 seconds each, making it around 200*120 = 24,000 samples. \n\nMeanwhile in validation for first 5 seconds ---> ~3400 samples\n\nThat makes me think maybe we can pad lesser sample (2,3) instead of padding = 5 could be a better approach to comment over CV keeping the ratio of padded samples / total samples balanced. \n\nOr we may use multiple samples for each audios for better judgement and also try out post processing at audio level.", "2184516": "Nice score. Do you mind sharing your CV setup @myso1987 ??", "2184520": "I agree we should respect the ratio of padded rows, so we are actually doing the same, maybe a good idea could be to expand the validation to all the 5 seconds chunks that the validation has for a better CV.\n\nI can't explain cv gab for now 😄", "2186586": "Model: tf_efficientnetv2_s\nCV:0.679\nLB:0.78", "2188773": "Did you train on 3-second samples and predicted on 5 secs?\nIf yes, then which 3 sec samples you choose to train?", "2189806": "Experiment 1:\n\nModel: eca_nfnet_l0\nNo Mixup\nSimple Aug: Horizontal and Vertical Flip on Spec.\nNo upsampling or downsampling\n20 Epochs.\n\nLB: 0.78\nCMAP@5: 0-15 sec validation 0.88\nCMAP@5: 0-5 sec validation 0.845\nCMAP@5: 5-10 sec validation 0.81", "2191650": "Experiment 2\n\nCV 0.875\nLB 0.8\n\neca_nfnet_l0\nno external data", "2193118": "Single model, simple split 80/20 (assuring all classes are in training set)\n\n- SED model, backbone tf_efficientnet_b0_ns\n- Audio augmentations, image augmentations, cutmix, mixup\n- CV: 0.78036 (padded cmap)\n- LB: 0.78", "2193693": "Baseline - Random split 80\\20\nefficientnet_b0\nCV: 0.767 -> LB: 0.76", "2210939": "efficientb0_ns, single fold, cv 0.87 lb 0.79. I took the cMAP method from your [notebook](https://www.kaggle.com/code/nischaydnk/birdclef-2023-pytorch-lightning-training-w-cmap), so it's a fair comparison to yours. Thanks for sharing it btw.", "2210975": "hey @nischaydnk have you tried eca_nfnet_l0 this model on your cv split ?", "2210977": "hi @hinepo can you share some details about SED Model architecture ?", "2211001": "Hi @vpkprasanna \n\nI am using a model similar to [this one](https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0), in cell 17\n\nFor architecture details, maybe you should visit [this link](https://www.kaggle.com/competitions/rfcx-species-audio-detection/discussion/211007#1151299)", "2237729": "Model: eca_nfnet_l0\n\nFeatures: melspectrogram\n\nCV (hold-out): 0.86 (with secondary label)\n\nLB : 0.8 (Near 0.81)\n\nno pretrain, no external data\n\nCV is by far better without secondary label (cv: 0.908), but LB is somewhat better with secondary label." }, "source": "meta" }