{ "id": 497563, "title": "Feathered Finale: Mels->Model->AUC Score", "url": "/competitions/birdclef-2024/discussion/497563", "author_name": "", "post_date": "2024-04-25T03:10:39.466139900Z", "votes": 5, "comment_count": 1, "views": 0, "content": "

I'm sharing my notebook that demonstrates using labeled soundscapes to generate a ROC-AUC score for your model:

\n

https://www.kaggle.com/code/richolson/feathered-finale-mels-model-auc-score/

\n

The ROC-AUC estimate for my efficientnetv2_b2-based model was 0.78 - higher than my actual LB of .59.

\n

This is a big difference - at least partly due to the soundscapes being generated on train data. I still think this approach may be useful to suggest if changes to a model are helping or hurting prior to submitting.

\n

Separate training and soundscape data should improve the ability to predict LB score. (The soundscape dataset already has a mechanism for this.)

\n

If you submit this notebook - it will automatically switch over to predicting on the test data.

\n

This notebook is the culmination of a series of notebooks on Mel generation, ImageNet training, scoring and soundscape generation:

\n

https://www.kaggle.com/code/richolson/birdclef-2024-contiguous-mel-spectrogram-generator
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-train-v2/
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-exploring-scoring
\nhttps://www.kaggle.com/code/richolson/birdtrax-birdclef-2024-labeled-soundscapes

\n

Hope this is useful to someone!

\n

-Rich

", "messages": [ { "id": "2774016", "postDate": "04/25/2024 03:10:39", "content": "

I'm sharing my notebook that demonstrates using labeled soundscapes to generate a ROC-AUC score for your model:

\n

https://www.kaggle.com/code/richolson/feathered-finale-mels-model-auc-score/

\n

The ROC-AUC estimate for my efficientnetv2_b2-based model was 0.78 - higher than my actual LB of .59.

\n

This is a big difference - at least partly due to the soundscapes being generated on train data. I still think this approach may be useful to suggest if changes to a model are helping or hurting prior to submitting.

\n

Separate training and soundscape data should improve the ability to predict LB score. (The soundscape dataset already has a mechanism for this.)

\n

If you submit this notebook - it will automatically switch over to predicting on the test data.

\n

This notebook is the culmination of a series of notebooks on Mel generation, ImageNet training, scoring and soundscape generation:

\n

https://www.kaggle.com/code/richolson/birdclef-2024-contiguous-mel-spectrogram-generator
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-train-v2/
\nhttps://www.kaggle.com/code/richolson/birdclef-2024-exploring-scoring
\nhttps://www.kaggle.com/code/richolson/birdtrax-birdclef-2024-labeled-soundscapes

\n

Hope this is useful to someone!

\n

-Rich

", "rawMarkdown": "I'm sharing my notebook that demonstrates using labeled soundscapes to generate a ROC-AUC score for your model:\n\nhttps://www.kaggle.com/code/richolson/feathered-finale-mels-model-auc-score/\n\nThe ROC-AUC estimate for my efficientnetv2_b2-based model was 0.78 - higher than my actual LB of .59.\n\nThis is a big difference - at least partly due to the soundscapes being generated on train data. I still think this approach may be useful to suggest if changes to a model are helping or hurting prior to submitting.\n\nSeparate training and soundscape data should improve the ability to predict LB score. (The soundscape dataset already has a mechanism for this.)\n\nIf you submit this notebook - it will automatically switch over to predicting on the test data.\n\nThis notebook is the culmination of a series of notebooks on Mel generation, ImageNet training, scoring and soundscape generation:\n\nhttps://www.kaggle.com/code/richolson/birdclef-2024-contiguous-mel-spectrogram-generator\nhttps://www.kaggle.com/code/richolson/birdclef-2024-train-v2/\nhttps://www.kaggle.com/code/richolson/birdclef-2024-exploring-scoring\nhttps://www.kaggle.com/code/richolson/birdtrax-birdclef-2024-labeled-soundscapes\n\nHope this is useful to someone!\n\n-Rich", "votes": null }, { "id": "2774222", "postDate": "04/25/2024 05:08:47", "content": "

This is great! Thanks for sharing

", "rawMarkdown": "This is great! Thanks for sharing", "votes": null } ], "comments": [ { "id": 2774222, "author_name": "elenilazaridou", "author_url": "", "post_date": "04/25/2024 05:08:47", "content": "

This is great! Thanks for sharing

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2774016": "I'm sharing my notebook that demonstrates using labeled soundscapes to generate a ROC-AUC score for your model:\n\nhttps://www.kaggle.com/code/richolson/feathered-finale-mels-model-auc-score/\n\nThe ROC-AUC estimate for my efficientnetv2_b2-based model was 0.78 - higher than my actual LB of .59.\n\nThis is a big difference - at least partly due to the soundscapes being generated on train data. I still think this approach may be useful to suggest if changes to a model are helping or hurting prior to submitting.\n\nSeparate training and soundscape data should improve the ability to predict LB score. (The soundscape dataset already has a mechanism for this.)\n\nIf you submit this notebook - it will automatically switch over to predicting on the test data.\n\nThis notebook is the culmination of a series of notebooks on Mel generation, ImageNet training, scoring and soundscape generation:\n\nhttps://www.kaggle.com/code/richolson/birdclef-2024-contiguous-mel-spectrogram-generator\nhttps://www.kaggle.com/code/richolson/birdclef-2024-train-v2/\nhttps://www.kaggle.com/code/richolson/birdclef-2024-exploring-scoring\nhttps://www.kaggle.com/code/richolson/birdtrax-birdclef-2024-labeled-soundscapes\n\nHope this is useful to someone!\n\n-Rich", "2774222": "This is great! Thanks for sharing" }, "source": "meta" }