{ "id": 498404, "title": "Why there is a gap between CV and LB", "url": "/competitions/birdclef-2024/discussion/498404", "author_name": "", "post_date": "2024-04-28T06:04:13.730163800Z", "votes": 52, "comment_count": 18, "views": 0, "content": "

I looked for a visual analogy of what happens and here is the closest I could think of.

\n

What we have in training data:
\n

\n

What we have in soundscapes:
\n

\n

Edit: I had to add some explanation in comments, reproducing it here for better sharing.

\n

Soundscape data is recording all sounds using omni directional mikes. It is different from Xeno Canto data where people recorded bird songs specifically with directional mikes. It is what I tried to capture in my visual analogy: train data is like an image focusing on one subject, and nothing else. soundscape is like a picture where the subject is buried among a myriad of irrelevant details.

", "messages": [ { "id": "2780297", "postDate": "04/28/2024 06:04:13", "content": "

I looked for a visual analogy of what happens and here is the closest I could think of.

\n

What we have in training data:
\n

\n

What we have in soundscapes:
\n

\n

Edit: I had to add some explanation in comments, reproducing it here for better sharing.

\n

Soundscape data is recording all sounds using omni directional mikes. It is different from Xeno Canto data where people recorded bird songs specifically with directional mikes. It is what I tried to capture in my visual analogy: train data is like an image focusing on one subject, and nothing else. soundscape is like a picture where the subject is buried among a myriad of irrelevant details.

", "rawMarkdown": "I looked for a visual analogy of what happens and here is the closest I could think of.\n\nWhat we have in training data:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F75976%2Fb5d7adbd06d2fb80257510f999d0206c%2FScreenshot%202024-04-28%20at%2008-00-02%2054%20Panneau%20Stop%20Stock%20Photos%20High-Res%20Pictures%20and%20Images%20-%20Getty%20Images.png?generation=1714284038161194&alt=media)\n\nWhat we have in soundscapes:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F75976%2Fea11d6325be2fba8a1aa6837071e6c76%2FScreenshot%202024-04-28%20at%2008-03-18%2054%20Panneau%20Stop%20Stock%20Photos%20High-Res%20Pictures%20and%20Images%20-%20Getty%20Images.png?generation=1714284232813950&alt=media)\n\nEdit: I had to add some explanation in comments, reproducing it here for better sharing.\n\nSoundscape data is recording all sounds using omni directional mikes. It is different from Xeno Canto data where people recorded bird songs specifically with directional mikes. It is what I tried to capture in my visual analogy: train data is like an image focusing on one subject, and nothing else. soundscape is like a picture where the subject is buried among a myriad of irrelevant details.", "votes": null }, { "id": "2780449", "postDate": "04/28/2024 08:18:41", "content": "

Is it a hint to which augs to use? :D

", "rawMarkdown": "Is it a hint to which augs to use? :D", "votes": null }, { "id": "2780451", "postDate": "04/28/2024 08:23:58", "content": "

No, but if you get a good idea out of it then please share! :D

", "rawMarkdown": "No, but if you get a good idea out of it then please share! :D", "votes": null }, { "id": "2780660", "postDate": "04/28/2024 11:01:14", "content": "

Use unlabeled soundscapes as background and paste xeno-canto birds onto it :]

", "rawMarkdown": "Use unlabeled soundscapes as background and paste xeno-canto birds onto it :]", "votes": null }, { "id": "2780775", "postDate": "04/28/2024 12:07:17", "content": "

I think this has been discussed in the forum already, see this post and its comments: https://www.kaggle.com/competitions/birdclef-2024/discussion/497050

", "rawMarkdown": "I think this has been discussed in the forum already, see this post and its comments: https://www.kaggle.com/competitions/birdclef-2024/discussion/497050", "votes": null }, { "id": "2781303", "postDate": "04/28/2024 17:45:47", "content": "

Do you mean to literally paste images of the birds onto the spectrograms. lol 😂

", "rawMarkdown": "Do you mean to literally paste images of the birds onto the spectrograms. lol 😂", "votes": null }, { "id": "2781437", "postDate": "04/28/2024 19:28:30", "content": "

My cv is 0.66 and lb is 0.64

", "rawMarkdown": "My cv is 0.66 and lb is 0.64", "votes": null }, { "id": "2782153", "postDate": "04/29/2024 06:55:27", "content": "

That's a great analogy :)

", "rawMarkdown": "That's a great analogy :)", "votes": null }, { "id": "2782348", "postDate": "04/29/2024 09:15:33", "content": "

Thank you, glad it resonated with you!

", "rawMarkdown": "Thank you, glad it resonated with you!", "votes": null }, { "id": "2785770", "postDate": "05/01/2024 03:00:02", "content": "

does that help in your case? I feel the unlabeled soundscapes not only include background but include some real bird sounds as well, won't it hurt the result?

", "rawMarkdown": "does that help in your case? I feel the unlabeled soundscapes not only include background but include some real bird sounds as well, won't it hurt the result?", "votes": null }, { "id": "2790013", "postDate": "05/03/2024 01:06:19", "content": "

My cv is 0.77 and lb is 0.58🤮

", "rawMarkdown": "My cv is 0.77 and lb is 0.58🤮", "votes": null }, { "id": "2809783", "postDate": "05/13/2024 01:55:22", "content": "

I think gap because there are no call chunks in test sample. And these chunks are labeled as 0. This snippet code demonstate how no call chunks effect to roc auc score https://ideone.com/uleNMv. Accuracy of label 1, 2 to 100% but ROC AUC < 0.8

", "rawMarkdown": "I think gap because there are no call chunks in test sample. And these chunks are labeled as 0. This snippet code demonstate how no call chunks effect to roc auc score https://ideone.com/uleNMv. Accuracy of label 1, 2 to 100% but ROC AUC < 0.8", "votes": null }, { "id": "2810629", "postDate": "05/13/2024 11:11:59", "content": "

I don't think this is the reason.

", "rawMarkdown": "I don't think this is the reason.", "votes": null }, { "id": "2814687", "postDate": "05/15/2024 13:55:06", "content": "

@cpmpml Is that based on the addition of nocall data having a negative effect on LB performance?

", "rawMarkdown": "cpmpml Is that based on the addition of nocall data having a negative effect on LB performance?", "votes": null }, { "id": "2817205", "postDate": "05/16/2024 18:53:18", "content": "

I don't undertsand what no call data has to do with the difficulty of the problem.

\n

Soundscape data is recording all sounds using omni directional mikes. It is different from Xeno Canto data where people recorded bird songs specifically with directional mikes. It is what I tried to capture in my visual analogy: train data is like an image focusing on one subject, and nothing else. soundscape is like a picture where the subject is buried among a myriad of irrelevant details.

", "rawMarkdown": "I don't undertsand what no call data has to do with the difficulty of the problem.\n\nSoundscape data is recording all sounds using omni directional mikes. It is different from Xeno Canto data where people recorded bird songs specifically with directional mikes. It is what I tried to capture in my visual analogy: train data is like an image focusing on one subject, and nothing else. soundscape is like a picture where the subject is buried among a myriad of irrelevant details.", "votes": null }, { "id": "2822555", "postDate": "05/18/2024 17:01:20", "content": "

After exp, I agree with you.

", "rawMarkdown": "After exp, I agree with you.", "votes": null }, { "id": "2822702", "postDate": "05/18/2024 17:56:58", "content": "

Obviously my hypotheses haven't paid off, but there are enough other calls in the unlabeled data that it seems like knowing the difference between a bird and a frog would benefit the model.

", "rawMarkdown": "Obviously my hypotheses haven't paid off, but there are enough other calls in the unlabeled data that it seems like knowing the difference between a bird and a frog would benefit the model.", "votes": null }, { "id": "2832945", "postDate": "05/24/2024 01:02:15", "content": "

@quan0095 How so? Your idea did help our team.

", "rawMarkdown": "quan0095 How so? Your idea did help our team.", "votes": null }, { "id": "2832971", "postDate": "05/24/2024 01:59:49", "content": "

No, my idea is not helpful. I add no call to training, but not improve LB.

", "rawMarkdown": "No, my idea is not helpful. I add no call to training, but not improve LB.", "votes": null } ], "comments": [ { "id": 2780449, "author_name": "martynoveduard", "author_url": "", "post_date": "04/28/2024 08:18:41", "content": "

Is it a hint to which augs to use? :D

", "votes": null, "replies": [ { "id": 2780451, "author_name": "cpmpml", "author_url": "", "post_date": "04/28/2024 08:23:58", "content": "

No, but if you get a good idea out of it then please share! :D

", "votes": null, "replies": [ { "id": 2780660, "author_name": "martynoveduard", "author_url": "", "post_date": "04/28/2024 11:01:14", "content": "

Use unlabeled soundscapes as background and paste xeno-canto birds onto it :]

", "votes": null, "replies": [ { "id": 2780775, "author_name": "cpmpml", "author_url": "", "post_date": "04/28/2024 12:07:17", "content": "

I think this has been discussed in the forum already, see this post and its comments: https://www.kaggle.com/competitions/birdclef-2024/discussion/497050

", "votes": null, "replies": [] }, { "id": 2781303, "author_name": "robbynevels", "author_url": "", "post_date": "04/28/2024 17:45:47", "content": "

Do you mean to literally paste images of the birds onto the spectrograms. lol 😂

", "votes": null, "replies": [] }, { "id": 2785770, "author_name": "leonshangguan", "author_url": "", "post_date": "05/01/2024 03:00:02", "content": "

does that help in your case? I feel the unlabeled soundscapes not only include background but include some real bird sounds as well, won't it hurt the result?

", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2781437, "author_name": "arunodhayan", "author_url": "", "post_date": "04/28/2024 19:28:30", "content": "

My cv is 0.66 and lb is 0.64

", "votes": null, "replies": [ { "id": 2790013, "author_name": "deepshark", "author_url": "", "post_date": "05/03/2024 01:06:19", "content": "

My cv is 0.77 and lb is 0.58🤮

", "votes": null, "replies": [] } ] }, { "id": 2782153, "author_name": "stefankahl", "author_url": "", "post_date": "04/29/2024 06:55:27", "content": "

That's a great analogy :)

", "votes": null, "replies": [ { "id": 2782348, "author_name": "cpmpml", "author_url": "", "post_date": "04/29/2024 09:15:33", "content": "

Thank you, glad it resonated with you!

", "votes": null, "replies": [] } ] }, { "id": 2809783, "author_name": "quan0095", "author_url": "", "post_date": "05/13/2024 01:55:22", "content": "

I think gap because there are no call chunks in test sample. And these chunks are labeled as 0. This snippet code demonstate how no call chunks effect to roc auc score https://ideone.com/uleNMv. Accuracy of label 1, 2 to 100% but ROC AUC < 0.8

", "votes": null, "replies": [ { "id": 2810629, "author_name": "cpmpml", "author_url": "", "post_date": "05/13/2024 11:11:59", "content": "

I don't think this is the reason.

", "votes": null, "replies": [ { "id": 2814687, "author_name": "willrice", "author_url": "", "post_date": "05/15/2024 13:55:06", "content": "

@cpmpml Is that based on the addition of nocall data having a negative effect on LB performance?

", "votes": null, "replies": [ { "id": 2817205, "author_name": "cpmpml", "author_url": "", "post_date": "05/16/2024 18:53:18", "content": "

I don't undertsand what no call data has to do with the difficulty of the problem.

\n

Soundscape data is recording all sounds using omni directional mikes. It is different from Xeno Canto data where people recorded bird songs specifically with directional mikes. It is what I tried to capture in my visual analogy: train data is like an image focusing on one subject, and nothing else. soundscape is like a picture where the subject is buried among a myriad of irrelevant details.

", "votes": null, "replies": [] } ] }, { "id": 2822555, "author_name": "quan0095", "author_url": "", "post_date": "05/18/2024 17:01:20", "content": "

After exp, I agree with you.

", "votes": null, "replies": [ { "id": 2822702, "author_name": "willrice", "author_url": "", "post_date": "05/18/2024 17:56:58", "content": "

Obviously my hypotheses haven't paid off, but there are enough other calls in the unlabeled data that it seems like knowing the difference between a bird and a frog would benefit the model.

", "votes": null, "replies": [] }, { "id": 2832945, "author_name": "levinguyen02", "author_url": "", "post_date": "05/24/2024 01:02:15", "content": "

@quan0095 How so? Your idea did help our team.

", "votes": null, "replies": [ { "id": 2832971, "author_name": "quan0095", "author_url": "", "post_date": "05/24/2024 01:59:49", "content": "

No, my idea is not helpful. I add no call to training, but not improve LB.

", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2780297": "I looked for a visual analogy of what happens and here is the closest I could think of.\n\nWhat we have in training data:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F75976%2Fb5d7adbd06d2fb80257510f999d0206c%2FScreenshot%202024-04-28%20at%2008-00-02%2054%20Panneau%20Stop%20Stock%20Photos%20High-Res%20Pictures%20and%20Images%20-%20Getty%20Images.png?generation=1714284038161194&alt=media)\n\nWhat we have in soundscapes:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F75976%2Fea11d6325be2fba8a1aa6837071e6c76%2FScreenshot%202024-04-28%20at%2008-03-18%2054%20Panneau%20Stop%20Stock%20Photos%20High-Res%20Pictures%20and%20Images%20-%20Getty%20Images.png?generation=1714284232813950&alt=media)\n\nEdit: I had to add some explanation in comments, reproducing it here for better sharing.\n\nSoundscape data is recording all sounds using omni directional mikes. It is different from Xeno Canto data where people recorded bird songs specifically with directional mikes. It is what I tried to capture in my visual analogy: train data is like an image focusing on one subject, and nothing else. soundscape is like a picture where the subject is buried among a myriad of irrelevant details.", "2780449": "Is it a hint to which augs to use? :D", "2780451": "No, but if you get a good idea out of it then please share! :D", "2780660": "Use unlabeled soundscapes as background and paste xeno-canto birds onto it :]", "2780775": "I think this has been discussed in the forum already, see this post and its comments: https://www.kaggle.com/competitions/birdclef-2024/discussion/497050", "2781303": "Do you mean to literally paste images of the birds onto the spectrograms. lol 😂", "2781437": "My cv is 0.66 and lb is 0.64", "2782153": "That's a great analogy :)", "2782348": "Thank you, glad it resonated with you!", "2785770": "does that help in your case? I feel the unlabeled soundscapes not only include background but include some real bird sounds as well, won't it hurt the result?", "2790013": "My cv is 0.77 and lb is 0.58🤮", "2809783": "I think gap because there are no call chunks in test sample. And these chunks are labeled as 0. This snippet code demonstate how no call chunks effect to roc auc score https://ideone.com/uleNMv. Accuracy of label 1, 2 to 100% but ROC AUC < 0.8", "2810629": "I don't think this is the reason.", "2814687": "cpmpml Is that based on the addition of nocall data having a negative effect on LB performance?", "2817205": "I don't undertsand what no call data has to do with the difficulty of the problem.\n\nSoundscape data is recording all sounds using omni directional mikes. It is different from Xeno Canto data where people recorded bird songs specifically with directional mikes. It is what I tried to capture in my visual analogy: train data is like an image focusing on one subject, and nothing else. soundscape is like a picture where the subject is buried among a myriad of irrelevant details.", "2822555": "After exp, I agree with you.", "2822702": "Obviously my hypotheses haven't paid off, but there are enough other calls in the unlabeled data that it seems like knowing the difference between a bird and a frog would benefit the model.", "2832945": "quan0095 How so? Your idea did help our team.", "2832971": "No, my idea is not helpful. I add no call to training, but not improve LB." }, "source": "meta" }