Why are CV and LB different?
\nI've come up with three reasons I think. But I may be wrong.
In the last competition, main theme is domain shift and missing labels.
\nIn this point, this competition is similar.
In this competition, I think domain shift exist. But this is not a big problem.
\nIn training sound, it contains noisy sound (white noise and pink noise).
\nBut test sound is clean(or a little noisy sound).
But I think this domain shift has no negative impact.
\nBecause it's a domain shift from noisy to clean.
I reported that missing labels exist in tp sound.
\nIf validation data contains missing labels, your CV is different from LB.
For example, y_label is not correct(contains missing label).
\n2nd annotation is missed in y_label. Then y_predict = [1,0,0] is best CV in your local. But y_predict=[1,1,0] is better in the test. In other words, no matter how much you improve your CV, it will not always work in a testing environment. The validation data is not reliable.
\nMany participants seem to use SED. And I also use SED.
\nIf you use SED(PANNs architecture), you should be careful.
If you use SED like below, there is a gap training and prediction.
\nframewise_outputWeak label training is fitted clipwise_output not framewise_output. clipwise_output is a time-compressed version of framewise_output. clipwise_output is correlated with framewise_output, but not equal.
Comparatively, framewise_output prediction is good at short sound event. In this competition, there is many short sound event. Therefore I use framewise_output for prediction. But there is a gap training and prediction. Then CV(clipwise_output training) is different from LB(framewise_output prediction).
If you change validation strategy, you may avoid this problem. For example, in validation phase you use np.max(framewise_output, axis=time) instead of clipwise_output. This strategy may improve a gap CV and LB.
Finally, I don't have any above countermeasures. Because I cannot fix 2. Missing labels.
\nMy strategy is \"trust LB\" not \"trust CV\". Fortunately, public LB is almost the same as private LB. I believe shaking almost never happens.
", "messages": [ { "id": "1144085", "postDate": "01/08/2021 08:17:50", "content": "Why are CV and LB different?
\nI've come up with three reasons I think. But I may be wrong.
In the last competition, main theme is domain shift and missing labels.
\nIn this point, this competition is similar.
In this competition, I think domain shift exist. But this is not a big problem.
\nIn training sound, it contains noisy sound (white noise and pink noise).
\nBut test sound is clean(or a little noisy sound).
But I think this domain shift has no negative impact.
\nBecause it's a domain shift from noisy to clean.
I reported that missing labels exist in tp sound.
\nIf validation data contains missing labels, your CV is different from LB.
For example, y_label is not correct(contains missing label).
\n2nd annotation is missed in y_label. Then y_predict = [1,0,0] is best CV in your local. But y_predict=[1,1,0] is better in the test. In other words, no matter how much you improve your CV, it will not always work in a testing environment. The validation data is not reliable.
\nMany participants seem to use SED. And I also use SED.
\nIf you use SED(PANNs architecture), you should be careful.
If you use SED like below, there is a gap training and prediction.
\nframewise_outputWeak label training is fitted clipwise_output not framewise_output. clipwise_output is a time-compressed version of framewise_output. clipwise_output is correlated with framewise_output, but not equal.
Comparatively, framewise_output prediction is good at short sound event. In this competition, there is many short sound event. Therefore I use framewise_output for prediction. But there is a gap training and prediction. Then CV(clipwise_output training) is different from LB(framewise_output prediction).
If you change validation strategy, you may avoid this problem. For example, in validation phase you use np.max(framewise_output, axis=time) instead of clipwise_output. This strategy may improve a gap CV and LB.
Finally, I don't have any above countermeasures. Because I cannot fix 2. Missing labels.
\nMy strategy is \"trust LB\" not \"trust CV\". Fortunately, public LB is almost the same as private LB. I believe shaking almost never happens.
", "rawMarkdown": "Why are CV and LB different?\nI've come up with three reasons I think. But I may be wrong.\n\n# 1. Domain shift\nIn the [last competition](https://www.kaggle.com/c/birdsong-recognition), main theme is [domain shift and missing labels](https://www.kaggle.com/c/birdsong-recognition/discussion/183204).\nIn this point, this competition is similar.\n\nIn this competition, I think domain shift exist. But this is not a big problem.\nIn training sound, it contains noisy sound (white noise and pink noise).\nBut test sound is clean(or a little noisy sound). \n\nBut I think this domain shift has **no negative impact.**\nBecause it's a domain shift from noisy to clean.\n\n# 2. Missing labels\nI [reported](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/209040) that missing labels exist in tp sound. \nIf validation data contains missing labels, your CV is different from LB.\n\nFor example, y_label is not correct(contains missing label).\n+ y_label = [1,0,0]\n+ y_true = [1,1,0]\n\n2nd annotation is missed in y_label. Then y_predict = [1,0,0] is best CV in your local. But y_predict=[1,1,0] is better in the test. In other words, no matter how much you improve your CV, it will not always work in a testing environment. **The validation data is not reliable.**\n\n# 3. A gap training and prediction\nMany participants seem to use [SED](https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection?scriptVersionId=40731755). And I also use SED.\nIf you use SED(PANNs architecture), you should be careful.\n\nIf you use SED like below, there is a gap training and prediction.\n+ training with weak label\n+ prediction with ```framewise_output```\n(What is framewise_output?(Now I'm writing \"How to use SED\". Coming soon.))\n\nWeak label training is fitted ```clipwise_output``` **not ```framewise_output```.** ```clipwise_output``` is a time-compressed version of ```framewise_output```. ```clipwise_output``` is correlated with ```framewise_output```, but not equal. \n\nComparatively, ```framewise_output``` prediction is good at short sound event. In this competition, there is many short sound event. Therefore I use ```framewise_output``` for prediction. But there is a gap training and prediction. Then CV(```clipwise_output``` training) is different from LB(```framewise_output``` prediction).\n\nIf you change validation strategy, you may avoid this problem. For example, in validation phase you use np.max(```framewise_output```, axis=time) instead of ```clipwise_output```. This strategy may improve a gap CV and LB.\n\n# 4. Conclusion\nFinally, I don't have any above countermeasures. Because I cannot fix **2. Missing labels**.\n\nMy strategy is \"trust LB\" not \"trust CV\". Fortunately, [public LB is almost the same as private LB](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/207901#1134198). I believe shaking almost never happens.", "votes": null }, { "id": "1144216", "postDate": "01/08/2021 10:08:07", "content": "@shinmurashinmura Thanks.
\nhow do you predict at inference time
", "rawMarkdown": "shinmurashinmura Thanks.\n\nhow do you predict at inference time", "votes": null }, { "id": "1144242", "postDate": "01/08/2021 10:44:15", "content": "Thank you. Are you train on clipwise_output?
", "rawMarkdown": "Thank you. Are you train on clipwise_output?", "votes": null }, { "id": "1144429", "postDate": "01/08/2021 13:09:38", "content": "Yes, I trained with clipwise_output and prediction is \"framewise_outuput.
", "rawMarkdown": "Yes, I trained with clipwise_output and prediction is \"framewise_outuput.", "votes": null }, { "id": "1144435", "postDate": "01/08/2021 13:11:53", "content": "https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/208830#1139552
", "rawMarkdown": "https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/208830#1139552", "votes": null }, { "id": "1144445", "postDate": "01/08/2021 13:18:39", "content": "@shinmura0 how is your individual fold CV?
", "rawMarkdown": "shinmura0 how is your individual fold CV?", "votes": null }, { "id": "1149876", "postDate": "01/12/2021 08:16:19", "content": "I used StratifiedKFold(sklearn).
\nAnd n_splits=5.
Makes sense @shinmurashinmura
", "rawMarkdown": "Makes sense @shinmurashinmura", "votes": null } ], "comments": [ { "id": 1144216, "author_name": "gopidurgaprasad", "author_url": "", "post_date": "01/08/2021 10:08:07", "content": "@shinmurashinmura Thanks.
\nhow do you predict at inference time
", "votes": null, "replies": [ { "id": 1144435, "author_name": "shinmurashinmura", "author_url": "", "post_date": "01/08/2021 13:11:53", "content": "https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/208830#1139552
", "votes": null, "replies": [] } ] }, { "id": 1144242, "author_name": "vlomme", "author_url": "", "post_date": "01/08/2021 10:44:15", "content": "Thank you. Are you train on clipwise_output?
", "votes": null, "replies": [ { "id": 1144429, "author_name": "shinmurashinmura", "author_url": "", "post_date": "01/08/2021 13:09:38", "content": "Yes, I trained with clipwise_output and prediction is \"framewise_outuput.
", "votes": null, "replies": [] } ] }, { "id": 1144445, "author_name": "tugstugi", "author_url": "", "post_date": "01/08/2021 13:18:39", "content": "@shinmura0 how is your individual fold CV?
", "votes": null, "replies": [ { "id": 1149876, "author_name": "shinmurashinmura", "author_url": "", "post_date": "01/12/2021 08:16:19", "content": "I used StratifiedKFold(sklearn).
\nAnd n_splits=5.
Makes sense @shinmurashinmura
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1144085": "Why are CV and LB different?\nI've come up with three reasons I think. But I may be wrong.\n\n# 1. Domain shift\nIn the [last competition](https://www.kaggle.com/c/birdsong-recognition), main theme is [domain shift and missing labels](https://www.kaggle.com/c/birdsong-recognition/discussion/183204).\nIn this point, this competition is similar.\n\nIn this competition, I think domain shift exist. But this is not a big problem.\nIn training sound, it contains noisy sound (white noise and pink noise).\nBut test sound is clean(or a little noisy sound). \n\nBut I think this domain shift has **no negative impact.**\nBecause it's a domain shift from noisy to clean.\n\n# 2. Missing labels\nI [reported](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/209040) that missing labels exist in tp sound. \nIf validation data contains missing labels, your CV is different from LB.\n\nFor example, y_label is not correct(contains missing label).\n+ y_label = [1,0,0]\n+ y_true = [1,1,0]\n\n2nd annotation is missed in y_label. Then y_predict = [1,0,0] is best CV in your local. But y_predict=[1,1,0] is better in the test. In other words, no matter how much you improve your CV, it will not always work in a testing environment. **The validation data is not reliable.**\n\n# 3. A gap training and prediction\nMany participants seem to use [SED](https://www.kaggle.com/hidehisaarai1213/introduction-to-sound-event-detection?scriptVersionId=40731755). And I also use SED.\nIf you use SED(PANNs architecture), you should be careful.\n\nIf you use SED like below, there is a gap training and prediction.\n+ training with weak label\n+ prediction with ```framewise_output```\n(What is framewise_output?(Now I'm writing \"How to use SED\". Coming soon.))\n\nWeak label training is fitted ```clipwise_output``` **not ```framewise_output```.** ```clipwise_output``` is a time-compressed version of ```framewise_output```. ```clipwise_output``` is correlated with ```framewise_output```, but not equal. \n\nComparatively, ```framewise_output``` prediction is good at short sound event. In this competition, there is many short sound event. Therefore I use ```framewise_output``` for prediction. But there is a gap training and prediction. Then CV(```clipwise_output``` training) is different from LB(```framewise_output``` prediction).\n\nIf you change validation strategy, you may avoid this problem. For example, in validation phase you use np.max(```framewise_output```, axis=time) instead of ```clipwise_output```. This strategy may improve a gap CV and LB.\n\n# 4. Conclusion\nFinally, I don't have any above countermeasures. Because I cannot fix **2. Missing labels**.\n\nMy strategy is \"trust LB\" not \"trust CV\". Fortunately, [public LB is almost the same as private LB](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/207901#1134198). I believe shaking almost never happens.", "1144216": "shinmurashinmura Thanks.\n\nhow do you predict at inference time", "1144242": "Thank you. Are you train on clipwise_output?", "1144429": "Yes, I trained with clipwise_output and prediction is \"framewise_outuput.", "1144435": "https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/208830#1139552", "1144445": "shinmura0 how is your individual fold CV?", "1149876": "I used StratifiedKFold(sklearn).\nAnd n_splits=5.", "1155222": "Makes sense @shinmurashinmura" }, "source": "meta" }