Hai…,
\nMy CV crossing 0.90+ but no luck in LB it's always < 0.83
\nPlease give some suggestions
Thank you 🙏
", "messages": [ { "id": "1184047", "postDate": "02/03/2021 11:16:15", "content": "Hai…,
\nMy CV crossing 0.90+ but no luck in LB it's always < 0.83
\nPlease give some suggestions
Thank you 🙏
", "rawMarkdown": "Hai...,\n\nMy CV crossing 0.90+ but no luck in LB it's always < 0.83 \nPlease give some suggestions \n\nThank you 🙏", "votes": null }, { "id": "1184373", "postDate": "02/03/2021 14:04:37", "content": "Hi, @gopidurgaprasad , for me, it's hard to make a local CV, since we don't have sufficient information from the train_tp and 'train_fp' to construct proper label for the 60s audios. The train_tp and 'train_fp' only tells us about what happens in the specified time-frequency box, but does not tell us about other region in the 60s audio, what bird call it contains. I don't know if it's the same problem for you.
I have the same problem for sure.
", "rawMarkdown": "I have the same problem for sure.", "votes": null }, { "id": "1184469", "postDate": "02/03/2021 14:39:35", "content": "everyone does I bet)
", "rawMarkdown": "everyone does I bet)", "votes": null }, { "id": "1184476", "postDate": "02/03/2021 14:45:47", "content": "@barnwellguy thank you
\ndid you find anything interesting
In terms of CV, I personally did not compute LRAP (the competition metric) locally for the above reason. Of course, as everybody knows, if we can classify each species in every small clip in the 60s audio correctly, then, the LRAP must be good, in some sense. So, the validation I have to tune parameters are just clip wise (not 60s audio wise) cross-entropy, AUC, precision/recall. etc.
\nI would say, I didn't look at local LRAP, so I didn't have the trouble of a different local LRAP and LB LRAP - an ostrich policy.
", "rawMarkdown": "In terms of CV, I personally did not compute LRAP (the competition metric) locally for the above reason. Of course, as everybody knows, if we can classify each species in every small clip in the 60s audio correctly, then, the LRAP must be good, in some sense. So, the validation I have to tune parameters are just clip wise (not 60s audio wise) cross-entropy, AUC, precision/recall. etc. \n\nI would say, I didn't look at local LRAP, so I didn't have the trouble of a different local LRAP and LB LRAP - an ostrich policy.", "votes": null }, { "id": "1184658", "postDate": "02/03/2021 16:53:49", "content": "@barnwellguy are you using 6s or 10s audios?
", "rawMarkdown": "barnwellguy are you using 6s or 10s audios?", "votes": null }, { "id": "1184742", "postDate": "02/03/2021 18:07:07", "content": "@tugstugi haha, I feel, that is one of my magic sauce. But I can say, it's neither 6s nor 10s :)
", "rawMarkdown": "tugstugi haha, I feel, that is one of my magic sauce. But I can say, it's neither 6s nor 10s :)", "votes": null }, { "id": "1184765", "postDate": "02/03/2021 18:26:19", "content": "\n\n\nhe validation I have to tune parameters are just clip wise (not 60s audio wise) cross-entropy, AUC, precision/recall. etc.>
\n
do you mean your validation is framewise i.e for a small time frame (6s nor 10s)
", "rawMarkdown": "barnwellguy \n> he validation I have to tune parameters are just clip wise (not 60s audio wise) cross-entropy, AUC, precision/recall. etc.> \n\ndo you mean your validation is framewise i.e for a small time frame (6s nor 10s)", "votes": null }, { "id": "1184877", "postDate": "02/03/2021 20:29:39", "content": "Yes, frame wise. Not 60s audio wise.
\nI didn't quite rely on the validation, apart from early stopping the neural network training.
My best score for the SED model (from your notebook) was 0.835. Thanks to Buffalo Spdwy's post, it finally dawned on me that our problem is that we are looking at VAL_LWLRAP. In the last example, I prepared two almost identical models of difference: the first LWLRAP averaged 0.95, VAL_LWLRAP - 0.89, the second LWLRAP - 0.89, VAL_LWLRAP - 0.89. The first model showed a much better result on a public dataset.
", "rawMarkdown": "My best score for the SED model (from your notebook) was 0.835. Thanks to Buffalo Spdwy's post, it finally dawned on me that our problem is that we are looking at VAL_LWLRAP. In the last example, I prepared two almost identical models of difference: the first LWLRAP averaged 0.95, VAL_LWLRAP - 0.89, the second LWLRAP - 0.89, VAL_LWLRAP - 0.89. The first model showed a much better result on a public dataset.", "votes": null }, { "id": "1185871", "postDate": "02/04/2021 12:47:33", "content": "Do you mean with VAL_LWLRAP whole 60s or cutted segments? For me, the whole clip CV doesn't have any correlation to LB, i.e. CV 0.86 is sometimes LB0.85 or sometimes only LB 0.80.
", "rawMarkdown": "Do you mean with VAL_LWLRAP whole 60s or cutted segments? For me, the whole clip CV doesn't have any correlation to LB, i.e. CV 0.86 is sometimes LB0.85 or sometimes only LB 0.80.", "votes": null }, { "id": "1186072", "postDate": "02/04/2021 15:17:16", "content": "I wrote about VAL_LWLRAP from this model: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle
", "rawMarkdown": "I wrote about VAL_LWLRAP from this model: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle", "votes": null } ], "comments": [ { "id": 1184373, "author_name": "barnwellguy", "author_url": "", "post_date": "02/03/2021 14:04:37", "content": "Hi, @gopidurgaprasad , for me, it's hard to make a local CV, since we don't have sufficient information from the train_tp and 'train_fp' to construct proper label for the 60s audios. The train_tp and 'train_fp' only tells us about what happens in the specified time-frequency box, but does not tell us about other region in the 60s audio, what bird call it contains. I don't know if it's the same problem for you.
I have the same problem for sure.
", "votes": null, "replies": [] }, { "id": 1184469, "author_name": "kupchanski", "author_url": "", "post_date": "02/03/2021 14:39:35", "content": "everyone does I bet)
", "votes": null, "replies": [] }, { "id": 1184476, "author_name": "gopidurgaprasad", "author_url": "", "post_date": "02/03/2021 14:45:47", "content": "@barnwellguy thank you
\ndid you find anything interesting
In terms of CV, I personally did not compute LRAP (the competition metric) locally for the above reason. Of course, as everybody knows, if we can classify each species in every small clip in the 60s audio correctly, then, the LRAP must be good, in some sense. So, the validation I have to tune parameters are just clip wise (not 60s audio wise) cross-entropy, AUC, precision/recall. etc.
\nI would say, I didn't look at local LRAP, so I didn't have the trouble of a different local LRAP and LB LRAP - an ostrich policy.
", "votes": null, "replies": [] }, { "id": 1184658, "author_name": "tugstugi", "author_url": "", "post_date": "02/03/2021 16:53:49", "content": "@barnwellguy are you using 6s or 10s audios?
", "votes": null, "replies": [] }, { "id": 1184742, "author_name": "barnwellguy", "author_url": "", "post_date": "02/03/2021 18:07:07", "content": "@tugstugi haha, I feel, that is one of my magic sauce. But I can say, it's neither 6s nor 10s :)
", "votes": null, "replies": [] }, { "id": 1184765, "author_name": "yuvaramsingh", "author_url": "", "post_date": "02/03/2021 18:26:19", "content": "\n\n\nhe validation I have to tune parameters are just clip wise (not 60s audio wise) cross-entropy, AUC, precision/recall. etc.>
\n
do you mean your validation is framewise i.e for a small time frame (6s nor 10s)
", "votes": null, "replies": [] }, { "id": 1184877, "author_name": "barnwellguy", "author_url": "", "post_date": "02/03/2021 20:29:39", "content": "Yes, frame wise. Not 60s audio wise.
\nI didn't quite rely on the validation, apart from early stopping the neural network training.
My best score for the SED model (from your notebook) was 0.835. Thanks to Buffalo Spdwy's post, it finally dawned on me that our problem is that we are looking at VAL_LWLRAP. In the last example, I prepared two almost identical models of difference: the first LWLRAP averaged 0.95, VAL_LWLRAP - 0.89, the second LWLRAP - 0.89, VAL_LWLRAP - 0.89. The first model showed a much better result on a public dataset.
", "votes": null, "replies": [ { "id": 1185871, "author_name": "tugstugi", "author_url": "", "post_date": "02/04/2021 12:47:33", "content": "Do you mean with VAL_LWLRAP whole 60s or cutted segments? For me, the whole clip CV doesn't have any correlation to LB, i.e. CV 0.86 is sometimes LB0.85 or sometimes only LB 0.80.
", "votes": null, "replies": [] }, { "id": 1186072, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "02/04/2021 15:17:16", "content": "I wrote about VAL_LWLRAP from this model: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1184047": "Hai...,\n\nMy CV crossing 0.90+ but no luck in LB it's always < 0.83 \nPlease give some suggestions \n\nThank you 🙏", "1184373": "Hi, @gopidurgaprasad , for me, it's hard to make a local CV, since we don't have sufficient information from the `train_tp` and 'train_fp' to construct proper label for the 60s audios. The `train_tp` and 'train_fp' only tells us about what happens in the specified time-frequency box, but does not tell us about other region in the 60s audio, what bird call it contains. I don't know if it's the same problem for you.", "1184388": "I have the same problem for sure.", "1184469": "everyone does I bet)", "1184476": "barnwellguy thank you \ndid you find anything interesting", "1184641": "In terms of CV, I personally did not compute LRAP (the competition metric) locally for the above reason. Of course, as everybody knows, if we can classify each species in every small clip in the 60s audio correctly, then, the LRAP must be good, in some sense. So, the validation I have to tune parameters are just clip wise (not 60s audio wise) cross-entropy, AUC, precision/recall. etc. \n\nI would say, I didn't look at local LRAP, so I didn't have the trouble of a different local LRAP and LB LRAP - an ostrich policy.", "1184658": "barnwellguy are you using 6s or 10s audios?", "1184742": "tugstugi haha, I feel, that is one of my magic sauce. But I can say, it's neither 6s nor 10s :)", "1184765": "barnwellguy \n> he validation I have to tune parameters are just clip wise (not 60s audio wise) cross-entropy, AUC, precision/recall. etc.> \n\ndo you mean your validation is framewise i.e for a small time frame (6s nor 10s)", "1184877": "Yes, frame wise. Not 60s audio wise.\nI didn't quite rely on the validation, apart from early stopping the neural network training.", "1185678": "My best score for the SED model (from your notebook) was 0.835. Thanks to Buffalo Spdwy's post, it finally dawned on me that our problem is that we are looking at VAL_LWLRAP. In the last example, I prepared two almost identical models of difference: the first LWLRAP averaged 0.95, VAL_LWLRAP - 0.89, the second LWLRAP - 0.89, VAL_LWLRAP - 0.89. The first model showed a much better result on a public dataset.", "1185871": "Do you mean with VAL_LWLRAP whole 60s or cutted segments? For me, the whole clip CV doesn't have any correlation to LB, i.e. CV 0.86 is sometimes LB0.85 or sometimes only LB 0.80.", "1186072": "I wrote about VAL_LWLRAP from this model: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle" }, "source": "meta" }