![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2F0607d9fe7840d738bf53c2b2c070419f%2FSW.png?generation=1596543515489705&alt=media\")
In this notebook, I tried to validate with the official 'example_test_audio' data, the most popular inference notebook, shared by @ttahara.
\n\nHowever, the obtained micro F1-Score is too low: F1-Score=0.0180
\n\nWhat is wrong?
\n\nThanks in advance
", "messages": [ { "id": "946221", "postDate": "07/26/2020 12:50:39", "content": "In this notebook, I tried to validate with the official 'example_test_audio' data, the most popular inference notebook, shared by @ttahara.
\n\nHowever, the obtained micro F1-Score is too low: F1-Score=0.0180
\n\nWhat is wrong?
\n\nThanks in advance
", "rawMarkdown": "In this [notebook](https://www.kaggle.com/jpison/inference-resnest50-fast-with-example-test), I tried to validate with the official 'example\\_test\\_audio' data, the most popular [inference notebook](https://www.kaggle.com/ttahara/inference-birdsong-baseline-resnest50-fast), shared by @ttahara.\n\nHowever, the obtained micro F1-Score is too low: F1-Score=0.0180\n\nWhat is wrong?\n\nThanks in advance", "votes": null }, { "id": "946346", "postDate": "07/26/2020 14:18:48", "content": "Hi,
\n\nYou can click on the right \"Version 2 of 2\" then press \"Compare Diff\" button. You can view the difference between your code and Tawara's code. For example, you just have if site==\"site_3\" but Tawara has if site == \"site_3\" or site == \"example\":
There are a few other differences. Maybe you can find it better than me, since it is your code.
", "rawMarkdown": "Hi,\n\nYou can click on the right \"Version 2 of 2\" then press \"Compare Diff\" button. You can view the difference between your code and Tawara's code. For example, you just have `if site==\"site_3\"` but Tawara has `if site == \"site_3\" or site == \"example\":`\n\nThere are a few other differences. Maybe you can find it better than me, since it is your code.", "votes": null }, { "id": "946563", "postDate": "07/26/2020 16:42:27", "content": "Hi,
\n\nIt is the opposite.
\n\nI included if site == \"site_3\" or site == \"example\": to preprocess the two .mp3 files in the 'example_test_audio' folder.
Thanks for sharing. I compared it with @hidehisaarai1213 's baseline resnet50 model and got a micro f1-score of 0.0166. Am really curious on why that's the case. From inspection, it seems that the model is overpredicting a lot of nocalls. This can really cause a massive shakeup, if the rest of the test data set has a lot of labelled birdcalls
", "rawMarkdown": "Thanks for sharing. I compared it with @hidehisaarai1213 's baseline resnet50 model and got a micro f1-score of 0.0166. Am really curious on why that's the case. From inspection, it seems that the model is overpredicting a lot of nocalls. This can really cause a massive shakeup, if the rest of the test data set has a lot of labelled birdcalls", "votes": null }, { "id": "947471", "postDate": "07/27/2020 09:43:03", "content": "Here is the notebook I forked from yours, then applied to @hidehisaarai1213 's baseline model if anyone is interested:\nhttps://www.kaggle.com/alanchn31/forked-resnet50-with-example-test
", "rawMarkdown": "Here is the notebook I forked from yours, then applied to @hidehisaarai1213 's baseline model if anyone is interested:\nhttps://www.kaggle.com/alanchn31/forked-resnet50-with-example-test", "votes": null }, { "id": "947573", "postDate": "07/27/2020 10:58:33", "content": "Yes, but the model obtain a 0.568 in LB.\nReducing the threshold I got a local micro f1-score of 0.058, but it is too far from LB! I have been checking the code and I cannot find errors in it. If the code is fine, then I think the files in the directory 'example_test_audio' may not be representative of the testing database.
", "rawMarkdown": "Yes, but the model obtain a 0.568 in LB.\nReducing the threshold I got a local micro f1-score of 0.058, but it is too far from LB! I have been checking the code and I cannot find errors in it. If the code is fine, then I think the files in the directory 'example_test_audio' may not be representative of the testing database.", "votes": null }, { "id": "947574", "postDate": "07/27/2020 10:58:47", "content": "Thanks!
", "rawMarkdown": "Thanks!", "votes": null }, { "id": "947588", "postDate": "07/27/2020 11:11:02", "content": "Yes, agree. We need to see if exampletestaudio is representative. Another point to consider is that, the current LB is only calculated on 27% of the test set. Furthermore, in the test set used to evaluate public LB now, a lot of recordings are nocall. I wonder how different the 73% will be from this 27% used to evaluate our standings
", "rawMarkdown": "Yes, agree. We need to see if exampletestaudio is representative. Another point to consider is that, the current LB is only calculated on 27% of the test set. Furthermore, in the test set used to evaluate public LB now, a lot of recordings are nocall. I wonder how different the 73% will be from this 27% used to evaluate our standings", "votes": null }, { "id": "947664", "postDate": "07/27/2020 12:16:03", "content": "1) the records(BLKFR-10 and ORANGE-7) contain labels not from the training sample, such as mouqua and squirrel \n2) the Public test contains a lot of nocall(54%), so the current best solutions are not very good\n3) nocall is also calculated in f1
", "rawMarkdown": "1) the records(BLKFR-10 and ORANGE-7) contain labels not from the training sample, such as mouqua and squirrel \n2) the Public test contains a lot of nocall(54%), so the current best solutions are not very good\n3) nocall is also calculated in f1", "votes": null }, { "id": "947709", "postDate": "07/27/2020 12:44:06", "content": "Can you explain how do you know test set contains 54% nocall?
", "rawMarkdown": "Can you explain how do you know test set contains 54% nocall?", "votes": null }, { "id": "947719", "postDate": "07/27/2020 12:52:32", "content": "If you send only nocall everywhere, the rating is 0.544
", "rawMarkdown": "If you send only nocall everywhere, the rating is 0.544", "votes": null }, { "id": "947725", "postDate": "07/27/2020 12:56:43", "content": "ruh roh
", "rawMarkdown": "ruh roh", "votes": null }, { "id": "948131", "postDate": "07/27/2020 17:25:04", "content": "Thank you @vlomme for your response! I didn't realise that.\nNew version 4, includes 'nocall' label into the f1-score calculation and considers only the labels of the competition.\nNow, the search of the best local micro f1-score=0.455090 and LB=0.566 is obtained with a threshold=0.70. Close to the @ttahara's solution (threshold=0.60, LB=0.568).
", "rawMarkdown": "Thank you @vlomme for your response! I didn't realise that.\nNew [version 4](https://www.kaggle.com/jpison/inference-resnest50-fast-with-example-test-audio), includes 'nocall' label into the f1-score calculation and considers only the labels of the competition.\nNow, the search of the best local micro f1-score=0.455090 and LB=0.566 is obtained with a threshold=0.70. Close to the @ttahara's solution (threshold=0.60, LB=0.568).", "votes": null }, { "id": "948151", "postDate": "07/27/2020 17:35:35", "content": "I think they should have similar distribution.
", "rawMarkdown": "I think they should have similar distribution.", "votes": null }, { "id": "957581", "postDate": "08/04/2020 12:23:54", "content": "If you run the same with BirdCLEF 2020 SSW*.wav audio provided at https://www.kaggle.com/c/birdsong-recognition/discussion/158877#911336 (17 species) then you will notice very low F1 score due to non-nocall in test set. It's the opposite for best threshold (more random for low values).
\n\n
I'm wondering if nocall distribution from public set will be similar to private set ...
", "rawMarkdown": "If you run the same with BirdCLEF 2020 SSW*.wav audio provided at https://www.kaggle.com/c/birdsong-recognition/discussion/158877#911336 (17 species) then you will notice very low F1 score due to non-nocall in test set. It's the opposite for best threshold (more random for low values).\n\n\n\nI'm wondering if nocall distribution from public set will be similar to private set ...", "votes": null }, { "id": "959300", "postDate": "08/05/2020 13:33:59", "content": "But, to calculate f1-score, did you remove the samples with birds out of the competition?
", "rawMarkdown": "But, to calculate f1-score, did you remove the samples with birds out of the competition?", "votes": null }, { "id": "959540", "postDate": "08/05/2020 17:16:33", "content": "@mpware Hi, I have a little question:\nWhy did you say there are only 17 species match this competition?\nI have checked the gt_birdclef2020_validation_data.csv file and found that all 76 species belong to the SSW audios are included in the training classes in this comp.\nLike the first row in the form in your picture, why cangoo is nocall? It is a species in training data.
Yes removed. \nBTW: For BLKFR-10 + ORANGE-7, predicting nocall for all gives a F1 score around 0.7.
", "rawMarkdown": "Yes removed. \nBTW: For BLKFR-10 + ORANGE-7, predicting nocall for all gives a F1 score around 0.7.", "votes": null }, { "id": "960194", "postDate": "08/06/2020 07:57:43", "content": "The ground truth matching to SSW* files contains only 17 species in this competition. The screenshot I provided is a ground truth + inference (cangoo not detected so it becomes nocall)
", "rawMarkdown": "The ground truth matching to SSW* files contains only 17 species in this competition. The screenshot I provided is a ground truth + inference (cangoo not detected so it becomes nocall)", "votes": null }, { "id": "960310", "postDate": "08/06/2020 09:54:39", "content": "I will check again. Is there any other file need to be used except gt_birdclef2020_validation_data.csv? I may miss some information.
The files you need are under BirdCLEF2020__Validation_Audio_and_Ground_Truth/gt folder.\nHere is an example:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F698363%2Fc35c92f736d0c7ec7975794e756870fd%2FSSWexample.png?generation=1596709110319567&alt=media\")
Oh! I totally use the wrong file. Thanks for helping.\nBut what is gt_birdclef2020_validation_data.csv under BirdCLEF2020__Validation_Audio_and_Ground_Truth/scripts?
I think the file gt_birdclef2020_validation_data.csv and the files under BirdCLEF2020__Validation_Audio_and_Ground_Truth/gt folder contain the same information.
", "rawMarkdown": "I think the file *gt_birdclef2020_validation_data.csv* and the files under *BirdCLEF2020__Validation_Audio_and_Ground_Truth/gt* folder contain the same information.", "votes": null } ], "comments": [ { "id": 946346, "author_name": "returnofsputnik", "author_url": "", "post_date": "07/26/2020 14:18:48", "content": "Hi,
\n\nYou can click on the right \"Version 2 of 2\" then press \"Compare Diff\" button. You can view the difference between your code and Tawara's code. For example, you just have if site==\"site_3\" but Tawara has if site == \"site_3\" or site == \"example\":
There are a few other differences. Maybe you can find it better than me, since it is your code.
", "votes": null, "replies": [ { "id": 946563, "author_name": "jpison", "author_url": "", "post_date": "07/26/2020 16:42:27", "content": "Hi,
\n\nIt is the opposite.
\n\nI included if site == \"site_3\" or site == \"example\": to preprocess the two .mp3 files in the 'example_test_audio' folder.
Thanks for sharing. I compared it with @hidehisaarai1213 's baseline resnet50 model and got a micro f1-score of 0.0166. Am really curious on why that's the case. From inspection, it seems that the model is overpredicting a lot of nocalls. This can really cause a massive shakeup, if the rest of the test data set has a lot of labelled birdcalls
", "votes": null, "replies": [ { "id": 947573, "author_name": "jpison", "author_url": "", "post_date": "07/27/2020 10:58:33", "content": "Yes, but the model obtain a 0.568 in LB.\nReducing the threshold I got a local micro f1-score of 0.058, but it is too far from LB! I have been checking the code and I cannot find errors in it. If the code is fine, then I think the files in the directory 'example_test_audio' may not be representative of the testing database.
", "votes": null, "replies": [] }, { "id": 947588, "author_name": "alanchn31", "author_url": "", "post_date": "07/27/2020 11:11:02", "content": "Yes, agree. We need to see if exampletestaudio is representative. Another point to consider is that, the current LB is only calculated on 27% of the test set. Furthermore, in the test set used to evaluate public LB now, a lot of recordings are nocall. I wonder how different the 73% will be from this 27% used to evaluate our standings
", "votes": null, "replies": [] }, { "id": 948151, "author_name": "jpison", "author_url": "", "post_date": "07/27/2020 17:35:35", "content": "I think they should have similar distribution.
", "votes": null, "replies": [] } ] }, { "id": 947471, "author_name": "alanchn31", "author_url": "", "post_date": "07/27/2020 09:43:03", "content": "Here is the notebook I forked from yours, then applied to @hidehisaarai1213 's baseline model if anyone is interested:\nhttps://www.kaggle.com/alanchn31/forked-resnet50-with-example-test
", "votes": null, "replies": [ { "id": 947574, "author_name": "jpison", "author_url": "", "post_date": "07/27/2020 10:58:47", "content": "Thanks!
", "votes": null, "replies": [] } ] }, { "id": 947664, "author_name": "vlomme", "author_url": "", "post_date": "07/27/2020 12:16:03", "content": "1) the records(BLKFR-10 and ORANGE-7) contain labels not from the training sample, such as mouqua and squirrel \n2) the Public test contains a lot of nocall(54%), so the current best solutions are not very good\n3) nocall is also calculated in f1
", "votes": null, "replies": [ { "id": 947709, "author_name": "returnofsputnik", "author_url": "", "post_date": "07/27/2020 12:44:06", "content": "Can you explain how do you know test set contains 54% nocall?
", "votes": null, "replies": [] }, { "id": 947719, "author_name": "vlomme", "author_url": "", "post_date": "07/27/2020 12:52:32", "content": "If you send only nocall everywhere, the rating is 0.544
", "votes": null, "replies": [] }, { "id": 947725, "author_name": "returnofsputnik", "author_url": "", "post_date": "07/27/2020 12:56:43", "content": "ruh roh
", "votes": null, "replies": [] }, { "id": 948131, "author_name": "jpison", "author_url": "", "post_date": "07/27/2020 17:25:04", "content": "Thank you @vlomme for your response! I didn't realise that.\nNew version 4, includes 'nocall' label into the f1-score calculation and considers only the labels of the competition.\nNow, the search of the best local micro f1-score=0.455090 and LB=0.566 is obtained with a threshold=0.70. Close to the @ttahara's solution (threshold=0.60, LB=0.568).
", "votes": null, "replies": [] }, { "id": 957581, "author_name": "mpware", "author_url": "", "post_date": "08/04/2020 12:23:54", "content": "If you run the same with BirdCLEF 2020 SSW*.wav audio provided at https://www.kaggle.com/c/birdsong-recognition/discussion/158877#911336 (17 species) then you will notice very low F1 score due to non-nocall in test set. It's the opposite for best threshold (more random for low values).
\n\n
I'm wondering if nocall distribution from public set will be similar to private set ...
", "votes": null, "replies": [] }, { "id": 959300, "author_name": "jpison", "author_url": "", "post_date": "08/05/2020 13:33:59", "content": "But, to calculate f1-score, did you remove the samples with birds out of the competition?
", "votes": null, "replies": [] }, { "id": 959540, "author_name": "karlyukang", "author_url": "", "post_date": "08/05/2020 17:16:33", "content": "@mpware Hi, I have a little question:\nWhy did you say there are only 17 species match this competition?\nI have checked the gt_birdclef2020_validation_data.csv file and found that all 76 species belong to the SSW audios are included in the training classes in this comp.\nLike the first row in the form in your picture, why cangoo is nocall? It is a species in training data.
Yes removed. \nBTW: For BLKFR-10 + ORANGE-7, predicting nocall for all gives a F1 score around 0.7.
", "votes": null, "replies": [] }, { "id": 960194, "author_name": "mpware", "author_url": "", "post_date": "08/06/2020 07:57:43", "content": "The ground truth matching to SSW* files contains only 17 species in this competition. The screenshot I provided is a ground truth + inference (cangoo not detected so it becomes nocall)
", "votes": null, "replies": [] }, { "id": 960310, "author_name": "karlyukang", "author_url": "", "post_date": "08/06/2020 09:54:39", "content": "I will check again. Is there any other file need to be used except gt_birdclef2020_validation_data.csv? I may miss some information.
The files you need are under BirdCLEF2020__Validation_Audio_and_Ground_Truth/gt folder.\nHere is an example:\n
Oh! I totally use the wrong file. Thanks for helping.\nBut what is gt_birdclef2020_validation_data.csv under BirdCLEF2020__Validation_Audio_and_Ground_Truth/scripts?
I think the file gt_birdclef2020_validation_data.csv and the files under BirdCLEF2020__Validation_Audio_and_Ground_Truth/gt folder contain the same information.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "946221": "In this [notebook](https://www.kaggle.com/jpison/inference-resnest50-fast-with-example-test), I tried to validate with the official 'example\\_test\\_audio' data, the most popular [inference notebook](https://www.kaggle.com/ttahara/inference-birdsong-baseline-resnest50-fast), shared by @ttahara.\n\nHowever, the obtained micro F1-Score is too low: F1-Score=0.0180\n\nWhat is wrong?\n\nThanks in advance", "946346": "Hi,\n\nYou can click on the right \"Version 2 of 2\" then press \"Compare Diff\" button. You can view the difference between your code and Tawara's code. For example, you just have `if site==\"site_3\"` but Tawara has `if site == \"site_3\" or site == \"example\":`\n\nThere are a few other differences. Maybe you can find it better than me, since it is your code.", "946563": "Hi,\n\nIt is the opposite. \n\nI included `if site == \"site_3\" or site == \"example\":` to preprocess the two .mp3 files in the 'example\\_test\\_audio' folder.", "947455": "Thanks for sharing. I compared it with @hidehisaarai1213 's baseline resnet50 model and got a micro f1-score of 0.0166. Am really curious on why that's the case. From inspection, it seems that the model is overpredicting a lot of nocalls. This can really cause a massive shakeup, if the rest of the test data set has a lot of labelled birdcalls", "947471": "Here is the notebook I forked from yours, then applied to @hidehisaarai1213 's baseline model if anyone is interested:\nhttps://www.kaggle.com/alanchn31/forked-resnet50-with-example-test", "947573": "Yes, but the model obtain a 0.568 in LB.\nReducing the threshold I got a local micro f1-score of 0.058, but it is too far from LB! I have been checking the code and I cannot find errors in it. If the code is fine, then I think the files in the directory 'example_test_audio' may not be representative of the testing database.", "947574": "Thanks!", "947588": "Yes, agree. We need to see if exampletestaudio is representative. Another point to consider is that, the current LB is only calculated on 27% of the test set. Furthermore, in the test set used to evaluate public LB now, a lot of recordings are nocall. I wonder how different the 73% will be from this 27% used to evaluate our standings", "947664": "1) the records(BLKFR-10 and ORANGE-7) contain labels not from the training sample, such as mouqua and squirrel \n2) the Public test contains a lot of nocall(54%), so the current best solutions are not very good\n3) nocall is also calculated in f1", "947709": "Can you explain how do you know test set contains 54% nocall?", "947719": "If you send only nocall everywhere, the rating is 0.544", "947725": "ruh roh", "948131": "Thank you @vlomme for your response! I didn't realise that.\nNew [version 4](https://www.kaggle.com/jpison/inference-resnest50-fast-with-example-test-audio), includes 'nocall' label into the f1-score calculation and considers only the labels of the competition.\nNow, the search of the best local micro f1-score=0.455090 and LB=0.566 is obtained with a threshold=0.70. Close to the @ttahara's solution (threshold=0.60, LB=0.568).", "948151": "I think they should have similar distribution.", "957581": "If you run the same with BirdCLEF 2020 SSW*.wav audio provided at https://www.kaggle.com/c/birdsong-recognition/discussion/158877#911336 (17 species) then you will notice very low F1 score due to non-nocall in test set. It's the opposite for best threshold (more random for low values).\n\n\n\nI'm wondering if nocall distribution from public set will be similar to private set ...", "959300": "But, to calculate f1-score, did you remove the samples with birds out of the competition?", "959540": "mpware Hi, I have a little question:\nWhy did you say there are only 17 species match this competition?\nI have checked the `gt_birdclef2020_validation_data.csv` file and found that all 76 species belong to the SSW audios are included in the training classes in this comp.\nLike the first row in the form in your picture, why `cangoo` is `nocall`? It is a species in training data.", "960191": "Yes removed. \nBTW: For BLKFR-10 + ORANGE-7, predicting nocall for all gives a F1 score around 0.7.", "960194": "The ground truth matching to SSW* files contains only 17 species in this competition. The screenshot I provided is a ground truth + inference (cangoo not detected so it becomes nocall)", "960310": "I will check again. Is there any other file need to be used except `gt_birdclef2020_validation_data.csv`? I may miss some information.", "960346": "The files you need are under `BirdCLEF2020__Validation_Audio_and_Ground_Truth/gt` folder.\nHere is an example:\n", "960360": "Oh! I totally use the wrong file. Thanks for helping.\nBut what is `gt_birdclef2020_validation_data.csv` under `BirdCLEF2020__Validation_Audio_and_Ground_Truth/scripts`?", "1003851": "I think the file *gt_birdclef2020_validation_data.csv* and the files under *BirdCLEF2020__Validation_Audio_and_Ground_Truth/gt* folder contain the same information." }, "source": "meta" }