Hello, I have found some discrepancy between Test CSV and provided test soundscape which prevents me to run testing. The test CSV file includes column file_id but the particular name does not exist in test_soundscapes folder. The sample test table is:
row_id,file_id,bird,end_time\nsoundscape_1000170626_akiapo_5,soundscape_1000170626,akiapo,5\nsoundscape_1000170626_akiapo_10,soundscape_1000170626,akiapo,10\nsoundscape_1000170626_akiapo_15,soundscape_1000170626,akiapo,15\nbut the provided data/dataset is:
\nbirdclef-2022\n |- test_soundscapes\n | L soundscape_453028782.ogg\n |- train_audio\n |- eBird_Taxonomy_v2021.csv\n |- sample_submission.csv\n |- scored_birds.json\n |- test.csv\n L train_metadata.csv\nSo I am wondering if it is a bug or I miss some part of the test data mapping… @stefankahl?
", "messages": [ { "id": "1699701", "postDate": "02/21/2022 11:22:24", "content": "Hello, I have found some discrepancy between Test CSV and provided test soundscape which prevents me to run testing. The test CSV file includes column file_id but the particular name does not exist in test_soundscapes folder. The sample test table is:
row_id,file_id,bird,end_time\nsoundscape_1000170626_akiapo_5,soundscape_1000170626,akiapo,5\nsoundscape_1000170626_akiapo_10,soundscape_1000170626,akiapo,10\nsoundscape_1000170626_akiapo_15,soundscape_1000170626,akiapo,15\nbut the provided data/dataset is:
\nbirdclef-2022\n |- test_soundscapes\n | L soundscape_453028782.ogg\n |- train_audio\n |- eBird_Taxonomy_v2021.csv\n |- sample_submission.csv\n |- scored_birds.json\n |- test.csv\n L train_metadata.csv\nSo I am wondering if it is a bug or I miss some part of the test data mapping… @stefankahl?
", "rawMarkdown": "Hello, I have found some discrepancy between Test CSV and provided test soundscape which prevents me to run testing. The test CSV file includes column `file_id` but the particular name does not exist in `test_soundscapes` folder. The sample test table is:\n\n```\nrow_id,file_id,bird,end_time\nsoundscape_1000170626_akiapo_5,soundscape_1000170626,akiapo,5\nsoundscape_1000170626_akiapo_10,soundscape_1000170626,akiapo,10\nsoundscape_1000170626_akiapo_15,soundscape_1000170626,akiapo,15\n```\n\nbut the provided data/dataset is:\n```\nbirdclef-2022\n |- test_soundscapes\n | L soundscape_453028782.ogg\n |- train_audio\n |- eBird_Taxonomy_v2021.csv\n |- sample_submission.csv\n |- scored_birds.json\n |- test.csv\n L train_metadata.csv\n```\n\nSo I am wondering if it is a bug or I miss some part of the test data mapping... @stefankahl?", "votes": null }, { "id": "1700121", "postDate": "02/21/2022 17:08:02", "content": "This is somewhat intentional and no bug, yet not a 100% ideal. For testing, you might need to parse the test_soundscape folder for files and then switch to the test.csv for a submission. Or stick with parsing the soundscape dir. Both should work fine when submitting.
", "rawMarkdown": "This is somewhat intentional and no bug, yet not a 100% ideal. For testing, you might need to parse the test_soundscape folder for files and then switch to the test.csv for a submission. Or stick with parsing the soundscape dir. Both should work fine when submitting.", "votes": null }, { "id": "1700147", "postDate": "02/21/2022 17:38:10", "content": "ok, maybe I miss something… to make a submission I need to have a valid kernel (which runs with the dummy test data). It means the kernel takes the test data and creates a submission file. So when I open the test.csv I have reference to an audio file named soundscape_1000170626 but there is no such file in the dataset (in the test folder is soundscape_453028782) so based on what shall I make the predictions?
My point is that the name in CSV does not match the audio file name…
", "rawMarkdown": "ok, maybe I miss something... to make a submission I need to have a valid kernel (which runs with the dummy test data). It means the kernel takes the test data and creates a submission file. So when I open the test.csv I have reference to an audio file named `soundscape_1000170626` but there is no such file in the dataset (in the test folder is `soundscape_453028782`) so based on what shall I make the predictions?\n\nMy point is that the name in CSV does not match the audio file name...", "votes": null }, { "id": "1700186", "postDate": "02/21/2022 18:18:38", "content": "I'd suggest to have a look on the
\nsubmission nb provided by host - it should make clear how to parse test audio files
I see, that the sample kernel runs prediction for all 5-secund frames for all birds… completely ignoring the test.csv so any idea why we should care about it for submission as descriptions states \"the full test.csv is provided in the hidden test set\"?
\ninterestingly, why do we need the submission.csv as mentioned \"the full submission.csv is provided in the hidden test set.\"
", "rawMarkdown": "I see, that the sample kernel runs prediction for all 5-secund frames for all birds... completely ignoring the test.csv so any idea why we should care about it for submission as descriptions states \"the full test.csv is provided in the hidden test set\"?\n\ninterestingly, why do we need the submission.csv as mentioned \"the full submission.csv is provided in the hidden test set.\"", "votes": null }, { "id": "1700335", "postDate": "02/21/2022 20:57:25", "content": "they mean that when you press the \"Submit\" they replace the sample submission csv with the full one that contain ALL audios (public and private a.k.a hidden test).. as usual in code competition format
", "rawMarkdown": "they mean that when you press the \"Submit\" they replace the sample submission csv with the full one that contain ALL audios (public and private a.k.a hidden test).. as usual in code competition format", "votes": null }, { "id": "1700345", "postDate": "02/21/2022 21:33:45", "content": "Yeah, I see your point, but please see mine too.. with the actual setting you are not able to use their test as a mock to develop your submission… for illustration you have the task of result a+b for inpouts 2 and 4 and result is 42
@jirkaborovec have you made any progress on the matter? I've just joined the competition and I'm kinda puzzled, too.
", "rawMarkdown": "jirkaborovec have you made any progress on the matter? I've just joined the competition and I'm kinda puzzled, too.", "votes": null }, { "id": "1713806", "postDate": "03/06/2022 12:04:20", "content": "seems there is no update on the obvious issue, so just continued with creating predictions for all birds and ignoring the test.csv file, see https://www.kaggle.com/jirkaborovec/birdclef-fake-predictions
", "rawMarkdown": "seems there is no update on the obvious issue, so just continued with creating predictions for all birds and ignoring the test.csv file, see https://www.kaggle.com/jirkaborovec/birdclef-fake-predictions", "votes": null }, { "id": "1714862", "postDate": "03/07/2022 12:27:07", "content": "Hey I did this to have an easy way around the problem :
\ntest = pd.read_csv('../input/birdclef-2022/test.csv')\nif len(test) == 3:\n test['file_id'] = 'soundscape_453028782'\nThen, when I save the notebook I am sure it will run without problem and that it can work for the actual submission.
", "rawMarkdown": "Hey I did this to have an easy way around the problem :\n\n```\ntest = pd.read_csv('../input/birdclef-2022/test.csv')\nif len(test) == 3:\n test['file_id'] = 'soundscape_453028782'\n```\n\n\nThen, when I save the notebook I am sure it will run without problem and that it can work for the actual submission.", "votes": null }, { "id": "1714882", "postDate": "03/07/2022 12:55:10", "content": "cool, but then you will run your predictions on one audio file instead of the expected test dataset…
", "rawMarkdown": "cool, but then you will run your predictions on one audio file instead of the expected test dataset...", "votes": null }, { "id": "1714910", "postDate": "03/07/2022 13:26:01", "content": "Yes, if the length of the dataset (test) is equal to 3.
\nBut when it comes to the actual submission the test dataframe will be longer than 3. Thus, you will use the real files*.
\n*Because the 'len(test) == 3' condition will not be true, so you will not change the file_id names.
", "rawMarkdown": "Yes, if the length of the dataset (test) is equal to 3.\n\nBut when it comes to the actual submission the test dataframe will be longer than 3. Thus, you will use the real files*.\n\n*Because the 'len(test) == 3' condition will not be true, so you will not change the file_id names.", "votes": null }, { "id": "1714946", "postDate": "03/07/2022 14:07:35", "content": "I know already how to generate the submission without the test.csv, I was just asking the hosted could fix it and make the dataset part related to submission rather useful… :)
", "rawMarkdown": "I know already how to generate the submission without the test.csv, I was just asking the hosted could fix it and make the dataset part related to submission rather useful... :)", "votes": null }, { "id": "1714949", "postDate": "03/07/2022 14:09:04", "content": "Ha sorry ! I misunderstood the problem then :) !
", "rawMarkdown": "Ha sorry ! I misunderstood the problem then :) !", "votes": null }, { "id": "1720691", "postDate": "03/13/2022 03:27:53", "content": "\n\nOr stick with parsing the soundscape dir. Both should work fine when submitting.
\n
Is this really correct?
\nI searched for soundscape dir and tried to create a submission file,
\nbut I got an error if I don't set the number of segments to 12 (equivalent to 60 seconds) for all files.
Are files with more than 60 seconds evaluated only for the first 60 seconds?
\nI haven't tried submission based on test.csv because I've exceeded today's submission limit.
\nBut I'm guessing that the maximum end_time of each file in the test.csv is \"_60\" even though there are files longer than 60 seconds..
yes, generating the submission purely on the folder with soundtracks and scored_birds.json seems to be working for me in this dummy submission: https://www.kaggle.com/jirkaborovec/birdclef-fake-predictions
Thank you for your reply!
\nThe only difference between my failed code and your code seems to be 'ceil' (yours) or 'floor' (mine).
\nSo it seems that the cause of my failure is a lack of segments, but..
\nI'm confused by the success of my code with a fixed number of segments of 12.
If all the test data are 60 seconds, there should be no difference between 'ceil' and 'floor'.
\nIf there is a test data that is not 60 seconds, I don't know why the submission is successful with the fixed 12 segments.
I will try it tomorrow with reference to your code!
\nThanks again!!
~~~
\nThis is my failed code. (Of course, sample submission works fine.)
for fpath in test_files:\n sig, rate = librosa.load(fpath, sr=32000, mono=True)\n num_segments = len(sig) // (rate*5) \n for seg_idx in range(num_segments): # <---\n ...\nThe following code was successful.
\nfor fpath in test_files:\n sig, rate = librosa.load(fpath, sr=32000, mono=True)\n for seg_idx in range(12): # <---\n ...\nfrom data tab
\n\n", "rawMarkdown": "from data tab\n\n> the test_soundscapes directory will be populated with approximately 5,500 recordings to be used for scoring. These are each **within a few milliseconds of 1 minute long** and in the ogg audio format.", "votes": null }, { "id": "1721036", "postDate": "03/13/2022 11:06:51", "content": "the test_soundscapes directory will be populated with approximately 5,500 recordings to be used for scoring. These are each within a few milliseconds of 1 minute long and in the ogg audio format.
\n
Thank you!
\nI completely missed it..
\nSince 'ceil' works well, the audio file is probably up to 1 minute long (not over 1 minute).
I tripped over the exact same thing but after some failed attempts it works fine with ceil. But good to know that they all seem to be <= 60 sec - within a few milliseconds of 1 minute long could be 60k ms +- n ms.
This is somewhat intentional and no bug, yet not a 100% ideal. For testing, you might need to parse the test_soundscape folder for files and then switch to the test.csv for a submission. Or stick with parsing the soundscape dir. Both should work fine when submitting.
", "votes": null, "replies": [ { "id": 1700147, "author_name": "jirkaborovec", "author_url": "", "post_date": "02/21/2022 17:38:10", "content": "ok, maybe I miss something… to make a submission I need to have a valid kernel (which runs with the dummy test data). It means the kernel takes the test data and creates a submission file. So when I open the test.csv I have reference to an audio file named soundscape_1000170626 but there is no such file in the dataset (in the test folder is soundscape_453028782) so based on what shall I make the predictions?
My point is that the name in CSV does not match the audio file name…
", "votes": null, "replies": [] }, { "id": 1700186, "author_name": "imeintanis", "author_url": "", "post_date": "02/21/2022 18:18:38", "content": "I'd suggest to have a look on the
\nsubmission nb provided by host - it should make clear how to parse test audio files
I see, that the sample kernel runs prediction for all 5-secund frames for all birds… completely ignoring the test.csv so any idea why we should care about it for submission as descriptions states \"the full test.csv is provided in the hidden test set\"?
\ninterestingly, why do we need the submission.csv as mentioned \"the full submission.csv is provided in the hidden test set.\"
", "votes": null, "replies": [] }, { "id": 1700335, "author_name": "imeintanis", "author_url": "", "post_date": "02/21/2022 20:57:25", "content": "they mean that when you press the \"Submit\" they replace the sample submission csv with the full one that contain ALL audios (public and private a.k.a hidden test).. as usual in code competition format
", "votes": null, "replies": [] }, { "id": 1700345, "author_name": "jirkaborovec", "author_url": "", "post_date": "02/21/2022 21:33:45", "content": "Yeah, I see your point, but please see mine too.. with the actual setting you are not able to use their test as a mock to develop your submission… for illustration you have the task of result a+b for inpouts 2 and 4 and result is 42
@jirkaborovec have you made any progress on the matter? I've just joined the competition and I'm kinda puzzled, too.
", "votes": null, "replies": [] }, { "id": 1713806, "author_name": "jirkaborovec", "author_url": "", "post_date": "03/06/2022 12:04:20", "content": "seems there is no update on the obvious issue, so just continued with creating predictions for all birds and ignoring the test.csv file, see https://www.kaggle.com/jirkaborovec/birdclef-fake-predictions
", "votes": null, "replies": [] }, { "id": 1714862, "author_name": "gabiboubibel", "author_url": "", "post_date": "03/07/2022 12:27:07", "content": "Hey I did this to have an easy way around the problem :
\ntest = pd.read_csv('../input/birdclef-2022/test.csv')\nif len(test) == 3:\n test['file_id'] = 'soundscape_453028782'\nThen, when I save the notebook I am sure it will run without problem and that it can work for the actual submission.
", "votes": null, "replies": [] }, { "id": 1714882, "author_name": "jirkaborovec", "author_url": "", "post_date": "03/07/2022 12:55:10", "content": "cool, but then you will run your predictions on one audio file instead of the expected test dataset…
", "votes": null, "replies": [] }, { "id": 1714910, "author_name": "gabiboubibel", "author_url": "", "post_date": "03/07/2022 13:26:01", "content": "Yes, if the length of the dataset (test) is equal to 3.
\nBut when it comes to the actual submission the test dataframe will be longer than 3. Thus, you will use the real files*.
\n*Because the 'len(test) == 3' condition will not be true, so you will not change the file_id names.
", "votes": null, "replies": [] }, { "id": 1714946, "author_name": "jirkaborovec", "author_url": "", "post_date": "03/07/2022 14:07:35", "content": "I know already how to generate the submission without the test.csv, I was just asking the hosted could fix it and make the dataset part related to submission rather useful… :)
", "votes": null, "replies": [] }, { "id": 1714949, "author_name": "gabiboubibel", "author_url": "", "post_date": "03/07/2022 14:09:04", "content": "Ha sorry ! I misunderstood the problem then :) !
", "votes": null, "replies": [] }, { "id": 1720691, "author_name": "octpath0302", "author_url": "", "post_date": "03/13/2022 03:27:53", "content": "\n\nOr stick with parsing the soundscape dir. Both should work fine when submitting.
\n
Is this really correct?
\nI searched for soundscape dir and tried to create a submission file,
\nbut I got an error if I don't set the number of segments to 12 (equivalent to 60 seconds) for all files.
Are files with more than 60 seconds evaluated only for the first 60 seconds?
\nI haven't tried submission based on test.csv because I've exceeded today's submission limit.
\nBut I'm guessing that the maximum end_time of each file in the test.csv is \"_60\" even though there are files longer than 60 seconds..
yes, generating the submission purely on the folder with soundtracks and scored_birds.json seems to be working for me in this dummy submission: https://www.kaggle.com/jirkaborovec/birdclef-fake-predictions
Thank you for your reply!
\nThe only difference between my failed code and your code seems to be 'ceil' (yours) or 'floor' (mine).
\nSo it seems that the cause of my failure is a lack of segments, but..
\nI'm confused by the success of my code with a fixed number of segments of 12.
If all the test data are 60 seconds, there should be no difference between 'ceil' and 'floor'.
\nIf there is a test data that is not 60 seconds, I don't know why the submission is successful with the fixed 12 segments.
I will try it tomorrow with reference to your code!
\nThanks again!!
~~~
\nThis is my failed code. (Of course, sample submission works fine.)
for fpath in test_files:\n sig, rate = librosa.load(fpath, sr=32000, mono=True)\n num_segments = len(sig) // (rate*5) \n for seg_idx in range(num_segments): # <---\n ...\nThe following code was successful.
\nfor fpath in test_files:\n sig, rate = librosa.load(fpath, sr=32000, mono=True)\n for seg_idx in range(12): # <---\n ...\nfrom data tab
\n\n", "votes": null, "replies": [] }, { "id": 1721036, "author_name": "octpath0302", "author_url": "", "post_date": "03/13/2022 11:06:51", "content": "the test_soundscapes directory will be populated with approximately 5,500 recordings to be used for scoring. These are each within a few milliseconds of 1 minute long and in the ogg audio format.
\n
Thank you!
\nI completely missed it..
\nSince 'ceil' works well, the audio file is probably up to 1 minute long (not over 1 minute).
I tripped over the exact same thing but after some failed attempts it works fine with ceil. But good to know that they all seem to be <= 60 sec - within a few milliseconds of 1 minute long could be 60k ms +- n ms.