![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10590800%2Fd5a8b3f1f76a0c3aa612dedaf13c603a%2Fmel-spectrogram.png?generation=1679430426479307&alt=media\")
Usually one of the first steps I take when exploring tabular data is to check for duplicates. I discovered that when the filename and url columns were dropped from train_meta.csv there were 2503 duplicate rows. No big deal right? Some of the files could have been recorded by the same author, in the same location, and have the same rating etc and still be totally different recordings. That's when I decided to dig a little deeper:
I ended up finding 5 duplicate audio files in total, but there could definitely be more as I only explored the prementioned 2503 duplicate rows of train_meta.csv. Additionally, it appears that some of the audio files seem to be split into multiple parts which could potentially mean some of the audio files overlap each other. NOTE: You can view all 5 duplicate pairs in the EDA notebook I recently posted.
Usually one of the first steps I take when exploring tabular data is to check for duplicates. I discovered that when the filename and url columns were dropped from train_meta.csv there were 2503 duplicate rows. No big deal right? Some of the files could have been recorded by the same author, in the same location, and have the same rating etc and still be totally different recordings. That's when I decided to dig a little deeper:
I ended up finding 5 duplicate audio files in total, but there could definitely be more as I only explored the prementioned 2503 duplicate rows of train_meta.csv. Additionally, it appears that some of the audio files seem to be split into multiple parts which could potentially mean some of the audio files overlap each other. NOTE: You can view all 5 duplicate pairs in the EDA notebook I recently posted.
Thanks for investigating. Yet, it seems there's not much we can do if duplicate files have a different XC ID (i.e., filename). Sometimes recordists might confuse files they upload, or (happens pretty often) might split a longer file into two before uploading.
", "rawMarkdown": "Thanks for investigating. Yet, it seems there's not much we can do if duplicate files have a different XC ID (i.e., filename). Sometimes recordists might confuse files they upload, or (happens pretty often) might split a longer file into two before uploading.", "votes": null }, { "id": "2194363", "postDate": "03/23/2023 22:05:36", "content": "Hello @stefankahl, thank you for your response. My original strategy for identifying these duplicate audio files was rather inefficient and I have recently came up with a much more efficient method. The key is to compute the audio duration for each file in train_meta.csv. This can be achieved using librosa.get_duration with good floating point precision. When the audio length, author and location (longitude, latitude) are the same, there is potential for a duplicate audio file. Using this method I have discovered an additional 5 duplicate audio files (for a total of 10) and I have added them to my EDA notebook. I have also created a dataset that is a copy of train_meta.csv but I added two additional columns: duration_seconds & duration_minutes that you can find here if anyone would like to attempt to find more duplicate audio files.
Thanks for investigating. Yet, it seems there's not much we can do if duplicate files have a different XC ID (i.e., filename). Sometimes recordists might confuse files they upload, or (happens pretty often) might split a longer file into two before uploading.
", "votes": null, "replies": [ { "id": 2194363, "author_name": "mattop", "author_url": "", "post_date": "03/23/2023 22:05:36", "content": "Hello @stefankahl, thank you for your response. My original strategy for identifying these duplicate audio files was rather inefficient and I have recently came up with a much more efficient method. The key is to compute the audio duration for each file in train_meta.csv. This can be achieved using librosa.get_duration with good floating point precision. When the audio length, author and location (longitude, latitude) are the same, there is potential for a duplicate audio file. Using this method I have discovered an additional 5 duplicate audio files (for a total of 10) and I have added them to my EDA notebook. I have also created a dataset that is a copy of train_meta.csv but I added two additional columns: duration_seconds & duration_minutes that you can find here if anyone would like to attempt to find more duplicate audio files.