The train_short_audio can be used to train a multi-label classifier. However, the model needs to be trained on 5 second audio clips. I went through the trouble of creating 5 second audio clips using individual audio files in the train_short_audio directory.
You can either create the clips on the fly or use this dataset. By using this dataset your input pipeline will be spared of clipping overhead.
\nI have used a 16 CPU core, 60 GB RAM virtual instance on GCP to create this dataset. You can use Multiprocessing 5 Second Audio Clips kernel to reproduce the result. It uses multiprocessing to create the dataset. The resulting dataset is 20.5 GB in size.
\nNote: I tried to upload the dataset as a Kaggle dataset but all attempts failed - Kaggle API as well as manually uploading the .zip files.
I have created a new W&B team account to upload the entire dataset as an Artifact. If you want to download this dataset please use the code snippet below:
\nimport wandb\nrun = wandb.init()\nartifact = run.use_artifact('birdclef21/audio_clips_zip/bird_audio_clips_zip:v0', type='dataset')\nartifact_dir = artifact.download()\nNote: You will require a W&B account.
\nNote: You will find 398 zipped files where each zipped file is associated to a bird species. I have mistakenly uploaded dataset-metadata.json file. Please ignore it.
\nNote: The download might take time according to the internet speed available to your system.
These are raw clips. It's up to you to do pre-processing, filter out unwanted clips, etc. I will share some of my strategies in near future.
\nI have also created train_clips.csv file that contains metadata for each clip. To download use the code snippet below:
import wandb\nrun = wandb.init()\nartifact = run.use_artifact('birdclef21/audio_clips_zip/bird_audio_clip_csv:v0', type='dataset')\nartifact_dir = artifact.download()\nDo let me know if you face any difficulty downloading the files.
", "messages": [ { "id": "1277020", "postDate": "04/18/2021 09:35:19", "content": "The train_short_audio can be used to train a multi-label classifier. However, the model needs to be trained on 5 second audio clips. I went through the trouble of creating 5 second audio clips using individual audio files in the train_short_audio directory.
You can either create the clips on the fly or use this dataset. By using this dataset your input pipeline will be spared of clipping overhead.
\nI have used a 16 CPU core, 60 GB RAM virtual instance on GCP to create this dataset. You can use Multiprocessing 5 Second Audio Clips kernel to reproduce the result. It uses multiprocessing to create the dataset. The resulting dataset is 20.5 GB in size.
\nNote: I tried to upload the dataset as a Kaggle dataset but all attempts failed - Kaggle API as well as manually uploading the .zip files.
I have created a new W&B team account to upload the entire dataset as an Artifact. If you want to download this dataset please use the code snippet below:
\nimport wandb\nrun = wandb.init()\nartifact = run.use_artifact('birdclef21/audio_clips_zip/bird_audio_clips_zip:v0', type='dataset')\nartifact_dir = artifact.download()\nNote: You will require a W&B account.
\nNote: You will find 398 zipped files where each zipped file is associated to a bird species. I have mistakenly uploaded dataset-metadata.json file. Please ignore it.
\nNote: The download might take time according to the internet speed available to your system.
These are raw clips. It's up to you to do pre-processing, filter out unwanted clips, etc. I will share some of my strategies in near future.
\nI have also created train_clips.csv file that contains metadata for each clip. To download use the code snippet below:
import wandb\nrun = wandb.init()\nartifact = run.use_artifact('birdclef21/audio_clips_zip/bird_audio_clip_csv:v0', type='dataset')\nartifact_dir = artifact.download()\nDo let me know if you face any difficulty downloading the files.
", "rawMarkdown": "The `train_short_audio` can be used to train a multi-label classifier. However, the model needs to be trained on 5 second audio clips. I went through the trouble of creating 5 second audio clips using individual audio files in the `train_short_audio` directory.\n\nYou can either create the clips on the fly or use this dataset. **By using this dataset your input pipeline will be spared of clipping overhead.** \n\nI have used a 16 CPU core, 60 GB RAM virtual instance on GCP to create this dataset. You can use [Multiprocessing 5 Second Audio Clips](https://www.kaggle.com/ayuraj/multiprocessing-5-second-audio-clips) kernel to reproduce the result. It uses multiprocessing to create the dataset. The resulting dataset is 20.5 GB in size. \n\n**Note**: I tried to upload the dataset as a Kaggle dataset but all attempts failed - Kaggle API as well as manually uploading the `.zip` files. \n\nI have created a new [W&B team account](https://docs.wandb.ai/ref/app/features/teams) to upload the entire dataset as an [Artifact](https://docs.wandb.ai/guides/artifacts). If you want to download this dataset please use the code snippet below:\n\n```\nimport wandb\nrun = wandb.init()\nartifact = run.use_artifact('birdclef21/audio_clips_zip/bird_audio_clips_zip:v0', type='dataset')\nartifact_dir = artifact.download()\n```\n\n**Note**: You will require a [W&B account](https://wandb.ai/site). \n**Note**: You will find 398 zipped files where each zipped file is associated to a bird species. I have mistakenly uploaded `dataset-metadata.json` file. Please ignore it. \n**Note**: The download might take time according to the internet speed available to your system.\n\nThese are raw clips. It's up to you to do pre-processing, filter out unwanted clips, etc. I will share some of my strategies in near future. \n\nI have also created `train_clips.csv` file that contains metadata for each clip. To download use the code snippet below:\n\n```\nimport wandb\nrun = wandb.init()\nartifact = run.use_artifact('birdclef21/audio_clips_zip/bird_audio_clip_csv:v0', type='dataset')\nartifact_dir = artifact.download()\n```\n\nDo let me know if you face any difficulty downloading the files.", "votes": null }, { "id": "1277319", "postDate": "04/18/2021 16:14:36", "content": "I believe the size limit on kaggle datasets is 20GB - likely this is the reason you cannot load the data set. I used some different logic and my 5 sec set of clips is 27GB.
\nI am planning to run mine again (I do it on local PC and takes almost a day on the machine I used). I am going to change my logic to extract one less 5 second increment (per file) and see if that reduction of 68K clips will be enough to get me down to less than 20GB.
", "rawMarkdown": "I believe the size limit on kaggle datasets is 20GB - likely this is the reason you cannot load the data set. I used some different logic and my 5 sec set of clips is 27GB. \n\nI am planning to run mine again (I do it on local PC and takes almost a day on the machine I used). I am going to change my logic to extract one less 5 second increment (per file) and see if that reduction of 68K clips will be enough to get me down to less than 20GB.", "votes": null }, { "id": "1277733", "postDate": "04/19/2021 06:27:17", "content": "Good luck with your endeavor. :)
\nTry to use multiprocessing to utilize all the available CPU cores if not already. Also if you think that my data creation logic is okay you can simply use the download the dataset and move on.
\nAlso I don't think it was the size limit due to which I couldn't upload the dataset. I tried uploading individual .zip files (say 40 MB in size) but to no luck.
I believe the size limit on kaggle datasets is 20GB - likely this is the reason you cannot load the data set. I used some different logic and my 5 sec set of clips is 27GB.
\nI am planning to run mine again (I do it on local PC and takes almost a day on the machine I used). I am going to change my logic to extract one less 5 second increment (per file) and see if that reduction of 68K clips will be enough to get me down to less than 20GB.
", "votes": null, "replies": [ { "id": 1277733, "author_name": "ayuraj", "author_url": "", "post_date": "04/19/2021 06:27:17", "content": "Good luck with your endeavor. :)
\nTry to use multiprocessing to utilize all the available CPU cores if not already. Also if you think that my data creation logic is okay you can simply use the download the dataset and move on.
\nAlso I don't think it was the size limit due to which I couldn't upload the dataset. I tried uploading individual .zip files (say 40 MB in size) but to no luck.