\nThe first crop can have equal paddings on both sides.
\n
![\"\"](\"https://raw.githubusercontent.com/dimitreOliveira/MachineLearning/master/Kaggle/Rainforest%20Connection%20Species%20Audio%20Detection/Audio%20crop%20diagram_2.png\")
When I started looking at this competition one thing that confused me a little was how to effectively crop the audio, especially using Tensorflow to maximize the efficiency, I studied a few public notebooks and ended up making a lot of experiments into that, and have written a few functions that should be able to crop the data with some flexibility.
\nBesides that, the importance of audio cropping has been discussed many times here, especially for creating an effective validation, references: [1], [2], [3], [4].
The Idea is to first crop the audio with a given crop size then apply random crops at this first crop. The idea is very similar to what most public notebooks uses, one thing that I tried to make sure of is that the first crop had reasonable padding in cases where the label was small.
You can find the notebook here: Rainforest-Audio classification TF Improved
\nTo help with the understanding let's look at these examples:
\n1st example
\nThe first crop can have equal paddings on both sides.
\n
The first crop can't have equal paddings, so one side will have larger padding.
\n![\"\"](\"https://raw.githubusercontent.com/dimitreOliveira/MachineLearning/master/Kaggle/Rainforest%20Connection%20Species%20Audio%20Detection/Audio%20crop%20diagram.png\")
def crop_audio(audio, tmin, tmax, min_crop_size=MIN_CROP_SIZE, sample_rate=48000, max_size=60.):\n \"\"\"\n Crops a 'waveform' file to have {min_crop_size} size given, {tmin}, {tmax}, {sample_rate} and {max_size}.\n \"\"\"\n label_size = tmax - tmin\n\n if label_size >= min_crop_size: # No padding needed\n cut_min = tmin\n cut_max = tmax \n else: # Needs padding\n pad_start = (min_crop_size - label_size) / 2\n cut_min = tf.maximum(0., (tmin - pad_start))\n\n pad_end = (min_crop_size - (label_size - cut_min))\n cut_max = tf.minimum(max_size, (tmax + pad_end))\n\n cut_size = cut_max - cut_min\n\n if cut_size < min_crop_size:\n cut_min = tf.maximum(0., (cut_min - (min_crop_size - cut_size)))\n\n cut_size = cut_max - cut_min\n\n # Casting tensors\n cut_min = tf.cast((cut_min * sample_rate), tf.int32)\n cut_max = tf.cast((cut_max * sample_rate), tf.int32)\n cut_size = tf.cast((cut_size * sample_rate), tf.int32)\n\n audio = audio[cut_min:cut_max] # croping the audio\n audio = audio[:cut_size] # making sure it has the max size\n\n audio = tf.reshape(audio, [cut_size]) # making sure it has the expected shape\n return audio\ndef random_crop_audio(audio, crop_size=CROP_SIZE, sample_rate=48000, max_size=60):\n \"\"\"\n Randomly crops a 'waveform' file to have {crop_size} size given, {sample_rate} and {max_size}.\n \"\"\"\n start = tf.random.uniform([], minval=0, \n maxval=(max_size - crop_size), \n dtype=tf.int32)\n cut_min = start * sample_rate\n cut_max = (start + crop_size) * sample_rate\n\n audio_size = len(audio)\n if cut_max > audio_size:\n cut_min -= cut_max - audio_size\n cut_max = cut_min + (crop_size * sample_rate)\n\n # Casting tensors\n cut_min = tf.cast(cut_min, tf.int32)\n cut_max = tf.cast(cut_max, tf.int32)\n cut_size = tf.cast((crop_size*sample_rate), tf.int32)\n\n audio = audio[cut_min:cut_max] # croping the audio\n audio = audio[:cut_size] # making sure it has the max size\n\n audio = tf.reshape(audio, [cut_size]) # making sure it has the expected shape\n return audio\nI am not very familiar with audio data but I will add a few observations here
\nEfficientNet seems to work better than ResNets, I could not get ResNets to perform as well as public notebooks.When I started looking at this competition one thing that confused me a little was how to effectively crop the audio, especially using Tensorflow to maximize the efficiency, I studied a few public notebooks and ended up making a lot of experiments into that, and have written a few functions that should be able to crop the data with some flexibility.
\nBesides that, the importance of audio cropping has been discussed many times here, especially for creating an effective validation, references: [1], [2], [3], [4].
The Idea is to first crop the audio with a given crop size then apply random crops at this first crop. The idea is very similar to what most public notebooks uses, one thing that I tried to make sure of is that the first crop had reasonable padding in cases where the label was small.
You can find the notebook here: Rainforest-Audio classification TF Improved
\nTo help with the understanding let's look at these examples:
\n1st example
\nThe first crop can have equal paddings on both sides.
\n
The first crop can't have equal paddings, so one side will have larger padding.
\n
def crop_audio(audio, tmin, tmax, min_crop_size=MIN_CROP_SIZE, sample_rate=48000, max_size=60.):\n \"\"\"\n Crops a 'waveform' file to have {min_crop_size} size given, {tmin}, {tmax}, {sample_rate} and {max_size}.\n \"\"\"\n label_size = tmax - tmin\n\n if label_size >= min_crop_size: # No padding needed\n cut_min = tmin\n cut_max = tmax \n else: # Needs padding\n pad_start = (min_crop_size - label_size) / 2\n cut_min = tf.maximum(0., (tmin - pad_start))\n\n pad_end = (min_crop_size - (label_size - cut_min))\n cut_max = tf.minimum(max_size, (tmax + pad_end))\n\n cut_size = cut_max - cut_min\n\n if cut_size < min_crop_size:\n cut_min = tf.maximum(0., (cut_min - (min_crop_size - cut_size)))\n\n cut_size = cut_max - cut_min\n\n # Casting tensors\n cut_min = tf.cast((cut_min * sample_rate), tf.int32)\n cut_max = tf.cast((cut_max * sample_rate), tf.int32)\n cut_size = tf.cast((cut_size * sample_rate), tf.int32)\n\n audio = audio[cut_min:cut_max] # croping the audio\n audio = audio[:cut_size] # making sure it has the max size\n\n audio = tf.reshape(audio, [cut_size]) # making sure it has the expected shape\n return audio\ndef random_crop_audio(audio, crop_size=CROP_SIZE, sample_rate=48000, max_size=60):\n \"\"\"\n Randomly crops a 'waveform' file to have {crop_size} size given, {sample_rate} and {max_size}.\n \"\"\"\n start = tf.random.uniform([], minval=0, \n maxval=(max_size - crop_size), \n dtype=tf.int32)\n cut_min = start * sample_rate\n cut_max = (start + crop_size) * sample_rate\n\n audio_size = len(audio)\n if cut_max > audio_size:\n cut_min -= cut_max - audio_size\n cut_max = cut_min + (crop_size * sample_rate)\n\n # Casting tensors\n cut_min = tf.cast(cut_min, tf.int32)\n cut_max = tf.cast(cut_max, tf.int32)\n cut_size = tf.cast((crop_size*sample_rate), tf.int32)\n\n audio = audio[cut_min:cut_max] # croping the audio\n audio = audio[:cut_size] # making sure it has the max size\n\n audio = tf.reshape(audio, [cut_size]) # making sure it has the expected shape\n return audio\nI am not very familiar with audio data but I will add a few observations here
\nEfficientNet seems to work better than ResNets, I could not get ResNets to perform as well as public notebooks.Thanks for all your contributions Dimitre. Your notebooks are excellent. You share many good ideas.
", "rawMarkdown": "Thanks for all your contributions Dimitre. Your notebooks are excellent. You share many good ideas.", "votes": null }, { "id": "1191313", "postDate": "02/08/2021 11:41:51", "content": "Hey @cdeotte , Thank you very much!
\nWorking with audio is very interesting, next time I will start earlier 😄
", "rawMarkdown": "Hey @cdeotte , Thank you very much!\n\nWorking with audio is very interesting, next time I will start earlier 😄", "votes": null }, { "id": "1195517", "postDate": "02/10/2021 21:24:27", "content": "Thanks for Sharing, I've been trying to crop audio with 2-4s length but don see any improved, maybe my model is not going well. Your notebook give more ideas, thanks again 👍
", "rawMarkdown": "Thanks for Sharing, I've been trying to crop audio with 2-4s length but don see any improved, maybe my model is not going well. Your notebook give more ideas, thanks again 👍", "votes": null }, { "id": "1195526", "postDate": "02/10/2021 21:36:59", "content": "You're welcome @scarecrow2020 ,
\nI did not have the TPU quota to experiment a lot, but I think that the best audio sizes should be somewhere around 4~10
", "rawMarkdown": "You're welcome @scarecrow2020 ,\n\nI did not have the TPU quota to experiment a lot, but I think that the best audio sizes should be somewhere around 4~10", "votes": null } ], "comments": [ { "id": 1190917, "author_name": "cdeotte", "author_url": "", "post_date": "02/08/2021 05:34:05", "content": "Thanks for all your contributions Dimitre. Your notebooks are excellent. You share many good ideas.
", "votes": null, "replies": [ { "id": 1191313, "author_name": "dimitreoliveira", "author_url": "", "post_date": "02/08/2021 11:41:51", "content": "Hey @cdeotte , Thank you very much!
\nWorking with audio is very interesting, next time I will start earlier 😄
", "votes": null, "replies": [] } ] }, { "id": 1195517, "author_name": "scarecrow2020", "author_url": "", "post_date": "02/10/2021 21:24:27", "content": "Thanks for Sharing, I've been trying to crop audio with 2-4s length but don see any improved, maybe my model is not going well. Your notebook give more ideas, thanks again 👍
", "votes": null, "replies": [ { "id": 1195526, "author_name": "dimitreoliveira", "author_url": "", "post_date": "02/10/2021 21:36:59", "content": "You're welcome @scarecrow2020 ,
\nI did not have the TPU quota to experiment a lot, but I think that the best audio sizes should be somewhere around 4~10
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1189261": "When I started looking at this competition one thing that confused me a little was how to effectively crop the audio, especially using Tensorflow to maximize the efficiency, I studied a few public notebooks and ended up making a lot of experiments into that, and have written a few functions that should be able to crop the data with some flexibility.\nBesides that, the importance of audio cropping has been discussed many times here, especially for creating an effective validation, references: [[1]](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/200922), [[2]](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/207624), [[3]](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/215233), [[4]](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/216564).\n\nThe Idea is to first crop the audio with a given `crop size` then apply **random crops** at this **first crop**. The idea is very similar to what most public notebooks uses, one thing that I tried to make sure of is that the first crop had **reasonable padding** in cases where the label was small.\n\nYou can find the notebook here: [Rainforest-Audio classification TF Improved](https://www.kaggle.com/dimitreoliveira/rainforest-audio-classification-tf-improved)\n\nTo help with the understanding let's look at these examples:\n\n**1st example**\nThe first crop can have equal paddings on both sides.\n\n\nThe first crop can't have equal paddings, so one side will have larger padding.\n\n\n---\n### The code\n\n```\ndef crop_audio(audio, tmin, tmax, min_crop_size=MIN_CROP_SIZE, sample_rate=48000, max_size=60.):\n \"\"\"\n Crops a 'waveform' file to have {min_crop_size} size given, {tmin}, {tmax}, {sample_rate} and {max_size}.\n \"\"\"\n label_size = tmax - tmin\n \n if label_size >= min_crop_size: # No padding needed\n cut_min = tmin\n cut_max = tmax \n else: # Needs padding\n pad_start = (min_crop_size - label_size) / 2\n cut_min = tf.maximum(0., (tmin - pad_start))\n \n pad_end = (min_crop_size - (label_size - cut_min))\n cut_max = tf.minimum(max_size, (tmax + pad_end))\n \n cut_size = cut_max - cut_min\n \n if cut_size < min_crop_size:\n cut_min = tf.maximum(0., (cut_min - (min_crop_size - cut_size)))\n \n cut_size = cut_max - cut_min\n \n # Casting tensors\n cut_min = tf.cast((cut_min * sample_rate), tf.int32)\n cut_max = tf.cast((cut_max * sample_rate), tf.int32)\n cut_size = tf.cast((cut_size * sample_rate), tf.int32)\n \n audio = audio[cut_min:cut_max] # croping the audio\n audio = audio[:cut_size] # making sure it has the max size\n \n audio = tf.reshape(audio, [cut_size]) # making sure it has the expected shape\n return audio\n```\n```\ndef random_crop_audio(audio, crop_size=CROP_SIZE, sample_rate=48000, max_size=60):\n \"\"\"\n Randomly crops a 'waveform' file to have {crop_size} size given, {sample_rate} and {max_size}.\n \"\"\"\n start = tf.random.uniform([], minval=0, \n maxval=(max_size - crop_size), \n dtype=tf.int32)\n cut_min = start * sample_rate\n cut_max = (start + crop_size) * sample_rate\n \n audio_size = len(audio)\n if cut_max > audio_size:\n cut_min -= cut_max - audio_size\n cut_max = cut_min + (crop_size * sample_rate)\n \n # Casting tensors\n cut_min = tf.cast(cut_min, tf.int32)\n cut_max = tf.cast(cut_max, tf.int32)\n cut_size = tf.cast((crop_size*sample_rate), tf.int32)\n \n audio = audio[cut_min:cut_max] # croping the audio\n audio = audio[:cut_size] # making sure it has the max size\n \n audio = tf.reshape(audio, [cut_size]) # making sure it has the expected shape\n return audio\n```\n\n---\nI am not very familiar with audio data but I will add a few observations here\n\n- For me `EfficientNet` seems to work better than `ResNets`, I could not get ResNets to perform as well as public notebooks.\n- Mel-spectrograms performed worse than regular spectrograms, but I might be doing something wrong.\n- Tweaking the crop sizes seems more relevant than the spectrograms resolution (height x width)\n- Incrementing the architecture at the top of the backbone matters very little.", "1190917": "Thanks for all your contributions Dimitre. Your notebooks are excellent. You share many good ideas.", "1191313": "Hey @cdeotte , Thank you very much!\n\nWorking with audio is very interesting, next time I will start earlier 😄", "1195517": "Thanks for Sharing, I've been trying to crop audio with 2-4s length but don see any improved, maybe my model is not going well. Your notebook give more ideas, thanks again 👍", "1195526": "You're welcome @scarecrow2020 ,\n\nI did not have the TPU quota to experiment a lot, but I think that the best audio sizes should be somewhere around 4~10" }, "source": "meta" }