I haven't worked with signal processing before, hence it is quite a new thing for me. Although I understand how models work/will work, I am more interested in some background knowledge. I was looking through the documentation of librosa but didn't find much. Here are some of my doubts:\n1. What is hop_length, fmin, fmax, n_mel, and n_fft means? \n2. What is their importance and how the values affect the signal overall? Is there any general guideline for these values or just a random search?\n3. What are the disadvantages of using mel_spectogram?
I haven't worked with signal processing before, hence it is quite a new thing for me. Although I understand how models work/will work, I am more interested in some background knowledge. I was looking through the documentation of librosa but didn't find much. Here are some of my doubts:\n1. What is hop_length, fmin, fmax, n_mel, and n_fft means? \n2. What is their importance and how the values affect the signal overall? Is there any general guideline for these values or just a random search?\n3. What are the disadvantages of using mel_spectogram?
Here's a blog post from someone who took part in last year's Freesound Kaggle challenge that might be helpful\nhttps://towardsdatascience.com/audio-classification-using-fastai-and-on-the-fly-frequency-transforms-4dbe1b540f89
\n\nThis is also a nice interactive way to learn some basic signal processing without getting drowned in theory\nhttps://jackschaedler.github.io/circles-sines-signals/dft_introduction.html
\n\nMel spectrograms are a good general-purpose audio feature and you could use the hyperparameters we have used in the Challenge Baseline as a sane starting point (see Input Features section of https://github.com/DCASE-REPO/dcase2019_task2_baseline/blob/master/README.md)
", "rawMarkdown": "Here's a blog post from someone who took part in last year's Freesound Kaggle challenge that might be helpful\nhttps://towardsdatascience.com/audio-classification-using-fastai-and-on-the-fly-frequency-transforms-4dbe1b540f89\n\nThis is also a nice interactive way to learn some basic signal processing without getting drowned in theory\nhttps://jackschaedler.github.io/circles-sines-signals/dft_introduction.html\n\nMel spectrograms are a good general-purpose audio feature and you could use the hyperparameters we have used in the Challenge Baseline as a sane starting point (see Input Features section of https://github.com/DCASE-REPO/dcase2019_task2_baseline/blob/master/README.md)", "votes": null }, { "id": "525788", "postDate": "05/01/2019 18:03:24", "content": "Raw audio is some digital representation of air pressure. It goes to microphone similar like to human ear.\nHowever, we can suspect (or prove?) that humans hearing is focused on frequencies. The different representation of raw audio samples is spectrogram which is a visualization of Short Time Fourier Transform, STFT, frequency representation.\nWe could calculate all the frequencies in all the audio at once using Fast Fourier Transform, FFT. But it will not show the dynamics of audio, the changes in it, just all the present frequencies. So we can split the audio signal into some smaller parts and then calculate frequencies in consecutive parts. This is STFT.
\n\nThe questions here are: how do we split audio? Into some windows.
\n\nThe length of that window is *n_fft* (in samples we can convert it to milliseconds. ~2200 samples, sampling freq 44100, is around 50 milliseconds. Do you like this value? For speech, it seems too long because some sounds last less than that. Think about the best value for sound classification).
\n\nIf we split raw audion in equal windows, we can miss some information that is present in the border. So we can overlap the windows. *hop_length* is the value in samples, how big is the \"jump\" between the consecutive windows.
\n\nWe don't use raw spectrogram. We instead filter the frequencies using filters designed to mimic human hearing. these are mel filters. You can choose how many of them to create, setting *n_mel*. The signal has a sampling rate 44100, so the frequencies in it are between 0 and 22050. Using 128 mels means (not really because filters are not equal in freq sizes) that you represent 22050/128=172Hz on average. So one feature is around 172Hz in space between 0 and 22015 Hz. Or, you can set fmin and fmax to set the frequencies between which you will create mel filters.
\n\nMaybe you will like this\nhttps://www.kaggle.com/davids1992/speech-representation-and-data-exploration\nhttps://www.kaggle.com/davids1992/audio-representation-what-it-s-all-about
", "rawMarkdown": "Raw audio is some digital representation of air pressure. It goes to microphone similar like to human ear.\nHowever, we can suspect (or prove?) that humans hearing is focused on frequencies. The different representation of raw audio samples is spectrogram which is a visualization of Short Time Fourier Transform, STFT, frequency representation.\nWe could calculate all the frequencies in all the audio at once using Fast Fourier Transform, FFT. But it will not show the dynamics of audio, the changes in it, just all the present frequencies. So we can split the audio signal into some smaller parts and then calculate frequencies in consecutive parts. This is STFT.\n\nThe questions here are: how do we split audio? Into some windows. \n\nThe length of that window is *n_fft* (in samples we can convert it to milliseconds. ~2200 samples, sampling freq 44100, is around 50 milliseconds. Do you like this value? For speech, it seems too long because some sounds last less than that. Think about the best value for sound classification). \n\nIf we split raw audion in equal windows, we can miss some information that is present in the border. So we can overlap the windows. *hop_length* is the value in samples, how big is the \"jump\" between the consecutive windows.\n\nWe don't use raw spectrogram. We instead filter the frequencies using filters designed to mimic human hearing. these are mel filters. You can choose how many of them to create, setting *n_mel*. The signal has a sampling rate 44100, so the frequencies in it are between 0 and 22050. Using 128 mels means (not really because filters are not equal in freq sizes) that you represent 22050/128=172Hz on average. So one feature is around 172Hz in space between 0 and 22015 Hz. Or, you can set *fmin* and *fmax* to set the frequencies between which you will create mel filters.\n\n\nMaybe you will like this\nhttps://www.kaggle.com/davids1992/speech-representation-and-data-exploration\nhttps://www.kaggle.com/davids1992/audio-representation-what-it-s-all-about", "votes": null }, { "id": "525990", "postDate": "05/02/2019 06:04:08", "content": "Thanks @davids1992 for the detailed info. This is quiet good for the start. Though I have one question. If you are using Conv1D, then it is better to pass raw audio to the convolution as it can figure out what to look for and in this case the input shape for conv would be (nframes, 1) but if you want to pass spectogram to Conv1D, you can do it as well but that won't be much useful as we already restricted the amount of information and the input shape in this case would be (features_dim_of_Spectogram, timesteps), right?
Thanks @plakal . I will look into it. The only weird thing is that you are still using slim, which I personally don't like as I more of a keras person but I will try to convert the code correspondingly
You don't have to use the code. My suggestion was to use the hyperparameter values for feature extraction (STFT window/hop size, number of mel bands, etc) which you can then use in your librosa or keras implementation.
", "rawMarkdown": "You don't have to use the code. My suggestion was to use the hyperparameter values for feature extraction (STFT window/hop size, number of mel bands, etc) which you can then use in your librosa or keras implementation.", "votes": null }, { "id": "526569", "postDate": "05/03/2019 09:42:41", "content": "I don't have big experience with Convolutional Neural Networks. I think Conv1D can be useful with spectrograms as well as Conv2D. 1D should filter a few consecutive frames, not frequencies. It makes sense when you think about what the features represent.\nI don't think it's feasible to work directly on raw audio. Notice, however, that I'm doing pretty bad on this competition :)
", "rawMarkdown": "I don't have big experience with Convolutional Neural Networks. I think *Conv1D* can be useful with spectrograms as well as Conv2D. 1D should filter a few consecutive frames, not frequencies. It makes sense when you think about what the features represent.\nI don't think it's feasible to work directly on raw audio. Notice, however, that I'm doing pretty bad on this competition :)", "votes": null } ], "comments": [ { "id": 525771, "author_name": "plakal", "author_url": "", "post_date": "05/01/2019 17:24:51", "content": "Here's a blog post from someone who took part in last year's Freesound Kaggle challenge that might be helpful\nhttps://towardsdatascience.com/audio-classification-using-fastai-and-on-the-fly-frequency-transforms-4dbe1b540f89
\n\nThis is also a nice interactive way to learn some basic signal processing without getting drowned in theory\nhttps://jackschaedler.github.io/circles-sines-signals/dft_introduction.html
\n\nMel spectrograms are a good general-purpose audio feature and you could use the hyperparameters we have used in the Challenge Baseline as a sane starting point (see Input Features section of https://github.com/DCASE-REPO/dcase2019_task2_baseline/blob/master/README.md)
", "votes": null, "replies": [ { "id": 525991, "author_name": "aakashnain", "author_url": "", "post_date": "05/02/2019 06:05:16", "content": "Thanks @plakal . I will look into it. The only weird thing is that you are still using slim, which I personally don't like as I more of a keras person but I will try to convert the code correspondingly
You don't have to use the code. My suggestion was to use the hyperparameter values for feature extraction (STFT window/hop size, number of mel bands, etc) which you can then use in your librosa or keras implementation.
", "votes": null, "replies": [] } ] }, { "id": 525788, "author_name": "davids1992", "author_url": "", "post_date": "05/01/2019 18:03:24", "content": "Raw audio is some digital representation of air pressure. It goes to microphone similar like to human ear.\nHowever, we can suspect (or prove?) that humans hearing is focused on frequencies. The different representation of raw audio samples is spectrogram which is a visualization of Short Time Fourier Transform, STFT, frequency representation.\nWe could calculate all the frequencies in all the audio at once using Fast Fourier Transform, FFT. But it will not show the dynamics of audio, the changes in it, just all the present frequencies. So we can split the audio signal into some smaller parts and then calculate frequencies in consecutive parts. This is STFT.
\n\nThe questions here are: how do we split audio? Into some windows.
\n\nThe length of that window is *n_fft* (in samples we can convert it to milliseconds. ~2200 samples, sampling freq 44100, is around 50 milliseconds. Do you like this value? For speech, it seems too long because some sounds last less than that. Think about the best value for sound classification).
\n\nIf we split raw audion in equal windows, we can miss some information that is present in the border. So we can overlap the windows. *hop_length* is the value in samples, how big is the \"jump\" between the consecutive windows.
\n\nWe don't use raw spectrogram. We instead filter the frequencies using filters designed to mimic human hearing. these are mel filters. You can choose how many of them to create, setting *n_mel*. The signal has a sampling rate 44100, so the frequencies in it are between 0 and 22050. Using 128 mels means (not really because filters are not equal in freq sizes) that you represent 22050/128=172Hz on average. So one feature is around 172Hz in space between 0 and 22015 Hz. Or, you can set fmin and fmax to set the frequencies between which you will create mel filters.
\n\nMaybe you will like this\nhttps://www.kaggle.com/davids1992/speech-representation-and-data-exploration\nhttps://www.kaggle.com/davids1992/audio-representation-what-it-s-all-about
", "votes": null, "replies": [ { "id": 525990, "author_name": "aakashnain", "author_url": "", "post_date": "05/02/2019 06:04:08", "content": "Thanks @davids1992 for the detailed info. This is quiet good for the start. Though I have one question. If you are using Conv1D, then it is better to pass raw audio to the convolution as it can figure out what to look for and in this case the input shape for conv would be (nframes, 1) but if you want to pass spectogram to Conv1D, you can do it as well but that won't be much useful as we already restricted the amount of information and the input shape in this case would be (features_dim_of_Spectogram, timesteps), right?
I don't have big experience with Convolutional Neural Networks. I think Conv1D can be useful with spectrograms as well as Conv2D. 1D should filter a few consecutive frames, not frequencies. It makes sense when you think about what the features represent.\nI don't think it's feasible to work directly on raw audio. Notice, however, that I'm doing pretty bad on this competition :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "525721": "I haven't worked with signal processing before, hence it is quite a new thing for me. Although I understand how models work/will work, I am more interested in some background knowledge. I was looking through the documentation of `librosa` but didn't find much. Here are some of my doubts:\n1. What is `hop_length`, `fmin`, `fmax`, `n_mel`, and `n_fft` means? \n2. What is their importance and how the values affect the signal overall? Is there any general guideline for these values or just a random search?\n3. What are the disadvantages of using `mel_spectogram`?", "525771": "Here's a blog post from someone who took part in last year's Freesound Kaggle challenge that might be helpful\nhttps://towardsdatascience.com/audio-classification-using-fastai-and-on-the-fly-frequency-transforms-4dbe1b540f89\n\nThis is also a nice interactive way to learn some basic signal processing without getting drowned in theory\nhttps://jackschaedler.github.io/circles-sines-signals/dft_introduction.html\n\nMel spectrograms are a good general-purpose audio feature and you could use the hyperparameters we have used in the Challenge Baseline as a sane starting point (see Input Features section of https://github.com/DCASE-REPO/dcase2019_task2_baseline/blob/master/README.md)", "525788": "Raw audio is some digital representation of air pressure. It goes to microphone similar like to human ear.\nHowever, we can suspect (or prove?) that humans hearing is focused on frequencies. The different representation of raw audio samples is spectrogram which is a visualization of Short Time Fourier Transform, STFT, frequency representation.\nWe could calculate all the frequencies in all the audio at once using Fast Fourier Transform, FFT. But it will not show the dynamics of audio, the changes in it, just all the present frequencies. So we can split the audio signal into some smaller parts and then calculate frequencies in consecutive parts. This is STFT.\n\nThe questions here are: how do we split audio? Into some windows. \n\nThe length of that window is *n_fft* (in samples we can convert it to milliseconds. ~2200 samples, sampling freq 44100, is around 50 milliseconds. Do you like this value? For speech, it seems too long because some sounds last less than that. Think about the best value for sound classification). \n\nIf we split raw audion in equal windows, we can miss some information that is present in the border. So we can overlap the windows. *hop_length* is the value in samples, how big is the \"jump\" between the consecutive windows.\n\nWe don't use raw spectrogram. We instead filter the frequencies using filters designed to mimic human hearing. these are mel filters. You can choose how many of them to create, setting *n_mel*. The signal has a sampling rate 44100, so the frequencies in it are between 0 and 22050. Using 128 mels means (not really because filters are not equal in freq sizes) that you represent 22050/128=172Hz on average. So one feature is around 172Hz in space between 0 and 22015 Hz. Or, you can set *fmin* and *fmax* to set the frequencies between which you will create mel filters.\n\n\nMaybe you will like this\nhttps://www.kaggle.com/davids1992/speech-representation-and-data-exploration\nhttps://www.kaggle.com/davids1992/audio-representation-what-it-s-all-about", "525990": "Thanks @davids1992 for the detailed info. This is quiet good for the start. Though I have one question. If you are using `Conv1D`, then it is better to pass raw audio to the convolution as it can figure out what to look for and in this case the input shape for conv would be `(nframes, 1)` but if you want to pass `spectogram` to `Conv1D`, you can do it as well but that won't be much useful as we already restricted the amount of information and the input shape in this case would be `(features_dim_of_Spectogram, timesteps)`, right?", "525991": "Thanks @plakal . I will look into it. The only weird thing is that you are still using `slim`, which I personally don't like as I more of a `keras` person but I will try to convert the code correspondingly", "526006": "You don't have to use the code. My suggestion was to use the hyperparameter values for feature extraction (STFT window/hop size, number of mel bands, etc) which you can then use in your librosa or keras implementation.", "526569": "I don't have big experience with Convolutional Neural Networks. I think *Conv1D* can be useful with spectrograms as well as Conv2D. 1D should filter a few consecutive frames, not frequencies. It makes sense when you think about what the features represent.\nI don't think it's feasible to work directly on raw audio. Notice, however, that I'm doing pretty bad on this competition :)" }, "source": "meta" }