{"cells":[{"metadata":{},"cell_type":"markdown","source":"## LSTM Sound Classification\n\n#### Why should you look through this notebook?\nThe performance of my model is not great. But I think my spectrogram generator is quite nice"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nfrom keras.models import load_model\nimport os\nfrom tqdm import tqdm\nFiles = {    \n        'Test':{\n                'csv':'../input/freesound-audio-tagging-2019/sample_submission.csv',\n                'wav_dir':'../input/freesound-audio-tagging-2019/test'}\n        }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(Files['Test']['csv'])\nwavs = df.loc[:,'fname'].tolist()\nPATH = Files['Test']['wav_dir']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Custom Spectrogram\n\nI started with Librosa's spectrogram generator but decided to create my own so that I could more easily control things like pixel:frequency ratios. \n\n#### Sidenote\nI found that Librosa.load(fname) was quite slow at unpacking .wav files. I also tried my own method, using python's Struct and Wave modules but saw no improvement. Scipy's read_wav consistently took 33% less time than Librosa's load function. This makes a big difference when reading so many files.\n\n### Technical details\n\nA spectrogram is just a bunch of frequency distributions placed side by side in order to show visually how the frequency distribution changes as time progresses. In order to create one, you must break up the audio at regular intervals and obtain the frequency spectrum for each of those segments. This is achieved using a (Fast) Fourier Transformation or (FFT).\n\n\n#### Window Size\nHow many samples long should the segments be?\n\n* Powers of 2 seem to speed this up a LOT. \n\n* Larger window --> higher frequency resolution but lower time evolution resolution\n    \n* A window length of 1 second (44100 samples) gives you a 1Hz resolution. Typical humans have about 3Hz resolution, so 0.3 seconds (14700 samples) is probably the largest window you would want. (2048 samples) gives you (~1/20 seconds) time resolution and(20 Hz) frequency resolution. Which I think is a nice middle ground.\n\n#### Cuttoff Frequency\n\n* To save some space and to direct the algorithm to the most important information, I send it only the frequency information from 0-3000Hz. This should probably be closer to 5000Hz\n\n#### Spectrogram Window\n* In the following IPython cell you can see the spectrogram function has an input variable `N`. This is the width of the output image in pixels, and the number of segments of audio over which we compute an FFT. This means length of the audio represented in the final image is fixed by `N` and `window_length`. The relationship is shown by\n\n$Time = N*\\dfrac{window\\_length}{sample\\_rate} = 50*\\dfrac{2048}{44100}=2.32$ \n\n* In order to fix the time dimension of our spectrograms I somewhat arbitrarily chose each spectrogram to cover 2.3 seconds of the audio file in question\n* For files shorter than 2.3 seconds I pad the waveform with zeros\n* For those larger, I select the loudest consecutive 2.3 seconds of the clip\n * This is probably a very large source of error\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.fftpack import rfft\nfrom scipy.io.wavfile import read as read_wav\n\n# N is the width of the output image and determines the duration of the audio which is processed\ndef spectrogram(y,sr,N=50): \n    \n    # ---------------Segment audio for FFTs-------------\n    window_length = 2048 #1024          \n    num_windows = len(y)//window_length\n    \n    \n    \n    # -------------Select Portion of Audio to use-----------------\n    # If we have the exact length for only one choice for our 2.3 second window\n    if num_windows==N:\n        y = y[:N*window_length]\n    \n    # If we need to add some zeros to bring it up to size\n    elif num_windows<N:\n        diff = N*window_length - len(y)\n        before = diff//2\n        after = diff-before\n        y = np.pad(y,(before,after), mode='constant', constant_values=0)\n    \n    #------------------------------------------\n    # If we need to select the best portion to use\n    \n    #  THIS IS THE PLACE TO START FOR IMPROVEMENT\n    \n    # A better approach would be to select one of the loud portions at random\n    #    in an image generator for the learning model. But due to some memory\n    #    issues I'm having in Kaggle Kernels, I could not get this to work. And\n    #    it takes much longer\n    #------------------------------------------\n    else:\n        \n        \n        volume = []\n        for i in range(0,len(y)-window_length*N+1,window_length):\n            volume.append(abs(y[i:i+window_length*N]).sum())\n        volume = np.array(volume)\n \n        m = max(np.array(volume).argmax()-5,0)\n        y = y[window_length*m:window_length*(m+N)]\n    \n\n    # --------------------Compute FFT----------------------\n    y = y.reshape((N,window_length))\n    Y = abs(rfft(y,axis=1)).T\n    \n\n    # ---------------------Normalize-----------------------\n    Y = (Y-Y.min())\n    if Y.max()==0:\n        pass\n    else:\n        Y = Y/Y.max()\n    \n    # ---------Apply Cutoff Frequency (20-3000Hz)----------\n    # Convert to np.float32 to save memory\n    return Y[1:150,:].astype(np.float32)\n\n\ndef make_spec(path, filename,N=50):\n    fname = path+'/'+filename\n    sr,y = read_wav(fname)\n    return np.flip(spectrogram(y,sr,N).T,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = []\nfor i in tqdm(wavs):\n    X.append(make_spec(PATH,i))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### That is the entire Test set loaded and converted into a spectrogram in 35 seconds\nThat is roughly $\\dfrac{1}{10}th$ the time it takes for librosa to load in the wav files (without computing the spectrograms)\n\nFirst of all Scipy's read_wav function is much faster than librosa.load(), so that is a great start. As for the spectrogram, I'm more concerned with the customizable pixel to frequency/time ratios, the speed is not really comparable to Librosa's implementation because I only compute FFTs on a fraction of the input audio. "},{"metadata":{"trusted":true},"cell_type":"code","source":"import librosa\ntmp = []\nfor i in tqdm(wavs[:100]):\n    y,sr = librosa.load(PATH+'/'+i)\n    tmp.append(y)\n    \ndel tmp","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Input Spectrograms\nHere is a sample of what I'm sending into the LSTM model"},{"metadata":{"trusted":true},"cell_type":"code","source":"for n in range(3):\n    plt.figure(1)\n    for i in range(5):\n        plt.subplot(151+i)\n        plt.imshow(make_spec(PATH,wavs[i+n*5]))\n        plt.yticks([])\n        plt.xticks([])\n        plt.title(wavs[i+n*5], fontsize=8)\n    plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.95, hspace=0.25,\n                    wspace=0.35)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"LSTM_model = load_model('../input/freesound-keras-lstm-model/Keras LSTM Model')\nLSTM_model.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = np.array(X)\ny_pred = LSTM_model.predict(X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fnames = np.array(wavs).reshape((len(wavs),1))\ndata = np.concatenate((fnames,y_pred),1)\npd.DataFrame(data, columns=list(df)).to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Spoiler\nThe accuracy I was able to get with these spectrograms is 0.299. Though, this is my first attempt at deep learning so you can almost certainly do much better. To conlude, if you want to use a similar spectrogram generator I would advise either; randomizing the location of the spectrogram window at each training epoch, or finding a more reliable method for determining the location of the window (more reliable than __the loudest one possible__ that is). But as it stands, it is a pretty convinient function. The entire dataset (Curated, Noisy, Train) takes up about 1.7 Gb of memory using the dimentional parameters (`N` and `window_length`) explained at the top of this notebook"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}