{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"**This notebook is meant to upload and visually explore the different classes of sounds in the FreeSound dataset**\n\nAudio samples (i.e. a sound) can be very different from each other, even if they are said to be the same class of sound (e.g. a drum). But can we get the sounds, in a given audio class, that are the most representative or most common in an audio class? Of course sounds won't be exactly the same, but there are sounds more like each other than the rest. \n\nThis notebook explores the audio data by visualizing the frame rate of different sounds in the dataset, like trumpet, bird chirps, and drums. Then, the most representative sounds from each audio class. \n\nBy visualizing the frame rate of different sounds, we can get a sense how the audio samples and the audio classes are distinct from each other. In other words, we can visually inspect certain features between samples of the audio classes. If the audio classes are distinct, we have confidence in classifying them. If they are not very distinct, then we have to do a sophisticated approach to classify. "},{"metadata":{"_cell_guid":"0d0f7437-e5d0-495f-86c6-015efe3cd0ad","_uuid":"757ae10808f94831c563cb787bad8e5a401ab8b2"},"cell_type":"markdown","source":"load libraries"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":2,"outputs":[]},{"metadata":{"_cell_guid":"ef63615c-3e8a-40f8-af3a-4542dcafccc0","_uuid":"8a2174b78e8a08bfed15546d0aea7d18a1aca8b5"},"cell_type":"markdown","source":"upload data"},{"metadata":{"_cell_guid":"1b6aefdc-d54c-4988-bb0a-b59ac28c9fb8","_uuid":"554f591eba992d256a027db4fe0c72b9251c2625","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\nprint(\"Number of training examples=\", train.shape[0], \"  Number of classes=\", len(train.label.unique()))\ntest = pd.read_csv(\"../input/sample_submission.csv\")","execution_count":3,"outputs":[]},{"metadata":{"_cell_guid":"a1901e14-e690-45f0-a029-840ac6180f74","_uuid":"fe7091f91b963aaf7fc76b12e4bf349105afda26","trusted":true},"cell_type":"code","source":"_,ax=plt.subplots(figsize=(7,7))\ndata = train.groupby(['label']).size().sort_values()\np = data.plot(kind='barh',ax=ax)\np = ax.set_xlabel(\"Count\")\np = ax.set_ylabel(\"Audio class\")","execution_count":4,"outputs":[]},{"metadata":{"_cell_guid":"75fbb00e-0d6c-43b1-84a8-d474005b2c40","_uuid":"3bb8c02cf5db0b7bdab0c8929427d9bf275ce4c2"},"cell_type":"markdown","source":"The audio classes are non-uniformly distributed, for more than majority of the classes."},{"metadata":{"_cell_guid":"8efe601b-40bb-4426-becb-f525f75b051e","_uuid":"56c87f5dae84ba6bcc83404b0fa10cafed7d18eb"},"cell_type":"markdown","source":"example sound"},{"metadata":{"_cell_guid":"9102e16d-8901-4949-a038-90acbf4b8046","_uuid":"c38fb5ef50f2810f855c31369e80e53964d28ab8","trusted":true},"cell_type":"code","source":"import IPython.display as ipd  # To play sound in the notebook\nfname = '../input/audio_train/' + '00044347.wav'   # Hi-hat\nipd.Audio(fname)","execution_count":5,"outputs":[]},{"metadata":{"_cell_guid":"7136c29a-27c7-44e9-858e-4b66c4630562","_uuid":"a6fe079fe99f45dae18117bcf842f8908a424e3e"},"cell_type":"markdown","source":"I don't care to listen to the individual sounds but I care about seeing the sounds for the whole class. I'd like to get a representative picture for each class of sounds. Hearing that average class sound will mean nothing to us...but if we can see that there might be distinctive properties between the different classes, that might help us to decide on a classifier.\n\nNow, I need to be able to correctly grab the audio files of particular classes. I first do this for one example class, and then extrapolate to all the other classes. I'm sorting the file names by the order found in the train csv file. This is useful so that I can quickly grab files by their class"},{"metadata":{"_cell_guid":"726fafa2-e060-4961-9a4c-ad6e69c0da70","_uuid":"f98238b29ba1793a7cbf1e6d40045bc846c3e760","collapsed":true,"trusted":true},"cell_type":"code","source":"import os","execution_count":6,"outputs":[]},{"metadata":{"_cell_guid":"fadcb663-07cb-4ff5-8909-b3a8d75c4c6c","_uuid":"9c4f30ca1e3ed0637d90c1938890b8b0cd459baf","collapsed":true,"trusted":true},"cell_type":"code","source":"train_files = os.listdir(\"../input/audio_train\")\ntrain_files_dict = dict(zip(train_files,range(len(train_files))))\nsorted_train = train.iloc[train.fname.map(train_files_dict)]","execution_count":7,"outputs":[]},{"metadata":{"_cell_guid":"77fbc82d-51a6-4fc5-923e-61f005b1db1c","_uuid":"29d06dedddf5ea43f0583d1e34bd2b0f55cd10bf","collapsed":true,"trusted":true},"cell_type":"code","source":"import ipywidgets as widget","execution_count":8,"outputs":[]},{"metadata":{"_cell_guid":"0a850e38-e77e-4780-af79-fc5a9bc383c2","_uuid":"a4ff4bfa2fda55a808a8262393eb088e5c0a84b9","collapsed":true,"trusted":true},"cell_type":"code","source":"labels = train['label'].unique()\nw = widget.Dropdown(options = labels, value=labels[0])","execution_count":9,"outputs":[]},{"metadata":{"_cell_guid":"e758aca7-a691-4899-80cc-6d8bd9e56a4c","_uuid":"475c97d4eaff013d8b2f59d35fea18b69dfde03f","trusted":true},"cell_type":"code","source":"w","execution_count":10,"outputs":[]},{"metadata":{"_cell_guid":"8d7cb01c-20c7-4d7c-9a20-98dd9ab33949","_uuid":"3eb8a476812e7907b388110f9850b5da61802e3e","collapsed":true,"trusted":true},"cell_type":"code","source":"train_files_inds = train['label'] == w.label","execution_count":11,"outputs":[]},{"metadata":{"_cell_guid":"b7ea56d3-6a83-4778-99be-b017d1ce924e","_uuid":"71ce2e5bcb69189b5c7624e07ffa90776cb05eb4"},"cell_type":"markdown","source":"These are the files associated to this sound class"},{"metadata":{"_cell_guid":"d0abce85-e31f-47ec-bb1c-fa936c531b13","_uuid":"cc8acd4a7b441a608333c4cefc09c8afcffa5d16","trusted":true},"cell_type":"code","source":"train['fname'][train_files_inds].head(5)","execution_count":12,"outputs":[]},{"metadata":{"_cell_guid":"e629978f-51b1-45bc-aef9-4ec274ee72ef","_uuid":"d3a5febf76912dfe322f50c574d2c64bb6f8e080"},"cell_type":"markdown","source":"Now I want to load one of these and show as a spectrogram"},{"metadata":{"_cell_guid":"786bd231-85aa-4863-876a-8d2863a3b2d5","_uuid":"567c7576adc9d10f69b99cbe81846fffcf9145fd","collapsed":true,"trusted":true},"cell_type":"code","source":"from scipy.io import wavfile\npath = \"../input/audio_train/\"\nfname = train['fname'][train_files_inds].values[0]\nrate, data = wavfile.read(path + fname)","execution_count":13,"outputs":[]},{"metadata":{"_cell_guid":"1f85a30b-55f4-49d8-89db-130f13857bf0","_uuid":"948f490952f8d5320bc7cfbfcdac12a6cc95e2eb","trusted":true},"cell_type":"code","source":"plt.plot(data, '-', );","execution_count":14,"outputs":[]},{"metadata":{"_cell_guid":"dd80871a-1a4e-4f66-b751-2cc139112ce1","_uuid":"cca3232b61154ce1a4934f7ccf2b7d4778f5afd0"},"cell_type":"markdown","source":"Now for n files of the classes.  I'm actually plotting for only 5 classes because it is a lot to plot"},{"metadata":{"_cell_guid":"963e527e-5dbc-4a7f-a7d8-f63a455f8576","_uuid":"f700dd1d0fdab3f25c3b1840e71dd65f52b80180","collapsed":true,"trusted":true},"cell_type":"code","source":"seed = 2\nnp.random.seed(seed)","execution_count":15,"outputs":[]},{"metadata":{"_cell_guid":"cc20d31c-8fad-4068-ad4a-fce12e3b992c","_uuid":"0c0ebc8a730dd8164b3512d582899e250379f326","scrolled":false,"trusted":true},"cell_type":"code","source":"show_df = train.query('manually_verified == 1').sort_values('label')\nlabels = show_df['label'].unique()\n\nfor label in labels[:5]:\n    \n    train_files_inds = show_df['label'] == label\n\n    rand_inds = np.random.randint(0,show_df['fname'][train_files_inds].count(),5)\n    fnames = show_df['fname'].iloc[rand_inds]\n\n    _, axs = plt.subplots(figsize=(17,4),nrows=1,ncols=5)\n\n    for i,fname in enumerate(fnames):\n        rate, data = wavfile.read(path + fname)\n        axs[i].plot(data, '-', label=fname)\n        axs[i].legend()\n    plt.suptitle(label,x=0.04,y=0.5,horizontalalignment='center', fontsize=20)\n    del rate\n    del data\ndel axs","execution_count":16,"outputs":[]},{"metadata":{"_cell_guid":"1e356b29-b7a6-4563-a987-90cdfd1492f1","_uuid":"b5dd67661bff6eae2e70a0bbc142bdf258d56d54"},"cell_type":"markdown","source":"Awesome! Now I can see just 5 random audio frames for each class (manually verified only). \n\nBut this kernel is for getting the average representative samples for each class. \n\nSo how do I get the representative samples? I get the median (I bet the distribution is skewed) of the samples for each class and I pick the samples that are closest to this value. \n\nWhat do I mean by median of the class? \n\nAn audio file is a sample. Each sample has 44100 frames per second of audio. Each of those frames has a different magnitude representing the \"loudness\". The higher the magnitude of the frame the higher the pitch. \n\nWithin a class, a sample may have different numbers of frames. In order to get the most representative or median sample, I need to take the median at each corresponding frame. \n\nLet's say I have a class with 5 samples with the same size-all just 5 frames each, where at each frame in a sample there is a certain magnitude. "},{"metadata":{"_cell_guid":"7ee93682-95c3-47cf-b779-aa822d8b9b83","_uuid":"76b427fef1afc0daeaa22c12f3329752b6e436ff","trusted":true},"cell_type":"code","source":"arrs = []\nlens = [5]*5\nfor i in lens:\n    arrs.append(np.random.randint(7,size=i))\narrs = np.array(arrs)\narrs","execution_count":17,"outputs":[]},{"metadata":{"_cell_guid":"be8b9bba-59c8-4b55-a4c4-824abc49a181","_uuid":"773b148a41cb8706edd5a3b187cad27c32ec9b1e"},"cell_type":"markdown","source":"Say the above are my five samples from an audio class. What's the median? "},{"metadata":{"_cell_guid":"782fb80d-6111-49fd-bf26-04f1986e1c52","_uuid":"13a83764cd482ac141144879a9dcb53faeddbbe2","trusted":true},"cell_type":"code","source":"np.median(arrs,axis=1)","execution_count":18,"outputs":[]},{"metadata":{"_cell_guid":"1022b535-5fea-44aa-8414-8e39324aadbd","_uuid":"3bee34171352934791736db89c02dd0e6e99fcbe"},"cell_type":"markdown","source":"That was easy! But what if the samples are of different lengths,"},{"metadata":{"_cell_guid":"73d96e11-caac-4beb-9ba3-ea0c6d48d40f","_uuid":"2b7d12a49ca9ab3734d270b2f7b7730f660a1298","trusted":true},"cell_type":"code","source":"arrs = []\nlens = [5,10,7,6,9]\nfor i in lens:\n    arrs.append(np.random.randint(7,size=i))\narrs = np.array(arrs)\narrs","execution_count":19,"outputs":[]},{"metadata":{"_cell_guid":"da0d7aeb-fe14-4e9c-ad1b-d1b6d9644d0e","_uuid":"3a6f0ff0d1cad68a5f85e256c0e92e1c216f55eb"},"cell_type":"markdown","source":"Taking the median doesn't work because it doesn't broadcast. I need to find the longest array, make an array of 0s of that length, and pad the other arrays"},{"metadata":{"_cell_guid":"d2ad28a0-823b-4952-98be-746c4271d6d2","_uuid":"5ff4ae9b40a15d2a765a2288205efd6a8830d827","trusted":true},"cell_type":"code","source":"maxarrn = np.max([len(i) for i in arrs])\npadded_arrs = [np.pad(arr,(0,maxarrn-len(arr)),mode='constant')for arr in arrs]\npadded_arrs = np.array(padded_arrs)\ndisplay(padded_arrs)\n\nmed_padded_arrs = np.median(padded_arrs,axis=0)\nmed_padded_arrs","execution_count":20,"outputs":[]},{"metadata":{"_cell_guid":"e893aed0-ea4c-4b62-93e1-8756a4b91627","_uuid":"1abdd301fa7797e697d0cfa66ae4a452421cd66c"},"cell_type":"markdown","source":"Groovy! But I want to choose actual samples that are close to the median array. This means I have to compute a distance. I use _np.linalg.norm_ to get the vector norm, which is a measure of the distance between two arrays. So I compute the distances of each sample to the median"},{"metadata":{"_cell_guid":"29e5114b-8e47-457f-8c7a-b0fc4b3392e7","_uuid":"1425b5b8f046e223ed4ca98897485b28580a7e3a","trusted":true},"cell_type":"code","source":"dists = [np.linalg.norm(np.pad(arr,(0,len(med_padded_arrs)-len(arr)),mode='constant')-med_padded_arrs) for arr in arrs]\ndists = np.array(dists)\ndists","execution_count":21,"outputs":[]},{"metadata":{"_cell_guid":"5a6b6791-e59d-4095-bcf2-7d6435ca9142","_uuid":"64cdbf1ce52533d9b1c828596b5b2b1e6cab7392"},"cell_type":"markdown","source":"Then I choose which sample I want by ordering so I can get those samples with the smallest distance"},{"metadata":{"_cell_guid":"2002d7f5-fd88-4bee-9b93-ebf99291e3bc","_uuid":"976aaac44cde441be840d54cdf6dd31c64c523ac","trusted":true},"cell_type":"code","source":"dists.argsort()[::-1]","execution_count":22,"outputs":[]},{"metadata":{"_cell_guid":"a3127011-f735-4dc3-8942-e758a02de2ae","_uuid":"5118b2927b424241b1c4c7ffc4f51a74a4d74ab4"},"cell_type":"markdown","source":"So I would like at the samples from the first indices in the above because they are closest to the median. Now what if I do this for a whole audio class"},{"metadata":{"_cell_guid":"5f6ca856-5b37-4776-9148-4ddf08839c91","_uuid":"ee54dac6840a0ca4a7e43f0d603fc8afb2365dbd","trusted":true},"cell_type":"code","source":"label = 'Acoustic_guitar'\nsub = show_df[show_df.label==label]\nfnames = sub.fname.values\narrs = []\nfor i,fname in enumerate(fnames):\n        rate, data = wavfile.read(path + fname)\n        arrs.append(data)\nmaxarrn = np.max([len(i) for i in arrs])\npadded_arrs = [np.pad(arr,(0,maxarrn-len(arr)),mode='constant')for arr in arrs]\npadded_arrs = np.array(padded_arrs)\nmed_padded_arrs = np.median(padded_arrs,axis=0)\ndists = [np.linalg.norm(np.pad(arr,(0,len(med_padded_arrs)-len(arr)),mode='constant')-med_padded_arrs) for arr in arrs]\ndists = np.array(dists)\nfnames = sub.iloc[dists.argsort()[::-1]].head(5).fname.values\n_, axs = plt.subplots(figsize=(17,4),nrows=1,ncols=5)\nfor i,fname in enumerate(fnames):\n    rate, data = wavfile.read(path + fname)\n    axs[i].plot(data, '-', label=fname)\n    axs[i].legend()\nplt.suptitle(label,x=0.04,y=0.5,horizontalalignment='center', fontsize=20)","execution_count":23,"outputs":[]},{"metadata":{"_cell_guid":"74fc4cb7-07e4-460b-bcd4-a011752173c9","_uuid":"2766f2edc6e4fb13625c493da6337d0c7a49b0cc"},"cell_type":"markdown","source":"The above are the 5 most representative (closest tov the median) for Acoustic_guitar. Now I can do this for all classes"},{"metadata":{"_cell_guid":"707ea143-8632-4b61-ade8-c3efef218a58","_uuid":"80dd54b81d0a4bec37be01d68a5fc10fd750899e","scrolled":false,"trusted":true},"cell_type":"code","source":"show_df = train.query('manually_verified == 1').sort_values('label')\nlabels = show_df['label'].unique()\n\nfor label in labels:\n    sub = show_df[show_df.label==label]\n    fnames = sub.fname.values\n    arrs = []\n    for i,fname in enumerate(fnames):\n            rate, data = wavfile.read(path + fname)\n            arrs.append(data)\n    maxarrn = np.max([len(i) for i in arrs])\n    padded_arrs = [np.pad(arr,(0,maxarrn-len(arr)),mode='constant')for arr in arrs]\n    padded_arrs = np.array(padded_arrs)\n    med_padded_arrs = np.median(padded_arrs,axis=0)\n    dists = [np.linalg.norm(np.pad(arr,(0,len(med_padded_arrs)-len(arr)),mode='constant')-med_padded_arrs) for arr in arrs]\n    dists = np.array(dists)\n    fnames = sub.iloc[dists.argsort()[::-1]].head(5).fname.values\n    _, axs = plt.subplots(figsize=(17,4),nrows=1,ncols=5)\n    for i,fname in enumerate(fnames):\n        rate, data = wavfile.read(path + fname)\n        axs[i].plot(data, '-', label=fname)\n        axs[i].legend()\n    plt.suptitle(label,x=0.04,y=0.5,horizontalalignment='center', fontsize=20)\ndel axs","execution_count":24,"outputs":[]},{"metadata":{"_cell_guid":"e4111702-0934-4070-b839-94d39f6612dc","_uuid":"cfcbc499b150c249693c6cb9eb00abda6780ee71","collapsed":true},"cell_type":"markdown","source":"There will be a follow up notebook exploring another aspect of the data by visualizing spectrograms."},{"metadata":{"_cell_guid":"35609cbd-e8d8-4ee5-b639-0148f7934a87","_uuid":"74ffcde4397ee2f43fd8f521c25cc53b5c663f2f","trusted":true},"cell_type":"code","source":"fname = '../input/audio_train/' + '991fa1d7.wav'   # Hi-hat\nipd.Audio(fname)","execution_count":25,"outputs":[]},{"metadata":{"_cell_guid":"524188a0-de41-4c23-98b3-5222494a1a0f","_uuid":"93ad1aeb26cf7faab7abdce659a4fa15e154025d","collapsed":true,"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}