{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import pkg_resources\n\ndef placeholder(x):\n raise pkg_resources.DistributionNotFound\npkg_resources.get_distribution = placeholder\n\n!pip uninstall fastai fastcore torchaudio -y\n#!pip install torch==1.8.1 torchaudio==0.8.1 fastcore==1.3.20\n!pip install torch==1.9.0 torchaudio==0.9.0\n!pip install fastaudio","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-24T15:31:15.937028Z","iopub.execute_input":"2022-02-24T15:31:15.937315Z","iopub.status.idle":"2022-02-24T15:32:42.171731Z","shell.execute_reply.started":"2022-02-24T15:31:15.937236Z","shell.execute_reply":"2022-02-24T15:32:42.170497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n import pycaret\nexcept:\n !pip install pycaret-nightly","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-24T15:32:42.173319Z","iopub.execute_input":"2022-02-24T15:32:42.173547Z","iopub.status.idle":"2022-02-24T15:32:42.182698Z","shell.execute_reply.started":"2022-02-24T15:32:42.173518Z","shell.execute_reply":"2022-02-24T15:32:42.182104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n# Introduction\n\n\n\nPicture Credit: https://i1.rgstatic.net\n\nWhen I first encountered this problem, I had many thoughts. It was hard to figure out where to start. The more I read and tried to understand the problem, the less confident I became. I am neither an ornithologist nor an aucostic engineer. In short, domain knowledge is lacking. \n\n**Shut up and Calculate!**\n\nThis is a famous saying in quantum mechanics. While I was thinking about various ways to solve the problem, this saying suddenly came to my mind.\nRather than giving up because you don't have domain knowledge, let's stop thinking for a moment and start.","metadata":{}},{"cell_type":"markdown","source":"**I would like to organize this notebook in the following order.**\n> 1. Understand each dataset. Check which features are important features using a simple model.\n> 2. Understand audio data.\n> 3. Transform audio data and design dataloader to process them.\n> 4. Design a model and train it by introducing various methods to improve performance.","metadata":{}},{"cell_type":"code","source":"import os\nimport librosa\nfrom tqdm import tqdm\n\nimport pandas as pd\nfrom fastaudio.all import *\nfrom fastai.vision.all import *\n\nimport torch\nimport torchaudio\nimport fastcore\nimport fastai\nimport fastaudio\nimport torchaudio\ntorchaudio.set_audio_backend(\"sox_io\")\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib import rcParams\nimport plotly.express as px\n\nfrom sklearn.preprocessing import LabelEncoder\n\nimport warnings\nwarnings.filterwarnings(action='ignore')","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-24T15:32:42.184052Z","iopub.execute_input":"2022-02-24T15:32:42.184293Z","iopub.status.idle":"2022-02-24T15:32:46.007108Z","shell.execute_reply.started":"2022-02-24T15:32:42.184264Z","shell.execute_reply":"2022-02-24T15:32:46.006247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set(style=\"ticks\", context=\"talk\",font_scale = 1.2)\nplt.style.use(\"seaborn-paper\")\nplt.subplots_adjust(wspace=1)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.008967Z","iopub.execute_input":"2022-02-24T15:32:46.009213Z","iopub.status.idle":"2022-02-24T15:32:46.025357Z","shell.execute_reply.started":"2022-02-24T15:32:46.009182Z","shell.execute_reply":"2022-02-24T15:32:46.024534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"------------------------------------------------------\n# EDA","metadata":{}},{"cell_type":"code","source":"# CONFIGURATIONS\nDATA_DIR = Path('../input/birdclef-2022/train_audio')","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.026487Z","iopub.execute_input":"2022-02-24T15:32:46.027079Z","iopub.status.idle":"2022-02-24T15:32:46.031386Z","shell.execute_reply.started":"2022-02-24T15:32:46.027048Z","shell.execute_reply":"2022-02-24T15:32:46.030852Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"audio_fns = get_audio_files(DATA_DIR)\nprint(f'No. of audio files: {len(audio_fns)}')\n\n# to save time, I subset training data\nmeta_df = pd.read_csv('../input/birdclef-2022/train_metadata.csv')\ntest_df = pd.read_csv('../input/birdclef-2022/test.csv')","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.032482Z","iopub.execute_input":"2022-02-24T15:32:46.032953Z","iopub.status.idle":"2022-02-24T15:32:46.481717Z","shell.execute_reply.started":"2022-02-24T15:32:46.032924Z","shell.execute_reply":"2022-02-24T15:32:46.481076Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"--------------------------------------------\n# Analyzing Meta Data","metadata":{}},{"cell_type":"code","source":"meta_df.head().T.style.set_properties(**{'background-color': 'black',\n 'color': 'white',\n 'border-color': 'white'})","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.482848Z","iopub.execute_input":"2022-02-24T15:32:46.483386Z","iopub.status.idle":"2022-02-24T15:32:46.549901Z","shell.execute_reply.started":"2022-02-24T15:32:46.483352Z","shell.execute_reply":"2022-02-24T15:32:46.549176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"meta_df.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.551212Z","iopub.execute_input":"2022-02-24T15:32:46.551412Z","iopub.status.idle":"2022-02-24T15:32:46.578307Z","shell.execute_reply.started":"2022-02-24T15:32:46.551387Z","shell.execute_reply":"2022-02-24T15:32:46.577344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df.head().T.style.set_properties(**{'background-color': 'black',\n 'color': 'white',\n 'border-color': 'white'})","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.579510Z","iopub.execute_input":"2022-02-24T15:32:46.579755Z","iopub.status.idle":"2022-02-24T15:32:46.595140Z","shell.execute_reply.started":"2022-02-24T15:32:46.579725Z","shell.execute_reply":"2022-02-24T15:32:46.594451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = meta_df.drop(['url','filename','scientific_name','license','time','common_name','secondary_labels']\n ,axis=1\n ,errors='ignore')\ntrain_df = train_df.sample(1000)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.596238Z","iopub.execute_input":"2022-02-24T15:32:46.596605Z","iopub.status.idle":"2022-02-24T15:32:46.613793Z","shell.execute_reply.started":"2022-02-24T15:32:46.596577Z","shell.execute_reply":"2022-02-24T15:32:46.613097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"enc_list = ['primary_label','author','type']\nfor feature in enc_list:\n le = LabelEncoder()\n le = le.fit(train_df[feature])\n train_df[feature] = le.transform(train_df[feature])","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.615390Z","iopub.execute_input":"2022-02-24T15:32:46.615690Z","iopub.status.idle":"2022-02-24T15:32:46.624444Z","shell.execute_reply.started":"2022-02-24T15:32:46.615647Z","shell.execute_reply":"2022-02-24T15:32:46.623798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"------------------------------------------------------------------------\n# Understaning Metadata from the simple model","metadata":{}},{"cell_type":"code","source":"from pycaret.classification import *","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:46.625686Z","iopub.execute_input":"2022-02-24T15:32:46.626106Z","iopub.status.idle":"2022-02-24T15:32:46.892822Z","shell.execute_reply.started":"2022-02-24T15:32:46.626074Z","shell.execute_reply":"2022-02-24T15:32:46.891953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"setup(data = train_df, \n target = 'primary_label',\n preprocess = True,\n silent = True\n )","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-24T15:32:46.897274Z","iopub.execute_input":"2022-02-24T15:32:46.897517Z","iopub.status.idle":"2022-02-24T15:32:47.955571Z","shell.execute_reply.started":"2022-02-24T15:32:46.897488Z","shell.execute_reply":"2022-02-24T15:32:47.954662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dt = create_model('dt',verbose = False)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-02-24T15:32:47.956966Z","iopub.execute_input":"2022-02-24T15:32:47.957292Z","iopub.status.idle":"2022-02-24T15:32:52.707539Z","shell.execute_reply.started":"2022-02-24T15:32:47.957262Z","shell.execute_reply":"2022-02-24T15:32:52.706532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_model(dt, plot='feature')","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:52.708922Z","iopub.execute_input":"2022-02-24T15:32:52.709198Z","iopub.status.idle":"2022-02-24T15:32:53.177020Z","shell.execute_reply.started":"2022-02-24T15:32:52.709160Z","shell.execute_reply":"2022-02-24T15:32:53.176340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* latitue and longitude are important features.\n* rating is of relatively low importance.\n \nI still don't know if this information is helpful for learning. If it is helpful, I still don't know how to use it for learning.","metadata":{}},{"cell_type":"code","source":"fig = px.scatter_geo(\n meta_df,\n lat=\"latitude\",\n lon=\"longitude\",\n color=\"primary_label\",\n width=1000,\n height=500,\n title=\"Bird Distribution\",\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:53.178082Z","iopub.execute_input":"2022-02-24T15:32:53.178946Z","iopub.status.idle":"2022-02-24T15:32:54.416447Z","shell.execute_reply.started":"2022-02-24T15:32:53.178906Z","shell.execute_reply":"2022-02-24T15:32:54.415508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"------------------------------------\n# Understanding Train Dataset\n\nTrain data are audio files. Let's hear it for ourselves and see the waveform to see what strategy can solve this problem.","metadata":{}},{"cell_type":"code","source":"def show_bird(audios):\n for fn in audios:\n audio = AudioTensor.create(fn)\n audio.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:54.417892Z","iopub.execute_input":"2022-02-24T15:32:54.420286Z","iopub.status.idle":"2022-02-24T15:32:54.424913Z","shell.execute_reply.started":"2022-02-24T15:32:54.420240Z","shell.execute_reply":"2022-02-24T15:32:54.424022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------\n## Normoc\n\n\n\nRef: https://cdn.download.ams.birds.cornell.edu","metadata":{}},{"cell_type":"code","source":"fig = px.scatter_geo(\n meta_df[meta_df['primary_label'] == 'normoc'],\n lat=\"latitude\",\n lon=\"longitude\",\n color=\"primary_label\",\n width=1000,\n height=500,\n title=\"Bird Distribution\",\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:54.426241Z","iopub.execute_input":"2022-02-24T15:32:54.426444Z","iopub.status.idle":"2022-02-24T15:32:54.487718Z","shell.execute_reply.started":"2022-02-24T15:32:54.426419Z","shell.execute_reply":"2022-02-24T15:32:54.486901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"normoc_fns = get_audio_files( '../input/birdclef-2022/train_audio/normoc')","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:54.489103Z","iopub.execute_input":"2022-02-24T15:32:54.489454Z","iopub.status.idle":"2022-02-24T15:32:54.496149Z","shell.execute_reply.started":"2022-02-24T15:32:54.489417Z","shell.execute_reply":"2022-02-24T15:32:54.495515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_bird(normoc_fns[:3])","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:54.497181Z","iopub.execute_input":"2022-02-24T15:32:54.497506Z","iopub.status.idle":"2022-02-24T15:32:58.726889Z","shell.execute_reply.started":"2022-02-24T15:32:54.497479Z","shell.execute_reply":"2022-02-24T15:32:58.726283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* They are of the same bird species, but sound different.\n* It seems that there are audio files in which noise is heard.\n* Even if you look at the Waveform, there is no similarity.","metadata":{}},{"cell_type":"markdown","source":"--------------------------------\n## Norcar\n\n\n\nRef: https://upload.wikimedia.org","metadata":{}},{"cell_type":"code","source":"fig = px.scatter_geo(\n meta_df[meta_df['primary_label'] == 'norcar'],\n lat=\"latitude\",\n lon=\"longitude\",\n color=\"primary_label\",\n width=1000,\n height=500,\n title=\"Bird Distribution\",\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:58.728616Z","iopub.execute_input":"2022-02-24T15:32:58.729199Z","iopub.status.idle":"2022-02-24T15:32:58.784831Z","shell.execute_reply.started":"2022-02-24T15:32:58.729146Z","shell.execute_reply":"2022-02-24T15:32:58.784014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"norcar_fns = get_audio_files( '../input/birdclef-2022/train_audio/norcar')","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:58.786448Z","iopub.execute_input":"2022-02-24T15:32:58.787028Z","iopub.status.idle":"2022-02-24T15:32:58.796022Z","shell.execute_reply.started":"2022-02-24T15:32:58.786978Z","shell.execute_reply":"2022-02-24T15:32:58.795457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_bird(norcar_fns[:3])","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:32:58.797462Z","iopub.execute_input":"2022-02-24T15:32:58.797664Z","iopub.status.idle":"2022-02-24T15:33:05.995241Z","shell.execute_reply.started":"2022-02-24T15:32:58.797640Z","shell.execute_reply":"2022-02-24T15:33:05.994224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* They are of the same bird species, but sound different.\n* It seems that there are audio files in which noise is heard.\n* Even if you look at the Waveform, there is no similarity.","metadata":{}},{"cell_type":"markdown","source":"----------------------------------------\n## Bcnher\n\n\n\nPicture Credit: https://www.birdingintaiwan.com","metadata":{}},{"cell_type":"code","source":"fig = px.scatter_geo(\n meta_df[meta_df['primary_label'] == 'bcnher'],\n lat=\"latitude\",\n lon=\"longitude\",\n color=\"primary_label\",\n width=1000,\n height=500,\n title=\"Bird Distribution\",\n)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:33:05.996719Z","iopub.execute_input":"2022-02-24T15:33:05.997323Z","iopub.status.idle":"2022-02-24T15:33:06.063999Z","shell.execute_reply.started":"2022-02-24T15:33:05.997274Z","shell.execute_reply":"2022-02-24T15:33:06.063016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bcnher_fns = get_audio_files( '../input/birdclef-2022/train_audio/bcnher')","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:33:06.065545Z","iopub.execute_input":"2022-02-24T15:33:06.065872Z","iopub.status.idle":"2022-02-24T15:33:06.075644Z","shell.execute_reply.started":"2022-02-24T15:33:06.065829Z","shell.execute_reply":"2022-02-24T15:33:06.074779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_bird(bcnher_fns[:3])","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:33:06.077040Z","iopub.execute_input":"2022-02-24T15:33:06.077968Z","iopub.status.idle":"2022-02-24T15:33:09.857180Z","shell.execute_reply.started":"2022-02-24T15:33:06.077921Z","shell.execute_reply":"2022-02-24T15:33:09.856324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* They are of the same bird species, but sound different.\n* It seems that there are audio files in which noise is heard.\n* Even if you look at the Waveform, there is no similarity.","metadata":{}},{"cell_type":"markdown","source":"--------------------------------------------\n# Making fingerprints of audios\n\n\n\nPicture Credit: https://www.researchgate.net\n\nWe try to solve the problem of audio by changing it to an image problem. Regarding Image, there are proven CNN models with high performance, and many methods have been devised to improve the performance.\n\nFortunately, there are APIs related to MFCC provided by fastai, so let's use them to transform audio into image. In other words, we are going to take a method that makes audio fingerprints so that we can classify them with these fingerprints.","metadata":{}},{"cell_type":"markdown","source":"**What is Mel-frequency cepstral coefficients (MFCCs)?**\n\n> In sound processing, the mel-frequency cepstrum (MFC) is a representation of the short-term power spectrum of a sound, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency.\n> \n> Mel-frequency cepstral coefficients (MFCCs) are coefficients that collectively make up an MFC. They are derived from a type of cepstral representation of the audio clip (a nonlinear \"spectrum-of-a-spectrum\"). The difference between the cepstrum and the mel-frequency cepstrum is that in the MFC, the frequency bands are equally spaced on the mel scale, which approximates the human auditory system's response more closely than the linearly-spaced frequency bands used in the normal spectrum. This frequency warping can allow for better representation of sound, for example, in audio compression.\n\nRef: https://en.wikipedia.org/wiki/Mel-frequency_cepstrum","metadata":{}},{"cell_type":"code","source":"aud2mfcc = AudioToMFCC(n_mfcc=40, melkwargs={'n_fft':2048, 'hop_length':256,'n_mels':128})","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:33:09.858762Z","iopub.execute_input":"2022-02-24T15:33:09.859268Z","iopub.status.idle":"2022-02-24T15:33:09.867943Z","shell.execute_reply.started":"2022-02-24T15:33:09.859221Z","shell.execute_reply":"2022-02-24T15:33:09.866791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"--------------------------------------------------------------\n# Making Dataloders\n\n\n\nPicture Credit: https://miro.medium.com","metadata":{}},{"cell_type":"markdown","source":"## Making Pipeline\n\nWe build the pipeline in the following order.\n> Resample -> DownmixMono -> RemoveSilence -> ResizeSignal -> AudioToMFCC -> Delta -> ToTensor","metadata":{}},{"cell_type":"code","source":"item_tfms = [RemoveSilence(),ResizeSignal(1000), aud2mfcc, Delta()]","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:33:09.869606Z","iopub.execute_input":"2022-02-24T15:33:09.870171Z","iopub.status.idle":"2022-02-24T15:33:10.304237Z","shell.execute_reply.started":"2022-02-24T15:33:09.870124Z","shell.execute_reply":"2022-02-24T15:33:10.303435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aud_digit = DataBlock(blocks=(AudioBlock, CategoryBlock), \n get_items=get_audio_files, \n splitter=RandomSplitter(),\n item_tfms = item_tfms,\n get_y=parent_label)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:33:10.305837Z","iopub.execute_input":"2022-02-24T15:33:10.306316Z","iopub.status.idle":"2022-02-24T15:33:10.583975Z","shell.execute_reply.started":"2022-02-24T15:33:10.306264Z","shell.execute_reply":"2022-02-24T15:33:10.583254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"aud_digit.summary(DATA_DIR)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:33:10.585289Z","iopub.execute_input":"2022-02-24T15:33:10.586106Z","iopub.status.idle":"2022-02-24T15:33:13.042787Z","shell.execute_reply.started":"2022-02-24T15:33:10.586046Z","shell.execute_reply":"2022-02-24T15:33:13.042173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls = aud_digit.dataloaders(DATA_DIR, bs=16)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:35:15.693650Z","iopub.execute_input":"2022-02-24T15:35:15.694348Z","iopub.status.idle":"2022-02-24T15:35:16.500443Z","shell.execute_reply.started":"2022-02-24T15:35:15.694301Z","shell.execute_reply":"2022-02-24T15:35:16.499584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls.c","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:35:16.501883Z","iopub.execute_input":"2022-02-24T15:35:16.502150Z","iopub.status.idle":"2022-02-24T15:35:16.508492Z","shell.execute_reply.started":"2022-02-24T15:35:16.502094Z","shell.execute_reply":"2022-02-24T15:35:16.507566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n \n* Target is composed of 152 classes.","metadata":{}},{"cell_type":"markdown","source":"--------------------------------------------------------------------\n## Checking batch","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(10, 8))\ndls.show_batch(max_n=3,figsize=(20,10))","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:35:16.509910Z","iopub.execute_input":"2022-02-24T15:35:16.510191Z","iopub.status.idle":"2022-02-24T15:35:38.737041Z","shell.execute_reply.started":"2022-02-24T15:35:16.510156Z","shell.execute_reply":"2022-02-24T15:35:38.735948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* We converted audio to image. In other words, the above figures can be regarded as fingerprints of each audio file.\n \nThe format of the question has been changed. Looking at the picture above, it can be seen that the problem of audio classification has been changed to the problem of image classification.\n \nQuantum mechanics cannot explain the behavior of quantum clearly, but as it can be solved mathematically, rather than understanding the above figures accurately, we will only think about how the model can understand and learn the above picture.","metadata":{}},{"cell_type":"markdown","source":"------------------------------------------------------\n# Modeling","metadata":{}},{"cell_type":"code","source":"def audio_learner(dls, arch, loss_func, metrics):\n \"Prepares a `Learner` for audio processing\"\n learn = Learner(dls, arch, loss_func, metrics=metrics, \n cbs = [EarlyStoppingCallback(monitor='accuracy', patience=5),ActivationStats(with_hist=True)]).to_fp16()\n n_c = dls.one_batch()[0].shape[1]\n if n_c == 1: alter_learner(learn)\n return learn","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:35:38.738931Z","iopub.execute_input":"2022-02-24T15:35:38.739209Z","iopub.status.idle":"2022-02-24T15:35:38.744420Z","shell.execute_reply.started":"2022-02-24T15:35:38.739173Z","shell.execute_reply":"2022-02-24T15:35:38.743578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = audio_learner(dls, \n xresnet18(), \n LabelSmoothingCrossEntropy(), \n accuracy)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:35:38.745777Z","iopub.execute_input":"2022-02-24T15:35:38.746480Z","iopub.status.idle":"2022-02-24T15:36:03.079893Z","shell.execute_reply.started":"2022-02-24T15:35:38.746434Z","shell.execute_reply":"2022-02-24T15:36:03.078927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"--------------------------------------------\n# Finding the proper learning rate\n\n\n\nPicture Credit: https://149695847.v2.pressablecdn.com\n\n> In machine learning and statistics, the learning rate is a tuning parameter in an optimization algorithm that determines the step size at each iteration while moving toward a minimum of a loss function. Since it influences to what extent newly acquired information overrides old information, it metaphorically represents the speed at which a machine learning model \"learns\". In the adaptive control literature, the learning rate is commonly referred to as gain.\n> \n> In setting a learning rate, there is a trade-off between the rate of convergence and overshooting. While the descent direction is usually determined from the gradient of the loss function, the learning rate determines how big a step is taken in that direction. A too high learning rate will make the learning jump over minima but a too low learning rate will either take too long to converge or get stuck in an undesirable local minimum.\n\nRef: https://en.wikipedia.org/wiki/Learning_rate\n\nLearning rate is one of the important parameters among hyperparameters. However, choosing a learning rate is not an easy task with many considerations.\nFastai finds the learning rate so that an appropriate learning rate can be determined.","metadata":{}},{"cell_type":"code","source":"sr = learn.lr_find()\nsr","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:36:03.081370Z","iopub.execute_input":"2022-02-24T15:36:03.081687Z","iopub.status.idle":"2022-02-24T15:42:43.709188Z","shell.execute_reply.started":"2022-02-24T15:36:03.081657Z","shell.execute_reply":"2022-02-24T15:42:43.705149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"------------------------------------------------\n# Training","metadata":{}},{"cell_type":"code","source":"learn.fit_one_cycle(10, sr.lr_steep)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:42:43.712897Z","iopub.status.idle":"2022-02-24T15:42:43.714539Z","shell.execute_reply.started":"2022-02-24T15:42:43.714214Z","shell.execute_reply":"2022-02-24T15:42:43.714263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-------------------------------------------\n# Interpreting Model","metadata":{}},{"cell_type":"code","source":"def plot_layer_stats(self, idx):\n plt,axs = subplots(1, 3, figsize=(15,3))\n plt.subplots_adjust(wspace=0.5)\n for o,ax,title in zip(self.layer_stats(idx),axs,('mean','std','% near zero')):\n ax.plot(o)\n ax.set_title(title)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:34:31.736839Z","iopub.status.idle":"2022-02-24T15:34:31.737505Z","shell.execute_reply.started":"2022-02-24T15:34:31.737250Z","shell.execute_reply":"2022-02-24T15:34:31.737278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_layer_stats(learn.activation_stats,-2)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:34:31.742704Z","iopub.status.idle":"2022-02-24T15:34:31.744936Z","shell.execute_reply.started":"2022-02-24T15:34:31.744660Z","shell.execute_reply":"2022-02-24T15:34:31.744695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_layer_stats(learn.activation_stats,-1)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:34:31.748675Z","iopub.status.idle":"2022-02-24T15:34:31.749395Z","shell.execute_reply.started":"2022-02-24T15:34:31.749155Z","shell.execute_reply":"2022-02-24T15:34:31.749182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* The activation distribution is well distributed.","metadata":{}},{"cell_type":"markdown","source":"-------------------------------------------\n# Checking Underfitting and Overfitting\n\n\n\nPicture Credit: https://vitalflux.com/wp-content","metadata":{}},{"cell_type":"code","source":"learn.recorder.plot_loss()","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:34:31.754394Z","iopub.status.idle":"2022-02-24T15:34:31.754803Z","shell.execute_reply.started":"2022-02-24T15:34:31.754600Z","shell.execute_reply":"2022-02-24T15:34:31.754622Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* It seems that learning has ended before overffing. However, I still don't know if it is the optimal point. It is burdensome to increase the epoch because the training time is too long.","metadata":{}},{"cell_type":"markdown","source":"----------------------------------------------\n# Checking Results","metadata":{}},{"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_confusion_matrix(figsize=(30,30), dpi=240)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:34:31.764389Z","iopub.status.idle":"2022-02-24T15:34:31.764925Z","shell.execute_reply.started":"2022-02-24T15:34:31.764622Z","shell.execute_reply":"2022-02-24T15:34:31.764646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* Because there are many classes of Target, it is not good to look at.\n* If you look at the diagonal of the matrix, it seems that it has learned to some extent. However, it is not satisfactory.","metadata":{}},{"cell_type":"markdown","source":"**Let's check the cases where the model made the most mistakes!**","metadata":{}},{"cell_type":"code","source":"interp.most_confused(min_val=10)","metadata":{"execution":{"iopub.status.busy":"2022-02-24T15:34:31.766311Z","iopub.status.idle":"2022-02-24T15:34:31.766773Z","shell.execute_reply.started":"2022-02-24T15:34:31.766554Z","shell.execute_reply":"2022-02-24T15:34:31.766583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" Observation:\n* In the combination of 'bcnher' and 'brnowl', there are many cases where the judgment is exceptionally wrong. I guess I'll have to check more on why. ","metadata":{}},{"cell_type":"markdown","source":"
\n\n# Conclusion\n\nThe accuracy confirmed by the validation dataset is not satisfactory.\nIt seems that you need to think about ways to improve performance while reading various materials again.\n\nThank you for reading! \n\n**I will be back!**","metadata":{}},{"cell_type":"markdown","source":"\n\n
","metadata":{}}]}