{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## What is this? 🤔\nExperiment to weave nnaudio, timm and fastai together\n\n## What I have tried to explore here? 🔍\n👉 Extend Fastai for signal processing and time series: \n\n    * To use nnAudio for faster processing than librosa or other signal/audio processing methods.\n    * To deal with time series data as images.\n    * To deal with cosmological data like gravitational waves.1\n\n👉 Create custom Transform.\n\n👉 Create custom block.\n\n👉 Create a dataloader.\n\n👉 Create a custom model with models from the timm library.\n\n👉 Create custom learner.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:00:52.319446Z","iopub.execute_input":"2021-10-01T09:00:52.319765Z","iopub.status.idle":"2021-10-01T09:00:52.451900Z","shell.execute_reply.started":"2021-10-01T09:00:52.319686Z","shell.execute_reply":"2021-10-01T09:00:52.451210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Installing all the required libraries⚙️\n👉 Spacy\n\n👉 Fastai\n\n👉 nnAudio\n\n👉 timm","metadata":{}},{"cell_type":"code","source":"#!pip install spacy==3.1.1","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:00:52.453546Z","iopub.execute_input":"2021-10-01T09:00:52.453800Z","iopub.status.idle":"2021-10-01T09:00:52.481253Z","shell.execute_reply.started":"2021-10-01T09:00:52.453769Z","shell.execute_reply":"2021-10-01T09:00:52.480443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!pip3 install torch==1.9.0 torchvision==0.10.0\n!pip3 install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:00:52.482485Z","iopub.execute_input":"2021-10-01T09:00:52.483110Z","iopub.status.idle":"2021-10-01T09:04:05.906868Z","shell.execute_reply.started":"2021-10-01T09:00:52.483065Z","shell.execute_reply":"2021-10-01T09:04:05.906080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#!yes Y|conda install -c fastai fastai=2.5.2\n!pip3 install fastai==2.5.2","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-10-01T09:04:05.909025Z","iopub.execute_input":"2021-10-01T09:04:05.909295Z","iopub.status.idle":"2021-10-01T09:04:15.008773Z","shell.execute_reply.started":"2021-10-01T09:04:05.909260Z","shell.execute_reply":"2021-10-01T09:04:15.007784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip3 install timm","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:15.012307Z","iopub.execute_input":"2021-10-01T09:04:15.012531Z","iopub.status.idle":"2021-10-01T09:04:22.809828Z","shell.execute_reply.started":"2021-10-01T09:04:15.012505Z","shell.execute_reply":"2021-10-01T09:04:22.808885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip3 install nnaudio","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:22.811624Z","iopub.execute_input":"2021-10-01T09:04:22.811917Z","iopub.status.idle":"2021-10-01T09:04:30.261959Z","shell.execute_reply.started":"2021-10-01T09:04:22.811880Z","shell.execute_reply":"2021-10-01T09:04:30.261118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import all required modules🖥️","metadata":{}},{"cell_type":"code","source":"#export\nfrom typing import Tuple\nfrom collections import namedtuple\nfrom nnAudio.Spectrogram import CQT\nfrom timm import create_model, list_models\nfrom pandas.core.frame import DataFrame\nfrom fastcore.foundation import *\nfrom fastai.vision.all import *\nfrom fastai.torch_core import show_image\nfrom fastai.vision.learner import _update_first_layer","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:30.265453Z","iopub.execute_input":"2021-10-01T09:04:30.265692Z","iopub.status.idle":"2021-10-01T09:04:33.136921Z","shell.execute_reply.started":"2021-10-01T09:04:30.265649Z","shell.execute_reply":"2021-10-01T09:04:33.135473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Get the files🏗️\nI will try to grab all the numpy files inside train folder","metadata":{}},{"cell_type":"code","source":"path = Path(\"../input\")","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:33.138857Z","iopub.execute_input":"2021-10-01T09:04:33.139210Z","iopub.status.idle":"2021-10-01T09:04:33.208711Z","shell.execute_reply.started":"2021-10-01T09:04:33.139175Z","shell.execute_reply":"2021-10-01T09:04:33.206816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Get labels🏗️\nTraining labels are in the 'training_labels.csv' file.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(path/'g2net-gravitational-wave-detection/training_labels.csv')","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:33.212042Z","iopub.execute_input":"2021-10-01T09:04:33.212399Z","iopub.status.idle":"2021-10-01T09:04:33.741148Z","shell.execute_reply.started":"2021-10-01T09:04:33.212365Z","shell.execute_reply":"2021-10-01T09:04:33.740373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(1)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:33.744729Z","iopub.execute_input":"2021-10-01T09:04:33.744943Z","iopub.status.idle":"2021-10-01T09:04:33.809562Z","shell.execute_reply.started":"2021-10-01T09:04:33.744909Z","shell.execute_reply":"2021-10-01T09:04:33.808962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## getfiles(path: Path, ext)\nGet numpy files in `path` recursively, only in `folders`, if specified.\n\n> The \"#export\" in the function below and all the rest of the functions/code are there to help me use nbdev to export the required code into a library later.","metadata":{}},{"cell_type":"code","source":"#export\ndef getfiles(path: Path, ext) -> L:\n    \"Get numpy files in `path` recursively, only in `folders`, if specified.\"\n    return L(path.glob(f'**/*.{ext}'))","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:33.810793Z","iopub.execute_input":"2021-10-01T09:04:33.811089Z","iopub.status.idle":"2021-10-01T09:04:33.851811Z","shell.execute_reply.started":"2021-10-01T09:04:33.811055Z","shell.execute_reply":"2021-10-01T09:04:33.851135Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I am using the previous function to get all the files under the train folder.","metadata":{}},{"cell_type":"code","source":"train_path = path/'g2net-gravitational-wave-detection/train'","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:33.853188Z","iopub.execute_input":"2021-10-01T09:04:33.853444Z","iopub.status.idle":"2021-10-01T09:04:33.891203Z","shell.execute_reply.started":"2021-10-01T09:04:33.853413Z","shell.execute_reply":"2021-10-01T09:04:33.890465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain_files = getfiles(train_path, \"npy\")","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:04:33.892694Z","iopub.execute_input":"2021-10-01T09:04:33.892987Z","iopub.status.idle":"2021-10-01T09:06:19.799558Z","shell.execute_reply.started":"2021-10-01T09:04:33.892954Z","shell.execute_reply":"2021-10-01T09:06:19.798091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Just a quick test to see if we got the correct files.","metadata":{}},{"cell_type":"code","source":"train_files[:2]","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:19.800753Z","iopub.execute_input":"2021-10-01T09:06:19.801018Z","iopub.status.idle":"2021-10-01T09:06:19.844327Z","shell.execute_reply.started":"2021-10-01T09:06:19.800983Z","shell.execute_reply":"2021-10-01T09:06:19.843660Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Picking labels from the dataframe. We may need these labels later.","metadata":{}},{"cell_type":"code","source":"labels = df.target.to_list()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:19.845562Z","iopub.execute_input":"2021-10-01T09:06:19.845817Z","iopub.status.idle":"2021-10-01T09:06:19.897794Z","shell.execute_reply.started":"2021-10-01T09:06:19.845785Z","shell.execute_reply":"2021-10-01T09:06:19.896970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Map path to labels🗺️\nTo make things easier I will try to map the file paths to their respective labels and create a datafrane out of it.","metadata":{}},{"cell_type":"code","source":"#export\ndef map_path_to_labels(data: L, cols: L=None ) -> DataFrame:\n    \"\"\"maps the files to their labels\"\"\"\n    if cols is None: raise ValueError(\"You forgot to provide the columns\")\n    data = dict(zip(cols, data))\n    return pd.DataFrame.from_dict(data)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:19.899039Z","iopub.execute_input":"2021-10-01T09:06:19.899307Z","iopub.status.idle":"2021-10-01T09:06:19.939327Z","shell.execute_reply.started":"2021-10-01T09:06:19.899274Z","shell.execute_reply":"2021-10-01T09:06:19.938508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndf = map_path_to_labels([train_files, labels], cols=[\"id\", \"target\"])","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:19.940509Z","iopub.execute_input":"2021-10-01T09:06:19.940793Z","iopub.status.idle":"2021-10-01T09:06:22.119857Z","shell.execute_reply.started":"2021-10-01T09:06:19.940760Z","shell.execute_reply":"2021-10-01T09:06:22.119052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head(1)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:22.120988Z","iopub.execute_input":"2021-10-01T09:06:22.121731Z","iopub.status.idle":"2021-10-01T09:06:22.172472Z","shell.execute_reply.started":"2021-10-01T09:06:22.121645Z","shell.execute_reply":"2021-10-01T09:06:22.171532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## mapxy(df)","metadata":{}},{"cell_type":"code","source":"#export\ndef mapxy(df: DataFrame):\n    \"\"\"Create a dictionary of file path and the label from a dataframe\"\"\"\n    return dict(df.values)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:22.173912Z","iopub.execute_input":"2021-10-01T09:06:22.174434Z","iopub.status.idle":"2021-10-01T09:06:22.216904Z","shell.execute_reply.started":"2021-10-01T09:06:22.174400Z","shell.execute_reply":"2021-10-01T09:06:22.216329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nfiles = mapxy(df)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:22.218139Z","iopub.execute_input":"2021-10-01T09:06:22.218717Z","iopub.status.idle":"2021-10-01T09:06:23.232786Z","shell.execute_reply.started":"2021-10-01T09:06:22.218681Z","shell.execute_reply":"2021-10-01T09:06:23.232034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nfiles[train_files[7]]","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:23.234297Z","iopub.execute_input":"2021-10-01T09:06:23.234801Z","iopub.status.idle":"2021-10-01T09:06:23.295732Z","shell.execute_reply.started":"2021-10-01T09:06:23.234763Z","shell.execute_reply":"2021-10-01T09:06:23.294809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## get_label(f: Path)","metadata":{}},{"cell_type":"code","source":"def get_label(f: Path):\n    \"\"\"get the label belonging to a file\"\"\"\n    label = None\n    if f in df.values:\n        label = df[df['id'] == f]['target'].values[0]\n    return label","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:23.297528Z","iopub.execute_input":"2021-10-01T09:06:23.298126Z","iopub.status.idle":"2021-10-01T09:06:23.357738Z","shell.execute_reply.started":"2021-10-01T09:06:23.298090Z","shell.execute_reply":"2021-10-01T09:06:23.356849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_label(train_files[0])","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:23.359696Z","iopub.execute_input":"2021-10-01T09:06:23.360288Z","iopub.status.idle":"2021-10-01T09:06:24.360756Z","shell.execute_reply.started":"2021-10-01T09:06:23.360251Z","shell.execute_reply":"2021-10-01T09:06:24.360099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_label(Path(\"/ff/ff.npy\"))","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:24.362090Z","iopub.execute_input":"2021-10-01T09:06:24.362341Z","iopub.status.idle":"2021-10-01T09:06:24.873219Z","shell.execute_reply.started":"2021-10-01T09:06:24.362308Z","shell.execute_reply":"2021-10-01T09:06:24.872230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Q transform using nnaudio⚗️\nWe will design a function that would get the q transform of the time series on the fly using nnaudio.\nThe result will be similar to converting the time series data into images.\n\nCode taken from [notebook](https://www.kaggle.com/yasufuminakama/g2net-efficientnet-b7-baseline-training) shared by [Y.Nakama](https://www.kaggle.com/yasufuminakama)","metadata":{}},{"cell_type":"code","source":"#export\ndef qtfm():\n    \"\"\"convert waves to images\"\"\"\n    cqt = CQT(sr= 2048, fmin= 20, fmax= 1024, hop_length= 32, bins_per_octave=8,verbose=False)\n    return cqt","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:24.876218Z","iopub.execute_input":"2021-10-01T09:06:24.876440Z","iopub.status.idle":"2021-10-01T09:06:24.920218Z","shell.execute_reply.started":"2021-10-01T09:06:24.876416Z","shell.execute_reply":"2021-10-01T09:06:24.919426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n**NOTE**\n\nRemember to set `verbose` False if you don't want all the string output to be displayed everytime dataloader loads the data.\n\n---","metadata":{}},{"cell_type":"markdown","source":"Quick test to see if this works.","metadata":{}},{"cell_type":"code","source":"waves = np.load(train_files[0])\nwaves = np.hstack(waves)\nwaves = waves / np.max(waves)\nwaves = torch.from_numpy(waves).float()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:24.923169Z","iopub.execute_input":"2021-10-01T09:06:24.923398Z","iopub.status.idle":"2021-10-01T09:06:24.978836Z","shell.execute_reply.started":"2021-10-01T09:06:24.923367Z","shell.execute_reply":"2021-10-01T09:06:24.978013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nqtfm()(waves)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:24.980952Z","iopub.execute_input":"2021-10-01T09:06:24.981269Z","iopub.status.idle":"2021-10-01T09:06:25.070988Z","shell.execute_reply.started":"2021-10-01T09:06:24.981234Z","shell.execute_reply":"2021-10-01T09:06:25.070334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nfor i in range(5):\n    waves = np.load(df.iloc[i].id)\n    waves = np.hstack(waves)\n    waves = waves / np.max(waves)\n    waves = torch.from_numpy(waves).float()\n    image = qtfm()(waves)\n    target = get_label(train_files[i])\n    plt.imshow(image[0])\n    plt.title(f\"target: {target}\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:25.076820Z","iopub.execute_input":"2021-10-01T09:06:25.077102Z","iopub.status.idle":"2021-10-01T09:06:30.405163Z","shell.execute_reply.started":"2021-10-01T09:06:25.077077Z","shell.execute_reply":"2021-10-01T09:06:30.404435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Cool! so we are able to plot the images now. IT is fast too.","metadata":{}},{"cell_type":"markdown","source":"## Creating the dataset🖫\nIf you want to use fastai's learner to train your model on the transfomed spectograms, you can do so by creating a custom Dataset in pytorch and then feeding that dataset with a dataloader to fastai's learner. However, if you create a pipeline using fastai's internals then you get to use some cool functionalities out-of-box. We will see that in a while.\n\nAll the code below are very heavily insipired by the original inspiration of this notebook (see the very first section), this [post](https://ohmeow.com/posts/2020/04/11/finding-datablock-nirvana-part-1.html) by Wayde Gilliam and the fastai s[iamese tutorial](https://docs.fast.ai/tutorial.siamese.html#Writing-your-custom-data-block).","metadata":{}},{"cell_type":"markdown","source":"## Spectogram","metadata":{}},{"cell_type":"code","source":"#export\ndef get_waves(f):\n    \"\"\"read numpy file, stack the timeseries and convert those into a tensor\"\"\"\n    waves = np.load(f)\n    waves = np.hstack(waves)\n    waves = waves / np.max(waves)\n    waves = torch.from_numpy(waves).float()\n    return waves\n\ndef create_spectrogram(x: Path):\n    \"\"\"Create an AudioSpectrogram from a torch tensor\"\"\"\n    waves = get_waves(x)\n    return qtfm()(waves)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:30.406399Z","iopub.execute_input":"2021-10-01T09:06:30.406912Z","iopub.status.idle":"2021-10-01T09:06:30.450696Z","shell.execute_reply.started":"2021-10-01T09:06:30.406874Z","shell.execute_reply":"2021-10-01T09:06:30.449879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## NNAudioImage(fastuple)\nFirst of all, we are going to create an \"Image type\" for our transformed object (it's the numpy data transformed into spectrogram). \n\nWe have to do this because our data is not an image data from get-go. Rather it's a signal data which we are transforming into an Image.  So, to tell fastai that this is a custom Image type which we are dealing with and ho we should be displaying it, we have to create an Image type.","metadata":{}},{"cell_type":"code","source":"#export\nclass AudioImage(fastuple):\n    \"\"\"Custom Image for nnAudio transformed signals\"\"\"\n    def show(self, figsize=None, ctx=None, **kwargs):\n        if len(self) > 1:\n            img,label = self\n        else:\n            img = self\n            label = ''\n    \n        if figsize is None: figsize=(10,10)\n        return show_image(img, \n                          title=label, figsize=figsize, ctx=ctx)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:30.452115Z","iopub.execute_input":"2021-10-01T09:06:30.452375Z","iopub.status.idle":"2021-10-01T09:06:30.496694Z","shell.execute_reply.started":"2021-10-01T09:06:30.452342Z","shell.execute_reply":"2021-10-01T09:06:30.495992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image = create_spectrogram(train_files[0])","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:30.498004Z","iopub.execute_input":"2021-10-01T09:06:30.498254Z","iopub.status.idle":"2021-10-01T09:06:30.554178Z","shell.execute_reply.started":"2021-10-01T09:06:30.498223Z","shell.execute_reply":"2021-10-01T09:06:30.553460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"type(image), image.shape","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:30.555492Z","iopub.execute_input":"2021-10-01T09:06:30.555753Z","iopub.status.idle":"2021-10-01T09:06:30.597207Z","shell.execute_reply.started":"2021-10-01T09:06:30.555723Z","shell.execute_reply":"2021-10-01T09:06:30.596294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:30.598860Z","iopub.execute_input":"2021-10-01T09:06:30.599135Z","iopub.status.idle":"2021-10-01T09:06:30.642232Z","shell.execute_reply.started":"2021-10-01T09:06:30.599104Z","shell.execute_reply":"2021-10-01T09:06:30.641383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_label(train_files[0])","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:30.643649Z","iopub.execute_input":"2021-10-01T09:06:30.643907Z","iopub.status.idle":"2021-10-01T09:06:31.540918Z","shell.execute_reply.started":"2021-10-01T09:06:30.643874Z","shell.execute_reply":"2021-10-01T09:06:31.540093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"s = AudioImage(image, get_label(train_files[0]))\ns.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:31.542277Z","iopub.execute_input":"2021-10-01T09:06:31.542530Z","iopub.status.idle":"2021-10-01T09:06:32.537573Z","shell.execute_reply.started":"2021-10-01T09:06:31.542498Z","shell.execute_reply":"2021-10-01T09:06:32.536769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## NNAudioDataset\nYou can create a Dataset in fastai by creating a custom Transform . Creating a `Transform` has come advantages as compared to a pytorch Dataset. For example, you don't need to have a `len` component or a `get_item` component.\n\nOn a very high level a `Transform` has an `encodes`, `decodes` and `setup` methods. For our purpose having an `encodes` methods only would suffice. This is the place where we would be transforming the numpy data into spectograms.\n\nTo know more about `Tranforms` refer these -->\n* [data block nirvana](https://ohmeow.com/posts/2020/04/11/finding-datablock-nirvana-part-1.html)\n* [Siamese tutorial](https://docs.fast.ai/tutorial.siamese.html#Writing-your-custom-data-block)\n* [Fastbook chapter-11](https://github.com/fastai/fastbook/blob/master/11_midlevel_data.ipynb)\n* [Albumentation tutorial](https://docs.fast.ai/tutorial.albumentations.html)","metadata":{}},{"cell_type":"code","source":"%%time\nvals = df.target.to_list()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:32.538981Z","iopub.execute_input":"2021-10-01T09:06:32.539237Z","iopub.status.idle":"2021-10-01T09:06:32.612564Z","shell.execute_reply.started":"2021-10-01T09:06:32.539202Z","shell.execute_reply":"2021-10-01T09:06:32.610878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nvocab,o2i = uniqueify(vals, sort=True, bidir=True)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:32.613920Z","iopub.execute_input":"2021-10-01T09:06:32.614302Z","iopub.status.idle":"2021-10-01T09:06:32.670190Z","shell.execute_reply.started":"2021-10-01T09:06:32.614154Z","shell.execute_reply":"2021-10-01T09:06:32.669499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lbl2path = mapxy(df)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:32.671442Z","iopub.execute_input":"2021-10-01T09:06:32.671760Z","iopub.status.idle":"2021-10-01T09:06:33.586513Z","shell.execute_reply.started":"2021-10-01T09:06:32.671726Z","shell.execute_reply":"2021-10-01T09:06:33.585777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nlbl2path[train_files[5]]","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:33.587727Z","iopub.execute_input":"2021-10-01T09:06:33.587999Z","iopub.status.idle":"2021-10-01T09:06:33.634234Z","shell.execute_reply.started":"2021-10-01T09:06:33.587967Z","shell.execute_reply":"2021-10-01T09:06:33.633407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"combining all above steps","metadata":{}},{"cell_type":"markdown","source":"## NNAudioTransform(ItemTransform)","metadata":{}},{"cell_type":"code","source":"#export\n#ItemTransform let's you work with tuple elements\nclass NNAudioTransform(ItemTransform):\n    \"\"\"Custom Transform which uses nnAudio transforms\n    to extract spectogram on the fly\"\"\"\n    def __init__(self, df: DataFrame, col: str = 'target'):\n        self.lbl2files = mapxy(df)\n        vals = df[col].to_list()\n        self.vocab,self.o2i = uniqueify(vals, sort=True, bidir=True)\n        \n    def encodes(self, o): return (create_spectrogram(o), self.o2i.get(self.lbl2files.get(o)))\n    def decodes(self, x): return AudioImage(x[0],self.vocab[x[1]])","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:33.635996Z","iopub.execute_input":"2021-10-01T09:06:33.636248Z","iopub.status.idle":"2021-10-01T09:06:33.679641Z","shell.execute_reply.started":"2021-10-01T09:06:33.636217Z","shell.execute_reply":"2021-10-01T09:06:33.678925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's walk through the code.\n\nIf you inherit from the `Transform` class, the resulting transform is applied to the item as a whole but when you inherit from the `ItemTransform` class then the resulting transform is applied to each element of the input.\n\nFor example, if you have a transform that is inherited from the `Transform` class and you have an input which is a tuple `(\"a\", 1)` then the transform would consider the tuple as a single element. But, when your transform is an `ItemTransform` then the transform is applied to \"a\" as well as \"1\" separately. \n\nThe __init__ method sets up our `mapxy` method as a class property. It then converts the target column values into a list and creates a vocab of our targets and a dictionary mapping our targets to indices.\n\nThe encodes method is where the magic occurs. Here, we return a tuple with our spectogram and the label related to our input. \n\nThe decodes method returns an `AudioImage` type which knows how to show itself whenever a `show` method is invoked.\n\nYou might notice that I have used a dataframe to create a list of our inputs and a dictionary of our labels. This was an engineering choice which I made because creating a list of labels from the input list of filenames was too slow. Doing it this was by using a dataframe  made things faster.\n\n> In deep learning a majority chunk of the speed boost comes from good engineering practices rather than having the best SOTA architectures or a faster computer.","metadata":{}},{"cell_type":"markdown","source":"We will also use a 'splitter' which tells fastai the way we want to split our data. For now we will use `RandomSplitter` to do this job. Additionally we will also instantiate the `NNAudioTransform` object.","metadata":{}},{"cell_type":"markdown","source":"We will take a few samples only to make our experiment quicker.","metadata":{}},{"cell_type":"code","source":"subset_for_dsets = train_files[:20000]","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:33.680661Z","iopub.execute_input":"2021-10-01T09:06:33.681553Z","iopub.status.idle":"2021-10-01T09:06:33.721663Z","shell.execute_reply.started":"2021-10-01T09:06:33.681518Z","shell.execute_reply":"2021-10-01T09:06:33.720914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"splits = TrainTestSplitter()(subset_for_dsets)\ntfm = NNAudioTransform(df)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:33.722995Z","iopub.execute_input":"2021-10-01T09:06:33.723358Z","iopub.status.idle":"2021-10-01T09:06:34.950608Z","shell.execute_reply.started":"2021-10-01T09:06:33.723325Z","shell.execute_reply":"2021-10-01T09:06:34.949913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `tfm` is a transform is would be applied to the input files to generate the spectogram. The second list has the transform which will be applied to our targets. ","metadata":{}},{"cell_type":"markdown","source":"Next, we have to tell fastai to take our 'sample' and apply the transform and the splitter to it.","metadata":{}},{"cell_type":"code","source":"%%time\ntls = TfmdLists(subset_for_dsets, tfm, splits=splits)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:34.952105Z","iopub.execute_input":"2021-10-01T09:06:34.952357Z","iopub.status.idle":"2021-10-01T09:06:35.045887Z","shell.execute_reply.started":"2021-10-01T09:06:34.952326Z","shell.execute_reply":"2021-10-01T09:06:35.045181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`TfmdLists` is a low-level API which creates a pipeline for us. It creates a pipeline that takes in our samples-->splits it --> applies our transform to the items.\n\nMore information on a `TfmdLists` can be found in this [tutorial](https://docs.fast.ai/tutorial.pets.html) fromt he official documentation.","metadata":{}},{"cell_type":"code","source":"tls.vocab","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:35.047071Z","iopub.execute_input":"2021-10-01T09:06:35.047312Z","iopub.status.idle":"2021-10-01T09:06:35.090786Z","shell.execute_reply.started":"2021-10-01T09:06:35.047280Z","shell.execute_reply":"2021-10-01T09:06:35.089908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_at(tls.train, 1)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:35.092399Z","iopub.execute_input":"2021-10-01T09:06:35.092673Z","iopub.status.idle":"2021-10-01T09:06:35.375398Z","shell.execute_reply.started":"2021-10-01T09:06:35.092640Z","shell.execute_reply":"2021-10-01T09:06:35.374586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create the Dataloader🖨️\nWe can use the `TfmdLists` to create a dataloader by calling `dataloaders()`. Here, we can't apply `item_tfms` or `batch_tfms` but we can get the hooks to different point of the pipeline and can put our transforms there.\n\nFor example, once items are grabbed then that moment is known as \"after_item\". We can use this hook to apply our transforms once items are grabbed.","metadata":{}},{"cell_type":"code","source":"dls = tls.dataloaders(after_item=[ToTensor()], after_batch=[IntToFloatTensor(), Normalize.from_stats(*imagenet_stats)])","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:35.376714Z","iopub.execute_input":"2021-10-01T09:06:35.377142Z","iopub.status.idle":"2021-10-01T09:06:38.707320Z","shell.execute_reply.started":"2021-10-01T09:06:35.377106Z","shell.execute_reply":"2021-10-01T09:06:38.706597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One more thing that we need to do is to make the `show_batch` method aware of the type of our Image. This can be easily done by using `typedispatch` to dispatch our `show_batch` (the one which we will override with our image type). ","metadata":{}},{"cell_type":"code","source":"#export\n@typedispatch\ndef show_batch(x:AudioImage, y, samples, ctxs=None, max_n=6, nrows=None, ncols=3, figsize=None, **kwargs):\n    if figsize is None: figsize = (ncols*6, max_n//ncols * 3)\n    if ctxs is None: ctxs = get_grid(min(x[0].shape[0], max_n), nrows=None, ncols=ncols, figsize=figsize)\n    for i,ctx in enumerate(ctxs):\n        AudioImage(x[0][i], ['0','1'][x[1][i].item()]).show(ctx=ctx)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:38.708534Z","iopub.execute_input":"2021-10-01T09:06:38.708783Z","iopub.status.idle":"2021-10-01T09:06:38.754518Z","shell.execute_reply.started":"2021-10-01T09:06:38.708753Z","shell.execute_reply":"2021-10-01T09:06:38.753722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`typedispatch` does something similar to [multi-dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch). So, that whenever we call the `show_batch` on our image type then fastai will call our version of `show_batch` after recognizing our image type.","metadata":{}},{"cell_type":"markdown","source":"Here we go","metadata":{}},{"cell_type":"code","source":"%%time\ndls.show_batch()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:38.755986Z","iopub.execute_input":"2021-10-01T09:06:38.756351Z","iopub.status.idle":"2021-10-01T09:06:41.093714Z","shell.execute_reply.started":"2021-10-01T09:06:38.756317Z","shell.execute_reply":"2021-10-01T09:06:41.093007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modularity 🧩\n\nThe way that we have created the above transform works well for a specific type of task. There are somethings which could not be answered by the above transform.\n\n* What is the categories are other than 0 and 1.\n* What if it's a multicategory problem.\n* How to handle the lack of targets during inference? \n  * This could be handled by having a `setups` method inside the transform and have it accept list of filenames. This could work well when data is small but for huge data mapping the labelling function to all the filenames in order to create a vocab and label maps would take lots of time. In short it doesn't scale well.\n  \nSo what do we do?\n\nThe solution is to create a custom datablock for our type of task which can then be plugged into a `Datablock` like this-->\n\n```\nDataBlock(blocks=(NNAudioBlock, MultiCategoryBlock),\n                   splitter=ColSplitter(),\n                   get_x=lambda x:pascal_source/\"train\"/f'{x[0]}',\n                   get_y=lambda x:x[1].split(' '),\n                   item_tfms=Resize(224),\n                   batch_tfms=aug_transforms())\n```","metadata":{}},{"cell_type":"markdown","source":"Let's create a type to represent our spectrogram","metadata":{}},{"cell_type":"code","source":"class Spectrogram(TensorImageBase):\n    \"\"\"Type to represent a spectogram which knows show itself\"\"\"\n    @classmethod\n    def create(cls, o):\n        waves = get_waves(o)\n        return cls(qtfm()(waves))\n    \n    def show(self, figsize=None, ctx=None, **kwargs): \n        t = self\n        if not isinstance(t, Tensor): return ctx\n        if figsize is None: figsize=(10,10)\n        return show_image(t, figsize=figsize, ctx=ctx)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:41.095069Z","iopub.execute_input":"2021-10-01T09:06:41.095523Z","iopub.status.idle":"2021-10-01T09:06:41.158691Z","shell.execute_reply.started":"2021-10-01T09:06:41.095486Z","shell.execute_reply":"2021-10-01T09:06:41.157853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the above class we use the functions `get_waves` and `qtfm()` defined in the previous sections to create a spectrogram. The `show` method is also similar to the `show` method which we had used in the previous section. The only difference is that in this show method we are not taking the label into account because the `Spectogram` is just a type of a file converted to a spectrogram.","metadata":{}},{"cell_type":"markdown","source":"but does it work?  let's test it.","metadata":{}},{"cell_type":"code","source":"spectrogram = Spectrogram.create(train_files[0])\ntype(spectrogram)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:41.159772Z","iopub.execute_input":"2021-10-01T09:06:41.160139Z","iopub.status.idle":"2021-10-01T09:06:41.223331Z","shell.execute_reply.started":"2021-10-01T09:06:41.160097Z","shell.execute_reply":"2021-10-01T09:06:41.222714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spectrogram.show()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:41.224587Z","iopub.execute_input":"2021-10-01T09:06:41.224822Z","iopub.status.idle":"2021-10-01T09:06:41.371964Z","shell.execute_reply.started":"2021-10-01T09:06:41.224793Z","shell.execute_reply":"2021-10-01T09:06:41.370779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Voila! it knows how to show itself.","metadata":{}},{"cell_type":"markdown","source":"Now, we can create a custom block for our data. A block is a set of default transforms which is supposed to be applied to your data in order to tell fastai about the type of your data. \n\nIn our custom block we will tell fastai how create a Spectrogram block and then apply `IntToFloatTensor` transform.\n\nThe source code an `ImageBlock` is like this-->","metadata":{}},{"cell_type":"code","source":"ImageBlock??","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:41.373261Z","iopub.execute_input":"2021-10-01T09:06:41.373539Z","iopub.status.idle":"2021-10-01T09:06:41.487765Z","shell.execute_reply.started":"2021-10-01T09:06:41.373505Z","shell.execute_reply":"2021-10-01T09:06:41.487134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will use the source code for `ImageBlock` to create our custom block.","metadata":{}},{"cell_type":"code","source":"def SpectrogramBlock(cls=Spectrogram) : \n    \"A `TransformBlock` for spectograms of `cls`\"\n    return TransformBlock(type_tfms=cls.create, batch_tfms=IntToFloatTensor)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:41.488926Z","iopub.execute_input":"2021-10-01T09:06:41.489177Z","iopub.status.idle":"2021-10-01T09:06:41.528962Z","shell.execute_reply.started":"2021-10-01T09:06:41.489147Z","shell.execute_reply":"2021-10-01T09:06:41.528170Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we have our custom block ready, we can test if a DataBlock can now be created.","metadata":{}},{"cell_type":"code","source":"g2net = DataBlock(blocks=(SpectrogramBlock, CategoryBlock),\n                   splitter=RandomSplitter(),\n                   get_x=ColReader(0),\n                   get_y=ColReader(1),\n                   batch_tfms=aug_transforms())","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:41.530210Z","iopub.execute_input":"2021-10-01T09:06:41.530500Z","iopub.status.idle":"2021-10-01T09:06:41.572749Z","shell.execute_reply.started":"2021-10-01T09:06:41.530469Z","shell.execute_reply":"2021-10-01T09:06:41.571874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we create the dataloader.","metadata":{}},{"cell_type":"code","source":"dls = g2net.dataloaders(df.iloc[:2000])","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:41.574025Z","iopub.execute_input":"2021-10-01T09:06:41.574366Z","iopub.status.idle":"2021-10-01T09:06:41.882793Z","shell.execute_reply.started":"2021-10-01T09:06:41.574332Z","shell.execute_reply":"2021-10-01T09:06:41.882074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ndls.show_batch()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:41.884081Z","iopub.execute_input":"2021-10-01T09:06:41.884340Z","iopub.status.idle":"2021-10-01T09:06:44.285610Z","shell.execute_reply.started":"2021-10-01T09:06:41.884309Z","shell.execute_reply":"2021-10-01T09:06:44.284761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we go. Now we have a custom block and we can create a DataBlock as well as dataloaders and then display it.","metadata":{}},{"cell_type":"markdown","source":"## Make your own model🍕\nWe are going to use the timm library as the source of our model. To weave it into fastai, we will create a custom fastai model.\n\nAll the code below is heavily inspired by-->\n\n* [Ayushman's](https://www.kaggle.com/benihime91) [notebook](https://www.kaggle.com/benihime91/fastai-timm-efficientnet-train-fold-0).\n* fastai siamese [tutorial](https://docs.fast.ai/tutorial.siamese.html).\n\nWe will also take into account the structure of fastai's `create_cnn_model` class. The code for which is as follows","metadata":{}},{"cell_type":"code","source":"create_cnn_model??","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:44.287073Z","iopub.execute_input":"2021-10-01T09:06:44.287499Z","iopub.status.idle":"2021-10-01T09:06:44.363842Z","shell.execute_reply.started":"2021-10-01T09:06:44.287464Z","shell.execute_reply":"2021-10-01T09:06:44.363055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's build our own.\n\nWe will cut off the head of a timm pretrained model using `create_body` and take the encoder only as this would be the portion of the pretrained model which I would like to use. Then I will top it off with a custom fastai head using `create_head` that we would need to train on our target data.\n\nTo know more about this flow have a look into the fastai siamese [tutorial](https://docs.fast.ai/tutorial.siamese.html).","metadata":{}},{"cell_type":"markdown","source":"But first we will create our custom `create_body` and `create_head` functions. the reason for this is that fastai in it's current state is not integrated with the timm library. So, creating custom versions of `create_body` and `create_head` makes the weaving of timm into fastai re-usable.\n\nThe insipration for this is the [post](https://walkwithfastai.com/vision.external.timm#create_timm_body) in 'walk with fastai'. Once again the code and the approach is based on this post.\n\n> I am recreating this again here instead of using the 'walk with fastai' library is to drill down into the concept and for my personal learning.","metadata":{}},{"cell_type":"markdown","source":"## create_timm_body(arch, n_in=3, pretrained=True, cut=None)","metadata":{}},{"cell_type":"code","source":"#export\ndef create_timm_body(arch, n_in=3, pretrained=True, cut=None):\n    \"Cut off the body of a typically pretrained timm library `arch` as determined by `cut`\"\n    model = create_model(arch, pretrained=pretrained, num_classes=0, in_chans=1,global_pool='')\n    _update_first_layer(model, n_in, pretrained)\n    #cut = ifnone(cut, cnn_config(arch)['cut'])\n    if cut is None:\n        ll = list(enumerate(model.children()))\n        cut = next(i for i,o in reversed(ll) if has_pool_type(o))\n    if   isinstance(cut, int):      return nn.Sequential(*list(model.children())[:cut])\n    elif callable(cut): return cut(model)\n    else:  raise NamedError(\"cut must be either integer or a function\")","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:44.365102Z","iopub.execute_input":"2021-10-01T09:06:44.365350Z","iopub.status.idle":"2021-10-01T09:06:44.412815Z","shell.execute_reply.started":"2021-10-01T09:06:44.365319Z","shell.execute_reply":"2021-10-01T09:06:44.412065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now that we have a way to create a body, we will use the code from `create_cnn_model` to build our custom `create_timm_model`.\n\nThe code for `create_timm_model` is as follows.","metadata":{}},{"cell_type":"code","source":"create_cnn_model??","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:44.414143Z","iopub.execute_input":"2021-10-01T09:06:44.415012Z","iopub.status.idle":"2021-10-01T09:06:44.472527Z","shell.execute_reply.started":"2021-10-01T09:06:44.414966Z","shell.execute_reply":"2021-10-01T09:06:44.471770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## create_timm_model","metadata":{}},{"cell_type":"code","source":"create_head?","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:44.473719Z","iopub.execute_input":"2021-10-01T09:06:44.474067Z","iopub.status.idle":"2021-10-01T09:06:44.522260Z","shell.execute_reply.started":"2021-10-01T09:06:44.474031Z","shell.execute_reply":"2021-10-01T09:06:44.521307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#export\n@delegates(create_head)\ndef create_timm_model(arch, n_out, pretrained=True, cut=None, n_in=3, init=nn.init.kaiming_normal_, custom_head=None,\n                     concat_pool=True, in_chans=1, **kwargs):\n    \"Create custom architecture from the timm library\"\n    body = create_timm_body(arch, n_in, pretrained, None)\n    if custom_head is None:\n        nf = num_features_model(nn.Sequential(*body.children()))\n        head = create_head(nf, n_out, concat_pool=concat_pool, **kwargs)\n    else: head = custom_head\n    model = nn.Sequential(body, head)\n    if init is not None: apply_init(model[1], init)\n    return model","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:44.523779Z","iopub.execute_input":"2021-10-01T09:06:44.524105Z","iopub.status.idle":"2021-10-01T09:06:44.572137Z","shell.execute_reply.started":"2021-10-01T09:06:44.524051Z","shell.execute_reply":"2021-10-01T09:06:44.571428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `@delegate` macro tells fastai to show the parameters of any `**kwargs` (which we would be using in the `create_body`) during function [introspection](https://fastcore.fast.ai/meta.html#delegates).","metadata":{}},{"cell_type":"markdown","source":"Let's do a quick test to check if our custom model works.","metadata":{}},{"cell_type":"code","source":"# num of classes\nn_out = 2","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:44.573562Z","iopub.execute_input":"2021-10-01T09:06:44.573857Z","iopub.status.idle":"2021-10-01T09:06:44.616088Z","shell.execute_reply.started":"2021-10-01T09:06:44.573818Z","shell.execute_reply":"2021-10-01T09:06:44.615248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = create_timm_model(\"efficientnet_b3a\", n_out)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:44.617545Z","iopub.execute_input":"2021-10-01T09:06:44.617909Z","iopub.status.idle":"2021-10-01T09:06:46.371067Z","shell.execute_reply.started":"2021-10-01T09:06:44.617871Z","shell.execute_reply":"2021-10-01T09:06:46.370319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"L(model.children())","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:46.372376Z","iopub.execute_input":"2021-10-01T09:06:46.372632Z","iopub.status.idle":"2021-10-01T09:06:46.418511Z","shell.execute_reply.started":"2021-10-01T09:06:46.372602Z","shell.execute_reply":"2021-10-01T09:06:46.417848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"cool! so it works.\n\nNow, we will build a learner which would enable us to do transfer learning with timm models. Once again we will port `cnn_learner` for our use and like before let's quickly take a look into the `cnn_learner` code","metadata":{}},{"cell_type":"code","source":"cnn_learner??","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:46.419735Z","iopub.execute_input":"2021-10-01T09:06:46.420155Z","iopub.status.idle":"2021-10-01T09:06:46.480079Z","shell.execute_reply.started":"2021-10-01T09:06:46.420119Z","shell.execute_reply":"2021-10-01T09:06:46.479064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#export\n@delegates(create_timm_model)\ndef timm_learner(dls, arch, n_out=None, pretrained=True,\n                # learner args\n                loss_func=None, opt_func=Adam, lr=defaults.lr, splitter=None, cbs=None, metrics=None, path=None,\n                model_dir='models', wd=None, wd_bn_bias=False, train_bn=True, moms=(0.95,0.85,0.95),\n                # other model args\n                **kwargs):\n    \"Build a convnet style learner from `dls` and `timm arch`\"\n\n    kwargs = {**kwargs}\n    if n_out is None: n_out = get_c(dls)\n    assert n_out, \"`n_out` is not defined, and could not be inferred from data, set `dls.c` or pass `n_out`\"\n    model = create_timm_model(arch, n_out, default_split, pretrained, **kwargs)\n\n    learn = Learner(dls=dls, model=model, loss_func=loss_func, opt_func=opt_func, lr=lr, splitter=default_split, cbs=cbs,\n                   metrics=metrics, path=path, model_dir=model_dir, wd=wd, wd_bn_bias=wd_bn_bias, train_bn=train_bn,\n                   moms=moms)\n    if pretrained: learn.freeze()\n    # keep track of args for loggers\n    store_attr('arch,n_out,pretrained', self=learn, **kwargs)\n    return learn","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:46.483090Z","iopub.execute_input":"2021-10-01T09:06:46.483281Z","iopub.status.idle":"2021-10-01T09:06:46.525289Z","shell.execute_reply.started":"2021-10-01T09:06:46.483258Z","shell.execute_reply":"2021-10-01T09:06:46.524566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we go. We have managed to get a port of the learner code which looks the part. Does it work?\n\nLet me find out.\n\n> To find the list of models available in the timm library use `list_models`","metadata":{}},{"cell_type":"code","source":"list_models(\"efficient*\")","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:06:46.526490Z","iopub.execute_input":"2021-10-01T09:06:46.526757Z","iopub.status.idle":"2021-10-01T09:06:46.567314Z","shell.execute_reply.started":"2021-10-01T09:06:46.526726Z","shell.execute_reply":"2021-10-01T09:06:46.566699Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The learner👩‍🏫\nNow that we have the model in place, we can go ahead and create the learner the usual way. We have kept the batch size to default.\n\nThere is one little thing that I would like to do before creating a learner. I will create a helper function which can help me to get the suggested learning rate quickly.","metadata":{}},{"cell_type":"code","source":"#export\ndef show_me_lrs(learn, num_it:int= 10):\n    Suggested_lrs = namedtuple('Suggested_lrs', [\"min\", \"steep\",\n                                            \"valley\", \"slide\"])\n    lrs = learn.lr_find(suggest_funcs=(minimum, steep,valley, slide))\n    suggested_lrs = Suggested_lrs(lrs[0], lrs[1], lrs[2], lrs[3])\n    \n    print(f\"Minimum/10:\\t{lrs[0]:.2e}\\\n          \\nSteepest point:\\t{lrs[1]:.2e}\\\n          \\nLongest valley:\\t{lrs[2]:.2e}\\\n          \\nSlide interval:\\t{lrs[3]:.2e}\")\n    \n    return suggested_lrs","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:13:42.733898Z","iopub.execute_input":"2021-10-01T09:13:42.734541Z","iopub.status.idle":"2021-10-01T09:13:42.778633Z","shell.execute_reply.started":"2021-10-01T09:13:42.734505Z","shell.execute_reply":"2021-10-01T09:13:42.777955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = timm_learner(dls, 'efficientnet_b7', loss_func=CrossEntropyLossFlat(), metrics=[RocAucBinary(axis=0)], n_out=2).to_fp16()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:29:42.716122Z","iopub.execute_input":"2021-10-01T09:29:42.716729Z","iopub.status.idle":"2021-10-01T09:29:43.928872Z","shell.execute_reply.started":"2021-10-01T09:29:42.716691Z","shell.execute_reply":"2021-10-01T09:29:43.928140Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fit one epoch to see how it behaves","metadata":{}},{"cell_type":"code","source":"learn.fit_one_cycle(1, 3e-3)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:29:45.447860Z","iopub.execute_input":"2021-10-01T09:29:45.448475Z","iopub.status.idle":"2021-10-01T09:30:17.748596Z","shell.execute_reply.started":"2021-10-01T09:29:45.448439Z","shell.execute_reply":"2021-10-01T09:30:17.747843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#to recover gpu ram\nlearn.save('epoch1')\nlearn.load('epoch1')","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:30:40.513483Z","iopub.execute_input":"2021-10-01T09:30:40.513755Z","iopub.status.idle":"2021-10-01T09:30:41.877321Z","shell.execute_reply.started":"2021-10-01T09:30:40.513725Z","shell.execute_reply":"2021-10-01T09:30:41.876638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using the learning rate finder to get the learning rate","metadata":{}},{"cell_type":"code","source":"import gc; gc.collect()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:30:42.109098Z","iopub.execute_input":"2021-10-01T09:30:42.109691Z","iopub.status.idle":"2021-10-01T09:30:42.610599Z","shell.execute_reply.started":"2021-10-01T09:30:42.109656Z","shell.execute_reply":"2021-10-01T09:30:42.609868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"suggested_lrs = show_me_lrs(learn)\n#learn.lr_find(suggest_funcs=(minimum, steep,valley, slide))","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:30:44.226476Z","iopub.execute_input":"2021-10-01T09:30:44.227055Z","iopub.status.idle":"2021-10-01T09:32:27.557818Z","shell.execute_reply.started":"2021-10-01T09:30:44.227019Z","shell.execute_reply":"2021-10-01T09:32:27.557080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I will use the slide algorithm here to get the optimal learning rate.","metadata":{}},{"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(3, suggested_lrs.slide)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:32:50.301986Z","iopub.execute_input":"2021-10-01T09:32:50.302401Z","iopub.status.idle":"2021-10-01T09:34:39.928001Z","shell.execute_reply.started":"2021-10-01T09:32:50.302349Z","shell.execute_reply":"2021-10-01T09:34:39.927167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok! The performance is not that great but the goal of this exercise was not to have a SOTA model but rather to learn how to create a custom code base by using Fastai internals.\n\nHowever, with proper data augmentation and more data the performance can be much better.","metadata":{}},{"cell_type":"code","source":"learn.export(\"./final\")","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:36:18.063125Z","iopub.execute_input":"2021-10-01T09:36:18.064073Z","iopub.status.idle":"2021-10-01T09:36:18.745135Z","shell.execute_reply.started":"2021-10-01T09:36:18.064035Z","shell.execute_reply":"2021-10-01T09:36:18.744310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference🧐\nFor inference you will need to use the previous dataloader to create a test dataloader by passing the test files to it.","metadata":{}},{"cell_type":"code","source":"%%time\ntest_path = path/'g2net-gravitational-wave-detection/test'\ntest_files = getfiles(test_path, \"npy\")","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:36:21.813926Z","iopub.execute_input":"2021-10-01T09:36:21.814669Z","iopub.status.idle":"2021-10-01T09:37:00.924300Z","shell.execute_reply.started":"2021-10-01T09:36:21.814635Z","shell.execute_reply":"2021-10-01T09:37:00.923532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference","metadata":{}},{"cell_type":"markdown","source":"For inference we first load the learner","metadata":{}},{"cell_type":"code","source":"learn = load_learner('./final', cpu=False)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:37:05.913571Z","iopub.execute_input":"2021-10-01T09:37:05.914151Z","iopub.status.idle":"2021-10-01T09:37:06.248381Z","shell.execute_reply.started":"2021-10-01T09:37:05.914115Z","shell.execute_reply":"2021-10-01T09:37:06.247603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create a test dataloader. This will take in the test files and apply the transforms that we had created during trainign timebut on the inference data and give you a dataloader.","metadata":{}},{"cell_type":"code","source":"test_dls = learn.dls.test_dl(test_files[:100])","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:37:24.696529Z","iopub.execute_input":"2021-10-01T09:37:24.696794Z","iopub.status.idle":"2021-10-01T09:37:24.741174Z","shell.execute_reply.started":"2021-10-01T09:37:24.696766Z","shell.execute_reply":"2021-10-01T09:37:24.740169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"check the batch","metadata":{}},{"cell_type":"code","source":"test_dls.show_batch()","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:37:28.938964Z","iopub.execute_input":"2021-10-01T09:37:28.939599Z","iopub.status.idle":"2021-10-01T09:37:31.354223Z","shell.execute_reply.started":"2021-10-01T09:37:28.939545Z","shell.execute_reply":"2021-10-01T09:37:31.353395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Use `get_preds` to get predictions in batches.","metadata":{}},{"cell_type":"code","source":"preds = learn.get_preds(dl=test_dls)","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:37:35.135121Z","iopub.execute_input":"2021-10-01T09:37:35.135693Z","iopub.status.idle":"2021-10-01T09:37:39.890891Z","shell.execute_reply.started":"2021-10-01T09:37:35.135658Z","shell.execute_reply":"2021-10-01T09:37:39.890169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Have a look at your predictions.","metadata":{}},{"cell_type":"code","source":"preds","metadata":{"execution":{"iopub.status.busy":"2021-10-01T09:37:44.268898Z","iopub.execute_input":"2021-10-01T09:37:44.269277Z","iopub.status.idle":"2021-10-01T09:37:44.351174Z","shell.execute_reply.started":"2021-10-01T09:37:44.269234Z","shell.execute_reply":"2021-10-01T09:37:44.350412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\nI prepared this post as part of my experimentation for the [g2net-gravitational-wave-detection](https://www.kaggle.com/c/g2net-gravitational-wave-detection) competition. My goal for preparing this notebook was to design an end-to-end flow to learn about extending fastai for a custom new task and how to extend the library to work well with other libraries.\n\nIt took quite a long time to get my head around the low-level and mid level API in fastai.\n\nPart of the reason being that I couldn't spend much time on this competition and the other part was that there are very few resources available at this moment which provide good detail about creating custom bits using fastai's mid-level and low-level APIs. \n\nI would like to say that the effort that it took to complete this post was worth it and I came to know how powerful the modular structure of fastai is.\n\nI would like to create an extension library using the code that I have developed for this post but at this moment I can't say how soon I would be able to do it and when but stay tuned as I would keep posting my progress on this.\n","metadata":{}}]}