{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Final Model Evaluation Notebook\n","metadata":{}},{"cell_type":"markdown","source":"## Final Model description\n\nThis notebook consists of an evaluation of the best model for the Freesound General-Purpose Audio Tagging Challenge Kaggle competition. The best model that will be discussed here today is the Freesound_Audio_EfficientNet_B1 model which had the best public score of 0.90531 and a private score of 0.87349 on the test dataset. The EfficientNetB1 model is the only model that contains an image size of 128 by 87 where the other models contains an image size of 128 by 32. It is also worth noting that the model also performed without any use of fine-tuning and managed to get a decent validation accuracy of about 79 percent. This notebook will further go into more details regarding the EfficientNetB1 model below. ","metadata":{}},{"cell_type":"markdown","source":"## Importing Packages \n\nThe two code cells below imports several different packages into the Kaggle notebook that will allow me to perform the necessary steps to acheive the best model. Few important packages that is needed for this project include; Librosa, IPython, and TensorFlow.","metadata":{}},{"cell_type":"code","source":"!pip install -U efficientnet -qq","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:01.403697Z","iopub.execute_input":"2022-04-28T07:27:01.404063Z","iopub.status.idle":"2022-04-28T07:27:12.403716Z","shell.execute_reply.started":"2022-04-28T07:27:01.403969Z","shell.execute_reply":"2022-04-28T07:27:12.402796Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport math\n\nimport os\nimport cv2\n\nimport IPython.display as ipd \n\nimport librosa \nimport librosa.display\n\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow import keras\n\nfrom tensorflow.keras.models import Sequential, load_model\nfrom tensorflow.keras.layers import *\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras import backend as K\n\nimport efficientnet.tfkeras as efn\n\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' ","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:12.405717Z","iopub.execute_input":"2022-04-28T07:27:12.406008Z","iopub.status.idle":"2022-04-28T07:27:19.205438Z","shell.execute_reply.started":"2022-04-28T07:27:12.405969Z","shell.execute_reply":"2022-04-28T07:27:19.204640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading the DataFrame\n\nNext, we'll put the training dataset into the Kaggle notebook, which will allow me to train the model using the various audio files. fnames, labels, and manually_verified are the three attributes in the dataset below. The rows of wav audio files are referred to as fnames. The labels are the names that each audio file could have. The manually_verified column, on the other hand, is a binary classification problem that indicates whether or not a specific file label has been manually verified.","metadata":{}},{"cell_type":"code","source":"train_path = '../input/freesound-audio-tagging/audio_train/'\n\nprint(len(os.listdir(train_path)))","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:19.207007Z","iopub.execute_input":"2022-04-28T07:27:19.207258Z","iopub.status.idle":"2022-04-28T07:27:19.669388Z","shell.execute_reply.started":"2022-04-28T07:27:19.207223Z","shell.execute_reply":"2022-04-28T07:27:19.668589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(\"../input/freesound-audio-tagging/train.csv\")\n\nprint('The shape of the training data is: ', train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:19.671434Z","iopub.execute_input":"2022-04-28T07:27:19.672782Z","iopub.status.idle":"2022-04-28T07:27:19.697654Z","shell.execute_reply.started":"2022-04-28T07:27:19.672738Z","shell.execute_reply":"2022-04-28T07:27:19.696962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:19.698998Z","iopub.execute_input":"2022-04-28T07:27:19.699238Z","iopub.status.idle":"2022-04-28T07:27:19.717553Z","shell.execute_reply.started":"2022-04-28T07:27:19.699205Z","shell.execute_reply":"2022-04-28T07:27:19.716855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Statistical Data Analysis\n\nThe next several section goes into further details regarding the kind of information that will be dealt with when training the model. Such that there are 41 different types of unqiue labels to go off of. Not to mention that there are only 39 percent of audio files that are manually verfied while only 61 percent of audio files are not verified. The notebook will also provide few audio file samples along with their spectrogram images. ","metadata":{}},{"cell_type":"markdown","source":"## Unique Labels","metadata":{}},{"cell_type":"code","source":"uniq_labels = train.label.unique()\nprint('There are a total of', len(uniq_labels), 'unique labels.\\n')\nprint(uniq_labels)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:19.718660Z","iopub.execute_input":"2022-04-28T07:27:19.719318Z","iopub.status.idle":"2022-04-28T07:27:19.731184Z","shell.execute_reply.started":"2022-04-28T07:27:19.719280Z","shell.execute_reply":"2022-04-28T07:27:19.730396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Label Distribution","metadata":{}},{"cell_type":"code","source":"print((train.manually_verified.value_counts() /len(train)).to_frame().T)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:19.732409Z","iopub.execute_input":"2022-04-28T07:27:19.732841Z","iopub.status.idle":"2022-04-28T07:27:19.743968Z","shell.execute_reply.started":"2022-04-28T07:27:19.732802Z","shell.execute_reply":"2022-04-28T07:27:19.742970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.manually_verified.value_counts().plot(kind='bar', xlabel='MGMT_value', ylabel='Count', \n                                     color=['#1E90FF', '#00C957'], edgecolor='black');","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:19.745174Z","iopub.execute_input":"2022-04-28T07:27:19.746050Z","iopub.status.idle":"2022-04-28T07:27:19.963492Z","shell.execute_reply.started":"2022-04-28T07:27:19.746011Z","shell.execute_reply":"2022-04-28T07:27:19.962738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring Samples","metadata":{}},{"cell_type":"markdown","source":"### Sample 1","metadata":{}},{"cell_type":"code","source":"gunshot = '../input/freesound-audio-tagging/audio_train/0048fd00.wav'\nipd.Audio(gunshot)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:19.964834Z","iopub.execute_input":"2022-04-28T07:27:19.966083Z","iopub.status.idle":"2022-04-28T07:27:19.980438Z","shell.execute_reply.started":"2022-04-28T07:27:19.966038Z","shell.execute_reply":"2022-04-28T07:27:19.979736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"signal, sr = librosa.load(gunshot)\nprint(type(signal))\nprint(type(sr))","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:19.983790Z","iopub.execute_input":"2022-04-28T07:27:19.984291Z","iopub.status.idle":"2022-04-28T07:27:20.778235Z","shell.execute_reply.started":"2022-04-28T07:27:19.984254Z","shell.execute_reply":"2022-04-28T07:27:20.777454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(signal.shape)\nprint(sr)\nprint(len(signal) / sr)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:20.779622Z","iopub.execute_input":"2022-04-28T07:27:20.780032Z","iopub.status.idle":"2022-04-28T07:27:20.785285Z","shell.execute_reply.started":"2022-04-28T07:27:20.779994Z","shell.execute_reply":"2022-04-28T07:27:20.784587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = [12,3])\nplt.subplot(2,1,1)\nplt.plot(signal)\nplt.subplot(2,1,2)\ninterval = range(2000, 3000)\nplt.plot(interval, signal[interval])\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:20.786597Z","iopub.execute_input":"2022-04-28T07:27:20.787009Z","iopub.status.idle":"2022-04-28T07:27:21.068560Z","shell.execute_reply.started":"2022-04-28T07:27:20.786972Z","shell.execute_reply":"2022-04-28T07:27:21.067819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x1 = librosa.feature.melspectrogram(y=signal, sr=22050)   \nx2 = librosa.power_to_db(x1, ref=np.max)   \n\nprint(x2.shape)\n\nlibrosa.display.specshow(x2, sr=22050, x_axis='time', y_axis='hz')\nplt.colorbar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:21.069931Z","iopub.execute_input":"2022-04-28T07:27:21.070347Z","iopub.status.idle":"2022-04-28T07:27:21.376972Z","shell.execute_reply.started":"2022-04-28T07:27:21.070305Z","shell.execute_reply":"2022-04-28T07:27:21.376290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sample 2","metadata":{}},{"cell_type":"code","source":"cello = '../input/freesound-audio-tagging/audio_train/0091fc7f.wav'\nipd.Audio(cello)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:21.378346Z","iopub.execute_input":"2022-04-28T07:27:21.378821Z","iopub.status.idle":"2022-04-28T07:27:21.405090Z","shell.execute_reply.started":"2022-04-28T07:27:21.378767Z","shell.execute_reply":"2022-04-28T07:27:21.404433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"signal, sr = librosa.load(cello)\nprint(type(signal))\nprint(type(sr))","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:21.406255Z","iopub.execute_input":"2022-04-28T07:27:21.406604Z","iopub.status.idle":"2022-04-28T07:27:21.630941Z","shell.execute_reply.started":"2022-04-28T07:27:21.406573Z","shell.execute_reply":"2022-04-28T07:27:21.630146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(signal.shape)\nprint(sr)\nprint(len(signal) / sr)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:21.632383Z","iopub.execute_input":"2022-04-28T07:27:21.632843Z","iopub.status.idle":"2022-04-28T07:27:21.637788Z","shell.execute_reply.started":"2022-04-28T07:27:21.632804Z","shell.execute_reply":"2022-04-28T07:27:21.637102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = [12,3])\nplt.subplot(2,1,1)\nplt.plot(signal)\nplt.subplot(2,1,2)\ninterval = range(2000, 3000)\nplt.plot(interval, signal[interval])\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:21.639225Z","iopub.execute_input":"2022-04-28T07:27:21.639669Z","iopub.status.idle":"2022-04-28T07:27:21.961833Z","shell.execute_reply.started":"2022-04-28T07:27:21.639630Z","shell.execute_reply":"2022-04-28T07:27:21.961152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x1 = librosa.feature.melspectrogram(y=signal, sr=22050)   \nx2 = librosa.power_to_db(x1, ref=np.max)   \n\nprint(x2.shape)\n\nlibrosa.display.specshow(x2, sr=22050, x_axis='time', y_axis='hz')\nplt.colorbar()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:21.962915Z","iopub.execute_input":"2022-04-28T07:27:21.963295Z","iopub.status.idle":"2022-04-28T07:27:22.288993Z","shell.execute_reply.started":"2022-04-28T07:27:21.963257Z","shell.execute_reply":"2022-04-28T07:27:22.288302Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Label Encoder","metadata":{}},{"cell_type":"code","source":"labels = np.unique(train.label.values)\nlabel_encoder = {label:i for i, label in enumerate(labels)}\nprint(label_encoder['Cello'])\nprint(label_encoder['Gunshot_or_gunfire'])","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:22.290084Z","iopub.execute_input":"2022-04-28T07:27:22.290446Z","iopub.status.idle":"2022-04-28T07:27:22.305853Z","shell.execute_reply.started":"2022-04-28T07:27:22.290402Z","shell.execute_reply":"2022-04-28T07:27:22.305124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Displaying Several Spectrogram Images","metadata":{}},{"cell_type":"code","source":"sample = train.sample(20)\n\nplt.figure(figsize=[20,9])\n\nfor i in range(20):\n    fname = train_path + sample.fname.iloc[i]\n    clip, sr = librosa.load(fname, sr=44100)\n    S1 = librosa.feature.melspectrogram(y=clip, sr=44100) \n    S2 = librosa.power_to_db(S1, ref=np.max)                \n    \n    plt.subplot(5, 4, i+1)\n    librosa.display.specshow(S2)\n    plt.title(f'{sample.label.iloc[i]} - {S2.shape[:2]} - {sample.fname.iloc[i]} ')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:22.307213Z","iopub.execute_input":"2022-04-28T07:27:22.307609Z","iopub.status.idle":"2022-04-28T07:27:25.363799Z","shell.execute_reply.started":"2022-04-28T07:27:22.307572Z","shell.execute_reply":"2022-04-28T07:27:25.363062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Generators\n\n'Freesound Melpec-128-512-2sec' is a pre-cropped dataset that I've included below. All spectrogram images of all audio files are included in this dataset. It's also worth noting that each images only has two second audio clips to balance out the model's performance. As mentioned earlier, I also set the image size of this model to 128 by 87 (The only model from the other that does not have an image size of 128 by 32).","metadata":{}},{"cell_type":"code","source":"SPEC_PATH = '../input/freesound-melpec-128-512-2sec/spectrograms'\nIMG_SIZE = (128,87)\n\nclass DataGenerator(keras.utils.Sequence):\n    \n    def __init__(self, df, batch_size=32, shuffle=True, is_train=True):\n        self.df = df\n        self.n = len(df)\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.is_train = is_train\n        self.on_epoch_end()\n        \n    def on_epoch_end(self):\n        self.indices = np.arange(self.n)\n        if self.shuffle == True:\n            np.random.shuffle(self.indices)   \n    \n    def __len__(self):\n        \n        return math.ceil( self.n / self.batch_size )\n    \n    def __getitem__(self, batch_index):\n        \n        start = batch_index * self.batch_size\n        end = (batch_index + 1) * self.batch_size\n        \n        indices = self.indices[start:end]\n        \n        return self.__data_generation(indices)\n    \n    def __data_generation(self, batch_indices):\n        batch_size = len(batch_indices)\n        \n        X = np.zeros(shape=(batch_size, IMG_SIZE[0], IMG_SIZE[1], 3))\n        y = np.zeros(batch_size)\n        \n        for i, idx in enumerate(batch_indices):\n            FILE = self.df.fname.values[idx]\n            LABEL = self.df.label.values[idx]\n            \n            SET = 'train_spec' if self.is_train else 'test_spec'\n            path = f'{SPEC_PATH}/{SET}/{FILE[:-4]}.npy'\n\n            try:\n                data_array = np.load(path)\n                resized = cv2.resize(data_array, (IMG_SIZE[1], IMG_SIZE[0]))\n                \n                for j in range(3):\n                    X[i,:,:,j] = resized \n                \n            except:\n                print('skipped')\n\n            if self.is_train:\n                y[i] = label_encoder[LABEL]\n\n        if self.is_train:    \n            return X, y\n        return X\n\n    \nGENERATOR_TEST = True\n\nif GENERATOR_TEST:\n    temp_gen = DataGenerator(train, batch_size=8, shuffle=False)\n    X,y = temp_gen.__getitem__(0)\n\n    print(X.shape)\n    print(y)\n    \n    librosa.display.specshow(X[0, :, :, 0])","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:25.365628Z","iopub.execute_input":"2022-04-28T07:27:25.365938Z","iopub.status.idle":"2022-04-28T07:27:25.535778Z","shell.execute_reply.started":"2022-04-28T07:27:25.365880Z","shell.execute_reply":"2022-04-28T07:27:25.534669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df, valid_df = train_test_split(train, test_size=0.2, random_state=1, stratify=train.label)\n\nprint(train_df.shape)\nprint(valid_df.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:25.541003Z","iopub.execute_input":"2022-04-28T07:27:25.541341Z","iopub.status.idle":"2022-04-28T07:27:25.589043Z","shell.execute_reply.started":"2022-04-28T07:27:25.541296Z","shell.execute_reply":"2022-04-28T07:27:25.587972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loader = DataGenerator(train_df, batch_size=64, shuffle=True)\nvalid_loader = DataGenerator(valid_df, batch_size=64, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:25.593982Z","iopub.execute_input":"2022-04-28T07:27:25.594411Z","iopub.status.idle":"2022-04-28T07:27:25.605726Z","shell.execute_reply.started":"2022-04-28T07:27:25.594361Z","shell.execute_reply":"2022-04-28T07:27:25.604610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TR_STEPS = len(train_loader)\nVA_STEPS = len(valid_loader)\n\nprint(TR_STEPS)\nprint(VA_STEPS)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:25.611745Z","iopub.execute_input":"2022-04-28T07:27:25.614450Z","iopub.status.idle":"2022-04-28T07:27:25.622000Z","shell.execute_reply.started":"2022-04-28T07:27:25.614396Z","shell.execute_reply":"2022-04-28T07:27:25.620928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CNN Model\n\nIn this section, I will apply transfer learning to this notebook and use a pre-trained model called EfficientNet_B1. This pre-trained model was the last transfer learning model worked on for this project in particular. The CNN model below will contain a few convolutional layers with a dropout rate of 0.45. ","metadata":{}},{"cell_type":"code","source":"ENB1_model = efn.EfficientNetB1(input_shape=(128,87,3), include_top=False, weights='imagenet')\nENB1_model.trainable = True","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:25.624017Z","iopub.execute_input":"2022-04-28T07:27:25.624422Z","iopub.status.idle":"2022-04-28T07:27:41.724337Z","shell.execute_reply.started":"2022-04-28T07:27:25.624376Z","shell.execute_reply":"2022-04-28T07:27:41.723072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cnn = Sequential([\n    ENB1_model,\n    \n    Flatten(),\n    \n    Dense(64, activation='relu'),\n    Dropout(0.45),\n    \n    Dense(32, activation='relu'),\n    Dropout(0.45),\n    \n    Dense(41, activation='softmax')\n])\n\ncnn.summary()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:41.726327Z","iopub.execute_input":"2022-04-28T07:27:41.726779Z","iopub.status.idle":"2022-04-28T07:27:42.515428Z","shell.execute_reply.started":"2022-04-28T07:27:41.726718Z","shell.execute_reply":"2022-04-28T07:27:42.514651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train Network\n\nIn the next three section below, I will begin training my model for a total of three training runs. Each training run the learning rate will be decreased starting from 0.001 to 0.00001. This model has ran a total of 40 epochs before stopping. ","metadata":{}},{"cell_type":"markdown","source":"### Training Run 1","metadata":{}},{"cell_type":"code","source":"opt = tf.keras.optimizers.Adam(0.001)\ncnn.compile(loss='sparse_categorical_crossentropy', optimizer=opt, metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:42.516956Z","iopub.execute_input":"2022-04-28T07:27:42.517415Z","iopub.status.idle":"2022-04-28T07:27:42.535117Z","shell.execute_reply.started":"2022-04-28T07:27:42.517374Z","shell.execute_reply":"2022-04-28T07:27:42.534363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n\nh1 = cnn.fit(train_loader, steps_per_epoch = TR_STEPS, epochs = 20, validation_data = valid_loader, \n             validation_steps = VA_STEPS, verbose = 1)\n","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:27:42.540331Z","iopub.execute_input":"2022-04-28T07:27:42.540547Z","iopub.status.idle":"2022-04-28T07:36:51.696104Z","shell.execute_reply.started":"2022-04-28T07:27:42.540521Z","shell.execute_reply":"2022-04-28T07:36:51.694708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def merge_history(hlist):\n    history = {}\n    for k in hlist[0].history.keys():\n        history[k] = sum([h.history[k] for h in hlist], [])\n    return history","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:36:51.697750Z","iopub.execute_input":"2022-04-28T07:36:51.698174Z","iopub.status.idle":"2022-04-28T07:36:51.704349Z","shell.execute_reply.started":"2022-04-28T07:36:51.698129Z","shell.execute_reply":"2022-04-28T07:36:51.703363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def vis_training(h, start=1):\n    epoch_range = range(start, len(h['loss'])+1)\n    s = slice(start-1, None)\n\n    plt.figure(figsize=[14,4])\n\n    n = int(len(h.keys()) / 2)\n\n    for i in range(n):\n        k = list(h.keys())[i]\n        plt.subplot(1,n,i+1)\n        plt.plot(epoch_range, h[k][s], label='Training')\n        plt.plot(epoch_range, h['val_' + k][s], label='Validation')\n        plt.xlabel('Epoch'); plt.ylabel(k); plt.title(k)\n        plt.grid()\n        plt.legend()\n\n    plt.tight_layout()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:36:51.705952Z","iopub.execute_input":"2022-04-28T07:36:51.706313Z","iopub.status.idle":"2022-04-28T07:36:51.716836Z","shell.execute_reply.started":"2022-04-28T07:36:51.706272Z","shell.execute_reply":"2022-04-28T07:36:51.715845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = merge_history([h1])\nvis_training(history)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:36:51.718457Z","iopub.execute_input":"2022-04-28T07:36:51.718799Z","iopub.status.idle":"2022-04-28T07:36:52.310162Z","shell.execute_reply.started":"2022-04-28T07:36:51.718759Z","shell.execute_reply":"2022-04-28T07:36:52.309430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**In the first training run, we can see a bit of noise occuring within out validation loss and validation accuracy. With the learning rate at 0.001, we can see that our validation accuracy started off small ranging between 0.10 and 0.44. According to the graphs displayed above, we can also see that a bit more training is needed.**","metadata":{}},{"cell_type":"markdown","source":"### Training Run 2","metadata":{}},{"cell_type":"code","source":"K.set_value(cnn.optimizer.learning_rate, 0.0001)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:36:52.313943Z","iopub.execute_input":"2022-04-28T07:36:52.316110Z","iopub.status.idle":"2022-04-28T07:36:52.325202Z","shell.execute_reply.started":"2022-04-28T07:36:52.316069Z","shell.execute_reply":"2022-04-28T07:36:52.324458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n\nh2 = cnn.fit(train_loader, steps_per_epoch = TR_STEPS, epochs = 10, validation_data = valid_loader, \n             validation_steps = VA_STEPS, verbose = 1)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:36:52.333580Z","iopub.execute_input":"2022-04-28T07:36:52.335475Z","iopub.status.idle":"2022-04-28T07:40:39.943761Z","shell.execute_reply.started":"2022-04-28T07:36:52.335433Z","shell.execute_reply":"2022-04-28T07:40:39.942962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def vis_training(h, start=1):\n    epoch_range = range(start, len(h['loss'])+1)\n    s = slice(start-1, None)\n\n    plt.figure(figsize=[14,4])\n\n    n = int(len(h.keys()) / 2)\n\n    for i in range(n):\n        k = list(h.keys())[i]\n        plt.subplot(1,n,i+1)\n        plt.plot(epoch_range, h[k][s], label='Training')\n        plt.plot(epoch_range, h['val_' + k][s], label='Validation')\n        plt.xlabel('Epoch'); plt.ylabel(k); plt.title(k)\n        plt.grid()\n        plt.legend()\n\n    plt.tight_layout()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:40:39.945559Z","iopub.execute_input":"2022-04-28T07:40:39.945811Z","iopub.status.idle":"2022-04-28T07:40:39.955045Z","shell.execute_reply.started":"2022-04-28T07:40:39.945776Z","shell.execute_reply":"2022-04-28T07:40:39.954315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = merge_history([h1, h2])\nvis_training(history, start=10)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:40:39.956257Z","iopub.execute_input":"2022-04-28T07:40:39.956651Z","iopub.status.idle":"2022-04-28T07:40:40.359946Z","shell.execute_reply.started":"2022-04-28T07:40:39.956613Z","shell.execute_reply":"2022-04-28T07:40:40.359241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**In the second training run, we can see a bit of noise still occuring within out validation loss and validation accuracy, but it begins to slowly level out after the 20th epoch. With the learning rate at 0.0001, we can see that our validation accuracy beginning to increase ranging between 0.47 and 0.53. According to the graphs displayed above, we can also see that a bit more training is still needed.**","metadata":{}},{"cell_type":"markdown","source":"### Training Run 3","metadata":{}},{"cell_type":"code","source":"K.set_value(cnn.optimizer.learning_rate, 0.00001)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:40:40.361166Z","iopub.execute_input":"2022-04-28T07:40:40.361490Z","iopub.status.idle":"2022-04-28T07:40:40.368682Z","shell.execute_reply.started":"2022-04-28T07:40:40.361452Z","shell.execute_reply":"2022-04-28T07:40:40.367973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time \n\nh3 = cnn.fit(train_loader, steps_per_epoch = TR_STEPS, epochs = 10, validation_data = valid_loader, \n             validation_steps = VA_STEPS, verbose = 1)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:40:40.374216Z","iopub.execute_input":"2022-04-28T07:40:40.374397Z","iopub.status.idle":"2022-04-28T07:44:46.791685Z","shell.execute_reply.started":"2022-04-28T07:40:40.374374Z","shell.execute_reply":"2022-04-28T07:44:46.791022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def vis_training(h, start=1):\n    epoch_range = range(start, len(h['loss'])+1)\n    s = slice(start-1, None)\n\n    plt.figure(figsize=[14,4])\n\n    n = int(len(h.keys()) / 2)\n\n    for i in range(n):\n        k = list(h.keys())[i]\n        plt.subplot(1,n,i+1)\n        plt.plot(epoch_range, h[k][s], label='Training')\n        plt.plot(epoch_range, h['val_' + k][s], label='Validation')\n        plt.xlabel('Epoch'); plt.ylabel(k); plt.title(k)\n        plt.grid()\n        plt.legend()\n\n    plt.tight_layout()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:44:46.795574Z","iopub.execute_input":"2022-04-28T07:44:46.795774Z","iopub.status.idle":"2022-04-28T07:44:46.803092Z","shell.execute_reply.started":"2022-04-28T07:44:46.795748Z","shell.execute_reply":"2022-04-28T07:44:46.802280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = merge_history([h1, h2, h3])\nvis_training(history, start=10)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:44:46.804567Z","iopub.execute_input":"2022-04-28T07:44:46.805163Z","iopub.status.idle":"2022-04-28T07:44:47.201452Z","shell.execute_reply.started":"2022-04-28T07:44:46.805127Z","shell.execute_reply":"2022-04-28T07:44:47.200716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**For the last training run seen here, we can see a bit of noise begininng to disappear after the 20th epoch for both the validation loss and validation accuracy. With the learning rate at 0.00001, we can see that our validation accuracy started off where we left off from the second training run and began to slowly increase. The difference seen here is that validation accuracy begin to level out ranging only between 0.53 and 0.54. According to the graphs displayed above, we can also see that not much more training is needed since both the training and validation accuracy is begininng to level out.**","metadata":{}},{"cell_type":"markdown","source":"## Saving Model\n\nFor our final step for the final model, the cells below are created to help save our best model. ","metadata":{}},{"cell_type":"code","source":"cnn.save(f'Freesound_Audio_EfficientNet_B1_v01.h5')","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:44:47.202918Z","iopub.execute_input":"2022-04-28T07:44:47.203183Z","iopub.status.idle":"2022-04-28T07:44:48.164645Z","shell.execute_reply.started":"2022-04-28T07:44:47.203146Z","shell.execute_reply":"2022-04-28T07:44:48.163839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Test DataFrame","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv('../input/freesound-audio-tagging/sample_submission.csv')\n\ntest_loader = DataGenerator(test, batch_size=64, shuffle=False, is_train=False)\n\nprobs = cnn.predict(test_loader)\nprint(probs.shape)","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:44:48.166761Z","iopub.execute_input":"2022-04-28T07:44:48.167047Z","iopub.status.idle":"2022-04-28T07:45:25.583743Z","shell.execute_reply.started":"2022-04-28T07:44:48.167008Z","shell.execute_reply":"2022-04-28T07:45:25.582988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(probs[0, :].round(2))","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:45:25.585019Z","iopub.execute_input":"2022-04-28T07:45:25.585261Z","iopub.status.idle":"2022-04-28T07:45:25.593665Z","shell.execute_reply.started":"2022-04-28T07:45:25.585226Z","shell.execute_reply":"2022-04-28T07:45:25.592711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluating using MAP@3\n\nAfter saving the model and applying it to the test dataset. We will then perform what is called 'Mean Average Precision @3'. The evaluation is performed using the formula displayed below:\n\n$$MAP@3 = \\frac{1}{U} \\sum{u=1}^{U} \\sum{k=1}^{min(n,3)} P(k)$$\n\nWithin the formula seen above, 'U' is the number of scored audio files in the test data, while 'P(k)' is the precision at cutoff 'k', and 'n' is the number predictions per audio file.\n\nAfter applying the formula to the code cells below, we can either submit 1 label prediction or we can submit a total of 3 label prediction, hence the title 'Mean Average Precision @3'.","metadata":{}},{"cell_type":"markdown","source":"## Submit Top 1 Prediction","metadata":{}},{"cell_type":"code","source":"submission_top1 = test.copy()\n\nN = len(test)\nfor i in range(N):\n    p = probs[i, :]\n    idx = np.argmax(p)\n    submission_top1.label[i] = labels[idx]\n\nsubmission_top1.to_csv('submission_top1.csv', index=False, header=True)\n\nsubmission_top1.head()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:45:25.595314Z","iopub.execute_input":"2022-04-28T07:45:25.596047Z","iopub.status.idle":"2022-04-28T07:45:26.370543Z","shell.execute_reply.started":"2022-04-28T07:45:25.595864Z","shell.execute_reply":"2022-04-28T07:45:26.369759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submit Top 3 Predictions","metadata":{}},{"cell_type":"code","source":"submission_top3 = test.copy()\n\nN = len(test)\nfor i in range(N):\n    p = probs[i, :]\n    idx = np.argsort(-p)[:3]\n    top3 = labels[idx]\n    submission_top3.label[i] = ' '.join(top3)\n\nsubmission_top3.to_csv('submission_top3.csv', index=False, header=True)\nsubmission_top3.head()","metadata":{"execution":{"iopub.status.busy":"2022-04-28T07:45:26.371986Z","iopub.execute_input":"2022-04-28T07:45:26.372225Z","iopub.status.idle":"2022-04-28T07:45:27.285536Z","shell.execute_reply.started":"2022-04-28T07:45:26.372191Z","shell.execute_reply":"2022-04-28T07:45:27.284817Z"},"trusted":true},"execution_count":null,"outputs":[]}]}