{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Gravitational Wave Analysis and Detection","metadata":{}},{"cell_type":"markdown","source":"In this notebook, an implementation is made on the analysis and detection of gravitational waves using spectrogram representation of the samples in the <a href='https://www.kaggle.com/competitions/g2net-gravitational-wave-detection/data'>G2NET dataset</a>.","metadata":{}},{"cell_type":"markdown","source":"In this implementation, the spectrogram of the wave signals are computed with the Constant Q Transform algorthim offered by <a href=\"https://kinwaicheuk.github.io/nnAudio/index.html\">nnAudio</a> module.","metadata":{}},{"cell_type":"code","source":"!pip install nnAudio","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:08.844829Z","iopub.execute_input":"2022-05-07T23:32:08.8452Z","iopub.status.idle":"2022-05-07T23:32:18.373761Z","shell.execute_reply.started":"2022-05-07T23:32:08.84516Z","shell.execute_reply":"2022-05-07T23:32:18.37287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below the relevant modules for this implementation are imported. A thing to note is that <a href='https://docs.cupy.dev/en/stable/'>cupy</a> and <a href='https://docs.rapids.ai/api/cudf/stable/'>cudf</a> modules are used instead of numpy and pandas, as they are GPU variants and execute much more faster.","metadata":{}},{"cell_type":"code","source":"import cupy as cp\nimport cudf as cd\nimport os, tqdm, torch, torchvision, gc, random\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score, confusion_matrix, ConfusionMatrixDisplay\nfrom nnAudio.features import CQT","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2022-05-07T23:32:18.377432Z","iopub.execute_input":"2022-05-07T23:32:18.37766Z","iopub.status.idle":"2022-05-07T23:32:18.383396Z","shell.execute_reply.started":"2022-05-07T23:32:18.377633Z","shell.execute_reply":"2022-05-07T23:32:18.38261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loading and Exploratory Data Analysis","metadata":{}},{"cell_type":"markdown","source":"The dataset consist of '.npy' files aranged in a nested manner inside the train folder. Here the os module is used to iterate efficiently through the nested files to get the file paths and their file names corresponding to the variables *data_paths* and *data_ids* in the program below.","metadata":{}},{"cell_type":"code","source":"def get_data_info(root_path: str):\n    data_paths = []\n    data_ids = []\n    for path, subdirs, files in tqdm.tqdm(os.walk(root_path)):\n        for name in files:\n            data_paths.append(os.path.join(path, name))\n            data_ids.append(name.replace('.npy', ''))\n            \n    return data_paths, data_ids\n\ntrain_dir = '../input/g2net-gravitational-wave-detection'\ndata_paths, data_ids = get_data_info(os.path.join(train_dir, 'train'))","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:18.384701Z","iopub.execute_input":"2022-05-07T23:32:18.385178Z","iopub.status.idle":"2022-05-07T23:32:23.945013Z","shell.execute_reply.started":"2022-05-07T23:32:18.385141Z","shell.execute_reply":"2022-05-07T23:32:23.944179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, the 'training_labels.csv' file is loaded. This spreadsheet file has two columns 'id' and 'target'. The 'id' column corresponds to the file names without their file extensions and the 'target' column contains annotations of 0s and 1s corresponding to whether gravitational wave signal is abscent in the signal waveform or not respectively.","metadata":{}},{"cell_type":"code","source":"training_labels = cd.read_csv(os.path.join(train_dir, 'training_labels.csv'))\ntraining_labels.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:23.946341Z","iopub.execute_input":"2022-05-07T23:32:23.948021Z","iopub.status.idle":"2022-05-07T23:32:23.98041Z","shell.execute_reply.started":"2022-05-07T23:32:23.947976Z","shell.execute_reply":"2022-05-07T23:32:23.979738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The code below checks for the presence of *NA* values in the 'train_labels.csv' file.","metadata":{}},{"cell_type":"code","source":"training_labels.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:23.982929Z","iopub.execute_input":"2022-05-07T23:32:23.983325Z","iopub.status.idle":"2022-05-07T23:32:23.993522Z","shell.execute_reply.started":"2022-05-07T23:32:23.983288Z","shell.execute_reply":"2022-05-07T23:32:23.992443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No *NA* values were found in the labels file. Next a countplot is made to visualise the level of target imbalance in the labels file if any.","metadata":{}},{"cell_type":"code","source":"target_0 = training_labels[training_labels['target'] == 0]\ntarget_1 = training_labels[training_labels['target'] == 1]\nprint(f'length of target 0: {len(target_0)} \\nlength of target 1: {len(target_1)}')\n\nax = sns.countplot(x=training_labels['target'].values.get())","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:23.994807Z","iopub.execute_input":"2022-05-07T23:32:23.995075Z","iopub.status.idle":"2022-05-07T23:32:24.246709Z","shell.execute_reply.started":"2022-05-07T23:32:23.995041Z","shell.execute_reply":"2022-05-07T23:32:24.246018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the plot, it is observed that the target classes are relatively balanced with 280070 0s and 279930 1s.\n\nBelow the waveform of some given samples are visualised in a graphical plot. Each sample consists of three waveforms corresponding to simulated signal + noise of three different Interferometers corresponding to Handford LIGO, Livington LIGO and Virgo.","metadata":{}},{"cell_type":"code","source":"plot_idx = [0, len(data_paths)//3, len(data_paths)//5, len(data_paths)-1]\nsensors = ['handford LIGO', 'livington LIGO', 'virgo']\nsample_freq = 2048\ntimesteps = cp.arange(0, 2, 1/sample_freq).get()\n\nfor i, sample_idx in enumerate(plot_idx):\n    signals = cp.load(data_paths[sample_idx]).get()\n    target = training_labels[training_labels['id'] == data_ids[sample_idx]]['target'].values[0]\n    fig = plt.figure(figsize=(20, 4))\n    for j, signal in enumerate(signals):\n        plt.plot(timesteps, signal, label=f'sensor {sensors[j]}')\n        plt.title(f'wave {sample_idx+1}. Target:{target}')\n        plt.legend(loc='upper right')\n    plt.plot()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:24.247994Z","iopub.execute_input":"2022-05-07T23:32:24.248231Z","iopub.status.idle":"2022-05-07T23:32:25.66393Z","shell.execute_reply.started":"2022-05-07T23:32:24.248198Z","shell.execute_reply":"2022-05-07T23:32:25.663246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The histograms of the selected samples from above are plotted in the next code cell so as to get an overview of the distribution and width of variance of the signal readings of each detector.","metadata":{}},{"cell_type":"code","source":"fig, axs = plt.subplots(1, 4, figsize=(20, 7))\n\nfor i, sample_idx in enumerate(plot_idx):\n    axs[i].hist(cp.load(data_paths[sample_idx]).get()[0], label=f'{sensors[0]}')\n    axs[i].hist(cp.load(data_paths[sample_idx]).get()[1], label=f'{sensors[1]}')\n    axs[i].hist(cp.load(data_paths[sample_idx]).get()[2], label=f'{sensors[2]}')\n    axs[i].legend()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:25.665019Z","iopub.execute_input":"2022-05-07T23:32:25.665707Z","iopub.status.idle":"2022-05-07T23:32:26.49551Z","shell.execute_reply.started":"2022-05-07T23:32:25.665666Z","shell.execute_reply":"2022-05-07T23:32:26.494819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is observed that the distribution of the signal readings of the Virgo detector look skewered with relatively very small variance compared to the Handford and Livington detectors.","metadata":{}},{"cell_type":"markdown","source":"In the code cell below, the *SignalDataset()* class is instantiated and is a derived class from the *torch.utils.data.Dataset()* class. This class object is responsible for lazy loading of the dataset. The lazy loading approach was adopted because the dataset was too big and could not fit into the available RAM, making eager loading of large number of samples difficult.\n\nThe class returns the spectrogram of any given sample and it's corresponding target class via the *__getitem__()* method. In the *SignalDataset()* class, an object of the *CQT()* class from the *nnAudio* module is instantiated. maximum and minimum frequencies of 20Hz and 500Hz respectively are assumed, the sampling rate (sr) attribute is also set to 2048Hz and the hop length to 64.\n\nThe spectrogram is computed in the *preprocess()* method, this method takes the single sample and horizontally stacks the signals of the three differenct detectors into a single vector and computes the spectrogram. The idea behind stacking all signals in a sample horizontally is that a given sample whose target is 1 corresponding to presence of gravitational wave will have similar repeating frequencies along some given time intervals in the spectrogram.","metadata":{}},{"cell_type":"code","source":"class SignalDataset(Dataset):\n    def __init__(self, paths, ids=None):\n        self.paths = paths\n        self.ids = ids\n        self.CQT_transform = CQT(\n            sr=2048,\n            fmin=20,           \n            fmax=500,\n            hop_length=64,\n            verbose=False\n        )\n        \n    def __getitem__(self, idx):\n        X = cp.load(self.paths[idx])\n        X = self.preprocess(X)\n        if self.ids != None:\n            y = training_labels[training_labels['id'] == self.ids[idx]]['target'].values[0]\n            return X, y.get()\n        return X\n        \n    def preprocess(self, signals):\n        for i, signal in enumerate(signals):\n            signals[i] /= signal.max()\n        signals = cp.hstack(signals)\n        signal = torch.FloatTensor(signals)\n        #shape: C, Freq, Time\n        spectrogram = self.CQT_transform(signal)\n        return spectrogram\n    \n    def __len__(self):\n        return len(self.paths)","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:26.496618Z","iopub.execute_input":"2022-05-07T23:32:26.497341Z","iopub.status.idle":"2022-05-07T23:32:26.507372Z","shell.execute_reply.started":"2022-05-07T23:32:26.497293Z","shell.execute_reply":"2022-05-07T23:32:26.50661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The generated spectrograms of some selected samples are plotted and viewed with the *imshow()* method of the *matplotlib.plyplot()* class below","metadata":{}},{"cell_type":"code","source":"training_dataset = SignalDataset(data_paths, data_ids)\n\nfor i in plot_idx:\n    sample_cqt, target = training_dataset[i]\n    fig = plt.figure(figsize=(15, 7))\n    plt.imshow(sample_cqt.permute(1, 2, 0))\n    plt.title(f'Spectrogram of time domain signal. Target:{target}')\n    plt.xlabel('time')\n    plt.ylabel('frequency')\n    plt.plot()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:26.508787Z","iopub.execute_input":"2022-05-07T23:32:26.509076Z","iopub.status.idle":"2022-05-07T23:32:27.363032Z","shell.execute_reply.started":"2022-05-07T23:32:26.50904Z","shell.execute_reply":"2022-05-07T23:32:27.362201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Modelling with Artifical Neural Networks","metadata":{}},{"cell_type":"markdown","source":"In this section the dataset is modelled with the <a href='https://arxiv.org/abs/1512.03385'>ResNet18 </a> neural network architecture with the <a href='https://pytorch.org/docs/stable/index.html'>Pytorch </a> framework. The ResNet18 architecure in this implementation is modified to have a single input channel at first layer, dropout inbetween blocks and a last fc layer with output feature size of 1","metadata":{}},{"cell_type":"code","source":"torch.manual_seed(3407)","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:27.364512Z","iopub.execute_input":"2022-05-07T23:32:27.364761Z","iopub.status.idle":"2022-05-07T23:32:27.370611Z","shell.execute_reply.started":"2022-05-07T23:32:27.364724Z","shell.execute_reply":"2022-05-07T23:32:27.369824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ClassifierNet(torch.jit.ScriptModule):\n    \n    def __init__(self, input_channels, output_features, dropout):\n        super(ClassifierNet, self).__init__()\n        self.input_channels = input_channels\n        self.output_features = output_features\n        self.dropout = dropout\n        \n        self.resnet = torchvision.models.resnet18(pretrained=False)\n\n        self.resnet.conv1 = nn.Conv2d(\n            self.input_channels, \n            64, \n            kernel_size=(7, 7), \n            stride=(2, 2), \n            padding=(3, 3), \n            bias=False)\n\n        self.resnet.fc = nn.Linear(\n            in_features=512, \n            out_features=self.output_features, \n            bias=True)\n        \n        self.add_dropout()\n        \n        self.sigmoid = nn.Sigmoid()\n\n        \n    def add_dropout(self, n_layers=4):\n        for i in range(1, n_layers+1):\n            layer = list(getattr(self.resnet, f'layer{i}'))\n            layer.append(nn.Dropout(self.dropout))\n            setattr(self.resnet, f'layer{i}', nn.Sequential(*layer))\n        \n    \n    @torch.jit.script_method\n    def forward(self, _input):\n        output = self.resnet(_input)\n        output = self.sigmoid(output)\n        return output\n","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:27.371949Z","iopub.execute_input":"2022-05-07T23:32:27.372189Z","iopub.status.idle":"2022-05-07T23:32:27.38587Z","shell.execute_reply.started":"2022-05-07T23:32:27.372157Z","shell.execute_reply":"2022-05-07T23:32:27.38523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next the pipeline class is defined for the neural network model training, testing and saving. The *ClassifierPipeline()* class takes as arguments the defined NN model object, the loss function, the optimizer and the device to perform computing on.","metadata":{}},{"cell_type":"code","source":"class ClassifierPipeline:\n    def __init__(self, classifier, lossfunc, optimizer, device='cpu'):\n        \n        self.classifier = classifier.to(device)\n        self.lossfunc = lossfunc\n        self.optimizer = optimizer\n        self.device = device\n        self.classifier.apply(self.init_weights)\n    \n    \n    def init_weights(self, m):\n        if isinstance(m, nn.Linear) or isinstance(m, nn.Conv2d):\n            nn.init.xavier_uniform_(m.weight)\n            if m.bias:\n                m.bias.data.fill_(0.01)\n    \n    \n    def save_model(self, dirname='./model_params', \n                   filename='gravitational_wave_detection_model.pth.tar'):\n        if not os.path.isdir(dirname):os.mkdir(dirname)\n        state_dicts = {\n            'model_params':self.classifier.state_dict(),\n            'optimizer_params':self.optimizer.state_dict(),\n        }\n        return torch.save(state_dicts, os.path.join(dirname, filename))\n        \n    \n    def train(self, dataloader, epochs=1, verbose=False):\n        self.classifier.train()\n        losses, accuracies = [], []\n        for epoch in range(epochs):\n            mean_loss, mean_accuracy = 0, 0\n            for idx, (signals, target) in tqdm.tqdm(enumerate(dataloader)):\n                signals, target = signals.float().to(self.device), target.float().to(self.device)\n                optimizer.zero_grad()\n                pred = self.classifier(signals)\n                pred = pred.reshape(-1)\n                loss = self.lossfunc(pred, target)\n                loss.backward()\n                self.optimizer.step()\n                mean_loss += loss.item()\n                target = target.to('cpu').numpy()\n                pred = torch.round(pred).detach().to('cpu').numpy()\n                accuracy = accuracy_score(target, pred)\n                mean_accuracy += accuracy\n            mean_loss = mean_loss / (idx+1)\n            mean_accuracy = mean_accuracy / (idx+1)\n            losses.append(mean_loss)\n            accuracies.append(mean_accuracy)\n            if verbose:\n                print(f'epoch: {epoch} \\nLoss: {mean_loss}, \\naccuracy: {mean_accuracy}')\n        return losses, accuracies\n    \n    \n    def test(self, dataloader, verbose=False, return_cm=False):\n        actual_vals, pred_vals = [], []\n        self.classifier.eval()\n        mean_loss, mean_accuracy = 0, 0\n        with torch.no_grad():\n            for idx, (signals, target) in tqdm.tqdm(enumerate(dataloader)):\n                signals, target = signals.float().to(self.device), target.float().to(self.device)\n                pred = self.classifier(signals)\n                pred = pred.reshape(-1)\n                loss = self.lossfunc(pred, target)\n                mean_loss += loss.item()\n                target = target.to('cpu').numpy()\n                pred = torch.round(pred).detach().to('cpu').numpy()\n                accuracy = accuracy_score(target, pred)\n                             \n                if return_cm:\n                    actual_vals.append(target)\n                    pred_vals.append(pred)\n                    \n                mean_accuracy += accuracy\n            mean_loss = mean_loss / (idx+1)\n            mean_accuracy = mean_accuracy / (idx+1)\n            if verbose: print(f'Loss: {mean_loss}, \\naccuracy: {mean_accuracy}')\n                \n        if return_cm: \n            actual_vals = torch.cat([torch.from_numpy(i) for i in actual_vals]).numpy().reshape(-1)\n            pred_vals = torch.cat([torch.from_numpy(i) for i in pred_vals]).numpy().reshape(-1)\n            cm = confusion_matrix(actual_vals, pred_vals)\n            return mean_loss, mean_accuracy, cm\n        return mean_loss, mean_accuracy","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:27.387176Z","iopub.execute_input":"2022-05-07T23:32:27.387622Z","iopub.status.idle":"2022-05-07T23:32:27.409146Z","shell.execute_reply.started":"2022-05-07T23:32:27.387586Z","shell.execute_reply":"2022-05-07T23:32:27.408379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The *Model_CFG* class is defined to hold configuration values for model, optimizer, training and testing proccess, etc.","metadata":{}},{"cell_type":"code","source":"class Model_CFG:\n    input_channels = 1\n    output_features = 1\n    learning_rate = 5e-2\n    weight_decay = 5e-7\n    dropout = 0.2\n    epochs=30\n    device = 'cuda' if torch.cuda.is_available() else 'cpu'\n    batch_size = 400","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:27.412083Z","iopub.execute_input":"2022-05-07T23:32:27.412563Z","iopub.status.idle":"2022-05-07T23:32:27.421133Z","shell.execute_reply.started":"2022-05-07T23:32:27.412525Z","shell.execute_reply":"2022-05-07T23:32:27.420371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next the *data_paths* and *data_ids* list objects are split into training and testing lists which are used to create two objects of the *SignalDataset* class for training and testing. These created *SignalDataset* objects are used to instantiate two *Dataloader* objects for training and testing.\n\nDue to limitations in computing, 40000 samples are used for training and 2000 for testing.","metadata":{}},{"cell_type":"code","source":"training_paths, testing_paths, training_ids, testing_ids = train_test_split(\n    data_paths, data_ids, shuffle=True, random_state=42, train_size=0.8\n)\n\ntrain_n_samples, test_n_samples = 50000, 5000\n\ntraining_dataset = SignalDataset(training_paths[:train_n_samples], training_ids[:train_n_samples])\ntraining_dataloader = DataLoader(training_dataset, batch_size=Model_CFG.batch_size, shuffle=True, num_workers=0)\n\ntesting_dataset = SignalDataset(testing_paths[:test_n_samples], testing_ids[:test_n_samples])\ntesting_dataloader = DataLoader(testing_dataset, batch_size=Model_CFG.batch_size, shuffle=True, num_workers=0)","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:27.423837Z","iopub.execute_input":"2022-05-07T23:32:27.424644Z","iopub.status.idle":"2022-05-07T23:32:27.887836Z","shell.execute_reply.started":"2022-05-07T23:32:27.424609Z","shell.execute_reply":"2022-05-07T23:32:27.887039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below, the *ClassifierNet* neural network object is instantiated as well as the loss function and the optimizer.\n\nThe loss fuction used in this implentation is the <a href='https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.html'>Binary Cross Entropy Loss function</a> and the optimizer used is the <a href='https://pytorch.org/docs/stable/optim.html'>Adam Optimizer</a>","metadata":{}},{"cell_type":"code","source":"classifier = ClassifierNet(Model_CFG.input_channels, Model_CFG.output_features, Model_CFG.dropout)     \n        \nlossfunc = nn.BCELoss()\n\noptimizer = torch.optim.Adam(\n    classifier.parameters(), lr=Model_CFG.learning_rate,\n    weight_decay=Model_CFG.weight_decay\n)","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:27.889316Z","iopub.execute_input":"2022-05-07T23:32:27.889605Z","iopub.status.idle":"2022-05-07T23:32:28.369946Z","shell.execute_reply.started":"2022-05-07T23:32:27.889566Z","shell.execute_reply":"2022-05-07T23:32:28.36922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The neural network model architecture is printed below for proper visualisation.","metadata":{}},{"cell_type":"code","source":"print(classifier)","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:32:28.371367Z","iopub.execute_input":"2022-05-07T23:32:28.371634Z","iopub.status.idle":"2022-05-07T23:32:28.378409Z","shell.execute_reply.started":"2022-05-07T23:32:28.371581Z","shell.execute_reply":"2022-05-07T23:32:28.376773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The pipeline class is then instantiated and the *classifier*, *lossfunc* and *optimizer* objects are passed. A learning rate scheduler object is also initialised to reduce the learning rate when the measured metric (accuracy in this case) has plateaued after 10 cycles.\n\nNext training and testing commences and the metric values are stored in lists to be visualised later on.","metadata":{}},{"cell_type":"code","source":"classifier_pipeline = ClassifierPipeline(\n    classifier, lossfunc, optimizer, device=Model_CFG.device\n)\nlr_scheduler = torch.optim.lr_scheduler.StepLR(\n    classifier_pipeline.optimizer, step_size=10, gamma=0.1, verbose=True)\n\ntraining_losses, training_accuracies = [], []\ntesting_losses, testing_accuracies = [], []\n\nbest_acc = 0\nfor i in range(Model_CFG.epochs):\n    print(f'epoch: {i}')\n    train_loss, train_acc = classifier_pipeline.train(training_dataloader)\n    training_losses.append(train_loss[0])\n    training_accuracies.append(train_acc[0])\n    print(f'training loss: {train_loss[0]}, \\ntraining accuracy: {round(train_acc[0]*100, 4)}%')\n    \n    test_loss, test_acc = classifier_pipeline.test(testing_dataloader)\n    testing_losses.append(test_loss)\n    testing_accuracies.append(test_acc)\n    print(f'testing loss: {test_loss}, \\ntesting accuracy: {round(test_acc*100, 4)}% \\n')\n    \n    lr_scheduler.step()\n    if test_acc > best_acc:\n        best_acc = test_acc\n        classifier_pipeline.save_model()\n        print(f'model saved at epoch {i}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The metric values obtained from training and testing are graphically plotted and visualised. These plots are used to get an overview of the overall improvement of the model over the epochs of training.","metadata":{}},{"cell_type":"code","source":"fig, axs = plt.subplots(1, 2, figsize=(20, 6))\n\naxs[0].plot(training_losses, label='training loss')\naxs[0].plot(testing_losses, label='testing loss')\naxs[0].set_xlabel('epochs')\naxs[0].set_ylabel('BCE Loss')\naxs[0].set_title('Loss Plot')\naxs[0].legend()\n\naxs[1].plot(training_accuracies, label='training accuracy')\naxs[1].plot(testing_accuracies, label='testing accuracy')\naxs[1].set_xlabel('epochs')\naxs[1].set_ylabel('accuracy')\naxs[1].set_title('Accuracy Plot')\naxs[1].legend()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:36:32.73627Z","iopub.status.idle":"2022-05-07T23:36:32.736771Z","shell.execute_reply.started":"2022-05-07T23:36:32.736521Z","shell.execute_reply":"2022-05-07T23:36:32.736546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Evaluation","metadata":{}},{"cell_type":"markdown","source":"Finally, the model is evaluated on an evaluation set and the confusion matrix for classification is computed and plotted as shown below.","metadata":{}},{"cell_type":"code","source":"\neval_n_samples = 1000\neval_dataset = SignalDataset(\n    testing_paths[test_n_samples:test_n_samples+eval_n_samples],\n    testing_ids[test_n_samples:test_n_samples+eval_n_samples])\n\neval_dataloader = DataLoader(eval_dataset, batch_size=Model_CFG.batch_size, shuffle=True, num_workers=0)\n\n\neval_loss, eval_acc, cm = classifier_pipeline.test(eval_dataloader, return_cm=True)\n\nprint(f'evaluation loss: {eval_loss}, \\nevaluation accuracy: {round(eval_acc*100, 4)}% \\n')\n\ncm_disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=[0, 1])\ncm_disp.plot()","metadata":{"execution":{"iopub.status.busy":"2022-05-07T23:36:32.738174Z","iopub.status.idle":"2022-05-07T23:36:32.739191Z","shell.execute_reply.started":"2022-05-07T23:36:32.73895Z","shell.execute_reply":"2022-05-07T23:36:32.738974Z"},"trusted":true},"execution_count":null,"outputs":[]}]}