{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<div class=\"list-group\" id=\"list-tab\" role=\"tablist\">\n  <h3 class=\"list-group-item list-group-item-action active\" data-toggle=\"list\"  role=\"tab\" aria-controls=\"home\" align='center'>Table of Content</h3>\n\n* [About the Competition](#section-one)\n* [Data Provided](#section-two)\n* [Import Libraries](#section-three)  \n* [Load Data](#section-four)     \n* [Class Distribution](#section-five)\n* [Data Analysis](#section-six)     ","metadata":{"execution":{"iopub.status.busy":"2021-07-05T19:13:18.696993Z","iopub.execute_input":"2021-07-05T19:13:18.697469Z","iopub.status.idle":"2021-07-05T19:13:18.705283Z","shell.execute_reply.started":"2021-07-05T19:13:18.697425Z","shell.execute_reply":"2021-07-05T19:13:18.703921Z"}}},{"cell_type":"markdown","source":"<a id=\"section-one\"></a>\n# About the competition\n\n`G2Net is a network of Gravitational Wave, Geophysics and Machine Learning.` \n\nVia an Action from COST (European Cooperation in Science and Technology), a funding agency for research and innovation networks, G2Net aims to create a broad network of scientists. From four different areas of expertise, namely GW physics, Geophysics, Computing Science and Robotics, these scientists have agreed on a common goal of tackling challenges in data analysis and noise characterization for GW detectors.\n\n`In this competition, we aim to detect GW signals from the mergers of binary black holes.` \n\nSpecifically, we will build a model to analyze simulated GW time-series data from a network of Earth-based detectors.\n\nThe series of images below are taken from the 2015 [paper](https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.116.061102) paper announcing the discovery of gravitational waves from a pair of merging black holes.\n\n<img src=\"https://storage.googleapis.com/kaggle-media/competitions/G2Net-gravitational-waves/800px-LIGO_measurement_of_gravitational_waves.svg.png\"/>\n\n****What are we predicting?****\n\nWe need to predict the probability whether the given observation contains a gravitational wave","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-two\"></a>\n# Data Provided\n\nIn this competition we are provided with a training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers:\n\n1. LIGO Hanford\n2. LIGO Livingston\n3. Virgo\n\n\nEach time series contains either detector noise or detector noise plus a simulated gravitational wave signal. \n\n`The task is to identify when a signal is present in the data (target=1).`\n\nEach data sample (npy file) contains 3 time series (1 for each detector) and each spans 2 sec and is sampled at 2,048 Hz.\n\n##### Files\n`train folder` - the training set files, one npy file per observation; labels are provided in a files shown below\n\n`test folder` - the test set files; you must predict the probability that the observation contains a gravitational wave\n\n`training_labels.csv` - target values of whether the associated signal contains a gravitational wave\n\n`sample_submission.csv` - a sample submission file in the correct format","metadata":{}},{"cell_type":"markdown","source":"<a id=\"section-three\"></a>\n# Import Libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom glob import glob\nimport random\nfrom colorama import Fore, Back, Style\n# Setting plot styling.\nplt.style.use('ggplot')\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2021-07-14T20:48:05.829903Z","iopub.execute_input":"2021-07-14T20:48:05.830347Z","iopub.status.idle":"2021-07-14T20:48:06.702852Z","shell.execute_reply.started":"2021-07-14T20:48:05.830252Z","shell.execute_reply":"2021-07-14T20:48:06.701749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-four\"></a>\n# Load Data","metadata":{}},{"cell_type":"code","source":"labels = pd.read_csv(\"../input/g2net-gravitational-wave-detection/training_labels.csv\")\n\nprint(Fore.BLUE + \"Dataset has \",Style.RESET_ALL + \"{} Observations\".format(labels.shape[0]))\n\nprint(Fore.GREEN + \"First 5 Observations:\",Style.RESET_ALL)\ndisplay(labels.head())","metadata":{"execution":{"iopub.status.busy":"2021-07-14T20:48:06.704118Z","iopub.execute_input":"2021-07-14T20:48:06.704469Z","iopub.status.idle":"2021-07-14T20:48:07.187356Z","shell.execute_reply.started":"2021-07-14T20:48:06.704437Z","shell.execute_reply":"2021-07-14T20:48:07.186407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# build a training dataframe for all the available .npy files along with their path\n\n# path of the files\npaths = glob(\"../input/g2net-gravitational-wave-detection/train/*/*/*/*\")\n\n# list of ids of .npy files \nids = [path.split(\"/\")[-1].split(\".\")[0] for path in paths]\n\n# data frame containing paths and ids of .npy files \npath_df = pd.DataFrame({\"path\":paths,\"id\":ids})\n\n# merge the dataframe built above with the dataset having target\ntrain_df = pd.merge(left=labels,right=path_df,on=\"id\")\n\n# this would a comprehensive df which would include \"id\",\"target\" and \"path\" for each of the .npy file in train folder\ndisplay(train_df.head())\n\n# lets confirm whether the dataframe built above has expected no. of rows(560000)\nprint(Fore.BLUE + \"No.of rows in the merged dataframe:\",train_df.shape[0],Style.RESET_ALL)","metadata":{"execution":{"iopub.status.busy":"2021-07-14T20:48:07.189773Z","iopub.execute_input":"2021-07-14T20:48:07.190037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# segregate dataframes for individual classes\ntarget_1 = train_df[train_df.target==1]\ntarget_0 = train_df[train_df.target==0]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-five\"></a>\n# Class Distribution","metadata":{}},{"cell_type":"markdown","source":"****Both the labels (target=0,target=1) have equal distribution in the dataset****","metadata":{}},{"cell_type":"code","source":"print(\"Class Distribution:\\n\",labels.target.value_counts())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize class distribution\nsns.countplot(x=\"target\", data=labels)\nplt.title(\"Class Distribution\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"section-six\"></a>\n# Data Analysis (Let's pick up a random file and analyze it)","metadata":{}},{"cell_type":"markdown","source":"**Note**\n1. We would call the 3 different serieses as SITE-1, SITE-2 & SITE-3 in this notebook!","metadata":{}},{"cell_type":"code","source":"# visualize the randomly selected series\ndef plot_series(series,plot,target):\n    if plot == \"box\" or plot == \"kde\":\n        plt.figure(figsize = (20,2))    \n    else:\n        plt.figure(figsize = (15,12))    \n    \n    for idx in range(3):\n        if plot == \"box\":\n            plt.subplot(1,3,idx+1)            \n            sns.boxplot(series[idx:idx+1],color = 'b')  \n            \n        elif plot == \"kde\":\n            plt.subplot(1,3,idx+1)            \n            sns.kdeplot(series[idx],color = 'r', shade=True,lw=2, alpha=0.5)\n        else:\n            plt.subplot(3,1,idx+1)            \n            plt.plot(series[idx:idx+1].T,color = 'g')\n            plt.title(\"\\nSite-\" + str(idx+1))        \n            \n    if plot == \"box\":    \n        plt.suptitle(\"Box Plots(target = \" + target + \")\")\n    elif plot == \"kde\":    \n        plt.suptitle(\"Probablity Distribution Plots(target = \" + target + \")\")\n    else:    \n        plt.suptitle(\"Time Distribution of Signals - Spans 2 sec, Sampled at 2,048 Hz(target = \" + target + \")\")\n\n        \n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pick a random series(target=1)\ntarget_1 = target_1.sample(1).path.values[0]\n\npos = np.load(target_1)\n\nprint(Fore.BLUE + \"Shape of the selected signal:\",pos.shape,Style.RESET_ALL)\nprint(\"\\n\\n\")\n\nplot_series(pos,\"time\",\"1\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# pick a random series(target=1)\ntarget_0 = target_0.sample(1).path.values[0]\n\nneg = np.load(target_0)\nprint(Fore.BLUE + \"Shape of the selected signal:\",neg.shape,Style.RESET_ALL)\nprint(\"\\n\\n\")\n\nplot_series(neg,\"plot\",\"0\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Points to Note:**\n1. We have 560000 files, each file has dimension of 3 * 4096, this turns out to be a huge time series\n\n2. There are some differences in the plots for series where signal is present or absent\n   The series with no signals have bigger fluctuations, while the series for which signal is absent,\n   have smaller more consistent ones.","metadata":{}},{"cell_type":"code","source":"# Probability Distribution plots for target == 1 (Signal is missing)\nplot_series(pos,\"box\",\"1\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Probability Distribution plots for target == 0 (Signal is missing)\nplot_series(neg,\"box\",\"0\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Points to Note:**\n1. The 3 sites have fairly similar distribution for both the class types.\n2. Only visible difference is with the third site, which seems to have difference in outliers than the other sites.","metadata":{}},{"cell_type":"code","source":"# Probability Distribution plots for target == 1 (Signal is present)\nplot_series(pos,\"kde\",\"1\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Probability Distribution plots for target == 0 (Signal is missing)\nplot_series(neg,\"kde\",\"0\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Note**\n1. KDE plots for both the classes looks almost similar for Site-3\n2. Site-2 has little bit more variation for target =0\n3. Site-1 has more variation for target=1 ","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nfrom keras.utils import Sequence\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten, Conv1D, MaxPool1D, BatchNormalization\nfrom keras.optimizers import RMSprop,Adam\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data generator\nWill be used for real-time data feeding to your Keras model.","metadata":{}},{"cell_type":"code","source":"class DataGenerator(Sequence):\n    def __init__(self, path, list_IDs, data, batch_size):\n        self.path = path\n        self.list_IDs = list_IDs\n        self.data = data\n        self.batch_size = batch_size\n        self.indexes = np.arange(len(self.list_IDs))\n        \n    def __len__(self):\n        len_ = int(len(self.list_IDs)/self.batch_size)\n        if len_*self.batch_size < len(self.list_IDs):\n            len_ += 1\n        return len_\n    \n    def __getitem__(self, index):\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n        list_IDs_temp = [self.list_IDs[k] for k in indexes]\n        X, y = self.__data_generation(list_IDs_temp)\n        return X, y\n    \n    def __data_generation(self, list_IDs_temp):\n        X = np.zeros((self.batch_size, 3, 4096))\n        y = np.zeros((self.batch_size, 1))\n        for i, ID in enumerate(list_IDs_temp):\n            id_ = self.data.loc[ID, 'id']\n            file = id_+'.npy'\n            path_in = '/'.join([self.path, id_[0], id_[1], id_[2]])+'/'\n            data_array = np.load(path_in+file)\n            data_array = (data_array-data_array.mean())/data_array.std()\n            X[i, ] = data_array\n            y[i, ] = self.data.loc[ID, 'target']\n        return X, y","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission = pd.read_csv('../input/g2net-gravitational-wave-detection/sample_submission.csv')\ntrain_idx =  labels['id'].values\ny = labels['target'].values\ntest_idx = sample_submission['id'].values\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_idx, train_Valx = train_test_split(list(labels.index), test_size=0.33, random_state=2021)\ntest_idx = list(sample_submission.index)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_generator = DataGenerator('/kaggle/input/g2net-gravitational-wave-detection/train/', train_idx, labels, 64)\nval_generator = DataGenerator('/kaggle/input/g2net-gravitational-wave-detection/train/', train_Valx, labels, 64)\ntest_generator = DataGenerator('/kaggle/input/g2net-gravitational-wave-detection/test/', test_idx, sample_submission, 64)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = Sequential()\nmodel.add(Conv1D(64, input_shape=(3, 4096,), kernel_size=3, activation='relu'))\nmodel.add(BatchNormalization())\nmodel.add(Flatten())\nmodel.add(Dense(64, activation='relu'))\nmodel.add(Dense(1, activation='sigmoid'))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.compile(optimizer = Adam(lr=2e-4),loss='binary_crossentropy',metrics=['acc'])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.summary()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit_generator(generator=train_generator, validation_data=val_generator, epochs = 1, workers=4)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you can see, we called from model the fit_generator method instead of fit, where we just had to give our training generator as one of the arguments. Keras takes care of the rest!","metadata":{}},{"cell_type":"code","source":"predict = model.predict_generator(test_generator, verbose=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission['target'] = predict[:len(sample_submission)]\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_submission.to_csv('submission.csv', index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}