{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Intro\n\nMachine learning can help astronomers sort big data recorded in space exploration.\n\n**Gravitational Wave:**\n\nVery simply,\n`A gravitational wave is like ripples in space time. It is usually caused by some of the most violent and energetic processes in the Universe.`\n\nThey are invisible but incredibly fast. \n\n**Why we need to detect GW?**\n\nDetecting and analyzing the information carried by gravitational waves is allowing us to observe the Universe in a way never before possible, providing astronomers and other scientists with their first glimpses of literally un-seeable wonders.\n When a gravitational wave passes by Earth, it squeezes and stretches space. LIGO can detect this squeezing and stretching. Each LIGO observatory has two “arms” that are each more than 2 miles (4 kilometers) long.\n \n##### **above info collected via some very rough googling**\n \n### Goal\n\nThe GW was first detected/seen when two blackholes merged into one big black whole back in Sept, 2015.\n\nIn this competition, our goal is to detect GW signals from the mergers of binary black holes.\n\nI am going to document this process as I start with zero idea about any of these.\n\nThe folowing two kernels have been my overall inspiration to understand this whole task to my best capability. These are really well explained and worth mentioning.\n\n- [kernel 1](https://www.kaggle.com/pranay1990/pranay-g2net-gw)\n- [kernel 2](https://github.com/SiddharthPatel45/gravitational-wave-detection/blob/main/code/gw-detection-modelling.ipynb)\n- [kernel 3](https://www.kaggle.com/atamazian/nnaudio-constant-q-transform-demonstration/comments)\n\nThank you for sharing your work.","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"!pip install -q nnAudio","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:32:57.783361Z","iopub.execute_input":"2021-09-16T05:32:57.783726Z","iopub.status.idle":"2021-09-16T05:33:08.841526Z","shell.execute_reply.started":"2021-09-16T05:32:57.783612Z","shell.execute_reply":"2021-09-16T05:33:08.840391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pylab as plt\nimport seaborn as sns\nfrom glob import glob\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import Conv2D\nfrom tensorflow.keras.layers import MaxPooling2D\nfrom tensorflow.keras.layers import GlobalAveragePooling2D\nfrom tensorflow.keras.layers import Flatten\nfrom tensorflow.keras.applications import EfficientNetB0\nfrom tensorflow.keras.metrics import AUC\n\nimport librosa.display\nimport torch\n\n# this is used for Contant Q Transform\nfrom nnAudio.Spectrogram import CQT1992v2\nfrom tensorflow.keras.applications import EfficientNetB0 as efn","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-09-16T05:33:08.844426Z","iopub.execute_input":"2021-09-16T05:33:08.844840Z","iopub.status.idle":"2021-09-16T05:33:20.533664Z","shell.execute_reply.started":"2021-09-16T05:33:08.844741Z","shell.execute_reply":"2021-09-16T05:33:20.532627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Files\n\n```\ntrain: contains one npy file per observation.\n\ntest: we have to predict the probability whether or not the observation contains a gravitational wave.\n\ntraining_labels: If associated signal contains a GW or not.\n```\n\nThe waves detected by GW detectors have noises in output signals. So researchers need to find out if the output signal is only **noise** or **signal+noise**.\n\n\nWe are provided with a training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo). \n\nThis problem is seen as a binary classification problem, if signal is detected or not. \n ","metadata":{}},{"cell_type":"markdown","source":"## Data Exploration","metadata":{}},{"cell_type":"code","source":"train_label_dataset = pd.read_csv(\"../input/g2net-gravitational-wave-detection/training_labels.csv\")\ntrain_label_dataset.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:33:20.535706Z","iopub.execute_input":"2021-09-16T05:33:20.536134Z","iopub.status.idle":"2021-09-16T05:33:20.977152Z","shell.execute_reply.started":"2021-09-16T05:33:20.536087Z","shell.execute_reply":"2021-09-16T05:33:20.976197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_label_dataset.shape","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:33:20.979984Z","iopub.execute_input":"2021-09-16T05:33:20.980410Z","iopub.status.idle":"2021-09-16T05:33:20.987927Z","shell.execute_reply.started":"2021-09-16T05:33:20.980367Z","shell.execute_reply":"2021-09-16T05:33:20.986611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"when `target = 1` it means that the signal (GW) is present","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=train_label_dataset, x=\"target\")","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:33:20.989958Z","iopub.execute_input":"2021-09-16T05:33:20.990531Z","iopub.status.idle":"2021-09-16T05:33:21.261557Z","shell.execute_reply.started":"2021-09-16T05:33:20.990479Z","shell.execute_reply":"2021-09-16T05:33:21.260556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_label_dataset['target'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:33:21.265059Z","iopub.execute_input":"2021-09-16T05:33:21.266443Z","iopub.status.idle":"2021-09-16T05:33:21.283366Z","shell.execute_reply.started":"2021-09-16T05:33:21.266400Z","shell.execute_reply":"2021-09-16T05:33:21.282309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking for null values:","metadata":{}},{"cell_type":"code","source":"train_label_dataset.isnull().sum() # no null","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:33:21.285056Z","iopub.execute_input":"2021-09-16T05:33:21.285590Z","iopub.status.idle":"2021-09-16T05:33:21.347582Z","shell.execute_reply.started":"2021-09-16T05:33:21.285548Z","shell.execute_reply":"2021-09-16T05:33:21.346402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_path = glob('../input/g2net-gravitational-wave-detection/train/*/*/*/*')","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:33:21.349754Z","iopub.execute_input":"2021-09-16T05:33:21.350074Z","iopub.status.idle":"2021-09-16T05:34:49.680282Z","shell.execute_reply.started":"2021-09-16T05:33:21.350034Z","shell.execute_reply":"2021-09-16T05:34:49.679319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 560,000 **.npy** files in the `train` set","metadata":{}},{"cell_type":"code","source":"len(train_path)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:49.681890Z","iopub.execute_input":"2021-09-16T05:34:49.684470Z","iopub.status.idle":"2021-09-16T05:34:49.695581Z","shell.execute_reply.started":"2021-09-16T05:34:49.684439Z","shell.execute_reply":"2021-09-16T05:34:49.694267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we want to take a took at how these data looks:\n\nlets see how data at index 3 looks","metadata":{}},{"cell_type":"code","source":"explore_sample_3 = np.load(train_path[3])\nexplore_sample_3","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:49.701579Z","iopub.execute_input":"2021-09-16T05:34:49.701829Z","iopub.status.idle":"2021-09-16T05:34:49.721810Z","shell.execute_reply.started":"2021-09-16T05:34:49.701801Z","shell.execute_reply":"2021-09-16T05:34:49.720572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that there are 3 rows to the data. This represents data extracted by 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo) respectively.","metadata":{}},{"cell_type":"code","source":"explore_sample_3.shape","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:49.723554Z","iopub.execute_input":"2021-09-16T05:34:49.723993Z","iopub.status.idle":"2021-09-16T05:34:49.753819Z","shell.execute_reply.started":"2021-09-16T05:34:49.723921Z","shell.execute_reply":"2021-09-16T05:34:49.752280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"each index of `explore_sample_3` has **4096** columns","metadata":{}},{"cell_type":"code","source":"print(len(explore_sample_3[0]), len(explore_sample_3[1]), len(explore_sample_3[2]))","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:49.756193Z","iopub.execute_input":"2021-09-16T05:34:49.756796Z","iopub.status.idle":"2021-09-16T05:34:49.764735Z","shell.execute_reply.started":"2021-09-16T05:34:49.756751Z","shell.execute_reply":"2021-09-16T05:34:49.763279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# just a tensor representation\ntf.convert_to_tensor(explore_sample_3[0])","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:49.767198Z","iopub.execute_input":"2021-09-16T05:34:49.767586Z","iopub.status.idle":"2021-09-16T05:34:51.769324Z","shell.execute_reply.started":"2021-09-16T05:34:49.767545Z","shell.execute_reply":"2021-09-16T05:34:51.766946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring the sample data with Librosa\n\nLibrosa is a python package for music and audio analysis, more about this awesome library can be found [here](https://librosa.org/doc/latest/index.html).\n\nThere is a very good kernel that can be found [here](https://www.kaggle.com/hinamimi/visualization-gravitational-wave-with-librosa).\nIt has really great demonstration of how to use Librosa.\n\nNow first I will find the `label` (id) of `explore_sample_3` from the `training_label.csv` dataset. After that I can find whether the target is 1 or 0.\n\n- 0 = negative sample\n- 1 = possotive sample","metadata":{}},{"cell_type":"code","source":"train_path[3]","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:51.771411Z","iopub.execute_input":"2021-09-16T05:34:51.771834Z","iopub.status.idle":"2021-09-16T05:34:51.781028Z","shell.execute_reply.started":"2021-09-16T05:34:51.771790Z","shell.execute_reply":"2021-09-16T05:34:51.779624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"the value of `train_path` at index 3 looks like:'\n\n'../input/g2net-gravitational-wave-detection/train/7/7/7/77727f6826.npy'\n\nSo, we know that the Id of `explore_sample_3` is **77727f6826**. To extract the Id the following code snippet has been written. \n\n","metadata":{}},{"cell_type":"code","source":"rind = train_path[3].rindex('/') # last index where the character '/' appeared\nextracted_id_for_explore_sample_3 = train_path[3][rind+1:].replace('.npy', '') # replaced .npy\nextracted_id_for_explore_sample_3","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:51.783590Z","iopub.execute_input":"2021-09-16T05:34:51.783935Z","iopub.status.idle":"2021-09-16T05:34:51.793964Z","shell.execute_reply.started":"2021-09-16T05:34:51.783895Z","shell.execute_reply":"2021-09-16T05:34:51.792646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that it is a positive sample","metadata":{}},{"cell_type":"code","source":"train_label_dataset[train_label_dataset['id']==extracted_id_for_explore_sample_3]['target']","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:51.796084Z","iopub.execute_input":"2021-09-16T05:34:51.797024Z","iopub.status.idle":"2021-09-16T05:34:51.882555Z","shell.execute_reply.started":"2021-09-16T05:34:51.796933Z","shell.execute_reply":"2021-09-16T05:34:51.881191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"positive_sample = explore_sample_3\n# index 1 od train_path has a target of 0 so it is a negative sample.\nnegative_sample = np.load(train_path[1])\nnegative_sample","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:51.885105Z","iopub.execute_input":"2021-09-16T05:34:51.885728Z","iopub.status.idle":"2021-09-16T05:34:51.903779Z","shell.execute_reply.started":"2021-09-16T05:34:51.885670Z","shell.execute_reply":"2021-09-16T05:34:51.902860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"samples = (positive_sample, negative_sample)\ntargets = (1, 0)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:51.906830Z","iopub.execute_input":"2021-09-16T05:34:51.907061Z","iopub.status.idle":"2021-09-16T05:34:51.914118Z","shell.execute_reply.started":"2021-09-16T05:34:51.907035Z","shell.execute_reply":"2021-09-16T05:34:51.912237Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using librosa.display() to view raw waves: \n\nKernel: https://www.kaggle.com/hinamimi/visualization-gravitational-wave-with-librosa","metadata":{}},{"cell_type":"code","source":"colors = (\"red\", \"green\", \"blue\")\nsignal_names = (\"LIGO Hanford\", \"LIGO Livingston\", \"Virgo\")\n\nfor x, i in tqdm(zip(samples, targets)):\n    figure = plt.figure(figsize=(16, 7))\n    figure.suptitle(f'Raw wave (target={i})', fontsize=20)\n    # range is 3 because we have 3 different rows for each interferometers\n    for j in range(3):\n        axes = figure.add_subplot(3, 1, j+1)\n        librosa.display.waveshow(x[j], sr=2048, ax=axes, color=colors[j])\n        axes.set_title(signal_names[j], fontsize=12)\n        axes.set_xlabel('Time[sec]')\n    plt.tight_layout()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:51.915889Z","iopub.execute_input":"2021-09-16T05:34:51.917177Z","iopub.status.idle":"2021-09-16T05:34:53.670917Z","shell.execute_reply.started":"2021-09-16T05:34:51.917135Z","shell.execute_reply":"2021-09-16T05:34:53.669855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(positive_sample[0,:])","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:53.672797Z","iopub.execute_input":"2021-09-16T05:34:53.673135Z","iopub.status.idle":"2021-09-16T05:34:54.178412Z","shell.execute_reply.started":"2021-09-16T05:34:53.673094Z","shell.execute_reply":"2021-09-16T05:34:54.177409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Working with a cleaner datset by merging `train` and `training_labels` datasets","metadata":{}},{"cell_type":"code","source":"pd.set_option('display.max_colwidth',None)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:54.180177Z","iopub.execute_input":"2021-09-16T05:34:54.180519Z","iopub.status.idle":"2021-09-16T05:34:54.186520Z","shell.execute_reply.started":"2021-09-16T05:34:54.180476Z","shell.execute_reply":"2021-09-16T05:34:54.185123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ids = []\nfor files in train_path:\n    ids.append(files[files.rindex('/')+1:].replace('.npy',''))\ndf = pd.DataFrame({\"id\":ids,\"path\":train_path})\ndf = pd.merge(df, train_label_dataset, on='id')","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:54.188797Z","iopub.execute_input":"2021-09-16T05:34:54.189307Z","iopub.status.idle":"2021-09-16T05:34:55.285882Z","shell.execute_reply.started":"2021-09-16T05:34:54.189245Z","shell.execute_reply":"2021-09-16T05:34:55.284673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:55.287682Z","iopub.execute_input":"2021-09-16T05:34:55.288101Z","iopub.status.idle":"2021-09-16T05:34:55.301559Z","shell.execute_reply.started":"2021-09-16T05:34:55.288056Z","shell.execute_reply":"2021-09-16T05:34:55.300220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.shape","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:55.303605Z","iopub.execute_input":"2021-09-16T05:34:55.304007Z","iopub.status.idle":"2021-09-16T05:34:55.316492Z","shell.execute_reply.started":"2021-09-16T05:34:55.303965Z","shell.execute_reply":"2021-09-16T05:34:55.315112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing\n\n#### Core Idea: \nIf any particular frequency is widespread in the signal or not. If true then our required GW is present.\n\nApproach:\n\n- convert original signal  -->  spectrogram signal\n- coverting from time domain  --> frequency domain\n    - done using **[Constant Q transformation](https://en.wikipedia.org/wiki/Constant-Q_transform)**\n    - **[kernel](https://www.kaggle.com/atamazian/nnaudio-constant-q-transform-demonstration/comments)**\n","metadata":{}},{"cell_type":"markdown","source":"I refer to the kernel [here](https://www.kaggle.com/atamazian/nnaudio-constant-q-transform-demonstration/comments) to define my CQT.\n\nPlease have a look. ","metadata":{}},{"cell_type":"code","source":"# CQT\ntransform = CQT1992v2(sr=2048,        # sample rate\n                fmin=20,        # min freq\n                fmax=500,      # max freq\n                hop_length=64,  # hop length\n                verbose=False)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:55.318803Z","iopub.execute_input":"2021-09-16T05:34:55.319356Z","iopub.status.idle":"2021-09-16T05:34:55.369797Z","shell.execute_reply.started":"2021-09-16T05:34:55.319312Z","shell.execute_reply":"2021-09-16T05:34:55.368767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the Cqt function\n# preprocess function\ndef preprocess_function_cqt(path):\n    signal = np.load(path.numpy())\n    # there are 3 signal as explained before for each interferometers\n    for i in range(signal.shape[0]):\n        # normalize signal\n        signal[i] /= np.max(signal[i])\n    # horizontal stack\n    signal = np.hstack(signal)\n    # tensor conversion\n    signal = torch.from_numpy(signal).float()\n    # getting the image from CQT transform\n    image = transform(signal)\n    # converting to array from tensor\n    image = np.array(image)\n    # transpose the image to get right orientation\n    image = np.transpose(image,(1,2,0))\n    \n    # conver the image to tf.tensor and return\n    return tf.convert_to_tensor(image)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:55.371538Z","iopub.execute_input":"2021-09-16T05:34:55.371868Z","iopub.status.idle":"2021-09-16T05:34:55.381566Z","shell.execute_reply.started":"2021-09-16T05:34:55.371831Z","shell.execute_reply":"2021-09-16T05:34:55.380402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image = preprocess_function_cqt(tf.convert_to_tensor(df['path'][2]))\nprint(image.shape)\nplt.imshow(image)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:55.383367Z","iopub.execute_input":"2021-09-16T05:34:55.383874Z","iopub.status.idle":"2021-09-16T05:34:55.754378Z","shell.execute_reply.started":"2021-09-16T05:34:55.383813Z","shell.execute_reply":"2021-09-16T05:34:55.753333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"for a different path","metadata":{}},{"cell_type":"code","source":"image = preprocess_function_cqt(tf.convert_to_tensor(df['path'][5069]))\nprint(image.shape)\nplt.imshow(image)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:55.760892Z","iopub.execute_input":"2021-09-16T05:34:55.761553Z","iopub.status.idle":"2021-09-16T05:34:56.035770Z","shell.execute_reply.started":"2021-09-16T05:34:55.761508Z","shell.execute_reply":"2021-09-16T05:34:56.034809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"we can see that the image shape is **(56, 193, 1)**, so thats our shpa eof the input.","metadata":{}},{"cell_type":"code","source":"input_shape = (56, 193, 1)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.037560Z","iopub.execute_input":"2021-09-16T05:34:56.037945Z","iopub.status.idle":"2021-09-16T05:34:56.042895Z","shell.execute_reply.started":"2021-09-16T05:34:56.037903Z","shell.execute_reply":"2021-09-16T05:34:56.041807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess_function_parse_tf(path, y=None):\n    [x] = tf.py_function(func=preprocess_function_cqt, inp=[path], Tout=[tf.float32])\n    x = tf.ensure_shape(x, input_shape)\n    if y is None:\n        return x\n    else:\n        return x,y","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.044702Z","iopub.execute_input":"2021-09-16T05:34:56.045443Z","iopub.status.idle":"2021-09-16T05:34:56.054054Z","shell.execute_reply.started":"2021-09-16T05:34:56.045399Z","shell.execute_reply":"2021-09-16T05:34:56.052988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# preprocess_function_parse_tf(tf.convert_to_tensor(df['path'][5069]))","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.055909Z","iopub.execute_input":"2021-09-16T05:34:56.056253Z","iopub.status.idle":"2021-09-16T05:34:56.064125Z","shell.execute_reply.started":"2021-09-16T05:34:56.056211Z","shell.execute_reply":"2021-09-16T05:34:56.063075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### I will define the `training` and `validation` dataset from `df`","metadata":{}},{"cell_type":"code","source":"X = df['id']\ny = df['target'].astype('int8').values","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.065640Z","iopub.execute_input":"2021-09-16T05:34:56.066402Z","iopub.status.idle":"2021-09-16T05:34:56.075415Z","shell.execute_reply.started":"2021-09-16T05:34:56.066355Z","shell.execute_reply":"2021-09-16T05:34:56.074359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.077406Z","iopub.execute_input":"2021-09-16T05:34:56.077840Z","iopub.status.idle":"2021-09-16T05:34:56.088180Z","shell.execute_reply.started":"2021-09-16T05:34:56.077795Z","shell.execute_reply":"2021-09-16T05:34:56.086743Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train, x_valid, y_train, y_valid = train_test_split(X, y, random_state = 42, stratify = y)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.090429Z","iopub.execute_input":"2021-09-16T05:34:56.091110Z","iopub.status.idle":"2021-09-16T05:34:56.504394Z","shell.execute_reply.started":"2021-09-16T05:34:56.091067Z","shell.execute_reply":"2021-09-16T05:34:56.503359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size = 250","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.506171Z","iopub.execute_input":"2021-09-16T05:34:56.506515Z","iopub.status.idle":"2021-09-16T05:34:56.511501Z","shell.execute_reply.started":"2021-09-16T05:34:56.506466Z","shell.execute_reply":"2021-09-16T05:34:56.510459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_npy_filepath(id_, is_train=True):\n    path = ''\n    if is_train:\n        return f'../input/g2net-gravitational-wave-detection/train/{id_[0]}/{id_[1]}/{id_[2]}/{id_}.npy'\n    else:\n        return f'../input/g2net-gravitational-wave-detection/test/{id_[0]}/{id_[1]}/{id_[2]}/{id_}.npy'","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.513464Z","iopub.execute_input":"2021-09-16T05:34:56.514167Z","iopub.status.idle":"2021-09-16T05:34:56.523640Z","shell.execute_reply.started":"2021-09-16T05:34:56.514124Z","shell.execute_reply":"2021-09-16T05:34:56.522645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = tf.data.Dataset.from_tensor_slices((x_train.apply(get_npy_filepath).values, y_train))\n# shuffle the dataset\ntrain_dataset = train_dataset.shuffle(len(x_train))\ntrain_dataset = train_dataset.map(preprocess_function_parse_tf, num_parallel_calls=tf.data.AUTOTUNE)\ntrain_dataset = train_dataset.batch(batch_size)\ntrain_dataset = train_dataset.prefetch(tf.data.AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:56.525499Z","iopub.execute_input":"2021-09-16T05:34:56.525952Z","iopub.status.idle":"2021-09-16T05:34:57.101448Z","shell.execute_reply.started":"2021-09-16T05:34:56.525911Z","shell.execute_reply":"2021-09-16T05:34:57.100506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_dataset = tf.data.Dataset.from_tensor_slices((x_valid.apply(get_npy_filepath).values, y_valid))\nvalid_dataset = valid_dataset.map(preprocess_function_parse_tf, num_parallel_calls=tf.data.AUTOTUNE)\nvalid_dataset = valid_dataset.batch(batch_size)\nvalid_dataset = valid_dataset.prefetch(tf.data.AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:57.103197Z","iopub.execute_input":"2021-09-16T05:34:57.103635Z","iopub.status.idle":"2021-09-16T05:34:57.254975Z","shell.execute_reply.started":"2021-09-16T05:34:57.103565Z","shell.execute_reply":"2021-09-16T05:34:57.253881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:57.256865Z","iopub.execute_input":"2021-09-16T05:34:57.257258Z","iopub.status.idle":"2021-09-16T05:34:57.265618Z","shell.execute_reply.started":"2021-09-16T05:34:57.257214Z","shell.execute_reply":"2021-09-16T05:34:57.264483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_dataset","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:57.267366Z","iopub.execute_input":"2021-09-16T05:34:57.267944Z","iopub.status.idle":"2021-09-16T05:34:57.279713Z","shell.execute_reply.started":"2021-09-16T05:34:57.267902Z","shell.execute_reply":"2021-09-16T05:34:57.278595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating the Model","metadata":{}},{"cell_type":"code","source":"train_dataset.take(1)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:57.281683Z","iopub.execute_input":"2021-09-16T05:34:57.282014Z","iopub.status.idle":"2021-09-16T05:34:57.292395Z","shell.execute_reply.started":"2021-09-16T05:34:57.281974Z","shell.execute_reply":"2021-09-16T05:34:57.291049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Model from [here](https://github.com/SiddharthPatel45/gravitational-wave-detection/blob/main/code/gw-detection-modelling.ipynb) ~","metadata":{}},{"cell_type":"code","source":"# Instantiate the Sequential model\nmodel_cnn = Sequential(name='CNN_model')\n\n# Add the first Convoluted2D layer w/ input_shape & MaxPooling2D layer followed by that\nmodel_cnn.add(Conv2D(filters=16,\n                     kernel_size=3,\n                     input_shape=input_shape,\n                     activation='relu',\n                     name='Conv_01'))\nmodel_cnn.add(MaxPooling2D(pool_size=2, name='Pool_01'))\n\n# Second pair of Conv1D and MaxPooling1D layers\nmodel_cnn.add(Conv2D(filters=32,\n                     kernel_size=3,\n                     input_shape=input_shape,\n                     activation='relu',\n                     name='Conv_02'))\nmodel_cnn.add(MaxPooling2D(pool_size=2, name='Pool_02'))\n\n# Third pair of Conv1D and MaxPooling1D layers\nmodel_cnn.add(Conv2D(filters=64,\n                     kernel_size=3,\n                     input_shape=input_shape,\n                     activation='relu',\n                     name='Conv_03'))\nmodel_cnn.add(MaxPooling2D(pool_size=2, name='Pool_03'))\n\n# Add the Flatten layer\nmodel_cnn.add(Flatten(name='Flatten'))\n\n# Add the Dense layers\nmodel_cnn.add(Dense(units=512,\n                activation='relu',\n                name='Dense_01'))\nmodel_cnn.add(Dense(units=64,\n                activation='relu',\n                name='Dense_02'))\n\n# Add the final Output layer\nmodel_cnn.add(Dense(1, activation='sigmoid', name='Output'))","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:57.294106Z","iopub.execute_input":"2021-09-16T05:34:57.294463Z","iopub.status.idle":"2021-09-16T05:34:57.720086Z","shell.execute_reply.started":"2021-09-16T05:34:57.294421Z","shell.execute_reply":"2021-09-16T05:34:57.719052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_cnn.summary()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:57.721605Z","iopub.execute_input":"2021-09-16T05:34:57.721901Z","iopub.status.idle":"2021-09-16T05:34:57.739036Z","shell.execute_reply.started":"2021-09-16T05:34:57.721863Z","shell.execute_reply":"2021-09-16T05:34:57.737997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_cnn.compile(optimizer=Adam(learning_rate=0.0001),\n                  loss='binary_crossentropy',\n                  metrics=[[AUC(), 'accuracy']])","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:57.742356Z","iopub.execute_input":"2021-09-16T05:34:57.742622Z","iopub.status.idle":"2021-09-16T05:34:58.032186Z","shell.execute_reply.started":"2021-09-16T05:34:57.742596Z","shell.execute_reply":"2021-09-16T05:34:58.031195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit the data\nhistory_cnn = model_cnn.fit(x=train_dataset,\n                            epochs=3,\n                            validation_data=valid_dataset,\n                            batch_size=batch_size,\n                            verbose=1)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T05:34:58.033716Z","iopub.execute_input":"2021-09-16T05:34:58.034068Z","iopub.status.idle":"2021-09-16T08:56:01.490063Z","shell.execute_reply.started":"2021-09-16T05:34:58.034027Z","shell.execute_reply":"2021-09-16T08:56:01.486841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"saving the model after training is complete","metadata":{}},{"cell_type":"code","source":"model_cnn.save('./model/cnn_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:56:01.494738Z","iopub.execute_input":"2021-09-16T08:56:01.495016Z","iopub.status.idle":"2021-09-16T08:56:01.715894Z","shell.execute_reply.started":"2021-09-16T08:56:01.494984Z","shell.execute_reply":"2021-09-16T08:56:01.714805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ls -a ./","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:56:01.717687Z","iopub.execute_input":"2021-09-16T08:56:01.718051Z","iopub.status.idle":"2021-09-16T08:56:02.692555Z","shell.execute_reply.started":"2021-09-16T08:56:01.718009Z","shell.execute_reply":"2021-09-16T08:56:02.691414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing Test ","metadata":{}},{"cell_type":"code","source":"ls -a ../input/g2net-gravitational-wave-detection/sample_submission.csv","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:56:02.694595Z","iopub.execute_input":"2021-09-16T08:56:02.695215Z","iopub.status.idle":"2021-09-16T08:56:03.540062Z","shell.execute_reply.started":"2021-09-16T08:56:02.695152Z","shell.execute_reply":"2021-09-16T08:56:03.538909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"assigning submission ids to the test set to make prediction on them","metadata":{}},{"cell_type":"code","source":"sub = pd.read_csv('../input/g2net-gravitational-wave-detection/sample_submission.csv')\nx_test = sub[['id']]","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:58:02.048735Z","iopub.execute_input":"2021-09-16T08:58:02.049102Z","iopub.status.idle":"2021-09-16T08:58:02.207780Z","shell.execute_reply.started":"2021-09-16T08:58:02.049066Z","shell.execute_reply":"2021-09-16T08:58:02.206742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_test.tail()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:58:03.307933Z","iopub.execute_input":"2021-09-16T08:58:03.308241Z","iopub.status.idle":"2021-09-16T08:58:03.322910Z","shell.execute_reply.started":"2021-09-16T08:58:03.308211Z","shell.execute_reply":"2021-09-16T08:58:03.321536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test dataset\ntest_dataset = tf.data.Dataset.from_tensor_slices((x_test['id'].apply(get_npy_filepath, is_train=False).values))\ntest_dataset = test_dataset.map(preprocess_function_parse_tf, num_parallel_calls=tf.data.AUTOTUNE)\ntest_dataset = test_dataset.batch(batch_size)\ntest_dataset = test_dataset.prefetch(tf.data.AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:58:05.742163Z","iopub.execute_input":"2021-09-16T08:58:05.742766Z","iopub.status.idle":"2021-09-16T08:58:06.136597Z","shell.execute_reply.started":"2021-09-16T08:58:05.742724Z","shell.execute_reply":"2021-09-16T08:58:06.135623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataset","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:58:07.272800Z","iopub.execute_input":"2021-09-16T08:58:07.273221Z","iopub.status.idle":"2021-09-16T08:58:07.290138Z","shell.execute_reply.started":"2021-09-16T08:58:07.273169Z","shell.execute_reply":"2021-09-16T08:58:07.288752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction","metadata":{}},{"cell_type":"markdown","source":"Now, we will load the cnn model that we saved after training to make prediction on `test_dataset`","metadata":{}},{"cell_type":"code","source":"ls -a ./model/","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:58:13.324251Z","iopub.execute_input":"2021-09-16T08:58:13.324819Z","iopub.status.idle":"2021-09-16T08:58:14.220156Z","shell.execute_reply.started":"2021-09-16T08:58:13.324785Z","shell.execute_reply":"2021-09-16T08:58:14.218985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"saved_cnn_model = tf.keras.models.load_model('./model/cnn_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:58:20.335101Z","iopub.execute_input":"2021-09-16T08:58:20.335469Z","iopub.status.idle":"2021-09-16T08:58:20.613905Z","shell.execute_reply.started":"2021-09-16T08:58:20.335426Z","shell.execute_reply":"2021-09-16T08:58:20.612869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"saved_cnn_model","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:58:32.563805Z","iopub.execute_input":"2021-09-16T08:58:32.564123Z","iopub.status.idle":"2021-09-16T08:58:32.575004Z","shell.execute_reply.started":"2021-09-16T08:58:32.564092Z","shell.execute_reply":"2021-09-16T08:58:32.573615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"retraining the saved model on `valid_dataset`\n\n> previously we set x = train_dataset","metadata":{}},{"cell_type":"code","source":"saved_cnn_model.fit(x=valid_dataset, epochs=3, batch_size=batch_size, verbose=1)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T08:58:39.091117Z","iopub.execute_input":"2021-09-16T08:58:39.091930Z","iopub.status.idle":"2021-09-16T09:45:21.194748Z","shell.execute_reply.started":"2021-09-16T08:58:39.091895Z","shell.execute_reply":"2021-09-16T09:45:21.191645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now saving the full model after training to make prediction on `test_dataset`","metadata":{}},{"cell_type":"code","source":"saved_cnn_model.save('./model/full_cnn_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-09-16T09:45:21.199995Z","iopub.execute_input":"2021-09-16T09:45:21.200243Z","iopub.status.idle":"2021-09-16T09:45:21.376769Z","shell.execute_reply.started":"2021-09-16T09:45:21.200214Z","shell.execute_reply":"2021-09-16T09:45:21.375626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_cnn_model = tf.keras.models.load_model('./model/full_cnn_model.h5')","metadata":{"execution":{"iopub.status.busy":"2021-09-16T09:45:21.378257Z","iopub.execute_input":"2021-09-16T09:45:21.379186Z","iopub.status.idle":"2021-09-16T09:45:21.663388Z","shell.execute_reply.started":"2021-09-16T09:45:21.379139Z","shell.execute_reply":"2021-09-16T09:45:21.662404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction = full_cnn_model.predict(test_dataset)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T09:45:21.666465Z","iopub.execute_input":"2021-09-16T09:45:21.666790Z","iopub.status.idle":"2021-09-16T10:12:52.950257Z","shell.execute_reply.started":"2021-09-16T09:45:21.666748Z","shell.execute_reply":"2021-09-16T10:12:52.944527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction","metadata":{"execution":{"iopub.status.busy":"2021-09-16T10:12:52.960809Z","iopub.execute_input":"2021-09-16T10:12:52.961147Z","iopub.status.idle":"2021-09-16T10:12:52.986169Z","shell.execute_reply.started":"2021-09-16T10:12:52.961105Z","shell.execute_reply":"2021-09-16T10:12:52.984950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction = prediction.flatten()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T10:12:52.989277Z","iopub.execute_input":"2021-09-16T10:12:52.990434Z","iopub.status.idle":"2021-09-16T10:12:52.998907Z","shell.execute_reply.started":"2021-09-16T10:12:52.990391Z","shell.execute_reply":"2021-09-16T10:12:52.997788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparing to Submit","metadata":{}},{"cell_type":"code","source":"submission = pd.DataFrame({'id': x_test.id, 'target': prediction})","metadata":{"execution":{"iopub.status.busy":"2021-09-16T10:12:53.000694Z","iopub.execute_input":"2021-09-16T10:12:53.001337Z","iopub.status.idle":"2021-09-16T10:12:53.031071Z","shell.execute_reply.started":"2021-09-16T10:12:53.001258Z","shell.execute_reply":"2021-09-16T10:12:53.029919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.shape","metadata":{"execution":{"iopub.status.busy":"2021-09-16T10:12:53.032869Z","iopub.execute_input":"2021-09-16T10:12:53.033185Z","iopub.status.idle":"2021-09-16T10:12:53.044498Z","shell.execute_reply.started":"2021-09-16T10:12:53.033142Z","shell.execute_reply":"2021-09-16T10:12:53.043156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-16T10:12:53.046530Z","iopub.execute_input":"2021-09-16T10:12:53.047056Z","iopub.status.idle":"2021-09-16T10:12:53.087770Z","shell.execute_reply.started":"2021-09-16T10:12:53.047013Z","shell.execute_reply":"2021-09-16T10:12:53.086859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('./submission.csv', index= False)","metadata":{"execution":{"iopub.status.busy":"2021-09-16T10:12:53.090695Z","iopub.execute_input":"2021-09-16T10:12:53.091116Z","iopub.status.idle":"2021-09-16T10:12:53.780187Z","shell.execute_reply.started":"2021-09-16T10:12:53.091061Z","shell.execute_reply":"2021-09-16T10:12:53.779176Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I started with zero idea and I ended up learning about a lot of new things. I am very much thankful to all these resources that help me increase my knowledge and give me more insight as I proceed to improve my skills on my coding journey.\n\nI tried referencing as much as I could.","metadata":{}}]}