{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Intro\n\nMachine learning can help astronomers sort big data recorded in space exploration.\n\n**Gravitational Wave:**\n\nVery simply,\n`A gravitational wave is like ripples in space time. It is usually caused by some of the most violent and energetic processes in the Universe.`\n\nThey are invisible but incredibly fast. \n\n**Why we need to detect GW?**\n\nDetecting and analyzing the information carried by gravitational waves is allowing us to observe the Universe in a way never before possible, providing astronomers and other scientists with their first glimpses of literally un-seeable wonders.\n When a gravitational wave passes by Earth, it squeezes and stretches space. LIGO can detect this squeezing and stretching. Each LIGO observatory has two “arms” that are each more than 2 miles (4 kilometers) long.\n \n##### **above info collected via some very rough googling**\n \n### Goal\n\nThe GW was first detected/seen when two blackholes merged into one big black whole back in Sept, 2015.\n\nIn this competition, our goal is to detect GW signals from the mergers of binary black holes.\n\nI am going to document this process as I start with zero idea about any of these.\n\nThe folowing two kernels have been my overall inspiration to understand this whole task to my best capability. These are really well explained and worth mentioning.\n\n- [kernel 1](https://www.kaggle.com/pranay1990/pranay-g2net-gw)\n- [kernel 2](https://github.com/SiddharthPatel45/gravitational-wave-detection/blob/main/code/gw-detection-modelling.ipynb)\n- [kernel 3](https://www.kaggle.com/atamazian/nnaudio-constant-q-transform-demonstration/comments)\n\nThank you for sharing your work.","metadata":{"papermill":{"duration":0.055008,"end_time":"2021-09-16T10:17:57.239502","exception":false,"start_time":"2021-09-16T10:17:57.184494","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Imports","metadata":{"papermill":{"duration":0.054052,"end_time":"2021-09-16T10:17:57.347416","exception":false,"start_time":"2021-09-16T10:17:57.293364","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install -q nnAudio","metadata":{"papermill":{"duration":8.716114,"end_time":"2021-09-16T10:18:06.116706","exception":false,"start_time":"2021-09-16T10:17:57.400592","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T05:58:46.549762Z","iopub.execute_input":"2021-09-27T05:58:46.550555Z","iopub.status.idle":"2021-09-27T05:58:54.354914Z","shell.execute_reply.started":"2021-09-27T05:58:46.550452Z","shell.execute_reply":"2021-09-27T05:58:54.354075Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pylab as plt\nimport seaborn as sns\nfrom glob import glob\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\n\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.layers import Dense\nfrom tensorflow.keras.layers import Conv2D\nfrom tensorflow.keras.layers import MaxPooling2D\nfrom tensorflow.keras.layers import GlobalAveragePooling2D\nfrom tensorflow.keras.layers import Flatten\nfrom tensorflow.keras.applications import EfficientNetB0\nfrom tensorflow.keras.metrics import AUC\n\nimport librosa.display\nimport torch\n\n# this is used for Contant Q Transform\nfrom nnAudio.Spectrogram import CQT1992v2\nfrom tensorflow.keras.applications import EfficientNetB0 as efn","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":10.295319,"end_time":"2021-09-16T10:18:16.466936","exception":false,"start_time":"2021-09-16T10:18:06.171617","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T05:59:49.772476Z","iopub.execute_input":"2021-09-27T05:59:49.772768Z","iopub.status.idle":"2021-09-27T05:59:57.405381Z","shell.execute_reply.started":"2021-09-27T05:59:49.772736Z","shell.execute_reply":"2021-09-27T05:59:57.404393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Files\n\n```\ntrain: contains one npy file per observation.\n\ntest: we have to predict the probability whether or not the observation contains a gravitational wave.\n\ntraining_labels: If associated signal contains a GW or not.\n```\n\nThe waves detected by GW detectors have noises in output signals. So researchers need to find out if the output signal is only **noise** or **signal+noise**.\n\n\nWe are provided with a training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo). \n\nThis problem is seen as a binary classification problem, if signal is detected or not. \n ","metadata":{"papermill":{"duration":0.054082,"end_time":"2021-09-16T10:18:16.576335","exception":false,"start_time":"2021-09-16T10:18:16.522253","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Data Exploration","metadata":{"papermill":{"duration":0.054328,"end_time":"2021-09-16T10:18:16.685073","exception":false,"start_time":"2021-09-16T10:18:16.630745","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_label_dataset = pd.read_csv(\"../input/g2net-gravitational-wave-detection/training_labels.csv\")\ntrain_label_dataset.head()","metadata":{"papermill":{"duration":0.449412,"end_time":"2021-09-16T10:18:17.188677","exception":false,"start_time":"2021-09-16T10:18:16.739265","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T05:59:57.407324Z","iopub.execute_input":"2021-09-27T05:59:57.407561Z","iopub.status.idle":"2021-09-27T05:59:57.838480Z","shell.execute_reply.started":"2021-09-27T05:59:57.407532Z","shell.execute_reply":"2021-09-27T05:59:57.837670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_label_dataset.shape","metadata":{"papermill":{"duration":0.06226,"end_time":"2021-09-16T10:18:17.307470","exception":false,"start_time":"2021-09-16T10:18:17.245210","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:00:02.574169Z","iopub.execute_input":"2021-09-27T06:00:02.574803Z","iopub.status.idle":"2021-09-27T06:00:02.581075Z","shell.execute_reply.started":"2021-09-27T06:00:02.574762Z","shell.execute_reply":"2021-09-27T06:00:02.580272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"when `target = 1` it means that the signal (GW) is present","metadata":{"papermill":{"duration":0.054585,"end_time":"2021-09-16T10:18:17.416828","exception":false,"start_time":"2021-09-16T10:18:17.362243","status":"completed"},"tags":[]}},{"cell_type":"code","source":"sns.countplot(data=train_label_dataset, x=\"target\")","metadata":{"papermill":{"duration":0.27069,"end_time":"2021-09-16T10:18:17.742083","exception":false,"start_time":"2021-09-16T10:18:17.471393","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:00:05.010630Z","iopub.execute_input":"2021-09-27T06:00:05.010904Z","iopub.status.idle":"2021-09-27T06:00:05.248985Z","shell.execute_reply.started":"2021-09-27T06:00:05.010876Z","shell.execute_reply":"2021-09-27T06:00:05.248109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_label_dataset['target'].value_counts()","metadata":{"papermill":{"duration":0.070086,"end_time":"2021-09-16T10:18:17.873794","exception":false,"start_time":"2021-09-16T10:18:17.803708","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:00:07.513341Z","iopub.execute_input":"2021-09-27T06:00:07.513597Z","iopub.status.idle":"2021-09-27T06:00:07.525479Z","shell.execute_reply.started":"2021-09-27T06:00:07.513570Z","shell.execute_reply":"2021-09-27T06:00:07.524707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looking for null values:","metadata":{"papermill":{"duration":0.056039,"end_time":"2021-09-16T10:18:17.986063","exception":false,"start_time":"2021-09-16T10:18:17.930024","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_label_dataset.isnull().sum() # no null","metadata":{"papermill":{"duration":0.117361,"end_time":"2021-09-16T10:18:18.158944","exception":false,"start_time":"2021-09-16T10:18:18.041583","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:00:10.410915Z","iopub.execute_input":"2021-09-27T06:00:10.411473Z","iopub.status.idle":"2021-09-27T06:00:10.446894Z","shell.execute_reply.started":"2021-09-27T06:00:10.411438Z","shell.execute_reply":"2021-09-27T06:00:10.446067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_path = glob('../input/g2net-gravitational-wave-detection/train/*/*/*/*')","metadata":{"papermill":{"duration":86.381617,"end_time":"2021-09-16T10:19:44.596221","exception":false,"start_time":"2021-09-16T10:18:18.214604","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:00:11.606351Z","iopub.execute_input":"2021-09-27T06:00:11.607276Z","iopub.status.idle":"2021-09-27T06:01:30.561888Z","shell.execute_reply.started":"2021-09-27T06:00:11.607232Z","shell.execute_reply":"2021-09-27T06:01:30.560817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 560,000 **.npy** files in the `train` set","metadata":{"papermill":{"duration":0.055774,"end_time":"2021-09-16T10:19:44.709290","exception":false,"start_time":"2021-09-16T10:19:44.653516","status":"completed"},"tags":[]}},{"cell_type":"code","source":"len(train_path)","metadata":{"papermill":{"duration":0.064331,"end_time":"2021-09-16T10:19:44.829375","exception":false,"start_time":"2021-09-16T10:19:44.765044","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:01:51.735930Z","iopub.execute_input":"2021-09-27T06:01:51.736233Z","iopub.status.idle":"2021-09-27T06:01:51.741740Z","shell.execute_reply.started":"2021-09-27T06:01:51.736202Z","shell.execute_reply":"2021-09-27T06:01:51.740995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we want to take a took at how these data looks:\n\nlets see how data at index 3 looks","metadata":{"papermill":{"duration":0.05705,"end_time":"2021-09-16T10:19:44.942288","exception":false,"start_time":"2021-09-16T10:19:44.885238","status":"completed"},"tags":[]}},{"cell_type":"code","source":"explore_sample_3 = np.load(train_path[3])\nexplore_sample_3","metadata":{"papermill":{"duration":0.076601,"end_time":"2021-09-16T10:19:45.074872","exception":false,"start_time":"2021-09-16T10:19:44.998271","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:02:00.320631Z","iopub.execute_input":"2021-09-27T06:02:00.321374Z","iopub.status.idle":"2021-09-27T06:02:00.335420Z","shell.execute_reply.started":"2021-09-27T06:02:00.321337Z","shell.execute_reply":"2021-09-27T06:02:00.334727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that there are 3 rows to the data. This represents data extracted by 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo) respectively.","metadata":{"papermill":{"duration":0.055933,"end_time":"2021-09-16T10:19:45.187996","exception":false,"start_time":"2021-09-16T10:19:45.132063","status":"completed"},"tags":[]}},{"cell_type":"code","source":"explore_sample_3.shape","metadata":{"papermill":{"duration":0.063857,"end_time":"2021-09-16T10:19:45.308045","exception":false,"start_time":"2021-09-16T10:19:45.244188","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:04:56.098287Z","iopub.execute_input":"2021-09-27T06:04:56.099227Z","iopub.status.idle":"2021-09-27T06:04:56.104428Z","shell.execute_reply.started":"2021-09-27T06:04:56.099181Z","shell.execute_reply":"2021-09-27T06:04:56.103701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"each index of `explore_sample_3` has **4096** columns","metadata":{"papermill":{"duration":0.056611,"end_time":"2021-09-16T10:19:45.422118","exception":false,"start_time":"2021-09-16T10:19:45.365507","status":"completed"},"tags":[]}},{"cell_type":"code","source":"print(len(explore_sample_3[0]), len(explore_sample_3[1]), len(explore_sample_3[2]))","metadata":{"papermill":{"duration":0.064508,"end_time":"2021-09-16T10:19:45.544021","exception":false,"start_time":"2021-09-16T10:19:45.479513","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:04:57.856951Z","iopub.execute_input":"2021-09-27T06:04:57.857254Z","iopub.status.idle":"2021-09-27T06:04:57.866134Z","shell.execute_reply.started":"2021-09-27T06:04:57.857226Z","shell.execute_reply":"2021-09-27T06:04:57.865290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# just a tensor representation\ntf.convert_to_tensor(explore_sample_3[0])","metadata":{"papermill":{"duration":1.725313,"end_time":"2021-09-16T10:19:47.326219","exception":false,"start_time":"2021-09-16T10:19:45.600906","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:04:59.917546Z","iopub.execute_input":"2021-09-27T06:04:59.918083Z","iopub.status.idle":"2021-09-27T06:04:59.941398Z","shell.execute_reply.started":"2021-09-27T06:04:59.918034Z","shell.execute_reply":"2021-09-27T06:04:59.940648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploring the sample data with Librosa\n\nLibrosa is a python package for music and audio analysis, more about this awesome library can be found [here](https://librosa.org/doc/latest/index.html).\n\nThere is a very good kernel that can be found [here](https://www.kaggle.com/hinamimi/visualization-gravitational-wave-with-librosa).\nIt has really great demonstration of how to use Librosa.\n\nNow first I will find the `label` (id) of `explore_sample_3` from the `training_label.csv` dataset. After that I can find whether the target is 1 or 0.\n\n- 0 = negative sample\n- 1 = possotive sample","metadata":{"papermill":{"duration":0.059457,"end_time":"2021-09-16T10:19:47.444424","exception":false,"start_time":"2021-09-16T10:19:47.384967","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_path[3]","metadata":{"papermill":{"duration":0.06627,"end_time":"2021-09-16T10:19:47.568505","exception":false,"start_time":"2021-09-16T10:19:47.502235","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:08.435624Z","iopub.execute_input":"2021-09-27T06:05:08.435897Z","iopub.status.idle":"2021-09-27T06:05:08.441422Z","shell.execute_reply.started":"2021-09-27T06:05:08.435868Z","shell.execute_reply":"2021-09-27T06:05:08.440710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"the value of `train_path` at index 3 looks like:'\n\n'../input/g2net-gravitational-wave-detection/train/7/7/7/77727f6826.npy'\n\nSo, we know that the Id of `explore_sample_3` is **77727f6826**. To extract the Id the following code snippet has been written. \n\n","metadata":{"papermill":{"duration":0.058027,"end_time":"2021-09-16T10:19:47.685037","exception":false,"start_time":"2021-09-16T10:19:47.627010","status":"completed"},"tags":[]}},{"cell_type":"code","source":"rind = train_path[3].rindex('/') # last index where the character '/' appeared\nextracted_id_for_explore_sample_3 = train_path[3][rind+1:].replace('.npy', '') # replaced .npy\nextracted_id_for_explore_sample_3","metadata":{"papermill":{"duration":0.06649,"end_time":"2021-09-16T10:19:47.809883","exception":false,"start_time":"2021-09-16T10:19:47.743393","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:12.255445Z","iopub.execute_input":"2021-09-27T06:05:12.255724Z","iopub.status.idle":"2021-09-27T06:05:12.262211Z","shell.execute_reply.started":"2021-09-27T06:05:12.255694Z","shell.execute_reply":"2021-09-27T06:05:12.261588Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that it is a positive sample","metadata":{"papermill":{"duration":0.058267,"end_time":"2021-09-16T10:19:47.926552","exception":false,"start_time":"2021-09-16T10:19:47.868285","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_label_dataset[train_label_dataset['id']==extracted_id_for_explore_sample_3]['target']","metadata":{"papermill":{"duration":0.145958,"end_time":"2021-09-16T10:19:48.131177","exception":false,"start_time":"2021-09-16T10:19:47.985219","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:15.032447Z","iopub.execute_input":"2021-09-27T06:05:15.033245Z","iopub.status.idle":"2021-09-27T06:05:15.097554Z","shell.execute_reply.started":"2021-09-27T06:05:15.033205Z","shell.execute_reply":"2021-09-27T06:05:15.096832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"positive_sample = explore_sample_3\n# index 1 od train_path has a target of 0 so it is a negative sample.\nnegative_sample = np.load(train_path[1])\nnegative_sample","metadata":{"papermill":{"duration":0.086927,"end_time":"2021-09-16T10:19:48.276435","exception":false,"start_time":"2021-09-16T10:19:48.189508","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:17.139662Z","iopub.execute_input":"2021-09-27T06:05:17.140326Z","iopub.status.idle":"2021-09-27T06:05:17.156579Z","shell.execute_reply.started":"2021-09-27T06:05:17.140288Z","shell.execute_reply":"2021-09-27T06:05:17.155851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"samples = (positive_sample, negative_sample)\ntargets = (1, 0)","metadata":{"papermill":{"duration":0.065375,"end_time":"2021-09-16T10:19:48.400750","exception":false,"start_time":"2021-09-16T10:19:48.335375","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:18.102690Z","iopub.execute_input":"2021-09-27T06:05:18.103028Z","iopub.status.idle":"2021-09-27T06:05:18.108920Z","shell.execute_reply.started":"2021-09-27T06:05:18.102996Z","shell.execute_reply":"2021-09-27T06:05:18.108238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Using librosa.display() to view raw waves: \n\nKernel: https://www.kaggle.com/hinamimi/visualization-gravitational-wave-with-librosa","metadata":{"papermill":{"duration":0.058708,"end_time":"2021-09-16T10:19:48.518674","exception":false,"start_time":"2021-09-16T10:19:48.459966","status":"completed"},"tags":[]}},{"cell_type":"code","source":"colors = (\"red\", \"green\", \"blue\")\nsignal_names = (\"LIGO Hanford\", \"LIGO Livingston\", \"Virgo\")\n\nfor x, i in tqdm(zip(samples, targets)):\n    figure = plt.figure(figsize=(16, 7))\n    figure.suptitle(f'Raw wave (target={i})', fontsize=20)\n    # range is 3 because we have 3 different rows for each interferometers\n    for j in range(3):\n        axes = figure.add_subplot(3, 1, j+1)\n        librosa.display.waveshow(x[j], sr=2048, ax=axes, color=colors[j])\n        axes.set_title(signal_names[j], fontsize=12)\n        axes.set_xlabel('Time[sec]')\n    plt.tight_layout()\n    plt.show()","metadata":{"papermill":{"duration":1.475215,"end_time":"2021-09-16T10:19:50.053549","exception":false,"start_time":"2021-09-16T10:19:48.578334","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:22.466961Z","iopub.execute_input":"2021-09-27T06:05:22.467386Z","iopub.status.idle":"2021-09-27T06:05:23.957209Z","shell.execute_reply.started":"2021-09-27T06:05:22.467356Z","shell.execute_reply":"2021-09-27T06:05:23.956460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.displot(positive_sample[0,:])","metadata":{"papermill":{"duration":0.431647,"end_time":"2021-09-16T10:19:50.553460","exception":false,"start_time":"2021-09-16T10:19:50.121813","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:28.429417Z","iopub.execute_input":"2021-09-27T06:05:28.430245Z","iopub.status.idle":"2021-09-27T06:05:28.797005Z","shell.execute_reply.started":"2021-09-27T06:05:28.430198Z","shell.execute_reply":"2021-09-27T06:05:28.796300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Working with a cleaner datset by merging `train` and `training_labels` datasets","metadata":{"papermill":{"duration":0.066087,"end_time":"2021-09-16T10:19:50.686287","exception":false,"start_time":"2021-09-16T10:19:50.620200","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pd.set_option('display.max_colwidth',None)","metadata":{"papermill":{"duration":0.074082,"end_time":"2021-09-16T10:19:50.826275","exception":false,"start_time":"2021-09-16T10:19:50.752193","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:30.172804Z","iopub.execute_input":"2021-09-27T06:05:30.173093Z","iopub.status.idle":"2021-09-27T06:05:30.176413Z","shell.execute_reply.started":"2021-09-27T06:05:30.173056Z","shell.execute_reply":"2021-09-27T06:05:30.175863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ids = []\nfor files in train_path:\n    ids.append(files[files.rindex('/')+1:].replace('.npy',''))\ndf = pd.DataFrame({\"id\":ids,\"path\":train_path})\ndf = pd.merge(df, train_label_dataset, on='id')","metadata":{"papermill":{"duration":1.006209,"end_time":"2021-09-16T10:19:51.899081","exception":false,"start_time":"2021-09-16T10:19:50.892872","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:31.847877Z","iopub.execute_input":"2021-09-27T06:05:31.848415Z","iopub.status.idle":"2021-09-27T06:05:32.790431Z","shell.execute_reply.started":"2021-09-27T06:05:31.848372Z","shell.execute_reply":"2021-09-27T06:05:32.789533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head()","metadata":{"papermill":{"duration":0.117554,"end_time":"2021-09-16T10:19:52.084688","exception":false,"start_time":"2021-09-16T10:19:51.967134","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:32.879478Z","iopub.execute_input":"2021-09-27T06:05:32.879727Z","iopub.status.idle":"2021-09-27T06:05:32.889147Z","shell.execute_reply.started":"2021-09-27T06:05:32.879702Z","shell.execute_reply":"2021-09-27T06:05:32.888308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.shape","metadata":{"papermill":{"duration":0.129448,"end_time":"2021-09-16T10:19:52.345639","exception":false,"start_time":"2021-09-16T10:19:52.216191","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:35.209713Z","iopub.execute_input":"2021-09-27T06:05:35.210289Z","iopub.status.idle":"2021-09-27T06:05:35.215897Z","shell.execute_reply.started":"2021-09-27T06:05:35.210253Z","shell.execute_reply":"2021-09-27T06:05:35.215194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing\n\n#### Core Idea: \nIf any particular frequency is widespread in the signal or not. If true then our required GW is present.\n\nApproach:\n\n- convert original signal  -->  spectrogram signal\n- coverting from time domain  --> frequency domain\n    - done using **[Constant Q transformation](https://en.wikipedia.org/wiki/Constant-Q_transform)**\n    - **[kernel](https://www.kaggle.com/atamazian/nnaudio-constant-q-transform-demonstration/comments)**\n","metadata":{"papermill":{"duration":0.113612,"end_time":"2021-09-16T10:19:52.573376","exception":false,"start_time":"2021-09-16T10:19:52.459764","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"I refer to the kernel [here](https://www.kaggle.com/atamazian/nnaudio-constant-q-transform-demonstration/comments) to define my CQT.\n\nPlease have a look. ","metadata":{"papermill":{"duration":0.066678,"end_time":"2021-09-16T10:19:52.710207","exception":false,"start_time":"2021-09-16T10:19:52.643529","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# CQT\ntransform = CQT1992v2(sr=2048,        # sample rate\n                fmin=20,        # min freq\n                fmax=500,      # max freq\n                hop_length=64,  # hop length\n                verbose=False)","metadata":{"papermill":{"duration":0.108119,"end_time":"2021-09-16T10:19:52.885261","exception":false,"start_time":"2021-09-16T10:19:52.777142","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:38.187357Z","iopub.execute_input":"2021-09-27T06:05:38.187831Z","iopub.status.idle":"2021-09-27T06:05:38.228030Z","shell.execute_reply.started":"2021-09-27T06:05:38.187791Z","shell.execute_reply":"2021-09-27T06:05:38.227404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# the Cqt function\n# preprocess function\ndef preprocess_function_cqt(path):\n    signal = np.load(path.numpy())\n    # there are 3 signal as explained before for each interferometers\n    for i in range(signal.shape[0]):\n        # normalize signal\n        signal[i] /= np.max(signal[i])\n    # horizontal stack\n    signal = np.hstack(signal)\n    # tensor conversion\n    signal = torch.from_numpy(signal).float()\n    # getting the image from CQT transform\n    image = transform(signal)\n    # converting to array from tensor\n    image = np.array(image)\n    # transpose the image to get right orientation\n    image = np.transpose(image,(1,2,0))\n    \n    # conver the image to tf.tensor and return\n    return tf.convert_to_tensor(image)","metadata":{"papermill":{"duration":0.076763,"end_time":"2021-09-16T10:19:53.029293","exception":false,"start_time":"2021-09-16T10:19:52.952530","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:40.906491Z","iopub.execute_input":"2021-09-27T06:05:40.907167Z","iopub.status.idle":"2021-09-27T06:05:40.912714Z","shell.execute_reply.started":"2021-09-27T06:05:40.907130Z","shell.execute_reply":"2021-09-27T06:05:40.912189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image = preprocess_function_cqt(tf.convert_to_tensor(df['path'][2]))\nprint(image.shape)\nplt.imshow(image)","metadata":{"papermill":{"duration":0.345878,"end_time":"2021-09-16T10:19:53.442209","exception":false,"start_time":"2021-09-16T10:19:53.096331","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:47.104488Z","iopub.execute_input":"2021-09-27T06:05:47.105178Z","iopub.status.idle":"2021-09-27T06:05:47.406367Z","shell.execute_reply.started":"2021-09-27T06:05:47.105137Z","shell.execute_reply":"2021-09-27T06:05:47.405366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"for a different path","metadata":{"papermill":{"duration":0.069296,"end_time":"2021-09-16T10:19:53.618735","exception":false,"start_time":"2021-09-16T10:19:53.549439","status":"completed"},"tags":[]}},{"cell_type":"code","source":"image = preprocess_function_cqt(tf.convert_to_tensor(df['path'][5069]))\nprint(image.shape)\nplt.imshow(image)","metadata":{"papermill":{"duration":0.272033,"end_time":"2021-09-16T10:19:53.958994","exception":false,"start_time":"2021-09-16T10:19:53.686961","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:50.580112Z","iopub.execute_input":"2021-09-27T06:05:50.580913Z","iopub.status.idle":"2021-09-27T06:05:50.790601Z","shell.execute_reply.started":"2021-09-27T06:05:50.580877Z","shell.execute_reply":"2021-09-27T06:05:50.789784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"we can see that the image shape is **(56, 193, 1)**, so thats our shpa eof the input.","metadata":{"papermill":{"duration":0.073419,"end_time":"2021-09-16T10:19:54.104333","exception":false,"start_time":"2021-09-16T10:19:54.030914","status":"completed"},"tags":[]}},{"cell_type":"code","source":"input_shape = (56, 193, 1)","metadata":{"papermill":{"duration":0.077423,"end_time":"2021-09-16T10:19:54.252949","exception":false,"start_time":"2021-09-16T10:19:54.175526","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:51.513692Z","iopub.execute_input":"2021-09-27T06:05:51.514505Z","iopub.status.idle":"2021-09-27T06:05:51.518086Z","shell.execute_reply.started":"2021-09-27T06:05:51.514469Z","shell.execute_reply":"2021-09-27T06:05:51.517424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess_function_parse_tf(path, y=None):\n    [x] = tf.py_function(func=preprocess_function_cqt, inp=[path], Tout=[tf.float32])\n    x = tf.ensure_shape(x, input_shape)\n    if y is None:\n        return x\n    else:\n        return x,y","metadata":{"papermill":{"duration":0.077716,"end_time":"2021-09-16T10:19:54.401060","exception":false,"start_time":"2021-09-16T10:19:54.323344","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:56.802784Z","iopub.execute_input":"2021-09-27T06:05:56.803082Z","iopub.status.idle":"2021-09-27T06:05:56.809114Z","shell.execute_reply.started":"2021-09-27T06:05:56.803036Z","shell.execute_reply":"2021-09-27T06:05:56.808305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# preprocess_function_parse_tf(tf.convert_to_tensor(df['path'][5069]))","metadata":{"papermill":{"duration":0.076554,"end_time":"2021-09-16T10:19:54.553699","exception":false,"start_time":"2021-09-16T10:19:54.477145","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:05:59.069469Z","iopub.execute_input":"2021-09-27T06:05:59.070167Z","iopub.status.idle":"2021-09-27T06:05:59.073591Z","shell.execute_reply.started":"2021-09-27T06:05:59.070116Z","shell.execute_reply":"2021-09-27T06:05:59.073052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### I will define the `training` and `validation` dataset from `df`","metadata":{"papermill":{"duration":0.070797,"end_time":"2021-09-16T10:19:54.695754","exception":false,"start_time":"2021-09-16T10:19:54.624957","status":"completed"},"tags":[]}},{"cell_type":"code","source":"X = df['id']\ny = df['target'].astype('int8').values","metadata":{"papermill":{"duration":0.078153,"end_time":"2021-09-16T10:19:54.845047","exception":false,"start_time":"2021-09-16T10:19:54.766894","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:00.090503Z","iopub.execute_input":"2021-09-27T06:06:00.090990Z","iopub.status.idle":"2021-09-27T06:06:00.095490Z","shell.execute_reply.started":"2021-09-27T06:06:00.090939Z","shell.execute_reply":"2021-09-27T06:06:00.094829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y","metadata":{"papermill":{"duration":0.078591,"end_time":"2021-09-16T10:19:54.994447","exception":false,"start_time":"2021-09-16T10:19:54.915856","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:01.272672Z","iopub.execute_input":"2021-09-27T06:06:01.273154Z","iopub.status.idle":"2021-09-27T06:06:01.277640Z","shell.execute_reply.started":"2021-09-27T06:06:01.273109Z","shell.execute_reply":"2021-09-27T06:06:01.277114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_train, x_valid, y_train, y_valid = train_test_split(X, y, random_state = 42, stratify = y)","metadata":{"papermill":{"duration":0.478095,"end_time":"2021-09-16T10:19:55.543834","exception":false,"start_time":"2021-09-16T10:19:55.065739","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:03.247031Z","iopub.execute_input":"2021-09-27T06:06:03.247444Z","iopub.status.idle":"2021-09-27T06:06:03.546926Z","shell.execute_reply.started":"2021-09-27T06:06:03.247406Z","shell.execute_reply":"2021-09-27T06:06:03.546107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_size = 250","metadata":{"papermill":{"duration":0.077965,"end_time":"2021-09-16T10:19:55.694194","exception":false,"start_time":"2021-09-16T10:19:55.616229","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:03.548162Z","iopub.execute_input":"2021-09-27T06:06:03.548362Z","iopub.status.idle":"2021-09-27T06:06:03.555297Z","shell.execute_reply.started":"2021-09-27T06:06:03.548339Z","shell.execute_reply":"2021-09-27T06:06:03.554319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_npy_filepath(id_, is_train=True):\n    path = ''\n    if is_train:\n        return f'../input/g2net-gravitational-wave-detection/train/{id_[0]}/{id_[1]}/{id_[2]}/{id_}.npy'\n    else:\n        return f'../input/g2net-gravitational-wave-detection/test/{id_[0]}/{id_[1]}/{id_[2]}/{id_}.npy'","metadata":{"papermill":{"duration":0.078519,"end_time":"2021-09-16T10:19:55.844468","exception":false,"start_time":"2021-09-16T10:19:55.765949","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:03.748802Z","iopub.execute_input":"2021-09-27T06:06:03.749066Z","iopub.status.idle":"2021-09-27T06:06:03.753646Z","shell.execute_reply.started":"2021-09-27T06:06:03.749026Z","shell.execute_reply":"2021-09-27T06:06:03.752681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset = tf.data.Dataset.from_tensor_slices((x_train.apply(get_npy_filepath).values, y_train))\n# shuffle the dataset\ntrain_dataset = train_dataset.shuffle(len(x_train))\ntrain_dataset = train_dataset.map(preprocess_function_parse_tf, num_parallel_calls=tf.data.AUTOTUNE)\ntrain_dataset = train_dataset.batch(batch_size)\ntrain_dataset = train_dataset.prefetch(tf.data.AUTOTUNE)","metadata":{"papermill":{"duration":0.57005,"end_time":"2021-09-16T10:19:56.485293","exception":false,"start_time":"2021-09-16T10:19:55.915243","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:05.998466Z","iopub.execute_input":"2021-09-27T06:06:05.998768Z","iopub.status.idle":"2021-09-27T06:06:06.454191Z","shell.execute_reply.started":"2021-09-27T06:06:05.998729Z","shell.execute_reply":"2021-09-27T06:06:06.453248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_dataset = tf.data.Dataset.from_tensor_slices((x_valid.apply(get_npy_filepath).values, y_valid))\nvalid_dataset = valid_dataset.map(preprocess_function_parse_tf, num_parallel_calls=tf.data.AUTOTUNE)\nvalid_dataset = valid_dataset.batch(batch_size)\nvalid_dataset = valid_dataset.prefetch(tf.data.AUTOTUNE)","metadata":{"papermill":{"duration":0.334146,"end_time":"2021-09-16T10:19:56.902365","exception":false,"start_time":"2021-09-16T10:19:56.568219","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:08.362855Z","iopub.execute_input":"2021-09-27T06:06:08.363483Z","iopub.status.idle":"2021-09-27T06:06:08.484722Z","shell.execute_reply.started":"2021-09-27T06:06:08.363440Z","shell.execute_reply":"2021-09-27T06:06:08.484124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_dataset","metadata":{"papermill":{"duration":0.130902,"end_time":"2021-09-16T10:19:57.153939","exception":false,"start_time":"2021-09-16T10:19:57.023037","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:08.551848Z","iopub.execute_input":"2021-09-27T06:06:08.552309Z","iopub.status.idle":"2021-09-27T06:06:08.559133Z","shell.execute_reply.started":"2021-09-27T06:06:08.552265Z","shell.execute_reply":"2021-09-27T06:06:08.558362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_dataset","metadata":{"papermill":{"duration":0.129375,"end_time":"2021-09-16T10:19:57.400899","exception":false,"start_time":"2021-09-16T10:19:57.271524","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:08.730742Z","iopub.execute_input":"2021-09-27T06:06:08.731165Z","iopub.status.idle":"2021-09-27T06:06:08.736905Z","shell.execute_reply.started":"2021-09-27T06:06:08.731125Z","shell.execute_reply":"2021-09-27T06:06:08.736152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating the Model","metadata":{"papermill":{"duration":0.119021,"end_time":"2021-09-16T10:19:57.637951","exception":false,"start_time":"2021-09-16T10:19:57.518930","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_dataset.take(1)","metadata":{"papermill":{"duration":0.080423,"end_time":"2021-09-16T10:19:57.809744","exception":false,"start_time":"2021-09-16T10:19:57.729321","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:11.652152Z","iopub.execute_input":"2021-09-27T06:06:11.652545Z","iopub.status.idle":"2021-09-27T06:06:11.658871Z","shell.execute_reply.started":"2021-09-27T06:06:11.652518Z","shell.execute_reply":"2021-09-27T06:06:11.658146Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Model from [here](https://github.com/SiddharthPatel45/gravitational-wave-detection/blob/main/code/gw-detection-modelling.ipynb) ~","metadata":{"papermill":{"duration":0.071675,"end_time":"2021-09-16T10:19:57.953872","exception":false,"start_time":"2021-09-16T10:19:57.882197","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Instantiate the Sequential model\nmodel_cnn = Sequential(name='CNN_model')\n\n# Add the first Convoluted2D layer w/ input_shape & MaxPooling2D layer followed by that\nmodel_cnn.add(Conv2D(filters=16,\n                     kernel_size=3,\n                     input_shape=input_shape,\n                     activation='relu',\n                     name='Conv_01'))\nmodel_cnn.add(MaxPooling2D(pool_size=2, name='Pool_01'))\n\n# Second pair of Conv1D and MaxPooling1D layers\nmodel_cnn.add(Conv2D(filters=32,\n                     kernel_size=3,\n                     input_shape=input_shape,\n                     activation='relu',\n                     name='Conv_02'))\nmodel_cnn.add(MaxPooling2D(pool_size=2, name='Pool_02'))\n\n# Third pair of Conv1D and MaxPooling1D layers\nmodel_cnn.add(Conv2D(filters=64,\n                     kernel_size=3,\n                     input_shape=input_shape,\n                     activation='relu',\n                     name='Conv_03'))\nmodel_cnn.add(MaxPooling2D(pool_size=2, name='Pool_03'))\n\n# Add the Flatten layer\nmodel_cnn.add(Flatten(name='Flatten'))\n\n# Add the Dense layers\nmodel_cnn.add(Dense(units=512,\n                activation='relu',\n                name='Dense_01'))\nmodel_cnn.add(Dense(units=64,\n                activation='relu',\n                name='Dense_02'))\n\n# Add the final Output layer\nmodel_cnn.add(Dense(1, activation='sigmoid', name='Output'))","metadata":{"papermill":{"duration":0.441072,"end_time":"2021-09-16T10:19:58.467996","exception":false,"start_time":"2021-09-16T10:19:58.026924","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:14.420206Z","iopub.execute_input":"2021-09-27T06:06:14.420724Z","iopub.status.idle":"2021-09-27T06:06:14.819693Z","shell.execute_reply.started":"2021-09-27T06:06:14.420673Z","shell.execute_reply":"2021-09-27T06:06:14.819084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_cnn.summary()","metadata":{"papermill":{"duration":0.091805,"end_time":"2021-09-16T10:19:58.632540","exception":false,"start_time":"2021-09-16T10:19:58.540735","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:15.763347Z","iopub.execute_input":"2021-09-27T06:06:15.763785Z","iopub.status.idle":"2021-09-27T06:06:15.775364Z","shell.execute_reply.started":"2021-09-27T06:06:15.763747Z","shell.execute_reply":"2021-09-27T06:06:15.774418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_cnn.compile(optimizer=Adam(learning_rate=0.0001),\n                  loss='binary_crossentropy',\n                  metrics=[[AUC(), 'accuracy']])","metadata":{"papermill":{"duration":0.327145,"end_time":"2021-09-16T10:19:59.037931","exception":false,"start_time":"2021-09-16T10:19:58.710786","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:17.837348Z","iopub.execute_input":"2021-09-27T06:06:17.837604Z","iopub.status.idle":"2021-09-27T06:06:17.859804Z","shell.execute_reply.started":"2021-09-27T06:06:17.837578Z","shell.execute_reply":"2021-09-27T06:06:17.859221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fit the data\nhistory_cnn = model_cnn.fit(x=train_dataset,\n                            epochs=3,\n                            validation_data=valid_dataset,\n                            batch_size=batch_size,\n                            verbose=1)","metadata":{"papermill":{"duration":9320.508043,"end_time":"2021-09-16T12:55:19.624112","exception":false,"start_time":"2021-09-16T10:19:59.116069","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T06:06:18.012119Z","iopub.execute_input":"2021-09-27T06:06:18.012424Z","iopub.status.idle":"2021-09-27T09:01:38.795836Z","shell.execute_reply.started":"2021-09-27T06:06:18.012394Z","shell.execute_reply":"2021-09-27T09:01:38.792973Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"saving the model after training is complete","metadata":{"papermill":{"duration":1.412747,"end_time":"2021-09-16T12:55:22.637163","exception":false,"start_time":"2021-09-16T12:55:21.224416","status":"completed"},"tags":[]}},{"cell_type":"code","source":"model_cnn.save('./model/cnn_model.h5')","metadata":{"papermill":{"duration":1.55418,"end_time":"2021-09-16T12:55:25.545638","exception":false,"start_time":"2021-09-16T12:55:23.991458","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:01:38.971484Z","iopub.execute_input":"2021-09-27T09:01:38.971722Z","iopub.status.idle":"2021-09-27T09:01:39.098992Z","shell.execute_reply.started":"2021-09-27T09:01:38.971693Z","shell.execute_reply":"2021-09-27T09:01:39.098300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ls -a ./","metadata":{"papermill":{"duration":2.166803,"end_time":"2021-09-16T12:55:29.072261","exception":false,"start_time":"2021-09-16T12:55:26.905458","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:35.553876Z","iopub.execute_input":"2021-09-27T09:02:35.554658Z","iopub.status.idle":"2021-09-27T09:02:36.364507Z","shell.execute_reply.started":"2021-09-27T09:02:35.554608Z","shell.execute_reply":"2021-09-27T09:02:36.363650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing Test ","metadata":{"papermill":{"duration":1.60315,"end_time":"2021-09-16T12:55:32.052500","exception":false,"start_time":"2021-09-16T12:55:30.449350","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ls -a ../input/g2net-gravitational-wave-detection/sample_submission.csv","metadata":{"papermill":{"duration":2.07027,"end_time":"2021-09-16T12:55:35.548503","exception":false,"start_time":"2021-09-16T12:55:33.478233","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:38.334908Z","iopub.execute_input":"2021-09-27T09:02:38.335209Z","iopub.status.idle":"2021-09-27T09:02:39.144988Z","shell.execute_reply.started":"2021-09-27T09:02:38.335180Z","shell.execute_reply":"2021-09-27T09:02:39.144134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"assigning submission ids to the test set to make prediction on them","metadata":{"papermill":{"duration":1.36968,"end_time":"2021-09-16T12:55:38.271399","exception":false,"start_time":"2021-09-16T12:55:36.901719","status":"completed"},"tags":[]}},{"cell_type":"code","source":"sub = pd.read_csv('../input/g2net-gravitational-wave-detection/sample_submission.csv')\nx_test = sub[['id']]","metadata":{"papermill":{"duration":1.574057,"end_time":"2021-09-16T12:55:41.188823","exception":false,"start_time":"2021-09-16T12:55:39.614766","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:41.330584Z","iopub.execute_input":"2021-09-27T09:02:41.331297Z","iopub.status.idle":"2021-09-27T09:02:41.554898Z","shell.execute_reply.started":"2021-09-27T09:02:41.331256Z","shell.execute_reply":"2021-09-27T09:02:41.554159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_test.tail()","metadata":{"papermill":{"duration":1.403808,"end_time":"2021-09-16T12:55:44.738228","exception":false,"start_time":"2021-09-16T12:55:43.334420","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:42.348979Z","iopub.execute_input":"2021-09-27T09:02:42.349253Z","iopub.status.idle":"2021-09-27T09:02:42.360814Z","shell.execute_reply.started":"2021-09-27T09:02:42.349227Z","shell.execute_reply":"2021-09-27T09:02:42.360193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# test dataset\ntest_dataset = tf.data.Dataset.from_tensor_slices((x_test['id'].apply(get_npy_filepath, is_train=False).values))\ntest_dataset = test_dataset.map(preprocess_function_parse_tf, num_parallel_calls=tf.data.AUTOTUNE)\ntest_dataset = test_dataset.batch(batch_size)\ntest_dataset = test_dataset.prefetch(tf.data.AUTOTUNE)","metadata":{"papermill":{"duration":1.680839,"end_time":"2021-09-16T12:55:47.764851","exception":false,"start_time":"2021-09-16T12:55:46.084012","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:42.640021Z","iopub.execute_input":"2021-09-27T09:02:42.640863Z","iopub.status.idle":"2021-09-27T09:02:42.856062Z","shell.execute_reply.started":"2021-09-27T09:02:42.640823Z","shell.execute_reply":"2021-09-27T09:02:42.855289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_dataset","metadata":{"papermill":{"duration":1.391257,"end_time":"2021-09-16T12:55:50.508771","exception":false,"start_time":"2021-09-16T12:55:49.117514","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:44.463501Z","iopub.execute_input":"2021-09-27T09:02:44.463836Z","iopub.status.idle":"2021-09-27T09:02:44.472217Z","shell.execute_reply.started":"2021-09-27T09:02:44.463800Z","shell.execute_reply":"2021-09-27T09:02:44.471169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prediction","metadata":{"papermill":{"duration":1.537217,"end_time":"2021-09-16T12:55:53.443666","exception":false,"start_time":"2021-09-16T12:55:51.906449","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"Now, we will load the cnn model that we saved after training to make prediction on `test_dataset`","metadata":{"papermill":{"duration":1.365152,"end_time":"2021-09-16T12:55:56.200566","exception":false,"start_time":"2021-09-16T12:55:54.835414","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ls -a ./model/","metadata":{"papermill":{"duration":2.04561,"end_time":"2021-09-16T12:55:59.591262","exception":false,"start_time":"2021-09-16T12:55:57.545652","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:45.526910Z","iopub.execute_input":"2021-09-27T09:02:45.527406Z","iopub.status.idle":"2021-09-27T09:02:46.344826Z","shell.execute_reply.started":"2021-09-27T09:02:45.527368Z","shell.execute_reply":"2021-09-27T09:02:46.344012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"saved_cnn_model = tf.keras.models.load_model('./model/cnn_model.h5')","metadata":{"papermill":{"duration":1.568635,"end_time":"2021-09-16T12:56:02.571769","exception":false,"start_time":"2021-09-16T12:56:01.003134","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:47.853302Z","iopub.execute_input":"2021-09-27T09:02:47.854111Z","iopub.status.idle":"2021-09-27T09:02:48.141338Z","shell.execute_reply.started":"2021-09-27T09:02:47.854057Z","shell.execute_reply":"2021-09-27T09:02:48.140162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"saved_cnn_model","metadata":{"papermill":{"duration":1.591922,"end_time":"2021-09-16T12:56:05.519543","exception":false,"start_time":"2021-09-16T12:56:03.927621","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:48.142983Z","iopub.execute_input":"2021-09-27T09:02:48.143277Z","iopub.status.idle":"2021-09-27T09:02:48.149659Z","shell.execute_reply.started":"2021-09-27T09:02:48.143240Z","shell.execute_reply":"2021-09-27T09:02:48.148853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"retraining the saved model on `valid_dataset`\n\n> previously we set x = train_dataset","metadata":{"papermill":{"duration":1.346718,"end_time":"2021-09-16T12:56:08.217488","exception":false,"start_time":"2021-09-16T12:56:06.870770","status":"completed"},"tags":[]}},{"cell_type":"code","source":"saved_cnn_model.fit(x=valid_dataset, epochs=3, batch_size=batch_size, verbose=1)","metadata":{"papermill":{"duration":2388.180379,"end_time":"2021-09-16T13:35:57.786415","exception":false,"start_time":"2021-09-16T12:56:09.606036","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:02:51.238229Z","iopub.execute_input":"2021-09-27T09:02:51.238510Z","iopub.status.idle":"2021-09-27T09:47:44.888441Z","shell.execute_reply.started":"2021-09-27T09:02:51.238480Z","shell.execute_reply":"2021-09-27T09:47:44.887495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now saving the full model after training to make prediction on `test_dataset`","metadata":{"papermill":{"duration":1.816094,"end_time":"2021-09-16T13:36:01.442353","exception":false,"start_time":"2021-09-16T13:35:59.626259","status":"completed"},"tags":[]}},{"cell_type":"code","source":"saved_cnn_model.save('./model/full_cnn_model.h5')","metadata":{"papermill":{"duration":2.347351,"end_time":"2021-09-16T13:36:05.893299","exception":false,"start_time":"2021-09-16T13:36:03.545948","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:50:27.243404Z","iopub.execute_input":"2021-09-27T09:50:27.243707Z","iopub.status.idle":"2021-09-27T09:50:27.338717Z","shell.execute_reply.started":"2021-09-27T09:50:27.243677Z","shell.execute_reply":"2021-09-27T09:50:27.338006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"full_cnn_model = tf.keras.models.load_model('./model/full_cnn_model.h5')","metadata":{"papermill":{"duration":2.004736,"end_time":"2021-09-16T13:36:09.723381","exception":false,"start_time":"2021-09-16T13:36:07.718645","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:50:27.980113Z","iopub.execute_input":"2021-09-27T09:50:27.980821Z","iopub.status.idle":"2021-09-27T09:50:28.148579Z","shell.execute_reply.started":"2021-09-27T09:50:27.980777Z","shell.execute_reply":"2021-09-27T09:50:28.148001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction = full_cnn_model.predict(test_dataset)","metadata":{"papermill":{"duration":1392.683713,"end_time":"2021-09-16T13:59:24.199682","exception":false,"start_time":"2021-09-16T13:36:11.515969","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T09:50:29.413784Z","iopub.execute_input":"2021-09-27T09:50:29.414500Z","iopub.status.idle":"2021-09-27T10:08:39.531077Z","shell.execute_reply.started":"2021-09-27T09:50:29.414446Z","shell.execute_reply":"2021-09-27T10:08:39.529967Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction","metadata":{"papermill":{"duration":1.81119,"end_time":"2021-09-16T13:59:27.893086","exception":false,"start_time":"2021-09-16T13:59:26.081896","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T10:08:39.532928Z","iopub.execute_input":"2021-09-27T10:08:39.533208Z","iopub.status.idle":"2021-09-27T10:08:39.545808Z","shell.execute_reply.started":"2021-09-27T10:08:39.533178Z","shell.execute_reply":"2021-09-27T10:08:39.544627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction = prediction.flatten()","metadata":{"papermill":{"duration":2.051946,"end_time":"2021-09-16T13:59:31.778431","exception":false,"start_time":"2021-09-16T13:59:29.726485","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T10:08:39.547507Z","iopub.execute_input":"2021-09-27T10:08:39.547941Z","iopub.status.idle":"2021-09-27T10:08:39.558819Z","shell.execute_reply.started":"2021-09-27T10:08:39.547897Z","shell.execute_reply":"2021-09-27T10:08:39.557762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preparing to Submit","metadata":{"papermill":{"duration":1.786033,"end_time":"2021-09-16T13:59:35.347226","exception":false,"start_time":"2021-09-16T13:59:33.561193","status":"completed"},"tags":[]}},{"cell_type":"code","source":"submission = pd.DataFrame({'id': x_test.id, 'target': prediction})","metadata":{"papermill":{"duration":1.820529,"end_time":"2021-09-16T13:59:39.004342","exception":false,"start_time":"2021-09-16T13:59:37.183813","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T10:09:36.944196Z","iopub.execute_input":"2021-09-27T10:09:36.944805Z","iopub.status.idle":"2021-09-27T10:09:36.962869Z","shell.execute_reply.started":"2021-09-27T10:09:36.944761Z","shell.execute_reply":"2021-09-27T10:09:36.962116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.shape","metadata":{"papermill":{"duration":2.038658,"end_time":"2021-09-16T13:59:42.889555","exception":false,"start_time":"2021-09-16T13:59:40.850897","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T10:09:38.106991Z","iopub.execute_input":"2021-09-27T10:09:38.107916Z","iopub.status.idle":"2021-09-27T10:09:38.112552Z","shell.execute_reply.started":"2021-09-27T10:09:38.107869Z","shell.execute_reply":"2021-09-27T10:09:38.111990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.head()","metadata":{"papermill":{"duration":1.897309,"end_time":"2021-09-16T13:59:46.578193","exception":false,"start_time":"2021-09-16T13:59:44.680884","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T10:09:39.182965Z","iopub.execute_input":"2021-09-27T10:09:39.184032Z","iopub.status.idle":"2021-09-27T10:09:39.200768Z","shell.execute_reply.started":"2021-09-27T10:09:39.183993Z","shell.execute_reply":"2021-09-27T10:09:39.199487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv('./submission.csv', index= False)","metadata":{"papermill":{"duration":2.674311,"end_time":"2021-09-16T13:59:51.365777","exception":false,"start_time":"2021-09-16T13:59:48.691466","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2021-09-27T10:09:41.185322Z","iopub.execute_input":"2021-09-27T10:09:41.185616Z","iopub.status.idle":"2021-09-27T10:09:41.619509Z","shell.execute_reply.started":"2021-09-27T10:09:41.185586Z","shell.execute_reply":"2021-09-27T10:09:41.618666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I started with zero idea and I ended up learning about a lot of new things. I am very much thankful to all these resources that help me increase my knowledge and give me more insight as I proceed to improve my skills on my coding journey.\n\nI tried referencing as much as I could.","metadata":{"papermill":{"duration":1.794236,"end_time":"2021-09-16T13:59:55.213466","exception":false,"start_time":"2021-09-16T13:59:53.419230","status":"completed"},"tags":[]}}]}