{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAYgAAACBCAMAAADt5d1oAAAAilBMVEX///8AAAAfGxwVDxAIAAB1c3MLAAPQz9Dt7Ozh4OHKycl+fHwRCQwyLy/NzMwXERPc29uhoKB7eXq2tLW8u7urqqokHyDW1dVfXV7o6Og3MzWNjIwcFxjy8vKDgYJCP0ApJSZNSkuVlJRVUlNHREVta2tjYGGmpaa4t7eTkpNMSUnBwcE8ODpxb3CuY8bJAAALAklEQVR4nO2dbWOyLBTHO0DaGkUP68HCsqfZfdW+/9e7sW0tFRDNzDn+b65rmkb+hHMOHKDVKkudyWG3Xv+bL7ul3dKqgFZAGI9EYWFRPE3+niCKg/X6LP7hMHx2ef6s1pjjoXf57+yDIJg8uTx/VW2H9zrXv44UkcETS/N39UIR69z8PSb48LTC/GX9R2EcO/DOHO9JZfnTWnOIH3gDayWeID9g8/gRj9Plc8ryp/WC8Ef8SKdHX59Tlj8tAaIdP2JBPEUWRE1kQdREFkRNZEHURBZETWRB1EQWRE1kQdREFkRNZEHURBZETWRB1EQWRE1kQdREFkRNZEHURKWCeHmbhMvX/8LJ2+zugv01lQaiv90DAHEopg4BoIfJS95b+AW+tjEqB8TslQLlnBIBglEClHFMYB3meLTdZe87Z6FDDhPNlf2cuQ3dcGKi8Duvzt8OcypfeeQqA8RxA1hAmO6W41PXHbin4/CwQA7mBNp9s1uMz0Bh+10CJCiulMlVLuRLlP4AYiIIv78enJwyLYk/mVPRbKCR5KncD6K/Ac6cxaobf4m97mtAOIadgbk4BeIWhH8/CX9BKKIwUtQKF1BHfkauEZsymq0bEDinDAsy7gFjDGOGYZ5qt+8F0fkQzxC/S99Rf7wT7RWMMh6bvwIkbnG6+Zi7ChxEe3KGLrCNeQEFCMzCmZup/rUKurOcMivHf8CdYD5atXccM5qsFHeC6PYod97VJenuCKfspLtF5+wgckjeorNFDCWSQb/kB4isjEsYgXDcHB9/lELxum0/X7aXNnCaqBP3gdiKG561j7n1tqbo2vpL1FkwzmQ5nt4/RzFFYAaIvJkW8QLC0FI9Ui5B/KfdGAJfx8/fBaINiK4yPaMVQTBXfcrvMR4o5gGI28vrxBDQ1DxRuh4gDsw53vy5oiT+0+4A4e8IC/TV4VOngOGzgsQGo6myZWsThKUn3ynbGZUxUi1AeJS9xw7QhKErDsLfULw2C9q8NcNrKYklQY7G1r1jfpaZer/HiXE0UQsQEyDxZnaEUaxSFwexo3hn6kb6e4z3EhIzmF69Rpk6AaJt2YmZAGhqJmoBYkkh/tJOCI25EIVBtAnemAfO/gZjSWOyY3yePnqjEygaJ2EmAsM+lFqAWFEnXtwxODGfvygIYfYXucKqM3dSjYkrKkTGhL05T04b+NKBysjKVA8QmD4EhIhueb45jx5GJGnZP1jCgqXVFV8kffP9hamZKAvEfmPuIKT0KBABytnhc2llgoTPKe6S6XWtERylJ1wHOSY+W2kggPHiFz8IRJtSeYOh05bg+PvvilAv86olZVJz/WkmTKKJskAQNC1+8WNAiIYpKFCYM4/3sJwgw1RH6gJXtQiGZqLBIPY8u0mRqAvxMGxI6X+ZF3WAJxzfay0wNBPNBTFWv6R6vUOsvV+ZTCT2p6gXd8+61wILM2HwRlQGor+abza7jzeZV/8QEGcEuYdBL/LC8BbEKQyzPS9/zROWwP2hadTpVBGI7prQaIUfRrikj/MRILqEVbhIhH9Og8DX33Qw6HSqBsQSOAaYBgRENU2Huo8AseMFK0Qh+YgnuqkEiKv3ZdLpVAmIFSBYXep3J0SSPs4HgHjBPNcA2Z3yUq6VCzcOrUsyzUQVIN4AkWvXkb/HNBmoPgBESOSDBA/SEXCiODEQkZmg+r6WKkBQRG4eq2hOk/HuA0Dsk6uoPFYrDIlu1jiI1jvOGMKuAMQb0NjgbT85+vAAEANepamOXi6U8IsSIPweAq2ZqABEmyUe84ZD3EqUD+IIpMqWyU1H3wkQl2hC1/FVAYgzX8QPbCnEe+/LB9FmkKv7+06NcMoiJUG0hgQ5mjJVACJIRrhHknAhygch4ivN2bIlfKZe0hNMgYhGVPfqe1QAYpqstidIWLbSQXiBYqDmMXqlTipMTYMQZkITTTwDxNvDQcywU+HCsC+AUhVCAqLV10UTzQQh6lyFtvogwy4BEUUTSjPRTBBDB6rLX+yn8uEiyUC05pj9U9ylmSCWuEKn6cxlg6FSEFFCrMJM3ILwlMrOSKkXCBHoVjaxZylfC1gKIho1VJiJGxAzUCo7rbleIEYsq4OjkyFjjl3FYIMcRGQmkjH4V5FvQShnR/w2EB9ZIE7ThVY90wxnHylSkBUgIjMhHZu4AeGtlipl5w3WC4QIrPWv9FH50n29eqY9VTtGR9ITKhDCTKRz2FpNNdavGPRjk+5HW6sPwzBkKfWYLt+gABGZCdm8iWaCWNJq3NcjKLdPUIJQZDo1E0ToFMqkyasXqjAQLR2IaGwibSaaCeJUSS94NBStdGM0IMR1aTPRTBAuraKvacM0OXwaEJGZSM2baCYIL8BZCdz3a+SgQO2b6UDIzEQzQbTWnGnOliLxMHXTJrQgJGaioSAOLGtuyb06ZmTI6EGkM50aCmJMyGN3vnEx0s6rywDR6ic7nRoKYsCSF5Qrb8qJPKL+VgaIlJloKIhod6gH9r/6C04zJk1kgUiaiaaCCAnRthz3aYeZomfjqkwQwkzAzcSLpoJ4wainO3+X2g6fZg08ZYJImImmgmjNGRT9Yf6pq11TZwmIZC66kw0iPm+isSD6xedHtPWpB6FylO1WBiBiZqKxICJzXawH9gV0iWDRBGC94/opExAdYSa+o4nmghAP7Kz/hEJtrItBZlyXJvYjExCtLr1WruaCiOYMFQnqXgAt1J6v1+PUqMkzAhElxH4NYTcYxAAQLRBL7LimQvhrhndGNzUD8TNvosEghHejXu1KqS1gTag2p/xstiqZIYjO4stMNBlEq52/Qsy0U3FXoFnJLC5DEFczUQEIlHzFKgORX1G4q3Zdt6KxMx2CNQXR2hLEvUpA/NsnSlRfEMIEOOrHF0KO3cyNQXwlxFYAwk82EPUF8Q/rDMRsMDAf5DAH8RlNPGMtjtqC2FFlsnZumYNodQmC7sqCuH7JnmJNBJFTOUBEZiJ4578UxEKR6lhYxymj+/KS+fOAaM0Z4tPfCcLfp9YvuEt+tBJymVOzc4Hwp2j6S0G0DizftgB6jRf6tcHzKxeIVh9+LYgJ0Jx7lqgVrZZPeuUmaeYDIUKU3woiWqEt7xKWUnUme8oZLDVmepC9r0OkWMydE0Rr9+M1+bpdILKagcpBiCqB2N0Jrp87qjDJ3iE3GjgGW51QSmJzfPOC8Dn5mTGk3hYFslJJqwcRjVg6u7E7KCa3exq3z9EeQw60tSNIHjXc/ieWgNmHnLa/Dz8g1BsFZYIAQo2/siQQrRAYJ5QVE3UIwdGuW71hRm13zfbDmkxi9bMThjmNzvb7J3uaDbW2WaOO2xzba0lA5F4b/KLBHMDJu+fRl6Jt6GA6GptvtdE8nSCxfvMrxvHnYQhCvHfj4aiQXoeT0+BP7+3XihIsnPjo+wEHBUFY3SNvGp8i7gfJHHULohpteGzSZzo2syCq0RFu1wL1OGIJN96CqEg75lwbJ28dS8a9yIKoSMIq0MXFhe0MRUSV6ke1IKqSd3Y4wPlfD8S/6QjUgqhM/iu69KhQ6EkSSS2ICuW9jXb7eXJf409ZEDWRBVETWRA1kQVRE1kQNZEFURNZEDWRBVETWRA10RFIEkTJCZZWRuomRrVdk20qrcpXB+I1YOJUulON1VVrHksS2vNKd6qxuiqEW+u8pSx7J2Orh6jHnetKniFFkH+erlUpmkX5lBfPaXYgRotfWD1GM4a5w3sLThnKTPi0eqA6c6AcIU4hKCX93qqwBsP5ItgtLYZy9D8xFwfb6ygWrQAAAABJRU5ErkJggg==\" width=\"600\" height=\"400\"></center>\n\n## <center>G2Net Gravitational Wave Detection</center>\n### <center>🔭Find gravitational wave signals from binary black hole collisions📡</center>","metadata":{}},{"cell_type":"markdown","source":"# Table of Contents  <a id='0.1'></a>\n\n1. [Introduction](#1)\n2. [Import Packages](#2)\n3. [Utility](#3)\n4. [Data Overview](#4)\n   * [Trainig Labels](#4.1)\n   * [Train Folder](#4.2)\n   * [Time Series Dataframe](#4.3)\n4. [Visualization](#5)\n   * [Target Variables](#5.1)\n   * [Time Series (Target=1)](#5.2)\n   * [Time Series (Target=0)](#5.3)\n   * [Spectrograms](#5.4)\n   * [Q-Transform](#5.5)\n   * [Constant Q-Transform](#5.6)\n6. [Reference](#6)\n7. [Glossary](#0)","metadata":{}},{"cell_type":"markdown","source":"# 1. <a id=\"1\">Introduction</a>\n[Table of contents](#0.1)\n\nThis competition is hosted by [European Gravitational Observatory (EGO)](https://www.ego-gw.it/). The **goal** in this competition is to detect **Gravitational Wave** signals from the collisions of binary black holes. We are suppose to build a model which can analyze simulated gravitational wave time-series data from a network of Earth-based detectors.\n\nWe are provided with huge **72 GB time-series** dataset containing simulated gravitational wave measurements from a network of 3 gravitational wave [interferometers](#0.2) (LIGO Hanford, LIGO Livingston, and Virgo). Each time series contains either detector noise or detector noise plus a simulated gravitational wave signal. The task is to identify when a signal is present in the data or not.","metadata":{}},{"cell_type":"markdown","source":"# 2. <a id=\"2\">Import Packages</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"!python -m pip install gwpy\n!pip install -q nnAudio","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-07T13:01:31.294388Z","iopub.execute_input":"2021-07-07T13:01:31.294971Z","iopub.status.idle":"2021-07-07T13:01:57.014278Z","shell.execute_reply.started":"2021-07-07T13:01:31.294872Z","shell.execute_reply":"2021-07-07T13:01:57.013043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# basic utility\nimport os, gc\nimport glob\nimport warnings\nimport numpy as np\nimport pandas as pd\n\n# visualization\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom matplotlib import gridspec\n\n# gw visualization\nfrom gwpy.plot import Plot\nfrom gwpy.timeseries import TimeSeries\nfrom nnAudio.Spectrogram import CQT\n\n# deep learning/modelling\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\n\n%matplotlib inline\nwarnings.filterwarnings('ignore')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:01:57.016177Z","iopub.execute_input":"2021-07-07T13:01:57.016473Z","iopub.status.idle":"2021-07-07T13:02:00.204261Z","shell.execute_reply.started":"2021-07-07T13:01:57.016443Z","shell.execute_reply":"2021-07-07T13:02:00.203423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's have quick look at directory structure and files we are going to deal with. ","metadata":{}},{"cell_type":"code","source":"# directory\nprint('Competition Data/Files')\nROOT = '../input/g2net-gravitational-wave-detection/'\nos.listdir(ROOT)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:00.215092Z","iopub.execute_input":"2021-07-07T13:02:00.215545Z","iopub.status.idle":"2021-07-07T13:02:00.224312Z","shell.execute_reply.started":"2021-07-07T13:02:00.215498Z","shell.execute_reply":"2021-07-07T13:02:00.223683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. <a id=\"3\">Utility</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"# get filepath\ndef get_ts_file(file_id):\n    return f\"../input/g2net-gravitational-wave-detection/train/{file_id[0]}/{file_id[1]}/{file_id[2]}/{file_id}.npy\"\n\n# load .npy files\ndef generate_ts(file_id):\n    return np.load(f\"../input/g2net-gravitational-wave-detection/train/{file_id[0]}/{file_id[1]}/{file_id[2]}/{file_id}.npy\")\n\n# apply q transform\ndef apply_qtransform(waves, transform):\n    waves = np.hstack(waves)\n    waves = waves / np.max(waves)\n    waves = torch.from_numpy(waves).float()\n    image = transform(waves)\n    return image\n\n# generate dataframe\ndef generate_ts_df(id):\n    data = pd.DataFrame(np.asmatrix(generate_ts(id))).T\n    data.index.name = \"Time\"\n    data.reset_index(inplace=True)\n    data.rename(columns={0:\"LIGO Hanford\", 1: \"LIGO Livingston\", 2: \"Virgo\"}, inplace=True)\n    return data\n\n# plot timeseries\ndef plotTimeseries(id, target): \n    \n    data = generate_ts_df(id)\n\n    f = plt.figure(figsize=(16, 8))\n    gs = f.add_gridspec(3, 1)\n\n    f.suptitle(f\"For Observation {id} and Target = {target}\", fontsize=20)\n    \n    with sns.axes_style(\"whitegrid\"):\n        sub_spec = gridspec.GridSpecFromSubplotSpec(1, 2, subplot_spec=gs[0], width_ratios = (.80, .20), wspace=0.1, hspace=0.1)\n\n        ax1 = plt.Subplot(f, sub_spec[0])\n        ax2 = plt.Subplot(f, sub_spec[1])\n\n        sns.lineplot(data['Time'], data['LIGO Hanford'], color='red', ax=ax1)\n        sns.kdeplot(y = data['LIGO Hanford'], color='red', shade=True, ax=ax2)\n\n        f.add_subplot(ax1)\n        f.add_subplot(ax2)\n\n        ax1.axes.get_xaxis().set_visible(False)\n        ax1.set(title=\"Time Series (LIGO Hanford)\", ylabel = \"\")\n\n        ax2.set_yticks([])\n        ax2.set(ylabel='', title=\"Time Series (LIGO Hanford)\")\n        ax2.axes.get_xaxis().set_visible(False)\n\n    with sns.axes_style(\"whitegrid\"):\n        sub_spec = gridspec.GridSpecFromSubplotSpec(1, 2, subplot_spec=gs[1], width_ratios = (.80, .20), wspace=0.1, hspace=0.1)\n\n        ax1 = plt.Subplot(f, sub_spec[0])\n        ax2 = plt.Subplot(f, sub_spec[1])\n\n        sns.lineplot(data['Time'], data['LIGO Livingston'], color='blue', ax=ax1)\n        sns.kdeplot(y = data['LIGO Livingston'], color='blue', shade=True, ax=ax2)\n\n        f.add_subplot(ax1)\n        f.add_subplot(ax2)\n\n        ax1.axes.get_xaxis().set_visible(False)\n        ax1.set(title=\"Time Series (LIGO Livingston)\", ylabel=\"Strain Noise(10^-20)\")\n\n        ax2.set_yticks([])\n        ax2.axes.get_xaxis().set_visible(False)\n        ax2.set(ylabel='', title=\"Time Series (LIGO Livingston)\")\n\n    with sns.axes_style(\"whitegrid\"):\n        sub_spec = gridspec.GridSpecFromSubplotSpec(1, 2, subplot_spec=gs[2], width_ratios = (.80, .20), wspace=0.1, hspace=0.1)\n\n        ax1 = plt.Subplot(f, sub_spec[0])\n        ax2 = plt.Subplot(f, sub_spec[1])\n\n        sns.lineplot(data['Time'], data['Virgo'], color='green', ax=ax1)\n        sns.kdeplot(y = data['Virgo'], color='green', shade=True, ax=ax2)\n\n        f.add_subplot(ax1)\n        f.add_subplot(ax2)\n\n        ax1.set(title=\"Time Series (Virgo)\", ylabel='')\n\n        ax2.set_yticks([])\n        ax2.set(ylabel='', title=\"Time Series (Virgo)\")\n\n# plot spectrogram\ndef plotSpectrogram(id, target): \n    \n    data = generate_ts_df(id)\n\n    f = plt.figure(figsize=(16, 8))\n    gs = f.add_gridspec(1, 3)\n\n    f.suptitle(f\"Spectrogram For Observation {id} and Target = {target}\", fontsize=14)\n    \n    with sns.axes_style(\"whitegrid\"):\n        f.add_subplot(gs[0, 0])\n        plt.specgram(data['LIGO Hanford'], Fs=2048)\n        plt.title(\"Spectrogram (LIGO Hanford)\")\n        plt.ylabel(\"Frequency (Hz)\")\n        plt.xlabel(\"\")\n        plt.xticks(range(0, 3))\n\n    with sns.axes_style(\"whitegrid\"):\n        f.add_subplot(gs[0, 1])\n        plt.specgram(data['LIGO Livingston'], Fs=2048)\n        plt.title(\"Spectrogram (LIGO Livingston)\")\n        plt.xlabel(\"Time(seconds)\")\n        plt.ylabel(\"\")\n        plt.xticks(range(0, 3))\n\n    with sns.axes_style(\"whitegrid\"):\n        f.add_subplot(gs[0, 2])\n        plt.specgram(data['Virgo'], Fs=2048)\n        plt.title(\"Spectrogram (Virgo)\")\n        plt.xlabel(\"\")\n        plt.ylabel(\"\")\n        plt.xticks(range(0, 3))\n        \n# plot CQT\ndef plotCQT(id, target, transform):\n    \n    image = apply_qtransform(generate_ts(id), transform)\n    \n    f = plt.figure(figsize=(16, 8))\n    plt.title(f\"Constant Q-Transform For Observation {id} and Target = {target}\", fontsize=14)  \n    plt.imshow(image[0])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:43.809832Z","iopub.execute_input":"2021-07-07T13:02:43.810239Z","iopub.status.idle":"2021-07-07T13:02:43.840926Z","shell.execute_reply.started":"2021-07-07T13:02:43.810193Z","shell.execute_reply":"2021-07-07T13:02:43.839935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. <a id=\"4\">Data Overview</a>\n[Table of contents](#0.1)\n\nWe are provided with **train** and **test** folders. For every observation in **train.csv** we have one **.npy** file. Labels are provided in a **training_labels.csv** file. Each data sample (npy file) contains 3 time series (1 for each detector) and each spans 2 sec and is sampled at 2,048 Hz.\n\n## 4.1 <a id=\"4.1\">Training Labels</a>","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv(os.path.join(ROOT, 'training_labels.csv'))\ntrain.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:00.256982Z","iopub.execute_input":"2021-07-07T13:02:00.257557Z","iopub.status.idle":"2021-07-07T13:02:00.742560Z","shell.execute_reply.started":"2021-07-07T13:02:00.257494Z","shell.execute_reply":"2021-07-07T13:02:00.741462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"We have total {train.shape[0]} rows and {train.shape[1]} columns in train.csv.\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:00.745465Z","iopub.execute_input":"2021-07-07T13:02:00.745809Z","iopub.status.idle":"2021-07-07T13:02:00.750713Z","shell.execute_reply.started":"2021-07-07T13:02:00.745776Z","shell.execute_reply":"2021-07-07T13:02:00.749875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Missing Values","metadata":{}},{"cell_type":"code","source":"print(f'Missing values in training_label.csv in each columns:\\n{train.isnull().sum()}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:00.752366Z","iopub.execute_input":"2021-07-07T13:02:00.752879Z","iopub.status.idle":"2021-07-07T13:02:00.822688Z","shell.execute_reply.started":"2021-07-07T13:02:00.752838Z","shell.execute_reply":"2021-07-07T13:02:00.821446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Unique Values","metadata":{}},{"cell_type":"code","source":"print('Unique Values in each column of training_labels.csv')\nprint('##########################################')\nfor col in train:\n    print(f'{col}: {train[col].nunique()}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:00.824133Z","iopub.execute_input":"2021-07-07T13:02:00.824570Z","iopub.status.idle":"2021-07-07T13:02:01.053426Z","shell.execute_reply.started":"2021-07-07T13:02:00.824526Z","shell.execute_reply":"2021-07-07T13:02:01.052178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**📌 Points to note**\n   * We have two columns in training label file i.e \"id\" and \"target\".\n   * There are total **5,60,000** observations and 2 columns in training label file.\n   * Also there are no missing values.\n   * We can see that all **id** are unique.","metadata":{}},{"cell_type":"markdown","source":"## 4.2 <a id=\"4.2\">Train Folder</a>\n[Table of contents](#0.1)\n\nWe are given train and test folders. As you can see in **Data** pane each **.npy file** is organised in nested structure inside this folders at 3 subsequent levels. Each file consists of three time series one from each detector (LIGO Hanford, LIGO Livingston, and Virgo) and each spands 2 second. Let us chck the shape of each **.npy** file.","metadata":{}},{"cell_type":"code","source":"for i in train.id[:10]:\n    print(f\"The shape of file {i}: {generate_ts(i).shape}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:01.057247Z","iopub.execute_input":"2021-07-07T13:02:01.057541Z","iopub.status.idle":"2021-07-07T13:02:01.134745Z","shell.execute_reply.started":"2021-07-07T13:02:01.057512Z","shell.execute_reply":"2021-07-07T13:02:01.134040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**📌 Points to note**\n   * We can expect that shape of every **.npy** is going to be (3, 4096). The 3 represent three time series from respective detectors (LIGO Hanford, LIGO Livingston, and Virgo).","metadata":{}},{"cell_type":"markdown","source":"## 4.3 <a id=\"4.3\">Time Series DataFrame</a>\n[Table of contents](#0.1)\n\nLet's see how the .npy files look like. I have converted the numpy arrays in dataframe for more efficient working. ","metadata":{}},{"cell_type":"code","source":"generate_ts_df(train.id[0]).head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:01.135827Z","iopub.execute_input":"2021-07-07T13:02:01.136230Z","iopub.status.idle":"2021-07-07T13:02:01.154493Z","shell.execute_reply.started":"2021-07-07T13:02:01.136195Z","shell.execute_reply":"2021-07-07T13:02:01.153718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**📌 Points to note**\n   * Here we can see that signal from each of the three detectors. \n   * Each column contain time-series data from 3 different detectors (LIGO Hanford, LIGO Livingston and Virgo).","metadata":{}},{"cell_type":"markdown","source":"# 5. <a id=\"5\">Data Visualization</a>\n[Table of contents](#0.1)\n\n## 5.1 <a id=\"5.1\">Target Variables</a>","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16,10))\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = sns.countplot(train['target'], palette='Pastel1')\n    for p in ax.patches:\n        height = p.get_height()\n        ax.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/len(train)*100),\n                ha=\"center\", fontsize=12)\n    plt.title(\"Target Distribution\", fontsize=16)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:01.155585Z","iopub.execute_input":"2021-07-07T13:02:01.156017Z","iopub.status.idle":"2021-07-07T13:02:01.626871Z","shell.execute_reply.started":"2021-07-07T13:02:01.155979Z","shell.execute_reply":"2021-07-07T13:02:01.625804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**📌 Points to note**\n   * We can see that **target** variable is evenly distributed. Both 1 and 0 are 50%. ","metadata":{}},{"cell_type":"markdown","source":"## 5.2 <a id=\"5.2\">Time Series (Target=1)</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"plotTimeseries(train.id[0], train.target[0])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:01.628043Z","iopub.execute_input":"2021-07-07T13:02:01.628371Z","iopub.status.idle":"2021-07-07T13:02:03.398891Z","shell.execute_reply.started":"2021-07-07T13:02:01.628341Z","shell.execute_reply":"2021-07-07T13:02:03.397998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.3 <a id=\"5.3\">Time Series (Target=0)</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"plotTimeseries(train.id[1], train.target[1])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:03.400053Z","iopub.execute_input":"2021-07-07T13:02:03.400330Z","iopub.status.idle":"2021-07-07T13:02:04.908010Z","shell.execute_reply.started":"2021-07-07T13:02:03.400303Z","shell.execute_reply":"2021-07-07T13:02:04.907007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.4 <a id=\"5.4\">Spectrograms</a>\n[Table of contents](#0.1)\n\nA [spectrogram](#0.3) is a visual representation of the spectrum of frequencies of a signal as it varies with time. ","metadata":{}},{"cell_type":"code","source":"plotSpectrogram(train.id[1], train.target[1])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:04.909259Z","iopub.execute_input":"2021-07-07T13:02:04.909550Z","iopub.status.idle":"2021-07-07T13:02:05.453058Z","shell.execute_reply.started":"2021-07-07T13:02:04.909522Z","shell.execute_reply":"2021-07-07T13:02:05.451965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plotSpectrogram(train.id[0], train.target[0])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:05.454280Z","iopub.execute_input":"2021-07-07T13:02:05.454573Z","iopub.status.idle":"2021-07-07T13:02:05.998425Z","shell.execute_reply.started":"2021-07-07T13:02:05.454546Z","shell.execute_reply":"2021-07-07T13:02:05.997380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.5 <a id=\"5.5\">Q-Transform</a>\n[Table of contents](#0.1)\n\nOne of the most useful tools for filtering and visualising short-duration features in a TimeSeries is the [Q-transform](#0.3).\n\n### Target = 1","metadata":{}},{"cell_type":"code","source":"d1 = TimeSeries(generate_ts_df(train.id[0])['LIGO Hanford'], sample_rate=2048).q_transform(outseg=(0, 2)).plot()\nd1 = d1.gca()\nd1.set_xscale('seconds')\nd1.set_yscale('log')\nd1.set_title('Q-Transform (LIGO Hanford)')\nd1.set_xlabel('Time (s)')\nd1.set_ylabel('Frequency (Hz)')\nd1.grid(False)\nd1.colorbar(cmap='viridis', label='Normalized energy')\n\nd2 = TimeSeries(generate_ts_df(train.id[0])['LIGO Livingston'], sample_rate=2048).q_transform(outseg=(0, 2)).plot()\nd2 = d2.gca()\nd2.set_xscale('seconds')\nd2.set_yscale('log')\nd2.set_title('Q-Transform (LIGO Livingston)')\nd2.set_xlabel('Time (s)')\nd2.set_ylabel('Frequency (Hz)')\nd2.grid(False)\nd2.colorbar(cmap='viridis', label='Normalized energy')\n\nd3 = TimeSeries(generate_ts_df(train.id[0])['Virgo'], sample_rate=2048).q_transform(outseg=(0, 2)).plot()\nd3 = d3.gca()\nd3.colorbar(cmap='viridis', label='Normalized energy')\nd3.set_xscale('seconds')\nd3.set_yscale('log')\nd3.set_title('Q-Transform (Virgo)')\nd3.set_xlabel('Time (s)')\nd3.set_ylabel('Frequency (Hz)')\nd3.grid(False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:05.999755Z","iopub.execute_input":"2021-07-07T13:02:06.000036Z","iopub.status.idle":"2021-07-07T13:02:12.045569Z","shell.execute_reply.started":"2021-07-07T13:02:06.000009Z","shell.execute_reply":"2021-07-07T13:02:12.044582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Target = 0","metadata":{}},{"cell_type":"code","source":"d1 = TimeSeries(generate_ts_df(train.id[1])['LIGO Hanford'], sample_rate=2048).q_transform(outseg=(0, 2)).plot()\nd1 = d1.gca()\nd1.set_xscale('seconds')\nd1.set_yscale('log')\nd1.set_title('Q-Transform (LIGO Hanford)')\nd1.set_xlabel('Time (s)')\nd1.set_ylabel('Frequency (Hz)')\nd1.grid(False)\nd1.colorbar(cmap='viridis', label='Normalized energy')\n\nd2 = TimeSeries(generate_ts_df(train.id[1])['LIGO Livingston'], sample_rate=2048).q_transform(outseg=(0, 2)).plot()\nd2 = d2.gca()\nd2.set_xscale('seconds')\nd2.set_yscale('log')\nd2.set_title('Q-Transform (LIGO Livingston)')\nd2.set_xlabel('Time (s)')\nd2.set_ylabel('Frequency (Hz)')\nd2.grid(False)\nd2.colorbar(cmap='viridis', label='Normalized energy')\n\nd3 = TimeSeries(generate_ts_df(train.id[1])['Virgo'], sample_rate=2048).q_transform(outseg=(0, 2)).plot()\nd3 = d3.gca()\nd3.colorbar(cmap='viridis', label='Normalized energy')\nd3.set_xscale('seconds')\nd3.set_yscale('log')\nd3.set_title('Q-Transform (Virgo)')\nd3.set_xlabel('Time (s)')\nd3.set_ylabel('Frequency (Hz)')\nd3.grid(False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:12.047022Z","iopub.execute_input":"2021-07-07T13:02:12.047328Z","iopub.status.idle":"2021-07-07T13:02:17.558121Z","shell.execute_reply.started":"2021-07-07T13:02:12.047297Z","shell.execute_reply":"2021-07-07T13:02:17.554789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.6 <a id=\"5.6\">Constant Q-Transform</a>\n[Table of contents](#0.1)","metadata":{}},{"cell_type":"code","source":"transform = CQT(sr=2048, fmin=20, fmax=1024, hop_length=64)  ","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:17.559554Z","iopub.execute_input":"2021-07-07T13:02:17.559992Z","iopub.status.idle":"2021-07-07T13:02:17.610495Z","shell.execute_reply.started":"2021-07-07T13:02:17.559935Z","shell.execute_reply":"2021-07-07T13:02:17.609328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plotCQT(train.id[0], train.target[0], transform)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:51.480956Z","iopub.execute_input":"2021-07-07T13:02:51.481555Z","iopub.status.idle":"2021-07-07T13:02:52.099922Z","shell.execute_reply.started":"2021-07-07T13:02:51.481503Z","shell.execute_reply":"2021-07-07T13:02:52.099094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plotCQT(train.id[1], train.target[1], transform)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-07T13:02:54.406876Z","iopub.execute_input":"2021-07-07T13:02:54.407386Z","iopub.status.idle":"2021-07-07T13:02:54.764575Z","shell.execute_reply.started":"2021-07-07T13:02:54.407342Z","shell.execute_reply":"2021-07-07T13:02:54.763388Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6 <a id=\"6\">References</a>\n[Table of contents](#0.1)\n\n   * [Constant Q-Tramsform Demonstartion](https://www.kaggle.com/atamazian/nnaudio-constant-q-transform-demonstration).\n   * https://www.kaggle.com/yasufuminakama/g2net-efficientnet-b7-baseline-training/notebook","metadata":{}},{"cell_type":"markdown","source":"# Glossary <a id='0'></a>\n[Table of contents](#0.1)\n\n* <a id='0.2'>Interferometers</a> -  are investigative tools used in many fields of science and engineering. They are called interferometers because they work by merging two or more sources of light to create an interference pattern, which can be measured and analyzed; hence 'Interfere-o-meter', or interferometer. They are used to measure everything from the smallest variations on the surface of a microscopic organism, to the structure of enormous expanses of gas and dust in the distant Universe, and now, to detect gravitational waves. [source](https://www.ligo.caltech.edu/page/what-is-interferometer)\n\n* <a id='0.3'>Spectrograms</a> - A spectrogram is a visual way of representing the signal strength, or “loudness”, of a signal over time at various frequencies present in a particular waveform. Not only can one see whether there is more or less energy at, for example, 2 Hz vs 10 Hz, but one can also see how energy levels vary over time. [source](https://pnsn.org/spectrograms/what-is-a-spectrogram)","metadata":{}},{"cell_type":"markdown","source":"# IN PROGRESS...","metadata":{}}]}