{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# G2Net\n\nQuick Exploratory Data Analysis for [G2Net Gravitational Wave Detection](https://www.kaggle.com/c/g2net-gravitational-wave-detection) challenge\n\n*Thanks*\n\n* [🕳️G2Net🕳️ - EDA and Modeling](https://www.kaggle.com/ihelon/g2net-eda-and-modeling)\n* [MatchedFiltering](https://github.com/moble/MatchedFiltering)","metadata":{}},{"cell_type":"markdown","source":"## Useful sources\n\n- Gravitational waves (GW)\n    - [First observation](https://journals.aps.org/prl/pdf/10.1103/PhysRevLett.116.061102)\n    - [Detecting methods](https://iopscience.iop.org/article/10.1088/0264-9381/21/20/024/pdf) или [🏴‍☠️](https://sci-hub.ru/10.1088/0264-9381/21/20/024)\n- Scientific python modules\n    - [`PyCBC`](http://pycbc.org/pycbc/latest/html/) + [tutorials](https://github.com/gwastro/PyCBC-Tutorials)\n    - [`GWpy`](https://gwpy.github.io/docs/stable/index.html#)\n- Matching filtering (classical way to detect GW)\n    - [Matching Matched Filtering with Deep Networks for Gravitational-Wave Astronomy](https://journals.aps.org/prl/pdf/10.1103/PhysRevLett.120.141103)\n    - [Matching filtering notebook](https://nbviewer.jupyter.org/github/moble/MatchedFiltering/blob/binder/MatchedFiltering.ipynb), [code](https://github.com/moble/MatchedFiltering/blob/binder/utilities.py)\n- Ideas\n    - [Convolutional neural networks: A magic bullet for gravitational-wave detection?](https://arxiv.org/pdf/1904.08693.pdf)\n    - [Deep Learning for real-time gravitational wave detection and parameter estimation: Results with Advanced LIGO data](https://reader.elsevier.com/reader/sd/pii/S0370269317310390?token=CE27A8104F32CF9E7B6C1C42755C4BD09041FDCD4C97466386736235D570DE1DA77622F7C26D54BFD43F0D1E4E732409&originRegion=eu-west-1&originCreation=20210908151446)\n    - [Gravitational-wave parameter estimation with autoregressive neural network flows](https://arxiv.org/pdf/2002.07656.pdf)\n    - [Improving Deep Speech Denoising by Noisy2Noisy Signal Mapping](https://arxiv.org/ftp/arxiv/papers/1904/1904.12069.pdf)\n- Kaggle notebooks\n    - [Data Transformations + EfficientNet](https://www.kaggle.com/ihelon/g2net-eda-and-modeling)\n- Kaggle Discussion\n    - [signal fixed time](https://www.kaggle.com/c/g2net-gravitational-wave-detection/discussion/268553)","metadata":{}},{"cell_type":"markdown","source":"## Setup environment\n\n### Files\n**train/** - the training set files, one npy file per observation; labels are provided in a files shown below   \n**test/** - the test set files; you must predict the probability that the observation contains a gravitational wave   \n**training_labels.csv** - target values of whether the associated signal contains a gravitational wave   \n**sample_submission.csv** - a sample submission file in the correct format","metadata":{}},{"cell_type":"code","source":"!pip install -q nnAudio -qq # to draw q transform\n!pip install efficientnet_pytorch -q # to train the model ","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:33:37.449550Z","iopub.execute_input":"2021-09-09T03:33:37.449979Z","iopub.status.idle":"2021-09-09T03:33:58.408561Z","shell.execute_reply.started":"2021-09-09T03:33:37.449889Z","shell.execute_reply":"2021-09-09T03:33:58.407426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\n\nimport numpy as np\nimport pandas as pd\n\nimport scipy\nimport scipy.signal\nfrom scipy.interpolate import InterpolatedUnivariateSpline\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport torch\nfrom torch import nn\nimport torch.nn.functional as F\nimport torchaudio\n\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(device)","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:33:58.410723Z","iopub.execute_input":"2021-09-09T03:33:58.411146Z","iopub.status.idle":"2021-09-09T03:34:00.664411Z","shell.execute_reply.started":"2021-09-09T03:33:58.411099Z","shell.execute_reply":"2021-09-09T03:34:00.663607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = pd.read_csv(\"../input/g2net-gravitational-wave-detection/training_labels.csv\")\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:00.666365Z","iopub.execute_input":"2021-09-09T03:34:00.666819Z","iopub.status.idle":"2021-09-09T03:34:01.179969Z","shell.execute_reply.started":"2021-09-09T03:34:00.666781Z","shell.execute_reply":"2021-09-09T03:34:01.179220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.countplot(data=train_df, x=\"target\"); # equal class fractions","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:01.181317Z","iopub.execute_input":"2021-09-09T03:34:01.181688Z","iopub.status.idle":"2021-09-09T03:34:01.517449Z","shell.execute_reply.started":"2021-09-09T03:34:01.181658Z","shell.execute_reply":"2021-09-09T03:34:01.516487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data exploration","metadata":{}},{"cell_type":"code","source":"!wget https://github.com/moble/MatchedFiltering/raw/binder/utilities.py # useful functions to preprocess data\n!wget https://github.com/moble/MatchedFiltering/raw/binder/Data/NR_GW150914.txt # gravity wave signal\n!wget -N https://gist.github.com/nikita-p/8c67b7228f7b3930025bab90ef4ae8ff/raw/1f439dde4b2459bbc524ad3021e011670c1c8552/g2net_models.py # 1Dcnns (import g2net_models)","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:01.519991Z","iopub.execute_input":"2021-09-09T03:34:01.520638Z","iopub.status.idle":"2021-09-09T03:34:08.861398Z","shell.execute_reply.started":"2021-09-09T03:34:01.520586Z","shell.execute_reply":"2021-09-09T03:34:08.860208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from utilities import bandpass, bump_function\nfrom scipy.signal import welch\nfrom nnAudio.Spectrogram import CQT1992v2\n\ndef fade(signal, fade_length=0.075):\n    \"\"\"Customize `utilities.fade` to work with 2-dim arrays\n    \"\"\"\n    n = signal.shape[1]\n    t = np.arange(n, dtype=float)\n    return signal * bump_function(t, t[0], t[int(fade_length*n)], t[int(-1-fade_length*n)], t[-1])\n\n\nclass Dataset(torch.utils.data.Dataset):\n    def __convert_image_id_2_path(self, image_id: str, is_train: bool = True) -> str:\n        folder = \"train\" if is_train else \"test\"\n        return \"../input/g2net-gravitational-wave-detection/{}/{}/{}/{}/{}.npy\".format(\n            folder, image_id[0], image_id[1], image_id[2], image_id \n        )\n    \n    def estimate_noise_spectrum(self, signal):\n        \"\"\"Estimates noise spectrum via scipy.signal.welch\n        \n        Parameters\n        ----------\n        signal : np.array\n            signal time representation\n            \n        Returns\n        -------\n        noise_interpolator\n            interpolator for whitening\n        \"\"\"\n        \n        number_of_chunks = 8\n        length = signal.shape[1]\n        points_per_chunk = 2**int(np.log2(length/number_of_chunks))\n        f_noise, noise_welch = welch(signal, self.sampling_rate, nperseg=points_per_chunk)\n        noise_spectral_density = np.sqrt(2 * length * noise_welch / self.sampling_rate)\n        noise_interpolator = [InterpolatedUnivariateSpline(f_noise, noise_spectral_density[i]) for i in range(signal.shape[0])]\n        return noise_interpolator\n    \n    def raw_signal(self, idx: int):\n        \"\"\"Prepares raw signal of item (times, freqs)\n        \n        Parameters\n        ----------\n        idx : int\n            event number\n            \n        Returns\n        -------\n        raw_signal : np.array\n            raw signal event from 3 detectors (LIGO_H, LIGO_L, Virgo), time representation\n        raw_freq : np.array\n            raw signal event, freq representation\n        \"\"\"\n        \n        image_id = self.img[idx]\n        path = self.__convert_image_id_2_path(image_id)\n        raw_signal = np.load(path)\n        raw_signal = fade(raw_signal)\n        raw_freq = np.fft.rfft(raw_signal) / self.sampling_rate\n        return raw_signal, raw_freq\n    \n    def filtered_signal(self, idx: int, return_interpolators: bool = False):\n        \"\"\"Prepares filtered signal of item (whitening + bandpass filter from 35 to upper bound)\n        \n        Parameters\n        ----------\n        idx : int\n            event number\n        return_interpolators : bool\n            return interpolators for whitening or not\n            \n        Returns\n        -------\n        filtered_signal : np.array\n            filtered signal (time representation)\n        filtered_freq : np.array\n            filtered signal (freq representation)\n        noise_interpolator\n            interpolators for whitening if return_interpolators is True  \n        \"\"\"\n        \n        raw_signal, raw_freq = self.raw_signal(idx)\n        \n        # Estimate the noise spectrum\n        noise_interpolator = self.estimate_noise_spectrum(raw_signal)\n        raw_frequencies = np.fft.rfftfreq(raw_signal.shape[1], d=1/self.sampling_rate)\n        raw_bkg_freq = np.array([noise_interpolator[i](raw_frequencies) for i in range(3)])\n\n        # Equalize the data using this noise estimate\n        filtered_freq = raw_freq / raw_bkg_freq\n        filtered_signal = fade(self.sampling_rate * np.fft.irfft(filtered_freq))\n        \n        # Finally, bandpass the equalized data\n        filtered_signal = bandpass(filtered_signal, self.sampling_rate, lower_end=35.0,\n                                   upper_end=self.upper_bandpass_frequency)\n        filtered_freq = np.fft.rfft(filtered_signal) / self.sampling_rate\n        \n        if return_interpolators:\n            return filtered_signal, filtered_freq, noise_interpolator\n        \n        return filtered_signal, filtered_freq\n    \n    def matching_filtering(self, idx: int, bh_signal_phase: complex = 0.1j) -> np.array:\n        \"\"\"Matching filtering for filtered signal with simulated signal from file. \n        ATTENTION: PRELIMINARY!\n        \n        Parameters\n        ----------\n        idx : int\n            event number\n        bh_signal_phase : complex\n            phase of the simulated signal\n        \n        Returns\n        -------\n        resulted_signal : np.array\n            filtered signal\n        \"\"\"\n        \n        filtered_signal, filtered_freq, noise_interpolator = self.filtered_signal(idx, return_interpolators = True)\n        \n        bh_signal = np.exp(bh_signal_phase*np.pi)*self.bh_signal.copy().view('complex')\n        bh_signal = fade(bh_signal.real.reshape(1, -1))\n        bh_freq = np.fft.rfft(bh_signal) / self.sampling_rate\n        \n        bh_frequencies = np.fft.rfftfreq(bh_signal.shape[1], d=1/self.sampling_rate)\n        bh_bkg_freq = np.array([noise_interpolator[i](bh_frequencies) for i in range(3)])\n\n        filtered_bh_freq = bh_freq.copy() / bh_bkg_freq\n        filtered_bh_signal = fade(self.sampling_rate * np.fft.irfft(filtered_bh_freq)\n                                  [:, bh_signal.shape[1]//2 - self.sampling_rate:bh_signal.shape[1]//2 + self.sampling_rate])\n\n        filtered_bh_signal = bandpass(filtered_bh_signal, self.sampling_rate, lower_end=35.0,\n                                   upper_end=self.upper_bandpass_frequency)\n        filtered_bh_freq = np.fft.rfft(filtered_bh_signal) / self.sampling_rate\n        \n        # Offset\n        match = np.fft.irfft(filtered_freq * filtered_bh_freq.conjugate())\n        offsets = np.argmax(abs(match), axis=1)\n        resulted_signal = np.array([np.roll(filtered_signal[i], -offsets[i])*filtered_bh_signal[i] for i in range(3)])\n        resulted_freq = np.fft.rfft(resulted_signal) / self.sampling_rate\n        \n        return resulted_signal, resulted_freq\n    \n    def q_transformed(self, idx: int, filtered: bool = True):\n        \"\"\"Q-transform signal\n        \n        Parameters\n        ----------\n        idx : int\n            event number\n        filtered : bool\n            use filtered signal from `self.filtered_signal` or raw from `self.raw_signal`\n            \n        Returns\n        -------\n        signal_transformed : torch.Tesnor\n            q-transformed signal\n        \"\"\"\n        \n        if filtered:\n            signal, freq = self.filtered_signal(idx)\n        else:\n            signal, freq = self.raw_signal(idx)\n        \n        if self.q_transform is None:\n            self.q_transform = CQT1992v2(sr=self.sampling_rate, fmin=35, fmax=self.upper_bandpass_frequency, hop_length=32)\n        \n        t_signal_torch = torch.from_numpy(signal.astype(np.float32))\n        signal_transformed = self.q_transform(t_signal_torch)\n        \n        return signal_transformed\n    \n    def event_spectrogram(self, idx: int, filtered: bool = True):\n        \"\"\"Returns spectrogram\n        \n        Parameters\n        ----------\n        idx : int\n            event number\n        filtered : bool\n            use filtered signal from `self.filtered_signal` or raw from `self.raw_signal`\n            \n        Returns\n        -------\n        spectrogram : torch.Tesnor\n            signal spectrogram\n        \"\"\"\n        if self.spectrogram is None:\n            self.spectrogram = torchaudio.transforms.Spectrogram(n_fft = 1024, hop_length=2, power=2, win_length = 150)\n        \n        if filtered:\n            signal, freq = self.filtered_signal(idx)\n        else:\n            signal, freq = self.raw_signal(idx)\n            signal *= 1e21\n        spectrogram = self.spectrogram(torch.from_numpy(signal.astype(np.float32)))\n        return spectrogram\n    \n    def mel_transformed(self, idx: int, filtered: bool = True):\n        \"\"\"Returns mel spectrogram\n        \n        Parameters\n        ----------\n        idx : int\n            event number\n        filtered : bool\n            use filtered signal from `self.filtered_signal` or raw from `self.raw_signal`\n            \n        Returns\n        -------\n        mel : torch.Tesnor\n            mel signal\n        \"\"\"\n        if self.mel_transform is None:\n            self.mel_transform = torchaudio.transforms.MelScale(256, sample_rate=2048, f_min=35, f_max=350)\n        \n        spectra = self.event_spectrogram(idx, filtered)\n        mel = self.mel_transform(spectra)\n        return mel\n        \n    def load_bh_signal(self, bh_signal_filename: str):\n        \"\"\"Loads simulated signal (black holes waves)\n        \"\"\"\n        \n        bh_signal_sampling_rate = 4096\n        assert bh_signal_sampling_rate%self.sampling_rate == 0\n        bh_signal = np.loadtxt(bh_signal_filename).view(dtype=complex)[::(bh_signal_sampling_rate//self.sampling_rate), 0]\n        return bh_signal\n    \n    def __init__(self, train_df_filename: str = \"../input/g2net-gravitational-wave-detection/training_labels.csv\", \n                 bh_signal_filename: str = 'NR_GW150914.txt'):\n        super().__init__()\n        self.sampling_rate = 2048\n        self.upper_bandpass_frequency = 350\n        \n        train_df = pd.read_csv(train_df_filename)\n        self.img, self.y = train_df.id.values, train_df.target.values\n        self.len = len(self.img)\n        \n        self.bh_signal = self.load_bh_signal(bh_signal_filename)\n        \n        self.q_transform = None\n        self.spectrogram = None\n        self.mel_transform = None\n        \n    def __getitem__(self, idx: int):\n        \"\"\"Get event using index (for model training)\n        \n        Parameters\n        ----------\n        idx : int\n            event number\n        \n        Returns\n        -------\n        tuple\n            filtered signal (time repr), true label\n        \"\"\"\n        x = self.filtered_signal(idx)[0].astype(np.float32)\n#         x = self.mel_transformed(idx)\n        true = self.y[idx]\n        return (x, true)\n    \n    def __len__(self):\n        \"\"\"Returns length of the dataset\n        \"\"\"\n        \n        return self.len\n    \ndef draw_1D_signal(t_signal: np.array, freq: np.array, title: str, event: int = None, target: int = None, t_xlim: tuple = (0, 4096)):\n    plt.figure(dpi=120, tight_layout=True, figsize=(9, 3))\n    event = '' if event is None else f', event {event}'\n    target = '' if target is None else f', target {target}'\n    plt.suptitle(f'{title}{event}{target}')\n    plt.subplot(121)\n    plt.plot(t_signal.T, alpha=0.8, label=['LIGO_H', 'LIGO_L', 'Virgo'], lw=1)\n    plt.xlabel('$\\sim t$, s')\n    plt.xlim(t_xlim)\n    plt.legend()\n    plt.grid(ls=':')\n    plt.subplot(122)\n    plt.loglog(np.abs(freq.T), alpha=0.8, lw=1)\n    plt.xlabel('$\\sim f$, Hz')\n    plt.grid(ls=':')\n\ndef draw_2D_signal(img: np.array, title: str, event: int = None, target: int = None):\n    event = '' if event is None else f', event {event}'\n    target = '' if target is None else f', target {target}'\n    fig, ax = plt.subplots(1, 3, dpi=120, figsize=(4*3, 4))\n    for i, (a, detector) in enumerate(zip(ax, ('LIGO_H', 'LIGO_L', 'Virgo'))):\n        a.imshow(img[i], interpolation='bicubic', aspect='auto', cmap='jet')\n        a.set(title=f'{title} {detector}{event}{target}')\n    \nfull_dataset = Dataset()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-09-09T03:34:08.863699Z","iopub.execute_input":"2021-09-09T03:34:08.864173Z","iopub.status.idle":"2021-09-09T03:34:10.697353Z","shell.execute_reply.started":"2021-09-09T03:34:08.864121Z","shell.execute_reply":"2021-09-09T03:34:10.696331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Look at the `14` event \n(the best that i've seen)","metadata":{"execution":{"iopub.status.busy":"2021-09-09T02:49:47.790123Z","iopub.execute_input":"2021-09-09T02:49:47.790855Z","iopub.status.idle":"2021-09-09T02:49:47.818446Z","shell.execute_reply.started":"2021-09-09T02:49:47.790800Z","shell.execute_reply":"2021-09-09T02:49:47.817528Z"}}},{"cell_type":"code","source":"idx = 14","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:10.698786Z","iopub.execute_input":"2021-09-09T03:34:10.699140Z","iopub.status.idle":"2021-09-09T03:34:10.702791Z","shell.execute_reply.started":"2021-09-09T03:34:10.699087Z","shell.execute_reply":"2021-09-09T03:34:10.701978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"t_signal, freq = full_dataset.raw_signal(idx)\ndraw_1D_signal(t_signal, freq, 'Raw signal', event=idx, target=full_dataset.y[idx])","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:10.704976Z","iopub.execute_input":"2021-09-09T03:34:10.705456Z","iopub.status.idle":"2021-09-09T03:34:12.131144Z","shell.execute_reply.started":"2021-09-09T03:34:10.705423Z","shell.execute_reply":"2021-09-09T03:34:12.129996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"t_signal, freq = full_dataset.filtered_signal(idx)\ndraw_1D_signal(t_signal, freq, 'Filtered signal', event=idx, target=full_dataset.y[idx])","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:12.132804Z","iopub.execute_input":"2021-09-09T03:34:12.133165Z","iopub.status.idle":"2021-09-09T03:34:13.401953Z","shell.execute_reply.started":"2021-09-09T03:34:12.133130Z","shell.execute_reply":"2021-09-09T03:34:13.400870Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"signal, freq = full_dataset.matching_filtering(idx, bh_signal_phase=0.2j)\ndraw_1D_signal(signal, freq, 'Matching filtering', t_xlim=(2.7e3, 3e3), event=idx, target=full_dataset.y[idx])\nimport scipy.integrate\nI = [scipy.integrate.simps(signal[i], np.arange(len(signal[i]))/full_dataset.sampling_rate) for i in range(3)]\nprint('Correlations:')\nfor title, i in zip(('LIGO_H', 'LIGO_L', 'Virgo'), I):\n    print(f'c ({title}) = {i:.3f}')","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:13.403332Z","iopub.execute_input":"2021-09-09T03:34:13.403639Z","iopub.status.idle":"2021-09-09T03:34:14.406510Z","shell.execute_reply.started":"2021-09-09T03:34:13.403609Z","shell.execute_reply":"2021-09-09T03:34:14.405291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"q_transformed = full_dataset.q_transformed(idx, filtered=True)\ndraw_2D_signal(q_transformed, 'CQT', event=idx, target=full_dataset.y[idx])","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:14.408201Z","iopub.execute_input":"2021-09-09T03:34:14.408624Z","iopub.status.idle":"2021-09-09T03:34:15.347554Z","shell.execute_reply.started":"2021-09-09T03:34:14.408580Z","shell.execute_reply":"2021-09-09T03:34:15.346317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spectra = full_dataset.event_spectrogram(idx, filtered=True)\ndraw_2D_signal(spectra, 'Spectrgm', event=idx, target=full_dataset.y[idx])","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:15.349513Z","iopub.execute_input":"2021-09-09T03:34:15.349875Z","iopub.status.idle":"2021-09-09T03:34:17.729202Z","shell.execute_reply.started":"2021-09-09T03:34:15.349841Z","shell.execute_reply":"2021-09-09T03:34:17.728022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mel_spectra = full_dataset.mel_transformed(idx, filtered=True)\ndraw_2D_signal(mel_spectra, 'mel', event=idx, target=full_dataset.y[idx])","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:17.730742Z","iopub.execute_input":"2021-09-09T03:34:17.731094Z","iopub.status.idle":"2021-09-09T03:34:19.719860Z","shell.execute_reply.started":"2021-09-09T03:34:17.731060Z","shell.execute_reply":"2021-09-09T03:34:19.718780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Models\n\n1D conv models\n\n2D conv models (more effiective i think)","metadata":{}},{"cell_type":"code","source":"full_dataset = Dataset()\ntrain_size = int(0.8 * len(full_dataset))\ntest_size = len(full_dataset) - train_size\ntrain_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])\n\ntrain_loader = torch.utils.data.DataLoader(train_dataset, batch_size = 128, shuffle = True)\ntest_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 128)\n\nprint('Train dataset len:', len(train_dataset))\nprint('Test dataset len:', len(test_dataset))","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:19.721323Z","iopub.execute_input":"2021-09-09T03:34:19.721896Z","iopub.status.idle":"2021-09-09T03:34:21.524773Z","shell.execute_reply.started":"2021-09-09T03:34:19.721853Z","shell.execute_reply":"2021-09-09T03:34:21.524060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nbatch0 = next(iter(train_loader)) # time to create batch on-the-fly (long time)","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:21.525745Z","iopub.execute_input":"2021-09-09T03:34:21.526144Z","iopub.status.idle":"2021-09-09T03:34:23.400676Z","shell.execute_reply.started":"2021-09-09T03:34:21.526114Z","shell.execute_reply":"2021-09-09T03:34:23.399350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import g2net_models # 1D convolutional models\nmodel = g2net_models.Model1DCNN().to(device) #AUC: 0.829","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:23.402310Z","iopub.execute_input":"2021-09-09T03:34:23.402732Z","iopub.status.idle":"2021-09-09T03:34:23.419346Z","shell.execute_reply.started":"2021-09-09T03:34:23.402678Z","shell.execute_reply":"2021-09-09T03:34:23.418343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sanity test","metadata":{}},{"cell_type":"code","source":"%%time\nwith torch.no_grad():\n    res = model(batch0[0].to(device))\n    print('Predictions slice')\n    print(res[:3])\n    print('\\nTrue labels shape:', batch0[1].shape)\n    print('Model predicts shape:', res.shape, '\\n')","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:36:53.772958Z","iopub.execute_input":"2021-09-09T03:36:53.773380Z","iopub.status.idle":"2021-09-09T03:36:53.904025Z","shell.execute_reply.started":"2021-09-09T03:36:53.773341Z","shell.execute_reply":"2021-09-09T03:36:53.902950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"optimizer = torch.optim.Adam(model.parameters(), lr = 1e-4)#, weight_decay = 1e-5)\nloss_fn = nn.CrossEntropyLoss()","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:23.554879Z","iopub.execute_input":"2021-09-09T03:34:23.555356Z","iopub.status.idle":"2021-09-09T03:34:23.560683Z","shell.execute_reply.started":"2021-09-09T03:34:23.555301Z","shell.execute_reply":"2021-09-09T03:34:23.559739Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import warnings\ndef train(model, optimizer, loss_fn, dataloader, device = 'cpu', demo: bool = False):\n    if demo:\n        warnings.warn('Demo mode. Using only first 30 batches')\n    model.train()\n    losses, accs = list(), list()\n    for i, batch in enumerate(dataloader):\n        x, y = batch\n        x, y = x.float().to(device), y.to(device)\n        \n        pred = model(x)\n        loss = loss_fn(pred, y)\n        losses.append(loss)\n        accs.append((pred.argmax(dim=1)==y).sum()/len(y))\n        \n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n        \n        if len(losses)==10:\n            mean_loss = sum(losses)/len(losses)\n            mean_acc = sum(accs)/len(accs)\n            print(f'Batch: {i+1}, Loss: {mean_loss:.4f}, acc: {mean_acc:.2%}')\n            losses, accs = list(), list()\n            \n        if demo and (i>30):\n             break\n\nfrom tqdm import tqdm\ndef test(model, loss_fn, dataloader, device = 'cpu'):\n    model.eval()\n    losses, accs = list(), list()\n    preds, trues = torch.Tensor().to(device), torch.Tensor().to(device)\n    with torch.no_grad():\n        for i, batch in tqdm(zip(range(50), dataloader)):\n            x, y = batch\n            x, y = x.to(device), y.to(device)\n\n            pred = model(x)\n            pred = nn.functional.softmax(pred, dim=1)\n            loss = loss_fn(pred, y)\n            losses.append(loss)\n            accs.append((pred.argmax(dim=1)==y).sum()/len(y))\n            preds = torch.cat([preds, pred[:, 1]])\n            trues = torch.cat([trues, y])\n    mean_loss = sum(losses)/len(losses)\n    mean_acc = sum(accs)/len(accs)\n    print(f'Test loss: {mean_loss:.4f}, acc: {mean_acc:.2%}')\n    return preds, trues","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:23.561916Z","iopub.execute_input":"2021-09-09T03:34:23.562230Z","iopub.status.idle":"2021-09-09T03:34:23.578637Z","shell.execute_reply.started":"2021-09-09T03:34:23.562200Z","shell.execute_reply":"2021-09-09T03:34:23.577273Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train(model, optimizer, loss_fn, train_loader, device, demo=True)","metadata":{"execution":{"iopub.status.busy":"2021-09-09T03:34:23.579961Z","iopub.execute_input":"2021-09-09T03:34:23.580301Z","iopub.status.idle":"2021-09-09T03:35:30.609063Z","shell.execute_reply.started":"2021-09-09T03:34:23.580258Z","shell.execute_reply":"2021-09-09T03:35:30.607992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}