{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Goal/ Explanation\n\nIn this competition, you’ll aim to detect GW signals from the mergers of binary black holes. Specifically, you'll build a model to analyze simulated GW time-series data from a network of Earth-based detectors.  \n(このコンテストでは、連星ブラックホールの合併によるGW信号の検出を目指します。具体的には、地球ベースの検出器のネットワークからシミュレートされたGW時系列データを分析するためのモデルを構築する)\n\nThese signals are unimaginably tiny ripples in the fabric of space-time and even though the global network of GW detectors are some of the most sensitive instruments on the planet, the signals are buried in detector noise.（要約:これらの信号はとても小さなさざなみであり、かなり精密なGW検知機をもってしても、ノイズに埋もれてしまう)  \n\nIn this competition you are provided with a training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo).   \n(このコンペティションでは、3つの重力波干渉計（LIGOハンフォード、LIGOリビングストン、およびバーゴ）のネットワークからのシミュレートされた重力波測定値を含む時系列データのトレーニングセットが提供されます。)\n\nEach time series contains either detector noise or detector noise plus a simulated gravitational wave signal.   \n(各時系列には、検出器ノイズまたは検出器ノイズのいずれかと、シミュレートされた重力波信号が含まれます。)  \nThe task is to identify when a signal is present in the data (target=1).  ※シグナルなので、重力波の波形をそのまま表してはいない？  \n(タスクは、信号がデータに存在するときを識別することです（target = 1）  )　⇨ 存在する確率を出す\n\nEach data sample (npy file) contains 3 time series (1 for each detector) and each spans 2 sec and is sampled at 2,048 Hz.  \n(各データサンプル（npyファイル）には3つの時系列（各検出器に1つ）が含まれ、それぞれが2秒にまたがり、2,048Hzでサンプリングされます。)\n  \n信号対雑音比（SNR） : 信号がどの程度検出可能であるかを示す最も有益な尺度","metadata":{}},{"cell_type":"markdown","source":"## Competition Metric\nAUC - ROC curve","metadata":{}},{"cell_type":"markdown","source":"## Reference\n* [kaggle日本語初心者取り組み](https://www.kaggle.com/tensorchoko/g2net-gravitational-timm-efn-train#%F0%9F%8E%B5import)\n* [top kaggler Modeling](https://www.kaggle.com/ihelon/g2net-eda-and-modeling)  \n* [Good Tutorial EDA](https://www.kaggle.com/andradaolteanu/g2net-searching-the-sky-pytorch-effnet-w-meta)","metadata":{}},{"cell_type":"markdown","source":"### Library","metadata":{}},{"cell_type":"code","source":"import os\nimport random\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport time","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:21.218496Z","iopub.execute_input":"2021-08-18T23:37:21.218933Z","iopub.status.idle":"2021-08-18T23:37:22.134892Z","shell.execute_reply.started":"2021-08-18T23:37:21.218842Z","shell.execute_reply":"2021-08-18T23:37:22.133877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Look at training_labels.csv","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('../input/g2net-gravitational-wave-detection/training_labels.csv')\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:22.136901Z","iopub.execute_input":"2021-08-18T23:37:22.137362Z","iopub.status.idle":"2021-08-18T23:37:22.603584Z","shell.execute_reply.started":"2021-08-18T23:37:22.137316Z","shell.execute_reply":"2021-08-18T23:37:22.602617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* target = 1 GW exist / target = 0 GW not exist","metadata":{}},{"cell_type":"markdown","source":"### Look at sample_submission.csv","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv('../input/g2net-gravitational-wave-detection/sample_submission.csv')\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:22.60574Z","iopub.execute_input":"2021-08-18T23:37:22.60617Z","iopub.status.idle":"2021-08-18T23:37:22.833965Z","shell.execute_reply.started":"2021-08-18T23:37:22.606127Z","shell.execute_reply":"2021-08-18T23:37:22.832673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 👉  the Goal of this competition is to show probability whether GW in the time serise data","metadata":{}},{"cell_type":"markdown","source":"### distribution train","metadata":{}},{"cell_type":"code","source":"sns.countplot(data=train, x=\"target\")","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:22.83627Z","iopub.execute_input":"2021-08-18T23:37:22.836722Z","iopub.status.idle":"2021-08-18T23:37:23.052789Z","shell.execute_reply.started":"2021-08-18T23:37:22.836672Z","shell.execute_reply":"2021-08-18T23:37:23.051612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### check the detail","metadata":{}},{"cell_type":"code","source":"# check the detail\nprint(train.shape)\nnum_target_exist = train[\"target\"] == 1\n# count target == 1\nprint(\"exist:{0}\".format(num_target_exist.sum()))\nnum_target_notexist = ~(train[\"target\"] == 1)\nprint(\"not exist:{0}\".format(num_target_notexist.sum()))","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:23.05455Z","iopub.execute_input":"2021-08-18T23:37:23.054994Z","iopub.status.idle":"2021-08-18T23:37:23.086067Z","shell.execute_reply.started":"2021-08-18T23:37:23.054952Z","shell.execute_reply":"2021-08-18T23:37:23.085009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def convert_image_id_2_path(image_id: str, is_train: bool = True) -> str:\n    folder = \"train\" if is_train else \"test\"\n    return \"../input/g2net-gravitational-wave-detection/{}/{}/{}/{}/{}.npy\".format(\n        folder, image_id[0],  image_id[1],  image_id[2],  image_id \n    )","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:23.087602Z","iopub.execute_input":"2021-08-18T23:37:23.088251Z","iopub.status.idle":"2021-08-18T23:37:23.094807Z","shell.execute_reply.started":"2021-08-18T23:37:23.088208Z","shell.execute_reply":"2021-08-18T23:37:23.093443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA","metadata":{}},{"cell_type":"markdown","source":"Each data sample (npy file) contains 3 time series (1 for each detector) and each spans 2 sec and is sampled at 2,048 Hz.  ","metadata":{}},{"cell_type":"code","source":"# なぜ事前に\"LIGO Hanford\", \"LIGO Livingston\", \"Virgo\"　の順番だとわかったのか？\ndef visualize_sample(_id, target, colors=(\"black\", \"red\", \"green\"), signal_names=(\"LIGO Hanford\", \"LIGO Livingston\", \"Virgo\")):\n    # visualize_train_data\n    path = convert_image_id_2_path(_id)\n    x = np.load(path)\n    # print(x) # show the source data\n    plt.figure(figsize=(16, 7))\n    for i in range(3):\n        plt.subplot(4, 1, i+1)\n        plt.plot(x[i], color=colors[i])\n        plt.legend([signal_names[i]], fontsize=12, loc=\"lower right\")\n        \n        plt.subplot(4, 1, 4)\n        plt.plot(x[i], color=colors[i])\n    \n    plt.subplot(4, 1, 4)\n    plt.legend(signal_names, fontsize=12, loc=\"lower right\")\n    \n    plt.suptitle(f\"id: {_id} target: {target}\", fontsize=16)\n    plt.show","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:23.096632Z","iopub.execute_input":"2021-08-18T23:37:23.097341Z","iopub.status.idle":"2021-08-18T23:37:23.108701Z","shell.execute_reply.started":"2021-08-18T23:37:23.097293Z","shell.execute_reply":"2021-08-18T23:37:23.107563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in random.sample(train.index.tolist(), 3):\n    _id = train.iloc[i]['id']\n    target = train.iloc[i]['target']\n    visualize_sample(_id, target)","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:23.112221Z","iopub.execute_input":"2021-08-18T23:37:23.112843Z","iopub.status.idle":"2021-08-18T23:37:26.093928Z","shell.execute_reply.started":"2021-08-18T23:37:23.112795Z","shell.execute_reply":"2021-08-18T23:37:26.092888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Signal transformation - Spectogram","metadata":{}},{"cell_type":"code","source":"import librosa\nimport librosa.display","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:26.099429Z","iopub.execute_input":"2021-08-18T23:37:26.101963Z","iopub.status.idle":"2021-08-18T23:37:27.694692Z","shell.execute_reply.started":"2021-08-18T23:37:26.101909Z","shell.execute_reply":"2021-08-18T23:37:27.693619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize_sample_spectogram(_id, target, signal_names=(\"LIGO Hanford\", \"LIGO Livingston\", \"Virgo\")):\n    x = np.load(convert_image_id_2_path(_id))\n    plt.figure(figsize=(16, 5))\n    for i in range(3):\n        X = librosa.stft(x[i] / x[i].max()) # why x[i] / x[i].max()?\n        Xdb = librosa.amplitude_to_db(abs(X))\n        plt.subplot(1, 3, i + 1)\n        librosa.display.specshow(Xdb, sr=2048, x_axis=\"time\", y_axis=\"hz\", vmin=-3, vmax=50)\n        plt.colorbar()\n        plt.title(signal_names[i], fontsize=14)\n        \n        \n    plt.suptitle(f\"id: {_id} target: {target}\", fontsize=16)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:27.698505Z","iopub.execute_input":"2021-08-18T23:37:27.698844Z","iopub.status.idle":"2021-08-18T23:37:27.708904Z","shell.execute_reply.started":"2021-08-18T23:37:27.698807Z","shell.execute_reply":"2021-08-18T23:37:27.707856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in random.sample(train.index.tolist(), 3):\n    _id = train.iloc[i]['id']\n    target = train.iloc[i]['target']\n    visualize_sample_spectogram(_id, target)","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:27.710527Z","iopub.execute_input":"2021-08-18T23:37:27.711174Z","iopub.status.idle":"2021-08-18T23:37:29.623767Z","shell.execute_reply.started":"2021-08-18T23:37:27.711126Z","shell.execute_reply":"2021-08-18T23:37:29.622807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Signal transformation - Q-transform","metadata":{}},{"cell_type":"markdown","source":"### nnAudio\n* nnAudio is an audio processing toolbox using PyTorch convolutional neural network as its backend.  \n* spectrograms can be generated from audio on-the-fly during neural network training and the Fourier kernels (e.g. or CQT kernels) can be trained","metadata":{}},{"cell_type":"code","source":"!pip install -q nnAudio -qq\nimport torch\nfrom nnAudio.Spectrogram import CQT1992v2 # 定数Q変換を計算するための効率的なアルゴリズム","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:29.625319Z","iopub.execute_input":"2021-08-18T23:37:29.625707Z","iopub.status.idle":"2021-08-18T23:37:40.281351Z","shell.execute_reply.started":"2021-08-18T23:37:29.625666Z","shell.execute_reply":"2021-08-18T23:37:40.28022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Q_TRANSFORM = CQT1992v2(sr=2048, fmin=20, fmax=1024, hop_length=32)\n\ndef visualize_sample_qtransform(_id, target, signal_names=(\"LIGO Hanford\", \"LIGO Livingston\", \"Virgo\"), sr=2048):\n    x = np.load(convert_image_id_2_path(_id))\n    plt.figure(figsize=(16, 5))\n    for i in range(3):\n        waves = x[i] / np.max(x[i])\n        waves = torch.from_numpy(waves).float()\n        image = Q_TRANSFORM(waves)\n        \n        plt.subplot(1, 3, i + 1)\n        plt.imshow(image.squeeze())\n        plt.title(signal_names[i], fontsize=14)\n        \n    plt.suptitle(f\"id: {_id} target: {target}\", fontsize=16)\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:40.282984Z","iopub.execute_input":"2021-08-18T23:37:40.283455Z","iopub.status.idle":"2021-08-18T23:37:40.535014Z","shell.execute_reply.started":"2021-08-18T23:37:40.283401Z","shell.execute_reply":"2021-08-18T23:37:40.533836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in random.sample(train.index.tolist(), 3):\n    _id = train.iloc[i]['id']\n    target = train.iloc[i]['target']\n    \n    visualize_sample(_id, target)\n    visualize_sample_qtransform(_id, target)","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:40.536694Z","iopub.execute_input":"2021-08-18T23:37:40.537149Z","iopub.status.idle":"2021-08-18T23:37:43.712221Z","shell.execute_reply.started":"2021-08-18T23:37:40.537103Z","shell.execute_reply":"2021-08-18T23:37:43.711009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"markdown","source":"#### EfficientNet  \nArrange 3 balance in \"depth\" / \"width\" / \"resolution\" properly  ※ not change the Arichitecture  \n(モデルの「深さ」と「広さ」と「解像度(=入力画像の大きさ)」の3つをバランスよく調整する ※レイヤーのアーキティクチャ自体は変えない)","metadata":{}},{"cell_type":"code","source":"!pip install efficientnet_pytorch -qq","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:43.714004Z","iopub.execute_input":"2021-08-18T23:37:43.714466Z","iopub.status.idle":"2021-08-18T23:37:53.673452Z","shell.execute_reply.started":"2021-08-18T23:37:43.714419Z","shell.execute_reply":"2021-08-18T23:37:53.672122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nfrom torch import nn\nfrom torch.utils import data as torch_data\nfrom sklearn import model_selection as sk_model_selection\nfrom torch.nn import functional as torch_functional\nimport efficientnet_pytorch\nfrom sklearn.model_selection import StratifiedKFold # 交差検証 分布に大きな不均衡がある場合に用いるKFold,  分布の比率を維持したままデータを訓練用とテスト用に分割する．","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:37:53.675396Z","iopub.execute_input":"2021-08-18T23:37:53.675911Z","iopub.status.idle":"2021-08-18T23:37:53.693272Z","shell.execute_reply.started":"2021-08-18T23:37:53.675859Z","shell.execute_reply":"2021-08-18T23:37:53.691976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### DataLoader\ndatasetsからバッチごとに取り出す  \ndatsets = [データセット全て]  \nDataloader = [[batch_1], [batch_2], ... [batch_n]]  \nthat is..   \nlen(datasets)=\"すべてのデータの数\"  \nlen(Dataloader)=\"イテレーションの数\"  ","metadata":{}},{"cell_type":"code","source":"class DataRetriever(torch_data.Dataset):\n    def __init__(self, paths, targets):\n        self.paths = paths\n        self.targets = targets\n        \n        self.q_transform = CQT1992v2(sr=2048, fmin=20, fmax=1024, hop_length=32)\n        \n    def __len__(self):\n        return len(self.paths)\n    \n    def __get_qtransform(self, x):\n        image = []\n        for i in range(3):\n            waves = x[i] / np.max(x[i])\n            waves = torch.from_numpy(waves).float()\n            channel = self.q_transform(waves).squeeze().numpy()\n            image.append(channel)\n        \n        return torch.tensor(image).float()\n    \n    def __getitem__(self, index):\n        # データのindexを指定すると、前処理(x: Q-Transformで画像化, y: 画像をTensor化)をして返す\n        file_path = convert_image_id_2_path(self.paths[index])\n        x = np.load(file_path)\n        image = self.__get_qtransform(x)\n        y = torch.tensor(self.targets[index], dtype=torch.float)\n        \n        return {\"X\": image, \"y\": y}","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:42:30.773905Z","iopub.execute_input":"2021-08-18T23:42:30.774297Z","iopub.status.idle":"2021-08-18T23:42:30.784123Z","shell.execute_reply.started":"2021-08-18T23:42:30.774266Z","shell.execute_reply":"2021-08-18T23:42:30.782735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Model Efficientnet","metadata":{}},{"cell_type":"code","source":"class Model(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.net = efficientnet_pytorch.EfficientNet.from_pretrained(\"efficientnet-b7\")\n        n_features = self.net._fc.in_features\n        self.net._fc = nn.Linear(in_features=n_features, out_features=1, bias=True)\n    \n    def forward(self, x):\n        out = self.net(x)\n        return out","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:42:31.114774Z","iopub.execute_input":"2021-08-18T23:42:31.115168Z","iopub.status.idle":"2021-08-18T23:42:31.122789Z","shell.execute_reply.started":"2021-08-18T23:42:31.115133Z","shell.execute_reply":"2021-08-18T23:42:31.121758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class LossMeter:\n    def __init__(self):\n        self.avg = 0\n        self.n = 0\n        \n    def update(self, val):\n        self.n += 1\n        # ↓Whats?  (incremental update)\n        \n        self.avg = val / self.n + (self.n - 1) / self.n * self.avg\n        \n        \nclass AccMeter:\n    def __init__(self):\n        self.avg = 0\n        self.n = 0\n        \n    def update(self, y_true, y_pred):\n        y_true = y_true.cpu().numpy().astype(int)\n        y_pred = y_pred.cpu().numpy() >= 0\n        last_n = self.n\n        self.n += len(y_true)\n        true_count = np.sum(y_true == y_pred)\n        # incremental update (unfamiliar)\n        self.avg = true_count / self.n + last_n / self.n * self.avg","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:42:31.275132Z","iopub.execute_input":"2021-08-18T23:42:31.275504Z","iopub.status.idle":"2021-08-18T23:42:31.284877Z","shell.execute_reply.started":"2021-08-18T23:42:31.275472Z","shell.execute_reply":"2021-08-18T23:42:31.283591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Trainer:\n    # do train\n    def __init__(self, model, device, optimizer, criterion, loss_meter, score_meter):\n        self.model = model\n        self.device = device\n        self.optimizer = optimizer\n        self.criterion = criterion\n        self.loss_meter = loss_meter\n        self.score_meter = score_meter\n        \n        self.best_valid_score = -np.inf\n        self.n_patience = 0 # early_stoppingのために、止める指標 (今回は100epochでstop)\n        \n        self.messages = {\n            \"epoch\": \"[Epoch {}: {}] loss: {:.5f}, score: {:.5f}, time: {} s\",\n            \"checkpoint\": \"The score improved from {:.5f} to {:.5f}. Save model to '{}'\",\n            \"patience\": \"\\nValid score didn't improve last {} epochs.\"\n        }\n        \n    def fit(self, epochs, train_loader, valid_loader, save_path, patience):\n        for n_epoch in range(1, epochs + 1):\n            self.info_message(\"EPOCH: {}\", n_epoch)\n            \n            train_loss, train_score, train_time = self.train_epoch(train_loader)\n            valid_loss, valid_score, valid_time = self.valid_epoch(valid_loader)\n            \n            self.info_message(\n                self.messages[\"epoch\"], \"Train\", n_epoch, train_loss, train_score, train_time\n            )\n            self.info_message(\n                self.messages[\"epoch\"], \"Valid\", n_epoch, valid_loss, valid_score, valid_time\n            )\n            \n            if True:\n                # if self.best_valid_score < valid_score:\n                self.info_message(\n                    self.messages[\"checkpoint\"], self.best_valid_score, valid_score, save_path\n                )\n                self.best_valid_score = valid_score\n                self.save_model(n_epoch, save_path)\n                self.n_patience = 0\n            else:\n                self.n_patience += 1\n            \n            if self.n_patience >= patience:\n                self.info_message(self.messages[\"patience\"], patience)\n                break\n            \n                \n    def train_epoch(self, train_loader):\n        self.model.train()\n        t = time.time()\n        train_loss = self.loss_meter()\n        train_score = self.score_meter()\n        \n        for step, batch in enumerate(train_loader, 1):\n            X = batch[\"X\"].to(self.device)\n            targets = batch[\"y\"].to(self.device)\n            self.optimizer.zero_grad()\n            outputs = self.model(X).squeeze(1)\n            \n            loss = self.criterion(outputs, targets)\n            loss.backward()\n            \n            train_loss.update(loss.detach().item())\n            train_score.update(targets, outputs.detach())\n            \n            self.optimizer.step() # update weight\n            \n            _loss, _score = train_loss.avg, train_score.avg\n            message = 'Train Step {}/{}, train_loss: {:.5f}, train_score: {:.5f}'\n            self.info_message(message, step, len(train_loader), _loss, _score, end=\"\\r\")\n                   \n        return train_loss.avg, train_score.avg, int(time.time() - t)\n            \n    def valid_epoch(self, valid_loader):\n        self.model.eval()\n        t = time.time()\n        valid_loss = self.loss_meter()\n        valid_score = self.score_meter()\n        \n        for step, batch in enumerate(valid_loader, 1):\n            with torch.no_grad():\n                X = batch[\"X\"].to(self.device)\n                targets = batch[\"y\"].to(self.device)\n\n                outputs = self.model(X).squeeze(1)\n                loss = self.criterion(outputs, targets)\n\n                valid_loss.update(loss.detach().item())\n                valid_score.update(targets, outputs)\n                \n            _loss, _score = valid_loss.avg, valid_score.avg\n            message = 'Valid Step {}/{}, valid_loss: {:.5f}, valid_score: {:.5f}' \n            self.info_message(message, step, len(valid_loader), _loss, _score, end=\"\\r\")\n            \n        return valid_loss.avg, valid_score.avg, int(time.time() - t)\n        \n    def save_model(self, n_epoch, save_path):\n        torch.save(\n        {\n            \"model_state_dict\" : self.model.state_dict(),\n            \"optimizer_state_dict\" : self.optimizer.state_dict(),\n            \"best_valid_score\" : self.best_valid_score,\n            \"n_epoch\" : n_epoch,\n        },\n        save_path\n        )\n    @staticmethod        \n    def info_message(message, *args, end=\"\\n\"):\n        print(message.format(*args), end=end)","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:42:31.445306Z","iopub.execute_input":"2021-08-18T23:42:31.445669Z","iopub.status.idle":"2021-08-18T23:42:31.469558Z","shell.execute_reply.started":"2021-08-18T23:42:31.445637Z","shell.execute_reply":"2021-08-18T23:42:31.468041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"code","source":"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nskf = StratifiedKFold(n_splits=2, random_state=42, shuffle=True)\nfor fold, (train_index, valid_index) in enumerate(skf.split(train, train[\"target\"])):\n    # prepare Dataset\n    train_X = train.iloc[train_index]\n    valid_X = train.iloc[valid_index][:20000]  # Reduce calculation time\n    print(train_X.shape, valid_X.shape)\n    \n    train_data_retriever = DataRetriever(\n        train_X[\"id\"].values,\n        train_X[\"target\"].values\n    )\n    valid_data_retriever = DataRetriever(\n        valid_X[\"id\"].values, \n        valid_X[\"target\"].values\n    )\n    \n    train_loader = torch_data.DataLoader(train_data_retriever, batch_size=32, shuffle=True, num_workers=8)\n    valid_loader = torch_data.DataLoader(valid_data_retriever, batch_size=32, shuffle=False, num_workers=8)\n    \n    model = Model()\n    model.to(device)\n    \n    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)\n    criterion = torch_functional.binary_cross_entropy_with_logits\n    \n    trainer = Trainer(model, device, optimizer, criterion, LossMeter, AccMeter)\n    \n    # train 1epoch\n    history = trainer.fit(1, train_loader, valid_loader, f\"best-model-{fold}.pth\" ,100)","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:42:31.749044Z","iopub.execute_input":"2021-08-18T23:42:31.749469Z","iopub.status.idle":"2021-08-18T23:42:37.332225Z","shell.execute_reply.started":"2021-08-18T23:42:31.749435Z","shell.execute_reply":"2021-08-18T23:42:37.328861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### choice best model","metadata":{}},{"cell_type":"code","source":"models = []\nfor i in range(2):\n    model = Model()\n    model.to(device)\n    \n    checkpoint = torch.load(f\"best-model-{i}.pth\")\n    model.load_state_dict(checkpoint[\"model_state_dict\"])\n    model.eval()\n    \n    models.append(model)","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:40:50.608651Z","iopub.execute_input":"2021-08-18T23:40:50.609118Z","iopub.status.idle":"2021-08-18T23:40:55.379602Z","shell.execute_reply.started":"2021-08-18T23:40:50.60905Z","shell.execute_reply":"2021-08-18T23:40:55.37854Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Prepare testdata","metadata":{}},{"cell_type":"code","source":"class DataRetriever(torch_data.Dataset):\n    def __init__(self, paths):\n        self.paths = paths\n\n        self.q_transform = CQT1992v2(\n            sr=2048, fmin=20, fmax=1024, hop_length=32\n        )\n    \n    def __len__(self):\n        return len(self.paths)\n    \n    def __get_qtransform(self, x):\n        image = []\n        for i in range(3):\n            waves = x[i] / np.max(x[i])\n            waves = torch.from_numpy(waves).float() # numpy to tensor\n            channel = self.q_transform(waves).squeeze().numpy() # .numpy() :This implicitly means that the converted tensor will be now processed on the CPU.\n            image.append(channel)\n        \n        return torch.tensor(image).float()\n    \n    def __getitem__(self, index):\n        file_path = convert_image_id_2_path(self.paths[index], is_train=False)\n        x = np.load(file_path)\n        image = self.__get_qtransform(x)\n        \n        return {\"X\": image, \"id\": self.paths[index]}","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:40:58.599858Z","iopub.execute_input":"2021-08-18T23:40:58.600246Z","iopub.status.idle":"2021-08-18T23:40:58.613071Z","shell.execute_reply.started":"2021-08-18T23:40:58.60021Z","shell.execute_reply":"2021-08-18T23:40:58.611884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_data_retriever = DataRetriever(\n    test[\"id\"].values,\n)\n\ntest_loader = torch_data.DataLoader(\n    test_data_retriever,\n    batch_size=32,\n    shuffle=False,\n    num_workers = 8\n)","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:40:59.007904Z","iopub.execute_input":"2021-08-18T23:40:59.008319Z","iopub.status.idle":"2021-08-18T23:40:59.04271Z","shell.execute_reply.started":"2021-08-18T23:40:59.008283Z","shell.execute_reply":"2021-08-18T23:40:59.041194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inference","metadata":{}},{"cell_type":"code","source":"y_pred = []\nids = []\n\nfor e, batch in enumerate(test_loader):\n    print(f\"{e}/{len(test_loader)}\", end=\"\\r\")\n    with torch.no_grad():\n        tmp_pred = np.zeros((batch[\"X\"].shape[0], ))\n        for model in models:\n            tmp_res = torch.sigmoid(model(batch[\"X\"].to(device))).cpu().numpy().squeeze()\n            tmp_pred += tmp_res / 2\n        y_pred.extend(tmp_pred)\n        ids.extend(batch[\"id\"])","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:41:06.747534Z","iopub.execute_input":"2021-08-18T23:41:06.747911Z","iopub.status.idle":"2021-08-18T23:41:29.389583Z","shell.execute_reply.started":"2021-08-18T23:41:06.747878Z","shell.execute_reply":"2021-08-18T23:41:29.386231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"submission = pd.DataFrame({\"id\": ids, \"target\": y_pred})\nsubmission.to_csv(\"model_submission.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:38:09.376486Z","iopub.status.idle":"2021-08-18T23:38:09.377157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission","metadata":{"execution":{"iopub.status.busy":"2021-08-18T23:38:09.378561Z","iopub.status.idle":"2021-08-18T23:38:09.379249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}