{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

G2Net Gravitational Waves

\n\n![](https://www.g2net.eu/wp-content/uploads/2019/07/2ndconference_V2-1170x600.jpg)\n\n\n\n

\nThe breakthrough discovery of gravitational waves on September 14, 2015 was made possible through synergy of techniques drawing from expertise in physics, mathematics, information science and computing. In this competition you are provided with a training set of time series data containing simulated gravitational wave measurements from a network of 3 gravitational wave interferometers (LIGO Hanford, LIGO Livingston, and Virgo). Each time series contains either detector noise or detector noise plus a simulated gravitational wave signal.

\n\n
\"Weights

","metadata":{}},{"cell_type":"markdown","source":"

What are we discussing today?

\n

Constant Q Transform
\n DeepSpeed Fused Adam optimizer
\n Pytorch Lightning
\n Stratified K-Fold with PL ⚡️
\n Weights and Biases for Experiment Tracking including W&B Tables","metadata":{}},{"cell_type":"markdown","source":"

Upvote the kernel if you find it insightful!

","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.append('../input/timm-pytorch-image-models/pytorch-image-models-master')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"execution":{"iopub.status.busy":"2021-07-18T07:15:29.231455Z","iopub.execute_input":"2021-07-18T07:15:29.231849Z","iopub.status.idle":"2021-07-18T07:15:29.242288Z","shell.execute_reply.started":"2021-07-18T07:15:29.231771Z","shell.execute_reply":"2021-07-18T07:15:29.241340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\"Weights

\n

Wandb is a developer tool for companies turn deep learning research projects into deployed software by helping teams track their models, visualize model performance and easily automate training and improving models.\nWe will use their tools to log hyperparameters and output metrics from your runs, then visualize and compare results and quickly share findings with your colleagues.

We'll be using this to train our K Fold Cross Validation and gain better insights about our training.

\n\n![img](https://i.imgur.com/BGgfZj3.png)","metadata":{}},{"cell_type":"code","source":"!pip install --upgrade -q wandb\n!pip install -q pytorch-lightning\n!pip install -q nnAudio\n!pip install -q deepspeed\n!pip install -q timm","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-18T07:15:29.243997Z","iopub.execute_input":"2021-07-18T07:15:29.244548Z","iopub.status.idle":"2021-07-18T07:16:17.488720Z","shell.execute_reply.started":"2021-07-18T07:15:29.244509Z","shell.execute_reply":"2021-07-18T07:16:17.487663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Import Libraries

","metadata":{}},{"cell_type":"code","source":"# Hide Warning\nimport warnings\nwarnings.filterwarnings('ignore', category=DeprecationWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\nwarnings.filterwarnings('ignore', category=UserWarning)\n\n# Python Libraries\nfrom tqdm import tqdm\nfrom collections import defaultdict\nimport pandas as pd\nimport numpy as np\nimport os\nimport random\nimport glob\nimport math\n\n\n# Visualizations\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2\nimport plotly.express as px\n%matplotlib inline\nsns.set(style=\"whitegrid\")\n\n# Utilities and Metrics\nfrom sklearn.model_selection import StratifiedKFold, train_test_split, KFold\nfrom sklearn.metrics import roc_auc_score\n\n# Pytorch \nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.optim import Adam, SGD\nfrom torch.utils.data import Dataset, DataLoader, WeightedRandomSampler\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\nfrom torch.optim.optimizer import Optimizer, required \nfrom pytorch_lightning.plugins import DeepSpeedPlugin\nfrom deepspeed.ops.adam import FusedAdam\n\n\n# Pytorch Lightning\nimport pytorch_lightning as pl\nfrom pytorch_lightning import seed_everything\nfrom pytorch_lightning import Callback\nfrom pytorch_lightning.callbacks import ModelCheckpoint, EarlyStopping\nfrom pytorch_lightning.loggers import WandbLogger\n\n# Pytorch Image Models\nimport timm\n\n# Image Augmentation Library\nimport albumentations\nfrom albumentations.pytorch import ToTensorV2\nfrom albumentations.core.transforms_interface import DualTransform\nfrom albumentations.augmentations import functional as AF\n\nfrom nnAudio.Spectrogram import CQT2010v2, MFCC, CQT1992v2, MelSpectrogram\n\n\n# Weights and Biases Tool\nimport wandb\nwandb.login()","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:16:17.490921Z","iopub.execute_input":"2021-07-18T07:16:17.491225Z","iopub.status.idle":"2021-07-18T07:16:50.773204Z","shell.execute_reply.started":"2021-07-18T07:16:17.491195Z","shell.execute_reply":"2021-07-18T07:16:50.772394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Define Configurations/Parameters

","metadata":{}},{"cell_type":"code","source":"params = {\n 'seed': 42,\n 'model': 'efficientnet_b1',\n 'size' : 224,\n 'inp_channels': 1,\n 'lr': 1e-3,\n 'weight_decay': 1e-6,\n 'batch_size': 64,\n 'num_workers' : 8,\n 'epochs': 5,\n 'out_features': 1,\n 'name': 'CosineAnnealingLR',\n 'T_max': 10,\n 'min_lr': 1e-6,\n 'nfolds': 5,\n 'precision': 16,\n 'competetion': 'G2Net',\n '_wandb_kernel':'tang'\n}","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:16:50.774808Z","iopub.execute_input":"2021-07-18T07:16:50.775178Z","iopub.status.idle":"2021-07-18T07:16:50.780373Z","shell.execute_reply.started":"2021-07-18T07:16:50.775142Z","shell.execute_reply":"2021-07-18T07:16:50.779439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Define Seed for Reproducibility

","metadata":{}},{"cell_type":"code","source":"def seed_everything(seed=42):\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = True\n \nseed_everything(params['seed'])","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:16:50.781976Z","iopub.execute_input":"2021-07-18T07:16:50.782544Z","iopub.status.idle":"2021-07-18T07:16:50.798458Z","shell.execute_reply.started":"2021-07-18T07:16:50.782507Z","shell.execute_reply":"2021-07-18T07:16:50.797608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Load Train and Test

","metadata":{}},{"cell_type":"code","source":"train_dir = ('../input/g2net-gravitational-wave-detection/train')\ntest_dir = ('../input/g2net-gravitational-wave-detection/test')\ntrain_df = pd.read_csv('../input/g2net-gravitational-wave-detection/training_labels.csv')\ntest_df = pd.read_csv('../input/g2net-gravitational-wave-detection/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:16:50.800186Z","iopub.execute_input":"2021-07-18T07:16:50.800456Z","iopub.status.idle":"2021-07-18T07:16:51.376800Z","shell.execute_reply.started":"2021-07-18T07:16:50.800429Z","shell.execute_reply":"2021-07-18T07:16:51.375924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def return_filpath(name, folder=train_dir):\n path = os.path.join(folder, f'{name[0]}/{name[1]}/{name[2]}/{name}.npy')\n return path","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:16:51.380432Z","iopub.execute_input":"2021-07-18T07:16:51.380707Z","iopub.status.idle":"2021-07-18T07:16:51.384445Z","shell.execute_reply.started":"2021-07-18T07:16:51.380680Z","shell.execute_reply":"2021-07-18T07:16:51.383666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['image_path'] = train_df['id'].apply(lambda x: return_filpath(x))\ntest_df['image_path'] = test_df['id'].apply(lambda x: return_filpath(x, folder=test_dir))\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:16:51.386007Z","iopub.execute_input":"2021-07-18T07:16:51.386477Z","iopub.status.idle":"2021-07-18T07:16:53.486605Z","shell.execute_reply.started":"2021-07-18T07:16:51.386438Z","shell.execute_reply":"2021-07-18T07:16:53.485808Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = train_df.sample(n=50000, random_state=params['seed']).reset_index(drop=True)","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:16:53.488080Z","iopub.execute_input":"2021-07-18T07:16:53.488442Z","iopub.status.idle":"2021-07-18T07:16:53.585853Z","shell.execute_reply.started":"2021-07-18T07:16:53.488402Z","shell.execute_reply":"2021-07-18T07:16:53.584883Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\"Weights

\n

W&B Tables accelerate the ML development lifecycle by giving users the ability to rapidly extract meaningful insights from data. The W&B Table Visualizer provides an interactive interface to perform powerful analytics functions like grouping, joining, and creating custom fields while simultaneously supporting rich media annotations such as bounding boxes and segmentation masks.

W&B Tables is designed generically to work well for a wide range of use cases - from analyzing intermediate data transformations to reviewing model predictions - while being directly integrated directly into the WB UI dashboard, allowing users to learn, adapt, and improve their models effectively and efficiently.

","metadata":{}},{"cell_type":"markdown","source":"

Constant Q Transform

\n\n

In mathematics and signal processing, the constant-Q transform, simply known as CQT transforms a data series to the frequency domain. It is related to the Fourier transform[1] and very closely related to the complex Morlet wavelet transform.

An efficient method of transforming a discrete Fourier transform (DFT) into a constant Q transform, where Q is the ratio of center frequency to bandwidth, has been devised. This method involves the calculation of kernels that are then applied to each subsequent DFT. Only a few multiples are involved in the calculation of each component of the constant Q transform, so this transformation adds a small amount to the computation. In effect, this method makes it possible to take full advantage of the computational efficiency of the fast Fourier transform (FFT).","metadata":{}},{"cell_type":"code","source":"run = wandb.init(project='G2Net-PL-Baseline', \n config=params,\n group='Effnet-CQT', \n job_type='Visualize')\n\nsample_data_tables = wandb.Table(columns=[\"ID\",\"CQT\", \"target\"])\n\n\ndef apply_qtransform(waves, transform=CQT1992v2(sr=2048, fmin=20, fmax=1024, hop_length=64)):\n waves = np.hstack(waves)\n waves = waves / np.max(waves)\n waves = torch.from_numpy(waves).float()\n image = transform(waves)\n return image\n\n\nfor i in range(10):\n waves = np.load(train_df.loc[i, 'image_path'])\n image = apply_qtransform(waves)\n target = train_df.loc[i, 'target']\n sample_data_tables.add_data(train_df.loc[i, 'id'],\n wandb.Image(image),\n target)\n \n \n\nwandb.log({'wandb_viz': sample_data_tables}) \nwandb.finish()","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-18T07:16:53.587101Z","iopub.execute_input":"2021-07-18T07:16:53.587668Z","iopub.status.idle":"2021-07-18T07:17:07.481792Z","shell.execute_reply.started":"2021-07-18T07:16:53.587602Z","shell.execute_reply":"2021-07-18T07:17:07.480779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

W&B Table for CQT Visualization


\n\n
\"metrics\"
","metadata":{}},{"cell_type":"code","source":"dist = train_df.target.map({0:'Target 0', 1:'Target 1'})\ndist = dist.value_counts()\nfig = px.pie(dist,\n values='target',\n names=dist.index,\n hole=.4,title=\"Target Distribution\")\nfig.update_traces(textinfo='percent+label', pull=0.05)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:17:07.483207Z","iopub.execute_input":"2021-07-18T07:17:07.483588Z","iopub.status.idle":"2021-07-18T07:17:08.424477Z","shell.execute_reply.started":"2021-07-18T07:17:07.483549Z","shell.execute_reply":"2021-07-18T07:17:08.423708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Custom Dataset

","metadata":{}},{"cell_type":"code","source":"def get_train_transforms():\n return albumentations.Compose(\n [\n\n ToTensorV2(p=1.0)\n ]\n )\n\ndef get_valid_transforms():\n return albumentations.Compose(\n [\n ToTensorV2(p=1.0)\n ]\n )\n\ndef get_test_transforms():\n return albumentations.Compose(\n [\n ToTensorV2(p=1.0)\n ]\n )","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:17:08.425941Z","iopub.execute_input":"2021-07-18T07:17:08.426315Z","iopub.status.idle":"2021-07-18T07:17:08.433672Z","shell.execute_reply.started":"2021-07-18T07:17:08.426279Z","shell.execute_reply":"2021-07-18T07:17:08.432906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class SETIDataset(Dataset):\n def __init__(self, images_filepaths, targets, transform=None):\n self.images_filepaths = images_filepaths\n self.targets = targets\n self.transform = transform\n self.wave_transform = CQT1992v2(sr=2048, fmin=20, fmax=1024, hop_length=64, verbose=True)\n\n def __len__(self):\n return len(self.images_filepaths)\n \n def apply_cqt(self, image, transform):\n image = np.hstack(image)\n image = image / np.max(image)\n image = torch.from_numpy(image).float()\n image = transform(image)\n return image\n\n def __getitem__(self, idx):\n image_filepath = self.images_filepaths[idx]\n image = np.load(image_filepath)\n image = self.apply_cqt(image, self.wave_transform)\n\n if self.transform is not None:\n image = image.squeeze().numpy()\n image = self.transform(image=image)[\"image\"]\n else:\n image = image[np.newaxis,:,:]\n image = torch.from_numpy(image).float()\n \n label = torch.tensor(self.targets[idx]).float()\n return image, label","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:17:08.435068Z","iopub.execute_input":"2021-07-18T07:17:08.435430Z","iopub.status.idle":"2021-07-18T07:17:08.445972Z","shell.execute_reply.started":"2021-07-18T07:17:08.435395Z","shell.execute_reply":"2021-07-18T07:17:08.445048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Custom Callback for Viewing Predictions in W&B

\n","metadata":{}},{"cell_type":"code","source":"class ImagePredictionLogger(Callback):\n def __init__(self, val_samples = None, num_samples=32):\n super().__init__()\n self.num_samples = num_samples\n self.val_imgs, self.val_labels = val_samples\n \n \n def on_validation_epoch_end(self, trainer, pl_module):\n val_imgs = self.val_imgs.to(device=pl_module.device)\n val_labels = self.val_labels.to(device=pl_module.device)\n logits = pl_module(val_imgs)\n preds = torch.argmax(logits, -1)\n trainer.logger.experiment.log({\n \"Predictions\":[wandb.Image(x, caption=f\"Pred:{pred}, Target:{y}\") \n for x, pred, y in zip(val_imgs[:self.num_samples], \n preds[:self.num_samples], \n val_labels[:self.num_samples])]\n }, commit=False)","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:17:08.447358Z","iopub.execute_input":"2021-07-18T07:17:08.447797Z","iopub.status.idle":"2021-07-18T07:17:08.458768Z","shell.execute_reply.started":"2021-07-18T07:17:08.447748Z","shell.execute_reply":"2021-07-18T07:17:08.457897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Pytorch Lightning DataModule

\n\n#
\"Weights
\n\n\n

A datamodule is a shareable, reusable class that encapsulates all the steps needed to process data. A datamodule encapsulates the five steps involved in data processing in PyTorch:
1. Download / tokenize / process.
2. Clean and (maybe) save to disk.
3. Load inside Dataset.
4. Apply transforms (rotate, tokenize, etc…).
5. Wrap inside a DataLoader.

\n📍 Why do I need a DataModule?
In normal PyTorch code, the data cleaning/preparation is usually scattered across many files. This makes sharing and reusing the exact splits and transforms across projects impossible.
Datamodules are for you if you ever asked the questions:
1. what splits did you use?
2. what transforms did you use?
3. what normalization did you use?
4. how did you prepare/tokenize the data?

To summarize: A DataModule is simply a collection of a train_dataloader, val_dataloader(s), test_dataloader(s) along with the matching transforms and data processing/downloads steps required.","metadata":{}},{"cell_type":"code","source":"class DataModule(pl.LightningDataModule):\n\n def __init__(self, train_data, valid_data, test_data):\n super().__init__()\n self.train_data = train_data\n self.valid_data = valid_data\n self.test_data = test_data\n \n def setup(self, stage=None):\n self.train_dataset = SETIDataset(\n images_filepaths=self.train_data['image_path'].values,\n targets=self.train_data['target'].values,\n transform = get_train_transforms() \n )\n \n self.val_dataset = SETIDataset(\n images_filepaths=self.valid_data['image_path'].values,\n targets=self.valid_data['target'].values,\n transform = get_valid_transforms()\n )\n\n self.test_dataset = SETIDataset(\n images_filepaths = self.test_data['image_path'].values,\n targets = self.test_data['target'].values,\n transform = get_test_transforms()\n )\n\n def train_dataloader(self):\n return DataLoader(\n self.train_dataset,\n batch_size=params['batch_size'],\n shuffle=True,\n num_workers=params['num_workers'],\n pin_memory=True\n )\n\n def val_dataloader(self):\n return DataLoader(\n self.val_dataset,\n batch_size=params['batch_size'],\n shuffle=False,\n num_workers=params['num_workers'],\n pin_memory=True\n )\n\n def test_dataloader(self):\n return DataLoader(\n self.test_dataset, batch_size=params['batch_size'],\n shuffle=False, num_workers=params['num_workers'],\n pin_memory=True\n )","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:17:08.460352Z","iopub.execute_input":"2021-07-18T07:17:08.460591Z","iopub.status.idle":"2021-07-18T07:17:08.472275Z","shell.execute_reply.started":"2021-07-18T07:17:08.460569Z","shell.execute_reply":"2021-07-18T07:17:08.471281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

DeepSpeed Fused Adam Optimizer

\n\n

DeepSpeed offers high-performance implementations of Adam optimizer on CPU; FusedAdam, FusedLamb, OnebitAdam, OnebitLamb optimizers on GPU.

Here we implement Fused Adam which implements Adam algorithm but this version implements 2 fusions.

1. Fusion of the Adam update’s elementwise operations.
2.A multi-tensor apply launch that batches the elementwise updates applied to all the model’s parameters into one or a few kernel launches.

The Adam optimizer in Pytorch (like all Pytorch optimizers) carries out optimizer.step() by looping over parameters, and launching a series of kernels for each parameter. This can require hundreds of small launches that are mostly bound by CPU-side Python looping and kernel launch overhead, resulting in poor device utilization. Currently, the FusedAdam implementation in Apex flattens the parameters for the optimization step, then carries out the optimization step itself via a fused kernel that combines all the Adam operations. In this way, the loop over parameters as well as the internal series of Adam operations for each parameter are fused such that optimizer.step() requires only a few kernel launches.","metadata":{}},{"cell_type":"markdown","source":"

Pytorch Lightning Module

\n
\"Weights

\n\n

A LightningModule organizes your PyTorch code into 5 sections
1. Computations (init).
2. Train loop (training_step)
3. Validation loop (validation_step)
4. Test loop (test_step)
5. Optimizers (configure_optimizers)

\nA LightningModule is a torch.nn.Module but with added functionality. ","metadata":{}},{"cell_type":"code","source":"class Trainer(pl.LightningModule):\n\n def __init__(self, model_name=params['model'],out_features=params['out_features'],\n inp_channels=params['inp_channels'],pretrained=True):\n super().__init__()\n \n self.model = timm.create_model(model_name, pretrained=pretrained,\n in_chans=inp_channels)\n \n if model_name == 'resnet18d':\n n_features = self.model.fc.in_features\n self.model.fc = nn.Linear(n_features, out_features, bias=True)\n \n if model_name == 'nfnet_f1':\n n_features = self.model.head.fc.in_features\n self.model.head.fc = nn.Linear(n_features, out_features, bias=True)\n \n elif model_name == 'efficientnet_b1':\n n_features = self.model.classifier.in_features\n self.model.classifier = nn.Linear(n_features, out_features, bias=True) \n \n self.criterion = nn.BCEWithLogitsLoss()\n \n def forward(self, x):\n output = self.model(x)\n return output\n\n\n def training_step(self, batch, batch_idx):\n\n x, y = batch\n output = self.model(x)\n labels = y.unsqueeze(1)\n loss = self.criterion(output, labels)\n \n try:\n auc=roc_auc_score(labels.detach().cpu(), output.sigmoid().detach().cpu()) \n\n self.log(\"auc\", auc, on_step= True, prog_bar=True, logger=True)\n self.log(\"Train Loss\", loss, on_step= True,prog_bar=True, logger=True)\n \n except:\n pass\n\n return {\"loss\": loss, \"predictions\": output, \"labels\": labels}\n\n def training_epoch_end(self, outputs):\n\n preds = []\n labels = []\n \n for output in outputs:\n \n preds += output['predictions']\n labels += output['labels']\n\n labels = torch.stack(labels)\n preds = torch.stack(preds)\n\n train_auc=roc_auc_score(labels.detach().cpu(), preds.sigmoid().detach().cpu())\n self.log(\"mean_train_auc\", train_auc, prog_bar=True, logger=True)\n\n def validation_step(self, batch, batch_idx):\n x, y = batch\n output = self.model(x)\n labels = y.unsqueeze(1)\n loss = self.criterion(output, labels)\n \n self.log('val_loss', loss, on_step= True, prog_bar=True, logger=True)\n return {\"predictions\": output, \"labels\": labels}\n \n\n def validation_epoch_end(self, outputs):\n\n preds = []\n labels = []\n \n for output in outputs:\n preds += output['predictions']\n labels += output['labels']\n\n labels = torch.stack(labels)\n preds = torch.stack(preds)\n\n val_auc=roc_auc_score(labels.detach().cpu(), preds.sigmoid().detach().cpu())\n self.log(\"val_auc\", val_auc, prog_bar=True, logger=True)\n \n\n def test_step(self, batch, batch_idx):\n x = batch \n output = self(x).sigmoid()\n return output \n\n def configure_optimizers(self):\n\n param_optimizer = list(self.model.named_parameters())\n no_decay = [\"bias\", \"LayerNorm.bias\", \"LayerNorm.weight\"]\n optimizer_parameters = [\n {\n \"params\": [\n p for n, p in param_optimizer if not any(nd in n for nd in no_decay)\n ],\n \"weight_decay\": params['weight_decay'],\n },\n {\n \"params\": [\n p for n, p in param_optimizer if any(nd in n for nd in no_decay)\n ],\n \"weight_decay\": 0.0,\n },\n ]\n \n optimizer = FusedAdam(optimizer_parameters, lr=params['lr'])\n \n \n scheduler = CosineAnnealingLR(optimizer,\n T_max=params['T_max'],\n eta_min=params['min_lr'],\n last_epoch=-1)\n\n return dict(\n optimizer=optimizer,\n lr_scheduler=scheduler\n )\n","metadata":{"execution":{"iopub.status.busy":"2021-07-18T07:17:08.473751Z","iopub.execute_input":"2021-07-18T07:17:08.474181Z","iopub.status.idle":"2021-07-18T07:17:08.499484Z","shell.execute_reply.started":"2021-07-18T07:17:08.474146Z","shell.execute_reply":"2021-07-18T07:17:08.498609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Pytorch Lightning Trainer

\n\n#
\"Weights
\n\n

Once you’ve organized your PyTorch code into a LightningModule, the Trainer automates everything else.

📍 This abstraction achieves the following:

1. You maintain control over all aspects via PyTorch code without an added abstraction.
2. The trainer uses best practices embedded by contributors and users from top AI labs such as Facebook AI Research, NYU, MIT, Stanford, etc…
3. The trainer allows overriding any key part that you don’t want automated.

\n📍 Under the hood
\nUnder the hood, the Lightning Trainer handles the training loop details for you, some examples include:
1. Automatically enabling/disabling grads
2. Running the training, validation and test dataloaders
3. Calling the Callbacks at the appropriate times
4. Putting batches and computations on the correct devices

\n📍 The code below applies Stratified K-Fold with PL ⚡️\n\n","metadata":{}},{"cell_type":"code","source":"kfolds = StratifiedKFold(n_splits=params['nfolds'], shuffle=True, random_state=params['seed'])\n\nmodel = Trainer()\n\nfor fold, (trn_idx, val_idx) in enumerate(kfolds.split(train_df[\"id\"], train_df[\"target\"])):\n \n if fold != 0:\n continue\n \n wandb_logger = WandbLogger(project='G2Net-PL-Baseline', \n config=params,\n group='Effnet-CQT', \n job_type='train',\n name = f'Fold{fold}')\n \n \n print(f\"{'='*38} Fold: {fold} {'='*38}\")\n \n train_data = train_df.loc[trn_idx]\n valid_data = train_df.loc[val_idx]\n \n data_module = DataModule(\n train_data,\n valid_data, \n valid_data, \n )\n \n data_module.setup()\n val_samples = next(iter(data_module.val_dataloader()))\n \n \n early_stopping_callback = EarlyStopping(monitor='val_auc',mode=\"max\", patience=1)\n \n checkpoint_callback = ModelCheckpoint(\n dirpath=\"checkpoints\",\n filename=\"best-checkpoint-fold{fold}-val_auc{val_auc:.3f}\",\n save_top_k=params['epochs'],\n verbose=True,\n monitor=\"val_auc\",\n mode=\"max\"\n )\n \n trainer = pl.Trainer(\n gpus = 1,\n checkpoint_callback=checkpoint_callback,\n callbacks=[early_stopping_callback,\n ImagePredictionLogger(val_samples)],\n max_epochs=params['epochs'],\n precision = params['precision'],\n progress_bar_refresh_rate=1, \n stochastic_weight_avg = True, \n logger=wandb_logger\n )\n \n trainer.fit(model, data_module)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-18T07:17:08.500948Z","iopub.execute_input":"2021-07-18T07:17:08.501487Z","iopub.status.idle":"2021-07-18T07:50:59.148178Z","shell.execute_reply.started":"2021-07-18T07:17:08.501448Z","shell.execute_reply":"2021-07-18T07:50:59.146702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Check out the Weights and Biases Dashboard here $\\rightarrow$

","metadata":{}},{"cell_type":"markdown","source":"

Final Metrics


\n\n
\"metrics\"
","metadata":{}},{"cell_type":"markdown","source":"

Single Fold Metrics


\n\n
\"metrics\"
","metadata":{}},{"cell_type":"markdown","source":"

Validation Loss per Epoch


\n\n
\"metrics\"
","metadata":{}},{"cell_type":"markdown","source":"

GPU Utilization


\n\n
\"metrics\"
","metadata":{}}]}