{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"}],"dockerImageVersionId":30646,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HMS - PyTorch Baseline Training\n\n**Comments welcome!**\n\nOne of my goals in this competition is to learn more PyTorch.\n\nThis is a **training** notebook; the respective inference notebook is [HMS - PyTorch Baseline Inference](https://www.kaggle.com/code/morodertobias/hms-pytorch-baseline-inference) and the saved models are uploaded as a dataset [HMS - PyTorch Baseline Training Dataset](https://www.kaggle.com/datasets/morodertobias/hms-pytorch-baseline-training-dataset) to keep versions clean.\n\nCurrent version fine-tunes EfficientNetB0 from noisy student weights in a 5-folding manner. It uses squashed spectrograms, as already done in the reference notebooks. I try to use my way of coding, but naturally it is similar.\n\n\n## References\n- [HMS baseline_resnet34d(512*512 Training 5 folds)](https://www.kaggle.com/code/yunsuxiaozi/hms-baseline-resnet34d-512-512-training-5-folds)\n- [HMS: Train EfficientNetB0](https://www.kaggle.com/code/andreasbis/hms-train-efficientnetb0/notebook)\n- [HMS-HBAC: ResNet34d Baseline [Training]](https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-training/)\n\n\n## Table of Contents\n- [Imports](#Imports)\n- [Config](#Config)\n- [Load data](#Load-data)\n- [Data Handling](#Data-Handling)\n- [Model](#Model)\n- [Training Utils](#Training-Utils)\n- [Training](#Training)\n- [OOF](#OOF)","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport pathlib\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision.transforms as transforms\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import CosineAnnealingLR\nimport timm","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:18.374782Z","iopub.execute_input":"2024-02-07T19:29:18.375230Z","iopub.status.idle":"2024-02-07T19:29:18.381214Z","shell.execute_reply.started":"2024-02-07T19:29:18.375201Z","shell.execute_reply":"2024-02-07T19:29:18.380292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"class CFG:\n    seed = 888\n    debug = False\n    one_fold = False\n    base_dir = pathlib.Path(\"/kaggle/input/hms-harmful-brain-activity-classification\")\n    path_train = base_dir / \"train.csv\"\n    spec_dir = base_dir / \"train_spectrograms\"\n    transform = transforms.Resize((512, 512), antialias=False)\n    model_name = \"tf_efficientnet_b0_ns\"\n    ckpt_name = \"tf_efficientnet_b0_ns_v1\"\n    n_fold = 5\n    epochs = 10\n    batch_size = 16    \n    lr = 0.0005","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:23.666870Z","iopub.execute_input":"2024-02-07T19:29:23.668068Z","iopub.status.idle":"2024-02-07T19:29:23.674095Z","shell.execute_reply.started":"2024-02-07T19:29:23.668029Z","shell.execute_reply":"2024-02-07T19:29:23.673082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load data\n- We classify the spectrogram directly.\n- As a label we use the aggregated the votes per spectrogram slice","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/train.csv\")\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:24.091834Z","iopub.execute_input":"2024-02-07T19:29:24.092176Z","iopub.status.idle":"2024-02-07T19:29:24.276473Z","shell.execute_reply.started":"2024-02-07T19:29:24.092150Z","shell.execute_reply":"2024-02-07T19:29:24.275414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_columns = train_df.filter(like=\"_vote\").columns.to_list()\nlabel_columns","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:24.328911Z","iopub.execute_input":"2024-02-07T19:29:24.329473Z","iopub.status.idle":"2024-02-07T19:29:24.336993Z","shell.execute_reply.started":"2024-02-07T19:29:24.329443Z","shell.execute_reply":"2024-02-07T19:29:24.336016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = train_df.groupby(\"spectrogram_id\")[label_columns].sum()\ndata = data.div(data.sum(axis=1), axis=0)\ndata[\"path\"] = data.index.map(lambda x: CFG.spec_dir / f\"{x}.parquet\")\ndata = data.reset_index()\ndata","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:24.532032Z","iopub.execute_input":"2024-02-07T19:29:24.532469Z","iopub.status.idle":"2024-02-07T19:29:24.636300Z","shell.execute_reply.started":"2024-02-07T19:29:24.532438Z","shell.execute_reply":"2024-02-07T19:29:24.635282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Handling\n- Spectrogram is loaded, which contains spectrograms in 4 different regions.\n- All together they are clipped, log-transformed and the standardized.\n- Finally it is resized as an image.\n- Note, since the spectrograms have sometimes very different lenghts this creates a certain distortion.","metadata":{}},{"cell_type":"code","source":"from torch.utils.data import Dataset, DataLoader","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:25.313518Z","iopub.execute_input":"2024-02-07T19:29:25.314336Z","iopub.status.idle":"2024-02-07T19:29:25.319525Z","shell.execute_reply.started":"2024-02-07T19:29:25.314299Z","shell.execute_reply":"2024-02-07T19:29:25.318497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess(x):\n    x = np.clip(x, np.exp(-6), np.exp(10))\n    x = np.log(x)\n    m, s = x.mean(), x.std()\n    x = (x - m) / (s + 1e-6)\n    return x\n\n\nclass SpecDataset(Dataset):\n    \n    def __init__(self, df, transform=CFG.transform):\n        self.df = df\n        self.transform = transform\n    \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, index):\n        row = self.df.iloc[index]\n        # input\n        x = pd.read_parquet(row.path)\n        x = x.fillna(-1).values[:, 1:].T\n        x = preprocess(x)\n        x = torch.Tensor(x[None, :])\n        if self.transform:\n            x = self.transform(x)\n        # output\n        y = np.array(row.loc[label_columns].values, 'float32')\n        y = torch.Tensor(y)\n        return x, y","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:25.602712Z","iopub.execute_input":"2024-02-07T19:29:25.603522Z","iopub.status.idle":"2024-02-07T19:29:25.614842Z","shell.execute_reply.started":"2024-02-07T19:29:25.603480Z","shell.execute_reply":"2024-02-07T19:29:25.613791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ds = SpecDataset(df=data.iloc[:50])\nds, len(ds)","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:25.950649Z","iopub.execute_input":"2024-02-07T19:29:25.951262Z","iopub.status.idle":"2024-02-07T19:29:25.957896Z","shell.execute_reply.started":"2024-02-07T19:29:25.951220Z","shell.execute_reply":"2024-02-07T19:29:25.956858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x, y = ds[0]\nx.shape, x, y.shape, y","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:26.306793Z","iopub.execute_input":"2024-02-07T19:29:26.307345Z","iopub.status.idle":"2024-02-07T19:29:26.348276Z","shell.execute_reply.started":"2024-02-07T19:29:26.307317Z","shell.execute_reply":"2024-02-07T19:29:26.347400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ld = DataLoader(dataset=ds, batch_size=CFG.batch_size, drop_last=True, num_workers=os.cpu_count())\nld, len(ld)","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:26.665992Z","iopub.execute_input":"2024-02-07T19:29:26.666695Z","iopub.status.idle":"2024-02-07T19:29:26.673237Z","shell.execute_reply.started":"2024-02-07T19:29:26.666663Z","shell.execute_reply":"2024-02-07T19:29:26.672278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x, y = next(iter(ld))\nx.shape, x, y.shape, y","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:27.236655Z","iopub.execute_input":"2024-02-07T19:29:27.237339Z","iopub.status.idle":"2024-02-07T19:29:28.209813Z","shell.execute_reply.started":"2024-02-07T19:29:27.237306Z","shell.execute_reply":"2024-02-07T19:29:28.208482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(x[0, 0])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:28.212268Z","iopub.execute_input":"2024-02-07T19:29:28.213101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"code","source":"DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(f\"DEVICE: {DEVICE}\")","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:28.962340Z","iopub.execute_input":"2024-02-07T19:29:28.962714Z","iopub.status.idle":"2024-02-07T19:29:28.989981Z","shell.execute_reply.started":"2024-02-07T19:29:28.962685Z","shell.execute_reply":"2024-02-07T19:29:28.988881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cuda.memory_allocated()","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:29.810174Z","iopub.execute_input":"2024-02-07T19:29:29.810561Z","iopub.status.idle":"2024-02-07T19:29:29.816979Z","shell.execute_reply.started":"2024-02-07T19:29:29.810531Z","shell.execute_reply":"2024-02-07T19:29:29.816011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = timm.create_model(model_name=CFG.model_name, pretrained=True, num_classes=6, in_chans=1)\nmodel.to(DEVICE)\nnum_parameter = sum(x.numel() for x in model.parameters())\nprint(f\"Model has {num_parameter} parameters.\")","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:30.130892Z","iopub.execute_input":"2024-02-07T19:29:30.131275Z","iopub.status.idle":"2024-02-07T19:29:30.870878Z","shell.execute_reply.started":"2024-02-07T19:29:30.131232Z","shell.execute_reply":"2024-02-07T19:29:30.869845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_out = model(x.to(DEVICE))\ny_out","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:31.363117Z","iopub.execute_input":"2024-02-07T19:29:31.363743Z","iopub.status.idle":"2024-02-07T19:29:32.308061Z","shell.execute_reply.started":"2024-02-07T19:29:31.363711Z","shell.execute_reply":"2024-02-07T19:29:32.306979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training Utils\n- Create Kullback-Leibler Divergence loss from logits.\n- Compute loss utility.","metadata":{}},{"cell_type":"code","source":"def KLDivLoss(logit, target):\n    log_prob = F.log_softmax(logit, dim=1)\n    return F.kl_div(log_prob, target, reduction=\"batchmean\")","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:32.969486Z","iopub.execute_input":"2024-02-07T19:29:32.970183Z","iopub.status.idle":"2024-02-07T19:29:32.975145Z","shell.execute_reply.started":"2024-02-07T19:29:32.970149Z","shell.execute_reply":"2024-02-07T19:29:32.973959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from reference\ndef KL_loss(p,q):\n    epsilon = 10**(-15)\n    p = torch.clip(p,epsilon,1-epsilon)\n    q = nn.functional.log_softmax(q,dim=1)\n    return torch.mean(torch.sum(p*(torch.log(p)-q),dim=1))","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:34.361689Z","iopub.execute_input":"2024-02-07T19:29:34.362071Z","iopub.status.idle":"2024-02-07T19:29:34.367872Z","shell.execute_reply.started":"2024-02-07T19:29:34.362045Z","shell.execute_reply":"2024-02-07T19:29:34.366953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"KLDivLoss(y_out, y.to(DEVICE)), KL_loss(p=y.to(DEVICE), q=y_out)","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:34.642138Z","iopub.execute_input":"2024-02-07T19:29:34.643035Z","iopub.status.idle":"2024-02-07T19:29:34.776636Z","shell.execute_reply.started":"2024-02-07T19:29:34.642998Z","shell.execute_reply":"2024-02-07T19:29:34.775727Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def compute_loss(model, data_loader):\n    model.eval()\n    l_loss = []\n    with torch.no_grad():\n        for x, y in data_loader:\n            y_pred = model(x.to(DEVICE))\n            loss = KLDivLoss(y_pred, y.to(DEVICE))\n            l_loss.append(loss.item())\n    return np.mean(l_loss) ","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:35.344990Z","iopub.execute_input":"2024-02-07T19:29:35.345868Z","iopub.status.idle":"2024-02-07T19:29:35.351277Z","shell.execute_reply.started":"2024-02-07T19:29:35.345834Z","shell.execute_reply":"2024-02-07T19:29:35.350293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"compute_loss(model, ld)","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:36.152697Z","iopub.execute_input":"2024-02-07T19:29:36.153700Z","iopub.status.idle":"2024-02-07T19:29:37.319389Z","shell.execute_reply.started":"2024-02-07T19:29:36.153663Z","shell.execute_reply":"2024-02-07T19:29:37.318221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cuda.memory_allocated()","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:37.321151Z","iopub.execute_input":"2024-02-07T19:29:37.321472Z","iopub.status.idle":"2024-02-07T19:29:37.327831Z","shell.execute_reply.started":"2024-02-07T19:29:37.321445Z","shell.execute_reply":"2024-02-07T19:29:37.326974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del model, x, y, y_out\ntorch.cuda.empty_cache()","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:37.328949Z","iopub.execute_input":"2024-02-07T19:29:37.329211Z","iopub.status.idle":"2024-02-07T19:29:37.360443Z","shell.execute_reply.started":"2024-02-07T19:29:37.329189Z","shell.execute_reply":"2024-02-07T19:29:37.359559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cuda.memory_allocated()","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:37.362147Z","iopub.execute_input":"2024-02-07T19:29:37.362409Z","iopub.status.idle":"2024-02-07T19:29:37.367925Z","shell.execute_reply.started":"2024-02-07T19:29:37.362387Z","shell.execute_reply":"2024-02-07T19:29:37.367083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training\n- Uses plain 5 fold training strategy.\n- Runs for all epochs and checkpoints model weights if the validation loss improves.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import KFold","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:38.729877Z","iopub.execute_input":"2024-02-07T19:29:38.730632Z","iopub.status.idle":"2024-02-07T19:29:39.307131Z","shell.execute_reply.started":"2024-02-07T19:29:38.730596Z","shell.execute_reply":"2024-02-07T19:29:39.306303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if CFG.debug:\n    data = data.iloc[:400]","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:29:45.368967Z","iopub.execute_input":"2024-02-07T19:29:45.369859Z","iopub.status.idle":"2024-02-07T19:29:45.374356Z","shell.execute_reply.started":"2024-02-07T19:29:45.369828Z","shell.execute_reply":"2024-02-07T19:29:45.373356Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nkf = KFold(n_splits=CFG.n_fold, shuffle=True, random_state=CFG.seed)\n\nl_best_loss = []\nfor fold, (iloc_train, iloc_valid) in enumerate(kf.split(data)):\n    print(f\"Fold {fold}:\")\n\n    # prepare data\n    train_ds = SpecDataset(df=data.iloc[iloc_train])\n    valid_ds = SpecDataset(df=data.iloc[iloc_valid])\n    train_loader = DataLoader(dataset=train_ds, shuffle=True, batch_size=CFG.batch_size, num_workers=os.cpu_count(), drop_last=True)\n    valid_loader = DataLoader(dataset=valid_ds, batch_size=CFG.batch_size, num_workers=os.cpu_count())\n    \n    # init training\n    model = timm.create_model(model_name=CFG.model_name, pretrained=True, num_classes=6, in_chans=1)\n    model.to(DEVICE)\n    optimizer = optim.AdamW(model.parameters(), lr=CFG.lr)\n    scheduler = CosineAnnealingLR(optimizer=optimizer, T_max=CFG.epochs)\n    optimizer, scheduler\n    best_loss = float(\"inf\")\n    history = []\n    \n    # run training\n    for epoch in tqdm(range(CFG.epochs)):\n        model.train()\n        l_loss = []\n        for x, y in train_loader:\n            x, y = x.to(DEVICE), y.to(DEVICE)\n            y_pred = model(x)\n            loss = KLDivLoss(y_pred, y)\n            l_loss.append(loss.item())\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n        train_loss = np.mean(l_loss)\n        valid_loss = compute_loss(model, valid_loader)\n        history.append((epoch, train_loss, valid_loss))\n        print(f\"Epoch {epoch}\")\n        print(f\"Train Loss: {train_loss:>10.6f}, Valid Loss: {valid_loss:>10.6}\")\n        if valid_loss < best_loss:\n            print(f\"Loss improves from {best_loss:>10.6f} to {valid_loss:>10.6}\")\n            torch.save(model.state_dict(), f\"{CFG.ckpt_name}__{fold}.pt\")\n            best_loss = valid_loss\n    print(f\"\\nBest loss Model training with {best_loss}\\n\")\n    l_best_loss.append(best_loss)\n    \n    # plot\n    history = pd.DataFrame(history, columns=[\"epoch\", \"loss\", \"val_loss\"]).set_index(\"epoch\")\n    history.plot(subplots=True, layout=(1, 2), sharey=\"row\", figsize=(14, 6))\n    plt.show()\n    \n    if CFG.one_fold:\n        break","metadata":{"execution":{"iopub.status.busy":"2024-02-07T19:30:04.084679Z","iopub.execute_input":"2024-02-07T19:30:04.085305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# OOF","metadata":{}},{"cell_type":"code","source":"l_best_loss, np.mean(l_best_loss)","metadata":{},"execution_count":null,"outputs":[]}]}