{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7550085,"sourceType":"datasetVersion","datasetId":4397253},{"sourceId":162349255,"sourceType":"kernelVersion"}],"dockerImageVersionId":30648,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HMS - PyTorch Baseline Inference\n\n**Comments welcome!**\n\nOne of my goals in this competition is to learn more PyTorch.\n\nThis is an **inference** notebook; the respetive training notebook is [HMS - PyTorch Baseline Training](https://www.kaggle.com/code/morodertobias/hms-pytorch-baseline-training/notebook), and its trained models have been registered as a versioned dataset [HMS - PyTorch Baseline Training Dataset](https://www.kaggle.com/datasets/morodertobias/hms-pytorch-baseline-training-dataset).\n\nThe model uses squashed spectrograms, as done in the reference notebooks. I try to use my way of coding, but naturally it is similar. \n\nThis version uses the current version of the notebook, version 1 of dataset, and the last successful notebook run, version 8, hence 10 models in total. Each one is an EfficientNetB0 which have been fined-tuned from noisy student weights.","metadata":{}},{"cell_type":"markdown","source":"> ACTUALIZACION Y MEJORAN EN EL MODELO\n\nAl cambiar el tamaño de entrada del modelo a 64, es probable que se esté modificando la arquitectura de la red neuronal, específicamente la cantidad de neuronas en la capa de entrada. Esto podría ayudar en el modelo de varias maneras:\n\n    Más capacidad para capturar patrones: Al aumentar el tamaño de entrada, el modelo tiene más capacidad para capturar patrones complejos en los datos de entrada. Esto puede ser útil cuando los datos tienen características detalladas que necesitan ser modeladas de manera más precisa.\n\n    Mejora en la representación de características: Una capa de entrada más grande puede proporcionar una representación más rica de las características de entrada. Esto puede ser beneficioso para problemas donde las características de entrada son altamente informativas pero requieren una representación más detallada para ser correctamente capturadas por el modelo.\n\n    Generalización mejorada: Aunque no siempre es el caso, aumentar el tamaño de entrada podría ayudar al modelo a generalizar mejor en datos de prueba no vistos. Esto puede suceder si el modelo puede capturar una representación más completa de las características relevantes en los datos de entrenamiento.\n\nEn resumen, al aumentar el tamaño de entrada a 64, potencialmente estás permitiendo que el modelo capture patrones más complejos y represente mejor las características de entrada, lo que podría llevar a una mejora en el rendimiento del modelo en términos de precisión y capacidad de generalización. Sin embargo, es importante experimentar con diferentes configuraciones y evaluar el rendimiento del modelo para determinar el impacto exacto de este cambio. actualizacion por Alejandro cuevas V https://www.kaggle.com/pollicio","metadata":{}},{"cell_type":"markdown","source":"## Core References\n- [HMS: Inference (LB: 0.42)](https://www.kaggle.com/code/andreasbis/hms-inference-lb-0-42)\n- [HMS-HBAC: ResNet34d Baseline [Inference]](https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-inference)\n- [HMS: Train EfficientNetB0](https://www.kaggle.com/code/andreasbis/hms-train-efficientnetb0)\n- [HMS baseline_resnet34d(512*512 Training 5 folds)](https://www.kaggle.com/code/yunsuxiaozi/hms-baseline-resnet34d-512-512-training-5-folds)\n- [https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-training/](https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-training/)","metadata":{}},{"cell_type":"markdown","source":"## Table of Contents\n- [Imports](#Imports)\n- [Config](#Config)\n- [Prepare data](#Prepare-data)\n- [Prepare model](#Prepare-model)\n- [Predict](#Predict)\n- [Finalize submission](#Finalize-submission)","metadata":{}},{"cell_type":"markdown","source":"# Imports","metadata":{}},{"cell_type":"code","source":"import os\nimport pathlib\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm\nimport torch\nimport torch.nn.functional as F\nimport torchvision.transforms as transforms\nimport timm\nfrom torch.utils.data import Dataset, DataLoader","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-02-11T05:51:32.316872Z","iopub.execute_input":"2024-02-11T05:51:32.317879Z","iopub.status.idle":"2024-02-11T05:51:32.323380Z","shell.execute_reply.started":"2024-02-11T05:51:32.317835Z","shell.execute_reply":"2024-02-11T05:51:32.322307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Config","metadata":{}},{"cell_type":"code","source":"class CFG:\n    base_dir = pathlib.Path(\"/kaggle/input/hms-harmful-brain-activity-classification\")\n    path_test = base_dir / \"test.csv\"\n    path_submission = base_dir / \"sample_submission.csv\"\n    spec_dir = base_dir / \"test_spectrograms\"\n    model_name = \"tf_efficientnet_b0_ns\"\n    model_weights = sorted(\n        list(pathlib.Path(\"/kaggle/input/hms-pytorch-baseline-training-dataset\").glob(\"*.pt\"))\n        + list(pathlib.Path(\"/kaggle/input/hms-pytorch-baseline-training\").glob(\"*.pt\"))\n    )\n    transform = transforms.Resize((512, 512), antialias=False)\n    batch_size = 64\n    label_columns = [\n        \"seizure_vote\",\n        \"lpd_vote\",\n        \"gpd_vote\",\n        \"lrda_vote\",\n        \"grda_vote\",\n        \"other_vote\",\n    ]\n\n\nCFG.model_weights","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:51:38.753610Z","iopub.execute_input":"2024-02-11T05:51:38.753971Z","iopub.status.idle":"2024-02-11T05:51:38.765521Z","shell.execute_reply.started":"2024-02-11T05:51:38.753942Z","shell.execute_reply":"2024-02-11T05:51:38.764484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare data\n- Load test dataframe.\n- Prepare Dataset and DataLoader.\n- Check one example to see that everything is correct.","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv(CFG.path_test)\nsubmission = pd.read_csv(CFG.path_submission)\nsubmission = pd.merge(submission, test, how=\"inner\", on=\"eeg_id\")\nsubmission[\"path\"] = submission[\"spectrogram_id\"].map(lambda x: CFG.spec_dir / f\"{x}.parquet\")\nsubmission","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:51:47.472320Z","iopub.execute_input":"2024-02-11T05:51:47.473050Z","iopub.status.idle":"2024-02-11T05:51:47.495497Z","shell.execute_reply.started":"2024-02-11T05:51:47.473019Z","shell.execute_reply":"2024-02-11T05:51:47.494590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def preprocess(x):\n    x = np.clip(x, np.exp(-6), np.exp(10))\n    x = np.log(x)\n    m, s = x.mean(), x.std()\n    x = (x - m) / (s + 1e-6)\n    return x\n\n\nclass SpecDataset(Dataset):\n    \n    def __init__(self, df, transform=CFG.transform):\n        self.df = df\n        self.transform = transform\n    \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, index):\n        row = self.df.iloc[index]\n        # input\n        x = pd.read_parquet(row.path)\n        x = x.fillna(-1).values[:, 1:].T\n        x = preprocess(x)\n        x = torch.Tensor(x[None, :])\n        if self.transform:\n            x = self.transform(x)\n        # output\n        y = np.array(row.loc[CFG.label_columns].values, 'float32')\n        y = torch.Tensor(y)\n        return x, y","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:51:57.976570Z","iopub.execute_input":"2024-02-11T05:51:57.977429Z","iopub.status.idle":"2024-02-11T05:51:57.985887Z","shell.execute_reply.started":"2024-02-11T05:51:57.977397Z","shell.execute_reply":"2024-02-11T05:51:57.984923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_ds = SpecDataset(df=submission)\ndata_loader = DataLoader(dataset=data_ds, num_workers=os.cpu_count())\ndata_loader","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:04.610283Z","iopub.execute_input":"2024-02-11T05:52:04.611123Z","iopub.status.idle":"2024-02-11T05:52:04.618861Z","shell.execute_reply.started":"2024-02-11T05:52:04.611082Z","shell.execute_reply":"2024-02-11T05:52:04.617786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x, y = next(iter(data_loader))\nx.shape, x","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:08.113894Z","iopub.execute_input":"2024-02-11T05:52:08.114288Z","iopub.status.idle":"2024-02-11T05:52:08.350082Z","shell.execute_reply.started":"2024-02-11T05:52:08.114258Z","shell.execute_reply":"2024-02-11T05:52:08.348966Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(x[0, 0])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:12.657385Z","iopub.execute_input":"2024-02-11T05:52:12.658115Z","iopub.status.idle":"2024-02-11T05:52:12.919894Z","shell.execute_reply.started":"2024-02-11T05:52:12.658078Z","shell.execute_reply":"2024-02-11T05:52:12.918958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare model","metadata":{}},{"cell_type":"code","source":"DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(f\"DEVICE: {DEVICE}\")","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:18.770783Z","iopub.execute_input":"2024-02-11T05:52:18.771402Z","iopub.status.idle":"2024-02-11T05:52:18.776247Z","shell.execute_reply.started":"2024-02-11T05:52:18.771370Z","shell.execute_reply":"2024-02-11T05:52:18.775193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = timm.create_model(model_name=CFG.model_name, pretrained=False, num_classes=6, in_chans=1)\nmodel.to(DEVICE)\nnum_parameter = sum(x.numel() for x in model.parameters())\nprint(f\"Model has {num_parameter} parameters.\")","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:24.397580Z","iopub.execute_input":"2024-02-11T05:52:24.397920Z","iopub.status.idle":"2024-02-11T05:52:24.523577Z","shell.execute_reply.started":"2024-02-11T05:52:24.397895Z","shell.execute_reply":"2024-02-11T05:52:24.522618Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Predict\n- Load weights and compute individual predictions.\n- Note, the output of the model are logits.\n- Final predicition is the ensemble of all invidiual predictions.","metadata":{}},{"cell_type":"code","source":"prediction = pd.DataFrame(0.0, columns=CFG.label_columns, index=submission.index)\nfor i, path_weight in enumerate(CFG.model_weights):\n    print(f\"Model {i}: {path_weight}\")\n    model.load_state_dict(torch.load(path_weight))\n    model.eval()\n    with torch.no_grad():\n        res = []\n        for x, y in data_loader:\n            x = x.to(DEVICE)\n            pred = model(x)\n            pred = F.softmax(pred, dim=1)\n            pred = pred.detach().cpu().numpy()\n            res.append(pred)\n        res = np.concatenate(res)\n        res = pd.DataFrame(res, columns=CFG.label_columns, index=submission.index)\n        display(res)\n        prediction = prediction + res\n        print(\"\\n\")\nprediction = prediction / len(CFG.model_weights)","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:29.874190Z","iopub.execute_input":"2024-02-11T05:52:29.874947Z","iopub.status.idle":"2024-02-11T05:52:33.682044Z","shell.execute_reply.started":"2024-02-11T05:52:29.874916Z","shell.execute_reply":"2024-02-11T05:52:33.681002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"prediction","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:39.938876Z","iopub.execute_input":"2024-02-11T05:52:39.939315Z","iopub.status.idle":"2024-02-11T05:52:39.951122Z","shell.execute_reply.started":"2024-02-11T05:52:39.939273Z","shell.execute_reply":"2024-02-11T05:52:39.950178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Finalize submission","metadata":{"execution":{"iopub.status.busy":"2024-02-04T07:41:51.765545Z","iopub.execute_input":"2024-02-04T07:41:51.765946Z","iopub.status.idle":"2024-02-04T07:41:51.774172Z","shell.execute_reply.started":"2024-02-04T07:41:51.765901Z","shell.execute_reply":"2024-02-04T07:41:51.773122Z"}}},{"cell_type":"code","source":"submission[CFG.label_columns] = prediction\nsubmission = submission[[\"eeg_id\"] + CFG.label_columns]\nsubmission","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:42.800206Z","iopub.execute_input":"2024-02-11T05:52:42.800593Z","iopub.status.idle":"2024-02-11T05:52:42.817160Z","shell.execute_reply.started":"2024-02-11T05:52:42.800564Z","shell.execute_reply":"2024-02-11T05:52:42.816058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\ninput_size = 64\nhidden_size = 128  # Define el tamaño de la capa oculta\noutput_size = 10\nmodel = nn.Sequential(\n    nn.Linear(input_size, hidden_size),\n    nn.ReLU(),\n    nn.Dropout(0.5),  # Ejemplo de dropout\n    nn.Linear(hidden_size, output_size)\n)\n","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:45.729713Z","iopub.execute_input":"2024-02-11T05:52:45.730069Z","iopub.status.idle":"2024-02-11T05:52:45.738709Z","shell.execute_reply.started":"2024-02-11T05:52:45.730036Z","shell.execute_reply":"2024-02-11T05:52:45.737821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.to_csv(\"submission.csv\", index=None)","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:48.743456Z","iopub.execute_input":"2024-02-11T05:52:48.744075Z","iopub.status.idle":"2024-02-11T05:52:48.750246Z","shell.execute_reply.started":"2024-02-11T05:52:48.744044Z","shell.execute_reply":"2024-02-11T05:52:48.749240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2024-02-11T05:52:52.666069Z","iopub.execute_input":"2024-02-11T05:52:52.666798Z","iopub.status.idle":"2024-02-11T05:52:53.618648Z","shell.execute_reply.started":"2024-02-11T05:52:52.666766Z","shell.execute_reply":"2024-02-11T05:52:53.617680Z"},"trusted":true},"execution_count":null,"outputs":[]}]}