{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7447509,"sourceType":"datasetVersion","datasetId":4334995},{"sourceId":7585255,"sourceType":"datasetVersion","datasetId":4415285},{"sourceId":7658681,"sourceType":"datasetVersion","datasetId":4465437},{"sourceId":7813680,"sourceType":"datasetVersion","datasetId":4430326}],"dockerImageVersionId":30646,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Brain Solver Python Training Notebook\n\nThis notebook utilizes the custom `brain_solver` package for analyzing brain activity data. Our data sources include official datasets from Kaggle competitions and additional datasets for enhanced model training and evaluation.\n\nThis is the Training notebook.\n\n## Data Sources\n\n### Official:\n\n- **HMS - Harmful Brain Activity Classification**\n  - **Source:** [Kaggle Competition](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification)\n  - **Description:** This competition focuses on classifying harmful brain activity. It includes a comprehensive dataset for training and testing models.\n\n- **Brain-Spectrograms**\n  - **Source:** [Kaggle Dataset](https://www.kaggle.com/datasets/cdeotte/brain-spectrograms)\n  - **Description:** The `specs.npy` file contains all the spectrograms from the HMS competition, offering a detailed view of brain activity through visual representations.\n\n### Additional:\n\n- **Brain-EEG-Spectrograms**\n  - **Source:** [Kaggle Dataset](https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms)\n  - **Description:** The `EEG_Spectrograms` folder includes one NumPy file per EEG ID, with each array shaped as (128x256x4), representing (frequency, time, montage chain). This dataset provides a more nuanced understanding of brain activity through EEG spectrograms.\n\n- **hms_efficientnetb0_pt_ckpts**\n  - **Source:** [Kaggle Dataset](https://www.kaggle.com/datasets/crackle/hms-efficientnetb0-pt-ckpts)\n  - **Description:** This dataset offers pre-trained checkpoints for EfficientNetB0 models, tailored for the HMS competition. It's intended for use in fine-tuning models on the specific task of harmful brain activity classification.\n","metadata":{}},{"cell_type":"code","source":"!pip install d2l --no-index --find-links=file:///kaggle/input/d2l-package/d2l/\n!pip install /kaggle/input/brain-solver/brain_solver-0.9.0-py3-none-any.whl","metadata":{"execution":{"iopub.status.busy":"2024-03-11T00:04:21.605428Z","iopub.execute_input":"2024-03-11T00:04:21.605705Z","iopub.status.idle":"2024-03-11T00:05:12.367706Z","shell.execute_reply.started":"2024-03-11T00:04:21.605679Z","shell.execute_reply":"2024-03-11T00:05:12.366488Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os, sys\nimport gc\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom torch.utils.data import DataLoader\nimport pytorch_lightning as pl\nfrom brain_solver import Helpers as hp, Trainer as tr, BrainModel as br, EEGDataset, Network\nfrom brain_solver import Wav2Vec2 as w2v\n\n# Suppress warnings if desired\nimport warnings\n\nwarnings.filterwarnings(\"ignore\")\n\n# Setup for CUDA device selection\nos.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0,1\"\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\ndevice","metadata":{"execution":{"iopub.status.busy":"2024-03-11T00:05:12.370185Z","iopub.execute_input":"2024-03-11T00:05:12.370571Z","iopub.status.idle":"2024-03-11T00:05:29.147555Z","shell.execute_reply.started":"2024-03-11T00:05:12.370535Z","shell.execute_reply":"2024-03-11T00:05:29.146475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Config Class Summary\n\nThe `Config` class manages configurations for a brain activity classification project. It includes:\n\n- **Data and Model Paths**: Centralizes paths for data (e.g., EEG, spectrograms) and model checkpoints.\n- **Training Parameters**: Configures training details like epochs, batch size, and learning rate.\n- **Feature Flags**: Toggles for using wavelets, spectrograms, and reading options.\n\nDesigned for easy adjustments to facilitate model development and experimentation.\n","metadata":{}},{"cell_type":"code","source":"from brain_solver import Config\n# full_path = \"/home/osloup/NoodleNappers/brain/data/\"\n# full_path = \"C:/Users/tygof/Documents/Semester 8/MLiP/NoodleNappers/brain/data/\"\n# config = Config(full_path,  full_path + \"out/\", USE_EEG_SPECTROGRAMS=True, USE_KAGGLE_SPECTROGRAMS=True, should_read_brain_spectograms=False, should_read_eeg_spectrogram_files=False, USE_PRETRAINED_MODEL=False, FINE_TUNE=False)\n\n\nfull_path = \"/kaggle/input/\"\nconfig = Config(full_path, \"/kaggle/working/\", USE_EEG_SPECTROGRAMS=True, USE_KAGGLE_SPECTROGRAMS=True, should_read_brain_spectograms=False, should_read_eeg_spectrogram_files=False, USE_PRETRAINED_MODEL=False, FINE_TUNE=False)\n\nimport sys\nsys.path.append(full_path + 'kaggle-kl-div')\nfrom kaggle_kl_div import score","metadata":{"tags":[],"execution":{"iopub.status.busy":"2024-03-11T00:05:29.152189Z","iopub.execute_input":"2024-03-11T00:05:29.152460Z","iopub.status.idle":"2024-03-11T00:05:29.178572Z","shell.execute_reply.started":"2024-03-11T00:05:29.152436Z","shell.execute_reply":"2024-03-11T00:05:29.177850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create Output folder if does not exist\nif not os.path.exists(config.output_path):\n    os.makedirs(config.output_path)\n\n# Initialize random environment\npl.seed_everything(config.seed, workers=True)","metadata":{"execution":{"iopub.status.busy":"2024-03-11T00:05:29.179558Z","iopub.execute_input":"2024-03-11T00:05:29.179813Z","iopub.status.idle":"2024-03-11T00:05:29.193757Z","shell.execute_reply.started":"2024-03-11T00:05:29.179789Z","shell.execute_reply":"2024-03-11T00:05:29.192963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df: pd.DataFrame = hp.load_csv(config.data_train_csv)\n\nif train_df is None:\n    print(\"Failed to load the CSV file.\")\n    exit()\nelse:\n    EEG_IDS = train_df.eeg_id.unique()\n    TARGETS = train_df.columns[-6:]\n    TARS = {\"Seizure\": 0, \"LPD\": 1, \"GPD\": 2, \"LRDA\": 3, \"GRDA\": 4, \"Other\": 5}\n    TARS_INV = {x: y for y, x in TARS.items()}\n    print(\"Train shape:\", train_df.shape)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:17:12.803263Z","iopub.execute_input":"2024-03-10T23:17:12.803947Z","iopub.status.idle":"2024-03-10T23:17:13.083267Z","shell.execute_reply.started":"2024-03-10T23:17:12.803911Z","shell.execute_reply":"2024-03-10T23:17:13.082365Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_preprocessed = hp.preprocess_eeg_data(train_df, TARGETS)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:17:13.084537Z","iopub.execute_input":"2024-03-10T23:17:13.085282Z","iopub.status.idle":"2024-03-10T23:17:13.295850Z","shell.execute_reply.started":"2024-03-10T23:17:13.085246Z","shell.execute_reply":"2024-03-10T23:17:13.294797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data_preprocessed.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:17:13.297433Z","iopub.execute_input":"2024-03-10T23:17:13.298025Z","iopub.status.idle":"2024-03-10T23:17:13.324003Z","shell.execute_reply.started":"2024-03-10T23:17:13.297983Z","shell.execute_reply":"2024-03-10T23:17:13.322910Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spectrograms = hp.read_spectrograms(\n    config.data_spectograms,\n    config.path_to_brain_spectrograms_npy,\n    config.should_read_brain_spectograms,\n)\n\ndata_eeg_spectograms = hp.read_eeg_spectrograms(\n   train_data_preprocessed,\n   config.path_to_eeg_spectrograms_folder,\n   config.path_to_eeg_spectrograms_npy,\n   config.should_read_eeg_spectrogram_files,\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:17:13.325219Z","iopub.execute_input":"2024-03-10T23:17:13.325532Z","iopub.status.idle":"2024-03-10T23:19:35.755060Z","shell.execute_reply.started":"2024-03-10T23:17:13.325505Z","shell.execute_reply":"2024-03-10T23:19:35.754088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# specs_wav = w2v.wav2vec2(spectrograms)\n# dataset2 = EEGDataset(\n#     train_data_preprocessed, specs_wav, data_eeg_spectograms, TARGETS\n# )\n# dataloader2 = DataLoader(dataset2, batch_size=32, shuffle=False)\n# hp.plot_spectrograms(\n#     dataloader2, train_data_preprocessed, ROWS=2, COLS=3, BATCHES=2\n# )","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:19:35.756365Z","iopub.execute_input":"2024-03-10T23:19:35.756667Z","iopub.status.idle":"2024-03-10T23:19:35.760721Z","shell.execute_reply.started":"2024-03-10T23:19:35.756642Z","shell.execute_reply":"2024-03-10T23:19:35.759890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\n    f\"Length of spectrograms: {spectrograms.__len__()}, Length of all EEGs: {data_eeg_spectograms.__len__()}\"\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:19:35.763668Z","iopub.execute_input":"2024-03-10T23:19:35.763981Z","iopub.status.idle":"2024-03-10T23:19:35.776771Z","shell.execute_reply.started":"2024-03-10T23:19:35.763958Z","shell.execute_reply":"2024-03-10T23:19:35.775757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dataset = EEGDataset(\n    train_data_preprocessed, spectrograms, data_eeg_spectograms, TARGETS\n)\ndataloader = DataLoader(dataset, batch_size=32, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:19:35.778033Z","iopub.execute_input":"2024-03-10T23:19:35.778332Z","iopub.status.idle":"2024-03-10T23:19:35.790062Z","shell.execute_reply.started":"2024-03-10T23:19:35.778308Z","shell.execute_reply":"2024-03-10T23:19:35.789379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"hp.plot_spectrograms(\n    dataloader, train_data_preprocessed, ROWS=2, COLS=3, BATCHES=2\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:19:35.791259Z","iopub.execute_input":"2024-03-10T23:19:35.791608Z","iopub.status.idle":"2024-03-10T23:19:38.820484Z","shell.execute_reply.started":"2024-03-10T23:19:35.791577Z","shell.execute_reply":"2024-03-10T23:19:38.819402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del dataset, dataloader, train_df\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:19:38.821581Z","iopub.execute_input":"2024-03-10T23:19:38.821857Z","iopub.status.idle":"2024-03-10T23:19:39.048562Z","shell.execute_reply.started":"2024-03-10T23:19:38.821833Z","shell.execute_reply":"2024-03-10T23:19:39.047654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# model = Network(\n#                 config.trained_weight_file,\n#                 config.USE_KAGGLE_SPECTROGRAMS,\n#                 config.USE_EEG_SPECTROGRAMS,\n#             ).to(device)\n# for param in model.base_model.parameters():\n#     param.requires_grad = False\n\n# for param in model.base_model.avgpool.parameters():\n#     param.requires_grad = True\n\n# for param in model.base_model.classifier.parameters():\n#     param.requires_grad = True\n\n     \n# # def print_layers(model, prefix, counter):\n# #     for name, module in reversed(list(model.named_children())):\n# #         if list(module.children()):  # If the module has children\n# #             print_layers(module, prefix + name + '.', counter)\n# #         else:\n# #             print(prefix + name, module)\n# #             counter+= 1\n# #         if counter > 15:\n# #             break\n\n# def set_requires_grad_for_first_n_layers(model, n):\n#     count = 0\n\n#     def set_grad(module):\n#         nonlocal count\n#         if isinstance(module, torch.nn.BatchNorm2d):\n#             return\n#         for param in module.parameters():\n#             param.requires_grad = True\n#         count += 1\n#         if count == n:\n#             return\n#         if isinstance(module, torch.nn.Module):\n#             for child_name, child_module in module.named_children():\n#                 set_grad(child_module)\n\n#     # Start the recursive traversal from the top-level modules\n#     for name, module in reversed(list(model.named_children())):\n#         set_grad(module)\n#         if count == n:\n#             break\n\n# # Assuming `model` is your PyTorch model\n# set_requires_grad_for_first_n_layers(model, 15)\n","metadata":{"execution":{"iopub.status.busy":"2024-03-11T00:31:41.424795Z","iopub.execute_input":"2024-03-11T00:31:41.425172Z","iopub.status.idle":"2024-03-11T00:31:41.622621Z","shell.execute_reply.started":"2024-03-11T00:31:41.425142Z","shell.execute_reply":"2024-03-11T00:31:41.621643Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_oof, all_true, valid_loaders = br.cross_validate_eeg(\n    config,\n    device,\n    train_data_preprocessed=train_data_preprocessed,\n    spectrograms=spectrograms,\n    data_eeg_spectograms=data_eeg_spectograms,\n    TARGETS=TARGETS,\n    n_splits=5,\n    batch_size_train=32,\n    batch_size_valid=64,\n    max_epochs_first_stage=8,\n    max_epochs_second_stage=5,\n    num_workers=3,\n)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T23:33:25.575289Z","iopub.execute_input":"2024-03-10T23:33:25.575682Z","iopub.status.idle":"2024-03-10T23:33:31.243704Z","shell.execute_reply.started":"2024-03-10T23:33:25.575650Z","shell.execute_reply":"2024-03-10T23:33:31.241889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_oof, all_true = br.validate_model_across_folds(config, device, all_oof, all_true, valid_loaders)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T19:13:12.284583Z","iopub.status.idle":"2024-03-10T19:13:12.285030Z","shell.execute_reply.started":"2024-03-10T19:13:12.284802Z","shell.execute_reply":"2024-03-10T19:13:12.284820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof.copy())\noof[\"id\"] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue[\"id\"] = np.arange(len(true))\n\n# Calculate the score\ncv = score(solution=true, submission=oof, row_id_column_name=\"id\")\nprint(\"CV Score KL-Div for EfficientNetB0 =\", cv)","metadata":{"execution":{"iopub.status.busy":"2024-03-10T19:13:12.286217Z","iopub.status.idle":"2024-03-10T19:13:12.286640Z","shell.execute_reply.started":"2024-03-10T19:13:12.286417Z","shell.execute_reply":"2024-03-10T19:13:12.286435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del data_eeg_spectograms, spectrograms\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-03-10T19:13:12.287921Z","iopub.status.idle":"2024-03-10T19:13:12.288355Z","shell.execute_reply.started":"2024-03-10T19:13:12.288124Z","shell.execute_reply":"2024-03-10T19:13:12.288142Z"},"trusted":true},"execution_count":null,"outputs":[]}]}