{"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7402356,"sourceType":"datasetVersion","datasetId":4304475},{"sourceId":7403069,"sourceType":"datasetVersion","datasetId":4304949},{"sourceId":7447509,"sourceType":"datasetVersion","datasetId":4334995},{"sourceId":7450712,"sourceType":"datasetVersion","datasetId":4336944},{"sourceId":158958765,"sourceType":"kernelVersion"}],"dockerImageVersionId":30636,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"papermill":{"default_parameters":{},"duration":270.012179,"end_time":"2024-01-14T22:56:02.916427","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-01-14T22:51:32.904248","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# | HMS: Harmful Brain Activity Classification [Train] \n\n***\n\n- [Inference Notebook](https://www.kaggle.com/alejopaullier/hms-efficientnetb0-pytorch-inference)\n- In case you don't want to train the model you can find my dataset with [5-fold trained Efficientnet models](https://www.kaggle.com/datasets/alejopaullier/hms-efficientnetb0-5-folds) that are the result of running this notebook.\n\nThis is the **PyTorch 🔥 version** of [Chris Deotte EfficientNetB0 Starter](https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43#Train-DataLoader) give him an upvote too ⬆️!\n\nYour goal in this competition is to detect and classify seizures and other types of harmful brain activity. You will develop a model trained on electroencephalography (EEG) signals recorded from critically ill hospital patients.\n\nIn this notebook you will learn how to train a `efficientnet` model for image classification using PyTorch. Hope you enjoy it and find it useful.\n\n### Table of Contents \n\n\n\n# | Introduction [↑](#top) \n\n***\n\n### What is an EEG waveform?\n\n**EEG** (Electroencephalogram) waveforms are the **patterns of electrical activity generated by the brain**, which are recorded using electrodes placed on the scalp. EEG is a non-invasive method that measures the electrical potentials produced by the firing of neurons in the brain. These electrical potentials are then amplified and displayed as waveforms on a computer or paper.\n\n- **Delta Waves (0.5-4 Hz):** Delta waves are slow-wave patterns associated with deep sleep and certain abnormal brain states. They are usually the dominant waves during deep sleep stages.\n- **Theta Waves (4-8 Hz):** Theta waves are associated with drowsiness, relaxation, and the early stages of sleep. They can also be present during deep meditation.\n- **Alpha Waves (8-13 Hz):** Alpha waves are dominant when a person is awake but relaxed and not actively processing information. They are commonly seen when a person's eyes are closed.\n- **Beta Waves (13-30 Hz):** Beta waves are associated with active, alert, and focused mental activity. They are commonly observed when a person is awake and engaged in cognitive tasks.\n- **Gamma Waves (30-100 Hz and above):** Gamma waves are associated with higher cognitive functions, such as perception, learning, and problem-solving. They are not always present and are often associated with specific cognitive tasks.\n\nIn this competition, EEG waveforms are 50 seconds long.\n\n### What is a spectogram?\n\nA spectrogram is a visual representation of the spectrum of frequencies in a signal as they vary with time. It is a three-dimensional plot that displays how the frequencies of a signal change over time. Spectrograms are commonly used in signal processing, audio analysis, and other fields to analyze the frequency content of a signal and how it evolves over time.\n\n### Useful References\n\n- [Understand this competition's data](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468010)\n","metadata":{}},{"cell_type":"markdown","source":"# | Import Libraries [↑](#top) \n\n***\n\nImport all the required libraries for this notebook.","metadata":{}},{"cell_type":"code","source":"import albumentations as A\nimport gc\nimport matplotlib.pyplot as plt\nimport math\nimport multiprocessing\nimport numpy as np\nimport os\nimport pandas as pd\nimport random\nimport time\nimport timm\nimport torch\nimport torch.nn as nn\n\n\nfrom albumentations.pytorch import ToTensorV2\nfrom glob import glob\nfrom torch.utils.data import DataLoader, Dataset\nfrom tqdm import tqdm\nfrom typing import Dict, List\n\nos.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0,1\"\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nprint('Using', torch.cuda.device_count(), 'GPU(s)')","metadata":{"execution":{"iopub.status.busy":"2024-02-08T15:16:57.866288Z","iopub.execute_input":"2024-02-08T15:16:57.866844Z","iopub.status.idle":"2024-02-08T15:17:06.642982Z","shell.execute_reply.started":"2024-02-08T15:16:57.866806Z","shell.execute_reply":"2024-02-08T15:17:06.641791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Configuration [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"class config:\n AMP = True\n BATCH_SIZE_TRAIN = 32\n BATCH_SIZE_VALID = 32\n EPOCHS = 4\n FOLDS = 5\n FREEZE = False\n GRADIENT_ACCUMULATION_STEPS = 1\n MAX_GRAD_NORM = 1e7\n MODEL = \"tf_efficientnet_b0\"\n NUM_FROZEN_LAYERS = 39\n NUM_WORKERS = 0 # multiprocessing.cpu_count()\n PRINT_FREQ = 20\n SEED = 20\n TRAIN_FULL_DATA = False\n VISUALIZE = True\n WEIGHT_DECAY = 0.01\n \n \nclass paths:\n OUTPUT_DIR = \"/kaggle/working/\"\n PRE_LOADED_EEGS = '/kaggle/input/brain-eeg-spectrograms/eeg_specs.npy'\n PRE_LOADED_SPECTOGRAMS = '/kaggle/input/brain-spectrograms/specs.npy'\n TRAIN_CSV = \"/kaggle/input/hms-harmful-brain-activity-classification/train.csv\"\n TRAIN_EEGS = \"/kaggle/input/brain-eeg-spectrograms/EEG_Spectrograms/\"\n TRAIN_SPECTOGRAMS = \"/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/\"","metadata":{"papermill":{"duration":0.016556,"end_time":"2024-01-14T22:51:51.671783","exception":false,"start_time":"2024-01-14T22:51:51.655227","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-08T15:17:06.645613Z","iopub.execute_input":"2024-02-08T15:17:06.645976Z","iopub.status.idle":"2024-02-08T15:17:06.654408Z","shell.execute_reply.started":"2024-02-08T15:17:06.645946Z","shell.execute_reply":"2024-02-08T15:17:06.653238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Utils [↑](#top) \n\n***\n\nUtility functions.","metadata":{}},{"cell_type":"code","source":"class AverageMeter(object):\n \"\"\"Computes and stores the average and current value\"\"\"\n def __init__(self):\n self.reset()\n\n def reset(self):\n self.val = 0\n self.avg = 0\n self.sum = 0\n self.count = 0\n\n def update(self, val, n=1):\n self.val = val\n self.sum += val * n\n self.count += n\n self.avg = self.sum / self.count\n\n\ndef asMinutes(s: float):\n \"Convert to minutes.\"\n m = math.floor(s / 60)\n s -= m * 60\n return '%dm %ds' % (m, s)\n\n\ndef timeSince(since: float, percent: float):\n now = time.time()\n s = now - since\n es = s / (percent)\n rs = es - s\n return '%s (remain %s)' % (asMinutes(s), asMinutes(rs))\n\n\ndef get_logger(filename=paths.OUTPUT_DIR):\n from logging import getLogger, INFO, StreamHandler, FileHandler, Formatter\n logger = getLogger(__name__)\n logger.setLevel(INFO)\n handler1 = StreamHandler()\n handler1.setFormatter(Formatter(\"%(message)s\"))\n handler2 = FileHandler(filename=f\"{filename}.log\")\n handler2.setFormatter(Formatter(\"%(message)s\"))\n logger.addHandler(handler1)\n logger.addHandler(handler2)\n return logger\n\n\ndef plot_spectrogram(spectrogram_path: str):\n \"\"\"\n Source: https://www.kaggle.com/code/mvvppp/hms-eda-and-domain-journey\n Visualize spectogram recordings from a parquet file.\n :param spectrogram_path: path to the spectogram parquet.\n \"\"\"\n sample_spect = pd.read_parquet(spectrogram_path)\n \n split_spect = {\n \"LL\": sample_spect.filter(regex='^LL', axis=1),\n \"RL\": sample_spect.filter(regex='^RL', axis=1),\n \"RP\": sample_spect.filter(regex='^RP', axis=1),\n \"LP\": sample_spect.filter(regex='^LP', axis=1),\n }\n \n fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(15, 12))\n axes = axes.flatten()\n label_interval = 5\n for i, split_name in enumerate(split_spect.keys()):\n ax = axes[i]\n img = ax.imshow(np.log(split_spect[split_name]).T, cmap='viridis', aspect='auto', origin='lower')\n cbar = fig.colorbar(img, ax=ax)\n cbar.set_label('Log(Value)')\n ax.set_title(split_name)\n ax.set_ylabel(\"Frequency (Hz)\")\n ax.set_xlabel(\"Time\")\n\n ax.set_yticks(np.arange(len(split_spect[split_name].columns)))\n ax.set_yticklabels([column_name[3:] for column_name in split_spect[split_name].columns])\n frequencies = [column_name[3:] for column_name in split_spect[split_name].columns]\n ax.set_yticks(np.arange(0, len(split_spect[split_name].columns), label_interval))\n ax.set_yticklabels(frequencies[::label_interval])\n plt.tight_layout()\n plt.show()\n \n \ndef seed_everything(seed: int):\n random.seed(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed) \n\n \ndef sep():\n print(\"-\"*100)\n \n\ntarget_preds = [x + \"_pred\" for x in ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']]\nlabel_to_num = {'Seizure': 0, 'LPD': 1, 'GPD': 2, 'LRDA': 3, 'GRDA': 4, 'Other':5}\nnum_to_label = {v: k for k, v in label_to_num.items()}\nLOGGER = get_logger()\nseed_everything(config.SEED)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-02-08T15:17:06.656262Z","iopub.execute_input":"2024-02-08T15:17:06.657015Z","iopub.status.idle":"2024-02-08T15:17:06.688262Z","shell.execute_reply.started":"2024-02-08T15:17:06.656982Z","shell.execute_reply":"2024-02-08T15:17:06.686646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Load Data [↑](#top) \n\n***\n\nLoad the competition's data.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(paths.TRAIN_CSV)\nlabel_cols = df.columns[-6:]\nprint(f\"Train cataframe shape is: {df.shape}\")\nprint(f\"Labels: {list(label_cols)}\")\ndf.head()","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.288611,"end_time":"2024-01-14T22:51:51.984993","exception":false,"start_time":"2024-01-14T22:51:51.696382","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-02-08T15:17:06.690003Z","iopub.execute_input":"2024-02-08T15:17:06.690515Z","iopub.status.idle":"2024-02-08T15:17:07.029677Z","shell.execute_reply.started":"2024-02-08T15:17:06.690470Z","shell.execute_reply":"2024-02-08T15:17:07.028638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Data pre-processing [↑](#top) \n\n***\n\n### Create Non-Overlapping Eeg Id Train Data\n\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{}},{"cell_type":"code","source":"aux = df.groupby('eeg_id')[['expert_consensus']].agg('first')\naux","metadata":{"execution":{"iopub.status.busy":"2024-02-08T15:22:23.159789Z","iopub.execute_input":"2024-02-08T15:22:23.160196Z","iopub.status.idle":"2024-02-08T15:22:23.196256Z","shell.execute_reply.started":"2024-02-08T15:22:23.160164Z","shell.execute_reply":"2024-02-08T15:22:23.195017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg({\n 'spectrogram_id':'first',\n 'spectrogram_label_offset_seconds':'min'\n})\ntrain_df.columns = ['spectogram_id','min']\n\naux = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg({\n 'spectrogram_label_offset_seconds':'max'\n})\ntrain_df['max'] = aux\n\naux = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain_df['patient_id'] = aux\n\naux = df.groupby('eeg_id')[label_cols].agg('sum')\nfor label in label_cols:\n train_df[label] = aux[label].values\n \ny_data = train_df[label_cols].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain_df[label_cols] = y_data\n\naux = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain_df['target'] = aux\n\ntrain_df = train_df.reset_index()\nprint('Train non-overlapp eeg_id shape:', train_df.shape )\ntrain_df.head()","metadata":{"papermill":{"duration":0.111621,"end_time":"2024-01-14T22:51:52.125134","exception":false,"start_time":"2024-01-14T22:51:52.013513","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-30T19:09:07.660802Z","iopub.execute_input":"2024-01-30T19:09:07.661627Z","iopub.status.idle":"2024-01-30T19:09:07.753959Z","shell.execute_reply.started":"2024-01-30T19:09:07.661596Z","shell.execute_reply":"2024-01-30T19:09:07.753139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Read Train Spectrograms\n\n\nFirst we need to read in all 11k train spectrogram files. Reading thousands of files takes 11 minutes with Pandas. Instead, we can read 1 file from my [Kaggle dataset here][1] which contains all the 11k spectrograms in less than 1 minute! To use my Kaggle dataset, set variable `READ_SPEC_FILES = False`. Thank you for upvoting my helpful [dataset][1] :-)\n\nThe resulting `all_spectograms` dictionary contains `spectogram_id` as keys (`int` keys) and the values are the spectogram sequences (as 2-dimensional `np.array`) of shape `(timesteps, 400)`.\n\nEach spectogram is a parquet file. This parquet, when converted to a pandas dataframe, results in a dataframe of shape `(time_steps, 401)`. First column is the `time` column and the remaining 400 columns are the recordings. There are 400 columns because there are, respectively, 100 rows associated to the 4 recording regions of the EEG electrodes: `LL`, `RL`, `LP`, `RP`. Column names also include the frequency in heartz.\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms","metadata":{"papermill":{"duration":0.00881,"end_time":"2024-01-14T22:51:52.142747","exception":false,"start_time":"2024-01-14T22:51:52.133937","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nREAD_SPEC_FILES = False\n\npaths_spectograms = glob(paths.TRAIN_SPECTOGRAMS + \"*.parquet\")\nprint(f'There are {len(paths_spectograms)} spectrogram parquets')\n\nif READ_SPEC_FILES: \n all_spectrograms = {}\n for file_path in tqdm(paths_spectograms):\n aux = pd.read_parquet(file_path)\n name = int(file_path.split(\"/\")[-1].split('.')[0])\n all_spectrograms[name] = aux.iloc[:,1:].values\n del aux\nelse:\n all_spectrograms = np.load(paths.PRE_LOADED_SPECTOGRAMS, allow_pickle=True).item()\n \nif config.VISUALIZE:\n idx = np.random.randint(0,len(paths_spectograms))\n spectrogram_path = paths_spectograms[idx]\n plot_spectrogram(spectrogram_path)","metadata":{"papermill":{"duration":55.16894,"end_time":"2024-01-14T22:52:47.320438","exception":false,"start_time":"2024-01-14T22:51:52.151498","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-30T19:09:08.520015Z","iopub.execute_input":"2024-01-30T19:09:08.520809Z","iopub.status.idle":"2024-01-30T19:10:00.147510Z","shell.execute_reply.started":"2024-01-30T19:09:08.520774Z","shell.execute_reply":"2024-01-30T19:10:00.146157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Read EEG Spectrograms\n\nThe resulting `all_eegs` dictionary contains `eeg_id` as keys (`int` keys) and the values are the eeg sequences (as 3-dimensional `np.array`) of shape `(128, 256, 4)`.\n\n","metadata":{}},{"cell_type":"code","source":"%%time\nREAD_EEG_SPEC_FILES = False\n\npaths_eegs = glob(paths.TRAIN_EEGS + \"*.npy\")\nprint(f'There are {len(paths_eegs)} EEG spectograms')\n\nif READ_EEG_SPEC_FILES:\n all_eegs = {}\n for file_path in tqdm(paths_eegs):\n eeg_id = file_path.split(\"/\")[-1].split(\".\")[0]\n eeg_spectogram = np.load(file_path)\n all_eegs[eeg_id] = eeg_spectogram\nelse:\n all_eegs = np.load(paths.PRE_LOADED_EEGS, allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-01-30T19:10:00.149293Z","iopub.execute_input":"2024-01-30T19:10:00.149574Z","iopub.status.idle":"2024-01-30T19:10:57.385552Z","shell.execute_reply.started":"2024-01-30T19:10:00.149549Z","shell.execute_reply":"2024-01-30T19:10:57.384509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Validation [↑](#top) \n\n***\n\nWe train using `GroupKFold` on `patient_id`.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold\n\n\ngkf = GroupKFold(n_splits=config.FOLDS)\nfor fold, (train_index, valid_index) in enumerate(gkf.split(train_df, train_df.target, train_df.patient_id)):\n train_df.loc[valid_index, \"fold\"] = int(fold)\n \ndisplay(train_df.groupby('fold').size()), sep()\ndisplay(train_df.head())","metadata":{"execution":{"iopub.status.busy":"2024-01-30T19:10:57.387170Z","iopub.execute_input":"2024-01-30T19:10:57.387493Z","iopub.status.idle":"2024-01-30T19:10:57.439059Z","shell.execute_reply.started":"2024-01-30T19:10:57.387466Z","shell.execute_reply":"2024-01-30T19:10:57.438031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Dataset [↑](#top) \n\n***\n\nCreate a custom `Dataset` to load data.\n\nOur dataloader outputs both Kaggle spectrograms and EEG spectrogams as 8 channel image of size `(128, 256, 8)`\n\n[1]: https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43/comments#2617811","metadata":{}},{"cell_type":"code","source":"class CustomDataset(Dataset):\n def __init__(\n self, df: pd.DataFrame, config,\n augment: bool = False, mode: str = 'train',\n specs: Dict[int, np.ndarray] = all_spectrograms,\n eeg_specs: Dict[int, np.ndarray] = all_eegs\n ): \n self.df = df\n self.config = config\n self.batch_size = self.config.BATCH_SIZE_TRAIN\n self.augment = augment\n self.mode = mode\n self.spectograms = all_spectrograms\n self.eeg_spectograms = eeg_specs\n \n def __len__(self):\n \"\"\"\n Denotes the number of batches per epoch.\n \"\"\"\n return len(self.df)\n \n def __getitem__(self, index):\n \"\"\"\n Generate one batch of data.\n \"\"\"\n X, y = self.__data_generation(index)\n if self.augment:\n X = self.__transform(X) \n return torch.tensor(X, dtype=torch.float32), torch.tensor(y, dtype=torch.float32)\n \n def __data_generation(self, index):\n \"\"\"\n Generates data containing batch_size samples.\n \"\"\"\n X = np.zeros((128, 256, 8), dtype='float32')\n y = np.zeros(6, dtype='float32')\n img = np.ones((128,256), dtype='float32')\n row = self.df.iloc[index]\n if self.mode=='test': \n r = 0\n else: \n r = int((row['min'] + row['max']) // 4)\n \n for region in range(4):\n img = self.spectograms[row.spectogram_id][r:r+300, region*100:(region+1)*100].T\n \n # Log transform spectogram\n img = np.clip(img, np.exp(-4), np.exp(8))\n img = np.log(img)\n\n # Standarize per image\n ep = 1e-6\n mu = np.nanmean(img.flatten())\n std = np.nanstd(img.flatten())\n img = (img-mu)/(std+ep)\n img = np.nan_to_num(img, nan=0.0)\n X[14:-14, :, region] = img[:, 22:-22] / 2.0\n img = self.eeg_spectograms[row.eeg_id]\n X[:, :, 4:] = img\n \n if self.mode != 'test':\n y = row[label_cols].values.astype(np.float32)\n \n return X, y\n \n def __transform(self, img):\n transforms = A.Compose([\n A.HorizontalFlip(p=0.5),\n ])\n return transforms(image=img)['image']","metadata":{"execution":{"iopub.status.busy":"2024-01-30T19:10:57.441819Z","iopub.execute_input":"2024-01-30T19:10:57.442277Z","iopub.status.idle":"2024-01-30T19:10:57.458569Z","shell.execute_reply.started":"2024-01-30T19:10:57.442219Z","shell.execute_reply":"2024-01-30T19:10:57.457527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | DataLoader [↑](#top) \n\n***\n\nBelow we display example dataloader spectrogram images.","metadata":{}},{"cell_type":"code","source":"train_dataset = CustomDataset(train_df, config, mode=\"train\")\ntrain_loader = DataLoader(\n train_dataset,\n batch_size=config.BATCH_SIZE_TRAIN,\n shuffle=False,\n num_workers=config.NUM_WORKERS, pin_memory=True, drop_last=True\n)\nX, y = train_dataset[0]\nprint(f\"X shape: {X.shape}\")\nprint(f\"y shape: {y.shape}\")","metadata":{"execution":{"iopub.status.busy":"2024-01-30T19:10:57.459847Z","iopub.execute_input":"2024-01-30T19:10:57.460159Z","iopub.status.idle":"2024-01-30T19:10:57.511931Z","shell.execute_reply.started":"2024-01-30T19:10:57.460135Z","shell.execute_reply":"2024-01-30T19:10:57.510933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize DataLoader\n","metadata":{}},{"cell_type":"code","source":"if config.VISUALIZE:\n ROWS = 2\n COLS = 3\n for (X, y) in train_loader:\n plt.figure(figsize=(20,8))\n for row in range(ROWS):\n for col in range(COLS):\n plt.subplot(ROWS, COLS, row*COLS + col+1)\n t = y[row*COLS + col]\n img = X[row*COLS + col, :, :, 0]\n mn = img.flatten().min()\n mx = img.flatten().max()\n img = (img-mn)/(mx-mn)\n plt.imshow(img)\n tars = f'[{t[0]:0.2f}'\n for s in t[1:]:\n tars += f', {s:0.2f}'\n eeg = train_df.eeg_id.values[row*config.BATCH_SIZE_TRAIN + row*COLS + col]\n plt.title(f'EEG = {eeg}\\nTarget = {tars}',size=12)\n plt.yticks([])\n plt.ylabel('Frequencies (Hz)',size=14)\n plt.xlabel('Time (sec)',size=16)\n plt.show()\n break","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-01-30T19:10:57.513220Z","iopub.execute_input":"2024-01-30T19:10:57.513534Z","iopub.status.idle":"2024-01-30T19:10:59.286952Z","shell.execute_reply.started":"2024-01-30T19:10:57.513507Z","shell.execute_reply":"2024-01-30T19:10:59.285723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Model [↑](#top) \n\n***\n\nWe will be using the [timm](https://github.com/huggingface/pytorch-image-models) library for our models.\n\nOur models receives both Kaggle spectrograms and EEG spectrograms from our data loader. We then reshape these 8 spectrograms into 1 large flat image and feed it into EfficientNet.","metadata":{}},{"cell_type":"code","source":"class CustomModel(nn.Module):\n def __init__(self, config, num_classes: int = 6, pretrained: bool = True):\n super(CustomModel, self).__init__()\n self.USE_KAGGLE_SPECTROGRAMS = True\n self.USE_EEG_SPECTROGRAMS = True\n self.model = timm.create_model(\n config.MODEL,\n pretrained=pretrained,\n drop_rate = 0.1,\n drop_path_rate = 0.2,\n )\n if config.FREEZE:\n for i,(name, param) in enumerate(list(self.model.named_parameters())\\\n [0:config.NUM_FROZEN_LAYERS]):\n param.requires_grad = False\n\n self.features = nn.Sequential(*list(self.model.children())[:-2])\n self.custom_layers = nn.Sequential(\n nn.AdaptiveAvgPool2d(1),\n nn.Flatten(),\n nn.Linear(self.model.num_features, num_classes)\n )\n\n def __reshape_input(self, x):\n \"\"\"\n Reshapes input (128, 256, 8) -> (512, 512, 3) monotone image.\n \"\"\" \n # === Get spectograms ===\n spectograms = [x[:, :, :, i:i+1] for i in range(4)]\n spectograms = torch.cat(spectograms, dim=1)\n \n # === Get EEG spectograms ===\n eegs = [x[:, :, :, i:i+1] for i in range(4,8)]\n eegs = torch.cat(eegs, dim=1)\n \n # === Reshape (512,512,3) ===\n if self.USE_KAGGLE_SPECTROGRAMS & self.USE_EEG_SPECTROGRAMS:\n x = torch.cat([spectograms, eegs], dim=2)\n elif self.USE_EEG_SPECTROGRAMS:\n x = eegs\n else:\n x = spectograms\n \n x = torch.cat([x,x,x], dim=3)\n x = x.permute(0, 3, 1, 2)\n return x\n \n def forward(self, x):\n x = self.__reshape_input(x)\n x = self.features(x)\n x = self.custom_layers(x)\n return x","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-01-30T19:10:59.289207Z","iopub.execute_input":"2024-01-30T19:10:59.289560Z","iopub.status.idle":"2024-01-30T19:10:59.302892Z","shell.execute_reply.started":"2024-01-30T19:10:59.289532Z","shell.execute_reply":"2024-01-30T19:10:59.301868Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Scheduler [↑](#top) \n\n***\n\nWe will train our model with a Step Train Schedule for 4 epochs. First 2 epochs are LR=1e-3. Then epochs 3 and 4 use LR=1e-4 and 1e-5 respectively. (Below we also provide a Cosine Train Schedule if you want to experiment with it. Note it is not used in this notebook).","metadata":{}},{"cell_type":"code","source":"from torch.optim.lr_scheduler import OneCycleLR\n\nEPOCHS = config.EPOCHS\nBATCHES = len(train_loader)\nsteps = []\nlrs = []\noptim_lrs = []\nmodel = CustomModel(config)\noptimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)\nscheduler = OneCycleLR(\n optimizer,\n max_lr=1e-3,\n epochs=config.EPOCHS,\n steps_per_epoch=len(train_loader),\n pct_start=0.05,\n anneal_strategy=\"cos\",\n final_div_factor=100,\n)\nfor epoch in range(EPOCHS):\n for batch in range(BATCHES):\n scheduler.step()\n lrs.append(scheduler.get_last_lr()[0])\n steps.append(epoch * BATCHES + batch)\n\nmax_lr = max(lrs)\nmin_lr = min(lrs)\nprint(f\"Maximum LR: {max_lr} | Minimum LR: {min_lr}\")\nplt.figure()\nplt.plot(steps, lrs, label='OneCycle')\nplt.ticklabel_format(axis='y', style='sci', scilimits=(0,0))\nplt.xlabel(\"Step\")\nplt.ylabel(\"Learning Rate\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-01-30T19:10:59.304099Z","iopub.execute_input":"2024-01-30T19:10:59.304412Z","iopub.status.idle":"2024-01-30T19:11:05.531483Z","shell.execute_reply.started":"2024-01-30T19:10:59.304386Z","shell.execute_reply":"2024-01-30T19:11:05.530512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Loss Function [↑](#top) \n\n***\n\nIn PyTorch's [KLDivLoss][1], the reduction parameter determines how the loss is aggregated across different dimensions. Two common options are `mean` and `batchmean`.\n\n- `reduction`='mean': When reduction is set to \"mean\", the Kullback-Leibler Divergence loss is computed and then averaged over all the elements in the input tensor. The result is a scalar value representing the mean loss.\n- `reduction`='batchmean': When reduction is set to \"batchmean\", the Kullback-Leibler Divergence loss is computed independently for each item in the batch, and then the mean is taken over the batch dimension. This is useful when you have a batch of samples, and you want the average loss per sample.\n\n[1]: https://pytorch.org/docs/stable/generated/torch.nn.KLDivLoss.html\n","metadata":{}},{"cell_type":"code","source":"import torch.nn.functional as F\n\n# === Reduction = \"mean\" ===\ncriterion = nn.KLDivLoss(reduction=\"mean\")\ny_pred = F.log_softmax(torch.randn(6, 2, requires_grad=True), dim=1)\ny_true = F.softmax(torch.rand(6, 2), dim=1)\nprint(f\"Predictions: {y_pred}\")\nprint(f\"Targets: {y_true}\")\noutput = criterion(y_pred, y_true)\nprint(f\"Output: {output}\")\n\nprint(\"\\n\", \"=\"*100, \"\\n\")\n\n# === Reduction = \"batchmean\" ===\ncriterion = nn.KLDivLoss(reduction=\"batchmean\")\ny_pred = F.log_softmax(torch.randn(2, 6, requires_grad=True), dim=1)\ny_true = F.softmax(torch.rand(2, 6), dim=1)\nprint(f\"Predictions: {y_pred}\")\nprint(f\"Targets: {y_true}\")\noutput = criterion(y_pred, y_true)\nprint(f\"Output: {output}\")","metadata":{"execution":{"iopub.status.busy":"2024-01-30T19:11:05.534771Z","iopub.execute_input":"2024-01-30T19:11:05.535464Z","iopub.status.idle":"2024-01-30T19:11:05.575150Z","shell.execute_reply.started":"2024-01-30T19:11:05.535427Z","shell.execute_reply":"2024-01-30T19:11:05.574238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Train and Validation Functions [↑](#top) \n\n***\n\nWe train using Group KFold on patient id. If `LOAD_MODELS_FROM = None`, then we will train new models in this notebook version. Otherwise we will load saved models from the path `LOAD_MODELS_FROM`.","metadata":{"papermill":{"duration":0.033717,"end_time":"2024-01-14T22:53:06.742557","exception":false,"start_time":"2024-01-14T22:53:06.70884","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def train_epoch(train_loader, model, criterion, optimizer, epoch, scheduler, device):\n \"\"\"One epoch training pass.\"\"\"\n model.train() \n criterion = nn.KLDivLoss(reduction=\"batchmean\")\n scaler = torch.cuda.amp.GradScaler(enabled=config.AMP)\n losses = AverageMeter()\n start = end = time.time()\n global_step = 0\n \n # ========== ITERATE OVER TRAIN BATCHES ============\n with tqdm(train_loader, unit=\"train_batch\", desc='Train') as tqdm_train_loader:\n for step, (X, y) in enumerate(tqdm_train_loader):\n X = X.to(device)\n y = y.to(device)\n batch_size = y.size(0)\n with torch.cuda.amp.autocast(enabled=config.AMP):\n y_preds = model(X) \n loss = criterion(F.log_softmax(y_preds, dim=1), y)\n if config.GRADIENT_ACCUMULATION_STEPS > 1:\n loss = loss / config.GRADIENT_ACCUMULATION_STEPS\n losses.update(loss.item(), batch_size)\n scaler.scale(loss).backward()\n grad_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), config.MAX_GRAD_NORM)\n\n if (step + 1) % config.GRADIENT_ACCUMULATION_STEPS == 0:\n scaler.step(optimizer)\n scaler.update()\n optimizer.zero_grad()\n global_step += 1\n scheduler.step()\n end = time.time()\n\n # ========== LOG INFO ==========\n if step % config.PRINT_FREQ == 0 or step == (len(train_loader)-1):\n print('Epoch: [{0}][{1}/{2}] '\n 'Elapsed {remain:s} '\n 'Loss: {loss.avg:.4f} '\n 'Grad: {grad_norm:.4f} '\n 'LR: {lr:.8f} '\n .format(epoch+1, step, len(train_loader), \n remain=timeSince(start, float(step+1)/len(train_loader)),\n loss=losses,\n grad_norm=grad_norm,\n lr=scheduler.get_last_lr()[0]))\n\n return losses.avg\n\n\ndef valid_epoch(valid_loader, model, criterion, device):\n model.eval()\n softmax = nn.Softmax(dim=1)\n losses = AverageMeter()\n prediction_dict = {}\n preds = []\n start = end = time.time()\n with tqdm(valid_loader, unit=\"valid_batch\", desc='Validation') as tqdm_valid_loader:\n for step, (X, y) in enumerate(tqdm_valid_loader):\n X = X.to(device)\n y = y.to(device)\n batch_size = y.size(0)\n with torch.no_grad():\n y_preds = model(X)\n loss = criterion(F.log_softmax(y_preds, dim=1), y)\n if config.GRADIENT_ACCUMULATION_STEPS > 1:\n loss = loss / config.GRADIENT_ACCUMULATION_STEPS\n losses.update(loss.item(), batch_size)\n y_preds = softmax(y_preds)\n preds.append(y_preds.to('cpu').numpy())\n end = time.time()\n\n # ========== LOG INFO ==========\n if step % config.PRINT_FREQ == 0 or step == (len(valid_loader)-1):\n print('EVAL: [{0}/{1}] '\n 'Elapsed {remain:s} '\n 'Loss: {loss.avg:.4f} '\n .format(step, len(valid_loader),\n remain=timeSince(start, float(step+1)/len(valid_loader)),\n loss=losses))\n \n prediction_dict[\"predictions\"] = np.concatenate(preds)\n return losses.avg, prediction_dict","metadata":{"execution":{"iopub.status.busy":"2024-01-30T19:11:05.576662Z","iopub.execute_input":"2024-01-30T19:11:05.577029Z","iopub.status.idle":"2024-01-30T19:11:05.594971Z","shell.execute_reply.started":"2024-01-30T19:11:05.576997Z","shell.execute_reply":"2024-01-30T19:11:05.594047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Train Loop [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"def train_loop(df, fold):\n \n LOGGER.info(f\"========== Fold: {fold} training ==========\")\n\n # ======== SPLIT ==========\n train_folds = df[df['fold'] != fold].reset_index(drop=True)\n valid_folds = df[df['fold'] == fold].reset_index(drop=True)\n \n # ======== DATASETS ==========\n train_dataset = CustomDataset(train_folds, config, mode=\"train\", augment=True)\n valid_dataset = CustomDataset(valid_folds, config, mode=\"train\", augment=False)\n \n # ======== DATALOADERS ==========\n train_loader = DataLoader(train_dataset,\n batch_size=config.BATCH_SIZE_TRAIN,\n shuffle=False,\n num_workers=config.NUM_WORKERS, pin_memory=True, drop_last=True)\n valid_loader = DataLoader(valid_dataset,\n batch_size=config.BATCH_SIZE_VALID,\n shuffle=False,\n num_workers=config.NUM_WORKERS, pin_memory=True, drop_last=False)\n \n # ======== MODEL ==========\n model = CustomModel(config)\n model.to(device)\n\n optimizer = torch.optim.AdamW(model.parameters(), lr=0.1, weight_decay=config.WEIGHT_DECAY)\n scheduler = OneCycleLR(\n optimizer,\n max_lr=1e-3,\n epochs=config.EPOCHS,\n steps_per_epoch=len(train_loader),\n pct_start=0.1,\n anneal_strategy=\"cos\",\n final_div_factor=100,\n )\n\n # ======= LOSS ==========\n criterion = nn.KLDivLoss(reduction=\"batchmean\")\n \n best_loss = np.inf\n # ====== ITERATE EPOCHS ========\n for epoch in range(config.EPOCHS):\n start_time = time.time()\n\n # ======= TRAIN ==========\n avg_train_loss = train_epoch(train_loader, model, criterion, optimizer, epoch, scheduler, device)\n\n # ======= EVALUATION ==========\n avg_val_loss, prediction_dict = valid_epoch(valid_loader, model, criterion, device)\n predictions = prediction_dict[\"predictions\"]\n \n # ======= SCORING ==========\n elapsed = time.time() - start_time\n\n LOGGER.info(f'Epoch {epoch+1} - avg_train_loss: {avg_train_loss:.4f} avg_val_loss: {avg_val_loss:.4f} time: {elapsed:.0f}s')\n \n if avg_val_loss < best_loss:\n best_loss = avg_val_loss\n LOGGER.info(f'Epoch {epoch+1} - Save Best Loss: {best_loss:.4f} Model')\n torch.save({'model': model.state_dict(),\n 'predictions': predictions},\n paths.OUTPUT_DIR + f\"/{config.MODEL.replace('/', '_')}_fold_{fold}_best.pth\")\n\n predictions = torch.load(paths.OUTPUT_DIR + f\"/{config.MODEL.replace('/', '_')}_fold_{fold}_best.pth\", \n map_location=torch.device('cpu'))['predictions']\n valid_folds[target_preds] = predictions\n\n torch.cuda.empty_cache()\n gc.collect()\n \n return valid_folds","metadata":{"execution":{"iopub.status.busy":"2024-01-30T19:11:05.596585Z","iopub.execute_input":"2024-01-30T19:11:05.596962Z","iopub.status.idle":"2024-01-30T19:11:05.611408Z","shell.execute_reply.started":"2024-01-30T19:11:05.596928Z","shell.execute_reply":"2024-01-30T19:11:05.610578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Train Full Data [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"def train_loop_full_data(df):\n train_dataset = CustomDataset(df, config, mode=\"train\", augment=True)\n train_loader = DataLoader(train_dataset,\n batch_size=config.BATCH_SIZE_TRAIN,\n shuffle=False,\n num_workers=config.NUM_WORKERS, pin_memory=True, drop_last=True)\n model = CustomModel(config)\n model.to(device)\n optimizer = torch.optim.AdamW(model.parameters(), lr=0.1, weight_decay=config.WEIGHT_DECAY)\n scheduler = OneCycleLR(\n optimizer,\n max_lr=1e-3,\n epochs=config.EPOCHS,\n steps_per_epoch=len(train_loader),\n pct_start=0.1,\n anneal_strategy=\"cos\",\n final_div_factor=100,\n )\n criterion = nn.KLDivLoss(reduction=\"batchmean\")\n best_loss = np.inf\n for epoch in range(config.EPOCHS):\n start_time = time.time()\n avg_train_loss = train_epoch(train_loader, model, criterion, optimizer, epoch, scheduler, device)\n elapsed = time.time() - start_time\n LOGGER.info(f'Epoch {epoch+1} - avg_train_loss: {avg_train_loss:.4f} time: {elapsed:.0f}s')\n torch.save(\n {'model': model.state_dict()},\n paths.OUTPUT_DIR + f\"/{config.MODEL.replace('/', '_')}_epoch_{epoch}.pth\")\n torch.cuda.empty_cache()\n gc.collect()\n return _","metadata":{"execution":{"iopub.status.busy":"2024-01-30T19:11:05.612555Z","iopub.execute_input":"2024-01-30T19:11:05.612846Z","iopub.status.idle":"2024-01-30T19:11:05.625577Z","shell.execute_reply.started":"2024-01-30T19:11:05.612821Z","shell.execute_reply":"2024-01-30T19:11:05.624710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# | Train [↑](#top) \n\n***","metadata":{}},{"cell_type":"code","source":"def get_result(oof_df):\n kl_loss = nn.KLDivLoss(reduction=\"batchmean\")\n labels = torch.tensor(oof_df[label_cols].values)\n preds = torch.tensor(oof_df[target_preds].values)\n preds = F.log_softmax(preds, dim=1)\n result = kl_loss(preds, labels)\n return result\n\nif not config.TRAIN_FULL_DATA:\n oof_df = pd.DataFrame()\n for fold in range(config.FOLDS):\n if fold in [0, 1, 2, 3, 4]:\n _oof_df = train_loop(train_df, fold)\n oof_df = pd.concat([oof_df, _oof_df])\n LOGGER.info(f\"========== Fold {fold} result: {get_result(_oof_df)} ==========\")\n print(f\"========== Fold {fold} result: {get_result(_oof_df)} ==========\")\n oof_df = oof_df.reset_index(drop=True)\n LOGGER.info(f\"========== CV: {get_result(oof_df)} ==========\")\n oof_df.to_csv(paths.OUTPUT_DIR + '/oof_df.csv', index=False)\nelse:\n train_loop_full_data(train_df)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}