{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7402356,"sourceType":"datasetVersion","datasetId":4304475},{"sourceId":7403069,"sourceType":"datasetVersion","datasetId":4304949},{"sourceId":7447509,"sourceType":"datasetVersion","datasetId":4334995},{"sourceId":158958765,"sourceType":"kernelVersion"},{"sourceId":390463,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":321603,"modelId":342209},{"sourceId":390685,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":321765,"modelId":342375},{"sourceId":390917,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":321934,"modelId":342553}],"dockerImageVersionId":30636,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"papermill":{"default_parameters":{},"duration":270.012179,"end_time":"2024-01-14T22:56:02.916427","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-01-14T22:51:32.904248","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EfficientNetB2 Starter for Brain Comp\nThis is an EfficientNetB2 starter notebook for Kaggle's brain comp. We use only spectrogram features. (The model does not use eeg features yet). This EfficientNet Starter achieves CV 0.72 and LB 0.56! This is +0.10 better than my CatBoost starter. My CatBoost starter is [here][1] (with discussion [here][3]) and achieves CV 0.82 and LB 0.67. \n\nBoth my EfficientNetB2 starter and CatBoost starter use my Kaggle dataset [here][2]. This dataset is a single file which contains all of Kaggle's 11,138 spectrogram parquets. Reading this single file is much faster than reading 11k separate files. Thank you everyonne for upvoting my Kaggle [dataset][2]!\n\n# Exciting UPDATE!\nVersion 5 of this notebook trains EfficientNet using **both** Kaggle spectrograms and my new **EEG spectrograms** from my Kaggle dataset [here][4] (which were created from my spectrogram starter [here][5]). Version 5 also uses KL Div loss (suggested by Awsaf in comments) instead of Cross Entropy loss. We also use EfficientNetB0 to save time versus EfficientNetB2 and train for 4 epochs instead of 3. Lastly we add horizontal flip data augmentation.\n* Version 1-2: **CV 0.73 LB 0.57** Kaggle spec only, CE loss\n* Version 3-4: EEG spec only **CV 0.63**, Kaggle spec only **CV 0.65**, KL-Div loss\n* Version 5-6: **CV 0.59 LB 0.44** Both spec, KL-Div loss\n\n[1]: https://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-67\n[2]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\n[3]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467576\n[4]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms\n[5]: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg","metadata":{"papermill":{"duration":0.008094,"end_time":"2024-01-14T22:51:36.811998","exception":false,"start_time":"2024-01-14T22:51:36.803904","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Kaggle Spectrograms vs. EEG Spectrograms\nIn this competition, Kaggle provides us with both spectrograms and raw eeg waveforms. The Kaggle spectrograms are 10 minutes long and the eeg waveforms are 50 seconds. The middle 50 seconds of both data are the same time window observing the same event. (i.e. the middle 50 seconds is same information displayed two different ways). Spectrograms are just visual representations of raw waveforms.\n\nI published a spectrogram starter notebook [here][1] that converts the raw eeg waveform into a spectrogram. The eeg spectrograms (from version 4) have been uploaded to my Kaggle dataset [here][2]. Version 4 (of my spectrogram starter notebook) onward uses a new powerful formula. The below table shows that the new created eeg spectrograms are equal or better than the Kaggle spectroms:\n\n| Spectrogram | EffNet 5Fold CV | EffNet LB | Notebook version | \n| --- | --- | --- | --- |\n| EEG Spectrograms | 0.63 | ??? | ver 3 |\n| Kaggle Spectrograms | 0.66 | ??? | ver 4 |\n| Both Spectrograms | 0.59 | 0.44 | ver 5|\n\nIn different versions of this notebook, we will train three EfficientNet models. Version 3 is using only EEG spectrograms. Version 4 is using only Kaggle spectrograms. And version 5 is using both EEG spectrograms and Kaggle spectrograms. Then we save the model weights into a Kaggle dataset and we make a submit notebook in version 6 for the version 5 model trained on both spectrograms. We achieve **CV 0.59 LB 0.44**! Wow!\n\n[1]: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg\n[2]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms","metadata":{}},{"cell_type":"markdown","source":"# Initialize 2xT4 GPUs\nWe will use both Kaggle T4 GPUs and we will use mixed precision.","metadata":{"papermill":{"duration":0.008572,"end_time":"2024-01-14T22:51:36.82846","exception":false,"start_time":"2024-01-14T22:51:36.819888","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os, gc\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport tensorflow as tf\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import confusion_matrix, classification_report\nimport seaborn as sns\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom sklearn.utils import resample\nprint('TensorFlow version =',tf.__version__)\n\n# USE MULTIPLE GPUS\ngpus = tf.config.list_physical_devices('GPU')\nif len(gpus)<=1: \n strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n print(f'Using {len(gpus)} GPU')\nelse: \n strategy = tf.distribute.MirroredStrategy()\n print(f'Using {len(gpus)} GPUs')\n\nVER = 3.3\n\n# IF THIS EQUALS NONE, THEN WE TRAIN NEW MODELS\n# IF THIS EQUALS DISK PATH, THEN WE LOAD PREVIOUSLY TRAINED MODELS\nLOAD_MODELS_FROM = None #'/kaggle/input/effnet_strat_freqshift_all/tensorflow2/default/1/' #'/kaggle/input/effnet_v3.3_stratified/tensorflow2/default/1/' #'/kaggle/input/effnetb0_all_v3.2/tensorflow2/default/1/'\n\nUSE_KAGGLE_SPECTROGRAMS = True\nUSE_EEG_SPECTROGRAMS = True\nSEED = 42","metadata":{"papermill":{"duration":14.80928,"end_time":"2024-01-14T22:51:51.64702","exception":false,"start_time":"2024-01-14T22:51:36.83774","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:21:15.088019Z","iopub.execute_input":"2024-01-21T19:21:15.088759Z","iopub.status.idle":"2024-01-21T19:21:34.058803Z","shell.execute_reply.started":"2024-01-21T19:21:15.088713Z","shell.execute_reply":"2024-01-21T19:21:34.05778Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"tf.random.set_seed(SEED)\nnp.random.seed(SEED)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# USE MIXED PRECISION\nMIX = True\nif MIX:\n tf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\n print('Mixed precision enabled')\nelse:\n print('Using full precision')","metadata":{"papermill":{"duration":0.016556,"end_time":"2024-01-14T22:51:51.671783","exception":false,"start_time":"2024-01-14T22:51:51.655227","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:21:34.061051Z","iopub.execute_input":"2024-01-21T19:21:34.06174Z","iopub.status.idle":"2024-01-21T19:21:34.067321Z","shell.execute_reply.started":"2024-01-21T19:21:34.06168Z","shell.execute_reply":"2024-01-21T19:21:34.066507Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Load Train Data","metadata":{"papermill":{"duration":0.007846,"end_time":"2024-01-14T22:51:51.688268","exception":false,"start_time":"2024-01-14T22:51:51.680422","status":"completed"},"tags":[]}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape )\nprint('Targets', list(TARGETS))\ndf.head()","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.288611,"end_time":"2024-01-14T22:51:51.984993","exception":false,"start_time":"2024-01-14T22:51:51.696382","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:21:34.068862Z","iopub.execute_input":"2024-01-21T19:21:34.069274Z","iopub.status.idle":"2024-01-21T19:21:34.407402Z","shell.execute_reply.started":"2024-01-21T19:21:34.069243Z","shell.execute_reply":"2024-01-21T19:21:34.406245Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Create Non-Overlapping Eeg Id Train Data\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{"papermill":{"duration":0.009407,"end_time":"2024-01-14T22:51:52.004075","exception":false,"start_time":"2024-01-14T22:51:51.994668","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\n\nsum_targets = tmp.sum(axis=1)\nmax_vote_percentage = tmp.max(axis=1) / sum_targets\ntrain['max_vote_percentage'] = max_vote_percentage\n\nfor t in TARGETS:\n train[t] = tmp[t].values\n \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\n# train = train[train['max_vote_percentage']>=.9]\nprint('Train non-overlapp eeg_id shape:', train.shape )\ntrain.head()","metadata":{"papermill":{"duration":0.111621,"end_time":"2024-01-14T22:51:52.125134","exception":false,"start_time":"2024-01-14T22:51:52.013513","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:21:34.408742Z","iopub.execute_input":"2024-01-21T19:21:34.409023Z","iopub.status.idle":"2024-01-21T19:21:34.526897Z","shell.execute_reply.started":"2024-01-21T19:21:34.408998Z","shell.execute_reply":"2024-01-21T19:21:34.526012Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # Compute class weights based on the training data\n# def compute_class_weights(y_data):\n# # Convert y_data (probabilities) to class labels\n# y_classes = np.argmax(y_data, axis=1)\n# classes = np.arange(6) # 6 classes\n# class_weights = compute_class_weight(class_weight='balanced', classes=classes, y=y_classes)\n# return dict(zip(classes, class_weights))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# min_class_count = train['target'].value_counts().min()\n\n# # Collect balanced subsets\n# balanced_train_parts = []\n\n# for cls in train['target'].unique():\n# cls_data = train[train['target'] == cls]\n# sampled_data = cls_data.sample(n=min_class_count, random_state=42)\n# balanced_train_parts.append(sampled_data)\n\n# train = pd.concat(balanced_train_parts).sample(frac=1, random_state=42).reset_index(drop=True)\n\n# print(\"Balanced class distribution:\\n\", train['target'].value_counts())","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Read Train Spectrograms \n\nFirst we need to read in all 11k train spectrogram files. Reading thousands of files takes 11 minutes with Pandas. Instead, we can read 1 file from my [Kaggle dataset here][1] which contains all the 11k spectrograms in less than 1 minute! To use my Kaggle dataset, set variable `READ_SPEC_FILES = False`. Thank you for upvoting my helpful [dataset][1] :-)\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms","metadata":{"papermill":{"duration":0.00881,"end_time":"2024-01-14T22:51:52.142747","exception":false,"start_time":"2024-01-14T22:51:52.133937","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\n\nspectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()","metadata":{"papermill":{"duration":55.16894,"end_time":"2024-01-14T22:52:47.320438","exception":false,"start_time":"2024-01-14T22:51:52.151498","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:21:34.529548Z","iopub.execute_input":"2024-01-21T19:21:34.529848Z","iopub.status.idle":"2024-01-21T19:22:44.997202Z","shell.execute_reply.started":"2024-01-21T19:21:34.529822Z","shell.execute_reply":"2024-01-21T19:22:44.996182Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Read EEG Spectrograms\nIn version 4 onward, we use EEG spectrograms in addition to Kaggle spectrograms. The EEG spectrograms come from my Kaggle dataset [here][4] (which were created from my spectrogram starter [here][5]). Thank you for upvoting my Kaggle dataset!\n\n[4]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms\n[5]: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg","metadata":{}},{"cell_type":"code","source":"%%time\n\nall_eegs = np.load('/kaggle/input/brain-eeg-spectrograms/eeg_specs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-01-21T19:22:44.998591Z","iopub.execute_input":"2024-01-21T19:22:44.999278Z","iopub.status.idle":"2024-01-21T19:24:11.93439Z","shell.execute_reply.started":"2024-01-21T19:22:44.999239Z","shell.execute_reply":"2024-01-21T19:24:11.933067Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train DataLoader\nThis dataloader outputs 4 spectrogram images as a 4 channel image of size 128x256x4 per train sample. This notebook version is not using data augmention but the code is available below to experiment with albumentations data augmention. Just add `augment = True` when creating the train data loader. And consider adding new transformations to the augment function below.\n\nUPDATE: In version 4 onward, our dataloader outputs both Kaggle spectrograms and EEG spectrogams as 8 channel image of size 128x256x8. And we use horizonal data augmentation.","metadata":{"papermill":{"duration":0.010384,"end_time":"2024-01-14T22:52:47.341581","exception":false,"start_time":"2024-01-14T22:52:47.331197","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class DataGenerator(tf.keras.utils.Sequence):\n 'Generates data for Keras'\n def __init__(self, data, batch_size=32, shuffle=False, augment=False, mode='train',\n specs = spectrograms, eeg_specs = all_eegs): \n\n self.data = data\n self.batch_size = batch_size\n self.shuffle = shuffle\n self.augment = augment\n self.mode = mode\n self.specs = specs\n self.eeg_specs = eeg_specs\n self.on_epoch_end()\n \n def __len__(self):\n 'Denotes the number of batches per epoch'\n ct = int( np.ceil( len(self.data) / self.batch_size ) )\n return ct\n\n def __getitem__(self, index):\n 'Generate one batch of data'\n indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n X, y = self.__data_generation(indexes)\n if self.augment:\n X = self.__augment_batch(X)\n X, y = self.__mixup(X, y)\n return X, y\n\n def on_epoch_end(self):\n 'Updates indexes after each epoch'\n self.indexes = np.arange( len(self.data) )\n if self.shuffle: np.random.shuffle(self.indexes)\n \n def __data_generation(self, indexes):\n 'Generates data containing batch_size samples' \n \n X = np.zeros((len(indexes),128,256,8),dtype='float32')\n y = np.zeros((len(indexes),6),dtype='float32')\n img = np.ones((128,256),dtype='float32')\n \n for j,i in enumerate(indexes):\n row = self.data.iloc[i]\n if self.mode=='test': \n r = 0\n else: \n r = int( (row['min'] + row['max'])//4 )\n\n for k in range(4):\n # EXTRACT 300 ROWS OF SPECTROGRAM\n img = self.specs[row.spec_id][r:r+300,k*100:(k+1)*100].T\n \n # LOG TRANSFORM SPECTROGRAM\n img = np.clip(img,np.exp(-4),np.exp(8))\n img = np.log(img)\n \n # STANDARDIZE PER IMAGE\n ep = 1e-6\n m = np.nanmean(img.flatten())\n s = np.nanstd(img.flatten())\n img = (img-m)/(s+ep)\n img = np.nan_to_num(img, nan=0.0)\n \n # CROP TO 256 TIME STEPS\n X[j,14:-14,:,k] = img[:,22:-22] / 2.0\n \n # EEG SPECTROGRAMS\n img = self.eeg_specs[row.eeg_id]\n X[j,:,:,4:] = img\n \n if self.mode!='test':\n y[j,] = row[TARGETS]\n \n return X,y\n\n def __random_transform(self, img):\n # Apply Frequency Shift (±3 bins) with 50% probability\n if np.random.rand() < 0.5:\n shift = np.random.choice([-3, 3])\n img = np.roll(img, shift=shift, axis=0) \n \n return img\n\n def __mixup(self, X, y, alpha=0.2):\n # Mixup augmentation: create synthetic samples by linear interpolation\n if X.shape[0] <= 1:\n return X, y\n lam = np.random.beta(alpha, alpha, X.shape[0])\n indices = np.random.permutation(X.shape[0])\n X_mixed = X * lam[:, None, None, None] + X[indices] * (1 - lam[:, None, None, None])\n y_mixed = y * lam[:, None] + y[indices] * (1 - lam[:, None])\n \n return X_mixed, y_mixed\n \n def __augment_batch(self, img_batch):\n for i in range(img_batch.shape[0]):\n img_batch[i, ] = self.__random_transform(img_batch[i, ])\n return img_batch","metadata":{"papermill":{"duration":2.369789,"end_time":"2024-01-14T22:52:49.721728","exception":false,"start_time":"2024-01-14T22:52:47.351939","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:24:11.935993Z","iopub.execute_input":"2024-01-21T19:24:11.936579Z","iopub.status.idle":"2024-01-21T19:24:15.396032Z","shell.execute_reply.started":"2024-01-21T19:24:11.936544Z","shell.execute_reply":"2024-01-21T19:24:15.395227Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Display DataLoader\nBelow we display example dataloader spectrogram images.","metadata":{"papermill":{"duration":0.00888,"end_time":"2024-01-14T22:52:49.739973","exception":false,"start_time":"2024-01-14T22:52:49.731093","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Train Scheduler\nWe will train our model with a Step Train Schedule for 4 epochs. First 2 epochs are LR=1e-3. Then epochs 3 and 4 use LR=1e-4 and 1e-5 respectively. (Below we also provide a Cosine Train Schedule if you want to experiment with it. Note it is not used in this notebook).","metadata":{"papermill":{"duration":0.026741,"end_time":"2024-01-14T22:52:52.251841","exception":false,"start_time":"2024-01-14T22:52:52.2251","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from tensorflow.keras.callbacks import ReduceLROnPlateau\n\nLR_START = 1e-4\nLR_MAX = 1e-3\nLR_RAMPUP_EPOCHS = 0\nLR_SUSTAIN_EPOCHS = 1\nLR_STEP_DECAY = 0.1\nEVERY = 1\nEPOCHS = 8\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)//EVERY)\n return lr\n\n# rng = [i for i in range(EPOCHS)]\n# y = [lrfn(x) for x in rng]\n# plt.figure(figsize=(10, 4))\n# plt.plot(rng, y, 'o-'); \n# plt.xlabel('epoch',size=14); plt.ylabel('learning rate',size=14)\n# plt.title('Step Training Schedule',size=16); plt.show()\n\nLR = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n\n# LR = ReduceLROnPlateau(\n# monitor='val_loss', # Metric to monitor\n# factor=0.5, # Factor by which the learning rate will be reduced. new_lr = lr * factor\n# patience=1, # Number of epochs with no improvement after which learning rate will be reduced.\n# # Since you only have 4 epochs, patience=1 might be more reactive, or patience=1\n# # to give it a chance before reducing.\n# min_lr=1e-6, # Lower bound on the learning rate.\n# verbose=1\n# )","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.309296,"end_time":"2024-01-14T22:52:52.92271","exception":false,"start_time":"2024-01-14T22:52:52.613414","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:24:18.456106Z","iopub.execute_input":"2024-01-21T19:24:18.456453Z","iopub.status.idle":"2024-01-21T19:24:18.74635Z","shell.execute_reply.started":"2024-01-21T19:24:18.456426Z","shell.execute_reply":"2024-01-21T19:24:18.745416Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"early_stopping = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=2, restore_best_weights=True)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# def focal_loss(gamma=2.0, alpha=0.25):\n# def focal_loss_fn(y_true, y_pred):\n# y_pred = tf.clip_by_value(y_pred, tf.keras.backend.epsilon(), 1.0 - tf.keras.backend.epsilon())\n# cross_entropy = -y_true * tf.math.log(y_pred)\n# weight = alpha * y_true * tf.pow((1 - y_pred), gamma)\n# loss = weight * cross_entropy\n# return tf.reduce_mean(tf.reduce_sum(loss, axis=1))\n# return focal_loss_fn","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Build EfficientNet Model\nVersion 1-3 uses EfficientNet B2. Version 4 uses EfficientNet B0. Our models receives both Kaggle spectrograms and EEG spectrograms from our data loader. We then reshape these 8 spectrograms into 1 large flat image and feed it into EfficientNet.","metadata":{"papermill":{"duration":0.027228,"end_time":"2024-01-14T22:52:52.97653","exception":false,"start_time":"2024-01-14T22:52:52.949302","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install --no-index --find-links=/kaggle/input/tf-efficientnet-whl-files /kaggle/input/tf-efficientnet-whl-files/efficientnet-1.1.1-py3-none-any.whl","metadata":{"_kg_hide-output":true,"papermill":{"duration":13.587235,"end_time":"2024-01-14T22:53:06.589596","exception":false,"start_time":"2024-01-14T22:52:53.002361","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:24:18.747441Z","iopub.execute_input":"2024-01-21T19:24:18.747746Z","iopub.status.idle":"2024-01-21T19:24:33.595312Z","shell.execute_reply.started":"2024-01-21T19:24:18.747696Z","shell.execute_reply":"2024-01-21T19:24:33.594119Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# !pip install efficientnet","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import efficientnet.tfkeras as efn\n\ndef build_model():\n \n inp = tf.keras.Input(shape=(128,256,8))\n base_model = efn.EfficientNetB0(include_top=False, weights=None, input_shape=None)\n base_model.load_weights('/kaggle/input/tf-efficientnet-imagenet-weights/efficientnet-b0_weights_tf_dim_ordering_tf_kernels_autoaugment_notop.h5')\n \n # RESHAPE INPUT 128x256x8 => 512x512x3 MONOTONE IMAGE\n # KAGGLE SPECTROGRAMS\n x1 = [inp[:,:,:,i:i+1] for i in range(4)]\n x1 = tf.keras.layers.Concatenate(axis=1)(x1)\n # EEG SPECTROGRAMS\n x2 = [inp[:,:,:,i+4:i+5] for i in range(4)]\n x2 = tf.keras.layers.Concatenate(axis=1)(x2)\n # MAKE 512X512X3\n if USE_KAGGLE_SPECTROGRAMS & USE_EEG_SPECTROGRAMS:\n x = tf.keras.layers.Concatenate(axis=2)([x1,x2])\n elif USE_EEG_SPECTROGRAMS: x = x2\n else: x = x1\n x = tf.keras.layers.Concatenate(axis=3)([x,x,x])\n \n # OUTPUT\n x = base_model(x)\n x = tf.keras.layers.GlobalAveragePooling2D()(x)\n x = tf.keras.layers.Dropout(0.3)(x)\n x = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(x)\n \n # COMPILE MODEL\n model = tf.keras.Model(inputs=inp, outputs=x)\n opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n loss = tf.keras.losses.KLDivergence() #focal_loss(gamma=1.0, alpha=0.5)\n\n model.compile(loss=loss, optimizer = opt, metrics='accuracy') \n \n return model","metadata":{"papermill":{"duration":0.057714,"end_time":"2024-01-14T22:53:06.682056","exception":false,"start_time":"2024-01-14T22:53:06.624342","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:24:33.596965Z","iopub.execute_input":"2024-01-21T19:24:33.597291Z","iopub.status.idle":"2024-01-21T19:24:33.620946Z","shell.execute_reply.started":"2024-01-21T19:24:33.597261Z","shell.execute_reply":"2024-01-21T19:24:33.620066Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Train Model\nWe train using Group KFold on patient id. If `LOAD_MODELS_FROM = None`, then we will train new models in this notebook version. Otherwise we will load saved models from the path `LOAD_MODELS_FROM`.","metadata":{"papermill":{"duration":0.033717,"end_time":"2024-01-14T22:53:06.742557","exception":false,"start_time":"2024-01-14T22:53:06.70884","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold, StratifiedGroupKFold\nimport tensorflow.keras.backend as K, gc\n\nall_oof = []\nall_true = []\n\n# gkf = GroupKFold(n_splits=5)\ngkf = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=SEED)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)): \n \n print('#'*25)\n print(f'### Fold {i+1}')\n\n train_fold = train.iloc[train_index].copy()\n # class_weights = compute_class_weights(train_fold[TARGETS].values)\n\n # Get class labels for undersampling\n y_true_fold_train = np.argmax(train_fold[TARGETS].values, axis=1)\n class_counts_fold = np.bincount(y_true_fold_train, minlength=len(TARGETS))\n min_class_size = min(class_counts_fold) # ~733 for LRDA\n multiplier = 3\n \n class_dfs = []\n for cls_idx in range(len(TARGETS)):\n cls_df = train_fold[y_true_fold_train == cls_idx]\n target_size = min(len(cls_df), min_class_size * multiplier) # Cap at 1,466 unless smaller\n if len(cls_df) > target_size:\n cls_df = resample(cls_df, replace=False, n_samples=target_size, random_state=SEED)\n class_dfs.append(cls_df)\n \n # Concatenate undersampled data\n undersampled_train_df = pd.concat(class_dfs).sample(frac=1, random_state=SEED).reset_index(drop=True)\n \n print(f\"Undersampled train size: {len(undersampled_train_df)}\")\n\n \n train_gen = DataGenerator(undersampled_train_df, shuffle=True, batch_size=32, augment=True)\n valid_gen = DataGenerator(train.iloc[valid_index], shuffle=False, batch_size=64, mode='valid')\n \n print(f'### train size {len(undersampled_train_df)}, valid size {len(valid_index)}')\n print('#'*25)\n \n K.clear_session()\n with strategy.scope():\n model = build_model()\n if LOAD_MODELS_FROM is None:\n model.fit(train_gen, verbose=1,\n validation_data = valid_gen,\n epochs=EPOCHS, callbacks = [LR, early_stopping],\n # class_weight = class_weights\n )\n model.save_weights(f'EffNetV2B1_v{VER}_f{i}.h5')\n else:\n model.load_weights(f'{LOAD_MODELS_FROM}EffNet_v{VER}_f{i}.h5')\n \n oof = model.predict(valid_gen, verbose=1)\n all_oof.append(oof)\n all_true.append(train.iloc[valid_index][TARGETS].values)\n\n y_pred = np.argmax(oof, axis=1)\n y_true = np.argmax(train.iloc[valid_index][TARGETS].values, axis=1)\n\n # Classification report\n print(\"\\nClassification Report:\")\n print(classification_report(y_true, y_pred, target_names=TARGETS))\n \n # Confusion matrix\n cm = confusion_matrix(y_true, y_pred)\n plt.figure(figsize=(10, 8))\n sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', \n xticklabels=TARGETS, yticklabels=TARGETS)\n plt.title(f'Confusion Matrix - Fold {i+1}')\n plt.ylabel('True Label')\n plt.xlabel('Predicted Label')\n plt.show()\n \n del model, oof\n gc.collect()\n \nall_oof = np.concatenate(all_oof)\nall_true = np.concatenate(all_true)","metadata":{"papermill":{"duration":161.172,"end_time":"2024-01-14T22:55:47.94776","exception":false,"start_time":"2024-01-14T22:53:06.77576","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:24:33.622111Z","iopub.execute_input":"2024-01-21T19:24:33.622478Z","iopub.status.idle":"2024-01-21T19:26:48.914343Z","shell.execute_reply.started":"2024-01-21T19:24:33.622453Z","shell.execute_reply":"2024-01-21T19:26:48.913512Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# CV Score for EfficientNet\nThis is CV score for our EfficientNet model.","metadata":{"papermill":{"duration":0.047893,"end_time":"2024-01-14T22:55:48.045405","exception":false,"start_time":"2024-01-14T22:55:47.997512","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score KL-Div for EfficientNetB2 =',cv)","metadata":{"papermill":{"duration":0.126007,"end_time":"2024-01-14T22:55:48.222599","exception":false,"start_time":"2024-01-14T22:55:48.096592","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:26:48.917528Z","iopub.execute_input":"2024-01-21T19:26:48.91782Z","iopub.status.idle":"2024-01-21T19:26:48.974204Z","shell.execute_reply.started":"2024-01-21T19:26:48.917795Z","shell.execute_reply":"2024-01-21T19:26:48.973351Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Infer Test and Create Submission CSV\nBelow we use our 5 EfficientNet fold models to infer the test data and create a `submission.csv` file.","metadata":{"papermill":{"duration":0.050491,"end_time":"2024-01-14T22:55:48.321932","exception":false,"start_time":"2024-01-14T22:55:48.271441","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# del all_eegs, spectrograms; gc.collect()\n# test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\n# print('Test shape',test.shape)\n# test.head()","metadata":{"papermill":{"duration":0.073698,"end_time":"2024-01-14T22:55:48.445914","exception":false,"start_time":"2024-01-14T22:55:48.372216","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:26:48.975432Z","iopub.execute_input":"2024-01-21T19:26:48.975796Z","iopub.status.idle":"2024-01-21T19:26:49.5313Z","shell.execute_reply.started":"2024-01-21T19:26:48.97577Z","shell.execute_reply":"2024-01-21T19:26:49.53033Z"},"trusted":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # READ ALL SPECTROGRAMS\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\n# files2 = os.listdir(PATH2)\n# print(f'There are {len(files2)} test spectrogram parquets')\n \n# spectrograms2 = {}\n# for i,f in enumerate(files2):\n# if i%100==0: print(i,', ',end='')\n# tmp = pd.read_parquet(f'{PATH2}{f}')\n# name = int(f.split('.')[0])\n# spectrograms2[name] = tmp.iloc[:,1:].values\n \n# # RENAME FOR DATALOADER\n# test = test.rename({'spectrogram_id':'spec_id'},axis=1)","metadata":{"papermill":{"duration":0.257975,"end_time":"2024-01-14T22:55:48.757931","exception":false,"start_time":"2024-01-14T22:55:48.499956","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-01-21T19:26:49.532661Z","iopub.execute_input":"2024-01-21T19:26:49.533958Z","iopub.status.idle":"2024-01-21T19:26:49.946019Z","shell.execute_reply.started":"2024-01-21T19:26:49.533929Z","shell.execute_reply":"2024-01-21T19:26:49.945209Z"},"trusted":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# import pywt, librosa\n\n# USE_WAVELET = None \n\n# NAMES = ['LL','LP','RP','RR']\n\n# FEATS = [['Fp1','F7','T3','T5','O1'],\n# ['Fp1','F3','C3','P3','O1'],\n# ['Fp2','F8','T4','T6','O2'],\n# ['Fp2','F4','C4','P4','O2']]\n\n# # DENOISE FUNCTION\n# def maddest(d, axis=None):\n# return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\n# def denoise(x, wavelet='haar', level=1): \n# coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n# sigma = (1/0.6745) * maddest(coeff[-level])\n\n# uthresh = sigma * np.sqrt(2*np.log(len(x)))\n# coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n# ret=pywt.waverec(coeff, wavelet, mode='per')\n \n# return ret\n\n# def spectrogram_from_eeg(parquet_path, display=False):\n \n# # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n# eeg = pd.read_parquet(parquet_path)\n# middle = (len(eeg)-10_000)//2\n# eeg = eeg.iloc[middle:middle+10_000]\n \n# # VARIABLE TO HOLD SPECTROGRAM\n# img = np.zeros((128,256,4),dtype='float32')\n \n# if display: plt.figure(figsize=(10,7))\n# signals = []\n# for k in range(4):\n# COLS = FEATS[k]\n \n# for kk in range(4):\n \n# # COMPUTE PAIR DIFFERENCES\n# x = eeg[COLS[kk]].values - eeg[COLS[kk+1]].values\n\n# # FILL NANS\n# m = np.nanmean(x)\n# if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n# else: x[:] = 0\n\n# # DENOISE\n# if USE_WAVELET:\n# x = denoise(x, wavelet=USE_WAVELET)\n# signals.append(x)\n\n# # RAW SPECTROGRAM\n# mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x)//256, \n# n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n\n# # LOG TRANSFORM\n# width = (mel_spec.shape[1]//32)*32\n# mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:,:width]\n\n# # STANDARDIZE TO -1 TO 1\n# mel_spec_db = (mel_spec_db+40)/40 \n# img[:,:,k] += mel_spec_db\n \n# # AVERAGE THE 4 MONTAGE DIFFERENCES\n# img[:,:,k] /= 4.0\n \n# if display:\n# plt.subplot(2,2,k+1)\n# plt.imshow(img[:,:,k],aspect='auto',origin='lower')\n# plt.title(f'EEG {eeg_id} - Spectrogram {NAMES[k]}')\n \n# if display: \n# plt.show()\n# plt.figure(figsize=(10,5))\n# offset = 0\n# for k in range(4):\n# if k>0: offset -= signals[3-k].min()\n# plt.plot(range(10_000),signals[k]+offset,label=NAMES[3-k])\n# offset += signals[3-k].max()\n# plt.legend()\n# plt.title(f'EEG {eeg_id} Signals')\n# plt.show()\n# print(); print('#'*25); print()\n \n# return img","metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-01-21T19:26:49.947542Z","iopub.execute_input":"2024-01-21T19:26:49.947862Z","iopub.status.idle":"2024-01-21T19:26:49.975242Z","shell.execute_reply.started":"2024-01-21T19:26:49.947835Z","shell.execute_reply":"2024-01-21T19:26:49.974407Z"},"trusted":true,"jupyter":{"source_hidden":true}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# # READ ALL EEG SPECTROGRAMS\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n# DISPLAY = 1\n# EEG_IDS2 = test.eeg_id.unique()\n# all_eegs2 = {}\n\n# print('Converting Test EEG to Spectrograms...'); print()\n# for i,eeg_id in enumerate(EEG_IDS2):\n \n# # CREATE SPECTROGRAM FROM EEG PARQUET\n# img = spectrogram_from_eeg(f'{PATH2}{eeg_id}.parquet', i