{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7402356,"sourceType":"datasetVersion","datasetId":4304475},{"sourceId":7403069,"sourceType":"datasetVersion","datasetId":4304949},{"sourceId":7447509,"sourceType":"datasetVersion","datasetId":4334995},{"sourceId":7450712,"sourceType":"datasetVersion","datasetId":4336944},{"sourceId":158958765,"sourceType":"kernelVersion"}],"dockerImageVersionId":30636,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"papermill":{"default_parameters":{},"duration":270.012179,"end_time":"2024-01-14T22:56:02.916427","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-01-14T22:51:32.904248","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"","metadata":{"papermill":{"duration":0.008094,"end_time":"2024-01-14T22:51:36.811998","exception":false,"start_time":"2024-01-14T22:51:36.803904","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Kaggle Spectrograms vs. EEG Spectrograms\nIn this competition, Kaggle provides us with both spectrograms and raw eeg waveforms. The Kaggle spectrograms are 10 minutes long and the eeg waveforms are 50 seconds. The middle 50 seconds of both data are the same time window observing the same event. (i.e. the middle 50 seconds is same information displayed two different ways). Spectrograms are just visual representations of raw waveforms.\n\nI published a spectrogram starter notebook [here][1] that converts the raw eeg waveform into a spectrogram. There is a discussion [here][3]. The eeg spectrograms (from version 4) have been uploaded to my Kaggle dataset [here][2]. Version 4 (of my spectrogram starter notebook) onward uses a new powerful formula. The below table shows that the new created eeg spectrograms are equal or better than the Kaggle spectrograms:\n\n| Spectrogram | EffNet 5Fold CV | EffNet LB | Notebook version | Model | Loss | Epochs | Data Augmention |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Kaggle Spectrograms | 0.73 | 0.57 | ver 1 | EffNetB2 | CE | 3 | none |\n| Kaggle Spectrograms | 0.66 | ??? | ver 4 | EffNetB0 | KL-Div | 4 | none |\n| EEG Spectrograms | 0.63 | ??? | ver 3 | EffNetB0 | KL-Div | 4 | none |\n| **Both Spectrograms** | **0.59** | **0.44** | ver 5| EffNetB0 | KL-Div | 4 | none |\n\n\nIn different versions of this (efficientnet starter) notebook, we will train three EfficientNet models. Version 3 is using only EEG spectrograms. Version 4 is using only Kaggle spectrograms. And version 5 is a single model using both EEG spectrograms and Kaggle spectrograms. Then we save the model weights into a Kaggle dataset [here][4] and we make a submit notebook in version 6 for the version 5 model trained on both spectrograms. We achieve **CV 0.59 LB 0.44**! Wow!\n\n[1]: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg\n[2]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms\n[3]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469760\n[4]: https://www.kaggle.com/datasets/cdeotte/brain-efficientnet-models-v3-v4-v5","metadata":{}},{"cell_type":"markdown","source":"# Initialize 2xT4 GPUs\nWe will use both Kaggle T4 GPUs and we will use mixed precision.","metadata":{"papermill":{"duration":0.008572,"end_time":"2024-01-14T22:51:36.82846","exception":false,"start_time":"2024-01-14T22:51:36.819888","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os, gc\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport tensorflow as tf\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\nprint('TensorFlow version =',tf.__version__)\n\n# USE MULTIPLE GPUS\ngpus = tf.config.list_physical_devices('GPU')\nif len(gpus)<=1: \n strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n print(f'Using {len(gpus)} GPU')\nelse: \n strategy = tf.distribute.MirroredStrategy()\n print(f'Using {len(gpus)} GPUs')\n\nVER = 5\n\n# IF THIS EQUALS NONE, THEN WE TRAIN NEW MODELS\n# IF THIS EQUALS DISK PATH, THEN WE LOAD PREVIOUSLY TRAINED MODELS\nLOAD_MODELS_FROM = '/kaggle/input/brain-efficientnet-models-v3-v4-v5/'\n\nUSE_KAGGLE_SPECTROGRAMS = True\nUSE_EEG_SPECTROGRAMS = True","metadata":{"papermill":{"duration":14.80928,"end_time":"2024-01-14T22:51:51.64702","exception":false,"start_time":"2024-01-14T22:51:36.83774","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-07T17:13:56.080659Z","iopub.execute_input":"2024-04-07T17:13:56.080959Z","iopub.status.idle":"2024-04-07T17:14:10.512878Z","shell.execute_reply.started":"2024-04-07T17:13:56.080917Z","shell.execute_reply":"2024-04-07T17:14:10.511867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# USE MIXED PRECISION\nMIX = True\nif MIX:\n tf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\n print('Mixed precision enabled')\nelse:\n print('Using full precision')","metadata":{"papermill":{"duration":0.016556,"end_time":"2024-01-14T22:51:51.671783","exception":false,"start_time":"2024-01-14T22:51:51.655227","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-07T17:14:10.514512Z","iopub.execute_input":"2024-04-07T17:14:10.515054Z","iopub.status.idle":"2024-04-07T17:14:10.520774Z","shell.execute_reply.started":"2024-04-07T17:14:10.515026Z","shell.execute_reply":"2024-04-07T17:14:10.519672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Train Data","metadata":{"papermill":{"duration":0.007846,"end_time":"2024-01-14T22:51:51.688268","exception":false,"start_time":"2024-01-14T22:51:51.680422","status":"completed"},"tags":[]}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape )\nprint('Targets', list(TARGETS))\ndf.head()","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.288611,"end_time":"2024-01-14T22:51:51.984993","exception":false,"start_time":"2024-01-14T22:51:51.696382","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-07T17:14:10.52247Z","iopub.execute_input":"2024-04-07T17:14:10.522843Z","iopub.status.idle":"2024-04-07T17:14:10.812255Z","shell.execute_reply.started":"2024-04-07T17:14:10.52281Z","shell.execute_reply":"2024-04-07T17:14:10.811383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Non-Overlapping Eeg Id Train Data\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{"papermill":{"duration":0.009407,"end_time":"2024-01-14T22:51:52.004075","exception":false,"start_time":"2024-01-14T22:51:51.994668","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n train[t] = tmp[t].values\n \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\nprint('Train non-overlapp eeg_id shape:', train.shape )\ntrain.head()","metadata":{"papermill":{"duration":0.111621,"end_time":"2024-01-14T22:51:52.125134","exception":false,"start_time":"2024-01-14T22:51:52.013513","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-07T17:14:10.81483Z","iopub.execute_input":"2024-04-07T17:14:10.815688Z","iopub.status.idle":"2024-04-07T17:14:10.915069Z","shell.execute_reply.started":"2024-04-07T17:14:10.815662Z","shell.execute_reply":"2024-04-07T17:14:10.914094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read Train Spectrograms \n\nFirst we need to read in all 11k train spectrogram files. Reading thousands of files takes 11 minutes with Pandas. Instead, we can read 1 file from my [Kaggle dataset here][1] which contains all the 11k spectrograms in less than 1 minute! To use my Kaggle dataset, set variable `READ_SPEC_FILES = False`. Thank you for upvoting my helpful [dataset][1] :-)\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms","metadata":{"papermill":{"duration":0.00881,"end_time":"2024-01-14T22:51:52.142747","exception":false,"start_time":"2024-01-14T22:51:52.133937","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nREAD_SPEC_FILES = False\n\n# READ ALL SPECTROGRAMS\nPATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/'\nfiles = os.listdir(PATH)\nprint(f'There are {len(files)} spectrogram parquets')\n\nif READ_SPEC_FILES: \n spectrograms = {}\n for i,f in enumerate(files):\n if i%100==0: print(i,', ',end='')\n tmp = pd.read_parquet(f'{PATH}{f}')\n name = int(f.split('.')[0])\n spectrograms[name] = tmp.iloc[:,1:].values\nelse:\n spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()","metadata":{"papermill":{"duration":55.16894,"end_time":"2024-01-14T22:52:47.320438","exception":false,"start_time":"2024-01-14T22:51:52.151498","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-07T17:14:10.916115Z","iopub.execute_input":"2024-04-07T17:14:10.916372Z","iopub.status.idle":"2024-04-07T17:15:13.557689Z","shell.execute_reply.started":"2024-04-07T17:14:10.916351Z","shell.execute_reply":"2024-04-07T17:15:13.556821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read EEG Spectrograms\nIn version 4 onward, we use EEG spectrograms in addition to Kaggle spectrograms. The EEG spectrograms come from my Kaggle dataset [here][4] (which were created from my spectrogram starter [here][5]). Thank you for upvoting my Kaggle dataset!\n\n[4]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms\n[5]: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg","metadata":{}},{"cell_type":"code","source":"%%time\nREAD_EEG_SPEC_FILES = False\n\nif READ_EEG_SPEC_FILES:\n all_eegs = {}\n for i,e in enumerate(train.eeg_id.values):\n if i%100==0: print(i,', ',end='')\n x = np.load(f'/kaggle/input/brain-eeg-spectrograms/EEG_Spectrograms/{e}.npy')\n all_eegs[e] = x\nelse:\n all_eegs = np.load('/kaggle/input/brain-eeg-spectrograms/eeg_specs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-04-07T17:15:13.559191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train DataLoader\nThis dataloader outputs 4 spectrogram images as a 4 channel image of size 128x256x4 per train sample. This notebook version is not using data augmention but the code is available below to experiment with albumentations data augmention. Just add `augment = True` when creating the train data loader. And consider adding new transformations to the augment function below.\n\nUPDATE: In version 4 onward, our dataloader outputs both Kaggle spectrograms and EEG spectrogams as 8 channel image of size 128x256x8.","metadata":{"papermill":{"duration":0.010384,"end_time":"2024-01-14T22:52:47.341581","exception":false,"start_time":"2024-01-14T22:52:47.331197","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import albumentations as albu\nTARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\nTARS2 = {x:y for y,x in TARS.items()}\n\nclass DataGenerator(tf.keras.utils.Sequence):\n 'Generates data for Keras'\n def __init__(self, data, batch_size=32, shuffle=False, augment=False, mode='train',\n specs = spectrograms, eeg_specs = all_eegs): \n\n self.data = data\n self.batch_size = batch_size\n self.shuffle = shuffle\n self.augment = augment\n self.mode = mode\n self.specs = specs\n self.eeg_specs = eeg_specs\n self.on_epoch_end()\n \n def __len__(self):\n 'Denotes the number of batches per epoch'\n ct = int( np.ceil( len(self.data) / self.batch_size ) )\n return ct\n\n def __getitem__(self, index):\n 'Generate one batch of data'\n indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n X, y = self.__data_generation(indexes)\n if self.augment: X = self.__augment_batch(X) \n return X, y\n\n def on_epoch_end(self):\n 'Updates indexes after each epoch'\n self.indexes = np.arange( len(self.data) )\n if self.shuffle: np.random.shuffle(self.indexes)\n \n def __data_generation(self, indexes):\n 'Generates data containing batch_size samples' \n \n X = np.zeros((len(indexes),128,256,8),dtype='float32')\n y = np.zeros((len(indexes),6),dtype='float32')\n img = np.ones((128,256),dtype='float32')\n \n for j,i in enumerate(indexes):\n row = self.data.iloc[i]\n if self.mode=='test': \n r = 0\n else: \n r = int( (row['min'] + row['max'])//4 )\n\n for k in range(4):\n # EXTRACT 300 ROWS OF SPECTROGRAM\n img = self.specs[row.spec_id][r:r+300,k*100:(k+1)*100].T\n \n # LOG TRANSFORM SPECTROGRAM\n img = np.clip(img,np.exp(-4),np.exp(8))\n img = np.log(img)\n \n # STANDARDIZE PER IMAGE\n ep = 1e-6\n m = np.nanmean(img.flatten())\n s = np.nanstd(img.flatten())\n img = (img-m)/(s+ep)\n img = np.nan_to_num(img, nan=0.0)\n \n # CROP TO 256 TIME STEPS\n X[j,14:-14,:,k] = img[:,22:-22] / 2.0\n \n # EEG SPECTROGRAMS\n img = self.eeg_specs[row.eeg_id]\n X[j,:,:,4:] = img\n \n if self.mode!='test':\n y[j,] = row[TARGETS]\n \n return X,y\n \n def __random_transform(self, img):\n composition = albu.Compose([\n albu.HorizontalFlip(p=0.5),\n #albu.CoarseDropout(max_holes=8,max_height=32,max_width=32,fill_value=0,p=0.5),\n ])\n return composition(image=img)['image']\n \n def __augment_batch(self, img_batch):\n for i in range(img_batch.shape[0]):\n img_batch[i, ] = self.__random_transform(img_batch[i, ])\n return img_batch","metadata":{"papermill":{"duration":2.369789,"end_time":"2024-01-14T22:52:49.721728","exception":false,"start_time":"2024-01-14T22:52:47.351939","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Display DataLoader\nBelow we display example dataloader spectrogram images.","metadata":{"papermill":{"duration":0.00888,"end_time":"2024-01-14T22:52:49.739973","exception":false,"start_time":"2024-01-14T22:52:49.731093","status":"completed"},"tags":[]}},{"cell_type":"code","source":"gen = DataGenerator(train, batch_size=32, shuffle=False)\nROWS=2; COLS=3; BATCHES=2\n\nfor i,(x,y) in enumerate(gen):\n plt.figure(figsize=(20,8))\n for j in range(ROWS):\n for k in range(COLS):\n plt.subplot(ROWS,COLS,j*COLS+k+1)\n t = y[j*COLS+k]\n img = x[j*COLS+k,:,:,0][::-1,]\n mn = img.flatten().min()\n mx = img.flatten().max()\n img = (img-mn)/(mx-mn)\n plt.imshow(img)\n tars = f'[{t[0]:0.2f}'\n for s in t[1:]: tars += f', {s:0.2f}'\n eeg = train.eeg_id.values[i*32+j*COLS+k]\n plt.title(f'EEG = {eeg}\\nTarget = {tars}',size=12)\n plt.yticks([])\n plt.ylabel('Frequencies (Hz)',size=14)\n plt.xlabel('Time (sec)',size=16)\n plt.show()\n if i==BATCHES-1: break","metadata":{"papermill":{"duration":2.448242,"end_time":"2024-01-14T22:52:52.197249","exception":false,"start_time":"2024-01-14T22:52:49.749007","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Scheduler\nWe will train our model with a Step Train Schedule for 4 epochs. First 2 epochs are LR=1e-3. Then epochs 3 and 4 use LR=1e-4 and 1e-5 respectively. (Below we also provide a Cosine Train Schedule if you want to experiment with it. Note it is not used in this notebook).","metadata":{"papermill":{"duration":0.026741,"end_time":"2024-01-14T22:52:52.251841","exception":false,"start_time":"2024-01-14T22:52:52.2251","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import math\nLR_START = 1e-6\nLR_MAX = 1e-3\nLR_MIN = 1e-6\nLR_RAMPUP_EPOCHS = 0\nLR_SUSTAIN_EPOCHS = 0\nEPOCHS2 = 10\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n decay_total_epochs = EPOCHS2 - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS - 1\n decay_epoch_index = epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS\n phase = math.pi * decay_epoch_index / decay_total_epochs\n cosine_decay = 0.5 * (1 + math.cos(phase))\n lr = (LR_MAX - LR_MIN) * cosine_decay + LR_MIN\n return lr\n\nrng = [i for i in range(EPOCHS2)]\nlr_y = [lrfn(x) for x in rng]\nplt.figure(figsize=(10, 4))\nplt.plot(rng, lr_y, '-o')\nplt.xlabel('epoch',size=14); plt.ylabel('learning rate',size=14)\nplt.title('Cosine Training Schedule',size=16); plt.show()\n\nLR2 = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.304824,"end_time":"2024-01-14T22:52:52.58355","exception":false,"start_time":"2024-01-14T22:52:52.278726","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LR_START = 1e-4\nLR_MAX = 1e-3\nLR_RAMPUP_EPOCHS = 0\nLR_SUSTAIN_EPOCHS = 1\nLR_STEP_DECAY = 0.1\nEVERY = 1\nEPOCHS = 4\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)//EVERY)\n return lr\n\nrng = [i for i in range(EPOCHS)]\ny = [lrfn(x) for x in rng]\nplt.figure(figsize=(10, 4))\nplt.plot(rng, y, 'o-'); \nplt.xlabel('epoch',size=14); plt.ylabel('learning rate',size=14)\nplt.title('Step Training Schedule',size=16); plt.show()\n\nLR = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.309296,"end_time":"2024-01-14T22:52:52.92271","exception":false,"start_time":"2024-01-14T22:52:52.613414","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build EfficientNet Model\nVersion 1-3 uses EfficientNet B2. Version 4 uses EfficientNet B0. Our models receives both Kaggle spectrograms and EEG spectrograms from our data loader. We then reshape these 8 spectrograms into 1 large flat image and feed it into EfficientNet.","metadata":{"papermill":{"duration":0.027228,"end_time":"2024-01-14T22:52:52.97653","exception":false,"start_time":"2024-01-14T22:52:52.949302","status":"completed"},"tags":[]}},{"cell_type":"code","source":"!pip install --no-index --find-links=/kaggle/input/tf-efficientnet-whl-files /kaggle/input/tf-efficientnet-whl-files/efficientnet-1.1.1-py3-none-any.whl","metadata":{"_kg_hide-output":true,"papermill":{"duration":13.587235,"end_time":"2024-01-14T22:53:06.589596","exception":false,"start_time":"2024-01-14T22:52:53.002361","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import efficientnet.tfkeras as efn\n\ndef build_model():\n \n inp = tf.keras.Input(shape=(128,256,8))\n base_model = efn.EfficientNetB0(include_top=False, weights=None, input_shape=None)\n base_model.load_weights('/kaggle/input/tf-efficientnet-imagenet-weights/efficientnet-b0_weights_tf_dim_ordering_tf_kernels_autoaugment_notop.h5')\n \n # RESHAPE INPUT 128x256x8 => 512x512x3 MONOTONE IMAGE\n # KAGGLE SPECTROGRAMS\n x1 = [inp[:,:,:,i:i+1] for i in range(4)]\n x1 = tf.keras.layers.Concatenate(axis=1)(x1)\n # EEG SPECTROGRAMS\n x2 = [inp[:,:,:,i+4:i+5] for i in range(4)]\n x2 = tf.keras.layers.Concatenate(axis=1)(x2)\n # MAKE 512X512X3\n if USE_KAGGLE_SPECTROGRAMS & USE_EEG_SPECTROGRAMS:\n x = tf.keras.layers.Concatenate(axis=2)([x1,x2])\n elif USE_EEG_SPECTROGRAMS: x = x2\n else: x = x1\n x = tf.keras.layers.Concatenate(axis=3)([x,x,x])\n \n # OUTPUT\n x = base_model(x)\n x = tf.keras.layers.GlobalAveragePooling2D()(x)\n x = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(x)\n \n # COMPILE MODEL\n model = tf.keras.Model(inputs=inp, outputs=x)\n opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n loss = tf.keras.losses.KLDivergence()\n\n model.compile(loss=loss, optimizer = opt) \n \n return model","metadata":{"papermill":{"duration":0.057714,"end_time":"2024-01-14T22:53:06.682056","exception":false,"start_time":"2024-01-14T22:53:06.624342","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Model\nWe train using Group KFold on patient id. If `LOAD_MODELS_FROM = None`, then we will train new models in this notebook version. Otherwise we will load saved models from the path `LOAD_MODELS_FROM`.","metadata":{"papermill":{"duration":0.033717,"end_time":"2024-01-14T22:53:06.742557","exception":false,"start_time":"2024-01-14T22:53:06.70884","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold\nimport tensorflow.keras.backend as K, gc\n\nall_oof = []\nall_true = []\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)): \n \n print('#'*25)\n print(f'### Fold {i+1}')\n \n train_gen = DataGenerator(train.iloc[train_index], shuffle=True, batch_size=32, augment=False)\n valid_gen = DataGenerator(train.iloc[valid_index], shuffle=False, batch_size=64, mode='valid')\n \n print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n print('#'*25)\n \n K.clear_session()\n with strategy.scope():\n model = build_model()\n if LOAD_MODELS_FROM is None:\n model.fit(train_gen, verbose=1,\n validation_data = valid_gen,\n epochs=EPOCHS, callbacks = [LR])\n model.save_weights(f'EffNet_v{VER}_f{i}.h5')\n else:\n model.load_weights(f'{LOAD_MODELS_FROM}EffNet_v{VER}_f{i}.h5')\n \n oof = model.predict(valid_gen, verbose=1)\n all_oof.append(oof)\n all_true.append(train.iloc[valid_index][TARGETS].values)\n \n del model, oof\n gc.collect()\n \nall_oof = np.concatenate(all_oof)\nall_true = np.concatenate(all_true)","metadata":{"papermill":{"duration":161.172,"end_time":"2024-01-14T22:55:47.94776","exception":false,"start_time":"2024-01-14T22:53:06.77576","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for EfficientNet\nThis is CV score for our EfficientNet model.","metadata":{"papermill":{"duration":0.047893,"end_time":"2024-01-14T22:55:48.045405","exception":false,"start_time":"2024-01-14T22:55:47.997512","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score KL-Div for EfficientNetB2 =',cv)","metadata":{"papermill":{"duration":0.126007,"end_time":"2024-01-14T22:55:48.222599","exception":false,"start_time":"2024-01-14T22:55:48.096592","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Infer Test and Create Submission CSV\nBelow we use our 5 EfficientNet fold models to infer the test data and create a `submission.csv` file.","metadata":{"papermill":{"duration":0.050491,"end_time":"2024-01-14T22:55:48.321932","exception":false,"start_time":"2024-01-14T22:55:48.271441","status":"completed"},"tags":[]}},{"cell_type":"code","source":"del all_eegs, spectrograms; gc.collect()\ntest = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\nprint('Test shape',test.shape)\ntest.head()","metadata":{"papermill":{"duration":0.073698,"end_time":"2024-01-14T22:55:48.445914","exception":false,"start_time":"2024-01-14T22:55:48.372216","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# READ ALL SPECTROGRAMS\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\nfiles2 = os.listdir(PATH2)\nprint(f'There are {len(files2)} test spectrogram parquets')\n \nspectrograms2 = {}\nfor i,f in enumerate(files2):\n if i%100==0: print(i,', ',end='')\n tmp = pd.read_parquet(f'{PATH2}{f}')\n name = int(f.split('.')[0])\n spectrograms2[name] = tmp.iloc[:,1:].values\n \n# RENAME FOR DATALOADER\ntest = test.rename({'spectrogram_id':'spec_id'},axis=1)","metadata":{"papermill":{"duration":0.257975,"end_time":"2024-01-14T22:55:48.757931","exception":false,"start_time":"2024-01-14T22:55:48.499956","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pywt, librosa\n\nUSE_WAVELET = None \n\nNAMES = ['LL','LP','RP','RR']\n\nFEATS = [['Fp1','F7','T3','T5','O1'],\n ['Fp1','F3','C3','P3','O1'],\n ['Fp2','F8','T4','T6','O2'],\n ['Fp2','F4','C4','P4','O2']]\n\n# DENOISE FUNCTION\ndef maddest(d, axis=None):\n return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\ndef denoise(x, wavelet='haar', level=1): \n coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n sigma = (1/0.6745) * maddest(coeff[-level])\n\n uthresh = sigma * np.sqrt(2*np.log(len(x)))\n coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n ret=pywt.waverec(coeff, wavelet, mode='per')\n \n return ret\n\ndef spectrogram_from_eeg(parquet_path, display=False):\n \n # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n eeg = pd.read_parquet(parquet_path)\n middle = (len(eeg)-10_000)//2\n eeg = eeg.iloc[middle:middle+10_000]\n \n # VARIABLE TO HOLD SPECTROGRAM\n img = np.zeros((128,256,4),dtype='float32')\n \n if display: plt.figure(figsize=(10,7))\n signals = []\n for k in range(4):\n COLS = FEATS[k]\n \n for kk in range(4):\n \n # COMPUTE PAIR DIFFERENCES\n x = eeg[COLS[kk]].values - eeg[COLS[kk+1]].values\n\n # FILL NANS\n m = np.nanmean(x)\n if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n else: x[:] = 0\n\n # DENOISE\n if USE_WAVELET:\n x = denoise(x, wavelet=USE_WAVELET)\n signals.append(x)\n\n # RAW SPECTROGRAM\n mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x)//256, \n n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n\n # LOG TRANSFORM\n width = (mel_spec.shape[1]//32)*32\n mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:,:width]\n\n # STANDARDIZE TO -1 TO 1\n mel_spec_db = (mel_spec_db+40)/40 \n img[:,:,k] += mel_spec_db\n \n # AVERAGE THE 4 MONTAGE DIFFERENCES\n img[:,:,k] /= 4.0\n \n if display:\n plt.subplot(2,2,k+1)\n plt.imshow(img[:,:,k],aspect='auto',origin='lower')\n plt.title(f'EEG {eeg_id} - Spectrogram {NAMES[k]}')\n \n if display: \n plt.show()\n plt.figure(figsize=(10,5))\n offset = 0\n for k in range(4):\n if k>0: offset -= signals[3-k].min()\n plt.plot(range(10_000),signals[k]+offset,label=NAMES[3-k])\n offset += signals[3-k].max()\n plt.legend()\n plt.title(f'EEG {eeg_id} Signals')\n plt.show()\n print(); print('#'*25); print()\n \n return img","metadata":{"_kg_hide-output":false,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# READ ALL EEG SPECTROGRAMS\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\nDISPLAY = 1\nEEG_IDS2 = test.eeg_id.unique()\nall_eegs2 = {}\n\nprint('Converting Test EEG to Spectrograms...'); print()\nfor i,eeg_id in enumerate(EEG_IDS2):\n \n # CREATE SPECTROGRAM FROM EEG PARQUET\n img = spectrogram_from_eeg(f'{PATH2}{eeg_id}.parquet', i 512x512x3 MONOTONE IMAGE\n# # KAGGLE SPECTROGRAMS\n# x1 = [inp[:, :, :, i:i+1] for i in range(4)]\n# x1 = tf.keras.layers.Concatenate(axis=1)(x1)\n# # EEG SPECTROGRAMS\n# x2 = [inp[:, :, :, i+4:i+5] for i in range(4)]\n# x2 = tf.keras.layers.Concatenate(axis=1)(x2)\n# # MAKE 512X512X3\n# if USE_KAGGLE_SPECTROGRAMS & USE_EEG_SPECTROGRAMS:\n# x = tf.keras.layers.Concatenate(axis=2)([x1, x2])\n# elif USE_EEG_SPECTROGRAMS:\n# x = x2\n# else:\n# x = x1\n# x = tf.keras.layers.Concatenate(axis=3)([x, x, x])\n\n# # OUTPUT\n# x = base_model(x)\n# x = tf.keras.layers.GlobalAveragePooling2D()(x)\n# x = tf.keras.layers.Dense(6, activation='softmax', dtype='float32')(x)\n\n# # COMPILE MODEL\n# model = tf.keras.Model(inputs=inp, outputs=x)\n# opt = tf.keras.optimizers.Adam(learning_rate=1e-3)\n# loss = tf.keras.losses.KLDivergence()\n\n# model.compile(loss=loss, optimizer=opt)\n\n# return model\n\n# # Train Model\n# from sklearn.model_selection import KFold, GroupKFold\n# import tensorflow.keras.backend as K, gc\n\n# all_oof = []\n# all_true = []\n\n# gkf = GroupKFold(n_splits=5)\n# for i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):\n# print('#' * 25)\n# print(f'### Fold {i+1}')\n\n# train_gen = DataGenerator(train.iloc[train_index], shuffle=True, batch_size=32, augment=False)\n# valid_gen = DataGenerator(train.iloc[valid_index], shuffle=False, batch_size=64, mode='valid')\n\n# print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n# print('#' * 25)\n\n# K.clear_session()\n# with strategy.scope():\n# model = build_model()\n# model.fit(train_gen, verbose=1,\n# validation_data=valid_gen,\n# epochs=EPOCHS, callbacks=[LR])\n# model.save_weights(f'VGG_v{VER}_f{i}.h5')\n\n# oof = model.predict(valid_gen, verbose=1)\n# all_oof.append(oof)\n# all_true.append(train.iloc[valid_index][TARGETS].values)\n\n# del model, oof\n# gc.collect()\n\n# all_oof = np.concatenate(all_oof)\n# all_true = np.concatenate(all_true)\n\n# # CV Score for VGG16\n# import sys\n# sys.path.append('/kaggle/input/kaggle-kl-div')\n# from kaggle_kl_div import score\n\n# oof = pd.DataFrame(all_oof.copy())\n# oof['id'] = np.arange(len(oof))\n\n# true = pd.DataFrame(all_true.copy())\n# true['id'] = np.arange(len(true))\n\n# cv = score(solution=true, submission=oof, row_id_column_name='id')\n# print('CV Score KL-Div for VGG16 =', cv)\n\n# # Infer Test and Create Submission CSV\n# del all_eegs, spectrograms\n# gc.collect()\n# test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\n# print('Test shape', test.shape)\n# test.head()\n\n# # READ ALL SPECTROGRAMS\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\n# files2 = os.listdir(PATH2)\n# print(f'There are {len(files2)} test spectrogram parquets')\n\n# spectrograms2 = {}\n# for i, f in enumerate(files2):\n# if i % 100 == 0:\n# print(i, ', ', end='')\n# tmp = pd.read_parquet(f'{PATH2}{f}')\n# name = int(f.split('.')[0])\n# spectrograms2[name] =tmp.iloc[:, 1:].values\n\n# # RENAME FOR DATALOADER\n# test = test.rename({'spectrogram_id': 'spec_id'}, axis=1)\n\n# import pywt, librosa\n\n# USE_WAVELET = None\n\n# NAMES = ['LL', 'LP', 'RP', 'RR']\n\n# FEATS = [['Fp1', 'F7', 'T3', 'T5', 'O1'],\n# ['Fp1', 'F3', 'C3', 'P3', 'O1'],\n# ['Fp2', 'F8', 'T4', 'T6', 'O2'],\n# ['Fp2', 'F4', 'C4', 'P4', 'O2']]\n\n# # DENOISE FUNCTION\n# def maddest(d, axis=None):\n# return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\n# def denoise(x, wavelet='haar', level=1):\n# coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n# sigma = (1 / 0.6745) * maddest(coeff[-level])\n\n# uthresh = sigma * np.sqrt(2 * np.log(len(x)))\n# coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n# ret = pywt.waverec(coeff, wavelet, mode='per')\n\n# return ret\n\n# def spectrogram_from_eeg(parquet_path, display=False):\n# # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n# eeg = pd.read_parquet(parquet_path)\n# middle = (len(eeg) - 10_000) // 2\n# eeg = eeg.iloc[middle:middle + 10_000]\n\n# # VARIABLE TO HOLD SPECTROGRAM\n# img = np.zeros((128, 256, 4), dtype='float32')\n\n# if display:\n# plt.figure(figsize=(10, 7))\n# signals = []\n# for k in range(4):\n# COLS = FEATS[k]\n\n# for kk in range(4):\n# # COMPUTE PAIR DIFFERENCES\n# x = eeg[COLS[kk]].values - eeg[COLS[kk + 1]].values\n\n# # FILL NANS\n# m = np.nanmean(x)\n# if np.isnan(x).mean() < 1:\n# x = np.nan_to_num(x, nan=m)\n# else:\n# x[:] = 0\n\n# # DENOISE\n# if USE_WAVELET:\n# x = denoise(x, wavelet=USE_WAVELET)\n# signals.append(x)\n\n# # RAW SPECTROGRAM\n# mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x) // 256,\n# n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n\n# # LOG TRANSFORM\n# width = (mel_spec.shape[1] // 32) * 32\n# mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:, :width]\n\n# # STANDARDIZE TO -1 TO 1\n# mel_spec_db = (mel_spec_db + 40) / 40\n# img[:, :, k] += mel_spec_db\n\n# # AVERAGE THE 4 MONTAGE DIFFERENCES\n# img[:, :, k] /= 4.0\n\n# if display:\n# plt.subplot(2, 2, k + 1)\n# plt.imshow(img[:, :, k], aspect='auto', origin='lower')\n# plt.title(f'EEG {eeg_id} - Spectrogram {NAMES[k]}')\n\n# if display:\n# plt.show()\n# plt.figure(figsize=(10, 5))\n# offset = 0\n# for k in range(4):\n# if k > 0:\n# offset -= signals[3 - k].min()\n# plt.plot(range(10_000), signals[k] + offset, label=NAMES[3 - k])\n# offset += signals[3 - k].max()\n# plt.legend()\n# plt.title(f'EEG {eeg_id} Signals')\n# plt.show()\n# print()\n# print('#' * 25)\n# print()\n\n# return img\n\n# # READ ALL EEG SPECTROGRAMS\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n# DISPLAY = 1\n# EEG_IDS2 = test.eeg_id.unique()\n# all_eegs2 = {}\n\n# print('Converting Test EEG to Spectrograms...')\n# print()\n# for i, eeg_id in enumerate(EEG_IDS2):\n# # CREATE SPECTROGRAM FROM EEG PARQUET\n# img = spectrogram_from_eeg(f'{PATH2}{eeg_id}.parquet', i < DISPLAY)\n# all_eegs2[eeg_id] = img\n\n# # INFER VGG16 ON TEST\n# preds = []\n# model = build_model()\n# test_gen = DataGenerator(test, shuffle=False, batch_size=64, mode='test',\n# specs=spectrograms2, eeg_specs=all_eegs2)\n\n# for i in range(5):\n# print(f'Fold {i + 1}')\n# model.load_weights(f'VGG_v{VER}_f{i}.h5')\n# pred = model.predict(test_gen, verbose=1)\n# preds.append(pred)\n# pred = np.mean(preds, axis=0)\n# print()\n# print('Test preds shape', pred.shape)\n\n# sub = pd.DataFrame({'eeg_id': test.eeg_id.values})\n# sub[TARGETS] = pred\n# sub.to_csv('submission.csv', index=False)\n# print('Submission shape', sub.shape)\n# sub.head()\n\n# # SANITY CHECK TO CONFIRM PREDICTIONS SUM TO ONE\n# sub.iloc[:, -6:].sum(axis=1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import os\n# import gc\n# os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0,1\"\n# import tensorflow as tf\n# import pandas as pd\n# import numpy as np\n# import matplotlib.pyplot as plt\n# from tensorflow.keras.applications import VGG16\n# from sklearn.model_selection import GroupKFold\n# from tensorflow.keras import layers\n# import albumentations as albu\n# import pywt\n# import librosa\n# import tensorflow.keras.backend as K\n\n\n# print('TensorFlow version =', tf.__version__)\n\n# # USE MULTIPLE GPUS\n# gpus = tf.config.list_physical_devices('GPU')\n# if len(gpus) <= 1:\n# strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n# print(f'Using {len(gpus)} GPU')\n# else:\n# strategy = tf.distribute.MirroredStrategy()\n# print(f'Using {len(gpus)} GPUs')\n\n# VER = 5\n\n# # IF THIS EQUALS NONE, THEN WE TRAIN NEW MODELS\n# # IF THIS EQUALS DISK PATH, THEN WE LOAD PREVIOUSLY TRAINED MODELS\n# LOAD_MODELS_FROM = None\n\n# USE_KAGGLE_SPECTROGRAMS = True\n# USE_EEG_SPECTROGRAMS = True\n\n# # USE MIXED PRECISION\n# MIX = True\n# if MIX:\n# tf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\n# print('Mixed precision enabled')\n# else:\n# print('Using full precision')\n\n# # Load Train Data\n# df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\n\n# # Define the target columns\n# TARGETS = df.columns[-6:]\n\n# # Check if all target columns exist in the DataFrame\n# missing_targets = [col for col in TARGETS if col not in df.columns]\n# if missing_targets:\n# print(\"Error: Missing target columns in DataFrame:\", missing_targets)\n# # Handle the error, possibly by modifying TARGETS or the DataFrame\n# # For example, you can remove the missing columns from TARGETS or handle them differently\n# # You could also choose to skip the subsequent code or raise an exception\n# else:\n# print(\"All target columns found in DataFrame.\")\n\n# # Continue with the rest of the code if no error occurred\n# print('Train shape:', df.shape)\n# print('Targets', list(TARGETS))\n# df.head()\n\n# # Continue with the rest of the code...\n\n\n# # Create Non-Overlapping Eeg Id Train Data\n# train = df.groupby('eeg_id')[['spectrogram_id', 'spectrogram_label_offset_seconds']].agg(\n# {'spectrogram_id': 'first', 'spectrogram_label_offset_seconds': 'min'})\n# train.columns = ['spec_id', 'min']\n\n# tmp = df.groupby('eeg_id')[['spectrogram_id', 'spectrogram_label_offset_seconds']].agg(\n# {'spectrogram_label_offset_seconds': 'max'})\n# train['max'] = tmp\n\n# tmp = df.groupby('eeg_id')[['patient_id']].agg('first')\n# train['patient_id'] = tmp\n\n# tmp = df.groupby('eeg_id')[TARGETS].agg('sum')\n# for t in TARGETS:\n# train[t] = tmp[t].values\n\n# y_data = train[TARGETS].values\n# y_data = y_data / y_data.sum(axis=1, keepdims=True)\n# train[TARGETS] = y_data\n\n# tmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\n# train['target'] = tmp\n\n# train = train.reset_index()\n# print('Train non-overlapping eeg_id shape:', train.shape)\n# train.head()\n\n# # Read Train Spectrograms\n# READ_SPEC_FILES = False\n# PATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/'\n# files = os.listdir(PATH)\n# print(f'There are {len(files)} spectrogram parquets')\n\n# if READ_SPEC_FILES:\n# spectrograms = {}\n# for i, f in enumerate(files):\n# if i % 100 == 0:\n# print(i, ', ', end='')\n# tmp = pd.read_parquet(f'{PATH}{f}')\n# name = int(f.split('.')[0])\n# spectrograms[name] = tmp.iloc[:, 1:].values\n# else:\n# spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy', allow_pickle=True).item()\n\n# # Read EEG Spectrograms\n# READ_EEG_SPEC_FILES = False\n\n# if READ_EEG_SPEC_FILES:\n# all_eegs = {}\n# for i, e in enumerate(train.eeg_id.values):\n# if i % 100 == 0:\n# print(i, ', ', end='')\n# x = np.load(f'/kaggle/input/brain-eeg-spectrograms/EEG_Spectrograms/{e}.npy')\n# all_eegs[e] = x\n# else:\n# all_eegs = np.load('/kaggle/input/brain-eeg-spectrograms/eeg_specs.npy', allow_pickle=True).item()\n\n# # Train DataLoader\n# TARS = {'Seizure': 0, 'LPD': 1, 'GPD': 2, 'LRDA': 3, 'GRDA': 4, 'Other': 5}\n# TARS2 = {x: y for y, x in TARS.items()}\n\n# class DataGenerator(tf.keras.utils.Sequence):\n# 'Generates data for Keras'\n# def __init__(self, data, batch_size=32, shuffle=False, augment=False, mode='train',\n# specs=spectrograms, eeg_specs=all_eegs):\n\n# self.data = data\n# self.batch_size = batch_size\n# self.shuffle = shuffle\n# self.augment = augment\n# self.mode = mode\n# self.specs = specs\n# self.eeg_specs = eeg_specs\n# self.on_epoch_end()\n\n# def __len__(self):\n# 'Denotes the number of batches per epoch'\n# ct = int(np.ceil(len(self.data) / self.batch_size))\n# return ct\n\n# def __getitem__(self, index):\n# 'Generate one batch of data'\n# indexes = self.indexes[index * self.batch_size:(index + 1) * self.batch_size]\n# X, y = self.__data_generation(indexes)\n# if self.augment:\n# X = self.__augment_batch(X)\n# return X, y\n\n# def on_epoch_end(self):\n# 'Updates indexes after each epoch'\n# self.indexes = np.arange(len(self.data))\n# if self.shuffle:\n# np.random.shuffle(self.indexes)\n\n# def __data_generation(self, indexes):\n# 'Generates data containing batch_size samples'\n\n# X = np.zeros((len(indexes), 128, 256, 8), dtype='float32')\n# y = np.zeros((len(indexes), 6), dtype='float32')\n# img = np.ones((128, 256), dtype='float32')\n\n# for j, i in enumerate(indexes):\n# row = self.data.iloc[i]\n# if self.mode == 'test':\n# r = 0\n# else:\n# r = int((row['min'] + row['max']) // 4)\n\n# for k in range(4):\n# # EXTRACT 300 ROWS OF SPECTROGRAM\n# img = self.specs[row.spec_id][r:r + 300, k * 100:(k + 1) * 100].T\n\n# # LOG TRANSFORM SPECTROGRAM\n# img = np.clip(img, np.exp(-4), np.exp(8))\n# img = np.log(img)\n\n# # STANDARDIZE PER IMAGE\n# ep = 1e-6\n# m = np.nanmean(img.flatten())\n# s = np.nanstd(img.flatten())\n# img = (img - m) / (s + ep)\n# img = np.nan_to_num(img, nan=0.0)\n\n# # CROP TO 256 TIME STEPS\n# X[j, 14:-14, :, k] = img[:, 22:-22] / 2.0\n\n# # EEG SPECTROGRAMS\n# img = self.eeg_specs[row.eeg_id]\n# X[j, :, :, 4:] = img\n\n# if self.mode != 'test':\n# y[j,] = row[TARGETS]\n\n# return X, y\n\n# def __random_transform(self, img):\n# composition = albu.Compose([\n# albu.HorizontalFlip(p=0.5),\n# # albu.CoarseDropout(max_holes=8,max_height=32,max_width=32,fill_value=0,p=0.5),\n# ])\n# return composition(image=img)['image']\n\n# def __augment_batch(self, img_batch):\n# for i in range(img_batch.shape[0]):\n# img_batch[i,] = self.__random_transform(img_batch[i,])\n# return img_batch\n\n# # Display DataLoader\n# gen = DataGenerator(train, batch_size=32, shuffle=False)\n# ROWS = 2\n# COLS = 3\n# BATCHES = 2\n\n# for i, (x, y) in enumerate(gen):\n# plt.figure(figsize=(20, 8))\n# for j in range(ROWS):\n# for k in range(COLS):\n# plt.subplot(ROWS, COLS, j * COLS + k + 1)\n# t = y[j * COLS + k]\n# img = x[j * COLS + k, :, :, 0][::-1, ]\n# mn = img.flatten().min()\n# mx = img.flatten().max()\n# img = (img - mn) / (mx - mn)\n# plt.imshow(img)\n# tars = f'[{t[0]:0.2f}'\n# for s in t[1:]:\n# tars += f', {s:0.2f}'\n# eeg = train.eeg_id.values[i * 32 + j * COLS + k]\n# plt.title(f'EEG = {eeg}\\\\nTarget = {tars}', size=12)\n# plt.yticks([])\n# plt.ylabel('Frequencies (Hz)', size=14)\n# plt.xlabel('Time (sec)', size=16)\n# plt.show()\n# if i == BATCHES - 1:\n# break\n\n# # Train Scheduler\n# import math\n\n# LR_START = 1e-4\n# LR_MAX = 1e-3\n# LR_RAMPUP_EPOCHS = 0\n# LR_SUSTAIN_EPOCHS = 1\n# LR_STEP_DECAY = 0.1\n# EVERY = 1\n# EPOCHS = 4\n\n# def lrfn(epoch):\n# if epoch < LR_RAMPUP_EPOCHS:\n# lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n# elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n# lr = LR_MAX\n# else:\n# lr = LR_MAX * LR_STEP_DECAY ** ((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) // EVERY)\n# return lr\n\n# rng = [i for i in range(EPOCHS)]\n# y = [lrfn(x) for x in rng]\n# plt.figure(figsize=(10, 4))\n# plt.plot(rng, y, 'o-')\n# plt.xlabel('epoch', size=14)\n# plt.ylabel('learning rate', size=14)\n# plt.title('Step Training Schedule', size=16)\n# plt.show()\n\n# LR = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)\n\n# # Build VGG16 Model\n# def build_model():\n# inp = tf.keras.Input(shape=(128, 256, 8))\n# base_model = VGG16(include_top=False, weights='imagenet', input_shape=(512, 512, 3))\n\n# # RESHAPE INPUT 128x256x8 => 512x512x3 MONOTONE IMAGE\n# # KAGGLE SPECTROGRAMS\n# x1 = [inp[:, :, :, i:i+1] for i in range(4)]\n# x1 = tf.keras.layers.Concatenate(axis=1)(x1)\n# # EEG SPECTROGRAMS\n# x2 = [inp[:, :, :, i+4:i+5] for i in range(4)]\n# x2 = tf.keras.layers.Concatenate(axis=1)(x2)\n# # MAKE 512X512X3\n# if USE_KAGGLE_SPECTROGRAMS & USE_EEG_SPECTROGRAMS:\n# x = tf.keras.layers.Concatenate(axis=2)([x1, x2])\n# elif USE_EEG_SPECTROGRAMS:\n# x = x2\n# else:\n# x = x1\n# x = tf.keras.layers.Concatenate(axis=3)([x, x, x])\n\n# # OUTPUT\n# x = base_model(x)\n# x = tf.keras.layers.GlobalAveragePooling2D()(x)\n# x = tf.keras.layers.Dense(6, activation='softmax', dtype='float32')(x)\n\n# # COMPILE MODEL\n# model = tf.keras.Model(inputs=inp, outputs=x)\n# opt = tf.keras.optimizers.Adam(learning_rate=1e-3)\n# loss = tf.keras.losses.KLDivergence()\n\n# model.compile(loss=loss, optimizer=opt)\n\n# return model\n\n# # Train Model\n# all_oof = []\n# all_true = []\n\n# gkf = GroupKFold(n_splits=5)\n# for i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):\n# print('#' * 25)\n# print(f'### Fold {i+1}')\n\n# train_gen = DataGenerator(train.iloc[train_index], shuffle=True, batch_size=32, augment=False)\n# valid_gen = DataGenerator(train.iloc[valid_index], shuffle=False, batch_size=64, mode='valid')\n\n# print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n# print('#' * 25)\n\n# K.clear_session()\n# with strategy.scope():\n# model = build_model()\n# model.fit(train_gen, verbose=1,\n# validation_data=valid_gen,\n# epochs=EPOCHS, callbacks=[LR])\n# model.save_weights(f'VGG_v{VER}_f{i}.h5')\n\n# oof = model.predict(valid_gen, verbose=1)\n# all_oof.append(oof)\n# all_true.append(train.iloc[valid_index].target.values)\n\n# del model\n# gc.collect()\n\n# oof = np.concatenate(all_oof)\n# true = np.concatenate(all_true)\n\n# np.save(f'oof_VGG_v{VER}.npy', oof)\n# np.save(f'true.npy', true)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # DENOISE FUNCTION\n# def maddest(d, axis=None):\n# return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\n# def denoise(x, wavelet='haar', level=1): \n# coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n# sigma = (1/0.6745) * maddest(coeff[-level])\n\n# uthresh = sigma * np.sqrt(2*np.log(len(x)))\n# coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n# ret = pywt.waverec(coeff, wavelet, mode='per')\n \n# return ret\n\n# def spectrogram_from_eeg(parquet_path, display=False):\n \n# # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n# eeg = pd.read_parquet(parquet_path)\n# middle = (len(eeg) - 10_000) // 2\n# eeg = eeg.iloc[middle:middle+10_000]\n \n# # VARIABLE TO HOLD SPECTROGRAM\n# img = np.zeros((128, 256, 4), dtype='float32')\n \n# if display:\n# plt.figure(figsize=(10, 7))\n# signals = []\n# for k in range(4):\n# COLS = FEATS[k]\n \n# for kk in range(4):\n# # COMPUTE PAIR DIFFERENCES\n# x = eeg[COLS[kk]].values - eeg[COLS[kk+1]].values\n\n# # FILL NANS\n# m = np.nanmean(x)\n# if np.isnan(x).mean() < 1:\n# x = np.nan_to_num(x, nan=m)\n# else:\n# x[:] = 0\n\n# # DENOISE\n# if USE_WAVELET:\n# x = denoise(x, wavelet=USE_WAVELET)\n# signals.append(x)\n\n# # RAW SPECTROGRAM\n# mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x) // 256, \n# n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n\n# # LOG TRANSFORM\n# width = (mel_spec.shape[1] // 32) * 32\n# mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:, :width]\n\n# # STANDARDIZE TO -1 TO 1\n# mel_spec_db = (mel_spec_db + 40) / 40 \n# img[:, :, k] += mel_spec_db\n \n# # AVERAGE THE 4 MONTAGE DIFFERENCES\n# img[:, :, k] /= 4.0\n \n# if display:\n# plt.subplot(2, 2, k+1)\n# plt.imshow(img[:, :, k], aspect='auto', origin='lower')\n# plt.title(f'EEG {eeg_id} - Spectrogram {NAMES[k]}')\n \n# if display: \n# plt.show()\n# plt.figure(figsize=(10, 5))\n# offset = 0\n# for k in range(4):\n# if k > 0:\n# offset -= signals[3-k].min()\n# plt.plot(range(10_000), signals[k]+offset, label=NAMES[3-k])\n# offset += signals[3-k].max()\n# plt.legend()\n# plt.title(f'EEG {eeg_id} Signals')\n# plt.show()\n# print()\n# print('#'*25)\n# print()\n \n# return img","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import tensorflow as tf\n# from tensorflow.keras.applications import VGG16\n\n# # Define the build_model function\n# def build_model():\n# inp = tf.keras.Input(shape=(128, 256, 8))\n# base_model = VGG16(include_top=False, weights='imagenet', input_shape=(512, 512, 3))\n\n# # RESHAPE INPUT 128x256x8 => 512x512x3 MONOTONE IMAGE\n# # KAGGLE SPECTROGRAMS\n# x1 = [inp[:, :, :, i:i+1] for i in range(4)]\n# x1 = tf.keras.layers.Concatenate(axis=1)(x1)\n# # EEG SPECTROGRAMS\n# x2 = [inp[:, :, :, i+4:i+5] for i in range(4)]\n# x2 = tf.keras.layers.Concatenate(axis=1)(x2)\n# # MAKE 512X512X3\n# if USE_KAGGLE_SPECTROGRAMS & USE_EEG_SPECTROGRAMS:\n# x = tf.keras.layers.Concatenate(axis=2)([x1, x2])\n# elif USE_EEG_SPECTROGRAMS:\n# x = x2\n# else:\n# x = x1\n# x = tf.keras.layers.Concatenate(axis=3)([x, x, x])\n\n# # OUTPUT\n# x = base_model(x)\n# x = tf.keras.layers.GlobalAveragePooling2D()(x)\n# x = tf.keras.layers.Dense(6, activation='softmax', dtype='float32')(x)\n\n# # COMPILE MODEL\n# model = tf.keras.Model(inputs=inp, outputs=x)\n# opt = tf.keras.optimizers.Adam(learning_rate=1e-3)\n# loss = tf.keras.losses.KLDivergence()\n\n# model.compile(loss=loss, optimizer=opt)\n\n# return model\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# print(train.columns)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import os\n# import pandas as pd\n# import numpy as np\n# import pywt\n# import librosa\n# import matplotlib.pyplot as plt\n# import tensorflow as tf\n\n# # DENOISE FUNCTION\n# def maddest(d, axis=None):\n# return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\n# def denoise(x, wavelet='haar', level=1): \n# coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n# sigma = (1/0.6745) * maddest(coeff[-level])\n\n# uthresh = sigma * np.sqrt(2*np.log(len(x)))\n# coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n# ret = pywt.waverec(coeff, wavelet, mode='per')\n \n# return ret\n\n# def spectrogram_from_eeg(parquet_path, display=False):\n \n# # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n# eeg = pd.read_parquet(parquet_path)\n# middle = (len(eeg) - 10_000) // 2\n# eeg = eeg.iloc[middle:middle+10_000]\n \n# # VARIABLE TO HOLD SPECTROGRAM\n# img = np.zeros((128, 256, 4), dtype='float32')\n \n# if display:\n# plt.figure(figsize=(10, 7))\n# signals = []\n# for k in range(4):\n# COLS = FEATS[k]\n \n# for kk in range(4):\n# # COMPUTE PAIR DIFFERENCES\n# x = eeg[COLS[kk]].values - eeg[COLS[kk+1]].values\n\n# # FILL NANS\n# m = np.nanmean(x)\n# if np.isnan(x).mean() < 1:\n# x = np.nan_to_num(x, nan=m)\n# else:\n# x[:] = 0\n\n# # DENOISE\n# if USE_WAVELET:\n# x = denoise(x, wavelet=USE_WAVELET)\n# signals.append(x)\n\n# # RAW SPECTROGRAM\n# mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x) // 256, \n# n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n\n# # LOG TRANSFORM\n# width = (mel_spec.shape[1] // 32) * 32\n# mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:, :width]\n\n# # STANDARDIZE TO -1 TO 1\n# mel_spec_db = (mel_spec_db + 40) / 40 \n# img[:, :, k] += mel_spec_db\n \n# # AVERAGE THE 4 MONTAGE DIFFERENCES\n# img[:, :, k] /= 4.0\n \n# if display:\n# plt.subplot(2, 2, k+1)\n# plt.imshow(img[:, :, k], aspect='auto', origin='lower')\n# plt.title(f'Spectrogram {NAMES[k]}')\n \n# if display: \n# plt.show()\n# plt.figure(figsize=(10, 5))\n# offset = 0\n# for k in range(4):\n# if k > 0:\n# offset -= signals[3-k].min()\n# plt.plot(range(10_000), signals[k]+offset, label=NAMES[3-k])\n# offset += signals[3-k].max()\n# plt.legend()\n# plt.title('Signals')\n# plt.show()\n# print()\n# print('#'*25)\n# print()\n \n# return img\n\n# # Define necessary variables\n# FEATS = [['Fp1', 'F7', 'T3', 'T5', 'O1'],\n# ['Fp1', 'F3', 'C3', 'P3', 'O1'],\n# ['Fp2', 'F8', 'T4', 'T6', 'O2'],\n# ['Fp2', 'F4', 'C4', 'P4', 'O2']]\n\n# NAMES = ['LL', 'LP', 'RP', 'RR']\n\n# USE_WAVELET = None\n\n# # Load the test data\n# test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\n# print('Test shape', test.shape)\n# test.head()\n# test = test.rename({'spectrogram_id': 'spec_id'}, axis=1)\n\n\n# # Generate spectrograms from EEG signals in the test data\n# import pywt, librosa\n\n# # Set DISPLAY to control the number of EEG spectrograms to display\n# DISPLAY = 1\n\n# # Read test spectrograms\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\n# files2 = os.listdir(PATH2)\n# print(f'There are {len(files2)} test spectrogram parquets')\n\n# spectrograms2 = {}\n# for i, f in enumerate(files2):\n# if i % 100 == 0:\n# print(i, ', ', end='')\n# tmp = pd.read_parquet(os.path.join(PATH2, f)) # Use os.path.join to construct the file path\n# name = int(f.split('.')[0])\n# spectrograms2[name] = tmp.iloc[:, 1:].values\n\n# # Generate EEG spectrograms\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n# EEG_IDS2 = test.eeg_id.unique()\n# all_eegs2 = {}\n\n# print('Converting Test EEG to Spectrograms...')\n# print()\n# for i, eeg_id in enumerate(EEG_IDS2):\n# img = spectrogram_from_eeg(os.path.join(PATH2, f'{eeg_id}.parquet'), i < DISPLAY) # Use os.path.join to construct the file path\n# all_eegs2[eeg_id] = img\n\n# # Perform model inference on the test data\n# preds = []\n# model = build_model() # Define this function to build your model\n# test_gen = DataGenerator(test, shuffle=False, batch_size=64, mode='test',\n# specs=spectrograms2, eeg_specs=all_eegs2)\n\n# for i in range(5):\n# print(f'Fold {i+1}')\n# model.load_weights(f'/kaggle/working/VGG_v5_f{i}.h5')\n# pred = model.predict(test_gen, verbose=1)\n# preds.append(pred)\n\n# pred = np.mean(preds, axis=0)\n# print()\n# print('Test preds shape', pred.shape)\n\n# # Submission Generation\n# sub = pd.DataFrame({'eeg_id': test.eeg_id.values})\n# sub[TARGETS] = pred\n# sub.to_csv('submission.csv', index=False)\n# print('Submission shape', sub.shape)\n# sub.head()\n\n# # Sanity Check\n# print('Prediction sum:', sub.iloc[:, -6:].sum(axis=1).mean())\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import os\n# import pandas as pd\n# import numpy as np\n# import pywt\n# import librosa\n# import matplotlib.pyplot as plt\n# import tensorflow as tf\n\n# # Define DENOISE FUNCTION\n# def maddest(d, axis=None):\n# return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\n# def denoise(x, wavelet='haar', level=1): \n# coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n# sigma = (1/0.6745) * maddest(coeff[-level])\n\n# uthresh = sigma * np.sqrt(2*np.log(len(x)))\n# coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n# ret = pywt.waverec(coeff, wavelet, mode='per')\n \n# return ret\n\n# # Function to generate spectrograms from EEG signals\n# def spectrogram_from_eeg(parquet_path, display=False):\n \n# # Load middle 50 seconds of EEG series\n# eeg = pd.read_parquet(parquet_path)\n# middle = (len(eeg) - 10_000) // 2\n# eeg = eeg.iloc[middle:middle+10_000]\n \n# # Variable to hold spectrogram\n# img = np.zeros((128, 256, 4), dtype='float32')\n \n# if display:\n# plt.figure(figsize=(10, 7))\n# signals = []\n# for k in range(4):\n# COLS = FEATS[k]\n \n# for kk in range(4):\n# # Compute pair differences\n# x = eeg[COLS[kk]].values - eeg[COLS[kk+1]].values\n\n# # Fill NaNs\n# m = np.nanmean(x)\n# if np.isnan(x).mean() < 1:\n# x = np.nan_to_num(x, nan=m)\n# else:\n# x[:] = 0\n\n# # Denoise\n# if USE_WAVELET:\n# x = denoise(x, wavelet=USE_WAVELET)\n# signals.append(x)\n\n# # Raw spectrogram\n# mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x) // 256, \n# n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n\n# # Log transform\n# width = (mel_spec.shape[1] // 32) * 32\n# mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:, :width]\n\n# # Standardize to -1 to 1\n# mel_spec_db = (mel_spec_db + 40) / 40 \n# img[:, :, k] += mel_spec_db\n \n# # Average the 4 montage differences\n# img[:, :, k] /= 4.0\n \n# if display:\n# plt.subplot(2, 2, k+1)\n# plt.imshow(img[:, :, k], aspect='auto', origin='lower')\n# plt.title(f'Spectrogram {NAMES[k]}')\n \n# if display: \n# plt.show()\n# plt.figure(figsize=(10, 5))\n# offset = 0\n# for k in range(4):\n# if k > 0:\n# offset -= signals[3-k].min()\n# plt.plot(range(10_000), signals[k]+offset, label=NAMES[3-k])\n# offset += signals[3-k].max()\n# plt.legend()\n# plt.title('Signals')\n# plt.show()\n# print()\n# print('#'*25)\n# print()\n \n# return img\n\n# # Define necessary variables\n# FEATS = [['Fp1', 'F7', 'T3', 'T5', 'O1'],\n# ['Fp1', 'F3', 'C3', 'P3', 'O1'],\n# ['Fp2', 'F8', 'T4', 'T6', 'O2'],\n# ['Fp2', 'F4', 'C4', 'P4', 'O2']]\n\n# NAMES = ['LL', 'LP', 'RP', 'RR']\n\n# USE_WAVELET = None\n\n# # Load the test data\n# test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\n# print('Test shape', test.shape)\n# test.head()\n# test = test.rename({'spectrogram_id': 'spec_id'}, axis=1)\n\n# # Generate spectrograms from EEG signals in the test data\n# import pywt, librosa\n\n# # Set DISPLAY to control the number of EEG spectrograms to display\n# DISPLAY = 1\n\n# # Read test spectrograms\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\n# files2 = os.listdir(PATH2)\n# print(f'There are {len(files2)} test spectrogram parquets')\n\n# spectrograms2 = {}\n# for i, f in enumerate(files2):\n# if i % 100 == 0:\n# print(i, ', ', end='')\n# tmp = pd.read_parquet(os.path.join(PATH2, f)) # Use os.path.join to construct the file path\n# name = int(f.split('.')[0])\n# spectrograms2[name] = tmp.iloc[:, 1:].values\n\n# # Generate EEG spectrograms\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n# EEG_IDS2 = test.eeg_id.unique()\n# all_eegs2 = {}\n\n# print('Converting Test EEG to Spectrograms...')\n# print()\n# for eeg_id in EEG_IDS2:\n# img = spectrogram_from_eeg(os.path.join(PATH2, f'{eeg_id}.parquet'), eeg_id == EEG_IDS2[0]) # Use os.path.join to construct the file path\n# all_eegs2[eeg_id] = img\n\n# # Perform model inference on the test data\n# preds = []\n# model = build_model() # Define this function to build your model\n# test_gen = DataGenerator(test, shuffle=False, batch_size=64, mode='test', specs=spectrograms2, eeg_specs=all_eegs2)\n\n# for i in range(5):\n# print(f'Fold {i}')\n# model.load_weights(f'/kaggle/working/VGG_v5_f{i}.h5')\n# pred = model.predict(test_gen, verbose=1)\n# preds.append(pred)\n\n# # Average predictions from different folds\n# pred = np.mean(preds, axis=0)\n# print()\n# print('Test preds shape', pred.shape)\n\n# # Create submission dataframe\n# sub = pd.DataFrame({'eeg_id': test.eeg_id.values})\n# TARGETS = ['Seizure', 'LPD', 'GPD', 'LRDA', 'GRDA', 'Other']\n# for i, target in enumerate(TARGETS):\n# sub[target] = pred[0][i]\n\n# # Display the submission dataframe\n# print('Submission shape', sub.shape)\n# print(sub.head())\n\n# # Sanity Check\n# print('Prediction sum:', sub.iloc[:, -6:].sum(axis=1).mean())\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Perform model inference on the test data\n# preds = []\n# model = build_model() # Define this function to build your model\n# test_gen = DataGenerator(test, shuffle=False, batch_size=64, mode='test', specs=spectrograms2, eeg_specs=all_eegs2)\n\n# for i in range(5):\n# print(f'Fold {i}')\n# model.load_weights(f'/kaggle/working/VGG_v5_f{i}.h5')\n# pred = model.predict(test_gen, verbose=1)\n# preds.append(pred)\n\n# # Average predictions from different folds\n# pred = np.mean(preds, axis=0)\n# print()\n# print('Test preds shape', pred.shape)\n\n# # Extract predicted values\n# predicted_values = pred[0]\n# print('Predicted values:', predicted_values)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rng = [i for i in range(EPOCHS2)]\nlr_y = [lrfn(x) for x in rng]\nplt.figure(figsize=(10, 4))\nplt.plot(rng, lr_y, '-o')\nplt.xlabel('epoch', size=14)\nplt.ylabel('learning rate', size=14)\nplt.title('Cosine Training Schedule', size=16)\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix\nimport seaborn as sns\ntrue_labels=['eeg_id', 'spec_id', 'min', 'max', 'patient_id', 'seizure_vote',\n 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote',\n 'target']\npredicted_labels=['eeg_id', 'spec_id', 'min', 'max', 'patient_id', 'seizure_vote',\n 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote',\n 'target']\n\n# Assuming `true_labels` and `predicted_labels` are available\nconf_matrix = confusion_matrix(true_labels, predicted_labels)\nplt.figure(figsize=(8, 6))\nsns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues')\nplt.xlabel('Predicted Labels')\nplt.ylabel('True Labels')\nplt.title('Confusion Matrix')\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import roc_curve, auc\n\n# Assuming `true_labels` and `predicted_probabilities` for binary classification\nfpr, tpr, _ = roc_curve(true_labels, predicted_probabilities)\nroc_auc = auc(fpr, tpr)\n\nplt.figure(figsize=(8, 6))\nplt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('Receiver Operating Characteristic')\nplt.legend(loc=\"lower right\")\nplt.show()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nimport gc\nimport tensorflow as tf\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport albumentations as albu\nimport math\nimport pywt\nimport librosa\nfrom sklearn.model_selection import GroupKFold\nfrom tensorflow.keras.applications import InceptionV3\n\nprint('TensorFlow version =', tf.__version__)\n\n# Set CUDA_VISIBLE_DEVICES\nos.environ[\"CUDA_VISIBLE_DEVICES\"] = \"0,1\"\n\n# Check GPU availability and decide strategy\ngpus = tf.config.list_physical_devices('GPU')\nif len(gpus) <= 1:\n strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n print(f'Using {len(gpus)} GPU')\nelse:\n strategy = tf.distribute.MirroredStrategy()\n print(f'Using {len(gpus)} GPUs')\n\nVER = 5\n\n# Check if mixed precision is enabled\nMIX = True\nif MIX:\n tf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\n print('Mixed precision enabled')\nelse:\n print('Using full precision')\n\n# Load and preprocess data\ndf = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape)\nprint('Targets', list(TARGETS))\n\ntrain = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n train[t] = tmp[t].values\n \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\nprint('Train non-overlapp eeg_id shape:', train.shape)\n\n# Define DataGenerator class\nclass DataGenerator(tf.keras.utils.Sequence):\n 'Generates data for Keras'\n def __init__(self, data, batch_size=32, shuffle=False, augment=False, mode='train',\n specs=spectrogram_id, eeg_specs=eegs_id): \n\n self.data = data\n self.batch_size = batch_size\n self.shuffle = shuffle\n self.augment = augment\n self.mode = mode\n self.specs = specs\n self.eeg_specs = eeg_specs\n self.on_epoch_end()\n \n def __len__(self):\n 'Denotes the number of batches per epoch'\n ct = int(np.ceil(len(self.data) / self.batch_size))\n return ct\n\n def __getitem__(self, index):\n 'Generate one batch of data'\n indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n X, y = self.__data_generation(indexes)\n if self.augment:\n X = self.__augment_batch(X) \n return X, y\n\n def on_epoch_end(self):\n 'Updates indexes after each epoch'\n self.indexes = np.arange(len(self.data))\n if self.shuffle:\n np.random.shuffle(self.indexes)\n \n def __data_generation(self, indexes):\n 'Generates data containing batch_size samples' \n \n X = np.zeros((len(indexes), 128, 256, 8), dtype='float32')\n y = np.zeros((len(indexes), 6), dtype='float32')\n img = np.ones((128, 256), dtype='float32')\n \n for j, i in enumerate(indexes):\n row = self.data.iloc[i]\n if self.mode == 'test': \n r = 0\n else: \n r = int((row['min'] + row['max']) // 4)\n\n for k in range(4):\n # Extract 300 rows of spectrogram\n img = self.specs[row.spec_id][r:r+300, k*100:(k+1)*100].T\n \n # Log transform spectrogram\n img = np.clip(img, np.exp(-4), np.exp(8))\n img = np.log(img)\n \n # Standardize per image\n ep = 1e-6\n m = np.nanmean(img.flatten())\n s = np.nanstd(img.flatten())\n img = (img - m) / (s + ep)\n img = np.nan_to_num(img, nan=0.0)\n \n # Crop to 256 time steps\n X[j, 14:-14, :, k] = img[:, 22:-22] / 2.0\n \n # EEG spectrograms\n img = self.eeg_specs[row.eeg_id]\n X[j, :, :, 4:] = img\n \n if self.mode != 'test':\n y[j,] = row[TARGETS]\n \n return X, y\n \n def __random_transform(self, img):\n composition = albu.Compose([\n albu.HorizontalFlip(p=0.5),\n #albu.CoarseDropout(max_holes=8,max_height=32,max_width=32,fill_value=0,p=0.5),\n ])\n return composition(image=img)['image']\n \n def __augment_batch(self, img_batch):\n for i in range(img_batch.shape[0]):\n img_batch[i, ] = self.__random_transform(img_batch[i, ])\n return img_batch\n\n# Define learning rate schedules\ndef cosine_decay_with_warmup(epoch, lr):\n \"\"\"Cosine decay schedule with warm up.\"\"\"\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n decay_total_epochs = EPOCHS2 - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS - 1\n decay_epoch_index = epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS\n phase = math.pi * decay_epoch_index / decay_total_epochs\n cosine_decay = 0.5 * (1 + math.cos(phase))\n lr = (LR_MAX - LR_MIN) * cosine_decay + LR_MIN\n return lr\n\ndef step_decay(epoch, lr):\n \"\"\"Step decay schedule.\"\"\"\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) // EVERY)\n return lr\n\n# Build model using InceptionV3\ndef build_model():\n \n inp = tf.keras.Input(shape=(128, 256, 8))\n base_model = InceptionV3(include_top=False, weights='imagenet', input_shape=None)\n \n # Reshape input 128x256x8 => 512x512x3 monotone image\n # Kaggle spectrograms\n x1 = [inp[:,:,:,i:i+1] for i in range(4)]\n x1 = tf.keras.layers.Concatenate(axis=1)(x1)\n # EEG spectrograms\n x2 = [inp[:,:,:,i+4:i+5] for i in range(4)]\n x2 = tf.keras.layers.Concatenate(axis=1)(x2)\n # Make 512x512x3\n if USE_KAGGLE_SPECTROGRAMS & USE_EEG_SPECTROGRAMS:\n x = tf.keras.layers.Concatenate(axis=2)([x1,x2])\n elif USE_EEG_SPECTROGRAMS: \n x = x2\n else: \n x = x1\n x = tf.keras.layers.Concatenate(axis=3)([x,x,x])\n \n # Output\n x = base_model(x)\n x = tf.keras.layers.GlobalAveragePooling2D()(x)\n x = tf.keras.layers.Dense(6, activation='softmax', dtype='float32')(x)\n \n # Compile model\n model = tf.keras.Model(inputs=inp, outputs=x)\n opt = tf.keras.optimizers.Adam(learning_rate=1e-3)\n loss = tf.keras.losses.KLDivergence()\n\n model.compile(loss=loss, optimizer=opt) \n \n return model\n\n# GroupKFold splitting\nall_oof = []\nall_true = []\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)): \n \n print('#'*25)\n print(f'### Fold {i+1}')\n \n train_gen = DataGenerator(train.iloc[train_index], shuffle=True, batch_size=32, augment=False)\n valid_gen = DataGenerator(train.iloc[valid_index], shuffle=False, batch_size=64, mode='valid')\n \n print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n print('#'*25)\n \n K.clear_session()\n with strategy.scope():\n model = build_model()\n \n if LOAD_MODELS_FROM is None:\n model.fit(train_gen, verbose=1,\n validation_data=valid_gen,\n epochs=EPOCHS, callbacks=[LR])\n model.save_weights(f'Inception_v{VER}_f{i}.h5')\n else:\n model.load_weights(f'{LOAD_MODELS_FROM}Inception_v{VER}_f{i}.h5')\n \n oof = model.predict(valid_gen, verbose=1)\n all_oof.append(oof)\n all_true.append(train.iloc[valid_index][TARGETS].values)\n \n del model, oof\n gc.collect()\n \nall_oof = np.concatenate(all_oof)\nall_true = np.concatenate(all_true)\n\n# Evaluate using KL-Div\nimport sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score KL-Div for InceptionV3 =', cv)\n\n# Inference on test data\ntest = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\nprint('Test shape', test.shape)\n\n# Read all test spectrograms\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\nfiles2 = os.listdir(PATH2)\nprint(f'There are {len(files2)} test spectrogram parquets')\n \nspectrograms2 = {}\nfor i, f in enumerate(files2):\n if i % 100 == 0: \n print(i, ', ', end='')\n tmp = pd.read_parquet(f'{PATH2}{f}')\n name = int(f.split('.')[0])\n spectrograms2[name] = tmp.iloc[:, 1:].values\n\n# Rename for DataLoader\ntest = test.rename({'spectrogram_id': 'spec_id'}, axis=1)\n\n# Read all test EEG spectrograms\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\nDISPLAY = 1\nEEG_IDS2 = test.eeg_id.unique()\nall_eegs2 = {}\n\nprint('Converting Test EEG to Spectrograms...')\nprint()\nfor i, eeg_id in enumerate(EEG_IDS2):\n \n # Create spectrogram from EEG parquet\n img = spectrogram_from_eeg(f'{PATH2}{eeg_id}.parquet', i < DISPLAY)\n all_eegs2[eeg_id] = img\n\n# Inference\npreds = []\nmodel = build_model()\ntest_gen = DataGenerator(test, shuffle=False, batch_size=64, mode='test',\n specs=spectrograms2, eeg_specs=all_eegs2)\n\nfor i in range(5):\n print(f'Fold {i+1}')\n if LOAD_MODELS_FROM:\n model.load_weights(f'{LOAD_MODELS_FROM}Inception_v{VER}_f{i}.h5')\n else:\n model.load_weights(f'Inception_v{VER}_f{i}.h5')\n pred = model.predict(test_gen, verbose=1)\n preds.append(pred)\npred = np.mean(preds, axis=0)\nprint()\nprint('Test preds shape', pred.shape)\n\n# Generate submission\nsub = pd.DataFrame({'eeg_id': test.eeg_id.values})\nsub[TARGETS] = pred\nsub.to_csv('submission.csv', index=False)\nprint('Submission shape', sub.shape)\nsub.head()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}