{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7465251,"sourceType":"datasetVersion","datasetId":4317718}],"dockerImageVersionId":30636,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# WaveNet Starter using RAW EEG Features!\nThis notebook is a WaveNet starter for Kaggle's Brain comp. It achieves `CV 0.91` and `LB 0.66`. Note that submitting train means achieves `CV 1.26` and `LB 0.97` [[here][1]]. So this notebook's WaveNet is successfully learning to predict brain events from raw EEG waveforms!\n\nThis model only uses two features. We can engineer more features and/or modify the model architecture to improve CV score and LB score. Furthermore we can build 1 model which inputs both spectrogram images and eeg waveforms. The two EEG features in this notebook are:\n* feature 1 : `Fp1 minus O1`\n* feature 2 : `Fp2 minus O2`\n\nFeature 1 is the beginning of the montage chains `LL` and `LP` minus the ending of montage `LL` and `LP`. And feature 2 is the beginning of the montage chains `RL` and `RP` minus the ending of montage `RL` and `RP`.\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/montage.png)\n\n# UPDATE\nIn version 7 and 8, we add more features and update the model architecture to evaluate each montage chain separately and then concatenate the features. This new architecture is motivated by the discovery of a better formula to utilize EEG explained in discussion [here][2]\n\n* Version 5,6: Use 2 features - CV 0.91 LB 0.66\n* Version 7,8: Use 8 features grouped as 4 chains. Downsample time 5x - **CV 0.81 LB 0.53**, wow!\n\nWe train our new model in version 7, then save model weights. Then load model into version 8 to submit to LB.\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave-model.png)\n\n# Train and Infer Tricks\nWe train the fold models in version 7 of the notebook and submit to Kaggle LB in version 8 of the notebook. This makes submission faster because we train the fold models for 30 minutes in version 7 then save them. In version 8, we just load the models without needing to retrain models during Kaggle submit. (And we train our old model in 5 annd submit in 6).\n\nVersion 4 uses `1xP100` GPU with full precision and takes 1 hour to train 5 folds 5 epochs of WaveNet. Version 5 uses `2xT4` GPU with mixed precision and takes 30 minutes to train 5 folds 5 epochs of WaveNet. \n\n[1]: https://www.kaggle.com/code/seshurajup/eda-train-csv\n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469760","metadata":{}},{"cell_type":"markdown","source":"# Load Train Data","metadata":{}},{"cell_type":"code","source":"import pandas as pd, numpy as np, os\nimport matplotlib.pyplot as plt\n\ntrain = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nprint( train.shape )\ndisplay( train.head() )\n\n# CHOICE TO CREATE OR LOAD EEGS FROM NOTEBOOK VERSION 1\nCREATE_EEGS = False\nTRAIN_MODEL = True","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:42:21.845444Z","iopub.execute_input":"2024-07-02T08:42:21.845986Z","iopub.status.idle":"2024-07-02T08:42:23.025644Z","shell.execute_reply.started":"2024-07-02T08:42:21.845959Z","shell.execute_reply":"2024-07-02T08:42:23.024700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Raw EEG Features","metadata":{}},{"cell_type":"code","source":"df = pd.read_parquet('/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/1000913311.parquet')\nFEATS = df.columns\nprint(f'There are {len(FEATS)} raw eeg features')\nprint( list(FEATS) )\n","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:42:23.027507Z","iopub.execute_input":"2024-07-02T08:42:23.028281Z","iopub.status.idle":"2024-07-02T08:42:23.268686Z","shell.execute_reply.started":"2024-07-02T08:42:23.028245Z","shell.execute_reply":"2024-07-02T08:42:23.267792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('We will use the following subset of raw EEG features:')\nFEATS = ['Fp1','T3','C3','O1','Fp2','C4','T4','O2']\nFEAT2IDX = {x:y for x,y in zip(FEATS,range(len(FEATS)))}\nprint( list(FEATS) )","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:42:23.269969Z","iopub.execute_input":"2024-07-02T08:42:23.270308Z","iopub.status.idle":"2024-07-02T08:42:23.276857Z","shell.execute_reply.started":"2024-07-02T08:42:23.270274Z","shell.execute_reply":"2024-07-02T08:42:23.275881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def eeg_from_parquet(parquet_path, display=False):\n    \n    # EXTRACT MIDDLE 50 SECONDS\n    eeg = pd.read_parquet(parquet_path, columns=FEATS)\n    rows = len(eeg)\n    offset = (rows-10_000)//2\n    eeg = eeg.iloc[offset:offset+10_000]\n    \n    if display: \n        plt.figure(figsize=(10,5))\n        offset = 0\n    \n    # CONVERT TO NUMPY\n    data = np.zeros((10_000,len(FEATS)))\n    for j,col in enumerate(FEATS):\n        \n        # FILL NAN\n        x = eeg[col].values.astype('float32')\n        m = np.nanmean(x)\n        if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n        else: x[:] = 0\n            \n        data[:,j] = x\n        \n        if display: \n            if j!=0: offset += x.max()\n            plt.plot(range(10_000),x-offset,label=col)\n            offset -= x.min()\n            \n    if display:\n        plt.legend()\n        name = parquet_path.split('/')[-1]\n        name = name.split('.')[0]\n        plt.title(f'EEG {name}',size=16)\n        plt.show()\n        \n    return data","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:42:23.278961Z","iopub.execute_input":"2024-07-02T08:42:23.279285Z","iopub.status.idle":"2024-07-02T08:42:23.289562Z","shell.execute_reply.started":"2024-07-02T08:42:23.279251Z","shell.execute_reply":"2024-07-02T08:42:23.288634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n\n# all_eegs = {}\n# DISPLAY = 4\n# EEG_IDS = train.eeg_id.unique()\n# PATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/'\n\n# for i,eeg_id in enumerate(EEG_IDS):\n#     if (i%100==0)&(i!=0): print(i,', ',end='') \n    \n#     # SAVE EEG TO PYTHON DICTIONARY OF NUMPY ARRAYS\n#     data = eeg_from_parquet(f'{PATH}{eeg_id}.parquet', display=i<DISPLAY)              \n#     all_eegs[eeg_id] = data\n    \n#     if i==DISPLAY:\n#         if CREATE_EEGS:\n#             print(f'Processing {train.eeg_id.nunique()} eeg parquets... ',end='')\n#         else:\n#             print(f'Reading {len(EEG_IDS)} eeg NumPys from disk.')\n#             break\n            \n# if CREATE_EEGS: \n#     np.save('eegs',all_eegs)\n# else:\nall_eegs = np.load('/kaggle/input/brain-eegs/eegs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:42:23.290837Z","iopub.execute_input":"2024-07-02T08:42:23.291180Z","iopub.status.idle":"2024-07-02T08:44:37.906199Z","shell.execute_reply.started":"2024-07-02T08:42:23.291155Z","shell.execute_reply":"2024-07-02T08:44:37.905185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Deduplicate Train EEG Id","metadata":{}},{"cell_type":"code","source":"# LOAD TRAIN \nDISPLAY = 4\nEEG_IDS = train.eeg_id.unique()\ndf = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nTARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\nTARS2 = {x:y for y,x in TARS.items()}\n\ntrain = df.groupby('eeg_id')[['patient_id']].agg('first')\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n    train[t] = tmp[t].values\n    \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\ntrain = train.loc[train.eeg_id.isin(EEG_IDS)]\nprint('Train Data with unique eeg_id shape:', train.shape )\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:44:37.907615Z","iopub.execute_input":"2024-07-02T08:44:37.908077Z","iopub.status.idle":"2024-07-02T08:44:38.170294Z","shell.execute_reply.started":"2024-07-02T08:44:37.908042Z","shell.execute_reply":"2024-07-02T08:44:38.169426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n# Butter Low-Pass Filter","metadata":{}},{"cell_type":"code","source":"# from scipy.signal import butter, lfilter\n\n# def butter_lowpass_filter(data, cutoff_freq=20, sampling_rate=200, order=4):\n#     nyquist = 0.5 * sampling_rate\n#     normal_cutoff = cutoff_freq / nyquist\n#     b, a = butter(order, normal_cutoff, btype='low', analog=False)\n#     filtered_data = lfilter(b, a, data, axis=0)\n#     return filtered_data","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:44:38.171428Z","iopub.execute_input":"2024-07-02T08:44:38.171687Z","iopub.status.idle":"2024-07-02T08:44:38.175648Z","shell.execute_reply.started":"2024-07-02T08:44:38.171664Z","shell.execute_reply":"2024-07-02T08:44:38.174803Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# FREQS = [1,2,4,8,16][::-1]\n# x = [all_eegs[EEG_IDS[0]][:,0]]\n\n# for k in FREQS:\n#     x.append( butter_lowpass_filter(x[0], cutoff_freq=k) )\n\n# plt.figure(figsize=(20,20))\n# plt.plot(range(10_000),x[0], label='without filter')\n# for k in range(1,len(x)):\n#     plt.plot(range(10_000),x[k]-k*(x[0].max()-x[0].min()), label=f'with filter {FREQS[k-1]}Hz')\n# plt.legend()\n# plt.title('Butter Low-Pass Filter Examples',size=18)\n# plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:44:38.176935Z","iopub.execute_input":"2024-07-02T08:44:38.177265Z","iopub.status.idle":"2024-07-02T08:44:38.185533Z","shell.execute_reply.started":"2024-07-02T08:44:38.177236Z","shell.execute_reply":"2024-07-02T08:44:38.184722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom scipy.signal import stft, spectrogram, welch, butter, lfilter\nimport matplotlib.colors\n\n# # Assuming all_eegs is already loaded\n# eeg_data = all_eegs[EEG_IDS[0]]  # Use the first EEG signal for this example\n# # Use the first channel of the EEG data\n# x = eeg_data[:, 0]\n\n# # Apply the low-pass filter\n# filtered_x = butter_lowpass_filter(x)\n# plt.plot(filtered_x)\n\n# #spectrogram calc\n# f, t, Sxx = spectrogram(np.array(filtered_x), fs=200, nperseg=200, noverlap=int(200*0.8), scaling='density', return_onesided=True)\n# Sxx_db = 10*np.log10(Sxx)\n\n# plt.figure(figsize=(10, 6))\n# plt.pcolormesh(Sxx_db, cmap='jet')# Plot the resultplt.title('STFT Magnitude of Filtered EEG Signal (dB)')\n# plt.ylabel('Frequency [Hz]')\n# plt.xlabel('Time [sec]')\n# plt.colorbar(label='Magnitude (dB)')\n\n","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:44:38.186575Z","iopub.execute_input":"2024-07-02T08:44:38.186853Z","iopub.status.idle":"2024-07-02T08:44:38.565473Z","shell.execute_reply.started":"2024-07-02T08:44:38.186829Z","shell.execute_reply":"2024-07-02T08:44:38.564671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import numpy as np\n# import matplotlib.pyplot as plt\n# from scipy.signal import spectrogram, butter, lfilter\n# import matplotlib\n\n# # Function for low-pass filtering\n# def butter_lowpass_filter(data, cutoff_freq=20, sampling_rate=200, order=4):\n#     nyquist = 0.5 * sampling_rate\n#     normal_cutoff = cutoff_freq / nyquist\n#     b, a = butter(order, normal_cutoff, btype='low', analog=False)\n#     filtered_data = lfilter(b, a, data, axis=0)\n#     return filtered_data\n\n# # Assuming all_eegs is already loaded\n# eeg_data = all_eegs[EEG_IDS[0]]  # Use the first EEG signal for this example\n\n# # Use the first channel of the EEG data\n# filtered_eeg_data = butter_lowpass_filter(eeg_data)\n# FEATS = ['Fp1', 'T3', 'C3', 'O1', 'Fp2', 'C4', 'T4', 'O2']\n# FEAT2IDX = {x: y for y, x in enumerate(FEATS)}\n\n# sample = np.zeros((filtered_eeg_data.shape[0], 8))\n# sample[:,0] = eeg_data[:,FEAT2IDX['Fp1']] - eeg_data[:,FEAT2IDX['T3']]\n# sample[:,1] = eeg_data[:,FEAT2IDX['T3']] - eeg_data[:,FEAT2IDX['O1']]\n# sample[:,2] = eeg_data[:,FEAT2IDX['Fp1']] - eeg_data[:,FEAT2IDX['C3']]\n# sample[:,3] = eeg_data[:,FEAT2IDX['C3']] - eeg_data[:,FEAT2IDX['O1']]\n# sample[:,4] = eeg_data[:,FEAT2IDX['Fp2']] - eeg_data[:,FEAT2IDX['C4']]\n# sample[:,5] = eeg_data[:,FEAT2IDX['C4']] - eeg_data[:,FEAT2IDX['O2']]\n# sample[:,6] = eeg_data[:,FEAT2IDX['Fp2']] - eeg_data[:,FEAT2IDX['T4']]\n# sample[:,7] = eeg_data[:,FEAT2IDX['T4']] - eeg_data[:,FEAT2IDX['O2']]\n\n# # Function to plot spectrogram\n# def plot_spectrogram(data, title, ax):\n#     f, t, Sxx = spectrogram(data, fs=200, nperseg=200, noverlap=int(200 * 0.8), scaling='density', return_onesided=True)\n#     Sxx_db = 10 * np.log10(np.abs(Sxx)**2 + 1e-10)  # Add a small value to avoid log(0)\n#     mn = np.min(Sxx_db)\n#     mx = np.max(Sxx_db)\n#     normalize_color= matplotlib.colors.Normalize(vmin=mn, vmax=mx)\n#     cax = ax.pcolormesh(t, f, Sxx_db, cmap='jet', norm=normalize_color)\n#     ax.set_title(title)\n#     ax.set_ylabel('Frequency [Hz]')\n#     ax.set_xlabel('Time [sec]')\n#     return cax\n\n# # Plot spectrograms for each feature in a 4x2 grid\n# fig, axs = plt.subplots(4, 2, figsize=(20, 20))\n\n# for i in range(8):\n#     row, col = divmod(i, 2)\n#     cax = plot_spectrogram(sample[:, i], f'Feature {i + 1}', axs[row, col])\n\n# # Adjust layout\n# plt.tight_layout(rect=[0, 0.03, 1, 0.95])\n# fig.colorbar(cax, ax=axs, orientation='horizontal', fraction=0.025, pad=0.04)\n# plt.show()\n\n\n","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:44:38.569110Z","iopub.execute_input":"2024-07-02T08:44:38.569366Z","iopub.status.idle":"2024-07-02T08:44:38.575043Z","shell.execute_reply.started":"2024-07-02T08:44:38.569344Z","shell.execute_reply":"2024-07-02T08:44:38.574138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def average_spectrogram(data1, data2):\n    f1, t1, Sxx1 = spectrogram(data1, fs=200, nperseg=200, nfft=512, noverlap=int(200 * 0.8), scaling='density', return_onesided=True)\n    f2, t2, Sxx2 = spectrogram(data2, fs=200, nperseg=200, nfft=512, noverlap=int(200 * 0.8), scaling='density', return_onesided=True)\n    Sxx_avg = (Sxx1 + Sxx2) / 2\n    Sxx_avg_db = 10 * np.log10(np.abs(Sxx_avg)**2 + 1e-10)  # Add a small value to avoid log(0)\n#     print(Sxx_avg_db.shape)\n    return t1, f1, Sxx_avg_db\n   \n\n","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:44:38.576197Z","iopub.execute_input":"2024-07-02T08:44:38.576488Z","iopub.status.idle":"2024-07-02T08:44:38.588065Z","shell.execute_reply.started":"2024-07-02T08:44:38.576465Z","shell.execute_reply":"2024-07-02T08:44:38.587259Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def spectrogram1(data1):\n    f1, t1, Sxx1 = spectrogram(data1, fs=200, nperseg=200, nfft=512, noverlap=int(200 * 0.8), scaling='density', return_onesided=True)\n#     f2, t2, Sxx2 = spectrogram(data2, fs=200, nperseg=200, nfft=512, noverlap=int(200 * 0.8), scaling='density', return_onesided=True)\n#     Sxx_avg = (Sxx1 + Sxx2) / 2\n    Sxx_avg_db = 10 * np.log10(np.abs(Sxx1)**2 + 1e-10)  # Add a small value to avoid log(0)\n#     print(Sxx_avg_db.shape)\n    return Sxx_avg_db","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:44:38.589132Z","iopub.execute_input":"2024-07-02T08:44:38.589444Z","iopub.status.idle":"2024-07-02T08:44:38.599949Z","shell.execute_reply.started":"2024-07-02T08:44:38.589413Z","shell.execute_reply":"2024-07-02T08:44:38.599169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loader with Butter Low-Pass Filter","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nepsilon = 1e-6\nclass DataGenerator(tf.keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, data, batch_size=32, shuffle=False, eegs=all_eegs, mode='train',\n                 downsample=1): \n\n        self.data = data\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.eegs = eegs\n        self.mode = mode\n        self.downsample = downsample\n        self.on_epoch_end()\n        \n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        ct = int( np.ceil( len(self.data) / self.batch_size ) )\n        return ct\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n        X, y = self.__data_generation(indexes)\n        return X[:,:,:,:], y\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange( len(self.data) )\n        if self.shuffle: np.random.shuffle(self.indexes)\n                        \n    def __data_generation(self, indexes):\n        'Generates data containing batch_size samples' \n    \n        X = np.zeros((len(indexes),257,246,8),dtype='float32')\n        y = np.zeros((len(indexes),6),dtype='float32')\n        \n        sample = np.zeros((257,246,X.shape[-1]))\n        \n        for j,i in enumerate(indexes):\n            row = self.data.iloc[i]      \n            data = self.eegs[row.eeg_id]\n            \n            \n            # FEATURE ENGINEER\n\n#             _,_,spec0 = average_spectrogram(data[:,FEAT2IDX['Fp1']] - data[:,FEAT2IDX['T3']],\n#                                                  data[:,FEAT2IDX['T3']] - data[:,FEAT2IDX['O1']])\n#             sample[:, :, 0] = spec0\n# #             print(f'the spec shape is {spec0.shape}')\n#             _,_,sample[:, :, 1] = average_spectrogram(data[:,FEAT2IDX['Fp1']] - data[:,FEAT2IDX['C3']],\n#                                                  data[:,FEAT2IDX['C3']] - data[:,FEAT2IDX['O1']])\n#             _,_,sample[:, :, 2] = average_spectrogram(data[:,FEAT2IDX['Fp2']] - data[:,FEAT2IDX['C4']],\n#                                                  data[:,FEAT2IDX['C4']] - data[:,FEAT2IDX['O2']])\n#             _,_,sample[:, :, 3] = average_spectrogram(data[:,FEAT2IDX['Fp2']] - data[:,FEAT2IDX['T4']],\n#                                                  data[:,FEAT2IDX['T4']] - data[:,FEAT2IDX['O2']])\n            sample[:, :, 0] = spectrogram1(np.nan_to_num(data[:,FEAT2IDX['Fp1']], nan=0) - np.nan_to_num(data[:,FEAT2IDX['T3']], nan=0))\n            sample[:, :, 1] = spectrogram1(np.nan_to_num(data[:,FEAT2IDX['T3']], nan=0) - np.nan_to_num(data[:,FEAT2IDX['O1']], nan=0))\n            sample[:, :, 2] = spectrogram1(np.nan_to_num(data[:,FEAT2IDX['Fp1']], nan=0) - np.nan_to_num(data[:,FEAT2IDX['C3']], nan=0))\n            sample[:, :, 3] = spectrogram1(np.nan_to_num(data[:,FEAT2IDX['C3']], nan=0) - np.nan_to_num(data[:,FEAT2IDX['O1']], nan=0))\n            sample[:, :, 4] = spectrogram1(np.nan_to_num(data[:,FEAT2IDX['Fp2']], nan=0) - np.nan_to_num(data[:,FEAT2IDX['C4']], nan=0))\n            sample[:, :, 5] = spectrogram1(np.nan_to_num(data[:,FEAT2IDX['C4']], nan=0) - np.nan_to_num(data[:,FEAT2IDX['O2']], nan=0))\n            sample[:, :, 6] = spectrogram1(np.nan_to_num(data[:,FEAT2IDX['Fp2']], nan=0) - np.nan_to_num(data[:,FEAT2IDX['T4']], nan=0))\n            sample[:, :, 7] = spectrogram1(np.nan_to_num(data[:,FEAT2IDX['T4']], nan=0) - np.nan_to_num(data[:,FEAT2IDX['O2']], nan=0))\n            sample = np.clip(sample,-1024, 1024)\n            sample[:, :, 0] = (sample[:, :, 0]-sample[:, :, 0].min())/(sample[:, :, 0].max()-sample[:, :, 0].min()+epsilon)\n            sample[:, :, 1] = (sample[:, :, 1]-sample[:, :, 1].min())/(sample[:, :, 1].max()-sample[:, :, 1].min()+epsilon)\n            sample[:, :, 2] = (sample[:, :, 2]-sample[:, :, 2].min())/(sample[:, :, 2].max()-sample[:, :, 2].min()+epsilon)\n            sample[:, :, 3] = (sample[:, :, 3]-sample[:, :, 3].min())/(sample[:, :, 3].max()-sample[:, :, 3].min()+epsilon)\n            sample[:, :, 4] = (sample[:, :, 4]-sample[:, :, 4].min())/(sample[:, :, 4].max()-sample[:, :, 4].min()+epsilon)\n            sample[:, :, 5] = (sample[:, :, 5]-sample[:, :, 5].min())/(sample[:, :, 5].max()-sample[:, :, 5].min()+epsilon)\n            sample[:, :, 6] = (sample[:, :, 6]-sample[:, :, 6].min())/(sample[:, :, 6].max()-sample[:, :, 6].min()+epsilon)\n            sample[:, :, 7] = (sample[:, :, 7]-sample[:, :, 7].min())/(sample[:, :, 7].max()-sample[:, :, 7].min()+epsilon)\n            \n#             sample = np.nan_to_num(sample, nan=0) / 32.0\n            X[j,] = sample\n            if self.mode!='test':\n                y[j] = row[TARGETS]\n            \n        return X[:,:224, :224,:],y","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:58:55.821464Z","iopub.execute_input":"2024-07-02T08:58:55.822325Z","iopub.status.idle":"2024-07-02T08:58:55.850498Z","shell.execute_reply.started":"2024-07-02T08:58:55.822293Z","shell.execute_reply":"2024-07-02T08:58:55.849497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train = pd.DataFrame({'eeg_id': [0, 1, 2], 'TARGETS': [0, 1, 1]})  # Example dataframe\n# eegs = {0: np.random.rand(500, 500), 1: np.random.rand(500, 500), 2: np.random.rand(500, 500)}  # Example EEG data\nvalid_index = [p for p in range(64)]\n# gen = DataGenerator(train, shuffle=False, eegs=eegs)\n# train_gen = DataGenerator(train.iloc[train_index], shuffle=True, batch_size=32)\ntrain_gen = DataGenerator(train.iloc[valid_index], shuffle=False, batch_size=64, mode='valid')\nfor X, y in train_gen:\n    print(X.shape, y.shape)\n    break","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:58:57.350901Z","iopub.execute_input":"2024-07-02T08:58:57.351692Z","iopub.status.idle":"2024-07-02T08:58:59.653420Z","shell.execute_reply.started":"2024-07-02T08:58:57.351658Z","shell.execute_reply":"2024-07-02T08:58:59.652222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nfor ii in range(8):\n    mn = np.min(X[ii,:,:,0])\n    mx = np.max(X[ii,:,:,0])\n    normalize_color= matplotlib.colors.Normalize(vmin=mn, vmax=mx)\n    plt.figure()\n    plt.pcolormesh(X[ii,:,:,0], cmap='jet', norm=normalize_color)\n    plt.title(f'{y[ii]}')","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:58:59.655788Z","iopub.execute_input":"2024-07-02T08:58:59.656164Z","iopub.status.idle":"2024-07-02T08:59:02.812362Z","shell.execute_reply.started":"2024-07-02T08:58:59.656130Z","shell.execute_reply":"2024-07-02T08:59:02.811383Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Display Data Loader","metadata":{}},{"cell_type":"markdown","source":"# Initialize GPUs","metadata":{}},{"cell_type":"code","source":"import os\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport tensorflow as tf\nprint('TensorFlow version =',tf.__version__)\n\n# USE MULTIPLE GPUS\ngpus = tf.config.list_physical_devices('GPU')\nif len(gpus)<=1: \n    strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n    print(f'Using {len(gpus)} GPU')\nelse: \n    strategy = tf.distribute.MirroredStrategy()\n    print(f'Using {len(gpus)} GPUs')","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:59:02.813868Z","iopub.execute_input":"2024-07-02T08:59:02.814151Z","iopub.status.idle":"2024-07-02T08:59:02.821192Z","shell.execute_reply.started":"2024-07-02T08:59:02.814126Z","shell.execute_reply":"2024-07-02T08:59:02.820244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# USE MIXED PRECISION\nMIX = True\nif MIX:\n    tf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\n    print('Mixed precision enabled')\nelse:\n    print('Using full precision')","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:59:02.822318Z","iopub.execute_input":"2024-07-02T08:59:02.822572Z","iopub.status.idle":"2024-07-02T08:59:02.831518Z","shell.execute_reply.started":"2024-07-02T08:59:02.822550Z","shell.execute_reply":"2024-07-02T08:59:02.830704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build CNN Model","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras.applications import VGG16, ConvNeXtSmall,MobileNetV3Small\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Flatten, Dense, Dropout\nfrom tensorflow.keras.optimizers import Adam\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:59:02.833477Z","iopub.execute_input":"2024-07-02T08:59:02.833775Z","iopub.status.idle":"2024-07-02T08:59:02.843474Z","shell.execute_reply.started":"2024-07-02T08:59:02.833720Z","shell.execute_reply":"2024-07-02T08:59:02.842658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model():\n        \n    # INPUT \n    inp = tf.keras.Input(shape=(224,224,8))\n    x = tf.keras.layers.Conv2D(3, kernel_size=(3,3), activation='gelu', padding='same')(inp)\n    # Load the VGG16 model\n    base_model = MobileNetV3Small(weights='imagenet', include_top=False, input_shape=(224, 224, 3))\n    x = base_model(x)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    x = Dropout(0.5)(x)\n#     x = Dense(256, activation='relu')(x)\n#     x = Dropout(0.5)(x)\n#     predictions = Dense(10, activation='softmax')(x)\n    y = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(x)\n    \n    # COMPILE MODEL\n    model = tf.keras.Model(inputs=inp, outputs=y)\n    model.summary()\n#     opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    opt = tf.keras.optimizers.AdamW(weight_decay=0.5, learning_rate = 1e-4, )\n    loss = tf.keras.losses.KLDivergence()\n    model.compile(loss=loss, optimizer = opt, metrics=[tf.keras.metrics.CategoricalAccuracy()], )\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:59:02.844452Z","iopub.execute_input":"2024-07-02T08:59:02.844814Z","iopub.status.idle":"2024-07-02T08:59:02.855496Z","shell.execute_reply.started":"2024-07-02T08:59:02.844782Z","shell.execute_reply":"2024-07-02T08:59:02.854578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Group KFold","metadata":{}},{"cell_type":"code","source":"VERBOSE = 1\nFOLDS_TO_TRAIN = 2\nif not os.path.exists('WaveNet_Model'):\n    os.makedirs('WaveNet_Model')\n\nfrom sklearn.model_selection import KFold, GroupKFold\nimport tensorflow.keras.backend as K, gc\n\nall_oof = []; all_oof2 = []; all_true = []\ngkf = GroupKFold(n_splits=2)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):\n    \n    print('#'*25)\n    print(f'### Fold {i+1}')\n    train_gen = DataGenerator(train.iloc[train_index], shuffle=True, batch_size=32)\n    valid_gen = DataGenerator(train.iloc[valid_index], shuffle=True, batch_size=64, mode='valid')\n    print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n    print('#'*25)\n    \n    # TRAIN MODEL\n    K.clear_session()\n    with strategy.scope():\n        model = build_model()\n    if TRAIN_MODEL:\n        model.fit(train_gen, verbose=VERBOSE,\n              validation_data = valid_gen,\n              epochs=50)\n        model.save_weights(f'WaveNet_Model/WaveNet_fold{i}.h5')\n    else:\n        model.load_weights(f'/kaggle/input/brain-eegs/WaveNet_Model/WaveNet_fold{i}.h5')\n    \n    # WAVENET OOF\n    oof = model.predict(valid_gen, verbose=VERBOSE)\n    all_oof.append(oof)\n    all_true.append(train.iloc[valid_index][TARGETS].values)\n    \n    # TRAIN MEAN OOF\n    y_train = train.iloc[train_index][TARGETS].values\n    y_valid = train.iloc[valid_index][TARGETS].values\n    oof = y_valid.copy()\n    for j in range(6):\n        oof[:,j] = y_train[:,j].mean()\n    oof = oof / oof.sum(axis=1,keepdims=True)\n    all_oof2.append(oof)\n    \n    del model, oof, y_train, y_valid\n    gc.collect()\n    \n    if i==FOLDS_TO_TRAIN-1: break\n    \nall_oof = np.concatenate(all_oof)\nall_oof2 = np.concatenate(all_oof2)\nall_true = np.concatenate(all_true)","metadata":{"execution":{"iopub.status.busy":"2024-07-02T08:59:24.719393Z","iopub.execute_input":"2024-07-02T08:59:24.719768Z","iopub.status.idle":"2024-07-02T10:35:40.583842Z","shell.execute_reply.started":"2024-07-02T08:59:24.719727Z","shell.execute_reply":"2024-07-02T10:35:40.583018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"valid_index","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:38:50.526132Z","iopub.execute_input":"2024-07-02T10:38:50.526511Z","iopub.status.idle":"2024-07-02T10:38:50.532949Z","shell.execute_reply.started":"2024-07-02T10:38:50.526482Z","shell.execute_reply":"2024-07-02T10:38:50.532067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_index","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:38:51.212231Z","iopub.execute_input":"2024-07-02T10:38:51.212564Z","iopub.status.idle":"2024-07-02T10:38:51.218365Z","shell.execute_reply.started":"2024-07-02T10:38:51.212537Z","shell.execute_reply":"2024-07-02T10:38:51.217576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for WaveNet","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score with WaveNet Raw EEG =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:38:54.232250Z","iopub.execute_input":"2024-07-02T10:38:54.232838Z","iopub.status.idle":"2024-07-02T10:38:54.316343Z","shell.execute_reply.started":"2024-07-02T10:38:54.232805Z","shell.execute_reply":"2024-07-02T10:38:54.315070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score using Train Means","metadata":{}},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof2.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score with Train Means =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:38:56.430327Z","iopub.execute_input":"2024-07-02T10:38:56.430678Z","iopub.status.idle":"2024-07-02T10:38:56.485730Z","shell.execute_reply.started":"2024-07-02T10:38:56.430650Z","shell.execute_reply":"2024-07-02T10:38:56.484957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit to Kaggle LB","metadata":{}},{"cell_type":"code","source":"del all_eegs, train; gc.collect()\ntest = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\nprint('Test shape:',test.shape)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:38:59.858996Z","iopub.execute_input":"2024-07-02T10:38:59.859593Z","iopub.status.idle":"2024-07-02T10:39:00.074957Z","shell.execute_reply.started":"2024-07-02T10:38:59.859562Z","shell.execute_reply":"2024-07-02T10:39:00.074112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_eegs2 = {}\nDISPLAY = 1\nEEG_IDS2 = test.eeg_id.unique()\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n\nprint('Processing Test EEG parquets...'); print()\nfor i,eeg_id in enumerate(EEG_IDS2):\n        \n    # SAVE EEG TO PYTHON DICTIONARY OF NUMPY ARRAYS\n    data = eeg_from_parquet(f'{PATH2}{eeg_id}.parquet', i<DISPLAY)\n    all_eegs2[eeg_id] = data","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:39:01.781273Z","iopub.execute_input":"2024-07-02T10:39:01.781638Z","iopub.status.idle":"2024-07-02T10:39:02.527884Z","shell.execute_reply.started":"2024-07-02T10:39:01.781609Z","shell.execute_reply":"2024-07-02T10:39:02.526906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER MLP ON TEST\npreds = []\nmodel = build_model()\ntest_gen = DataGenerator(test, shuffle=False, batch_size=64, eegs=all_eegs2, mode='test')\n\nprint('Inferring test... ',end='')\nfor i in range(FOLDS_TO_TRAIN):\n    print(f'fold {i+1}, ',end='')\n    if TRAIN_MODEL:\n        model.load_weights(f'WaveNet_Model/WaveNet_fold{i}.h5')\n    else:\n        model.load_weights(f'/kaggle/input/brain-eegs/WaveNet_Model/WaveNet_fold{i}.h5')\n    pred = model.predict(test_gen, verbose=0)\n    preds.append(pred)\npred = np.mean(preds,axis=0)\nprint()\nprint('Test preds shape',pred.shape)","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:39:05.024239Z","iopub.execute_input":"2024-07-02T10:39:05.024861Z","iopub.status.idle":"2024-07-02T10:39:08.553214Z","shell.execute_reply.started":"2024-07-02T10:39:05.024829Z","shell.execute_reply":"2024-07-02T10:39:08.552250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CREATE SUBMISSION.CSV\nfrom IPython.display import display\n\nsub = pd.DataFrame({'eeg_id':test.eeg_id.values})\nsub[TARGETS] = pred\nsub.to_csv('submission.csv',index=False)\nprint('Submission shape',sub.shape)\ndisplay( sub.head() )\n\n# SANITY CHECK TO CONFIRM PREDICTIONS SUM TO ONE\nprint('Sub row 0 sums to:',sub.iloc[0,-6:].sum())","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:39:08.554576Z","iopub.execute_input":"2024-07-02T10:39:08.554887Z","iopub.status.idle":"2024-07-02T10:39:08.576693Z","shell.execute_reply.started":"2024-07-02T10:39:08.554861Z","shell.execute_reply":"2024-07-02T10:39:08.575608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score with WaveNet Raw EEG =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:42:47.112323Z","iopub.execute_input":"2024-07-02T10:42:47.112746Z","iopub.status.idle":"2024-07-02T10:42:47.183846Z","shell.execute_reply.started":"2024-07-02T10:42:47.112715Z","shell.execute_reply":"2024-07-02T10:42:47.182924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score with WaveNet Raw EEG =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:45:28.879307Z","iopub.execute_input":"2024-07-02T10:45:28.880282Z","iopub.status.idle":"2024-07-02T10:45:28.940193Z","shell.execute_reply.started":"2024-07-02T10:45:28.880249Z","shell.execute_reply":"2024-07-02T10:45:28.939248Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof2.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score with Train Means =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-07-02T10:45:53.940374Z","iopub.execute_input":"2024-07-02T10:45:53.941158Z","iopub.status.idle":"2024-07-02T10:45:54.004167Z","shell.execute_reply.started":"2024-07-02T10:45:53.941119Z","shell.execute_reply":"2024-07-02T10:45:54.003205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}