{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7465251,"sourceType":"datasetVersion","datasetId":4317718}],"dockerImageVersionId":30636,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# WaveNet Starter using RAW EEG Features!\nThis notebook is a WaveNet starter for Kaggle's Brain comp. It achieves `CV 0.91` and `LB 0.66`. Note that submitting train means achieves `CV 1.26` and `LB 0.97` [[here][1]]. So this notebook's WaveNet is successfully learning to predict brain events from raw EEG waveforms!\n\nThis model only uses two features. We can engineer more features and/or modify the model architecture to improve CV score and LB score. Furthermore we can build 1 model which inputs both spectrogram images and eeg waveforms. The two EEG features in this notebook are:\n* feature 1 : `Fp1 minus O1`\n* feature 2 : `Fp2 minus O2`\n\nFeature 1 is the beginning of the montage chains `LL` and `LP` minus the ending of montage `LL` and `LP`. And feature 2 is the beginning of the montage chains `RL` and `RP` minus the ending of montage `RL` and `RP`.\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/montage.png)\n\n# UPDATE\nIn version 7 and 8, we add more features and update the model architecture to evaluate each montage chain separately and then concatenate the features. This new architecture is motivated by the discovery of a better formula to utilize EEG explained in discussion [here][2]\n\n* Version 5,6: Use 2 features - CV 0.91 LB 0.66\n* Version 7,8: Use 8 features grouped as 4 chains. Downsample time 5x - **CV 0.81 LB 0.53**, wow!\n\nWe train our new model in version 7, then save model weights. Then load model into version 8 to submit to LB.\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave-model.png)\n\n# Train and Infer Tricks\nWe train the fold models in version 7 of the notebook and submit to Kaggle LB in version 8 of the notebook. This makes submission faster because we train the fold models for 30 minutes in version 7 then save them. In version 8, we just load the models without needing to retrain models during Kaggle submit. (And we train our old model in 5 annd submit in 6).\n\nVersion 4 uses `1xP100` GPU with full precision and takes 1 hour to train 5 folds 5 epochs of WaveNet. Version 5 uses `2xT4` GPU with mixed precision and takes 30 minutes to train 5 folds 5 epochs of WaveNet. \n\n[1]: https://www.kaggle.com/code/seshurajup/eda-train-csv\n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469760","metadata":{}},{"cell_type":"markdown","source":"# Load Train Data","metadata":{}},{"cell_type":"code","source":"import pandas as pd, numpy as np, os\nimport matplotlib.pyplot as plt\n\ntrain = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nprint( train.shape )\ndisplay( train.head() )\n\n# CHOICE TO CREATE OR LOAD EEGS FROM NOTEBOOK VERSION 1\nCREATE_EEGS = False\nTRAIN_MODEL = False","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:13:36.847819Z","iopub.execute_input":"2024-02-18T04:13:36.848266Z","iopub.status.idle":"2024-02-18T04:13:38.002226Z","shell.execute_reply.started":"2024-02-18T04:13:36.848227Z","shell.execute_reply":"2024-02-18T04:13:38.001258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Raw EEG Features","metadata":{}},{"cell_type":"markdown","source":"随便读了一个.parquet文件，然后列出了它有什么feature","metadata":{}},{"cell_type":"code","source":"df = pd.read_parquet('/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/1000913311.parquet')\nFEATS = df.columns\nprint(f'There are {len(FEATS)} raw eeg features')\nprint( list(FEATS) )","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:13:38.004209Z","iopub.execute_input":"2024-02-18T04:13:38.004878Z","iopub.status.idle":"2024-02-18T04:13:38.229218Z","shell.execute_reply.started":"2024-02-18T04:13:38.004842Z","shell.execute_reply":"2024-02-18T04:13:38.228323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"挑选我们想要的特征，并且得到它在真实文件中的列号，方便后续取用","metadata":{}},{"cell_type":"code","source":"print('We will use the following subset of raw EEG features:')\nFEATS = ['Fp1','T3','C3','O1','Fp2','C4','T4','O2']\nFEAT2IDX = {x:y for x,y in zip(FEATS,range(len(FEATS)))}\nprint( list(FEATS) )\nprint( FEAT2IDX )","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:13:38.230335Z","iopub.execute_input":"2024-02-18T04:13:38.230608Z","iopub.status.idle":"2024-02-18T04:13:38.236429Z","shell.execute_reply.started":"2024-02-18T04:13:38.230582Z","shell.execute_reply":"2024-02-18T04:13:38.235556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"下面的函数仅仅是取了中间的10000行，并且用平均值替换掉了nan","metadata":{}},{"cell_type":"code","source":"def eeg_from_parquet(parquet_path, display=False):\n    \n    # parquet是一种广泛使用的数据存储格式，FEATS表示我们感兴趣的features\n    eeg = pd.read_parquet(parquet_path, columns=FEATS)\n    rows = len(eeg)\n    offset = (rows-10_000)//2 # 从python3.6开始，数字可以使用_作为分隔符，以此来提高数字的可读性\n    \n    # 只取了中间的10000个数据\n    eeg = eeg.iloc[offset:offset+10_000]\n    \n    # display表示用数据绘制波形图，这里不重要，没用上\n    if display: \n        plt.figure(figsize=(10,5))\n        offset = 0\n    \n    # 10000 × features的二维numpy\n    data = np.zeros((10_000,len(FEATS)))\n    for j,col in enumerate(FEATS):\n        \n        x = eeg[col].values.astype('float32')\n        # 计算数组忽略nan值后的平均值\n        m = np.nanmean(x)\n        # np.isnan(x)是布尔数组，也可以看做01数组来进行取平均值，这样的结果就是nan的比例\n        # nan_to_num的用法是用m填充x中的nan值\n        if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n        # else的意思是，如果全是nan，就填成全0\n        else: x[:] = 0\n            \n        data[:,j] = x\n        \n        if display: \n            if j!=0: offset += x.max()\n            plt.plot(range(10_000),x-offset,label=col)\n            offset -= x.min()\n            \n    if display:\n        plt.legend()\n        name = parquet_path.split('/')[-1]\n        name = name.split('.')[0]\n        plt.title(f'EEG {name}',size=16)\n        plt.show()\n        \n    return data","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:13:38.238475Z","iopub.execute_input":"2024-02-18T04:13:38.238778Z","iopub.status.idle":"2024-02-18T04:13:38.248858Z","shell.execute_reply.started":"2024-02-18T04:13:38.238753Z","shell.execute_reply":"2024-02-18T04:13:38.248001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"%%time是notebook的一个魔术命令，带上它之后，就可以在运行该单元格时输出包含执行时间的统计信息\n下面的block其实就是调用前面的读取parquet函数，把每个egg_id对应的parquet读进来","metadata":{}},{"cell_type":"code","source":"%%time\n\nall_eegs = {}\nDISPLAY = 4\n# 所有不同的egg_id\nEEG_IDS = train.eeg_id.unique()\nPATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/'\n\nfor i,eeg_id in enumerate(EEG_IDS):\n    if (i%100==0)&(i!=0): print(i,', ',end='') \n    \n    # 按照egg_id来读取parquet文件，用all_eggs这个字典存储数据\n    data = eeg_from_parquet(f'{PATH}{eeg_id}.parquet', display=i<DISPLAY)              \n    all_eegs[eeg_id] = data\n    \n    if i==DISPLAY:\n        if CREATE_EEGS:\n            print(f'Processing {train.eeg_id.nunique()} eeg parquets... ',end='')\n        else:\n            print(f'Reading {len(EEG_IDS)} eeg NumPys from disk.')\n            break\n# 要么把刚刚读到的存进磁盘，要么从之前存过的地方重新加载          \nif CREATE_EEGS: \n    np.save('eegs',all_eegs)\nelse:\n    all_eegs = np.load('/kaggle/input/brain-eegs/eegs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:13:38.249995Z","iopub.execute_input":"2024-02-18T04:13:38.250615Z","iopub.status.idle":"2024-02-18T04:15:12.336434Z","shell.execute_reply.started":"2024-02-18T04:13:38.250583Z","shell.execute_reply":"2024-02-18T04:15:12.335423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Deduplicate Train EEG Id","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\n# 最后六列是target列\nTARGETS = df.columns[-6:]\nTARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\nTARS2 = {x:y for y,x in TARS.items()}\n\n# train是eeg_id和patient_id的对应关系\ntrain = df.groupby('eeg_id')[['patient_id']].agg('first')\n\n# 每个id合起来，然后取每段投票数之和\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n    train[t] = tmp[t].values\n\n# train现在以egg_id为序，patient_\ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\ntrain = train.loc[train.eeg_id.isin(EEG_IDS)]\nprint('Train Data with unique eeg_id shape:', train.shape )\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:12.337503Z","iopub.execute_input":"2024-02-18T04:15:12.337792Z","iopub.status.idle":"2024-02-18T04:15:12.598966Z","shell.execute_reply.started":"2024-02-18T04:15:12.337767Z","shell.execute_reply":"2024-02-18T04:15:12.598039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TARGETS","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:16:38.382006Z","iopub.execute_input":"2024-02-18T04:16:38.382883Z","iopub.status.idle":"2024-02-18T04:16:38.388819Z","shell.execute_reply.started":"2024-02-18T04:16:38.382848Z","shell.execute_reply":"2024-02-18T04:16:38.387947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Butter Low-Pass Filter","metadata":{}},{"cell_type":"code","source":"from scipy.signal import butter, lfilter\n\ndef butter_lowpass_filter(data, cutoff_freq=20, sampling_rate=200, order=4):\n    nyquist = 0.5 * sampling_rate\n    normal_cutoff = cutoff_freq / nyquist\n    b, a = butter(order, normal_cutoff, btype='low', analog=False)\n    filtered_data = lfilter(b, a, data, axis=0)\n    return filtered_data","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:12.599924Z","iopub.execute_input":"2024-02-18T04:15:12.600158Z","iopub.status.idle":"2024-02-18T04:15:13.319026Z","shell.execute_reply.started":"2024-02-18T04:15:12.600137Z","shell.execute_reply":"2024-02-18T04:15:13.318226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FREQS = [1,2,4,8,16][::-1]\nx = [all_eegs[EEG_IDS[0]][:,0]]\nfor k in FREQS:\n    x.append( butter_lowpass_filter(x[0], cutoff_freq=k) )\n\nplt.figure(figsize=(20,20))\nplt.plot(range(10_000),x[0], label='without filter')\nfor k in range(1,len(x)):\n    plt.plot(range(10_000),x[k]-k*(x[0].max()-x[0].min()), label=f'with filter {FREQS[k-1]}Hz')\nplt.legend()\nplt.title('Butter Low-Pass Filter Examples',size=18)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:13.320126Z","iopub.execute_input":"2024-02-18T04:15:13.320398Z","iopub.status.idle":"2024-02-18T04:15:14.163954Z","shell.execute_reply.started":"2024-02-18T04:15:13.320374Z","shell.execute_reply":"2024-02-18T04:15:14.163071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Loader with Butter Low-Pass Filter","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\n\nclass DataGenerator(tf.keras.utils.Sequence):\n    'Generates data for Keras'\n    def __init__(self, data, batch_size=32, shuffle=False, eegs=all_eegs, mode='train',\n                 downsample=5): \n\n        self.data = data\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.eegs = eegs\n        self.mode = mode\n        self.downsample = downsample\n        self.on_epoch_end()\n        \n    def __len__(self):\n        'Denotes the number of batches per epoch'\n        ct = int( np.ceil( len(self.data) / self.batch_size ) )\n        return ct\n\n    def __getitem__(self, index):\n        'Generate one batch of data'\n        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n        X, y = self.__data_generation(indexes)\n        return X[:,::self.downsample,:], y\n\n    def on_epoch_end(self):\n        'Updates indexes after each epoch'\n        self.indexes = np.arange( len(self.data) )\n        if self.shuffle: np.random.shuffle(self.indexes)\n                        \n    def __data_generation(self, indexes):\n        'Generates data containing batch_size samples' \n    \n        X = np.zeros((len(indexes),10_000,8),dtype='float32')\n        y = np.zeros((len(indexes),6),dtype='float32')\n        \n        sample = np.zeros((10_000,X.shape[-1]))\n        for j,i in enumerate(indexes):\n            row = self.data.iloc[i]      \n            data = self.eegs[row.eeg_id]\n            \n            # FEATURE ENGINEER\n            sample[:,0] = data[:,FEAT2IDX['Fp1']] - data[:,FEAT2IDX['T3']]\n            sample[:,1] = data[:,FEAT2IDX['T3']] - data[:,FEAT2IDX['O1']]\n            \n            sample[:,2] = data[:,FEAT2IDX['Fp1']] - data[:,FEAT2IDX['C3']]\n            sample[:,3] = data[:,FEAT2IDX['C3']] - data[:,FEAT2IDX['O1']]\n            \n            sample[:,4] = data[:,FEAT2IDX['Fp2']] - data[:,FEAT2IDX['C4']]\n            sample[:,5] = data[:,FEAT2IDX['C4']] - data[:,FEAT2IDX['O2']]\n            \n            sample[:,6] = data[:,FEAT2IDX['Fp2']] - data[:,FEAT2IDX['T4']]\n            sample[:,7] = data[:,FEAT2IDX['T4']] - data[:,FEAT2IDX['O2']]\n            \n            # STANDARDIZE\n            sample = np.clip(sample,-1024,1024)\n            sample = np.nan_to_num(sample, nan=0) / 32.0\n            \n            # BUTTER LOW-PASS FILTER\n            sample = butter_lowpass_filter(sample)\n            \n            X[j,] = sample\n            if self.mode!='test':\n                y[j] = row[TARGETS]\n            \n        return X,y","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:14.165060Z","iopub.execute_input":"2024-02-18T04:15:14.165336Z","iopub.status.idle":"2024-02-18T04:15:25.314475Z","shell.execute_reply.started":"2024-02-18T04:15:14.165311Z","shell.execute_reply":"2024-02-18T04:15:25.313637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Display Data Loader","metadata":{}},{"cell_type":"code","source":"gen = DataGenerator(train, shuffle=False)\n\nfor x,y in gen:\n    for k in range(4):\n        plt.figure(figsize=(20,4))\n        offset = 0\n        for j in range(x.shape[-1]):\n            if j!=0: offset -= x[k,:,j].min()\n            plt.plot(range(2_000),x[k,:,j]+offset,label=f'feature {j+1}')\n            offset += x[k,:,j].max()\n        tt = f'{y[k][0]:0.1f}'\n        for t in y[k][1:]:\n            tt += f', {t:0.1f}'\n        plt.title(f'EEG_Id = {EEG_IDS[k]}\\nTarget = {tt}',size=14)\n        plt.legend()\n        plt.show()\n    break","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:25.317511Z","iopub.execute_input":"2024-02-18T04:15:25.318363Z","iopub.status.idle":"2024-02-18T04:15:27.223772Z","shell.execute_reply.started":"2024-02-18T04:15:25.318333Z","shell.execute_reply":"2024-02-18T04:15:27.222556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Initialize GPUs","metadata":{}},{"cell_type":"code","source":"import os\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport tensorflow as tf\nprint('TensorFlow version =',tf.__version__)\n\n# USE MULTIPLE GPUS\ngpus = tf.config.list_physical_devices('GPU')\nif len(gpus)<=1: \n    strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n    print(f'Using {len(gpus)} GPU')\nelse: \n    strategy = tf.distribute.MirroredStrategy()\n    print(f'Using {len(gpus)} GPUs')","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:27.224946Z","iopub.execute_input":"2024-02-18T04:15:27.225216Z","iopub.status.idle":"2024-02-18T04:15:27.707584Z","shell.execute_reply.started":"2024-02-18T04:15:27.225193Z","shell.execute_reply":"2024-02-18T04:15:27.706501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# USE MIXED PRECISION\nMIX = True\nif MIX:\n    tf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\n    print('Mixed precision enabled')\nelse:\n    print('Using full precision')","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:27.708878Z","iopub.execute_input":"2024-02-18T04:15:27.709195Z","iopub.status.idle":"2024-02-18T04:15:27.720310Z","shell.execute_reply.started":"2024-02-18T04:15:27.709168Z","shell.execute_reply":"2024-02-18T04:15:27.719428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build WaveNet Model","metadata":{}},{"cell_type":"code","source":"# TRAIN SCHEDULE\ndef lrfn(epoch):\n        return [1e-3,1e-3,1e-4,1e-4,1e-5][epoch]\nLR = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\nEPOCHS = 5","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:27.721337Z","iopub.execute_input":"2024-02-18T04:15:27.721619Z","iopub.status.idle":"2024-02-18T04:15:27.730548Z","shell.execute_reply.started":"2024-02-18T04:15:27.721594Z","shell.execute_reply":"2024-02-18T04:15:27.729702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.layers import Input, Dense, Multiply, Add, Conv1D, Concatenate\n\ndef wave_block(x, filters, kernel_size, n):\n    dilation_rates = [2**i for i in range(n)]\n    x = Conv1D(filters = filters,\n               kernel_size = 1,\n               padding = 'same')(x)\n    res_x = x\n    for dilation_rate in dilation_rates:\n        tanh_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same', \n                          activation = 'tanh', \n                          dilation_rate = dilation_rate)(x)\n        sigm_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same',\n                          activation = 'sigmoid', \n                          dilation_rate = dilation_rate)(x)\n        x = Multiply()([tanh_out, sigm_out])\n        x = Conv1D(filters = filters,\n                   kernel_size = 1,\n                   padding = 'same')(x)\n        res_x = Add()([res_x, x])\n    return res_x","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:27.731614Z","iopub.execute_input":"2024-02-18T04:15:27.731924Z","iopub.status.idle":"2024-02-18T04:15:27.743481Z","shell.execute_reply.started":"2024-02-18T04:15:27.731895Z","shell.execute_reply":"2024-02-18T04:15:27.742620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jan-2024/wave-model.png)","metadata":{}},{"cell_type":"code","source":"def build_model():\n        \n    # INPUT \n    inp = tf.keras.Input(shape=(2_000,8))\n    \n    ############\n    # FEATURE EXTRACTION SUB MODEL\n    inp2 = tf.keras.Input(shape=(2_000,1))\n    x = wave_block(inp2, 8, 3, 12)\n    x = wave_block(x, 16, 3, 8)\n    x = wave_block(x, 32, 3, 4)\n    x = wave_block(x, 64, 3, 1)\n    model2 = tf.keras.Model(inputs=inp2, outputs=x)\n    ###########\n    \n    # LEFT TEMPORAL CHAIN\n    x1 = model2(inp[:,:,0:1])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,1:2])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z1 = tf.keras.layers.Average()([x1,x2])\n    \n    # LEFT PARASAGITTAL CHAIN\n    x1 = model2(inp[:,:,2:3])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,3:4])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z2 = tf.keras.layers.Average()([x1,x2])\n    \n    # RIGHT PARASAGITTAL CHAIN\n    x1 = model2(inp[:,:,4:5])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,5:6])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z3 = tf.keras.layers.Average()([x1,x2])\n    \n    # RIGHT TEMPORAL CHAIN\n    x1 = model2(inp[:,:,6:7])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,7:8])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z4 = tf.keras.layers.Average()([x1,x2])\n    \n    # COMBINE CHAINS\n    y = tf.keras.layers.Concatenate()([z1,z2,z3,z4])\n    y = tf.keras.layers.Dense(64, activation='relu')(y)\n    y = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(y)\n    \n    # COMPILE MODEL\n    model = tf.keras.Model(inputs=inp, outputs=y)\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.KLDivergence()\n    model.compile(loss=loss, optimizer = opt)\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:27.744552Z","iopub.execute_input":"2024-02-18T04:15:27.744864Z","iopub.status.idle":"2024-02-18T04:15:27.760188Z","shell.execute_reply.started":"2024-02-18T04:15:27.744840Z","shell.execute_reply":"2024-02-18T04:15:27.759337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Group KFold","metadata":{}},{"cell_type":"code","source":"VERBOSE = 1\nFOLDS_TO_TRAIN = 5\nif not os.path.exists('WaveNet_Model'):\n    os.makedirs('WaveNet_Model')\n\nfrom sklearn.model_selection import KFold, GroupKFold\nimport tensorflow.keras.backend as K, gc\n\nall_oof = []; all_oof2 = []; all_true = []\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):   \n    \n    print('#'*25)\n    print(f'### Fold {i+1}')\n    train_gen = DataGenerator(train.iloc[train_index], shuffle=True, batch_size=32)\n    valid_gen = DataGenerator(train.iloc[valid_index], shuffle=False, batch_size=64, mode='valid')\n    print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n    print('#'*25)\n    \n    # TRAIN MODEL\n    K.clear_session()\n    with strategy.scope():\n        model = build_model()\n    if TRAIN_MODEL:\n        model.fit(train_gen, verbose=VERBOSE,\n              validation_data = valid_gen,\n              epochs=EPOCHS, callbacks = [LR])\n        model.save_weights(f'WaveNet_Model/WaveNet_fold{i}.h5')\n    else:\n        model.load_weights(f'/kaggle/input/brain-eegs/WaveNet_Model/WaveNet_fold{i}.h5')\n    \n    # WAVENET OOF\n    oof = model.predict(valid_gen, verbose=VERBOSE)\n    all_oof.append(oof)\n    all_true.append(train.iloc[valid_index][TARGETS].values)\n    \n    # TRAIN MEAN OOF\n    y_train = train.iloc[train_index][TARGETS].values\n    y_valid = train.iloc[valid_index][TARGETS].values\n    oof = y_valid.copy()\n    for j in range(6):\n        oof[:,j] = y_train[:,j].mean()\n    oof = oof / oof.sum(axis=1,keepdims=True)\n    all_oof2.append(oof)\n    \n    del model, oof, y_train, y_valid\n    gc.collect()\n    \n    if i==FOLDS_TO_TRAIN-1: break\n    \nall_oof = np.concatenate(all_oof)\nall_oof2 = np.concatenate(all_oof2)\nall_true = np.concatenate(all_true)","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:15:27.761202Z","iopub.execute_input":"2024-02-18T04:15:27.761462Z","iopub.status.idle":"2024-02-18T04:16:29.505563Z","shell.execute_reply.started":"2024-02-18T04:15:27.761439Z","shell.execute_reply":"2024-02-18T04:16:29.503889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for WaveNet","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score with WaveNet Raw EEG =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:16:29.506667Z","iopub.status.idle":"2024-02-18T04:16:29.507110Z","shell.execute_reply.started":"2024-02-18T04:16:29.506886Z","shell.execute_reply":"2024-02-18T04:16:29.506906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score using Train Means","metadata":{}},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof2.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score with Train Means =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:16:29.508320Z","iopub.status.idle":"2024-02-18T04:16:29.508697Z","shell.execute_reply.started":"2024-02-18T04:16:29.508500Z","shell.execute_reply":"2024-02-18T04:16:29.508516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit to Kaggle LB","metadata":{}},{"cell_type":"code","source":"del all_eegs, train; gc.collect()\ntest = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\nprint('Test shape:',test.shape)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:16:29.509885Z","iopub.status.idle":"2024-02-18T04:16:29.510276Z","shell.execute_reply.started":"2024-02-18T04:16:29.510087Z","shell.execute_reply":"2024-02-18T04:16:29.510105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_eegs2 = {}\nDISPLAY = 1\nEEG_IDS2 = test.eeg_id.unique()\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n\nprint('Processing Test EEG parquets...'); print()\nfor i,eeg_id in enumerate(EEG_IDS2):\n        \n    # SAVE EEG TO PYTHON DICTIONARY OF NUMPY ARRAYS\n    data = eeg_from_parquet(f'{PATH2}{eeg_id}.parquet', i<DISPLAY)\n    all_eegs2[eeg_id] = data","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:16:29.511475Z","iopub.status.idle":"2024-02-18T04:16:29.511879Z","shell.execute_reply.started":"2024-02-18T04:16:29.511688Z","shell.execute_reply":"2024-02-18T04:16:29.511707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER MLP ON TEST\npreds = []\nmodel = build_model()\ntest_gen = DataGenerator(test, shuffle=False, batch_size=64, eegs=all_eegs2, mode='test')\n\nprint('Inferring test... ',end='')\nfor i in range(FOLDS_TO_TRAIN):\n    print(f'fold {i+1}, ',end='')\n    if TRAIN_MODEL:\n        model.load_weights(f'WaveNet_Model/WaveNet_fold{i}.h5')\n    else:\n        model.load_weights(f'/kaggle/input/brain-eegs/WaveNet_Model/WaveNet_fold{i}.h5')\n    pred = model.predict(test_gen, verbose=0)\n    preds.append(pred)\npred = np.mean(preds,axis=0)\nprint()\nprint('Test preds shape',pred.shape)","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:16:29.514316Z","iopub.status.idle":"2024-02-18T04:16:29.514856Z","shell.execute_reply.started":"2024-02-18T04:16:29.514570Z","shell.execute_reply":"2024-02-18T04:16:29.514595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CREATE SUBMISSION.CSV\nfrom IPython.display import display\n\nsub = pd.DataFrame({'eeg_id':test.eeg_id.values})\nsub[TARGETS] = pred\nsub.to_csv('submission.csv',index=False)\nprint('Submission shape',sub.shape)\ndisplay( sub.head() )\n\n# SANITY CHECK TO CONFIRM PREDICTIONS SUM TO ONE\nprint('Sub row 0 sums to:',sub.iloc[0,-6:].sum())","metadata":{"execution":{"iopub.status.busy":"2024-02-18T04:16:29.516580Z","iopub.status.idle":"2024-02-18T04:16:29.517122Z","shell.execute_reply.started":"2024-02-18T04:16:29.516851Z","shell.execute_reply":"2024-02-18T04:16:29.516877Z"},"trusted":true},"execution_count":null,"outputs":[]}]}