{"metadata":{"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7447509,"sourceType":"datasetVersion","datasetId":4334995},{"sourceId":7570342,"sourceType":"datasetVersion","datasetId":4407194},{"sourceId":7752462,"sourceType":"datasetVersion","datasetId":4382744},{"sourceId":7776446,"sourceType":"datasetVersion","datasetId":4550181},{"sourceId":7818976,"sourceType":"datasetVersion","datasetId":4417235}],"dockerImageVersionId":30636,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"papermill":{"default_parameters":{},"duration":101.739864,"end_time":"2024-03-03T17:15:15.283704","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-03-03T17:13:33.54384","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"We made an ensemble model of the following notebooks:\n- [Catboost Starter](http://www.kaggle.com/code/hzhangsg/wavenet-tensorflow-hms)\n- [Wavenet-Tenserflow-HMS](http://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-60)\n- [Features+Head Starter](http://www.kaggle.com/code/nartaa/features-head-starter)","metadata":{}},{"cell_type":"markdown","source":"## Features+Head Ensemble Starter [LB 0.34] for HMS Brain Comp\nWe can train 7 model and data variations: (all models included)\n\n**The Ensemble achieves LB 0.34**\n\n| MODEL | DATA TYPE | CV | LB | NOTES |\n|----------------|-----------|----------|---|--|\n| EfficientNetB2 | K | 0.6123 | 0.41 |K: Kaggle's spectrograms|\n| EfficientNetB2 | E | 0.6288 | 0.39 |E: EEG's spectrograms|\n| WaveNet | R | 0.6992 | 0.41 |R: Raw EEG signals|\n| EfficientNetB2 | KE | 0.5646 | 0.37 |KE: Kaggle's and EEG's spectrograms|\n| EfficientNetB2 + WaveNet | KR | 0.5912 | 0.39 |KR: Kaggle's spectrograms and Raw EEG signals|\n| EfficientNetB2 + WaveNet | ER | 0.6085 | 0.38 |ER: EEG's spectrograms and Raw EEG signals|\n| EfficientNetB2 + WaveNet | KER | - | 0.36 |KER: Kaggle's, EEG's spectrograms and Raw EEG signals|\n\nGreat discussion [here][5] by @KOLOO that led to the latest scores!\n\nFeatures+Head Starter uses Chris Deotte's Kaggle dataset [here][1]. Also Uses Chris's EEG spectrograms [here][3] (modified version). The Raw EEG signals can be found [here][6]. This notebook is a direct descendent of Chris's notebooks [EfficientNet][2] and [WaveNet][4]\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\n[2]: https://www.kaggle.com/code/cdeotte/efficientnetb2-starter-lb-0-57\n[3]: https://www.kaggle.com/datasets/nartaa/eeg-spectrograms\n[4]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468684\n[5]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461\n[6]: https://www.kaggle.com/datasets/nartaa/hms-eeg","metadata":{"papermill":{"duration":0.009301,"end_time":"2024-03-03T17:13:37.231229","exception":false,"start_time":"2024-03-03T17:13:37.221928","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os, random\nimport tensorflow as tf\nimport tensorflow\nimport tensorflow.keras.backend as K\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras.models import load_model\n\nLOAD_BACKBONE_FROM = '/kaggle/input/efficientnetb-tf-keras/EfficientNetB2.h5'\nLOAD_MODELS_FROM = '/kaggle/input/features-head-starter-models'\nHMS_PATH = '/kaggle/input/hms-harmful-brain-activity-classification'\nVER = 50\nDATA_TYPE = 'KER' # K|E|R|KE|KR|ER|KER\nUSE_PROCESSED = True # Use processed downsampled Raw EEG \nsubmission = True\n\n# Setup for ensemble\nENSEMBLE = True\nLBs = [0.41,0.39,0.41,0.37,0.39,0.38,0.36] # K|E|R|KE|KR|ER|KER for weighted ensemble we use LBs of each model\nVER_K = 43 # Kaggle's spectrogram model version\nVER_E = 42 # EEG's spectrogram model version\nVER_R = 37 # EEG's Raw wavenet model version, trained on single GPU\nVER_KE = 47 # Kaggle's and EEG's spectrogram model version\nVER_KR = 48 # Kaggle's spectrogram and Raw model version\nVER_ER = 49 # EEG's spectrogram and Raw model version\nVER_KER = 50 # EEG's, Kaggle's spectrograms and Raw model version\n\nnp.random.seed(42)\nrandom.seed(42)\ntf.random.set_seed(42)\n\n# USE SINGLE GPU, MULTIPLE GPUS \ngpus = tf.config.list_physical_devices('GPU')\n# WE USE MIXED PRECISION\ntf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\nif len(gpus)>1:\n    strategy = tf.distribute.MirroredStrategy()\n    print(f'Using {len(gpus)} GPUs')\nelse:\n    strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n    print(f'Using {len(gpus)} GPU')","metadata":{"papermill":{"duration":15.019425,"end_time":"2024-03-03T17:13:52.258732","exception":false,"start_time":"2024-03-03T17:13:37.239307","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:10:48.476755Z","iopub.execute_input":"2024-03-07T23:10:48.477169Z","iopub.status.idle":"2024-03-07T23:11:03.044131Z","shell.execute_reply.started":"2024-03-07T23:10:48.477138Z","shell.execute_reply":"2024-03-07T23:11:03.04319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load and create Non-Overlapping Eeg Id Train Data\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{"papermill":{"duration":0.008576,"end_time":"2024-03-03T17:13:52.276274","exception":false,"start_time":"2024-03-03T17:13:52.267698","status":"completed"},"tags":[]}},{"cell_type":"code","source":"TARGETS = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\n\ndef add_kl(data):\n    labels = data[TARGETS].values + 1e-5\n    data['kl'] = tf.keras.losses.KLDivergence(reduction='none')(\n        np.array([[1/6]*6]*len(data)),labels)\n    return data\n    \nif not submission:\n    train = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\n    TARGETS = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\n    META = ['spectrogram_id','spectrogram_label_offset_seconds','patient_id','expert_consensus']\n    train = train.groupby('eeg_id')[META+TARGETS\n                           ].agg({**{m:'first' for m in META},**{t:'sum' for t in TARGETS}}).reset_index() \n    train[TARGETS] = train[TARGETS]/train[TARGETS].values.sum(axis=1,keepdims=True)\n    train.columns = ['eeg_id','spec_id','offset','patient_id','target'] + TARGETS\n    train = add_kl(train)\n    print(train.head(1).to_string())","metadata":{"papermill":{"duration":0.024968,"end_time":"2024-03-03T17:13:52.310258","exception":false,"start_time":"2024-03-03T17:13:52.28529","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:03.045825Z","iopub.execute_input":"2024-03-07T23:11:03.046336Z","iopub.status.idle":"2024-03-07T23:11:03.05597Z","shell.execute_reply.started":"2024-03-07T23:11:03.04631Z","shell.execute_reply":"2024-03-07T23:11:03.055043Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read Train Spectrograms and EEGs\n\nWe can read 3 file from Chris's [Kaggle dataset here][1] which contains all the 11k spectrograms. From Chris's modified EEG spectrogram [here][2]. From Raw EEG signals [here][3]\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\n[2]: https://www.kaggle.com/datasets/nartaa/eeg-spectrograms\n[3]: https://www.kaggle.com/datasets/nartaa/hms-eeg","metadata":{"papermill":{"duration":0.007831,"end_time":"2024-03-03T17:13:52.325843","exception":false,"start_time":"2024-03-03T17:13:52.318012","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nif not submission:\n    spectrograms = None\n    all_eegs = None\n    all_raw_eegs = None\n    if DATA_TYPE in ['K','KE','KR','KER']:\n        spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()\n    if DATA_TYPE in ['E','KE','ER','KER']:\n        all_eegs = np.load('/kaggle/input/eeg-spectrograms/eeg_specs.npy',allow_pickle=True).item()\n    if DATA_TYPE in ['R','KR','ER','KER']:\n        if USE_PROCESSED:\n            all_raw_eegs = np.load('/kaggle/input/hms-eeg/eegs_processed.npy',allow_pickle=True).item()\n        else:\n            all_raw_eegs = np.load('/kaggle/input/hms-eeg/eegs.npy',allow_pickle=True).item()","metadata":{"papermill":{"duration":0.021122,"end_time":"2024-03-03T17:13:52.354762","exception":false,"start_time":"2024-03-03T17:13:52.33364","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:03.057104Z","iopub.execute_input":"2024-03-07T23:11:03.057366Z","iopub.status.idle":"2024-03-07T23:11:03.099351Z","shell.execute_reply.started":"2024-03-07T23:11:03.057343Z","shell.execute_reply":"2024-03-07T23:11:03.09843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DATA GENERATOR\nThis data generator outputs 512x512x3, the spectrogram and eeg images are concatenated all togother in a single image. For using data augmention you can set `augment = True` when creating the train data generator.","metadata":{"papermill":{"duration":0.007968,"end_time":"2024-03-03T17:13:52.37087","exception":false,"start_time":"2024-03-03T17:13:52.362902","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import albumentations as albu\nfrom scipy.signal import butter, lfilter\nimport librosa\n\nFEATS2 = ['Fp1','T3','C3','O1','Fp2','C4','T4','O2']\nFEAT2IDX = {x:y for x,y in zip(FEATS2,range(len(FEATS2)))}\nFEATS = [['Fp1','F7','T3','T5','O1'],\n         ['Fp1','F3','C3','P3','O1'],\n         ['Fp2','F8','T4','T6','O2'],\n         ['Fp2','F4','C4','P4','O2']]\n    \nclass DataGenerator():\n    'Generates data for Keras'\n    def __init__(self, data, specs=None, eeg_specs=None, raw_eegs=None , augment=False, mode='train', data_type=DATA_TYPE): \n        self.data = data\n        self.augment = augment\n        self.mode = mode\n        self.data_type = data_type\n        self.specs = specs\n        self.eeg_specs = eeg_specs\n        self.raw_eegs = raw_eegs\n        self.on_epoch_end()\n        \n    def __len__(self):\n        return self.data.shape[0]\n\n    def __getitem__(self, index):\n        X, y = self.data_generation(index)\n        if self.augment: X = self.augmentation(X)\n        return X, y\n    \n    def __call__(self):\n        for i in range(self.__len__()):\n            yield self.__getitem__(i)\n            \n            if i == self.__len__()-1:\n                self.on_epoch_end()\n                \n    def on_epoch_end(self):\n        if self.mode=='train': \n            self.data = self.data.sample(frac=1).reset_index(drop=True)\n    \n    def data_generation(self, index):\n        if self.data_type == 'KE':\n            X,y = self.generate_all_specs(index)\n        elif self.data_type == 'E' or self.data_type == 'K':\n            X,y = self.generate_specs(index)\n        elif self.data_type == 'R':\n            X,y = self.generate_raw(index)\n        elif self.data_type in ['ER','KR']:\n            X1,y = self.generate_specs(index)\n            X2,y = self.generate_raw(index)\n            X = (X1,X2)\n        elif self.data_type in ['KER']:\n            X1,y = self.generate_all_specs(index)\n            X2,y = self.generate_raw(index)\n            X = (X1,X2)\n        return X,y\n    \n    def generate_all_specs(self, index):\n        X = np.zeros((512,512,3),dtype='float32')\n        y = np.zeros((6,),dtype='float32')\n        \n        row = self.data.iloc[index]\n        if self.mode=='test': \n            offset = 0\n        else:\n            offset = int(row.offset/2)\n        \n        eeg = self.eeg_specs[row.eeg_id]\n        spec = self.specs[row.spec_id]\n        \n        imgs = [spec[offset:offset+300,k*100:(k+1)*100].T for k in [0,2,1,3]] # to match kaggle with eeg\n        img = np.stack(imgs,axis=-1)\n        # LOG TRANSFORM SPECTROGRAM\n        img = np.clip(img,np.exp(-4),np.exp(8))\n        img = np.log(img)\n            \n        # STANDARDIZE PER IMAGE\n        img = np.nan_to_num(img, nan=0.0)    \n            \n        mn = img.flatten().min()\n        mx = img.flatten().max()\n        ep = 1e-5\n        img = 255 * (img - mn) / (mx - mn + ep)\n        \n        X[0_0+56:100+56,:256,0] = img[:,22:-22,0] # LL_k\n        X[100+56:200+56,:256,0] = img[:,22:-22,2] # RL_k\n        X[0_0+56:100+56,:256,1] = img[:,22:-22,1] # LP_k\n        X[100+56:200+56,:256,1] = img[:,22:-22,3] # RP_k\n        X[0_0+56:100+56,:256,2] = img[:,22:-22,2] # RL_k\n        X[100+56:200+56,:256,2] = img[:,22:-22,1] # LP_k\n        \n        X[0_0+56:100+56,256:,0] = img[:,22:-22,0] # LL_k\n        X[100+56:200+56,256:,0] = img[:,22:-22,2] # RL_k\n        X[0_0+56:100+56,256:,1] = img[:,22:-22,1] # LP_k\n        X[100+56:200+56,256:,1] = img[:,22:-22,3] # RP_K\n        \n        # EEG\n        img = eeg\n        mn = img.flatten().min()\n        mx = img.flatten().max()\n        ep = 1e-5\n        img = 255 * (img - mn) / (mx - mn + ep)\n        X[200+56:300+56,:256,0] = img[:,22:-22,0] # LL_e\n        X[300+56:400+56,:256,0] = img[:,22:-22,2] # RL_e\n        X[200+56:300+56,:256,1] = img[:,22:-22,1] # LP_e\n        X[300+56:400+56,:256,1] = img[:,22:-22,3] # RP_e\n        X[200+56:300+56,:256,2] = img[:,22:-22,2] # RL_e\n        X[300+56:400+56,:256,2] = img[:,22:-22,1] # LP_e\n        \n        X[200+56:300+56,256:,0] = img[:,22:-22,0] # LL_e\n        X[300+56:400+56,256:,0] = img[:,22:-22,2] # RL_e\n        X[200+56:300+56,256:,1] = img[:,22:-22,1] # LP_e\n        X[300+56:400+56,256:,1] = img[:,22:-22,3] # RP_e\n\n        if self.mode!='test':\n            y[:] = row[TARGETS]\n        \n        return X,y\n    \n    def generate_specs(self, index):\n        X = np.zeros((512,512,3),dtype='float32')\n        y = np.zeros((6,),dtype='float32')\n        \n        row = self.data.iloc[index]\n        if self.mode=='test': \n            offset = 0\n        else:\n            offset = int(row.offset/2)\n            \n        if self.data_type in ['E','ER']:\n            img = self.eeg_specs[row.eeg_id]\n        elif self.data_type in ['K','KR']:\n            spec = self.specs[row.spec_id]\n            imgs = [spec[offset:offset+300,k*100:(k+1)*100].T for k in [0,2,1,3]] # to match kaggle with eeg\n            img = np.stack(imgs,axis=-1)\n            # LOG TRANSFORM SPECTROGRAM\n            img = np.clip(img,np.exp(-4),np.exp(8))\n            img = np.log(img)\n            \n            # STANDARDIZE PER IMAGE\n            img = np.nan_to_num(img, nan=0.0)    \n            \n        mn = img.flatten().min()\n        mx = img.flatten().max()\n        ep = 1e-5\n        img = 255 * (img - mn) / (mx - mn + ep)\n        \n        X[0_0+56:100+56,:256,0] = img[:,22:-22,0]\n        X[100+56:200+56,:256,0] = img[:,22:-22,2]\n        X[0_0+56:100+56,:256,1] = img[:,22:-22,1]\n        X[100+56:200+56,:256,1] = img[:,22:-22,3]\n        X[0_0+56:100+56,:256,2] = img[:,22:-22,2]\n        X[100+56:200+56,:256,2] = img[:,22:-22,1]\n        \n        X[0_0+56:100+56,256:,0] = img[:,22:-22,0]\n        X[100+56:200+56,256:,0] = img[:,22:-22,1]\n        X[0_0+56:100+56,256:,1] = img[:,22:-22,2]\n        X[100+56:200+56,256:,1] = img[:,22:-22,3]\n        \n        X[200+56:300+56,:256,0] = img[:,22:-22,0]\n        X[300+56:400+56,:256,0] = img[:,22:-22,1]\n        X[200+56:300+56,:256,1] = img[:,22:-22,2]\n        X[300+56:400+56,:256,1] = img[:,22:-22,3]\n        X[200+56:300+56,:256,2] = img[:,22:-22,3]\n        X[300+56:400+56,:256,2] = img[:,22:-22,2]\n        \n        X[200+56:300+56,256:,0] = img[:,22:-22,0]\n        X[300+56:400+56,256:,0] = img[:,22:-22,2]\n        X[200+56:300+56,256:,1] = img[:,22:-22,1]\n        X[300+56:400+56,256:,1] = img[:,22:-22,3]\n        \n        if self.mode!='test':\n            y[:] = row[TARGETS]\n        \n        return X,y\n    \n    def generate_raw(self,index):\n        if USE_PROCESSED and self.mode!='test':\n            X = np.zeros((2_000,8),dtype='float32')\n            y = np.zeros((6,),dtype='float32')\n            row = self.data.iloc[index]\n            X = self.raw_eegs[row.eeg_id]\n            y[:] = row[TARGETS]\n            return X,y\n        \n        X = np.zeros((10_000,8),dtype='float32')\n        y = np.zeros((6,),dtype='float32')\n        \n        row = self.data.iloc[index]\n        eeg = self.raw_eegs[row.eeg_id]\n            \n        # FEATURE ENGINEER\n        X[:,0] = eeg[:,FEAT2IDX['Fp1']] - eeg[:,FEAT2IDX['T3']]\n        X[:,1] = eeg[:,FEAT2IDX['T3']] - eeg[:,FEAT2IDX['O1']]\n            \n        X[:,2] = eeg[:,FEAT2IDX['Fp1']] - eeg[:,FEAT2IDX['C3']]\n        X[:,3] = eeg[:,FEAT2IDX['C3']] - eeg[:,FEAT2IDX['O1']]\n            \n        X[:,4] = eeg[:,FEAT2IDX['Fp2']] - eeg[:,FEAT2IDX['C4']]\n        X[:,5] = eeg[:,FEAT2IDX['C4']] - eeg[:,FEAT2IDX['O2']]\n            \n        X[:,6] = eeg[:,FEAT2IDX['Fp2']] - eeg[:,FEAT2IDX['T4']]\n        X[:,7] = eeg[:,FEAT2IDX['T4']] - eeg[:,FEAT2IDX['O2']]\n            \n        # STANDARDIZE\n        X = np.clip(X,-1024,1024)\n        X = np.nan_to_num(X, nan=0) / 32.0\n            \n        # BUTTER LOW-PASS FILTER\n        X = self.butter_lowpass_filter(X)\n        # Downsample\n        X = X[::5,:]\n        \n        if self.mode!='test':\n            y[:] = row[TARGETS]\n                \n        return X,y\n        \n    def butter_lowpass_filter(self, data, cutoff_freq=20, sampling_rate=200, order=4):\n        nyquist = 0.5 * sampling_rate\n        normal_cutoff = cutoff_freq / nyquist\n        b, a = butter(order, normal_cutoff, btype='low', analog=False)\n        filtered_data = lfilter(b, a, data, axis=0)\n        return filtered_data\n    \n    def resize(self, img,size):\n        composition = albu.Compose([\n                albu.Resize(size[0],size[1])\n            ])\n        return composition(image=img)['image']\n            \n    def augmentation(self, img):\n        composition = albu.Compose([\n                albu.HorizontalFlip(p=0.4)\n            ])\n        return composition(image=img)['image']\n\ndef spectrogram_from_eeg(parquet_path):\n    \n    # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n    eeg = pd.read_parquet(parquet_path)\n    middle = (len(eeg)-10_000)//2\n    eeg = eeg.iloc[middle:middle+10_000]\n    \n    # VARIABLE TO HOLD SPECTROGRAM\n    img = np.zeros((100,300,4),dtype='float32')\n\n    for k in range(4):\n        COLS = FEATS[k]\n        \n        for kk in range(4):\n            # FILL NANS\n            x1 = eeg[COLS[kk]].values\n            x2 = eeg[COLS[kk+1]].values\n            m = np.nanmean(x1)\n            if np.isnan(x1).mean()<1: x1 = np.nan_to_num(x1,nan=m)\n            else: x1[:] = 0\n            m = np.nanmean(x2)\n            if np.isnan(x2).mean()<1: x2 = np.nan_to_num(x2,nan=m)\n            else: x2[:] = 0\n                \n            # COMPUTE PAIR DIFFERENCES\n            x = x1 - x2\n\n            # RAW SPECTROGRAM\n            mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x)//300, \n                  n_fft=1024, n_mels=100, fmin=0, fmax=20, win_length=128)\n            \n            # LOG TRANSFORM\n            width = (mel_spec.shape[1]//30)*30\n            mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:,:width]\n            img[:,:,k] += mel_spec_db\n                \n        # AVERAGE THE 4 MONTAGE DIFFERENCES\n        img[:,:,k] /= 4.0\n          \n    return img\n\ndef eeg_from_parquet(parquet_path):\n\n    eeg = pd.read_parquet(parquet_path, columns=FEATS2)\n    rows = len(eeg)\n    offset = (rows-10_000)//2\n    eeg = eeg.iloc[offset:offset+10_000]\n    data = np.zeros((10_000,len(FEATS2)))\n    for j,col in enumerate(FEATS2):\n        \n        # FILL NAN\n        x = eeg[col].values.astype('float32')\n        m = np.nanmean(x)\n        if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n        else: x[:] = 0\n        \n        data[:,j] = x\n\n    return data","metadata":{"papermill":{"duration":2.248907,"end_time":"2024-03-03T17:13:54.628134","exception":false,"start_time":"2024-03-03T17:13:52.379227","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:03.100584Z","iopub.execute_input":"2024-03-07T23:11:03.100842Z","iopub.status.idle":"2024-03-07T23:11:04.904451Z","shell.execute_reply.started":"2024-03-07T23:11:03.100819Z","shell.execute_reply":"2024-03-07T23:11:04.903616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DISPLAY DATA GENERATOR\nBelow we display example data generator spectrogram images and raw EEG signals.","metadata":{"papermill":{"duration":0.008885,"end_time":"2024-03-03T17:13:54.646135","exception":false,"start_time":"2024-03-03T17:13:54.63725","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if not submission: \n    params = {'specs':spectrograms, 'eeg_specs':all_eegs, 'raw_eegs':all_raw_eegs}\n    gen = DataGenerator(train, augment=False, **params)\n    for x,y in gen:\n        break\n        \n    if DATA_TYPE in ['E','K','KE','KR','ER','KER']:\n        x1 = x[0] if DATA_TYPE in ['KR','ER','KER'] else x\n        plt.imshow(x1[:,:,0])\n        plt.title(f'Target = {y.round(1)}',size=12)\n        plt.yticks([])\n        plt.ylabel('Frequencies (Hz)',size=12)\n        plt.xlabel('Time (sec)',size=12)\n    \n    if DATA_TYPE in ['R','KR','ER','KER']:\n        x1 = x[1] if DATA_TYPE in ['KR','ER','KER'] else x\n        plt.figure(figsize=(20,4))\n        offset = 0\n        for j in range(x1.shape[-1]):\n            if j!=0: offset -= x1[:,j].min()\n            plt.plot(range(2_000),x1[:,j]+offset,label=f'feature {j+1}')\n            offset += x1[:,j].max()\n        plt.legend()\n        \n    plt.show()","metadata":{"papermill":{"duration":0.021571,"end_time":"2024-03-03T17:13:54.676501","exception":false,"start_time":"2024-03-03T17:13:54.65493","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:04.907025Z","iopub.execute_input":"2024-03-07T23:11:04.907505Z","iopub.status.idle":"2024-03-07T23:11:04.916547Z","shell.execute_reply.started":"2024-03-07T23:11:04.907478Z","shell.execute_reply":"2024-03-07T23:11:04.915549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TRAINING","metadata":{"papermill":{"duration":0.009379,"end_time":"2024-03-03T17:13:54.694693","exception":false,"start_time":"2024-03-03T17:13:54.685314","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## LEARNING RATE","metadata":{"papermill":{"duration":0.007933,"end_time":"2024-03-03T17:13:54.710721","exception":false,"start_time":"2024-03-03T17:13:54.702788","status":"completed"},"tags":[]}},{"cell_type":"code","source":"\nif not submission:\n\n    def lrfn(epoch):\n        e3 = 1e-3 if DATA_TYPE in ['R'] else 1e-4\n        return [1e-3,1e-3,e3,1e-4,1e-5][epoch]\n\n    LR = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n    \n    def lrfn2(epoch):\n        return [1e-5,1e-5,1e-6][epoch]\n\n    LR2 = tf.keras.callbacks.LearningRateScheduler(lrfn2, verbose = True)","metadata":{"papermill":{"duration":0.017031,"end_time":"2024-03-03T17:13:54.736108","exception":false,"start_time":"2024-03-03T17:13:54.719077","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:04.919844Z","iopub.execute_input":"2024-03-07T23:11:04.920209Z","iopub.status.idle":"2024-03-07T23:11:04.989598Z","shell.execute_reply.started":"2024-03-07T23:11:04.920184Z","shell.execute_reply":"2024-03-07T23:11:04.988716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## MODEL AND UTILITY FUNCTIONS","metadata":{"papermill":{"duration":0.007987,"end_time":"2024-03-03T17:13:54.752857","exception":false,"start_time":"2024-03-03T17:13:54.74487","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from tensorflow.keras.layers import Input, Dense, Multiply, Add, Conv1D, Concatenate, LayerNormalization\n\n\ndef build_model():\n    K.clear_session()\n    with strategy.scope():\n        if DATA_TYPE in ['R']:\n            model = build_wave_model()\n        elif DATA_TYPE in ['K','E','KE']:\n            model = build_spec_model()\n        elif DATA_TYPE in ['KR','ER','KER']:\n            model = build_hybrid_model()\n    return model\n\ndef build_spec_model(hybrid=False):  \n    inp = tf.keras.layers.Input((512,512,3))\n    base_model = load_model(f'{LOAD_BACKBONE_FROM}')    \n    x = base_model(inp)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    if not hybrid:\n        x = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(x)\n    model = tf.keras.Model(inputs=inp, outputs=x)\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.KLDivergence()\n    model.compile(loss=loss, optimizer=opt)  \n    return model\n\ndef wave_block(x, filters, kernel_size, n):\n    dilation_rates = [2**i for i in range(n)]\n    x = Conv1D(filters = filters,\n               kernel_size = 1,\n               padding = 'same')(x)\n    res_x = x\n    for dilation_rate in dilation_rates:\n        tanh_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same', \n                          activation = 'tanh', \n                          dilation_rate = dilation_rate)(x)\n        sigm_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same',\n                          activation = 'sigmoid', \n                          dilation_rate = dilation_rate)(x)\n        x = Multiply()([tanh_out, sigm_out])\n        x = Conv1D(filters = filters,\n                   kernel_size = 1,\n                   padding = 'same')(x)\n        res_x = Add()([res_x, x])\n    return res_x\n\ndef build_wave_model(hybrid=False):\n        \n    # INPUT \n    inp = tf.keras.Input(shape=(2_000,8))\n    \n    ############\n    # FEATURE EXTRACTION SUB MODEL\n    inp2 = tf.keras.Input(shape=(2_000,1))\n    x = wave_block(inp2, 8, 4, 6)\n    x = wave_block(x, 16, 4, 6)\n    x = wave_block(x, 32, 4, 6)\n    x = wave_block(x, 64, 4, 6)\n    model2 = tf.keras.Model(inputs=inp2, outputs=x)\n    ###########\n    \n    # LEFT TEMPORAL CHAIN\n    x1 = model2(inp[:,:,0:1])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,1:2])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z1 = tf.keras.layers.Average()([x1,x2])\n    \n    # LEFT PARASAGITTAL CHAIN\n    x1 = model2(inp[:,:,2:3])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,3:4])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z2 = tf.keras.layers.Average()([x1,x2])\n    \n    # RIGHT PARASAGITTAL CHAIN\n    x1 = model2(inp[:,:,4:5])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,5:6])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z3 = tf.keras.layers.Average()([x1,x2])\n    \n    # RIGHT TEMPORAL CHAIN\n    x1 = model2(inp[:,:,6:7])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,7:8])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z4 = tf.keras.layers.Average()([x1,x2])\n    \n    # COMBINE CHAINS\n    y = tf.keras.layers.Concatenate()([z1,z2,z3,z4])\n    if not hybrid:\n        y = tf.keras.layers.Dense(64, activation='relu')(y)\n        y = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(y)\n    \n    # COMPILE MODEL\n    model = tf.keras.Model(inputs=inp, outputs=y)\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.KLDivergence()\n    model.compile(loss=loss, optimizer = opt)\n    \n    return model\n\ndef build_hybrid_model():\n    model_spec = build_spec_model(True)\n    model_wave = build_wave_model(True)\n    inputs = [model_spec.input, model_wave.input]\n    x = [model_spec.output, model_wave.output]\n    x = tf.keras.layers.Concatenate()(x)\n    x = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(x)\n    \n    # COMPILE MODEL\n    model = tf.keras.Model(inputs=inputs, outputs=x)\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.KLDivergence()\n    model.compile(loss=loss, optimizer = opt)\n    \n    return model\n\ndef score(y_true, y_pred):\n    kl = tf.keras.metrics.KLDivergence()\n    return kl(y_true, y_pred)\n\ndef plot_hist(hist):\n    metrics = ['loss']\n    for i,metric in enumerate(metrics):\n        plt.figure(figsize=(10,4))\n        plt.subplot(1,2,i+1)\n        plt.plot(hist[metric])\n        plt.plot(hist[f'val_{metric}'])\n        plt.title(f'{metric}',size=12)\n        plt.ylabel(f'{metric}',size=12)\n        plt.xlabel('epoch',size=12)\n        plt.legend([\"train\", \"validation\"], loc=\"upper left\")\n        plt.show()\n        \ndef dataset(data, mode='train', batch_size=8, data_type=DATA_TYPE, \n            augment=False, specs=None, eeg_specs=None, raw_eegs=None):\n    \n    BATCH_SIZE_PER_REPLICA = batch_size\n    BATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync\n    gen = DataGenerator(data,mode=mode, data_type=data_type, augment=augment,\n                       specs=specs, eeg_specs=eeg_specs, raw_eegs=raw_eegs)\n    if data_type in ['K','E','KE']: \n        inp = tf.TensorSpec(shape=(512,512,3), dtype=tf.float32)\n    elif data_type in ['KR','ER','KER']:\n        inp = (tf.TensorSpec(shape=(512,512,3), dtype=tf.float32),tf.TensorSpec(shape=(2000,8), dtype=tf.float32))\n    elif data_type in ['R']:\n        inp = tf.TensorSpec(shape=(2000,8), dtype=tf.float32)\n        \n    output_signature = (inp,tf.TensorSpec(shape=(6,), dtype=tf.float32))\n    dataset = tf.data.Dataset.from_generator(generator=gen, output_signature=output_signature).batch(\n        BATCH_SIZE)\n    return dataset\n\ndef reset_seed(seed):\n    np.random.seed(seed)\n    random.seed(seed)\n    tf.random.set_seed(seed)","metadata":{"papermill":{"duration":0.038778,"end_time":"2024-03-03T17:13:54.800238","exception":false,"start_time":"2024-03-03T17:13:54.76146","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:04.991078Z","iopub.execute_input":"2024-03-07T23:11:04.991412Z","iopub.status.idle":"2024-03-07T23:11:05.027228Z","shell.execute_reply.started":"2024-03-07T23:11:04.991382Z","shell.execute_reply":"2024-03-07T23:11:05.02646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TRANSFER LEARNING","metadata":{"papermill":{"duration":0.008661,"end_time":"2024-03-03T17:13:54.817886","exception":false,"start_time":"2024-03-03T17:13:54.809225","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold\nimport tensorflow.keras.backend as K, gc\n\nif not submission:\n    reset_seed(42)\n    all_oof = []\n    all_true = []\n    losses = []\n    val_losses = []\n    total_hist = {}\n\n    gkf = GroupKFold(n_splits=5)\n    for i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):   \n        print('#'*25)\n        print(f'### Fold {i+1}')\n        \n        EPOCHS = 5\n        params = {'specs':spectrograms, 'eeg_specs':all_eegs, 'raw_eegs':all_raw_eegs}\n        data, val = train.iloc[train_index],train.iloc[valid_index]\n        train_dataset = dataset(data, **params)\n        val_dataset = dataset(val,mode='valid', **params)\n        data = data[data['kl']<5.5]\n        train_dataset2 = dataset(data, **params)\n        \n        print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n        print('#'*25)\n        model = build_model()\n        hist = model.fit(train_dataset, validation_data = val_dataset, \n                         epochs=EPOCHS, callbacks=[LR])\n        print(f'### seconds stage train size {len(data)}, valid size {len(val)}')\n        print('#'*25)\n        hist2 = model.fit(train_dataset2, validation_data = val_dataset, \n                         epochs=3, callbacks=[LR2])\n        losses.append(hist.history['loss']+hist2.history['loss'])\n        val_losses.append(hist.history['val_loss']+hist2.history['val_loss'])\n        with strategy.scope():\n            model.save_weights(f'model_{DATA_TYPE}_{VER}_{i}.weights.h5')\n        oof = model.predict(val_dataset, verbose=1)\n        all_oof.append(oof)\n        all_true.append(train.iloc[valid_index][TARGETS].values)    \n        del model, oof\n        gc.collect()\n        \n    total_hist['loss'] = np.mean(losses,axis=0)\n    total_hist['val_loss'] = np.mean(val_losses,axis=0)\n    all_oof = np.concatenate(all_oof)\n    all_true = np.concatenate(all_true)\n    plot_hist(total_hist)\n    print('#'*25)\n    print(f'CV KL SCORE: {score(all_true,all_oof)}')","metadata":{"papermill":{"duration":0.032338,"end_time":"2024-03-03T17:13:54.859116","exception":false,"start_time":"2024-03-03T17:13:54.826778","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:05.028336Z","iopub.execute_input":"2024-03-07T23:11:05.02865Z","iopub.status.idle":"2024-03-07T23:11:05.04365Z","shell.execute_reply.started":"2024-03-07T23:11:05.028621Z","shell.execute_reply":"2024-03-07T23:11:05.042821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Infer Test and Create Submission CSV\nInfer the test data and create a `submission.csv` file.","metadata":{"papermill":{"duration":0.008533,"end_time":"2024-03-03T17:13:54.87667","exception":false,"start_time":"2024-03-03T17:13:54.868137","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if submission:\n    test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\n    print('Test shape',test.shape)\n    test.head()","metadata":{"papermill":{"duration":0.028182,"end_time":"2024-03-03T17:13:54.959119","exception":false,"start_time":"2024-03-03T17:13:54.930937","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:05.044679Z","iopub.execute_input":"2024-03-07T23:11:05.044975Z","iopub.status.idle":"2024-03-07T23:11:05.069843Z","shell.execute_reply.started":"2024-03-07T23:11:05.044952Z","shell.execute_reply":"2024-03-07T23:11:05.068862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# READ ALL SPECTROGRAMS\nif submission:\n    PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms'\n    files2 = os.listdir(PATH2)\n    print(f'There are {len(files2)} test spectrogram parquets')\n    \n    spectrograms2 = {}\n    for i,f in enumerate(files2):\n        if i%100==0: print(i,', ',end='')\n        tmp = pd.read_parquet(f'{PATH2}/{f}')\n        name = int(f.split('.')[0])\n        spectrograms2[name] = tmp.iloc[:,1:].values\n    \n    # RENAME FOR DATA GENERATOR\n    test = test.rename({'spectrogram_id':'spec_id'},axis=1)","metadata":{"papermill":{"duration":0.237311,"end_time":"2024-03-03T17:13:55.20519","exception":false,"start_time":"2024-03-03T17:13:54.967879","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:05.070946Z","iopub.execute_input":"2024-03-07T23:11:05.071308Z","iopub.status.idle":"2024-03-07T23:11:05.329493Z","shell.execute_reply.started":"2024-03-07T23:11:05.071276Z","shell.execute_reply":"2024-03-07T23:11:05.3287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# READ ALL EEG SPECTROGRAMS\nif submission:\n    PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs'\n    DISPLAY = 0\n    EEG_IDS2 = test.eeg_id.unique()\n    all_eegs2 = {}\n\n    print('Converting Test EEG to Spectrograms...'); print()\n    for i,eeg_id in enumerate(EEG_IDS2):\n        \n        # CREATE SPECTROGRAM FROM EEG PARQUET\n        img = spectrogram_from_eeg(f'{PATH2}/{eeg_id}.parquet')\n        all_eegs2[eeg_id] = img","metadata":{"papermill":{"duration":10.319308,"end_time":"2024-03-03T17:14:05.533196","exception":false,"start_time":"2024-03-03T17:13:55.213888","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:05.330596Z","iopub.execute_input":"2024-03-07T23:11:05.330883Z","iopub.status.idle":"2024-03-07T23:11:15.058635Z","shell.execute_reply.started":"2024-03-07T23:11:05.330858Z","shell.execute_reply":"2024-03-07T23:11:15.05737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# READ ALL RAW EEG SIGNALS\nif submission :\n    all_raw_eegs2 = {}\n    EEG_IDS2 = test.eeg_id.unique()\n    PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs'\n\n    print('Processing Test EEG parquets...'); print()\n    for i,eeg_id in enumerate(EEG_IDS2):\n        \n        # SAVE EEG TO PYTHON DICTIONARY OF NUMPY ARRAYS\n        data = eeg_from_parquet(f'{PATH2}/{eeg_id}.parquet')\n        all_raw_eegs2[eeg_id] = data","metadata":{"papermill":{"duration":0.061557,"end_time":"2024-03-03T17:14:05.615731","exception":false,"start_time":"2024-03-03T17:14:05.554174","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:15.06089Z","iopub.execute_input":"2024-03-07T23:11:15.06233Z","iopub.status.idle":"2024-03-07T23:11:15.091025Z","shell.execute_reply.started":"2024-03-07T23:11:15.062278Z","shell.execute_reply":"2024-03-07T23:11:15.089906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Submission ON TEST without ensemble\nif submission and not ENSEMBLE:\n    preds = []\n    \n    test_dataset = dataset(test,mode='test',specs=spectrograms2, eeg_specs=all_eegs2, raw_eegs=all_raw_eegs2)\n    model = build_model()\n\n    for i in range(5):\n        print(f'Fold {i+1}')\n        model.load_weights(f'{LOAD_MODELS_FROM}/model_{DATA_TYPE}_{VER}_{i}.weights.h5')\n        pred = model.predict(test_dataset, verbose=1)\n        preds.append(pred)\n        \n    pred = np.mean(preds,axis=0)\n    print('Test preds shape',pred.shape)","metadata":{"papermill":{"duration":0.033962,"end_time":"2024-03-03T17:14:05.671441","exception":false,"start_time":"2024-03-03T17:14:05.637479","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:15.096454Z","iopub.execute_input":"2024-03-07T23:11:15.100162Z","iopub.status.idle":"2024-03-07T23:11:15.113741Z","shell.execute_reply.started":"2024-03-07T23:11:15.100103Z","shell.execute_reply":"2024-03-07T23:11:15.112335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Submission ON TEST with ensemble\nif submission and ENSEMBLE:\n    preds = []\n    params = {'specs':spectrograms2, 'eeg_specs':all_eegs2, 'raw_eegs':all_raw_eegs2}\n    test_dataset_K = dataset(test, data_type='K', mode='test', **params)\n    test_dataset_E = dataset(test, data_type='E', mode='test', **params)\n    test_dataset_R = dataset(test, data_type='R', mode='test', **params)\n    test_dataset_KE = dataset(test, data_type='KE', mode='test', **params)\n    test_dataset_KR = dataset(test, data_type='KR', mode='test', **params)\n    test_dataset_ER = dataset(test, data_type='ER', mode='test', **params)\n    test_dataset_KER = dataset(test, data_type='KER', mode='test', **params)\n\n    # LB SCORE WEIGHTS FOR EACH MODEL\n    lbs = 1 - np.array(LBs)\n    weights = lbs/lbs.sum()\n    model_spec = build_spec_model()\n    model_wave = build_wave_model()\n    model_hybrid = build_hybrid_model()\n    \n    for i in range(5):\n        print(f'Fold {i+1}')\n        \n        model_spec.load_weights(f'{LOAD_MODELS_FROM}/model_K_{VER_K}_{i}.weights.h5')\n        pred_K = model_spec.predict(test_dataset_K, verbose=1)\n        \n        model_spec.load_weights(f'{LOAD_MODELS_FROM}/model_E_{VER_E}_{i}.weights.h5')\n        pred_E = model_spec.predict(test_dataset_E, verbose=1)\n\n        model_wave.load_weights(f'{LOAD_MODELS_FROM}/model_R_{VER_R}_{i}.weights.h5')\n        pred_R = model_wave.predict(test_dataset_R, verbose=1)\n        \n        model_spec.load_weights(f'{LOAD_MODELS_FROM}/model_KE_{VER_KE}_{i}.weights.h5')\n        pred_KE = model_spec.predict(test_dataset_KE, verbose=1)\n        \n        model_hybrid.load_weights(f'{LOAD_MODELS_FROM}/model_KR_{VER_KR}_{i}.weights.h5')\n        pred_KR = model_hybrid.predict(test_dataset_KR, verbose=1)\n        \n        model_hybrid.load_weights(f'{LOAD_MODELS_FROM}/model_ER_{VER_ER}_{i}.weights.h5')\n        pred_ER = model_hybrid.predict(test_dataset_ER, verbose=1)\n        \n        model_hybrid.load_weights(f'{LOAD_MODELS_FROM}/model_KER_{VER_KER}_{i}.weights.h5')\n        pred_KER = model_hybrid.predict(test_dataset_KER, verbose=1)\n        \n        pred = np.array([pred_K,pred_E,pred_R,pred_KE,pred_KR,pred_ER,pred_KER])\n        pred = np.average(pred,axis=0,weights=weights)\n        preds.append(pred)\n        \n    pred = np.mean(preds,axis=0)\n    print('Test preds shape',pred.shape)","metadata":{"papermill":{"duration":66.385954,"end_time":"2024-03-03T17:15:12.067289","exception":false,"start_time":"2024-03-03T17:14:05.681335","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-07T23:11:15.123974Z","iopub.execute_input":"2024-03-07T23:11:15.127565Z","iopub.status.idle":"2024-03-07T23:13:26.839773Z","shell.execute_reply.started":"2024-03-07T23:11:15.127516Z","shell.execute_reply":"2024-03-07T23:13:26.8387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Catboost notebook**","metadata":{}},{"cell_type":"code","source":"import os, gc\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\n\nVER = 3","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape )\nprint('Targets', list(TARGETS))\ndf.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n    train[t] = tmp[t].values\n    \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\nprint('Train non-overlapp eeg_id shape:', train.shape )\ntrain.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"READ_SPEC_FILES = False\nREAD_EEG_SPEC_FILES = False","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# READ ALL SPECTROGRAMS\nPATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/'\nfiles = os.listdir(PATH)\nprint(f'There are {len(files)} spectrogram parquets')\n\nif READ_SPEC_FILES:    \n    spectrograms = {}\n    for i,f in enumerate(files):\n        if i%100==0: print(i,', ',end='')\n        tmp = pd.read_parquet(f'{PATH}{f}')\n        name = int(f.split('.')[0])\n        spectrograms[name] = tmp.iloc[:,1:].values\nelse:\n    spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# READ ALL EEG SPECTROGRAMS\nif READ_EEG_SPEC_FILES:\n    all_eegs = {}\n    for i,e in enumerate(train.eeg_id.values):\n        if i%100==0: print(i,', ',end='')\n        x = np.load(f'/kaggle/input/brain-eeg-spectrograms/EEG_Spectrograms/{e}.npy')\n        all_eegs[e] = x\nelse:\n    all_eegs = np.load('/kaggle/input/brain-eeg-spectrograms/eeg_specs.npy',allow_pickle=True).item()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time\n# ENGINEER FEATURES\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# FEATURE NAMES\nSPEC_COLS = pd.read_parquet(f'{PATH}1000086677.parquet').columns[1:]\nFEATURES = [f'{c}_mean_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_mean_20s' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_20s' for c in SPEC_COLS]\nFEATURES += [f'eeg_mean_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_min_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_max_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_std_f{x}_10s' for x in range(512)]\nprint(f'We are creating {len(FEATURES)} features for {len(train)} rows... ',end='')\n\ndata = np.zeros((len(train),len(FEATURES)))\nfor k in range(len(train)):\n    if k%100==0: print(k,', ',end='')\n    row = train.iloc[k]\n    r = int( (row['min'] + row['max'])//4 ) \n\n    # 10 MINUTE WINDOW FEATURES (MEANS and MINS)\n    x = np.nanmean(spectrograms[row.spec_id][r:r+300,:],axis=0)\n    data[k,:400] = x\n    x = np.nanmin(spectrograms[row.spec_id][r:r+300,:],axis=0)\n    data[k,400:800] = x\n\n    # 20 SECOND WINDOW FEATURES (MEANS and MINS)\n    x = np.nanmean(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n    data[k,800:1200] = x\n    x = np.nanmin(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n    data[k,1200:1600] = x\n\n    # RESHAPE EEG SPECTROGRAMS 128x256x4 => 512x256\n    eeg_spec = np.zeros((512,256),dtype='float32')\n    xx = all_eegs[row.eeg_id]\n    for j in range(4): eeg_spec[128*j:128*(j+1),] = xx[:,:,j]\n\n    # 10 SECOND WINDOW FROM EEG SPECTROGRAMS \n    x = np.nanmean(eeg_spec.T[100:-100,:],axis=0)\n    data[k,1600:2112] = x\n    x = np.nanmin(eeg_spec.T[100:-100,:],axis=0)\n    data[k,2112:2624] = x\n    x = np.nanmax(eeg_spec.T[100:-100,:],axis=0)\n    data[k,2624:3136] = x\n    x = np.nanstd(eeg_spec.T[100:-100,:],axis=0)\n    data[k,3136:3648] = x\n\ntrain[FEATURES] = data\nprint(); print('New train shape:',train.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# FREE MEMORY\ndel all_eegs, spectrograms, data\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import catboost as cat\nfrom catboost import CatBoostClassifier, Pool\nprint('CatBoost version',cat.__version__)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold\n\nall_oof = []\nall_true = []\nTARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):   \n    \n    print('#'*25)\n    print(f'### Fold {i+1}')\n    print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n    print('#'*25)\n    \n    model = CatBoostClassifier(task_type='GPU',\n                               loss_function='MultiClass')\n    \n    train_pool = Pool(\n        data = train.loc[train_index,FEATURES],\n        label = train.loc[train_index,'target'].map(TARS),\n    )\n    \n    valid_pool = Pool(\n        data = train.loc[valid_index,FEATURES],\n        label = train.loc[valid_index,'target'].map(TARS),\n    )\n    \n    model.fit(train_pool,\n             verbose=100,\n             eval_set=valid_pool,\n             )\n    model.save_model(f'CAT_v{VER}_f{i}.cat')\n    \n    oof = model.predict_proba(valid_pool)\n    all_oof.append(oof)\n    all_true.append(train.loc[valid_index, TARGETS].values)\n    \n    del train_pool, valid_pool, oof #model\n    gc.collect()\n    \n    #break\n    \nall_oof = np.concatenate(all_oof)\nall_true = np.concatenate(all_true)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TOP = 25\n\nfeature_importance = model.feature_importances_\nsorted_idx = np.argsort(feature_importance)\nfig = plt.figure(figsize=(10, 8))\nplt.barh(np.arange(len(sorted_idx))[-TOP:], feature_importance[sorted_idx][-TOP:], align='center')\nplt.yticks(np.arange(len(sorted_idx))[-TOP:], np.array(FEATURES)[sorted_idx][-TOP:])\nplt.title(f'Feature Importance - Top {TOP}')\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score KL-Div for CatBoost =',cv)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof.copy())\nfor c in oof.columns:\n    oof[c] = 1/6.\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score for \"Use Equal Preds 1/6\" =',cv)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_oof2 = []\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):  \n    #print('#'*25)\n    #print(f'### Fold {i+1}')\n        \n    y_train = train.iloc[train_index][TARGETS].values\n    y_valid = train.iloc[valid_index][TARGETS].values\n    \n    #print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n    #print('#'*25)\n        \n    oof = y_valid.copy()\n    for j in range(6):\n        oof[:,j] = y_train[:,j].mean()\n    oof = oof / oof.sum(axis=1,keepdims=True)\n    all_oof2.append(oof)\n    \nall_oof2 = np.concatenate(all_oof2)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof2.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score for \"Use Train Means\" =',cv)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train; gc.collect()\ntest = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\nprint('Test shape',test.shape)\ntest.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pywt, librosa\n\nUSE_WAVELET = None \n\nNAMES = ['LL','LP','RP','RR']\n\nFEATS = [['Fp1','F7','T3','T5','O1'],\n         ['Fp1','F3','C3','P3','O1'],\n         ['Fp2','F8','T4','T6','O2'],\n         ['Fp2','F4','C4','P4','O2']]\n\n# DENOISE FUNCTION\ndef maddest(d, axis=None):\n    return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\ndef denoise(x, wavelet='haar', level=1):    \n    coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n    sigma = (1/0.6745) * maddest(coeff[-level])\n\n    uthresh = sigma * np.sqrt(2*np.log(len(x)))\n    coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n    ret=pywt.waverec(coeff, wavelet, mode='per')\n    \n    return ret\n\ndef spectrogram_from_eeg(parquet_path, display=False):\n    \n    # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n    eeg = pd.read_parquet(parquet_path)\n    middle = (len(eeg)-10_000)//2\n    eeg = eeg.iloc[middle:middle+10_000]\n    \n    # VARIABLE TO HOLD SPECTROGRAM\n    img = np.zeros((128,256,4),dtype='float32')\n    \n    if display: plt.figure(figsize=(10,7))\n    signals = []\n    for k in range(4):\n        COLS = FEATS[k]\n        \n        for kk in range(4):\n        \n            # COMPUTE PAIR DIFFERENCES\n            x = eeg[COLS[kk]].values - eeg[COLS[kk+1]].values\n\n            # FILL NANS\n            m = np.nanmean(x)\n            if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n            else: x[:] = 0\n\n            # DENOISE\n            if USE_WAVELET:\n                x = denoise(x, wavelet=USE_WAVELET)\n            signals.append(x)\n\n            # RAW SPECTROGRAM\n            mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x)//256, \n                  n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n\n            # LOG TRANSFORM\n            width = (mel_spec.shape[1]//32)*32\n            mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:,:width]\n\n            # STANDARDIZE TO -1 TO 1\n            mel_spec_db = (mel_spec_db+40)/40 \n            img[:,:,k] += mel_spec_db\n                \n        # AVERAGE THE 4 MONTAGE DIFFERENCES\n        img[:,:,k] /= 4.0\n        \n        if display:\n            plt.subplot(2,2,k+1)\n            plt.imshow(img[:,:,k],aspect='auto',origin='lower')\n            plt.title(f'EEG {eeg_id} - Spectrogram {NAMES[k]}')\n            \n    if display: \n        plt.show()\n        plt.figure(figsize=(10,5))\n        offset = 0\n        for k in range(4):\n            if k>0: offset -= signals[3-k].min()\n            plt.plot(range(10_000),signals[k]+offset,label=NAMES[3-k])\n            offset += signals[3-k].max()\n        plt.legend()\n        plt.title(f'EEG {eeg_id} Signals')\n        plt.show()\n        print(); print('#'*25); print()\n        \n    return img","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CREATE ALL EEG SPECTROGRAMS\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\nDISPLAY = 0\nEEG_IDS2 = test.eeg_id.unique()\nall_eegs2 = {}\n\nprint('Converting Test EEG to Spectrograms...'); print()\nfor i,eeg_id in enumerate(EEG_IDS2):\n        \n    # CREATE SPECTROGRAM FROM EEG PARQUET\n    img = spectrogram_from_eeg(f'{PATH2}{eeg_id}.parquet', i<DISPLAY)\n    all_eegs2[eeg_id] = img","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# FEATURE ENGINEER TEST\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\ndata = np.zeros((len(test),len(FEATURES)))\n    \nfor k in range(len(test)):\n    row = test.iloc[k]\n    s = int( row.spectrogram_id )\n    spec = pd.read_parquet(f'{PATH2}{s}.parquet')\n    \n    # 10 MINUTE WINDOW FEATURES\n    x = np.nanmean( spec.iloc[:,1:].values, axis=0)\n    data[k,:400] = x\n    x = np.nanmin( spec.iloc[:,1:].values, axis=0)\n    data[k,400:800] = x\n\n    # 20 SECOND WINDOW FEATURES\n    x = np.nanmean( spec.iloc[145:155,1:].values, axis=0)\n    data[k,800:1200] = x\n    x = np.nanmin( spec.iloc[145:155,1:].values, axis=0)\n    data[k,1200:1600] = x\n    \n    # RESHAPE EEG SPECTROGRAMS 128x256x4 => 512x256\n    eeg_spec = np.zeros((512,256),dtype='float32')\n    xx = all_eegs2[row.eeg_id]\n    for j in range(4): eeg_spec[128*j:128*(j+1),] = xx[:,:,j]\n\n    # 10 SECOND WINDOW FROM EEG SPECTROGRAMS \n    x = np.nanmean(eeg_spec.T[100:-100,:],axis=0)\n    data[k,1600:2112] = x\n    x = np.nanmin(eeg_spec.T[100:-100,:],axis=0)\n    data[k,2112:2624] = x\n    x = np.nanmax(eeg_spec.T[100:-100,:],axis=0)\n    data[k,2624:3136] = x\n    x = np.nanstd(eeg_spec.T[100:-100,:],axis=0)\n    data[k,3136:3648] = x\n\ntest[FEATURES] = data\nprint('New test shape',test.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER CATBOOST ON TEST\npreds = []\n\nfor i in range(5):\n    print(i,', ',end='')\n    model = CatBoostClassifier(task_type='GPU')\n    model.load_model(f'CAT_v{VER}_f{i}.cat')\n    \n    test_pool = Pool(\n        data = test[FEATURES]\n    )\n    \n    pred_1 = model.predict_proba(test_pool)\n    preds.append(pred_1)\npred_1 = np.mean(preds,axis=0)\nprint()\nprint('Test preds shape',pred_1.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**wavenet-tensorflow notebook**","metadata":{}},{"cell_type":"code","source":"import numpy as np, pandas as pd, tensorflow as tf","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols = ['eeg_id', 'spectrogram_id', 'seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\nlabel_cols = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\n\ntrain_df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv', usecols=cols)\ntrain_df.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data = train_df.groupby(['eeg_id', 'spectrogram_id'], as_index=False).sum()\ndata['total_vote'] = data[label_cols].sum(axis=1)\n\nfor label in label_cols:\n    data[label] = data[label] / data['total_vote']\n    \ndata['eeg_path'] = data['eeg_id'].apply(lambda x: '/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/'+str(x)+'.parquet')\ndata['spectrogram_path'] = data['spectrogram_id'].apply(lambda x: '/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/'+str(x)+'.parquet')\ndata.drop(columns=['eeg_id', 'spectrogram_id', 'total_vote'], inplace=True)\ndata.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_num = int(0.8 * data.shape[0])\ntrain_data, val_data = data.iloc[:train_num], data.iloc[train_num:]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from scipy.signal import butter, lfilter\n\ndef butter_lowpass_filter(data, cutoff_freq=20, sampling_rate=200, order=4):\n    nyquist = 0.5 * sampling_rate\n    normal_cutoff = cutoff_freq / nyquist\n    b, a = butter(order, normal_cutoff, btype='low', analog=False)\n    filtered_data = lfilter(b, a, data, axis=1)\n    return filtered_data","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class DataGenerator(tf.keras.utils.Sequence):\n    def __init__(self, tmp_df, batch_size=32, shuffle=False, mode='train'): \n\n        self.batch_size = batch_size\n        self.mode = mode\n        \n        eeg_data = [] \n        for i, row in tmp_df.iterrows():\n            \n            FEATS = ['Fp1','T3','C3','O1','Fp2','C4','T4','O2']\n            eeg_tmp = pd.read_parquet(row['eeg_path'], columns=FEATS).fillna(0).values.astype('float32')\n            eeg_offset = (len(eeg_tmp)-10000) // 2\n            eeg_data.append(eeg_tmp[eeg_offset:(eeg_offset+10000):5])\n            \n        eeg_data = np.array(eeg_data)\n        \n        self.eeg_data_agg = np.empty(shape=(eeg_data.shape[0], 2000, 8))\n        self.eeg_data_agg[:,:,0] = eeg_data[:,:,0] - eeg_data[:,:,1]\n        self.eeg_data_agg[:,:,1] = eeg_data[:,:,1] - eeg_data[:,:,3]\n        self.eeg_data_agg[:,:,2] = eeg_data[:,:,0] - eeg_data[:,:,2]\n        self.eeg_data_agg[:,:,3] = eeg_data[:,:,2] - eeg_data[:,:,3]\n        self.eeg_data_agg[:,:,4] = eeg_data[:,:,4] - eeg_data[:,:,5]\n        self.eeg_data_agg[:,:,5] = eeg_data[:,:,5] - eeg_data[:,:,7]\n        self.eeg_data_agg[:,:,6] = eeg_data[:,:,4] - eeg_data[:,:,6]\n        self.eeg_data_agg[:,:,7] = eeg_data[:,:,6] - eeg_data[:,:,7]\n        \n        del eeg_data\n\n        self.eeg_data_agg = np.clip(self.eeg_data_agg, -1024, 1024) \n        self.eeg_data_agg = np.nan_to_num(self.eeg_data_agg, nan=0) / 32.0\n        self.eeg_data_agg = butter_lowpass_filter(self.eeg_data_agg)\n     \n        if self.mode != 'test':\n            self.labels = tmp_df[label_cols].values\n       \n    def __len__(self):\n        return int(np.ceil(len(self.eeg_data_agg) / self.batch_size))\n\n    def __getitem__(self, idx):\n        X = self.eeg_data_agg[idx*self.batch_size:(idx+1)*self.batch_size]\n        y = np.zeros((self.batch_size,6),dtype='float32')\n        if self.mode != 'test':\n            y = self.labels[idx*self.batch_size:(idx+1)*self.batch_size]\n        return X, y ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.layers import Input, Dense, Multiply, Add, Conv1D, Concatenate\n\ndef wave_block(x, filters, kernel_size, n):\n    dilation_rates = [2**i for i in range(n)]\n    x = Conv1D(filters = filters,\n               kernel_size = 1,\n               padding = 'same')(x)\n    res_x = x\n    for dilation_rate in dilation_rates:\n        tanh_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same', \n                          activation = 'tanh', \n                          dilation_rate = dilation_rate)(x)\n        sigm_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same',\n                          activation = 'sigmoid', \n                          dilation_rate = dilation_rate)(x)\n        x = Multiply()([tanh_out, sigm_out])\n        x = Conv1D(filters = filters,\n                   kernel_size = 1,\n                   padding = 'same')(x)\n        res_x = Add()([res_x, x])\n    return res_x","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model():\n\n    inp = tf.keras.Input(shape=(2_000,8))\n\n    inp2 = tf.keras.Input(shape=(2_000,1))\n    x = wave_block(inp2, 8, 3, 12)\n    x = wave_block(x, 16, 3, 8)\n    x = wave_block(x, 32, 3, 4)\n    x = wave_block(x, 64, 3, 1)\n    model2 = tf.keras.Model(inputs=inp2, outputs=x)\n    \n    x1 = model2(inp[:,:,0:1])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,1:2])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z1 = tf.keras.layers.Average()([x1,x2])\n    \n    x1 = model2(inp[:,:,2:3])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,3:4])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z2 = tf.keras.layers.Average()([x1,x2])\n    \n    x1 = model2(inp[:,:,4:5])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,5:6])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z3 = tf.keras.layers.Average()([x1,x2])\n\n    x1 = model2(inp[:,:,6:7])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,7:8])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z4 = tf.keras.layers.Average()([x1,x2])\n\n    y = tf.keras.layers.Concatenate()([z1,z2,z3,z4])\n    y = tf.keras.layers.Dense(64, activation='relu')(y)\n    y = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(y)\n    \n    model = tf.keras.Model(inputs=inp, outputs=y)\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.KLDivergence()\n    model.compile(loss=loss, optimizer = opt)\n    \n    return model","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nos.environ['CUDA_VISIBLE_DEVICES']='0,1'\ngpus = tf.config.list_physical_devices('GPU')\nif len(gpus)<=1: \n    strategy = tf.distribute.OneDeviceStrategy(device='/gpu:0')\nelse: \n    strategy = tf.distribute.MirroredStrategy()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_gen = DataGenerator(train_data, shuffle=True, batch_size=32, mode='train') \nval_gen = DataGenerator(val_data, shuffle=False, batch_size=32, mode='val')\n\nwith strategy.scope():\n    model = build_model() \n    \nmodel.fit(train_gen, verbose=1, validation_data = val_gen, epochs=5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\ntest_df['eeg_path'] = test_df['eeg_id'].apply(lambda x: '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'+str(x)+'.parquet')\ntest_df['spectrogram_path'] = test_df['spectrogram_id'].apply(lambda x: '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'+str(x)+'.parquet')\ntest_df\n\ntest_gen = DataGenerator(test_df, shuffle=True, batch_size=32, mode='test')\ntest_preds = model.predict(test_gen)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission=pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/sample_submission.csv\")\nlabels=['seizure','lpd','gpd','lrda','grda','other']\nfor i in range(len(labels)):\n    submission[f'{labels[i]}_vote']=(pred[:,i]*0.9 + test_preds[:, i]*0.04 + pred_1[:, i]*0.06)\nsubmission.to_csv(\"submission.csv\",index=None)\ndisplay(submission.head())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission.iloc[:,-6:].sum(axis=1)","metadata":{},"execution_count":null,"outputs":[]}]}