{"metadata":{"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7465251,"sourceType":"datasetVersion","datasetId":4317718},{"sourceId":7570342,"sourceType":"datasetVersion","datasetId":4407194},{"sourceId":7679679,"sourceType":"datasetVersion","datasetId":4480381},{"sourceId":7752462,"sourceType":"datasetVersion","datasetId":4382744},{"sourceId":7818976,"sourceType":"datasetVersion","datasetId":4417235}],"dockerImageVersionId":30635,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"papermill":{"default_parameters":{},"duration":98.666323,"end_time":"2024-02-28T19:49:51.36372","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-02-28T19:48:12.697397","version":"2.4.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Features+Head Ensemble Starter [LB 0.36] for HMS Brain Comp\nThis is Features+Head is a combination and ensemble Starter notebook for Kaggle's HMS brain comp. We can train 4 different models using:\n- Kaggle's spectrograms (CV 0.6123 – LB 0.41)\n- Chris's EEG spectrograms(modified version) (CV 0.6288 – LB 0.39)\n- Both Kaggle and EEG spectrograms (CV 0.5768 – LB 0.37)\n- Chris's [WaveNet][4] (CV 0.6992 - LB 0.41)\n- SPaRCNet (LB 2.87)\n\n**The Ensemble achieves LB 0.34** \n\nGreat discussion [here][5] by @KOLOO that led to the latest score!\n\nFeatures+Head Starter uses Chris Deotte's Kaggle dataset [here][1]. Also Uses Chris's EEG spectrograms [here][3] (modified version) \n\n### Train and Infer Tips\n\nThis notebook can be used both to train and submit (infer) to Kaggle LB. When training, you can set variable `submission = False` , you can also set `TEST_MODE = TRUE` to upload 500 samples queckly instead of the whole dataset for testing. \n\nTo train a specific model type, you should set `DATA_TYPE = 'both|eeg|kaggle|raw'`, `kaggle` to train on Kaggle's spectrograms, `eeg` to train on EEG's spectrograms, `both` to train on Kaggle's and EEG's spectrograms, `raw` to train on EEG's signal with WaveNet,\n\nFor submission after training models, you should save them in the LOAD_MODELS_FROM dataset, then run this notebook with `submission = True`.\n\nOnce we have all the models saved to LOAD_MODELS_FROM and ready ensemble, we should set `submission = True` and `ENSEMBLE = True` and set the models versions that we prior specified, as well as their `LBs` for weighted ensemble.\n\nThis notebook is made as generic as possible to expand and try different experiments.\n\nWhat you could do:\n- Change EfficientNetB(0-7) with `LOAD_BACKBONE_FROM`\n- Data augmentation by setting DataGenerator's parameter to `augment = True`\n- Different image configurations as input.\n- WaveNet model tuning.\n\n\nThis notebook is a direct descendent of Chris's notebook [here][2]\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\n[2]: https://www.kaggle.com/code/cdeotte/efficientnetb2-starter-lb-0-57\n[3]: https://www.kaggle.com/datasets/nartaa/eeg-spectrograms\n[4]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468684\n[5]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461","metadata":{"papermill":{"duration":0.00905,"end_time":"2024-02-28T19:48:16.229318","exception":false,"start_time":"2024-02-28T19:48:16.220268","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os, random\nimport tensorflow as tf\nimport tensorflow\nimport tensorflow.keras.backend as K\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras.models import load_model\n\nLOAD_BACKBONE_FROM = '/kaggle/input/efficientnetb-tf-keras/EfficientNetB2.h5'\nLOAD_MODELS_FROM = '/kaggle/input/features-head-starter-models/'\nVER = 47\nDATA_TYPE = 'both' # both|eeg|kaggle|raw\nTEST_MODE = False\nsubmission = True\n\n# Setup for ensemble\nENSEMBLE = True\nLBs = [0.37,0.39,0.41,0.41, 0.9] # for weighted ensemble we use LBs of each model\nVERK = 43 # Kaggle's spectrogram model version\nVERB = 47 # Kaggle's and EEG's spectrogram model version\nVERE = 42 # EEG's spectrogram model version\nVERR = 37 # EEG's raw wavenet model version, trained on single GPU\n\nnp.random.seed(42)\nrandom.seed(42)\ntf.random.set_seed(42)\n\n# USE SINGLE GPU, MULTIPLE GPUS \ngpus = tf.config.list_physical_devices('GPU')\n# WE USE MIXED PRECISION\ntf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\nif len(gpus)>1:\n    strategy = tf.distribute.MirroredStrategy()\n    print(f'Using {len(gpus)} GPUs')\nelse:\n    strategy = tf.distribute.OneDeviceStrategy(device=\"/gpu:0\")\n    print(f'Using {len(gpus)} GPU')","metadata":{"papermill":{"duration":13.504974,"end_time":"2024-02-28T19:48:29.742388","exception":false,"start_time":"2024-02-28T19:48:16.237414","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:21.478012Z","iopub.execute_input":"2024-03-15T19:18:21.478369Z","iopub.status.idle":"2024-03-15T19:18:35.224555Z","shell.execute_reply.started":"2024-03-15T19:18:21.478338Z","shell.execute_reply":"2024-03-15T19:18:35.223690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load and create Non-Overlapping Eeg Id Train Data\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{"papermill":{"duration":0.011883,"end_time":"2024-02-28T19:48:29.765736","exception":false,"start_time":"2024-02-28T19:48:29.753853","status":"completed"},"tags":[]}},{"cell_type":"code","source":"TARGETS = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\nFEATS2 = ['Fp1','T3','C3','O1','Fp2','C4','T4','O2']\nFEAT2IDX = {x:y for x,y in zip(FEATS2,range(len(FEATS2)))}\n\ndef eeg_from_parquet(parquet_path):\n\n    eeg = pd.read_parquet(parquet_path, columns=FEATS2)\n    rows = len(eeg)\n    offset = (rows-10_000)//2\n    eeg = eeg.iloc[offset:offset+10_000]\n    data = np.zeros((10_000,len(FEATS2)))\n    for j,col in enumerate(FEATS2):\n        \n        # FILL NAN\n        x = eeg[col].values.astype('float32')\n        m = np.nanmean(x)\n        if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n        else: x[:] = 0\n        \n        data[:,j] = x\n\n    return data\n\ndef add_kl(data):\n    import torch\n    labels = data[TARGETS].values + 1e-5\n\n    # compute kl-loss with uniform distribution by pytorch\n    data['kl'] = torch.nn.functional.kl_div(\n        torch.log(torch.tensor(labels)),\n        torch.tensor([1 / 6] * 6),\n        reduction='none'\n    ).sum(dim=1).numpy()\n    return data\n\ndef reset_seed(seed):\n    np.random.seed(seed)\n    random.seed(seed)\n    tf.random.set_seed(seed)\n    \nif not submission:\n    train = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\n    TARGETS = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\n    META = ['spectrogram_id','spectrogram_label_offset_seconds','patient_id','expert_consensus']\n    train = train.groupby('eeg_id')[META+TARGETS\n                           ].agg({**{m:'first' for m in META},**{t:'sum' for t in TARGETS}}).reset_index() \n    train[TARGETS] = train[TARGETS]/train[TARGETS].values.sum(axis=1,keepdims=True)\n    train.columns = ['eeg_id','spec_id','offset','patient_id','target'] + TARGETS\n    train = add_kl(train)\n    print(train.head(1).to_string())","metadata":{"papermill":{"duration":0.025421,"end_time":"2024-02-28T19:48:29.800372","exception":false,"start_time":"2024-02-28T19:48:29.774951","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:35.226147Z","iopub.execute_input":"2024-03-15T19:18:35.226968Z","iopub.status.idle":"2024-03-15T19:18:35.246787Z","shell.execute_reply.started":"2024-03-15T19:18:35.226928Z","shell.execute_reply":"2024-03-15T19:18:35.245443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Read Train Spectrograms and EEGs\n\nWe can read 3 file from Chris's [Kaggle dataset here][1] which contains all the 11k spectrograms. From Chris's modified EEG spectrogram [here][2]. From Chris's EEG signals [here][3]\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\n[2]: https://www.kaggle.com/datasets/nartaa/eeg-spectrograms\n[3]: https://www.kaggle.com/datasets/cdeotte/brain-eegs","metadata":{"papermill":{"duration":0.007453,"end_time":"2024-02-28T19:48:29.815771","exception":false,"start_time":"2024-02-28T19:48:29.808318","status":"completed"},"tags":[]}},{"cell_type":"code","source":"%%time\nif not submission:\n    # FOR TESTING SET TEST_MODE TO TRUE\n    if TEST_MODE:\n        train = train.sample(500,random_state=42).reset_index(drop=True)\n        spectrograms = {}\n        for i,e in enumerate(train.spec_id.values):\n            if i%100==0: print(i,', ',end='')\n            x = pd.read_parquet(f'/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/{e}.parquet')\n            spectrograms[e] = x.values\n        all_eegs = {}\n        for i,e in enumerate(train.eeg_id.values):\n            if i%100==0: print(i,', ',end='')\n            x = np.load(f'/kaggle/input/eeg-spectrograms/EEG_Spectrograms/{e}.npy')\n            all_eegs[e] = x\n        all_raw_eegs = {}\n        for i,e in enumerate(train.eeg_id.values):\n            if i%100==0: print(i,', ',end='')\n            x = eeg_from_parquet(f'/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/{e}.parquet')              \n            all_raw_eegs[e] = x\n    else:\n        spectrograms = None\n        all_eegs = None\n        all_raw_eegs = None\n        if DATA_TYPE=='both' or DATA_TYPE=='kaggle':\n            spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()\n        if DATA_TYPE=='both' or DATA_TYPE=='eeg':\n            all_eegs = np.load('/kaggle/input/eeg-spectrograms/eeg_specs.npy',allow_pickle=True).item()\n        if DATA_TYPE=='raw':\n            all_raw_eegs = np.load('/kaggle/input/brain-eegs/eegs.npy',allow_pickle=True).item()","metadata":{"papermill":{"duration":0.02106,"end_time":"2024-02-28T19:48:29.844607","exception":false,"start_time":"2024-02-28T19:48:29.823547","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:35.248394Z","iopub.execute_input":"2024-03-15T19:18:35.249579Z","iopub.status.idle":"2024-03-15T19:18:35.297887Z","shell.execute_reply.started":"2024-03-15T19:18:35.249538Z","shell.execute_reply":"2024-03-15T19:18:35.296371Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DATA GENERATOR\nThis data generator outputs 512x512x3, the spectrogram and eeg images are concatenated all togother in a single image. For using data augmention you can set `augment = True` when creating the train data generator.","metadata":{"papermill":{"duration":0.007648,"end_time":"2024-02-28T19:48:29.860169","exception":false,"start_time":"2024-02-28T19:48:29.852521","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import albumentations as albu\nfrom scipy.signal import butter, lfilter\n\nclass DataGenerator():\n    'Generates data for Keras'\n    def __init__(self, data, specs=None, eeg_specs=None, raw_eegs=None, augment=False, mode='train', data_type=DATA_TYPE): \n        self.data = data\n        self.augment = augment\n        self.mode = mode\n        self.data_type = data_type\n        self.specs = specs\n        self.eeg_specs = eeg_specs\n        self.raw_eegs = raw_eegs\n        self.on_epoch_end()\n        \n    def __len__(self):\n        return self.data.shape[0]\n\n    def __getitem__(self, index):\n        X, y = self.data_generation(index)\n        if self.augment: X = self.augmentation(X)\n        return X, y\n    \n    def __call__(self):\n        for i in range(self.__len__()):\n            yield self.__getitem__(i)\n            \n            if i == self.__len__()-1:\n                self.on_epoch_end()\n                \n    def on_epoch_end(self):\n        if self.mode=='train': \n            self.data = self.data.sample(frac=1).reset_index(drop=True)\n    \n    def data_generation(self, index):\n        if self.data_type == 'both':\n            X,y = self.generate_all_specs(index)\n        elif self.data_type == 'eeg' or self.data_type == 'kaggle':\n            X,y = self.generate_specs(index)\n        elif self.data_type == 'raw':\n            X,y = self.generate_raw(index)\n\n        return X,y\n    \n    def generate_all_specs(self, index):\n        X = np.zeros((512,512,3),dtype='float32')\n        y = np.zeros((6,),dtype='float32')\n        \n        row = self.data.iloc[index]\n        if self.mode=='test': \n            offset = 0\n        else:\n            offset = int(row.offset/2)\n            \n        eeg = self.eeg_specs[row.eeg_id]\n        spec = self.specs[row.spec_id]\n        \n        imgs = [spec[offset:offset+300,k*100:(k+1)*100].T for k in [0,2,1,3]] # to match kaggle with eeg\n        img = np.stack(imgs,axis=-1)\n        # LOG TRANSFORM SPECTROGRAM\n        img = np.clip(img,np.exp(-4),np.exp(8))\n        img = np.log(img)\n            \n        # STANDARDIZE PER IMAGE\n        img = np.nan_to_num(img, nan=0.0)    \n            \n        mn = img.flatten().min()\n        mx = img.flatten().max()\n        ep = 1e-5\n        img = 255 * (img - mn) / (mx - mn + ep)\n        \n        X[0_0+56:100+56,:256,0] = img[:,22:-22,0] # LL_k\n        X[100+56:200+56,:256,0] = img[:,22:-22,2] # RL_k\n        X[0_0+56:100+56,:256,1] = img[:,22:-22,1] # LP_k\n        X[100+56:200+56,:256,1] = img[:,22:-22,3] # RP_k\n        X[0_0+56:100+56,:256,2] = img[:,22:-22,2] # RL_k\n        X[100+56:200+56,:256,2] = img[:,22:-22,1] # LP_k\n        \n        X[0_0+56:100+56,256:,0] = img[:,22:-22,0] # LL_k\n        X[100+56:200+56,256:,0] = img[:,22:-22,2] # RL_k\n        X[0_0+56:100+56,256:,1] = img[:,22:-22,1] # LP_k\n        X[100+56:200+56,256:,1] = img[:,22:-22,3] # RP_K\n        \n        # EEG\n        img = eeg\n        mn = img.flatten().min()\n        mx = img.flatten().max()\n        ep = 1e-5\n        img = 255 * (img - mn) / (mx - mn + ep)\n        X[200+56:300+56,:256,0] = img[:,22:-22,0] # LL_e\n        X[300+56:400+56,:256,0] = img[:,22:-22,2] # RL_e\n        X[200+56:300+56,:256,1] = img[:,22:-22,1] # LP_e\n        X[300+56:400+56,:256,1] = img[:,22:-22,3] # RP_e\n        X[200+56:300+56,:256,2] = img[:,22:-22,2] # RL_e\n        X[300+56:400+56,:256,2] = img[:,22:-22,1] # LP_e\n        \n        X[200+56:300+56,256:,0] = img[:,22:-22,0] # LL_e\n        X[300+56:400+56,256:,0] = img[:,22:-22,2] # RL_e\n        X[200+56:300+56,256:,1] = img[:,22:-22,1] # LP_e\n        X[300+56:400+56,256:,1] = img[:,22:-22,3] # RP_e\n\n        if self.mode!='test':\n            y[:] = row[TARGETS]\n        \n        return X,y\n    \n    def generate_specs(self, index):\n        X = np.zeros((512,512,3),dtype='float32')\n        y = np.zeros((6,),dtype='float32')\n        \n        row = self.data.iloc[index]\n        if self.mode=='test': \n            offset = 0\n        else:\n            offset = int(row.offset/2)\n            \n        if self.data_type == 'eeg':\n            img = self.eeg_specs[row.eeg_id]\n        elif self.data_type == 'kaggle':\n            spec = self.specs[row.spec_id]\n            imgs = [spec[offset:offset+300,k*100:(k+1)*100].T for k in [0,2,1,3]] # to match kaggle with eeg\n            img = np.stack(imgs,axis=-1)\n            # LOG TRANSFORM SPECTROGRAM\n            img = np.clip(img,np.exp(-4),np.exp(8))\n            img = np.log(img)\n            \n            # STANDARDIZE PER IMAGE\n            img = np.nan_to_num(img, nan=0.0)    \n            \n        mn = img.flatten().min()\n        mx = img.flatten().max()\n        ep = 1e-5\n        img = 255 * (img - mn) / (mx - mn + ep)\n        \n        X[0_0+56:100+56,:256,0] = img[:,22:-22,0]\n        X[100+56:200+56,:256,0] = img[:,22:-22,2]\n        X[0_0+56:100+56,:256,1] = img[:,22:-22,1]\n        X[100+56:200+56,:256,1] = img[:,22:-22,3]\n        X[0_0+56:100+56,:256,2] = img[:,22:-22,2]\n        X[100+56:200+56,:256,2] = img[:,22:-22,1]\n        \n        X[0_0+56:100+56,256:,0] = img[:,22:-22,0]\n        X[100+56:200+56,256:,0] = img[:,22:-22,1]\n        X[0_0+56:100+56,256:,1] = img[:,22:-22,2]\n        X[100+56:200+56,256:,1] = img[:,22:-22,3]\n        \n        X[200+56:300+56,:256,0] = img[:,22:-22,0]\n        X[300+56:400+56,:256,0] = img[:,22:-22,1]\n        X[200+56:300+56,:256,1] = img[:,22:-22,2]\n        X[300+56:400+56,:256,1] = img[:,22:-22,3]\n        X[200+56:300+56,:256,2] = img[:,22:-22,3]\n        X[300+56:400+56,:256,2] = img[:,22:-22,2]\n        \n        X[200+56:300+56,256:,0] = img[:,22:-22,0]\n        X[300+56:400+56,256:,0] = img[:,22:-22,2]\n        X[200+56:300+56,256:,1] = img[:,22:-22,1]\n        X[300+56:400+56,256:,1] = img[:,22:-22,3]\n        \n        if self.mode!='test':\n            y[:] = row[TARGETS]\n        \n        return X,y\n    \n    def generate_raw(self,index):\n        X = np.zeros((10_000,8),dtype='float32')\n        y = np.zeros((6,),dtype='float32')\n        \n        row = self.data.iloc[index]\n        eeg = self.raw_eegs[row.eeg_id]\n            \n        # FEATURE ENGINEER\n        X[:,0] = eeg[:,FEAT2IDX['Fp1']] - eeg[:,FEAT2IDX['T3']]\n        X[:,1] = eeg[:,FEAT2IDX['T3']] - eeg[:,FEAT2IDX['O1']]\n            \n        X[:,2] = eeg[:,FEAT2IDX['Fp1']] - eeg[:,FEAT2IDX['C3']]\n        X[:,3] = eeg[:,FEAT2IDX['C3']] - eeg[:,FEAT2IDX['O1']]\n            \n        X[:,4] = eeg[:,FEAT2IDX['Fp2']] - eeg[:,FEAT2IDX['C4']]\n        X[:,5] = eeg[:,FEAT2IDX['C4']] - eeg[:,FEAT2IDX['O2']]\n            \n        X[:,6] = eeg[:,FEAT2IDX['Fp2']] - eeg[:,FEAT2IDX['T4']]\n        X[:,7] = eeg[:,FEAT2IDX['T4']] - eeg[:,FEAT2IDX['O2']]\n            \n        # STANDARDIZE\n        X = np.clip(X,-1024,1024)\n        X = np.nan_to_num(X, nan=0) / 32.0\n            \n        # BUTTER LOW-PASS FILTER\n        X = self.butter_lowpass_filter(X)\n        # Downsample\n        X = X[::5,:]\n        \n        if self.mode!='test':\n            y[:] = row[TARGETS]\n                \n        return X,y\n        \n    def butter_lowpass_filter(self, data, cutoff_freq=20, sampling_rate=200, order=4):\n        nyquist = 0.5 * sampling_rate\n        normal_cutoff = cutoff_freq / nyquist\n        b, a = butter(order, normal_cutoff, btype='low', analog=False)\n        filtered_data = lfilter(b, a, data, axis=0)\n        return filtered_data\n    \n    def resize(self, img,size):\n        composition = albu.Compose([\n                albu.Resize(size[0],size[1])\n            ])\n        return composition(image=img)['image']\n            \n    def augmentation(self, img):\n        composition = albu.Compose([\n                albu.HorizontalFlip(p=0.4)\n            ])\n        return composition(image=img)['image']","metadata":{"papermill":{"duration":1.920135,"end_time":"2024-02-28T19:48:31.788415","exception":false,"start_time":"2024-02-28T19:48:29.86828","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:35.301289Z","iopub.execute_input":"2024-03-15T19:18:35.302807Z","iopub.status.idle":"2024-03-15T19:18:37.203455Z","shell.execute_reply.started":"2024-03-15T19:18:35.302638Z","shell.execute_reply":"2024-03-15T19:18:37.202532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DISPLAY DATA GENERATOR\nBelow we display example data generator spectrogram images and raw EEG signals.","metadata":{"papermill":{"duration":0.008418,"end_time":"2024-02-28T19:48:31.805954","exception":false,"start_time":"2024-02-28T19:48:31.797536","status":"completed"},"tags":[]}},{"cell_type":"code","source":"if not submission and DATA_TYPE!='raw':\n    gen = DataGenerator(train, augment=False, specs=spectrograms, eeg_specs=all_eegs, data_type=DATA_TYPE)\n    for x,y in gen:\n        break\n    plt.imshow(x[:,:,0])\n    plt.title(f'Target = {y.round(1)}',size=12)\n    plt.yticks([])\n    plt.ylabel('Frequencies (Hz)',size=12)\n    plt.xlabel('Time (sec)',size=12)\n    plt.show()\n    \nif not submission and DATA_TYPE=='raw':\n    gen = DataGenerator(train, raw_eegs=all_raw_eegs, data_type=DATA_TYPE)\n    for x,y in gen:\n        plt.figure(figsize=(20,4))\n        offset = 0\n        for j in range(x.shape[-1]):\n            if j!=0: offset -= x[:,j].min()\n            plt.plot(range(2_000),x[:,j]+offset,label=f'feature {j+1}')\n            offset += x[:,j].max()\n        plt.legend()\n        plt.show()\n        break","metadata":{"papermill":{"duration":0.019565,"end_time":"2024-02-28T19:48:31.833755","exception":false,"start_time":"2024-02-28T19:48:31.81419","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:37.204694Z","iopub.execute_input":"2024-03-15T19:18:37.205412Z","iopub.status.idle":"2024-03-15T19:18:37.215931Z","shell.execute_reply.started":"2024-03-15T19:18:37.205374Z","shell.execute_reply":"2024-03-15T19:18:37.214525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TRAINING","metadata":{"papermill":{"duration":0.008667,"end_time":"2024-02-28T19:48:31.850562","exception":false,"start_time":"2024-02-28T19:48:31.841895","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## LEARNING RATE","metadata":{"papermill":{"duration":0.007995,"end_time":"2024-02-28T19:48:31.866605","exception":false,"start_time":"2024-02-28T19:48:31.85861","status":"completed"},"tags":[]}},{"cell_type":"code","source":"\nif not submission:\n\n    def lrfn(epoch):\n        e2 = 1e-3 if DATA_TYPE=='raw' else 1e-4\n        e3 = 1e-3 if DATA_TYPE=='raw' else 1e-4\n        e4 = 1e-4 if DATA_TYPE=='raw' else 1e-5\n        return [1e-3,e2,e3,e4,1e-5][epoch]\n\n    LR = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = True)\n    \n    def lrfn2(epoch):\n        return [1e-5,1e-5,1e-6][epoch]\n\n    LR2 = tf.keras.callbacks.LearningRateScheduler(lrfn2, verbose = True)","metadata":{"papermill":{"duration":0.01697,"end_time":"2024-02-28T19:48:31.891645","exception":false,"start_time":"2024-02-28T19:48:31.874675","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:37.217173Z","iopub.execute_input":"2024-03-15T19:18:37.218059Z","iopub.status.idle":"2024-03-15T19:18:37.230019Z","shell.execute_reply.started":"2024-03-15T19:18:37.218017Z","shell.execute_reply":"2024-03-15T19:18:37.229055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## MODEL AND UTILITY FUNCTIONS","metadata":{"papermill":{"duration":0.008034,"end_time":"2024-02-28T19:48:31.907771","exception":false,"start_time":"2024-02-28T19:48:31.899737","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from tensorflow.keras.layers import Input, Dense, Multiply, Add, Conv1D, Concatenate\n\ndef build_model():  \n    inp = tf.keras.layers.Input((512,512,3))\n    base_model = load_model(f'{LOAD_BACKBONE_FROM}')    \n    x = base_model(inp)\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n    output = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(x)\n    model = tf.keras.Model(inputs=inp, outputs=output)\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.KLDivergence()\n    model.compile(loss=loss, optimizer=opt)  \n    return model\n\ndef score(y_true, y_pred):\n    kl = tf.keras.metrics.KLDivergence()\n    return kl(y_true, y_pred)\n\ndef wave_block(x, filters, kernel_size, n):\n    dilation_rates = [2**i for i in range(n)]\n    x = Conv1D(filters = filters,\n               kernel_size = 1,\n               padding = 'same')(x)\n    res_x = x\n    for dilation_rate in dilation_rates:\n        tanh_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same', \n                          activation = 'tanh', \n                          dilation_rate = dilation_rate)(x)\n        sigm_out = Conv1D(filters = filters,\n                          kernel_size = kernel_size,\n                          padding = 'same',\n                          activation = 'sigmoid', \n                          dilation_rate = dilation_rate)(x)\n        x = Multiply()([tanh_out, sigm_out])\n        x = Conv1D(filters = filters,\n                   kernel_size = 1,\n                   padding = 'same')(x)\n        res_x = Add()([res_x, x])\n    return res_x\n\ndef build_wave_model():\n        \n    # INPUT \n    inp = tf.keras.Input(shape=(2_000,8))\n    \n    ############\n    # FEATURE EXTRACTION SUB MODEL\n    inp2 = tf.keras.Input(shape=(2_000,1))\n    x = wave_block(inp2, 8, 4, 6)\n    x = wave_block(x, 16, 4, 6)\n    x = wave_block(x, 32, 4, 6)\n    x = wave_block(x, 64, 4, 6)\n    model2 = tf.keras.Model(inputs=inp2, outputs=x)\n    ###########\n    \n    # LEFT TEMPORAL CHAIN\n    x1 = model2(inp[:,:,0:1])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,1:2])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z1 = tf.keras.layers.Average()([x1,x2])\n    \n    # LEFT PARASAGITTAL CHAIN\n    x1 = model2(inp[:,:,2:3])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,3:4])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z2 = tf.keras.layers.Average()([x1,x2])\n    \n    # RIGHT PARASAGITTAL CHAIN\n    x1 = model2(inp[:,:,4:5])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,5:6])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z3 = tf.keras.layers.Average()([x1,x2])\n    \n    # RIGHT TEMPORAL CHAIN\n    x1 = model2(inp[:,:,6:7])\n    x1 = tf.keras.layers.GlobalAveragePooling1D()(x1)\n    x2 = model2(inp[:,:,7:8])\n    x2 = tf.keras.layers.GlobalAveragePooling1D()(x2)\n    z4 = tf.keras.layers.Average()([x1,x2])\n    \n    # COMBINE CHAINS\n    y = tf.keras.layers.Concatenate()([z1,z2,z3,z4])\n    y = tf.keras.layers.Dense(64, activation='relu')(y)\n    y = tf.keras.layers.Dense(6,activation='softmax', dtype='float32')(y)\n    \n    # COMPILE MODEL\n    model = tf.keras.Model(inputs=inp, outputs=y)\n    opt = tf.keras.optimizers.Adam(learning_rate = 1e-3)\n    loss = tf.keras.losses.KLDivergence()\n    model.compile(loss=loss, optimizer = opt)\n    \n    return model\n\ndef plot_hist(hist):\n    metrics = ['loss']\n    for i,metric in enumerate(metrics):\n        plt.figure(figsize=(10,4))\n        plt.subplot(1,2,i+1)\n        plt.plot(hist[metric])\n        plt.plot(hist[f'val_{metric}'])\n        plt.title(f'{metric}',size=12)\n        plt.ylabel(f'{metric}',size=12)\n        plt.xlabel('epoch',size=12)\n        plt.legend([\"train\", \"validation\"], loc=\"upper left\")\n        plt.show()","metadata":{"papermill":{"duration":0.035214,"end_time":"2024-02-28T19:48:31.951261","exception":false,"start_time":"2024-02-28T19:48:31.916047","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:37.231575Z","iopub.execute_input":"2024-03-15T19:18:37.232312Z","iopub.status.idle":"2024-03-15T19:18:37.263859Z","shell.execute_reply.started":"2024-03-15T19:18:37.232269Z","shell.execute_reply":"2024-03-15T19:18:37.262779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TRANSFER LEARNING","metadata":{"papermill":{"duration":0.007821,"end_time":"2024-02-28T19:48:31.967235","exception":false,"start_time":"2024-02-28T19:48:31.959414","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold\nimport tensorflow.keras.backend as K, gc\n\nif not submission:\n    # for CV scores setting random seed works for single GPU only\n    reset_seed(42)\n    all_oof = []\n    all_true = []\n    losses = []\n    val_losses = []\n    total_hist = {}\n\n    gkf = GroupKFold(n_splits=5)\n    for i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):   \n        \n        print('#'*25)\n        print(f'### Fold {i+1}')\n        \n        data, val = train.iloc[train_index],train.iloc[valid_index]\n        train_gen = DataGenerator(data, augment=False, specs=spectrograms, eeg_specs=all_eegs, raw_eegs=all_raw_eegs)\n        valid_gen = DataGenerator(val, mode='valid', specs=spectrograms, eeg_specs=all_eegs, raw_eegs=all_raw_eegs)\n        data = data[data['kl']<5.5]\n        train_gen2 = DataGenerator(data, augment=False, specs=spectrograms, eeg_specs=all_eegs, raw_eegs=all_raw_eegs)\n        in_shape = (2000,8) if DATA_TYPE=='raw' else (512,512,3)\n        EPOCHS = 5\n        BATCH_SIZE_PER_REPLICA = 8\n        BATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync\n\n        train_dataset = tf.data.Dataset.from_generator(generator=train_gen, \n                                                   output_signature=(tf.TensorSpec(shape=in_shape, dtype=tf.float32),\n                                                                     tf.TensorSpec(shape=(6,), dtype=tf.float32))).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n        val_dataset = tf.data.Dataset.from_generator(generator=valid_gen, \n                                                   output_signature=(tf.TensorSpec(shape=in_shape, dtype=tf.float32),\n                                                                     tf.TensorSpec(shape=(6,), dtype=tf.float32))).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n        train_dataset2 = tf.data.Dataset.from_generator(generator=train_gen2, \n                                                   output_signature=(tf.TensorSpec(shape=in_shape, dtype=tf.float32),\n                                                                     tf.TensorSpec(shape=(6,), dtype=tf.float32))).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n            \n        print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n        print('#'*25)\n        \n        K.clear_session()\n        with strategy.scope():\n            if DATA_TYPE=='raw':\n                model = build_wave_model()\n            else:\n                model = build_model()\n        \n        hist = model.fit(train_dataset, validation_data = val_dataset, \n                         epochs=EPOCHS, callbacks=[LR])\n        print(f'### seconds stage train size {len(data)}, valid size {len(val)}')\n        print('#'*25)\n        hist2 = model.fit(train_dataset2, validation_data = val_dataset, \n                         epochs=3, callbacks=[LR2])\n        losses.append(hist.history['loss']+hist2.history['loss'])\n        val_losses.append(hist.history['val_loss']+hist2.history['val_loss'])\n        with strategy.scope():\n            model.save_weights(f'model_{DATA_TYPE}_{VER}_{i}.weights.h5')\n        oof = model.predict(val_dataset, verbose=1)\n        all_oof.append(oof)\n        all_true.append(train.iloc[valid_index][TARGETS].values)    \n        del model, oof\n        gc.collect()\n        \n    total_hist['loss'] = np.mean(losses,axis=0)\n    total_hist['val_loss'] = np.mean(val_losses,axis=0)\n    all_oof = np.concatenate(all_oof)\n    all_true = np.concatenate(all_true)\n    plot_hist(total_hist)\n    print('#'*25)\n    print(f'CV KL SCORE: {score(all_true,all_oof)}')","metadata":{"papermill":{"duration":0.029686,"end_time":"2024-02-28T19:48:32.005088","exception":false,"start_time":"2024-02-28T19:48:31.975402","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:37.265413Z","iopub.execute_input":"2024-03-15T19:18:37.265858Z","iopub.status.idle":"2024-03-15T19:18:37.286851Z","shell.execute_reply.started":"2024-03-15T19:18:37.265827Z","shell.execute_reply":"2024-03-15T19:18:37.285635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Infer Test and Create Submission CSV\nInfer the test data and create a `submission.csv` file.","metadata":{"papermill":{"duration":0.007921,"end_time":"2024-02-28T19:48:32.021376","exception":false,"start_time":"2024-02-28T19:48:32.013455","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import pywt, librosa\n\nUSE_WAVELET = None \n\nNAMES = ['LL','LP','RP','RR']\n\nFEATS = [['Fp1','F7','T3','T5','O1'],\n         ['Fp1','F3','C3','P3','O1'],\n         ['Fp2','F8','T4','T6','O2'],\n         ['Fp2','F4','C4','P4','O2']]\n\n# DENOISE FUNCTION\ndef maddest(d, axis=None):\n    return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\ndef denoise(x, wavelet='haar', level=1):    \n    coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n    sigma = (1/0.6745) * maddest(coeff[-level])\n\n    uthresh = sigma * np.sqrt(2*np.log(len(x)))\n    coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n    ret=pywt.waverec(coeff, wavelet, mode='per')\n    \n    return ret\n\nimport librosa\n\ndef spectrogram_from_eeg(parquet_path, display=False):\n    \n    # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n    eeg = pd.read_parquet(parquet_path)\n    middle = (len(eeg)-10_000)//2\n    eeg = eeg.iloc[middle:middle+10_000]\n    \n    # VARIABLE TO HOLD SPECTROGRAM\n    img = np.zeros((100,300,4),dtype='float32')\n    \n    if display: plt.figure(figsize=(10,7))\n    signals = []\n    for k in range(4):\n        COLS = FEATS[k]\n        \n        for kk in range(4):\n            # FILL NANS\n            x1 = eeg[COLS[kk]].values\n            x2 = eeg[COLS[kk+1]].values\n            m = np.nanmean(x1)\n            if np.isnan(x1).mean()<1: x1 = np.nan_to_num(x1,nan=m)\n            else: x1[:] = 0\n            m = np.nanmean(x2)\n            if np.isnan(x2).mean()<1: x2 = np.nan_to_num(x2,nan=m)\n            else: x2[:] = 0\n                \n            # COMPUTE PAIR DIFFERENCES\n            x = x1 - x2\n\n            # DENOISE\n            if USE_WAVELET:\n                x = denoise(x, wavelet=USE_WAVELET)\n            signals.append(x)\n\n            # RAW SPECTROGRAM\n            mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x)//300, \n                  n_fft=1024, n_mels=100, fmin=0, fmax=20, win_length=128)\n            \n            # LOG TRANSFORM\n            width = (mel_spec.shape[1]//30)*30\n            mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:,:width]\n            img[:,:,k] += mel_spec_db\n                \n        # AVERAGE THE 4 MONTAGE DIFFERENCES\n        img[:,:,k] /= 4.0\n        \n        if display:\n            plt.subplot(2,2,k+1)\n            plt.imshow(img[:,:,k],aspect='auto',origin='lower')\n            \n    if display: \n        plt.show()\n        plt.figure(figsize=(10,5))\n        offset = 0\n        for k in range(4):\n            if k>0: offset -= signals[3-k].min()\n            plt.plot(range(10_000),signals[k]+offset,label=NAMES[3-k])\n            offset += signals[3-k].max()\n        plt.legend()\n        plt.show()\n        \n    return img","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.032151,"end_time":"2024-02-28T19:48:32.061684","exception":false,"start_time":"2024-02-28T19:48:32.029533","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:37.288080Z","iopub.execute_input":"2024-03-15T19:18:37.288381Z","iopub.status.idle":"2024-03-15T19:18:37.316423Z","shell.execute_reply.started":"2024-03-15T19:18:37.288355Z","shell.execute_reply":"2024-03-15T19:18:37.315206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if submission:\n    test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\n    print('Test shape',test.shape)\n    test.head()","metadata":{"papermill":{"duration":0.026805,"end_time":"2024-02-28T19:48:32.096416","exception":false,"start_time":"2024-02-28T19:48:32.069611","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:37.322579Z","iopub.execute_input":"2024-03-15T19:18:37.322939Z","iopub.status.idle":"2024-03-15T19:18:37.339906Z","shell.execute_reply.started":"2024-03-15T19:18:37.322909Z","shell.execute_reply":"2024-03-15T19:18:37.338766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# READ ALL SPECTROGRAMS\nif submission:\n    PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\n    files2 = os.listdir(PATH2)\n    print(f'There are {len(files2)} test spectrogram parquets')\n    \n    spectrograms2 = {}\n    for i,f in enumerate(files2):\n        if i%100==0: print(i,', ',end='')\n        tmp = pd.read_parquet(f'{PATH2}{f}')\n        name = int(f.split('.')[0])\n        spectrograms2[name] = tmp.iloc[:,1:].values\n    \n    # RENAME FOR DATA GENERATOR\n    test = test.rename({'spectrogram_id':'spec_id'},axis=1)","metadata":{"papermill":{"duration":0.192747,"end_time":"2024-02-28T19:48:32.297204","exception":false,"start_time":"2024-02-28T19:48:32.104457","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:37.341375Z","iopub.execute_input":"2024-03-15T19:18:37.341765Z","iopub.status.idle":"2024-03-15T19:18:37.525087Z","shell.execute_reply.started":"2024-03-15T19:18:37.341733Z","shell.execute_reply":"2024-03-15T19:18:37.524274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# READ ALL EEG SPECTROGRAMS\nif submission:\n    PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n    DISPLAY = 0\n    EEG_IDS2 = test.eeg_id.unique()\n    all_eegs2 = {}\n\n    print('Converting Test EEG to Spectrograms...'); print()\n    for i,eeg_id in enumerate(EEG_IDS2):\n        \n        # CREATE SPECTROGRAM FROM EEG PARQUET\n        img = spectrogram_from_eeg(f'{PATH2}{eeg_id}.parquet', i<DISPLAY)\n        all_eegs2[eeg_id] = img","metadata":{"papermill":{"duration":10.034698,"end_time":"2024-02-28T19:48:42.34085","exception":false,"start_time":"2024-02-28T19:48:32.306152","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:37.526168Z","iopub.execute_input":"2024-03-15T19:18:37.526658Z","iopub.status.idle":"2024-03-15T19:18:51.378252Z","shell.execute_reply.started":"2024-03-15T19:18:37.526628Z","shell.execute_reply":"2024-03-15T19:18:51.376747Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# READ ALL RAW EEG SIGNALS\nif submission :\n    all_raw_eegs2 = {}\n    EEG_IDS2 = test.eeg_id.unique()\n    PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n\n    print('Processing Test EEG parquets...'); print()\n    for i,eeg_id in enumerate(EEG_IDS2):\n        \n        # SAVE EEG TO PYTHON DICTIONARY OF NUMPY ARRAYS\n        data = eeg_from_parquet(f'{PATH2}{eeg_id}.parquet')\n        all_raw_eegs2[eeg_id] = data","metadata":{"papermill":{"duration":0.051468,"end_time":"2024-02-28T19:48:42.411372","exception":false,"start_time":"2024-02-28T19:48:42.359904","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:51.379910Z","iopub.execute_input":"2024-03-15T19:18:51.381166Z","iopub.status.idle":"2024-03-15T19:18:51.409872Z","shell.execute_reply.started":"2024-03-15T19:18:51.381128Z","shell.execute_reply":"2024-03-15T19:18:51.408779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Submission ON TEST without ensemble\nif submission and not ENSEMBLE:\n    preds = []\n    \n    if DATA_TYPE=='raw':\n        test_gen = DataGenerator(test, mode='test', raw_eegs=all_raw_eegs2)\n        in_shape = (2000,8)\n    else:\n        test_gen = DataGenerator(test, mode='test', specs = spectrograms2, eeg_specs = all_eegs2)\n        in_shape = (512,512,3)\n    \n    test_dataset = tf.data.Dataset.from_generator(generator=test_gen, \n                                               output_signature=(tf.TensorSpec(shape=in_shape, dtype=tf.float32),\n                                                                 tf.TensorSpec(shape=(6,), dtype=tf.float32))).batch(64).prefetch(tf.data.AUTOTUNE)\n    if DATA_TYPE=='raw':\n        model = build_wave_model()\n    else:\n        model = build_model()\n\n    for i in range(5):\n        print(f'Fold {i+1}')\n        model.load_weights(f'{LOAD_MODELS_FROM}model_{DATA_TYPE}_{VER}_{i}.weights.h5')\n        pred = model.predict(test_dataset, verbose=1)\n        preds.append(pred)\n        \n    pred = np.mean(preds,axis=0)\n    print('Test preds shape',pred.shape)","metadata":{"papermill":{"duration":0.040625,"end_time":"2024-02-28T19:48:42.472193","exception":false,"start_time":"2024-02-28T19:48:42.431568","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:18:51.411448Z","iopub.execute_input":"2024-03-15T19:18:51.412125Z","iopub.status.idle":"2024-03-15T19:18:51.427368Z","shell.execute_reply.started":"2024-03-15T19:18:51.412076Z","shell.execute_reply":"2024-03-15T19:18:51.425831Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SparCeNet","metadata":{}},{"cell_type":"code","source":"!pip install mne","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:18:51.429864Z","iopub.execute_input":"2024-03-15T19:18:51.430690Z","iopub.status.idle":"2024-03-15T19:19:26.068765Z","shell.execute_reply.started":"2024-03-15T19:18:51.430654Z","shell.execute_reply":"2024-03-15T19:19:26.067332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\n\ncurrent_directory = os.getcwd()\nprint(\"Current Working Directory:\", current_directory)","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:26.071018Z","iopub.execute_input":"2024-03-15T19:19:26.071413Z","iopub.status.idle":"2024-03-15T19:19:26.088007Z","shell.execute_reply.started":"2024-03-15T19:19:26.071379Z","shell.execute_reply":"2024-03-15T19:19:26.077171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport re\nfrom mne.filter import filter_data, notch_filter\nimport time\nimport numpy as np\nimport random\nimport re\nimport pandas as pd\nfrom collections import OrderedDict\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom collections import Counter\nimport os\n\nprint (\"\")\nprint (\"lib finish\")\nprint (\"\")","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:26.090048Z","iopub.execute_input":"2024-03-15T19:19:26.090510Z","iopub.status.idle":"2024-03-15T19:19:29.225699Z","shell.execute_reply.started":"2024-03-15T19:19:26.090451Z","shell.execute_reply":"2024-03-15T19:19:29.224522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BASE_PATH = '/kaggle/input/sparcnet/SPaRCNet'","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.226996Z","iopub.execute_input":"2024-03-15T19:19:29.227720Z","iopub.status.idle":"2024-03-15T19:19:29.232597Z","shell.execute_reply.started":"2024-03-15T19:19:29.227688Z","shell.execute_reply":"2024-03-15T19:19:29.231501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"################################################## nets\nclass _DenseLayer(nn.Sequential):\n\tdef __init__(self, num_input_features, growth_rate, bn_size, drop_rate, conv_bias, batch_norm):\n\t\tsuper(_DenseLayer, self).__init__()\n\t\tif batch_norm:\n\t\t\tself.add_module('norm1', nn.BatchNorm1d(num_input_features)),\n\t\t# self.add_module('relu1', nn.ReLU()),\n\t\tself.add_module('elu1', nn.ELU()),\n\t\tself.add_module('conv1', nn.Conv1d(num_input_features, bn_size * growth_rate, kernel_size=1, stride=1, bias=conv_bias)),\n\t\tif batch_norm:\n\t\t\tself.add_module('norm2', nn.BatchNorm1d(bn_size * growth_rate)),\n\t\t# self.add_module('relu2', nn.ReLU()),\n\t\tself.add_module('elu2', nn.ELU()),\n\t\tself.add_module('conv2', nn.Conv1d(bn_size * growth_rate, growth_rate, kernel_size=3, stride=1, padding=1, bias=conv_bias)),\n\t\t# self.add_module('conv2', nn.Conv1d(bn_size * growth_rate, growth_rate, kernel_size=7, stride=1, padding=3, bias=conv_bias)),\n\t\tself.drop_rate = drop_rate\n\n\tdef forward(self, x):\n\t\t# print(\"Dense Layer Input: \")\n\t\t# print(x.size())\n\t\tnew_features = super(_DenseLayer, self).forward(x)\n\t\t# print(\"Dense Layer Output:\")\n\t\t# print(new_features.size())\n\t\tif self.drop_rate > 0:\n\t\t\tnew_features = F.dropout(new_features, p=self.drop_rate, training=self.training)\n\t\treturn torch.cat([x, new_features], 1)\n\n\nclass _DenseBlock(nn.Sequential):\n\tdef __init__(self, num_layers, num_input_features, bn_size, growth_rate, drop_rate, conv_bias, batch_norm):\n\t\tsuper(_DenseBlock, self).__init__()\n\t\tfor i in range(num_layers):\n\t\t\tlayer = _DenseLayer(num_input_features + i * growth_rate, growth_rate, bn_size, drop_rate, conv_bias, batch_norm)\n\t\t\tself.add_module('denselayer%d' % (i + 1), layer)\n\n\nclass _Transition(nn.Sequential):\n\tdef __init__(self, num_input_features, num_output_features, conv_bias, batch_norm):\n\t\tsuper(_Transition, self).__init__()\n\t\tif batch_norm:\n\t\t\tself.add_module('norm', nn.BatchNorm1d(num_input_features))\n\t\t# self.add_module('relu', nn.ReLU())\n\t\tself.add_module('elu', nn.ELU())\n\t\tself.add_module('conv', nn.Conv1d(num_input_features, num_output_features, kernel_size=1, stride=1, bias=conv_bias))\n\t\tself.add_module('pool', nn.AvgPool1d(kernel_size=2, stride=2))\n\n\nclass DenseNetEnconder(nn.Module):\n\tdef __init__(self, growth_rate=32, block_config=(4, 4, 4, 4, 4, 4, 4),  #block_config=(6, 12, 24, 48, 24, 20, 16),  #block_config=(6, 12, 24, 16),\n\t\t\t\t in_channels=16, num_init_features=64, bn_size=4, drop_rate=0.2, conv_bias=True, batch_norm=False):\n\n\t\tsuper(DenseNetEnconder, self).__init__()\n\n\t\t# First convolution\n\t\tfirst_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, kernel_size=7, stride=2, padding=3, bias=conv_bias))])\n\t\t# first_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, groups=in_channels, kernel_size=7, stride=2, padding=3, bias=conv_bias))])\n\t\t# first_conv = OrderedDict([('conv0', nn.Conv1d(in_channels, num_init_features, kernel_size=15, stride=2, padding=7, bias=conv_bias))])\n\n\t\t# first_conv = OrderedDict([\n\t\t# \t('conv0-depth', nn.Conv1d(in_channels, 32, groups=in_channels, kernel_size=7, stride=2, padding=3, bias=conv_bias)),\n\t\t# \t('conv0-point', nn.Conv1d(32, num_init_features, kernel_size=1, stride=1, bias=conv_bias)),\n\t\t# ])\n\n\t\tif batch_norm:\n\t\t\tfirst_conv['norm0'] = nn.BatchNorm1d(num_init_features)\n\t\t# first_conv['relu0'] = nn.ReLU()\n\t\tfirst_conv['elu0'] = nn.ELU()\n\t\tfirst_conv['pool0'] = nn.MaxPool1d(kernel_size=3, stride=2, padding=1)\n\n\t\tself.densenet = nn.Sequential(first_conv)\n\n\t\tnum_features = num_init_features\n\t\tfor i, num_layers in enumerate(block_config):\n\t\t\tblock = _DenseBlock(num_layers=num_layers, num_input_features=num_features,\n\t\t\t\t\t\t\t\tbn_size=bn_size, growth_rate=growth_rate, drop_rate=drop_rate, conv_bias=conv_bias, batch_norm=batch_norm)\n\t\t\tself.densenet.add_module('denseblock%d' % (i + 1), block)\n\t\t\tnum_features = num_features + num_layers * growth_rate\n\t\t\tif i != len(block_config) - 1:\n\t\t\t\ttrans = _Transition(num_input_features=num_features, num_output_features=num_features // 2, conv_bias=conv_bias, batch_norm=batch_norm)\n\t\t\t\tself.densenet.add_module('transition%d' % (i + 1), trans)\n\t\t\t\tnum_features = num_features // 2\n\n\t\t# Final batch norm\n\t\tif batch_norm:\n\t\t\tself.densenet.add_module('norm{}'.format(len(block_config) + 1), nn.BatchNorm1d(num_features))\n\t\t# self.features.add_module('norm5', BatchReNorm1d(num_features))\n\n\t\tself.densenet.add_module('relu{}'.format(len(block_config) + 1), nn.ReLU())\n\t\tself.densenet.add_module('pool{}'.format(len(block_config) + 1), nn.AvgPool1d(kernel_size=7, stride=3))  # stride originally 1\n\n\t\tself.num_features = num_features\n\n\t\t# Official init from torch repo.\n\t\tfor m in self.modules():\n\t\t\tif isinstance(m, nn.Conv1d):\n\t\t\t\tnn.init.kaiming_normal_(m.weight.data)\n\t\t\telif isinstance(m, nn.BatchNorm1d):\n\t\t\t\tm.weight.data.fill_(1)\n\t\t\t\tm.bias.data.zero_()\n\t\t\telif isinstance(m, nn.Linear):\n\t\t\t\tm.bias.data.zero_()\n\n\tdef forward(self, x):\n\t\tfeatures = self.densenet(x)\n\t\t# print(\"Final Output\")\n\t\t# print(features.size())\n\t\treturn features.view(features.size(0), -1)\n\n\nclass DenseNetClassifier(nn.Module):\n\t# def __init__(self, growth_rate=16, block_config=(3, 6, 12, 8),  #block_config=(6, 12, 24, 48, 24, 20, 16),  #block_config=(6, 12, 24, 16),\n\t# \t\t\t in_channels=16, num_init_features=32, bn_size=2, drop_rate=0, conv_bias=False, drop_fc=0.5, num_classes=6):\n\tdef __init__(self, growth_rate=32, block_config=(4, 4, 4, 4, 4, 4, 4),\n\t\t\t\t in_channels=16, num_init_features=64, bn_size=4, drop_rate=0.2, conv_bias=True, batch_norm=False, drop_fc=0.5, num_classes=6):\n\n\t\tsuper(DenseNetClassifier, self).__init__()\n\n\t\tself.features = DenseNetEnconder(growth_rate=growth_rate, block_config=block_config, in_channels=in_channels,\n\t\t\t\t\t\t\t\t\t\t num_init_features=num_init_features, bn_size=bn_size, drop_rate=drop_rate,\n\t\t\t\t\t\t\t\t\t\t conv_bias=conv_bias, batch_norm=batch_norm)\n\n\t\t# Linear layer\n\t\tself.classifier = nn.Sequential(\n\t\t\tnn.Dropout(p=drop_fc),\n\t\t\tnn.Linear(self.features.num_features, num_classes)\n\t\t)\n\n\t\t# Official init from torch repo.\n\t\tfor m in self.modules():\n\t\t\tif isinstance(m, nn.Conv1d):\n\t\t\t\tnn.init.kaiming_normal_(m.weight.data)\n\t\t\telif isinstance(m, nn.BatchNorm1d):\n\t\t\t\tm.weight.data.fill_(1)\n\t\t\t\tm.bias.data.zero_()\n\t\t\telif isinstance(m, nn.Linear):\n\t\t\t\tm.bias.data.zero_()\n\n\tdef forward(self, x):\n\t\tfeatures = self.features(x)\n\t\tout = self.classifier(features)\n\t\treturn out, features\n\n    \ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\nprint (\"device: \", device)\nprint (\"\")\n\nDATA_RAW_PATH = os.path.join(BASE_PATH, 'Data', 'Raw/') \nmodel_cnn = torch.load('/kaggle/input/iiic-sparcnet/SPaRCNet/model_1130.pt', map_location=torch.device('cpu'))\nmodel_cnn.eval()\n\nprint('so far so good!')","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.234337Z","iopub.execute_input":"2024-03-15T19:19:29.234954Z","iopub.status.idle":"2024-03-15T19:19:29.476051Z","shell.execute_reply.started":"2024-03-15T19:19:29.234924Z","shell.execute_reply":"2024-03-15T19:19:29.475240Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_cnn.to(device)","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.477444Z","iopub.execute_input":"2024-03-15T19:19:29.478063Z","iopub.status.idle":"2024-03-15T19:19:29.496451Z","shell.execute_reply.started":"2024-03-15T19:19:29.478032Z","shell.execute_reply":"2024-03-15T19:19:29.495422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(BASE_PATH + \"/model_1130.pt\")","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.497858Z","iopub.execute_input":"2024-03-15T19:19:29.498163Z","iopub.status.idle":"2024-03-15T19:19:29.502937Z","shell.execute_reply.started":"2024-03-15T19:19:29.498135Z","shell.execute_reply":"2024-03-15T19:19:29.502012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BASE_PATH = '/kaggle/input/sparcnet/SPaRCNet'","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.504501Z","iopub.execute_input":"2024-03-15T19:19:29.505254Z","iopub.status.idle":"2024-03-15T19:19:29.510811Z","shell.execute_reply.started":"2024-03-15T19:19:29.505224Z","shell.execute_reply":"2024-03-15T19:19:29.509823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"DATA_PATH = '/kaggle/input/sparcnet/SPaRCNet/Data'\nprint(DATA_PATH)","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.512364Z","iopub.execute_input":"2024-03-15T19:19:29.512749Z","iopub.status.idle":"2024-03-15T19:19:29.520327Z","shell.execute_reply.started":"2024-03-15T19:19:29.512720Z","shell.execute_reply":"2024-03-15T19:19:29.519081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\n\n# Define the directory path\noutput_directory = '/kaggle/working/sparcnet/SPaRCNet/Data/iiic'\n\n# Create the directory if it doesn't exist\nos.makedirs(output_directory, exist_ok=True)\n\"\"\"\n# Define the file path\nfile_path = os.path.join(output_directory, 'sample_cEEG_score.csv')\n\n# Create the file\nwith open(file_path, 'w'):\n    pass  # This will create an empty file-\n\nprint(\"File created successfully at:\", file_path)\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.521962Z","iopub.execute_input":"2024-03-15T19:19:29.522549Z","iopub.status.idle":"2024-03-15T19:19:29.530926Z","shell.execute_reply.started":"2024-03-15T19:19:29.522517Z","shell.execute_reply":"2024-03-15T19:19:29.529779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/'\ntrain = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nEEG_IDS = train.eeg_id.unique()\ndf = pd.read_parquet(f'{PATH}{EEG_IDS[1]}.parquet')\nFEATS = df.columns\n\ndef eeg_from_parquet(parquet_path, display=False):\n    \n    # EXTRACT MIDDLE 50 SECONDS\n    eeg = pd.read_parquet(parquet_path, columns=FEATS)\n    rows = len(eeg)\n    offset = (rows-10_000)//2\n    eeg = eeg.iloc[offset:offset+10_000]\n\n    if display: \n        plt.figure(figsize=(10,5))\n        offset = 0\n\n    # CONVERT TO NUMPY\n    data = np.zeros((10_000,len(FEATS)))\n    for j,col in enumerate(FEATS):\n\n        # FILL NAN\n        x = eeg[col].values.astype('float32')\n        m = np.nanmean(x)\n        if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n        else: x[:] = 0\n\n        data[:,j] = x\n\n        if display: \n            if j!=0: offset += x.max()\n            plt.plot(range(10_000),x-offset,label=col)\n            offset -= x.min()\n\n    if display:\n        plt.legend()\n        name = parquet_path.split('/')[-1]\n        name = name.split('.')[0]\n        plt.title(f'EEG {name}',size=16)\n        plt.show()\n\n    return data.T","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.532369Z","iopub.execute_input":"2024-03-15T19:19:29.532748Z","iopub.status.idle":"2024-03-15T19:19:29.826981Z","shell.execute_reply.started":"2024-03-15T19:19:29.532718Z","shell.execute_reply":"2024-03-15T19:19:29.825912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_predictions(DATA_RAW_PATH):\n      \n    ##################################################################################################################################      \n\n    print (\"\")\n    print (\"$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$\")\n    print (\"read data\")\n    print (\"$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$\")\n    print (\"\")    \n\n    ######################################################################################\n\n    total_file_list = os.listdir(DATA_RAW_PATH)\n    all_predictions = []\n\n    print (\"len(total_file_list):\" , len(total_file_list))\n    print (\"\")\n\n    T = len(total_file_list)\n\n    print (\"T: \", T)\n    print (\"\")\n\n\n    for t in range(T):\n\n        print (\"***************************************\")\n        print (\"t: \", t)\n        print (\"\")\n\n\n        file_name = total_file_list[t]\n        save_name = file_name.rstrip(\".parquet\")\n\n        if os.path.isfile(DATA_PATH+\"/iiic/\"+ save_name + \"_score.csv\"):\n            print('--alr done ' + save_name)\n\n        else:\n            print (\"file_name: \", file_name)\n            print (\"\")\n            print (\"save_name: \", save_name)\n            print (\"\")\n\n            path1 = DATA_RAW_PATH + file_name\n            print (\"path1: \", path1)\n            print (\"\")\n\n            X = eeg_from_parquet(f'{PATH}{EEG_IDS[t]}.parquet')\n            print (\"X.shape: \", X.shape)\n            print (\"\")\n\n            print (\"************ Montage\")\n            print (\"\")\n\n            X2 = X[[0,4,5,6, 11,15,16,17, 0,1,2,3, 11,12,13,14]] - X[[4,5,6,7, 15,16,17,18, 1,2,3,7, 12,13,14,18]]\n            print (\"X2.shape: \", X2.shape)\n\n            print (\"************ filtering\")\n            print (\"\")\n\n            X2 = notch_filter(X2, 200, 60, n_jobs=-1, verbose='ERROR')\n            X2 = filter_data(X2, 200, 0.5, 40, n_jobs=-1, verbose='ERROR') \n\n            print (\"X2.shape: \", X2.shape)\n            N = int(X2.shape[1]/400)\n\n            print (\"N: \", N)\n            print (\"\")\n\n            print (\"************ reshaping\")\n            print (\"\")\n\n            X3 = np.zeros((N-5,16,2000))\n\n            for n in range(N-5):\n                start_sn = n*400\n                end_sn = start_sn + 2000\n                x = X2[:,start_sn:end_sn]\n                X3[n,:,:] = x\n\n            print (\"X3.shape: \", X3.shape)\n\n            X = X3\n\n            print (\"X.shape: \", X.shape)\n            print (\"\")\n            print (\"np.isnan(X).sum(): \", np.isnan(X).sum())\n            print (\"np.max(X): \", np.max(X))\n            print (\"np.min(X): \", np.min(X))\n            print (\"\")\n\n            X = np.where(X<=500, X, 500)\n            X = np.where(X>=-500, X, -500)\n\n            print (\"X.shape: \", X.shape)\n            print (\"\")\n            print (\"np.isnan(X).sum(): \", np.isnan(X).sum())\n            print (\"np.max(X): \", np.max(X))\n            print (\"np.min(X): \", np.min(X))\n            print (\"\")\n\n            X4 = X\n\n            del X\n            del X2\n            del X3\n\n            print (\"X4.shape: \", X4.shape)\n            print (\"\")\n\n\n            ######################### evaluation\n            batch_size = 1000\n            def get_unlabeled_batch_list(X_train,batch_size):\n                N = X_train.shape[0]\n                sn_list = list(range(N))\n                K = int(N/batch_size)\n                X_list = list()\n                end_sn = 0\n\n                for k in range(K):\n                    start_sn = k*batch_size\n                    end_sn = start_sn + batch_size\n\n                    X = X_train[start_sn:end_sn,:,:]\n                    X_list.append(X)   \n                if not end_sn == N:\n                    X = X_train[end_sn:N,:,:]\n                    X_list.append(X)   \n\n                return (X_list)\n\n            ################### scanning\n            model_cnn.eval() #*\n\n            (X_batch_list) = get_unlabeled_batch_list(X4,batch_size)\n            K = len(X_batch_list)\n\n            print (\"K: \", K)\n            print (\"\")\n\n            S_list = list() \n            V_list = list()\n\n            for k in range(K):\n                if k%100 == 0:\n                    print (k)\n\n                X = X_batch_list[k]\n\n                X = torch.from_numpy(X).float()\n                X = X.to(device)\n\n                output, v = model_cnn(X)\n\n                S_list.append(output.detach().to('cpu'))\n                V_list.append(v.detach().to('cpu'))\n\n                del X\n                del output\n                del v\n\n            S2 = torch.cat(S_list,dim=0)\n            prob = F.softmax(S2, 1)\n            unlabeled_score = prob.numpy()\n            averaged_unlabeled_score = np.mean(unlabeled_score, axis=0)\n            all_predictions.append(averaged_unlabeled_score)\n\n            print (\"\")\n            print (\"average_unlabeled_score.shape: \", averaged_unlabeled_score.shape)\n            print (\"\")\n\n            V2 = torch.cat(V_list,dim=0)\n            unlabeled_V = V2.numpy()\n\n            print (\"unlabeled_V.shape: \", unlabeled_V.shape)\n            print (\"\")\n     \n        \n            print (\"writing finish\")\n            print (\"\")\n\n            del X4\n            del X_batch_list\n            \n        all_predictions_df = pd.DataFrame(np.vstack(all_predictions))\n\n        # Save the DataFrame to a single CSV file\n        output_path = '/kaggle/working/sparcnet/combined_predictions.csv'\n        all_predictions_csv = all_predictions_df.to_csv(output_path, index=False)\n\n        print(\"All predictions saved to:\", output_path)\n\n        print (\"\")\n        print (\"Done!\")\n        print (\"\")\n    return all_predictions_df.to_numpy()\n","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.828537Z","iopub.execute_input":"2024-03-15T19:19:29.828866Z","iopub.status.idle":"2024-03-15T19:19:29.862553Z","shell.execute_reply.started":"2024-03-15T19:19:29.828837Z","shell.execute_reply":"2024-03-15T19:19:29.861578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Generate predictions","metadata":{}},{"cell_type":"code","source":"sparcnetPred = make_predictions('/kaggle/input/hms-harmful-brain-activity-classification/test_eegs')","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:19:29.864092Z","iopub.execute_input":"2024-03-15T19:19:29.864970Z","iopub.status.idle":"2024-03-15T19:19:32.567530Z","shell.execute_reply.started":"2024-03-15T19:19:29.864930Z","shell.execute_reply":"2024-03-15T19:19:32.566568Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Submission ON TEST with ensemble\nif submission and ENSEMBLE:\n    preds = []\n    test_gen_kaggle = DataGenerator(test, mode='test', data_type='kaggle', specs = spectrograms2, eeg_specs = all_eegs2)\n    test_dataset_kaggle = tf.data.Dataset.from_generator(generator=test_gen_kaggle, \n                                               output_signature=(tf.TensorSpec(shape=(512,512,3), dtype=tf.float32),\n                                                                 tf.TensorSpec(shape=(6,), dtype=tf.float32))).batch(64).prefetch(tf.data.AUTOTUNE)\n    test_gen_both = DataGenerator(test, mode='test', data_type='both', specs = spectrograms2, eeg_specs = all_eegs2)\n    test_dataset_both = tf.data.Dataset.from_generator(generator=test_gen_both, \n                                               output_signature=(tf.TensorSpec(shape=(512,512,3), dtype=tf.float32),\n                                                                 tf.TensorSpec(shape=(6,), dtype=tf.float32))).batch(64).prefetch(tf.data.AUTOTUNE)\n\n    test_gen_eeg = DataGenerator(test, mode='test', data_type='eeg', specs = spectrograms2, eeg_specs = all_eegs2)\n    test_dataset_eeg = tf.data.Dataset.from_generator(generator=test_gen_eeg, \n                                               output_signature=(tf.TensorSpec(shape=(512,512,3), dtype=tf.float32),\n                                                                 tf.TensorSpec(shape=(6,), dtype=tf.float32))).batch(64).prefetch(tf.data.AUTOTUNE)\n    test_gen_raw = DataGenerator(test, mode='test', data_type='raw', raw_eegs=all_raw_eegs2)\n    test_dataset_raw = tf.data.Dataset.from_generator(generator=test_gen_raw, \n                                               output_signature=(tf.TensorSpec(shape=(2000,8), dtype=tf.float32),\n                                                                 tf.TensorSpec(shape=(6,), dtype=tf.float32))).batch(64).prefetch(tf.data.AUTOTUNE)\n \n    # LB SCORE FOR EACH MODEL\n    lbs = 1 - np.array(LBs)\n    weights = lbs/lbs.sum()\n    model = build_model()\n    model_wave = build_wave_model()\n\n    for i in range(5):\n        print(f'Fold {i+1}')\n        \n        model.load_weights(f'{LOAD_MODELS_FROM}model_K_{VERK}_{i}.weights.h5')\n        pred_kaggle = model.predict(test_dataset_kaggle, verbose=1)\n        \n        model.load_weights(f'{LOAD_MODELS_FROM}model_KE_{VERB}_{i}.weights.h5')\n        pred_both = model.predict(test_dataset_both, verbose=1)\n        \n        model.load_weights(f'{LOAD_MODELS_FROM}model_E_{VERE}_{i}.weights.h5')\n        pred_eeg = model.predict(test_dataset_eeg, verbose=1)\n        \n        model_wave.load_weights(f'{LOAD_MODELS_FROM}model_R_{VERR}_{i}.weights.h5')\n        pred_raw = model_wave.predict(test_dataset_raw, verbose=1)\n        \n        pred = np.array([pred_both,pred_eeg,pred_kaggle,pred_raw, sparcnetPred])\n        pred = np.average(pred,axis=0,weights=weights)\n        preds.append(pred)\n        \n    pred = np.mean(preds,axis=0)\n    print('Test preds shape',pred.shape)","metadata":{"papermill":{"duration":65.637583,"end_time":"2024-02-28T19:49:48.119265","exception":false,"start_time":"2024-02-28T19:48:42.481682","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:19:32.573799Z","iopub.execute_input":"2024-03-15T19:19:32.574347Z","iopub.status.idle":"2024-03-15T19:20:46.898103Z","shell.execute_reply.started":"2024-03-15T19:19:32.574315Z","shell.execute_reply":"2024-03-15T19:20:46.896856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nprint(os.listdir('/kaggle/input/features-head-starter-models'))\n","metadata":{"execution":{"iopub.status.busy":"2024-03-15T19:20:46.899424Z","iopub.execute_input":"2024-03-15T19:20:46.901356Z","iopub.status.idle":"2024-03-15T19:20:46.923587Z","shell.execute_reply.started":"2024-03-15T19:20:46.901324Z","shell.execute_reply":"2024-03-15T19:20:46.922450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if submission:\n    sub = pd.DataFrame({'eeg_id':test.eeg_id.values})\n    sub[TARGETS] = pred\n    sub.to_csv('submission.csv',index=False)\n    print('Submissionn shape',sub.shape)\n    print()\n    print(sub.head().to_string())","metadata":{"papermill":{"duration":0.030654,"end_time":"2024-02-28T19:49:48.162622","exception":false,"start_time":"2024-02-28T19:49:48.131968","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:20:46.925051Z","iopub.execute_input":"2024-03-15T19:20:46.926753Z","iopub.status.idle":"2024-03-15T19:20:46.944084Z","shell.execute_reply.started":"2024-03-15T19:20:46.926711Z","shell.execute_reply":"2024-03-15T19:20:46.942688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# SANITY CHECK TO CONFIRM PREDICTIONS SUM TO ONE\nif submission:\n    print(sub.iloc[:,-6:].sum(axis=1).to_string())","metadata":{"papermill":{"duration":0.022274,"end_time":"2024-02-28T19:49:48.196838","exception":false,"start_time":"2024-02-28T19:49:48.174564","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-03-15T19:20:46.946115Z","iopub.execute_input":"2024-03-15T19:20:46.946553Z","iopub.status.idle":"2024-03-15T19:20:46.956672Z","shell.execute_reply.started":"2024-03-15T19:20:46.946514Z","shell.execute_reply":"2024-03-15T19:20:46.955516Z"},"trusted":true},"execution_count":null,"outputs":[]}]}