{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7447509,"sourceType":"datasetVersion","datasetId":4334995}],"dockerImageVersionId":30636,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"raw","source":"# CatBoost Starter for Brain Comp\nThis is a CatBoost starter notebook for Kaggle's brain comp. We use only spectrogram features. (The model does not use eeg features yet). We can improve the CV and LB score by engineering more (spectrogram and/or eeg) features and we can tune the CatBoost model (and/or use other ML DL models). Discussion about this starter is [here][2].\n\nIn this notebook, we also compare five CV scores. Kaggle's sample submission uses equal predictions of 1/6 for all targets and achieves CV 1.46, LB 1.09. The best public notebook (on Jan 12th) [here][1] uses train means and achieves CV 1.26 LB 0.97. Our CatBoost model version 1 achieves CV 1.01 LB 0.81. Our CatBoost model version 2 achieves CV 0.82 LB 0.67. Then version 3 adds features from **EEG spectrograms** and achieves CV 0.74, wow! Let's see what LB is...\n\n# Exciting UPDATE!\nVersion 3 of this notebook trains using **both** Kaggle spectrograms and my new **EEG spectrograms** from my Kaggle dataset [here][3] (which were created from my spectrogram starter [here][4]). We boost the CV score and (most likely) LB score by almost `+0.10`, wow! \n\n### Version Notes\n* Version 1 - Uses spectrogram features from 10 minute window `means`. Achieves CV 1.01, LB 0.81\n* Version 2 - Uses spectrogram features from 10 minute and 20 second `means` and `mins`. Achieves CV 0.82, LB 0.67\n* Version 3 - Uses Kaggle spectrogrms **plus EEG spectrograms**. Achieves 0.74, LB to be determined...\n\n[1]: https://www.kaggle.com/code/seshurajup/eda-train-csv\n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467576\n[3]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms\n[4]: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg","metadata":{}},{"cell_type":"markdown","source":"# Load Libraries","metadata":{}},{"cell_type":"code","source":"import os, gc\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\n\nVER = 3","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:48:53.858876Z","iopub.execute_input":"2024-02-05T07:48:53.859206Z","iopub.status.idle":"2024-02-05T07:48:55.644409Z","shell.execute_reply.started":"2024-02-05T07:48:53.859178Z","shell.execute_reply":"2024-02-05T07:48:55.643292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Train Data","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape )\nprint('Targets', list(TARGETS))\ndf.head()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-02-05T07:48:55.646764Z","iopub.execute_input":"2024-02-05T07:48:55.647710Z","iopub.status.idle":"2024-02-05T07:48:56.026131Z","shell.execute_reply.started":"2024-02-05T07:48:55.647668Z","shell.execute_reply":"2024-02-05T07:48:56.025044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Non-Overlapping Eeg Id Train Data\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{}},{"cell_type":"code","source":"train = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n    train[t] = tmp[t].values\n    \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\nprint('Train non-overlapp eeg_id shape:', train.shape )\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:48:56.027542Z","iopub.execute_input":"2024-02-05T07:48:56.027865Z","iopub.status.idle":"2024-02-05T07:48:56.152978Z","shell.execute_reply.started":"2024-02-05T07:48:56.027838Z","shell.execute_reply":"2024-02-05T07:48:56.151777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineer\nIn this section, we create features for our CatBoost model. \n\nFirst we need to read in all 11k train spectrogram files. Reading thousands of files takes 11 minutes with Pandas. Instead, we can read 1 file from my [Kaggle dataset here][1] which contains all the 11k spectrograms in less than 1 minute! To use my [Kaggle dataset][1], set variable `READ_SPEC_FILES = False`. Thanks for upvoting my Kaggle dataset!\n\nNext we need to engineer features for our CatBoost model. In version 1 notebook, we just take the mean (over time) of each of the 400 spectrogram frequencies (using middle 10 minutes). This produces 400 features (per each unique eeg id). We can improve CV and LB score by engineering new features (and/or tuning CatBoost).\n\nUPDATE: Version 2 creates features from `means` and `mins`. And version 2 uses `10 minute windows` and `20 second windows`.\n\nUPDATE: Version 3 uses **both** Kaggle spectrograms and **EEG spectrograms**. We load EEG spectrograms from my Kaggle dataset [here][2]. These EEG spectrograms were created from EEG raw waveforms in my spectrogram starter [here][3]. Thank you everyone for upvoting my new Kaggle dataset!\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\n[2]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms\n[3]: https://www.kaggle.com/code/cdeotte/efficientnetb2-starter-lb-0-57","metadata":{}},{"cell_type":"code","source":"READ_SPEC_FILES = False\nREAD_EEG_SPEC_FILES = False","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:48:56.155628Z","iopub.execute_input":"2024-02-05T07:48:56.156029Z","iopub.status.idle":"2024-02-05T07:48:56.161307Z","shell.execute_reply.started":"2024-02-05T07:48:56.155992Z","shell.execute_reply":"2024-02-05T07:48:56.160123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# READ ALL SPECTROGRAMS\nPATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/'\nfiles = os.listdir(PATH)\nprint(f'There are {len(files)} spectrogram parquets')\n\nif READ_SPEC_FILES:    \n    spectrograms = {}\n    for i,f in enumerate(files):\n        if i%100==0: print(i,', ',end='')\n        tmp = pd.read_parquet(f'{PATH}{f}')\n        name = int(f.split('.')[0])\n        spectrograms[name] = tmp.iloc[:,1:].values\nelse:\n    spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:48:56.162675Z","iopub.execute_input":"2024-02-05T07:48:56.163041Z","iopub.status.idle":"2024-02-05T07:50:04.568164Z","shell.execute_reply.started":"2024-02-05T07:48:56.163005Z","shell.execute_reply":"2024-02-05T07:50:04.567276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# READ ALL EEG SPECTROGRAMS\nif READ_EEG_SPEC_FILES:\n    all_eegs = {}\n    for i,e in enumerate(train.eeg_id.values):\n        if i%100==0: print(i,', ',end='')\n        x = np.load(f'/kaggle/input/brain-eeg-spectrograms/EEG_Spectrograms/{e}.npy')\n        all_eegs[e] = x\nelse:\n    all_eegs = np.load('/kaggle/input/brain-eeg-spectrograms/eeg_specs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:50:04.569457Z","iopub.execute_input":"2024-02-05T07:50:04.570107Z","iopub.status.idle":"2024-02-05T07:51:26.402180Z","shell.execute_reply.started":"2024-02-05T07:50:04.570068Z","shell.execute_reply":"2024-02-05T07:51:26.401268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time\n# ENGINEER FEATURES\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# FEATURE NAMES\nSPEC_COLS = pd.read_parquet(f'{PATH}1000086677.parquet').columns[1:]\nFEATURES = [f'{c}_mean_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_mean_20s' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_20s' for c in SPEC_COLS]\nFEATURES += [f'eeg_mean_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_min_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_max_f{x}_10s' for x in range(512)]\nFEATURES += [f'eeg_std_f{x}_10s' for x in range(512)]\nprint(f'We are creating {len(FEATURES)} features for {len(train)} rows... ',end='')\n\ndata = np.zeros((len(train),len(FEATURES)))\nfor k in range(len(train)):\n    if k%100==0: print(k,', ',end='')\n    row = train.iloc[k]\n    r = int( (row['min'] + row['max'])//4 ) \n\n    # 10 MINUTE WINDOW FEATURES (MEANS and MINS)\n    x = np.nanmean(spectrograms[row.spec_id][r:r+300,:],axis=0)\n    data[k,:400] = x\n    x = np.nanmin(spectrograms[row.spec_id][r:r+300,:],axis=0)\n    data[k,400:800] = x\n\n    # 20 SECOND WINDOW FEATURES (MEANS and MINS)\n    x = np.nanmean(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n    data[k,800:1200] = x\n    x = np.nanmin(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n    data[k,1200:1600] = x\n\n    # RESHAPE EEG SPECTROGRAMS 128x256x4 => 512x256\n    eeg_spec = np.zeros((512,256),dtype='float32')\n    xx = all_eegs[row.eeg_id]\n    for j in range(4): eeg_spec[128*j:128*(j+1),] = xx[:,:,j]\n\n    # 10 SECOND WINDOW FROM EEG SPECTROGRAMS \n    x = np.nanmean(eeg_spec.T[100:-100,:],axis=0)\n    data[k,1600:2112] = x\n    x = np.nanmin(eeg_spec.T[100:-100,:],axis=0)\n    data[k,2112:2624] = x\n    x = np.nanmax(eeg_spec.T[100:-100,:],axis=0)\n    data[k,2624:3136] = x\n    x = np.nanstd(eeg_spec.T[100:-100,:],axis=0)\n    data[k,3136:3648] = x\n\ntrain[FEATURES] = data\nprint(); print('New train shape:',train.shape)","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:51:26.403475Z","iopub.execute_input":"2024-02-05T07:51:26.404272Z","iopub.status.idle":"2024-02-05T07:52:17.025513Z","shell.execute_reply.started":"2024-02-05T07:51:26.404235Z","shell.execute_reply":"2024-02-05T07:52:17.024579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# FREE MEMORY\ndel all_eegs, spectrograms, data\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:52:17.026711Z","iopub.execute_input":"2024-02-05T07:52:17.027006Z","iopub.status.idle":"2024-02-05T07:52:17.201363Z","shell.execute_reply.started":"2024-02-05T07:52:17.026979Z","shell.execute_reply":"2024-02-05T07:52:17.200377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.to_parquet('train.parquet')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train CatBoost\nWe use the default settings for CatBoost which are pretty good. We can tune CatBoost manually to improve CV and LB score. Note that CatBoost will automatically use both Kaggle T4 GPUs (when we add parameter `task_type='GPU'`)  for super fast training!","metadata":{}},{"cell_type":"code","source":"import catboost as cat\nfrom catboost import CatBoostClassifier, Pool\nprint('CatBoost version',cat.__version__)","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:52:41.360142Z","iopub.execute_input":"2024-02-05T07:52:41.360509Z","iopub.status.idle":"2024-02-05T07:52:43.431653Z","shell.execute_reply.started":"2024-02-05T07:52:41.360478Z","shell.execute_reply":"2024-02-05T07:52:43.430762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from sklearn.model_selection import KFold, GroupKFold\n\n# all_oof = []\n# all_true = []\n# TARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\n\n# gkf = GroupKFold(n_splits=5)\n# for i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):   \n    \n#     print('#'*25)\n#     print(f'### Fold {i+1}')\n#     print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n#     print('#'*25)\n    \n#     model = CatBoostClassifier(task_type='GPU',\n#                                loss_function='MultiClass')\n    \n#     train_pool = Pool(\n#         data = train.loc[train_index,FEATURES],\n#         label = train.loc[train_index,'target'].map(TARS),\n#     )\n    \n#     valid_pool = Pool(\n#         data = train.loc[valid_index,FEATURES],\n#         label = train.loc[valid_index,'target'].map(TARS),\n#     )\n    \n#     print(train.loc[valid_index,'target'].map(TARS))\n    \n#     model.fit(train_pool,\n#              verbose=100,\n#              eval_set=valid_pool,\n#              )\n#     model.save_model(f'CAT_v{VER}_f{i}.cat')\n    \n#     oof = model.predict_proba(valid_pool)\n#     all_oof.append(oof)\n#     all_true.append(train.loc[valid_index, TARGETS].values)\n    \n#     del train_pool, valid_pool, oof #model\n#     gc.collect()\n    \n    #break\n    \n# all_oof = np.concatenate(all_oof)\n# all_true = np.concatenate(all_true)","metadata":{"execution":{"iopub.status.busy":"2024-02-05T07:53:51.060478Z","iopub.execute_input":"2024-02-05T07:53:51.060863Z","iopub.status.idle":"2024-02-05T07:53:59.839321Z","shell.execute_reply.started":"2024-02-05T07:53:51.060827Z","shell.execute_reply":"2024-02-05T07:53:59.838311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Importance\nBelow we display the CatBoost top 25 feature importance for the last fold we trained.","metadata":{}},{"cell_type":"code","source":"# TOP = 25\n\n# feature_importance = model.feature_importances_\n# sorted_idx = np.argsort(feature_importance)\n# fig = plt.figure(figsize=(10, 8))\n# plt.barh(np.arange(len(sorted_idx))[-TOP:], feature_importance[sorted_idx][-TOP:], align='center')\n# plt.yticks(np.arange(len(sorted_idx))[-TOP:], np.array(FEATURES)[sorted_idx][-TOP:])\n# plt.title(f'Feature Importance - Top {TOP}')\n# plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:45:53.56356Z","iopub.execute_input":"2024-01-21T18:45:53.564191Z","iopub.status.idle":"2024-01-21T18:45:54.196157Z","shell.execute_reply.started":"2024-01-21T18:45:53.564161Z","shell.execute_reply":"2024-01-21T18:45:54.195278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for CatBoost\nThis is CV score for our CatBoost model.","metadata":{}},{"cell_type":"code","source":"# import sys\n# sys.path.append('/kaggle/input/kaggle-kl-div')\n# from kaggle_kl_div import score\n\n# oof = pd.DataFrame(all_oof.copy())\n# oof['id'] = np.arange(len(oof))\n\n# true = pd.DataFrame(all_true.copy())\n# true['id'] = np.arange(len(true))\n\n# cv = score(solution=true, submission=oof, row_id_column_name='id')\n# print('CV Score KL-Div for CatBoost =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:45:54.197132Z","iopub.execute_input":"2024-01-21T18:45:54.197462Z","iopub.status.idle":"2024-01-21T18:45:54.254727Z","shell.execute_reply.started":"2024-01-21T18:45:54.197433Z","shell.execute_reply":"2024-01-21T18:45:54.253934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for Preds 1/6\nThis is CV score for Kaggle's sample submission.csv which uses equal predictions of 1/6 for all targets.","metadata":{}},{"cell_type":"code","source":"# oof = pd.DataFrame(all_oof.copy())\n# for c in oof.columns:\n#     oof[c] = 1/6.\n# oof['id'] = np.arange(len(oof))\n\n# true = pd.DataFrame(all_true.copy())\n# true['id'] = np.arange(len(true))\n\n# cv = score(solution=true, submission=oof, row_id_column_name='id')\n# print('CV Score for \"Use Equal Preds 1/6\" =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:45:54.255743Z","iopub.execute_input":"2024-01-21T18:45:54.256214Z","iopub.status.idle":"2024-01-21T18:45:54.293065Z","shell.execute_reply.started":"2024-01-21T18:45:54.256185Z","shell.execute_reply":"2024-01-21T18:45:54.292336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for EEG_Id Means\nThis is CV score for current highest scoring public notebook [here][1] which uses train means as predictions.\n\n[1]: https://www.kaggle.com/code/seshurajup/eda-train-csv","metadata":{}},{"cell_type":"code","source":"# all_oof2 = []\n\n# gkf = GroupKFold(n_splits=5)\n# for i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):  \n#     #print('#'*25)\n#     #print(f'### Fold {i+1}')\n        \n#     y_train = train.iloc[train_index][TARGETS].values\n#     y_valid = train.iloc[valid_index][TARGETS].values\n    \n#     #print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n#     #print('#'*25)\n        \n#     oof = y_valid.copy()\n#     for j in range(6):\n#         oof[:,j] = y_train[:,j].mean()\n#     oof = oof / oof.sum(axis=1,keepdims=True)\n#     all_oof2.append(oof)\n    \n# all_oof2 = np.concatenate(all_oof2)","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:45:54.293989Z","iopub.execute_input":"2024-01-21T18:45:54.294735Z","iopub.status.idle":"2024-01-21T18:45:55.121332Z","shell.execute_reply.started":"2024-01-21T18:45:54.294707Z","shell.execute_reply":"2024-01-21T18:45:55.120557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# oof = pd.DataFrame(all_oof2.copy())\n# oof['id'] = np.arange(len(oof))\n\n# true = pd.DataFrame(all_true.copy())\n# true['id'] = np.arange(len(true))\n\n# cv = score(solution=true, submission=oof, row_id_column_name='id')\n# print('CV Score for \"Use Train Means\" =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:45:55.122536Z","iopub.execute_input":"2024-01-21T18:45:55.122874Z","iopub.status.idle":"2024-01-21T18:45:55.155713Z","shell.execute_reply.started":"2024-01-21T18:45:55.122847Z","shell.execute_reply":"2024-01-21T18:45:55.154717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Infer Test and Create Submission CSV\nBelow we use our 5 CatBoost fold models to infer the test data and create a `submission.csv` file.","metadata":{}},{"cell_type":"code","source":"# del train; gc.collect()\n# test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\n# print('Test shape',test.shape)\n# test.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:45:55.156777Z","iopub.execute_input":"2024-01-21T18:45:55.15707Z","iopub.status.idle":"2024-01-21T18:45:55.240099Z","shell.execute_reply.started":"2024-01-21T18:45:55.157047Z","shell.execute_reply":"2024-01-21T18:45:55.239185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import pywt, librosa\n\n# USE_WAVELET = None \n\n# NAMES = ['LL','LP','RP','RR']\n\n# FEATS = [['Fp1','F7','T3','T5','O1'],\n#          ['Fp1','F3','C3','P3','O1'],\n#          ['Fp2','F8','T4','T6','O2'],\n#          ['Fp2','F4','C4','P4','O2']]\n\n# # DENOISE FUNCTION\n# def maddest(d, axis=None):\n#     return np.mean(np.absolute(d - np.mean(d, axis)), axis)\n\n# def denoise(x, wavelet='haar', level=1):    \n#     coeff = pywt.wavedec(x, wavelet, mode=\"per\")\n#     sigma = (1/0.6745) * maddest(coeff[-level])\n\n#     uthresh = sigma * np.sqrt(2*np.log(len(x)))\n#     coeff[1:] = (pywt.threshold(i, value=uthresh, mode='hard') for i in coeff[1:])\n\n#     ret=pywt.waverec(coeff, wavelet, mode='per')\n    \n#     return ret\n\n# def spectrogram_from_eeg(parquet_path, display=False):\n    \n#     # LOAD MIDDLE 50 SECONDS OF EEG SERIES\n#     eeg = pd.read_parquet(parquet_path)\n#     middle = (len(eeg)-10_000)//2\n#     eeg = eeg.iloc[middle:middle+10_000]\n    \n#     # VARIABLE TO HOLD SPECTROGRAM\n#     img = np.zeros((128,256,4),dtype='float32')\n    \n#     if display: plt.figure(figsize=(10,7))\n#     signals = []\n#     for k in range(4):\n#         COLS = FEATS[k]\n        \n#         for kk in range(4):\n        \n#             # COMPUTE PAIR DIFFERENCES\n#             x = eeg[COLS[kk]].values - eeg[COLS[kk+1]].values\n\n#             # FILL NANS\n#             m = np.nanmean(x)\n#             if np.isnan(x).mean()<1: x = np.nan_to_num(x,nan=m)\n#             else: x[:] = 0\n\n#             # DENOISE\n#             if USE_WAVELET:\n#                 x = denoise(x, wavelet=USE_WAVELET)\n#             signals.append(x)\n\n#             # RAW SPECTROGRAM\n#             mel_spec = librosa.feature.melspectrogram(y=x, sr=200, hop_length=len(x)//256, \n#                   n_fft=1024, n_mels=128, fmin=0, fmax=20, win_length=128)\n\n#             # LOG TRANSFORM\n#             width = (mel_spec.shape[1]//32)*32\n#             mel_spec_db = librosa.power_to_db(mel_spec, ref=np.max).astype(np.float32)[:,:width]\n\n#             # STANDARDIZE TO -1 TO 1\n#             mel_spec_db = (mel_spec_db+40)/40 \n#             img[:,:,k] += mel_spec_db\n                \n#         # AVERAGE THE 4 MONTAGE DIFFERENCES\n#         img[:,:,k] /= 4.0\n        \n#         if display:\n#             plt.subplot(2,2,k+1)\n#             plt.imshow(img[:,:,k],aspect='auto',origin='lower')\n#             plt.title(f'EEG {eeg_id} - Spectrogram {NAMES[k]}')\n            \n#     if display: \n#         plt.show()\n#         plt.figure(figsize=(10,5))\n#         offset = 0\n#         for k in range(4):\n#             if k>0: offset -= signals[3-k].min()\n#             plt.plot(range(10_000),signals[k]+offset,label=NAMES[3-k])\n#             offset += signals[3-k].max()\n#         plt.legend()\n#         plt.title(f'EEG {eeg_id} Signals')\n#         plt.show()\n#         print(); print('#'*25); print()\n        \n#     return img","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-01-21T18:45:55.241316Z","iopub.execute_input":"2024-01-21T18:45:55.241642Z","iopub.status.idle":"2024-01-21T18:45:55.360661Z","shell.execute_reply.started":"2024-01-21T18:45:55.241616Z","shell.execute_reply":"2024-01-21T18:45:55.35991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CREATE ALL EEG SPECTROGRAMS\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_eegs/'\n# DISPLAY = 0\n# EEG_IDS2 = test.eeg_id.unique()\n# all_eegs2 = {}\n\n# print('Converting Test EEG to Spectrograms...'); print()\n# for i,eeg_id in enumerate(EEG_IDS2):\n        \n#     # CREATE SPECTROGRAM FROM EEG PARQUET\n#     img = spectrogram_from_eeg(f'{PATH2}{eeg_id}.parquet', i<DISPLAY)\n#     all_eegs2[eeg_id] = img","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:45:55.36159Z","iopub.execute_input":"2024-01-21T18:45:55.361827Z","iopub.status.idle":"2024-01-21T18:46:04.307631Z","shell.execute_reply.started":"2024-01-21T18:45:55.361806Z","shell.execute_reply":"2024-01-21T18:46:04.306258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# FEATURE ENGINEER TEST\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\n# data = np.zeros((len(test),len(FEATURES)))\n    \n# for k in range(len(test)):\n#     row = test.iloc[k]\n#     s = int( row.spectrogram_id )\n#     spec = pd.read_parquet(f'{PATH2}{s}.parquet')\n    \n#     # 10 MINUTE WINDOW FEATURES\n#     x = np.nanmean( spec.iloc[:,1:].values, axis=0)\n#     data[k,:400] = x\n#     x = np.nanmin( spec.iloc[:,1:].values, axis=0)\n#     data[k,400:800] = x\n\n#     # 20 SECOND WINDOW FEATURES\n#     x = np.nanmean( spec.iloc[145:155,1:].values, axis=0)\n#     data[k,800:1200] = x\n#     x = np.nanmin( spec.iloc[145:155,1:].values, axis=0)\n#     data[k,1200:1600] = x\n    \n#     # RESHAPE EEG SPECTROGRAMS 128x256x4 => 512x256\n#     eeg_spec = np.zeros((512,256),dtype='float32')\n#     xx = all_eegs2[row.eeg_id]\n#     for j in range(4): eeg_spec[128*j:128*(j+1),] = xx[:,:,j]\n\n#     # 10 SECOND WINDOW FROM EEG SPECTROGRAMS \n#     x = np.nanmean(eeg_spec.T[100:-100,:],axis=0)\n#     data[k,1600:2112] = x\n#     x = np.nanmin(eeg_spec.T[100:-100,:],axis=0)\n#     data[k,2112:2624] = x\n#     x = np.nanmax(eeg_spec.T[100:-100,:],axis=0)\n#     data[k,2624:3136] = x\n#     x = np.nanstd(eeg_spec.T[100:-100,:],axis=0)\n#     data[k,3136:3648] = x\n\n# test[FEATURES] = data\n# print('New test shape',test.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:46:04.309474Z","iopub.execute_input":"2024-01-21T18:46:04.310288Z","iopub.status.idle":"2024-01-21T18:46:07.577789Z","shell.execute_reply.started":"2024-01-21T18:46:04.310252Z","shell.execute_reply":"2024-01-21T18:46:07.576758Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER CATBOOST ON TEST\n# preds = []\n\n# for i in range(5):\n#     print(i,', ',end='')\n#     model = CatBoostClassifier(task_type='GPU')\n#     model.load_model(f'CAT_v{VER}_f{i}.cat')\n    \n#     test_pool = Pool(\n#         data = test[FEATURES]\n#     )\n    \n#     pred = model.predict_proba(test_pool)\n#     preds.append(pred)\n# pred = np.mean(preds,axis=0)\n# print()\n# print('Test preds shape',pred.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:46:07.578905Z","iopub.execute_input":"2024-01-21T18:46:07.579215Z","iopub.status.idle":"2024-01-21T18:46:09.552075Z","shell.execute_reply.started":"2024-01-21T18:46:07.579189Z","shell.execute_reply":"2024-01-21T18:46:09.551139Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# sub = pd.DataFrame({'eeg_id':test.eeg_id.values})\n# sub[TARGETS] = pred\n# sub.to_csv('submission.csv',index=False)\n# print('Submissionn shape',sub.shape)\n# sub.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:46:09.553317Z","iopub.execute_input":"2024-01-21T18:46:09.553591Z","iopub.status.idle":"2024-01-21T18:46:09.573227Z","shell.execute_reply.started":"2024-01-21T18:46:09.553569Z","shell.execute_reply":"2024-01-21T18:46:09.57242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# SANITY CHECK TO CONFIRM PREDICTIONS SUM TO ONE\n# sub.iloc[:,-6:].sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-01-21T18:46:09.574196Z","iopub.execute_input":"2024-01-21T18:46:09.574453Z","iopub.status.idle":"2024-01-21T18:46:09.583849Z","shell.execute_reply.started":"2024-01-21T18:46:09.57443Z","shell.execute_reply":"2024-01-21T18:46:09.582992Z"},"trusted":true},"execution_count":null,"outputs":[]}]}