{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782},{"sourceId":7447509,"sourceType":"datasetVersion","datasetId":4334995}],"dockerImageVersionId":30636,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Source:\nhttps://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-60","metadata":{"execution":{"iopub.status.busy":"2024-01-24T03:13:05.025830Z","iopub.execute_input":"2024-01-24T03:13:05.026197Z","iopub.status.idle":"2024-01-24T03:13:05.032680Z","shell.execute_reply.started":"2024-01-24T03:13:05.026169Z","shell.execute_reply":"2024-01-24T03:13:05.031256Z"}}},{"cell_type":"markdown","source":"# CatBoost Starter for Brain Comp\nThis is a CatBoost starter notebook for Kaggle's brain comp. We use only spectrogram features. (The model does not use eeg features yet). We can improve the CV and LB score by engineering more (spectrogram and/or eeg) features and we can tune the CatBoost model (and/or use other ML DL models). Discussion about this starter is [here][2].\n\nIn this notebook, we also compare five CV scores. Kaggle's sample submission uses equal predictions of 1/6 for all targets and achieves CV 1.46, LB 1.09. The best public notebook (on Jan 12th) [here][1] uses train means and achieves CV 1.26 LB 0.97. Our CatBoost model version 1 achieves CV 1.01 LB 0.81. Our CatBoost model version 2 achieves CV 0.82 LB 0.67. Then version 3 adds features from **EEG spectrograms** and achieves CV 0.74, wow! Let's see what LB is...\n\n# Exciting UPDATE!\nVersion 3 of this notebook trains using **both** Kaggle spectrograms and my new **EEG spectrograms** from my Kaggle dataset [here][3] (which were created from my spectrogram starter [here][4]). We boost the CV score and (most likely) LB score by almost `+0.10`, wow! \n\n### Version Notes\n* Version 1 - Uses spectrogram features from 10 minute window `means`. Achieves CV 1.01, LB 0.81\n* Version 2 - Uses spectrogram features from 10 minute and 20 second `means` and `mins`. Achieves CV 0.82, LB 0.67\n* Version 3 - Uses Kaggle spectrogrms **plus EEG spectrograms**. Achieves 0.74, LB to be determined...\n\n[1]: https://www.kaggle.com/code/seshurajup/eda-train-csv\n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467576\n[3]: https://www.kaggle.com/datasets/cdeotte/brain-eeg-spectrograms\n[4]: https://www.kaggle.com/code/cdeotte/how-to-make-spectrogram-from-eeg","metadata":{}},{"cell_type":"markdown","source":"# Load Libraries","metadata":{}},{"cell_type":"code","source":"import os, gc\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\n\nVER = 3","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:58:34.406695Z","iopub.execute_input":"2024-01-24T14:58:34.407509Z","iopub.status.idle":"2024-01-24T14:58:34.772380Z","shell.execute_reply.started":"2024-01-24T14:58:34.407475Z","shell.execute_reply":"2024-01-24T14:58:34.771569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Train Data","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape )\nprint('Targets', list(TARGETS))\ndf.head()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-24T14:58:34.773855Z","iopub.execute_input":"2024-01-24T14:58:34.774261Z","iopub.status.idle":"2024-01-24T14:58:35.038241Z","shell.execute_reply.started":"2024-01-24T14:58:34.774233Z","shell.execute_reply":"2024-01-24T14:58:35.037213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Non-Overlapping Eeg Id Train Data\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{}},{"cell_type":"code","source":"train = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg({'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg({'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n    train[t] = tmp[t].values\n    \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\nprint('Train non-overlapp eeg_id shape:', train.shape )","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:58:35.039656Z","iopub.execute_input":"2024-01-24T14:58:35.040070Z","iopub.status.idle":"2024-01-24T14:58:35.126206Z","shell.execute_reply.started":"2024-01-24T14:58:35.040032Z","shell.execute_reply":"2024-01-24T14:58:35.125289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Adding the count of eeg per patient based on following link (https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021#2599328)","metadata":{}},{"cell_type":"code","source":"tmp = df.groupby('patient_id').size()\neeg_id_count_per_patient_id = pd.DataFrame({'patient_id':tmp.index,'eeg_count':tmp.values})\nmean_value = np.mean(eeg_id_count_per_patient_id[\"eeg_count\"])\nstd_value = np.std(eeg_id_count_per_patient_id[\"eeg_count\"])\n\neeg_id_count_per_patient_id[\"eeg_count_z_score\"] = (eeg_id_count_per_patient_id[\"eeg_count\"] - mean_value)/std_value\neeg_id_count_per_patient_id = eeg_id_count_per_patient_id.drop(columns=[\"eeg_count\"])","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:58:35.128393Z","iopub.execute_input":"2024-01-24T14:58:35.128687Z","iopub.status.idle":"2024-01-24T14:58:35.141204Z","shell.execute_reply.started":"2024-01-24T14:58:35.128660Z","shell.execute_reply":"2024-01-24T14:58:35.140241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = train.merge(eeg_id_count_per_patient_id, on=\"patient_id\")","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:58:35.142517Z","iopub.execute_input":"2024-01-24T14:58:35.142883Z","iopub.status.idle":"2024-01-24T14:58:35.154634Z","shell.execute_reply.started":"2024-01-24T14:58:35.142845Z","shell.execute_reply":"2024-01-24T14:58:35.153630Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:58:35.155973Z","iopub.execute_input":"2024-01-24T14:58:35.156417Z","iopub.status.idle":"2024-01-24T14:58:35.174861Z","shell.execute_reply.started":"2024-01-24T14:58:35.156380Z","shell.execute_reply":"2024-01-24T14:58:35.173917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train CatBoost\nWe use the default settings for CatBoost which are pretty good. We can tune CatBoost manually to improve CV and LB score. Note that CatBoost will automatically use both Kaggle T4 GPUs (when we add parameter `task_type='GPU'`)  for super fast training!","metadata":{}},{"cell_type":"code","source":"import catboost as cat\nfrom catboost import CatBoostClassifier, Pool\nprint('CatBoost version',cat.__version__)","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:58:35.176170Z","iopub.execute_input":"2024-01-24T14:58:35.176689Z","iopub.status.idle":"2024-01-24T14:58:36.134808Z","shell.execute_reply.started":"2024-01-24T14:58:35.176652Z","shell.execute_reply":"2024-01-24T14:58:36.133784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold\n\nall_oof = []\nall_true = []\nTARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\n\ncolumns = [\"eeg_count_z_score\"]\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):   \n    \n    print('#'*25)\n    print(f'### Fold {i+1}')\n    print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n    print('#'*25)\n    \n    model = CatBoostClassifier(task_type='GPU', loss_function='MultiClass')\n    \n    train_pool = Pool(\n        data = train.loc[train_index, columns],\n        label = train.loc[train_index,'target'].map(TARS),\n    )\n    \n    valid_pool = Pool(\n        data = train.loc[valid_index, columns],\n        label = train.loc[valid_index,'target'].map(TARS),\n    )\n    \n    model.fit(train_pool,\n             verbose=100,\n             eval_set=valid_pool,\n             )\n    model.save_model(f'CAT_v{VER}_f{i}.cat')\n    \n    oof = model.predict_proba(valid_pool)\n    all_oof.append(oof)\n    all_true.append(train.loc[valid_index, TARGETS].values)\n    \n    del train_pool, valid_pool, oof #model\n    gc.collect()\n    \n    #break\n    \nall_oof = np.concatenate(all_oof)\nall_true = np.concatenate(all_true)","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:58:36.136252Z","iopub.execute_input":"2024-01-24T14:58:36.136998Z","iopub.status.idle":"2024-01-24T14:59:21.082008Z","shell.execute_reply.started":"2024-01-24T14:58:36.136959Z","shell.execute_reply":"2024-01-24T14:59:21.081006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Importance\nBelow we display the CatBoost top 25 feature importance for the last fold we trained.","metadata":{}},{"cell_type":"code","source":"TOP = 25\n\nfeature_importance = model.feature_importances_\nsorted_idx = np.argsort(feature_importance)\nfig = plt.figure(figsize=(10, 8))\nplt.barh(np.arange(len(sorted_idx))[-TOP:], feature_importance[sorted_idx][-TOP:], align='center')\nplt.yticks(np.arange(len(sorted_idx))[-TOP:], np.array(columns)[sorted_idx][-TOP:])\nplt.title(f'Feature Importance - Top {TOP}')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.083243Z","iopub.execute_input":"2024-01-24T14:59:21.084926Z","iopub.status.idle":"2024-01-24T14:59:21.367559Z","shell.execute_reply.started":"2024-01-24T14:59:21.084885Z","shell.execute_reply":"2024-01-24T14:59:21.366684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for CatBoost\nThis is CV score for our CatBoost model.","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score KL-Div for CatBoost =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.374351Z","iopub.execute_input":"2024-01-24T14:59:21.374731Z","iopub.status.idle":"2024-01-24T14:59:21.459664Z","shell.execute_reply.started":"2024-01-24T14:59:21.374699Z","shell.execute_reply":"2024-01-24T14:59:21.458731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for Preds 1/6\nThis is CV score for Kaggle's sample submission.csv which uses equal predictions of 1/6 for all targets.","metadata":{}},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof.copy())\nfor c in oof.columns:\n    oof[c] = 1/6.\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score for \"Use Equal Preds 1/6\" =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.461088Z","iopub.execute_input":"2024-01-24T14:59:21.461399Z","iopub.status.idle":"2024-01-24T14:59:21.526158Z","shell.execute_reply.started":"2024-01-24T14:59:21.461371Z","shell.execute_reply":"2024-01-24T14:59:21.525260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for EEG_Id Means\nThis is CV score for current highest scoring public notebook [here][1] which uses train means as predictions.\n\n[1]: https://www.kaggle.com/code/seshurajup/eda-train-csv","metadata":{}},{"cell_type":"code","source":"all_oof2 = []\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):  \n    #print('#'*25)\n    #print(f'### Fold {i+1}')\n        \n    y_train = train.iloc[train_index][TARGETS].values\n    y_valid = train.iloc[valid_index][TARGETS].values\n    \n    #print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n    #print('#'*25)\n        \n    oof = y_valid.copy()\n    for j in range(6):\n        oof[:,j] = y_train[:,j].mean()\n    oof = oof / oof.sum(axis=1,keepdims=True)\n    all_oof2.append(oof)\n    \nall_oof2 = np.concatenate(all_oof2)","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.527495Z","iopub.execute_input":"2024-01-24T14:59:21.527787Z","iopub.status.idle":"2024-01-24T14:59:21.569240Z","shell.execute_reply.started":"2024-01-24T14:59:21.527760Z","shell.execute_reply":"2024-01-24T14:59:21.568389Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof2.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score for \"Use Train Means\" =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.570748Z","iopub.execute_input":"2024-01-24T14:59:21.571134Z","iopub.status.idle":"2024-01-24T14:59:21.642473Z","shell.execute_reply.started":"2024-01-24T14:59:21.571103Z","shell.execute_reply":"2024-01-24T14:59:21.641340Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Infer Test and Create Submission CSV\nBelow we use our 5 CatBoost fold models to infer the test data and create a `submission.csv` file.","metadata":{}},{"cell_type":"code","source":"del train; gc.collect()\ntest = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\nprint('Test shape',test.shape)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.644153Z","iopub.execute_input":"2024-01-24T14:59:21.644533Z","iopub.status.idle":"2024-01-24T14:59:21.743523Z","shell.execute_reply.started":"2024-01-24T14:59:21.644499Z","shell.execute_reply":"2024-01-24T14:59:21.742737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tmp = test.groupby('patient_id').size()\neeg_id_count_per_patient_id = pd.DataFrame({'patient_id':tmp.index,'eeg_count':tmp.values})\nmean_value = np.mean(eeg_id_count_per_patient_id[\"eeg_count\"])\nstd_value = np.std(eeg_id_count_per_patient_id[\"eeg_count\"])\neeg_id_count_per_patient_id[\"eeg_count_z_score\"] = (eeg_id_count_per_patient_id[\"eeg_count\"] - mean_value)/std_value\neeg_id_count_per_patient_id = eeg_id_count_per_patient_id.drop(columns=[\"eeg_count\"])\n\ntest = test.merge(eeg_id_count_per_patient_id, on=\"patient_id\")","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.744787Z","iopub.execute_input":"2024-01-24T14:59:21.745719Z","iopub.status.idle":"2024-01-24T14:59:21.758385Z","shell.execute_reply.started":"2024-01-24T14:59:21.745685Z","shell.execute_reply":"2024-01-24T14:59:21.757431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.759899Z","iopub.execute_input":"2024-01-24T14:59:21.760691Z","iopub.status.idle":"2024-01-24T14:59:21.771072Z","shell.execute_reply.started":"2024-01-24T14:59:21.760656Z","shell.execute_reply":"2024-01-24T14:59:21.769857Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER CATBOOST ON TEST\npreds = []\n\ncolumns = [\"eeg_count_z_score\"]\n\nfor i in range(5):\n    print(i,', ',end='')\n    model = CatBoostClassifier(task_type='GPU')\n    model.load_model(f'CAT_v{VER}_f{i}.cat')\n    \n    test_pool = Pool(\n        data = test[columns]\n    )\n    \n    pred = model.predict_proba(test_pool)\n    preds.append(pred)\npred = np.mean(preds,axis=0)\nprint()\nprint('Test preds shape',pred.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.772488Z","iopub.execute_input":"2024-01-24T14:59:21.772955Z","iopub.status.idle":"2024-01-24T14:59:21.796273Z","shell.execute_reply.started":"2024-01-24T14:59:21.772910Z","shell.execute_reply":"2024-01-24T14:59:21.795418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.DataFrame({'eeg_id':test.eeg_id.values})\nsub[TARGETS] = pred\nsub.to_csv('submission.csv',index=False)\nprint('Submissionn shape',sub.shape)\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.797448Z","iopub.execute_input":"2024-01-24T14:59:21.797988Z","iopub.status.idle":"2024-01-24T14:59:21.816960Z","shell.execute_reply.started":"2024-01-24T14:59:21.797939Z","shell.execute_reply":"2024-01-24T14:59:21.816250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# SANITY CHECK TO CONFIRM PREDICTIONS SUM TO ONE\nsub.iloc[:,-6:].sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-01-24T14:59:21.818354Z","iopub.execute_input":"2024-01-24T14:59:21.818997Z","iopub.status.idle":"2024-01-24T14:59:21.827455Z","shell.execute_reply.started":"2024-01-24T14:59:21.818949Z","shell.execute_reply":"2024-01-24T14:59:21.826599Z"},"trusted":true},"execution_count":null,"outputs":[]}]}