{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392733,"sourceType":"datasetVersion","datasetId":4297749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782}],"dockerImageVersionId":30636,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **I just added that the catboost hyperparameter (learning rate,iteration); all credit goes to Chris Deotte @cdeotte. LB reached 0.65! Superb Chris Deotte, I never expected this outcome. Your code and approach inspire me and help me learn a lot.** \n\n# **https://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-67**","metadata":{}},{"cell_type":"markdown","source":"# CatBoost Starter for Brain Comp\nThis is a CatBoost starter notebook for Kaggle's brain comp. We use only spectrogram features. (The model does not use eeg features yet). We can improve the CV and LB score by engineering more (spectrogram and/or eeg) features and we can tune the CatBoost model (and/or use other ML DL models). Discussion about this starter is [here][2].\n\nIn this notebook, we also compare four CV scores. Kaggle's sample submission uses equal predictions of 1/6 for all targets and achieves CV 1.46, LB 1.09. The best public notebook (on Jan 12th) [here][1] uses train means and achieves CV 1.26 LB 0.97. Our CatBoost model version 1 achieves CV 1.01 LB 0.81. Our CatBoost model version 2 achieves CV 0.82. Let's submit version 2 and see what the LB score is!\n\n### Version Notes\n* Version 1 - Uses spectrogram features from 10 minute window `means`. Achieves CV 1.01, LB 0.81\n* Version 2 - Uses spectrogram features from 10 minute and 20 second `means` and `mins`. Achieves CV 0.82, LB to be determined...\n\n[1]: https://www.kaggle.com/code/seshurajup/eda-train-csv\n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467576\n","metadata":{}},{"cell_type":"markdown","source":"# Load Libraries","metadata":{}},{"cell_type":"code","source":"import os\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0,1\"\nimport pandas as pd, numpy as np\nimport matplotlib.pyplot as plt\n\nVER = 2","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:11:08.437441Z","iopub.execute_input":"2024-01-14T05:11:08.438271Z","iopub.status.idle":"2024-01-14T05:11:10.093675Z","shell.execute_reply.started":"2024-01-14T05:11:08.438237Z","shell.execute_reply":"2024-01-14T05:11:10.092703Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Train Data","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape )\nprint('Targets', list(TARGETS))\ndf.head()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-14T05:11:10.095533Z","iopub.execute_input":"2024-01-14T05:11:10.095942Z","iopub.status.idle":"2024-01-14T05:11:10.398194Z","shell.execute_reply.started":"2024-01-14T05:11:10.095914Z","shell.execute_reply":"2024-01-14T05:11:10.396605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Non-Overlapping Eeg Id Train Data\nThe competition data description says that test data does not have multiple crops from the same `eeg_id`. Therefore we will train and validate using only 1 crop per `eeg_id`. There is a discussion about this [here][1].\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021","metadata":{}},{"cell_type":"code","source":"train = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n    train[t] = tmp[t].values\n    \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\nprint('Train non-overlapp eeg_id shape:', train.shape )\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:11:10.399453Z","iopub.execute_input":"2024-01-14T05:11:10.399758Z","iopub.status.idle":"2024-01-14T05:11:10.517139Z","shell.execute_reply.started":"2024-01-14T05:11:10.399731Z","shell.execute_reply":"2024-01-14T05:11:10.515986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Engineer\nIn this section, we create features for our CatBoost model. \n\nFirst we need to read in all 11k train spectrogram files. Reading thousands of files takes 11 minutes with Pandas. Instead, we can read 1 file from my [Kaggle dataset here][1] which contains all the 11k spectrograms in less than 1 minute! To use my [Kaggle dataset][1], set variable `READ_SPEC_FILES = False`. Don't forget to upvote this helpful [dataset][1] :-)\n\nNext we need to engineer features for our CatBoost model. In this notebook, we just take the mean (over time) of each of the 400 spectrogram frequencies (using middle 10 minutes). This produces 400 features (per each unique eeg id). We can improve CV and LB score by engineering new features (and/or tuning CatBoost).\n\nUPDATE: Version 2 creates features from `means` and `mins`. And version 2 uses `10 minute windows` and `20 second windows`.\n\n[1]: https://www.kaggle.com/datasets/cdeotte/brain-spectrograms","metadata":{}},{"cell_type":"code","source":"READ_SPEC_FILES = False\nFEATURE_ENGINEER = True","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:11:10.519249Z","iopub.execute_input":"2024-01-14T05:11:10.519552Z","iopub.status.idle":"2024-01-14T05:11:10.524078Z","shell.execute_reply.started":"2024-01-14T05:11:10.519527Z","shell.execute_reply":"2024-01-14T05:11:10.522958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# READ ALL SPECTROGRAMS\nPATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/'\nfiles = os.listdir(PATH)\nprint(f'There are {len(files)} spectrogram parquets')\n\nif READ_SPEC_FILES:    \n    spectrograms = {}\n    for i,f in enumerate(files):\n        if i%100==0: print(i,', ',end='')\n        tmp = pd.read_parquet(f'{PATH}{f}')\n        name = int(f.split('.')[0])\n        spectrograms[name] = tmp.iloc[:,1:].values\nelse:\n    spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:11:10.525656Z","iopub.execute_input":"2024-01-14T05:11:10.526004Z","iopub.status.idle":"2024-01-14T05:12:13.843910Z","shell.execute_reply.started":"2024-01-14T05:11:10.525960Z","shell.execute_reply":"2024-01-14T05:12:13.842811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time\n# ENGINEER FEATURES\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# FEATURE NAMES\nSPEC_COLS = pd.read_parquet(f'{PATH}1000086677.parquet').columns[1:]\nFEATURES = [f'{c}_mean_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_mean_20s' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_20s' for c in SPEC_COLS]\nprint(f'We are creating {len(FEATURES)} features for {len(train)} rows... ',end='')\n\nif FEATURE_ENGINEER:\n    data = np.zeros((len(train),len(FEATURES)))\n    for k in range(len(train)):\n        if k%100==0: print(k,', ',end='')\n        row = train.iloc[k]\n        r = int( (row['min'] + row['max'])//4 ) \n        \n        # 10 MINUTE WINDOW FEATURES (MEANS and MINS)\n        x = np.nanmean(spectrograms[row.spec_id][r:r+300,:],axis=0)\n        data[k,:400] = x\n        x = np.nanmin(spectrograms[row.spec_id][r:r+300,:],axis=0)\n        data[k,400:800] = x\n        \n        # 20 SECOND WINDOW FEATURES (MEANS and MINS)\n        x = np.nanmean(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n        data[k,800:1200] = x\n        x = np.nanmin(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n        data[k,1200:1600] = x\n\n    train[FEATURES] = data\nelse:\n    train = pd.read_parquet('/kaggle/input/brain-spectrograms/train.pqt')\nprint()\nprint('New train shape:',train.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:13.845139Z","iopub.execute_input":"2024-01-14T05:12:13.845423Z","iopub.status.idle":"2024-01-14T05:12:31.955030Z","shell.execute_reply.started":"2024-01-14T05:12:13.845398Z","shell.execute_reply":"2024-01-14T05:12:31.954045Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train CatBoost\nWe use the default settings for CatBoost which are pretty good. We can tune CatBoost manually to improve CV and LB score. Note that CatBoost will automatically use both Kaggle T4 GPUs (when we add parameter `task_type='GPU'`)  for super fast training!","metadata":{}},{"cell_type":"code","source":"import catboost as cat, gc\nfrom catboost import CatBoostClassifier, Pool\nprint('CatBoost version',cat.__version__)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:31.956358Z","iopub.execute_input":"2024-01-14T05:12:31.956733Z","iopub.status.idle":"2024-01-14T05:12:33.488168Z","shell.execute_reply.started":"2024-01-14T05:12:31.956697Z","shell.execute_reply":"2024-01-14T05:12:33.487199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import KFold, GroupKFold\n\nall_oof = []\nall_true = []\nTARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):   \n    \n    print('#'*25)\n    print(f'### Fold {i+1}')\n    print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n    print('#'*25)\n\n    \n    model = CatBoostClassifier(task_type='GPU',\n                               loss_function='MultiClass',iterations= 30000,       \n                               learning_rate = 0.00013589, random_state= 42,\n\n\n)\n    \n    train_pool = Pool(\n        data = train.loc[train_index,FEATURES],\n        label = train.loc[train_index,'target'].map(TARS),\n    )\n    \n    valid_pool = Pool(\n        data = train.loc[valid_index,FEATURES],\n        label = train.loc[valid_index,'target'].map(TARS),\n    )\n    \n    model.fit(train_pool,\n             verbose=100,\n             eval_set=valid_pool,\n             )\n    model.save_model(f'CAT_v{VER}_f{i}.cat')\n    \n    oof = model.predict_proba(valid_pool)\n    all_oof.append(oof)\n    all_true.append(train.loc[valid_index, TARGETS].values)\n    \n    del train_pool, valid_pool, oof #model\n    gc.collect()\n    \n    #break\n    \nall_oof = np.concatenate(all_oof)\nall_true = np.concatenate(all_true)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:16:37.962790Z","iopub.execute_input":"2024-01-14T05:16:37.963708Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Feature Importance\nBelow we display the CatBoost top 25 feature importance for the last fold we trained.","metadata":{}},{"cell_type":"code","source":"TOP = 25\n\nfeature_importance = model.feature_importances_\nsorted_idx = np.argsort(feature_importance)\nfig = plt.figure(figsize=(10, 8))\nplt.barh(np.arange(len(sorted_idx))[-TOP:], feature_importance[sorted_idx][-TOP:], align='center')\nplt.yticks(np.arange(len(sorted_idx))[-TOP:], np.array(FEATURES)[sorted_idx][-TOP:])\nplt.title(f'Feature Importance - Top {TOP}')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.503686Z","iopub.status.idle":"2024-01-14T05:12:33.504063Z","shell.execute_reply.started":"2024-01-14T05:12:33.503878Z","shell.execute_reply":"2024-01-14T05:12:33.503897Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for CatBoost\nThis is CV score for our CatBoost model.","metadata":{}},{"cell_type":"code","source":"import sys\nsys.path.append('/kaggle/input/kaggle-kl-div')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(all_oof.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score KL-Div for CatBoost =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.505684Z","iopub.status.idle":"2024-01-14T05:12:33.506209Z","shell.execute_reply.started":"2024-01-14T05:12:33.505958Z","shell.execute_reply":"2024-01-14T05:12:33.505980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for Preds 1/6\nThis is CV score for Kaggle's sample submission.csv which uses equal predictions of 1/6 for all targets.","metadata":{}},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof.copy())\nfor c in oof.columns:\n    oof[c] = 1/6.\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score for \"Use Equal Preds 1/6\" =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.508086Z","iopub.status.idle":"2024-01-14T05:12:33.508515Z","shell.execute_reply.started":"2024-01-14T05:12:33.508293Z","shell.execute_reply":"2024-01-14T05:12:33.508313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CV Score for EEG_Id Means\nThis is CV score for current highest scoring public notebook [here][1] which uses train means as predictions.\n\n[1]: https://www.kaggle.com/code/seshurajup/eda-train-csv","metadata":{}},{"cell_type":"code","source":"all_oof2 = []\n\ngkf = GroupKFold(n_splits=10)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train, train.target, train.patient_id)):  \n    #print('#'*25)\n    #print(f'### Fold {i+1}')\n        \n    y_train = train.iloc[train_index][TARGETS].values\n    y_valid = train.iloc[valid_index][TARGETS].values\n    \n    #print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n    #print('#'*25)\n        \n    oof = y_valid.copy()\n    for j in range(6):\n        oof[:,j] = y_train[:,j].mean()\n    oof = oof / oof.sum(axis=1,keepdims=True)\n    all_oof2.append(oof)\n    \nall_oof2 = np.concatenate(all_oof2)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.510362Z","iopub.status.idle":"2024-01-14T05:12:33.510796Z","shell.execute_reply.started":"2024-01-14T05:12:33.510573Z","shell.execute_reply":"2024-01-14T05:12:33.510594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof = pd.DataFrame(all_oof2.copy())\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(all_true.copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score for \"Use Train Means\" =',cv)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.512070Z","iopub.status.idle":"2024-01-14T05:12:33.512541Z","shell.execute_reply.started":"2024-01-14T05:12:33.512308Z","shell.execute_reply":"2024-01-14T05:12:33.512330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Infer Test and Create Submission CSV\nBelow we use our 5 CatBoost fold models to infer the test data and create a `submission.csv` file.","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\nprint('Test shape',test.shape)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.514200Z","iopub.status.idle":"2024-01-14T05:12:33.514575Z","shell.execute_reply.started":"2024-01-14T05:12:33.514371Z","shell.execute_reply":"2024-01-14T05:12:33.514387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# FEATURE ENGINEER TEST\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\ndata = np.zeros((len(test),len(FEATURES)))\n    \nfor k in range(len(test)):\n    row = test.iloc[k]\n    s = int( row.spectrogram_id )\n    spec = pd.read_parquet(f'{PATH2}{s}.parquet')\n    \n    # 10 MINUTE WINDOW FEATURES\n    x = np.nanmean( spec.iloc[:,1:].values, axis=0)\n    data[k,:400] = x\n    x = np.nanmin( spec.iloc[:,1:].values, axis=0)\n    data[k,400:800] = x\n\n    # 20 SECOND WINDOW FEATURES\n    x = np.nanmean( spec.iloc[145:155,1:].values, axis=0)\n    data[k,800:1200] = x\n    x = np.nanmin( spec.iloc[145:155,1:].values, axis=0)\n    data[k,1200:1600] = x\n\ntest[FEATURES] = data\nprint('New test shape',test.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.516220Z","iopub.status.idle":"2024-01-14T05:12:33.516630Z","shell.execute_reply.started":"2024-01-14T05:12:33.516393Z","shell.execute_reply":"2024-01-14T05:12:33.516417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER CATBOOST ON TEST\npreds = []\n\nfor i in range(5):\n    print(i,', ',end='')\n    model = CatBoostClassifier(task_type='GPU',\n                               loss_function='MultiClass',iterations= 15000,       \n                               learning_rate = 0.0013589, random_state= 42,\n)\n    model.load_model(f'CAT_v{VER}_f{i}.cat')\n    \n    test_pool = Pool(\n        data = test[FEATURES]\n    )\n    \n    pred = model.predict_proba(test_pool)\n    preds.append(pred)\npred = np.mean(preds,axis=0)\nprint()\nprint('Test preds shape',pred.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.518368Z","iopub.status.idle":"2024-01-14T05:12:33.518831Z","shell.execute_reply.started":"2024-01-14T05:12:33.518578Z","shell.execute_reply":"2024-01-14T05:12:33.518599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.DataFrame({'eeg_id':test.eeg_id.values})\nsub[TARGETS] = pred\nsub.to_csv('submission.csv',index=False)\nprint('Submissionn shape',sub.shape)\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.520254Z","iopub.status.idle":"2024-01-14T05:12:33.520682Z","shell.execute_reply.started":"2024-01-14T05:12:33.520462Z","shell.execute_reply":"2024-01-14T05:12:33.520481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# SANITY CHECK TO CONFIRM PREDICTIONS SUM TO ONE\nsub.iloc[:,-6:].sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T05:12:33.522401Z","iopub.status.idle":"2024-01-14T05:12:33.522729Z","shell.execute_reply.started":"2024-01-14T05:12:33.522571Z","shell.execute_reply":"2024-01-14T05:12:33.522586Z"},"trusted":true},"execution_count":null,"outputs":[]}]}