{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782}],"dockerImageVersionId":30635,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\nfrom discussion - https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021 ( **Power of mean to prob leaderboard** )\n\nwith Equal probability 1/6: **LB 1.09**\n\nWith Overlapping **LB: 1.1**\n```python\nseizure_vote  0.196002,\ngpd_vote  0.156386,\nlrda_vote 0.155805,\nother_vote 0.17610,\ngrda_vote 0.17660,\nlpd_vote 0.139101\n```\n## Conclusion: Test set have means closer to Train Set\n\nwith non-overlapping based on spectrogram id\n**LB: 1.0**\n```python\nseizure_vote    0.174031\nlpd_vote        0.112700\ngpd_vote        0.090854\nlrda_vote       0.071484\ngrda_vote       0.136408\nother_vote      0.414523\n```\nwith non-overlapping based on eeg id\n**LB: 0.97**\n```python\nseizure_vote    0.152810\nlpd_vote        0.142456\ngpd_vote        0.104062\nlrda_vote       0.065407\ngrda_vote       0.114851\nother_vote      0.420414\n```\n\n## Conclusion: Test set have non-overlapping egg based sequences\nwith non-overlapping based on p id\n**LB: 1.28**\n```python\nseizure_vote 0.310718\nlpd_vote 0.046279\ngpd_vote 0.051885\nlrda_vote 0.081796\ngrda_vote 0.231471\nother_vote 0.277851\n```\n\n## Conclusion: Test set have multiple patient_id sequences\n\n# Overall : Test set have repeat patients with \"non overlapping eeg\".\n\n\n### **Most of the code copied from https://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-8#Create-Non-Overlapping-Eeg-Id-Train-Data upvode @cdeotte notebook**","metadata":{}},{"cell_type":"markdown","source":"## Brain activity notebook series\n\n### [EEGS 10–20 system](https://www.kaggle.com/code/seshurajup/eegs-10-20-system)\nBetter understanding eegs 10-20 system\n### [Missing Eeg_ids Train.csv vs train_eegs [Resolved]](https://www.kaggle.com/code/seshurajup/missing-eeg-ids-in-train-csv-vs-train-eegs-parquet)\nExtra training eggs [Resolved] as we can ignore it\n### [EDA train.csv](https://www.kaggle.com/code/seshurajup/eda-train-csv)\nDetailed analysis of the train.csv\n### [Eegs Pairing Analysis & Features](https://www.kaggle.com/code/seshurajup/eegs-pairing-analysis-features)\nPairing features analysis and build features\n### [Eegs Target Analysis - Correct way to merge target](https://www.kaggle.com/code/seshurajup/eegs-target-analysis-correct-way-to-merge-target)\nHow to choice the target votes for training\n### [Eegs Train Split (CV)](https://www.kaggle.com/seshurajup/eegs-train-splits-cv)\ngenerate better train split without patient_id overlap\n\n#### **Upvote my work if it is useful**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.model_selection import KFold, GroupKFold\nfrom sklearn.model_selection import StratifiedGroupKFold\n\nsns.set_style(\"whitegrid\")\n\ndf = pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/train.csv\")\nTARGETS = df.columns[-6:]\nstr(list(TARGETS))","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:39:48.197288Z","iopub.execute_input":"2024-01-14T06:39:48.197819Z","iopub.status.idle":"2024-01-14T06:39:50.066817Z","shell.execute_reply.started":"2024-01-14T06:39:48.197794Z","shell.execute_reply":"2024-01-14T06:39:50.065761Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n    {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first')\ntrain['patient_id'] = tmp\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum')\nfor t in TARGETS:\n    train[t] = tmp[t].values\n    \ny_data = train[TARGETS].values\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first')\ntrain['target'] = tmp\n\ntrain = train.reset_index()\nprint('Train non-overlapp eeg_id shape:', train.shape )\ntrain.head(5)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:39:50.068764Z","iopub.execute_input":"2024-01-14T06:39:50.069101Z","iopub.status.idle":"2024-01-14T06:39:50.186722Z","shell.execute_reply.started":"2024-01-14T06:39:50.069071Z","shell.execute_reply":"2024-01-14T06:39:50.185656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:39:54.263504Z","iopub.execute_input":"2024-01-14T06:39:54.263859Z","iopub.status.idle":"2024-01-14T06:39:54.270400Z","shell.execute_reply.started":"2024-01-14T06:39:54.263831Z","shell.execute_reply":"2024-01-14T06:39:54.268559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def calculate_balance_score(distribution):\n    flat_distribution = distribution.values.flatten()\n    mean_distribution = np.mean(flat_distribution)\n    balance_score = mean_absolute_error(flat_distribution, [mean_distribution] * len(flat_distribution))\n    return balance_score\nfrom sklearn.model_selection import StratifiedGroupKFold\nn_splits = 5\nsgkf = StratifiedGroupKFold(n_splits=n_splits, shuffle=True, random_state=2024)\nfor fold, (train_idx, val_idx) in enumerate(sgkf.split(train, train['target'], groups=train['patient_id'])):\n    train_fold = train.iloc[train_idx]\n    val_fold = train.iloc[val_idx]\n    train.loc[val_idx,'fold'] = fold\nfold_distributions = train.groupby('fold')['target'].value_counts(normalize=True).unstack().fillna(0)\nbalance_score = calculate_balance_score(fold_distributions)\nbalance_score","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:39:57.064937Z","iopub.execute_input":"2024-01-14T06:39:57.065317Z","iopub.status.idle":"2024-01-14T06:39:57.881664Z","shell.execute_reply.started":"2024-01-14T06:39:57.065287Z","shell.execute_reply":"2024-01-14T06:39:57.880301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"palette = sns.color_palette(\"husl\", n_colors=len(train['target'].unique()))\nn_splits = 5\nsgkf = GroupKFold(n_splits=n_splits)\nfor fold, (train_idx, val_idx) in enumerate(sgkf.split(train, train['target'], groups=train['patient_id'])):\n    train_fold = train.iloc[train_idx]\n    val_fold = train.iloc[val_idx]\n    train.loc[val_idx,'fold'] = fold\n    \nplt.figure(figsize=(14, 4))\noverall_target_order = train['target'].value_counts().sort_index().index\nsns.countplot(x='target', data=train, palette=palette, order=overall_target_order)\nplt.title('Overall Target Distribution')\nplt.show()\n\nfig, axes = plt.subplots(nrows=2, ncols=3, figsize=(15, 10))\naxes = axes.flatten()\nfor i in range(n_splits):\n    fold_data = train[train['fold'] == i]\n    if not fold_data.empty:\n        target_order = fold_data['target'].value_counts().sort_index().index\n        sns.countplot(x='target', data=fold_data, ax=axes[i], palette=palette, order=target_order)\n        axes[i].set_title(f'Fold {i} Target Distribution')\n    else:\n        axes[i].set_title('')\n        axes[i].set_visible(False)\nfor j in range(i + 1, len(axes)):\n    fig.delaxes(axes[j])\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:39:59.839335Z","iopub.execute_input":"2024-01-14T06:39:59.839745Z","iopub.status.idle":"2024-01-14T06:40:01.413047Z","shell.execute_reply.started":"2024-01-14T06:39:59.839710Z","shell.execute_reply":"2024-01-14T06:40:01.411903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_distribution = None\nbest_balance_score = float('inf')\nbest_iteration = -1\n\niterations = 100\nfor iteration in tqdm(range(iterations), total=iterations):\n    shuffled_patient_ids = train['patient_id'].sample(frac=1).reset_index(drop=True)\n    train['shuffled_patient_id'] = shuffled_patient_ids\n\n    for fold, (train_idx, val_idx) in enumerate(sgkf.split(train, train['target'], groups=train['shuffled_patient_id'])):\n        train.loc[val_idx, 'fold'] = fold\n\n    fold_distributions = train.groupby('fold')['target'].value_counts(normalize=True).unstack().fillna(0)\n    balance_score = calculate_balance_score(fold_distributions)\n\n    if balance_score < best_balance_score:\n        best_balance_score = balance_score\n        best_distribution = fold_distributions\n        best_iteration = iteration\n        print(\"Improved\", best_balance_score)\n\nbest_distribution, best_iteration, best_balance_score","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:40:19.918563Z","iopub.execute_input":"2024-01-14T06:40:19.918922Z","iopub.status.idle":"2024-01-14T06:40:21.726029Z","shell.execute_reply.started":"2024-01-14T06:40:19.918894Z","shell.execute_reply":"2024-01-14T06:40:21.725177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(14, 4))\noverall_target_order = train['target'].value_counts().sort_index().index\nsns.countplot(x='target', data=train, palette=palette, order=overall_target_order)\nplt.title('Target Distribution')\nplt.show()\n\nfig, axes = plt.subplots(nrows=2, ncols=3, figsize=(15, 10))\naxes = axes.flatten()\npalette = sns.color_palette(\"husl\", n_colors=len(train['target'].unique()))\nfor i in range(n_splits):\n    fold_data = train[train['fold'] == i]\n    if not fold_data.empty:\n        target_order = fold_data['target'].value_counts().sort_index().index\n        sns.countplot(x='target', data=fold_data, ax=axes[i], palette=palette, order=target_order)\n        axes[i].set_title(f'Fold {i} Target Distribution')\n    else:\n        axes[i].set_title(f'')\n        axes[i].set_visible(False)\nfor j in range(i + 1, len(axes)):\n    fig.delaxes(axes[j])\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:40:26.894833Z","iopub.execute_input":"2024-01-14T06:40:26.895214Z","iopub.status.idle":"2024-01-14T06:40:28.292547Z","shell.execute_reply.started":"2024-01-14T06:40:26.895184Z","shell.execute_reply":"2024-01-14T06:40:28.291446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"palette = sns.color_palette(\"husl\", n_colors=len(train['target'].unique()))\nn_splits = 5\nsgkf = GroupKFold(n_splits=n_splits)\nfor fold, (train_idx, val_idx) in enumerate(sgkf.split(train, train['target'], groups=train['patient_id'])):\n    train_fold = train.iloc[train_idx]\n    val_fold = train.iloc[val_idx]\n    train.loc[val_idx,'fold'] = fold\n    \nplt.figure(figsize=(14, 4))\noverall_target_order = train['target'].value_counts().sort_index().index\nsns.countplot(x='target', data=train, palette=palette, order=overall_target_order)\nplt.title('Overall Target Distribution')\nplt.show()\n\nfig, axes = plt.subplots(nrows=2, ncols=3, figsize=(15, 10))\naxes = axes.flatten()\nfor i in range(n_splits):\n    fold_data = train[train['fold'] == i]\n    if not fold_data.empty:\n        target_order = fold_data['target'].value_counts().sort_index().index\n        sns.countplot(x='target', data=fold_data, ax=axes[i], palette=palette, order=target_order)\n        axes[i].set_title(f'Fold {i} Target Distribution')\n    else:\n        axes[i].set_title('')\n        axes[i].set_visible(False)\nfor j in range(i + 1, len(axes)):\n    fig.delaxes(axes[j])\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:41:05.422600Z","iopub.execute_input":"2024-01-14T06:41:05.423064Z","iopub.status.idle":"2024-01-14T06:41:06.875085Z","shell.execute_reply.started":"2024-01-14T06:41:05.423027Z","shell.execute_reply":"2024-01-14T06:41:06.873979Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_distribution = None\nbest_balance_score = float('inf')\nbest_iteration = -1\n\niterations = len(train)*10\nfor iteration in tqdm(range(iterations), total=iterations):\n    shuffled_patient_ids = train['patient_id'].sample(frac=1).reset_index(drop=True)\n    train['shuffled_patient_id'] = shuffled_patient_ids\n\n    for fold, (train_idx, val_idx) in enumerate(sgkf.split(train, train['target'], groups=train['shuffled_patient_id'])):\n        train.loc[val_idx, 'fold'] = fold\n\n    fold_distributions = train.groupby('fold')['target'].value_counts(normalize=True).unstack().fillna(0)\n    balance_score = calculate_balance_score(fold_distributions)\n\n    if balance_score < best_balance_score:\n        best_balance_score = balance_score\n        best_distribution = fold_distributions\n        best_iteration = iteration\n        print(\"Improved\", best_balance_score)\n\nbest_distribution, best_iteration, best_balance_score","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:42:17.800165Z","iopub.execute_input":"2024-01-14T06:42:17.800501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_balance_score","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:41:38.957670Z","iopub.execute_input":"2024-01-14T06:41:38.958119Z","iopub.status.idle":"2024-01-14T06:41:38.964841Z","shell.execute_reply.started":"2024-01-14T06:41:38.958092Z","shell.execute_reply":"2024-01-14T06:41:38.963468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(14, 4))\noverall_target_order = train['target'].value_counts().sort_index().index\nsns.countplot(x='target', data=train, palette=palette, order=overall_target_order)\nplt.title('Target Distribution')\nplt.show()\n\nfig, axes = plt.subplots(nrows=2, ncols=3, figsize=(15, 10))\naxes = axes.flatten()\npalette = sns.color_palette(\"husl\", n_colors=len(train['target'].unique()))\nfor i in range(n_splits):\n    fold_data = train[train['fold'] == i]\n    if not fold_data.empty:\n        target_order = fold_data['target'].value_counts().sort_index().index\n        sns.countplot(x='target', data=fold_data, ax=axes[i], palette=palette, order=target_order)\n        axes[i].set_title(f'Fold {i} Target Distribution')\n    else:\n        axes[i].set_title(f'')\n        axes[i].set_visible(False)\nfor j in range(i + 1, len(axes)):\n    fig.delaxes(axes[j])\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:41:41.600096Z","iopub.execute_input":"2024-01-14T06:41:41.600446Z","iopub.status.idle":"2024-01-14T06:41:42.988163Z","shell.execute_reply.started":"2024-01-14T06:41:41.600420Z","shell.execute_reply":"2024-01-14T06:41:42.987051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Now split distributions much better ","metadata":{}},{"cell_type":"code","source":"train","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:41:49.637300Z","iopub.execute_input":"2024-01-14T06:41:49.637651Z","iopub.status.idle":"2024-01-14T06:41:49.663172Z","shell.execute_reply.started":"2024-01-14T06:41:49.637615Z","shell.execute_reply":"2024-01-14T06:41:49.662262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train['shuffled_patient_id']","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:42:01.033585Z","iopub.execute_input":"2024-01-14T06:42:01.033986Z","iopub.status.idle":"2024-01-14T06:42:01.039283Z","shell.execute_reply.started":"2024-01-14T06:42:01.033954Z","shell.execute_reply":"2024-01-14T06:42:01.038049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.to_csv(\"train_splits.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2024-01-14T06:42:02.422216Z","iopub.execute_input":"2024-01-14T06:42:02.422592Z","iopub.status.idle":"2024-01-14T06:42:02.526623Z","shell.execute_reply.started":"2024-01-14T06:42:02.422562Z","shell.execute_reply":"2024-01-14T06:42:02.525653Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Generally I use Stratified Group KFold in the following two situations:**\n\n**one target class is very rare and we want to make sure to include this target class in both train and valid (for each K fold split)\nthe test data has the same proportion of target classes as train data\nIn this competition, we are not sure if the test data has the same proportions (of 6 target classes) as train data. Therefore if we use Group KFold (instead of Stratified Group KFold) then we are evaluating how well our model can perform when the test data may have slightly different proportions as train data.**\n\n**If we use Stratified Group KFold, then we are optimizing a model to make predictions on test data that has the same proportion of target classes as train data. In conclusion, they will both work well, but perhaps Group KFold will produce a model that generalizes slightly better to an unknown test proportion.**\n\n- by [@cdeotte](https://www.kaggle.com/cdeotte) in [Discussion CatBoost Starter Notebook and Kaggle Dataset - LB 0.67 - CV 0.82](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467576#2600727)","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}