{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"}],"dockerImageVersionId":30635,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction and EDA","metadata":{}},{"cell_type":"markdown","source":"## 1. Introduction \n\n### 1.1 Quick Summary \n\n**Goal:** Classification of seizures + other harmful brain activity <br>\n**Data:** electroencephalography (EEG) signals <br>\n**Submission Date:** 19th of April ","metadata":{}},{"cell_type":"markdown","source":"### 1.2 What is electroencephalography (EEG)? \n\nAn electroencephalogram (EEG) is a test that measures electrical activity in the brain using small, metal discs (electrodes) attached to the scalp. Brain cells communicate via electrical impulses and are active all the time, even during asleep. This activity shows up as wavy lines on an EEG recording. <br> <br>\n\n<img src=\"https://assets-global.website-files.com/621e95f9ac30687a56e4297e/64a8d6171e48d618d6eb1f61_V2_1677770554425_847b3a5e-24b7-4152-ab9b-01ef841697d5.png\" alt=\"level of mesuarements\" width=\"720\"/>","metadata":{}},{"cell_type":"markdown","source":"### 1.3 Which patterns are there? \n\n**patterns:** \n- seizure (SZ)\n- generalized periodic discharges (GPD)\n- lateralized periodic discharges (LPD)\n- lateralized rhythmic delta activity (LRDA)\n- generalized rhythmic delta activity (GRDA)\n\n**weighting:**\n\n- idealized &rarr; clear expert vote\n- proto &rarr; half of experts for one pattern\n- egde &rarr; 50/50 vote between two patterns ","metadata":{}},{"cell_type":"markdown","source":"### 1.4 Evaluation metrics\n\n**Kullbeck Liebler Divergence** <br>\n\n$D_{KL}(P \\parallel Q) = \\sum_{i} P(i) \\log\\left(\\frac{P(i)}{Q(i)}\\right)$\n\n&rarr; measures the difference between two distributions \n- log is used get the same divergence for $\\frac{1}{x}$ and $x$\n- $P(i)$ is used as weighting to set a focus an cases we will find more often","metadata":{}},{"cell_type":"markdown","source":"### 1.5 Target Output \n&rarr; as csv file in the following format \n\n|eeg_id|seizure_vote|lpd_vote|gpd_vote|lrda_vote|grda_vote|other_vote|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 0 | 0.166 | 0.166 | 0.167 | 0.167 | 0.167 | 0.167 |\n| 1 | 0.166 | 0.166 | 0.167 | 0.167 | 0.167 | 0.167 |\n| ... | ... | ... | ... | ... | ... | ... |","metadata":{}},{"cell_type":"markdown","source":"## 2. EDA\n\n### 2.1 train.csv  \n\n#### 2.1.1 Data Import","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np \nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\ndf_train = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\ndf_train","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:46.119852Z","iopub.execute_input":"2024-02-19T14:45:46.120424Z","iopub.status.idle":"2024-02-19T14:45:48.086890Z","shell.execute_reply.started":"2024-02-19T14:45:46.120389Z","shell.execute_reply":"2024-02-19T14:45:48.085970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.1.2 Data Description","metadata":{}},{"cell_type":"code","source":"df_train.describe().T","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:48.088774Z","iopub.execute_input":"2024-02-19T14:45:48.089314Z","iopub.status.idle":"2024-02-19T14:45:48.243431Z","shell.execute_reply.started":"2024-02-19T14:45:48.089280Z","shell.execute_reply":"2024-02-19T14:45:48.242539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"| Column Name                  | Description                                                         |\n|------------------------------|---------------------------------------------------------------------|\n| eeg_id                       | A unique identifier for the entire EEG recording.                   |\n| eeg_sub_id                   | An ID for the specific 50-second long subsample this row's labels apply to. |\n| eeg_label_offset_seconds     | The time between the beginning of the consolidated EEG and this subsample. |\n| spectrogram_id               | A unique identifier for the entire EEG recording.                   |\n| spectrogram_sub_id           | An ID for the specific 10-minute subsample this row's labels apply to. |\n| spectogram_label_offset_seconds | The time between the beginning of the consolidated spectrogram and this subsample. |\n| label_id                     | An ID for this set of labels.                                       |\n| patient_id                   | An ID for the patient who donated the data.                         |\n| expert_consensus             | The consensus annotator label. Provided for convenience only.       |\n\n\nthe rest are the categories to predict ","metadata":{}},{"cell_type":"code","source":"df_train.info()","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:48.244818Z","iopub.execute_input":"2024-02-19T14:45:48.245372Z","iopub.status.idle":"2024-02-19T14:45:48.278159Z","shell.execute_reply.started":"2024-02-19T14:45:48.245336Z","shell.execute_reply":"2024-02-19T14:45:48.276500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.1.3 train.csv EDA ","metadata":{}},{"cell_type":"code","source":"print('There are ', df_train['eeg_id'].nunique(), ' eeg data entries')\nprint('There are ', df_train['spectrogram_id'].nunique(), ' spectrogram data entries')\nprint('There are ', df_train['patient_id'].nunique(), ' different patients')","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:48.280495Z","iopub.execute_input":"2024-02-19T14:45:48.281602Z","iopub.status.idle":"2024-02-19T14:45:48.294069Z","shell.execute_reply.started":"2024-02-19T14:45:48.281555Z","shell.execute_reply":"2024-02-19T14:45:48.292985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"columns_to_plot = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\n\ndf_normalized = df_train[columns_to_plot].div(df_train[columns_to_plot].sum(axis=1), axis=0)\n\ncolumn_means = df_normalized.mean()\n\ndata_for_barplot = pd.DataFrame({'Columns': column_means.index, 'Mean Percentage': column_means.values})\n\nsns.set(style=\"whitegrid\")\n\nplt.figure(figsize=(10, 6))\nsns.barplot(x='Columns', y='Mean Percentage', data=data_for_barplot, palette=\"viridis\")\n\nplt.title('Comparison of Mean Percentages for Each Normalized Expert Vote')\nplt.xlabel('Columns')\nplt.ylabel('Mean Percentage')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:48.295294Z","iopub.execute_input":"2024-02-19T14:45:48.295761Z","iopub.status.idle":"2024-02-19T14:45:48.657743Z","shell.execute_reply.started":"2024-02-19T14:45:48.295730Z","shell.execute_reply":"2024-02-19T14:45:48.656908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(14, 16))\nfor i, column in enumerate(columns_to_plot, 1):\n    plt.subplot(3, 2, i)\n    sns.violinplot(x=df_train[column])\n    plt.title(f'Violin Plot for {column}')\n\nplt.suptitle('Violin Plots for Absolute Values of Each Column', y=1.02)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:48.659105Z","iopub.execute_input":"2024-02-19T14:45:48.659605Z","iopub.status.idle":"2024-02-19T14:45:51.981472Z","shell.execute_reply.started":"2024-02-19T14:45:48.659573Z","shell.execute_reply":"2024-02-19T14:45:51.980558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(4, 3))\ncolumn = 'eeg_label_offset_seconds'\nsns.violinplot(x=df_train[column])\n\nplt.title('The time between the beginning of the consolidated EEG and this subsample')\nplt.suptitle('EEG Offset Seconds Distribution', y=1.02)\nplt.tight_layout()\nplt.show()\n\ndf_train[[column]].describe().T\n","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:51.982601Z","iopub.execute_input":"2024-02-19T14:45:51.983121Z","iopub.status.idle":"2024-02-19T14:45:52.482418Z","shell.execute_reply.started":"2024-02-19T14:45:51.983087Z","shell.execute_reply":"2024-02-19T14:45:52.481435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(4, 3))\ncolumn = 'spectrogram_label_offset_seconds'\nsns.violinplot(x=df_train[column])\n\nplt.title('The time between the beginning of the consolidated spectrogram and this subsample')\nplt.suptitle('Spectrogram Offset Seconds Distribution', y=1.02)\nplt.tight_layout()\nplt.show()\n\ndf_train[[column]].describe().T","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:52.483699Z","iopub.execute_input":"2024-02-19T14:45:52.483993Z","iopub.status.idle":"2024-02-19T14:45:52.992607Z","shell.execute_reply.started":"2024-02-19T14:45:52.483967Z","shell.execute_reply":"2024-02-19T14:45:52.991521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(6, 5))\nsns.heatmap(df_train[columns_to_plot].corr(), annot=True, cmap='coolwarm', fmt=\".2f\", linewidths=.5)\n\nplt.title('correlation matrix for possible patterns')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:52.993977Z","iopub.execute_input":"2024-02-19T14:45:52.994328Z","iopub.status.idle":"2024-02-19T14:45:53.380140Z","shell.execute_reply.started":"2024-02-19T14:45:52.994298Z","shell.execute_reply":"2024-02-19T14:45:53.378734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.2 train_eegs\n\n#### 2.2.1 Data Import","metadata":{}},{"cell_type":"code","source":"df_eeg = pd.read_parquet('/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/1628180742.parquet')\ndf_eeg","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:53.383802Z","iopub.execute_input":"2024-02-19T14:45:53.384190Z","iopub.status.idle":"2024-02-19T14:45:53.551577Z","shell.execute_reply.started":"2024-02-19T14:45:53.384153Z","shell.execute_reply":"2024-02-19T14:45:53.550390Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.2.2 Data Description","metadata":{}},{"cell_type":"markdown","source":"**corresponding meta date from train.csv**","metadata":{}},{"cell_type":"code","source":"df_train[df_train.eeg_id == 1628180742]","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:53.553869Z","iopub.execute_input":"2024-02-19T14:45:53.554823Z","iopub.status.idle":"2024-02-19T14:45:53.575252Z","shell.execute_reply.started":"2024-02-19T14:45:53.554786Z","shell.execute_reply":"2024-02-19T14:45:53.574156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**how to map the eeg_sub_id** <br>\n\nUsually the eeg data for an eeg_id contains 10k rows because we have 50 seconds x 200 records per second. If we have multiple entries for our eeg_id in the metadate, it will have eeg_sub_ids. Next to the sub_id you can find the time (in seconds) when this subrecord starts. If we want to find the starting row, we have to use the mentioned formula minus one because the index starts with 0.\n\nSo let's imagine our last sub record starts after 20 seconds. To find the starting row, we multiply 20 by 200 and subtract one. So we know our first line for this sub id is 3999. Our last row for this sub_id is always 10k lines later which means row 13999.\n\nFor our case we will reduce the eeg data to the first subsample (only first 50 seconds)","metadata":{}},{"cell_type":"code","source":"df_eeg = df_eeg.head(10000)\n\ndf_eeg.describe().T","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:53.576563Z","iopub.execute_input":"2024-02-19T14:45:53.576883Z","iopub.status.idle":"2024-02-19T14:45:53.630885Z","shell.execute_reply.started":"2024-02-19T14:45:53.576856Z","shell.execute_reply":"2024-02-19T14:45:53.629471Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Based on 10-20 System**\n\nThe 10–20 system, also known as the International 10–20 system, is a standardized method for placing scalp electrodes during EEG exams, polysomnograph sleep studies, or lab research. It ensures consistency in testing, allowing for reproducibility and effective analysis. Electrodes are positioned based on their relationship to underlying areas of the brain, particularly the cerebral cortex. The system derives its name from the fact that electrode distances are either 10% or 20% of the total front–back or right–left distance of the skull. This method facilitates the detection of distinct electrical patterns in the brain during sleep and wake cycles. Various extrinsic factors can influence these patterns, such as age, medication, health conditions, neurological history, and substance use. The measurements are taken from specific anatomical locations, like the tragus, auricle, and mastoid, and are crucial for consistent and comparable results in scientific studies.\n\n<img src=\"https://info.tmsi.com/hs-fs/hubfs/Blogs/0.1%20The%2010-20%20System/the-10-20-system-1-new.jpg?width=1307&height=689&name=the-10-20-system-1-new.jpg\" alt=\"level of mesuarements\" width=\"720\"/>\n\n\nThe EKG column is for an electrocardiogram lead that records data from the heart.\n\n<img src=\"https://my.clevelandclinic.org/-/scassets/images/org/health/articles/16953-electrocardiogram\" alt=\"level of mesuarements\" width=\"720\"/>","metadata":{}},{"cell_type":"code","source":"columns = df_eeg.columns\n\nplt.figure(figsize=(14, 30))\nfor i, column in enumerate(columns, 1):\n    plt.subplot(20, 1, i)\n    sns.violinplot(x=df_eeg[column])\n    plt.title(f'Violin Plot for {column}')\n    plt.xlim(-1600, 250)\n\nplt.suptitle('Violin Plots for the electrode values per position', y=1.02)\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:53.632517Z","iopub.execute_input":"2024-02-19T14:45:53.632801Z","iopub.status.idle":"2024-02-19T14:45:58.557148Z","shell.execute_reply.started":"2024-02-19T14:45:53.632774Z","shell.execute_reply":"2024-02-19T14:45:58.555957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(14, 12))\nsns.heatmap(df_eeg.corr(), annot=True, cmap='coolwarm', fmt=\".2f\", linewidths=.5)\n\nplt.title('Correlation Matrix')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:45:58.558212Z","iopub.execute_input":"2024-02-19T14:45:58.558756Z","iopub.status.idle":"2024-02-19T14:45:59.749287Z","shell.execute_reply.started":"2024-02-19T14:45:58.558728Z","shell.execute_reply":"2024-02-19T14:45:59.748171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### 2.3. train_spectorgrams \n\n#### 2.3.1 Data Import\n\n- it will be the corresponding spectrogram to the eeg data before ","metadata":{}},{"cell_type":"code","source":"spectrogram = pd.read_parquet('/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/353733.parquet')\nspectrogram","metadata":{"execution":{"iopub.status.busy":"2024-02-19T14:48:17.974101Z","iopub.execute_input":"2024-02-19T14:48:17.974450Z","iopub.status.idle":"2024-02-19T14:48:18.046679Z","shell.execute_reply.started":"2024-02-19T14:48:17.974420Z","shell.execute_reply":"2024-02-19T14:48:18.045485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.3.2 Data Explanation\n\n### Column Names Description:\n- `LL_x`: Frequency (Hz) data from left lateral region\n- `RL_x`: Frequency (Hz) data from right lateral region\n- `LP_x`: Frequency (Hz) data from left parasagittal region\n- `RP_x`: Frequency (Hz) data from right parasagittal region\n\n**the connection to the metadate i based on a similar conecpt**","metadata":{}},{"cell_type":"code","source":"spectrogram.describe().T","metadata":{"execution":{"iopub.status.busy":"2024-02-19T15:20:19.054202Z","iopub.execute_input":"2024-02-19T15:20:19.054592Z","iopub.status.idle":"2024-02-19T15:20:19.966641Z","shell.execute_reply.started":"2024-02-19T15:20:19.054561Z","shell.execute_reply":"2024-02-19T15:20:19.965528Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### 2.3.3 Data Analysis","metadata":{}},{"cell_type":"code","source":"def plot_spectrogram(spectrogram_path):\n    '''\n    source --> https://www.kaggle.com/code/clehmann10/plot-spectrograms\n    '''\n    \n    sample_spect = pd.read_parquet(spectrogram_path)\n    \n    split_spect = {\n        \"LL\": sample_spect.filter(regex='^LL', axis=1),\n        \"RL\": sample_spect.filter(regex='^RL', axis=1),\n        \"RP\": sample_spect.filter(regex='^RP', axis=1),\n        \"LP\": sample_spect.filter(regex='^LP', axis=1),\n    }\n    \n    fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(15, 12))\n    axes = axes.flatten()\n    label_interval = 5\n    for i, split_name in enumerate(split_spect.keys()):\n        ax = axes[i]\n        img = ax.imshow(np.log(split_spect[split_name]).T, cmap='viridis', aspect='auto', origin='lower')  # You can choose any colormap (cmap) that suits your preferences\n        cbar = fig.colorbar(img, ax=ax)\n        cbar.set_label('Log(Value)')\n        ax.set_title(split_name)\n        ax.set_ylabel(\"Frequency (Hz)\")\n        ax.set_xlabel(\"Time\")\n\n        ax.set_yticks(np.arange(len(split_spect[split_name].columns)))\n        ax.set_yticklabels([column_name[3:] for column_name in split_spect[split_name].columns])\n        frequencies = [column_name[3:] for column_name in split_spect[split_name].columns]\n        ax.set_yticks(np.arange(0, len(split_spect[split_name].columns), label_interval))\n        ax.set_yticklabels(frequencies[::label_interval])\n    plt.tight_layout()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-02-19T15:00:31.771918Z","iopub.execute_input":"2024-02-19T15:00:31.772277Z","iopub.status.idle":"2024-02-19T15:00:31.783720Z","shell.execute_reply.started":"2024-02-19T15:00:31.772244Z","shell.execute_reply":"2024-02-19T15:00:31.782481Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_spectrogram('/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/353733.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-02-19T15:00:56.402978Z","iopub.execute_input":"2024-02-19T15:00:56.403402Z","iopub.status.idle":"2024-02-19T15:00:59.425937Z","shell.execute_reply.started":"2024-02-19T15:00:56.403368Z","shell.execute_reply":"2024-02-19T15:00:59.424904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}