{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782}],"dockerImageVersionId":30635,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#
1 | INTRODUCTION
\n\n👋 Welcome to \"🧠Exploring EEG: A Beginner's Guide\"! \n\nIf you're fascinated by the wonders of the human brain and the intricate patterns of brainwaves, but find the world of Electroencephalography (EEG) analysis daunting, you're in the right place. \n\nThis notebook is designed for beginners like me & you, aiming to demystify the complexities of EEG data and make your learning journey both enjoyable and informative.\n\n### 1.1 | Intention of the notebook\nIn this notebook, we will embark on an exploratory journey into the realm of EEG data analysis. Our goal is to provide a clear, step-by-step guide to understanding and analyzing EEG signals, which are crucial in detecting and classifying brain activities, such as seizures. We aim to:\n\n* Break down complex concepts into easily digestible sections.\n* Illustrate each step with practical code examples.\n* Reference public notebooks and discussions to enhance your learning experience.\n\n\n### 1.2 | Learning Objective\nBy the end of this notebook, you will have a foundational understanding of:\n\n* The basics of EEG signals and their significance in medical research and neurology.\n* How to preprocess and analyze EEG data.\n* Run through the basic code to build a machine learning model for EEG data classification.","metadata":{}},{"cell_type":"markdown","source":"#
2 | REFERENCE & ACKNOWLEDGEMENT
\n\nThis notebook wouldn't be possible without the valuable insights and contributions from the Kaggle community. I've leveraged several resources to compile the most effective learning path for us:\n\n* https://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-8\n* https://www.kaggle.com/code/mvvppp/hms-eda-and-domain-journey\n* https://www.kaggle.com/code/ksooklall/hms-banana-montage\n* https://www.kaggle.com/code/mpwolke/seizures-classification-parquet\n\n\nFeel free to explore these resources alongside this notebook to deepen your understanding.","metadata":{}},{"cell_type":"markdown","source":"#
3 | LOAD LIBARIES
","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd, numpy as np\nfrom glob import glob\nimport matplotlib.pyplot as plt\nVER = 1","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:39:13.560653Z","iopub.execute_input":"2024-01-17T13:39:13.561071Z","iopub.status.idle":"2024-01-17T13:39:13.954701Z","shell.execute_reply.started":"2024-01-17T13:39:13.561035Z","shell.execute_reply":"2024-01-17T13:39:13.953802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
4 | INTRODUCTION TO EEG AND SEIZURE DETECTION
\n\n 4.1 | Electroencephalography (EEG) - The Window into Brain Activity\n\n* Electroencephalography, commonly known as EEG, is a non-invasive method used by medical professionals to record electrical activity in the brain. \n* This is done using electrodes placed along the scalp. \n* EEG is a crucial tool in diagnosing neurological disorders, especially epilepsy, which is characterized by recurrent seizures.\n\n\"EEG\"\n\n","metadata":{"execution":{"iopub.status.busy":"2024-01-14T15:14:51.032997Z","iopub.execute_input":"2024-01-14T15:14:51.033378Z","iopub.status.idle":"2024-01-14T15:14:51.038874Z","shell.execute_reply.started":"2024-01-14T15:14:51.033348Z","shell.execute_reply":"2024-01-14T15:14:51.037454Z"}}},{"cell_type":"code","source":"# check the reading of one parquet for understanding\n\nBASE_PATH = '/kaggle/input/hms-harmful-brain-activity-classification/'\n\ndf = pd.DataFrame({'path': glob(BASE_PATH + '**/*.parquet')})\ndf['test_type'] = df['path'].str.split('/').str.get(-2).str.split('_').str.get(-1)\ndf['id'] = df['path'].str.split('/').str.get(-1).str.split('.').str.get(0)\n\ndf_eeg = pd.read_parquet(BASE_PATH + 'train_eegs/1000913311.parquet')\ndf_eeg.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:39:13.956738Z","iopub.execute_input":"2024-01-17T13:39:13.957752Z","iopub.status.idle":"2024-01-17T13:39:15.156288Z","shell.execute_reply.started":"2024-01-17T13:39:13.957708Z","shell.execute_reply":"2024-01-17T13:39:15.155182Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Determine the number of channels\n# Assuming each row is a time point and each column is a channel\nn_channels = df_eeg.shape[1]\nn_channels","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:39:15.157718Z","iopub.execute_input":"2024-01-17T13:39:15.158038Z","iopub.status.idle":"2024-01-17T13:39:15.164204Z","shell.execute_reply.started":"2024-01-17T13:39:15.158010Z","shell.execute_reply":"2024-01-17T13:39:15.163225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The headers in the dataset (Fp1, F3, C3, P3, F7, T3, T5, O1, Fz, Cz, Pz, Fp2, F4, C4, P4, F8, T4, T6, O2, EKG) are standard electrode placement labels used in electroencephalography (EEG). \n* These labels correspond to specific positions on the scalp where EEG electrodes are placed to record brain activity. \n* Here's a brief overview of what they represent:\n\n1. **Fp1, Fp2:** Frontopolar electrodes, located on the forehead, left and right side.\n2. **F3, F4:** Frontal electrodes, on the left and right side of the forehead.\n3. **C3, C4:** Central electrodes, placed above the left and right hemispheres of the brain.\n4. **P3, P4:** Parietal electrodes, located on the upper back portion of the head, left and right sides.\n5. **O1, O2:** Occipital electrodes, positioned at the back of the head near the visual cortex.\n6. **T3, T4, T5, T6:** Temporal electrodes, situated on the left and right sides of the head near the ears. They are often involved in monitoring auditory functions.\n7. **F7, F8:** Frontal-temporal electrodes, located at the front of the temporal lobes.\n8. **Fz, Cz, Pz:** Midline electrodes, located at the frontal (Fz), central (Cz), and parietal (Pz) positions on the midline of the head.\n9. **EKG:** Electrocardiogram electrode, which records the heart’s electrical activity. It's not directly related to brain activity but can be important in some EEG analyses.\n\n\n\"10-20-system-of-electrode-placement\"","metadata":{}},{"cell_type":"markdown","source":" 4.2 | Seizures and Their Impact\n* Seizures are sudden, uncontrolled electrical disturbances in the brain that can cause changes in behavior, feelings, movements, and levels of consciousness. \n* Detecting and classifying seizures accurately is vital for appropriate treatment and care, especially in critically ill patients.\n\n 4.3 | The Challenge of Manual EEG Analysis\n\n* Traditionally, EEG data analysis relies on visual inspection by trained neurologists. \n* This process is not only time-consuming and labor-intensive but also prone to errors due to fatigue and subjective interpretation.\n\n\"Manual\nSource: Automated Identification of Abnormal Adult EEG, S. López, G. Suarez, D. Jungreis, I. Obeid and J. Picone, Neural Engineering Data Consortium, Temple University\n","metadata":{}},{"cell_type":"markdown","source":" 4.4 | The Role of Data Science in EEG Analysis\n\n* Automating EEG Interpretation\nThe advent of machine learning and data science offers an opportunity to automate the interpretation of EEG data. By developing algorithms that can detect and classify different patterns in EEG signals, we can aid neurologists in making faster, more accurate diagnoses.\n\n* The Data Science Approach\nData scientists approach this challenge by first preprocessing the EEG data, which involves filtering out noise and extracting relevant features. Machine learning models are then trained on these features to distinguish between different types of brain activity.\n\n\"flowchart\n","metadata":{}},{"cell_type":"markdown","source":" 4.5 | Understanding EEG Patterns\n\nIn the realm of EEG analysis for seizure detection, certain patterns are of particular interest:\n\n1. **Seizure (SZ):** Characterized by abnormal rhythmic activity, indicative of a seizure.\n2. **Generalized Periodic Discharges (GPD):** Patterns that may be seen in various encephalopathies.\n3. **Lateralized Periodic Discharges (LPD):** Often associated with focal brain lesions.\n4. **Lateralized Rhythmic Delta Activity (LRDA):** Can be observed in focal brain dysfunction.\n5. **Generalized Rhythmic Delta Activity (GRDA):** Typically related to diffuse brain dysfunction.\n6. **\"Other\" Patterns:** Any other type of activity not falling into the above categories.\n\n 4.6 | Interpreting Complex EEG Data\n\nEEG data interpretation can be complex, especially in edge cases where expert neurologists may not agree on a classification. This is where machine learning models can particularly shine by providing an additional layer of analysis.\n\n\"flowchart\n\nSource: Development of Expert-Level Classification of Seizures and Rhythmic and Periodic Patterns During EEG Interpretation https://www.neurology.org/doi/10.1212/WNL.0000000000207127\n","metadata":{}},{"cell_type":"markdown","source":"#
5 | LOAD TRAIN DATA
","metadata":{"execution":{"iopub.status.busy":"2024-01-14T15:16:07.035029Z","iopub.execute_input":"2024-01-14T15:16:07.035699Z","iopub.status.idle":"2024-01-14T15:16:07.040799Z","shell.execute_reply.started":"2024-01-14T15:16:07.035656Z","shell.execute_reply":"2024-01-14T15:16:07.039623Z"}}},{"cell_type":"code","source":"df = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\nTARGETS = df.columns[-6:]\nprint('Train shape:', df.shape )\nprint('Targets', list(TARGETS))\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:39:15.166320Z","iopub.execute_input":"2024-01-17T13:39:15.166645Z","iopub.status.idle":"2024-01-17T13:39:15.418459Z","shell.execute_reply.started":"2024-01-17T13:39:15.166616Z","shell.execute_reply":"2024-01-17T13:39:15.417378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
6 | CREATE NON-OVERLAPPING EEG ID TRAIN DATA
\n\nFollowing the notebook from Chris Deotte: https://www.kaggle.com/code/cdeotte/catboost-starter-lb-0-8,\nInitial discussion found here https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467021\n\nWe perform the following because:\n\n* **Match Training Data with Test Data Format:** The competition states that the test data does not have multiple segments from the same eeg_id. To make the training data similar to the test data, we also use only one segment per eeg_id in the training data.\n\n* **Remove Redundancies:** This approach ensures that the training data does not have overlapping or redundant information, which can lead to a more accurate and generalizable machine learning model.\n\n* **Consistency in Data:** By standardizing how we handle the EEG segments in training, we ensure that our model learns from data that is consistent in format with the data it will be tested on.\n\n* **Data Preparation for Machine Learning:** The normalization of target variables and inclusion of relevant features like patient_id and expert_consensus prepare the dataset for effective machine learning modeling.","metadata":{"execution":{"iopub.status.busy":"2024-01-14T15:17:26.898038Z","iopub.execute_input":"2024-01-14T15:17:26.898867Z","iopub.status.idle":"2024-01-14T15:17:26.903917Z","shell.execute_reply.started":"2024-01-14T15:17:26.898818Z","shell.execute_reply":"2024-01-14T15:17:26.902565Z"}}},{"cell_type":"code","source":"# Creating a Unique EEG Segment per eeg_id:\n# The code groups (groupby) the EEG data (df) by eeg_id. Each eeg_id represents a different EEG recording.\n# It then picks the first spectrogram_id and the earliest (min) spectrogram_label_offset_seconds for each eeg_id. This helps in identifying the starting point of each EEG segment.\n# The resulting DataFrame train has columns spec_id (first spectrogram_id) and min (earliest spectrogram_label_offset_seconds).\ntrain = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n {'spectrogram_id':'first','spectrogram_label_offset_seconds':'min'})\ntrain.columns = ['spec_id','min']\n\n\n# Finding the Latest Point in Each EEG Segment:\n# The code again groups the data by eeg_id and finds the latest (max) spectrogram_label_offset_seconds for each segment.\n# This max value is added to the train DataFrame, representing the end point of each EEG segment.\ntmp = df.groupby('eeg_id')[['spectrogram_id','spectrogram_label_offset_seconds']].agg(\n {'spectrogram_label_offset_seconds':'max'})\ntrain['max'] = tmp\n\n\ntmp = df.groupby('eeg_id')[['patient_id']].agg('first') # The code adds the patient_id for each eeg_id to the train DataFrame. This links each EEG segment to a specific patient.\ntrain['patient_id'] = tmp\n\n\ntmp = df.groupby('eeg_id')[TARGETS].agg('sum') # The code sums up the target variable counts (like votes for seizure, LPD, etc.) for each eeg_id.\nfor t in TARGETS:\n train[t] = tmp[t].values\n \ny_data = train[TARGETS].values # It then normalizes these counts so that they sum up to 1. This step converts the counts into probabilities, which is a common practice in classification tasks.\ny_data = y_data / y_data.sum(axis=1,keepdims=True)\ntrain[TARGETS] = y_data\n\ntmp = df.groupby('eeg_id')[['expert_consensus']].agg('first') # For each eeg_id, the code includes the expert_consensus on the EEG segment's classification.\ntrain['target'] = tmp\n\ntrain = train.reset_index() # This makes eeg_id a regular column, making the DataFrame easier to work with.\nprint('Train non-overlapp eeg_id shape:', train.shape )\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:39:15.419607Z","iopub.execute_input":"2024-01-17T13:39:15.419990Z","iopub.status.idle":"2024-01-17T13:39:15.515608Z","shell.execute_reply.started":"2024-01-17T13:39:15.419957Z","shell.execute_reply":"2024-01-17T13:39:15.514644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
7 | FEATURE ENGINEERING
\n\n","metadata":{}},{"cell_type":"markdown","source":" 7.1 | 10 min and 20 sec windows\n\n* The code belows efficiently reads spectrogram data, from a single combined file, based on the set variable. We relied on the dataset by Chris Deotte to save time. https://www.kaggle.com/datasets/cdeotte/brain-spectrograms\n* It then performs feature engineering by calculating mean and minimum values over two different time windows for each frequency in the spectrogram.\nIt produce produces in 1600 features (400 features × 4 calculations) for each EEG ID.\n* The new features are intended to help the model better understand and classify the EEG data.\n* This approach is designed to enhance the model's performance by providing it with more detailed information derived from the spectrogram data.","metadata":{}},{"cell_type":"code","source":"READ_SPEC_FILES = False # If READ_SPEC_FILES is False, the code reads the combined file instead of individual files.\nFEATURE_ENGINEER = True","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:39:15.516627Z","iopub.execute_input":"2024-01-17T13:39:15.516955Z","iopub.status.idle":"2024-01-17T13:39:15.521390Z","shell.execute_reply.started":"2024-01-17T13:39:15.516918Z","shell.execute_reply":"2024-01-17T13:39:15.520335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# READ ALL SPECTROGRAMS\nPATH = '/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/'\nfiles = os.listdir(PATH)\nprint(f'There are {len(files)} spectrogram parquets')\n\nif READ_SPEC_FILES: \n spectrograms = {}\n for i,f in enumerate(files):\n if i%100==0: print(i,', ',end='')\n tmp = pd.read_parquet(f'{PATH}{f}')\n name = int(f.split('.')[0])\n spectrograms[name] = tmp.iloc[:,1:].values\nelse:\n spectrograms = np.load('/kaggle/input/brain-spectrograms/specs.npy',allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:39:15.522641Z","iopub.execute_input":"2024-01-17T13:39:15.523043Z","iopub.status.idle":"2024-01-17T13:40:17.405646Z","shell.execute_reply.started":"2024-01-17T13:39:15.523005Z","shell.execute_reply":"2024-01-17T13:40:17.404521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%time\n# ENGINEER FEATURES\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# The code generates features from the spectrogram data for use in a model \n# The features are derived by calculating the mean and minimum values over time for each of the 400 spectrogram frequencies.\n# Two types of windows are used for these calculations:\n# A 10-minute window (_mean_10m, _min_10m).\n# A 20-second window (_mean_20s, _min_20s).\n# This process results in 1600 features (400 features × 4 calculations) for each EEG ID.\n\nSPEC_COLS = pd.read_parquet(f'{PATH}1000086677.parquet').columns[1:]\nFEATURES = [f'{c}_mean_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_10m' for c in SPEC_COLS]\nFEATURES += [f'{c}_mean_20s' for c in SPEC_COLS]\nFEATURES += [f'{c}_min_20s' for c in SPEC_COLS]\nprint(f'We are creating {len(FEATURES)} features for {len(train)} rows... ',end='')\n\n\n# A data matrix data is initialized to store the new features for each eeg_id in the train DataFrame.\n# For each row in train, the code calculates the mean and minimum values within the specified 10-minute and 20-second windows.\n# These calculated values are then stored in the data matrix.\n# Finally, the matrix is added to the train DataFrame as new columns.\n\nif FEATURE_ENGINEER:\n data = np.zeros((len(train),len(FEATURES)))\n for k in range(len(train)):\n if k%100==0: print(k,', ',end='')\n row = train.iloc[k]\n r = int( (row['min'] + row['max'])//4 ) \n \n # 10 MINUTE WINDOW FEATURES (MEANS and MINS)\n x = np.nanmean(spectrograms[row.spec_id][r:r+300,:],axis=0)\n data[k,:400] = x\n x = np.nanmin(spectrograms[row.spec_id][r:r+300,:],axis=0)\n data[k,400:800] = x\n \n # 20 SECOND WINDOW FEATURES (MEANS and MINS)\n x = np.nanmean(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n data[k,800:1200] = x\n x = np.nanmin(spectrograms[row.spec_id][r+145:r+155,:],axis=0)\n data[k,1200:1600] = x\n\n train[FEATURES] = data\nelse:\n train = pd.read_parquet('/kaggle/input/brain-spectrograms/train.pqt')\nprint()\nprint('New train shape:',train.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:17.410051Z","iopub.execute_input":"2024-01-17T13:40:17.410383Z","iopub.status.idle":"2024-01-17T13:40:35.676525Z","shell.execute_reply.started":"2024-01-17T13:40:17.410355Z","shell.execute_reply":"2024-01-17T13:40:35.675443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" 7.2 | Frequency Band Analysis\n\n#### Frequency Band Feature Extraction:\n\n* The function extract_frequency_band_features is designed to process a segment of EEG data. EEG data is a complex signal that represents the electrical activity of the brain.\n* This function divides the EEG signal into different frequency bands: Delta, Theta, Alpha, Beta, and Gamma. These bands are significant in neuroscientific studies as they are associated with different brain states and activities.\n\n![](https://ars.els-cdn.com/content/image/3-s2.0-B9780128044902000026-f02-01-9780128044902.jpg)\n\n\n1. **Delta (0.5 – 4 Hz):**\nDelta waves are the slowest brainwaves and are typically associated with deep sleep and restorative processes in the body. They are most prominent during dreamless sleep and play a role in healing and regeneration.\n2. **Theta (4 – 8 Hz):**\nTheta waves occur during light sleep, deep meditation, and REM (Rapid Eye Movement) sleep. They are linked to creativity, intuition, daydreaming, and fantasizing. Theta states are often associated with subconscious mind activities.\n3. **Alpha (8 – 12 Hz):**\nAlpha waves are present during physically and mentally relaxed states but still alert. They are typical in wakeful states that involve a relaxed and effortless alertness. Alpha waves aid in mental coordination, calmness, alertness, and learning.\n4. **Beta (12 – 30 Hz):**\nBeta waves dominate our normal waking state of consciousness when attention is directed towards cognitive tasks and the outside world. They are associated with active, busy or anxious thinking and active concentration.\n5. **Gamma (30 – 45 Hz):**\nGamma waves are involved in higher mental activity and consolidation of information. They are important for learning, memory, and information processing. Gamma waves are thought to be the fastest brainwave frequency and relate to simultaneous processing of information from different brain areas.\n\n\n\n\n* For each frequency band, the function applies a bandpass filter to isolate that band's signal. It then computes statistical features (mean, standard deviation, maximum, and minimum) for each band, effectively capturing the characteristics of the EEG signal in these different frequency ranges.\n* The use of np.nanmean, np.nanstd, np.nanmax, and np.nanmin ensures that the calculations are robust to NaN (Not a Number) values in the data, which might occur due to various reasons like signal loss or artifacts.\n\n#### Feature Aggregation and PCA:\n\n* The main script initializes a Principal Component Analysis (PCA) model with the intention of reducing the dimensionality of the extracted features. PCA is a common technique used to transform high-dimensional datasets into a lower-dimensional space while retaining most of the variance in the data.\n* The script iterates over rows in the train dataset, extracting EEG segments and applying the extract_frequency_band_features function to each channel in these segments. The extracted features from all channels are aggregated.\n* However, before applying PCA, any NaN values in the aggregated data (data_original) are handled using mean imputation. This step ensures that the PCA algorithm, which cannot handle NaN values, receives a clean dataset.\n* After imputation, PCA is applied to transform the features into a principal component space, and these transformed features are added back into the train DataFrame.\n* This process ultimately results in a feature set that's potentially more informative and concise for machine learning models, helping in tasks like classification or anomaly detection in EEG data.","metadata":{"execution":{"iopub.status.busy":"2024-01-17T03:03:50.993523Z","iopub.execute_input":"2024-01-17T03:03:50.993878Z","iopub.status.idle":"2024-01-17T03:03:51.029962Z","shell.execute_reply.started":"2024-01-17T03:03:50.993848Z","shell.execute_reply":"2024-01-17T03:03:51.028783Z"}}},{"cell_type":"code","source":"from scipy import signal\nfrom sklearn.decomposition import PCA","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:35.677918Z","iopub.execute_input":"2024-01-17T13:40:35.678281Z","iopub.status.idle":"2024-01-17T13:40:36.516010Z","shell.execute_reply.started":"2024-01-17T13:40:35.678250Z","shell.execute_reply":"2024-01-17T13:40:36.515134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def extract_frequency_band_features(segment):\n # Define EEG frequency bands\n eeg_bands = {'Delta': (0.5, 4), 'Theta': (4, 8), 'Alpha': (8, 12), 'Beta': (12, 30), 'Gamma': (30, 45)}\n \n band_features = []\n for band in eeg_bands:\n low, high = eeg_bands[band]\n # Filter signal for the specific band\n band_pass_filter = signal.butter(3, [low, high], btype='bandpass', fs=200, output='sos')\n filtered = signal.sosfilt(band_pass_filter, segment)\n # Extract features like mean, standard deviation, etc.\n band_features.extend([np.nanmean(filtered), np.nanstd(filtered), np.nanmax(filtered), np.nanmin(filtered)])\n \n return band_features","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:36.519187Z","iopub.execute_input":"2024-01-17T13:40:36.519926Z","iopub.status.idle":"2024-01-17T13:40:36.526465Z","shell.execute_reply.started":"2024-01-17T13:40:36.519894Z","shell.execute_reply":"2024-01-17T13:40:36.525330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import time\n# from sklearn.impute import SimpleImputer\n\n# # Initialize a PCA model\n# pca = PCA(n_components=0.95)\n# print(\"PCA model initialized.\")\n\n# # Initialize an array for original features\n# num_rows = len(train)\n# num_features = 20 * n_channels # 20 features per channel\n# data_original = np.zeros((num_rows, num_features))\n\n# print(\"Starting feature extraction and PCA processing...\")\n# start_time = time.time()\n\n# for k in range(num_rows):\n# if k % 1000 == 0:\n# print(f\"Processing row {k} of {num_rows}...\")\n\n# row = train.iloc[k]\n# r = int((row['min'] + row['max']) // 4)\n# eeg_segment = spectrograms[row.spec_id][r:r+300, :]\n\n# # Apply the feature extraction function to each EEG channel\n# all_channel_features = []\n# for i in range(n_channels):\n# channel_features = extract_frequency_band_features(eeg_segment[:, i])\n# all_channel_features.extend(channel_features)\n \n# data_original[k, :] = all_channel_features\n\n# print(\"Data matrix constructed\")\n\n# # Impute NaN values in the data matrix\n# imputer = SimpleImputer(strategy='mean')\n# data_imputed = imputer.fit_transform(data_original)\n\n# print(f\"NaN values handled. Imputed data matrix shape: {data_imputed.shape}\")\n\n# # Apply PCA on the imputed data\n# pca.fit(data_imputed)\n# print(\"PCA fitting completed.\")\n\n# # Transform data using PCA\n# data_pca = pca.transform(data_imputed)\n\n# # Add PCA features to DataFrame\n# pca_feature_columns = [f'pca_feature_{i}' for i in range(data_pca.shape[1])]\n# train[pca_feature_columns] = data_pca\n\n# # Measure total processing time\n# total_time = time.time() - start_time\n# print(f\"Total processing time: {total_time:.2f} seconds.\")","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:36.528124Z","iopub.execute_input":"2024-01-17T13:40:36.528463Z","iopub.status.idle":"2024-01-17T13:40:36.540152Z","shell.execute_reply.started":"2024-01-17T13:40:36.528426Z","shell.execute_reply":"2024-01-17T13:40:36.538243Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:36.541454Z","iopub.execute_input":"2024-01-17T13:40:36.541814Z","iopub.status.idle":"2024-01-17T13:40:36.590096Z","shell.execute_reply.started":"2024-01-17T13:40:36.541776Z","shell.execute_reply":"2024-01-17T13:40:36.589007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from sklearn.preprocessing import StandardScaler\n\n# # Columns to be excluded from scaling\n# excluded_columns = ['eeg_id', 'spec_id', 'min', 'max', 'patient_id', 'seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote','target']\n\n# # Save the columns to be excluded\n# excluded_data = train[excluded_columns]\n\n# # DataFrame with only the columns to be scaled\n# features = train.drop(columns=excluded_columns)\n\n# # Initialize the StandardScaler\n# scaler = StandardScaler()\n\n# # Fit the scaler to the features and transform them\n# features_scaled = scaler.fit_transform(features)\n\n# # Create a DataFrame from the scaled features\n# features_scaled_df = pd.DataFrame(features_scaled, columns=features.columns)\n\n# # Concatenate the scaled features with the excluded columns\n# train_scaled_df = pd.concat([excluded_data.reset_index(drop=True),features_scaled_df,], axis=1)\n# train_scaled_df \n","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:36.591581Z","iopub.execute_input":"2024-01-17T13:40:36.592454Z","iopub.status.idle":"2024-01-17T13:40:36.598136Z","shell.execute_reply.started":"2024-01-17T13:40:36.592410Z","shell.execute_reply":"2024-01-17T13:40:36.597088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train_scaled_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:36.599942Z","iopub.execute_input":"2024-01-17T13:40:36.600704Z","iopub.status.idle":"2024-01-17T13:40:36.619112Z","shell.execute_reply.started":"2024-01-17T13:40:36.600663Z","shell.execute_reply":"2024-01-17T13:40:36.617906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
8 | TRAIN MODEL
\n\n* Original work uses catboost, let's try with XGBoost in this version to see the difference in model performance.","metadata":{"execution":{"iopub.status.busy":"2024-01-14T15:48:13.199779Z","iopub.execute_input":"2024-01-14T15:48:13.200329Z","iopub.status.idle":"2024-01-14T15:48:13.206828Z","shell.execute_reply.started":"2024-01-14T15:48:13.200282Z","shell.execute_reply":"2024-01-14T15:48:13.205408Z"}}},{"cell_type":"code","source":"import xgboost as xgb\nimport gc\nfrom sklearn.model_selection import KFold, GroupKFold\n\nprint('XGBoost version', xgb.__version__)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:36.621382Z","iopub.execute_input":"2024-01-17T13:40:36.622373Z","iopub.status.idle":"2024-01-17T13:40:36.820884Z","shell.execute_reply.started":"2024-01-17T13:40:36.622328Z","shell.execute_reply":"2024-01-17T13:40:36.819697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_oof = []\nall_true = []\nTARS = {'Seizure':0, 'LPD':1, 'GPD':2, 'LRDA':3, 'GRDA':4, 'Other':5}\n\ngkf = GroupKFold(n_splits=5)\nfor i, (train_index, valid_index) in enumerate(gkf.split(train , train .target, train .patient_id)): \n \n print('#'*25)\n print(f'### Fold {i+1}')\n print(f'### train size {len(train_index)}, valid size {len(valid_index)}')\n print('#'*25)\n \n model = xgb.XGBClassifier(\n objective='multi:softprob', \n num_class=len(TARS),\n learning_rate = 0.1, \n \n# tree_method='gpu_hist', #skip GPU acceleration\n )\n \n # Prepare training and validation data\n X_train = train.loc[train_index, FEATURES]\n y_train = train.loc[train_index, 'target'].map(TARS)\n X_valid = train.loc[valid_index, FEATURES]\n y_valid = train.loc[valid_index, 'target'].map(TARS)\n \n model.fit(X_train, y_train, \n eval_set=[(X_valid, y_valid)], \n verbose=True, \n early_stopping_rounds=10)\n model.save_model(f'XGB_v{VER}_f{i}.model')\n \n oof = model.predict_proba(X_valid)\n all_oof.append(oof)\n all_true.append(train.loc[valid_index, TARGETS].values)\n \n del X_train, y_train, X_valid, y_valid, oof\n gc.collect()\n \nall_oof = np.concatenate(all_oof)\nall_true = np.concatenate(all_true)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:45:00.748977Z","iopub.execute_input":"2024-01-17T13:45:00.749394Z","iopub.status.idle":"2024-01-17T14:06:02.331533Z","shell.execute_reply.started":"2024-01-17T13:45:00.749361Z","shell.execute_reply":"2024-01-17T14:06:02.330233Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
9 | HYPERPARAMETER TUNING
\n\n### 9.1 | Import Libraries and Set Up Optuna\n* First, you import necessary libraries: optuna for hyperparameter optimization, xgboost for the machine learning model, log_loss from scikit-learn for the evaluation metric, and GroupKFold for cross-validation.\n* optuna.create_study(direction='minimize') creates a new optimization study. The direction='minimize' means you want to minimize the value returned by the objective function, which in this case is the log loss.\n\n### 9.2 | Define the Objective Function\n* The objective function is what Optuna will optimize. This function takes a trial object, which is used to suggest values for the hyperparameters.\n* Inside this function, you set up the hyperparameter space. Optuna will test different combinations of these parameters:\n1. lambda, alpha: Regularization parameters.\n2. colsample_bytree, subsample: Ratios for column and row sampling.\n3. learning_rate: Step size shrinkage used to prevent overfitting.\n4. n_estimators: Number of gradient boosted trees.\n5. max_depth: Maximum depth of a tree.\n6. min_child_weight: Minimum sum of instance weight needed in a child.\n\n### 9.3 | Cross-Validation Loop\n\n* The function uses GroupKFold for splitting the data. This method is suitable when you have groups in your data (like patient IDs) that should not be split across the training and validation sets.\n* For each fold in the cross-validation, the function:\n1. Splits the data into training and validation sets.\n2. Trains an XGBoost model using the parameters suggested by Optuna.\n3. Computes the log loss on the validation set.\n4. The average log loss across all folds is returned. Optuna will use this value to decide which hyperparameters are best.\n\n### 9.4 | Running the Optuna Study\n\n* study.optimize(objective, n_trials=100) tells Optuna to optimize the objective function. It will try 100 different combinations of hyperparameters (n_trials=100) to find the best ones.\n* It is best to start with small trials before investing time to run on more trials to manage time invested\n* Once the optimization is complete, the best hyperparameters found are printed.","metadata":{}},{"cell_type":"code","source":"# import optuna\n# from sklearn.metrics import log_loss\n\n\n# def objective(trial):\n# # Hyperparameters to be tuned by Optuna\n# param = {\n# 'objective': 'multi:softprob',\n# 'num_class': len(TARS),\n# 'tree_method': 'gpu_hist', # use 'gpu_hist' for GPU\n# 'lambda': trial.suggest_loguniform('lambda', 1e-4, 10.0),\n# 'alpha': trial.suggest_loguniform('alpha', 1e-4, 10.0),\n# 'colsample_bytree': trial.suggest_categorical('colsample_bytree', [0.5, 0.6, 0.7, 0.8, 0.9, 1.0]),\n# 'subsample': trial.suggest_categorical('subsample', [0.6, 0.7, 0.8, 0.9, 1.0]),\n# 'learning_rate': trial.suggest_categorical('learning_rate', [0.008, 0.01, 0.02, 0.05, 0.1]),\n# 'n_estimators': 1000,\n# 'max_depth': trial.suggest_categorical('max_depth', [5, 7, 9, 11, 13]),\n# 'min_child_weight': trial.suggest_int('min_child_weight', 1, 300),\n# }\n\n# gkf = GroupKFold(n_splits=5)\n# cv_scores = []\n\n# for train_index, valid_index in gkf.split(train, train.target, train.patient_id):\n# X_train, X_valid = train.loc[train_index, FEATURES], train.loc[valid_index, FEATURES]\n# y_train, y_valid = train.loc[train_index, 'target'].map(TARS), train.loc[valid_index, 'target'].map(TARS)\n\n# model = xgb.XGBClassifier(**param)\n# model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], verbose=False, early_stopping_rounds=10)\n# preds = model.predict_proba(X_valid)\n# cv_scores.append(log_loss(y_valid, preds))\n\n# return np.mean(cv_scores)\n\n# study = optuna.create_study(direction='minimize')\n# study.optimize(objective, n_trials=10) # Increase n_trials for more extensive search\n\n# print('Number of finished trials:', len(study.trials))\n# print('Best trial:', study.best_trial.params)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:37.295221Z","iopub.status.idle":"2024-01-17T13:40:37.295669Z","shell.execute_reply.started":"2024-01-17T13:40:37.295467Z","shell.execute_reply":"2024-01-17T13:40:37.295489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* [I 2024-01-15 06:06:55,585] A new study created in memory with name: no-name-21c5e987-7941-4f05-8e92-2903b9a7e304\n* [I 2024-01-15 06:20:36,414] Trial 0 finished with value: 1.0734795198486586 and parameters: {'lambda': 8.578460041884545, 'alpha': 1.0420327876364774, 'colsample_bytree': 1.0, 'subsample': 1.0, 'learning_rate': 0.02, 'max_depth': 7, 'min_child_weight': 30}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 06:39:35,894] Trial 1 finished with value: 1.0792154540612213 and parameters: {'lambda': 0.0029055528938438085, 'alpha': 0.03244985452963714, 'colsample_bytree': 0.5, 'subsample': 1.0, 'learning_rate': 0.01, 'max_depth': 11, 'min_child_weight': 77}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 06:42:19,332] Trial 2 finished with value: 1.107279543251576 and parameters: {'lambda': 1.5919274718287213, 'alpha': 0.042136459342788604, 'colsample_bytree': 0.7, 'subsample': 0.7, 'learning_rate': 0.1, 'max_depth': 5, 'min_child_weight': 152}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 06:57:56,577] Trial 3 finished with value: 1.107956784549986 and parameters: {'lambda': 6.873357111288177, 'alpha': 0.05538406375764404, 'colsample_bytree': 0.5, 'subsample': 0.8, 'learning_rate': 0.008, 'max_depth': 7, 'min_child_weight': 108}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 07:12:36,921] Trial 4 finished with value: 1.1453593669298952 and parameters: {'lambda': 0.0012348624625841455, 'alpha': 6.698933350539058, 'colsample_bytree': 0.8, 'subsample': 0.9, 'learning_rate': 0.01, 'max_depth': 9, 'min_child_weight': 258}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 07:27:45,543] Trial 5 finished with value: 1.1234561427497631 and parameters: {'lambda': 0.0779496745099949, 'alpha': 0.001997110519034328, 'colsample_bytree': 0.5, 'subsample': 0.7, 'learning_rate': 0.008, 'max_depth': 11, 'min_child_weight': 145}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 07:30:45,104] Trial 6 finished with value: 1.139466297500727 and parameters: {'lambda': 0.011643281906929, 'alpha': 0.06334100511005662, 'colsample_bytree': 0.6, 'subsample': 0.7, 'learning_rate': 0.1, 'max_depth': 9, 'min_child_weight': 274}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 07:33:47,478] Trial 7 finished with value: 1.074140292040741 and parameters: {'lambda': 5.487810272015954, 'alpha': 2.266845998962579, 'colsample_bytree': 0.6, 'subsample': 0.8, 'learning_rate': 0.1, 'max_depth': 7, 'min_child_weight': 51}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 07:48:19,863] Trial 8 finished with value: 1.1705891200455967 and parameters: {'lambda': 0.03333036846711228, 'alpha': 0.0004482362892025373, 'colsample_bytree': 0.9, 'subsample': 0.8, 'learning_rate': 0.008, 'max_depth': 13, 'min_child_weight': 294}. Best is trial 0 with value: 1.0734795198486586.\n* [I 2024-01-15 07:53:41,442] Trial 9 finished with value: 1.115599617296683 and parameters: {'lambda': 0.000284724944614318, 'alpha': 0.020480207664040264, 'colsample_bytree': 0.6, 'subsample': 0.9, 'learning_rate': 0.05, 'max_depth': 9, 'min_child_weight': 248}. Best is trial 0 with value: 1.0734795198486586.\nNumber of finished trials: 10\n\n* **Best trial: {'lambda': 8.578460041884545, 'alpha': 1.0420327876364774, 'colsample_bytree': 1.0, 'subsample': 1.0, 'learning_rate': 0.02, 'max_depth': 7, 'min_child_weight': 30}**","metadata":{}},{"cell_type":"markdown","source":"#
10 | FEATURE IMPORTANCE
","metadata":{"execution":{"iopub.status.busy":"2024-01-14T16:08:35.106237Z","iopub.execute_input":"2024-01-14T16:08:35.106804Z","iopub.status.idle":"2024-01-14T16:08:35.112735Z","shell.execute_reply.started":"2024-01-14T16:08:35.106757Z","shell.execute_reply":"2024-01-14T16:08:35.111659Z"}}},{"cell_type":"code","source":"TOP = 30\n\n# Assuming 'model' is your trained model\nfeature_importance = model.feature_importances_\n\n# Get the feature names from 'train'\nfeature_names = train.columns\n\n# Sort the feature importances and get the indices of the sorted array\nsorted_idx = np.argsort(feature_importance)\n\n# Plot only the top 'TOP' features\nfig = plt.figure(figsize=(10, 8))\nplt.barh(np.arange(len(sorted_idx))[-TOP:], feature_importance[sorted_idx][-TOP:], align='center')\nplt.yticks(np.arange(len(sorted_idx))[-TOP:], feature_names[sorted_idx][-TOP:])\nplt.title(f'Feature Importance - Top {TOP}')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T14:12:19.822466Z","iopub.execute_input":"2024-01-17T14:12:19.823028Z","iopub.status.idle":"2024-01-17T14:12:20.406006Z","shell.execute_reply.started":"2024-01-17T14:12:19.822988Z","shell.execute_reply":"2024-01-17T14:12:20.404813Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
11 | INFER TEST
","metadata":{"execution":{"iopub.status.busy":"2024-01-14T16:17:09.800209Z","iopub.execute_input":"2024-01-14T16:17:09.802007Z","iopub.status.idle":"2024-01-14T16:17:09.809141Z","shell.execute_reply.started":"2024-01-14T16:17:09.801957Z","shell.execute_reply":"2024-01-14T16:17:09.807551Z"}}},{"cell_type":"code","source":"test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\nprint('Test shape',test.shape)\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T14:12:32.465945Z","iopub.execute_input":"2024-01-17T14:12:32.466394Z","iopub.status.idle":"2024-01-17T14:12:32.485369Z","shell.execute_reply.started":"2024-01-17T14:12:32.466357Z","shell.execute_reply":"2024-01-17T14:12:32.484210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\n# spec = pd.read_parquet(f'{PATH2}{s}.parquet')\n# spec","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:37.300846Z","iopub.status.idle":"2024-01-17T13:40:37.301226Z","shell.execute_reply.started":"2024-01-17T13:40:37.301049Z","shell.execute_reply":"2024-01-17T13:40:37.301068Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %%time\n# # READ ALL TEST SPECTROGRAMS\n# PATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\n# files = os.listdir(PATH2)\n# print(f'There are {len(files)} spectrogram parquets')\n\n# spectrograms = {}\n# for i,f in enumerate(files):\n# if i%100==0: print(i,', ',end='')\n# tmp = pd.read_parquet(f'{PATH2}{f}')\n# name = int(f.split('.')[0])\n# spectrograms_test[name] = tmp.iloc[:,1:].values\n","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:37.302384Z","iopub.status.idle":"2024-01-17T13:40:37.302745Z","shell.execute_reply.started":"2024-01-17T13:40:37.302572Z","shell.execute_reply":"2024-01-17T13:40:37.302589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# %time\n# # ENGINEER FEATURES\n# import warnings\n# warnings.filterwarnings('ignore')\n\n# # The code generates features from the spectrogram data for use in a model \n# # The features are derived by calculating the mean and minimum values over time for each of the 400 spectrogram frequencies.\n# # Two types of windows are used for these calculations:\n# # A 10-minute window (_mean_10m, _min_10m).\n# # A 20-second window (_mean_20s, _min_20s).\n# # This process results in 1600 features (400 features × 4 calculations) for each EEG ID.\n\n# SPEC_COLS = pd.read_parquet(f'{PATH}1000086677.parquet').columns[1:]\n# FEATURES = [f'{c}_mean_10m' for c in SPEC_COLS]\n# FEATURES += [f'{c}_min_10m' for c in SPEC_COLS]\n# FEATURES += [f'{c}_mean_20s' for c in SPEC_COLS]\n# FEATURES += [f'{c}_min_20s' for c in SPEC_COLS]\n# print(f'We are creating {len(FEATURES)} features for {len(test)} rows... ',end='')\n\n\n# # A data matrix data is initialized to store the new features for each eeg_id in the train DataFrame.\n# # For each row in train, the code calculates the mean and minimum values within the specified 10-minute and 20-second windows.\n# # These calculated values are then stored in the data matrix.\n# # Finally, the matrix is added to the train DataFrame as new columns.\n\n# data = np.zeros((len(test),len(FEATURES)))\n# for k in range(len(test)):\n# if k%100==0: print(k,', ',end='')\n# row = test.iloc[k]\n \n# # 10 MINUTE WINDOW FEATURES\n# x = np.nanmean( spec.iloc[:,1:].values, axis=0)\n# data[k,:400] = x\n# x = np.nanmin( spec.iloc[:,1:].values, axis=0)\n# data[k,400:800] = x\n\n# # 20 SECOND WINDOW FEATURES\n# x = np.nanmean( spec.iloc[145:155,1:].values, axis=0)\n# data[k,800:1200] = x\n# x = np.nanmin( spec.iloc[145:155,1:].values, axis=0)\n# data[k,1200:1600] = x\n\n# test[FEATURES] = data\n\n \n# print()\n# print('New test shape:',test.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:37.303729Z","iopub.status.idle":"2024-01-17T13:40:37.304126Z","shell.execute_reply.started":"2024-01-17T13:40:37.303944Z","shell.execute_reply":"2024-01-17T13:40:37.303963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# from sklearn.impute import SimpleImputer\n\n# # Initialize a PCA model\n# pca = PCA(n_components=0.95)\n# print(\"PCA model initialized.\")\n\n# # Initialize an array for original features\n# num_rows = len(test)\n# num_features = 20 * n_channels # 20 features per channel\n# data_original = np.zeros((num_rows, num_features))\n\n# print(\"Starting feature extraction and PCA processing...\")\n# start_time = time.time()\n\n# for k in range(num_rows):\n# if k % 1000 == 0:\n# print(f\"Processing row {k} of {num_rows}...\")\n\n# row = train.iloc[k]\n# eeg_segment = spectrograms_test[853520][r:r+300, :]\n\n# # Apply the feature extraction function to each EEG channel\n# all_channel_features = []\n# for i in range(n_channels):\n# channel_features = extract_frequency_band_features(eeg_segment[:, i])\n# all_channel_features.extend(channel_features)\n \n# data_original[k, :] = all_channel_features\n\n# print(\"Data matrix constructed\")\n\n# # Impute NaN values in the data matrix\n# imputer = SimpleImputer(strategy='mean')\n# data_imputed = imputer.fit_transform(data_original)\n\n# print(f\"NaN values handled. Imputed data matrix shape: {data_imputed.shape}\")\n\n# # Apply PCA on the imputed data\n# pca.fit(data_imputed)\n# print(\"PCA fitting completed.\")\n\n# # Transform data using PCA\n# data_pca = pca.transform(data_imputed)\n\n# # Add PCA features to DataFrame\n# pca_feature_columns = [f'pca_feature_{i}' for i in range(data_pca.shape[1])]\n# test[pca_feature_columns] = data_pca\n\n# # Measure total processing time\n# total_time = time.time() - start_time\n# print(f\"Total processing time: {total_time:.2f} seconds.\")\n\n# test.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:37.305186Z","iopub.status.idle":"2024-01-17T13:40:37.305554Z","shell.execute_reply.started":"2024-01-17T13:40:37.305379Z","shell.execute_reply":"2024-01-17T13:40:37.305396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # Columns to be excluded from scaling\n# excluded_columns = ['eeg_id', 'spectrogram_id', 'patient_id']\n\n# # Save the columns to be excluded\n# excluded_data = test[excluded_columns]\n\n# # DataFrame with only the columns to be scaled\n# features = test.drop(columns=excluded_columns)\n\n# # Initialize the StandardScaler\n# scaler = StandardScaler()\n\n# # Fit the scaler to the features and transform them\n# features_scaled = scaler.fit_transform(features)\n\n# # Create a DataFrame from the scaled features\n# features_scaled_df = pd.DataFrame(features_scaled, columns=features.columns)\n\n# # Concatenate the scaled features with the excluded columns\n# test_scaled_df = pd.concat([excluded_data.reset_index(drop=True),features_scaled_df,], axis=1)\n# test_scaled_df \n","metadata":{"execution":{"iopub.status.busy":"2024-01-17T13:40:37.308779Z","iopub.status.idle":"2024-01-17T13:40:37.309167Z","shell.execute_reply.started":"2024-01-17T13:40:37.308993Z","shell.execute_reply":"2024-01-17T13:40:37.309011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# FEATURE ENGINEER TEST\nPATH2 = '/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms/'\ndata = np.zeros((len(test),len(FEATURES)))\n \nfor k in range(len(test)):\n row = test.iloc[k]\n s = int( row.spectrogram_id )\n spec = pd.read_parquet(f'{PATH2}{s}.parquet')\n \n # 10 MINUTE WINDOW FEATURES\n x = np.nanmean( spec.iloc[:,1:].values, axis=0)\n data[k,:400] = x\n x = np.nanmin( spec.iloc[:,1:].values, axis=0)\n data[k,400:800] = x\n\n # 20 SECOND WINDOW FEATURES\n x = np.nanmean( spec.iloc[145:155,1:].values, axis=0)\n data[k,800:1200] = x\n x = np.nanmin( spec.iloc[145:155,1:].values, axis=0)\n data[k,1200:1600] = x\n\ntest[FEATURES] = data\nprint('New test shape',test.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T14:12:38.149388Z","iopub.execute_input":"2024-01-17T14:12:38.149775Z","iopub.status.idle":"2024-01-17T14:12:38.894043Z","shell.execute_reply.started":"2024-01-17T14:12:38.149746Z","shell.execute_reply":"2024-01-17T14:12:38.893069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER XGBOOST ON TEST\npreds = []\n\nfor i in range(5):\n print(i, ', ', end='')\n \n # Load the XGBoost model\n model = xgb.XGBClassifier()\n model.load_model(f'XGB_v{VER}_f{i}.model')\n \n # Make predictions\n pred = model.predict_proba(test[FEATURES])\n preds.append(pred)\n\n# Average the predictions from each fold\npred = np.mean(preds, axis=0)\nprint()\nprint('Test preds shape', pred.shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T14:12:50.631050Z","iopub.execute_input":"2024-01-17T14:12:50.631491Z","iopub.status.idle":"2024-01-17T14:12:51.908914Z","shell.execute_reply.started":"2024-01-17T14:12:50.631455Z","shell.execute_reply":"2024-01-17T14:12:51.907722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
12 | SUBMISSION
","metadata":{}},{"cell_type":"code","source":"sub = pd.DataFrame({'eeg_id':test.eeg_id.values})\nsub[TARGETS] = pred\nsub.to_csv('submission.csv',index=False)\nprint('Submission shape',sub.shape)\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T14:12:56.955648Z","iopub.execute_input":"2024-01-17T14:12:56.956094Z","iopub.status.idle":"2024-01-17T14:12:56.979024Z","shell.execute_reply.started":"2024-01-17T14:12:56.956057Z","shell.execute_reply":"2024-01-17T14:12:56.978179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# SANITY CHECK TO CONFIRM PREDICTIONS SUM TO ONE\nsub.iloc[:,-6:].sum(axis=1)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T14:13:01.825709Z","iopub.execute_input":"2024-01-17T14:13:01.826129Z","iopub.status.idle":"2024-01-17T14:13:01.837955Z","shell.execute_reply.started":"2024-01-17T14:13:01.826094Z","shell.execute_reply":"2024-01-17T14:13:01.836590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}