{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"}],"dockerImageVersionId":30635,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"> The goal of this competition is to detect and classify seizures and other types of harmful brain activity in electroencephalography (EEG) data. Even experts find this to be a challenging task and often disagree about the correct labels.","metadata":{}},{"cell_type":"markdown","source":"# Setup","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport polars as pl\npl.Config.set_tbl_cols(-1);\nfrom pathlib import Path","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:33.031380Z","iopub.execute_input":"2024-01-18T07:33:33.031839Z","iopub.status.idle":"2024-01-18T07:33:33.304193Z","shell.execute_reply.started":"2024-01-18T07:33:33.031796Z","shell.execute_reply":"2024-01-18T07:33:33.302817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load dataset","metadata":{}},{"cell_type":"code","source":"df_train = pl.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\ndf_test = pl.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\ndf_sample_submission = pl.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:33.306857Z","iopub.execute_input":"2024-01-18T07:33:33.307353Z","iopub.status.idle":"2024-01-18T07:33:33.547792Z","shell.execute_reply.started":"2024-01-18T07:33:33.307310Z","shell.execute_reply":"2024-01-18T07:33:33.546675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data description","metadata":{}},{"cell_type":"markdown","source":"> **train.csv** Metadata for the train set. The expert annotators reviewed 50 second long EEG samples plus matched spectrograms covering 10 a minute window centered at the same time and labeled the central 10 seconds. Many of these samples overlapped and have been consolidated. train.csv provides the metadata that allows you to extract the original subsets that the raters annotated.\n> \n> - `eeg_id` - A unique identifier for the entire EEG recording.\n> - `eeg_sub_id` - An ID for the specific 50 second long subsample this row's labels apply to.\n> - `eeg_label_offset_seconds` - The time between the beginning of the consolidated EEG and this subsample.\n> - `spectrogram_id` - A unique identifier for the entire EEG recording.\n> - `spectrogram_sub_id` - An ID for the specific 10 minute subsample this row's labels apply to.\n> - `spectogram_label_offset_seconds` - The time between the beginning of the consolidated spectrogram and this subsample.\n> - `label_id` - An ID for this set of labels.\n> - `patient_id` - An ID for the patient who donated the data.\n> - `expert_consensus` - The consensus annotator label. Provided for convenience only.\n> - `[seizure/lpd/gpd/lrda/grda/other]_vote` - The count of annotator votes for a given brain activity class. The full names of the activity classes are as follows: lpd: lateralized periodic discharges, gpd: generalized periodic discharges, lrd: lateralized rhythmic delta activity, and grda: generalized rhythmic delta activity . A detailed explanations of these patterns is [available here](https://www.acns.org/UserFiles/file/ACNSStandardizedCriticalCareEEGTerminology_rev2021.pdf).","metadata":{}},{"cell_type":"code","source":"display(df_train)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:42.075107Z","iopub.execute_input":"2024-01-18T07:33:42.075492Z","iopub.status.idle":"2024-01-18T07:33:42.109340Z","shell.execute_reply.started":"2024-01-18T07:33:42.075463Z","shell.execute_reply":"2024-01-18T07:33:42.107802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **test.csv** Metadata for the test set. As there are no overlapping samples in the test set, many columns in the train metadata don't apply.\n> \n> - `eeg_id`\n> - `spectrogram_id`\n> - `patient_id`","metadata":{}},{"cell_type":"code","source":"display(df_test)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:43.619813Z","iopub.execute_input":"2024-01-18T07:33:43.620182Z","iopub.status.idle":"2024-01-18T07:33:43.629249Z","shell.execute_reply.started":"2024-01-18T07:33:43.620154Z","shell.execute_reply":"2024-01-18T07:33:43.628420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **sample_submission.csv**\n> \n> - `eeg_id`\n> - `[seizure/lpd/gpd/lrda/grda/other]_vote` - The target columns. Your predictions must be probabilities. Note that the test samples had between 3 and 20 annotators.","metadata":{}},{"cell_type":"code","source":"display(df_sample_submission)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:44.010306Z","iopub.execute_input":"2024-01-18T07:33:44.011586Z","iopub.status.idle":"2024-01-18T07:33:44.017769Z","shell.execute_reply.started":"2024-01-18T07:33:44.011524Z","shell.execute_reply":"2024-01-18T07:33:44.017107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **train_eegs/** EEG data from one or more overlapping samples. Use the metadata in train.csv to select specific annotated subsets. The column names are [the names of the individual electrode locations for EEG leads](https://en.wikipedia.org/wiki/10%E2%80%9320_system_%28EEG%29), with one exception. The EKG column is for an electrocardiogram lead that records data from the heart. All of the EEG data (for both train and test) was collected at a frequency of 200 samples per second.","metadata":{}},{"cell_type":"code","source":"sample_eeg_id = 1628180742","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:44.388608Z","iopub.execute_input":"2024-01-18T07:33:44.389699Z","iopub.status.idle":"2024-01-18T07:33:44.394280Z","shell.execute_reply.started":"2024-01-18T07:33:44.389662Z","shell.execute_reply":"2024-01-18T07:33:44.392860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_eeg = pl.read_parquet(\n    Path('/kaggle/input/hms-harmful-brain-activity-classification/train_eegs')\n    / f'{sample_eeg_id}.parquet'\n)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:44.579546Z","iopub.execute_input":"2024-01-18T07:33:44.579921Z","iopub.status.idle":"2024-01-18T07:33:44.652251Z","shell.execute_reply.started":"2024-01-18T07:33:44.579889Z","shell.execute_reply":"2024-01-18T07:33:44.651562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_sample_eeg)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:44.763054Z","iopub.execute_input":"2024-01-18T07:33:44.763423Z","iopub.status.idle":"2024-01-18T07:33:44.779106Z","shell.execute_reply.started":"2024-01-18T07:33:44.763395Z","shell.execute_reply":"2024-01-18T07:33:44.778315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **test_eegs/** Exactly 50 seconds of EEG data.","metadata":{}},{"cell_type":"code","source":"sample_test_eeg_id = 3911565283\ndf_sample_test_eeg = pl.read_parquet(\n    Path('/kaggle/input/hms-harmful-brain-activity-classification/test_eegs')\n    / f'{sample_test_eeg_id}.parquet'\n)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:45.179430Z","iopub.execute_input":"2024-01-18T07:33:45.180010Z","iopub.status.idle":"2024-01-18T07:33:45.244796Z","shell.execute_reply.started":"2024-01-18T07:33:45.179977Z","shell.execute_reply":"2024-01-18T07:33:45.244004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_sample_test_eeg)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:45.386775Z","iopub.execute_input":"2024-01-18T07:33:45.387441Z","iopub.status.idle":"2024-01-18T07:33:45.402775Z","shell.execute_reply.started":"2024-01-18T07:33:45.387405Z","shell.execute_reply":"2024-01-18T07:33:45.401067Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **train_spectrograms/** Spectrograms assembled EEG data. Use the metadata in train.csv to select specific annotated subsets. The column names indicate the frequency in hertz and the recording regions of the EEG electrodes. The latter are abbreviated as LL = left lateral; RL = right lateral; LP = left parasagittal; RP = right parasagittal.","metadata":{}},{"cell_type":"code","source":"sample_spectrogram_id = 353733","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:45.810807Z","iopub.execute_input":"2024-01-18T07:33:45.811444Z","iopub.status.idle":"2024-01-18T07:33:45.816827Z","shell.execute_reply.started":"2024-01-18T07:33:45.811407Z","shell.execute_reply":"2024-01-18T07:33:45.815749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample_spectrogram = pl.read_parquet(\n    Path('/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms')\n    / f'{sample_spectrogram_id}.parquet'\n)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:46.171469Z","iopub.execute_input":"2024-01-18T07:33:46.171827Z","iopub.status.idle":"2024-01-18T07:33:46.204699Z","shell.execute_reply.started":"2024-01-18T07:33:46.171798Z","shell.execute_reply":"2024-01-18T07:33:46.203590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_sample_spectrogram)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:46.370618Z","iopub.execute_input":"2024-01-18T07:33:46.370979Z","iopub.status.idle":"2024-01-18T07:33:46.539909Z","shell.execute_reply.started":"2024-01-18T07:33:46.370950Z","shell.execute_reply":"2024-01-18T07:33:46.538114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **test_spectrograms/** Spectrograms assembled using exactly 10 minutes of EEG data.","metadata":{}},{"cell_type":"code","source":"sample_test_spectrogram_id = 853520\ndf_sample_test_spectrogram = pl.read_parquet(\n    Path('/kaggle/input/hms-harmful-brain-activity-classification/test_spectrograms')\n    / f'{sample_test_spectrogram_id}.parquet'\n)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:46.770723Z","iopub.execute_input":"2024-01-18T07:33:46.771121Z","iopub.status.idle":"2024-01-18T07:33:46.805843Z","shell.execute_reply.started":"2024-01-18T07:33:46.771090Z","shell.execute_reply":"2024-01-18T07:33:46.804584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_sample_test_spectrogram)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:47.011104Z","iopub.execute_input":"2024-01-18T07:33:47.011457Z","iopub.status.idle":"2024-01-18T07:33:47.175963Z","shell.execute_reply.started":"2024-01-18T07:33:47.011427Z","shell.execute_reply":"2024-01-18T07:33:47.174624Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **example_figures/** Larger copies of the example case images used on the overview tab.","metadata":{}},{"cell_type":"code","source":"# the following code should display PDF, but seems not on Kaggle\nfrom IPython.display import display_pdf\n\nwith open('/kaggle/input/hms-harmful-brain-activity-classification/example_figures/Sample01.pdf', 'rb') as f:\n    display_pdf(f.read(),raw=True)","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:47.475637Z","iopub.execute_input":"2024-01-18T07:33:47.476095Z","iopub.status.idle":"2024-01-18T07:33:47.507112Z","shell.execute_reply.started":"2024-01-18T07:33:47.476064Z","shell.execute_reply":"2024-01-18T07:33:47.506418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualization","metadata":{}},{"cell_type":"code","source":"display(df_train.describe())","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:50.043619Z","iopub.execute_input":"2024-01-18T07:33:50.043960Z","iopub.status.idle":"2024-01-18T07:33:50.092367Z","shell.execute_reply.started":"2024-01-18T07:33:50.043930Z","shell.execute_reply":"2024-01-18T07:33:50.090694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_sample_eeg.describe())","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:33:59.828007Z","iopub.execute_input":"2024-01-18T07:33:59.828367Z","iopub.status.idle":"2024-01-18T07:33:59.852188Z","shell.execute_reply.started":"2024-01-18T07:33:59.828337Z","shell.execute_reply":"2024-01-18T07:33:59.851192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_sample_spectrogram.describe())","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:34:00.321032Z","iopub.execute_input":"2024-01-18T07:34:00.321622Z","iopub.status.idle":"2024-01-18T07:34:00.451175Z","shell.execute_reply.started":"2024-01-18T07:34:00.321588Z","shell.execute_reply":"2024-01-18T07:34:00.450408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train['expert_consensus'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:34:01.804726Z","iopub.execute_input":"2024-01-18T07:34:01.805381Z","iopub.status.idle":"2024-01-18T07:34:01.824745Z","shell.execute_reply.started":"2024-01-18T07:34:01.805340Z","shell.execute_reply":"2024-01-18T07:34:01.823310Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_train['label_id'].n_unique())","metadata":{"execution":{"iopub.status.busy":"2024-01-18T07:34:11.187015Z","iopub.execute_input":"2024-01-18T07:34:11.187453Z","iopub.status.idle":"2024-01-18T07:34:11.204536Z","shell.execute_reply.started":"2024-01-18T07:34:11.187417Z","shell.execute_reply":"2024-01-18T07:34:11.203597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eeg_samples_per_second = 200","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:15.741055Z","iopub.execute_input":"2024-01-17T04:39:15.741430Z","iopub.status.idle":"2024-01-17T04:39:15.746685Z","shell.execute_reply.started":"2024-01-17T04:39:15.741391Z","shell.execute_reply":"2024-01-17T04:39:15.745337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_eeg_subsamples(df_eeg, offset):\n    return (\n        df_eeg\n        .with_row_count('index')\n        .filter(pl.col('index').is_between(\n            offset * eeg_samples_per_second,\n            (offset + 50) * eeg_samples_per_second,\n            closed = 'left',\n        ))\n        .drop('index')\n    )","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:15.748498Z","iopub.execute_input":"2024-01-17T04:39:15.748885Z","iopub.status.idle":"2024-01-17T04:39:15.758561Z","shell.execute_reply.started":"2024-01-17T04:39:15.748842Z","shell.execute_reply":"2024-01-17T04:39:15.757169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_subsample_eeg = get_eeg_subsamples(df_sample_eeg, 0)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:15.760240Z","iopub.execute_input":"2024-01-17T04:39:15.761003Z","iopub.status.idle":"2024-01-17T04:39:15.787193Z","shell.execute_reply.started":"2024-01-17T04:39:15.760959Z","shell.execute_reply":"2024-01-17T04:39:15.785935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:15.788836Z","iopub.execute_input":"2024-01-17T04:39:15.789563Z","iopub.status.idle":"2024-01-17T04:39:17.080386Z","shell.execute_reply.started":"2024-01-17T04:39:15.789526Z","shell.execute_reply":"2024-01-17T04:39:17.078818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.lineplot(df_subsample_eeg.to_pandas());","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:17.081684Z","iopub.execute_input":"2024-01-17T04:39:17.082245Z","iopub.status.idle":"2024-01-17T04:39:21.914387Z","shell.execute_reply.started":"2024-01-17T04:39:17.082210Z","shell.execute_reply":"2024-01-17T04:39:21.913011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.lineplot(df_subsample_eeg.head(eeg_samples_per_second * 5).to_pandas());","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:21.916446Z","iopub.execute_input":"2024-01-17T04:39:21.916820Z","iopub.status.idle":"2024-01-17T04:39:23.352101Z","shell.execute_reply.started":"2024-01-17T04:39:21.916784Z","shell.execute_reply":"2024-01-17T04:39:23.350772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.lineplot(df_subsample_eeg.head(eeg_samples_per_second * 1).to_pandas());","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:23.353723Z","iopub.execute_input":"2024-01-17T04:39:23.354146Z","iopub.status.idle":"2024-01-17T04:39:24.388019Z","shell.execute_reply.started":"2024-01-17T04:39:23.354111Z","shell.execute_reply":"2024-01-17T04:39:24.386475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.lineplot(df_subsample_eeg.drop('EKG').to_pandas());","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:24.389786Z","iopub.execute_input":"2024-01-17T04:39:24.390213Z","iopub.status.idle":"2024-01-17T04:39:29.306721Z","shell.execute_reply.started":"2024-01-17T04:39:24.390176Z","shell.execute_reply":"2024-01-17T04:39:29.305422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.lineplot(df_subsample_eeg.drop('EKG').head(eeg_samples_per_second * 5).to_pandas());","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:29.312695Z","iopub.execute_input":"2024-01-17T04:39:29.313160Z","iopub.status.idle":"2024-01-17T04:39:31.057393Z","shell.execute_reply.started":"2024-01-17T04:39:29.313123Z","shell.execute_reply":"2024-01-17T04:39:31.056028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.lineplot(df_subsample_eeg.drop('EKG').head(eeg_samples_per_second * 1).to_pandas());","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:31.059426Z","iopub.execute_input":"2024-01-17T04:39:31.059886Z","iopub.status.idle":"2024-01-17T04:39:32.245854Z","shell.execute_reply.started":"2024-01-17T04:39:31.059844Z","shell.execute_reply":"2024-01-17T04:39:32.244611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_spectrogram_subsamples(df_spectrogram, offset, region):\n    return (\n        df_spectrogram\n        .filter(pl.col('time').is_between(\n            offset, offset + 600,\n            closed = 'left',\n        ))\n        .select(pl.col(f'^{region}_.*$'))\n    )","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:32.247725Z","iopub.execute_input":"2024-01-17T04:39:32.248115Z","iopub.status.idle":"2024-01-17T04:39:32.255211Z","shell.execute_reply.started":"2024-01-17T04:39:32.248079Z","shell.execute_reply":"2024-01-17T04:39:32.253644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_subsample_spectrogram = get_spectrogram_subsamples(df_sample_spectrogram, 0, 'LL')","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:32.257060Z","iopub.execute_input":"2024-01-17T04:39:32.257418Z","iopub.status.idle":"2024-01-17T04:39:32.269853Z","shell.execute_reply.started":"2024-01-17T04:39:32.257384Z","shell.execute_reply":"2024-01-17T04:39:32.268812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_subsample_spectrogram)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:32.271461Z","iopub.execute_input":"2024-01-17T04:39:32.272688Z","iopub.status.idle":"2024-01-17T04:39:32.350640Z","shell.execute_reply.started":"2024-01-17T04:39:32.272647Z","shell.execute_reply":"2024-01-17T04:39:32.349615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.heatmap(df_subsample_spectrogram.to_pandas().T);","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:32.352222Z","iopub.execute_input":"2024-01-17T04:39:32.352634Z","iopub.status.idle":"2024-01-17T04:39:33.637963Z","shell.execute_reply.started":"2024-01-17T04:39:32.352599Z","shell.execute_reply":"2024-01-17T04:39:33.636500Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.heatmap(get_spectrogram_subsamples(df_sample_spectrogram, 0, 'RL').to_pandas().T);","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:33.639418Z","iopub.execute_input":"2024-01-17T04:39:33.639771Z","iopub.status.idle":"2024-01-17T04:39:34.915635Z","shell.execute_reply.started":"2024-01-17T04:39:33.639740Z","shell.execute_reply":"2024-01-17T04:39:34.914194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.heatmap(get_spectrogram_subsamples(df_sample_spectrogram, 0, 'LP').to_pandas().T);","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:34.917269Z","iopub.execute_input":"2024-01-17T04:39:34.917613Z","iopub.status.idle":"2024-01-17T04:39:36.201033Z","shell.execute_reply.started":"2024-01-17T04:39:34.917583Z","shell.execute_reply":"2024-01-17T04:39:36.199527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (16, 8))\nsns.heatmap(get_spectrogram_subsamples(df_sample_spectrogram, 0, 'RP').to_pandas().T);","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:36.202772Z","iopub.execute_input":"2024-01-17T04:39:36.203180Z","iopub.status.idle":"2024-01-17T04:39:37.693725Z","shell.execute_reply.started":"2024-01-17T04:39:36.203146Z","shell.execute_reply":"2024-01-17T04:39:37.692452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Stats","metadata":{}},{"cell_type":"code","source":"train_eeg_dir = Path('/kaggle/input/hms-harmful-brain-activity-classification/train_eegs')","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:39:37.695675Z","iopub.execute_input":"2024-01-17T04:39:37.697329Z","iopub.status.idle":"2024-01-17T04:39:37.704888Z","shell.execute_reply.started":"2024-01-17T04:39:37.697275Z","shell.execute_reply":"2024-01-17T04:39:37.703479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_eeg_subsamples(eeg_id, offset):\n    path = train_eeg_dir / f'{eeg_id}.parquet'\n    df = pl.read_parquet(path)\n    return get_eeg_subsamples(df, offset)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:40:19.308342Z","iopub.execute_input":"2024-01-17T04:40:19.308822Z","iopub.status.idle":"2024-01-17T04:40:19.315233Z","shell.execute_reply.started":"2024-01-17T04:40:19.308784Z","shell.execute_reply":"2024-01-17T04:40:19.314169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_eeg_stats(df_train = df_train):\n    acc = pl.DataFrame()\n    df = df_train.select('eeg_id', 'eeg_label_offset_seconds')\n    for r in df.iter_rows():\n        eeg_id = r[0]\n        offset = r[1]\n        eeg = read_eeg_subsamples(eeg_id, offset)\n        eeg_min = eeg.min().select(pl.col('*').name.suffix('_min'))\n        eeg_max = eeg.max().select(pl.col('*').name.suffix('_max'))\n        eeg_null = eeg.null_count().select(pl.col('*').name.suffix('_null_count'))\n        acc = pl.concat([acc, pl.concat([eeg_min, eeg_max, eeg_null], how = 'horizontal')],\n                        how = 'vertical')\n    return pl.concat([df, acc], how = 'horizontal')","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:50:35.389751Z","iopub.execute_input":"2024-01-17T04:50:35.391026Z","iopub.status.idle":"2024-01-17T04:50:35.400569Z","shell.execute_reply.started":"2024-01-17T04:50:35.390970Z","shell.execute_reply":"2024-01-17T04:50:35.398833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_eeg_stats = make_eeg_stats()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T04:50:45.404663Z","iopub.execute_input":"2024-01-17T04:50:45.405145Z","iopub.status.idle":"2024-01-17T05:26:28.745727Z","shell.execute_reply.started":"2024-01-17T04:50:45.405104Z","shell.execute_reply":"2024-01-17T05:26:28.743678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_eeg_stats)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T05:27:28.728284Z","iopub.execute_input":"2024-01-17T05:27:28.728758Z","iopub.status.idle":"2024-01-17T05:27:28.772961Z","shell.execute_reply.started":"2024-01-17T05:27:28.728712Z","shell.execute_reply":"2024-01-17T05:27:28.771325Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_eeg_stats.write_parquet('train_eeg_stats.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-01-17T05:38:44.992429Z","iopub.execute_input":"2024-01-17T05:38:44.992882Z","iopub.status.idle":"2024-01-17T05:38:45.257951Z","shell.execute_reply.started":"2024-01-17T05:38:44.992846Z","shell.execute_reply":"2024-01-17T05:38:45.256453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_eeg_stats.describe())","metadata":{"execution":{"iopub.status.busy":"2024-01-17T05:27:33.162166Z","iopub.execute_input":"2024-01-17T05:27:33.162628Z","iopub.status.idle":"2024-01-17T05:27:33.272739Z","shell.execute_reply.started":"2024-01-17T05:27:33.162588Z","shell.execute_reply":"2024-01-17T05:27:33.271118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_spec_dir = Path('/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms')","metadata":{"execution":{"iopub.status.busy":"2024-01-17T05:41:44.775754Z","iopub.execute_input":"2024-01-17T05:41:44.776285Z","iopub.status.idle":"2024-01-17T05:41:44.782296Z","shell.execute_reply.started":"2024-01-17T05:41:44.776249Z","shell.execute_reply":"2024-01-17T05:41:44.780836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_spec_subsamples(spec_id, offset):\n    path = train_spec_dir / f'{spec_id}.parquet'\n    df = pl.read_parquet(path)\n    return pl.concat(\n        [\n            get_spectrogram_subsamples(df, offset, region)\n            for region in ['LL', 'RL', 'LP', 'RP']\n        ],\n        how = 'horizontal',\n    )","metadata":{"execution":{"iopub.status.busy":"2024-01-17T05:42:55.968435Z","iopub.execute_input":"2024-01-17T05:42:55.968864Z","iopub.status.idle":"2024-01-17T05:42:55.975694Z","shell.execute_reply.started":"2024-01-17T05:42:55.968828Z","shell.execute_reply":"2024-01-17T05:42:55.974285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def make_spec_stats(df_train = df_train):\n    acc = pl.DataFrame()\n    df = df_train.select('spectrogram_id', 'spectrogram_label_offset_seconds')\n    for r in df.iter_rows():\n        spec_id = r[0]\n        offset = r[1]\n        spec = read_spec_subsamples(spec_id, offset)\n        spec_min = spec.min().select(pl.col('*').name.suffix('_min'))\n        spec_max = spec.max().select(pl.col('*').name.suffix('_max'))\n        spec_null = spec.null_count().select(pl.col('*').name.suffix('_null_count'))\n        acc = pl.concat([acc, pl.concat([spec_min, spec_max, spec_null], how = 'horizontal')],\n                        how = 'vertical')\n    return pl.concat([df, acc], how = 'horizontal')","metadata":{"execution":{"iopub.status.busy":"2024-01-17T05:43:00.511546Z","iopub.execute_input":"2024-01-17T05:43:00.512032Z","iopub.status.idle":"2024-01-17T05:43:00.521206Z","shell.execute_reply.started":"2024-01-17T05:43:00.511986Z","shell.execute_reply":"2024-01-17T05:43:00.519595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_spec_stats = make_spec_stats()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T05:43:01.199726Z","iopub.execute_input":"2024-01-17T05:43:01.201138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_spec_stats)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_spec_stats.write_parquet('train_spectrogram_stats.parquet')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display(df_spec_stats.describe())","metadata":{},"execution_count":null,"outputs":[]}]}