{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7457433,"sourceType":"competition"}],"dockerImageVersionId":30626,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Published on January 09, 2023. By Marília Prata, mpwolke.","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport plotly.graph_objs as go\nimport plotly.offline as py\nimport plotly.express as px\n\nimport dask.dataframe as dd\nimport dask.array as da\nfrom dask.diagnostics import ProgressBar\n\n#Ignore warnings\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-01-09T19:11:37.536332Z","iopub.execute_input":"2024-01-09T19:11:37.536708Z","iopub.status.idle":"2024-01-09T19:11:54.116874Z","shell.execute_reply.started":"2024-01-09T19:11:37.536680Z","shell.execute_reply":"2024-01-09T19:11:54.115828Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Competition Citation\n\n@misc{hms-harmful-brain-activity-classification,\n\n    author = {Jin Jing, Zhen Lin, Chaoqi Yang, Ashley Chow, Sohier Dane, Jimeng Sun, M. Brandon Westover},\n    \n    title = {HMS - Harmful Brain Activity Classification },\n    \n    publisher = {Kaggle},\n    year = {2024},\n    url = {https://kaggle.com/competitions/hms-harmful-brain-activity-classification}","metadata":{}},{"cell_type":"markdown","source":"#The Ictal-Interictal Continuum:\n\n![](https://i0.wp.com/emcrit.org/wp-content/uploads/2021/10/Screen-Shot-2021-11-02-at-2.32.31-PM.png?resize=614%2C474&ssl=1)https://emcrit.org/emcrit/eeg-terminology/\n\n\"This category includes rhythmic patterns that disrupt the background. They may contribute to poor mental status and perhaps contribute to neuronal injury. The four most mentioned are Generalized Rhythmic Delta Activity (GRDA), Lateralized Rhythmic Delta Activity (LRDA), Generalized Periodic Discharges (GPDs), and Lateralized Periodic Discharges (LPDs). There are others.\"\n\n\"Generalized Rhythmic Delta (GRDA):\n\n\"A non-specific pattern that is slow (<5 hz) and may be seen in profound encephalopathy, post-ictally or with inflammatory, degenerative, traumatic, or toxic-metabolic disorders.\"\n\nLateralized Rhythmic Delta (LRDA):\n\n\"Another lateralized slow (<5hz) pattern that usually reflects the presence of a focal lesion; associated with risk of acute seizures, especially non-convulsive status epilepticus.\"\n\nGeneralized Periodic Discharges (GPDs):\n\n\"Sharp waves or spikes that occur rhythmically in the right and left brain.  They may occur infrequently (0.5hz) or quite frequently (2hz). Tend to be seen in diffuse processes: toxic-metabolic encephalopathy, sepsis.  But may also be seen in herpes simplex virus encephalitis and autoimmune encephalopathies. Can be associated with NCSE.\"\n\nLateralized Periodic Discharges (LPDs):\n\n\"Lateralized sharp waves or spikes which made have associated slow waves.  Also can be seen infrequently or frequently (2hz).  Commonly encountered in stroke, intracerebral hemorrhage, subarachnoid hemorrhage, tumors, abscesses, Creutzfeldt-Jakob disease, herpes simplex virus and other infectious/autoimmune pathology.  LPDs are highly associated with seizures, especially in the setting of acute illness, metabolic disturbances or focal lesions.\"\n\nCitation: NeuroEMCrit Team (Casey & Neha). NeuroEMCrit – Demystifying the EEG Report. EMCrit Blog. Published on November 11, 2021. Accessed on January 9th 2024. Available at [https://emcrit.org/emcrit/eeg-terminology/ ].","metadata":{}},{"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/train.csv')\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-09T18:36:20.769392Z","iopub.execute_input":"2024-01-09T18:36:20.769752Z","iopub.status.idle":"2024-01-09T18:36:20.787100Z","shell.execute_reply.started":"2024-01-09T18:36:20.769723Z","shell.execute_reply":"2024-01-09T18:36:20.786175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#ACNS Critical Care EEG Terminology 2021\n\n[American Clinical Neurophysiology Society’s Standardized\nCritical Care EEG Terminology: 2021 Version](https://www.acns.org/UserFiles/file/ACNSStandardizedCriticalCareEEGTerminology_rev2021.pdf)\n\n2021 ACNS Critical Care EEG Terminology \n\nCONTENTS\n\n* EEG BACKGROUND\n*  SPORADIC EPILEPTIFORM DISCHARGES\n*  RHYTHMIC AND PERIODIC PATTERNS (RPPs)\n*  ELECTROGRAPHIC AND ELECTROCLINICAL SEIZURES [NEW, 2021]\n*  BRIEF POTENTIALLY ICTAL RHYTHMIC DISCHARGES (BIRDs) [NEW, 2021]\n*  ICTAL-INTERICTAL CONTINUUM (IIC) [NEW, 2021]\n*  MINIMUM REPORTING REQUIREMENTS\n*  OTHER TERMS\n\nABBREVIATION LIST\n\nACNS ¼ American Clinical Neurophysiology Society\n\nBI ¼ Bilateral Independent\n\nBIRDs ¼ Brief Potentially Ictal Rhythmic Discharges\n\nBTC ¼ Bilateral Tonic-Clonic\n\nCAPE ¼ Cyclic Alternating Pattern of Encephalopathy\n\nCCEMRC ¼ Critical Care EEG Monitoring Research\n\nConsortium\n\nECSz ¼ Electroclinical seizure\n\nECSE ¼ Electroclinical status epilepticus\n\nEDs ¼ Epileptiform Discharges\n\nEDB ¼ Extreme Delta Brush\n\nEEG ¼ Electroencephalography\n\nESE ¼ Electrographic status epilepticus\n\nESz ¼ Electrographic seizure\n\nG ¼ Generalized\n\nGPFA ¼ Generalized Paroxysmal Fast Activity\n\nHz ¼ Hertz (i.e., per second)\n\nIIC ¼ Ictal-Interictal Continuum\n\nL ¼ Lateralized\n\nMf ¼ Multifocal\n\nPDs ¼ Periodic Discharges\n\nRDA ¼ Rhythmic Delta Activity\n\nRPP ¼ Rhythmic or Periodic Pattern (i.e., PDs, RDA or SW)\n\nSE ¼ Status epilepticus\n\nSI ¼ Stimulus-Induced.\n\nSIRPIDs ¼ Stimulus-Induced Rhythmic, Periodic or IctalAppearing Discharges\n\nST ¼ Stimulus-Terminated\n\nSW ¼ Spike-and-wave or sharp-and-wave\n\nUI ¼ Unilateral Independent\n\n1 ¼ Plus ¼ Additional feature which renders the pattern\n\nmore ictal-appearing than the usual term without the plus\n\n1F ¼ Superimposed fast activity\n\n1R ¼ Superimposed rhythmic activity\n\n1S ¼ Superimposed sharp waves or spikes, or sharply contoured\n\nhttps://www.acns.org/UserFiles/file/ACNSStandardizedCriticalCareEEGTerminology_rev2021.pdf","metadata":{}},{"cell_type":"code","source":"test = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/test.csv')\ntest.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-09T18:36:42.411918Z","iopub.execute_input":"2024-01-09T18:36:42.412638Z","iopub.status.idle":"2024-01-09T18:36:42.430873Z","shell.execute_reply.started":"2024-01-09T18:36:42.412553Z","shell.execute_reply":"2024-01-09T18:36:42.429773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.read_csv('/kaggle/input/hms-harmful-brain-activity-classification/sample_submission.csv')\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-09T18:37:23.411949Z","iopub.execute_input":"2024-01-09T18:37:23.412368Z","iopub.status.idle":"2024-01-09T18:37:23.434103Z","shell.execute_reply.started":"2024-01-09T18:37:23.412336Z","shell.execute_reply":"2024-01-09T18:37:23.432906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Read one electroencephalography (EEG) parquet with Dask","metadata":{}},{"cell_type":"code","source":"ddf = dd.read_parquet(\"/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/1712674008.parquet\")","metadata":{"execution":{"iopub.status.busy":"2024-01-09T19:12:01.981982Z","iopub.execute_input":"2024-01-09T19:12:01.982409Z","iopub.status.idle":"2024-01-09T19:12:02.184277Z","shell.execute_reply.started":"2024-01-09T19:12:01.982372Z","shell.execute_reply":"2024-01-09T19:12:02.183413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nlen(ddf)","metadata":{"execution":{"iopub.status.busy":"2024-01-09T19:20:13.379275Z","iopub.execute_input":"2024-01-09T19:20:13.380425Z","iopub.status.idle":"2024-01-09T19:20:13.789582Z","shell.execute_reply.started":"2024-01-09T19:20:13.380373Z","shell.execute_reply":"2024-01-09T19:20:13.788577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ddf.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-09T19:51:30.485922Z","iopub.execute_input":"2024-01-09T19:51:30.486376Z","iopub.status.idle":"2024-01-09T19:51:30.527880Z","shell.execute_reply.started":"2024-01-09T19:51:30.486340Z","shell.execute_reply":"2024-01-09T19:51:30.526975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ddf.EKG.describe().compute()","metadata":{"execution":{"iopub.status.busy":"2024-01-09T19:24:26.095715Z","iopub.execute_input":"2024-01-09T19:24:26.096086Z","iopub.status.idle":"2024-01-09T19:24:26.152835Z","shell.execute_reply.started":"2024-01-09T19:24:26.096058Z","shell.execute_reply":"2024-01-09T19:24:26.151862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ddf.Fp1.describe().compute()","metadata":{"execution":{"iopub.status.busy":"2024-01-09T19:38:04.107507Z","iopub.execute_input":"2024-01-09T19:38:04.108281Z","iopub.status.idle":"2024-01-09T19:38:04.153203Z","shell.execute_reply.started":"2024-01-09T19:38:04.108243Z","shell.execute_reply":"2024-01-09T19:38:04.152566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Spectogram Parquet file - Reading them without Dask","metadata":{}},{"cell_type":"code","source":"spec_train = pd.read_parquet('../input/hms-harmful-brain-activity-classification/train_spectrograms/1000086677.parquet')","metadata":{"execution":{"iopub.status.busy":"2024-01-09T19:41:23.080982Z","iopub.execute_input":"2024-01-09T19:41:23.081942Z","iopub.status.idle":"2024-01-09T19:41:23.145511Z","shell.execute_reply.started":"2024-01-09T19:41:23.081904Z","shell.execute_reply":"2024-01-09T19:41:23.144612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spec_train.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-09T19:41:51.944205Z","iopub.execute_input":"2024-01-09T19:41:51.945321Z","iopub.status.idle":"2024-01-09T19:41:51.976375Z","shell.execute_reply.started":"2024-01-09T19:41:51.945270Z","shell.execute_reply":"2024-01-09T19:41:51.975345Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#3rd row, 3rd colunm\n\nspec_train.iloc[2,2]","metadata":{"execution":{"iopub.status.busy":"2024-01-09T19:44:48.984391Z","iopub.execute_input":"2024-01-09T19:44:48.984758Z","iopub.status.idle":"2024-01-09T19:44:48.991272Z","shell.execute_reply.started":"2024-01-09T19:44:48.984728Z","shell.execute_reply":"2024-01-09T19:44:48.990552Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#Almost 3 hours trying to find out to deliver a Spectogram with Parquet Files: TY Corey Lehmann\n\nI've alredy learned spectograms with wav/audio/hdf5/oog files though I was suffering to plot with Parquet files.  Anyway, my head is acking so I've just copied Corey's Lehmann snippets. And changed the cmap to Spectral, In fact, I should have applied Inferno, cause that's how I'm felling now. Please vote Corey's:\n\nhttps://www.kaggle.com/code/clehmann10/plot-spectrograms/notebook","metadata":{}},{"cell_type":"code","source":"#By Corey Lehmann https://www.kaggle.com/code/clehmann10/plot-spectrograms/notebook\n\ncomp_path = '/kaggle/input/hms-harmful-brain-activity-classification'\ntrain_spect_dir = '/'.join([comp_path, 'train_spectrograms'])\ntrain_spect_path_list = [entry.path for entry in os.scandir(train_spect_dir)]","metadata":{"execution":{"iopub.status.busy":"2024-01-09T21:10:31.128465Z","iopub.execute_input":"2024-01-09T21:10:31.128880Z","iopub.status.idle":"2024-01-09T21:10:31.146211Z","shell.execute_reply.started":"2024-01-09T21:10:31.128846Z","shell.execute_reply":"2024-01-09T21:10:31.144972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#By Corey Lehmann https://www.kaggle.com/code/clehmann10/plot-spectrograms/notebook\n\ndef plot_spectrogram(spectrogram_path):\n    sample_spect = pd.read_parquet(spectrogram_path)\n    \n    split_spect = {\n        \"LL\": sample_spect.filter(regex='^LL', axis=1),\n        \"RL\": sample_spect.filter(regex='^RL', axis=1),\n        \"RP\": sample_spect.filter(regex='^RP', axis=1),\n        \"LP\": sample_spect.filter(regex='^LP', axis=1),\n    }\n    \n    fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(15, 12))\n    axes = axes.flatten()\n    label_interval = 5\n    for i, split_name in enumerate(split_spect.keys()):\n        ax = axes[i]\n        img = ax.imshow(np.log(split_spect[split_name]).T, cmap='Spectral', aspect='auto', origin='lower')  # You can choose any colormap (cmap) that suits your preferences\n        cbar = fig.colorbar(img, ax=ax)\n        cbar.set_label('Log(Value)')\n        ax.set_title(split_name)\n        ax.set_ylabel(\"Frequency (Hz)\")\n        ax.set_xlabel(\"Time\")\n\n        ax.set_yticks(np.arange(len(split_spect[split_name].columns)))\n        ax.set_yticklabels([column_name[3:] for column_name in split_spect[split_name].columns])\n        frequencies = [column_name[3:] for column_name in split_spect[split_name].columns]\n        ax.set_yticks(np.arange(0, len(split_spect[split_name].columns), label_interval))\n        ax.set_yticklabels(frequencies[::label_interval])\n    plt.tight_layout()\n    plt.show()\n    \nplot_spectrogram(train_spect_path_list[0])","metadata":{"execution":{"iopub.status.busy":"2024-01-09T21:12:12.433946Z","iopub.execute_input":"2024-01-09T21:12:12.434418Z","iopub.status.idle":"2024-01-09T21:12:15.919752Z","shell.execute_reply.started":"2024-01-09T21:12:12.434377Z","shell.execute_reply":"2024-01-09T21:12:15.918807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Another Message: \"It looks like you ran out of disk space. Upgrade to run on larger machines with more memory. You can export your notebook to get started.\"","metadata":{}},{"cell_type":"markdown","source":"#Seizures Competitions on Kaggle\n\n[UPenn and Mayo Clinic's Seizure Detection Challenge](https://www.kaggle.com/competitions/seizure-detection)\n\n[American Epilepsy Society Seizure Prediction Challenge](https://www.kaggle.com/competitions/seizure-prediction)\n\n[Melbourne University AES/MathWorks/NIH Seizure Prediction](https://www.kaggle.com/competitions/melbourne-university-seizure-prediction)\n\n[Cornell ECE/BME 5040 Project](https://www.kaggle.com/competitions/cornell-ecebme-5040-project)\n\n[EEG Seizure Analysis](https://www.kaggle.com/competitions/eeg-seizure-analysis)","metadata":{}},{"cell_type":"markdown","source":"#Acknowledgements:\n\nCorey Lehmann https://www.kaggle.com/code/clehmann10/plot-spectrograms/notebook","metadata":{}}]}