{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"}],"dockerImageVersionId":30635,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport time\nimport matplotlib.pyplot as plt\nimport seaborn as sns","metadata":{"execution":{"iopub.status.busy":"2024-01-11T14:08:24.778560Z","iopub.execute_input":"2024-01-11T14:08:24.779088Z","iopub.status.idle":"2024-01-11T14:08:26.116773Z","shell.execute_reply.started":"2024-01-11T14:08:24.779034Z","shell.execute_reply":"2024-01-11T14:08:26.115406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction\n\nThis notebook gives a basic introduction into the data of the Harmful Brain Activity competion, describing how you get from the training data to making an actual submission.\n\nIn this competition we need to analyse EEG and Spectograms to predict the probablities of how experts have classified each sample of brain activity.","metadata":{}},{"cell_type":"markdown","source":"# Training Data\n\nThe easiest way to understand this competition is by looking at the training data:","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/train.csv\")\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-11T14:08:27.622167Z","iopub.execute_input":"2024-01-11T14:08:27.622872Z","iopub.status.idle":"2024-01-11T14:08:28.068053Z","shell.execute_reply.started":"2024-01-11T14:08:27.622826Z","shell.execute_reply":"2024-01-11T14:08:28.066476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each row of the training CSV file contains a reference to an EEG file and to a Spectrogram, along with time offsets into each of these images, to the point in the image that corresponds to the data for the current row.
\n\nAdditionally the last 6 columns (the \"vote\" columns) contain the target values that we're trying to predict. These represent the number of experts that have chosen each classification.
\n\nSo, for example, for the first row entry, 3 experts have analysed this data and all have decided that the data shows that the patient has suffered a seizure.
","metadata":{}},{"cell_type":"markdown","source":"## Training EEG\n\nFor the first row in the training data we can look at the matching EEG data - which in this case has ID = 1628180742","metadata":{}},{"cell_type":"code","source":"first_eeg_df = train_df[train_df['eeg_id']==1628180742]\nfirst_eeg_df","metadata":{"execution":{"iopub.status.busy":"2024-01-11T14:43:31.055149Z","iopub.execute_input":"2024-01-11T14:43:31.055673Z","iopub.status.idle":"2024-01-11T14:43:31.080024Z","shell.execute_reply.started":"2024-01-11T14:43:31.055635Z","shell.execute_reply":"2024-01-11T14:43:31.078639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the training CSV if we look at all the rows that match this EEG file name we can see that there are 9 predictions using this EEG, with each prediction being based on a 50 second long period.
\n\nSo the first prediction will be from the start of the EEG (0.0 seconds) until the 50 second mark, the second will be from 6 seconds until 56 seconds and the last prediction is from 40 seconds until 90 seconds.\n\n\nHere's the matching EEG data...","metadata":{}},{"cell_type":"code","source":"train_eeg = \"/kaggle/input/hms-harmful-brain-activity-classification/train_eegs/1628180742.parquet\"\ntrain_eeg_df = pd.read_parquet(train_eeg)\nprint(train_eeg_df.shape)\ntrain_eeg_df.head(3)","metadata":{"execution":{"iopub.status.busy":"2024-01-11T14:28:05.278074Z","iopub.execute_input":"2024-01-11T14:28:05.278564Z","iopub.status.idle":"2024-01-11T14:28:05.513336Z","shell.execute_reply.started":"2024-01-11T14:28:05.278527Z","shell.execute_reply":"2024-01-11T14:28:05.512031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The rows represent each EEG sample and there are 200 EEG samples taken per second, so the total number of seconds in this EEG data = (18000/200) = 90 seconds. This matches with where we expect the 50 second window of the last prediction to end.
\n\n* Each column represents a different location on the skull, from where the EEG readings have been taken, except the last column 'EKG' which represents a heart reading. The various positions on the skull are as shown below...
\n
\n
\n
\n\"drawing\"\n\n(Image by: By トマトン124 (talk) - Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=10489987)\n
","metadata":{}},{"cell_type":"markdown","source":"We can look at the EEG reading for the first location 'Fp1'. In the plot below, the green vertical lines represent the start of each prediction period and the red lines represent the period ends","metadata":{}},{"cell_type":"code","source":"f = 'Fp1'\nplt.figure(figsize=(12,3))\nplt.plot(train_eeg_df[f])\nplt.title(f)\nplt.grid()\nfor i,row in first_eeg_df.iterrows():\n plt.axvline(x=(row['eeg_label_offset_seconds']*200),color='green') \n plt.axvline(x=((row['eeg_label_offset_seconds']+50)*200),color='red') \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-11T14:47:59.920488Z","iopub.execute_input":"2024-01-11T14:47:59.920965Z","iopub.status.idle":"2024-01-11T14:48:00.322177Z","shell.execute_reply.started":"2024-01-11T14:47:59.920930Z","shell.execute_reply":"2024-01-11T14:48:00.320856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"So, for the very first prediction in the training data, it represents the time period from 0 to 50 seconds, or from 0 to (50*200)= 10000 samples and is shown in the 'Fp1' plot below:","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12,3))\nplt.plot(train_eeg_df[f][:10000])\nplt.title(f)\nplt.grid()\nrow = first_eeg_df.iloc[0]\nplt.axvline(x=(row['eeg_label_offset_seconds']*200),color='green') \nplt.axvline(x=((row['eeg_label_offset_seconds']+50)*200),color='red') \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-11T14:53:59.672605Z","iopub.execute_input":"2024-01-11T14:53:59.673089Z","iopub.status.idle":"2024-01-11T14:54:00.038947Z","shell.execute_reply.started":"2024-01-11T14:53:59.673052Z","shell.execute_reply":"2024-01-11T14:54:00.037515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Spectrogram\n\nWe can examine the Spectograms in exactly the same way as the EEGs.","metadata":{}},{"cell_type":"code","source":"first_spectrogram_df = train_df[train_df['spectrogram_id']==353733]\nfirst_spectrogram_df","metadata":{"execution":{"iopub.status.busy":"2024-01-11T14:57:46.440983Z","iopub.execute_input":"2024-01-11T14:57:46.441473Z","iopub.status.idle":"2024-01-11T14:57:46.467362Z","shell.execute_reply.started":"2024-01-11T14:57:46.441437Z","shell.execute_reply":"2024-01-11T14:57:46.466087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As with the EEGs, each Spectrogram can contain data relating to multiple predictions. So, as with the first EEG data, the first Spectrogram also contains data from 9 predictions.
\n\nOpening the Spectrogram referenced on the first line of the training CSV, gives the following data:","metadata":{}},{"cell_type":"code","source":"train_spect = \"/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/353733.parquet\"\ntrain_spect_df = pd.read_parquet(train_spect).dropna()\nprint(train_spect_df.shape)\ntrain_spect_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-11T15:02:10.106623Z","iopub.execute_input":"2024-01-11T15:02:10.107097Z","iopub.status.idle":"2024-01-11T15:02:10.214152Z","shell.execute_reply.started":"2024-01-11T15:02:10.107059Z","shell.execute_reply":"2024-01-11T15:02:10.212757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The columns of the Spectogram data represent the location at which the reading was taken and frequency in Hertz. The four possible recording locations are:\n\n* LL = left lateral\n* RL = right lateral\n* LP = left parasagittal\n* RP = right parasagittal\n\nAs shown in the image below, where the figure shows three major planes annotated as follows:
\n
\n* a: Axial or horizontal plane (blue), that contains the lateral axis \n and also the medial axis \n* c: Coronal plane (green), containing the axial axis and the lateral axis\n* s: Sagittal plane (red), containing the axial axis and the medial axis\n\n
\nThere also is:
\n\n* e: The eye position showing the anterior end of the brain
\n* p: an example of a parasagittal plane (yellow); \n parasagittal planes comprise the class of planes \n parallel to (and therefore lateral to) \n the sagittal plane.\n\n
\n
\n
\n \n \n\n(Image by: https://commons.wikimedia.org/wiki/File:Human_brain_anatomical_planes_letter_annotations.jpg)\n
\n","metadata":{}},{"cell_type":"markdown","source":"For each of these 4 regions we can plot the Spectrogram to see how the frequencies change over time.","metadata":{}},{"cell_type":"code","source":"def plot_spectrogram(spectrogram_path):\n \"\"\" Code from: https://www.kaggle.com/code/clehmann10/plot-spectrograms\n \"\"\"\n sample_spect = pd.read_parquet(spectrogram_path)\n \n split_spect = {\n \"LL\": sample_spect.filter(regex='^LL', axis=1),\n \"RL\": sample_spect.filter(regex='^RL', axis=1),\n \"RP\": sample_spect.filter(regex='^RP', axis=1),\n \"LP\": sample_spect.filter(regex='^LP', axis=1),\n }\n \n fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(15, 12))\n axes = axes.flatten()\n label_interval = 5\n for i, split_name in enumerate(split_spect.keys()):\n ax = axes[i]\n img = ax.imshow(np.log(split_spect[split_name]).T, cmap='viridis', aspect='auto', origin='lower') \n cbar = fig.colorbar(img, ax=ax)\n cbar.set_label('Log(Value)')\n ax.set_title(split_name)\n ax.set_ylabel(\"Frequency (Hz)\")\n ax.set_xlabel(\"Time\")\n \n ax.set_yticks(np.arange(len(split_spect[split_name].columns)))\n ax.set_yticklabels([column_name[3:] for column_name in split_spect[split_name].columns])\n frequencies = [column_name[3:] for column_name in split_spect[split_name].columns]\n ax.set_yticks(np.arange(0, len(split_spect[split_name].columns), label_interval))\n ax.set_yticklabels(frequencies[::label_interval]) \n \n plt.tight_layout()\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-01-11T15:45:05.759552Z","iopub.execute_input":"2024-01-11T15:45:05.760076Z","iopub.status.idle":"2024-01-11T15:45:05.774435Z","shell.execute_reply.started":"2024-01-11T15:45:05.760037Z","shell.execute_reply":"2024-01-11T15:45:05.772752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Note: the time axis should be multiplied by 2, to match with the time column in the Spectrogram data\nplot_spectrogram(train_spect)","metadata":{"execution":{"iopub.status.busy":"2024-01-11T15:45:09.150612Z","iopub.execute_input":"2024-01-11T15:45:09.151077Z","iopub.status.idle":"2024-01-11T15:45:13.745610Z","shell.execute_reply.started":"2024-01-11T15:45:09.151042Z","shell.execute_reply":"2024-01-11T15:45:13.744156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the case of the Spectrograms, they contain a 10 minute window that is centered at the same time as the associated EEG image. The 'time' column represents the time in seconds. So the first row of the train CSV file is for the Spectogram period from 0 to (10*60)= 600 seconds. Whereas the last is from 40 to 640 seconds. If we look at the last row of this Spectrogram data we can see that its time matches with this:\n\n(Note - in the spectrograms shown above, their time axis should be multiplied by 2 to match with the time column of the spectrogram data)","metadata":{}},{"cell_type":"code","source":"train_spect_df.tail()","metadata":{"execution":{"iopub.status.busy":"2024-01-11T15:35:29.063718Z","iopub.execute_input":"2024-01-11T15:35:29.064137Z","iopub.status.idle":"2024-01-11T15:35:29.105877Z","shell.execute_reply.started":"2024-01-11T15:35:29.064103Z","shell.execute_reply":"2024-01-11T15:35:29.104426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission\n\nAs with the training data shown above, we want to predict the how the experts categorize each sample in the test data. However, in the training data, for each row the number of experts that made each prediction is given, whereas in the submission we want to give the probability for each type of prediction.\n\nThis can be seen by looking at the sample submission file:","metadata":{}},{"cell_type":"code","source":"sub_df = pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/sample_submission.csv\")\nsub_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-11T15:51:56.687989Z","iopub.execute_input":"2024-01-11T15:51:56.688455Z","iopub.status.idle":"2024-01-11T15:51:56.739606Z","shell.execute_reply.started":"2024-01-11T15:51:56.688421Z","shell.execute_reply":"2024-01-11T15:51:56.738244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This sample row gives the ID of the associated EEG file to look at when making the prediction and the predicted probabilities for each category of diagnosis (the description for each of these categories can be found in the competition description: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/overview)\n\nSo, in this case, each vote is set to 0.1667 = 1/6, so we're saying that each category is equally likely.\n\nTo make actual predictions we take the 'test.csv' file and, for each row, examine the specified EEG and Spectrogram files, before predicting the probability for each of the six categories.","metadata":{}},{"cell_type":"markdown","source":"In the given test file there's only a single row - but this expands to have multiple rows when you submit your notebook.","metadata":{}},{"cell_type":"code","source":"test_df = pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/test.csv\")\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-11T15:57:59.734478Z","iopub.execute_input":"2024-01-11T15:57:59.735021Z","iopub.status.idle":"2024-01-11T15:57:59.755427Z","shell.execute_reply.started":"2024-01-11T15:57:59.734966Z","shell.execute_reply":"2024-01-11T15:57:59.754147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"If we imaging that there are a total of 60 experts making predictions and they have an even split over each of the 6 possible voting categories, then we'll have the following votes and probabilities:","metadata":{}},{"cell_type":"code","source":"seizure_vote = 10\nlpd_vote = 10\ngpd_vote = 10\nlrda_vote = 10\ngrda_vote = 10\nother_vote = 10\n\ntotal_vote = seizure_vote + lpd_vote + gpd_vote + lrda_vote + grda_vote + other_vote\npreds = [seizure_vote/total_vote, lpd_vote/total_vote, gpd_vote/total_vote, lrda_vote/total_vote, grda_vote/total_vote, other_vote/total_vote]\n\nprint(f\"Total Number of Votes = {total_vote}\")\nprint(f\"Predictions = {preds}\")","metadata":{"execution":{"iopub.status.busy":"2024-01-11T16:04:49.301929Z","iopub.execute_input":"2024-01-11T16:04:49.302386Z","iopub.status.idle":"2024-01-11T16:04:49.311612Z","shell.execute_reply.started":"2024-01-11T16:04:49.302341Z","shell.execute_reply":"2024-01-11T16:04:49.309837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can then use these predictions to generate the same entry for each row in the test data:","metadata":{}},{"cell_type":"code","source":"# generate the same predictions for every entry in the test data\ndf = pd.DataFrame(columns=sub_df.columns)\nfor i, row in test_df.iterrows(): \n df.loc[i] = [str(row['eeg_id'])] + preds\n \ndf ","metadata":{"execution":{"iopub.status.busy":"2024-01-11T16:06:12.335677Z","iopub.execute_input":"2024-01-11T16:06:12.336135Z","iopub.status.idle":"2024-01-11T16:06:12.358284Z","shell.execute_reply.started":"2024-01-11T16:06:12.336091Z","shell.execute_reply":"2024-01-11T16:06:12.356736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To get a slightly better score we can take the mean vote ratio from the training data and use these values for the predctions (see: https://www.kaggle.com/code/seshurajup/eda-train-csv):\n\nseizure_vote 0.152810
\nlpd_vote 0.142456
\ngpd_vote 0.104062
\nlrda_vote 0.065407
\ngrda_vote 0.114851
\nother_vote 0.420414
","metadata":{}},{"cell_type":"code","source":"preds = [0.152810,0.142456,0.104062,0.065407,0.114851,0.420414]\ndf = pd.DataFrame(columns=sub_df.columns)\nfor i, row in test_df.iterrows(): \n df.loc[i] = [str(row['eeg_id'])] + preds\n \ndf ","metadata":{"execution":{"iopub.status.busy":"2024-01-11T16:09:20.991027Z","iopub.execute_input":"2024-01-11T16:09:20.991552Z","iopub.status.idle":"2024-01-11T16:09:21.015253Z","shell.execute_reply.started":"2024-01-11T16:09:20.991512Z","shell.execute_reply":"2024-01-11T16:09:21.013235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# write the final submission file\ndf.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2024-01-11T16:06:21.870545Z","iopub.execute_input":"2024-01-11T16:06:21.871616Z","iopub.status.idle":"2024-01-11T16:06:21.882375Z","shell.execute_reply.started":"2024-01-11T16:06:21.871563Z","shell.execute_reply":"2024-01-11T16:06:21.881009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nWe've looked at the basics of how to get from the training data to making an actual submission - although the submission is only a baseline. To get a real submission you'll need to take the EEG and Spectrogram data and do a proper prediction, but hopefully this will get you on your way.\n\nIf you've found this notebook useful please consider giving it a vote. Thanks!","metadata":{}}]}