{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7378374,"sourceType":"datasetVersion","datasetId":4287749},{"sourceId":7392775,"sourceType":"datasetVersion","datasetId":4297782}],"dockerImageVersionId":30626,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n

Harmful Brain Activity Classification

\n

- data understanding -

\n\n> ๐Ÿ“Œ **Competition Scope**: Detect and classify seizures and other types of harmful brain activity in electroencephalography (EEG) data. Even experts find this to be a challenging task and *often disagree* about the correct labels.\n\n### About the Problem\n\n**There are 6 patterns to be identified**:\n* seizure (SZ)\n* generalized periodic discharges (GPD)\n* lateralized periodic discharges (LPD)\n* lateralized rhythmic delta activity (LRDA)\n* generalized rhythmic delta activity (GRDA)\n* other\n\nThe annotations were made by a group of experts, *however* the challenge is that not even the experts can fully agree on a case 100% of the time. Hence, the competition creates a second set of labels:\n* where there are high levels of agreement => โ€œidealizedโ€ patterns\n* where ~1-2 experts give a label as โ€œotherโ€ and ~1-2 give one of the remaining five labels => โ€œprotoโ€ patterns\n* where experts are approximately split between 2 of the 5 named patterns => โ€œedgeโ€ cases\n\n\n\n> ๐Ÿ“Œ **Note**: so there are patterns that look both like a Seizure or like an LPD or GPD. There are patterns that look like a LRDA and a GRDA. And so on.\n\n### โ—‹ Libraries","metadata":{}},{"cell_type":"code","source":"# general\nimport os\nimport gc\nimport wandb\nimport random\nimport math\nfrom glob import glob\nfrom tqdm import tqdm\nfrom time import time\nfrom pprint import pprint\nimport warnings\nimport pandas as pd\nimport numpy as np\nfrom scipy.signal import spectrogram\n\n# visuals\nimport seaborn as sns\nimport matplotlib as mpl\nfrom matplotlib import cm\nimport matplotlib.patches as patches\nimport matplotlib.pyplot as plt\nplt.rcParams.update({'font.size': 18})\n\n# env check\nwarnings.filterwarnings('ignore')\nos.environ[\"WANDB_SILENT\"] = \"true\"\nCOMP_ID = '2024_hms'\nCONFIG = {'competition': COMP_ID, '_wandb_kernel': 'aot', \"source_type\": \"artifact\"}\n\n# color\nclass clr:\n S = '\\033[1m' + '\\033[90m'\n E = '\\033[0m'\n \nmy_colors = [\"#FECF72\", \"#DB8C0F\", \"#E39A7F\",\n \"#D87AA0\", \"#91D5DF\", \"#7BAEC8\",]\n\nprint(clr.S+\"Notebook Color Schemes:\"+clr.E)\nsns.palplot(sns.color_palette(my_colors))\nplt.show()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-17T15:51:32.681761Z","iopub.execute_input":"2024-01-17T15:51:32.682021Z","iopub.status.idle":"2024-01-17T15:51:34.988015Z","shell.execute_reply.started":"2024-01-17T15:51:32.681997Z","shell.execute_reply":"2024-01-17T15:51:34.986560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ๐Ÿ W&B Fork & Run\n\nIn order to run this notebook you will need to input your own **secret API key** within the `! wandb login $secret_value_0` line. \n\n๐Ÿ**How do you get your own API key?**\n\nSuper simple! Go to **https://wandb.ai/site** -> Login -> Click on your profile in the top right corner -> Settings -> Scroll down to API keys -> copy your very own key (for more info check [this amazing notebook for ML Experiment Tracking on Kaggle](https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases)).\n\n
","metadata":{}},{"cell_type":"code","source":"# ๐Ÿ secrets\nfrom kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nwandb_key = user_secrets.get_secret(\"wandb_key\")\n\n! wandb login $wandb_key","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:51:34.991153Z","iopub.execute_input":"2024-01-17T15:51:34.992382Z","iopub.status.idle":"2024-01-17T15:51:38.822661Z","shell.execute_reply.started":"2024-01-17T15:51:34.992327Z","shell.execute_reply":"2024-01-17T15:51:38.821368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โ—‹ Helper Functions Below","metadata":{}},{"cell_type":"code","source":"# === data discover ===\n\ndef jitter(values,j):\n return values + np.random.normal(j,0.05,values.shape)\n\n\ndef find_rectangles(arr):\n '''\n return indices where the rectangle starts and ends\n '''\n rectangles = []\n start = None\n for i, val in enumerate(arr):\n if val == 3:\n if start is None:\n start = i\n elif start is not None:\n rectangles.append((start, i - 1))\n start = None\n if start is not None:\n rectangles.append((start, len(arr) - 1))\n return rectangles\n\n\ndef get_general_info(df, desc=None):\n \n # ๐Ÿ new exp\n run = wandb.init(project=COMP_ID, name=f'{desc}_data_summary', config=CONFIG)\n\n print(clr.S+\"--- General Info ---\"+clr.E)\n print(clr.S+\"Data Shape:\"+clr.E, df.shape)\n print(clr.S+\"Data Cols:\"+clr.E, df.columns.tolist())\n print(clr.S+\"Total No. of Cols:\"+clr.E, len(df.columns.tolist()))\n print(clr.S+\"No. Missing Values:\"+clr.E, df.isna().sum().sum())\n print(clr.S+\"Columns with missing data:\"+clr.E, \"\\n\",\n df.isna().sum()[df.isna().sum() != 0], \"\\n\")\n\n for col in df.columns:\n if is_string_dtype(df[col]):\n print(clr.S+f\"--- {col} --- is type string\"+clr.E)\n print(clr.S+f\"[nunique] {col}:\"+clr.E, \n df[col].nunique())\n \n elif is_numeric_dtype(df[col]):\n print(clr.S+f\"--- {col} --- is type numeric\"+clr.E)\n print(clr.S+f\"[describe] {col}:\"+clr.E, \"\\n\",\n df[col].describe())\n \n # log data\n wandb.log\n (\n {\"data_shape\": len(df),\n \"missing_values\": df.isna().sum().sum()\n }\n )\n wandb.finish()\n print(\"๐Ÿ Info saved to dashboard.\")\n \n\ndef get_missing_values_plot(df):\n '''\n Plots missing values barchart for a given dataframe.\n '''\n \n # count missing values\n missing_counts = df.isnull().sum().reset_index()\\\n .sort_values(0, ascending=False)\\\n .reset_index(drop=True)\n missing_counts.columns = [\"col_name\", \"missing_count\"]\n\n # plot\n plt.figure(figsize=(24, 16))\n axs = sns.barplot(y=missing_counts.col_name, x=missing_counts.missing_count, \n color=my_colors[0])\n show_values_on_bars(axs, h_v=\"h\", space=0.4)\n plt.xlabel('no. missing values', size=20, weight=\"bold\")\n plt.ylabel('column name', size=20, weight=\"bold\")\n plt.title('Missing Values', size=22, weight=\"bold\")\n plt.show();\n \n \n# === plots ===\ndef show_values_on_bars(axs, h_v=\"v\", space=0.4):\n '''Plots the value at the end of the a seaborn barplot.\n axs: the ax of the plot\n h_v: weather or not the barplot is vertical/ horizontal'''\n \n def _show_on_single_plot(ax):\n if h_v == \"v\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() / 2\n _y = p.get_y() + p.get_height()\n value = int(p.get_height())\n ax.text(_x, _y, format(value, ','), ha=\"center\") \n elif h_v == \"h\":\n for p in ax.patches:\n _x = p.get_x() + p.get_width() + float(space)\n _y = p.get_y() + p.get_height()\n value = int(p.get_width())\n ax.text(_x, _y, format(value, ','), ha=\"left\")\n\n if isinstance(axs, np.ndarray):\n for idx, ax in np.ndenumerate(axs):\n _show_on_single_plot(ax)\n else:\n _show_on_single_plot(axs)\n \n \n# === ๐Ÿ w&b ===\ndef save_dataset_artifact(run_name, artifact_name, path, data_type=\"dataset\"):\n '''Saves dataset to W&B Artifactory.\n run_name: name of the experiment\n artifact_name: under what name should the dataset be stored\n path: path to the dataset'''\n \n run = wandb.init(project=COMP_ID, \n name=run_name, \n config=CONFIG)\n artifact = wandb.Artifact(name=artifact_name, \n type=data_type)\n artifact.add_file(path)\n\n wandb.log_artifact(artifact)\n wandb.finish()\n print(f\"๐ŸArtifact {artifact_name} has been saved successfully.\")\n \n \ndef create_wandb_plot(x_data=None, y_data=None, x_name=None, y_name=None, title=None, log=None, plot=\"line\"):\n '''Create and save lineplot/barplot in W&B Environment.\n x_data & y_data: Pandas Series containing x & y data\n x_name & y_name: strings containing axis names\n title: title of the graph\n log: string containing name of log'''\n \n data = [[label, val] for (label, val) in zip(x_data, y_data)]\n table = wandb.Table(data=data, columns = [x_name, y_name])\n \n if plot == \"line\":\n wandb.log({log : wandb.plot.line(table, x_name, y_name, title=title)})\n elif plot == \"bar\":\n wandb.log({log : wandb.plot.bar(table, x_name, y_name, title=title)})\n elif plot == \"scatter\":\n wandb.log({log : wandb.plot.scatter(table, x_name, y_name, title=title)})\n \n \ndef create_wandb_hist(x_data=None, x_name=None, title=None, log=None):\n '''Create and save histogram in W&B Environment.\n x_data: Pandas Series containing x values\n x_name: strings containing axis name\n title: title of the graph\n log: string containing name of log'''\n \n data = [[x] for x in x_data]\n table = wandb.Table(data=data, columns=[x_name])\n wandb.log({log : wandb.plot.histogram(table, x_name, title=title)})","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2024-01-17T15:51:38.824704Z","iopub.execute_input":"2024-01-17T15:51:38.825058Z","iopub.status.idle":"2024-01-17T15:51:38.856753Z","shell.execute_reply.started":"2024-01-17T15:51:38.825031Z","shell.execute_reply":"2024-01-17T15:51:38.855814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ log cover\nrun = wandb.init(project=COMP_ID, name='cover', config=CONFIG)\ncover = plt.imread(\"/kaggle/input/hmd-additional-data/leonardo_ai_cover.jpg\")\nwandb.log({\"cover\": wandb.Image(cover)})\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:51:38.859580Z","iopub.execute_input":"2024-01-17T15:51:38.859896Z","iopub.status.idle":"2024-01-17T15:52:39.671237Z","shell.execute_reply.started":"2024-01-17T15:51:38.859872Z","shell.execute_reply":"2024-01-17T15:52:39.670482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Understanding the train columns\n\n**train.csv**:\n* all `_vote` cols are our target columns\n* `eeg_id` marks one recording (17,089 in total)\n* `spectrogram_id` represents the \"training\" data available to predict the classification - there are 11,138 spectrograms in total available in the training set\n* `patient_id` is the ID of the patient who this data is about - 1950 in total\n* `expert_consensus` contains the votes for each of these subsegments - most for seizures.","metadata":{}},{"cell_type":"code","source":"# ๐Ÿ\nrun = wandb.init(project=COMP_ID, name='understanding', config=CONFIG)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:52:39.672315Z","iopub.execute_input":"2024-01-17T15:52:39.672574Z","iopub.status.idle":"2024-01-17T15:53:11.656515Z","shell.execute_reply.started":"2024-01-17T15:52:39.672550Z","shell.execute_reply":"2024-01-17T15:53:11.655649Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/train.csv\")\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:11.657538Z","iopub.execute_input":"2024-01-17T15:53:11.657858Z","iopub.status.idle":"2024-01-17T15:53:12.587607Z","shell.execute_reply.started":"2024-01-17T15:53:11.657832Z","shell.execute_reply":"2024-01-17T15:53:12.586798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Train shape:\", train.shape, \"\\n\")\nprint(\"Unique eeg_ids: \", train.eeg_id.nunique())\nprint(train.groupby(\"eeg_id\")[\"eeg_sub_id\"].count().describe(), \"\\n\")\nprint(\"Unique spectrogram_ids: \", train.spectrogram_id.nunique())\nprint(\"Unique patient_ids: \", train.patient_id.nunique(), \"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:12.588796Z","iopub.execute_input":"2024-01-17T15:53:12.589123Z","iopub.status.idle":"2024-01-17T15:53:13.304923Z","shell.execute_reply.started":"2024-01-17T15:53:12.589090Z","shell.execute_reply":"2024-01-17T15:53:13.303968Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# overall expert consensus\n# data\ndt = train.expert_consensus.value_counts().reset_index()\ndt.columns = [\"consensus\", \"frequency\"]\n\n# plot\nplt.figure(figsize=(20, 10))\n\nfigure = sns.barplot(data=dt,\n x=\"consensus\", y=\"frequency\", palette=my_colors[1:])\nshow_values_on_bars(figure, h_v=\"v\", space=0.4)\nplt.title('[train] Expert Consensus - Frequency', weight=\"bold\", size=20)\n\nplt.xlabel(\"Consensus\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True);","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2024-01-17T15:53:13.306292Z","iopub.execute_input":"2024-01-17T15:53:13.306707Z","iopub.status.idle":"2024-01-17T15:53:14.477550Z","shell.execute_reply.started":"2024-01-17T15:53:13.306670Z","shell.execute_reply":"2024-01-17T15:53:14.476658Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ log plot\ncreate_wandb_plot(x_data=dt[\"consensus\"],\n y_data=dt[\"frequency\"],\n x_name=\"Consensus\", y_name=\"Count\",\n title=\"[train] Expert Consensus Frequency\",\n log=\"bar_consensus\", plot=\"bar\")","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:14.478767Z","iopub.execute_input":"2024-01-17T15:53:14.479053Z","iopub.status.idle":"2024-01-17T15:53:15.229534Z","shell.execute_reply.started":"2024-01-17T15:53:14.479028Z","shell.execute_reply":"2024-01-17T15:53:15.228656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The `test.csv` dataset contains only the columns `eeg_id`, `spectrogram_id`, `patient_id`. This is because, in the end, this is the format we will be needing to train out `train.csv` data too.\n\nWe will also need to create another column that will contain the \"patterns\"\n* idealized - high level of agreement\n* proto - some say \"other\" and some agree on another activity\n* edge - split ~equally between two activities","metadata":{}},{"cell_type":"code","source":"# Grouped train.csv\nvote_cols = [col for col in train.columns if '_vote' in col]\nprint(\"vote cols:\", vote_cols)\n\ntrain_group = train.groupby(by=[\"eeg_id\", \"spectrogram_id\", \"patient_id\"])\\\n [vote_cols].sum().reset_index()\ntrain_group.head(7)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:15.234161Z","iopub.execute_input":"2024-01-17T15:53:15.234444Z","iopub.status.idle":"2024-01-17T15:53:15.890453Z","shell.execute_reply.started":"2024-01-17T15:53:15.234417Z","shell.execute_reply":"2024-01-17T15:53:15.889616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def categorize_votes(row):\n # compute max and sum\n col_names = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\n max_vote = row[col_names].max()\n total_votes = row[col_names].sum()\n\n # % votes for max fruit\n percentage = max_vote / total_votes * 100\n\n high_agreement_threshold = 70\n equal_splitting_threshold = 40\n\n if percentage >= high_agreement_threshold:\n return 'idealized'\n elif row['other_vote'] / total_votes >= 0.4 and percentage >= equal_splitting_threshold:\n return 'proto'\n elif row['other_vote'] == 0 and percentage >= equal_splitting_threshold:\n return 'edge'\n else:\n return 'undecided'\n\n# create new set of \"pattern\" labels\ntrain_group['pattern'] = train_group.apply(categorize_votes, axis=1)\ntrain_group.head(7)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:15.891691Z","iopub.execute_input":"2024-01-17T15:53:15.892297Z","iopub.status.idle":"2024-01-17T15:53:30.811642Z","shell.execute_reply.started":"2024-01-17T15:53:15.892261Z","shell.execute_reply":"2024-01-17T15:53:30.810571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# data\ndt = train_group.pattern.value_counts().reset_index()\ndt.columns = [\"pattern\", \"frequency\"]\n\n# plot\nplt.figure(figsize=(20, 10))\n\nfigure = sns.barplot(data=dt,\n x=\"pattern\", y=\"frequency\", palette=my_colors[1:])\nshow_values_on_bars(figure, h_v=\"v\", space=0.4)\nplt.title('[train] Categorized Pattern - Frequency', weight=\"bold\", size=20)\n\nplt.xlabel(\"Pattern\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True);","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2024-01-17T15:53:30.813034Z","iopub.execute_input":"2024-01-17T15:53:30.813311Z","iopub.status.idle":"2024-01-17T15:53:31.886514Z","shell.execute_reply.started":"2024-01-17T15:53:30.813286Z","shell.execute_reply":"2024-01-17T15:53:31.885332Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ log plot\ncreate_wandb_plot(x_data=dt[\"pattern\"],\n y_data=dt[\"frequency\"],\n x_name=\"Pattern\", y_name=\"Count\",\n title=\"[train] Categorized pattern Frequency\",\n log=\"bar_pattern\", plot=\"bar\")","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:31.887806Z","iopub.execute_input":"2024-01-17T15:53:31.888097Z","iopub.status.idle":"2024-01-17T15:53:32.769416Z","shell.execute_reply.started":"2024-01-17T15:53:31.888071Z","shell.execute_reply":"2024-01-17T15:53:32.768408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"However, why the extra information in the `sub_ids`, `spectrogram_sub_id`, `spectrogram_label_offset` and `eeg_label_offset_seconds`?\n\nBecause the specialists can disagree with the overall assessment of an entire spectrogram, the 10mins/spectrogram data was split in multiple sub-segments and evaluated individually.","metadata":{}},{"cell_type":"code","source":"# an example of spectrogram id with one EEG recording\ntrain[train.eeg_id==722738444]","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:32.770709Z","iopub.execute_input":"2024-01-17T15:53:32.771057Z","iopub.status.idle":"2024-01-17T15:53:33.525679Z","shell.execute_reply.started":"2024-01-17T15:53:32.771023Z","shell.execute_reply":"2024-01-17T15:53:33.524677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"However, not all cases look like the one above (as the `nunique` number of `eeg_id` differs than the `nunique` number for `spectrogram_id`).\n\nOne spectrogram (like the case below - 1219001) can be a part of multiple EEG recordings, all being from the same `patient_id`.","metadata":{}},{"cell_type":"code","source":"# an example of spectrogram id with multiple EEG records\n# train[train.eeg_sub_id != train.spectrogram_sub_id]\ntrain[train.spectrogram_id==1219001]","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:33.526934Z","iopub.execute_input":"2024-01-17T15:53:33.527289Z","iopub.status.idle":"2024-01-17T15:53:34.336013Z","shell.execute_reply.started":"2024-01-17T15:53:33.527253Z","shell.execute_reply":"2024-01-17T15:53:34.335008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hence, our \"consolidated\" `train.csv` dataset looks something like this (from a schematic point of view):\n\n","metadata":{}},{"cell_type":"markdown","source":"# 2. Understanding the Spectrograms\n\nThe `train_spectrograms` folder contains a .parquet file for each spectrogram.\n\nThe column names indicate the *frequency in hertz* (400 cols in total) and the recording regions of the EEG electrodes:\n* LL = left lateral;\n* RL = right lateral;\n* LP = left parasagittal; \n* RP = right parasagittal.","metadata":{}},{"cell_type":"code","source":"spectrogram_id = 789577333\n\n# read in the data\nspec_base_path = \"/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/\"\nspec_data = pd.read_parquet(spec_base_path + str(spectrogram_id) + \".parquet\")\n\nprint(spec_data.shape)\nspec_data.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:34.337339Z","iopub.execute_input":"2024-01-17T15:53:34.337701Z","iopub.status.idle":"2024-01-17T15:53:35.247935Z","shell.execute_reply.started":"2024-01-17T15:53:34.337669Z","shell.execute_reply":"2024-01-17T15:53:35.246913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To visualize a dataset this big, I will use the `specgram` module within `scipy` library. We call `data.T` to transpose the dataset and have the \"time\" variable on the x axis.\n\nLet's look at some spectrograms from each category. I will *select only the spectrograms that have the highest votes of confidence*, meaning that there were close to no disagreements between the experts.","metadata":{}},{"cell_type":"code","source":"# number of spectrograms for each category\nN = 5\n\nspec_dict = {\n \"seizure_vote\": 0,\n \"lpd_vote\": 0,\n \"gpd_vote\": 0,\n \"lrda_vote\":0, \n \"grda_vote\":0,\n \"other_vote\":0\n}\nidealized_df = train_group[train_group.pattern==\"idealized\"].reset_index(drop=True)\n\nfor key in spec_dict.keys():\n col_idx = idealized_df[key].sort_values(ascending=False).head(N).index\n spec_dict[key] = idealized_df.loc[col_idx, \"spectrogram_id\"].values\n \npprint(spec_dict)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:35.250158Z","iopub.execute_input":"2024-01-17T15:53:35.251057Z","iopub.status.idle":"2024-01-17T15:53:35.972200Z","shell.execute_reply.started":"2024-01-17T15:53:35.251029Z","shell.execute_reply":"2024-01-17T15:53:35.971278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### โฌ‡๏ธ plot function below","metadata":{}},{"cell_type":"code","source":"def plot_spectrograms_by_category(spectrogram_ids, category):\n \n # read in the data\n spec_base_path = \"/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/\"\n spec_data = [pd.read_parquet(spec_base_path + str(id) + \".parquet\") for id in spectrogram_ids]\n\n # create plots\n fs = 1000\n nfft = 1024\n\n fig, axes = plt.subplots(1, N, figsize=(20, 5), sharey=True)\n plt.suptitle(f\"{category}\", weight=\"bold\")\n axes = axes.flatten()\n\n for i in range(N):\n axes[i].imshow(np.log(spec_data[i].T), cmap='magma', aspect='auto')\n axes[i].set_title(f'id {spectrogram_ids[i]}', size=15)\n axes[i].set_xlabel('Time', size=15)\n axes[i].set_ylabel('(Hz)', size=15)\n \n# axes[i].axis(\"off\")\n axes[i].tick_params(axis='both', which='both', labelsize=10)\n\n plt.subplots_adjust(top=0.85)\n plt.show()","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2024-01-17T15:53:35.973426Z","iopub.execute_input":"2024-01-17T15:53:35.973768Z","iopub.status.idle":"2024-01-17T15:53:36.737636Z","shell.execute_reply.started":"2024-01-17T15:53:35.973737Z","shell.execute_reply":"2024-01-17T15:53:36.736538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for key, values in spec_dict.items():\n plot_spectrograms_by_category(spectrogram_ids=values,\n category=key)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:36.738947Z","iopub.execute_input":"2024-01-17T15:53:36.739303Z","iopub.status.idle":"2024-01-17T15:53:46.472086Z","shell.execute_reply.started":"2024-01-17T15:53:36.739265Z","shell.execute_reply":"2024-01-17T15:53:46.471003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I would like to look at some **edge** cases.\n\nThe reason I want to do that is to see if there are any groups of activities that come up the most (e.g. is seizure usually the most similar with grda?).","metadata":{}},{"cell_type":"code","source":"# filter only edge cases\nedge_df = train_group[train_group.pattern==\"edge\"].reset_index(drop=True)\n\n# get the names of the first two columns with the largest values\ndef top_columns(row, n=2):\n l = row.nlargest(n).index.tolist()\n return str(l[0]) + \", \" + str(l[1])\n\nedge_df[\"edge_cases\"] = edge_df.iloc[:, 3:-1].apply(top_columns, axis=1)\nedge_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:46.473437Z","iopub.execute_input":"2024-01-17T15:53:46.473813Z","iopub.status.idle":"2024-01-17T15:53:47.630648Z","shell.execute_reply.started":"2024-01-17T15:53:46.473778Z","shell.execute_reply":"2024-01-17T15:53:47.629567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# data\ndt = edge_df.edge_cases.value_counts().reset_index()\ndt.columns = [\"edge_cases\", \"frequency\"]\n\n# plot\nplt.figure(figsize=(20, 15))\nfigure = sns.barplot(data=dt,\n y=\"edge_cases\", x=\"frequency\", color=my_colors[4])\nshow_values_on_bars(figure, h_v=\"h\", space=0.4)\nplt.title('[train] Edge Cases - Frequency', weight=\"bold\", size=20)\n\nplt.xlabel(\"cases\", size = 18, weight=\"bold\")\nplt.ylabel(\"Count\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True);","metadata":{"_kg_hide-input":true,"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2024-01-17T15:53:47.632008Z","iopub.execute_input":"2024-01-17T15:53:47.632312Z","iopub.status.idle":"2024-01-17T15:53:49.282585Z","shell.execute_reply.started":"2024-01-17T15:53:47.632284Z","shell.execute_reply":"2024-01-17T15:53:49.281535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ log plot\ncreate_wandb_plot(x_data=dt[\"edge_cases\"],\n y_data=dt[\"frequency\"],\n x_name=\"edge_cases\", y_name=\"Count\",\n title=\"[train] Edge Cases Frequency\",\n log=\"bar_edge_cases\", plot=\"bar\")","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:49.283896Z","iopub.execute_input":"2024-01-17T15:53:49.284183Z","iopub.status.idle":"2024-01-17T15:53:50.149934Z","shell.execute_reply.started":"2024-01-17T15:53:49.284158Z","shell.execute_reply":"2024-01-17T15:53:50.148938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let us now see a few of these spectrograms that contain disagreement between experts.","metadata":{}},{"cell_type":"code","source":"spec_dict2 = {key:0 for \n key in edge_df.edge_cases.value_counts()[:6].index}\n\n# get top N ids\nN = 5\nfor key in spec_dict2.keys():\n col_idx = edge_df[edge_df.edge_cases == key].head(N).index\n spec_dict2[key] = edge_df.loc[col_idx, \"spectrogram_id\"].values\npprint(spec_dict2)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:50.151029Z","iopub.execute_input":"2024-01-17T15:53:50.151275Z","iopub.status.idle":"2024-01-17T15:53:50.951643Z","shell.execute_reply.started":"2024-01-17T15:53:50.151253Z","shell.execute_reply":"2024-01-17T15:53:50.950656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for key, values in spec_dict2.items():\n plot_spectrograms_by_category(spectrogram_ids=values,\n category=key)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:53:50.953130Z","iopub.execute_input":"2024-01-17T15:53:50.953467Z","iopub.status.idle":"2024-01-17T15:54:00.930272Z","shell.execute_reply.started":"2024-01-17T15:53:50.953436Z","shell.execute_reply":"2024-01-17T15:54:00.929272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ\nwandb.finish()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:54:00.932039Z","iopub.execute_input":"2024-01-17T15:54:00.932400Z","iopub.status.idle":"2024-01-17T15:54:08.772283Z","shell.execute_reply.started":"2024-01-17T15:54:00.932366Z","shell.execute_reply":"2024-01-17T15:54:08.771501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

- XGBoost using RAPIDS -

\n\nI am using the [rapids library](https://rapids.ai/) to handle the data and for preprocessing (much faster) XGBoost for training.\n\nAs for the spectrograms dataset (as there are 11,138 `.parquet` files with 400 columns each), I will be using the dataset Chris has already put together (you can find it [here](https://www.kaggle.com/datasets/cdeotte/brain-spectrograms)).\n\n### โ—‹ ML Libraries\n\n> ๐Ÿ“Œ**Note**: CuML doesn't work with the newest pandas version - there are a few fixes available, but they are too of an overhead so as of now I'll just use `sklearn`.","metadata":{}},{"cell_type":"code","source":"# import cuml\nimport cupy\nimport cudf\nimport xgboost as xgb\n\n# from cuml.model_selection import train_test_split\nfrom sklearn.model_selection import train_test_split","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:54:08.773483Z","iopub.execute_input":"2024-01-17T15:54:08.773769Z","iopub.status.idle":"2024-01-17T15:54:11.104508Z","shell.execute_reply.started":"2024-01-17T15:54:08.773743Z","shell.execute_reply":"2024-01-17T15:54:11.103736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Feature Engineering\n\nThe `spectrogram` files contain information about the hertz, on time, for multiple recording regions of the EEG electrodes. We can take these and create features out of them, which we can use afterwards to train the model.","metadata":{}},{"cell_type":"code","source":"# import spectrogram info\nspect_data = np.load(\"/kaggle/input/brain-spectrograms/specs.npy\", allow_pickle=True).item()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:54:11.105626Z","iopub.execute_input":"2024-01-17T15:54:11.105885Z","iopub.status.idle":"2024-01-17T15:55:11.387713Z","shell.execute_reply.started":"2024-01-17T15:54:11.105862Z","shell.execute_reply":"2024-01-17T15:55:11.386654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# example data\n\npprint(spect_data[319287046])\npprint(spect_data[319287046].shape)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:55:11.392323Z","iopub.execute_input":"2024-01-17T15:55:11.392643Z","iopub.status.idle":"2024-01-17T15:55:11.399228Z","shell.execute_reply.started":"2024-01-17T15:55:11.392617Z","shell.execute_reply":"2024-01-17T15:55:11.398111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get all column names\nsample_path = \"/kaggle/input/hms-harmful-brain-activity-classification/train_spectrograms/1000086677.parquet\"\nfeature_col_names = cudf.read_parquet(sample_path).columns[1:]\n\nprint(feature_col_names)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T15:55:11.400557Z","iopub.execute_input":"2024-01-17T15:55:11.400909Z","iopub.status.idle":"2024-01-17T15:55:11.896215Z","shell.execute_reply.started":"2024-01-17T15:55:11.400879Z","shell.execute_reply":"2024-01-17T15:55:11.895172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create features across all cols\n# ~ 1min to run\nfe_data = {}\n\nfor spect_id, data in tqdm(spect_data.items()):\n fe_data[spect_id] = {}\n \n for k, feature in enumerate(feature_col_names):\n fe_data[spect_id][f\"{feature}_mean\"] = data[:, k].mean()\n fe_data[spect_id][f\"{feature}_min\"] = data[:, k].min()\n fe_data[spect_id][f\"{feature}_max\"] = data[:, k].max()\n fe_data[spect_id][f\"{feature}_std\"] = data[:, k].std()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:05:19.645719Z","iopub.execute_input":"2024-01-17T16:05:19.646228Z","iopub.status.idle":"2024-01-17T16:10:19.022842Z","shell.execute_reply.started":"2024-01-17T16:05:19.646190Z","shell.execute_reply":"2024-01-17T16:10:19.021823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The end result is a dictionary of dicts of the format:\n\n```\n{\n spectrogram_id: \n {\n 'LL_0.59_mean': 51.703323,\n 'LL_0.78_mean': 66.76726,\n 'LL_0.98_mean': 78.36359,\n ...\n }\n}\n```\n\nTODO: create vote target labels - convert from classification to regression task","metadata":{}},{"cell_type":"code","source":"# convert to df\nfe_data_df = pd.DataFrame.from_dict(fe_data, orient='index').reset_index()\n\n# append target labels\n# target_df = train_group\\\n# .groupby(\"spectrogram_id\")[train_group.filter(regex='_vote$').columns]\\\n# .sum().reset_index()\ntarget_df = train\\\n .groupby(\"spectrogram_id\")[\"expert_consensus\"]\\\n .first().reset_index()\n# encoding from string to numbers\ntarget_df['expert_consensus'] = pd.factorize(target_df['expert_consensus'])[0]\n\nfinal_df = pd.merge(left=fe_data_df, right=target_df, \n left_on=\"index\", right_on=\"spectrogram_id\")\n\nfinal_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:10:19.024744Z","iopub.execute_input":"2024-01-17T16:10:19.025038Z","iopub.status.idle":"2024-01-17T16:10:38.641853Z","shell.execute_reply.started":"2024-01-17T16:10:19.025012Z","shell.execute_reply":"2024-01-17T16:10:38.640972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Model Training\n\nTODO: code a better data validation strategy","metadata":{}},{"cell_type":"code","source":"# data validation\ndtrain, dvalid = train_test_split(final_df, train_size=0.8, random_state=42)\n\nFEATURE_COLS = final_df.columns[1:-2]\nTARGET_COL = final_df.columns[-1]","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:10:38.642909Z","iopub.execute_input":"2024-01-17T16:10:38.643196Z","iopub.status.idle":"2024-01-17T16:10:38.686066Z","shell.execute_reply.started":"2024-01-17T16:10:38.643173Z","shell.execute_reply":"2024-01-17T16:10:38.685267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### XGBoost","metadata":{}},{"cell_type":"code","source":"# xgboost train function\ndef train_xgboost(dtrain, dvalid, config):\n '''\n Train the XGBoost model.\n ''' \n params = {\n 'objective': config.objective,\n 'eval_metric': config.eval_metric,\n 'num_class': config.num_class,\n 'tree_method': config.tree_method,\n \"random_state\": config.random_state,\n \"learning_rate\": config.learning_rate,\n \"max_depth\": config.max_depth,\n \"min_child_weight\": config.min_child_weight,\n }\n \n\n # Matrix\n dtrain_matrix = xgb.DMatrix(dtrain[FEATURE_COLS], label=dtrain[TARGET_COL])\n dvalid_matrix = xgb.DMatrix(dvalid[FEATURE_COLS], label=dvalid[TARGET_COL])\n\n # Training ...\n model = xgb.train(params, dtrain_matrix, \n evals=[(dvalid_matrix, 'test')], \n num_boost_round=100,\n verbose_eval=False)\n\n # Evaluate ...\n y_pred = model.predict(dvalid_matrix)\n y_pred = np.asarray(y_pred)\n\n # Compute accuracy\n y_true = np.asarray(dvalid[TARGET_COL])\n accuracy = np.sum(y_pred == y_true) / len(y_pred)\n wandb.log({\"accuracy\": np.float64(accuracy)})\n\n print(clr.S+f\"Accuracy: {accuracy:.4f}\"+clr.E)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:10:38.688105Z","iopub.execute_input":"2024-01-17T16:10:38.688389Z","iopub.status.idle":"2024-01-17T16:10:38.698382Z","shell.execute_reply.started":"2024-01-17T16:10:38.688365Z","shell.execute_reply":"2024-01-17T16:10:38.697642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train pipeline","metadata":{}},{"cell_type":"code","source":"def train_pipeline():\n \n # XGBoost hyperparameters\n config_defaults = {\n 'objective': 'multi:softmax',\n 'eval_metric': 'mlogloss',\n 'num_class': 6,\n 'tree_method': 'hist',\n 'device': 'cuda',\n \"random_state\": 24,\n \"learning_rate\": 0.1,\n \"max_depth\": 1,\n \"min_child_weight\": 1,\n }\n \n # ๐Ÿ W&B Experiment\n config_defaults.update(CONFIG)\n run = wandb.init(project=COMP_ID, config=config_defaults)\n config = wandb.config\n \n train_xgboost(dtrain, dvalid, config)\n \n # ๐Ÿ\n wandb.finish()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:10:38.699442Z","iopub.execute_input":"2024-01-17T16:10:38.699730Z","iopub.status.idle":"2024-01-17T16:10:38.713229Z","shell.execute_reply.started":"2024-01-17T16:10:38.699706Z","shell.execute_reply":"2024-01-17T16:10:38.712357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### First iteration:","metadata":{}},{"cell_type":"code","source":"train_pipeline()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:10:38.714315Z","iopub.execute_input":"2024-01-17T16:10:38.714631Z","iopub.status.idle":"2024-01-17T16:12:08.938478Z","shell.execute_reply.started":"2024-01-17T16:10:38.714583Z","shell.execute_reply":"2024-01-17T16:12:08.937655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Sweeps\n\n> ๐Ÿ“Œ**Note**: I am fine tuning the model using [Sweeps](https://docs.wandb.ai/guides/sweeps). The run from the image below can be found [here](https://wandb.ai/andrada/2024_hms/sweeps/txg90cxn?workspace=user-andrada). **Best run has an accuracy of 0.57.**\n\n","metadata":{}},{"cell_type":"code","source":"# Sweep Config\nsweep_config = {\n \"method\": \"random\",\n \"metric\": {\n \"name\": \"accuracy\",\n \"goal\": \"maximize\" \n },\n \"parameters\": {\n \"max_depth\": {\n \"values\": [1, 4, 6, 10, 15, 20]\n },\n \"min_child_weight\": {\n \"values\": [1, 2, 3, 4, 5, 8, 10]\n },\n \"learning_rate\": {\n \"values\": [0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.5, 0.7]\n },\n \"random_state\": {\n \"values\": [10, 24, 30, 45, 50, 75, 80, 100]\n }\n }\n}\n\n# Sweep ID\nsweep_id = wandb.sweep(sweep_config, project=COMP_ID)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:12:08.939711Z","iopub.execute_input":"2024-01-17T16:12:08.940017Z","iopub.status.idle":"2024-01-17T16:12:31.140767Z","shell.execute_reply.started":"2024-01-17T16:12:08.939992Z","shell.execute_reply":"2024-01-17T16:12:31.139775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ๐Ÿ RUN SWEEPS\nstart = time()\n\n# count = the number of trials/experiments to run\nwandb.agent(sweep_id, train_pipeline, count=30)\nprint(\"Sweeping took:\", round((time()-start)/60, 1), \"mins\")","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:12:31.141875Z","iopub.execute_input":"2024-01-17T16:12:31.142137Z","iopub.status.idle":"2024-01-17T16:12:49.354409Z","shell.execute_reply.started":"2024-01-17T16:12:31.142114Z","shell.execute_reply":"2024-01-17T16:12:49.353343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Feature Importance after Sweeps\n\nAfter we've run the feature importance, I want to create a final model and further analyse it to see which of the features that we have created yield the most entropy during training.\n\n**I am using the metrics of one of the best sweeps so far**:\n* lr: 0.1\n* max_depth: 20\n* min_child_weight: 10\n* random_state: 100","metadata":{}},{"cell_type":"code","source":"# best model train\nparams = {\n 'objective': 'multi:softmax',\n 'eval_metric': 'mlogloss',\n 'num_class': 6,\n 'tree_method': 'hist',\n 'device': 'cuda',\n \"random_state\": 100,\n \"learning_rate\": 0.1,\n \"max_depth\": 20,\n \"min_child_weight\": 10,\n }\n \n\n# matrix\ndtrain_matrix = xgb.DMatrix(dtrain[FEATURE_COLS], label=dtrain[TARGET_COL])\ndvalid_matrix = xgb.DMatrix(dvalid[FEATURE_COLS], label=dvalid[TARGET_COL])\n\n# training ...\nbest_model = xgb.train(params, dtrain_matrix, \n evals=[(dvalid_matrix, 'test')], \n num_boost_round=100,\n verbose_eval=False)\n\n# evaluate ...\ny_pred = best_model.predict(dvalid_matrix)\ny_pred = np.asarray(y_pred)","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:16:27.971900Z","iopub.execute_input":"2024-01-17T16:16:27.972300Z","iopub.status.idle":"2024-01-17T16:16:46.138555Z","shell.execute_reply.started":"2024-01-17T16:16:27.972269Z","shell.execute_reply":"2024-01-17T16:16:46.137543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Accuracy\ny_true = np.asarray(dvalid[TARGET_COL])\nprint(np.sum(y_pred == y_true) / len(y_pred))","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:17:05.782725Z","iopub.execute_input":"2024-01-17T16:17:05.783450Z","iopub.status.idle":"2024-01-17T16:17:05.788778Z","shell.execute_reply.started":"2024-01-17T16:17:05.783415Z","shell.execute_reply":"2024-01-17T16:17:05.787745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature Importance\n\n**What do the scores mean?**\n\nThey show the importance - the idea is as they get bigger, the more important the feature is. They are computed based on how many times the decission split was made based on that feature (hence, it means the feature itself gave a lot of information).","metadata":{}},{"cell_type":"code","source":"importance_dict = best_model.get_score(importance_type='weight')\nimportance_df = pd.DataFrame(list(importance_dict.items()), \n columns=['Feature', 'Importance'])\\\n .sort_values('Importance', ascending=False)\\\n .reset_index(drop=True)\nimportance_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:28:17.954227Z","iopub.execute_input":"2024-01-17T16:28:17.954645Z","iopub.status.idle":"2024-01-17T16:28:17.973365Z","shell.execute_reply.started":"2024-01-17T16:28:17.954613Z","shell.execute_reply":"2024-01-17T16:28:17.972334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"importance_df.describe()","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:28:20.313126Z","iopub.execute_input":"2024-01-17T16:28:20.313507Z","iopub.status.idle":"2024-01-17T16:28:20.326954Z","shell.execute_reply.started":"2024-01-17T16:28:20.313477Z","shell.execute_reply":"2024-01-17T16:28:20.325926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plot\nplt.figure(figsize=(20, 15))\n\nfigure = sns.barplot(data=importance_df[importance_df.Importance >= 100],\n x=\"Importance\", y=\"Feature\", color=my_colors[0])\nshow_values_on_bars(figure, h_v=\"h\", space=0.4)\nplt.title('Top Features in terms of Importance', weight=\"bold\", size=20)\n\nplt.xlabel(\"Importance\", size = 18, weight=\"bold\")\nplt.ylabel(\"Feature Name\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True);","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:31:16.768192Z","iopub.execute_input":"2024-01-17T16:31:16.768563Z","iopub.status.idle":"2024-01-17T16:31:17.430299Z","shell.execute_reply.started":"2024-01-17T16:31:16.768535Z","shell.execute_reply":"2024-01-17T16:31:17.429369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"importance_df[\"kpi\"] = importance_df[\"Feature\"].apply(lambda x: x.split(\"_\")[-1])\ndt = importance_df[importance_df.Importance>=30][\"kpi\"]\\\n .value_counts().reset_index()\n\n# Plot\nplt.figure(figsize=(20, 15))\n\nfigure = sns.barplot(data=dt,\n x=\"kpi\", y=\"count\", palette=my_colors)\nshow_values_on_bars(figure, h_v=\"v\", space=0.4)\nplt.title('KPI that yields the most information in top 25% features', weight=\"bold\", size=20)\n\nplt.xlabel(\"KPI\", size = 18, weight=\"bold\")\nplt.ylabel(\"Importance\", size = 18, weight=\"bold\")\n \nsns.despine(right=True, top=True, left=True);","metadata":{"execution":{"iopub.status.busy":"2024-01-17T16:36:02.074005Z","iopub.execute_input":"2024-01-17T16:36:02.074384Z","iopub.status.idle":"2024-01-17T16:36:02.479113Z","shell.execute_reply.started":"2024-01-17T16:36:02.074353Z","shell.execute_reply":"2024-01-17T16:36:02.478183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### ๐Ÿ [my W&B dash](https://wandb.ai/andrada/2024_hms?workspace=user-andrada)\n \n
\n\n------\n\n
\n\n### My Specs\n\n* ๐Ÿ–ฅ Z8 G4 Workstation\n* ๐Ÿ’พ 2 CPUs & 96GB Memory\n* ๐ŸŽฎ 2x NVIDIA A6000\n* ๐Ÿ’ป Zbook Studio G9 on the go","metadata":{}}]}