\nThis starter notebook is provided by the Keras team.
","metadata":{"execution":{"iopub.execute_input":"2024-01-10T05:24:31.308329Z","iopub.status.busy":"2024-01-10T05:24:31.307595Z","iopub.status.idle":"2024-01-10T05:24:31.313088Z","shell.execute_reply":"2024-01-10T05:24:31.312113Z","shell.execute_reply.started":"2024-01-10T05:24:31.308287Z"},"papermill":{"duration":0.014007,"end_time":"2024-02-01T08:23:39.222448","exception":false,"start_time":"2024-02-01T08:23:39.208441","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# HMS - Harmful Brain Activity Classification with [KerasCV](https://github.com/keras-team/keras-cv) and [Keras](https://github.com/keras-team/keras)\n\n> The objective of this competition is to classify seizures and other patterns of harmful brain activity in critically ill patients\n\nThis notebook guides you through the process of training and inferring a Deep Learning model, specifically EfficientNetV2, using KerasCV on the competition dataset. Specificaclly, this notebook uses spectrogram of the eeg data to classify the patterns.\n\nFun fact: This notebook is backend-agnostic, supporting TensorFlow, PyTorch, and JAX. Utilizing KerasCV and Keras allows us to choose our preferred backend. Explore more details on [Keras](https://keras.io/keras_core/announcement/).\n\nIn this notebook, you will learn:\n\n* Loading the data efficiently using [`tf.data`](https://www.tensorflow.org/guide/data).\n* Creating the model using KerasCV presets.\n* Training the model.\n* Inference and Submission on test data.\n\n**Note**: For a more in-depth understanding of KerasCV, refer to the [KerasCV guides](https://keras.io/guides/keras_cv/).","metadata":{"papermill":{"duration":0.012942,"end_time":"2024-02-01T08:23:39.249157","exception":false,"start_time":"2024-02-01T08:23:39.236215","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# ๐ | Install Libraries \n\nSince internet access is **disabled** during inference, we cannot install libraries in the usual `!pip install ` manner. Instead, we need to install libraries from local files. In the following cell, we will install libraries from our local files. The installation code stays very similar - we just use the `filepath` instead of the `filename` of the library. So now the code is `!pip install `. \n\n> The `filepath` of these local libraries look quite complicated, but don't be intimidated! Also `--no-deps` argument ensures that we are not installing any additional libraries.","metadata":{"papermill":{"duration":0.013441,"end_time":"2024-02-01T08:23:39.275869","exception":false,"start_time":"2024-02-01T08:23:39.262428","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import time\n\ns_time = time.time()","metadata":{"execution":{"iopub.status.busy":"2024-04-29T21:28:36.570538Z","iopub.execute_input":"2024-04-29T21:28:36.571105Z","iopub.status.idle":"2024-04-29T21:28:36.585230Z","shell.execute_reply.started":"2024-04-29T21:28:36.571066Z","shell.execute_reply":"2024-04-29T21:28:36.584111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -q /kaggle/input/kerasv3-lib-ds/keras_cv-0.8.2-py3-none-any.whl --no-deps\n!pip install -q /kaggle/input/kerasv3-lib-ds/tensorflow-2.15.0.post1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl --no-deps\n!pip install -q /kaggle/input/kerasv3-lib-ds/keras-3.0.4-py3-none-any.whl --no-deps\n\nprint('done')","metadata":{"papermill":{"duration":112.31422,"end_time":"2024-02-01T08:25:31.603108","exception":false,"start_time":"2024-02-01T08:23:39.288888","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:28:36.592189Z","iopub.execute_input":"2024-04-29T21:28:36.592725Z","iopub.status.idle":"2024-04-29T21:30:24.016475Z","shell.execute_reply.started":"2024-04-29T21:28:36.592686Z","shell.execute_reply":"2024-04-29T21:30:24.015342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ | Import Libraries ","metadata":{"papermill":{"duration":0.013298,"end_time":"2024-02-01T08:25:31.629826","exception":false,"start_time":"2024-02-01T08:25:31.616528","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os\nos.environ[\"KERAS_BACKEND\"] = \"jax\" # you can also use tensorflow or torch\n\nimport keras_cv\nimport keras\nfrom keras import ops\nimport tensorflow as tf\n\nimport cv2\nimport pandas as pd\nimport numpy as np\nfrom glob import glob\nfrom tqdm.notebook import tqdm\nimport joblib\n\nimport matplotlib.pyplot as plt \n\nprint('done')","metadata":{"_kg_hide-output":true,"papermill":{"duration":11.396813,"end_time":"2024-02-01T08:25:43.039862","exception":false,"start_time":"2024-02-01T08:25:31.643049","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:30:24.018973Z","iopub.execute_input":"2024-04-29T21:30:24.019365Z","iopub.status.idle":"2024-04-29T21:30:34.554414Z","shell.execute_reply.started":"2024-04-29T21:30:24.019328Z","shell.execute_reply":"2024-04-29T21:30:34.553463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Library Versions","metadata":{"papermill":{"duration":0.013126,"end_time":"2024-02-01T08:25:43.066591","exception":false,"start_time":"2024-02-01T08:25:43.053465","status":"completed"},"tags":[]}},{"cell_type":"code","source":"'''\nprint(\"TensorFlow:\", tf.__version__)\nprint(\"Keras:\", keras.__version__)\nprint(\"KerasCV:\", keras_cv.__version__)\n'''","metadata":{"papermill":{"duration":0.022163,"end_time":"2024-02-01T08:25:43.101932","exception":false,"start_time":"2024-02-01T08:25:43.079769","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:30:34.555657Z","iopub.execute_input":"2024-04-29T21:30:34.556185Z","iopub.status.idle":"2024-04-29T21:30:34.563212Z","shell.execute_reply.started":"2024-04-29T21:30:34.556158Z","shell.execute_reply":"2024-04-29T21:30:34.562225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# โ๏ธ | Configuration","metadata":{"papermill":{"duration":0.013239,"end_time":"2024-02-01T08:25:43.128915","exception":false,"start_time":"2024-02-01T08:25:43.115676","status":"completed"},"tags":[]}},{"cell_type":"code","source":"class CFG:\n verbose = 1 # Verbosity\n seed = 42 # Random seed\n preset = \"efficientnetv2_b2_imagenet\" # Name of pretrained classifier\n image_size = [400, 300] # Input image size\n epochs = 13 # Training epochs\n batch_size = 64 # Batch size\n lr_mode = \"cos\" # LR scheduler mode from one of \"cos\", \"step\", \"exp\"\n drop_remainder = True # Drop incomplete batches\n num_classes = 6 # Number of classes in the dataset\n fold = 0 # Which fold to set as validation data\n class_names = ['Seizure', 'LPD', 'GPD', 'LRDA','GRDA', 'Other']\n label2name = dict(enumerate(class_names))\n name2label = {v:k for k, v in label2name.items()}\n \nprint('done')","metadata":{"papermill":{"duration":0.023244,"end_time":"2024-02-01T08:25:43.166779","exception":false,"start_time":"2024-02-01T08:25:43.143535","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:30:34.566023Z","iopub.execute_input":"2024-04-29T21:30:34.566322Z","iopub.status.idle":"2024-04-29T21:30:34.578392Z","shell.execute_reply.started":"2024-04-29T21:30:34.566294Z","shell.execute_reply":"2024-04-29T21:30:34.577550Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# โป๏ธ | Reproducibility \nSets value for random seed to produce similar result in each run.","metadata":{"papermill":{"duration":0.01362,"end_time":"2024-02-01T08:25:43.193679","exception":false,"start_time":"2024-02-01T08:25:43.180059","status":"completed"},"tags":[]}},{"cell_type":"code","source":"keras.utils.set_random_seed(CFG.seed)\n\nprint('done')","metadata":{"papermill":{"duration":0.021308,"end_time":"2024-02-01T08:25:43.228374","exception":false,"start_time":"2024-02-01T08:25:43.207066","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:30:34.579309Z","iopub.execute_input":"2024-04-29T21:30:34.579536Z","iopub.status.idle":"2024-04-29T21:30:34.589994Z","shell.execute_reply.started":"2024-04-29T21:30:34.579516Z","shell.execute_reply":"2024-04-29T21:30:34.589062Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ | Dataset Path ","metadata":{"papermill":{"duration":0.01325,"end_time":"2024-02-01T08:25:43.255073","exception":false,"start_time":"2024-02-01T08:25:43.241823","status":"completed"},"tags":[]}},{"cell_type":"code","source":"BASE_PATH = \"/kaggle/input/hms-harmful-brain-activity-classification\"\n\nSPEC_DIR = \"/tmp/dataset/hms-hbac\"\nos.makedirs(SPEC_DIR+'/train_spectrograms', exist_ok=True)\nos.makedirs(SPEC_DIR+'/test_spectrograms', exist_ok=True)\n\nprint('done')","metadata":{"papermill":{"duration":0.021773,"end_time":"2024-02-01T08:25:43.290219","exception":false,"start_time":"2024-02-01T08:25:43.268446","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:30:34.590971Z","iopub.execute_input":"2024-04-29T21:30:34.591225Z","iopub.status.idle":"2024-04-29T21:30:34.602903Z","shell.execute_reply.started":"2024-04-29T21:30:34.591204Z","shell.execute_reply":"2024-04-29T21:30:34.601988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ | Meta Data ","metadata":{"papermill":{"duration":0.01301,"end_time":"2024-02-01T08:25:43.316629","exception":false,"start_time":"2024-02-01T08:25:43.303619","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Train + Valid + Test2 (check accuracy)\ndf = pd.read_csv(f'{BASE_PATH}/train.csv')\ndf['eeg_path'] = f'{BASE_PATH}/train_eegs/'+df['eeg_id'].astype(str)+'.parquet'\ndf['spec_path'] = f'{BASE_PATH}/train_spectrograms/'+df['spectrogram_id'].astype(str)+'.parquet'\ndf['spec2_path'] = f'{SPEC_DIR}/train_spectrograms/'+df['spectrogram_id'].astype(str)+'.npy'\ndf['class_name'] = df.expert_consensus.copy()\ndf['class_label'] = df.expert_consensus.map(CFG.name2label)\ndisplay(df.head(2))\n\n'''\n# Test - used for submission\ntest_df = pd.read_csv(f'{BASE_PATH}/test.csv')\ntest_df['eeg_path'] = f'{BASE_PATH}/test_eegs/'+test_df['eeg_id'].astype(str)+'.parquet'\ntest_df['spec_path'] = f'{BASE_PATH}/test_spectrograms/'+test_df['spectrogram_id'].astype(str)+'.parquet'\ntest_df['spec2_path'] = f'{SPEC_DIR}/test_spectrograms/'+test_df['spectrogram_id'].astype(str)+'.npy'\ndisplay(test_df.head(2))\n\n'''\nprint('done')","metadata":{"papermill":{"duration":0.626473,"end_time":"2024-02-01T08:25:43.956328","exception":false,"start_time":"2024-02-01T08:25:43.329855","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:30:34.604004Z","iopub.execute_input":"2024-04-29T21:30:34.604262Z","iopub.status.idle":"2024-04-29T21:30:35.138216Z","shell.execute_reply.started":"2024-04-29T21:30:34.604240Z","shell.execute_reply":"2024-04-29T21:30:35.137307Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Convert `.parquet` to `.npy`\n\nTo facilitate easier data loading, we will convert the EEG spectrograms from `parquet` to `npy` format. This process involves saving the spectrogram data, and since the content of the files remains the same, no significant changes are made. \n\n> It's worth noting that the `time` column is excluded, as it is not part of the spectrogram.","metadata":{"papermill":{"duration":0.013448,"end_time":"2024-02-01T08:25:43.984679","exception":false,"start_time":"2024-02-01T08:25:43.971231","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Define a function to process a single eeg_id\ndef process_spec(spec_id, split=\"train\"):\n spec_path = f\"{BASE_PATH}/{split}_spectrograms/{spec_id}.parquet\"\n spec = pd.read_parquet(spec_path)\n spec = spec.fillna(0).values[:, 1:].T # fill NaN values with 0, transpose for (Time, Freq) -> (Freq, Time)\n spec = spec.astype(\"float32\")\n np.save(f\"{SPEC_DIR}/{split}_spectrograms/{spec_id}.npy\", spec)\n\n# Get unique spec_ids of train and valid data\nspec_ids = df[\"spectrogram_id\"].unique()\n\n# Parallelize the processing using joblib for training data\n_ = joblib.Parallel(n_jobs=-1, backend=\"loky\")(\n joblib.delayed(process_spec)(spec_id, \"train\")\n for spec_id in tqdm(spec_ids, total=len(spec_ids))\n)\n\n'''\n# Get unique spec_ids of test data\ntest_spec_ids = test_df[\"spectrogram_id\"].unique()\n\n# Parallelize the processing using joblib for test data\n_ = joblib.Parallel(n_jobs=-1, backend=\"loky\")(\n joblib.delayed(process_spec)(spec_id, \"test\")\n for spec_id in tqdm(test_spec_ids, total=len(test_spec_ids))\n)\n'''\nprint('done')","metadata":{"papermill":{"duration":204.012172,"end_time":"2024-02-01T08:29:08.010508","exception":false,"start_time":"2024-02-01T08:25:43.998336","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:30:35.139717Z","iopub.execute_input":"2024-04-29T21:30:35.140174Z","iopub.status.idle":"2024-04-29T21:33:42.794268Z","shell.execute_reply.started":"2024-04-29T21:30:35.140133Z","shell.execute_reply":"2024-04-29T21:33:42.793171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ | DataLoader\n\nThis DataLoader first reads `npy` spectrogram files and extracts labeled subsamples using specified `offset` values. Then, it converts the spectrogram data into `log spectrogram` and applies the popular signal augmentation `MixUp`.\n\n> Note that, we are converting the mono channel signal to a 3-channel signal for using \"ImageNet\" weights of pretrained model.","metadata":{"papermill":{"duration":0.085349,"end_time":"2024-02-01T08:29:08.117563","exception":false,"start_time":"2024-02-01T08:29:08.032214","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def build_augmenter(dim=CFG.image_size):\n augmenters = [\n keras_cv.layers.MixUp(alpha=2.0),\n keras_cv.layers.RandomCutout(height_factor=(1.0, 1.0),\n width_factor=(0.06, 0.1)), # freq-masking\n keras_cv.layers.RandomCutout(height_factor=(0.06, 0.1),\n width_factor=(1.0, 1.0)), # time-masking\n ]\n \n def augment(img, label):\n data = {\"images\":img, \"labels\":label}\n for augmenter in augmenters:\n if tf.random.uniform([]) < 0.5:\n data = augmenter(data, training=True)\n return data[\"images\"], data[\"labels\"]\n \n return augment\n\n\ndef build_decoder(with_labels=True, target_size=CFG.image_size, dtype=32):\n def decode_signal(path, offset=None):\n # Read .npy files and process the signal\n file_bytes = tf.io.read_file(path)\n sig = tf.io.decode_raw(file_bytes, tf.float32)\n sig = sig[1024//dtype:] # Remove header tag\n sig = tf.reshape(sig, [400, -1])\n \n # Extract labeled subsample from full spectrogram using \"offset\"\n if offset is not None: \n offset = offset // 2 # Only odd values are given\n sig = sig[:, offset:offset+300]\n \n # Pad spectrogram to ensure the same input shape of [400, 300]\n pad_size = tf.math.maximum(0, 300 - tf.shape(sig)[1])\n sig = tf.pad(sig, [[0, 0], [0, pad_size]])\n sig = tf.reshape(sig, [400, 300])\n \n # Log spectrogram \n sig = tf.clip_by_value(sig, tf.math.exp(-4.0), tf.math.exp(8.0)) # avoid 0 in log\n sig = tf.math.log(sig)\n \n # Normalize spectrogram\n sig -= tf.math.reduce_mean(sig)\n sig /= tf.math.reduce_std(sig) + 1e-6\n \n # Mono channel to 3 channels to use \"ImageNet\" weights\n sig = tf.tile(sig[..., None], [1, 1, 3])\n return sig\n \n def decode_label(label):\n label = tf.one_hot(label, CFG.num_classes)\n label = tf.cast(label, tf.float32)\n label = tf.reshape(label, [CFG.num_classes])\n return label\n \n def decode_with_labels(path, offset=None, label=None):\n sig = decode_signal(path, offset)\n label = decode_label(label)\n return (sig, label)\n \n return decode_with_labels if with_labels else decode_signal\n\n\ndef build_dataset(paths, offsets=None, labels=None, batch_size=32, cache=True,\n decode_fn=None, augment_fn=None,\n augment=False, repeat=True, shuffle=1024, \n cache_dir=\"\", drop_remainder=False):\n if cache_dir != \"\" and cache is True:\n os.makedirs(cache_dir, exist_ok=True)\n \n if decode_fn is None:\n decode_fn = build_decoder(labels is not None)\n \n if augment_fn is None:\n augment_fn = build_augmenter()\n \n AUTO = tf.data.experimental.AUTOTUNE\n slices = (paths, offsets) if labels is None else (paths, offsets, labels)\n \n ds = tf.data.Dataset.from_tensor_slices(slices)\n ds = ds.map(decode_fn, num_parallel_calls=AUTO)\n ds = ds.cache(cache_dir) if cache else ds\n ds = ds.repeat() if repeat else ds\n if shuffle: \n ds = ds.shuffle(shuffle, seed=CFG.seed)\n opt = tf.data.Options()\n opt.experimental_deterministic = False\n ds = ds.with_options(opt)\n ds = ds.batch(batch_size, drop_remainder=drop_remainder)\n ds = ds.map(augment_fn, num_parallel_calls=AUTO) if augment else ds\n ds = ds.prefetch(AUTO)\n return ds\n\nprint('done')","metadata":{"papermill":{"duration":0.049068,"end_time":"2024-02-01T08:29:08.182732","exception":false,"start_time":"2024-02-01T08:29:08.133664","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:33:42.796101Z","iopub.execute_input":"2024-04-29T21:33:42.796781Z","iopub.status.idle":"2024-04-29T21:33:42.818436Z","shell.execute_reply.started":"2024-04-29T21:33:42.796735Z","shell.execute_reply":"2024-04-29T21:33:42.817226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ช | Data Split\n\nIn the following code snippet, the data is divided into `5` folds. Note that, the `groups` argument is used to prevent any overlap of patients between the training and validation sets, thus avoiding potential **data leakage** issues. Additionally, each split is stratified based on the `class_label`, ensuring a uniform distribution of class labels in each fold.","metadata":{"papermill":{"duration":0.018049,"end_time":"2024-02-01T08:29:08.221643","exception":false,"start_time":"2024-02-01T08:29:08.203594","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from sklearn.model_selection import StratifiedGroupKFold\n\nsgkf = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=CFG.seed)\n\ndf[\"fold\"] = -1\ndf.reset_index(drop=True, inplace=True)\nfor fold, (train_idx, valid_idx) in enumerate(\n sgkf.split(df, y=df[\"class_label\"], groups=df[\"patient_id\"])\n):\n df.loc[valid_idx, \"fold\"] = fold\ndf.groupby([\"fold\", \"class_name\"])[[\"eeg_id\"]].count().T\n\nprint('done')","metadata":{"papermill":{"duration":2.550523,"end_time":"2024-02-01T08:29:10.791064","exception":false,"start_time":"2024-02-01T08:29:08.240541","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:33:42.822666Z","iopub.execute_input":"2024-04-29T21:33:42.822965Z","iopub.status.idle":"2024-04-29T21:33:45.376163Z","shell.execute_reply.started":"2024-04-29T21:33:42.822941Z","shell.execute_reply":"2024-04-29T21:33:45.375172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build Train & Valid Dataset\n\nOnly first sample for each `spectrogram_id` is used in order to keep the dataset size managable. Feel free to train on full data.","metadata":{"papermill":{"duration":0.016026,"end_time":"2024-02-01T08:29:10.823438","exception":false,"start_time":"2024-02-01T08:29:10.807412","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Sample from full data\ntest_df_2 = df.groupby(\"spectrogram_id\").head(1).reset_index(drop=True)\ntest_df_2 = test_df_2[test_df_2.fold == CFG.fold]\n#test_df_2.to_csv(\"test_df_2.csv\", index=False)\n#sample_df = df.groupby(\"spectrogram_id\").head(2).reset_index(drop=True) - completed\nsample_df = df.groupby(\"spectrogram_id\").head(10).reset_index(drop=True)\nsample_df = sample_df.drop(test_df_2.index)\n\ntrain_df = sample_df[sample_df.fold != CFG.fold]\nvalid_df = sample_df[sample_df.fold == CFG.fold]\nprint(f\"# Num Train: {len(train_df)} | Num Valid: {len(valid_df)}\")\n\n# Train\ntrain_paths = train_df.spec2_path.values\ntrain_offsets = train_df.spectrogram_label_offset_seconds.values.astype(int)\ntrain_labels = train_df.class_label.values\ntrain_ds = build_dataset(train_paths, train_offsets, train_labels, batch_size=CFG.batch_size,\n repeat=True, shuffle=True, augment=True, cache=False) #cache=True)\n\n# Valid\nvalid_paths = valid_df.spec2_path.values\nvalid_offsets = valid_df.spectrogram_label_offset_seconds.values.astype(int)\nvalid_labels = valid_df.class_label.values\nvalid_ds = build_dataset(valid_paths, valid_offsets, valid_labels, batch_size=CFG.batch_size,\n repeat=False, shuffle=False, augment=False, cache=False) #cache=True)\n\nprint('done')","metadata":{"papermill":{"duration":2.847538,"end_time":"2024-02-01T08:29:13.688891","exception":false,"start_time":"2024-02-01T08:29:10.841353","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:33:45.377358Z","iopub.execute_input":"2024-04-29T21:33:45.377664Z","iopub.status.idle":"2024-04-29T21:33:48.168498Z","shell.execute_reply.started":"2024-04-29T21:33:45.377640Z","shell.execute_reply":"2024-04-29T21:33:48.167326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n'''\nhead(5)\n(1972, 21)\n(30824, 21)\n(6390, 21)\n\nhead(10)\n(1972, 21)\n(46156, 21)\n(9324, 21)\n'''\n\nprint(test_df_2.shape)\nprint(train_df.shape)\nprint(valid_df.shape)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-29T21:33:48.170208Z","iopub.execute_input":"2024-04-29T21:33:48.170993Z","iopub.status.idle":"2024-04-29T21:33:48.176646Z","shell.execute_reply.started":"2024-04-29T21:33:48.170963Z","shell.execute_reply":"2024-04-29T21:33:48.175802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Dataset Check\n\nLet's visualize some samples from the dataset.","metadata":{"papermill":{"duration":0.014561,"end_time":"2024-02-01T08:29:13.721268","exception":false,"start_time":"2024-02-01T08:29:13.706707","status":"completed"},"tags":[]}},{"cell_type":"code","source":"'''\nimgs, tars = next(iter(train_ds))\n\nnum_imgs = 8\nplt.figure(figsize=(4*4, num_imgs//4*5))\nfor i in range(num_imgs):\n plt.subplot(num_imgs//4, 4, i + 1)\n img = imgs[i].numpy()[...,0] # Adjust as per your image data format\n img -= img.min()\n img /= img.max() + 1e-4\n tar = CFG.label2name[np.argmax(tars[i].numpy())]\n plt.imshow(img)\n plt.title(f\"Target: {tar}\")\n plt.axis('off')\n \nplt.tight_layout()\nplt.show()\n'''\nprint('done')","metadata":{"_kg_hide-input":true,"papermill":{"duration":3.091679,"end_time":"2024-02-01T08:29:16.827306","exception":false,"start_time":"2024-02-01T08:29:13.735627","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:33:48.178143Z","iopub.execute_input":"2024-04-29T21:33:48.178818Z","iopub.status.idle":"2024-04-29T21:33:48.837392Z","shell.execute_reply.started":"2024-04-29T21:33:48.178772Z","shell.execute_reply":"2024-04-29T21:33:48.836314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ | Loss & Metric\n\nThe evaluation metric in this competition is **KL Divergence**, defined as,\n\n$$\nD_{\\text{KL}}(P \\parallel Q) = \\sum_{i} P(i) \\log\\left(\\frac{P(i)}{Q(i)}\\right)\n$$\n\nWhere:\n- $P$ is the true distribution.\n- $Q$ is the predicted distribution.\n\nInterestingly, as KL Divergence is differentiable, we can directly use it as our loss function. Thus, we don't need to use a third-party metric like **Accuracy** to evaluate our model. Therefore, `valid_loss` can stand alone as an indicator for our evaluation. In keras, we already have impelementation for KL Divergence loss so we only need to import it.","metadata":{"papermill":{"duration":0.04787,"end_time":"2024-02-01T08:29:16.923832","exception":false,"start_time":"2024-02-01T08:29:16.875962","status":"completed"},"tags":[]}},{"cell_type":"code","source":"LOSS = keras.losses.KLDivergence()\n\nprint('done')","metadata":{"papermill":{"duration":0.059431,"end_time":"2024-02-01T08:29:17.033766","exception":false,"start_time":"2024-02-01T08:29:16.974335","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:33:48.838810Z","iopub.execute_input":"2024-04-29T21:33:48.839467Z","iopub.status.idle":"2024-04-29T21:33:48.882348Z","shell.execute_reply.started":"2024-04-29T21:33:48.839418Z","shell.execute_reply":"2024-04-29T21:33:48.881142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ค | Modeling\n\nThis notebook uses the `EfficientNetV2 B2` from KerasCV's collection of pretrained models. To explore other models, simply modify the `preset` in the `CFG` (config). Check the [KerasCV website](https://keras.io/api/keras_cv/models/tasks/image_classifier/) for a list of available pretrained models.","metadata":{"papermill":{"duration":0.049049,"end_time":"2024-02-01T08:29:17.133352","exception":false,"start_time":"2024-02-01T08:29:17.084303","status":"completed"},"tags":[]}},{"cell_type":"code","source":"#import tensorflowas tf\n\n","metadata":{"execution":{"iopub.status.busy":"2024-04-29T21:33:48.883947Z","iopub.execute_input":"2024-04-29T21:33:48.884318Z","iopub.status.idle":"2024-04-29T21:33:48.924434Z","shell.execute_reply.started":"2024-04-29T21:33:48.884286Z","shell.execute_reply":"2024-04-29T21:33:48.923522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\nimport tensorflow_cloud as tfc\n\n# Define TPU strategy\ntpu_strategy = tf.distribute.TPUStrategy(resolver)\n\n# Define and compile the model within the TPU strategy scope\nwith tpu_strategy.scope():\n # Build Classifier\n model = keras_cv.models.ImageClassifier.from_preset(\n CFG.preset, num_classes=CFG.num_classes\n )\n \n # Compile the model\n model.compile(optimizer=keras.optimizers.Adam(learning_rate=1e-4),\n loss=LOSS)\n\n# Model Summary\nmodel.summary()\n'''\nprint('done')","metadata":{"execution":{"iopub.status.busy":"2024-04-29T21:33:48.926120Z","iopub.execute_input":"2024-04-29T21:33:48.926395Z","iopub.status.idle":"2024-04-29T21:33:48.969427Z","shell.execute_reply.started":"2024-04-29T21:33:48.926372Z","shell.execute_reply":"2024-04-29T21:33:48.968209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n# Build Classifier\nmodel = keras_cv.models.ImageClassifier.from_preset(\n CFG.preset, num_classes=CFG.num_classes\n)\n\n# Compile the model \nmodel.compile(optimizer=keras.optimizers.Adam(learning_rate=1e-4),\n loss=LOSS)\n\n# Model Sumamry\nmodel.summary()\n\nprint('done')","metadata":{"papermill":{"duration":23.194951,"end_time":"2024-02-01T08:29:40.378919","exception":false,"start_time":"2024-02-01T08:29:17.183968","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:33:48.970767Z","iopub.execute_input":"2024-04-29T21:33:48.971158Z","iopub.status.idle":"2024-04-29T21:34:11.462817Z","shell.execute_reply.started":"2024-04-29T21:33:48.971124Z","shell.execute_reply":"2024-04-29T21:34:11.461701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# โ | LR Schedule\n\nA well-structured learning rate schedule is essential for efficient model training, ensuring optimal convergence and avoiding issues such as overshooting or stagnation.","metadata":{"papermill":{"duration":0.051318,"end_time":"2024-02-01T08:29:40.481346","exception":false,"start_time":"2024-02-01T08:29:40.430028","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import math\n\ndef get_lr_callback(batch_size=8, mode='cos', epochs=10, plot=False):\n lr_start, lr_max, lr_min = 5e-5, 6e-6 * batch_size, 1e-5\n lr_ramp_ep, lr_sus_ep, lr_decay = 3, 0, 0.75\n\n def lrfn(epoch): # Learning rate update function\n if epoch < lr_ramp_ep: lr = (lr_max - lr_start) / lr_ramp_ep * epoch + lr_start\n elif epoch < lr_ramp_ep + lr_sus_ep: lr = lr_max\n elif mode == 'exp': lr = (lr_max - lr_min) * lr_decay**(epoch - lr_ramp_ep - lr_sus_ep) + lr_min\n elif mode == 'step': lr = lr_max * lr_decay**((epoch - lr_ramp_ep - lr_sus_ep) // 2)\n elif mode == 'cos':\n decay_total_epochs, decay_epoch_index = epochs - lr_ramp_ep - lr_sus_ep + 3, epoch - lr_ramp_ep - lr_sus_ep\n phase = math.pi * decay_epoch_index / decay_total_epochs\n lr = (lr_max - lr_min) * 0.5 * (1 + math.cos(phase)) + lr_min\n return lr\n\n if plot: # Plot lr curve if plot is True\n plt.figure(figsize=(10, 5))\n plt.plot(np.arange(epochs), [lrfn(epoch) for epoch in np.arange(epochs)], marker='o')\n plt.xlabel('epoch'); plt.ylabel('lr')\n plt.title('LR Scheduler')\n plt.show()\n\n return keras.callbacks.LearningRateScheduler(lrfn, verbose=False) # Create lr callback\n\nprint('done')","metadata":{"papermill":{"duration":0.064566,"end_time":"2024-02-01T08:29:40.597373","exception":false,"start_time":"2024-02-01T08:29:40.532807","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:34:11.467033Z","iopub.execute_input":"2024-04-29T21:34:11.467485Z","iopub.status.idle":"2024-04-29T21:34:11.480652Z","shell.execute_reply.started":"2024-04-29T21:34:11.467459Z","shell.execute_reply":"2024-04-29T21:34:11.479704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lr_cb = get_lr_callback(CFG.batch_size, mode=CFG.lr_mode, plot=True)\n\nprint('done')","metadata":{"papermill":{"duration":0.271771,"end_time":"2024-02-01T08:29:40.919864","exception":false,"start_time":"2024-02-01T08:29:40.648093","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:34:11.481748Z","iopub.execute_input":"2024-04-29T21:34:11.482072Z","iopub.status.idle":"2024-04-29T21:34:12.057874Z","shell.execute_reply.started":"2024-04-29T21:34:11.482029Z","shell.execute_reply":"2024-04-29T21:34:12.057001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐พ | Model Checkpointing","metadata":{"papermill":{"duration":0.050253,"end_time":"2024-02-01T08:29:41.021653","exception":false,"start_time":"2024-02-01T08:29:40.971400","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ckpt_cb = keras.callbacks.ModelCheckpoint(\"best_model.keras\",\n monitor='val_loss',\n save_best_only=True,\n save_weights_only=False,\n mode='min')\n\nprint('done')","metadata":{"papermill":{"duration":0.060079,"end_time":"2024-02-01T08:29:41.132069","exception":false,"start_time":"2024-02-01T08:29:41.071990","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:34:12.059205Z","iopub.execute_input":"2024-04-29T21:34:12.059859Z","iopub.status.idle":"2024-04-29T21:34:12.066078Z","shell.execute_reply.started":"2024-04-29T21:34:12.059724Z","shell.execute_reply":"2024-04-29T21:34:12.064980Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ | Training","metadata":{"papermill":{"duration":0.052404,"end_time":"2024-02-01T08:29:41.236033","exception":false,"start_time":"2024-02-01T08:29:41.183629","status":"completed"},"tags":[]}},{"cell_type":"code","source":"history = model.fit(\n train_ds, \n epochs=CFG.epochs,\n callbacks=[lr_cb, ckpt_cb], \n steps_per_epoch=len(train_df)//CFG.batch_size,\n validation_data=valid_ds, \n verbose=CFG.verbose\n)","metadata":{"papermill":{"duration":1005.294016,"end_time":"2024-02-01T08:46:26.588516","exception":false,"start_time":"2024-02-01T08:29:41.294500","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T21:34:12.067072Z","iopub.execute_input":"2024-04-29T21:34:12.068410Z","iopub.status.idle":"2024-04-29T22:49:44.240741Z","shell.execute_reply.started":"2024-04-29T21:34:12.068373Z","shell.execute_reply":"2024-04-29T22:49:44.239974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐งช | Prediction","metadata":{"papermill":{"duration":0.208758,"end_time":"2024-02-01T08:46:26.999476","exception":false,"start_time":"2024-02-01T08:46:26.790718","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Load Best Model","metadata":{"papermill":{"duration":0.203244,"end_time":"2024-02-01T08:46:27.406021","exception":false,"start_time":"2024-02-01T08:46:27.202777","status":"completed"},"tags":[]}},{"cell_type":"code","source":"model.load_weights(\"best_model.keras\")","metadata":{"papermill":{"duration":8.079925,"end_time":"2024-02-01T08:46:35.688531","exception":false,"start_time":"2024-02-01T08:46:27.608606","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T22:49:44.242397Z","iopub.execute_input":"2024-04-29T22:49:44.242692Z","iopub.status.idle":"2024-04-29T22:49:51.668938Z","shell.execute_reply.started":"2024-04-29T22:49:44.242666Z","shell.execute_reply":"2024-04-29T22:49:51.668127Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test Accuracy","metadata":{}},{"cell_type":"code","source":"'''\nimport gc\ngc.collect()\n'''\nprint('done')","metadata":{"execution":{"iopub.status.busy":"2024-04-29T22:49:51.670609Z","iopub.execute_input":"2024-04-29T22:49:51.670936Z","iopub.status.idle":"2024-04-29T22:49:51.675911Z","shell.execute_reply.started":"2024-04-29T22:49:51.670910Z","shell.execute_reply":"2024-04-29T22:49:51.674922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\ntrain_df.shape\nsample_df.shape\nvalid_df.shape\n\n\nvalid_ds\ntrain_ds\n\n\ndel train_df\ndel sample_df\ndel valid_df\ndel valid_ds\ndel train_ds\n'''\n\nprint('done')","metadata":{"execution":{"iopub.status.busy":"2024-04-29T22:49:51.676950Z","iopub.execute_input":"2024-04-29T22:49:51.677256Z","iopub.status.idle":"2024-04-29T22:49:51.685075Z","shell.execute_reply.started":"2024-04-29T22:49:51.677232Z","shell.execute_reply":"2024-04-29T22:49:51.684195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Test\ntest_paths_2 = test_df_2.spec2_path.values\ntest_offsets_2 = test_df_2.spectrogram_label_offset_seconds.values.astype(int)\ntest_labels_2 = test_df_2.class_label.values\ntest_ds_2 = build_dataset(test_paths_2, test_offsets_2, test_labels_2, batch_size=CFG.batch_size,\n repeat=False, shuffle=False, augment=False, cache=False) #cache=True)\n\nprint('done')","metadata":{"execution":{"iopub.status.busy":"2024-04-29T22:49:51.686108Z","iopub.execute_input":"2024-04-29T22:49:51.686434Z","iopub.status.idle":"2024-04-29T22:49:51.762313Z","shell.execute_reply.started":"2024-04-29T22:49:51.686387Z","shell.execute_reply":"2024-04-29T22:49:51.761283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"preds = model.predict(test_ds_2)","metadata":{"execution":{"iopub.status.busy":"2024-04-29T22:49:51.763519Z","iopub.execute_input":"2024-04-29T22:49:51.763827Z","iopub.status.idle":"2024-04-29T22:50:31.008770Z","shell.execute_reply.started":"2024-04-29T22:49:51.763784Z","shell.execute_reply":"2024-04-29T22:50:31.008028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_labels = np.argmax(preds, axis=1)\ncorrect_predictions = np.sum(predicted_labels == test_labels_2)\ntotal_samples = len(test_labels_2)\ntest_accuracy = correct_predictions / total_samples\n\nprint(\"Test Accuracy:\", test_accuracy)","metadata":{"execution":{"iopub.status.busy":"2024-04-29T22:50:31.010094Z","iopub.execute_input":"2024-04-29T22:50:31.010395Z","iopub.status.idle":"2024-04-29T22:50:31.017161Z","shell.execute_reply.started":"2024-04-29T22:50:31.010369Z","shell.execute_reply":"2024-04-29T22:50:31.015774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build Test Dataset","metadata":{"papermill":{"duration":0.200664,"end_time":"2024-02-01T08:46:36.090526","exception":false,"start_time":"2024-02-01T08:46:35.889862","status":"completed"},"tags":[]}},{"cell_type":"code","source":"'''\ntest_paths = test_df.spec2_path.values\ntest_ds = build_dataset(test_paths, batch_size=min(CFG.batch_size, len(test_df)),\n repeat=False, shuffle=False, cache=False, augment=False)\n\n'''\nprint('done')","metadata":{"papermill":{"duration":0.252329,"end_time":"2024-02-01T08:46:36.546799","exception":false,"start_time":"2024-02-01T08:46:36.294470","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T22:50:31.021969Z","iopub.execute_input":"2024-04-29T22:50:31.022254Z","iopub.status.idle":"2024-04-29T22:50:31.029115Z","shell.execute_reply.started":"2024-04-29T22:50:31.022220Z","shell.execute_reply":"2024-04-29T22:50:31.028160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference","metadata":{"papermill":{"duration":0.19961,"end_time":"2024-02-01T08:46:36.950400","exception":false,"start_time":"2024-02-01T08:46:36.750790","status":"completed"},"tags":[]}},{"cell_type":"code","source":"'''\npreds = model.predict(test_ds)\n'''\nprint('done')","metadata":{"papermill":{"duration":25.997706,"end_time":"2024-02-01T08:47:03.148947","exception":false,"start_time":"2024-02-01T08:46:37.151241","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T22:50:31.030167Z","iopub.execute_input":"2024-04-29T22:50:31.030408Z","iopub.status.idle":"2024-04-29T22:50:31.040528Z","shell.execute_reply.started":"2024-04-29T22:50:31.030387Z","shell.execute_reply":"2024-04-29T22:50:31.039686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ฉ | Submission","metadata":{"papermill":{"duration":0.203281,"end_time":"2024-02-01T08:47:03.557421","exception":false,"start_time":"2024-02-01T08:47:03.354140","status":"completed"},"tags":[]}},{"cell_type":"code","source":"'''\npred_df = test_df[[\"eeg_id\"]].copy()\ntarget_cols = [x.lower()+'_vote' for x in CFG.class_names]\npred_df[target_cols] = preds.tolist()\n\nsub_df = pd.read_csv(f'{BASE_PATH}/sample_submission.csv')\nsub_df = sub_df[[\"eeg_id\"]].copy()\nsub_df = sub_df.merge(pred_df, on=\"eeg_id\", how=\"left\")\nsub_df.to_csv(\"submission.csv\", index=False)\nsub_df.head()\n'''\nprint('done')","metadata":{"papermill":{"duration":0.256018,"end_time":"2024-02-01T08:47:04.017822","exception":false,"start_time":"2024-02-01T08:47:03.761804","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-04-29T22:50:31.041616Z","iopub.execute_input":"2024-04-29T22:50:31.042195Z","iopub.status.idle":"2024-04-29T22:50:31.051367Z","shell.execute_reply.started":"2024-04-29T22:50:31.042170Z","shell.execute_reply":"2024-04-29T22:50:31.050485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"'''\n\nhead(10)\nElapsed time: 4914.4774034023285\n\n'''\n\nend_time = time.time()\nelapsed_time = end_time - s_time\nprint(\"Elapsed time:\", elapsed_time)","metadata":{"execution":{"iopub.status.busy":"2024-04-29T22:50:31.052341Z","iopub.execute_input":"2024-04-29T22:50:31.052602Z","iopub.status.idle":"2024-04-29T22:50:31.062487Z","shell.execute_reply.started":"2024-04-29T22:50:31.052579Z","shell.execute_reply":"2024-04-29T22:50:31.061635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ๐ | Reference\n* [HMS-HBAC: ResNet34d Baseline [Training]](https://www.kaggle.com/code/ttahara/hms-hbac-resnet34d-baseline-training) \n* [EfficientNetB2 Starter - [LB 0.57]](https://www.kaggle.com/code/cdeotte/efficientnetb2-starter-lb-0-57)","metadata":{"papermill":{"duration":0.20353,"end_time":"2024-02-01T08:47:04.424508","exception":false,"start_time":"2024-02-01T08:47:04.220978","status":"completed"},"tags":[]}}]}
![\"Keras](\"https://keras.io/img/logo-small.png\"){kind=logo}