{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":7781194,"sourceType":"datasetVersion","datasetId":4553461},{"sourceId":18120,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":15062},{"sourceId":18121,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":15063},{"sourceId":18123,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":15065},{"sourceId":18124,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":15066},{"sourceId":18125,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":15067}],"dockerImageVersionId":30665,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Setup","metadata":{}},{"cell_type":"code","source":"import warnings\nimport timm\nfrom fastai.vision.all import *\nfrom fastcore.parallel import *\n\npath = Path('/kaggle/input/hms-harmful-brain-activity-classification')\n\npath.ls()","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-03-19T21:01:23.554503Z","iopub.execute_input":"2024-03-19T21:01:23.554896Z","iopub.status.idle":"2024-03-19T21:01:33.848055Z","shell.execute_reply.started":"2024-03-19T21:01:23.554868Z","shell.execute_reply":"2024-03-19T21:01:33.846997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Background\n\nThis is the fifth notebook in a series of 5 notebooks where I train different `vit_small_patch16_224` image classifiers on training spectrogram images using the fastai library:\n\n- [Part 0](https://www.kaggle.com/code/vishalbakshi/hms-hbac-fastai-planning-small-model-experiments): I plan out my initial small model experiments and create a [Kaggle dataset](https://www.kaggle.com/datasets/vishalbakshi/hms-hbac-training-spectrogram-images) with training spectrogram images.\n- [Part 1 [Train]](): I train 57 variants of `vit_small_patch16_224` using different `ImageDataLoaders`.\n- [Part 1 [Analysis]](): I analyze the training results from Part 1, run a few more trainings and pick the top vit models for submission.\n- [Part 2 Train](): I train those top vit models and export them to Kaggle.\n- **Part 2 [Submit] (You are here): I submit those models individually and as ensembles, and document their Kaggle Public Score.**","metadata":{}},{"cell_type":"markdown","source":"In this notebook I submit predictions from each of the 5 `vit_small_patch16_224` models I've trained, as well as ensembles of all 5 models.","metadata":{}},{"cell_type":"markdown","source":"## Submission Results","metadata":{}},{"cell_type":"markdown","source":"Here are the results for the single-model submissions I made using my `vit_small_patch16_224` models, sorted from best to worse Public Score:\n\n|Model Name|item method|item img size|batch_tfms|Public Score|\n|:-:|:-:|:-:|:-:|:-:|\n|AW|crop|256 --> (400, 311) --> (320, 512)|RandomResizedCropGPU(size=224, min_scale=1.0)|1.65|\n|BB|pad|256 --> 400 --> (320, 512)|aug_transforms(size=224, min_scale=0.75)|1.73|\n|AU|squish|256 --> (400, 311) --> (320, 512)|aug_transforms(size=224, min_scale=0.75)|1.79|\n|BA|squish|256 --> 400 --> (320, 512)|aug_transforms(size=224, min_scale=0.75)|1.80|\n|AO|squish|256 --> (400, 311) --> (320, 512)|aug_transforms(size=224, min_scale=0.75)|1.87|\n\nHere are the results of 5- and 6-model ensemble submissions using the above models. In the 6-model submissions I weighted one of the models twice. \n\n|Model Weighted Twice|Public Score|\n|:-:|:-:|\n|AW|1.58|\n|BB|1.60|\n|AU|1.62|\n|BA|1.62|\n|--|1.62|\n|AO|1.63|\n\nI submitted 10 three-model combinations as well, here are the results:\n\n|Ensemble|Public Score|\n|:-:|:-:|\n|AW-BB-AU|1.57|\n|AW-BB-BA|1.57|\n|AW-BB-AO|1.59|\n|AW-AU-BA|1.60|\n|AW-AU-AO|1.62|\n|AW-BA-AO|1.62|\n|BB-AU-BA|1.68|\n|BB-AU-AO|1.70|\n|BB-BA-AO|1.70|\n|AU-BA-AO|1.73|\n\nI'll pick AW, BB and AU as the three top models. AW and BB had the best scores in all three categories: individual, weighted-twice-in-5-model-ensemble and 3-ensemble submissions. AU and BA performed equally well (1.57) in the 3-model ensemble, and when weighted twice in the 5-model ensemble. AU had the better single model submission Kaggle Public Score so it edges out with the win.\n\nSummarizing the three top models---interesting to note that each item method is represented:\n\n|Model Name|item method|item img size|batch_tfms|Public Score|\n|:-:|:-:|:-:|:-:|:-:|\n|AW|crop|256 --> (400, 311) --> (320, 512)|RandomResizedCropGPU(size=224, min_scale=1.0)|1.65|\n|BB|pad|256 --> 400 --> (320, 512)|aug_transforms(size=224, min_scale=0.75)|1.73|\n|AU|squish|256 --> (400, 311) --> (320, 512)|aug_transforms(size=224, min_scale=0.75)|1.79|","metadata":{}},{"cell_type":"markdown","source":"## Generate Test Data Images","metadata":{}},{"cell_type":"markdown","source":"My models are image classifiers trained on spectrogram images so I need to convert the test parquet data to images. I'm referencing the following notebooks:\n\n- [HMS - HBAC - Fastai Starter](https://www.kaggle.com/code/sonujha090/hms-hbac-fastai-starter)\n- [HMS-HBAC: KerasCV Starter Notebook](https://www.kaggle.com/code/awsaf49/hms-hbac-kerascv-starter-notebook)","metadata":{}},{"cell_type":"code","source":"# create temporary folders to hold spectrograms\nSPEC_DIR = \"/tmp/dataset/hms-hbac\"\nos.makedirs(SPEC_DIR+'/test_spectrograms', exist_ok=True)","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:33.849882Z","iopub.execute_input":"2024-03-19T21:01:33.850354Z","iopub.status.idle":"2024-03-19T21:01:33.855345Z","shell.execute_reply.started":"2024-03-19T21:01:33.850326Z","shell.execute_reply":"2024-03-19T21:01:33.854341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def process_spec(spec_id, split=\"train\"):\n    # read the data\n    data = pd.read_parquet(path/f'{split}_spectrograms'/f'{spec_id}.parquet')\n    \n    # replace NA with 0\n    data = data.fillna(0)\n    \n    # convert DataFrame to array\n    data = data.values[:, 1:]\n    \n    # transpose\n    data = data.T\n    data = data.astype(\"float32\")\n    \n    # convert array to PILImage\n    im = PILImage.create(Image.fromarray((data * 255).astype(np.uint8)))\n    im.save(f\"{SPEC_DIR}/{split}_spectrograms/{spec_id}.png\")","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:33.856882Z","iopub.execute_input":"2024-03-19T21:01:33.857362Z","iopub.status.idle":"2024-03-19T21:01:33.864450Z","shell.execute_reply.started":"2024-03-19T21:01:33.857328Z","shell.execute_reply":"2024-03-19T21:01:33.863532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.read_csv(path/'test.csv')\ntest_df.head(3)","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:33.866314Z","iopub.execute_input":"2024-03-19T21:01:33.866636Z","iopub.status.idle":"2024-03-19T21:01:33.887753Z","shell.execute_reply.started":"2024-03-19T21:01:33.866613Z","shell.execute_reply":"2024-03-19T21:01:33.886783Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spec_ids = test_df['spectrogram_id'].unique()\nlen(spec_ids)","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:33.888759Z","iopub.execute_input":"2024-03-19T21:01:33.889029Z","iopub.status.idle":"2024-03-19T21:01:33.898126Z","shell.execute_reply.started":"2024-03-19T21:01:33.889007Z","shell.execute_reply":"2024-03-19T21:01:33.897019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"warnings.filterwarnings(\"ignore\")\nparallel(process_spec, spec_ids, split='test', n_workers=4)\nwarnings.filterwarnings(\"default\")","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:33.899254Z","iopub.execute_input":"2024-03-19T21:01:33.899512Z","iopub.status.idle":"2024-03-19T21:01:34.269252Z","shell.execute_reply.started":"2024-03-19T21:01:33.899492Z","shell.execute_reply":"2024-03-19T21:01:34.267862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PILImage.create(Path('/tmp/dataset/hms-hbac/test_spectrograms').ls()[0])","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:34.270741Z","iopub.execute_input":"2024-03-19T21:01:34.271079Z","iopub.status.idle":"2024-03-19T21:01:34.337684Z","shell.execute_reply.started":"2024-03-19T21:01:34.271051Z","shell.execute_reply":"2024-03-19T21:01:34.336679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating the `DataLoaders` Object","metadata":{}},{"cell_type":"markdown","source":"I already have created a dataset with training images, so I'll load that into my notebook and create a training path `trn_path` to use in my `DataLoaders`.","metadata":{}},{"cell_type":"code","source":"trn_path = Path('/kaggle/input/hms-hbac-training-spectrogram-images/train_spectrograms')","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:34.340798Z","iopub.execute_input":"2024-03-19T21:01:34.341234Z","iopub.status.idle":"2024-03-19T21:01:34.345998Z","shell.execute_reply.started":"2024-03-19T21:01:34.341203Z","shell.execute_reply":"2024-03-19T21:01:34.345011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Each model was trained with different `item_tfms` and `batch_tfms` so I'll list them out here.","metadata":{}},{"cell_type":"code","source":"item1 = Resize((320,512), method='squish') # Model BA\nitem2 = Resize((320,512), method='squish') # Model AO\nitem3 = Resize((320,512), method='squish') # Model AU\nitem4 = Resize((320,512), method=ResizeMethod.Pad, pad_mode=PadMode.Zeros) # Model BB\nitem5 = Resize((320,512), method='crop') # Model AW","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:34.347302Z","iopub.execute_input":"2024-03-19T21:01:34.347680Z","iopub.status.idle":"2024-03-19T21:01:34.355928Z","shell.execute_reply.started":"2024-03-19T21:01:34.347647Z","shell.execute_reply":"2024-03-19T21:01:34.355028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch1 = aug_transforms(size=224, min_scale=0.75) # Model BA\nbatch2 = aug_transforms(size=224, min_scale=0.75)# Model AO\nbatch3 = aug_transforms(size=224, min_scale=0.75)# Model AU\nbatch4 = aug_transforms(size=224, min_scale=0.75)# Model BB\nbatch5 = RandomResizedCropGPU(size=224, min_scale=1.0) # Model AW","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:34.357416Z","iopub.execute_input":"2024-03-19T21:01:34.357845Z","iopub.status.idle":"2024-03-19T21:01:34.372838Z","shell.execute_reply.started":"2024-03-19T21:01:34.357812Z","shell.execute_reply":"2024-03-19T21:01:34.371947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Ensemble","metadata":{}},{"cell_type":"markdown","source":"I'll create a list of `DataLoaders`, one for each of the five models that I am using in this ensemble.","metadata":{}},{"cell_type":"code","source":"dls_list = []\nitems = [item1, item2, item3, item4, item5]\nbatches = [batch1, batch2, batch3, batch4, batch5]\n\nfor i in range(5):\n    dls = ImageDataLoaders.from_folder(\n        trn_path, \n        valid_pct=0.2, \n        item_tfms=items[i],\n        batch_tfms=batches[i],\n        bs=16)\n    \n    dls_list.append(dls)","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:34.374085Z","iopub.execute_input":"2024-03-19T21:01:34.374691Z","iopub.status.idle":"2024-03-19T21:01:46.673524Z","shell.execute_reply.started":"2024-03-19T21:01:34.374658Z","shell.execute_reply":"2024-03-19T21:01:46.672520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dls_list","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:46.674685Z","iopub.execute_input":"2024-03-19T21:01:46.674961Z","iopub.status.idle":"2024-03-19T21:01:46.681038Z","shell.execute_reply.started":"2024-03-19T21:01:46.674938Z","shell.execute_reply":"2024-03-19T21:01:46.680104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, I'll create a list of `Learner`s, one for each model.","metadata":{}},{"cell_type":"code","source":"model_paths = [\n    Path('/kaggle/input/hms-hbac-fine-tuned-vit_small_patch16_224/pytorch/ba/1/hms_hbac_vit_small_patch16_224_BA'),\n    Path('/kaggle/input/hms-hbac-fine-tuned-vit_small_patch16_224/pytorch/ao/1/hms_hbac_vit_small_patch16_224_AO'),\n    Path('/kaggle/input/hms-hbac-fine-tuned-vit_small_patch16_224/pytorch/au/1/hms_hbac_vit_small_patch16_224_AU'),\n    Path('/kaggle/input/hms-hbac-fine-tuned-vit_small_patch16_224/pytorch/bb/1/hms_hbac_vit_small_patch16_224_BB'),\n    Path('/kaggle/input/hms-hbac-fine-tuned-vit_small_patch16_224/pytorch/aw/1/hms_hbac_vit_small_patch16_224_AW')\n]","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:46.684378Z","iopub.execute_input":"2024-03-19T21:01:46.684716Z","iopub.status.idle":"2024-03-19T21:01:46.690173Z","shell.execute_reply.started":"2024-03-19T21:01:46.684688Z","shell.execute_reply":"2024-03-19T21:01:46.689126Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learners = []\n\nfor idx, model_path in enumerate(model_paths):\n    learn = vision_learner(dls_list[idx], 'vit_small_patch16_224', pretrained=False)\n    learn.model_dir = '/kaggle/working/'\n    learn.load(model_path)\n    learners.append(learn)","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:46.691196Z","iopub.execute_input":"2024-03-19T21:01:46.691458Z","iopub.status.idle":"2024-03-19T21:01:53.811957Z","shell.execute_reply.started":"2024-03-19T21:01:46.691436Z","shell.execute_reply":"2024-03-19T21:01:53.811153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learners","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:53.813160Z","iopub.execute_input":"2024-03-19T21:01:53.813450Z","iopub.status.idle":"2024-03-19T21:01:53.819673Z","shell.execute_reply.started":"2024-03-19T21:01:53.813419Z","shell.execute_reply":"2024-03-19T21:01:53.818646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, I'll get a `DataFrame` of probabilities for each test image from each `Learner` in my 3-model ensemble.","metadata":{}},{"cell_type":"code","source":"probs_df_list = []\n\ntst_files = get_image_files(SPEC_DIR+'/test_spectrograms')\n\nfor idx in range(5):\n    # create test DataLoader\n    tst_dl = dls_list[idx].test_dl(tst_files)\n    \n    # get TTA predictions\n    probs,_= learners[idx].tta(dl=tst_dl)\n    \n    # formatting\n    probs_df = pd.DataFrame(probs, columns=dls_list[idx].vocab)\n    probs_df['eeg_id'] = test_df['eeg_id']\n    probs_df = probs_df[['eeg_id', 'seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']]\n    \n    probs_df_list.append(probs_df)","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:53.820899Z","iopub.execute_input":"2024-03-19T21:01:53.821182Z","iopub.status.idle":"2024-03-19T21:01:59.137539Z","shell.execute_reply.started":"2024-03-19T21:01:53.821159Z","shell.execute_reply":"2024-03-19T21:01:59.136357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Concatenate all `DataFrames`:","metadata":{}},{"cell_type":"code","source":"all_probs_df = pd.concat([probs_df_list[i] for i in [2, 0, 1]])\nall_probs_df.shape","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:01:59.139501Z","iopub.execute_input":"2024-03-19T21:01:59.140516Z","iopub.status.idle":"2024-03-19T21:01:59.148526Z","shell.execute_reply.started":"2024-03-19T21:01:59.140479Z","shell.execute_reply":"2024-03-19T21:01:59.147426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Take the mean value of each vote by `eeg_id`","metadata":{}},{"cell_type":"code","source":"final_probs = all_probs_df.groupby('eeg_id').mean().reset_index()","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:02:31.886085Z","iopub.execute_input":"2024-03-19T21:02:31.886464Z","iopub.status.idle":"2024-03-19T21:02:31.898197Z","shell.execute_reply.started":"2024-03-19T21:02:31.886430Z","shell.execute_reply":"2024-03-19T21:02:31.897019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_probs.head()","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:02:33.401522Z","iopub.execute_input":"2024-03-19T21:02:33.401930Z","iopub.status.idle":"2024-03-19T21:02:33.416878Z","shell.execute_reply.started":"2024-03-19T21:02:33.401898Z","shell.execute_reply":"2024-03-19T21:02:33.415280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And export it:","metadata":{}},{"cell_type":"code","source":"final_probs.to_csv('submission.csv', index=False)\n!head submission.csv","metadata":{"execution":{"iopub.status.busy":"2024-03-19T21:02:43.856064Z","iopub.execute_input":"2024-03-19T21:02:43.856918Z","iopub.status.idle":"2024-03-19T21:02:44.883356Z","shell.execute_reply.started":"2024-03-19T21:02:43.856875Z","shell.execute_reply":"2024-03-19T21:02:44.881894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}