{"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.13"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install timm huggingface_hub kaggle -Uqq\n\nimport warnings\nimport timm\nimport gc\n\nfrom fastai.vision.all import *\nfrom fastcore.parallel import *\n\n#path = Path('/kaggle/input/hms-hbac-training-spectrogram-images/train_spectrograms')\npath = Path('/notebooks/hms-hbac-training-spectrogram-images/train_spectrograms')\npath.ls()","metadata":{"execution":{"iopub.execute_input":"2024-03-11T21:37:58.322058Z","iopub.status.busy":"2024-03-11T21:37:58.321356Z","iopub.status.idle":"2024-03-11T21:38:10.350853Z","shell.execute_reply":"2024-03-11T21:38:10.350218Z","shell.execute_reply.started":"2024-03-11T21:37:58.321984Z"}},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Background\n\nThis is the fourth notebook in a series of 5 notebooks where I train different `convnext` image classifiers on training spectrogram images using the fastai library:\n\n- [Part 0](https://www.kaggle.com/code/vishalbakshi/hms-hbac-fastai-planning-small-model-experiments): I plan out my initial small model experiments and create a [Kaggle dataset](https://www.kaggle.com/datasets/vishalbakshi/hms-hbac-training-spectrogram-images) with training spectrogram images.\n- [Part 1 [Train]](https://www.kaggle.com/code/vishalbakshi/hms-hbac-fastai-convnext-small-pt-1-train): I train 48 variants of `convnext_small_in22k` using different `ImageDataLoaders`.\n- [Part 1 [Analysis]](https://www.kaggle.com/code/vishalbakshi/hms-hbac-fastai-convnext-small-pt-1-analysis): I analyze the results from Part 1, run a few more trainings and pick the top convnext models for submission.**\n- **Part 2 [Train] (You are here): I train those top convnext models and export them to Kaggle.**\n- [Part 2 [Submit]](https://www.kaggle.com/code/vishalbakshi/hms-hbac-fastai-convnext-small-pt-2-submit): I submit those models individually and as ensembles, and document their Kaggle Public Score.\n\nI'll follow the same approach (experiment -> train and export top models -> submit -> document Kaggle Public Score) for three other families: `vit`, `swin` and `swinv2`. Once I have identified the best `small` models, I'll train their `large` versions, submit them and document the results. I have taken this general approach from Jeremy Howard's [Road to the Top](https://www.kaggle.com/code/jhoward/first-steps-road-to-the-top-part-1) notebook series (although his notebooks and presentation is much more efficient).","metadata":{}},{"cell_type":"markdown","source":"## Models Training Results","metadata":{}},{"cell_type":"markdown","source":"Here are the parameters and results of the models that I have trained in this notebook. It's good to see that even though each one is trained on a randomly set training and validation set, the TTA Validation Error Rate is similar to previous runs. \n\n|`item_tfms` method|`item_tfms` size|`batch_tfms` size|TTA Validation Error Rate|Final Epoch Validation Error Rate|Minutes/Epoch|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|squish|(311, 400)|None|0.3651|\t0.365065|1.78|\n|pad|400|None|0.3696|0.380332|2.20|\n|crop|(400, 311)|None|0.3529|0.365963|1.77|\n|squish|(400, 311)|None|0.3642|0.364167|1.77|\n|pad|(320, 512)|None|0.3642|0.368657|2.23|","metadata":{}},{"cell_type":"markdown","source":"## Training Runs","metadata":{}},{"cell_type":"markdown","source":"I'll create my helper function as done in the previous notebooks (copied from Jeremy Howard's [Road to the Top series](https://www.kaggle.com/code/jhoward/first-steps-road-to-the-top-part-1)). \n\nSince none of my `DataLoaders` will have `batch_tfms` I'll assign that as `None`. I'm also sticking with `Gradient Accumulation` so that will occur every `64` images and the batch size will be `16`.\n\nI am not comparing performance across models so I'll be removing the `seed`. Because of this, I'll plot my losses for each model to see how it performs in the different validation sets.\n\nFinally, I'll export the model so that I can then upload it to Kaggle for use in my submission notebook.","metadata":{}},{"cell_type":"code","source":"def train(arch, item, fn):\n        \n    dls = ImageDataLoaders.from_folder(\n        path, \n        valid_pct=0.2, \n        item_tfms=item,\n        batch_tfms=None,\n        bs=16)\n    \n    cbs = GradientAccumulation(64)\n    learn = vision_learner(dls, arch, metrics=error_rate, cbs=cbs).to_fp16()\n    learn.fine_tune(7, 0.01)\n    learn.recorder.plot_loss()\n    learn.save(fn, with_opt=False)\n    \n    print(\"TTA Validation Error Rate:\", error_rate(*learn.tta(dl=dls.valid)))\n    ","metadata":{"execution":{"iopub.execute_input":"2024-03-11T21:57:51.575564Z","iopub.status.busy":"2024-03-11T21:57:51.575099Z","iopub.status.idle":"2024-03-11T21:57:51.579329Z","shell.execute_reply":"2024-03-11T21:57:51.578826Z","shell.execute_reply.started":"2024-03-11T21:57:51.575541Z"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"arch = 'convnext_small_in22k'","metadata":{"execution":{"iopub.execute_input":"2024-03-11T21:58:06.009745Z","iopub.status.busy":"2024-03-11T21:58:06.009247Z","iopub.status.idle":"2024-03-11T21:58:06.012573Z","shell.execute_reply":"2024-03-11T21:58:06.011966Z","shell.execute_reply.started":"2024-03-11T21:58:06.009722Z"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train(arch, item=Resize((311,400), method='squish'), fn=f\"/notebooks/hms_hbac_{arch}_1\")","metadata":{"execution":{"iopub.execute_input":"2024-03-11T21:58:11.942197Z","iopub.status.busy":"2024-03-11T21:58:11.941726Z","iopub.status.idle":"2024-03-11T22:13:35.118869Z","shell.execute_reply":"2024-03-11T22:13:35.117893Z","shell.execute_reply.started":"2024-03-11T21:58:11.942197Z"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train(arch, item=Resize((400), method=ResizeMethod.Pad, pad_mode=PadMode.Zeros), fn=f\"/notebooks/hms_hbac_{arch}_2\")","metadata":{"execution":{"iopub.execute_input":"2024-03-11T22:13:35.120252Z","iopub.status.busy":"2024-03-11T22:13:35.120081Z","iopub.status.idle":"2024-03-11T22:31:49.821140Z","shell.execute_reply":"2024-03-11T22:31:49.820513Z","shell.execute_reply.started":"2024-03-11T22:13:35.120234Z"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train(arch, item=Resize((400,311)), fn=f\"/notebooks/hms_hbac_{arch}_3\")","metadata":{"execution":{"iopub.execute_input":"2024-03-11T22:31:49.822255Z","iopub.status.busy":"2024-03-11T22:31:49.822073Z","iopub.status.idle":"2024-03-11T22:46:32.849560Z","shell.execute_reply":"2024-03-11T22:46:32.848776Z","shell.execute_reply.started":"2024-03-11T22:31:49.822237Z"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train(arch, item=Resize((400,311), method='squish'), fn=f\"/notebooks/hms_hbac_{arch}_4\")","metadata":{"execution":{"iopub.execute_input":"2024-03-11T22:46:32.851874Z","iopub.status.busy":"2024-03-11T22:46:32.851211Z","iopub.status.idle":"2024-03-11T23:01:19.579832Z","shell.execute_reply":"2024-03-11T23:01:19.579199Z","shell.execute_reply.started":"2024-03-11T22:46:32.851850Z"}},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train(arch, item=Resize((320,512), method=ResizeMethod.Pad, pad_mode=PadMode.Zeros), fn=f\"/notebooks/hms_hbac_{arch}_5\")","metadata":{"execution":{"iopub.execute_input":"2024-03-11T23:01:19.581111Z","iopub.status.busy":"2024-03-11T23:01:19.580898Z","iopub.status.idle":"2024-03-11T23:19:56.761399Z","shell.execute_reply":"2024-03-11T23:19:56.760840Z","shell.execute_reply.started":"2024-03-11T23:01:19.581091Z"}},"execution_count":null,"outputs":[]}]}