{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceType":"competition","sourceId":11848,"databundleVersionId":862157}],"dockerImageVersionId":23026,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\n`fastai` is a free deep learning API built on [PyTorch V1](https://pytorch.org/). The [fast.ai team](https://www.fast.ai/2018/10/02/fastai-ai/) incorporates their reseach breakthroughs into the software, enabling users to achieve more accurate results faster and with fewer lines of code.\n\nThis kernel illustrates the simplicity of deploying the `fastai.vision` package for image classification tasks. I am in no way a domain expert in this topic, in fact having no domain knowledge at all before this competition! I will heavily rely on published kernels (which are all referenced under [Acknowledgements](#Acknowledgements)) in guidance for setting hyperparameters in this task.\n\nI will be deploying standard techniques taught in the fast.ai course to see how well these techniques can perform without needing expert knowledge. The techniques are:\n1. Learning rate finder\n2. 1-cycle learning\n3. Differential learning rates for model finetuning\n4. Data augmentation\n5. Test time augmentation\n6. Transfer learning via low-resolution images\n\nThis kernel had the previous name of **Minimal fast.ai kit for image classification**, which is a slight misnomer now, considering the detailed techniques being deployed in this image classification task.","metadata":{"_uuid":"bfee4bed92c0290324e0b3f8287f32a3a679bda8"}},{"cell_type":"code","source":"%reload_ext autoreload\n%autoreload 2\n%matplotlib inline","metadata":{"trusted":true,"_uuid":"5199f8344f95af378935c472b568ec3c2ede273c","execution":{"iopub.status.busy":"2026-03-17T10:55:54.750736Z","iopub.execute_input":"2026-03-17T10:55:54.751019Z","iopub.status.idle":"2026-03-17T10:55:54.823936Z","shell.execute_reply.started":"2026-03-17T10:55:54.750976Z","shell.execute_reply":"2026-03-17T10:55:54.823145Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Contents\n1. [Skeleton code](#Skeleton-code)\n* [Import packages](#Import-packages)\n* [Exploratory data analysis](#Exploratory-data-analysis)\n* [Data loading and preparation](#Data-loading-and-preparation)\n* [Model creation](#Model-creation)\n* [Model training](#Model-training).\n* [Model interpretation](#Model-interpretation)\n* [Transfer learning](#Transfer-learning)\n* [Generating submission](#Generating-submission)\n* [Future work](#Future-work)\n* [Acknowledgements](#Acknowledgements)","metadata":{"_uuid":"97c86a45e1c6efbfd043f231d1830c0ef5e01001"}},{"cell_type":"markdown","source":"# Skeleton code","metadata":{"_uuid":"002376decfa096a06d59a2549946241395e4796b"}},{"cell_type":"markdown","source":"Following from the initial idea of showing the simplicity of using the `fastai` library, below is a code snippet containing 27 lines of code using default settings for a base model generation. Of these 27 lines, 10 lines are used to generate the submission file for the required format.\n```\nfrom fastai import *\nfrom fastai.vision import *\n\ndata = ImageDataBunch.from_csv(path, folder = 'train', csv_labels = \"train_labels.csv\",\n                               test = 'test',suffix=\".tif\", size = 36, ds_tfms = get_transforms())\ndata.path = pathlib.Path('.')\ndata.normalize(imagenet_stats)\n\nlearn = create_cnn(data,resnet50,pretrained = True,metrics = accuracy)\nlearn.fit_one_cycle(5)\n\nlearn.unfreeze()\nlearn.lr_find()\nlearn.recorder.plot()\nlearn.fit_one_cycle(3,max_lr = slice(1e-6,3e-4))\n\ninterp = ClassificationInterpretation.from_learner(learn)\ninterp.plot_top_losses(9)\ninterp.plot_confusion_matrix()\npreds,y = learn.TTA()\nacc = accuracy(preds, y)\nprint('The validation accuracy is {} %.'.format(acc * 100))\n\ndef generateSubmission(learner):\n    submissions = pd.read_csv('../input/sample_submission.csv')\n    id_list = list(submissions.id)\n    preds,y = learner.TTA(ds_type=DatasetType.Test)\n    pred_list = list(preds[:,1])\n    pred_dict = dict((key, value.item()) for (key, value) in zip(learner.data.test_ds.items,pred_list))\n    pred_ordered = [pred_dict[Path('../input/test/' + id + '.tif')] for id in id_list]\n    submissions = pd.DataFrame({'id':id_list,'label':pred_ordered})\n    submissions.to_csv(\"submission_{}.csv\".format(pred_score),index = False)\n \n generateSubmission(learn)\n```","metadata":{"_uuid":"3dda38b3b35637e4586578030193b03577042b46"}},{"cell_type":"markdown","source":"# Import packages","metadata":{"_uuid":"66ba8baf0a9169ffa65cb5d929f9a7c71a110a07"}},{"cell_type":"code","source":"from fastai import * \nfrom fastai.vision import *\nfrom torchvision.models import * \n\nimport os\nimport matplotlib.pyplot as plt","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-03-17T09:18:59.792239Z","iopub.execute_input":"2026-03-17T09:18:59.792488Z","iopub.status.idle":"2026-03-17T09:19:01.604947Z","shell.execute_reply.started":"2026-03-17T09:18:59.792452Z","shell.execute_reply":"2026-03-17T09:19:01.604151Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Exploratory data analysis","metadata":{"_uuid":"ad369e648b31bd9f30a1669304039abbba45bc8b"}},{"cell_type":"markdown","source":"Exploratory data analysis should be the first step of every data science task. Due to the lack of domain knowledge, I will only check for the number of classes and the number of items per class. Imbalanced datasets may require resampling of the data to ensure proper training.","metadata":{"_uuid":"ba2868a0e0708ee05e30bdb2cbc02fcc4ec435af"}},{"cell_type":"code","source":"path = Path(\"../input\")\nlabels = pd.read_csv(path/\"train_labels.csv\")\nlabels.head()","metadata":{"trusted":true,"_uuid":"fd9cb538747b30912bc948454a7fdfb717c2f9c4","execution":{"iopub.status.busy":"2026-03-17T09:19:14.349154Z","iopub.execute_input":"2026-03-17T09:19:14.349412Z","iopub.status.idle":"2026-03-17T09:19:14.821674Z","shell.execute_reply.started":"2026-03-17T09:19:14.349373Z","shell.execute_reply":"2026-03-17T09:19:14.820919Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(labels[\"label\"].nunique()); classes = list(set(labels[\"label\"])); classes","metadata":{"trusted":true,"_uuid":"6e895075bf84201f1990692769dd2b9e22ec11c0","execution":{"iopub.status.busy":"2026-03-17T10:56:02.777746Z","iopub.execute_input":"2026-03-17T10:56:02.778062Z","iopub.status.idle":"2026-03-17T10:56:02.845268Z","shell.execute_reply.started":"2026-03-17T10:56:02.778Z","shell.execute_reply":"2026-03-17T10:56:02.844384Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for i in classes:\n    print(\"Number of items in class {} is {}\".format(i,len(labels[labels[\"label\"] == i])))","metadata":{"trusted":true,"_uuid":"6fb77e492275f2c3cbbdc63c2f2e2de24699ba66","execution":{"iopub.status.busy":"2026-03-17T09:19:36.01017Z","iopub.execute_input":"2026-03-17T09:19:36.010415Z","iopub.status.idle":"2026-03-17T09:19:36.056186Z","shell.execute_reply.started":"2026-03-17T09:19:36.010375Z","shell.execute_reply":"2026-03-17T09:19:36.055438Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data loading and preparation","metadata":{"_uuid":"bc506c3e1586e44fc51f717f3a3e99778cd06219"}},{"cell_type":"code","source":"tfms = get_transforms(do_flip = True,flip_vert = True,max_zoom = 1.1)","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"execution":{"iopub.status.busy":"2026-03-17T09:19:50.771155Z","iopub.execute_input":"2026-03-17T09:19:50.771427Z","iopub.status.idle":"2026-03-17T09:19:50.801088Z","shell.execute_reply.started":"2026-03-17T09:19:50.771385Z","shell.execute_reply":"2026-03-17T09:19:50.800165Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"np.random.seed(123)\nsz = 32\ndata = ImageDataBunch.from_csv(path, folder = 'train', csv_labels = \"train_labels.csv\",\n                               test = 'test',suffix=\".tif\", size = sz,bs = 256,\n                               ds_tfms = tfms)\ndata.path = pathlib.Path('.')\ndata.normalize(imagenet_stats)","metadata":{"trusted":true,"_uuid":"91b943362c6451ea74111faff902ce73d0bccb85","execution":{"iopub.status.busy":"2026-03-17T09:19:54.340767Z","iopub.execute_input":"2026-03-17T09:19:54.341022Z","iopub.status.idle":"2026-03-17T09:21:43.995633Z","shell.execute_reply.started":"2026-03-17T09:19:54.340984Z","shell.execute_reply":"2026-03-17T09:21:43.990565Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(data.classes); data.c","metadata":{"trusted":true,"_uuid":"fed6259eb419198319bd7d5d6d9544444629f603","execution":{"iopub.status.busy":"2026-03-17T09:33:11.57741Z","iopub.execute_input":"2026-03-17T09:33:11.577733Z","iopub.status.idle":"2026-03-17T09:33:11.604592Z","shell.execute_reply.started":"2026-03-17T09:33:11.577677Z","shell.execute_reply":"2026-03-17T09:33:11.603869Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data.show_batch(rows = 3)","metadata":{"trusted":true,"_uuid":"aeffe6d8793b6b224490949bfe1d47bb3ca03e74","execution":{"iopub.status.busy":"2026-03-17T09:33:14.591481Z","iopub.execute_input":"2026-03-17T09:33:14.591816Z","iopub.status.idle":"2026-03-17T09:33:24.000615Z","shell.execute_reply.started":"2026-03-17T09:33:14.591749Z","shell.execute_reply":"2026-03-17T09:33:23.999291Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model creation","metadata":{"_uuid":"b66e8bb47853bf47cdd98fdfed31900b70f84d1b"}},{"cell_type":"markdown","source":"Submissions into the competition are [evaluated on the area under the ROC curve](https://www.kaggle.com/c/histopathologic-cancer-detection#evaluation) between the predicted probability and the observed target. Since we have a limited number of submissions per day, implementing a metric for the ROC AUC (which is non-standard in the fast.ai v1 library) allows us to run as many experiments we want.\n\nAt this point, I am not sure if changing the metric changes the loss function in the `Learner` to optimize the metric. I will be doing more reading up in that area. If anyone knows the answer to this, leave something in the comments below!","metadata":{"_uuid":"8e814e755b16e489d9de8ef9f89f889b911e5662"}},{"cell_type":"code","source":"from sklearn.metrics import roc_auc_score\n\ndef auc_score(y_pred,y_true,tens=True):\n    score = roc_auc_score(y_true,torch.sigmoid(y_pred)[:,1])\n    if tens:\n        score = tensor(score)\n    return score","metadata":{"trusted":true,"_uuid":"ff61e57537f8f611a2efc11e2e7b846179e00c10","execution":{"iopub.status.busy":"2026-03-17T09:33:31.419389Z","iopub.execute_input":"2026-03-17T09:33:31.419724Z","iopub.status.idle":"2026-03-17T09:33:31.70852Z","shell.execute_reply.started":"2026-03-17T09:33:31.419645Z","shell.execute_reply":"2026-03-17T09:33:31.707757Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"arch = models.resnet50\nlearn = cnn_learner(data,arch,pretrained = True,ps = 0.45,metrics = [auc_score,accuracy])","metadata":{"trusted":true,"_uuid":"038f7bb44f5e850827e718aade8715d44e33e35e","execution":{"iopub.status.busy":"2026-03-17T09:33:39.659729Z","iopub.execute_input":"2026-03-17T09:33:39.660018Z","iopub.status.idle":"2026-03-17T09:33:41.70226Z","shell.execute_reply.started":"2026-03-17T09:33:39.659975Z","shell.execute_reply":"2026-03-17T09:33:41.701367Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\n\nclass SEBlock(nn.Module):\n    def __init__(self, channels, reduction=16):\n        super(SEBlock, self).__init__()\n        self.avg_pool = nn.AdaptiveAvgPool2d(1)\n        self.fc = nn.Sequential(\n            nn.Conv2d(channels, channels // reduction, kernel_size=1, bias=False),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(channels // reduction, channels, kernel_size=1, bias=False),\n            nn.Sigmoid()\n        )\n\n    def forward(self, x):\n        b, c, _, _ = x.size()\n        y = self.avg_pool(x).view(b, c, 1, 1)\n        y = self.fc(y)\n        return x * y.expand_as(x)\n\nprint(\"SEBlock class defined successfully.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-17T09:33:46.459455Z","iopub.execute_input":"2026-03-17T09:33:46.459814Z","iopub.status.idle":"2026-03-17T09:33:46.490643Z","shell.execute_reply.started":"2026-03-17T09:33:46.459742Z","shell.execute_reply":"2026-03-17T09:33:46.489599Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"class SEBottleneck(nn.Module):\n    expansion = 4\n\n    def __init__(self, inplanes, planes, stride=1, downsample=None, reduction=16):\n        super(SEBottleneck, self).__init__()\n        sel\n        \n        \n        \n        \n        \n        f.conv1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)\n        self.bn1 = nn.BatchNorm2d(planes)\n        self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)\n        self.bn2 = nn.BatchNorm2d(planes)\n        self.conv3 = nn.Conv2d(planes, planes * self.expansion, kernel_size=1, bias=False)\n        self.bn3 = nn.BatchNorm2d(planes * self.expansion)\n        self.relu = nn.ReLU(inplace=True)\n        self.se_block = SEBlock(planes * self.expansion, reduction)\n        self.downsample = downsample\n        self.stride = stride\n\n    def forward(self, x):\n        identity = x\n\n        out = self.conv1(x)\n        out = self.bn1(out)\n        out = self.relu(out)\n\n        out = self.conv2(out)\n        out = self.bn2(out)\n        out = self.relu(out)\n\n        out = self.conv3(out)\n        out = self.bn3(out)\n        \n        # Apply SEBlock here\n        out = self.se_block(out)\n\n        if self.downsample is not None:\n            identity = self.downsample(x)\n\n        out += identity\n        out = self.relu(out)\n\n        return out\n\nprint(\"SEBottleneck class defined successfully.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-17T09:35:05.984995Z","iopub.execute_input":"2026-03-17T09:35:05.985251Z","iopub.status.idle":"2026-03-17T09:35:06.01636Z","shell.execute_reply.started":"2026-03-17T09:35:05.985214Z","shell.execute_reply":"2026-03-17T09:35:06.015516Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import torch.utils.model_zoo as model_zoo\nfrom torchvision.models.resnet import ResNet, Bottleneck\n\n__all__ = ['se_resnet50']\n\nmodel_urls = {\n    'resnet50': 'https://download.pytorch.org/models/resnet50-19c8e357.pth',\n}\n\ndef _resnet(arch, block, layers, pretrained, progress, **kwargs):\n    model = ResNet(block, layers, **kwargs)\n    if pretrained:\n        state_dict = model_zoo.load_url(model_urls[arch], progress=progress)\n        \n        # Remove keys that are specific to the original ResNet and not in SE-ResNet50\n        # Or keys that need special handling due to SEBlock insertion\n        # For now, we'll load and then update based on matching keys\n        \n        # A simpler approach for demonstration: load state_dict and then update only matching keys\n        # This will randomly initialize SEBlock weights and final FC layer if num_classes differs\n        \n        model_state_dict = model.state_dict()\n        for k, v in state_dict.items():\n            if k in model_state_dict:\n                if model_state_dict[k].shape == v.shape:\n                    model_state_dict[k].copy_(v)\n                else:\n                    print(f\"Skipping {k}: shape mismatch {model_state_dict[k].shape} vs {v.shape}\")\n            else:\n                print(f\"Skipping {k}: not found in model state dict\")\n        model.load_state_dict(model_state_dict)\n\n    return model\n\ndef se_resnet50(pretrained=False, progress=True, **kwargs):\n    \"\"\"Constructs a ResNet-50 model with Squeeze-and-Excitation blocks.\n    Args:\n        pretrained (bool): If True, returns a model pre-trained on ImageNet\n        progress (bool): If True, displays a progress bar of the download to stderr\n    \"\"\"\n    return _resnet('resnet50', SEBottleneck, [3, 4, 6, 3], pretrained, progress,\n                   **kwargs)\n\nprint(\"se_resnet50 function defined successfully.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-17T09:35:10.278008Z","iopub.execute_input":"2026-03-17T09:35:10.278279Z","iopub.status.idle":"2026-03-17T09:35:10.308158Z","shell.execute_reply.started":"2026-03-17T09:35:10.278227Z","shell.execute_reply":"2026-03-17T09:35:10.307277Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn_se  = cnn_learner(data,se_resnet50,pretrained = True,ps = 0.45,metrics = [auc_score,accuracy])\nprint(\"FastAI cnn_learner with SE-ResNet50 architecture instantiated successfully.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-17T09:35:19.490454Z","iopub.execute_input":"2026-03-17T09:35:19.490762Z","iopub.status.idle":"2026-03-17T09:35:20.547062Z","shell.execute_reply.started":"2026-03-17T09:35:19.490715Z","shell.execute_reply":"2026-03-17T09:35:20.546127Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model training","metadata":{"_uuid":"431fffedbcaf668ca622f08b035c39d14903f344"}},{"cell_type":"markdown","source":"The most important hyperparameter in training neural networks in general is the **learning rate**. Unfortunately as of now, there is no way of finding a good learning rate without trial-and-error. \n\nThe library has made it convenient to test different learning rates. We find a good learning rate using the method `lr_find`, then plotting the graph of learning rates against losses. As a rule of thumb, the learning rate is chosen from a part of the graph where it is **steepest** and **most consistent**.","metadata":{"_uuid":"d2c822c626b464fb5048525c9926ff26a29fca0f"}},{"cell_type":"markdown","source":"`fit_one_cycle`is a method implemented by the library and proposed in [this paper](https://arxiv.org/pdf/1803.09820.pdf) to produce more accurate results and faster convergence. [This post](https://sgugger.github.io/the-1cycle-policy.html) is a great explanation of why `fit_one_cycle`works over the standard `fit`.","metadata":{"_uuid":"9280bde7debfdd73e080b8e432f339c3bd9ab6fa"}},{"cell_type":"code","source":"learn_se.lr_find()\nlearn_se.recorder.plot()","metadata":{"trusted":true,"_uuid":"dd46bafbb68249b4f32ae1778b6d216c2e12c7a8","execution":{"iopub.status.busy":"2026-03-17T09:35:32.02011Z","iopub.execute_input":"2026-03-17T09:35:32.020386Z","iopub.status.idle":"2026-03-17T09:36:27.812498Z","shell.execute_reply.started":"2026-03-17T09:35:32.020345Z","shell.execute_reply":"2026-03-17T09:36:27.811799Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn_se.fit_one_cycle(8,1e-2)","metadata":{"trusted":true,"_uuid":"e93f38c672f3d18f11bfb95d2a37fea686787655","_kg_hide-output":false,"scrolled":true,"execution":{"iopub.status.busy":"2026-03-17T11:00:46.301612Z","iopub.execute_input":"2026-03-17T11:00:46.301907Z","iopub.status.idle":"2026-03-17T11:32:00.162688Z","shell.execute_reply.started":"2026-03-17T11:00:46.301868Z","shell.execute_reply":"2026-03-17T11:32:00.161824Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"\nfrom fastai.vision import DatasetType\nimport numpy as np\nfrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, classification_report\n\n# 1) Get predictions on validation set\nprobs, targets = learn_se.get_preds(ds_type=DatasetType.Valid)\ny_true = targets.numpy()\ny_pred = np.argmax(probs.numpy(), axis=1)\n\n# 2) Accuracy\naccuracy = accuracy_score(y_true, y_pred)\n\n# 3) Per-class precision/recall/f1 (average=None)\nprecision_per = precision_score(y_true, y_pred, average=None)\nrecall_per = recall_score(y_true, y_pred, average=None)\nf1_per = f1_score(y_true, y_pred, average=None)\n\n# 4) Handle possible NaNs (older np: use np.where + np.isnan)\nprecision_per = np.where(np.isnan(precision_per), 0.0, precision_per)\nrecall_per = np.where(np.isnan(recall_per), 0.0, recall_per)\nf1_per = np.where(np.isnan(f1_per), 0.0, f1_per)\n\nprecision_macro = precision_per.mean()\nrecall_macro = recall_per.mean()\nf1_macro = f1_per.mean()\n\n# 5) Confusion matrix and specificity per class\ncm = confusion_matrix(y_true, y_pred)\ntotal = cm.sum()\nspecificity_per = []\nfor i in range(len(cm)):\n    TP = cm[i, i]\n    FP = cm[:, i].sum() - TP\n    FN = cm[i, :].sum() - TP\n    TN = total - TP - FP - FN\n    spec = TN / (TN + FP) if (TN + FP) > 0 else 0.0\n    specificity_per.append(spec)\nspecificity_per = np.array(specificity_per)\nspecificity_macro = specificity_per.mean()\n\n# 6) Print results\nprint(f\"Accuracy: {accuracy:.4f}\")\nprint(f\"Precision (macro): {precision_macro:.4f}\")\nprint(f\"Recall / Sensitivity (macro): {recall_macro:.4f}\")\nprint(f\"F1 (macro): {f1_macro:.4f}\")\nprint(f\"Specificity (macro): {specificity_macro:.4f}\\n\")\n\nprint(\"Per-class metrics:\")\nfor i, (p, r, f, s) in enumerate(zip(precision_per, recall_per, f1_per, specificity_per)):\n    print(f\" Class {i}: Precision={p:.4f}, Recall={r:.4f}, F1={f:.4f}, Specificity={s:.4f}\")\n\nprint(\"\\nConfusion matrix:\\n\", cm)\nprint(\"\\nClassification report:\\n\", classification_report(y_true, y_pred))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-17T11:47:50.123715Z","iopub.execute_input":"2026-03-17T11:47:50.124003Z","iopub.status.idle":"2026-03-17T11:48:20.731467Z","shell.execute_reply.started":"2026-03-17T11:47:50.123962Z","shell.execute_reply":"2026-03-17T11:48:20.730649Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn_se.unfreeze()\nlearn_se.lr_find()\nlearn_se.recorder.plot()","metadata":{"trusted":true,"_uuid":"58d788cdcf477f8c0206a4a42931314c16d6fd0f","execution":{"iopub.status.busy":"2026-03-16T12:53:10.326263Z","iopub.execute_input":"2026-03-16T12:53:10.326596Z","iopub.status.idle":"2026-03-16T12:53:33.662194Z","shell.execute_reply.started":"2026-03-16T12:53:10.326545Z","shell.execute_reply":"2026-03-16T12:53:33.661524Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"A [simple rule of thumb suggested](https://github.com/hiromis/notes/blob/master/Lesson1.md) by Jeremy when selecting differential learning rates is to:\n1. Set the first part of the slice (corresponding to the earlier layers of the model) to a learning rate much smaller than where the loss starts increasing.\n2. Set the final slice to a learning rate 0.1x of that used when training the frozen model.","metadata":{"_uuid":"f7762d1bd059c3e753c573c95bedced002d3495b"}},{"cell_type":"code","source":"learn_se.fit_one_cycle(2,max_lr = slice(1e-5,1e-3))","metadata":{"trusted":true,"_uuid":"db5f375fb32dcfcabbec27e03f4e6034ff22df34","execution":{"execution_failed":"2026-03-16T13:33:10.62Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Transfer learning","metadata":{"_uuid":"0924635ed1bafd9f5a9705a11b5408b5c3ebad3c"}},{"cell_type":"markdown","source":"We will now train the model on the same dataset, except we are using images of higher resolution. Intuitively, the 'concepts' learnt by the neural network will continue to be applied in training with the new set of images.","metadata":{"_uuid":"f31a8824c512bc6d960d3208469918f98a7dbb83"}},{"cell_type":"markdown","source":"## 64x64 images","metadata":{"_uuid":"a10d000a128d89c0f76acd6de1b8c35789a52bd4"}},{"cell_type":"code","source":"newTfms = get_transforms(do_flip = True,flip_vert = True,max_zoom = 1.25)\nnewSz = 64\nnewData = ImageDataBunch.from_csv(path, folder = 'train', csv_labels = \"train_labels.csv\",\n                               test = 'test',suffix=\".tif\", size = newSz, ds_tfms = newTfms)\nnewData.path = pathlib.Path('.')\nnewData.normalize(imagenet_stats)\nlearn.data = newData","metadata":{"trusted":true,"_uuid":"e4a1083ac91ea44c51ed3e685bccab6ae1b35d0f","execution":{"iopub.status.busy":"2026-03-16T13:35:12.805189Z","iopub.execute_input":"2026-03-16T13:35:12.805366Z","iopub.status.idle":"2026-03-16T13:35:12.906074Z","shell.execute_reply.started":"2026-03-16T13:35:12.805331Z","shell.execute_reply":"2026-03-16T13:35:12.905011Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.freeze()\nlearn.lr_find()\nlearn.recorder.plot()","metadata":{"trusted":true,"_uuid":"983a45f4c18b30f1bdd41bccbe174bbaa0770a1c"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.fit_one_cycle(8,1e-2)","metadata":{"trusted":true,"_uuid":"f095501de823a620787bd39312b456bacde6c970"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.unfreeze()\nlearn.lr_find()\nlearn.recorder.plot()","metadata":{"trusted":true,"_uuid":"4d55a70b26fb3b82c37167b26b2d7cdc8a57e992"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.fit_one_cycle(2,max_lr = slice(1e-3/3,1e-3))","metadata":{"trusted":true,"_uuid":"9d8a1a4584b38a76a430c7f73da14f25dd474c1b"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.save('stage-2')","metadata":{"trusted":true,"_uuid":"b832cfd443486ea402e1c6a8aee0ccdb4555b07f"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"preds,y = learn.TTA()\nacc = accuracy(preds, y)\nprint('The validation accuracy is {} %.'.format(acc * 100))\npred_score = auc_score(preds,y).item()\nprint('The validation AUC is {}.'.format(pred_score))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 96x96 images","metadata":{"_uuid":"4fb761ab64be1d77e52c9a969bc8747986ed08bb"}},{"cell_type":"code","source":"newTfms = get_transforms(do_flip = True,flip_vert = True,max_zoom = 1.5)\nnewSz = 96\nnewData = ImageDataBunch.from_csv(path, folder = 'train', csv_labels = \"train_labels.csv\",\n                               test = 'test',suffix=\".tif\", size = newSz, ds_tfms = newTfms)\nnewData.path = pathlib.Path('.')\nnewData.normalize(imagenet_stats)\nlearn.data = newData","metadata":{"trusted":true,"_uuid":"e4a1083ac91ea44c51ed3e685bccab6ae1b35d0f"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.freeze()\nlearn.lr_find()\nlearn.recorder.plot()","metadata":{"trusted":true,"_uuid":"983a45f4c18b30f1bdd41bccbe174bbaa0770a1c"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.fit_one_cycle(4,1e-4)","metadata":{"trusted":true,"_uuid":"f095501de823a620787bd39312b456bacde6c970"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.unfreeze()\nlearn.lr_find()\nlearn.recorder.plot()","metadata":{"trusted":true,"_uuid":"4d55a70b26fb3b82c37167b26b2d7cdc8a57e992"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.fit_one_cycle(2,max_lr = slice(1e-5/3,1e-5))","metadata":{"trusted":true,"_uuid":"9d8a1a4584b38a76a430c7f73da14f25dd474c1b"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"learn.save('stage-3')","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"preds,y = learn.TTA()\nacc = accuracy(preds, y)\nprint('The validation accuracy is {} %.'.format(acc * 100))\npred_score = auc_score(preds,y).item()\nprint('The validation AUC is {}.'.format(pred_score))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Model interpretation","metadata":{"_uuid":"04ca87c9e4a02c27bd8f6709fead846737969404"}},{"cell_type":"markdown","source":"At this stage, we would like to check the effectiveness of the `learn` model against our validation set (which is automatically generated by the `ImageDataBunch` object). We will use the following methods to evaluate the effectiveness.\n1. Confusion matrix.\n2. Accuracy.\n3. ROC-AUC, as dictated in the competition evaluation.","metadata":{"_uuid":"ca3b4c9a079e9f4de451b69403304a9cad42fb73"}},{"cell_type":"code","source":"interp = ClassificationInterpretation.from_learner(learn)","metadata":{"trusted":true,"_uuid":"39fc6026a35e64f0a5cb322d289370d9f09e4ef8"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"interp.plot_top_losses(6)","metadata":{"trusted":true,"_uuid":"f5eb5f8d55cf443599c055401f27af339ddd40ce"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"interp.plot_confusion_matrix()","metadata":{"trusted":true,"_uuid":"f6e9fa6dc75e76174a47c7cedcc8d134ce8f8566"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"preds,y = learn.TTA()\nacc = accuracy(preds, y)\nprint('The validation accuracy is {} %.'.format(acc * 100))\npred_score = auc_score(preds,y).item()\nprint('The validation AUC is {}.'.format(pred_score))","metadata":{"trusted":true,"_uuid":"1372e6934cc77d14ad0e3e1037833add222cf4bb"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Generating submission","metadata":{"_uuid":"885a8278d83b8a0aa6f3573a30391bab7a4fe68a"}},{"cell_type":"markdown","source":"As the order of the images loaded into `data` is not necessarily the same order as that in the required submission, we will need to rearrange the predictions on the test set.\n\nThe desired order for submission is that of `sample_submission.csv`. while the order of the test set loaded into `data` can be accessed by calling `learn.data.test_ds.items`. We will first create a dictionary assigning each image in the test to its prediction by the model, then call the keys in the order in  `sample_submission.csv`.","metadata":{"_uuid":"cf9128cc20cb44b48b580d65c23f66dd2a6fe81f"}},{"cell_type":"code","source":"def generateSubmission(learner):\n    submissions = pd.read_csv('../input/sample_submission.csv')\n    id_list = list(submissions.id)\n    preds,y = learner.TTA(ds_type=DatasetType.Test)\n    pred_list = list(preds[:,1])\n    pred_dict = dict((key, value.item()) for (key, value) in zip(learner.data.test_ds.items,pred_list))\n    pred_ordered = [pred_dict[Path('../input/test/' + id + '.tif')] for id in id_list]\n    submissions = pd.DataFrame({'id':id_list,'label':pred_ordered})\n    submissions.to_csv(\"submission_transferLearning_{}.csv\".format(pred_score),index = False)","metadata":{"trusted":true,"_uuid":"58a5d027f1fbe1212305d6eea9f28f248bd3759e"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"generateSubmission(learn)","metadata":{"trusted":true,"_uuid":"5f51eebd339f0d331a7942187a15e23e0b24047c"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Future work\n\n1. [DONE] Generate sample submission to ensure functional code. (0.6007)\n2. [DONE] Implement skeleton model for baseline (0.9106).\n3. [DONE] Prepare AUC metric (0.9233).\n4. [DONE] Deploy reasoned data augmentation (0.9241).\n5. [DONE] Deploy test-time augmentation (0.9364).\n6. Test other architectures (which are found [here in the Pytorch docs](https://pytorch.org/docs/stable/torchvision/models.html))\n    1. [DONE] ResNet-34 (0.9362)\n    2. [DONE] DenseNet-169 (0.9370)\n7. [DONE] Implement weight decay. (0.9368)\n8. [DONE] Retrain model with higher resolution images. (0.9573)","metadata":{"_uuid":"ee301f6cb7d4c9d9ae5921ecf42c6e08c785871c"}},{"cell_type":"markdown","source":"# Acknowledgements\n\n* [qitvision](https://www.kaggle.com/qitvision/) for his [extremely well-explained kernel](https://www.kaggle.com/qitvision/a-complete-ml-pipeline-fast-ai) on the same competition and for answering my questions on data loading.\n* [Gunther](https://www.kaggle.com/guntherthepenguin) for providing the implementation of the AUC metric in [his kernel in the same competition.](https://www.kaggle.com/guntherthepenguin/fastai-v1-densenet169)\n* The [fast.ai team](https://www.fast.ai/about/) for creating the [library](https://docs.fast.ai/index.html) and [the v3 course](https://course.fast.ai/index.html) for teaching deep learning in a very accessible manner.","metadata":{"_uuid":"56eaf8be52be9665272c073b3e5b86986514fffc"}}]}