SIIM COVID-19 Fastai + EfficientNetV2 + TIMM Models
","metadata":{}},{"cell_type":"markdown","source":"Overview\n\n ✔️ Create & Visualize Fastai Datablocks \n ✔️ Stratified K-fold Split \n ✔️ Porting TIMM Models \n ✔️ Creating Fastai Learner \n ✔️ Weights & Biases Integration \n ✔️ Experiment Tracking and Logging with Weights & Biases \n\n#### **Version Notes**:\n**v7**: Multi-label with probabilities for all classes | `tf_efficientnetv2_m_in21ft1k` (pre-trained on ImageNet21K and fine-tuned with ImageNet1K)\n\n**v4**: Multi-class baseline | `efficientnetv2_rw_s`\n\n \n\nWe will cover some of the most useful and time-saving features of fastai. With little effort, fastai will help in acheiving state-of-the-art results with much speed and ease. That's the whole motivation behind the framework.\n\nLet's also go through how any model from the TIMM package can be ported into fastai. This opens up the possibility to try over 600+ different architectures including some of the very recent ones such as the EfficientNetV2, ViT and NFNet.\n\nFastai\n\nWith fastai, there is less boilerplate code and it is very straightforward to develop models. Fastai strives to give us the best settings and configs right from the start! It doesn't just develop a baseline but, delivers SOTA models with just starter level code!\n\nIt is also flexibile enough to build tune and tweak any aspect of the training or the model architecture and there are pre-built utils already part of the framework for this. However, there is a slight learning curve for some the advanced features as it is still code that we did not author. So let's also explore a few tricks to dissect the code for building custom components.\n\nFastai Naming Conventions:\n\nFastai naming conventions are a little different from the PEP Python coding standards and the reasoning behind this practice is because the name symbols following the Huffman Coding principle, which basically means\n\n Commonly used and generic concepts should be named shorter. You shouldn’t waste short sequences on less common concepts.\n\nFastai also follows the life-cycle naming principle:\n\n The shorter life a symbol, the shorter name it should have.\n \n \nI personally did not find any difficulty in reading and understanding the code. Here are some of the most commonly used shorthand names:\n\n \n \n\n| Concept | Abbreviation | Combination Examples |\n| --------------------------- | -------------------- | ---------------------------------------------- |\n| transforms | tfms | tfm\\_y, TfmType |\n| evaluate | eval | eval() |\n| configuration | cfg | |\n| threshold | thresh | |\n| size | sz | |\n| array | arr | label\\_arr |\n| dataset | ds | train\\_ds |\n| dataloader | dl | train\\_dl |\n| dataframe | df | train\\_df |\n| train | train | train\\_ds, train\\_dl, train\\_x, train\\_y |\n| validation | valid | valid\\_ds, valid\\_dl, valid\\_x, valid\\_y |\n| test | test | test\\_ds, test\\_dl |\n| number of classes | c | |\n| batch | b | |\n| batch size | bs | |\n| augment | aug | |\n| optimizer (e.g. Adam) | opt | |\n| schedule | sched | |\n\n \n \n\nFor further reference, the link below has the list of all shorthand conventions.\n\nhttps://docs.fast.ai/dev/abbr\n\n \n \n\n\nEfficientNet V2\n\nEfficientNet V2 uses a combination of training-aware neural architecture search and scaling, to jointly optimize training speed and parameter efficiency. The models were searched from the search space enriched with new ops such as Fused-MBConv. Experiments by the authors show that EfficientNetV2 models train much faster than state-of-the-art models while being up to 6.8x smaller.\n\n \n \n\n
\n\n \n \n\n\n\n \n \n \n\n🔗 Notebook on SIIM Covid-19 Dataset creation:\n\n[SIIM Covid-19 Resize Process Coco Dataset](https://www.kaggle.com/sreevishnudamodaran/siim-covid-19-resize-process-coco-dataset)\n\n \n\n
\n\n \n \n\n
","metadata":{}},{"cell_type":"markdown","source":"🏷️ Environment Setup\n","metadata":{}},{"cell_type":"code","source":"import os\n\ndevice = 'CPU'\nif 'CUDA_VERSION' in os.environ.keys():\n if os.environ['CUDA_VERSION'] is not None: device = 'GPU'\nprint(device)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:14:46.967109Z","iopub.execute_input":"2021-07-23T04:14:46.967546Z","iopub.status.idle":"2021-07-23T04:14:46.979977Z","shell.execute_reply.started":"2021-07-23T04:14:46.967410Z","shell.execute_reply":"2021-07-23T04:14:46.978931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install -q git+https://github.com/rwightman/pytorch-image-models.git\n# !pip install wandb --upgrade\n!pip install -q jupyterlab-git --upgrade\n!pip install -q nbdev","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:14:46.981733Z","iopub.execute_input":"2021-07-23T04:14:46.982392Z","iopub.status.idle":"2021-07-23T04:15:19.737708Z","shell.execute_reply.started":"2021-07-23T04:14:46.982159Z","shell.execute_reply":"2021-07-23T04:15:19.736453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip list | grep \"torch\\|cuda\\|fast\"","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:19.741392Z","iopub.execute_input":"2021-07-23T04:15:19.741821Z","iopub.status.idle":"2021-07-23T04:15:21.877429Z","shell.execute_reply.started":"2021-07-23T04:15:19.741778Z","shell.execute_reply":"2021-07-23T04:15:21.876505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Imports and Seeding\n","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport timm\nfrom timm import *\n\nfrom fastai.vision.all import *\nfrom fastai.vision.learner import _update_first_layer\nfrom fastai.callback.wandb import *\nfrom nbdev.showdoc import show_doc","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:21.879345Z","iopub.execute_input":"2021-07-23T04:15:21.879721Z","iopub.status.idle":"2021-07-23T04:15:26.947905Z","shell.execute_reply.started":"2021-07-23T04:15:21.879681Z","shell.execute_reply":"2021-07-23T04:15:26.946944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Data Preparation","metadata":{}},{"cell_type":"code","source":"df_study_lvl = pd.read_csv(\"../input/siim-covid19-detection/train_study_level.csv\")\ndf_study_lvl.rename({'id':'study_id',\n 'Negative for Pneumonia':'negative',\n 'Typical Appearance':'typical',\n 'Indeterminate Appearance':'indeterminate',\n 'Atypical Appearance':'atypical'}, axis=1, inplace=True)\n\ndf_image_lvl = pd.read_csv('/kaggle/input/siim-covid19-detection/train_image_level.csv')\ndf_image_lvl['study_id'] = df_image_lvl['StudyInstanceUID'].apply(lambda idx: idx+\"_study\")\n\ndf_annotations = df_image_lvl.merge(df_study_lvl, on='study_id', how='outer')\ndf_annotations.head(3)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:26.951571Z","iopub.execute_input":"2021-07-23T04:15:26.951876Z","iopub.status.idle":"2021-07-23T04:15:27.041883Z","shell.execute_reply.started":"2021-07-23T04:15:26.951845Z","shell.execute_reply":"2021-07-23T04:15:27.041097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PATH = '../input/siim-covid19-images-metadata-256-512-768/images_metadata_256_512_768/train_512x512'\n\ndf_annotations['image_path'] = df_annotations['id'].map(lambda x:os.path.join(PATH,\n str(x)+'.png'))","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.043118Z","iopub.execute_input":"2021-07-23T04:15:27.043443Z","iopub.status.idle":"2021-07-23T04:15:27.068602Z","shell.execute_reply.started":"2021-07-23T04:15:27.043408Z","shell.execute_reply":"2021-07-23T04:15:27.067700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_names = ['negative','typical','indeterminate','atypical']\n\ndef get_labels(row):\n labels_str = ''\n for key in label_names:\n if row[key]==1:\n labels_str = labels_str+' '+key if labels_str else key\n return labels_str\n\ndf_annotations['labels'] = df_annotations[label_names].apply(get_labels, axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.070018Z","iopub.execute_input":"2021-07-23T04:15:27.070377Z","iopub.status.idle":"2021-07-23T04:15:27.256445Z","shell.execute_reply.started":"2021-07-23T04:15:27.070339Z","shell.execute_reply":"2021-07-23T04:15:27.255616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_annotations.sample(4)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.257731Z","iopub.execute_input":"2021-07-23T04:15:27.258086Z","iopub.status.idle":"2021-07-23T04:15:27.284343Z","shell.execute_reply.started":"2021-07-23T04:15:27.258042Z","shell.execute_reply":"2021-07-23T04:15:27.283105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### TIMM\n\nTIMM s a deep-learning toolkit and model repository created by Ross Wightman. It has a collection of over 600+ SOTA computer vision models, layers, utilities, optimizers, schedulers, data-loaders, augmentations and even training and scoring scripts.\n\n#### Architectures Suppported by TIMM\n\n| | | | | | | | |\n| ------------------------------------ | ------------------------------- | ---------------------------- | ------------------------------------ | ---------------------------------- | ----------------------------------------- | ------------------------------------ | --------------------------------- |\n| adv\\_inception\\_v3 | ecaresnet26t | gluon\\_resnet152\\_v1s | mobilenetv2\\_140 | repvgg\\_b2g4 | rexnetr\\_130 | tf\\_efficientnet\\_b7\\_ns | vit\\_base\\_r50\\_s16\\_384 |\n| bat\\_resnext26ts | ecaresnet50d | gluon\\_resnext50\\_32x4d | mobilenetv3\\_large\\_075 | repvgg\\_b3 | rexnetr\\_150 | tf\\_efficientnet\\_b8 | vit\\_base\\_resnet26d\\_224 |\n| botnet26t\\_256 | ecaresnet50d\\_pruned | gluon\\_resnext101\\_32x4d | mobilenetv3\\_large\\_100 | repvgg\\_b3g4 | rexnetr\\_200 | tf\\_efficientnet\\_b8\\_ap | vit\\_base\\_resnet50\\_224\\_in21k |\n| botnet50ts\\_256 | ecaresnet50t | gluon\\_resnext101\\_64x4d | mobilenetv3\\_large\\_100\\_miil | res2net50\\_14w\\_8s | selecsls42 | tf\\_efficientnet\\_cc\\_b0\\_4e | vit\\_base\\_resnet50\\_384 |\n| cait\\_m36\\_384 | ecaresnet101d | gluon\\_senet154 | mobilenetv3\\_large\\_100\\_miil\\_in21k | res2net50\\_26w\\_4s | selecsls42b | tf\\_efficientnet\\_cc\\_b0\\_8e | vit\\_base\\_resnet50d\\_224 |\n| cait\\_m48\\_448 | ecaresnet101d\\_pruned | gluon\\_seresnext50\\_32x4d | mobilenetv3\\_rw | res2net50\\_26w\\_6s | selecsls60 | tf\\_efficientnet\\_cc\\_b1\\_8e | vit\\_huge\\_patch14\\_224\\_in21k |\n| cait\\_s24\\_224 | ecaresnet200d | gluon\\_seresnext101\\_32x4d | mobilenetv3\\_small\\_075 | res2net50\\_26w\\_8s | selecsls60b | tf\\_efficientnet\\_el | vit\\_large\\_patch16\\_224 |\n| cait\\_s24\\_384 | ecaresnet269d | gluon\\_seresnext101\\_64x4d | mobilenetv3\\_small\\_100 | res2net50\\_48w\\_2s | selecsls84 | tf\\_efficientnet\\_em | vit\\_large\\_patch16\\_224\\_in21k |\n| cait\\_s36\\_384 | ecaresnetlight | gluon\\_xception65 | nasnetalarge | res2net101\\_26w\\_4s | semnasnet\\_050 | tf\\_efficientnet\\_es | vit\\_large\\_patch16\\_384 |\n| cait\\_xs24\\_384 | ecaresnext26t\\_32x4d | gmixer\\_12\\_224 | nf\\_ecaresnet26 | res2next50 | semnasnet\\_075 | tf\\_efficientnet\\_l2\\_ns | vit\\_large\\_patch32\\_224 |\n| cait\\_xxs24\\_224 | ecaresnext50t\\_32x4d | gmixer\\_24\\_224 | nf\\_ecaresnet50 | resmlp\\_12\\_224 | semnasnet\\_100 | tf\\_efficientnet\\_l2\\_ns\\_475 | vit\\_large\\_patch32\\_224\\_in21k |\n| cait\\_xxs24\\_384 | efficientnet\\_b0 | gmlp\\_b16\\_224 | nf\\_ecaresnet101 | resmlp\\_12\\_distilled\\_224 | semnasnet\\_140 | tf\\_efficientnet\\_lite0 | vit\\_large\\_patch32\\_384 |\n| cait\\_xxs36\\_224 | efficientnet\\_b1 | gmlp\\_s16\\_224 | nf\\_regnet\\_b0 | resmlp\\_24\\_224 | senet154 | tf\\_efficientnet\\_lite1 | vit\\_large\\_r50\\_s32\\_224 |\n| cait\\_xxs36\\_384 | efficientnet\\_b1\\_pruned | gmlp\\_ti16\\_224 | nf\\_regnet\\_b1 | resmlp\\_24\\_distilled\\_224 | seresnet18 | tf\\_efficientnet\\_lite2 | vit\\_large\\_r50\\_s32\\_224\\_in21k |\n| coat\\_lite\\_mini | efficientnet\\_b2 | halonet26t | nf\\_regnet\\_b2 | resmlp\\_36\\_224 | seresnet34 | tf\\_efficientnet\\_lite3 | vit\\_large\\_r50\\_s32\\_384 |\n| coat\\_lite\\_small | efficientnet\\_b2\\_pruned | halonet50ts | nf\\_regnet\\_b3 | resmlp\\_36\\_distilled\\_224 | seresnet50 | tf\\_efficientnet\\_lite4 | vit\\_small\\_patch16\\_224 |\n| coat\\_lite\\_tiny | efficientnet\\_b2a | halonet\\_h1 | nf\\_regnet\\_b4 | resmlp\\_big\\_24\\_224 | seresnet50t | tf\\_efficientnetv2\\_b0 | vit\\_small\\_patch16\\_224\\_in21k |\n| coat\\_mini | efficientnet\\_b3 | halonet\\_h1\\_c4c5 | nf\\_regnet\\_b5 | resmlp\\_big\\_24\\_224\\_in22ft1k | seresnet101 | tf\\_efficientnetv2\\_b1 | vit\\_small\\_patch16\\_384 |\n| coat\\_tiny | efficientnet\\_b3\\_pruned | hardcorenas\\_a | nf\\_resnet26 | resmlp\\_big\\_24\\_distilled\\_224 | seresnet152 | tf\\_efficientnetv2\\_b2 | vit\\_small\\_patch32\\_224 |\n| convit\\_base | efficientnet\\_b3a | hardcorenas\\_b | nf\\_resnet50 | resnest14d | seresnet152d | tf\\_efficientnetv2\\_b3 | vit\\_small\\_patch32\\_224\\_in21k |\n| convit\\_small | efficientnet\\_b4 | hardcorenas\\_c | nf\\_resnet101 | resnest26d | seresnet200d | tf\\_efficientnetv2\\_l | vit\\_small\\_patch32\\_384 |\n| convit\\_tiny | efficientnet\\_b5 | hardcorenas\\_d | nf\\_seresnet26 | resnest50d | seresnet269d | tf\\_efficientnetv2\\_l\\_in21ft1k | vit\\_small\\_r26\\_s32\\_224 |\n| cspdarknet53 | efficientnet\\_b6 | hardcorenas\\_e | nf\\_seresnet50 | resnest50d\\_1s4x24d | seresnext26d\\_32x4d | tf\\_efficientnetv2\\_l\\_in21k | vit\\_small\\_r26\\_s32\\_224\\_in21k |\n| cspdarknet53\\_iabn | efficientnet\\_b7 | hardcorenas\\_f | nf\\_seresnet101 | resnest50d\\_4s2x40d | seresnext26t\\_32x4d | tf\\_efficientnetv2\\_m | vit\\_small\\_r26\\_s32\\_384 |\n| cspresnet50 | efficientnet\\_b8 | hrnet\\_w18 | nfnet\\_f0 | resnest101e | seresnext26tn\\_32x4d | tf\\_efficientnetv2\\_m\\_in21ft1k | vit\\_small\\_resnet26d\\_224 |\n| cspresnet50d | efficientnet\\_cc\\_b0\\_4e | hrnet\\_w18\\_small | nfnet\\_f0s | resnest200e | seresnext50\\_32x4d | tf\\_efficientnetv2\\_m\\_in21k | vit\\_small\\_resnet50d\\_s16\\_224 |\n| cspresnet50w | efficientnet\\_cc\\_b0\\_8e | hrnet\\_w18\\_small\\_v2 | nfnet\\_f1 | resnest269e | seresnext101\\_32x4d | tf\\_efficientnetv2\\_s | vit\\_tiny\\_patch16\\_224 |\n| cspresnext50 | efficientnet\\_cc\\_b1\\_8e | hrnet\\_w30 | nfnet\\_f1s | resnet18 | seresnext101\\_32x8d | tf\\_efficientnetv2\\_s\\_in21ft1k | vit\\_tiny\\_patch16\\_224\\_in21k |\n| cspresnext50\\_iabn | efficientnet\\_el | hrnet\\_w32 | nfnet\\_f2 | resnet18d | skresnet18 | tf\\_efficientnetv2\\_s\\_in21k | vit\\_tiny\\_patch16\\_384 |\n| darknet53 | efficientnet\\_el\\_pruned | hrnet\\_w40 | nfnet\\_f2s | resnet26 | skresnet34 | tf\\_inception\\_v3 | vit\\_tiny\\_r\\_s16\\_p8\\_224 |\n| deit\\_base\\_distilled\\_patch16\\_224 | efficientnet\\_em | hrnet\\_w44 | nfnet\\_f3 | resnet26d | skresnet50 | tf\\_mixnet\\_l | vit\\_tiny\\_r\\_s16\\_p8\\_224\\_in21k |\n| deit\\_base\\_distilled\\_patch16\\_384 | efficientnet\\_es | hrnet\\_w48 | nfnet\\_f3s | resnet26t | skresnet50d | tf\\_mixnet\\_m | vit\\_tiny\\_r\\_s16\\_p8\\_384 |\n| deit\\_base\\_patch16\\_224 | efficientnet\\_es\\_pruned | hrnet\\_w64 | nfnet\\_f4 | resnet34 | skresnext50\\_32x4d | tf\\_mixnet\\_s | vovnet39a |\n| deit\\_base\\_patch16\\_384 | efficientnet\\_l2 | ig\\_resnext101\\_32x8d | nfnet\\_f4s | resnet34d | spnasnet\\_100 | tf\\_mobilenetv3\\_large\\_075 | vovnet57a |\n| deit\\_small\\_distilled\\_patch16\\_224 | efficientnet\\_lite0 | ig\\_resnext101\\_32x16d | nfnet\\_f5 | resnet50 | ssl\\_resnet18 | tf\\_mobilenetv3\\_large\\_100 | wide\\_resnet50\\_2 |\n| deit\\_small\\_patch16\\_224 | efficientnet\\_lite1 | ig\\_resnext101\\_32x32d | nfnet\\_f5s | resnet50d | ssl\\_resnet50 | tf\\_mobilenetv3\\_large\\_minimal\\_100 | wide\\_resnet101\\_2 |\n| deit\\_tiny\\_distilled\\_patch16\\_224 | efficientnet\\_lite2 | ig\\_resnext101\\_32x48d | nfnet\\_f6 | resnet50t | ssl\\_resnext50\\_32x4d | tf\\_mobilenetv3\\_small\\_075 | xception |\n| deit\\_tiny\\_patch16\\_224 | efficientnet\\_lite3 | inception\\_resnet\\_v2 | nfnet\\_f6s | resnet51q | ssl\\_resnext101\\_32x4d | tf\\_mobilenetv3\\_small\\_100 | xception41 |\n| densenet121 | efficientnet\\_lite4 | inception\\_v3 | nfnet\\_f7 | resnet61q | ssl\\_resnext101\\_32x8d | tf\\_mobilenetv3\\_small\\_minimal\\_100 | xception65 |\n| densenet121d | efficientnetv2\\_l | inception\\_v4 | nfnet\\_f7s | resnet101 | ssl\\_resnext101\\_32x16d | tnt\\_b\\_patch16\\_224 | xception71 |\n| densenet161 | efficientnetv2\\_m | lambda\\_resnet26t | nfnet\\_l0 | resnet101d | swin\\_base\\_patch4\\_window7\\_224 | tnt\\_s\\_patch16\\_224 | |\n| densenet169 | efficientnetv2\\_rw\\_m | lambda\\_resnet50t | pit\\_b\\_224 | resnet152 | swin\\_base\\_patch4\\_window7\\_224\\_in22k | tresnet\\_l | |\n| densenet201 | efficientnetv2\\_rw\\_s | legacy\\_senet154 | pit\\_b\\_distilled\\_224 | resnet152d | swin\\_base\\_patch4\\_window12\\_384 | tresnet\\_l\\_448 | |\n| densenet264 | efficientnetv2\\_s | legacy\\_seresnet18 | pit\\_s\\_224 | resnet200 | swin\\_base\\_patch4\\_window12\\_384\\_in22k | tresnet\\_m | |\n| densenet264d\\_iabn | ens\\_adv\\_inception\\_resnet\\_v2 | legacy\\_seresnet34 | pit\\_s\\_distilled\\_224 | resnet200d | swin\\_large\\_patch4\\_window7\\_224 | tresnet\\_m\\_448 | |\n| densenetblur121d | ese\\_vovnet19b\\_dw | legacy\\_seresnet50 | pit\\_ti\\_224 | resnetblur18 | swin\\_large\\_patch4\\_window7\\_224\\_in22k | tresnet\\_m\\_miil\\_in21k | |\n| dla34 | ese\\_vovnet19b\\_slim | legacy\\_seresnet101 | pit\\_ti\\_distilled\\_224 | resnetblur50 | swin\\_large\\_patch4\\_window12\\_384 | tresnet\\_xl | |\n| dla46\\_c | ese\\_vovnet19b\\_slim\\_dw | legacy\\_seresnet152 | pit\\_xs\\_224 | resnetrs50 | swin\\_large\\_patch4\\_window12\\_384\\_in22k | tresnet\\_xl\\_448 | |\n| dla46x\\_c | ese\\_vovnet39b | legacy\\_seresnext26\\_32x4d | pit\\_xs\\_distilled\\_224 | resnetrs101 | swin\\_small\\_patch4\\_window7\\_224 | tv\\_densenet121 | |\n| dla60 | ese\\_vovnet39b\\_evos | legacy\\_seresnext50\\_32x4d | pnasnet5large | resnetrs152 | swin\\_tiny\\_patch4\\_window7\\_224 | tv\\_resnet34 | |\n| dla60\\_res2net | ese\\_vovnet57b | legacy\\_seresnext101\\_32x4d | rednet26t | resnetrs200 | swinnet26t\\_256 | tv\\_resnet50 | |\n| dla60\\_res2next | ese\\_vovnet99b | levit\\_128 | rednet50ts | resnetrs270 | swinnet50ts\\_256 | tv\\_resnet101 | |\n| dla60x | ese\\_vovnet99b\\_iabn | levit\\_128s | regnetx\\_002 | resnetrs350 | swsl\\_resnet18 | tv\\_resnet152 | |\n| dla60x\\_c | fbnetc\\_100 | levit\\_192 | regnetx\\_004 | resnetrs420 | swsl\\_resnet50 | tv\\_resnext50\\_32x4d | |\n| dla102 | fbnetv3\\_b | levit\\_256 | regnetx\\_006 | resnetv2\\_50 | swsl\\_resnext50\\_32x4d | twins\\_pcpvt\\_base | |\n| dla102x | fbnetv3\\_d | levit\\_384 | regnetx\\_008 | resnetv2\\_50d | swsl\\_resnext101\\_32x4d | twins\\_pcpvt\\_large | |\n| dla102x2 | fbnetv3\\_g | mixer\\_b16\\_224 | regnetx\\_016 | resnetv2\\_50x1\\_bit\\_distilled | swsl\\_resnext101\\_32x8d | twins\\_pcpvt\\_small | |\n| dla169 | gc\\_efficientnet\\_b0 | mixer\\_b16\\_224\\_in21k | regnetx\\_032 | resnetv2\\_50x1\\_bitm | swsl\\_resnext101\\_32x16d | twins\\_svt\\_base | |\n| dm\\_nfnet\\_f0 | gcresnet50t | mixer\\_b16\\_224\\_miil | regnetx\\_040 | resnetv2\\_50x1\\_bitm\\_in21k | tf\\_efficientnet\\_b0 | twins\\_svt\\_large | |\n| dm\\_nfnet\\_f1 | gcresnext26ts | mixer\\_b16\\_224\\_miil\\_in21k | regnetx\\_064 | resnetv2\\_50x3\\_bitm | tf\\_efficientnet\\_b0\\_ap | twins\\_svt\\_small | |\n| dm\\_nfnet\\_f2 | geresnet50t | mixer\\_b32\\_224 | regnetx\\_080 | resnetv2\\_50x3\\_bitm\\_in21k | tf\\_efficientnet\\_b0\\_ns | vgg11 | |\n| dm\\_nfnet\\_f3 | gernet\\_l | mixer\\_l16\\_224 | regnetx\\_120 | resnetv2\\_101x1\\_bitm | tf\\_efficientnet\\_b1 | vgg11\\_bn | |\n| dm\\_nfnet\\_f4 | gernet\\_m | mixer\\_l16\\_224\\_in21k | regnetx\\_160 | resnetv2\\_101x1\\_bitm\\_in21k | tf\\_efficientnet\\_b1\\_ap | vgg13 | |\n| dm\\_nfnet\\_f5 | gernet\\_s | mixer\\_l32\\_224 | regnetx\\_320 | resnetv2\\_101x3\\_bitm | tf\\_efficientnet\\_b1\\_ns | vgg13\\_bn | |\n| dm\\_nfnet\\_f6 | ghostnet\\_050 | mixer\\_s16\\_224 | regnety\\_002 | resnetv2\\_101x3\\_bitm\\_in21k | tf\\_efficientnet\\_b2 | vgg16 | |\n| dpn68 | ghostnet\\_100 | mixer\\_s32\\_224 | regnety\\_004 | resnetv2\\_152x2\\_bit\\_teacher | tf\\_efficientnet\\_b2\\_ap | vgg16\\_bn | |\n| dpn68b | ghostnet\\_130 | mixnet\\_l | regnety\\_006 | resnetv2\\_152x2\\_bit\\_teacher\\_384 | tf\\_efficientnet\\_b2\\_ns | vgg19 | |\n| dpn92 | gluon\\_inception\\_v3 | mixnet\\_m | regnety\\_008 | resnetv2\\_152x2\\_bitm | tf\\_efficientnet\\_b3 | vgg19\\_bn | |\n| dpn98 | gluon\\_resnet18\\_v1b | mixnet\\_s | regnety\\_016 | resnetv2\\_152x2\\_bitm\\_in21k | tf\\_efficientnet\\_b3\\_ap | visformer\\_small | |\n| dpn107 | gluon\\_resnet34\\_v1b | mixnet\\_xl | regnety\\_032 | resnetv2\\_152x4\\_bitm | tf\\_efficientnet\\_b3\\_ns | visformer\\_tiny | |\n| dpn131 | gluon\\_resnet50\\_v1b | mixnet\\_xxl | regnety\\_040 | resnetv2\\_152x4\\_bitm\\_in21k | tf\\_efficientnet\\_b4 | vit\\_base\\_patch16\\_224 | |\n| eca\\_botnext26ts\\_256 | gluon\\_resnet50\\_v1c | mnasnet\\_050 | regnety\\_064 | resnext50\\_32x4d | tf\\_efficientnet\\_b4\\_ap | vit\\_base\\_patch16\\_224\\_in21k | |\n| eca\\_efficientnet\\_b0 | gluon\\_resnet50\\_v1d | mnasnet\\_075 | regnety\\_080 | resnext50d\\_32x4d | tf\\_efficientnet\\_b4\\_ns | vit\\_base\\_patch16\\_224\\_miil | |\n| eca\\_halonext26ts | gluon\\_resnet50\\_v1s | mnasnet\\_100 | regnety\\_120 | resnext101\\_32x4d | tf\\_efficientnet\\_b5 | vit\\_base\\_patch16\\_224\\_miil\\_in21k | |\n| eca\\_lambda\\_resnext26ts | gluon\\_resnet101\\_v1b | mnasnet\\_140 | regnety\\_160 | resnext101\\_32x8d | tf\\_efficientnet\\_b5\\_ap | vit\\_base\\_patch16\\_384 | |\n| eca\\_nfnet\\_l0 | gluon\\_resnet101\\_v1c | mnasnet\\_a1 | regnety\\_320 | resnext101\\_64x4d | tf\\_efficientnet\\_b5\\_ns | vit\\_base\\_patch32\\_224 | |\n| eca\\_nfnet\\_l1 | gluon\\_resnet101\\_v1d | mnasnet\\_b1 | repvgg\\_a2 | rexnet\\_100 | tf\\_efficientnet\\_b6 | vit\\_base\\_patch32\\_224\\_in21k | |\n| eca\\_nfnet\\_l2 | gluon\\_resnet101\\_v1s | mnasnet\\_small | repvgg\\_b0 | rexnet\\_130 | tf\\_efficientnet\\_b6\\_ap | vit\\_base\\_patch32\\_384 | |\n| eca\\_nfnet\\_l3 | gluon\\_resnet152\\_v1b | mobilenetv2\\_100 | repvgg\\_b1 | rexnet\\_150 | tf\\_efficientnet\\_b6\\_ns | vit\\_base\\_r26\\_s32\\_224 | |\n| eca\\_swinnext26ts\\_256 | gluon\\_resnet152\\_v1c | mobilenetv2\\_110d | repvgg\\_b1g4 | rexnet\\_200 | tf\\_efficientnet\\_b7 | vit\\_base\\_r50\\_s16\\_224 | |\n| eca\\_vovnet39b | gluon\\_resnet152\\_v1d | mobilenetv2\\_120d | repvgg\\_b2 | rexnetr\\_100 | tf\\_efficientnet\\_b7\\_ap | vit\\_base\\_r50\\_s16\\_224\\_in21k | |","metadata":{}},{"cell_type":"code","source":"# from pprint import pprint\n# model_names = timm.list_models()\n# pprint(model_names)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-07-23T04:15:27.288244Z","iopub.execute_input":"2021-07-23T04:15:27.288567Z","iopub.status.idle":"2021-07-23T04:15:27.292800Z","shell.execute_reply.started":"2021-07-23T04:15:27.288537Z","shell.execute_reply":"2021-07-23T04:15:27.291406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"timm.list_models('*efficientnet*')","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.295226Z","iopub.execute_input":"2021-07-23T04:15:27.295796Z","iopub.status.idle":"2021-07-23T04:15:27.307192Z","shell.execute_reply.started":"2021-07-23T04:15:27.295755Z","shell.execute_reply":"2021-07-23T04:15:27.306048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📌 Architecture with the tf_ prefix are the original weights ported from Google, so it uses manual padding to match TensorFlow's \"same\" padding, which adds GPU overhead and a general slowdown.","metadata":{}},{"cell_type":"markdown","source":"⚙️ Global Configuration & Seed\n","metadata":{}},{"cell_type":"code","source":"def seed_everything(seed):\n random.seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n np.random.seed(seed)\n torch.manual_seed(seed)\n \n if device=='GPU':\n torch.cuda.manual_seed(seed)\n torch.backends.cudnn.deterministic = True","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.309090Z","iopub.execute_input":"2021-07-23T04:15:27.309516Z","iopub.status.idle":"2021-07-23T04:15:27.317492Z","shell.execute_reply.started":"2021-07-23T04:15:27.309473Z","shell.execute_reply":"2021-07-23T04:15:27.316671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class Config:\n seed_val = 111\n seed_everything(seed_val)\n fold_num = 0\n job = 1\n num_classes = 4\n input_dims = 512\n model_arch = 'tf_efficientnetv2_m_in21ft1k'\n batch_size = 18\n num_workers = 0\n kfold = 5\n loss_func = nn.BCEWithLogitsLoss() # CrossEntropyLossFlat() or LabelSmoothingCrossEntropyFlat() for multi-class\n metrics = [accuracy_multi, RocAucMulti(average='macro'), F1ScoreMulti(average='macro')]\n job_name = f'{model_arch}_fold{fold_num}_job{job}'\n print(\"Job Name:\", job_name)\n\n wandb_project = 'SIIM_classifier_public'\n wandb_run_name = job_name\n \n if device=='GPU':\n fp16 = True\n else:\n fp16 = False\n \ncfg = Config()","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.318923Z","iopub.execute_input":"2021-07-23T04:15:27.319320Z","iopub.status.idle":"2021-07-23T04:15:27.328650Z","shell.execute_reply.started":"2021-07-23T04:15:27.319280Z","shell.execute_reply":"2021-07-23T04:15:27.327786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting global config class object to a dictionary to log using Wandb\nconfig_dict = dict(vars(Config))\nconfig_dict = {k:(v if type(v)==int else str(v)) for (k,v) in config_dict.items() if '__' not in k}\nconfig_dict","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.329930Z","iopub.execute_input":"2021-07-23T04:15:27.330325Z","iopub.status.idle":"2021-07-23T04:15:27.339384Z","shell.execute_reply.started":"2021-07-23T04:15:27.330287Z","shell.execute_reply":"2021-07-23T04:15:27.338205Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"KFold Split","metadata":{}},{"cell_type":"markdown","source":"Apparantly, only one class is assigned to each image and therefore this can also be modelled as a multi-class classification problem. Modelling as a multi-label problem in this version. Version v4 is multi-class.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GroupKFold, train_test_split\n\ndf_annotations['fold'] = -1\ngrp_kfold = GroupKFold(n_splits = cfg.kfold)\nfor fold, (train_index, val_index) in enumerate(grp_kfold.split(df_annotations,\n groups=df_annotations.study_id.tolist())):\n df_annotations.loc[val_index, 'fold'] = fold\ndf_annotations.sample(3)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.341467Z","iopub.execute_input":"2021-07-23T04:15:27.342148Z","iopub.status.idle":"2021-07-23T04:15:27.402427Z","shell.execute_reply.started":"2021-07-23T04:15:27.342094Z","shell.execute_reply":"2021-07-23T04:15:27.400754Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Augmentation and Dataloader Preparation","metadata":{}},{"cell_type":"code","source":"import albumentations as A\n\n# Source: https://forums.fast.ai/t/albumentation-transformations-for-train-and-test-dataset/82642\nclass AlbumentationsTransform(RandTransform):\n split_idx,order=None,2\n def __init__(self, train_aug, valid_aug): store_attr()\n \n def before_call(self, b, split_idx):\n self.idx = split_idx\n \n def encodes(self, img: PILImage):\n if self.idx == 0:\n aug_img = self.train_aug(image=np.array(img))['image']\n else:\n aug_img = self.valid_aug(image=np.array(img))['image']\n return PILImage.create(aug_img)\n \ndef get_train_aug(): \n return A.Compose([\n# A.RandomResizedCrop(cfg.input_dims,cfg.input_dims), \n A.Resize(cfg.input_dims, cfg.input_dims, p=1.0),\n A.HorizontalFlip(p=0.5),\n A.ShiftScaleRotate(shift_limit=0.02, scale_limit=0.1, rotate_limit=10, p=0.5),\n A.IAAPerspective(scale=(0.02, 0.04), p=0.5),\n A.RandomBrightnessContrast(0.1, 0.1, p=0.5),\n A.OneOf([A.CLAHE(),\n A.HueSaturationValue(0.2, 0.2, 0.2, p=0.5)\n ],p=0.4),\n A.OneOf([A.CoarseDropout(),\n A.Cutout()], p=0.5)\n ])\n\ndef get_valid_aug():\n return A.Compose([A.Resize(cfg.input_dims, cfg.input_dims, p=1.0)], p=1.0)\n\nitem_tfms = AlbumentationsTransform(get_train_aug(), get_valid_aug())\nbatch_tfms = [Normalize.from_stats(*imagenet_stats)]","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:27.404634Z","iopub.execute_input":"2021-07-23T04:15:27.405242Z","iopub.status.idle":"2021-07-23T04:15:33.090352Z","shell.execute_reply.started":"2021-07-23T04:15:27.405198Z","shell.execute_reply":"2021-07-23T04:15:33.089317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📌 Fastai provideds two ways to perform augmentations:\n\n - Item transforms (item_tfms) which are computed on the CPU\n\n - Batch transforms (batch_tfms) are done on the GPU on whole batches.\n\nIf there is sufficient memory left in the GPU after loading the model into memory, batch transforms can be used to perform augmentations as it would result in a major boost in training speed.","metadata":{}},{"cell_type":"code","source":"val_indices = list(df_annotations[df_annotations['fold'] == cfg.fold_num].image_path.unique())\n\ndata_block = DataBlock(blocks=(ImageBlock, MultiCategoryBlock(vocab=['negative', 'typical', 'indeterminate', 'atypical'],\n encoded=True)),\n splitter=MaskSplitter(list(df_annotations['fold'] == fold)),\n get_x=ColReader('image_path'),\n get_y=ColReader(['negative', 'typical', 'indeterminate', 'atypical']),\n item_tfms=item_tfms,\n batch_tfms=batch_tfms)\n\ndls = data_block.dataloaders(df_annotations,\n bs=cfg.batch_size,\n num_workers=cfg.num_workers)\n\ndls.show_batch(figsize=(18,15), max_n=8, nrows=2)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:33.091867Z","iopub.execute_input":"2021-07-23T04:15:33.092254Z","iopub.status.idle":"2021-07-23T04:15:34.912764Z","shell.execute_reply.started":"2021-07-23T04:15:33.092214Z","shell.execute_reply":"2021-07-23T04:15:34.911820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create Fastai Learner\n\nEssentially to use TIMM models in fastai, we have to take the model architecuture and slice off the last Linear layer (resulting in a \"body\" that outputs unpooled features). We devise a function to acheive this by taking the `create_body` fastai function called in [create_cnn_model](https://github.com/fastai/fastai/blob/eda1a2e50980b1ec2df127ae431b8bdbf1a84877/fastai/vision/learner.py#L139) as an example.\n\nNow we need to create a head and for this we need to calculate the number of input features our head needs to have with the `num_features_model` method. We then pass concat_pool=True to have fastai create a head with two pooling layers: AdaptiveConcatPool2d and nn.AdaptiveAvgPool2d.\n\nWe then wrap the two in a `nn.Sequential` and we now have a PyTorch model ready to be trained.\n\nWe pass this to a `Learner`, specifying our splitter to be the `default_splitter` which expects the body in `model[0]` and the head in `model[1]` to split our layer groups and we enable mixed precision training by calling the `to_non_native_fp16()` method. \n\nPlease note that if you're facing NaN issues while using mixed precision, you may need to use try the `native to_fp16()` method or even try tuning the growth_factor parameter of the `to_non_native_fp16()` method to resolve it. The mixed precision feature does have some compatibility issues and it may throw errors if used with some fastai metrics.\n\nFor further reference: https://walkwithfastai.com/vision.external.timm\n\n \n\n📌 Let's look at the original fastai functions before we build ours. An easy way to see the usage and details of a fastai class or function is to use the `doc()` function","metadata":{}},{"cell_type":"code","source":"doc(create_body)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:34.914041Z","iopub.execute_input":"2021-07-23T04:15:34.914411Z","iopub.status.idle":"2021-07-23T04:15:34.935317Z","shell.execute_reply.started":"2021-07-23T04:15:34.914373Z","shell.execute_reply":"2021-07-23T04:15:34.934329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"doc(create_model)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:34.936817Z","iopub.execute_input":"2021-07-23T04:15:34.937338Z","iopub.status.idle":"2021-07-23T04:15:34.953813Z","shell.execute_reply.started":"2021-07-23T04:15:34.937291Z","shell.execute_reply":"2021-07-23T04:15:34.952702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"doc(create_body)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:34.955734Z","iopub.execute_input":"2021-07-23T04:15:34.956046Z","iopub.status.idle":"2021-07-23T04:15:34.971740Z","shell.execute_reply.started":"2021-07-23T04:15:34.956016Z","shell.execute_reply":"2021-07-23T04:15:34.970726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Adapted from https://walkwithfastai.com/vision.external.timm\n\nfrom fastai.vision.learner import _add_norm\n\ndef create_timm_body(arch:str, pretrained=True, cut=None, n_in=3):\n \"Creates a body from any model in the `timm` library.\"\n model = create_model(arch, pretrained=pretrained, num_classes=0, global_pool='')\n _update_first_layer(model, n_in, pretrained)\n if cut is None:\n ll = list(enumerate(model.children()))\n cut = next(i for i,o in reversed(ll) if has_pool_type(o))\n if isinstance(cut, int): return nn.Sequential(*list(model.children())[:cut])\n elif callable(cut): return cut(model)\n else: raise NamedError(\"cut must be either integer or function\")\n\ndef create_timm_model(arch:str, n_out, cut=None, pretrained=True, n_in=3,\n init=nn.init.kaiming_normal_, custom_head=None,\n concat_pool=True, **kwargs):\n \"Create custom architecture using `arch`, `n_in` and `n_out` from the `timm` library\"\n body = create_timm_body(arch, pretrained, None, n_in)\n if custom_head is None:\n nf = num_features_model(nn.Sequential(*body.children()))\n head = create_head(nf, n_out, concat_pool=concat_pool, **kwargs)\n else: head = custom_head\n model = nn.Sequential(body, head)\n if init is not None: apply_init(model[1], init)\n return model\n\ndef timm_learner(dls, arch:str, loss_func=None, pretrained=True, cut=None, splitter=None,\n y_range=None, config=None, n_out=None, normalize=True, fp16=False, **kwargs):\n \"Build a convnet style learner from `dls` and `arch` using the `timm` library\"\n if config is None: config = {}\n if n_out is None: n_out = get_c(dls)\n assert n_out, \"`n_out` is not defined, and could not be inferred from data, set `dls.c` or pass `n_out`\"\n if y_range is None and 'y_range' in config: y_range = config.pop('y_range')\n model = create_timm_model(arch, n_out, default_split, pretrained, y_range=y_range, **config)\n learn = Learner(dls, model, loss_func=loss_func, splitter=default_split, **kwargs)\n if pretrained: learn.freeze()\n \n # Enable Mixed Precision Training\n if fp16: learn.to_non_native_fp16()\n# if fp16: learn.to_fp16(growth_factor=1.0)\n return learn","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:34.974963Z","iopub.execute_input":"2021-07-23T04:15:34.975273Z","iopub.status.idle":"2021-07-23T04:15:34.990363Z","shell.execute_reply.started":"2021-07-23T04:15:34.975242Z","shell.execute_reply":"2021-07-23T04:15:34.989375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Configure Wandb Callback","metadata":{}},{"cell_type":"code","source":"doc(WandbCallback)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:34.991915Z","iopub.execute_input":"2021-07-23T04:15:34.992721Z","iopub.status.idle":"2021-07-23T04:15:35.021763Z","shell.execute_reply.started":"2021-07-23T04:15:34.992679Z","shell.execute_reply":"2021-07-23T04:15:35.020832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import wandb\nfrom kaggle_secrets import UserSecretsClient\n\nuser_secrets = UserSecretsClient()\n\n# I have saved my API token with \"wandb_api\" as the Label. \n# If you use some other Label make sure to change the same below. \nwandb_api = user_secrets.get_secret(\"wandb_key\") \n\nwandb.login(key=wandb_api)\n\nwandb.init(project=cfg.wandb_project, name=cfg.wandb_run_name, config=config_dict)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:35.024758Z","iopub.execute_input":"2021-07-23T04:15:35.025112Z","iopub.status.idle":"2021-07-23T04:15:42.630122Z","shell.execute_reply.started":"2021-07-23T04:15:35.025075Z","shell.execute_reply":"2021-07-23T04:15:42.629017Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"💽 Training","metadata":{}},{"cell_type":"code","source":"cbs = [\n WandbCallback(log='gradients',\n log_preds=True,\n log_model=True,\n log_dataset=False,\n dataset_name=None,\n valid_dl=None,\n n_preds=36,\n seed=cfg.seed_val,\n reorder=True),\n \n SaveModelCallback(monitor='valid_loss',\n comp=None,\n min_delta=0.0,\n fname=cfg.job_name,\n every_epoch=False,\n with_opt=False,\n reset_on_fit=True)\n ]","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:42.631934Z","iopub.execute_input":"2021-07-23T04:15:42.632304Z","iopub.status.idle":"2021-07-23T04:15:42.640716Z","shell.execute_reply.started":"2021-07-23T04:15:42.632263Z","shell.execute_reply":"2021-07-23T04:15:42.639242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"learn = timm_learner(dls,\n cfg.model_arch,\n loss_func=cfg.loss_func,\n pretrained=True,\n opt_func=ranger,\n splitter=default_split,\n fp16=cfg.fp16,\n metrics=cfg.metrics,\n cbs=cbs)\n# learn.summary()","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:42.646504Z","iopub.execute_input":"2021-07-23T04:15:42.646803Z","iopub.status.idle":"2021-07-23T04:15:48.142722Z","shell.execute_reply.started":"2021-07-23T04:15:42.646771Z","shell.execute_reply":"2021-07-23T04:15:48.141614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Transfer Learning with Pre-trained Weights\n","metadata":{}},{"cell_type":"code","source":"frozen_params = filter(lambda p: not p.requires_grad, learn.model.parameters())\nunfrozen_params = filter(lambda p: p.requires_grad, learn.model.parameters())\n\nprint(f'Total Parameters: {sum([np.prod(p.size()) for p in learn.model.parameters()])}')\nprint(f'Frozen Parameters: {sum([np.prod(p.size()) for p in frozen_params])}')\nprint(f'Unfrozen Parameters: {sum([np.prod(p.size()) for p in unfrozen_params])}')","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:48.144062Z","iopub.execute_input":"2021-07-23T04:15:48.144451Z","iopub.status.idle":"2021-07-23T04:15:48.207820Z","shell.execute_reply.started":"2021-07-23T04:15:48.144413Z","shell.execute_reply":"2021-07-23T04:15:48.206988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The default learning rates may not be the best match for every architecture we try. We want the learning rate to be high so that the training happens quickly and the model converges faster but, not so high as to skip minimums.\n\nFastai uses a idea developed by Leslie Smith (yes, the same person who invented the learning rate finder!) in his article \"Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates\". He designed a schedule for learning rate separated into two phases: one where the learning rate grows from the minimum value to the maximum value (warmup), and one where it decreases back to the minimum value (annealing). Smith called this combination of approaches 1cycle training. We also also call this schedule cosine annealing.\n\n1cycle training allows us to use a much higher maximum learning rate than other types of training, which gives two benefits:\n\n* By training with higher learning rates, we train faster—a phenomenon Smith named super-convergence.\n* By training with higher learning rates, we overfit less because we skip over the sharp local minima to end up in a smoother (and therefore more generalizable) part of the loss.\n\nWe can use 1cycle training in fastai by calling `fit_one_cycle`.","metadata":{}},{"cell_type":"code","source":"# learn.unfreeze()\nlearn.fit_one_cycle(10, 5e-3)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T04:15:48.208987Z","iopub.execute_input":"2021-07-23T04:15:48.209295Z","iopub.status.idle":"2021-07-23T05:16:57.277791Z","shell.execute_reply.started":"2021-07-23T04:15:48.209268Z","shell.execute_reply":"2021-07-23T05:16:57.276900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Fine-tuning of All Layers by Unfreezing\n\n📌 Fastai lets us pass a Python slice object anywhere that a learning rate is expected. The first value passed will be the learning rate in the earliest layer of the neural network. The second value will be the learning rate in the final layer and the layers in between will have learning rates that are multiplicatively equidistant throughout that range. \n\nThe deepest layers of our pretrained model might not need as high a learning rate as the last ones, so we should probably use different learning rates for those. This is known as using Discriminative Learning Rates.\n","metadata":{}},{"cell_type":"code","source":"learn.unfreeze()\nlearn.fit_one_cycle(2, lr_max=slice(1e-7, 5e-5))","metadata":{"execution":{"iopub.status.busy":"2021-07-23T05:16:57.282089Z","iopub.execute_input":"2021-07-23T05:16:57.282507Z","iopub.status.idle":"2021-07-23T05:47:28.827723Z","shell.execute_reply.started":"2021-07-23T05:16:57.282448Z","shell.execute_reply":"2021-07-23T05:47:28.826791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"📌 Detailed plots and metrics of the run and the model artificats which are uploaded automatically by the callback, can be found in the Wanb project page below\n\nhttps://wandb.ai/sreevishnu-damodaran/SIIM_classifier_public","metadata":{}},{"cell_type":"markdown","source":"📌 Learner already has everything it needs to perform inference","metadata":{}},{"cell_type":"code","source":"# # Predict\nimage_paths = df_annotations[df_annotations.fold==0].image_path.tolist()\ntest_dl = learn.dls.test_dl(image_paths)\npreds = nn.Sigmoid()(learn.get_preds(dl=test_dl)[0]).detach().cpu().numpy()\n\nthr = 0.5\nprint(\"Predicted Before Threshold:\\n\", preds)\npreds = np.where(preds>thr, 1, 0)\n\nlabel_cols = ['negative', 'typical', 'indeterminate', 'atypical']\nactual = df_annotations[df_annotations.fold==0][label_cols].to_numpy()\n\nprint(\"Actual Labels:\\n\", actual)\nprint(\"Predicted Labels:\\n\", preds)","metadata":{"execution":{"iopub.status.busy":"2021-07-23T05:47:28.829307Z","iopub.execute_input":"2021-07-23T05:47:28.829735Z","iopub.status.idle":"2021-07-23T05:48:00.706966Z","shell.execute_reply.started":"2021-07-23T05:47:28.829690Z","shell.execute_reply":"2021-07-23T05:48:00.705540Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nfrom sklearn.metrics import multilabel_confusion_matrix\n\ndef plot_confusion_matrix(con_matrix, axes, label, class_names, fontsize=16):\n df = pd.DataFrame(con_matrix,\n index=class_names,\n columns=class_names\n )\n heatmap = sns.heatmap(df, annot=True, fmt=\"d\", cbar=False, ax=axes)\n heatmap.yaxis.set_ticklabels(heatmap.yaxis.get_ticklabels(),\n rotation=0,\n ha='right',\n fontsize=fontsize)\n heatmap.xaxis.set_ticklabels(heatmap.xaxis.get_ticklabels(),\n rotation=45,\n ha='right',\n fontsize=fontsize)\n axes.set_ylabel('True label')\n axes.set_xlabel('Predicted label')\n axes.set_title(f\"Confusion Matrix for the class: {label}\", fontsize=fontsize, pad=20)\n \ncon_matrices = multilabel_confusion_matrix(actual, preds)\nfig, ax = plt.subplots(2, 2, figsize=(12, 12))\n\nfor axes, con_matrix, label in zip(ax.flatten(), con_matrices, label_cols):\n plot_confusion_matrix(con_matrix, axes, label, [\"N\", \"Y\"])\n\nfig.tight_layout()\nplt.savefig('con_matrix.png')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-07-23T05:48:00.708548Z","iopub.execute_input":"2021-07-23T05:48:00.708952Z","iopub.status.idle":"2021-07-23T05:48:01.605276Z","shell.execute_reply.started":"2021-07-23T05:48:00.708911Z","shell.execute_reply":"2021-07-23T05:48:01.601167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!rm ./wandb/run*/files/media/images/*.png","metadata":{"execution":{"iopub.status.busy":"2021-07-23T05:50:51.903775Z","iopub.execute_input":"2021-07-23T05:50:51.904166Z","iopub.status.idle":"2021-07-23T05:50:52.661633Z","shell.execute_reply.started":"2021-07-23T05:50:51.904125Z","shell.execute_reply":"2021-07-23T05:50:52.660378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"🔖 References\n\n* I highly recommend the Fastai course https://course.fast.ai/ and the book [Deep Learning for Coders with Fastai and PyTorch](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527)\n* Walk with Fastai: https://walkwithfastai.com\n* Fastai Docs: https://docs.fast.ai\n* EfficientNet V2 Paper: https://arxiv.org/pdf/2104.00298.pdf","metadata":{}},{"cell_type":"markdown","source":"
![](\"https://i.ibb.co/qxmZypN/Screenshot-62.png\")
![\"Ask](\"https://img.shields.io/badge/Ask%20me-something-1abc9c.svg?style=flat-square&logo=kaggle\"){kind=icon}
![\"Ask](\"https://img.shields.io/badge/Please-Upvote%20If%20you%20like%20this-07b3c8?style=for-the-badge&logo=kaggle\"){kind=icon}