{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://scontent.fblr2-1.fna.fbcdn.net/v/t39.2365-6/131715266_303592217719628_2247522524990492321_n.png?_nc_cat=101&ccb=1-3&_nc_sid=ad8a9d&_nc_ohc=Pt6GSI30vO0AX_YI3Nd&_nc_ht=scontent.fblr2-1.fna&oh=a39cdc80e4052f0a8507b596e83318bf&oe=606E10D5)\n\n\n\nThis new technique — Data-efficient image Transformers (DeiT) — requires less data and less computing resources to produce a high-performance image classification model. Training a DeiT model over 3 days, achieved 84.2 top-1 accuracy on the widely used ImageNet benchmark without using any external data for training. This result is competitive with the performance of cutting-edge convolutional neural networks (CNNs), which have been the dominant approach to image classification for many years.\n\nDeiT is an important step forward in using Transformers to advance computer vision. Its performance is already competitive with that of CNNs, even though the latter have been the dominant approach for computer vision tasks for the last eight years and have benefited from many improvements and adjustments. This indicates that additional research will produce significant additional gains."},{"metadata":{},"cell_type":"markdown","source":"# *Upvote the kernel if you find it insightful!*"},{"metadata":{},"cell_type":"markdown","source":"# Install and Import Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -q timm","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Python library used for working with arrays.\nimport numpy as np\n\n# Python library to interact with the file system.\nimport os\n\n# Software library written for data manipulation and analysis. \nimport pandas as pd\n\n# fastai library for computer vision tasks\nfrom fastai.vision.all import *\nfrom fastai.metrics import *\n\n# Developing and training neural network based deep learning models.\nimport torch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Import Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset_path = Path('../input/ranzcr-clip-catheter-line-classification')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(dataset_path/'train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['path'] = train_df['StudyInstanceUID'].map(lambda x:str(dataset_path/'train'/x)+'.jpg')\ntrain_df = train_df.drop(columns=['StudyInstanceUID'])\ntrain_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Transforms we need to do for each image in the dataset (ex: resizing).\nitem_tfms = RandomResizedCrop(384, min_scale=0.75, ratio=(1.,1.)) \n\n# Transforms that can take place on a batch of images (ex: many augmentations).\nbatch_tfms = [*aug_transforms(size=384, max_warp=0), Normalize.from_stats(*imagenet_stats)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"label_names = list(train_df.columns[:11])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data = DataBlock(blocks=(ImageBlock, MultiCategoryBlock(encoded=True, vocab=label_names)), # multi-label target\n                 splitter = RandomSplitter(seed=42),# split data into training and validation subsets.\n                 get_x = ColReader(12),# obtain the input images.\n                 get_y = ColReader(list(range(11))), # obtain the targets.\n                 item_tfms = item_tfms,\n                 batch_tfms = batch_tfms)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get dataloader and show the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"dls = data.dataloaders(train_df,bs=16)\n\n# We can call show_batch() to see what a sample of a batch looks like.\ndls.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = torch.hub.load('facebookresearch/deit:main', 'deit_base_patch16_384', pretrained=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.head","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.head = nn.Sequential(nn.Dropout(0.25), \n                           nn.Linear(768, 11))\n\nmodel.head","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn = Learner(dls, model, metrics = [accuracy_multi])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.lr_find()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"learn.fine_tune(1, base_lr=1.2022644114040304e-05)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission File"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_df = pd.read_csv(dataset_path/'sample_submission.csv')\nsample_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_sample_df = sample_df.copy()\n_sample_df['PatientID'] = 'None'\n_sample_df['path'] = _sample_df['StudyInstanceUID'].map(lambda x:str(dataset_path/'test'/x)+'.jpg')\n_sample_df = _sample_df.drop(columns=['StudyInstanceUID'])\ntest_dl = dls.test_dl(_sample_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dl.show_batch()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Test Time Augmentation (TTA)\nSimilar to what Data Augmentation is doing to the training set, the purpose of Test Time Augmentation is to perform random modifications to the test images. Thus, instead of showing the regular, “clean” images, only once to the trained model, we will show it the augmented images several times. We will then average the predictions of each corresponding image and take that as our final guess.\n\nThe reason why it works is that, by averaging our predictions, on randomly modified images, we are also averaging the errors. The error can be big in a single vector, leading to a wrong answer, but when averaged, only the correct answer stand out."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Return predictions on the ds_idx dataset or dl using Test Time Augmentation\npreds, _ = learn.tta(dl=test_dl,n=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = sample_df\nfor i in range(len(submission_df)):\n    for j in range(len(label_names)):\n        submission_df.iloc[i, j+1] = preds[i][j].numpy().astype(np.float32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.to_csv(f'submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}