{"cells":[{"metadata":{},"cell_type":"markdown","source":"

Vision Transformers but Simpler 😉

\n\n"},{"metadata":{},"cell_type":"markdown","source":"

(Well not talking about transformers here but I am surely
trying to not bore you to death already, lol.)

"},{"metadata":{},"cell_type":"markdown","source":"

Importing Libraries

\n\n

Lets quickly import the libraries that are needed for this model to work.

"},{"metadata":{"trusted":true,"collapsed":true},"cell_type":"code","source":"!pip -q install vit-pytorch\n!pip -q install linformer","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Important for Printing stuff:\nfrom __future__ import print_function\n\n# Basic native Python Tool:\nimport glob\nfrom itertools import chain\nimport os\nimport random\nimport zipfile\n\n# Our all season best friends:\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom PIL import Image\n\n# PyTorch because survival:\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import StepLR\nfrom torch.utils.data import DataLoader, Dataset\nfrom torchvision import datasets, transforms\n\n# To keep time in check:\nfrom tqdm.notebook import tqdm\n\n# For Preprocessing of the Data:\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import OrdinalEncoder\n\n# For grabbing our pretrained Model:\nfrom linformer import Linformer\nfrom vit_pytorch.efficient import ViT","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Some Hyperparameters

\n

Let's define some of the values before hand right now so that later we know what we are working with.

"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting up some HyperParameters\n\nbatch_size = 64\nepochs = 5\nlr = 3e-5\ngamma = 0.7\nseed = 42","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Seeding

"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining a function to seed everything.\ndef seed_everything(seed):\n random.seed(seed)\n os.environ[\"PYTHONSEED\"] = str(seed)\n np.random.seed(seed)\n torch.cuda.manual_seed(seed)\n torch.cuda.manual_seed_all(seed)\n torch.backends.cudnn.deterministic = True\n\n# Running the function:\nseed_everything(seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Setting the device as CUDA or moving stuff to the GPU:\ndevice = \"cuda\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Data Preprocessing


\n

In this section we basically edit the annotated dataset \n \nthat has been provided to us and form two seperate lists which have two things: Paths for Training Images and Labels.\nIn order to use those labels, I converted them from strings to numbers. I have no idea if that helped or not but again, you don't question your code if it is working just fine.

\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining the directories for our files/\ntrain_dir = \"../input/ranzcr-clip-catheter-line-classification/train\"\ntest_dir = \"../input/ranzcr-clip-catheter-line-classification/test\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loading the data CSV.\ndata = pd.read_csv(\"../input/ranzcr-clip-catheter-line-classification/train_annotations.csv\")\ndata.columns\ndata = data.drop([\"data\"], axis = 1)\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Converting the columns into integers.\nord_enc = OrdinalEncoder()\ndata[['label']] = ord_enc.fit_transform(data[['label']])\n\n# Converting the Labels from floats to integers.\ndata.label = data.label.astype(\"int\")\n\n# Grabbing the labels as a list.\nlabel = data[\"label\"]\nlabel = label.to_list()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Producing the list of paths for the trainingset image files:\ntrain_list = []\nfor i in data.index:\n \n # Grabbing the file name.\n a = data[\"StudyInstanceUID\"].loc[i]\n \n # Attaching the file's path to it.\n b = train_dir + \"/\" + a + \".jpg\"\n \n # Puttting it in a tupple along with it's label.\n train_list.append((b, data['label'].loc[i]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

If you upvoted the notebook without fully going through it, you will find $10,000 under your pillow tomorrow morning.

"},{"metadata":{},"cell_type":"markdown","source":"

Visualizing Data


\n

Let's check out our data.

"},{"metadata":{"trusted":true},"cell_type":"code","source":"random_idx = np.random.randint(1, len(train_list), size=9)\nfig, axes = plt.subplots(3,3, figsize=(16,12))\n\nfor idx, ax in enumerate(axes.ravel()):\n img = Image.open(train_list[idx][0])\n ax.set_title(train_list[idx][1])\n ax.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Splitting the data into training and validation datasets.

"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_list, valid_list = train_test_split(train_list, \n test_size=0.2,\n random_state=seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Data Augmentation

\n

Defining our transforms presets for data augmentation.

\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_transforms = transforms.Compose(\n [\n transforms.Resize((224,224)),\n transforms.RandomResizedCrop(224),\n transforms.RandomHorizontalFlip(),\n transforms.ToTensor(),\n transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))\n ])\n\nvalid_transforms = transforms.Compose(\n [\n transforms.Resize((224,224)),\n transforms.RandomResizedCrop(224),\n transforms.RandomHorizontalFlip(),\n transforms.ToTensor(),\n transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))\n ])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

PyTorch Dataset

\n

Finally defining a PyTorch dataset, this one is as simple to understand as it could get.
Basically grab the transforms presets that we defined above. Other than that we have the file length function and the function that grabs the labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"class RANZCRDataset(Dataset):\n \n # Grabbing the transform presets.\n def __init__(self, file_list, transform=None):\n self.file_list = file_list\n self.transform = transform\n \n def __len__(self):\n self.filelength = len(self.file_list)\n return self.filelength\n \n # Returning the transformed images and the image's label.\n def __getitem__(self, idx):\n \n # Note that file list consists of tuples.\n # The first item in tuple is the image.\n img_path = self.file_list[idx][0]\n img = Image.open(img_path).convert(\"RGB\")\n img_transformed = self.transform(img)\n \n # The second item in the tuple is the label.\n label = self.file_list[idx][1]\n \n return img_transformed, label\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining the Datasets.\ntrain_data = RANZCRDataset(train_list, transform = train_transforms)\nvalid_data = RANZCRDataset(valid_list, transform = valid_transforms)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining the Dataloaders.\ntrain_loader = DataLoader(dataset = train_data, batch_size = batch_size, shuffle = True)\nvalid_loader = DataLoader(dataset = valid_data, batch_size = batch_size, shuffle = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Efficient Attention"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Grabbing the transformer.\nefficient_transformer = Linformer(\n dim=128,\n seq_len=49+1, # 7x7 patches + 1 cls-token\n depth=12,\n heads=8,\n k=64\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Grabbing the model.\nmodel = ViT(\n dim=128,\n image_size=224,\n patch_size=32,\n num_classes=11,\n transformer=efficient_transformer,\n channels=3,\n).to(device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining some other presets:\ncriterion = nn.CrossEntropyLoss()\n\noptimizer = optim.Adam(model.parameters(), lr = lr)\n\nscheduler = StepLR(optimizer, step_size=1, gamma=gamma)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

I can't believe you came all over here. (That's what she said)

\n

An Upvote from you would probably get me a job,
please help me move out of my parent's house by hitting the upvote button! 😏

"},{"metadata":{},"cell_type":"markdown","source":"

Training the Model.

\n

Training the model Alas.

"},{"metadata":{"trusted":true},"cell_type":"code","source":"for epoch in range(epochs):\n epoch_loss = 0.0\n epoch_accuracy = 0.0\n \n for data, label in tqdm(train_loader):\n data = data.to(device)\n label = label.to(device)\n \n output = model(data)\n \n label = torch.nn.functional.one_hot(label, num_classes = 11)\n label = label.type_as(output)\n \n loss = criterion(output, label)\n \n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n \n acc = (output.argmax(dim=1) == label).float().mean()\n epoch_accuracy += acc / len(train_loader)\n epoch_loss += loss / len(train_loader)\n \n with torch.no_grad():\n epoch_val_accuracy = 0\n epoch_val_loss = 0\n for data, label in valid_loader:\n data = data.to(device)\n label = label.to(device)\n \n val_output = model(data)\n val_loss = criterion(val_output, label)\n \n acc = (val_output.argmax(dim=1) == label).float().mean()\n epoch_val_accuracy += acc / len(valid_loader)\n epoch_val_loss += val_loss / len(valid_loader)\n \n \n print(\n f\"Epoch: {epoch+1} - loss: {epoch_loss:.4f} - acc : {epoch_accuracy:.4f} - val_loss : {epoch_val_loss:.4f} - val_acc: {epoch_val_accuracy: .4f}\\n\"\n )","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}