{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":6799,"databundleVersionId":4225553,"sourceType":"competition"},{"sourceId":502445,"sourceType":"datasetVersion","datasetId":236239},{"sourceId":998277,"sourceType":"datasetVersion","datasetId":547506},{"sourceId":8208466,"sourceType":"datasetVersion","datasetId":4864112},{"sourceId":8212198,"sourceType":"datasetVersion","datasetId":4864677}],"dockerImageVersionId":30699,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"pip install git+https://github.com/openai/CLIP.git","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-04-25T16:22:38.789853Z","iopub.execute_input":"2024-04-25T16:22:38.790164Z","iopub.status.idle":"2024-04-25T16:22:57.043516Z","shell.execute_reply.started":"2024-04-25T16:22:38.790136Z","shell.execute_reply":"2024-04-25T16:22:57.041922Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Question-2

\n\n

PART-1

\n\n* We first load in the instances of the ResNET and CLIP models.\n* The ResNet-50 model has been pretrained on the ImageNet dataset.\n* The CLIP model also used the ResNet-50 model as its visual encoder. However, their architectures will not be the exact same. This is because CLIP's visual encoder is trained in conjunction with a text encoder to understand the relationship between images and text. Thus, the ResNet-50 model used here will be such that its input and output has been optimised for the CLIP architecture. \n* The difference in their architecture is also made clear when we print the number of parameters for each model instance. \n\nCLIP = 38316896
\nR-50 = 25557032\n\n","metadata":{}},{"cell_type":"code","source":"import torch\nimport torchvision.models as models\nfrom torchvision import transforms\nimport clip\nimport gc\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nimport cv2 as cv","metadata":{"execution":{"iopub.status.busy":"2024-04-25T16:22:57.050308Z","iopub.execute_input":"2024-04-25T16:22:57.050796Z","iopub.status.idle":"2024-04-25T16:23:01.953212Z","shell.execute_reply.started":"2024-04-25T16:22:57.050747Z","shell.execute_reply":"2024-04-25T16:23:01.952251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(device)\n\n# Load ResNet-50 model with ImageNet pretraining\nresnet50_imagenet = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V1)\nresnet50_imagenet.eval()\nresnet50_imagenet.to('cuda')\n\nclip_model, _ = clip.load(\"RN50\")\nclip_model.eval()\nclip_model.to('cuda')\nclip_visual_encoder = clip_model.visual","metadata":{"execution":{"iopub.status.busy":"2024-04-25T16:23:01.954703Z","iopub.execute_input":"2024-04-25T16:23:01.955754Z","iopub.status.idle":"2024-04-25T16:23:10.636889Z","shell.execute_reply.started":"2024-04-25T16:23:01.955717Z","shell.execute_reply":"2024-04-25T16:23:10.635889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clip_params = sum(p.numel() for p in clip_visual_encoder.parameters())\nprint(clip_params)\n\nr50_params = sum(p.numel() for p in resnet50_imagenet.parameters())\nprint(r50_params)","metadata":{"execution":{"iopub.status.busy":"2024-04-25T16:24:51.828131Z","iopub.execute_input":"2024-04-25T16:24:51.828824Z","iopub.status.idle":"2024-04-25T16:24:51.836519Z","shell.execute_reply.started":"2024-04-25T16:24:51.82879Z","shell.execute_reply":"2024-04-25T16:24:51.835482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

PART-2

\n\n1. The ImageNet challenge uses a label hierarchy that is based on the WordNet hierarchy. Essential, images that are based on synonyms/similar concepts (also called synsets) are grouped together. Thus, the synsets are organised in a hierarchial structure based on hypernyms and hyponyms - they start from broad categories such as \"animal\" and narrow down to specific categories like \"dog\".\"\n\n2. A synset stands for 'synonym set' and it is a group of words that represent the same/similar conecpt. In ImageNet, each synset is a set of images representing the concept or object described by the synset. For example, the synset for \"dog\" includes various breeds of dogs.\n\n3. There are likely some problems that could arise when we group based on synsets:\n* There could be a lot of intra-class variability as objects within the same synset (for example cars) could vary in colour, size, shape etc.\n* Eventhough two objects belong to different synsets, they could have similar characteristics (for example breeds of dogs) leading to inter-class similarity.\n\n4. Objects within the same synset could also have visual differences:\n* The object could be in different oritentations or positions.\n* The illumination and camera perspective also will not be constant and could change within a synset.\n* Finally, the background variations will also vary how the object is percieved.\n","metadata":{}},{"cell_type":"code","source":"categories = \"/kaggle/input/labels/imagenet1000_clsidx_to_labels.txt\" \nwith open(categories, 'r') as file:\n imagenet_categories = [line.strip() for line in file]\n\ntext_labels = []\nfor text in imagenet_categories:\n text = text.strip().strip('{}').strip(\",\")\n key, value = text.split(': ')\n value = value.strip().strip(\"'\")\n first_word = value.split(',')[0]\n text_labels.append(first_word)\n\ntext_features = []\nnum_categories = len(text_labels)\nfor i in range(0, 10):\n category = text_labels[i]\n text_inputs = torch.cat([clip.tokenize(f\"a photo of a {category}\")])\n text_inputs = text_inputs.to('cuda')\n text_feature = clip_model.encode_text(text_inputs)\n text_features.append(text_feature)\n\ntext_features = torch.cat(text_features)\nprint(text_features)","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:18:57.089746Z","iopub.execute_input":"2024-04-24T19:18:57.090214Z","iopub.status.idle":"2024-04-24T19:18:57.229638Z","shell.execute_reply.started":"2024-04-24T19:18:57.090181Z","shell.execute_reply":"2024-04-24T19:18:57.228725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01440764/n01440764_15071.JPEG\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01496331/n01496331_18272.JPEG\"]\n\ntransformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n plt.subplot(1,2,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n i+=1\n\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor)\n\n \n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:19:00.415417Z","iopub.execute_input":"2024-04-24T19:19:00.41655Z","iopub.status.idle":"2024-04-24T19:19:01.00491Z","shell.execute_reply.started":"2024-04-24T19:19:00.416513Z","shell.execute_reply":"2024-04-24T19:19:01.004029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

PART-3

\n\n* We test out the CLIP model on two images from the ImageNet dataset.\n* The cosine similarities of the image features and the text features are computed and then a softmax function is applied to it to retirieve the corresponding probabilities. \n* The model is correctly able to correlate the image to the text encoding. ","metadata":{}},{"cell_type":"code","source":"cosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\n\nfor i, prob in enumerate(probabilities):\n max_index = prob.argmax().item()\n predicted_category = text_labels[max_index]\n print(f\"Image {i + 1} is predicted to be in category: {predicted_category}\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:19:08.112463Z","iopub.execute_input":"2024-04-24T19:19:08.113099Z","iopub.status.idle":"2024-04-24T19:19:08.120311Z","shell.execute_reply.started":"2024-04-24T19:19:08.113067Z","shell.execute_reply":"2024-04-24T19:19:08.119364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_ten = [text_labels[22], text_labels[29], text_labels[39], text_labels[63], text_labels[84], text_labels[71], text_labels[79], text_labels[100], text_labels[245], text_labels[288]]\nprint(labels_ten)\nprint(len(labels_ten))","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:19:10.938766Z","iopub.execute_input":"2024-04-24T19:19:10.939928Z","iopub.status.idle":"2024-04-24T19:19:10.94614Z","shell.execute_reply.started":"2024-04-24T19:19:10.939891Z","shell.execute_reply":"2024-04-24T19:19:10.945097Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

PART-4

\n\n* 10 different imagenet classes have been considered: ['bald eagle', 'axolotl', 'common iguana', 'Indian cobra', 'peacock', 'scorpion', 'centipede', 'black swan', 'French bulldog', 'leopard']\n\n* Images from all these classes have been tested on both models to compare their performance.\n\n\n* We notice that the CLIP model performs much better than the ResNet one at images from the ImageNet Sketch and ImageNet-r datasets. Thus, the CLIP model is much more robust to dristribution shifts than a generic ImageNet model. CLIP’s zero-shot learning ability allows it to generalize across various tasks and datasets without needing specific training for each one (because it correlates the text encodings and image encodings without having to rely on learning the image features)\n\n* The ImageNet trained ResNet will perform better than CLIP if it is given images directly from the ImageNet dataset. However, this difference is very minute and CLIP performs on par with the ResNet model. \n\n

Bald Eagle

\n\nResNet 50:","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n01614925/sketch_2.JPEG\", \"/kaggle/input/baldeagle/art_6.jpg\", \"/kaggle/input/imagenet-object-localization-challenge/ILSVRC/Data/CLS-LOC/train/n01614925/n01614925_10607.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:19:13.56585Z","iopub.execute_input":"2024-04-24T19:19:13.566217Z","iopub.status.idle":"2024-04-24T19:19:14.256034Z","shell.execute_reply.started":"2024-04-24T19:19:13.56619Z","shell.execute_reply":"2024-04-24T19:19:14.255121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP:","metadata":{}},{"cell_type":"code","source":"text_features = []\nfor i in range(0, 10):\n category = labels_ten[i]\n text_inputs = torch.cat([clip.tokenize(f\"a photo of a {category}\")])\n text_inputs = text_inputs.to('cuda')\n text_feature = clip_model.encode_text(text_inputs)\n text_features.append(text_feature)\n\ntext_features = torch.cat(text_features)","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:19:25.864727Z","iopub.execute_input":"2024-04-24T19:19:25.86553Z","iopub.status.idle":"2024-04-24T19:19:25.989814Z","shell.execute_reply.started":"2024-04-24T19:19:25.865481Z","shell.execute_reply":"2024-04-24T19:19:25.98877Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nprint(\"\\nTop 5 matches with CLIP for image {i+1}:\")\nfor i, top5 in enumerate(clip_top5):\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:19:28.564778Z","iopub.execute_input":"2024-04-24T19:19:28.565485Z","iopub.status.idle":"2024-04-24T19:19:29.232024Z","shell.execute_reply.started":"2024-04-24T19:19:28.565451Z","shell.execute_reply":"2024-04-24T19:19:29.23115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Axolotl

\n\nResNet 50:","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n01632777/sketch_17.JPEG\", \"/kaggle/input/baldeagle/art_7.jpg\", \"/kaggle/input/imagenet-object-localization-challenge/ILSVRC/Data/CLS-LOC/train/n01632777/n01632777_10179.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:19:49.058753Z","iopub.execute_input":"2024-04-24T19:19:49.059633Z","iopub.status.idle":"2024-04-24T19:19:49.792465Z","shell.execute_reply.started":"2024-04-24T19:19:49.059596Z","shell.execute_reply":"2024-04-24T19:19:49.791549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:19:55.672773Z","iopub.execute_input":"2024-04-24T19:19:55.673603Z","iopub.status.idle":"2024-04-24T19:19:56.437781Z","shell.execute_reply.started":"2024-04-24T19:19:55.673561Z","shell.execute_reply":"2024-04-24T19:19:56.436542Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Common Iguana

\nRESNET","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n01677366/sketch_14.JPEG\", \"/kaggle/input/baldeagle/deviantart_17.jpg\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01677366/n01677366_1884.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n image = image.convert(\"RGB\")\n \n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:20:04.058111Z","iopub.execute_input":"2024-04-24T19:20:04.058875Z","iopub.status.idle":"2024-04-24T19:20:04.778957Z","shell.execute_reply.started":"2024-04-24T19:20:04.05883Z","shell.execute_reply":"2024-04-24T19:20:04.778036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP","metadata":{}},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:20:18.828865Z","iopub.execute_input":"2024-04-24T19:20:18.829276Z","iopub.status.idle":"2024-04-24T19:20:19.50216Z","shell.execute_reply.started":"2024-04-24T19:20:18.829228Z","shell.execute_reply":"2024-04-24T19:20:19.501245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Indian Cobra

\nResNET","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n01748264/sketch_23.JPEG\", \"/kaggle/input/baldeagle/deviantart_7.jpg\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01748264/n01748264_1664.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:20:55.433197Z","iopub.execute_input":"2024-04-24T19:20:55.434038Z","iopub.status.idle":"2024-04-24T19:20:56.500202Z","shell.execute_reply.started":"2024-04-24T19:20:55.434006Z","shell.execute_reply":"2024-04-24T19:20:56.499272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP","metadata":{}},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:21:03.03499Z","iopub.execute_input":"2024-04-24T19:21:03.036005Z","iopub.status.idle":"2024-04-24T19:21:03.688227Z","shell.execute_reply.started":"2024-04-24T19:21:03.035956Z","shell.execute_reply":"2024-04-24T19:21:03.687277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Peacock

\nResNET","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n01806143/sketch_19.JPEG\", \"/kaggle/input/baldeagle/art_3.jpg\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01806143/n01806143_6599.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n image = image.convert(\"RGB\")\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:21:51.275779Z","iopub.execute_input":"2024-04-24T19:21:51.276191Z","iopub.status.idle":"2024-04-24T19:21:52.382015Z","shell.execute_reply.started":"2024-04-24T19:21:51.27616Z","shell.execute_reply":"2024-04-24T19:21:52.381122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP","metadata":{}},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:22:00.583078Z","iopub.execute_input":"2024-04-24T19:22:00.583844Z","iopub.status.idle":"2024-04-24T19:22:01.456996Z","shell.execute_reply.started":"2024-04-24T19:22:00.58381Z","shell.execute_reply":"2024-04-24T19:22:01.456035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Centipede

\nResNET","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n01784675/sketch_24.JPEG\", \"/kaggle/input/baldeagle/graffiti_0.jpg\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01784675/n01784675_21526.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:24:04.619145Z","iopub.execute_input":"2024-04-24T19:24:04.619573Z","iopub.status.idle":"2024-04-24T19:24:05.475897Z","shell.execute_reply.started":"2024-04-24T19:24:04.619538Z","shell.execute_reply":"2024-04-24T19:24:05.474983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP","metadata":{}},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:24:17.545844Z","iopub.execute_input":"2024-04-24T19:24:17.546258Z","iopub.status.idle":"2024-04-24T19:24:18.291796Z","shell.execute_reply.started":"2024-04-24T19:24:17.546228Z","shell.execute_reply":"2024-04-24T19:24:18.290917Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Black Swan

\nResNET","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n01860187/sketch_20.JPEG\", \"/kaggle/input/baldeagle/art_1.jpg\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01860187/n01860187_375.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:28:23.653806Z","iopub.execute_input":"2024-04-24T19:28:23.654249Z","iopub.status.idle":"2024-04-24T19:28:24.651257Z","shell.execute_reply.started":"2024-04-24T19:28:23.65419Z","shell.execute_reply":"2024-04-24T19:28:24.650309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP","metadata":{}},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:28:34.229094Z","iopub.execute_input":"2024-04-24T19:28:34.229477Z","iopub.status.idle":"2024-04-24T19:28:34.976678Z","shell.execute_reply.started":"2024-04-24T19:28:34.229446Z","shell.execute_reply":"2024-04-24T19:28:34.975735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

French Bulldog

\nResNET","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n02108915/sketch_19.JPEG\", \"/kaggle/input/baldeagle/misc_25.jpg\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n02108915/n02108915_2118.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:32:58.799925Z","iopub.execute_input":"2024-04-24T19:32:58.800669Z","iopub.status.idle":"2024-04-24T19:33:01.375114Z","shell.execute_reply.started":"2024-04-24T19:32:58.800635Z","shell.execute_reply":"2024-04-24T19:33:01.374152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP","metadata":{}},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:33:37.829393Z","iopub.execute_input":"2024-04-24T19:33:37.830405Z","iopub.status.idle":"2024-04-24T19:33:39.767988Z","shell.execute_reply.started":"2024-04-24T19:33:37.830361Z","shell.execute_reply":"2024-04-24T19:33:39.767041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Leopard

\nResNET","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n02128385/sketch_12.JPEG\", \"/kaggle/input/baldeagle/embroidery_2.jpg\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n02128385/n02128385_2965.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:36:24.007542Z","iopub.execute_input":"2024-04-24T19:36:24.008097Z","iopub.status.idle":"2024-04-24T19:36:25.073538Z","shell.execute_reply.started":"2024-04-24T19:36:24.008053Z","shell.execute_reply":"2024-04-24T19:36:25.072602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP","metadata":{}},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:36:34.080974Z","iopub.execute_input":"2024-04-24T19:36:34.081939Z","iopub.status.idle":"2024-04-24T19:36:34.937667Z","shell.execute_reply.started":"2024-04-24T19:36:34.081905Z","shell.execute_reply":"2024-04-24T19:36:34.936746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Scorpion

\nResNET","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetsketch/sketch/n01770393/sketch_33.JPEG\", \"/kaggle/input/baldeagle/art_0.jpg\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01770393/n01770393_7489.JPEG\"]\n\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = Image.open(image_path)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n transform = transforms.Compose([\n transforms.ToTensor(), \n transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n ])\n image = transform(image)\n image = image.unsqueeze(0)\n image = image.to('cuda')\n \n resnet_outputs = resnet50_imagenet(image)\n temp, resnet_top5 = torch.topk(resnet_outputs, k=5, dim=1)\n resnet_top5 = resnet_top5.squeeze().cpu().numpy()\n \n print(f\"Top 5 matches with ResNet for image {i+1}:\")\n for idx in resnet_top5:\n print(f\"Class: {text_labels[idx]}, Probability: {torch.softmax(resnet_outputs, dim=1).squeeze()[idx]:.4f}\")\n i += 1\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:39:11.628931Z","iopub.execute_input":"2024-04-24T19:39:11.629878Z","iopub.status.idle":"2024-04-24T19:39:12.834687Z","shell.execute_reply.started":"2024-04-24T19:39:11.629834Z","shell.execute_reply":"2024-04-24T19:39:12.8337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"CLIP","metadata":{}},{"cell_type":"code","source":"transformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,3,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = clip_model.encode_image(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T19:39:41.480884Z","iopub.execute_input":"2024-04-24T19:39:41.481288Z","iopub.status.idle":"2024-04-24T19:39:42.445014Z","shell.execute_reply.started":"2024-04-24T19:39:41.481258Z","shell.execute_reply":"2024-04-24T19:39:42.444088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

PART-5

\n\n* To convert the visual encoder model to fp16, we half the inputs and the parameters in the model. \n* The difference between the fp32 and fp16 models is apparent from the mean and standard deviation computed. \n\nFP32 mean inference time: 0.0087s, std dev: 0.0008s
\nFP16 mean inference time: 0.0085s, std dev: 0.0003s\n\n* Thus, we see that the fp16 does take lesser time to run. It's lower standard deviation also indicates that it is stable and consistent ","metadata":{}},{"cell_type":"code","source":"import time\nimport numpy as np\n\nimage_path = \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01518878/n01518878_11113.JPEG\"\n\nimage = cv.imread(image_path)\nimage = cv.cvtColor(image, cv.COLOR_BGR2RGB)\nimage = Image.fromarray(image)\nimage_tensor = _(image)\nimage_tensor = image_tensor.unsqueeze(0)\nimage_tensor = image_tensor.to('cuda')\n\n\nclip_visual = clip_model.visual\nclip_visual.to('cuda')\nfp16 = clip_visual.half()\nfp16.to('cuda')\n\ntimes = []\nfor i in range(100):\n start_time = time.time()\n with torch.no_grad():\n output = fp16(image_tensor)\n times.append(time.time() - start_time)\nmean_fp16 = np.mean(times)\nstd_fp16 = np.std(times)\n# sum([(x - mean_time) ** 2 for x in times])\n# std_dev_time = (summation/100) ** 0.5\n\ntimes = []\nfor i in range(100):\n start_time = time.time()\n with torch.no_grad():\n output = clip_visual(image_tensor)\n times.append(time.time() - start_time)\nmean_fp32 = np.mean(times)\nstd_fp32 = np.std(times)\n\nprint(f\"FP32 mean inference time: {mean_fp32:.4f} s, std dev: {std_fp32:.4f} s\")\nprint(f\"FP16 mean inference time: {mean_fp16:.4f}s, std dev: {std_fp16:.4f}s\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-24T20:19:15.223421Z","iopub.execute_input":"2024-04-24T20:19:15.224139Z","iopub.status.idle":"2024-04-24T20:19:16.969089Z","shell.execute_reply.started":"2024-04-24T20:19:15.224108Z","shell.execute_reply":"2024-04-24T20:19:16.968082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* The top 5 rankings for both the fp16 and fp32 model are the same. However, we do notice a difference in the probability scores. This difference can be attributed to the fact that the fp32 model will be more accurate. \n\n* However, we can see that the accuracy for the fp16 model is not very bad and it still computes the correct class for the corresponding images. Hence, there is a good trade-off between the memory saved and accuracy.","metadata":{}},{"cell_type":"code","source":"image_paths = [\"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01770393/n01770393_7489.JPEG\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01784675/n01784675_21526.JPEG\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01806143/n01806143_6599.JPEG\", \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01748264/n01748264_1664.JPEG\", \"/kaggle/input/imagenet-object-localization-challenge/ILSVRC/Data/CLS-LOC/train/n01632777/n01632777_10179.JPEG\"]\n\ntransformed_images = []\nplt.figure()\ni = 0\nfor image_path in image_paths:\n image = cv.imread(image_path)\n image = cv.cvtColor(image, cv.COLOR_BGR2RGB)\n plt.subplot(1,5,i+1)\n plt.imshow(image)\n plt.title(f\"Image {i+1}\")\n \n image = Image.fromarray(image)\n image_tensor = _(image)\n image_tensor = image_tensor.unsqueeze(0)\n image_tensor = image_tensor.to('cuda')\n transformed_images.append(image_tensor) \n i+=1\n \nimages_tensor = torch.cat(transformed_images)\nimage_features = fp16(images_tensor)\n\ncosine_similarities = torch.nn.functional.cosine_similarity(image_features.unsqueeze(1), text_features.unsqueeze(0), dim=2)\nprobabilities = torch.nn.functional.softmax(cosine_similarities, dim=1)\nclip_top5 = torch.topk(probabilities, k=5, dim=1).indices.squeeze().cpu().numpy()\n\nfor i, top5 in enumerate(clip_top5):\n print(f\"\\nTop 5 matches with fp16 CLIP for image {i+1}:\")\n for idx in top5:\n print(f\" Class: {labels_ten[idx]}, Similarity: {probabilities[i, idx]:.4f}\")\n print(\"\\n\")","metadata":{"execution":{"iopub.status.busy":"2024-04-24T20:23:56.24494Z","iopub.execute_input":"2024-04-24T20:23:56.245657Z","iopub.status.idle":"2024-04-24T20:23:57.314617Z","shell.execute_reply.started":"2024-04-24T20:23:56.245623Z","shell.execute_reply":"2024-04-24T20:23:57.313664Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"* Using the pytorch memory usage function, we see that the forward pass of the fp16 model uses significantly less memory than the fp32 mdoel. This is because it is not as computationally expensive and also uses lesser number of bits (16-bit floating point numbers).","metadata":{}},{"cell_type":"code","source":"image_path = \"/kaggle/input/imagenetmini-1000/imagenet-mini/train/n01518878/n01518878_11113.JPEG\"\n\nimage = cv.imread(image_path)\nimage = cv.cvtColor(image, cv.COLOR_BGR2RGB)\nimage = Image.fromarray(image)\nimage_tensor = _(image)\nimage_tensor = image_tensor.unsqueeze(0)\nimage_tensor = image_tensor.to('cuda')","metadata":{"execution":{"iopub.status.busy":"2024-04-24T20:42:22.618725Z","iopub.execute_input":"2024-04-24T20:42:22.619224Z","iopub.status.idle":"2024-04-24T20:42:22.63879Z","shell.execute_reply.started":"2024-04-24T20:42:22.619192Z","shell.execute_reply":"2024-04-24T20:42:22.638034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cuda.reset_peak_memory_stats()\n\ntorch.cuda.memory._record_memory_history(max_entries=100000)\noutput = fp16(image_tensor)\ntorch.cuda.memory._dump_snapshot(\"fp16-model\")\n\ntorch.cuda.memory._record_memory_history(enabled=None)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"torch.cuda.reset_peak_memory_stats()\n\ntorch.cuda.memory._record_memory_history(max_entries=100000)\noutput = clip_visual(image_tensor)\ntorch.cuda.memory._dump_snapshot(\"fp32-model\")\n\ntorch.cuda.memory._record_memory_history(enabled=None)","metadata":{"execution":{"iopub.status.busy":"2024-04-24T20:52:32.501509Z","iopub.execute_input":"2024-04-24T20:52:32.502235Z","iopub.status.idle":"2024-04-24T20:52:32.554017Z","shell.execute_reply.started":"2024-04-24T20:52:32.502202Z","shell.execute_reply":"2024-04-24T20:52:32.553174Z"},"trusted":true},"execution_count":null,"outputs":[]}]}