{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Image Encoding extraction using ResNet & Nested Dictionnary !\n\n**March 2022**\n\n**If you use parts of this notebook in your scripts/notebooks, giving  some kind of credit would be very much appreciated :)  You can for instance link back to this `notebook`, and `upvote it`. Thanks!**\n\n\n\nIn this notebook, there is three crucials part:\n1.  Use of ``ImagesFolder`` and ``DataLoader`` with a Transform Pipeline that resizes and scales all the dataset ( >100k images ) **within 1 minutes**.\n2.  Use of ``Resnet18`` ( or any pretrained model ) to encode every image to a feature vector of 512 dimensions.\n3. Use of a ``nested dictionnary`` that mimics perfectly the same structure as the image folder contents. This allows you to select the feature vector of any images within a second, and use it for your model directly using the keys of the dictionary( more details in the last  section ).\n\n**The particularity of this work is that it allows you to use the information of the images directly in your model as complementary features, unlike notebooks that propose only a resized version of the images, this notebook proposes directly the encoding of any pretrained model using an appropriate structure thanks to nested dictionnary.**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport torch \nimport random\nimport torch.nn as nn\nimport torch.optim as optim \nimport os\nimport torchvision\nfrom torch.utils.data import DataLoader, Dataset\nimport torch.utils.data as utils\nfrom torchvision import transforms\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom tqdm import tqdm\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2022-03-05T18:56:30.363667Z","iopub.execute_input":"2022-03-05T18:56:30.364042Z","iopub.status.idle":"2022-03-05T18:56:31.793067Z","shell.execute_reply.started":"2022-03-05T18:56:30.363929Z","shell.execute_reply":"2022-03-05T18:56:31.792220Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# ImageFolder & DataLoader","metadata":{}},{"cell_type":"code","source":"images_path = \"../input/h-and-m-personalized-fashion-recommendations/images\"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_data(images_path, batch_size):\n\n    transformer = transforms.Compose(\n        [transforms.Resize((224,224)), # Resize images to (3,64,64) to speed up the process \n         transforms.ToTensor(),        # Transform numpy to torch tensors\n         transforms.Normalize(mean=[0.485, 0.456, 0.406],  # Normalize images using the standardization used for Resnet Training\n                              std=[0.229, 0.224, 0.225]),\n         ])\n    data = torchvision.datasets.ImageFolder(root=images_path, transform=transformer) # Extract all images for the subfolders\n    data_loader = torch.utils.data.DataLoader(data, batch_size=batch_size, shuffle=False, drop_last=False, num_workers=4) # Initialize a dataloader from the latter\n    del data\n    return data_loader\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check for GPU \nuse_cuda = torch.cuda.is_available()\ndevice = torch.device(\"cuda\" if use_cuda else \"cpu\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creation of the dataloader\nbatch_size = 32\nimgs_dataloader = load_data(images_path, batch_size)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Resnet18","metadata":{}},{"cell_type":"code","source":"from torchvision.models import resnet18\n\nmodel = resnet18(pretrained=True) \nresnet = torch.nn.Sequential(\n    # Extract only the encoding part of the Resnet and omit the classifier part\n    nn.Sequential(*list(model.children())[:-1]),\n    torch.nn.Flatten()\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This function allows to automatically adapt the shape of the placeholder tensor \"encoding\"\n# to the encoding size of any pretrained model : vgg,resnet,mobilenet...\ndef get_output_shape(model_, image_dim=(1,3,224,224)): \n    return model_(torch.rand(*(image_dim))).data.shape[1]\n\n# For example the output of this function for Resnet18 is 512.\nencoding_dim =  get_output_shape(resnet) \n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"initialize placeholder ....\") \n\n# Initialize an empty tensor\n# Here, because of GPU limits, I had to initialize the empty tensor on the CPU\n\nencoding = np.zeros((len(imgs_dataloader.dataset),encoding_dim))\nprint(\"Finish !\")\n# I choose a numpy array to enable the use of the encoding for both pytorch and tensorflow community\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Extract the resnet encoding for each image\ndef resnet_encoding(dataloader,encoding,model_):  \n\n\n    # Compute number of batches\n    print(\"Resnet18 inference begins ...\")\n    nb_batches = int(len(dataloader.dataset) / batch_size) + 1\n    generator = iter(dataloader)\n    del dataloader\n    model_ = model_.to(device)\n    for i in tqdm(range(nb_batches), position=0, leave=True):\n        x,_ = next(generator)\n        imgs = x.to(device)\n        # for each batch, we do the inference of the Resnet18 to extract the encoding\n        # Because the empty tensor is on CPU, I had to transfer the encoding to cpu\n        encoding[i*batch_size:(i+1)*batch_size,:] = model_(imgs).cpu().detach().numpy()\n        del imgs,x,_\n    print(\"Finish !\")\n    \n    return encoding\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"encoding = resnet_encoding(imgs_dataloader,encoding,resnet)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Nested Dictionaries for the encoding ( Same structure as the image folder )","metadata":{}},{"cell_type":"code","source":"\n# This functions outputs nested dictionnary that contains subfolder id, image id, and its corresponding encoding,\n# For example:\n# final_dict = { '010' : { '0108775015': encoding_of_0108775015\n#                          '0108775044': encoding_of_0108775044\n#                           ...\n#                        },\n#                '011': {....},\n#                '012': {....},\n#                ...,\n#                ...,\n#\n#               '095': {....},\n#\n#              }\n\ndef encoding_dictionnary(images_path,encoding):\n    \n## EXTRACT NAMES OF EVERY SUBDIRECTORY OF THE IMAGE ROOT DIRECTORY. (ex : 010,011,....042,043) \n    print(\"Creation of the first dictionnary ..\")\n    subdirs_path_list = []\n    for path, subdirs, files in os.walk(images_path,topdown=False):\n        if subdirs != []: # Here with empirical tests, I noticed that there is a lot of empty lists, I didn't manage to check the reason \n            subdirs_path_list.append(subdirs)\n\n\n            \n    dict_imgs = {key: None for key in sorted(subdirs_path_list[0])}  # dictionnary that store subfolder name as dictionnary key. \n                                                                    # For the moment, the element of each key is empty\n    print(\"Finish !\")\n\n\n## EXTRACT IMAGES ID OF EACH SUBDIRECTORY AS KEYS \n    l = []\n    print(\"Extract Images ID of each subdirectory and encoding vectors.. \")\n    for (_,_,files) in os.walk(images_path,topdown=False):\n\n        l.append(files) # Append every subdir contents in a list\n\n    ll = sorted(l)\n    ll.pop(0)\n\n    l_len = 0\n    for key,d in zip(dict_imgs,ll):\n        l = l_len\n        l_len = len(d)\n        dict_imgs[key] = {key_file: None for key_file in sorted(d)}\n        for enc,dd in zip(encoding[l:l_len],sorted(d)):\n            ## Add encoding vectors of each subfolders to the associate key and subkey\n            dict_imgs[key][dd] = enc\n\n    return dict(sorted(dict_imgs.items())) # Because the order is not respected at the first sight, I do this to have a structure similar to the initial tree of the dir/subdir of the images","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"encoded_dataset_dict = encoding_dictionnary(images_path,encoding)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Dump the encoding & the nested dictionary  ","metadata":{}},{"cell_type":"code","source":"import pickle\n\n### Resnet Encoding dump\nencoding_file = open(\"H&M_IMAGES_ENCODING_Resnet18.pkl\", \"wb\")\n\npickle.dump(encoding, encoding_file)\n\nencoding_file.close()\n\n\n### Nested dictionary dump\ndict_file = open(\"H&M_IMAGES_ENCODING_dict.pkl\", \"wb\")\n\npickle.dump(encoded_dataset_dict, dict_file)\n\ndict_file.close()\n\n\na_file = open(\"H&M_IMAGES_ENCODING_Resnet18.pkl\", \"rb\")\n\noutput = pickle.load(a_file)\noutput.shape","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test ","metadata":{}},{"cell_type":"markdown","source":"I wrote this section to make sure that every item of each subkey is the same as the encoding ","metadata":{}},{"cell_type":"code","source":"x,y = next(iter(imgs_dataloader))\nx = x.to(device)\nencode_x = resnet(x)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check for correspondance\n# You can try it for any item\nencode_x[0].cpu().detach().numpy() == encoded_dataset_dict['010']['0108775015.jpg']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can use ``PCA`` or ``Kernel PCA`` or any kind of dimensionality reduction technique to compress more the feature vector of the Resnet18 encoding...","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}