{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Deep Learning for Painter Classification\n\nWritten by: A. Kakampakos, C. Nikou, S. Rigas","metadata":{}},{"cell_type":"markdown","source":"### Part A: Basic Explorations & Data Loading\n\nThe dataset used is painter-by-numbers, obtained [here](https://www.kaggle.com/c/painter-by-numbers/data). The competition was designed for a pairwise comparison scheme in order to be applied in forgery recognition, however we are going to be using the data to design a classifier that performs painter recognition, for 20 different artists.\n\nThe data is available in the `train.zip` and `test.zip` folders and all the relevant information is located in `all_data_info.csv`. Since we are not designing a model for this specific competition, we may retrieve all of the data (both train and test) and then decide what to do with them. Let us first load the `all_data_info.csv` file in order to investigate it further.","metadata":{}},{"cell_type":"code","source":"# All the relevant modules and libraries\nimport itertools\nimport zipfile\nimport re\n\nimport numpy as np\nimport pandas as pd\n\nimport seaborn as sns\nimport matplotlib\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.axes_grid1 import make_axes_locatable\nfrom matplotlib.colors import ListedColormap, LinearSegmentedColormap\n\nimport cv2\nfrom PIL import Image\nfrom skimage.feature import hog\n\nfrom sklearn import preprocessing\nfrom sklearn import svm\nfrom sklearn.cluster import KMeans\nfrom sklearn.metrics import confusion_matrix\nfrom sklearn.metrics import classification_report\nfrom sklearn.model_selection import train_test_split\n\nimport torch\nimport torch.nn as nn\nfrom torch.utils.data import Dataset\nfrom torch.utils.data import SubsetRandomSampler, DataLoader\nfrom torchvision import transforms, models\n\npath = '../input/painter-by-numbers/'\n\ndf = pd.read_csv(path+'all_data_info.csv')\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:50:54.630797Z","iopub.execute_input":"2022-03-27T15:50:54.631391Z","iopub.status.idle":"2022-03-27T15:50:54.920509Z","shell.execute_reply.started":"2022-03-27T15:50:54.631353Z","shell.execute_reply":"2022-03-27T15:50:54.919789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some basic exploratory tasks follow. Our task is to find data for 20 artists, so we will need 20 artists for whom there is enough data for training. Furthermore, we need a balanced dataset, so we will have to base our sample-per-painter size on the artists with the least paintings.","metadata":{}},{"cell_type":"code","source":"print(f\"The full dataset contains a total of {len(df['artist'].unique())} different artists and {len(df['genre'].unique())} unique painting genres.\\n\")\nash = 5\nprint(f\"The {ash} artists with the most paintings available in the dataset are:\")\ndf['artist'].value_counts().head(ash)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:50:56.8948Z","iopub.execute_input":"2022-03-27T15:50:56.89548Z","iopub.status.idle":"2022-03-27T15:50:56.939492Z","shell.execute_reply.started":"2022-03-27T15:50:56.895444Z","shell.execute_reply":"2022-03-27T15:50:56.938779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It appears that the most paintings per artist available are 500. Before proceeding, we need to remove the corrupted datafiles from the dataset.","metadata":{}},{"cell_type":"code","source":"# This cell creates a list of the corrupted files' IDs, in order to ensure\n# that we don't use any of them in our dataset.\nfile_path = '../input/painter-by-numbers/'\n\narchive = zipfile.ZipFile(file_path+'replacements_for_corrupted_files.zip', 'r')\ncorrupted_ids = set()\n\nfor item in archive.namelist():\n    ID = re.sub(\"[^0-9]\", \"\", item)\n    if ID != \"\":\n        corrupted_ids.add(ID)\n\ndrop_idx = []\nfor index, row in df.iterrows():\n    id_check = re.sub(\"[^0-9]\", \"\", row['new_filename'])\n    if id_check in corrupted_ids:\n        drop_idx.append(index)\n        \ndf = df.drop(drop_idx)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:50:58.63679Z","iopub.execute_input":"2022-03-27T15:50:58.637696Z","iopub.status.idle":"2022-03-27T15:51:03.416909Z","shell.execute_reply.started":"2022-03-27T15:50:58.637651Z","shell.execute_reply":"2022-03-27T15:51:03.416078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we may construct a list of artists we're interested in classifying and check:\n1. If they have available work in this dataset\n2. If the corresponding sample is large enough for training and testing","metadata":{}},{"cell_type":"code","source":"painter_dict = {'Kandinsky':'','Dali':'','Picasso':'','Delacroix':'','Rembrandt':'','Gogh':'',\n               'Kuniyoshi':'','Dore':'','Steinlen':'','Saryan':'','Goya':'','Lautrec':'',\n               'Modigliani':'','Beksinski':'','Pissarro':'','Kirchner':'','Renoir':'','Piranesi':'',\n               'Degas':'','Chagall':''}\n\npaintings_dict = painter_dict.copy()\n\n# Find the correspondence between our names and the names in the dataframe\n# Also count the number of paintings\nfor artist in painter_dict:\n    for painter in df['artist']:\n        if artist in painter:\n            painter_dict[artist] = painter\n            paintings = df[df['artist'] == painter].shape[0]\n            paintings_dict[artist] = paintings\n            break\n\nfor artist in painter_dict:\n    print(f'The artist named {painter_dict[artist]} has a total of {paintings_dict[artist]} paintings in this dataset.')\n\nsample_size = min(paintings_dict.values())\nmin_a = list(paintings_dict.keys())[list(paintings_dict.values()).index(sample_size)]\nprint(f'\\nThe artist with the smallest number of paintings is {min_a} with {sample_size} paintings.')","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:03.418616Z","iopub.execute_input":"2022-03-27T15:51:03.418881Z","iopub.status.idle":"2022-03-27T15:51:03.728695Z","shell.execute_reply.started":"2022-03-27T15:51:03.418828Z","shell.execute_reply":"2022-03-27T15:51:03.7279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It therefore seems that, if we're not interested in generating new paintings from the existing ones, we will be using a sub-dataset containing 214 paintings per artist (so that it is balanced), with a total of 4280 paintings.\n\nNote that, alternatively, one may decide to keep the 20 artists with the most paintings, instead of handpicking them, even though the latter is more fun. For example, Kandinsky is the third writer's favourite painter, so this is why the dataset is limited to 214 samples per artist. However, this also helps speed up the training process, since it is a good enough sample number. In the *boring* case, the following code suffices:\n\n```\npaintings = df['artist'].value_counts().head(20)\nartists = paintings.index.tolist()\nsample_size = min(paintings)\n```\n\nand then, in the next cell, use `for artist in artists:`.","metadata":{}},{"cell_type":"code","source":"active_df = pd.DataFrame({}) # Reduce the large dataframe into the one containing only relevant data\n\n#for artist in artists: # <- use this in the boring case\nfor artist in painter_dict.values(): # <- use this in the non boring case\n    # small size = not many pixels = good (the reduction won't be as bad (?))\n    # if the opposite is true, set ascending = [False,False]\n    # or remove the contraint alltogether\n    #tr_df = df[(df['artist']==artist)].sort_values(by=['in_train'], ascending=[False])\n    tr_df = df[(df['artist']==artist)].sort_values(by=['in_train','size_bytes'], ascending=[False, True])\n    active_df = pd.concat([active_df,tr_df.iloc[:sample_size]])","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:03.732345Z","iopub.execute_input":"2022-03-27T15:51:03.733992Z","iopub.status.idle":"2022-03-27T15:51:04.202761Z","shell.execute_reply.started":"2022-03-27T15:51:03.733947Z","shell.execute_reply":"2022-03-27T15:51:04.202061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At the moment, `active_df` contains 4280 rows, where each row is a datum to be used for the model's training and evaluation. The next part of this preprocessing phase is to load only the relevant data, i.e. the elements of `active_df['new_filename']`, properly transform it and create the dataset that will be used.\n\nFirst, we create a LabelEncoder to transform artist names into integers.","metadata":{}},{"cell_type":"code","source":"artists = list(painter_dict.values())\n\n# Label Encoder to transform artist names into integers from 0 to 19\nLabEnc = preprocessing.LabelEncoder()\nLabEnc.fit(artists)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:04.813589Z","iopub.execute_input":"2022-03-27T15:51:04.816547Z","iopub.status.idle":"2022-03-27T15:51:04.827938Z","shell.execute_reply.started":"2022-03-27T15:51:04.81651Z","shell.execute_reply":"2022-03-27T15:51:04.827003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we create a transformer function that receives images, processes them and transforms them into tensors. Note that this transformer function is required for the training of \"classic\" classifiers as well, apart from CNNs. However, it's name ends with `_nn` simply because it corresponds to *all* the transformations required for the case of CNNs, while the classic classifiers require further modifications.","metadata":{}},{"cell_type":"code","source":"matplotlib.rc_file_defaults()\n\ndef image_transformer_nn(image, apply_norm=True, crop_img=True, new_dim=224):\n    \"\"\"\n    Args:\n        resize_num (int):\n            Dimension (pixels) to resize image\n        apply_norm (bool):\n            Choose whether to apply the normalization or not\n        crop_img (bool):\n            Choose whether to resize the image into the new_dim size, or crop\n            a square from its center, sized new_dim x new_dim\n    \"\"\"\n    if crop_img:\n        cropper = transforms.CenterCrop(new_dim)\n        image = cropper(image)\n    # Using transforms.Compose() is another option to perform these sequentially, but\n    # let's keep it like this until we find the \"final\" transformations sequence\n    tensoring = transforms.ToTensor()\n    image = tensoring(image) # shape is now (channels, height, width), see next line\n    channels, height, width = image.shape\n    \n    # This check was added because some images are automatically loaded as grayscale\n    if image.shape[0] < 3:\n        image = image.expand(3, -1, -1)\n    # This check is for images like 18807 that have extra channels with zero information\n    if image.shape[0] > 3:\n        image = image[0:3,:,:]\n    \n    # This is the imagenet normalizer, maybe define our own?\n    if apply_norm:\n        normalizer = transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))\n        image = normalizer(image)\n    \n    if not crop_img:\n        if width < height:\n            # Convolutions are invariant to rotations, so we choose to pad everything\n            # \"down\". This means that for landscape images (width > height) no rotation\n            # needs to be performed we just pad \"down\". For vertical images (width < height),\n            # in order to perform the \"down\" padding we have to rotate them first.\n            image = image.transpose(1,2)\n        channels, height, width = image.shape\n        res_percent = float(new_dim/width) # done to keep aspect ratio, width is max dim\n        height = round(height*res_percent)\n        resizer = transforms.Resize((height,new_dim))\n        image = resizer(image)\n        # Now that the image is resized by keeping aspect ratio, we pad \"down\"\n        padder = transforms.Pad([0,0,0,int(new_dim-height)])\n        image = padder(image)\n        \n    return image\n\n# Example:\narchive = zipfile.ZipFile(file_path+'train.zip', 'r')\nimg_path = 'train/'\nimgdata = archive.open(img_path+'69382.jpg')\nimage = Image.open(imgdata)\n\nprint(\"This is the original image:\\n\")\nplt.imshow(image)\nplt.show()\n\nprint(\"This is the cropped part of the transformed image:\\n\")\nimage2 = image_transformer_nn(image, apply_norm=False, crop_img=True, new_dim=224)\nplt.imshow(image2.numpy().transpose(1,2,0))\nplt.show()\n\nprint(\"This is the transformed, resized image:\\n\")\nimage3a = image_transformer_nn(image, apply_norm=False, crop_img=False, new_dim=224)\nplt.imshow(image3a.numpy().transpose(1,2,0))\nplt.show()\n\nprint(\"Transformed and resized, but with normalization as well:\\n\")\nimage3b = image_transformer_nn(image, apply_norm=True, crop_img=False, new_dim=224)\nplt.imshow(image3b.numpy().transpose(1,2,0))\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:06.031149Z","iopub.execute_input":"2022-03-27T15:51:06.031951Z","iopub.status.idle":"2022-03-27T15:51:08.291316Z","shell.execute_reply.started":"2022-03-27T15:51:06.031915Z","shell.execute_reply":"2022-03-27T15:51:08.28937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Two options are given: either perform resizing by respecting aspect ratio and then pad the data to apply the convolutional filters, or avoid padding alltogether and simply resize into squares via cropping square parts of the image. The former is usually preferred.\n\nThe next part corresponds to the collection of the data. This collection is done separately for data used for CNNs and data used for other classifiers.\n\nThe class seen below is a Dataset type class, that constructs the Dataset to be used for the CNNs training and evaluation.","metadata":{}},{"cell_type":"code","source":"# Creating one such class object currently requires 5-6 minutes of runtime.\n\nimport torch\nfrom torch.utils.data import Dataset\n\n# For Neural Networks\nclass ImageDataset(Dataset):\n    def __init__(self, path, dataframe, lab_encoder, img_size=224, normalize=True, crop=False):\n        \"\"\"\n        Args:\n            path (string):\n                Where to look for the files to extract\n            dataframe (pd.DataFrame):\n                dataframe to use for the IDs\n            lab_encoder:\n                label encoder to transform artist names into integers\n            img_size (int):\n                size to be used\n            normalize (bool):\n                perform normalization during transformation or not\n            crop (bool):\n                True: crop only a center from the image\n                False: Resize image with respect to aspect ratio and pad\n        \"\"\"\n        self.encoder = lab_encoder\n        self.img_size = img_size\n        self.normalize = normalize\n        self.crop = crop\n        self.feats, self.labels = self.get_all_items(path,dataframe)\n        \n    def get_all_items(self,path,dataframe):\n\n        # We begin with the train.zip\n        curr_df = dataframe[dataframe['in_train']==True]\n        archive = zipfile.ZipFile(path+'train.zip', 'r')\n        img_path = 'train/'\n\n        feats = []\n        labels = []\n\n        for index, row in curr_df.iterrows():\n            # Features\n            file = row['new_filename']\n            imgdata = archive.open(img_path+file)\n            try:\n                image = Image.open(imgdata)\n                datum = image_transformer_nn(image, apply_norm=self.normalize, \n                                          crop_img=self.crop, new_dim=self.img_size)\n                feats.append(datum)\n\n                # Label\n                artist = row['artist']\n                label = self.encoder.transform([artist])[0]\n                labels.append(label)\n            except Image.DecompressionBombError:\n                print(f\"Skipped loading image {file} to avoid a DecompressionBombError.\")\n\n        # Same for the test.zip\n        curr_df = dataframe[dataframe['in_train']==False]\n        archive = zipfile.ZipFile(path+'test.zip', 'r')\n        img_path = 'test/'\n\n        for index, row in curr_df.iterrows():\n            # Features\n            file = row['new_filename']\n            imgdata = archive.open(img_path+file)\n            try:\n                image = Image.open(imgdata)\n                datum = image_transformer_nn(image, apply_norm=self.normalize, \n                                          crop_img=self.crop, new_dim=self.img_size)\n                feats.append(datum)\n\n                # Label\n                artist = row['artist']\n                label = self.encoder.transform([artist])[0]\n                labels.append(label)\n            except Image.DecompressionBombError:\n                print(f\"Skipped loading image {file} to avoid a DecompressionBombError.\")\n\n        feats = torch.stack(feats)\n        labels = torch.LongTensor(labels)\n        return feats, labels\n\n    def __len__(self):\n        return len(self.labels)\n\n    def __getitem__(self, item):\n        return self.feats[item], self.labels[item]","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:08.293054Z","iopub.execute_input":"2022-03-27T15:51:08.293291Z","iopub.status.idle":"2022-03-27T15:51:08.309303Z","shell.execute_reply.started":"2022-03-27T15:51:08.29326Z","shell.execute_reply":"2022-03-27T15:51:08.308413Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On the other hand, the following function creates a tuple of the type (X, y) with data to be used in classic classifiers. The features used here are not the pictures themselves (unlike in the case of CNNs), but HOG or SIFT type features (these are two built-in options, however they can be expanded).","metadata":{}},{"cell_type":"code","source":"# For other models\ndef ImageData(path, dataframe, lab_encoder, hog_mode, sift_mode, img_size=224):\n\n    # We begin with the train.zip\n    curr_df = dataframe[dataframe['in_train']==True]\n    archive = zipfile.ZipFile(path+'train.zip', 'r')\n    img_path = 'train/'\n\n    PaintFeats = []\n    PaintLabels = []\n\n    for index, row in curr_df.iterrows():\n        # Features\n        file = row['new_filename']\n        imgdata = archive.open(img_path+file)\n        try:\n            image = Image.open(imgdata)\n            datum = image_transformer_nn(image, apply_norm=False, crop_img=False, new_dim=img_size)\n            \n            # Up to that point, the output was the same as in the NN case\n            np_datum = datum.numpy().transpose(1,2,0)\n            \n            if hog_mode:\n                orients, ppc, cpb = hog_mode[0], hog_mode[1], hog_mode[2]\n                # orients = number of grad orientations, default = 9\n                # ppc = pixels per cell, default = (8,8)\n                # cpb = cells per block, default = (2,2)\n                datum = hog(np_datum, orientations=orients, pixels_per_cell=ppc,\n                            cells_per_block=cpb, feature_vector=True, channel_axis=2)\n                PaintFeats.append(datum)\n                \n            elif sift_mode:\n                np_datum = cv2.normalize(np_datum, None, 0, 255, cv2.NORM_MINMAX).astype('uint8')\n                imgtogray = cv2.cvtColor(np_datum, cv2.COLOR_BGR2GRAY)\n                PaintFeats.append(imgtogray)\n    \n            # Label\n            artist = row['artist']\n            label = lab_encoder.transform([artist])[0]\n            PaintLabels.append(label)\n        except Image.DecompressionBombError:\n            print(f\"Skipped loading image {file} to avoid a DecompressionBombError.\")\n\n    # Same for the test.zip\n    curr_df = dataframe[dataframe['in_train']==False]\n    archive = zipfile.ZipFile(path+'test.zip', 'r')\n    img_path = 'test/'\n\n    for index, row in curr_df.iterrows():\n        # Features\n        file = row['new_filename']\n        imgdata = archive.open(img_path+file)\n        try:\n            image = Image.open(imgdata)\n            datum = image_transformer_nn(image, apply_norm=False, crop_img=False, new_dim=img_size)\n            \n            # Up to that point, the output was the same as in the NN case\n            np_datum = datum.numpy().transpose(1,2,0)\n            \n            if hog_mode:\n                orients, ppc, cpb = hog_mode[0], hog_mode[1], hog_mode[2]\n                # orients = number of grad orientations, default = 9\n                # ppc = pixels per cell, default = (8,8)\n                # cpb = cells per block, default = (2,2)\n                datum = hog(np_datum, orientations=orients, pixels_per_cell=ppc,\n                            cells_per_block=cpb, feature_vector=True, channel_axis=2)\n                PaintFeats.append(datum)\n                \n            elif sift_mode:\n                np_datum = cv2.normalize(np_datum, None, 0, 255, cv2.NORM_MINMAX).astype('uint8')\n                imgtogray = cv2.cvtColor(np_datum, cv2.COLOR_BGR2GRAY)\n                PaintFeats.append(imgtogray)\n            \n            # Label\n            artist = row['artist']\n            label = lab_encoder.transform([artist])[0]\n            PaintLabels.append(label) # datum is a tensor by default\n        except Image.DecompressionBombError:\n            print(f\"Skipped loading image {file} to avoid a DecompressionBombError.\")\n    \n    return np.asarray(PaintFeats), np.asarray(PaintLabels)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:09.000108Z","iopub.execute_input":"2022-03-27T15:51:09.000391Z","iopub.status.idle":"2022-03-27T15:51:09.021725Z","shell.execute_reply.started":"2022-03-27T15:51:09.00036Z","shell.execute_reply":"2022-03-27T15:51:09.020815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At this point, the data can be loaded either in the form of a pytorch Dataset via\n\n```\nnn_data = ImageDataset(file_path, active_df, LabEnc, img_size=224, resize_aspect=False)\n```\n\nfor example, or in the form of two numpy arrays (the traditional X and y), via\n\n```\nX, y = ImageData(path, dataframe, lab_encoder, hog_mode, sift_mode, img_size=224)\n```\n\nWhat is left to conclude this first part is the construction of proper splits for the data and the corresponding DataLoaders in the case of NN models. This is done through the function defined below, which is common for both types of data.","metadata":{}},{"cell_type":"code","source":"# This cell splits data into train-val-test and creates DataLoaders in the case of NNs\ndef DataSplitter(data, ratios=[60,20,20], need_val=True, batches=None, shuffle=True, seed=None):\n    \"\"\"\n    Args:\n        data (Dataset or List):\n            In the case of NN, it's the dataset to be loaded into loaders. In the case\n            of other models, it's a list containing the Features and Labels lists\n        batches (int):\n            batch size for loaders in case of NN\n        ratios (list):\n            list of integers, containing the ratios [train,val,test] for splitting\n            example: [60,25,15] means 60% train data, 25% val data and 15% test data\n        shuffle (bool):\n            option to shuffle data\n        seed (None or int):\n            seed for shuffling\n    \"\"\"\n    first_ratio = (ratios[1]+ratios[2])/sum(ratios)\n    second_ratio = ratios[2]/(ratios[1]+ratios[2])\n    \n    if isinstance(data,ImageDataset): # NN case\n        \n        labels = data.labels.numpy()\n        \n        train_indices, rest_indices = train_test_split(np.arange(len(labels)),\n                                               test_size=first_ratio, shuffle=shuffle,\n                                               random_state=seed, stratify=labels)\n        \n        rest_labels = data[rest_indices][1]\n        \n        val_indices, test_indices = train_test_split(rest_indices,\n                                            test_size=second_ratio, shuffle=shuffle,\n                                            random_state=seed, stratify=rest_labels)\n\n        train_sampler = SubsetRandomSampler(train_indices)\n        val_sampler = SubsetRandomSampler(val_indices)\n        test_sampler = SubsetRandomSampler(test_indices)\n\n        train_loader = DataLoader(data,batch_size=batches,sampler=train_sampler)\n        val_loader = DataLoader(data,batch_size=batches,sampler=val_sampler)\n        test_loader = DataLoader(data,batch_size=batches,sampler=test_sampler)\n        \n        return train_loader, val_loader, test_loader\n        \n    elif isinstance(data,tuple): # other model case\n        X_train, X_rest, y_train, y_rest = train_test_split(data[0], data[1],\n                                                            test_size=first_ratio,\n                                                            shuffle=shuffle,\n                                                            random_state=seed,\n                                                            stratify=data[1])\n        if need_val:\n            X_val, X_test, y_val, y_test = train_test_split(X_rest, y_rest,\n                                                            test_size=second_ratio,\n                                                            shuffle=shuffle, random_state=seed,\n                                                            stratify=y_rest)\n        \n            return X_train, X_val, X_test, y_train, y_val, y_test\n        return X_train, X_rest, y_train, y_rest\n    else:\n        print('Invalid data Type. Either insert Dataset to create DataLoaders or a list of two numpy arrays to perform the splitting.')\n        return","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:10.293417Z","iopub.execute_input":"2022-03-27T15:51:10.293668Z","iopub.status.idle":"2022-03-27T15:51:10.306028Z","shell.execute_reply.started":"2022-03-27T15:51:10.293639Z","shell.execute_reply":"2022-03-27T15:51:10.304765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Auxiliary Functions\n\nWe leave some auxiliary functions here, for uniformity in graphs.","metadata":{}},{"cell_type":"code","source":"sns.set(style = \"darkgrid\") # Personal preference\n\ndef CustomCmap(from_rgb,to_rgb):\n\n    # from color r,g,b\n    r1,g1,b1 = from_rgb\n\n    # to color r,g,b\n    r2,g2,b2 = to_rgb\n\n    cdict = {'red': ((0, r1, r1),\n                   (1, r2, r2)),\n           'green': ((0, g1, g1),\n                    (1, g2, g2)),\n           'blue': ((0, b1, b1),\n                   (1, b2, b2))}\n\n    cmap = LinearSegmentedColormap('custom_cmap', cdict)\n    return cmap\n\nmycmap = CustomCmap([1.0, 1.0, 1.0], [72/255, 99/255, 147/255])\nmycmap_r = CustomCmap([72/255, 99/255, 147/255], [1.0, 1.0, 1.0])\n\nmycol = (72/255, 99/255, 147/255)\nmycomplcol = (129/255, 143/255, 163/255)\n\ndef plot_cm(cfmatrix,title,classes):\n    fig, ax1 = plt.subplots(1,1) #, figsize=(5,5)\n\n    for ax,cm in zip([ax1],[cfmatrix]):\n        im = ax.imshow(cm, interpolation='nearest', cmap=mycmap)\n        divider = make_axes_locatable(ax)\n        cax = divider.append_axes(\"right\", size=\"5%\", pad=.2)\n        plt.colorbar(im, cax=cax) #, ticks=[-1,-0.5,0,0.5,1]\n        ax.set_title(title,fontsize=14)\n        tick_marks = np.arange(len(classes))\n        ax.set_xticks(tick_marks)\n        ax.set_xticklabels(classes, rotation=90)\n        ax.set_yticks(tick_marks)\n        ax.set_yticklabels(classes)\n\n        fmt = 'd'\n        thresh = cm.max() / 2.\n\n        for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n            ax.text(j, i, format(cm[i, j], fmt), horizontalalignment=\"center\", color=\"white\" if cm[i, j] > thresh else \"black\")\n\n        ax.set_ylabel('True label',fontsize=14)\n        ax.set_xlabel('Predicted label',fontsize=14)\n\n    plt.savefig(title+'.pdf', bbox_inches='tight')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:11.343597Z","iopub.execute_input":"2022-03-27T15:51:11.34387Z","iopub.status.idle":"2022-03-27T15:51:11.360254Z","shell.execute_reply.started":"2022-03-27T15:51:11.343827Z","shell.execute_reply":"2022-03-27T15:51:11.358184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Part B: Neural Networks Training\n\nWith the DataLoaders at hand, we may proceed to the NN training process. A first attempt is to define a new CNN Class and try to evaluate its performance on the data.","metadata":{}},{"cell_type":"code","source":"%%time\n# Load the Dataset for NN training\nnn_data = ImageDataset(file_path, active_df, LabEnc, img_size=224, normalize=True, crop=False)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:51:12.705432Z","iopub.execute_input":"2022-03-27T15:51:12.706183Z","iopub.status.idle":"2022-03-27T15:55:58.907266Z","shell.execute_reply.started":"2022-03-27T15:51:12.706143Z","shell.execute_reply":"2022-03-27T15:55:58.90651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create Data Loaders for NN training\ntrain_loader, val_loader, test_loader = DataSplitter(nn_data, ratios=[60,25,15], need_val=True, batches=32, shuffle=True, seed=420)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:55:58.908789Z","iopub.execute_input":"2022-03-27T15:55:58.909053Z","iopub.status.idle":"2022-03-27T15:55:59.587393Z","shell.execute_reply.started":"2022-03-27T15:55:58.909018Z","shell.execute_reply":"2022-03-27T15:55:59.586665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CNNBackbone(nn.Module):\n    def __init__(self, input_height, input_width, conv_channels, kernels, maxpools, lin_channels, dropout, batchnorm):\n        \"\"\"\n        Agrs:\n            input_height (int):\n                image height in pixels\n            input_width (int):\n                image width in pixels\n            conv_channels (list):\n                contains the input and output channels for each\n                convolutional layer, therefore using a total of\n                len(channels)-1 convolutional layers\n            kernels (list):\n                contains the kernel sizes to be considered per\n                convolution. Must have length len(channels)-1\n            maxpools (list):\n                contains the MaxPool2d kernel sizes to be considered\n                per convolution. Must have length len(channels)-1\n            lin_channels (list):\n                contains the output channels for each linear layer\n                following the convolutions, therefore using a total of\n                len(lin_channels) linear layers.\n                Note that the last element must be equal to the number\n                of classes to be determined.\n            classes (int):\n                number of output features\n            dropout (float):\n                dropout probability, 0 <= dropout <= 1\n            batchnorm (bool):\n                boolean parameter to control whether batch normalization\n                is applied or not.\n        \"\"\"\n        super(CNNBackbone, self).__init__()\n        self.num_conv_layers = len(kernels)\n        self.batchnorm = batchnorm\n        \n        seq = []\n        for i in range(self.num_conv_layers):\n            seq.append(nn.Conv2d(in_channels=conv_channels[i], \n                                 out_channels=conv_channels[i+1],\n                                 kernel_size=kernels[i], stride=1, padding=1))\n            seq.append(nn.ReLU())\n            if self.batchnorm:\n                seq.append(nn.BatchNorm2d(num_features=conv_channels[i+1],track_running_stats=False))\n            seq.append(nn.MaxPool2d(kernel_size=maxpools[i]))\n            \n        # Flatten the output of the final convolution layer\n        seq.append(nn.Flatten())\n        \n        convolutions = nn.Sequential(*seq)\n        \n        # Calculation of first linear layer dimensions\n        # We build an empty tensor of appropriate size and let him go through\n        # the above sequence, in order to calculate the output's size automatically\n        first_lin = convolutions(torch.empty(1,conv_channels[0],input_height,input_width)).size(-1)\n        \n        self.num_lin_layers = len(lin_channels)\n        for i in range(self.num_lin_layers):\n            if i == self.num_lin_layers-1:\n                seq.append(nn.Linear(lin_channels[i-1], lin_channels[i]))\n                break\n            elif i == 0:\n                seq.append(nn.Linear(first_lin, lin_channels[i]))\n            else:\n                seq.append(nn.Linear(lin_channels[i-1], lin_channels[i]))\n            seq.append(nn.ReLU())\n            seq.append(nn.Dropout(dropout))\n        seq.append(nn.Softmax(1))\n                \n        self.fitter = nn.Sequential(*seq)\n\n    def forward(self, x):\n        \"\"\"CNN forward\n        Args:\n            x (torch.Tensor):\n                [B, S, F] Batch size x sequence length x feature size\n                padded inputs\n        Returns:\n            torch.Tensor: [B, O] Batch size x CNN output size cnn outputs\n        \"\"\"\n        out = self.fitter(x)\n        return out\n    \ndef load_backbone_from_checkpoint(model, checkpoint_path):\n    model.load_state_dict(torch.load(checkpoint_path))\n    \n# adapted code from this repository: https://github.com/Bjarten/early-stopping-pytorch\nclass EarlyStopping:\n    def __init__(self, patience=7, verbose=False, delta=0, path='checkpoint.pt', trace_func=print):\n        self.patience = patience\n        self.verbose = verbose\n        self.counter = 0\n        self.best_score = None\n        self.early_stop = False\n        self.val_loss_min = np.Inf\n        self.delta = delta\n        self.path = path\n        self.trace_func = trace_func\n    def __call__(self, val_loss, model):\n\n        score = -val_loss\n\n        if self.best_score is None:\n            self.best_score = score\n            self.save_checkpoint(val_loss, model)\n        elif score < self.best_score + self.delta:\n            self.counter += 1\n            self.trace_func(f'Validation loss increase spotted. Early stopping counter: {self.counter} out of {self.patience}')\n            if self.counter >= self.patience:\n                self.early_stop = True\n        else:\n            self.best_score = score\n            self.save_checkpoint(val_loss, model)\n            self.counter = 0\n\n    def save_checkpoint(self, val_loss, model):\n        '''Saves model when validation loss decrease.'''\n        if self.verbose:\n            self.trace_func(f'Validation loss decreased ({self.val_loss_min:.6f} --> {val_loss:.6f}).  Saving model ...')\n        torch.save(model.state_dict(), self.path)\n        self.val_loss_min = val_loss\n\ndef training_loop(model, train_dataloader, optimizer, device=\"cuda\"):\n    model.train()\n    batch_losses = []\n            \n    for batch in train_dataloader:\n        x_batch, y_batch = batch\n                \n        # Move to device\n        x_batch, y_batch = x_batch.to(device), y_batch.to(device)\n                \n        # Clear the previous gradients first\n        optimizer.zero_grad()\n        \n        # forward pass\n        yhat = model(x_batch) # No unpacking occurs in CNNs\n        \n        # loss calculation\n        loss = loss_function(yhat, y_batch)\n        \n        # Backward pass\n        loss.backward()\n        \n        # Update weights\n        optimizer.step()\n        \n        batch_losses.append(loss.data.item())\n        \n    train_loss = np.mean(batch_losses)\n\n    return train_loss\n\n\ndef validation_loop(model, val_dataloader, device=\"cuda\"):\n    \n    model.eval()\n    batch_losses = []\n    \n    for batch in val_dataloader:\n        x_batch, y_batch = batch\n                \n        # Move to device\n        x_batch, y_batch = x_batch.to(device), y_batch.to(device)\n        \n        yhat = model(x_batch) # No unpacking occurs in CNNs\n        \n        loss = loss_function(yhat, y_batch)\n        \n        batch_losses.append(loss.data.item())\n        \n    val_loss = np.mean(batch_losses)\n\n    return val_loss # Return validation_loss and anything else you need\n\n\ndef train(model, train_dataloader, val_dataloader, optimizer, epochs, device=\"cuda\", patience=-1, verbose_ct=100):\n\n    train_losses = []\n    val_losses = []\n    print(f\"Initiating CNN training.\")\n    model_path = f'CNN.pt'\n    checkpoint_path = 'checkpoint.pt'\n        \n    if patience != -1:\n        early_stopping = EarlyStopping(patience=patience, verbose=False, path=checkpoint_path)\n\n    for epoch in range(epochs):\n        \n        # Training loop\n        train_loss = training_loop(model, train_dataloader, optimizer, device)    \n        train_losses.append(train_loss)\n\n        # Validation loop\n        with torch.no_grad():\n\n            val_loss = validation_loop(model, val_dataloader, device)\n            val_losses.append(val_loss)\n\n        if patience != -1:\n            early_stopping(val_loss, model)\n\n            if early_stopping.early_stop:\n                print(\"Patience limit reached. Early stopping and going back to last checkpoint.\")\n                break\n\n        if epoch % verbose_ct == 0:        \n            print(f\"[{epoch+1}/{epochs}] Training loss: {train_loss:.4f}\\t Validation loss: {val_loss:.4f}.\")\n\n    if patience != -1 and early_stopping.early_stop == True:\n        load_backbone_from_checkpoint(model,checkpoint_path)        \n\n    torch.save(model.state_dict(), model_path)\n\n    print(f\"CNN training finished.\\n\")\n    \n    return train_losses, val_losses\n    \ndef evaluate(model, test_dataloader, device=\"cuda\"):\n    model.eval()\n    predictions = []\n    labels = []\n    \n    with torch.no_grad():\n        for batch in test_dataloader:\n            \n            x_batch, y_batch = batch\n                \n            # Move to device\n            x_batch, y_batch = x_batch.to(device), y_batch.to(device)\n            \n            yhat = model(x_batch) # No unpacking occurs in CNNs\n            \n            # Calculate the index of the maximum argument\n            yhat_idx = torch.argmax(yhat, dim=1)\n            \n            predictions.append(yhat_idx.cpu().numpy())\n            labels.append(y_batch.cpu().numpy())\n    \n    return predictions, labels  # Return the model predictions\n\n# Small code to plot losses after training\ndef plot_losses(train_losses,val_losses,title):\n    plt.plot(train_losses, label=\"Training loss\", color=mycol)\n    plt.plot(val_losses, label=\"Validation loss\", color=mycomplcol)\n    plt.legend(loc='best')\n    plt.ylabel('Mean Loss')\n    plt.xlabel('Epochs')\n    plt.title(f\"Loss graph during the process of training the CNN.\")\n    plt.savefig(title, bbox_inches='tight')\n    plt.show() \n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:55:59.588686Z","iopub.execute_input":"2022-03-27T15:55:59.588954Z","iopub.status.idle":"2022-03-27T15:55:59.678204Z","shell.execute_reply.started":"2022-03-27T15:55:59.588917Z","shell.execute_reply":"2022-03-27T15:55:59.677507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ninput_height = nn_data[0][0].shape[1]\ninput_width = nn_data[0][0].shape[2]\nconv_channels = [nn_data[0][0].shape[0],4,16,64,128]\nkernels = [3,3,3,3]\nmaxpools = [2,2,2,2]\nlin_channels = [256,128,20]\ndropout = 0.25\nlearning_rate = 0.00001\nweight_decay = 1e-6\npatience = 10\nverbose_ct = 1\n\nepochs = 2500\n\nmodel = CNNBackbone(input_height = input_height, input_width = input_width,\n                    conv_channels = conv_channels, kernels = kernels, maxpools = maxpools,\n                    lin_channels = lin_channels, dropout = dropout, batchnorm=True)\nmodel.to(device)\nloss_function = nn.CrossEntropyLoss()\noptimizer = torch.optim.Adam(model.parameters(), lr = learning_rate, weight_decay = weight_decay)\n#optimizer = torch.optim.SGD(model.parameters(), lr = 0.0001, momentum=0.9)\n\n# Train the model\nt_losses, v_losses = train(model, train_loader, val_loader, optimizer, epochs,\n                           device=device, patience=patience, verbose_ct = verbose_ct)\n\n# Plot the loss diagram\nplot_losses(t_losses, v_losses, 'CNN_Training_Loss.pdf')\n\n# Evaluate the model\npredictions, labels = evaluate(model, test_loader, device=device)\n\ny_true = np.concatenate(labels, axis=0)\ny_pred = np.concatenate(predictions, axis=0)\n\nprint(classification_report(y_true, y_pred))","metadata":{"execution":{"iopub.status.busy":"2022-03-27T15:56:17.43405Z","iopub.execute_input":"2022-03-27T15:56:17.434813Z","iopub.status.idle":"2022-03-27T16:01:41.123557Z","shell.execute_reply.started":"2022-03-27T15:56:17.434764Z","shell.execute_reply":"2022-03-27T16:01:41.122701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# A depiction of the Confusion Matrix\nmatplotlib.rc_file_defaults() # to remove the sns darkgrid style\ncfmatrix = confusion_matrix(y_true, y_pred)\nplot_cm(cfmatrix,'CNN Confusion Matrix',artists)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T16:17:13.038034Z","iopub.execute_input":"2022-03-02T16:17:13.038517Z","iopub.status.idle":"2022-03-02T16:17:16.48474Z","shell.execute_reply.started":"2022-03-02T16:17:13.038475Z","shell.execute_reply":"2022-03-02T16:17:16.484167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Part C: Transfer learning\n\nWhile the first attempt was a good one, we may also utilize transfer learning methods, by using powerful pre-trained CNNs from [here](https://pytorch.org/vision/stable/models.html).\n\n#### ResNet","metadata":{}},{"cell_type":"code","source":"%%time\nlearning_rate = 0.00005\nweight_decay = 1e-6\npatience = -1\nverbose_ct = 1\n\nepochs = 27\n\nmodel_conv = models.resnet18(pretrained=True)\n#for param in model_conv.parameters():\n#    param.requires_grad = False\n\nnum_ftrs = model_conv.fc.in_features\nmodel_conv.fc = nn.Linear(num_ftrs, 256)\nmodel_conv.fc2 = nn.Linear(256,20)\nmodel_conv.sfact = nn.Softmax(1)\n\nmodel_conv = model_conv.to(device)\n\nloss_function = nn.CrossEntropyLoss()\noptimizer = torch.optim.Adam(model_conv.parameters(), lr = learning_rate, weight_decay = weight_decay)\n\n# Train the model\nt_losses, v_losses = train(model_conv, train_loader, val_loader, optimizer, epochs,\n                           device=device, patience=patience, verbose_ct = verbose_ct)\n\nsns.set(style = \"darkgrid\")\n\n# Plot the loss diagram\nplot_losses(t_losses, v_losses, 'CNN_Training_Loss_transfer.pdf')\n\ntorch.save(model_conv.state_dict(), 'ResNet-Trained.pt')\n\n# Evaluate the model\npredictions, labels = evaluate(model_conv, test_loader, device=device)\n\ny_true = np.concatenate(labels, axis=0)\ny_pred = np.concatenate(predictions, axis=0)\n\nprint(classification_report(y_true, y_pred))\n\n# A depiction of the Confusion Matrix\nmatplotlib.rc_file_defaults() # to remove the sns darkgrid style\ncfmatrix = confusion_matrix(y_true, y_pred)\nplot_cm(cfmatrix,'CNN Confusion Matrix - Transfer',artists)","metadata":{"execution":{"iopub.status.busy":"2022-03-27T16:03:27.12553Z","iopub.execute_input":"2022-03-27T16:03:27.12817Z","iopub.status.idle":"2022-03-27T16:06:01.877377Z","shell.execute_reply.started":"2022-03-27T16:03:27.128125Z","shell.execute_reply":"2022-03-27T16:06:01.876744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Part D: Classic Classifiers\n\nWhile somewhat unorthodox, since we usually start with the less efficient classifiers, we close this study by training \"classic\" classifiers to identify painters. The classifiers used are:\n\n- An SVM Classifier trained on HOG features extracted from the images.\n\n- An SVM & k-Means hybrid trained on SIFT features extracted from the images in a Bag-of-Words approximation.","metadata":{}},{"cell_type":"markdown","source":"#### Case 1: HOG","metadata":{}},{"cell_type":"code","source":"%%time\n# Load the Dataset for classic training - HOG\nhog_data = ImageData(file_path, active_df, LabEnc, hog_mode=[9, (8,8), (2,2)], sift_mode=False, img_size=224)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T12:48:17.60521Z","iopub.execute_input":"2022-03-02T12:48:17.605679Z","iopub.status.idle":"2022-03-02T12:54:30.939742Z","shell.execute_reply.started":"2022-03-02T12:48:17.605646Z","shell.execute_reply":"2022-03-02T12:54:30.938702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the data - no validation set needed, unless we want to perform hyperparameter tuning\nX_train, X_test, y_train, y_test = DataSplitter(hog_data, ratios=[85,5,10], need_val=False, batches=32, shuffle=True, seed=420)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T12:54:30.941901Z","iopub.execute_input":"2022-03-02T12:54:30.942821Z","iopub.status.idle":"2022-03-02T12:54:31.187798Z","shell.execute_reply.started":"2022-03-02T12:54:30.942775Z","shell.execute_reply":"2022-03-02T12:54:31.186844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Train the SVM on the hog features\nhog_classifier = svm.SVC(kernel='rbf', gamma=1.5, C=0.3)\nhog_classifier.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T12:54:31.189104Z","iopub.execute_input":"2022-03-02T12:54:31.189454Z","iopub.status.idle":"2022-03-02T13:12:52.992669Z","shell.execute_reply.started":"2022-03-02T12:54:31.189412Z","shell.execute_reply":"2022-03-02T13:12:52.991776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Draw predictions and evaluate the HOG classifier\ny_pred = hog_classifier.predict(X_test)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T13:12:52.995006Z","iopub.execute_input":"2022-03-02T13:12:52.995415Z","iopub.status.idle":"2022-03-02T13:14:34.658018Z","shell.execute_reply.started":"2022-03-02T13:12:52.995364Z","shell.execute_reply":"2022-03-02T13:14:34.65701Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"matplotlib.rc_file_defaults() # to remove the sns darkgrid style\nprint(classification_report(y_test, y_pred))\n\n# A depiction of the Confusion Matrix\ncfmatrix = confusion_matrix(y_test, y_pred)\nplot_cm(cfmatrix,'HOG - SVM Confusion Matrix',artists)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T15:48:18.632832Z","iopub.execute_input":"2022-03-02T15:48:18.633134Z","iopub.status.idle":"2022-03-02T15:48:21.948244Z","shell.execute_reply.started":"2022-03-02T15:48:18.633101Z","shell.execute_reply":"2022-03-02T15:48:21.944855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Case 2: SIFT","metadata":{}},{"cell_type":"code","source":"%%time\n# Load the Dataset for classic training\nsift_data = ImageData(file_path, active_df, LabEnc, hog_mode=False, sift_mode=True, img_size=224)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T13:15:37.357503Z","iopub.execute_input":"2022-03-02T13:15:37.358144Z","iopub.status.idle":"2022-03-02T13:19:29.734254Z","shell.execute_reply.started":"2022-03-02T13:15:37.358089Z","shell.execute_reply":"2022-03-02T13:19:29.731949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Split the data - no validation set needed, unless we want to perform hyperparameter tuning\nX_train_s, X_test_s, y_train_s, y_test_s = DataSplitter(sift_data, ratios=[85,5,10], need_val=False, batches=32, shuffle=True, seed=420)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T13:19:29.736639Z","iopub.execute_input":"2022-03-02T13:19:29.737186Z","iopub.status.idle":"2022-03-02T13:19:29.816839Z","shell.execute_reply.started":"2022-03-02T13:19:29.737136Z","shell.execute_reply":"2022-03-02T13:19:29.815849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As of this point we have the full dataset in the form of grayscale images from the SIFT procedure and the data split has already been performed. The pipeline to create a classifier is the following:\n\n- For each image in the train set, we acquire all of its descriptors ((X,128) array per image) and then stack all the descriptors together. This leads to a lengthy list of features (descriptors).\n\n- Due to the large number of features (descriptors) we perform k-Means clustering on the points created on the 128-dimensional space to reduce the dimension of our features to $k$, where $k$ is the number of clusters to be considered. It's common practice to use $k$ = `unique_labels` x `a`, where a is an integer (for example a = 10).\n\n- Having acquired the cluster's centroids, we re-visit the descriptors of every image. For every image, we calculate a histogram, which is a (1,k) shaped array with contribution elements, $t_i$, where $0 < t_i < 1$, $i \\in \\{0,\\dots,k\\}$. These contribution elements are acquired by applying the k-Means classifier on the image's descriptors.\n\n- A classifier (for example an SVM) is trained using the constructed histograms as features. In order to classify new data, the process of extracting descriptors and classifying them using the k-Means classifier needs to be repeated.","metadata":{}},{"cell_type":"code","source":"%%time\n# Draw all descriptors\nDescriptors = []\nfor image in X_train_s:\n    sift = cv2.xfeatures2d.SIFT_create()\n    keypoints, descriptors = sift.detectAndCompute(image, None)\n    Descriptors.append(descriptors)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T13:19:29.818088Z","iopub.execute_input":"2022-03-02T13:19:29.818317Z","iopub.status.idle":"2022-03-02T13:20:22.278014Z","shell.execute_reply.started":"2022-03-02T13:19:29.818291Z","shell.execute_reply":"2022-03-02T13:20:22.277109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Perform the clustering\nfeatures = np.vstack(Descriptors) # List of features to perform the clustering\n\nk = 500 # Number of clusters\nkmeans = KMeans(n_clusters=k, random_state=0)\nkmeans.fit(features)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T13:20:22.279843Z","iopub.execute_input":"2022-03-02T13:20:22.280066Z","iopub.status.idle":"2022-03-02T15:30:35.657009Z","shell.execute_reply.started":"2022-03-02T13:20:22.280041Z","shell.execute_reply":"2022-03-02T15:30:35.65516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# We now have k centroids\nbag_of_histograms = []\n\nfor descriptor_list in Descriptors:\n    histogram = np.zeros(k)\n    desc_length = descriptor_list.shape[0]\n    \n    for desc in descriptor_list:\n        idx = kmeans.predict(np.array(desc).reshape(1, -1))\n        histogram[idx] += 1/desc_length # for normalization\n        \n    bag_of_histograms.append(histogram)\n\nX_train_new = np.asarray(bag_of_histograms)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T15:30:35.661326Z","iopub.execute_input":"2022-03-02T15:30:35.662207Z","iopub.status.idle":"2022-03-02T15:38:40.020982Z","shell.execute_reply.started":"2022-03-02T15:30:35.662153Z","shell.execute_reply":"2022-03-02T15:38:40.019819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Transform the test data so that they can be classified\ntest_histograms = []\n\nfor image in X_test_s:\n    sift = cv2.xfeatures2d.SIFT_create()\n    keypoints, descriptors = sift.detectAndCompute(image, None)\n    \n    histogram = np.zeros(k)\n    desc_length = descriptors.shape[0]\n    \n    for desc in descriptors:\n        idx = kmeans.predict(np.array(desc).reshape(1, -1))\n        histogram[idx] += 1/desc_length\n        \n    test_histograms.append(histogram)\n    \nX_test_new = np.asarray(test_histograms)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T15:38:40.02249Z","iopub.execute_input":"2022-03-02T15:38:40.022789Z","iopub.status.idle":"2022-03-02T15:40:14.558506Z","shell.execute_reply.started":"2022-03-02T15:38:40.022758Z","shell.execute_reply":"2022-03-02T15:40:14.556276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n# Train the classifier on the new X_train\nsift_classifier = svm.SVC(kernel='rbf', gamma=6.1, C=8.3)\nsift_classifier.fit(X_train_new, y_train_s)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T15:59:13.570335Z","iopub.execute_input":"2022-03-02T15:59:13.570806Z","iopub.status.idle":"2022-03-02T15:59:27.972298Z","shell.execute_reply.started":"2022-03-02T15:59:13.570758Z","shell.execute_reply":"2022-03-02T15:59:27.971167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Perform the classification and draw predictions\ny_pred_s = sift_classifier.predict(X_test_new)\n\nprint(classification_report(y_test_s, y_pred_s))\n\n# A depiction of the Confusion Matrix\ncfmatrix_s = confusion_matrix(y_test_s, y_pred_s)\nplot_cm(cfmatrix_s,'SIFT - SVM Confusion Matrix',artists)","metadata":{"execution":{"iopub.status.busy":"2022-03-02T15:59:27.974681Z","iopub.execute_input":"2022-03-02T15:59:27.975032Z","iopub.status.idle":"2022-03-02T15:59:33.114138Z","shell.execute_reply.started":"2022-03-02T15:59:27.974986Z","shell.execute_reply":"2022-03-02T15:59:33.113225Z"},"trusted":true},"execution_count":null,"outputs":[]}]}