{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Duplicate Image Prediction\n\nPlease visit the articile on my website for full write up. \n* https://www.jocampo.com/projects/2019/3/3/finding-duplicate-images-using-transfer-learning-on-deep-neural-networks\n\nConcept: \nA discussion with a co-worker after a meeting generated this idea. He mentioned it would be nice if there was an easy way to identify duplicate images with a folder containing many image files. Seemed like a greate idea for a project. I think of this in 2 ways: \n1. Identifying similar/duplicate images \n2. The user interface to execute this application. \nI'm going to primarily focus on the former for now, and maybe circle back for the latter at some point. \n\nI'm re-using some of the code that I used for my previous project that explored deep learning on images using a transfer learning approach. Check that out if you haven't already. \n* https://www.jocampo.com/projects/2018/11/4/image-recognition-and-transfer-learning\n\n### Sections \n* <a href='#Prep'>Prep</a>\n* <a href='#Development'>Development</a>\n* <a href='#Further Testing'>Further Testing</a>\n* <a href='#Enhancements'>Enhancements</a>\n* <a href='#Results'>Results</a>"},{"metadata":{},"cell_type":"markdown","source":"<a name='Prep'></a>\n# Prep\n* Import the necessary packages\n* Set working directory\n* Set random seed to make reproducable"},{"metadata":{"trusted":false},"cell_type":"code","source":"import pandas as pd\nimport keras\nfrom keras.applications.inception_v3 import InceptionV3\nfrom keras.preprocessing import image\nfrom keras.models import Model\nfrom keras.layers import Input, Dense, GlobalAveragePooling2D\nfrom keras import backend as K\nimport os\nimport numpy as np\nfrom numpy.random import seed\nimport json\nfrom collections import Counter\nfrom keras.optimizers import SGD\nfrom PIL import Image\nfrom PIL import ImageEnhance\nfrom PIL import ImageFilter\nimport matplotlib.pyplot as plt\nfrom IPython.display import Image as Image2\nfrom IPython.display import display\nfrom matplotlib.pyplot import imshow\nimport urllib\nfrom tensorflow import set_random_seed\nimport tensorflow as tf\nfrom scipy.spatial import distance_matrix","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Checking the GPU is running\nGiven the type of model I'll be using, I want to make sure the environment can take advantage of my computer's GPUs. \nIf I've configured correctly, I should see a device_type called 'GPU'"},{"metadata":{"trusted":false},"cell_type":"code","source":"from tensorflow.python.client import device_lib\nprint(device_lib.list_local_devices())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Variable declaration"},{"metadata":{"trusted":false},"cell_type":"code","source":"working_path = 'd:/projects/python/dupe_image_pred/'\nos.chdir(working_path)\n\nseed(1)\nset_random_seed(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name='Development'></a>\n# Development\n* Load sample images\n* Import pre-trained deep learning model\n* Edit deep learning model\n* Get model output for each image\n* Check distances between them\n* Show how duplicates would present\n"},{"metadata":{},"cell_type":"markdown","source":"Get sample image to work. I'm using a naming convention that should make this easy to work with. The beginning is some name, like 'Goose' or 'NotGoose'. This is followed by an underscore and number. The number is simply a sequential number for the file, which creates a file ID. Later, I'll apply this method as an iterator for a larger number of images in a folder, as I did with the prior project. Lastly, for the development piece, I'm adding an underscore 1/0 to distinguish duplicates, where both duplicate images would have a 1. This makes it easy, as for the training set, I'm manually creating the duplicates. "},{"metadata":{"trusted":false},"cell_type":"code","source":"# change working directory\nos.chdir(working_path+'/Images/Sample_Goose')\n\n# get list of images (.jpg only)\nsample_images = [fn for fn in os.listdir() if fn.upper().endswith('.JPG')]\n\n# initialize\nlabels = []\ndupe = []\nidx_to_labels = []\nlabel_to_idx = {}\n\n# iterate\nfor fn in sample_images:\n    if not fn in label_to_idx:\n        label_to_idx[fn] = len(idx_to_labels)\n        idx_to_labels.append(fn)\n    labels.append(label_to_idx[fn])\n    dupe.append(int(fn.replace('.jpg','').rsplit('_', 2)[2]))\nlen(idx_to_labels)\n\nprint('')\nprint('Number of images: ')\nprint(len(sample_images))\nprint('')\nprint('Sample images used: ')\nprint([i for i in idx_to_labels])\nprint('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display images being used"},{"metadata":{"trusted":false},"cell_type":"code","source":"for pic in sample_images:\n    display(Image2(pic, width = 150))\n    print(pic)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, the first two images are duplicates of my dog, Goose. Grumpy cat is used for the last image, which is clearly not Goose. Here you can see the list of dupe tags derived from the file name: "},{"metadata":{"trusted":false},"cell_type":"code","source":"dupe","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I now want to normalize my images into the same size and black & white coloring. "},{"metadata":{},"cell_type":"markdown","source":"This will iterate over my images, normalize them, and create a new image file in a new folder. "},{"metadata":{"trusted":false},"cell_type":"code","source":"for i in range(0,len(sample_images)): \n    os.chdir(working_path+'/Images/Sample_Goose')\n    image_name = sample_images[i]\n    temp_load = Image.open(image_name,'r')\n    temp_load = temp_load.resize( (299,299), Image.ANTIALIAS )\n    temp_load = temp_load.convert('L') # converts to monochrome\n    temp_load = temp_load.convert('1') # converts to black and white\n    os.chdir(working_path+'/Images/Sample_Goose/Norm')\n    temp_load.save(image_name, temp_load.format)\n    temp_load.close()\n    \nos.chdir(working_path+'/Images/Sample_Goose/Norm')\n         \nsample_images_rs = [fn for fn in os.listdir() if fn.upper().endswith('.JPG')]\n         \norg_img_count = len(sample_images)\nrs_img_count = len(sample_images_rs)\n\nprint('\\n Original Images: ' + str(org_img_count) + '\\n Resized Images: ' + str(rs_img_count))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see the normalized images"},{"metadata":{"trusted":false},"cell_type":"code","source":"for pic in sample_images_rs:\n    display(Image2(pic, width = 150))\n    print(pic)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check that the images have been resized"},{"metadata":{"trusted":false},"cell_type":"code","source":"os.chdir(working_path+'/Images/Sample_Goose/Norm')\n\nw_min = 1000\nh_min = 1000\nfor i in range(1,len(sample_images_rs)):\n    temp_img = Image.open(sample_images_rs[i])\n    w, h = temp_img.size\n    if w < w_min:\n        w_min = w\n    if h < h_min:\n        h_min = h\n    \nprint('\\n Minimum image width: ' + str(w_min))\nprint('\\n Minimum image height: ' + str(h_min) + '\\n' )\n\nfor pic in sample_images_rs: \n    temp_img = Image.open(pic)\n    print(pic + ' ' + str(temp_img.size))\n\nprint('\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, let's import pre-trained model that I'll use for feature engineering"},{"metadata":{"trusted":false},"cell_type":"code","source":"# trainable = False is import because we'll be using this system for features but not output\n\nbase_model = InceptionV3(weights='imagenet', include_top=False, input_shape=(299, 299, 3))\nfor layer in base_model.layers:\n    layer.trainable = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Specify an additional layer that will be the output from the model"},{"metadata":{"trusted":false},"cell_type":"code","source":"pool_2d = GlobalAveragePooling2D(name='pool_2d')(base_model.output)\ndense = Dense(1024, name='dense', activation='relu')(pool_2d)\npredictions = Dense(1000, activation='relu')(dense)\nmodel = Model(inputs=base_model.input, outputs=predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load images"},{"metadata":{"trusted":false},"cell_type":"code","source":"os.chdir(working_path+'/Images/Sample_Goose/Norm')\n\nuse_images = [image.load_img(c, target_size=(299,299))\n         for c in sample_images]\n\nuse_tensor = np.array([image.img_to_array(img) for img in use_images])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the pre-trained model that I edited before, capture the final layer for each image. "},{"metadata":{"trusted":false},"cell_type":"code","source":"model_output = model.predict(use_tensor, batch_size=32, verbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check the shape of the output. For each image, there should be a row. The model's last layer that I added was 1000 nodes, so the resulting columns should be output of those 1000 nodes."},{"metadata":{"trusted":false},"cell_type":"code","source":"model_output.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With the results, put it into a dataframe and create a distance matrix from it. "},{"metadata":{"trusted":false},"cell_type":"code","source":"df = pd.DataFrame(model_output, index = sample_images)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"dist_mat = pd.DataFrame(distance_matrix(df.values,df.values),index=df.index,columns=df.index)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"See the resulting distance matrix to view the results."},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"dist_mat","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From this, we can see that NotGoose (aka Grumpy Cat) has a non-zero value, indicating it is not a duplicate. The two Goose files have a distance of zero, meaning they are duplicates. "},{"metadata":{},"cell_type":"markdown","source":"<a name='Further Testing'></a>\n# Further Testing\n* Run distance matrices for two more sets of images that are created to be slightly different for the other pictures in the set"},{"metadata":{},"cell_type":"markdown","source":"For a different set of images of my other dog, Sadie, I'd like to see how to distances look for pictures that are slightly different from one another. "},{"metadata":{},"cell_type":"markdown","source":"Get list of files for Sadie"},{"metadata":{"trusted":false},"cell_type":"code","source":"# change working directory\nos.chdir(working_path+'/Images/Sample_Sadie')\n\n# get list of images (.jpg only)\nsadie_images = [fn for fn in os.listdir() if fn.upper().endswith('.JPG')]\n\n# initialize\nsadie_labels = []\nsadie_dupe = []\nsadie_idx_to_labels = []\nsadie_label_to_idx = {}\n\n# iterate\nfor fn in sadie_images:\n    if not fn in sadie_label_to_idx:\n        sadie_label_to_idx[fn] = len(sadie_idx_to_labels)\n        sadie_idx_to_labels.append(fn)\n    sadie_labels.append(sadie_label_to_idx[fn])\n    sadie_dupe.append(int(fn.replace('.jpg','').rsplit('_', 2)[2]))\nlen(sadie_idx_to_labels)\n\nprint('')\nprint('Number of images: ')\nprint(len(sadie_images))\nprint('')\nprint('Sample images used: ')\nprint([i for i in sadie_idx_to_labels])\nprint('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"See the pictures of Sadie."},{"metadata":{"trusted":false},"cell_type":"code","source":"for pic in sadie_images:\n    display(Image2(pic, width = 150))\n    print(pic)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see these are basically the same picture, but from a slightly different closeness. What I'd expect to see is smaller distance as compared to that of the Grumpy Cat compared to Goose. Additionally, I'd think that Sadie_3 is very similar to Sadie_5. "},{"metadata":{},"cell_type":"markdown","source":"I want to do the same thing for a new set of images of Goose. "},{"metadata":{"trusted":false},"cell_type":"code","source":"# change working directory\nos.chdir(working_path+'/Images/Sample_Goose2')\n\n# get list of images (.jpg only)\ngoose2_images = [fn for fn in os.listdir() if fn.upper().endswith('.JPG')]\n\n# initialize\ngoose2_labels = []\ngoose2_dupe = []\ngoose2_idx_to_labels = []\ngoose2_label_to_idx = {}\n\n# iterate\nfor fn in goose2_images:\n    if not fn in goose2_label_to_idx:\n        goose2_label_to_idx[fn] = len(goose2_idx_to_labels)\n        goose2_idx_to_labels.append(fn)\n    goose2_labels.append(goose2_label_to_idx[fn])\n    goose2_dupe.append(int(fn.replace('.jpg','').rsplit('_', 2)[2]))\nlen(goose2_idx_to_labels)\n\nprint('')\nprint('Number of images: ')\nprint(len(goose2_images))\nprint('')\nprint('Sample images used: ')\nprint([i for i in goose2_idx_to_labels])\nprint('')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see the pictures of Goose. "},{"metadata":{"trusted":false},"cell_type":"code","source":"for pic in goose2_images:\n    display(Image2(pic, width = 150))\n    print(pic)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Similarly for the series of Sadie, I'd expect Goos2_1 and Goose2_5 to have close distance values. "},{"metadata":{},"cell_type":"markdown","source":"Now, I need to normalize the images of Sadie"},{"metadata":{"trusted":false},"cell_type":"code","source":"for i in range(0,len(sadie_images)): \n    os.chdir(working_path+'/Images/Sample_Sadie')\n    image_name = sadie_images[i]\n    temp_load = Image.open(image_name,'r')\n    temp_load = temp_load.resize( (299,299), Image.ANTIALIAS )\n    temp_load = temp_load.convert('L') # converts to monochrome\n    temp_load = temp_load.convert('1') # converts to black and white\n    os.chdir(working_path+'/Images/Sample_Sadie/Norm')\n    temp_load.save(image_name, temp_load.format)\n    temp_load.close()\n    \nos.chdir(working_path+'/Images/Sample_Sadie/Norm')\n         \nsadie_images_rs = [fn for fn in os.listdir() if fn.upper().endswith('.JPG')]\n         \norg_img_count = len(sadie_images)\nrs_img_count = len(sadie_images_rs)\n\nprint('\\n Original Images: ' + str(org_img_count) + '\\n Resized Images: ' + str(rs_img_count))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Similarly, I need to do this for Goose2. "},{"metadata":{"trusted":false},"cell_type":"code","source":"for i in range(0,len(goose2_images)): \n    os.chdir(working_path+'/Images/Sample_Goose2')\n    image_name = goose2_images[i]\n    temp_load = Image.open(image_name,'r')\n    temp_load = temp_load.resize( (299,299), Image.ANTIALIAS )\n    temp_load = temp_load.convert('L') # converts to monochrome\n    temp_load = temp_load.convert('1') # converts to black and white\n    os.chdir(working_path+'/Images/Sample_Goose2/Norm')\n    temp_load.save(image_name, temp_load.format)\n    temp_load.close()\n    \nos.chdir(working_path+'/Images/Sample_Goose2/Norm')\n         \ngoose2_images_rs = [fn for fn in os.listdir() if fn.upper().endswith('.JPG')]\n         \norg_img_count = len(goose2_images)\nrs_img_count = len(goose2_images_rs)\n\nprint('\\n Original Images: ' + str(org_img_count) + '\\n Resized Images: ' + str(rs_img_count))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'm going to skip displaying the output of the normalizing. I've already shown what that looks like and I quickly looked at the results in the /norm folder. "},{"metadata":{},"cell_type":"markdown","source":"I'm now going to run the model output and distance matrix of Sadie. I'm combining the steps into one code section and removing the display of the shape."},{"metadata":{"trusted":false},"cell_type":"code","source":"os.chdir(working_path+'/Images/Sample_Sadie/Norm')\n\nuse_images_sadie = [image.load_img(c, target_size=(299,299))\n         for c in sadie_images]\n\nuse_tensor_sadie = np.array([image.img_to_array(img) for img in use_images_sadie])\n\nmodel_output_sadie = model.predict(use_tensor_sadie, batch_size=32, verbose=1)\n\ndf_sadie = pd.DataFrame(model_output_sadie, index = sadie_images)\n\ndist_mat_sadie = pd.DataFrame(distance_matrix(df_sadie.values,df_sadie.values),index=df_sadie.index,columns=df_sadie.index)\n\ndist_mat_sadie","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These results show what I'd expect. Sadie_1 (most zoomed out) is most different from Sadie_6 (most zoomed in). Additionally, the middle images (Sadie_2 through Sadie_5) had similar distances. "},{"metadata":{},"cell_type":"markdown","source":"Run the same thing for Goose2. "},{"metadata":{"trusted":false},"cell_type":"code","source":"os.chdir(working_path+'/Images/Sample_Goose2/Norm')\n\nuse_images_goose2 = [image.load_img(c, target_size=(299,299))\n         for c in goose2_images]\n\nuse_tensor_goose2 = np.array([image.img_to_array(img) for img in use_images_goose2])\n\nmodel_output_goose2 = model.predict(use_tensor_goose2, batch_size=32, verbose=1)\n\ndf_goose2 = pd.DataFrame(model_output_goose2, index = goose2_images)\n\ndist_mat_goose2 = pd.DataFrame(distance_matrix(df_goose2.values,df_goose2.values),index=df_goose2.index,columns=df_goose2.index)\n\ndist_mat_goose2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the results, it makes sense that Goose2_2 and Goose2_3 are the most similar, given the composition of the pictures. "},{"metadata":{},"cell_type":"markdown","source":"<a name='Enhancements'></a>\n# Enhancements\n* Transform the resulting distance matrix to be a list of the closest neighbor (not including itself)\n* Run technique against a much larger data set of vehicles that were used in the previous project"},{"metadata":{},"cell_type":"markdown","source":"These are the steps to grab the smallest distance across all other files."},{"metadata":{"trusted":false},"cell_type":"code","source":"# create a temp dataframe\nstg_sadie = dist_mat_sadie\n\n# add a column for file name equal to the index, which enables the melt\nstg_sadie['File1']=stg_sadie.index\n\n# grab column names\nstg_cols = dist_mat_sadie.columns\n\n# complete the melt\nstg1 = pd.melt(stg_sadie, id_vars='File1',value_vars=stg_cols[stg_cols != 'File1'])\n\n# clean up column names\nstg1.columns = ['File1','File2','Distance']\n\n# remove rows where files are the same, which removes self duplicates\nstg2 = stg1[stg1.File1 != stg1.File2]\n\nfnl_sadie = stg2.groupby('File1')['Distance'].min()\n\nfnl_sadie","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For pictures of vehicles used in the prior project, add them to another folder and standardize naming conventions. "},{"metadata":{"trusted":false},"cell_type":"code","source":"vehicle_path = 'd:/projects/python/TL_logos/Vehicles/'\nos.chdir(vehicle_path)\n\nvehicle_list = [n for n in os.listdir() if n.upper().endswith('.JPG')]\n\nfor i in range(0,len(vehicle_list)):\n    new_name = working_path + 'Images/Sample_Vehicles/' + 'Vehicle_' + str(i) + '_0.JPG'\n    old_name = vehicle_path + vehicle_list[i]\n    os.rename(old_name, new_name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With those, I chose 30 images to duplicate. For these files, I changed the naming convention to Vehicle_D#_1 to indicate it is a duplicate."},{"metadata":{},"cell_type":"markdown","source":"Read in the new data set. "},{"metadata":{"trusted":false},"cell_type":"code","source":"# change working directory\nos.chdir(working_path+'/Images/Sample_Vehicles')\n\n# get list of images (.jpg only)\nvehicles_images = [fn for fn in os.listdir() if fn.upper().endswith('.JPG')]\n\n# initialize\nvehicles_labels = []\nvehicles_dupe = []\nvehicles_idx_to_labels = []\nvehicles_label_to_idx = {}\n\n# iterate\nfor fn in vehicles_images:\n    if not fn in vehicles_label_to_idx:\n        vehicles_label_to_idx[fn] = len(vehicles_idx_to_labels)\n        vehicles_idx_to_labels.append(fn)\n    vehicles_labels.append(vehicles_label_to_idx[fn])\n    vehicles_dupe.append(int(fn.replace('.jpg','').replace('.JPG','').rsplit('_', 2)[2]))\nlen(vehicles_idx_to_labels)\n\nprint('')\nprint('Number of images: ')\nprint(len(vehicles_images))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"See the images that I entered into the folder as duplicates. "},{"metadata":{"trusted":false},"cell_type":"code","source":"dupes_in_folder = 0 \n\nfor i in vehicles_images: \n    if i.endswith('1.JPG'):\n        print(i)\n        dupes_in_folder += 1\n\nprint('\\n Duplicates in the folder: ' + str(dupes_in_folder))\nprint('\\n We should see ' + str(dupes_in_folder * 2) + ' duplicates in the resulting list. ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Normalize the vehicle images."},{"metadata":{"trusted":false},"cell_type":"code","source":"for i in range(0,len(vehicles_images)): \n    os.chdir(working_path+'/Images/Sample_Vehicles')\n    image_name = vehicles_images[i]\n    temp_load = Image.open(image_name,'r')\n    temp_load = temp_load.resize( (299,299), Image.ANTIALIAS )\n    temp_load = temp_load.convert('L') # converts to monochrome\n    temp_load = temp_load.convert('1') # converts to black and white\n    os.chdir(working_path+'/Images/Sample_Vehicles/Norm')\n    temp_load.save(image_name, temp_load.format)\n    temp_load.close()\n    \nos.chdir(working_path+'/Images/Sample_Vehicles/Norm')\n         \nvehicles_images_rs = [fn for fn in os.listdir() if fn.upper().endswith('.JPG')]\n         \norg_img_count = len(vehicles_images)\nrs_img_count = len(vehicles_images_rs)\n\nprint('\\n Original Images: ' + str(org_img_count) + '\\n Resized Images: ' + str(rs_img_count))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Run the model for each image. "},{"metadata":{"trusted":false},"cell_type":"code","source":"os.chdir(working_path+'/Images/Sample_Vehicles/Norm')\n\nuse_images_vehicles = [image.load_img(c, target_size=(299,299))\n         for c in vehicles_images]\n\nuse_tensor_vehicles = np.array([image.img_to_array(img) for img in use_images_vehicles])\n\nmodel_output_vehicles = model.predict(use_tensor_vehicles, batch_size=32, verbose=1)\n\ndf_vehicles = pd.DataFrame(model_output_vehicles, index = vehicles_images)\n\ndist_mat_vehicles = pd.DataFrame(distance_matrix(df_vehicles.values,df_vehicles.values),index=df_vehicles.index,columns=df_vehicles.index)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get the minimum distance per image. "},{"metadata":{"trusted":false},"cell_type":"code","source":"# create a temp dataframe\nstg_vehicles = dist_mat_vehicles\n\n# add a column for file name equal to the index, which enables the melt\nstg_vehicles['File1']=stg_vehicles.index\n\n# grab column names\nstg_cols = dist_mat_vehicles.columns\n\n# complete the melt\nstg1 = pd.melt(stg_vehicles, id_vars='File1',value_vars=stg_cols[stg_cols != 'File1'])\n\n# clean up column names\nstg1.columns = ['File1','File2','Distance']\n\n# remove rows where files are the same, which removes self duplicates\nstg2 = stg1[stg1.File1 != stg1.File2]\n\nfnl_vehicles = stg2.groupby('File1')['Distance'].min()\n\nfnl_vehicles","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a name='Results'></a>\n# Results\n* Identified duplicate images\n* Show near-duplicate images"},{"metadata":{},"cell_type":"markdown","source":"See the duplicates. "},{"metadata":{"trusted":false},"cell_type":"code","source":"vehicle_dupes = fnl_vehicles[fnl_vehicles<1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"len(vehicle_dupes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The 58 length shows that it correctly identified the 29 duplicates and 29 original images that had a duplicate made. "},{"metadata":{"trusted":false},"cell_type":"code","source":"vehicle_dupes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The interesting thing about this technique isn't finding perfect duplicates, which could be accomplished other ways. If we select distances greater than 0 and less than a threshold, we can find similar images. "},{"metadata":{"trusted":false},"cell_type":"code","source":"vehicles_similar = fnl_vehicles[fnl_vehicles<26] # 26 was derived based on results examination","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"vehicles_similar_not_same = vehicles_similar[vehicles_similar>=1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"vehicles_similar_not_same","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the list of vehicle pictures that are similar, but not the same, get the file name that it most closely matches to. "},{"metadata":{"trusted":false},"cell_type":"code","source":"similar_df = pd.DataFrame(vehicles_similar_not_same)\njoined_results = pd.merge(similar_df,stg2,how='left',on=['File1','Distance'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display the groupings of similar, but not exact duplicate, vehicles. "},{"metadata":{"trusted":false},"cell_type":"code","source":"joined_results","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"os.chdir(working_path+'/Images/Sample_Vehicles')\n\nfor i in range(0,len(joined_results)):\n    print('Similar group ' + str(i+1))\n    pic = joined_results.File1[i]\n    display(Image2(pic, width = 150))\n    pic = joined_results.File2[i]\n    display(Image2(pic, width = 150))\n    print(pic)\n    print('\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that using a low distance metric was successful in finding some similar images. Further review results also shows that background of the image impacts results. That is to say that humans focus on the foreground, while the models look at the entire image. "},{"metadata":{},"cell_type":"markdown","source":"If we had the training data, we would be able to set up a logistic regression to predict duplicate and near-duplicate images. At this time, I'm not interested in creating that training data :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}