{"cells":[{"metadata":{},"cell_type":"markdown","source":"# EDA - Finding Similar Images in Train and Test Datasets\n\n![](http://)In this notebook, we are going to use [imagededup](https://github.com/idealo/imagededup) to find similar images in the test dataset of [Peking University/Baidu - Autonomous Driving](https://www.kaggle.com/c/pku-autonomous-driving) competition.\n\n### Install imagedup"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install imagededup","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Import modules"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pylab as plt\nimport imagededup\nfrom imagededup.methods import PHash\nfrom imagededup.utils import plot_duplicates","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"image_dir='../input/pku-autonomous-driving/test_images/'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Find similar images"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"phasher = PHash()\nduplicates = phasher.find_duplicates(image_dir=image_dir, scores=True, max_distance_threshold=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"__Note:__ `max_distance_threshold` defines the threshold of differences between two images to consider them similar, the higher the value, the more tolerant it is in differences.\n\nBelow we list the first 15 images found having similar content according to imagededup. To get the full list, you have to display the content of variable `duplicates`."},{"metadata":{"trusted":true},"cell_type":"code","source":"{y: duplicates[y] for y in [x for x in duplicates if duplicates[x] != []][:15]}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('There are', len([x for x in duplicates if duplicates[x] != []]), 'images with similar images over', len(duplicates), 'images.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow! It seems there are a lot of similar images in this dataset. Let's have a look.\n\n### Visualize results\n\nFirst, we visualise an image for witch imagededup found two similar images with a 0 threshold."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,20))\nplot_duplicates(image_dir=image_dir, duplicate_map=duplicates, filename='ID_5bf531cf3.jpg')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"imagededup is correct. Except some image pre-processing and enhancement, these 3 images are identical.\n\nNow we'll visualise an image for witch imagededup found two images with a threshold of 0 and two others with a threshold of 2."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,20))\nplot_duplicates(image_dir=image_dir, duplicate_map=duplicates, filename='ID_ca20646c5.jpg')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The results are impressive:\n - it found two duplications of `ID_ca20646c5.jpg` but with different contrast\n - it found two other images (identical to each other) which were probably taken at the same place as `ID_ca20646c5.jpg`  but at a slightly different time."},{"metadata":{},"cell_type":"markdown","source":"# Conclusions\n\n1. [imagededup](https://github.com/idealo/imagededup) is a powerful tool in your toolbox for various tasks like finding similar images in your dataset, find duplicated images to avoid leaks from train to test dataset...\n\n2. The test dataset has duplicated images. Some image filters were applied to hide this duplication.\n\n3. The test dataset also contains highly correlated images because they were taken at the same location within a short period of time.\n\nThanks for reading! "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}