{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### Summary\nNote book này được sao chép và chỉnh sửa lại từ project của [Nick Kuzmenkov](https://www.kaggle.com/nickuzmenkov).\n\nViệc trùng lặp ảnh gây khó khăn cho việc huấn luyện dữ liệu. Ảnh trùng lặp nhưng cùng nhãn gây hiện tương data leakage, trong khi ảnh trùng lặp nhưng khác nhãn gây nhiễu.\nNote book này sử dụng để tìm cách ảnh trùng lặp. Kết quả tìm được 56 cặp ảnh trùng lặp.\n### Các note book khác của nhóm:\n\n1. [Revealing Duplicates notebook](https://www.kaggle.com/nvlinhh/int3414-22-n11-revealing-duplicate)\n2. [Preprocessing notebook](https://www.kaggle.com/congnguyen8201/int3414-22-n11-preprocessing)\n3. [Training notebook](https://www.kaggle.com/congnguyen8201/int3414-22-n11-training)","metadata":{}},{"cell_type":"markdown","source":"### Imports","metadata":{}},{"cell_type":"code","source":"from tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport pandas as pd\nimport numpy as np\nimport imagehash\nimport PIL\nimport os","metadata":{"_uuid":"4a78d0be9bc398c6f69dffdb9709c7e32361f47d","scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Saving downscaled images to boost performance\nComputing hash over original images of very high quality would take nearly 5 hours, thus we downscaling first.","metadata":{}},{"cell_type":"code","source":"class CFG():\n    \n    threshold = .9\n    img_size = 600\n    seed = 42","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#đọc ảnh vào\nroot = '/kaggle/input/plant-pathology-2021-fgvc8/train_images'\n#lấy path của file jpg\npaths = os.listdir(root)\n\ndf = pd.read_csv('/kaggle/input/plant-pathology-2021-fgvc8/train.csv', index_col='image')\n\n#decode ảnh, chuyển ảnh sang dạng 600X600X3 (rgb), cast sang unit8\nfor path in tqdm(paths, total=len(paths)):\n    image = tf.io.read_file(os.path.join(root, path))\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.image.resize(image, [CFG.img_size, CFG.img_size])\n    image = tf.cast(image, tf.uint8).numpy()\n    plt.imsave(path, image)","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Hash computation","metadata":{}},{"cell_type":"code","source":"hash_functions = [#grey scale, down scale sang 8x8\n    imagehash.average_hash,#tính trung bình cộng tất cả các điểm ảnh rồi so sánh từng điểm ảnh, >tb ->1, <tb -> 0\n    imagehash.phash,#\n    imagehash.dhash,\n    imagehash.whash]\n#mỗi hàm hash đưa ra tập 64 bit 0,1\n\nimage_ids = []\nhashes = []#kết quả sau khi áp dụng 4 cách hash cho ảnh.\n\npaths = tf.io.gfile.glob('./*.jpg')\n\nfor path in tqdm(paths, total=len(paths)):\n\n    image = PIL.Image.open(path)\n\n    hashes.append(np.array([x(image).hash for x in hash_functions]).reshape(-1,))\n    image_ids.append(path.split('/')[-1])#thêm id ảnh\n    \nhashes = np.array(hashes)\nimage_ids = np.array(image_ids)","metadata":{"_uuid":"3a94ec9c45f58e7bc62bfeee6c2cdf06d7d92d92","scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Run search across hashed images\nWe firstly compare each image hash with all the hashes and then leave only unique pairs of matches","metadata":{}},{"cell_type":"code","source":"#so sánh các ảnh\nduplicate_ids = []\n# ảnh đầu: 110 ->ttf\n#so sanh ảnh a2: 100>tff\n#similarity: 101 -> mean(similarity) = 2/3\nfor i in tqdm(range(len(hashes)), total=len(hashes)):\n    similarity = (hashes[i] == hashes).mean(axis=1)#mảng độ tương đồng của các điểm ảnh\n    duplicate_ids.append(list(image_ids[similarity > CFG.threshold]))#nếu độ giống nhau > 0.9 thì coi là giống nhau\nduplicate_ids","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"duplicates = [frozenset([x] + y) for x, y in zip(image_ids, duplicate_ids)]\nduplicates = set([x for x in duplicates if len(x) > 1])#do sau khi so sánh duplicate_id có 1 phần tử là chính ảnh đó\nlen(duplicates)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we add some of the duplicates spotted by @kingofarmy in the corresponding **[discussion](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/229851)**:","metadata":{}},{"cell_type":"markdown","source":"## 4. Let's see what is found","metadata":{}},{"cell_type":"code","source":"#cho ra một file csv để dùng cho note book sau\nprint(f'Found {len(duplicates)} duplicate pairs:')\nfor row in duplicates:\n    print(', '.join(row))","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Writing duplicates to \"duplicates.csv\".')\nwith open('duplicates.csv', 'w') as file:\n    for row in duplicates:\n        file.write(','.join(row) + '\\n')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for row in duplicates:\n    \n    figure, axes = plt.subplots(1, len(row), figsize=[5 * len(row), 5])\n\n    for i, image_id in enumerate(row):\n        image = plt.imread(os.path.join('../input/plant-pathology-2021-fgvc8/train_images', image_id))\n        axes[i].imshow(image)\n\n        axes[i].set_title(f'{image_id} - {df.loc[image_id, \"labels\"]}')\n        axes[i].axis('off')\n\n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Clear working folder to avoid output pollution","metadata":{}},{"cell_type":"code","source":"for file in tf.io.gfile.glob('./*.jpg'):\n    os.remove(file)","metadata":{"scrolled":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Acknowledgements\n\n* This work is Copy&Edit form @appian **[notebook](https://www.kaggle.com/appian/let-s-find-out-duplicate-images-with-imagehash)** with a lot of changes, but still highly inspired. If you find this notebook useful, please, upvote his work too.\n* Thanks to @kingofarmy for spotting more duplicates in **[his thread](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/229851)**.","metadata":{}}]}