{
  "id": 227829,
  "title": "50 duplicate image pairs revealed",
  "url": "/competitions/plant-pathology-2021-fgvc8/discussion/227829",
  "author_name": "",
  "post_date": "2021-03-22T12:43:48.332925300Z",
  "votes": 13,
  "comment_count": 8,
  "views": 0,
  "content": "<h3>Hello!</h3>\n<p>I decided to implement a pretty basic duplicate search via the <code>image_hash</code> library, and got interesting results: there are at least 50 pairs of identical images, and most of them are labeled differently, e.g.:<br>\n<img src=\"https://drive.google.com/uc?id=1bZ0WWO1A_f0qyXwaRX3fwX8MALPdy6BG\"><br>\n<img src=\"https://drive.google.com/uc?id=1UJUDOIudvnM33tcvWFFgw-HOw3uZjKIZ\"><br>\n<img src=\"https://drive.google.com/uc?id=1Z5Vziho1wfcQUxh2dmSXr08-ZaNzarVP\"><br>\nThe entire list of duplicate ids can be found in the <strong><a href=\"https://www.kaggle.com/nickuzmenkov/pp2021-duplicates-revealing\" target=\"_blank\">notebook</a></strong> and output file <code>duplicates.csv</code>.</p>\n<h3>Update</h3>\n<p>I've tried to find duplicates among original images and horizontally flipped, vertically flipped, 90, 180, and 270 degrees rotated images, revealing no more duplicates (but I suspect there are still a lot). As I'm a novice in image hashing and duplicates detection, I'm looking forward for experienced users to find and reveal more duplicates.</p>",
  "messages": [
    {
      "id": "1248218",
      "postDate": "03/22/2021 12:43:48",
      "content": "<h3>Hello!</h3>\n<p>I decided to implement a pretty basic duplicate search via the <code>image_hash</code> library, and got interesting results: there are at least 50 pairs of identical images, and most of them are labeled differently, e.g.:<br>\n<img src=\"https://drive.google.com/uc?id=1bZ0WWO1A_f0qyXwaRX3fwX8MALPdy6BG\"><br>\n<img src=\"https://drive.google.com/uc?id=1UJUDOIudvnM33tcvWFFgw-HOw3uZjKIZ\"><br>\n<img src=\"https://drive.google.com/uc?id=1Z5Vziho1wfcQUxh2dmSXr08-ZaNzarVP\"><br>\nThe entire list of duplicate ids can be found in the <strong><a href=\"https://www.kaggle.com/nickuzmenkov/pp2021-duplicates-revealing\" target=\"_blank\">notebook</a></strong> and output file <code>duplicates.csv</code>.</p>\n<h3>Update</h3>\n<p>I've tried to find duplicates among original images and horizontally flipped, vertically flipped, 90, 180, and 270 degrees rotated images, revealing no more duplicates (but I suspect there are still a lot). As I'm a novice in image hashing and duplicates detection, I'm looking forward for experienced users to find and reveal more duplicates.</p>",
      "rawMarkdown": "### Hello!\n\nI decided to implement a pretty basic duplicate search via the `image_hash` library, and got interesting results: there are at least 50 pairs of identical images, and most of them are labeled differently, e.g.:\n<img src='https://drive.google.com/uc?id=1bZ0WWO1A_f0qyXwaRX3fwX8MALPdy6BG' />\n<img src='https://drive.google.com/uc?id=1UJUDOIudvnM33tcvWFFgw-HOw3uZjKIZ' />\n<img src='https://drive.google.com/uc?id=1Z5Vziho1wfcQUxh2dmSXr08-ZaNzarVP' />\nThe entire list of duplicate ids can be found in the **[notebook](https://www.kaggle.com/nickuzmenkov/pp2021-duplicates-revealing)** and output file `duplicates.csv`.\n\n### Update\nI've tried to find duplicates among original images and horizontally flipped, vertically flipped, 90, 180, and 270 degrees rotated images, revealing no more duplicates (but I suspect there are still a lot). As I'm a novice in image hashing and duplicates detection, I'm looking forward for experienced users to find and reveal more duplicates.",
      "votes": null
    },
    {
      "id": "1249001",
      "postDate": "03/23/2021 02:22:41",
      "content": "<p>Let's share all duplicate images filenames to make sure to get them all. I found 54 in this <a href=\"https://www.kaggle.com/datasciencegeek/eda-plantpathology2021-fgvc8\" target=\"_blank\">notebook</a> . Here is a list of filenames:</p>\n<p>[9ada65c5b0cc9a3c.jpg, e9e8ad35c5871d06.jpg]<br>\n[94b42695f24dad36.jpg, f7d233d11aa8496c.jpg]<br>\n[d64460a3b19f1cfc.jpg, f4cba4a7294d6585.jpg]<br>\n[c5aff2e545d0a129.jpg, dbee22785d9285c4.jpg]<br>\n[ca782c131f5ce0fc.jpg, e5f27342ce1cb9a0.jpg]<br>\n[848d9f9e82d18e5e.jpg, 949b6595a5b8ca3c.jpg]<br>\n[d3945c098edc9dd1.jpg, dea33c5c0347aa4d.jpg]<br>\n[85897ea7f064b26a.jpg, ec83c7c04a1e9db9.jpg]<br>\n[8f5263c61a8cdbe2.jpg, c9d63696629107ed.jpg]<br>\n[b7b0edd58a858217.jpg, ba99260ef46b18cb.jpg]<br>\n[c1a56069de75b899.jpg, f2c798246a0a9bdd.jpg]<br>\n[965295a64a97859f.jpg, ae5074cfc9183763.jpg]<br>\n[aa0e969518e15d3f.jpg, bb1891fa8c61e663.jpg]<br>\n[a3bb604d8a55995d.jpg, e7ecc583d571d202.jpg]<br>\n[86f963a16e70345e.jpg, 9fc321ce60a69f0b.jpg]<br>\n[d0c766bc38e1978c.jpg, e83f715e56462a31.jpg]<br>\n[9fad869f21b5b240.jpg, d7ec32d72c84dc18.jpg]<br>\n[bd8088cdad5f65e0.jpg, f90738a0d2f2c1db.jpg]<br>\n[90f99ee1c1132bb6.jpg, 94c83b2d5fad6924.jpg]<br>\n[90a5341f227ce5f9.jpg, f3dc17a2b20d48ec.jpg]<br>\n[8c0731fff680c51e.jpg, df90901ccb664de6.jpg]<br>\n[c1753cad2f31d492.jpg, d2266ac99461e9db.jpg]<br>\n[a9b2e4b09297bc2b.jpg, d1f01e8d8a73274d.jpg]<br>\n[a7e613159fb22859.jpg, b5ec52491e3349ce.jpg]<br>\n[82ba3e933947b097.jpg, cdf892b5b4865859.jpg]<br>\n[e51690d8ad307dcb.jpg, eec64e8cf1c8d0c6.jpg]<br>\n[d1dec0973c2cf4a1.jpg, d2d9c2bca7a72458.jpg]</p>",
      "rawMarkdown": "Let's share all duplicate images filenames to make sure to get them all. I found 54 in this [notebook](https://www.kaggle.com/datasciencegeek/eda-plantpathology2021-fgvc8) . Here is a list of filenames:\n\n[9ada65c5b0cc9a3c.jpg, e9e8ad35c5871d06.jpg]\n[94b42695f24dad36.jpg, f7d233d11aa8496c.jpg]\n[d64460a3b19f1cfc.jpg, f4cba4a7294d6585.jpg]\n[c5aff2e545d0a129.jpg, dbee22785d9285c4.jpg]\n[ca782c131f5ce0fc.jpg, e5f27342ce1cb9a0.jpg]\n[848d9f9e82d18e5e.jpg, 949b6595a5b8ca3c.jpg]\n[d3945c098edc9dd1.jpg, dea33c5c0347aa4d.jpg]\n[85897ea7f064b26a.jpg, ec83c7c04a1e9db9.jpg]\n[8f5263c61a8cdbe2.jpg, c9d63696629107ed.jpg]\n[b7b0edd58a858217.jpg, ba99260ef46b18cb.jpg]\n[c1a56069de75b899.jpg, f2c798246a0a9bdd.jpg]\n[965295a64a97859f.jpg, ae5074cfc9183763.jpg]\n[aa0e969518e15d3f.jpg, bb1891fa8c61e663.jpg]\n[a3bb604d8a55995d.jpg, e7ecc583d571d202.jpg]\n[86f963a16e70345e.jpg, 9fc321ce60a69f0b.jpg]\n[d0c766bc38e1978c.jpg, e83f715e56462a31.jpg]\n[9fad869f21b5b240.jpg, d7ec32d72c84dc18.jpg]\n[bd8088cdad5f65e0.jpg, f90738a0d2f2c1db.jpg]\n[90f99ee1c1132bb6.jpg, 94c83b2d5fad6924.jpg]\n[90a5341f227ce5f9.jpg, f3dc17a2b20d48ec.jpg]\n[8c0731fff680c51e.jpg, df90901ccb664de6.jpg]\n[c1753cad2f31d492.jpg, d2266ac99461e9db.jpg]\n[a9b2e4b09297bc2b.jpg, d1f01e8d8a73274d.jpg]\n[a7e613159fb22859.jpg, b5ec52491e3349ce.jpg]\n[82ba3e933947b097.jpg, cdf892b5b4865859.jpg]\n[e51690d8ad307dcb.jpg, eec64e8cf1c8d0c6.jpg]\n[d1dec0973c2cf4a1.jpg, d2d9c2bca7a72458.jpg]",
      "votes": null
    },
    {
      "id": "1249095",
      "postDate": "03/23/2021 05:21:16",
      "content": "<p>Nice work! </p>\n<p>Actually, I meant that I found 50 <strong>pairs</strong> of duplicates, i.e. 100 duplicated images total. I've done a quick check with your duplicates (let's say <code>a = set(...)</code>) and my duplicates (let's say <code>b=set(...)</code>) showing that all your duplicates <code>a</code> belong to my duplicates <code>b</code>:</p>\n<pre><code>print(len(a))\nprint(len(b))\nprint(len(b.intersection(a)))\n&gt;&gt;&gt; 27\n&gt;&gt;&gt; 50\n&gt;&gt;&gt; 27\n</code></pre>\n<p>And that's great news that two different methods give the same results.</p>",
      "rawMarkdown": "Nice work! \n\nActually, I meant that I found 50 **pairs** of duplicates, i.e. 100 duplicated images total. I've done a quick check with your duplicates (let's say `a = set(...)`) and my duplicates (let's say `b=set(...)`) showing that all your duplicates `a` belong to my duplicates `b`:\n\n```\nprint(len(a))\nprint(len(b))\nprint(len(b.intersection(a)))\n>>> 27\n>>> 50\n>>> 27\n```\n\nAnd that's great news that two different methods give the same results.",
      "votes": null
    },
    {
      "id": "1249850",
      "postDate": "03/23/2021 15:10:48",
      "content": "<p>Gotcha. Thank you for sharing! I have not heard of the method you used to detect duplicates and came up with an add-hoc method myself, nice to learn new approaches.</p>",
      "rawMarkdown": "Gotcha. Thank you for sharing! I have not heard of the method you used to detect duplicates and came up with an add-hoc method myself, nice to learn new approaches.",
      "votes": null
    },
    {
      "id": "1250344",
      "postDate": "03/24/2021 01:16:17",
      "content": "<p>Thanks for sharing these tags, much appreciated!</p>",
      "rawMarkdown": "Thanks for sharing these tags, much appreciated!",
      "votes": null
    },
    {
      "id": "1258967",
      "postDate": "04/01/2021 04:06:50",
      "content": "<p>There are around 104 pairs of duplicate images. I found this using <code>Imagededup</code> library. I often use this library to check the duplicates. I used Perceptual hashing method to find duplicates.</p>\n<p>The link about <a href=\"http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html\" target=\"_blank\">Perceptual Hashing</a></p>\n<p>The embeddings are as follows:<br>\n{'bd0a909f69c97865.jpg': ['d58ad09e28df7825.jpg'],<br>\n 'eec64e8cf1c8d0c6.jpg': ['e51690d8ad307dcb.jpg'],<br>\n 'd2266ac99461e9db.jpg': ['c1753cad2f31d492.jpg'],<br>\n 'd64460a3b19f1cfc.jpg': ['f4cba4a7294d6585.jpg'],<br>\n 'a8e291e1da57596a.jpg': ['a8eb95c7d85e482a.jpg'],<br>\n 'df90901ccb664de6.jpg': ['8c0731fff680c51e.jpg'],<br>\n 'cc73e025cc36d31e.jpg': ['cc73e005cc36f31e.jpg'],<br>\n 'a484c39f9fd861d8.jpg': ['b0e0cf9f9e98e188.jpg'],<br>\n 'af08a3a0ff35c0c7.jpg': ['fe0883b0bb75a0cd.jpg'],<br>\n 'cc73e005cc36f31e.jpg': ['cc73e025cc36d31e.jpg'],<br>\n 'f5e4c2f7849a8c91.jpg': ['f4a0c077c79a8cd9.jpg'],<br>\n '9ada65c5b0cc9a3c.jpg': ['e9e8ad35c5871d06.jpg'],<br>\n 'd0b487c76e8e3278.jpg': ['e0b4a7c3e88e7332.jpg'],<br>\n 'f585d0d2caeaeb02.jpg': ['bd84d4dacaf8cb40.jpg'],<br>\n 'fe0883b0bb75a0cd.jpg': ['af08a3a0ff35c0c7.jpg'],<br>\n 'b4e4c9424f79335a.jpg': ['b465cbd64c19725a.jpg'],<br>\n 'b7b0edd58a858217.jpg': ['ba99260ef46b18cb.jpg'],<br>\n 'e90e1ac8b7e7aa50.jpg': ['e90e1acab7e5aa50.jpg'],<br>\n 'e0f4c0f48aca6b9e.jpg': ['a065c2f4ca986fdc.jpg'],<br>\n '9c968ac989ed332e.jpg': ['8dbeda49894d522e.jpg'],<br>\n 'aba437d04a0997dd.jpg': ['afa074d04a8d25df.jpg'],<br>\n 'ca782c131f5ce0fc.jpg': ['e5f27342ce1cb9a0.jpg'],<br>\n 'e0fec0c0838bcf8f.jpg': ['f0daf0c0828f9f0f.jpg'],<br>\n 'bd92c0694632d73b.jpg': ['bf9ac0e9c6628633.jpg'],<br>\n 'c738d13e7146798c.jpg': ['c738d13e73467988.jpg'],<br>\n 'ff88c51c70b0e2d6.jpg': ['ff08c71cf0b42ba2.jpg'],<br>\n '8c0731fff680c51e.jpg': ['df90901ccb664de6.jpg'],<br>\n 'f196f11a99d91ce0.jpg': ['f392f11919991cea.jpg'],<br>\n 'dbee22785d9285c4.jpg': ['c5aff2e545d0a129.jpg'],<br>\n 'abb0d5a9d4b4985c.jpg': ['ab91d1a9d4b49a6c.jpg'],<br>\n 'aaf5d596828adac8.jpg': ['faf5d5868a0ac9b0.jpg'],<br>\n 'd0c766bc38e1978c.jpg': ['e83f715e56462a31.jpg'],<br>\n 'f392f11919991cea.jpg': ['f196f11a99d91ce0.jpg'],<br>\n 'd89c9cda9274322f.jpg': ['d89d9ede82743229.jpg'],<br>\n '9fad869f21b5b240.jpg': ['d7ec32d72c84dc18.jpg'],<br>\n 'e8212ddeec024fec.jpg': ['e8210ddefc024fec.jpg'],<br>\n 'fcc4c8fe30300dcf.jpg': ['fcc4c8bf387015c5.jpg'],<br>\n 'c1753cad2f31d492.jpg': ['d2266ac99461e9db.jpg'],<br>\n 'b53c9ab0c9e4f0c9.jpg': ['f528cbb0c5c1f0cd.jpg'],<br>\n 'ba2e601c8979afe2.jpg': ['9a2e269c1969bfe0.jpg'],<br>\n 'e11be7f110de8f80.jpg': ['e13be7f1105e8f80.jpg'],<br>\n 'f891839e8f342e72.jpg': ['f891c19fbf301233.jpg'],<br>\n 'afbe5641896d522a.jpg': ['8dbeda49894d522e.jpg'],<br>\n '9a2e269c1969bfe0.jpg': ['ba2e601c8979afe2.jpg'],<br>\n 'a59497a1da8a49de.jpg': ['a5a687b1ca8a595e.jpg'],<br>\n 'c15d3d3ae123c547.jpg': ['c1583d30e0e1d5f7.jpg'],<br>\n 'dea33c5c0347aa4d.jpg': ['d3945c098edc9dd1.jpg'],<br>\n '82ba3e933947b097.jpg': ['cdf892b5b4865859.jpg'],<br>\n 'a1a1d5a4cada586f.jpg': ['a9b095bedada5805.jpg'],<br>\n 'ae5074cfc9183763.jpg': ['965295a64a97859f.jpg'],<br>\n 'd1f01e8d8a73274d.jpg': ['a9b2e4b09297bc2b.jpg'],<br>\n 'f0cb859beb1b30c1.jpg': ['f0cb879e4a1b35c1.jpg'],<br>\n 'a7e613159fb22859.jpg': ['b5ec52491e3349ce.jpg'],<br>\n 'bf9ac0e9c6628633.jpg': ['bd92c0694632d73b.jpg'],<br>\n 'cdf892b5b4865859.jpg': ['82ba3e933947b097.jpg'],<br>\n 'd70f2230fb5a093d.jpg': ['d70f6330fb5a090d.jpg'],<br>\n 'ff3fd02a82ad8a84.jpg': ['fc5fd028d2a988f4.jpg'],<br>\n 'c6e4efce38d81419.jpg': ['c2ccaf86b8d9549a.jpg'],<br>\n 'dfc86b180e3ac8f8.jpg': ['dfcc6b28063a88fc.jpg'],<br>\n 'f474927ad9c0e88d.jpg': ['b5369279d9c1c949.jpg'],<br>\n 'bc2a94a2af52da69.jpg': ['b4ae98a3ad52c939.jpg'],<br>\n 'f90738a0d2f2c1db.jpg': ['bd8088cdad5f65e0.jpg'],<br>\n 'c738d13e73467988.jpg': ['c738d13e7146798c.jpg'],<br>\n 'c9d63696629107ed.jpg': ['8f5263c61a8cdbe2.jpg'],<br>\n 'c5339e87235c6ac5.jpg': ['c53b9e87215c6ac5.jpg'],<br>\n 'b0e0cf9f9e98e188.jpg': ['a484c39f9fd861d8.jpg'],<br>\n 'b80898e11f81ff1f.jpg': ['b80898e11fa17f1f.jpg'],<br>\n 'aaac95caca4b8d2d.jpg': ['a8bdb4cac84b98cd.jpg'],<br>\n 'ff08c71cf0b42ba2.jpg': ['ff88c51c70b0e2d6.jpg'],<br>\n '8dbeda49894d522e.jpg': ['afbe5641896d522a.jpg', '9c968ac989ed332e.jpg'],<br>\n 'd7ec32d72c84dc18.jpg': ['9fad869f21b5b240.jpg'],<br>\n 'f6f1160be38a48d5.jpg': ['f6f11e0fe38a08c5.jpg'],<br>\n 'd1dec0973c2cf4a1.jpg': ['d2d9c2bca7a72458.jpg'],<br>\n '9886c704f93a79b3.jpg': ['98864714f96a75d5.jpg'],<br>\n 'a8bdb4cac84b98cd.jpg': ['aaac95caca4b8d2d.jpg'],<br>\n 'f8228796cfdae848.jpg': ['b8828287cfdafc49.jpg'],<br>\n 'f528cbb0c5c1f0cd.jpg': ['b53c9ab0c9e4f0c9.jpg'],<br>\n '965295a64a97859f.jpg': ['ae5074cfc9183763.jpg'],<br>\n 'afa074d04a8d25df.jpg': ['aba437d04a0997dd.jpg'],<br>\n 'c4e8843569247fcf.jpg': ['c46a85b46b643acf.jpg'],<br>\n 'f4cba4a7294d6585.jpg': ['d64460a3b19f1cfc.jpg'],<br>\n 'ebd19f1c549a1ad0.jpg': ['ebd19f9c549a12d0.jpg'],<br>\n 'f8ab8597dc92d411.jpg': ['faca978fda90d011.jpg'],<br>\n 'abf0b5819f028f0f.jpg': ['abf0b5a0df028b17.jpg'],<br>\n 'dfbdc047068b063d.jpg': ['cebdc20f67838631.jpg'],<br>\n 'f47ed2a14ac36998.jpg': ['f47ad2a2ca934d8a.jpg'],<br>\n 'afe996c30a0e0d3e.jpg': ['afebd604ca06093f.jpg'],<br>\n 'f0cb879e4a1b35c1.jpg': ['f0cb859beb1b30c1.jpg'],<br>\n 'a8eb95c7d85e482a.jpg': ['a8e291e1da57596a.jpg'],<br>\n 'a8e895f5aa54d2a5.jpg': ['a8e895e5aad453a5.jpg'],<br>\n 'e0b4a7c3e88e7332.jpg': ['d0b487c76e8e3278.jpg'],<br>\n 'bcdad04d9d24ab4c.jpg': ['b4dad04d896dcb25.jpg'],<br>\n 'ec83c7c04a1e9db9.jpg': ['85897ea7f064b26a.jpg'],<br>\n 'e385830ecacd2d9e.jpg': ['c335971e8acd609e.jpg'],<br>\n 'bfa2d0bad0d22959.jpg': ['ba2092abd0faa95b.jpg'],<br>\n 'c1583d30e0e1d5f7.jpg': ['c15d3d3ae123c547.jpg'],<br>\n 'a3bb604d8a55995d.jpg': ['e7ecc583d571d202.jpg'],<br>\n 'bb1891fa8c61e663.jpg': ['aa0e969518e15d3f.jpg'],<br>\n 'e8a58ad4d9829f33.jpg': ['e8bd8ac4d982dd83.jpg'],<br>\n 'e90e1acab7e5aa50.jpg': ['e90e1ac8b7e7aa50.jpg'],<br>\n 'a5a687b1ca8a595e.jpg': ['a59497a1da8a49de.jpg'],<br>\n 'af6292db1b611d98.jpg': ['a56292dadb618d95.jpg'],<br>\n 'd70f6330fb5a090d.jpg': ['d70f2230fb5a093d.jpg'],<br>\n 'bd84d4dacaf8cb40.jpg': ['f585d0d2caeaeb02.jpg'],<br>\n 'b4ae98a3ad52c939.jpg': ['bc2a94a2af52da69.jpg'],<br>\n '98864714f96a75d5.jpg': ['9886c704f93a79b3.jpg'],<br>\n 'a5add7c29c8a9899.jpg': ['a4a5d6d69e9a981c.jpg'],<br>\n 'e51690d8ad307dcb.jpg': ['eec64e8cf1c8d0c6.jpg'],<br>\n 'f09697d6dada8805.jpg': ['fa9f95c2daca0905.jpg'],<br>\n '95bc383078d273b3.jpg': ['953f3074707063b7.jpg'],<br>\n 'b8a7cac8958b98d9.jpg': ['b88dc8ce8d8f0ad9.jpg'],<br>\n '8f5263c61a8cdbe2.jpg': ['c9d63696629107ed.jpg'],<br>\n 'd89062fbf4452a9e.jpg': ['d0806bfbf9452693.jpg'],<br>\n 'ba99260ef46b18cb.jpg': ['b7b0edd58a858217.jpg'],<br>\n 'ffe4c0878d50b4b8.jpg': ['b4c0c0b7cdd0b4f9.jpg'],<br>\n 'dfcc6b28063a88fc.jpg': ['dfc86b180e3ac8f8.jpg'],<br>\n 'faf5d5868a0ac9b0.jpg': ['aaf5d596828adac8.jpg'],<br>\n 'e087f27e83f008e7.jpg': ['e087a7fe83c00acf.jpg'],<br>\n 'faca978fda90d011.jpg': ['f8ab8597dc92d411.jpg'],<br>\n 'a8e895e5aad453a5.jpg': ['a8e895f5aa54d2a5.jpg'],<br>\n 'e8bd8ac4d982dd83.jpg': ['e8a58ad4d9829f33.jpg'],<br>\n 'd89d9ede82743229.jpg': ['d89c9cda9274322f.jpg'],<br>\n 'b490da8ddc8dc4ce.jpg': ['b594d286ddcfc0c8.jpg'],<br>\n 'c53b9e87215c6ac5.jpg': ['c5339e87235c6ac5.jpg'],<br>\n 'bd8088cdad5f65e0.jpg': ['f90738a0d2f2c1db.jpg'],<br>\n 'c1a56069de75b899.jpg': ['f2c798246a0a9bdd.jpg'],<br>\n 'e586c25e0f1c3c79.jpg': ['e787c35e0f0c34e8.jpg'],<br>\n 'c46a85b46b643acf.jpg': ['c4e8843569247fcf.jpg'],<br>\n 'aa142048eff78bf4.jpg': ['aa542048ebf78bf4.jpg'],<br>\n 'a56292dadb618d95.jpg': ['af6292db1b611d98.jpg'],<br>\n 'fc5fd028d2a988f4.jpg': ['ff3fd02a82ad8a84.jpg'],<br>\n 'a4a5d6d69e9a981c.jpg': ['a5add7c29c8a9899.jpg'],<br>\n 'b8828287cfdafc49.jpg': ['f8228796cfdae848.jpg'],<br>\n 'e1e39f8809c7349d.jpg': ['e9c39fa809c7341d.jpg'],<br>\n 'e83f715e56462a31.jpg': ['d0c766bc38e1978c.jpg'],<br>\n '86f963a16e70345e.jpg': ['9fc321ce60a69f0b.jpg'],<br>\n 'f6f11e0fe38a08c5.jpg': ['f6f1160be38a48d5.jpg'],<br>\n 'b465cbd64c19725a.jpg': ['b4e4c9424f79335a.jpg'],<br>\n '9fc321ce60a69f0b.jpg': ['86f963a16e70345e.jpg'],<br>\n 'd0806bfbf9452693.jpg': ['d89062fbf4452a9e.jpg'],<br>\n '94c83b2d5fad6924.jpg': ['90f99ee1c1132bb6.jpg'],<br>\n 'e9e8ad35c5871d06.jpg': ['9ada65c5b0cc9a3c.jpg'],<br>\n 'a9b2e4b09297bc2b.jpg': ['d1f01e8d8a73274d.jpg'],<br>\n 'f0daf0c0828f9f0f.jpg': ['e0fec0c0838bcf8f.jpg'],<br>\n 'd2d9c2bca7a72458.jpg': ['d1dec0973c2cf4a1.jpg'],<br>\n 'fc01c79c9ee107e1.jpg': ['fc01c2dc8fa99de0.jpg'],<br>\n '953f3074707063b7.jpg': ['95bc383078d273b3.jpg'],<br>\n 'e03496c3d9ddc1c9.jpg': ['a03497c399c9c7d9.jpg'],<br>\n 'f2c798246a0a9bdd.jpg': ['c1a56069de75b899.jpg'],<br>\n 'e8e5c38f8f18a05d.jpg': ['e8a5c38f9f9828d8.jpg'],<br>\n 'e087a7fe83c00acf.jpg': ['e087f27e83f008e7.jpg'],<br>\n '892ec13c8b783fca.jpg': ['a91ec1bc0b783f43.jpg'],<br>\n '83e3d2369933c5d2.jpg': ['83e3d2329933c5f2.jpg'],<br>\n '90a5341f227ce5f9.jpg': ['f3dc17a2b20d48ec.jpg'],<br>\n 'c335971e8acd609e.jpg': ['e385830ecacd2d9e.jpg'],<br>\n 'e9c39fa809c7341d.jpg': ['e1e39f8809c7349d.jpg'],<br>\n 'b80898e11fa17f1f.jpg': ['b80898e11f81ff1f.jpg'],<br>\n 'e22e9a365cd1ea58.jpg': ['e22e8a365cd1ea5c.jpg'],<br>\n 'fa9f95c2daca0905.jpg': ['f09697d6dada8805.jpg'],<br>\n 'f4a0c077c79a8cd9.jpg': ['f5e4c2f7849a8c91.jpg'],<br>\n 'b5ec52491e3349ce.jpg': ['a7e613159fb22859.jpg'],<br>\n 'd58ad09e28df7825.jpg': ['bd0a909f69c97865.jpg'],<br>\n 'e8a5c38f9f9828d8.jpg': ['e8e5c38f8f18a05d.jpg'],<br>\n 'f7d233d11aa8496c.jpg': ['94b42695f24dad36.jpg'],<br>\n 'c5aff2e545d0a129.jpg': ['dbee22785d9285c4.jpg'],<br>\n 'f47ad2a2ca934d8a.jpg': ['f47ed2a14ac36998.jpg'],<br>\n 'fcc4c8bf387015c5.jpg': ['fcc4c8fe30300dcf.jpg'],<br>\n 'e5f27342ce1cb9a0.jpg': ['ca782c131f5ce0fc.jpg'],<br>\n 'e7ecc583d571d202.jpg': ['a3bb604d8a55995d.jpg'],<br>\n '83e3d2329933c5f2.jpg': ['83e3d2369933c5d2.jpg'],<br>\n 'a9b095bedada5805.jpg': ['a1a1d5a4cada586f.jpg'],<br>\n 'b88dc8ce8d8f0ad9.jpg': ['b8a7cac8958b98d9.jpg'],<br>\n 'e787c35e0f0c34e8.jpg': ['e586c25e0f1c3c79.jpg'],<br>\n 'abf0b5a0df028b17.jpg': ['abf0b5819f028f0f.jpg'],<br>\n 'afebd604ca06093f.jpg': ['afe996c30a0e0d3e.jpg'],<br>\n 'f3dc17a2b20d48ec.jpg': ['90a5341f227ce5f9.jpg'],<br>\n 'ab83c0d147c3db66.jpg': ['ab83c05147c3cfe6.jpg'],<br>\n 'e8210ddefc024fec.jpg': ['e8212ddeec024fec.jpg'],<br>\n 'a8aa915abe68b565.jpg': ['aaa3955ad262b56c.jpg'],<br>\n 'aaa3955ad262b56c.jpg': ['a8aa915abe68b565.jpg'],<br>\n '848d9f9e82d18e5e.jpg': ['949b6595a5b8ca3c.jpg'],<br>\n 'fc01c2dc8fa99de0.jpg': ['fc01c79c9ee107e1.jpg'],<br>\n 'd3945c098edc9dd1.jpg': ['dea33c5c0347aa4d.jpg'],<br>\n 'ba2092abd0faa95b.jpg': ['bfa2d0bad0d22959.jpg'],<br>\n 'f891c19fbf301233.jpg': ['f891839e8f342e72.jpg'],<br>\n '90f99ee1c1132bb6.jpg': ['94c83b2d5fad6924.jpg'],<br>\n 'a03497c399c9c7d9.jpg': ['e03496c3d9ddc1c9.jpg'],<br>\n 'b594d286ddcfc0c8.jpg': ['b490da8ddc8dc4ce.jpg'],<br>\n '85897ea7f064b26a.jpg': ['ec83c7c04a1e9db9.jpg'],<br>\n 'd03fcce03447353e.jpg': ['c03ec9c12c87377e.jpg'],<br>\n 'ab91d1a9d4b49a6c.jpg': ['abb0d5a9d4b4985c.jpg'],<br>\n 'b5369279d9c1c949.jpg': ['f474927ad9c0e88d.jpg'],<br>\n '949b6595a5b8ca3c.jpg': ['848d9f9e82d18e5e.jpg'],<br>\n 'a91ec1bc0b783f43.jpg': ['892ec13c8b783fca.jpg'],<br>\n 'ebd19f9c549a12d0.jpg': ['ebd19f1c549a1ad0.jpg'],<br>\n 'ab83c05147c3cfe6.jpg': ['ab83c0d147c3db66.jpg'],<br>\n 'c2ccaf86b8d9549a.jpg': ['c6e4efce38d81419.jpg'],<br>\n '94b42695f24dad36.jpg': ['f7d233d11aa8496c.jpg'],<br>\n 'aa542048ebf78bf4.jpg': ['aa142048eff78bf4.jpg'],<br>\n 'a065c2f4ca986fdc.jpg': ['e0f4c0f48aca6b9e.jpg'],<br>\n 'e13fb40087e0fec6.jpg': ['e0bbf620c7c0ce87.jpg'],<br>\n 'cebdc20f67838631.jpg': ['dfbdc047068b063d.jpg'],<br>\n 'aa0e969518e15d3f.jpg': ['bb1891fa8c61e663.jpg'],<br>\n 'e13be7f1105e8f80.jpg': ['e11be7f110de8f80.jpg'],<br>\n 'e0bbf620c7c0ce87.jpg': ['e13fb40087e0fec6.jpg'],<br>\n 'c03ec9c12c87377e.jpg': ['d03fcce03447353e.jpg'],<br>\n 'e22e8a365cd1ea5c.jpg': ['e22e9a365cd1ea58.jpg'],<br>\n 'b4dad04d896dcb25.jpg': ['bcdad04d9d24ab4c.jpg'],<br>\n 'b4c0c0b7cdd0b4f9.jpg': ['ffe4c0878d50b4b8.jpg']}</p>",
      "rawMarkdown": "There are around 104 pairs of duplicate images. I found this using `Imagededup` library. I often use this library to check the duplicates. I used Perceptual hashing method to find duplicates.\n\nThe link about [Perceptual Hashing](http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html)\n\nThe embeddings are as follows:\n{'bd0a909f69c97865.jpg': ['d58ad09e28df7825.jpg'],\n 'eec64e8cf1c8d0c6.jpg': ['e51690d8ad307dcb.jpg'],\n 'd2266ac99461e9db.jpg': ['c1753cad2f31d492.jpg'],\n 'd64460a3b19f1cfc.jpg': ['f4cba4a7294d6585.jpg'],\n 'a8e291e1da57596a.jpg': ['a8eb95c7d85e482a.jpg'],\n 'df90901ccb664de6.jpg': ['8c0731fff680c51e.jpg'],\n 'cc73e025cc36d31e.jpg': ['cc73e005cc36f31e.jpg'],\n 'a484c39f9fd861d8.jpg': ['b0e0cf9f9e98e188.jpg'],\n 'af08a3a0ff35c0c7.jpg': ['fe0883b0bb75a0cd.jpg'],\n 'cc73e005cc36f31e.jpg': ['cc73e025cc36d31e.jpg'],\n 'f5e4c2f7849a8c91.jpg': ['f4a0c077c79a8cd9.jpg'],\n '9ada65c5b0cc9a3c.jpg': ['e9e8ad35c5871d06.jpg'],\n 'd0b487c76e8e3278.jpg': ['e0b4a7c3e88e7332.jpg'],\n 'f585d0d2caeaeb02.jpg': ['bd84d4dacaf8cb40.jpg'],\n 'fe0883b0bb75a0cd.jpg': ['af08a3a0ff35c0c7.jpg'],\n 'b4e4c9424f79335a.jpg': ['b465cbd64c19725a.jpg'],\n 'b7b0edd58a858217.jpg': ['ba99260ef46b18cb.jpg'],\n 'e90e1ac8b7e7aa50.jpg': ['e90e1acab7e5aa50.jpg'],\n 'e0f4c0f48aca6b9e.jpg': ['a065c2f4ca986fdc.jpg'],\n '9c968ac989ed332e.jpg': ['8dbeda49894d522e.jpg'],\n 'aba437d04a0997dd.jpg': ['afa074d04a8d25df.jpg'],\n 'ca782c131f5ce0fc.jpg': ['e5f27342ce1cb9a0.jpg'],\n 'e0fec0c0838bcf8f.jpg': ['f0daf0c0828f9f0f.jpg'],\n 'bd92c0694632d73b.jpg': ['bf9ac0e9c6628633.jpg'],\n 'c738d13e7146798c.jpg': ['c738d13e73467988.jpg'],\n 'ff88c51c70b0e2d6.jpg': ['ff08c71cf0b42ba2.jpg'],\n '8c0731fff680c51e.jpg': ['df90901ccb664de6.jpg'],\n 'f196f11a99d91ce0.jpg': ['f392f11919991cea.jpg'],\n 'dbee22785d9285c4.jpg': ['c5aff2e545d0a129.jpg'],\n 'abb0d5a9d4b4985c.jpg': ['ab91d1a9d4b49a6c.jpg'],\n 'aaf5d596828adac8.jpg': ['faf5d5868a0ac9b0.jpg'],\n 'd0c766bc38e1978c.jpg': ['e83f715e56462a31.jpg'],\n 'f392f11919991cea.jpg': ['f196f11a99d91ce0.jpg'],\n 'd89c9cda9274322f.jpg': ['d89d9ede82743229.jpg'],\n '9fad869f21b5b240.jpg': ['d7ec32d72c84dc18.jpg'],\n 'e8212ddeec024fec.jpg': ['e8210ddefc024fec.jpg'],\n 'fcc4c8fe30300dcf.jpg': ['fcc4c8bf387015c5.jpg'],\n 'c1753cad2f31d492.jpg': ['d2266ac99461e9db.jpg'],\n 'b53c9ab0c9e4f0c9.jpg': ['f528cbb0c5c1f0cd.jpg'],\n 'ba2e601c8979afe2.jpg': ['9a2e269c1969bfe0.jpg'],\n 'e11be7f110de8f80.jpg': ['e13be7f1105e8f80.jpg'],\n 'f891839e8f342e72.jpg': ['f891c19fbf301233.jpg'],\n 'afbe5641896d522a.jpg': ['8dbeda49894d522e.jpg'],\n '9a2e269c1969bfe0.jpg': ['ba2e601c8979afe2.jpg'],\n 'a59497a1da8a49de.jpg': ['a5a687b1ca8a595e.jpg'],\n 'c15d3d3ae123c547.jpg': ['c1583d30e0e1d5f7.jpg'],\n 'dea33c5c0347aa4d.jpg': ['d3945c098edc9dd1.jpg'],\n '82ba3e933947b097.jpg': ['cdf892b5b4865859.jpg'],\n 'a1a1d5a4cada586f.jpg': ['a9b095bedada5805.jpg'],\n 'ae5074cfc9183763.jpg': ['965295a64a97859f.jpg'],\n 'd1f01e8d8a73274d.jpg': ['a9b2e4b09297bc2b.jpg'],\n 'f0cb859beb1b30c1.jpg': ['f0cb879e4a1b35c1.jpg'],\n 'a7e613159fb22859.jpg': ['b5ec52491e3349ce.jpg'],\n 'bf9ac0e9c6628633.jpg': ['bd92c0694632d73b.jpg'],\n 'cdf892b5b4865859.jpg': ['82ba3e933947b097.jpg'],\n 'd70f2230fb5a093d.jpg': ['d70f6330fb5a090d.jpg'],\n 'ff3fd02a82ad8a84.jpg': ['fc5fd028d2a988f4.jpg'],\n 'c6e4efce38d81419.jpg': ['c2ccaf86b8d9549a.jpg'],\n 'dfc86b180e3ac8f8.jpg': ['dfcc6b28063a88fc.jpg'],\n 'f474927ad9c0e88d.jpg': ['b5369279d9c1c949.jpg'],\n 'bc2a94a2af52da69.jpg': ['b4ae98a3ad52c939.jpg'],\n 'f90738a0d2f2c1db.jpg': ['bd8088cdad5f65e0.jpg'],\n 'c738d13e73467988.jpg': ['c738d13e7146798c.jpg'],\n 'c9d63696629107ed.jpg': ['8f5263c61a8cdbe2.jpg'],\n 'c5339e87235c6ac5.jpg': ['c53b9e87215c6ac5.jpg'],\n 'b0e0cf9f9e98e188.jpg': ['a484c39f9fd861d8.jpg'],\n 'b80898e11f81ff1f.jpg': ['b80898e11fa17f1f.jpg'],\n 'aaac95caca4b8d2d.jpg': ['a8bdb4cac84b98cd.jpg'],\n 'ff08c71cf0b42ba2.jpg': ['ff88c51c70b0e2d6.jpg'],\n '8dbeda49894d522e.jpg': ['afbe5641896d522a.jpg', '9c968ac989ed332e.jpg'],\n 'd7ec32d72c84dc18.jpg': ['9fad869f21b5b240.jpg'],\n 'f6f1160be38a48d5.jpg': ['f6f11e0fe38a08c5.jpg'],\n 'd1dec0973c2cf4a1.jpg': ['d2d9c2bca7a72458.jpg'],\n '9886c704f93a79b3.jpg': ['98864714f96a75d5.jpg'],\n 'a8bdb4cac84b98cd.jpg': ['aaac95caca4b8d2d.jpg'],\n 'f8228796cfdae848.jpg': ['b8828287cfdafc49.jpg'],\n 'f528cbb0c5c1f0cd.jpg': ['b53c9ab0c9e4f0c9.jpg'],\n '965295a64a97859f.jpg': ['ae5074cfc9183763.jpg'],\n 'afa074d04a8d25df.jpg': ['aba437d04a0997dd.jpg'],\n 'c4e8843569247fcf.jpg': ['c46a85b46b643acf.jpg'],\n 'f4cba4a7294d6585.jpg': ['d64460a3b19f1cfc.jpg'],\n 'ebd19f1c549a1ad0.jpg': ['ebd19f9c549a12d0.jpg'],\n 'f8ab8597dc92d411.jpg': ['faca978fda90d011.jpg'],\n 'abf0b5819f028f0f.jpg': ['abf0b5a0df028b17.jpg'],\n 'dfbdc047068b063d.jpg': ['cebdc20f67838631.jpg'],\n 'f47ed2a14ac36998.jpg': ['f47ad2a2ca934d8a.jpg'],\n 'afe996c30a0e0d3e.jpg': ['afebd604ca06093f.jpg'],\n 'f0cb879e4a1b35c1.jpg': ['f0cb859beb1b30c1.jpg'],\n 'a8eb95c7d85e482a.jpg': ['a8e291e1da57596a.jpg'],\n 'a8e895f5aa54d2a5.jpg': ['a8e895e5aad453a5.jpg'],\n 'e0b4a7c3e88e7332.jpg': ['d0b487c76e8e3278.jpg'],\n 'bcdad04d9d24ab4c.jpg': ['b4dad04d896dcb25.jpg'],\n 'ec83c7c04a1e9db9.jpg': ['85897ea7f064b26a.jpg'],\n 'e385830ecacd2d9e.jpg': ['c335971e8acd609e.jpg'],\n 'bfa2d0bad0d22959.jpg': ['ba2092abd0faa95b.jpg'],\n 'c1583d30e0e1d5f7.jpg': ['c15d3d3ae123c547.jpg'],\n 'a3bb604d8a55995d.jpg': ['e7ecc583d571d202.jpg'],\n 'bb1891fa8c61e663.jpg': ['aa0e969518e15d3f.jpg'],\n 'e8a58ad4d9829f33.jpg': ['e8bd8ac4d982dd83.jpg'],\n 'e90e1acab7e5aa50.jpg': ['e90e1ac8b7e7aa50.jpg'],\n 'a5a687b1ca8a595e.jpg': ['a59497a1da8a49de.jpg'],\n 'af6292db1b611d98.jpg': ['a56292dadb618d95.jpg'],\n 'd70f6330fb5a090d.jpg': ['d70f2230fb5a093d.jpg'],\n 'bd84d4dacaf8cb40.jpg': ['f585d0d2caeaeb02.jpg'],\n 'b4ae98a3ad52c939.jpg': ['bc2a94a2af52da69.jpg'],\n '98864714f96a75d5.jpg': ['9886c704f93a79b3.jpg'],\n 'a5add7c29c8a9899.jpg': ['a4a5d6d69e9a981c.jpg'],\n 'e51690d8ad307dcb.jpg': ['eec64e8cf1c8d0c6.jpg'],\n 'f09697d6dada8805.jpg': ['fa9f95c2daca0905.jpg'],\n '95bc383078d273b3.jpg': ['953f3074707063b7.jpg'],\n 'b8a7cac8958b98d9.jpg': ['b88dc8ce8d8f0ad9.jpg'],\n '8f5263c61a8cdbe2.jpg': ['c9d63696629107ed.jpg'],\n 'd89062fbf4452a9e.jpg': ['d0806bfbf9452693.jpg'],\n 'ba99260ef46b18cb.jpg': ['b7b0edd58a858217.jpg'],\n 'ffe4c0878d50b4b8.jpg': ['b4c0c0b7cdd0b4f9.jpg'],\n 'dfcc6b28063a88fc.jpg': ['dfc86b180e3ac8f8.jpg'],\n 'faf5d5868a0ac9b0.jpg': ['aaf5d596828adac8.jpg'],\n 'e087f27e83f008e7.jpg': ['e087a7fe83c00acf.jpg'],\n 'faca978fda90d011.jpg': ['f8ab8597dc92d411.jpg'],\n 'a8e895e5aad453a5.jpg': ['a8e895f5aa54d2a5.jpg'],\n 'e8bd8ac4d982dd83.jpg': ['e8a58ad4d9829f33.jpg'],\n 'd89d9ede82743229.jpg': ['d89c9cda9274322f.jpg'],\n 'b490da8ddc8dc4ce.jpg': ['b594d286ddcfc0c8.jpg'],\n 'c53b9e87215c6ac5.jpg': ['c5339e87235c6ac5.jpg'],\n 'bd8088cdad5f65e0.jpg': ['f90738a0d2f2c1db.jpg'],\n 'c1a56069de75b899.jpg': ['f2c798246a0a9bdd.jpg'],\n 'e586c25e0f1c3c79.jpg': ['e787c35e0f0c34e8.jpg'],\n 'c46a85b46b643acf.jpg': ['c4e8843569247fcf.jpg'],\n 'aa142048eff78bf4.jpg': ['aa542048ebf78bf4.jpg'],\n 'a56292dadb618d95.jpg': ['af6292db1b611d98.jpg'],\n 'fc5fd028d2a988f4.jpg': ['ff3fd02a82ad8a84.jpg'],\n 'a4a5d6d69e9a981c.jpg': ['a5add7c29c8a9899.jpg'],\n 'b8828287cfdafc49.jpg': ['f8228796cfdae848.jpg'],\n 'e1e39f8809c7349d.jpg': ['e9c39fa809c7341d.jpg'],\n 'e83f715e56462a31.jpg': ['d0c766bc38e1978c.jpg'],\n '86f963a16e70345e.jpg': ['9fc321ce60a69f0b.jpg'],\n 'f6f11e0fe38a08c5.jpg': ['f6f1160be38a48d5.jpg'],\n 'b465cbd64c19725a.jpg': ['b4e4c9424f79335a.jpg'],\n '9fc321ce60a69f0b.jpg': ['86f963a16e70345e.jpg'],\n 'd0806bfbf9452693.jpg': ['d89062fbf4452a9e.jpg'],\n '94c83b2d5fad6924.jpg': ['90f99ee1c1132bb6.jpg'],\n 'e9e8ad35c5871d06.jpg': ['9ada65c5b0cc9a3c.jpg'],\n 'a9b2e4b09297bc2b.jpg': ['d1f01e8d8a73274d.jpg'],\n 'f0daf0c0828f9f0f.jpg': ['e0fec0c0838bcf8f.jpg'],\n 'd2d9c2bca7a72458.jpg': ['d1dec0973c2cf4a1.jpg'],\n 'fc01c79c9ee107e1.jpg': ['fc01c2dc8fa99de0.jpg'],\n '953f3074707063b7.jpg': ['95bc383078d273b3.jpg'],\n 'e03496c3d9ddc1c9.jpg': ['a03497c399c9c7d9.jpg'],\n 'f2c798246a0a9bdd.jpg': ['c1a56069de75b899.jpg'],\n 'e8e5c38f8f18a05d.jpg': ['e8a5c38f9f9828d8.jpg'],\n 'e087a7fe83c00acf.jpg': ['e087f27e83f008e7.jpg'],\n '892ec13c8b783fca.jpg': ['a91ec1bc0b783f43.jpg'],\n '83e3d2369933c5d2.jpg': ['83e3d2329933c5f2.jpg'],\n '90a5341f227ce5f9.jpg': ['f3dc17a2b20d48ec.jpg'],\n 'c335971e8acd609e.jpg': ['e385830ecacd2d9e.jpg'],\n 'e9c39fa809c7341d.jpg': ['e1e39f8809c7349d.jpg'],\n 'b80898e11fa17f1f.jpg': ['b80898e11f81ff1f.jpg'],\n 'e22e9a365cd1ea58.jpg': ['e22e8a365cd1ea5c.jpg'],\n 'fa9f95c2daca0905.jpg': ['f09697d6dada8805.jpg'],\n 'f4a0c077c79a8cd9.jpg': ['f5e4c2f7849a8c91.jpg'],\n 'b5ec52491e3349ce.jpg': ['a7e613159fb22859.jpg'],\n 'd58ad09e28df7825.jpg': ['bd0a909f69c97865.jpg'],\n 'e8a5c38f9f9828d8.jpg': ['e8e5c38f8f18a05d.jpg'],\n 'f7d233d11aa8496c.jpg': ['94b42695f24dad36.jpg'],\n 'c5aff2e545d0a129.jpg': ['dbee22785d9285c4.jpg'],\n 'f47ad2a2ca934d8a.jpg': ['f47ed2a14ac36998.jpg'],\n 'fcc4c8bf387015c5.jpg': ['fcc4c8fe30300dcf.jpg'],\n 'e5f27342ce1cb9a0.jpg': ['ca782c131f5ce0fc.jpg'],\n 'e7ecc583d571d202.jpg': ['a3bb604d8a55995d.jpg'],\n '83e3d2329933c5f2.jpg': ['83e3d2369933c5d2.jpg'],\n 'a9b095bedada5805.jpg': ['a1a1d5a4cada586f.jpg'],\n 'b88dc8ce8d8f0ad9.jpg': ['b8a7cac8958b98d9.jpg'],\n 'e787c35e0f0c34e8.jpg': ['e586c25e0f1c3c79.jpg'],\n 'abf0b5a0df028b17.jpg': ['abf0b5819f028f0f.jpg'],\n 'afebd604ca06093f.jpg': ['afe996c30a0e0d3e.jpg'],\n 'f3dc17a2b20d48ec.jpg': ['90a5341f227ce5f9.jpg'],\n 'ab83c0d147c3db66.jpg': ['ab83c05147c3cfe6.jpg'],\n 'e8210ddefc024fec.jpg': ['e8212ddeec024fec.jpg'],\n 'a8aa915abe68b565.jpg': ['aaa3955ad262b56c.jpg'],\n 'aaa3955ad262b56c.jpg': ['a8aa915abe68b565.jpg'],\n '848d9f9e82d18e5e.jpg': ['949b6595a5b8ca3c.jpg'],\n 'fc01c2dc8fa99de0.jpg': ['fc01c79c9ee107e1.jpg'],\n 'd3945c098edc9dd1.jpg': ['dea33c5c0347aa4d.jpg'],\n 'ba2092abd0faa95b.jpg': ['bfa2d0bad0d22959.jpg'],\n 'f891c19fbf301233.jpg': ['f891839e8f342e72.jpg'],\n '90f99ee1c1132bb6.jpg': ['94c83b2d5fad6924.jpg'],\n 'a03497c399c9c7d9.jpg': ['e03496c3d9ddc1c9.jpg'],\n 'b594d286ddcfc0c8.jpg': ['b490da8ddc8dc4ce.jpg'],\n '85897ea7f064b26a.jpg': ['ec83c7c04a1e9db9.jpg'],\n 'd03fcce03447353e.jpg': ['c03ec9c12c87377e.jpg'],\n 'ab91d1a9d4b49a6c.jpg': ['abb0d5a9d4b4985c.jpg'],\n 'b5369279d9c1c949.jpg': ['f474927ad9c0e88d.jpg'],\n '949b6595a5b8ca3c.jpg': ['848d9f9e82d18e5e.jpg'],\n 'a91ec1bc0b783f43.jpg': ['892ec13c8b783fca.jpg'],\n 'ebd19f9c549a12d0.jpg': ['ebd19f1c549a1ad0.jpg'],\n 'ab83c05147c3cfe6.jpg': ['ab83c0d147c3db66.jpg'],\n 'c2ccaf86b8d9549a.jpg': ['c6e4efce38d81419.jpg'],\n '94b42695f24dad36.jpg': ['f7d233d11aa8496c.jpg'],\n 'aa542048ebf78bf4.jpg': ['aa142048eff78bf4.jpg'],\n 'a065c2f4ca986fdc.jpg': ['e0f4c0f48aca6b9e.jpg'],\n 'e13fb40087e0fec6.jpg': ['e0bbf620c7c0ce87.jpg'],\n 'cebdc20f67838631.jpg': ['dfbdc047068b063d.jpg'],\n 'aa0e969518e15d3f.jpg': ['bb1891fa8c61e663.jpg'],\n 'e13be7f1105e8f80.jpg': ['e11be7f110de8f80.jpg'],\n 'e0bbf620c7c0ce87.jpg': ['e13fb40087e0fec6.jpg'],\n 'c03ec9c12c87377e.jpg': ['d03fcce03447353e.jpg'],\n 'e22e8a365cd1ea5c.jpg': ['e22e9a365cd1ea58.jpg'],\n 'b4dad04d896dcb25.jpg': ['bcdad04d9d24ab4c.jpg'],\n 'b4c0c0b7cdd0b4f9.jpg': ['ffe4c0878d50b4b8.jpg']}",
      "votes": null
    },
    {
      "id": "1263765",
      "postDate": "04/05/2021 16:59:20",
      "content": "<p>Hello!</p>\n<ul>\n<li>I've done a quick check revealing that all my duplicates (let's say <code>b=set(...)</code>) belong to your duplicates (let's say <code>a=set(...)</code>):</li>\n</ul>\n<pre><code>print(len(a))\nprint(len(b))\nprint(len(a.intersection(a))\n&gt;&gt;&gt; 106\n&gt;&gt;&gt; 50\n&gt;&gt;&gt; 50\n</code></pre>\n<p>which is great.</p>\n<ul>\n<li>As for the remainder of 56 duplicate pairs (and even one triplet), most of them <strong>aren't duplicates</strong> and even do not look alike, e.g.:<br>\n<img src=\"https://drive.google.com/uc?id=1O3Ra6ui_lkc-yW98n-6fxvRg4d4anFho\"><br>\n<img src=\"https://drive.google.com/uc?id=1BK8-RyRh2GTnyhZqpXnFg0XT32FhhZdN\"><br>\nthat's why I suggest using different types of hash or setting a higher threshold and <strong>always manually check the results</strong>. I've found that only 6 pairs of these are actual duplicates:</li>\n</ul>\n<pre><code>('8dbeda49894d522e.jpg', 'afbe5641896d522a.jpg')\n('af6292db1b611d98.jpg', 'a56292dadb618d95.jpg')\n('abf0b5a0df028b17.jpg', 'abf0b5819f028f0f.jpg')\n('e385830ecacd2d9e.jpg', 'c335971e8acd609e.jpg')\n('cebdc20f67838631.jpg', 'dfbdc047068b063d.jpg')\n('f392f11919991cea.jpg', 'f196f11a99d91ce0.jpg')\n</code></pre>",
      "rawMarkdown": "Hello!\n\n* I've done a quick check revealing that all my duplicates (let's say `b=set(...)`) belong to your duplicates (let's say `a=set(...)`):\n```\nprint(len(a))\nprint(len(b))\nprint(len(a.intersection(a))\n>>> 106\n>>> 50\n>>> 50\n```\nwhich is great.\n* As for the remainder of 56 duplicate pairs (and even one triplet), most of them **aren't duplicates** and even do not look alike, e.g.:\n<img src='https://drive.google.com/uc?id=1O3Ra6ui_lkc-yW98n-6fxvRg4d4anFho' />\n<img src='https://drive.google.com/uc?id=1BK8-RyRh2GTnyhZqpXnFg0XT32FhhZdN' />\nthat's why I suggest using different types of hash or setting a higher threshold and **always manually check the results**. I've found that only 6 pairs of these are actual duplicates:\n```\n('8dbeda49894d522e.jpg', 'afbe5641896d522a.jpg')\n('af6292db1b611d98.jpg', 'a56292dadb618d95.jpg')\n('abf0b5a0df028b17.jpg', 'abf0b5819f028f0f.jpg')\n('e385830ecacd2d9e.jpg', 'c335971e8acd609e.jpg')\n('cebdc20f67838631.jpg', 'dfbdc047068b063d.jpg')\n('f392f11919991cea.jpg', 'f196f11a99d91ce0.jpg')\n```",
      "votes": null
    },
    {
      "id": "1264103",
      "postDate": "04/05/2021 21:51:37",
      "content": "<p>Good catch! Finding duplicates requires quite a lot of manual work.</p>",
      "rawMarkdown": "Good catch! Finding duplicates requires quite a lot of manual work.",
      "votes": null
    },
    {
      "id": "1264105",
      "postDate": "04/05/2021 21:54:22",
      "content": "<p><a href=\"https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/230853\" target=\"_blank\">Here</a> is an update on more duplicates findings with methods described in details and tried in this <a href=\"https://www.kaggle.com/datasciencegeek/detect-duplicates-via-image-embeddings\" target=\"_blank\">notebook</a>.</p>",
      "rawMarkdown": "[Here](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/230853) is an update on more duplicates findings with methods described in details and tried in this [notebook](https://www.kaggle.com/datasciencegeek/detect-duplicates-via-image-embeddings).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1249001,
      "author_name": "datasciencegeek",
      "author_url": "",
      "post_date": "03/23/2021 02:22:41",
      "content": "<p>Let's share all duplicate images filenames to make sure to get them all. I found 54 in this <a href=\"https://www.kaggle.com/datasciencegeek/eda-plantpathology2021-fgvc8\" target=\"_blank\">notebook</a> . Here is a list of filenames:</p>\n<p>[9ada65c5b0cc9a3c.jpg, e9e8ad35c5871d06.jpg]<br>\n[94b42695f24dad36.jpg, f7d233d11aa8496c.jpg]<br>\n[d64460a3b19f1cfc.jpg, f4cba4a7294d6585.jpg]<br>\n[c5aff2e545d0a129.jpg, dbee22785d9285c4.jpg]<br>\n[ca782c131f5ce0fc.jpg, e5f27342ce1cb9a0.jpg]<br>\n[848d9f9e82d18e5e.jpg, 949b6595a5b8ca3c.jpg]<br>\n[d3945c098edc9dd1.jpg, dea33c5c0347aa4d.jpg]<br>\n[85897ea7f064b26a.jpg, ec83c7c04a1e9db9.jpg]<br>\n[8f5263c61a8cdbe2.jpg, c9d63696629107ed.jpg]<br>\n[b7b0edd58a858217.jpg, ba99260ef46b18cb.jpg]<br>\n[c1a56069de75b899.jpg, f2c798246a0a9bdd.jpg]<br>\n[965295a64a97859f.jpg, ae5074cfc9183763.jpg]<br>\n[aa0e969518e15d3f.jpg, bb1891fa8c61e663.jpg]<br>\n[a3bb604d8a55995d.jpg, e7ecc583d571d202.jpg]<br>\n[86f963a16e70345e.jpg, 9fc321ce60a69f0b.jpg]<br>\n[d0c766bc38e1978c.jpg, e83f715e56462a31.jpg]<br>\n[9fad869f21b5b240.jpg, d7ec32d72c84dc18.jpg]<br>\n[bd8088cdad5f65e0.jpg, f90738a0d2f2c1db.jpg]<br>\n[90f99ee1c1132bb6.jpg, 94c83b2d5fad6924.jpg]<br>\n[90a5341f227ce5f9.jpg, f3dc17a2b20d48ec.jpg]<br>\n[8c0731fff680c51e.jpg, df90901ccb664de6.jpg]<br>\n[c1753cad2f31d492.jpg, d2266ac99461e9db.jpg]<br>\n[a9b2e4b09297bc2b.jpg, d1f01e8d8a73274d.jpg]<br>\n[a7e613159fb22859.jpg, b5ec52491e3349ce.jpg]<br>\n[82ba3e933947b097.jpg, cdf892b5b4865859.jpg]<br>\n[e51690d8ad307dcb.jpg, eec64e8cf1c8d0c6.jpg]<br>\n[d1dec0973c2cf4a1.jpg, d2d9c2bca7a72458.jpg]</p>",
      "votes": null,
      "replies": [
        {
          "id": 1249095,
          "author_name": "nickuzmenkov",
          "author_url": "",
          "post_date": "03/23/2021 05:21:16",
          "content": "<p>Nice work! </p>\n<p>Actually, I meant that I found 50 <strong>pairs</strong> of duplicates, i.e. 100 duplicated images total. I've done a quick check with your duplicates (let's say <code>a = set(...)</code>) and my duplicates (let's say <code>b=set(...)</code>) showing that all your duplicates <code>a</code> belong to my duplicates <code>b</code>:</p>\n<pre><code>print(len(a))\nprint(len(b))\nprint(len(b.intersection(a)))\n&gt;&gt;&gt; 27\n&gt;&gt;&gt; 50\n&gt;&gt;&gt; 27\n</code></pre>\n<p>And that's great news that two different methods give the same results.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1249850,
          "author_name": "datasciencegeek",
          "author_url": "",
          "post_date": "03/23/2021 15:10:48",
          "content": "<p>Gotcha. Thank you for sharing! I have not heard of the method you used to detect duplicates and came up with an add-hoc method myself, nice to learn new approaches.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1250344,
      "author_name": "brendanartley",
      "author_url": "",
      "post_date": "03/24/2021 01:16:17",
      "content": "<p>Thanks for sharing these tags, much appreciated!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1258967,
      "author_name": "kingofarmy",
      "author_url": "",
      "post_date": "04/01/2021 04:06:50",
      "content": "<p>There are around 104 pairs of duplicate images. I found this using <code>Imagededup</code> library. I often use this library to check the duplicates. I used Perceptual hashing method to find duplicates.</p>\n<p>The link about <a href=\"http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html\" target=\"_blank\">Perceptual Hashing</a></p>\n<p>The embeddings are as follows:<br>\n{'bd0a909f69c97865.jpg': ['d58ad09e28df7825.jpg'],<br>\n 'eec64e8cf1c8d0c6.jpg': ['e51690d8ad307dcb.jpg'],<br>\n 'd2266ac99461e9db.jpg': ['c1753cad2f31d492.jpg'],<br>\n 'd64460a3b19f1cfc.jpg': ['f4cba4a7294d6585.jpg'],<br>\n 'a8e291e1da57596a.jpg': ['a8eb95c7d85e482a.jpg'],<br>\n 'df90901ccb664de6.jpg': ['8c0731fff680c51e.jpg'],<br>\n 'cc73e025cc36d31e.jpg': ['cc73e005cc36f31e.jpg'],<br>\n 'a484c39f9fd861d8.jpg': ['b0e0cf9f9e98e188.jpg'],<br>\n 'af08a3a0ff35c0c7.jpg': ['fe0883b0bb75a0cd.jpg'],<br>\n 'cc73e005cc36f31e.jpg': ['cc73e025cc36d31e.jpg'],<br>\n 'f5e4c2f7849a8c91.jpg': ['f4a0c077c79a8cd9.jpg'],<br>\n '9ada65c5b0cc9a3c.jpg': ['e9e8ad35c5871d06.jpg'],<br>\n 'd0b487c76e8e3278.jpg': ['e0b4a7c3e88e7332.jpg'],<br>\n 'f585d0d2caeaeb02.jpg': ['bd84d4dacaf8cb40.jpg'],<br>\n 'fe0883b0bb75a0cd.jpg': ['af08a3a0ff35c0c7.jpg'],<br>\n 'b4e4c9424f79335a.jpg': ['b465cbd64c19725a.jpg'],<br>\n 'b7b0edd58a858217.jpg': ['ba99260ef46b18cb.jpg'],<br>\n 'e90e1ac8b7e7aa50.jpg': ['e90e1acab7e5aa50.jpg'],<br>\n 'e0f4c0f48aca6b9e.jpg': ['a065c2f4ca986fdc.jpg'],<br>\n '9c968ac989ed332e.jpg': ['8dbeda49894d522e.jpg'],<br>\n 'aba437d04a0997dd.jpg': ['afa074d04a8d25df.jpg'],<br>\n 'ca782c131f5ce0fc.jpg': ['e5f27342ce1cb9a0.jpg'],<br>\n 'e0fec0c0838bcf8f.jpg': ['f0daf0c0828f9f0f.jpg'],<br>\n 'bd92c0694632d73b.jpg': ['bf9ac0e9c6628633.jpg'],<br>\n 'c738d13e7146798c.jpg': ['c738d13e73467988.jpg'],<br>\n 'ff88c51c70b0e2d6.jpg': ['ff08c71cf0b42ba2.jpg'],<br>\n '8c0731fff680c51e.jpg': ['df90901ccb664de6.jpg'],<br>\n 'f196f11a99d91ce0.jpg': ['f392f11919991cea.jpg'],<br>\n 'dbee22785d9285c4.jpg': ['c5aff2e545d0a129.jpg'],<br>\n 'abb0d5a9d4b4985c.jpg': ['ab91d1a9d4b49a6c.jpg'],<br>\n 'aaf5d596828adac8.jpg': ['faf5d5868a0ac9b0.jpg'],<br>\n 'd0c766bc38e1978c.jpg': ['e83f715e56462a31.jpg'],<br>\n 'f392f11919991cea.jpg': ['f196f11a99d91ce0.jpg'],<br>\n 'd89c9cda9274322f.jpg': ['d89d9ede82743229.jpg'],<br>\n '9fad869f21b5b240.jpg': ['d7ec32d72c84dc18.jpg'],<br>\n 'e8212ddeec024fec.jpg': ['e8210ddefc024fec.jpg'],<br>\n 'fcc4c8fe30300dcf.jpg': ['fcc4c8bf387015c5.jpg'],<br>\n 'c1753cad2f31d492.jpg': ['d2266ac99461e9db.jpg'],<br>\n 'b53c9ab0c9e4f0c9.jpg': ['f528cbb0c5c1f0cd.jpg'],<br>\n 'ba2e601c8979afe2.jpg': ['9a2e269c1969bfe0.jpg'],<br>\n 'e11be7f110de8f80.jpg': ['e13be7f1105e8f80.jpg'],<br>\n 'f891839e8f342e72.jpg': ['f891c19fbf301233.jpg'],<br>\n 'afbe5641896d522a.jpg': ['8dbeda49894d522e.jpg'],<br>\n '9a2e269c1969bfe0.jpg': ['ba2e601c8979afe2.jpg'],<br>\n 'a59497a1da8a49de.jpg': ['a5a687b1ca8a595e.jpg'],<br>\n 'c15d3d3ae123c547.jpg': ['c1583d30e0e1d5f7.jpg'],<br>\n 'dea33c5c0347aa4d.jpg': ['d3945c098edc9dd1.jpg'],<br>\n '82ba3e933947b097.jpg': ['cdf892b5b4865859.jpg'],<br>\n 'a1a1d5a4cada586f.jpg': ['a9b095bedada5805.jpg'],<br>\n 'ae5074cfc9183763.jpg': ['965295a64a97859f.jpg'],<br>\n 'd1f01e8d8a73274d.jpg': ['a9b2e4b09297bc2b.jpg'],<br>\n 'f0cb859beb1b30c1.jpg': ['f0cb879e4a1b35c1.jpg'],<br>\n 'a7e613159fb22859.jpg': ['b5ec52491e3349ce.jpg'],<br>\n 'bf9ac0e9c6628633.jpg': ['bd92c0694632d73b.jpg'],<br>\n 'cdf892b5b4865859.jpg': ['82ba3e933947b097.jpg'],<br>\n 'd70f2230fb5a093d.jpg': ['d70f6330fb5a090d.jpg'],<br>\n 'ff3fd02a82ad8a84.jpg': ['fc5fd028d2a988f4.jpg'],<br>\n 'c6e4efce38d81419.jpg': ['c2ccaf86b8d9549a.jpg'],<br>\n 'dfc86b180e3ac8f8.jpg': ['dfcc6b28063a88fc.jpg'],<br>\n 'f474927ad9c0e88d.jpg': ['b5369279d9c1c949.jpg'],<br>\n 'bc2a94a2af52da69.jpg': ['b4ae98a3ad52c939.jpg'],<br>\n 'f90738a0d2f2c1db.jpg': ['bd8088cdad5f65e0.jpg'],<br>\n 'c738d13e73467988.jpg': ['c738d13e7146798c.jpg'],<br>\n 'c9d63696629107ed.jpg': ['8f5263c61a8cdbe2.jpg'],<br>\n 'c5339e87235c6ac5.jpg': ['c53b9e87215c6ac5.jpg'],<br>\n 'b0e0cf9f9e98e188.jpg': ['a484c39f9fd861d8.jpg'],<br>\n 'b80898e11f81ff1f.jpg': ['b80898e11fa17f1f.jpg'],<br>\n 'aaac95caca4b8d2d.jpg': ['a8bdb4cac84b98cd.jpg'],<br>\n 'ff08c71cf0b42ba2.jpg': ['ff88c51c70b0e2d6.jpg'],<br>\n '8dbeda49894d522e.jpg': ['afbe5641896d522a.jpg', '9c968ac989ed332e.jpg'],<br>\n 'd7ec32d72c84dc18.jpg': ['9fad869f21b5b240.jpg'],<br>\n 'f6f1160be38a48d5.jpg': ['f6f11e0fe38a08c5.jpg'],<br>\n 'd1dec0973c2cf4a1.jpg': ['d2d9c2bca7a72458.jpg'],<br>\n '9886c704f93a79b3.jpg': ['98864714f96a75d5.jpg'],<br>\n 'a8bdb4cac84b98cd.jpg': ['aaac95caca4b8d2d.jpg'],<br>\n 'f8228796cfdae848.jpg': ['b8828287cfdafc49.jpg'],<br>\n 'f528cbb0c5c1f0cd.jpg': ['b53c9ab0c9e4f0c9.jpg'],<br>\n '965295a64a97859f.jpg': ['ae5074cfc9183763.jpg'],<br>\n 'afa074d04a8d25df.jpg': ['aba437d04a0997dd.jpg'],<br>\n 'c4e8843569247fcf.jpg': ['c46a85b46b643acf.jpg'],<br>\n 'f4cba4a7294d6585.jpg': ['d64460a3b19f1cfc.jpg'],<br>\n 'ebd19f1c549a1ad0.jpg': ['ebd19f9c549a12d0.jpg'],<br>\n 'f8ab8597dc92d411.jpg': ['faca978fda90d011.jpg'],<br>\n 'abf0b5819f028f0f.jpg': ['abf0b5a0df028b17.jpg'],<br>\n 'dfbdc047068b063d.jpg': ['cebdc20f67838631.jpg'],<br>\n 'f47ed2a14ac36998.jpg': ['f47ad2a2ca934d8a.jpg'],<br>\n 'afe996c30a0e0d3e.jpg': ['afebd604ca06093f.jpg'],<br>\n 'f0cb879e4a1b35c1.jpg': ['f0cb859beb1b30c1.jpg'],<br>\n 'a8eb95c7d85e482a.jpg': ['a8e291e1da57596a.jpg'],<br>\n 'a8e895f5aa54d2a5.jpg': ['a8e895e5aad453a5.jpg'],<br>\n 'e0b4a7c3e88e7332.jpg': ['d0b487c76e8e3278.jpg'],<br>\n 'bcdad04d9d24ab4c.jpg': ['b4dad04d896dcb25.jpg'],<br>\n 'ec83c7c04a1e9db9.jpg': ['85897ea7f064b26a.jpg'],<br>\n 'e385830ecacd2d9e.jpg': ['c335971e8acd609e.jpg'],<br>\n 'bfa2d0bad0d22959.jpg': ['ba2092abd0faa95b.jpg'],<br>\n 'c1583d30e0e1d5f7.jpg': ['c15d3d3ae123c547.jpg'],<br>\n 'a3bb604d8a55995d.jpg': ['e7ecc583d571d202.jpg'],<br>\n 'bb1891fa8c61e663.jpg': ['aa0e969518e15d3f.jpg'],<br>\n 'e8a58ad4d9829f33.jpg': ['e8bd8ac4d982dd83.jpg'],<br>\n 'e90e1acab7e5aa50.jpg': ['e90e1ac8b7e7aa50.jpg'],<br>\n 'a5a687b1ca8a595e.jpg': ['a59497a1da8a49de.jpg'],<br>\n 'af6292db1b611d98.jpg': ['a56292dadb618d95.jpg'],<br>\n 'd70f6330fb5a090d.jpg': ['d70f2230fb5a093d.jpg'],<br>\n 'bd84d4dacaf8cb40.jpg': ['f585d0d2caeaeb02.jpg'],<br>\n 'b4ae98a3ad52c939.jpg': ['bc2a94a2af52da69.jpg'],<br>\n '98864714f96a75d5.jpg': ['9886c704f93a79b3.jpg'],<br>\n 'a5add7c29c8a9899.jpg': ['a4a5d6d69e9a981c.jpg'],<br>\n 'e51690d8ad307dcb.jpg': ['eec64e8cf1c8d0c6.jpg'],<br>\n 'f09697d6dada8805.jpg': ['fa9f95c2daca0905.jpg'],<br>\n '95bc383078d273b3.jpg': ['953f3074707063b7.jpg'],<br>\n 'b8a7cac8958b98d9.jpg': ['b88dc8ce8d8f0ad9.jpg'],<br>\n '8f5263c61a8cdbe2.jpg': ['c9d63696629107ed.jpg'],<br>\n 'd89062fbf4452a9e.jpg': ['d0806bfbf9452693.jpg'],<br>\n 'ba99260ef46b18cb.jpg': ['b7b0edd58a858217.jpg'],<br>\n 'ffe4c0878d50b4b8.jpg': ['b4c0c0b7cdd0b4f9.jpg'],<br>\n 'dfcc6b28063a88fc.jpg': ['dfc86b180e3ac8f8.jpg'],<br>\n 'faf5d5868a0ac9b0.jpg': ['aaf5d596828adac8.jpg'],<br>\n 'e087f27e83f008e7.jpg': ['e087a7fe83c00acf.jpg'],<br>\n 'faca978fda90d011.jpg': ['f8ab8597dc92d411.jpg'],<br>\n 'a8e895e5aad453a5.jpg': ['a8e895f5aa54d2a5.jpg'],<br>\n 'e8bd8ac4d982dd83.jpg': ['e8a58ad4d9829f33.jpg'],<br>\n 'd89d9ede82743229.jpg': ['d89c9cda9274322f.jpg'],<br>\n 'b490da8ddc8dc4ce.jpg': ['b594d286ddcfc0c8.jpg'],<br>\n 'c53b9e87215c6ac5.jpg': ['c5339e87235c6ac5.jpg'],<br>\n 'bd8088cdad5f65e0.jpg': ['f90738a0d2f2c1db.jpg'],<br>\n 'c1a56069de75b899.jpg': ['f2c798246a0a9bdd.jpg'],<br>\n 'e586c25e0f1c3c79.jpg': ['e787c35e0f0c34e8.jpg'],<br>\n 'c46a85b46b643acf.jpg': ['c4e8843569247fcf.jpg'],<br>\n 'aa142048eff78bf4.jpg': ['aa542048ebf78bf4.jpg'],<br>\n 'a56292dadb618d95.jpg': ['af6292db1b611d98.jpg'],<br>\n 'fc5fd028d2a988f4.jpg': ['ff3fd02a82ad8a84.jpg'],<br>\n 'a4a5d6d69e9a981c.jpg': ['a5add7c29c8a9899.jpg'],<br>\n 'b8828287cfdafc49.jpg': ['f8228796cfdae848.jpg'],<br>\n 'e1e39f8809c7349d.jpg': ['e9c39fa809c7341d.jpg'],<br>\n 'e83f715e56462a31.jpg': ['d0c766bc38e1978c.jpg'],<br>\n '86f963a16e70345e.jpg': ['9fc321ce60a69f0b.jpg'],<br>\n 'f6f11e0fe38a08c5.jpg': ['f6f1160be38a48d5.jpg'],<br>\n 'b465cbd64c19725a.jpg': ['b4e4c9424f79335a.jpg'],<br>\n '9fc321ce60a69f0b.jpg': ['86f963a16e70345e.jpg'],<br>\n 'd0806bfbf9452693.jpg': ['d89062fbf4452a9e.jpg'],<br>\n '94c83b2d5fad6924.jpg': ['90f99ee1c1132bb6.jpg'],<br>\n 'e9e8ad35c5871d06.jpg': ['9ada65c5b0cc9a3c.jpg'],<br>\n 'a9b2e4b09297bc2b.jpg': ['d1f01e8d8a73274d.jpg'],<br>\n 'f0daf0c0828f9f0f.jpg': ['e0fec0c0838bcf8f.jpg'],<br>\n 'd2d9c2bca7a72458.jpg': ['d1dec0973c2cf4a1.jpg'],<br>\n 'fc01c79c9ee107e1.jpg': ['fc01c2dc8fa99de0.jpg'],<br>\n '953f3074707063b7.jpg': ['95bc383078d273b3.jpg'],<br>\n 'e03496c3d9ddc1c9.jpg': ['a03497c399c9c7d9.jpg'],<br>\n 'f2c798246a0a9bdd.jpg': ['c1a56069de75b899.jpg'],<br>\n 'e8e5c38f8f18a05d.jpg': ['e8a5c38f9f9828d8.jpg'],<br>\n 'e087a7fe83c00acf.jpg': ['e087f27e83f008e7.jpg'],<br>\n '892ec13c8b783fca.jpg': ['a91ec1bc0b783f43.jpg'],<br>\n '83e3d2369933c5d2.jpg': ['83e3d2329933c5f2.jpg'],<br>\n '90a5341f227ce5f9.jpg': ['f3dc17a2b20d48ec.jpg'],<br>\n 'c335971e8acd609e.jpg': ['e385830ecacd2d9e.jpg'],<br>\n 'e9c39fa809c7341d.jpg': ['e1e39f8809c7349d.jpg'],<br>\n 'b80898e11fa17f1f.jpg': ['b80898e11f81ff1f.jpg'],<br>\n 'e22e9a365cd1ea58.jpg': ['e22e8a365cd1ea5c.jpg'],<br>\n 'fa9f95c2daca0905.jpg': ['f09697d6dada8805.jpg'],<br>\n 'f4a0c077c79a8cd9.jpg': ['f5e4c2f7849a8c91.jpg'],<br>\n 'b5ec52491e3349ce.jpg': ['a7e613159fb22859.jpg'],<br>\n 'd58ad09e28df7825.jpg': ['bd0a909f69c97865.jpg'],<br>\n 'e8a5c38f9f9828d8.jpg': ['e8e5c38f8f18a05d.jpg'],<br>\n 'f7d233d11aa8496c.jpg': ['94b42695f24dad36.jpg'],<br>\n 'c5aff2e545d0a129.jpg': ['dbee22785d9285c4.jpg'],<br>\n 'f47ad2a2ca934d8a.jpg': ['f47ed2a14ac36998.jpg'],<br>\n 'fcc4c8bf387015c5.jpg': ['fcc4c8fe30300dcf.jpg'],<br>\n 'e5f27342ce1cb9a0.jpg': ['ca782c131f5ce0fc.jpg'],<br>\n 'e7ecc583d571d202.jpg': ['a3bb604d8a55995d.jpg'],<br>\n '83e3d2329933c5f2.jpg': ['83e3d2369933c5d2.jpg'],<br>\n 'a9b095bedada5805.jpg': ['a1a1d5a4cada586f.jpg'],<br>\n 'b88dc8ce8d8f0ad9.jpg': ['b8a7cac8958b98d9.jpg'],<br>\n 'e787c35e0f0c34e8.jpg': ['e586c25e0f1c3c79.jpg'],<br>\n 'abf0b5a0df028b17.jpg': ['abf0b5819f028f0f.jpg'],<br>\n 'afebd604ca06093f.jpg': ['afe996c30a0e0d3e.jpg'],<br>\n 'f3dc17a2b20d48ec.jpg': ['90a5341f227ce5f9.jpg'],<br>\n 'ab83c0d147c3db66.jpg': ['ab83c05147c3cfe6.jpg'],<br>\n 'e8210ddefc024fec.jpg': ['e8212ddeec024fec.jpg'],<br>\n 'a8aa915abe68b565.jpg': ['aaa3955ad262b56c.jpg'],<br>\n 'aaa3955ad262b56c.jpg': ['a8aa915abe68b565.jpg'],<br>\n '848d9f9e82d18e5e.jpg': ['949b6595a5b8ca3c.jpg'],<br>\n 'fc01c2dc8fa99de0.jpg': ['fc01c79c9ee107e1.jpg'],<br>\n 'd3945c098edc9dd1.jpg': ['dea33c5c0347aa4d.jpg'],<br>\n 'ba2092abd0faa95b.jpg': ['bfa2d0bad0d22959.jpg'],<br>\n 'f891c19fbf301233.jpg': ['f891839e8f342e72.jpg'],<br>\n '90f99ee1c1132bb6.jpg': ['94c83b2d5fad6924.jpg'],<br>\n 'a03497c399c9c7d9.jpg': ['e03496c3d9ddc1c9.jpg'],<br>\n 'b594d286ddcfc0c8.jpg': ['b490da8ddc8dc4ce.jpg'],<br>\n '85897ea7f064b26a.jpg': ['ec83c7c04a1e9db9.jpg'],<br>\n 'd03fcce03447353e.jpg': ['c03ec9c12c87377e.jpg'],<br>\n 'ab91d1a9d4b49a6c.jpg': ['abb0d5a9d4b4985c.jpg'],<br>\n 'b5369279d9c1c949.jpg': ['f474927ad9c0e88d.jpg'],<br>\n '949b6595a5b8ca3c.jpg': ['848d9f9e82d18e5e.jpg'],<br>\n 'a91ec1bc0b783f43.jpg': ['892ec13c8b783fca.jpg'],<br>\n 'ebd19f9c549a12d0.jpg': ['ebd19f1c549a1ad0.jpg'],<br>\n 'ab83c05147c3cfe6.jpg': ['ab83c0d147c3db66.jpg'],<br>\n 'c2ccaf86b8d9549a.jpg': ['c6e4efce38d81419.jpg'],<br>\n '94b42695f24dad36.jpg': ['f7d233d11aa8496c.jpg'],<br>\n 'aa542048ebf78bf4.jpg': ['aa142048eff78bf4.jpg'],<br>\n 'a065c2f4ca986fdc.jpg': ['e0f4c0f48aca6b9e.jpg'],<br>\n 'e13fb40087e0fec6.jpg': ['e0bbf620c7c0ce87.jpg'],<br>\n 'cebdc20f67838631.jpg': ['dfbdc047068b063d.jpg'],<br>\n 'aa0e969518e15d3f.jpg': ['bb1891fa8c61e663.jpg'],<br>\n 'e13be7f1105e8f80.jpg': ['e11be7f110de8f80.jpg'],<br>\n 'e0bbf620c7c0ce87.jpg': ['e13fb40087e0fec6.jpg'],<br>\n 'c03ec9c12c87377e.jpg': ['d03fcce03447353e.jpg'],<br>\n 'e22e8a365cd1ea5c.jpg': ['e22e9a365cd1ea58.jpg'],<br>\n 'b4dad04d896dcb25.jpg': ['bcdad04d9d24ab4c.jpg'],<br>\n 'b4c0c0b7cdd0b4f9.jpg': ['ffe4c0878d50b4b8.jpg']}</p>",
      "votes": null,
      "replies": [
        {
          "id": 1263765,
          "author_name": "nickuzmenkov",
          "author_url": "",
          "post_date": "04/05/2021 16:59:20",
          "content": "<p>Hello!</p>\n<ul>\n<li>I've done a quick check revealing that all my duplicates (let's say <code>b=set(...)</code>) belong to your duplicates (let's say <code>a=set(...)</code>):</li>\n</ul>\n<pre><code>print(len(a))\nprint(len(b))\nprint(len(a.intersection(a))\n&gt;&gt;&gt; 106\n&gt;&gt;&gt; 50\n&gt;&gt;&gt; 50\n</code></pre>\n<p>which is great.</p>\n<ul>\n<li>As for the remainder of 56 duplicate pairs (and even one triplet), most of them <strong>aren't duplicates</strong> and even do not look alike, e.g.:<br>\n<img src=\"https://drive.google.com/uc?id=1O3Ra6ui_lkc-yW98n-6fxvRg4d4anFho\"><br>\n<img src=\"https://drive.google.com/uc?id=1BK8-RyRh2GTnyhZqpXnFg0XT32FhhZdN\"><br>\nthat's why I suggest using different types of hash or setting a higher threshold and <strong>always manually check the results</strong>. I've found that only 6 pairs of these are actual duplicates:</li>\n</ul>\n<pre><code>('8dbeda49894d522e.jpg', 'afbe5641896d522a.jpg')\n('af6292db1b611d98.jpg', 'a56292dadb618d95.jpg')\n('abf0b5a0df028b17.jpg', 'abf0b5819f028f0f.jpg')\n('e385830ecacd2d9e.jpg', 'c335971e8acd609e.jpg')\n('cebdc20f67838631.jpg', 'dfbdc047068b063d.jpg')\n('f392f11919991cea.jpg', 'f196f11a99d91ce0.jpg')\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1264103,
          "author_name": "datasciencegeek",
          "author_url": "",
          "post_date": "04/05/2021 21:51:37",
          "content": "<p>Good catch! Finding duplicates requires quite a lot of manual work.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1264105,
      "author_name": "datasciencegeek",
      "author_url": "",
      "post_date": "04/05/2021 21:54:22",
      "content": "<p><a href=\"https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/230853\" target=\"_blank\">Here</a> is an update on more duplicates findings with methods described in details and tried in this <a href=\"https://www.kaggle.com/datasciencegeek/detect-duplicates-via-image-embeddings\" target=\"_blank\">notebook</a>.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1248218": "### Hello!\n\nI decided to implement a pretty basic duplicate search via the `image_hash` library, and got interesting results: there are at least 50 pairs of identical images, and most of them are labeled differently, e.g.:\n<img src='https://drive.google.com/uc?id=1bZ0WWO1A_f0qyXwaRX3fwX8MALPdy6BG' />\n<img src='https://drive.google.com/uc?id=1UJUDOIudvnM33tcvWFFgw-HOw3uZjKIZ' />\n<img src='https://drive.google.com/uc?id=1Z5Vziho1wfcQUxh2dmSXr08-ZaNzarVP' />\nThe entire list of duplicate ids can be found in the **[notebook](https://www.kaggle.com/nickuzmenkov/pp2021-duplicates-revealing)** and output file `duplicates.csv`.\n\n### Update\nI've tried to find duplicates among original images and horizontally flipped, vertically flipped, 90, 180, and 270 degrees rotated images, revealing no more duplicates (but I suspect there are still a lot). As I'm a novice in image hashing and duplicates detection, I'm looking forward for experienced users to find and reveal more duplicates.",
    "1249001": "Let's share all duplicate images filenames to make sure to get them all. I found 54 in this [notebook](https://www.kaggle.com/datasciencegeek/eda-plantpathology2021-fgvc8) . Here is a list of filenames:\n\n[9ada65c5b0cc9a3c.jpg, e9e8ad35c5871d06.jpg]\n[94b42695f24dad36.jpg, f7d233d11aa8496c.jpg]\n[d64460a3b19f1cfc.jpg, f4cba4a7294d6585.jpg]\n[c5aff2e545d0a129.jpg, dbee22785d9285c4.jpg]\n[ca782c131f5ce0fc.jpg, e5f27342ce1cb9a0.jpg]\n[848d9f9e82d18e5e.jpg, 949b6595a5b8ca3c.jpg]\n[d3945c098edc9dd1.jpg, dea33c5c0347aa4d.jpg]\n[85897ea7f064b26a.jpg, ec83c7c04a1e9db9.jpg]\n[8f5263c61a8cdbe2.jpg, c9d63696629107ed.jpg]\n[b7b0edd58a858217.jpg, ba99260ef46b18cb.jpg]\n[c1a56069de75b899.jpg, f2c798246a0a9bdd.jpg]\n[965295a64a97859f.jpg, ae5074cfc9183763.jpg]\n[aa0e969518e15d3f.jpg, bb1891fa8c61e663.jpg]\n[a3bb604d8a55995d.jpg, e7ecc583d571d202.jpg]\n[86f963a16e70345e.jpg, 9fc321ce60a69f0b.jpg]\n[d0c766bc38e1978c.jpg, e83f715e56462a31.jpg]\n[9fad869f21b5b240.jpg, d7ec32d72c84dc18.jpg]\n[bd8088cdad5f65e0.jpg, f90738a0d2f2c1db.jpg]\n[90f99ee1c1132bb6.jpg, 94c83b2d5fad6924.jpg]\n[90a5341f227ce5f9.jpg, f3dc17a2b20d48ec.jpg]\n[8c0731fff680c51e.jpg, df90901ccb664de6.jpg]\n[c1753cad2f31d492.jpg, d2266ac99461e9db.jpg]\n[a9b2e4b09297bc2b.jpg, d1f01e8d8a73274d.jpg]\n[a7e613159fb22859.jpg, b5ec52491e3349ce.jpg]\n[82ba3e933947b097.jpg, cdf892b5b4865859.jpg]\n[e51690d8ad307dcb.jpg, eec64e8cf1c8d0c6.jpg]\n[d1dec0973c2cf4a1.jpg, d2d9c2bca7a72458.jpg]",
    "1249095": "Nice work! \n\nActually, I meant that I found 50 **pairs** of duplicates, i.e. 100 duplicated images total. I've done a quick check with your duplicates (let's say `a = set(...)`) and my duplicates (let's say `b=set(...)`) showing that all your duplicates `a` belong to my duplicates `b`:\n\n```\nprint(len(a))\nprint(len(b))\nprint(len(b.intersection(a)))\n>>> 27\n>>> 50\n>>> 27\n```\n\nAnd that's great news that two different methods give the same results.",
    "1249850": "Gotcha. Thank you for sharing! I have not heard of the method you used to detect duplicates and came up with an add-hoc method myself, nice to learn new approaches.",
    "1250344": "Thanks for sharing these tags, much appreciated!",
    "1258967": "There are around 104 pairs of duplicate images. I found this using `Imagededup` library. I often use this library to check the duplicates. I used Perceptual hashing method to find duplicates.\n\nThe link about [Perceptual Hashing](http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html)\n\nThe embeddings are as follows:\n{'bd0a909f69c97865.jpg': ['d58ad09e28df7825.jpg'],\n 'eec64e8cf1c8d0c6.jpg': ['e51690d8ad307dcb.jpg'],\n 'd2266ac99461e9db.jpg': ['c1753cad2f31d492.jpg'],\n 'd64460a3b19f1cfc.jpg': ['f4cba4a7294d6585.jpg'],\n 'a8e291e1da57596a.jpg': ['a8eb95c7d85e482a.jpg'],\n 'df90901ccb664de6.jpg': ['8c0731fff680c51e.jpg'],\n 'cc73e025cc36d31e.jpg': ['cc73e005cc36f31e.jpg'],\n 'a484c39f9fd861d8.jpg': ['b0e0cf9f9e98e188.jpg'],\n 'af08a3a0ff35c0c7.jpg': ['fe0883b0bb75a0cd.jpg'],\n 'cc73e005cc36f31e.jpg': ['cc73e025cc36d31e.jpg'],\n 'f5e4c2f7849a8c91.jpg': ['f4a0c077c79a8cd9.jpg'],\n '9ada65c5b0cc9a3c.jpg': ['e9e8ad35c5871d06.jpg'],\n 'd0b487c76e8e3278.jpg': ['e0b4a7c3e88e7332.jpg'],\n 'f585d0d2caeaeb02.jpg': ['bd84d4dacaf8cb40.jpg'],\n 'fe0883b0bb75a0cd.jpg': ['af08a3a0ff35c0c7.jpg'],\n 'b4e4c9424f79335a.jpg': ['b465cbd64c19725a.jpg'],\n 'b7b0edd58a858217.jpg': ['ba99260ef46b18cb.jpg'],\n 'e90e1ac8b7e7aa50.jpg': ['e90e1acab7e5aa50.jpg'],\n 'e0f4c0f48aca6b9e.jpg': ['a065c2f4ca986fdc.jpg'],\n '9c968ac989ed332e.jpg': ['8dbeda49894d522e.jpg'],\n 'aba437d04a0997dd.jpg': ['afa074d04a8d25df.jpg'],\n 'ca782c131f5ce0fc.jpg': ['e5f27342ce1cb9a0.jpg'],\n 'e0fec0c0838bcf8f.jpg': ['f0daf0c0828f9f0f.jpg'],\n 'bd92c0694632d73b.jpg': ['bf9ac0e9c6628633.jpg'],\n 'c738d13e7146798c.jpg': ['c738d13e73467988.jpg'],\n 'ff88c51c70b0e2d6.jpg': ['ff08c71cf0b42ba2.jpg'],\n '8c0731fff680c51e.jpg': ['df90901ccb664de6.jpg'],\n 'f196f11a99d91ce0.jpg': ['f392f11919991cea.jpg'],\n 'dbee22785d9285c4.jpg': ['c5aff2e545d0a129.jpg'],\n 'abb0d5a9d4b4985c.jpg': ['ab91d1a9d4b49a6c.jpg'],\n 'aaf5d596828adac8.jpg': ['faf5d5868a0ac9b0.jpg'],\n 'd0c766bc38e1978c.jpg': ['e83f715e56462a31.jpg'],\n 'f392f11919991cea.jpg': ['f196f11a99d91ce0.jpg'],\n 'd89c9cda9274322f.jpg': ['d89d9ede82743229.jpg'],\n '9fad869f21b5b240.jpg': ['d7ec32d72c84dc18.jpg'],\n 'e8212ddeec024fec.jpg': ['e8210ddefc024fec.jpg'],\n 'fcc4c8fe30300dcf.jpg': ['fcc4c8bf387015c5.jpg'],\n 'c1753cad2f31d492.jpg': ['d2266ac99461e9db.jpg'],\n 'b53c9ab0c9e4f0c9.jpg': ['f528cbb0c5c1f0cd.jpg'],\n 'ba2e601c8979afe2.jpg': ['9a2e269c1969bfe0.jpg'],\n 'e11be7f110de8f80.jpg': ['e13be7f1105e8f80.jpg'],\n 'f891839e8f342e72.jpg': ['f891c19fbf301233.jpg'],\n 'afbe5641896d522a.jpg': ['8dbeda49894d522e.jpg'],\n '9a2e269c1969bfe0.jpg': ['ba2e601c8979afe2.jpg'],\n 'a59497a1da8a49de.jpg': ['a5a687b1ca8a595e.jpg'],\n 'c15d3d3ae123c547.jpg': ['c1583d30e0e1d5f7.jpg'],\n 'dea33c5c0347aa4d.jpg': ['d3945c098edc9dd1.jpg'],\n '82ba3e933947b097.jpg': ['cdf892b5b4865859.jpg'],\n 'a1a1d5a4cada586f.jpg': ['a9b095bedada5805.jpg'],\n 'ae5074cfc9183763.jpg': ['965295a64a97859f.jpg'],\n 'd1f01e8d8a73274d.jpg': ['a9b2e4b09297bc2b.jpg'],\n 'f0cb859beb1b30c1.jpg': ['f0cb879e4a1b35c1.jpg'],\n 'a7e613159fb22859.jpg': ['b5ec52491e3349ce.jpg'],\n 'bf9ac0e9c6628633.jpg': ['bd92c0694632d73b.jpg'],\n 'cdf892b5b4865859.jpg': ['82ba3e933947b097.jpg'],\n 'd70f2230fb5a093d.jpg': ['d70f6330fb5a090d.jpg'],\n 'ff3fd02a82ad8a84.jpg': ['fc5fd028d2a988f4.jpg'],\n 'c6e4efce38d81419.jpg': ['c2ccaf86b8d9549a.jpg'],\n 'dfc86b180e3ac8f8.jpg': ['dfcc6b28063a88fc.jpg'],\n 'f474927ad9c0e88d.jpg': ['b5369279d9c1c949.jpg'],\n 'bc2a94a2af52da69.jpg': ['b4ae98a3ad52c939.jpg'],\n 'f90738a0d2f2c1db.jpg': ['bd8088cdad5f65e0.jpg'],\n 'c738d13e73467988.jpg': ['c738d13e7146798c.jpg'],\n 'c9d63696629107ed.jpg': ['8f5263c61a8cdbe2.jpg'],\n 'c5339e87235c6ac5.jpg': ['c53b9e87215c6ac5.jpg'],\n 'b0e0cf9f9e98e188.jpg': ['a484c39f9fd861d8.jpg'],\n 'b80898e11f81ff1f.jpg': ['b80898e11fa17f1f.jpg'],\n 'aaac95caca4b8d2d.jpg': ['a8bdb4cac84b98cd.jpg'],\n 'ff08c71cf0b42ba2.jpg': ['ff88c51c70b0e2d6.jpg'],\n '8dbeda49894d522e.jpg': ['afbe5641896d522a.jpg', '9c968ac989ed332e.jpg'],\n 'd7ec32d72c84dc18.jpg': ['9fad869f21b5b240.jpg'],\n 'f6f1160be38a48d5.jpg': ['f6f11e0fe38a08c5.jpg'],\n 'd1dec0973c2cf4a1.jpg': ['d2d9c2bca7a72458.jpg'],\n '9886c704f93a79b3.jpg': ['98864714f96a75d5.jpg'],\n 'a8bdb4cac84b98cd.jpg': ['aaac95caca4b8d2d.jpg'],\n 'f8228796cfdae848.jpg': ['b8828287cfdafc49.jpg'],\n 'f528cbb0c5c1f0cd.jpg': ['b53c9ab0c9e4f0c9.jpg'],\n '965295a64a97859f.jpg': ['ae5074cfc9183763.jpg'],\n 'afa074d04a8d25df.jpg': ['aba437d04a0997dd.jpg'],\n 'c4e8843569247fcf.jpg': ['c46a85b46b643acf.jpg'],\n 'f4cba4a7294d6585.jpg': ['d64460a3b19f1cfc.jpg'],\n 'ebd19f1c549a1ad0.jpg': ['ebd19f9c549a12d0.jpg'],\n 'f8ab8597dc92d411.jpg': ['faca978fda90d011.jpg'],\n 'abf0b5819f028f0f.jpg': ['abf0b5a0df028b17.jpg'],\n 'dfbdc047068b063d.jpg': ['cebdc20f67838631.jpg'],\n 'f47ed2a14ac36998.jpg': ['f47ad2a2ca934d8a.jpg'],\n 'afe996c30a0e0d3e.jpg': ['afebd604ca06093f.jpg'],\n 'f0cb879e4a1b35c1.jpg': ['f0cb859beb1b30c1.jpg'],\n 'a8eb95c7d85e482a.jpg': ['a8e291e1da57596a.jpg'],\n 'a8e895f5aa54d2a5.jpg': ['a8e895e5aad453a5.jpg'],\n 'e0b4a7c3e88e7332.jpg': ['d0b487c76e8e3278.jpg'],\n 'bcdad04d9d24ab4c.jpg': ['b4dad04d896dcb25.jpg'],\n 'ec83c7c04a1e9db9.jpg': ['85897ea7f064b26a.jpg'],\n 'e385830ecacd2d9e.jpg': ['c335971e8acd609e.jpg'],\n 'bfa2d0bad0d22959.jpg': ['ba2092abd0faa95b.jpg'],\n 'c1583d30e0e1d5f7.jpg': ['c15d3d3ae123c547.jpg'],\n 'a3bb604d8a55995d.jpg': ['e7ecc583d571d202.jpg'],\n 'bb1891fa8c61e663.jpg': ['aa0e969518e15d3f.jpg'],\n 'e8a58ad4d9829f33.jpg': ['e8bd8ac4d982dd83.jpg'],\n 'e90e1acab7e5aa50.jpg': ['e90e1ac8b7e7aa50.jpg'],\n 'a5a687b1ca8a595e.jpg': ['a59497a1da8a49de.jpg'],\n 'af6292db1b611d98.jpg': ['a56292dadb618d95.jpg'],\n 'd70f6330fb5a090d.jpg': ['d70f2230fb5a093d.jpg'],\n 'bd84d4dacaf8cb40.jpg': ['f585d0d2caeaeb02.jpg'],\n 'b4ae98a3ad52c939.jpg': ['bc2a94a2af52da69.jpg'],\n '98864714f96a75d5.jpg': ['9886c704f93a79b3.jpg'],\n 'a5add7c29c8a9899.jpg': ['a4a5d6d69e9a981c.jpg'],\n 'e51690d8ad307dcb.jpg': ['eec64e8cf1c8d0c6.jpg'],\n 'f09697d6dada8805.jpg': ['fa9f95c2daca0905.jpg'],\n '95bc383078d273b3.jpg': ['953f3074707063b7.jpg'],\n 'b8a7cac8958b98d9.jpg': ['b88dc8ce8d8f0ad9.jpg'],\n '8f5263c61a8cdbe2.jpg': ['c9d63696629107ed.jpg'],\n 'd89062fbf4452a9e.jpg': ['d0806bfbf9452693.jpg'],\n 'ba99260ef46b18cb.jpg': ['b7b0edd58a858217.jpg'],\n 'ffe4c0878d50b4b8.jpg': ['b4c0c0b7cdd0b4f9.jpg'],\n 'dfcc6b28063a88fc.jpg': ['dfc86b180e3ac8f8.jpg'],\n 'faf5d5868a0ac9b0.jpg': ['aaf5d596828adac8.jpg'],\n 'e087f27e83f008e7.jpg': ['e087a7fe83c00acf.jpg'],\n 'faca978fda90d011.jpg': ['f8ab8597dc92d411.jpg'],\n 'a8e895e5aad453a5.jpg': ['a8e895f5aa54d2a5.jpg'],\n 'e8bd8ac4d982dd83.jpg': ['e8a58ad4d9829f33.jpg'],\n 'd89d9ede82743229.jpg': ['d89c9cda9274322f.jpg'],\n 'b490da8ddc8dc4ce.jpg': ['b594d286ddcfc0c8.jpg'],\n 'c53b9e87215c6ac5.jpg': ['c5339e87235c6ac5.jpg'],\n 'bd8088cdad5f65e0.jpg': ['f90738a0d2f2c1db.jpg'],\n 'c1a56069de75b899.jpg': ['f2c798246a0a9bdd.jpg'],\n 'e586c25e0f1c3c79.jpg': ['e787c35e0f0c34e8.jpg'],\n 'c46a85b46b643acf.jpg': ['c4e8843569247fcf.jpg'],\n 'aa142048eff78bf4.jpg': ['aa542048ebf78bf4.jpg'],\n 'a56292dadb618d95.jpg': ['af6292db1b611d98.jpg'],\n 'fc5fd028d2a988f4.jpg': ['ff3fd02a82ad8a84.jpg'],\n 'a4a5d6d69e9a981c.jpg': ['a5add7c29c8a9899.jpg'],\n 'b8828287cfdafc49.jpg': ['f8228796cfdae848.jpg'],\n 'e1e39f8809c7349d.jpg': ['e9c39fa809c7341d.jpg'],\n 'e83f715e56462a31.jpg': ['d0c766bc38e1978c.jpg'],\n '86f963a16e70345e.jpg': ['9fc321ce60a69f0b.jpg'],\n 'f6f11e0fe38a08c5.jpg': ['f6f1160be38a48d5.jpg'],\n 'b465cbd64c19725a.jpg': ['b4e4c9424f79335a.jpg'],\n '9fc321ce60a69f0b.jpg': ['86f963a16e70345e.jpg'],\n 'd0806bfbf9452693.jpg': ['d89062fbf4452a9e.jpg'],\n '94c83b2d5fad6924.jpg': ['90f99ee1c1132bb6.jpg'],\n 'e9e8ad35c5871d06.jpg': ['9ada65c5b0cc9a3c.jpg'],\n 'a9b2e4b09297bc2b.jpg': ['d1f01e8d8a73274d.jpg'],\n 'f0daf0c0828f9f0f.jpg': ['e0fec0c0838bcf8f.jpg'],\n 'd2d9c2bca7a72458.jpg': ['d1dec0973c2cf4a1.jpg'],\n 'fc01c79c9ee107e1.jpg': ['fc01c2dc8fa99de0.jpg'],\n '953f3074707063b7.jpg': ['95bc383078d273b3.jpg'],\n 'e03496c3d9ddc1c9.jpg': ['a03497c399c9c7d9.jpg'],\n 'f2c798246a0a9bdd.jpg': ['c1a56069de75b899.jpg'],\n 'e8e5c38f8f18a05d.jpg': ['e8a5c38f9f9828d8.jpg'],\n 'e087a7fe83c00acf.jpg': ['e087f27e83f008e7.jpg'],\n '892ec13c8b783fca.jpg': ['a91ec1bc0b783f43.jpg'],\n '83e3d2369933c5d2.jpg': ['83e3d2329933c5f2.jpg'],\n '90a5341f227ce5f9.jpg': ['f3dc17a2b20d48ec.jpg'],\n 'c335971e8acd609e.jpg': ['e385830ecacd2d9e.jpg'],\n 'e9c39fa809c7341d.jpg': ['e1e39f8809c7349d.jpg'],\n 'b80898e11fa17f1f.jpg': ['b80898e11f81ff1f.jpg'],\n 'e22e9a365cd1ea58.jpg': ['e22e8a365cd1ea5c.jpg'],\n 'fa9f95c2daca0905.jpg': ['f09697d6dada8805.jpg'],\n 'f4a0c077c79a8cd9.jpg': ['f5e4c2f7849a8c91.jpg'],\n 'b5ec52491e3349ce.jpg': ['a7e613159fb22859.jpg'],\n 'd58ad09e28df7825.jpg': ['bd0a909f69c97865.jpg'],\n 'e8a5c38f9f9828d8.jpg': ['e8e5c38f8f18a05d.jpg'],\n 'f7d233d11aa8496c.jpg': ['94b42695f24dad36.jpg'],\n 'c5aff2e545d0a129.jpg': ['dbee22785d9285c4.jpg'],\n 'f47ad2a2ca934d8a.jpg': ['f47ed2a14ac36998.jpg'],\n 'fcc4c8bf387015c5.jpg': ['fcc4c8fe30300dcf.jpg'],\n 'e5f27342ce1cb9a0.jpg': ['ca782c131f5ce0fc.jpg'],\n 'e7ecc583d571d202.jpg': ['a3bb604d8a55995d.jpg'],\n '83e3d2329933c5f2.jpg': ['83e3d2369933c5d2.jpg'],\n 'a9b095bedada5805.jpg': ['a1a1d5a4cada586f.jpg'],\n 'b88dc8ce8d8f0ad9.jpg': ['b8a7cac8958b98d9.jpg'],\n 'e787c35e0f0c34e8.jpg': ['e586c25e0f1c3c79.jpg'],\n 'abf0b5a0df028b17.jpg': ['abf0b5819f028f0f.jpg'],\n 'afebd604ca06093f.jpg': ['afe996c30a0e0d3e.jpg'],\n 'f3dc17a2b20d48ec.jpg': ['90a5341f227ce5f9.jpg'],\n 'ab83c0d147c3db66.jpg': ['ab83c05147c3cfe6.jpg'],\n 'e8210ddefc024fec.jpg': ['e8212ddeec024fec.jpg'],\n 'a8aa915abe68b565.jpg': ['aaa3955ad262b56c.jpg'],\n 'aaa3955ad262b56c.jpg': ['a8aa915abe68b565.jpg'],\n '848d9f9e82d18e5e.jpg': ['949b6595a5b8ca3c.jpg'],\n 'fc01c2dc8fa99de0.jpg': ['fc01c79c9ee107e1.jpg'],\n 'd3945c098edc9dd1.jpg': ['dea33c5c0347aa4d.jpg'],\n 'ba2092abd0faa95b.jpg': ['bfa2d0bad0d22959.jpg'],\n 'f891c19fbf301233.jpg': ['f891839e8f342e72.jpg'],\n '90f99ee1c1132bb6.jpg': ['94c83b2d5fad6924.jpg'],\n 'a03497c399c9c7d9.jpg': ['e03496c3d9ddc1c9.jpg'],\n 'b594d286ddcfc0c8.jpg': ['b490da8ddc8dc4ce.jpg'],\n '85897ea7f064b26a.jpg': ['ec83c7c04a1e9db9.jpg'],\n 'd03fcce03447353e.jpg': ['c03ec9c12c87377e.jpg'],\n 'ab91d1a9d4b49a6c.jpg': ['abb0d5a9d4b4985c.jpg'],\n 'b5369279d9c1c949.jpg': ['f474927ad9c0e88d.jpg'],\n '949b6595a5b8ca3c.jpg': ['848d9f9e82d18e5e.jpg'],\n 'a91ec1bc0b783f43.jpg': ['892ec13c8b783fca.jpg'],\n 'ebd19f9c549a12d0.jpg': ['ebd19f1c549a1ad0.jpg'],\n 'ab83c05147c3cfe6.jpg': ['ab83c0d147c3db66.jpg'],\n 'c2ccaf86b8d9549a.jpg': ['c6e4efce38d81419.jpg'],\n '94b42695f24dad36.jpg': ['f7d233d11aa8496c.jpg'],\n 'aa542048ebf78bf4.jpg': ['aa142048eff78bf4.jpg'],\n 'a065c2f4ca986fdc.jpg': ['e0f4c0f48aca6b9e.jpg'],\n 'e13fb40087e0fec6.jpg': ['e0bbf620c7c0ce87.jpg'],\n 'cebdc20f67838631.jpg': ['dfbdc047068b063d.jpg'],\n 'aa0e969518e15d3f.jpg': ['bb1891fa8c61e663.jpg'],\n 'e13be7f1105e8f80.jpg': ['e11be7f110de8f80.jpg'],\n 'e0bbf620c7c0ce87.jpg': ['e13fb40087e0fec6.jpg'],\n 'c03ec9c12c87377e.jpg': ['d03fcce03447353e.jpg'],\n 'e22e8a365cd1ea5c.jpg': ['e22e9a365cd1ea58.jpg'],\n 'b4dad04d896dcb25.jpg': ['bcdad04d9d24ab4c.jpg'],\n 'b4c0c0b7cdd0b4f9.jpg': ['ffe4c0878d50b4b8.jpg']}",
    "1263765": "Hello!\n\n* I've done a quick check revealing that all my duplicates (let's say `b=set(...)`) belong to your duplicates (let's say `a=set(...)`):\n```\nprint(len(a))\nprint(len(b))\nprint(len(a.intersection(a))\n>>> 106\n>>> 50\n>>> 50\n```\nwhich is great.\n* As for the remainder of 56 duplicate pairs (and even one triplet), most of them **aren't duplicates** and even do not look alike, e.g.:\n<img src='https://drive.google.com/uc?id=1O3Ra6ui_lkc-yW98n-6fxvRg4d4anFho' />\n<img src='https://drive.google.com/uc?id=1BK8-RyRh2GTnyhZqpXnFg0XT32FhhZdN' />\nthat's why I suggest using different types of hash or setting a higher threshold and **always manually check the results**. I've found that only 6 pairs of these are actual duplicates:\n```\n('8dbeda49894d522e.jpg', 'afbe5641896d522a.jpg')\n('af6292db1b611d98.jpg', 'a56292dadb618d95.jpg')\n('abf0b5a0df028b17.jpg', 'abf0b5819f028f0f.jpg')\n('e385830ecacd2d9e.jpg', 'c335971e8acd609e.jpg')\n('cebdc20f67838631.jpg', 'dfbdc047068b063d.jpg')\n('f392f11919991cea.jpg', 'f196f11a99d91ce0.jpg')\n```",
    "1264103": "Good catch! Finding duplicates requires quite a lot of manual work.",
    "1264105": "[Here](https://www.kaggle.com/c/plant-pathology-2021-fgvc8/discussion/230853) is an update on more duplicates findings with methods described in details and tried in this [notebook](https://www.kaggle.com/datasciencegeek/detect-duplicates-via-image-embeddings)."
  },
  "source": "meta"
}