{
  "id": 324699,
  "title": "Duplicate images in dataset?",
  "url": "/competitions/hotel-id-to-combat-human-trafficking-2022-fgvc9/discussion/324699",
  "author_name": "",
  "post_date": "2022-05-12T18:48:46.655204800Z",
  "votes": 5,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi, I am looking at the data and it seems there are many duplicate images. Here are some examples:</p>\n<pre><code>1111,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/34906/000010614.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/34906/000010606.jpg,1.0,train,train,train-train\n1110,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/35579/000025696.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/35579/000025697.jpg,1.0,train,train,train-train\n110,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/23095/000022016.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/23095/000022012.jpg,1.0,train,train,train-train\n1099,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/107503/000040226.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/107503/000040222.jpg,1.0,train,train,train-train\n1094,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/141562/000037037.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/141562/000037026.jpg,1.0,train,train,train-train\n1079,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/32635/000038710.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/32635/000038714.jpg,1.0,train,train,train-train\n1077,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/307892/000033824.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/307892/000033820.jpg,1.0,train,train,train-train\n1073,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/309388/000009802.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/309388/000009806.jpg,1.0,train,train,train-train\n1071,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/205762/000042388.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/205762/000042385.jpg,1.0,train,train,train-train\n1067,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29985/000018856.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29985/000018855.jpg,1.0,train,train,train-train\n1066,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11522/000007429.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11522/000007433.jpg,1.0,train,train,train-train\n1054,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/2891/000008183.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/2891/000008180.jpg,1.0,train,train,train-train\n1046,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011724.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011718.jpg,1.0,train,train,train-train\n1045,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011729.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011723.jpg,1.0,train,train,train-train\n1041,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/91611/000015441.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/91611/000015443.jpg,1.0,train,train,train-train\n1034,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11600/000020889.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11600/000020885.jpg,1.0,train,train,train-train\n1033,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/203154/000034308.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/203154/000034306.jpg,1.0,train,train,train-train\n1028,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/31263/000024342.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/31263/000024339.jpg,1.0,train,train,train-train\n1024,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29396/000021449.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29396/000021453.jpg,1.0,train,train,train-train\n1018,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/592374/000040439.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/592374/000040443.jpg,1.0,train,train,train-train\n1003,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/210388/000026537.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/210388/000026536.jpg,1.0,train,train,train-train\n1001,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/197510/000031222.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/197510/000031232.jpg,1.0,train,train,train-train\n</code></pre>\n<p>I wonder if this is a mistake on data preperation?</p>",
  "messages": [
    {
      "id": "1786306",
      "postDate": "05/12/2022 18:48:46",
      "content": "<p>Hi, I am looking at the data and it seems there are many duplicate images. Here are some examples:</p>\n<pre><code>1111,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/34906/000010614.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/34906/000010606.jpg,1.0,train,train,train-train\n1110,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/35579/000025696.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/35579/000025697.jpg,1.0,train,train,train-train\n110,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/23095/000022016.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/23095/000022012.jpg,1.0,train,train,train-train\n1099,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/107503/000040226.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/107503/000040222.jpg,1.0,train,train,train-train\n1094,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/141562/000037037.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/141562/000037026.jpg,1.0,train,train,train-train\n1079,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/32635/000038710.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/32635/000038714.jpg,1.0,train,train,train-train\n1077,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/307892/000033824.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/307892/000033820.jpg,1.0,train,train,train-train\n1073,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/309388/000009802.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/309388/000009806.jpg,1.0,train,train,train-train\n1071,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/205762/000042388.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/205762/000042385.jpg,1.0,train,train,train-train\n1067,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29985/000018856.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29985/000018855.jpg,1.0,train,train,train-train\n1066,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11522/000007429.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11522/000007433.jpg,1.0,train,train,train-train\n1054,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/2891/000008183.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/2891/000008180.jpg,1.0,train,train,train-train\n1046,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011724.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011718.jpg,1.0,train,train,train-train\n1045,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011729.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011723.jpg,1.0,train,train,train-train\n1041,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/91611/000015441.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/91611/000015443.jpg,1.0,train,train,train-train\n1034,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11600/000020889.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11600/000020885.jpg,1.0,train,train,train-train\n1033,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/203154/000034308.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/203154/000034306.jpg,1.0,train,train,train-train\n1028,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/31263/000024342.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/31263/000024339.jpg,1.0,train,train,train-train\n1024,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29396/000021449.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29396/000021453.jpg,1.0,train,train,train-train\n1018,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/592374/000040439.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/592374/000040443.jpg,1.0,train,train,train-train\n1003,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/210388/000026537.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/210388/000026536.jpg,1.0,train,train,train-train\n1001,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/197510/000031222.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/197510/000031232.jpg,1.0,train,train,train-train\n</code></pre>\n<p>I wonder if this is a mistake on data preperation?</p>",
      "rawMarkdown": "Hi, I am looking at the data and it seems there are many duplicate images. Here are some examples:\n```\n1111,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/34906/000010614.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/34906/000010606.jpg,1.0,train,train,train-train\n1110,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/35579/000025696.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/35579/000025697.jpg,1.0,train,train,train-train\n110,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/23095/000022016.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/23095/000022012.jpg,1.0,train,train,train-train\n1099,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/107503/000040226.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/107503/000040222.jpg,1.0,train,train,train-train\n1094,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/141562/000037037.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/141562/000037026.jpg,1.0,train,train,train-train\n1079,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/32635/000038710.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/32635/000038714.jpg,1.0,train,train,train-train\n1077,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/307892/000033824.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/307892/000033820.jpg,1.0,train,train,train-train\n1073,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/309388/000009802.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/309388/000009806.jpg,1.0,train,train,train-train\n1071,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/205762/000042388.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/205762/000042385.jpg,1.0,train,train,train-train\n1067,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29985/000018856.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29985/000018855.jpg,1.0,train,train,train-train\n1066,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11522/000007429.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11522/000007433.jpg,1.0,train,train,train-train\n1054,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/2891/000008183.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/2891/000008180.jpg,1.0,train,train,train-train\n1046,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011724.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011718.jpg,1.0,train,train,train-train\n1045,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011729.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011723.jpg,1.0,train,train,train-train\n1041,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/91611/000015441.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/91611/000015443.jpg,1.0,train,train,train-train\n1034,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11600/000020889.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11600/000020885.jpg,1.0,train,train,train-train\n1033,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/203154/000034308.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/203154/000034306.jpg,1.0,train,train,train-train\n1028,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/31263/000024342.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/31263/000024339.jpg,1.0,train,train,train-train\n1024,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29396/000021449.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29396/000021453.jpg,1.0,train,train,train-train\n1018,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/592374/000040439.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/592374/000040443.jpg,1.0,train,train,train-train\n1003,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/210388/000026537.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/210388/000026536.jpg,1.0,train,train,train-train\n1001,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/197510/000031222.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/197510/000031232.jpg,1.0,train,train,train-train\n```\nI wonder if this is a mistake on data preperation?",
      "votes": null
    },
    {
      "id": "1786316",
      "postDate": "05/12/2022 18:55:13",
      "content": "<p>Hmm, looks like those are duplicates in our database that we didn't catch. It looks like the pairs are always in the same hotel so it shouldn't have too big an impact on any of your approaches, but it's a good catch and thank you for pointing it out!</p>",
      "rawMarkdown": "Hmm, looks like those are duplicates in our database that we didn't catch. It looks like the pairs are always in the same hotel so it shouldn't have too big an impact on any of your approaches, but it's a good catch and thank you for pointing it out!",
      "votes": null
    },
    {
      "id": "1786363",
      "postDate": "05/12/2022 19:47:57",
      "content": "<p>It seems like there are about 6,200 of these duplicate pairs.<br>\nHere are some with examples of pairs with different hotel ids:</p>\n<p>/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043031.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg   1.0 1013    688864<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043131.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043077.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043118.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043031.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 1013    723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043032.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 1013    723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043032.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg   1.0 1013    688864<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043030.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 1013    723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043030.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg   1.0 1013    688864<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043076.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/208080/000034714.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/310349/000034724.jpg   1.0 208080  310349<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/208080/000034715.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/310349/000034725.jpg   1.0 208080  310349<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30552/000002831.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043029.jpg 0.999999    30552   1013<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30552/000002830.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043029.jpg 0.999999    30552   1013</p>",
      "rawMarkdown": "It seems like there are about 6,200 of these duplicate pairs.\nHere are some with examples of pairs with different hotel ids:\n\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043031.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg\t1.0\t1013\t688864\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043131.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043077.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043118.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043031.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t1013\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043032.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t1013\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043032.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg\t1.0\t1013\t688864\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043030.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t1013\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043030.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg\t1.0\t1013\t688864\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043076.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/208080/000034714.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/310349/000034724.jpg\t1.0\t208080\t310349\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/208080/000034715.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/310349/000034725.jpg\t1.0\t208080\t310349\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30552/000002831.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043029.jpg\t0.999999\t30552\t1013\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30552/000002830.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043029.jpg\t0.999999\t30552\t1013",
      "votes": null
    },
    {
      "id": "1787099",
      "postDate": "05/13/2022 15:16:22",
      "content": "<p>These are indeed duplicates in the dataset. It won't have too big of an impact on your models but it's still good to be aware of!</p>",
      "rawMarkdown": "These are indeed duplicates in the dataset. It won't have too big of an impact on your models but it's still good to be aware of!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1786316,
      "author_name": "abbystylianou",
      "author_url": "",
      "post_date": "05/12/2022 18:55:13",
      "content": "<p>Hmm, looks like those are duplicates in our database that we didn't catch. It looks like the pairs are always in the same hotel so it shouldn't have too big an impact on any of your approaches, but it's a good catch and thank you for pointing it out!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1786363,
      "author_name": "amirus",
      "author_url": "",
      "post_date": "05/12/2022 19:47:57",
      "content": "<p>It seems like there are about 6,200 of these duplicate pairs.<br>\nHere are some with examples of pairs with different hotel ids:</p>\n<p>/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043031.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg   1.0 1013    688864<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043131.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043077.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043118.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043031.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 1013    723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043032.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 1013    723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043032.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg   1.0 1013    688864<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043030.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 1013    723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043030.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg   1.0 1013    688864<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043076.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg  1.0 688864  723<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/208080/000034714.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/310349/000034724.jpg   1.0 208080  310349<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/208080/000034715.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/310349/000034725.jpg   1.0 208080  310349<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30552/000002831.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043029.jpg 0.999999    30552   1013<br>\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30552/000002830.jpg    /home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043029.jpg 0.999999    30552   1013</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1787099,
      "author_name": "",
      "author_url": "",
      "post_date": "05/13/2022 15:16:22",
      "content": "<p>These are indeed duplicates in the dataset. It won't have too big of an impact on your models but it's still good to be aware of!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1786306": "Hi, I am looking at the data and it seems there are many duplicate images. Here are some examples:\n```\n1111,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/34906/000010614.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/34906/000010606.jpg,1.0,train,train,train-train\n1110,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/35579/000025696.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/35579/000025697.jpg,1.0,train,train,train-train\n110,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/23095/000022016.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/23095/000022012.jpg,1.0,train,train,train-train\n1099,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/107503/000040226.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/107503/000040222.jpg,1.0,train,train,train-train\n1094,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/141562/000037037.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/141562/000037026.jpg,1.0,train,train,train-train\n1079,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/32635/000038710.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/32635/000038714.jpg,1.0,train,train,train-train\n1077,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/307892/000033824.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/307892/000033820.jpg,1.0,train,train,train-train\n1073,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/309388/000009802.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/309388/000009806.jpg,1.0,train,train,train-train\n1071,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/205762/000042388.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/205762/000042385.jpg,1.0,train,train,train-train\n1067,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29985/000018856.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29985/000018855.jpg,1.0,train,train,train-train\n1066,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11522/000007429.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11522/000007433.jpg,1.0,train,train,train-train\n1054,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/2891/000008183.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/2891/000008180.jpg,1.0,train,train,train-train\n1046,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011724.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011718.jpg,1.0,train,train,train-train\n1045,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011729.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30968/000011723.jpg,1.0,train,train,train-train\n1041,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/91611/000015441.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/91611/000015443.jpg,1.0,train,train,train-train\n1034,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11600/000020889.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/11600/000020885.jpg,1.0,train,train,train-train\n1033,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/203154/000034308.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/203154/000034306.jpg,1.0,train,train,train-train\n1028,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/31263/000024342.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/31263/000024339.jpg,1.0,train,train,train-train\n1024,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29396/000021449.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/29396/000021453.jpg,1.0,train,train,train-train\n1018,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/592374/000040439.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/592374/000040443.jpg,1.0,train,train,train-train\n1003,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/210388/000026537.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/210388/000026536.jpg,1.0,train,train,train-train\n1001,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/197510/000031222.jpg,/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/197510/000031232.jpg,1.0,train,train,train-train\n```\nI wonder if this is a mistake on data preperation?",
    "1786316": "Hmm, looks like those are duplicates in our database that we didn't catch. It looks like the pairs are always in the same hotel so it shouldn't have too big an impact on any of your approaches, but it's a good catch and thank you for pointing it out!",
    "1786363": "It seems like there are about 6,200 of these duplicate pairs.\nHere are some with examples of pairs with different hotel ids:\n\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043031.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg\t1.0\t1013\t688864\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043131.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043077.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043118.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043031.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t1013\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043032.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t1013\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043032.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg\t1.0\t1013\t688864\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043030.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t1013\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043030.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg\t1.0\t1013\t688864\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043117.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/688864/000043076.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/723/000043140.jpg\t1.0\t688864\t723\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/208080/000034714.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/310349/000034724.jpg\t1.0\t208080\t310349\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/208080/000034715.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/310349/000034725.jpg\t1.0\t208080\t310349\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30552/000002831.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043029.jpg\t0.999999\t30552\t1013\n/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/30552/000002830.jpg\t/home/jupyter/data2/hotel-id-to-combat-human-trafficking-2022-fgvc9/train_images/1013/000043029.jpg\t0.999999\t30552\t1013",
    "1787099": "These are indeed duplicates in the dataset. It won't have too big of an impact on your models but it's still good to be aware of!"
  },
  "source": "meta"
}