{
  "id": 69979,
  "title": "Duplicate images and Near-Duplicate images",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/69979",
  "author_name": "",
  "post_date": "2018-10-29T15:03:31.995587800Z",
  "votes": 8,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I found odd facts.</p>\n\n<h1>Near-Duplicate images</h1>\n\n<p>Brian reported about duplicate images(<a href=\"https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69594\">https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69594</a>). So, I visualized the image and confirmed it. \nStrange to say, it looks slightly different if you look closely. (Maybe the brightness is different?)</p>\n\n<h1>Duplicate images</h1>\n\n<p>On the other hand, I found duplicate images using the hash function. Further verification was done with 8 times the data which can be made by flip and rotation, but no more duplication was found. </p>\n\n<p>I report \"Duplicate images\".</p>\n\n<p>img1    img2    img1_original_size  img2_original_size</p>\n\n<p>0   train/abfa727e-bba4-11e8-b2ba-ac1f6b6435d0  train/7d7566b6-bbb8-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>1   train/10b37fe0-bb9a-11e8-b2b9-ac1f6b6435d0  train/be92e108-bbb5-11e8-b2ba-ac1f6b6435d0  2048    2048</p>\n\n<p>2   train/0f9a26dc-bb9e-11e8-b2b9-ac1f6b6435d0  train/be856d8e-bba1-11e8-b2b9-ac1f6b6435d0  2048    2048</p>\n\n<p>3   train/3556a318-bbbd-11e8-b2ba-ac1f6b6435d0  train/aca475ec-bbbf-11e8-b2bb-ac1f6b6435d0  3072    3072</p>\n\n<p>4   train/310e3162-bbbe-11e8-b2ba-ac1f6b6435d0  train/8a10533e-bba6-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>5   train/26a0e4f6-bbc6-11e8-b2bc-ac1f6b6435d0  train/50f4626e-bba8-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>6   train/31551386-bbc1-11e8-b2bb-ac1f6b6435d0  train/8549d2f0-bb9f-11e8-b2b9-ac1f6b6435d0  2048    2048</p>\n\n<p>7   train/c90611fe-bba2-11e8-b2b9-ac1f6b6435d0  train/aa197268-bba7-11e8-b2ba-ac1f6b6435d0  2048    2048</p>\n\n<p>8   train/76494604-bbaf-11e8-b2ba-ac1f6b6435d0  train/c7a7a462-bbb1-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>9   train/1c980c8e-bb9c-11e8-b2b9-ac1f6b6435d0  train/e30040c2-bbb9-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>10  train/f7afc760-bbbc-11e8-b2ba-ac1f6b6435d0  train/2384acac-bbae-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>11  train/260ace10-bbbe-11e8-b2ba-ac1f6b6435d0  train/7138c4aa-bb9b-11e8-b2b9-ac1f6b6435d0  3072    3072</p>\n\n<p>12  train/439f3e7a-bbaa-11e8-b2ba-ac1f6b6435d0  train/dca7670e-bbc4-11e8-b2bc-ac1f6b6435d0  2048    2048</p>\n\n<p>13  train/6f094b0a-bbbe-11e8-b2ba-ac1f6b6435d0  train/c3200bb2-bbc2-11e8-b2bc-ac1f6b6435d0  2048    2048</p>\n\n<p>14  train/74b52b52-bbbc-11e8-b2ba-ac1f6b6435d0  train/559f7ce0-bbb2-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>15  train/c43c72e0-bba6-11e8-b2ba-ac1f6b6435d0  train/82936850-bb9f-11e8-b2b9-ac1f6b6435d0  3072    3072</p>\n\n<p>16  train/642c90e8-bba0-11e8-b2b9-ac1f6b6435d0  train/8e2962fa-bba4-11e8-b2b9-ac1f6b6435d0  3072    3072</p>\n\n<p>17  test/280f99a0-bad5-11e8-b2b8-ac1f6b6435d0   test/93dde07a-baca-11e8-b2b8-ac1f6b6435d0   3072    3072</p>\n\n<p>18  test/09b910ec-bac8-11e8-b2b7-ac1f6b6435d0   test/66606722-bace-11e8-b2b8-ac1f6b6435d0   2048    2048</p>",
  "messages": [
    {
      "id": "412096",
      "postDate": "10/29/2018 15:03:31",
      "content": "<p>I found odd facts.</p>\n\n<h1>Near-Duplicate images</h1>\n\n<p>Brian reported about duplicate images(<a href=\"https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69594\">https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69594</a>). So, I visualized the image and confirmed it. \nStrange to say, it looks slightly different if you look closely. (Maybe the brightness is different?)</p>\n\n<h1>Duplicate images</h1>\n\n<p>On the other hand, I found duplicate images using the hash function. Further verification was done with 8 times the data which can be made by flip and rotation, but no more duplication was found. </p>\n\n<p>I report \"Duplicate images\".</p>\n\n<p>img1    img2    img1_original_size  img2_original_size</p>\n\n<p>0   train/abfa727e-bba4-11e8-b2ba-ac1f6b6435d0  train/7d7566b6-bbb8-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>1   train/10b37fe0-bb9a-11e8-b2b9-ac1f6b6435d0  train/be92e108-bbb5-11e8-b2ba-ac1f6b6435d0  2048    2048</p>\n\n<p>2   train/0f9a26dc-bb9e-11e8-b2b9-ac1f6b6435d0  train/be856d8e-bba1-11e8-b2b9-ac1f6b6435d0  2048    2048</p>\n\n<p>3   train/3556a318-bbbd-11e8-b2ba-ac1f6b6435d0  train/aca475ec-bbbf-11e8-b2bb-ac1f6b6435d0  3072    3072</p>\n\n<p>4   train/310e3162-bbbe-11e8-b2ba-ac1f6b6435d0  train/8a10533e-bba6-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>5   train/26a0e4f6-bbc6-11e8-b2bc-ac1f6b6435d0  train/50f4626e-bba8-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>6   train/31551386-bbc1-11e8-b2bb-ac1f6b6435d0  train/8549d2f0-bb9f-11e8-b2b9-ac1f6b6435d0  2048    2048</p>\n\n<p>7   train/c90611fe-bba2-11e8-b2b9-ac1f6b6435d0  train/aa197268-bba7-11e8-b2ba-ac1f6b6435d0  2048    2048</p>\n\n<p>8   train/76494604-bbaf-11e8-b2ba-ac1f6b6435d0  train/c7a7a462-bbb1-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>9   train/1c980c8e-bb9c-11e8-b2b9-ac1f6b6435d0  train/e30040c2-bbb9-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>10  train/f7afc760-bbbc-11e8-b2ba-ac1f6b6435d0  train/2384acac-bbae-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>11  train/260ace10-bbbe-11e8-b2ba-ac1f6b6435d0  train/7138c4aa-bb9b-11e8-b2b9-ac1f6b6435d0  3072    3072</p>\n\n<p>12  train/439f3e7a-bbaa-11e8-b2ba-ac1f6b6435d0  train/dca7670e-bbc4-11e8-b2bc-ac1f6b6435d0  2048    2048</p>\n\n<p>13  train/6f094b0a-bbbe-11e8-b2ba-ac1f6b6435d0  train/c3200bb2-bbc2-11e8-b2bc-ac1f6b6435d0  2048    2048</p>\n\n<p>14  train/74b52b52-bbbc-11e8-b2ba-ac1f6b6435d0  train/559f7ce0-bbb2-11e8-b2ba-ac1f6b6435d0  3072    3072</p>\n\n<p>15  train/c43c72e0-bba6-11e8-b2ba-ac1f6b6435d0  train/82936850-bb9f-11e8-b2b9-ac1f6b6435d0  3072    3072</p>\n\n<p>16  train/642c90e8-bba0-11e8-b2b9-ac1f6b6435d0  train/8e2962fa-bba4-11e8-b2b9-ac1f6b6435d0  3072    3072</p>\n\n<p>17  test/280f99a0-bad5-11e8-b2b8-ac1f6b6435d0   test/93dde07a-baca-11e8-b2b8-ac1f6b6435d0   3072    3072</p>\n\n<p>18  test/09b910ec-bac8-11e8-b2b7-ac1f6b6435d0   test/66606722-bace-11e8-b2b8-ac1f6b6435d0   2048    2048</p>",
      "rawMarkdown": "I found odd facts.\n\n# Near-Duplicate images\nBrian reported about duplicate images(https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69594). So, I visualized the image and confirmed it. \nStrange to say, it looks slightly different if you look closely. (Maybe the brightness is different?)\n\n# Duplicate images\nOn the other hand, I found duplicate images using the hash function. Further verification was done with 8 times the data which can be made by flip and rotation, but no more duplication was found. \n\nI report \"Duplicate images\".\n\nimg1 \timg2 \timg1_original_size \timg2_original_size\n\n0 \ttrain/abfa727e-bba4-11e8-b2ba-ac1f6b6435d0 \ttrain/7d7566b6-bbb8-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n1 \ttrain/10b37fe0-bb9a-11e8-b2b9-ac1f6b6435d0 \ttrain/be92e108-bbb5-11e8-b2ba-ac1f6b6435d0 \t2048 \t2048\n\n2 \ttrain/0f9a26dc-bb9e-11e8-b2b9-ac1f6b6435d0 \ttrain/be856d8e-bba1-11e8-b2b9-ac1f6b6435d0 \t2048 \t2048\n\n3 \ttrain/3556a318-bbbd-11e8-b2ba-ac1f6b6435d0 \ttrain/aca475ec-bbbf-11e8-b2bb-ac1f6b6435d0 \t3072 \t3072\n\n4 \ttrain/310e3162-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/8a10533e-bba6-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n5 \ttrain/26a0e4f6-bbc6-11e8-b2bc-ac1f6b6435d0 \ttrain/50f4626e-bba8-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n6 \ttrain/31551386-bbc1-11e8-b2bb-ac1f6b6435d0 \ttrain/8549d2f0-bb9f-11e8-b2b9-ac1f6b6435d0 \t2048 \t2048\n\n7 \ttrain/c90611fe-bba2-11e8-b2b9-ac1f6b6435d0 \ttrain/aa197268-bba7-11e8-b2ba-ac1f6b6435d0 \t2048 \t2048\n\n8 \ttrain/76494604-bbaf-11e8-b2ba-ac1f6b6435d0 \ttrain/c7a7a462-bbb1-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n9 \ttrain/1c980c8e-bb9c-11e8-b2b9-ac1f6b6435d0 \ttrain/e30040c2-bbb9-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n10 \ttrain/f7afc760-bbbc-11e8-b2ba-ac1f6b6435d0 \ttrain/2384acac-bbae-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n11 \ttrain/260ace10-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/7138c4aa-bb9b-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072\n\n12 \ttrain/439f3e7a-bbaa-11e8-b2ba-ac1f6b6435d0 \ttrain/dca7670e-bbc4-11e8-b2bc-ac1f6b6435d0 \t2048 \t2048\n\n13 \ttrain/6f094b0a-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/c3200bb2-bbc2-11e8-b2bc-ac1f6b6435d0 \t2048 \t2048\n\n14 \ttrain/74b52b52-bbbc-11e8-b2ba-ac1f6b6435d0 \ttrain/559f7ce0-bbb2-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n15 \ttrain/c43c72e0-bba6-11e8-b2ba-ac1f6b6435d0 \ttrain/82936850-bb9f-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072\n\n16 \ttrain/642c90e8-bba0-11e8-b2b9-ac1f6b6435d0 \ttrain/8e2962fa-bba4-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072\n\n17 \ttest/280f99a0-bad5-11e8-b2b8-ac1f6b6435d0 \ttest/93dde07a-baca-11e8-b2b8-ac1f6b6435d0 \t3072 \t3072\n\n18 \ttest/09b910ec-bac8-11e8-b2b7-ac1f6b6435d0 \ttest/66606722-bace-11e8-b2b8-ac1f6b6435d0 \t2048 \t2048",
      "votes": null
    },
    {
      "id": "412108",
      "postDate": "10/29/2018 15:18:16",
      "content": "<p>Interesting. I just checked these images.</p>",
      "rawMarkdown": "Interesting. I just checked these images.",
      "votes": null
    },
    {
      "id": "412317",
      "postDate": "10/30/2018 01:29:38",
      "content": "<p>The number of near duplicates I found is easily over 100, if not more. I didn't look much further than that into it. I'm not sure how the data can  be used, I still include all images in my training.</p>",
      "rawMarkdown": "The number of near duplicates I found is easily over 100, if not more. I didn't look much further than that into it. I'm not sure how the data can  be used, I still include all images in my training.",
      "votes": null
    },
    {
      "id": "412484",
      "postDate": "10/30/2018 09:18:12",
      "content": "<p>Does all of duplicates (or almost duplicates) have same targets?</p>",
      "rawMarkdown": "Does all of duplicates (or almost duplicates) have same targets?",
      "votes": null
    },
    {
      "id": "412559",
      "postDate": "10/30/2018 11:39:55",
      "content": "<p>I confimed it. Fortunately, all of duplicates have same targets!</p>\n\n<p>img1    img2    file_size_x     file_size_y     Target_x    Target_y</p>\n\n<p>0   train/abfa727e-bba4-11e8-b2ba-ac1f6b6435d0  train/7d7566b6-bbb8-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0 21   25 2 0 21</p>\n\n<p>1   train/10b37fe0-bb9a-11e8-b2b9-ac1f6b6435d0  train/be92e108-bbb5-11e8-b2ba-ac1f6b6435d0  2048    2048    25 3    25 3</p>\n\n<p>2   train/0f9a26dc-bb9e-11e8-b2b9-ac1f6b6435d0  train/be856d8e-bba1-11e8-b2b9-ac1f6b6435d0  2048    2048    25 3    25 3</p>\n\n<p>3   train/3556a318-bbbd-11e8-b2ba-ac1f6b6435d0  train/aca475ec-bbbf-11e8-b2bb-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>4   train/310e3162-bbbe-11e8-b2ba-ac1f6b6435d0  train/8a10533e-bba6-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>5   train/26a0e4f6-bbc6-11e8-b2bc-ac1f6b6435d0  train/50f4626e-bba8-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>6   train/31551386-bbc1-11e8-b2bb-ac1f6b6435d0  train/8549d2f0-bb9f-11e8-b2b9-ac1f6b6435d0  2048    2048    25 3    25 3</p>\n\n<p>7   train/c90611fe-bba2-11e8-b2b9-ac1f6b6435d0  train/aa197268-bba7-11e8-b2ba-ac1f6b6435d0  2048    2048    24  24</p>\n\n<p>8   train/76494604-bbaf-11e8-b2ba-ac1f6b6435d0  train/c7a7a462-bbb1-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0 21   25 2 0 21</p>\n\n<p>9   train/1c980c8e-bb9c-11e8-b2b9-ac1f6b6435d0  train/e30040c2-bbb9-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0 21   25 2 0 21</p>\n\n<p>10  train/f7afc760-bbbc-11e8-b2ba-ac1f6b6435d0  train/2384acac-bbae-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>11  train/260ace10-bbbe-11e8-b2ba-ac1f6b6435d0  train/7138c4aa-bb9b-11e8-b2b9-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>12  train/439f3e7a-bbaa-11e8-b2ba-ac1f6b6435d0  train/dca7670e-bbc4-11e8-b2bc-ac1f6b6435d0  2048    2048    14  14</p>\n\n<p>13  train/6f094b0a-bbbe-11e8-b2ba-ac1f6b6435d0  train/c3200bb2-bbc2-11e8-b2bc-ac1f6b6435d0  2048    2048    24  24</p>\n\n<p>14  train/74b52b52-bbbc-11e8-b2ba-ac1f6b6435d0  train/559f7ce0-bbb2-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>15  train/c43c72e0-bba6-11e8-b2ba-ac1f6b6435d0  train/82936850-bb9f-11e8-b2b9-ac1f6b6435d0  3072    3072    25 2 0 21   25 2 0 21</p>\n\n<p>16  train/642c90e8-bba0-11e8-b2b9-ac1f6b6435d0  train/8e2962fa-bba4-11e8-b2b9-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>I haven't implemented a script that search near duplicates. (I'm thinking of using ImageHash)</p>",
      "rawMarkdown": "I confimed it. Fortunately, all of duplicates have same targets!\n\nimg1 \timg2 \tfile_size_x \tfile_size_y \tTarget_x \tTarget_y\n\n0 \ttrain/abfa727e-bba4-11e8-b2ba-ac1f6b6435d0 \ttrain/7d7566b6-bbb8-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 21 \t25 2 0 21\n\n1 \ttrain/10b37fe0-bb9a-11e8-b2b9-ac1f6b6435d0 \ttrain/be92e108-bbb5-11e8-b2ba-ac1f6b6435d0 \t2048 \t2048 \t25 3 \t25 3\n\n2 \ttrain/0f9a26dc-bb9e-11e8-b2b9-ac1f6b6435d0 \ttrain/be856d8e-bba1-11e8-b2b9-ac1f6b6435d0 \t2048 \t2048 \t25 3 \t25 3\n\n3 \ttrain/3556a318-bbbd-11e8-b2ba-ac1f6b6435d0 \ttrain/aca475ec-bbbf-11e8-b2bb-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n4 \ttrain/310e3162-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/8a10533e-bba6-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n5 \ttrain/26a0e4f6-bbc6-11e8-b2bc-ac1f6b6435d0 \ttrain/50f4626e-bba8-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n6 \ttrain/31551386-bbc1-11e8-b2bb-ac1f6b6435d0 \ttrain/8549d2f0-bb9f-11e8-b2b9-ac1f6b6435d0 \t2048 \t2048 \t25 3 \t25 3\n\n7 \ttrain/c90611fe-bba2-11e8-b2b9-ac1f6b6435d0 \ttrain/aa197268-bba7-11e8-b2ba-ac1f6b6435d0 \t2048 \t2048 \t24 \t24\n\n8 \ttrain/76494604-bbaf-11e8-b2ba-ac1f6b6435d0 \ttrain/c7a7a462-bbb1-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 21 \t25 2 0 21\n\n9 \ttrain/1c980c8e-bb9c-11e8-b2b9-ac1f6b6435d0 \ttrain/e30040c2-bbb9-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 21 \t25 2 0 21\n\n10 \ttrain/f7afc760-bbbc-11e8-b2ba-ac1f6b6435d0 \ttrain/2384acac-bbae-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n11 \ttrain/260ace10-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/7138c4aa-bb9b-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n12 \ttrain/439f3e7a-bbaa-11e8-b2ba-ac1f6b6435d0 \ttrain/dca7670e-bbc4-11e8-b2bc-ac1f6b6435d0 \t2048 \t2048 \t14 \t14\n\n13 \ttrain/6f094b0a-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/c3200bb2-bbc2-11e8-b2bc-ac1f6b6435d0 \t2048 \t2048 \t24 \t24\n\n14 \ttrain/74b52b52-bbbc-11e8-b2ba-ac1f6b6435d0 \ttrain/559f7ce0-bbb2-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n15 \ttrain/c43c72e0-bba6-11e8-b2ba-ac1f6b6435d0 \ttrain/82936850-bb9f-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 21 \t25 2 0 21\n\n16 \ttrain/642c90e8-bba0-11e8-b2b9-ac1f6b6435d0 \ttrain/8e2962fa-bba4-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n\nI haven't implemented a script that search near duplicates. (I'm thinking of using ImageHash)",
      "votes": null
    },
    {
      "id": "415047",
      "postDate": "11/04/2018 08:47:36",
      "content": "<p>this give you an idea how to do do augmentation:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/415047/10596/duplicates.png\" alt=\"enter image description here\"></p>",
      "rawMarkdown": "this give you an idea how to do do augmentation:\n\n\n  ![enter image description here][1]\n\n\n  [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/415047/10596/duplicates.png",
      "votes": null
    },
    {
      "id": "417559",
      "postDate": "11/08/2018 13:01:56",
      "content": "<p>Thank you for valuable info!! \nWe should think about \n1. How to deal with such augmentation data.\n2. Why do their dataset contains such augmentation data.</p>",
      "rawMarkdown": "Thank you for valuable info!! \nWe should think about \n1. How to deal with such augmentation data.\n2. Why do their dataset contains such augmentation data.",
      "votes": null
    },
    {
      "id": "417785",
      "postDate": "11/08/2018 19:28:01",
      "content": "<p>My approach from this is to include everything. Like most datasets, this one is not 100% clean.  Artifacts, duplicates, and other noise exists pretty much everywhere and cleaning a dataset is still resource intensive. Part of the value of these competitions to the sponsors is having hundreds of people dig through  the data and find this sort of thing.</p>",
      "rawMarkdown": "My approach from this is to include everything. Like most datasets, this one is not 100% clean.  Artifacts, duplicates, and other noise exists pretty much everywhere and cleaning a dataset is still resource intensive. Part of the value of these competitions to the sponsors is having hundreds of people dig through  the data and find this sort of thing.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 412108,
      "author_name": "vignam",
      "author_url": "",
      "post_date": "10/29/2018 15:18:16",
      "content": "<p>Interesting. I just checked these images.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 412317,
      "author_name": "ldm314",
      "author_url": "",
      "post_date": "10/30/2018 01:29:38",
      "content": "<p>The number of near duplicates I found is easily over 100, if not more. I didn't look much further than that into it. I'm not sure how the data can  be used, I still include all images in my training.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 412484,
      "author_name": "mortido",
      "author_url": "",
      "post_date": "10/30/2018 09:18:12",
      "content": "<p>Does all of duplicates (or almost duplicates) have same targets?</p>",
      "votes": null,
      "replies": [
        {
          "id": 412559,
          "author_name": "tamreff3290",
          "author_url": "",
          "post_date": "10/30/2018 11:39:55",
          "content": "<p>I confimed it. Fortunately, all of duplicates have same targets!</p>\n\n<p>img1    img2    file_size_x     file_size_y     Target_x    Target_y</p>\n\n<p>0   train/abfa727e-bba4-11e8-b2ba-ac1f6b6435d0  train/7d7566b6-bbb8-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0 21   25 2 0 21</p>\n\n<p>1   train/10b37fe0-bb9a-11e8-b2b9-ac1f6b6435d0  train/be92e108-bbb5-11e8-b2ba-ac1f6b6435d0  2048    2048    25 3    25 3</p>\n\n<p>2   train/0f9a26dc-bb9e-11e8-b2b9-ac1f6b6435d0  train/be856d8e-bba1-11e8-b2b9-ac1f6b6435d0  2048    2048    25 3    25 3</p>\n\n<p>3   train/3556a318-bbbd-11e8-b2ba-ac1f6b6435d0  train/aca475ec-bbbf-11e8-b2bb-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>4   train/310e3162-bbbe-11e8-b2ba-ac1f6b6435d0  train/8a10533e-bba6-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>5   train/26a0e4f6-bbc6-11e8-b2bc-ac1f6b6435d0  train/50f4626e-bba8-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>6   train/31551386-bbc1-11e8-b2bb-ac1f6b6435d0  train/8549d2f0-bb9f-11e8-b2b9-ac1f6b6435d0  2048    2048    25 3    25 3</p>\n\n<p>7   train/c90611fe-bba2-11e8-b2b9-ac1f6b6435d0  train/aa197268-bba7-11e8-b2ba-ac1f6b6435d0  2048    2048    24  24</p>\n\n<p>8   train/76494604-bbaf-11e8-b2ba-ac1f6b6435d0  train/c7a7a462-bbb1-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0 21   25 2 0 21</p>\n\n<p>9   train/1c980c8e-bb9c-11e8-b2b9-ac1f6b6435d0  train/e30040c2-bbb9-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0 21   25 2 0 21</p>\n\n<p>10  train/f7afc760-bbbc-11e8-b2ba-ac1f6b6435d0  train/2384acac-bbae-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>11  train/260ace10-bbbe-11e8-b2ba-ac1f6b6435d0  train/7138c4aa-bb9b-11e8-b2b9-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>12  train/439f3e7a-bbaa-11e8-b2ba-ac1f6b6435d0  train/dca7670e-bbc4-11e8-b2bc-ac1f6b6435d0  2048    2048    14  14</p>\n\n<p>13  train/6f094b0a-bbbe-11e8-b2ba-ac1f6b6435d0  train/c3200bb2-bbc2-11e8-b2bc-ac1f6b6435d0  2048    2048    24  24</p>\n\n<p>14  train/74b52b52-bbbc-11e8-b2ba-ac1f6b6435d0  train/559f7ce0-bbb2-11e8-b2ba-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>15  train/c43c72e0-bba6-11e8-b2ba-ac1f6b6435d0  train/82936850-bb9f-11e8-b2b9-ac1f6b6435d0  3072    3072    25 2 0 21   25 2 0 21</p>\n\n<p>16  train/642c90e8-bba0-11e8-b2b9-ac1f6b6435d0  train/8e2962fa-bba4-11e8-b2b9-ac1f6b6435d0  3072    3072    25 2 0  25 2 0</p>\n\n<p>I haven't implemented a script that search near duplicates. (I'm thinking of using ImageHash)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 415047,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "11/04/2018 08:47:36",
      "content": "<p>this give you an idea how to do do augmentation:</p>\n\n<p><img src=\"https://storage.googleapis.com/kaggle-forum-message-attachments/415047/10596/duplicates.png\" alt=\"enter image description here\"></p>",
      "votes": null,
      "replies": [
        {
          "id": 417559,
          "author_name": "tamreff3290",
          "author_url": "",
          "post_date": "11/08/2018 13:01:56",
          "content": "<p>Thank you for valuable info!! \nWe should think about \n1. How to deal with such augmentation data.\n2. Why do their dataset contains such augmentation data.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 417785,
          "author_name": "ldm314",
          "author_url": "",
          "post_date": "11/08/2018 19:28:01",
          "content": "<p>My approach from this is to include everything. Like most datasets, this one is not 100% clean.  Artifacts, duplicates, and other noise exists pretty much everywhere and cleaning a dataset is still resource intensive. Part of the value of these competitions to the sponsors is having hundreds of people dig through  the data and find this sort of thing.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "412096": "I found odd facts.\n\n# Near-Duplicate images\nBrian reported about duplicate images(https://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/69594). So, I visualized the image and confirmed it. \nStrange to say, it looks slightly different if you look closely. (Maybe the brightness is different?)\n\n# Duplicate images\nOn the other hand, I found duplicate images using the hash function. Further verification was done with 8 times the data which can be made by flip and rotation, but no more duplication was found. \n\nI report \"Duplicate images\".\n\nimg1 \timg2 \timg1_original_size \timg2_original_size\n\n0 \ttrain/abfa727e-bba4-11e8-b2ba-ac1f6b6435d0 \ttrain/7d7566b6-bbb8-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n1 \ttrain/10b37fe0-bb9a-11e8-b2b9-ac1f6b6435d0 \ttrain/be92e108-bbb5-11e8-b2ba-ac1f6b6435d0 \t2048 \t2048\n\n2 \ttrain/0f9a26dc-bb9e-11e8-b2b9-ac1f6b6435d0 \ttrain/be856d8e-bba1-11e8-b2b9-ac1f6b6435d0 \t2048 \t2048\n\n3 \ttrain/3556a318-bbbd-11e8-b2ba-ac1f6b6435d0 \ttrain/aca475ec-bbbf-11e8-b2bb-ac1f6b6435d0 \t3072 \t3072\n\n4 \ttrain/310e3162-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/8a10533e-bba6-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n5 \ttrain/26a0e4f6-bbc6-11e8-b2bc-ac1f6b6435d0 \ttrain/50f4626e-bba8-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n6 \ttrain/31551386-bbc1-11e8-b2bb-ac1f6b6435d0 \ttrain/8549d2f0-bb9f-11e8-b2b9-ac1f6b6435d0 \t2048 \t2048\n\n7 \ttrain/c90611fe-bba2-11e8-b2b9-ac1f6b6435d0 \ttrain/aa197268-bba7-11e8-b2ba-ac1f6b6435d0 \t2048 \t2048\n\n8 \ttrain/76494604-bbaf-11e8-b2ba-ac1f6b6435d0 \ttrain/c7a7a462-bbb1-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n9 \ttrain/1c980c8e-bb9c-11e8-b2b9-ac1f6b6435d0 \ttrain/e30040c2-bbb9-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n10 \ttrain/f7afc760-bbbc-11e8-b2ba-ac1f6b6435d0 \ttrain/2384acac-bbae-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n11 \ttrain/260ace10-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/7138c4aa-bb9b-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072\n\n12 \ttrain/439f3e7a-bbaa-11e8-b2ba-ac1f6b6435d0 \ttrain/dca7670e-bbc4-11e8-b2bc-ac1f6b6435d0 \t2048 \t2048\n\n13 \ttrain/6f094b0a-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/c3200bb2-bbc2-11e8-b2bc-ac1f6b6435d0 \t2048 \t2048\n\n14 \ttrain/74b52b52-bbbc-11e8-b2ba-ac1f6b6435d0 \ttrain/559f7ce0-bbb2-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072\n\n15 \ttrain/c43c72e0-bba6-11e8-b2ba-ac1f6b6435d0 \ttrain/82936850-bb9f-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072\n\n16 \ttrain/642c90e8-bba0-11e8-b2b9-ac1f6b6435d0 \ttrain/8e2962fa-bba4-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072\n\n17 \ttest/280f99a0-bad5-11e8-b2b8-ac1f6b6435d0 \ttest/93dde07a-baca-11e8-b2b8-ac1f6b6435d0 \t3072 \t3072\n\n18 \ttest/09b910ec-bac8-11e8-b2b7-ac1f6b6435d0 \ttest/66606722-bace-11e8-b2b8-ac1f6b6435d0 \t2048 \t2048",
    "412108": "Interesting. I just checked these images.",
    "412317": "The number of near duplicates I found is easily over 100, if not more. I didn't look much further than that into it. I'm not sure how the data can  be used, I still include all images in my training.",
    "412484": "Does all of duplicates (or almost duplicates) have same targets?",
    "412559": "I confimed it. Fortunately, all of duplicates have same targets!\n\nimg1 \timg2 \tfile_size_x \tfile_size_y \tTarget_x \tTarget_y\n\n0 \ttrain/abfa727e-bba4-11e8-b2ba-ac1f6b6435d0 \ttrain/7d7566b6-bbb8-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 21 \t25 2 0 21\n\n1 \ttrain/10b37fe0-bb9a-11e8-b2b9-ac1f6b6435d0 \ttrain/be92e108-bbb5-11e8-b2ba-ac1f6b6435d0 \t2048 \t2048 \t25 3 \t25 3\n\n2 \ttrain/0f9a26dc-bb9e-11e8-b2b9-ac1f6b6435d0 \ttrain/be856d8e-bba1-11e8-b2b9-ac1f6b6435d0 \t2048 \t2048 \t25 3 \t25 3\n\n3 \ttrain/3556a318-bbbd-11e8-b2ba-ac1f6b6435d0 \ttrain/aca475ec-bbbf-11e8-b2bb-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n4 \ttrain/310e3162-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/8a10533e-bba6-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n5 \ttrain/26a0e4f6-bbc6-11e8-b2bc-ac1f6b6435d0 \ttrain/50f4626e-bba8-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n6 \ttrain/31551386-bbc1-11e8-b2bb-ac1f6b6435d0 \ttrain/8549d2f0-bb9f-11e8-b2b9-ac1f6b6435d0 \t2048 \t2048 \t25 3 \t25 3\n\n7 \ttrain/c90611fe-bba2-11e8-b2b9-ac1f6b6435d0 \ttrain/aa197268-bba7-11e8-b2ba-ac1f6b6435d0 \t2048 \t2048 \t24 \t24\n\n8 \ttrain/76494604-bbaf-11e8-b2ba-ac1f6b6435d0 \ttrain/c7a7a462-bbb1-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 21 \t25 2 0 21\n\n9 \ttrain/1c980c8e-bb9c-11e8-b2b9-ac1f6b6435d0 \ttrain/e30040c2-bbb9-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 21 \t25 2 0 21\n\n10 \ttrain/f7afc760-bbbc-11e8-b2ba-ac1f6b6435d0 \ttrain/2384acac-bbae-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n11 \ttrain/260ace10-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/7138c4aa-bb9b-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n12 \ttrain/439f3e7a-bbaa-11e8-b2ba-ac1f6b6435d0 \ttrain/dca7670e-bbc4-11e8-b2bc-ac1f6b6435d0 \t2048 \t2048 \t14 \t14\n\n13 \ttrain/6f094b0a-bbbe-11e8-b2ba-ac1f6b6435d0 \ttrain/c3200bb2-bbc2-11e8-b2bc-ac1f6b6435d0 \t2048 \t2048 \t24 \t24\n\n14 \ttrain/74b52b52-bbbc-11e8-b2ba-ac1f6b6435d0 \ttrain/559f7ce0-bbb2-11e8-b2ba-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n15 \ttrain/c43c72e0-bba6-11e8-b2ba-ac1f6b6435d0 \ttrain/82936850-bb9f-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 21 \t25 2 0 21\n\n16 \ttrain/642c90e8-bba0-11e8-b2b9-ac1f6b6435d0 \ttrain/8e2962fa-bba4-11e8-b2b9-ac1f6b6435d0 \t3072 \t3072 \t25 2 0 \t25 2 0\n\n\nI haven't implemented a script that search near duplicates. (I'm thinking of using ImageHash)",
    "415047": "this give you an idea how to do do augmentation:\n\n\n  ![enter image description here][1]\n\n\n  [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/415047/10596/duplicates.png",
    "417559": "Thank you for valuable info!! \nWe should think about \n1. How to deal with such augmentation data.\n2. Why do their dataset contains such augmentation data.",
    "417785": "My approach from this is to include everything. Like most datasets, this one is not 100% clean.  Artifacts, duplicates, and other noise exists pretty much everywhere and cleaning a dataset is still resource intensive. Part of the value of these competitions to the sponsors is having hundreds of people dig through  the data and find this sort of thing."
  },
  "source": "meta"
}