{
  "id": 32051,
  "title": "List of highly similar images",
  "url": "/competitions/intel-mobileodt-cervical-cancer-screening/discussion/32051",
  "author_name": "",
  "post_date": "2017-04-25T10:34:20.911495200Z",
  "votes": 7,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I noticed that some images in the dataset are highly similar (especially in the additional data).</p>\n\n<p>I decided to use VisiPics to identify groups of highly similar images, and I am sharing here the results. It found 97 duplicates in 47 groups.</p>\n\n<h2>Type 1:</h2>\n\n<p>4028, 4031</p>\n\n<p>5851 , 5845</p>\n\n<p>5988, 5986</p>\n\n<p>5989, 553</p>\n\n<p>6182,6180</p>\n\n<p>6411,6412</p>\n\n<h2>Type 2:</h2>\n\n<p>2376, 2227, 268</p>\n\n<p>3521,3158</p>\n\n<p>354,2938</p>\n\n<p>3555,3372</p>\n\n<p>3725,3051</p>\n\n<p>4068,4067 </p>\n\n<p>4070,4069 (I would group with previous)</p>\n\n<p>5163,5162</p>\n\n<p>6065,6064</p>\n\n<p>6067,6066,6069</p>\n\n<p>6070,6068</p>\n\n<p>6071,1303</p>\n\n<p>6079,6076,6080</p>\n\n<p>6081,6078</p>\n\n<p>6175,6166</p>\n\n<p>6176,6171 (Should group with previous)</p>\n\n<p>6355,6354 (6355 is blurry, so mayber keep 6354)</p>\n\n<p>6404,6402</p>\n\n<p>6519,6517</p>\n\n<p>722,2564</p>\n\n<p>991,3273</p>\n\n<h2>Type 3:</h2>\n\n<p>1729,1402</p>\n\n<p>3824,1407</p>\n\n<p>3855,3710</p>\n\n<p>3984,3983</p>\n\n<p>4464,4462</p>\n\n<p>4616,4613</p>\n\n<p>5107,5106</p>\n\n<p>5439,5436</p>\n\n<p>5781,5780</p>\n\n<p>5951,5950</p>\n\n<p>5956,5952</p>\n\n<p>5957,5953</p>\n\n<p>624,3443</p>\n\n<p>6264,259</p>\n\n<p>6268,6263</p>\n\n<p>6269,6267</p>\n\n<p>6284,6283</p>\n\n<p>6542,6541</p>\n\n<p>693,224</p>\n\n<p>697,6287</p>\n\n<p>If you find more of them, don't hesitate to share.</p>",
  "messages": [
    {
      "id": "177605",
      "postDate": "04/25/2017 10:34:20",
      "content": "<p>I noticed that some images in the dataset are highly similar (especially in the additional data).</p>\n\n<p>I decided to use VisiPics to identify groups of highly similar images, and I am sharing here the results. It found 97 duplicates in 47 groups.</p>\n\n<h2>Type 1:</h2>\n\n<p>4028, 4031</p>\n\n<p>5851 , 5845</p>\n\n<p>5988, 5986</p>\n\n<p>5989, 553</p>\n\n<p>6182,6180</p>\n\n<p>6411,6412</p>\n\n<h2>Type 2:</h2>\n\n<p>2376, 2227, 268</p>\n\n<p>3521,3158</p>\n\n<p>354,2938</p>\n\n<p>3555,3372</p>\n\n<p>3725,3051</p>\n\n<p>4068,4067 </p>\n\n<p>4070,4069 (I would group with previous)</p>\n\n<p>5163,5162</p>\n\n<p>6065,6064</p>\n\n<p>6067,6066,6069</p>\n\n<p>6070,6068</p>\n\n<p>6071,1303</p>\n\n<p>6079,6076,6080</p>\n\n<p>6081,6078</p>\n\n<p>6175,6166</p>\n\n<p>6176,6171 (Should group with previous)</p>\n\n<p>6355,6354 (6355 is blurry, so mayber keep 6354)</p>\n\n<p>6404,6402</p>\n\n<p>6519,6517</p>\n\n<p>722,2564</p>\n\n<p>991,3273</p>\n\n<h2>Type 3:</h2>\n\n<p>1729,1402</p>\n\n<p>3824,1407</p>\n\n<p>3855,3710</p>\n\n<p>3984,3983</p>\n\n<p>4464,4462</p>\n\n<p>4616,4613</p>\n\n<p>5107,5106</p>\n\n<p>5439,5436</p>\n\n<p>5781,5780</p>\n\n<p>5951,5950</p>\n\n<p>5956,5952</p>\n\n<p>5957,5953</p>\n\n<p>624,3443</p>\n\n<p>6264,259</p>\n\n<p>6268,6263</p>\n\n<p>6269,6267</p>\n\n<p>6284,6283</p>\n\n<p>6542,6541</p>\n\n<p>693,224</p>\n\n<p>697,6287</p>\n\n<p>If you find more of them, don't hesitate to share.</p>",
      "rawMarkdown": "I noticed that some images in the dataset are highly similar (especially in the additional data).\n\nI decided to use VisiPics to identify groups of highly similar images, and I am sharing here the results. It found 97 duplicates in 47 groups.\n\n## Type 1:\n\n4028, 4031\n\n5851 , 5845\n\n5988, 5986\n\n5989, 553\n\n6182,6180\n\n6411,6412\n\n## Type 2:\n2376, 2227, 268\n\n3521,3158\n\n354,2938\n\n3555,3372\n\n3725,3051\n\n4068,4067 \n\n4070,4069 (I would group with previous)\n\n5163,5162\n\n6065,6064\n\n6067,6066,6069\n\n6070,6068\n\n6071,1303\n\n6079,6076,6080\n\n6081,6078\n\n6175,6166\n\n6176,6171 (Should group with previous)\n\n6355,6354 (6355 is blurry, so mayber keep 6354)\n\n6404,6402\n\n6519,6517\n\n722,2564\n\n991,3273\n\n## Type 3:\n\n1729,1402\n\n3824,1407\n\n3855,3710\n\n3984,3983\n\n4464,4462\n\n4616,4613\n\n5107,5106\n\n5439,5436\n\n5781,5780\n\n5951,5950\n\n5956,5952\n\n5957,5953\n\n624,3443\n\n6264,259\n\n6268,6263\n\n6269,6267\n\n6284,6283\n\n6542,6541\n\n693,224\n\n697,6287\n\n\nIf you find more of them, don't hesitate to share.",
      "votes": null
    },
    {
      "id": "178441",
      "postDate": "04/27/2017 20:29:16",
      "content": "<p>I made a <a href=\"https://www.kaggle.com/vfdev5/intel-mobileodt-cervical-cancer-screening/type-1-clustering\">Type 1 clustering</a> (which can be easily generalized to Type 2 and Type 3 ), where you can see similar images are grouped together or not far ... \nHIH </p>",
      "rawMarkdown": "I made a [Type 1 clustering](https://www.kaggle.com/vfdev5/intel-mobileodt-cervical-cancer-screening/type-1-clustering) (which can be easily generalized to Type 2 and Type 3 ), where you can see similar images are grouped together or not far ... \nHIH",
      "votes": null
    },
    {
      "id": "179136",
      "postDate": "04/30/2017 10:05:08",
      "content": "<p>Your kernel is really nice. However, it feels like it mostly dicrimates between types of images (lugol/green light/ normal and presence or absence of black circle). What I wanted to have here is a list of images that could impact training performances because they are too similar.</p>",
      "rawMarkdown": "Your kernel is really nice. However, it feels like it mostly dicrimates between types of images (lugol/green light/ normal and presence or absence of black circle). What I wanted to have here is a list of images that could impact training performances because they are too similar.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 178441,
      "author_name": "vfdev5",
      "author_url": "",
      "post_date": "04/27/2017 20:29:16",
      "content": "<p>I made a <a href=\"https://www.kaggle.com/vfdev5/intel-mobileodt-cervical-cancer-screening/type-1-clustering\">Type 1 clustering</a> (which can be easily generalized to Type 2 and Type 3 ), where you can see similar images are grouped together or not far ... \nHIH </p>",
      "votes": null,
      "replies": [
        {
          "id": 179136,
          "author_name": "deveaup",
          "author_url": "",
          "post_date": "04/30/2017 10:05:08",
          "content": "<p>Your kernel is really nice. However, it feels like it mostly dicrimates between types of images (lugol/green light/ normal and presence or absence of black circle). What I wanted to have here is a list of images that could impact training performances because they are too similar.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "177605": "I noticed that some images in the dataset are highly similar (especially in the additional data).\n\nI decided to use VisiPics to identify groups of highly similar images, and I am sharing here the results. It found 97 duplicates in 47 groups.\n\n## Type 1:\n\n4028, 4031\n\n5851 , 5845\n\n5988, 5986\n\n5989, 553\n\n6182,6180\n\n6411,6412\n\n## Type 2:\n2376, 2227, 268\n\n3521,3158\n\n354,2938\n\n3555,3372\n\n3725,3051\n\n4068,4067 \n\n4070,4069 (I would group with previous)\n\n5163,5162\n\n6065,6064\n\n6067,6066,6069\n\n6070,6068\n\n6071,1303\n\n6079,6076,6080\n\n6081,6078\n\n6175,6166\n\n6176,6171 (Should group with previous)\n\n6355,6354 (6355 is blurry, so mayber keep 6354)\n\n6404,6402\n\n6519,6517\n\n722,2564\n\n991,3273\n\n## Type 3:\n\n1729,1402\n\n3824,1407\n\n3855,3710\n\n3984,3983\n\n4464,4462\n\n4616,4613\n\n5107,5106\n\n5439,5436\n\n5781,5780\n\n5951,5950\n\n5956,5952\n\n5957,5953\n\n624,3443\n\n6264,259\n\n6268,6263\n\n6269,6267\n\n6284,6283\n\n6542,6541\n\n693,224\n\n697,6287\n\n\nIf you find more of them, don't hesitate to share.",
    "178441": "I made a [Type 1 clustering](https://www.kaggle.com/vfdev5/intel-mobileodt-cervical-cancer-screening/type-1-clustering) (which can be easily generalized to Type 2 and Type 3 ), where you can see similar images are grouped together or not far ... \nHIH",
    "179136": "Your kernel is really nice. However, it feels like it mostly dicrimates between types of images (lugol/green light/ normal and presence or absence of black circle). What I wanted to have here is a list of images that could impact training performances because they are too similar."
  },
  "source": "meta"
}