Hi everyone!
\n\nThanks a lot @zakajd and ODS for observation about some duplicates in ISIC Archive.
\n\nI would like to share with you my kernel about searhing duplicated images in my dataset (using EfficientNet Embeddings + DBSCAN) and checking some correctness of metadata in marking.
\n\nI have found ~ 430 duplicated samples from ISIC2020! It means that any spliiting of data incorrect: folds contains not uniques samples! I believe that It is one of the reasons for the problem unstable CV. You can see approach and simple eda in my kernel:
\n\n\n\nWelcome!
\n\nP.S. I will add updated StratifyGroupKFold in kernel [Merge External Data] in near future
\n\n[UPDATE]:\nI have seen manually all 486 pairs of duplicates: precision 100%! Recall is not known. It is not \"similar\" images, it is duplcated images. You can trust version5. I will make some experiments with epsilon for finding more duplicates in train and test sets.
\n\n[UPDATE2]:\nIn version 7 you will find ~2369 duplicated images (1132 clusters) in train set. Also you will find information about comparison imagededup and custom DBSCAN approaches for clustering duplicates.
```
\n\n```
", "messages": [ { "id": "882287", "postDate": "06/11/2020 17:29:59", "content": "Hi everyone!
\n\nThanks a lot @zakajd and ODS for observation about some duplicates in ISIC Archive.
\n\nI would like to share with you my kernel about searhing duplicated images in my dataset (using EfficientNet Embeddings + DBSCAN) and checking some correctness of metadata in marking.
\n\nI have found ~ 430 duplicated samples from ISIC2020! It means that any spliiting of data incorrect: folds contains not uniques samples! I believe that It is one of the reasons for the problem unstable CV. You can see approach and simple eda in my kernel:
\n\n\n\nWelcome!
\n\nP.S. I will add updated StratifyGroupKFold in kernel [Merge External Data] in near future
\n\n[UPDATE]:\nI have seen manually all 486 pairs of duplicates: precision 100%! Recall is not known. It is not \"similar\" images, it is duplcated images. You can trust version5. I will make some experiments with epsilon for finding more duplicates in train and test sets.
\n\n[UPDATE2]:\nIn version 7 you will find ~2369 duplicated images (1132 clusters) in train set. Also you will find information about comparison imagededup and custom DBSCAN approaches for clustering duplicates.
```
\n\n```
", "rawMarkdown": "Hi everyone!\n\nThanks a lot @zakajd and ODS for observation about some duplicates in ISIC Archive.\n\nI would like to share with you my kernel about searhing duplicated images in [my dataset](https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg) (using EfficientNet Embeddings + DBSCAN) and checking some correctness of metadata in marking. \n\nI have found ~ 430 duplicated samples from ISIC2020! It means that any spliiting of data incorrect: folds contains not uniques samples! I believe that It is one of the reasons for the problem unstable CV. You can see approach and simple eda in my kernel:\n\n- [[DBSCAN Clustering] Check Marking](https://www.kaggle.com/shonenkov/dbscan-clustering-check-marking)\n\nWelcome!\n\nP.S. I will add updated StratifyGroupKFold in kernel [[Merge External Data]](https://www.kaggle.com/shonenkov/merge-external-data) in near future\n\n[UPDATE]:\nI have seen manually all 486 pairs of duplicates: precision 100%! Recall is not known. It is not \"similar\" images, it is duplcated images. You can trust [version5](https://www.kaggle.com/shonenkov/dbscan-clustering-check-marking?scriptVersionId=36026391). I will make some experiments with epsilon for finding more duplicates in train and test sets.\n\n[UPDATE2]:\nIn [version 7](https://www.kaggle.com/shonenkov/dbscan-clustering-check-marking?scriptVersionId=36206353) you will find ~2369 duplicated images (1132 clusters) in train set. Also you will find information about comparison `imagededup` and custom `DBSCAN` approaches for clustering duplicates.\n\n```\n----------[imagededup]----------\n[Clusters found]: 540\n[Precision]: ~0.954\n--------------------------------\n\n----------[DBSCAN]----------\n[Clusters found]: 1097\n[Precision]: ~0.989\n--------------------------------\n```", "votes": null }, { "id": "882296", "postDate": "06/11/2020 17:41:38", "content": "Thanks, I was really struggling with it...
", "rawMarkdown": "Thanks, I was really struggling with it...", "votes": null }, { "id": "882358", "postDate": "06/11/2020 18:32:25", "content": "WoW great work! Youve helped me so much with your info since the beginning of this comp!
", "rawMarkdown": "WoW great work! Youve helped me so much with your info since the beginning of this comp!", "votes": null }, { "id": "882378", "postDate": "06/11/2020 18:53:38", "content": "@yannmajewski welcome!
", "rawMarkdown": "yannmajewski welcome!", "votes": null }, { "id": "882521", "postDate": "06/11/2020 22:06:52", "content": "Both train and test datasets are affected by duplicated images. I found 223 images in test dataset which are VERY similar 😧
\n\nI used imagededup , see my notebook from another competition. It takes less than a minutes on 512x512 resized images.
Here is the images and their potential duplicates (tool is not perfect):\n```\nImage name -> Similar images\nISIC_7262158 -> ISIC_5746049\nISIC_8746376 -> ISIC_2507875\nISIC_6844611 -> ISIC_0924293\nISIC_3091968 -> ISIC_5914041\nISIC_3341347 -> ISIC_2376329\nISIC_4364994 -> ISIC_0762091\nISIC_0557285 -> ISIC_3256190\nISIC_4892807 -> ISIC_5472411\nISIC_6456904 -> ISIC_8200009\nISIC_1663074 -> ISIC_6192610\nISIC_3555756 -> ISIC_0635396\nISIC_9232173 -> ISIC_7270127\nISIC_5753616 -> ISIC_2689674\nISIC_6082446 -> ISIC_0388802\nISIC_7480177 -> ISIC_8986079\nISIC_0631039 -> ISIC_8668157\nISIC_4874456 -> ISIC_2200299\nISIC_4535526 -> ISIC_3588900\nISIC_0536393 -> ISIC_1975904\nISIC_5176953 -> ISIC_5099850\nISIC_2192031 -> ISIC_8240408\nISIC_8315094 -> ISIC_0554819\nISIC_2234584 -> ISIC_1240474\nISIC_0704435 -> ISIC_2735323\nISIC_2200299 -> ISIC_4874456\nISIC_4751297 -> ISIC_6503142\nISIC_3046977 -> ISIC_3447990\nISIC_5472411 -> ISIC_4892807\nISIC_6901140 -> ISIC_1115450\nISIC_0506082 -> ISIC_1259317\nISIC_4718575 -> ISIC_2391447\nISIC_8965291 -> ISIC_4545487\nISIC_3588900 -> ISIC_4535526\nISIC_4308112 -> ISIC_4818658\nISIC_0559101 -> ISIC_1437807\nISIC_6946895 -> ISIC_4294619\nISIC_9693172 -> ISIC_7556912\nISIC_1706022 -> ISIC_9259972\nISIC_5265808 -> ISIC_7106452\nISIC_7113586 -> ISIC_4443268\nISIC_9931481 -> ISIC_5006966\nISIC_9155599 -> ISIC_8166745\nISIC_0388802 -> ISIC_6082446\nISIC_9552129 -> ISIC_5269066\nISIC_2403393 -> ISIC_0517531\nISIC_2757628 -> ISIC_8284060\nISIC_3508787 -> ISIC_3597521\nISIC_8750817 -> ISIC_4990272\nISIC_0462336 -> ISIC_2866924\nISIC_6976452 -> ISIC_4819603\nISIC_6250185 -> ISIC_3659414\nISIC_9075192 -> ISIC_6668182\nISIC_3447990 -> ISIC_3046977\nISIC_6204751 -> ISIC_4829472\nISIC_7249001 -> ISIC_5788972\nISIC_4872245 -> ISIC_4537621\nISIC_2689674 -> ISIC_5753616\nISIC_3597521 -> ISIC_3508787\nISIC_2391447 -> ISIC_4718575\nISIC_4808024 -> ISIC_4061347\nISIC_4819603 -> ISIC_6976452\nISIC_5914041 -> ISIC_3091968\nISIC_6850984 -> ISIC_5866452\nISIC_4247203 -> ISIC_4967356\nISIC_6503142 -> ISIC_4751297\nISIC_0904702 -> ISIC_4223719\nISIC_0674769 -> ISIC_4720030\nISIC_8874925 -> ISIC_0158761\nISIC_5788972 -> ISIC_7249001\nISIC_3796609 -> ISIC_3945172\nISIC_3276396 -> ISIC_6084895\nISIC_7793652 -> ISIC_6209867\nISIC_6192610 -> ISIC_1663074\nISIC_7556912 -> ISIC_9693172\nISIC_3377854 -> ISIC_6827668\nISIC_6668182 -> ISIC_9075192\nISIC_1115450 -> ISIC_6901140\nISIC_2568903 -> ISIC_5541445\nISIC_4545487 -> ISIC_8965291\nISIC_0434890 -> ISIC_5421651\nISIC_2131119 -> ISIC_9002552\nISIC_9850411 -> ISIC_4156931\nISIC_0924293 -> ISIC_6844611\nISIC_9535767 -> ISIC_6275427\nISIC_7373877 -> ISIC_3819795\nISIC_2394733 -> ISIC_9635083\nISIC_1950045 -> ISIC_6614779\nISIC_9084283 -> ISIC_8683473\nISIC_5198297 -> ISIC_2190511\nISIC_5747282 -> ISIC_1437627\nISIC_4818658 -> ISIC_4308112\nISIC_5078133 -> ISIC_1848959\nISIC_3971415 -> ISIC_2313463\nISIC_9846473 -> ISIC_9640394\nISIC_5911171 -> ISIC_4631399\nISIC_9656943 -> ISIC_1142863\nISIC_8562507 -> ISIC_6717898\nISIC_2862111 -> ISIC_3666478\nISIC_0762091 -> ISIC_4364994\nISIC_4294619 -> ISIC_6946895\nISIC_9388406 -> ISIC_4019458\nISIC_7548504 -> ISIC_4495527\nISIC_3761540 -> ISIC_2890008\nISIC_0936437 -> ISIC_9948221\nISIC_7529810 -> ISIC_3314065\nISIC_6717898 -> ISIC_8562507\nISIC_2866924 -> ISIC_0462336\nISIC_1142863 -> ISIC_9656943\nISIC_5541445 -> ISIC_2568903\nISIC_2722040 -> ISIC_8753442, ISIC_4471121\nISIC_3716403 -> ISIC_8328232\nISIC_8253521 -> ISIC_6248112\nISIC_1808187 -> ISIC_2596120\nISIC_8284060 -> ISIC_2757628\nISIC_9002552 -> ISIC_2131119\nISIC_3819795 -> ISIC_7373877\nISIC_5569800 -> ISIC_1425639\nISIC_6249626 -> ISIC_8648368\nISIC_9008728 -> ISIC_9982577\nISIC_8340210 -> ISIC_3083302\nISIC_7414621 -> ISIC_7005958\nISIC_3884657 -> ISIC_6014626\nISIC_5866452 -> ISIC_6850984\nISIC_8715911 -> ISIC_5467408\nISIC_2890008 -> ISIC_3761540\nISIC_3083302 -> ISIC_8340210\nISIC_7005958 -> ISIC_7414621\nISIC_4829472 -> ISIC_6204751\nISIC_2964421 -> ISIC_8749647\nISIC_9982577 -> ISIC_9008728\nISIC_8661785 -> ISIC_8061920\nISIC_7270127 -> ISIC_9232173\nISIC_3945172 -> ISIC_3796609\nISIC_6014626 -> ISIC_3884657\nISIC_1026838 -> ISIC_4907927\nISIC_4223719 -> ISIC_0904702\nISIC_2190511 -> ISIC_5198297\nISIC_6248112 -> ISIC_8253521\nISIC_8166745 -> ISIC_9155599\nISIC_5006966 -> ISIC_9931481\nISIC_1975904 -> ISIC_0536393\nISIC_1270937 -> ISIC_8748932\nISIC_8683473 -> ISIC_9084283\nISIC_5995007 -> ISIC_7194273\nISIC_6614779 -> ISIC_1950045\nISIC_6275427 -> ISIC_9535767\nISIC_9259972 -> ISIC_1706022\nISIC_4495527 -> ISIC_7548504\nISIC_1848959 -> ISIC_5078133\nISIC_4967356 -> ISIC_4247203\nISIC_5099850 -> ISIC_5176953\nISIC_7194273 -> ISIC_5995007\nISIC_8454848 -> ISIC_6170663\nISIC_0517531 -> ISIC_2403393\nISIC_4443268 -> ISIC_7113586\nISIC_8328232 -> ISIC_3716403\nISIC_7036799 -> ISIC_5735204\nISIC_2507875 -> ISIC_8746376\nISIC_4631399 -> ISIC_5911171\nISIC_5735204 -> ISIC_7036799\nISIC_2313463 -> ISIC_3971415\nISIC_6827668 -> ISIC_3377854\nISIC_6209867 -> ISIC_7793652\nISIC_5872962 -> ISIC_5193073\nISIC_3004197 -> ISIC_5859817\nISIC_5859817 -> ISIC_3004197\nISIC_1171499 -> ISIC_2485658\nISIC_9948221 -> ISIC_0936437\nISIC_4261613 -> ISIC_2258860\nISIC_5576295 -> ISIC_3656810\nISIC_0635396 -> ISIC_3555756\nISIC_1259317 -> ISIC_0506082\nISIC_4061347 -> ISIC_4808024\nISIC_1437627 -> ISIC_5747282\nISIC_2735323 -> ISIC_0704435\nISIC_7106452 -> ISIC_5265808\nISIC_5467408 -> ISIC_8715911\nISIC_1425639 -> ISIC_5569800\nISIC_4720030 -> ISIC_0674769\nISIC_0158761 -> ISIC_8874925\nISIC_3656810 -> ISIC_5576295\nISIC_8209167 -> ISIC_0596499\nISIC_2258860 -> ISIC_4261613\nISIC_1240474 -> ISIC_2234584\nISIC_8061920 -> ISIC_8661785\nISIC_9174152 -> ISIC_0735344\nISIC_2485658 -> ISIC_1171499\nISIC_4156931 -> ISIC_9850411\nISIC_1437807 -> ISIC_0559101\nISIC_2376329 -> ISIC_3341347\nISIC_9247289 -> ISIC_4133898\nISIC_2817101 -> ISIC_5185780\nISIC_5269066 -> ISIC_9552129\nISIC_6084895 -> ISIC_3276396\nISIC_3659414 -> ISIC_6250185\nISIC_8986079 -> ISIC_7480177\nISIC_0596499 -> ISIC_8209167\nISIC_9635083 -> ISIC_2394733\nISIC_8648368 -> ISIC_6249626\nISIC_8240408 -> ISIC_2192031\nISIC_4907927 -> ISIC_1026838\nISIC_9640394 -> ISIC_9846473\nISIC_5193073 -> ISIC_5872962\nISIC_3666478 -> ISIC_2862111\nISIC_8200009 -> ISIC_6456904\nISIC_4019458 -> ISIC_9388406\nISIC_8748932 -> ISIC_1270937\nISIC_8753442 -> ISIC_2722040, ISIC_4471121\nISIC_4990272 -> ISIC_8750817\nISIC_4133898 -> ISIC_9247289\nISIC_8749647 -> ISIC_2964421\nISIC_5421651 -> ISIC_0434890\nISIC_5746049 -> ISIC_7262158\nISIC_6170663 -> ISIC_8454848\nISIC_3256190 -> ISIC_0557285\nISIC_5185780 -> ISIC_2817101\nISIC_4471121 -> ISIC_2722040, ISIC_8753442\nISIC_8668157 -> ISIC_0631039\nISIC_0554819 -> ISIC_8315094\nISIC_3314065 -> ISIC_7529810\nISIC_4537621 -> ISIC_4872245\nISIC_0735344 -> ISIC_9174152\nISIC_2596120 -> ISIC_1808187
\n\nThere are 223 images with similar images over 10982 images.\n```
", "rawMarkdown": "Both train and test datasets are affected by duplicated images. I found 223 images in test dataset which are VERY similar 😧 \n\nI used `imagededup` , see my [notebook from another competition](https://www.kaggle.com/ebouteillon/eda-find-similar-images-in-datasets). It takes less than a minutes on 512x512 resized images.\n\nHere is the images and their potential duplicates (tool is not perfect):\n```\nImage name -> Similar images\nISIC_7262158 -> ISIC_5746049\nISIC_8746376 -> ISIC_2507875\nISIC_6844611 -> ISIC_0924293\nISIC_3091968 -> ISIC_5914041\nISIC_3341347 -> ISIC_2376329\nISIC_4364994 -> ISIC_0762091\nISIC_0557285 -> ISIC_3256190\nISIC_4892807 -> ISIC_5472411\nISIC_6456904 -> ISIC_8200009\nISIC_1663074 -> ISIC_6192610\nISIC_3555756 -> ISIC_0635396\nISIC_9232173 -> ISIC_7270127\nISIC_5753616 -> ISIC_2689674\nISIC_6082446 -> ISIC_0388802\nISIC_7480177 -> ISIC_8986079\nISIC_0631039 -> ISIC_8668157\nISIC_4874456 -> ISIC_2200299\nISIC_4535526 -> ISIC_3588900\nISIC_0536393 -> ISIC_1975904\nISIC_5176953 -> ISIC_5099850\nISIC_2192031 -> ISIC_8240408\nISIC_8315094 -> ISIC_0554819\nISIC_2234584 -> ISIC_1240474\nISIC_0704435 -> ISIC_2735323\nISIC_2200299 -> ISIC_4874456\nISIC_4751297 -> ISIC_6503142\nISIC_3046977 -> ISIC_3447990\nISIC_5472411 -> ISIC_4892807\nISIC_6901140 -> ISIC_1115450\nISIC_0506082 -> ISIC_1259317\nISIC_4718575 -> ISIC_2391447\nISIC_8965291 -> ISIC_4545487\nISIC_3588900 -> ISIC_4535526\nISIC_4308112 -> ISIC_4818658\nISIC_0559101 -> ISIC_1437807\nISIC_6946895 -> ISIC_4294619\nISIC_9693172 -> ISIC_7556912\nISIC_1706022 -> ISIC_9259972\nISIC_5265808 -> ISIC_7106452\nISIC_7113586 -> ISIC_4443268\nISIC_9931481 -> ISIC_5006966\nISIC_9155599 -> ISIC_8166745\nISIC_0388802 -> ISIC_6082446\nISIC_9552129 -> ISIC_5269066\nISIC_2403393 -> ISIC_0517531\nISIC_2757628 -> ISIC_8284060\nISIC_3508787 -> ISIC_3597521\nISIC_8750817 -> ISIC_4990272\nISIC_0462336 -> ISIC_2866924\nISIC_6976452 -> ISIC_4819603\nISIC_6250185 -> ISIC_3659414\nISIC_9075192 -> ISIC_6668182\nISIC_3447990 -> ISIC_3046977\nISIC_6204751 -> ISIC_4829472\nISIC_7249001 -> ISIC_5788972\nISIC_4872245 -> ISIC_4537621\nISIC_2689674 -> ISIC_5753616\nISIC_3597521 -> ISIC_3508787\nISIC_2391447 -> ISIC_4718575\nISIC_4808024 -> ISIC_4061347\nISIC_4819603 -> ISIC_6976452\nISIC_5914041 -> ISIC_3091968\nISIC_6850984 -> ISIC_5866452\nISIC_4247203 -> ISIC_4967356\nISIC_6503142 -> ISIC_4751297\nISIC_0904702 -> ISIC_4223719\nISIC_0674769 -> ISIC_4720030\nISIC_8874925 -> ISIC_0158761\nISIC_5788972 -> ISIC_7249001\nISIC_3796609 -> ISIC_3945172\nISIC_3276396 -> ISIC_6084895\nISIC_7793652 -> ISIC_6209867\nISIC_6192610 -> ISIC_1663074\nISIC_7556912 -> ISIC_9693172\nISIC_3377854 -> ISIC_6827668\nISIC_6668182 -> ISIC_9075192\nISIC_1115450 -> ISIC_6901140\nISIC_2568903 -> ISIC_5541445\nISIC_4545487 -> ISIC_8965291\nISIC_0434890 -> ISIC_5421651\nISIC_2131119 -> ISIC_9002552\nISIC_9850411 -> ISIC_4156931\nISIC_0924293 -> ISIC_6844611\nISIC_9535767 -> ISIC_6275427\nISIC_7373877 -> ISIC_3819795\nISIC_2394733 -> ISIC_9635083\nISIC_1950045 -> ISIC_6614779\nISIC_9084283 -> ISIC_8683473\nISIC_5198297 -> ISIC_2190511\nISIC_5747282 -> ISIC_1437627\nISIC_4818658 -> ISIC_4308112\nISIC_5078133 -> ISIC_1848959\nISIC_3971415 -> ISIC_2313463\nISIC_9846473 -> ISIC_9640394\nISIC_5911171 -> ISIC_4631399\nISIC_9656943 -> ISIC_1142863\nISIC_8562507 -> ISIC_6717898\nISIC_2862111 -> ISIC_3666478\nISIC_0762091 -> ISIC_4364994\nISIC_4294619 -> ISIC_6946895\nISIC_9388406 -> ISIC_4019458\nISIC_7548504 -> ISIC_4495527\nISIC_3761540 -> ISIC_2890008\nISIC_0936437 -> ISIC_9948221\nISIC_7529810 -> ISIC_3314065\nISIC_6717898 -> ISIC_8562507\nISIC_2866924 -> ISIC_0462336\nISIC_1142863 -> ISIC_9656943\nISIC_5541445 -> ISIC_2568903\nISIC_2722040 -> ISIC_8753442, ISIC_4471121\nISIC_3716403 -> ISIC_8328232\nISIC_8253521 -> ISIC_6248112\nISIC_1808187 -> ISIC_2596120\nISIC_8284060 -> ISIC_2757628\nISIC_9002552 -> ISIC_2131119\nISIC_3819795 -> ISIC_7373877\nISIC_5569800 -> ISIC_1425639\nISIC_6249626 -> ISIC_8648368\nISIC_9008728 -> ISIC_9982577\nISIC_8340210 -> ISIC_3083302\nISIC_7414621 -> ISIC_7005958\nISIC_3884657 -> ISIC_6014626\nISIC_5866452 -> ISIC_6850984\nISIC_8715911 -> ISIC_5467408\nISIC_2890008 -> ISIC_3761540\nISIC_3083302 -> ISIC_8340210\nISIC_7005958 -> ISIC_7414621\nISIC_4829472 -> ISIC_6204751\nISIC_2964421 -> ISIC_8749647\nISIC_9982577 -> ISIC_9008728\nISIC_8661785 -> ISIC_8061920\nISIC_7270127 -> ISIC_9232173\nISIC_3945172 -> ISIC_3796609\nISIC_6014626 -> ISIC_3884657\nISIC_1026838 -> ISIC_4907927\nISIC_4223719 -> ISIC_0904702\nISIC_2190511 -> ISIC_5198297\nISIC_6248112 -> ISIC_8253521\nISIC_8166745 -> ISIC_9155599\nISIC_5006966 -> ISIC_9931481\nISIC_1975904 -> ISIC_0536393\nISIC_1270937 -> ISIC_8748932\nISIC_8683473 -> ISIC_9084283\nISIC_5995007 -> ISIC_7194273\nISIC_6614779 -> ISIC_1950045\nISIC_6275427 -> ISIC_9535767\nISIC_9259972 -> ISIC_1706022\nISIC_4495527 -> ISIC_7548504\nISIC_1848959 -> ISIC_5078133\nISIC_4967356 -> ISIC_4247203\nISIC_5099850 -> ISIC_5176953\nISIC_7194273 -> ISIC_5995007\nISIC_8454848 -> ISIC_6170663\nISIC_0517531 -> ISIC_2403393\nISIC_4443268 -> ISIC_7113586\nISIC_8328232 -> ISIC_3716403\nISIC_7036799 -> ISIC_5735204\nISIC_2507875 -> ISIC_8746376\nISIC_4631399 -> ISIC_5911171\nISIC_5735204 -> ISIC_7036799\nISIC_2313463 -> ISIC_3971415\nISIC_6827668 -> ISIC_3377854\nISIC_6209867 -> ISIC_7793652\nISIC_5872962 -> ISIC_5193073\nISIC_3004197 -> ISIC_5859817\nISIC_5859817 -> ISIC_3004197\nISIC_1171499 -> ISIC_2485658\nISIC_9948221 -> ISIC_0936437\nISIC_4261613 -> ISIC_2258860\nISIC_5576295 -> ISIC_3656810\nISIC_0635396 -> ISIC_3555756\nISIC_1259317 -> ISIC_0506082\nISIC_4061347 -> ISIC_4808024\nISIC_1437627 -> ISIC_5747282\nISIC_2735323 -> ISIC_0704435\nISIC_7106452 -> ISIC_5265808\nISIC_5467408 -> ISIC_8715911\nISIC_1425639 -> ISIC_5569800\nISIC_4720030 -> ISIC_0674769\nISIC_0158761 -> ISIC_8874925\nISIC_3656810 -> ISIC_5576295\nISIC_8209167 -> ISIC_0596499\nISIC_2258860 -> ISIC_4261613\nISIC_1240474 -> ISIC_2234584\nISIC_8061920 -> ISIC_8661785\nISIC_9174152 -> ISIC_0735344\nISIC_2485658 -> ISIC_1171499\nISIC_4156931 -> ISIC_9850411\nISIC_1437807 -> ISIC_0559101\nISIC_2376329 -> ISIC_3341347\nISIC_9247289 -> ISIC_4133898\nISIC_2817101 -> ISIC_5185780\nISIC_5269066 -> ISIC_9552129\nISIC_6084895 -> ISIC_3276396\nISIC_3659414 -> ISIC_6250185\nISIC_8986079 -> ISIC_7480177\nISIC_0596499 -> ISIC_8209167\nISIC_9635083 -> ISIC_2394733\nISIC_8648368 -> ISIC_6249626\nISIC_8240408 -> ISIC_2192031\nISIC_4907927 -> ISIC_1026838\nISIC_9640394 -> ISIC_9846473\nISIC_5193073 -> ISIC_5872962\nISIC_3666478 -> ISIC_2862111\nISIC_8200009 -> ISIC_6456904\nISIC_4019458 -> ISIC_9388406\nISIC_8748932 -> ISIC_1270937\nISIC_8753442 -> ISIC_2722040, ISIC_4471121\nISIC_4990272 -> ISIC_8750817\nISIC_4133898 -> ISIC_9247289\nISIC_8749647 -> ISIC_2964421\nISIC_5421651 -> ISIC_0434890\nISIC_5746049 -> ISIC_7262158\nISIC_6170663 -> ISIC_8454848\nISIC_3256190 -> ISIC_0557285\nISIC_5185780 -> ISIC_2817101\nISIC_4471121 -> ISIC_2722040, ISIC_8753442\nISIC_8668157 -> ISIC_0631039\nISIC_0554819 -> ISIC_8315094\nISIC_3314065 -> ISIC_7529810\nISIC_4537621 -> ISIC_4872245\nISIC_0735344 -> ISIC_9174152\nISIC_2596120 -> ISIC_1808187\n\nThere are 223 images with similar images over 10982 images.\n```", "votes": null }, { "id": "882645", "postDate": "06/12/2020 02:55:13", "content": "@ebouteillon thanks for sharing! You are right, I didn’t use it for test dataset, as I tried to solve problem with unstable validation. Your information is very useful, I will try it.
\n\nP.S. DBSCAN looks like a perfect 😂 all of them are really duplicates, maybe It didn’t find all cases. I think value of epsilon is important.
", "rawMarkdown": "ebouteillon thanks for sharing! You are right, I didn’t use it for test dataset, as I tried to solve problem with unstable validation. Your information is very useful, I will try it. \n\nP.S. DBSCAN looks like a perfect 😂 all of them are really duplicates, maybe It didn’t find all cases. I think value of epsilon is important.", "votes": null }, { "id": "883227", "postDate": "06/12/2020 13:16:11", "content": "thanks
", "rawMarkdown": "thanks", "votes": null }, { "id": "883282", "postDate": "06/12/2020 14:09:37", "content": "I think having duplicates in test set is an issue. If some of these images are present in public leaderboard and the \"similar\" images are in the private leaderboard, then you can leak some results from the private leaderboard by probing the public leaderboard.
\n\n@juliaelliott @philculliton @veronicarotemberg can you confirm there is no such leak?
", "rawMarkdown": "I think having duplicates in test set is an issue. If some of these images are present in public leaderboard and the \"similar\" images are in the private leaderboard, then you can leak some results from the private leaderboard by probing the public leaderboard.\n\n@juliaelliott @philculliton @veronicarotemberg can you confirm there is no such leak?", "votes": null }, { "id": "883517", "postDate": "06/12/2020 17:51:34", "content": "Since there are duplicate images with differing meta data, does it even make sense to use the meta data for training?
", "rawMarkdown": "Since there are duplicate images with differing meta data, does it even make sense to use the meta data for training?", "votes": null }, { "id": "888093", "postDate": "06/16/2020 05:50:45", "content": "@ebouteillon We are looking into this with the hosts and will respond as soon as possible.
", "rawMarkdown": "ebouteillon We are looking into this with the hosts and will respond as soon as possible.", "votes": null }, { "id": "903474", "postDate": "06/26/2020 21:19:14", "content": "Alright, the host has issued this explanation that addresses dupes, both that you have identified and some additional ones uncovered. The short of it is that they acknowledge these dupes exist and have confirmed they are okay to use. Obviously, manually labeling the test set with this knowledge is still prohibited, but these images are still free to include in training.
", "rawMarkdown": "Alright, the host has [issued this explanation](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161943) that addresses dupes, both that you have identified and some additional ones uncovered. The short of it is that they acknowledge these dupes exist and have confirmed they are okay to use. Obviously, manually labeling the test set with this knowledge is still prohibited, but these images are still free to include in training.", "votes": null }, { "id": "903513", "postDate": "06/26/2020 22:41:10", "content": "Thanks for the clarification!
", "rawMarkdown": "Thanks for the clarification!", "votes": null } ], "comments": [ { "id": 882296, "author_name": "awsaf49", "author_url": "", "post_date": "06/11/2020 17:41:38", "content": "Thanks, I was really struggling with it...
", "votes": null, "replies": [] }, { "id": 882358, "author_name": "yannmajewski", "author_url": "", "post_date": "06/11/2020 18:32:25", "content": "WoW great work! Youve helped me so much with your info since the beginning of this comp!
", "votes": null, "replies": [ { "id": 882378, "author_name": "shonenkov", "author_url": "", "post_date": "06/11/2020 18:53:38", "content": "@yannmajewski welcome!
", "votes": null, "replies": [] } ] }, { "id": 882521, "author_name": "ebouteillon", "author_url": "", "post_date": "06/11/2020 22:06:52", "content": "Both train and test datasets are affected by duplicated images. I found 223 images in test dataset which are VERY similar 😧
\n\nI used imagededup , see my notebook from another competition. It takes less than a minutes on 512x512 resized images.
Here is the images and their potential duplicates (tool is not perfect):\n```\nImage name -> Similar images\nISIC_7262158 -> ISIC_5746049\nISIC_8746376 -> ISIC_2507875\nISIC_6844611 -> ISIC_0924293\nISIC_3091968 -> ISIC_5914041\nISIC_3341347 -> ISIC_2376329\nISIC_4364994 -> ISIC_0762091\nISIC_0557285 -> ISIC_3256190\nISIC_4892807 -> ISIC_5472411\nISIC_6456904 -> ISIC_8200009\nISIC_1663074 -> ISIC_6192610\nISIC_3555756 -> ISIC_0635396\nISIC_9232173 -> ISIC_7270127\nISIC_5753616 -> ISIC_2689674\nISIC_6082446 -> ISIC_0388802\nISIC_7480177 -> ISIC_8986079\nISIC_0631039 -> ISIC_8668157\nISIC_4874456 -> ISIC_2200299\nISIC_4535526 -> ISIC_3588900\nISIC_0536393 -> ISIC_1975904\nISIC_5176953 -> ISIC_5099850\nISIC_2192031 -> ISIC_8240408\nISIC_8315094 -> ISIC_0554819\nISIC_2234584 -> ISIC_1240474\nISIC_0704435 -> ISIC_2735323\nISIC_2200299 -> ISIC_4874456\nISIC_4751297 -> ISIC_6503142\nISIC_3046977 -> ISIC_3447990\nISIC_5472411 -> ISIC_4892807\nISIC_6901140 -> ISIC_1115450\nISIC_0506082 -> ISIC_1259317\nISIC_4718575 -> ISIC_2391447\nISIC_8965291 -> ISIC_4545487\nISIC_3588900 -> ISIC_4535526\nISIC_4308112 -> ISIC_4818658\nISIC_0559101 -> ISIC_1437807\nISIC_6946895 -> ISIC_4294619\nISIC_9693172 -> ISIC_7556912\nISIC_1706022 -> ISIC_9259972\nISIC_5265808 -> ISIC_7106452\nISIC_7113586 -> ISIC_4443268\nISIC_9931481 -> ISIC_5006966\nISIC_9155599 -> ISIC_8166745\nISIC_0388802 -> ISIC_6082446\nISIC_9552129 -> ISIC_5269066\nISIC_2403393 -> ISIC_0517531\nISIC_2757628 -> ISIC_8284060\nISIC_3508787 -> ISIC_3597521\nISIC_8750817 -> ISIC_4990272\nISIC_0462336 -> ISIC_2866924\nISIC_6976452 -> ISIC_4819603\nISIC_6250185 -> ISIC_3659414\nISIC_9075192 -> ISIC_6668182\nISIC_3447990 -> ISIC_3046977\nISIC_6204751 -> ISIC_4829472\nISIC_7249001 -> ISIC_5788972\nISIC_4872245 -> ISIC_4537621\nISIC_2689674 -> ISIC_5753616\nISIC_3597521 -> ISIC_3508787\nISIC_2391447 -> ISIC_4718575\nISIC_4808024 -> ISIC_4061347\nISIC_4819603 -> ISIC_6976452\nISIC_5914041 -> ISIC_3091968\nISIC_6850984 -> ISIC_5866452\nISIC_4247203 -> ISIC_4967356\nISIC_6503142 -> ISIC_4751297\nISIC_0904702 -> ISIC_4223719\nISIC_0674769 -> ISIC_4720030\nISIC_8874925 -> ISIC_0158761\nISIC_5788972 -> ISIC_7249001\nISIC_3796609 -> ISIC_3945172\nISIC_3276396 -> ISIC_6084895\nISIC_7793652 -> ISIC_6209867\nISIC_6192610 -> ISIC_1663074\nISIC_7556912 -> ISIC_9693172\nISIC_3377854 -> ISIC_6827668\nISIC_6668182 -> ISIC_9075192\nISIC_1115450 -> ISIC_6901140\nISIC_2568903 -> ISIC_5541445\nISIC_4545487 -> ISIC_8965291\nISIC_0434890 -> ISIC_5421651\nISIC_2131119 -> ISIC_9002552\nISIC_9850411 -> ISIC_4156931\nISIC_0924293 -> ISIC_6844611\nISIC_9535767 -> ISIC_6275427\nISIC_7373877 -> ISIC_3819795\nISIC_2394733 -> ISIC_9635083\nISIC_1950045 -> ISIC_6614779\nISIC_9084283 -> ISIC_8683473\nISIC_5198297 -> ISIC_2190511\nISIC_5747282 -> ISIC_1437627\nISIC_4818658 -> ISIC_4308112\nISIC_5078133 -> ISIC_1848959\nISIC_3971415 -> ISIC_2313463\nISIC_9846473 -> ISIC_9640394\nISIC_5911171 -> ISIC_4631399\nISIC_9656943 -> ISIC_1142863\nISIC_8562507 -> ISIC_6717898\nISIC_2862111 -> ISIC_3666478\nISIC_0762091 -> ISIC_4364994\nISIC_4294619 -> ISIC_6946895\nISIC_9388406 -> ISIC_4019458\nISIC_7548504 -> ISIC_4495527\nISIC_3761540 -> ISIC_2890008\nISIC_0936437 -> ISIC_9948221\nISIC_7529810 -> ISIC_3314065\nISIC_6717898 -> ISIC_8562507\nISIC_2866924 -> ISIC_0462336\nISIC_1142863 -> ISIC_9656943\nISIC_5541445 -> ISIC_2568903\nISIC_2722040 -> ISIC_8753442, ISIC_4471121\nISIC_3716403 -> ISIC_8328232\nISIC_8253521 -> ISIC_6248112\nISIC_1808187 -> ISIC_2596120\nISIC_8284060 -> ISIC_2757628\nISIC_9002552 -> ISIC_2131119\nISIC_3819795 -> ISIC_7373877\nISIC_5569800 -> ISIC_1425639\nISIC_6249626 -> ISIC_8648368\nISIC_9008728 -> ISIC_9982577\nISIC_8340210 -> ISIC_3083302\nISIC_7414621 -> ISIC_7005958\nISIC_3884657 -> ISIC_6014626\nISIC_5866452 -> ISIC_6850984\nISIC_8715911 -> ISIC_5467408\nISIC_2890008 -> ISIC_3761540\nISIC_3083302 -> ISIC_8340210\nISIC_7005958 -> ISIC_7414621\nISIC_4829472 -> ISIC_6204751\nISIC_2964421 -> ISIC_8749647\nISIC_9982577 -> ISIC_9008728\nISIC_8661785 -> ISIC_8061920\nISIC_7270127 -> ISIC_9232173\nISIC_3945172 -> ISIC_3796609\nISIC_6014626 -> ISIC_3884657\nISIC_1026838 -> ISIC_4907927\nISIC_4223719 -> ISIC_0904702\nISIC_2190511 -> ISIC_5198297\nISIC_6248112 -> ISIC_8253521\nISIC_8166745 -> ISIC_9155599\nISIC_5006966 -> ISIC_9931481\nISIC_1975904 -> ISIC_0536393\nISIC_1270937 -> ISIC_8748932\nISIC_8683473 -> ISIC_9084283\nISIC_5995007 -> ISIC_7194273\nISIC_6614779 -> ISIC_1950045\nISIC_6275427 -> ISIC_9535767\nISIC_9259972 -> ISIC_1706022\nISIC_4495527 -> ISIC_7548504\nISIC_1848959 -> ISIC_5078133\nISIC_4967356 -> ISIC_4247203\nISIC_5099850 -> ISIC_5176953\nISIC_7194273 -> ISIC_5995007\nISIC_8454848 -> ISIC_6170663\nISIC_0517531 -> ISIC_2403393\nISIC_4443268 -> ISIC_7113586\nISIC_8328232 -> ISIC_3716403\nISIC_7036799 -> ISIC_5735204\nISIC_2507875 -> ISIC_8746376\nISIC_4631399 -> ISIC_5911171\nISIC_5735204 -> ISIC_7036799\nISIC_2313463 -> ISIC_3971415\nISIC_6827668 -> ISIC_3377854\nISIC_6209867 -> ISIC_7793652\nISIC_5872962 -> ISIC_5193073\nISIC_3004197 -> ISIC_5859817\nISIC_5859817 -> ISIC_3004197\nISIC_1171499 -> ISIC_2485658\nISIC_9948221 -> ISIC_0936437\nISIC_4261613 -> ISIC_2258860\nISIC_5576295 -> ISIC_3656810\nISIC_0635396 -> ISIC_3555756\nISIC_1259317 -> ISIC_0506082\nISIC_4061347 -> ISIC_4808024\nISIC_1437627 -> ISIC_5747282\nISIC_2735323 -> ISIC_0704435\nISIC_7106452 -> ISIC_5265808\nISIC_5467408 -> ISIC_8715911\nISIC_1425639 -> ISIC_5569800\nISIC_4720030 -> ISIC_0674769\nISIC_0158761 -> ISIC_8874925\nISIC_3656810 -> ISIC_5576295\nISIC_8209167 -> ISIC_0596499\nISIC_2258860 -> ISIC_4261613\nISIC_1240474 -> ISIC_2234584\nISIC_8061920 -> ISIC_8661785\nISIC_9174152 -> ISIC_0735344\nISIC_2485658 -> ISIC_1171499\nISIC_4156931 -> ISIC_9850411\nISIC_1437807 -> ISIC_0559101\nISIC_2376329 -> ISIC_3341347\nISIC_9247289 -> ISIC_4133898\nISIC_2817101 -> ISIC_5185780\nISIC_5269066 -> ISIC_9552129\nISIC_6084895 -> ISIC_3276396\nISIC_3659414 -> ISIC_6250185\nISIC_8986079 -> ISIC_7480177\nISIC_0596499 -> ISIC_8209167\nISIC_9635083 -> ISIC_2394733\nISIC_8648368 -> ISIC_6249626\nISIC_8240408 -> ISIC_2192031\nISIC_4907927 -> ISIC_1026838\nISIC_9640394 -> ISIC_9846473\nISIC_5193073 -> ISIC_5872962\nISIC_3666478 -> ISIC_2862111\nISIC_8200009 -> ISIC_6456904\nISIC_4019458 -> ISIC_9388406\nISIC_8748932 -> ISIC_1270937\nISIC_8753442 -> ISIC_2722040, ISIC_4471121\nISIC_4990272 -> ISIC_8750817\nISIC_4133898 -> ISIC_9247289\nISIC_8749647 -> ISIC_2964421\nISIC_5421651 -> ISIC_0434890\nISIC_5746049 -> ISIC_7262158\nISIC_6170663 -> ISIC_8454848\nISIC_3256190 -> ISIC_0557285\nISIC_5185780 -> ISIC_2817101\nISIC_4471121 -> ISIC_2722040, ISIC_8753442\nISIC_8668157 -> ISIC_0631039\nISIC_0554819 -> ISIC_8315094\nISIC_3314065 -> ISIC_7529810\nISIC_4537621 -> ISIC_4872245\nISIC_0735344 -> ISIC_9174152\nISIC_2596120 -> ISIC_1808187
\n\nThere are 223 images with similar images over 10982 images.\n```
", "votes": null, "replies": [ { "id": 882645, "author_name": "shonenkov", "author_url": "", "post_date": "06/12/2020 02:55:13", "content": "@ebouteillon thanks for sharing! You are right, I didn’t use it for test dataset, as I tried to solve problem with unstable validation. Your information is very useful, I will try it.
\n\nP.S. DBSCAN looks like a perfect 😂 all of them are really duplicates, maybe It didn’t find all cases. I think value of epsilon is important.
", "votes": null, "replies": [] }, { "id": 883282, "author_name": "ebouteillon", "author_url": "", "post_date": "06/12/2020 14:09:37", "content": "I think having duplicates in test set is an issue. If some of these images are present in public leaderboard and the \"similar\" images are in the private leaderboard, then you can leak some results from the private leaderboard by probing the public leaderboard.
\n\n@juliaelliott @philculliton @veronicarotemberg can you confirm there is no such leak?
", "votes": null, "replies": [] }, { "id": 888093, "author_name": "juliaelliott", "author_url": "", "post_date": "06/16/2020 05:50:45", "content": "@ebouteillon We are looking into this with the hosts and will respond as soon as possible.
", "votes": null, "replies": [] }, { "id": 903474, "author_name": "juliaelliott", "author_url": "", "post_date": "06/26/2020 21:19:14", "content": "Alright, the host has issued this explanation that addresses dupes, both that you have identified and some additional ones uncovered. The short of it is that they acknowledge these dupes exist and have confirmed they are okay to use. Obviously, manually labeling the test set with this knowledge is still prohibited, but these images are still free to include in training.
", "votes": null, "replies": [] }, { "id": 903513, "author_name": "ebouteillon", "author_url": "", "post_date": "06/26/2020 22:41:10", "content": "Thanks for the clarification!
", "votes": null, "replies": [] } ] }, { "id": 883227, "author_name": "bixiaopeng", "author_url": "", "post_date": "06/12/2020 13:16:11", "content": "thanks
", "votes": null, "replies": [] }, { "id": 883517, "author_name": "cronoz30", "author_url": "", "post_date": "06/12/2020 17:51:34", "content": "Since there are duplicate images with differing meta data, does it even make sense to use the meta data for training?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "882287": "Hi everyone!\n\nThanks a lot @zakajd and ODS for observation about some duplicates in ISIC Archive.\n\nI would like to share with you my kernel about searhing duplicated images in [my dataset](https://www.kaggle.com/shonenkov/melanoma-merged-external-data-512x512-jpeg) (using EfficientNet Embeddings + DBSCAN) and checking some correctness of metadata in marking. \n\nI have found ~ 430 duplicated samples from ISIC2020! It means that any spliiting of data incorrect: folds contains not uniques samples! I believe that It is one of the reasons for the problem unstable CV. You can see approach and simple eda in my kernel:\n\n- [[DBSCAN Clustering] Check Marking](https://www.kaggle.com/shonenkov/dbscan-clustering-check-marking)\n\nWelcome!\n\nP.S. I will add updated StratifyGroupKFold in kernel [[Merge External Data]](https://www.kaggle.com/shonenkov/merge-external-data) in near future\n\n[UPDATE]:\nI have seen manually all 486 pairs of duplicates: precision 100%! Recall is not known. It is not \"similar\" images, it is duplcated images. You can trust [version5](https://www.kaggle.com/shonenkov/dbscan-clustering-check-marking?scriptVersionId=36026391). I will make some experiments with epsilon for finding more duplicates in train and test sets.\n\n[UPDATE2]:\nIn [version 7](https://www.kaggle.com/shonenkov/dbscan-clustering-check-marking?scriptVersionId=36206353) you will find ~2369 duplicated images (1132 clusters) in train set. Also you will find information about comparison `imagededup` and custom `DBSCAN` approaches for clustering duplicates.\n\n```\n----------[imagededup]----------\n[Clusters found]: 540\n[Precision]: ~0.954\n--------------------------------\n\n----------[DBSCAN]----------\n[Clusters found]: 1097\n[Precision]: ~0.989\n--------------------------------\n```", "882296": "Thanks, I was really struggling with it...", "882358": "WoW great work! Youve helped me so much with your info since the beginning of this comp!", "882378": "yannmajewski welcome!", "882521": "Both train and test datasets are affected by duplicated images. I found 223 images in test dataset which are VERY similar 😧 \n\nI used `imagededup` , see my [notebook from another competition](https://www.kaggle.com/ebouteillon/eda-find-similar-images-in-datasets). It takes less than a minutes on 512x512 resized images.\n\nHere is the images and their potential duplicates (tool is not perfect):\n```\nImage name -> Similar images\nISIC_7262158 -> ISIC_5746049\nISIC_8746376 -> ISIC_2507875\nISIC_6844611 -> ISIC_0924293\nISIC_3091968 -> ISIC_5914041\nISIC_3341347 -> ISIC_2376329\nISIC_4364994 -> ISIC_0762091\nISIC_0557285 -> ISIC_3256190\nISIC_4892807 -> ISIC_5472411\nISIC_6456904 -> ISIC_8200009\nISIC_1663074 -> ISIC_6192610\nISIC_3555756 -> ISIC_0635396\nISIC_9232173 -> ISIC_7270127\nISIC_5753616 -> ISIC_2689674\nISIC_6082446 -> ISIC_0388802\nISIC_7480177 -> ISIC_8986079\nISIC_0631039 -> ISIC_8668157\nISIC_4874456 -> ISIC_2200299\nISIC_4535526 -> ISIC_3588900\nISIC_0536393 -> ISIC_1975904\nISIC_5176953 -> ISIC_5099850\nISIC_2192031 -> ISIC_8240408\nISIC_8315094 -> ISIC_0554819\nISIC_2234584 -> ISIC_1240474\nISIC_0704435 -> ISIC_2735323\nISIC_2200299 -> ISIC_4874456\nISIC_4751297 -> ISIC_6503142\nISIC_3046977 -> ISIC_3447990\nISIC_5472411 -> ISIC_4892807\nISIC_6901140 -> ISIC_1115450\nISIC_0506082 -> ISIC_1259317\nISIC_4718575 -> ISIC_2391447\nISIC_8965291 -> ISIC_4545487\nISIC_3588900 -> ISIC_4535526\nISIC_4308112 -> ISIC_4818658\nISIC_0559101 -> ISIC_1437807\nISIC_6946895 -> ISIC_4294619\nISIC_9693172 -> ISIC_7556912\nISIC_1706022 -> ISIC_9259972\nISIC_5265808 -> ISIC_7106452\nISIC_7113586 -> ISIC_4443268\nISIC_9931481 -> ISIC_5006966\nISIC_9155599 -> ISIC_8166745\nISIC_0388802 -> ISIC_6082446\nISIC_9552129 -> ISIC_5269066\nISIC_2403393 -> ISIC_0517531\nISIC_2757628 -> ISIC_8284060\nISIC_3508787 -> ISIC_3597521\nISIC_8750817 -> ISIC_4990272\nISIC_0462336 -> ISIC_2866924\nISIC_6976452 -> ISIC_4819603\nISIC_6250185 -> ISIC_3659414\nISIC_9075192 -> ISIC_6668182\nISIC_3447990 -> ISIC_3046977\nISIC_6204751 -> ISIC_4829472\nISIC_7249001 -> ISIC_5788972\nISIC_4872245 -> ISIC_4537621\nISIC_2689674 -> ISIC_5753616\nISIC_3597521 -> ISIC_3508787\nISIC_2391447 -> ISIC_4718575\nISIC_4808024 -> ISIC_4061347\nISIC_4819603 -> ISIC_6976452\nISIC_5914041 -> ISIC_3091968\nISIC_6850984 -> ISIC_5866452\nISIC_4247203 -> ISIC_4967356\nISIC_6503142 -> ISIC_4751297\nISIC_0904702 -> ISIC_4223719\nISIC_0674769 -> ISIC_4720030\nISIC_8874925 -> ISIC_0158761\nISIC_5788972 -> ISIC_7249001\nISIC_3796609 -> ISIC_3945172\nISIC_3276396 -> ISIC_6084895\nISIC_7793652 -> ISIC_6209867\nISIC_6192610 -> ISIC_1663074\nISIC_7556912 -> ISIC_9693172\nISIC_3377854 -> ISIC_6827668\nISIC_6668182 -> ISIC_9075192\nISIC_1115450 -> ISIC_6901140\nISIC_2568903 -> ISIC_5541445\nISIC_4545487 -> ISIC_8965291\nISIC_0434890 -> ISIC_5421651\nISIC_2131119 -> ISIC_9002552\nISIC_9850411 -> ISIC_4156931\nISIC_0924293 -> ISIC_6844611\nISIC_9535767 -> ISIC_6275427\nISIC_7373877 -> ISIC_3819795\nISIC_2394733 -> ISIC_9635083\nISIC_1950045 -> ISIC_6614779\nISIC_9084283 -> ISIC_8683473\nISIC_5198297 -> ISIC_2190511\nISIC_5747282 -> ISIC_1437627\nISIC_4818658 -> ISIC_4308112\nISIC_5078133 -> ISIC_1848959\nISIC_3971415 -> ISIC_2313463\nISIC_9846473 -> ISIC_9640394\nISIC_5911171 -> ISIC_4631399\nISIC_9656943 -> ISIC_1142863\nISIC_8562507 -> ISIC_6717898\nISIC_2862111 -> ISIC_3666478\nISIC_0762091 -> ISIC_4364994\nISIC_4294619 -> ISIC_6946895\nISIC_9388406 -> ISIC_4019458\nISIC_7548504 -> ISIC_4495527\nISIC_3761540 -> ISIC_2890008\nISIC_0936437 -> ISIC_9948221\nISIC_7529810 -> ISIC_3314065\nISIC_6717898 -> ISIC_8562507\nISIC_2866924 -> ISIC_0462336\nISIC_1142863 -> ISIC_9656943\nISIC_5541445 -> ISIC_2568903\nISIC_2722040 -> ISIC_8753442, ISIC_4471121\nISIC_3716403 -> ISIC_8328232\nISIC_8253521 -> ISIC_6248112\nISIC_1808187 -> ISIC_2596120\nISIC_8284060 -> ISIC_2757628\nISIC_9002552 -> ISIC_2131119\nISIC_3819795 -> ISIC_7373877\nISIC_5569800 -> ISIC_1425639\nISIC_6249626 -> ISIC_8648368\nISIC_9008728 -> ISIC_9982577\nISIC_8340210 -> ISIC_3083302\nISIC_7414621 -> ISIC_7005958\nISIC_3884657 -> ISIC_6014626\nISIC_5866452 -> ISIC_6850984\nISIC_8715911 -> ISIC_5467408\nISIC_2890008 -> ISIC_3761540\nISIC_3083302 -> ISIC_8340210\nISIC_7005958 -> ISIC_7414621\nISIC_4829472 -> ISIC_6204751\nISIC_2964421 -> ISIC_8749647\nISIC_9982577 -> ISIC_9008728\nISIC_8661785 -> ISIC_8061920\nISIC_7270127 -> ISIC_9232173\nISIC_3945172 -> ISIC_3796609\nISIC_6014626 -> ISIC_3884657\nISIC_1026838 -> ISIC_4907927\nISIC_4223719 -> ISIC_0904702\nISIC_2190511 -> ISIC_5198297\nISIC_6248112 -> ISIC_8253521\nISIC_8166745 -> ISIC_9155599\nISIC_5006966 -> ISIC_9931481\nISIC_1975904 -> ISIC_0536393\nISIC_1270937 -> ISIC_8748932\nISIC_8683473 -> ISIC_9084283\nISIC_5995007 -> ISIC_7194273\nISIC_6614779 -> ISIC_1950045\nISIC_6275427 -> ISIC_9535767\nISIC_9259972 -> ISIC_1706022\nISIC_4495527 -> ISIC_7548504\nISIC_1848959 -> ISIC_5078133\nISIC_4967356 -> ISIC_4247203\nISIC_5099850 -> ISIC_5176953\nISIC_7194273 -> ISIC_5995007\nISIC_8454848 -> ISIC_6170663\nISIC_0517531 -> ISIC_2403393\nISIC_4443268 -> ISIC_7113586\nISIC_8328232 -> ISIC_3716403\nISIC_7036799 -> ISIC_5735204\nISIC_2507875 -> ISIC_8746376\nISIC_4631399 -> ISIC_5911171\nISIC_5735204 -> ISIC_7036799\nISIC_2313463 -> ISIC_3971415\nISIC_6827668 -> ISIC_3377854\nISIC_6209867 -> ISIC_7793652\nISIC_5872962 -> ISIC_5193073\nISIC_3004197 -> ISIC_5859817\nISIC_5859817 -> ISIC_3004197\nISIC_1171499 -> ISIC_2485658\nISIC_9948221 -> ISIC_0936437\nISIC_4261613 -> ISIC_2258860\nISIC_5576295 -> ISIC_3656810\nISIC_0635396 -> ISIC_3555756\nISIC_1259317 -> ISIC_0506082\nISIC_4061347 -> ISIC_4808024\nISIC_1437627 -> ISIC_5747282\nISIC_2735323 -> ISIC_0704435\nISIC_7106452 -> ISIC_5265808\nISIC_5467408 -> ISIC_8715911\nISIC_1425639 -> ISIC_5569800\nISIC_4720030 -> ISIC_0674769\nISIC_0158761 -> ISIC_8874925\nISIC_3656810 -> ISIC_5576295\nISIC_8209167 -> ISIC_0596499\nISIC_2258860 -> ISIC_4261613\nISIC_1240474 -> ISIC_2234584\nISIC_8061920 -> ISIC_8661785\nISIC_9174152 -> ISIC_0735344\nISIC_2485658 -> ISIC_1171499\nISIC_4156931 -> ISIC_9850411\nISIC_1437807 -> ISIC_0559101\nISIC_2376329 -> ISIC_3341347\nISIC_9247289 -> ISIC_4133898\nISIC_2817101 -> ISIC_5185780\nISIC_5269066 -> ISIC_9552129\nISIC_6084895 -> ISIC_3276396\nISIC_3659414 -> ISIC_6250185\nISIC_8986079 -> ISIC_7480177\nISIC_0596499 -> ISIC_8209167\nISIC_9635083 -> ISIC_2394733\nISIC_8648368 -> ISIC_6249626\nISIC_8240408 -> ISIC_2192031\nISIC_4907927 -> ISIC_1026838\nISIC_9640394 -> ISIC_9846473\nISIC_5193073 -> ISIC_5872962\nISIC_3666478 -> ISIC_2862111\nISIC_8200009 -> ISIC_6456904\nISIC_4019458 -> ISIC_9388406\nISIC_8748932 -> ISIC_1270937\nISIC_8753442 -> ISIC_2722040, ISIC_4471121\nISIC_4990272 -> ISIC_8750817\nISIC_4133898 -> ISIC_9247289\nISIC_8749647 -> ISIC_2964421\nISIC_5421651 -> ISIC_0434890\nISIC_5746049 -> ISIC_7262158\nISIC_6170663 -> ISIC_8454848\nISIC_3256190 -> ISIC_0557285\nISIC_5185780 -> ISIC_2817101\nISIC_4471121 -> ISIC_2722040, ISIC_8753442\nISIC_8668157 -> ISIC_0631039\nISIC_0554819 -> ISIC_8315094\nISIC_3314065 -> ISIC_7529810\nISIC_4537621 -> ISIC_4872245\nISIC_0735344 -> ISIC_9174152\nISIC_2596120 -> ISIC_1808187\n\nThere are 223 images with similar images over 10982 images.\n```", "882645": "ebouteillon thanks for sharing! You are right, I didn’t use it for test dataset, as I tried to solve problem with unstable validation. Your information is very useful, I will try it. \n\nP.S. DBSCAN looks like a perfect 😂 all of them are really duplicates, maybe It didn’t find all cases. I think value of epsilon is important.", "883227": "thanks", "883282": "I think having duplicates in test set is an issue. If some of these images are present in public leaderboard and the \"similar\" images are in the private leaderboard, then you can leak some results from the private leaderboard by probing the public leaderboard.\n\n@juliaelliott @philculliton @veronicarotemberg can you confirm there is no such leak?", "883517": "Since there are duplicate images with differing meta data, does it even make sense to use the meta data for training?", "888093": "ebouteillon We are looking into this with the hosts and will respond as soon as possible.", "903474": "Alright, the host has [issued this explanation](https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161943) that addresses dupes, both that you have identified and some additional ones uncovered. The short of it is that they acknowledge these dupes exist and have confirmed they are okay to use. Obviously, manually labeling the test set with this knowledge is still prohibited, but these images are still free to include in training.", "903513": "Thanks for the clarification!" }, "source": "meta" }