![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F172c5a9ff81f2ad7a74fe05620b98817%2FScreen%20Shot%202020-08-02%20at%206.52.03%20PM.png?generation=1596419578076730&alt=media\")
This is a continuation of @zaharch discussion here
\n\nIt has been published here by the host that there are 107 pairs of images in the test data that are identical images. Of these 107 pairs, 97 pairs have the same patient_id. Presumably ( @jwebermsk please correct me if I'm wrong), all images from one patient are either all in public test or all in private test and therefore these 97 pairs are not leaks.
There are 10 pairs of test images that are duplicate images but each image has a different patient_id. That means that one image may be in public test and one image may be in private test. As such, we can probe public LB and determine the truth of the following 20 potential private test leaks:
['ISIC_0434890', 'ISIC_5421651', 'ISIC_0559101', 'ISIC_1437807',\n 'ISIC_0924293', 'ISIC_6844611', 'ISIC_1663074', 'ISIC_6192610',\n 'ISIC_1950045', 'ISIC_6614779', 'ISIC_2568903', 'ISIC_5541445',\n 'ISIC_3377854', 'ISIC_6827668', 'ISIC_3884657', 'ISIC_6014626',\n 'ISIC_5099850', 'ISIC_5176953', 'ISIC_6456904', 'ISIC_8200009']
Instead of everyone wasting 20 submissions, we could work together and post the results in this discussion. Or better yet, Kaggle can remove these 10 images from private test dataset. Does this sound fair @juliaelliott @philculliton to remove these leaks from private test? (Another discussion about these leaks is here).
\n\nIt has been confirmed by Sirish (in comments below) that none of these 20 images are both malignant and in public test. Therefore they do not have the ability to significantly affect private LB. No action is required regarding these images. (It would still be good to get confirmation that the other 97 pairs with same patient_id are not split between public and private @jwebermsk @juliaelliott @philculliton )
It has been confirmed by @ucanmakeit175 @benboren @coreacasa @ajtryt2 that none of the 194 images (97 pairs with same patient id) contain any private test leaks.
\n\nEach potential private test leak, has the potential of increasing private LB AUC score by 0.0027! (because 7687 private images with hypothesized 2.367% malignant equals 182 malignant. Then 0.5 / 182 = 0.00274 AUC) Thus maximum (although unlikely) increase in private LB AUC from using these leaks is 0.0274 = 10 * 0.00274 !
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F2882cfa11fbf2c115e68bed07cb93e7a%2FScreen%20Shot%202020-08-02%20at%206.52.12%20PM.png?generation=1596419586275440&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fcb61ff78694b13dfe387dd12ef493c51%2FScreen%20Shot%202020-08-02%20at%206.52.22%20PM.png?generation=1596419595338472&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Fc2d16fdaccfbc07dffbfdfcee64dc6d6%2FScreen%20Shot%202020-08-02%20at%206.52.30%20PM.png?generation=1596419603579481&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2Ffb0843cf7521ebdf25db72c374c91bfc%2FScreen%20Shot%202020-08-02%20at%206.52.38%20PM.png?generation=1596419612120365&alt=media\")
This is a continuation of @zaharch discussion here
\n\nIt has been published here by the host that there are 107 pairs of images in the test data that are identical images. Of these 107 pairs, 97 pairs have the same patient_id. Presumably ( @jwebermsk please correct me if I'm wrong), all images from one patient are either all in public test or all in private test and therefore these 97 pairs are not leaks.
There are 10 pairs of test images that are duplicate images but each image has a different patient_id. That means that one image may be in public test and one image may be in private test. As such, we can probe public LB and determine the truth of the following 20 potential private test leaks:
['ISIC_0434890', 'ISIC_5421651', 'ISIC_0559101', 'ISIC_1437807',\n 'ISIC_0924293', 'ISIC_6844611', 'ISIC_1663074', 'ISIC_6192610',\n 'ISIC_1950045', 'ISIC_6614779', 'ISIC_2568903', 'ISIC_5541445',\n 'ISIC_3377854', 'ISIC_6827668', 'ISIC_3884657', 'ISIC_6014626',\n 'ISIC_5099850', 'ISIC_5176953', 'ISIC_6456904', 'ISIC_8200009']
Instead of everyone wasting 20 submissions, we could work together and post the results in this discussion. Or better yet, Kaggle can remove these 10 images from private test dataset. Does this sound fair @juliaelliott @philculliton to remove these leaks from private test? (Another discussion about these leaks is here).
\n\nIt has been confirmed by Sirish (in comments below) that none of these 20 images are both malignant and in public test. Therefore they do not have the ability to significantly affect private LB. No action is required regarding these images. (It would still be good to get confirmation that the other 97 pairs with same patient_id are not split between public and private @jwebermsk @juliaelliott @philculliton )
It has been confirmed by @ucanmakeit175 @benboren @coreacasa @ajtryt2 that none of the 194 images (97 pairs with same patient id) contain any private test leaks.
\n\nEach potential private test leak, has the potential of increasing private LB AUC score by 0.0027! (because 7687 private images with hypothesized 2.367% malignant equals 182 malignant. Then 0.5 / 182 = 0.00274 AUC) Thus maximum (although unlikely) increase in private LB AUC from using these leaks is 0.0274 = 10 * 0.00274 !
I would like to strongly encourage Kaggle to remove the 10 problematic images from the private test set. This is a very good solution of the problem which does not have any obvious downsides. Why not do it for the sake of fairness? I would like to spend the remaining time of the competition refining my ML models instead of dealing with this unnecessary headache.
", "rawMarkdown": "I would like to strongly encourage Kaggle to remove the 10 problematic images from the private test set. This is a very good solution of the problem which does not have any obvious downsides. Why not do it for the sake of fairness? I would like to spend the remaining time of the competition refining my ML models instead of dealing with this unnecessary headache. ", "votes": 41, "replies": [ { "id": 955869, "postDate": "2020-08-03T03:10:17.280Z", "content": "Agree +1
", "rawMarkdown": "Agree +1", "votes": 6 }, { "id": 956365, "postDate": "2020-08-03T12:48:20.397Z", "content": "I dont understand why they dont do it. It is literally just a flag they need to set and they can keep public LB like it is.
", "rawMarkdown": "I dont understand why they dont do it. It is literally just a flag they need to set and they can keep public LB like it is.", "votes": 1 } ] }, { "id": 958004, "postDate": "2020-08-04T17:27:28.350Z", "content": "Dear Chris and all other challenge participants,
\n\nWe have confirmed with Kaggle that these duplicates won't have a significant impact on the scoring of the private test set. And so, at this point, we believe that no changes are warranted or advisable, and that the impact is so small that we feel it would be more disruptive and create other problems that outweigh the benefit of removing this minimal bias.
\n\nAgain, we apologize for having these duplicates slip through the cracks and not being picked up! As mentioned in an earlier thread, we have put in place outlined procedures for next year's challenge that will run the entire dataset (train and test) through a thorough duplicate detection process.
\n\nAnd also again thank you for your diligence and sleuthing for looking into this!! As one of the organizers, I think I can speak for the team that we have been extremely impressed by the amount of work and commitment that has gone into delving into the depths of the dataset!
\n\n/jochen
", "rawMarkdown": "Dear Chris and all other challenge participants,\n\nWe have confirmed with Kaggle that these duplicates won't have a significant impact on the scoring of the private test set. And so, at this point, we believe that no changes are warranted or advisable, and that the impact is so small that we feel it would be more disruptive and create other problems that outweigh the benefit of removing this minimal bias.\n\nAgain, we apologize for having these duplicates slip through the cracks and not being picked up! As mentioned in an earlier thread, we have put in place outlined procedures for next year's challenge that will run the entire dataset (train and test) through a thorough duplicate detection process.\n\nAnd also again thank you for your diligence and sleuthing for looking into this!! As one of the organizers, I think I can speak for the team that we have been extremely impressed by the amount of work and commitment that has gone into delving into the depths of the dataset!\n\n/jochen", "votes": 19, "replies": [ { "id": 959525, "postDate": "2020-08-05T16:54:55.917Z", "content": "A duplicate detection process would be appreciable👍
", "rawMarkdown": "A duplicate detection process would be appreciable👍 " } ] }, { "id": 956437, "postDate": "2020-08-03T13:55:12.427Z", "content": "@cdeotte As far as I can tell, these 20 images are (i) either not M or (ii) not in Public LB part\ni.e. their labels cannot be known by LB probing!
\n\n['ISIC_0434890', 'ISIC_5421651', 'ISIC_0559101', 'ISIC_1437807', 'ISIC_0924293', 'ISIC_6844611', 'ISIC_1663074', 'ISIC_6192610', 'ISIC_1950045', 'ISIC_6614779', 'ISIC_2568903', 'ISIC_5541445', 'ISIC_3377854', 'ISIC_6827668', 'ISIC_3884657', 'ISIC_6014626', 'ISIC_5099850', 'ISIC_5176953', 'ISIC_6456904', 'ISIC_8200009']
\n\nMy proof: I have wasted 1-submission by setting above 20 images to 1 (and rest of test images to 0), and obtained score of 0.4987 (less than 0.5). If even one image was an M, I would expect to get a score greater than 0.5!
", "rawMarkdown": "@cdeotte As far as I can tell, these 20 images are (i) either **not M** or (ii) **not in Public LB** part\ni.e. their labels cannot be known by LB probing!\n\n['ISIC_0434890', 'ISIC_5421651', 'ISIC_0559101', 'ISIC_1437807', 'ISIC_0924293', 'ISIC_6844611', 'ISIC_1663074', 'ISIC_6192610', 'ISIC_1950045', 'ISIC_6614779', 'ISIC_2568903', 'ISIC_5541445', 'ISIC_3377854', 'ISIC_6827668', 'ISIC_3884657', 'ISIC_6014626', 'ISIC_5099850', 'ISIC_5176953', 'ISIC_6456904', 'ISIC_8200009']\n\n**My proof:** I have wasted 1-submission by setting above 20 images to 1 (and rest of test images to 0), and obtained score of 0.4987 (less than 0.5). If even one image was an M, I would expect to get a score greater than 0.5!", "votes": 17, "replies": [ { "id": 956451, "postDate": "2020-08-03T14:11:31.373Z", "content": "@sirishks thanks for sharing this! (I consider that you paid your debt towards me about my wasted submission last time ^^)
\n\nI agree with you that if one of them was malginant and in Public LB you would have ended up with something >0.5.
\n\nIt seems that what this means is at least one of them is in Public LB and is benign (otherwise you would have had a 0.5 AUC LB). Does not seem like a very dangerous leak!
", "rawMarkdown": "@sirishks thanks for sharing this! (I consider that you paid your debt towards me about my wasted submission last time ^^) \n\nI agree with you that if one of them was malginant and in Public LB you would have ended up with something >0.5.\n\nIt seems that what this means is at least one of them is in Public LB and is benign (otherwise you would have had a 0.5 AUC LB). Does not seem like a very dangerous leak!\n\n\n", "votes": 1 }, { "id": 956481, "postDate": "2020-08-03T14:34:20.163Z", "content": "thanks for sharing. I can confirm that what you said is right. @sirishks
", "rawMarkdown": "thanks for sharing. I can confirm that what you said is right. @sirishks ", "votes": 2 }, { "id": 956491, "postDate": "2020-08-03T14:41:02.363Z", "content": "@sirishks Thank you for your test. Your conclusions are correct. If you set target=1 for a set of images and target=0 for the rest the following holds. When one image is both malignant and in public LB, it will increase public LB AUC by 0.006410. When one image is both benign and in public LB, it will decrease public LB AUC by 0.000155. Your experiment concludes that 8 of the 20 images are in public test and those 8 images are benign. Thus their leak ability to affect private LB is insignificant.
We're grateful to Sirish for checking the potential 10 leaks for us and determining that they are not problems. And we're grateful to @ucanmakeit175 @benboren @coreacasa @ajtryt2 for determining the other 97 pairs are not problems either.
\n\nWe finished checking all 107 pairs. There are NO LEAKS that will give anyone an advantage on private LB.
\n\nThe lists below contain the 97 pairs of duplicates. (The other 10 pairs are in another comment below). I have submitted the first, @ucanmakeit175 submitted the second, and @benboren submitted the third, @coreacasa the fourth, and @ajtryt2 the fifth. Thank you . Our results confirmed that there are no leaks in private test.
\n\nNAMES1 = ['ISIC_0158761', 'ISIC_8874925', 'ISIC_0375431', 'ISIC_5921042',\n 'ISIC_0388802', 'ISIC_6082446', 'ISIC_0462336', 'ISIC_2866924',\n 'ISIC_0506082', 'ISIC_1259317', 'ISIC_0517531', 'ISIC_2403393',\n 'ISIC_0554819', 'ISIC_8315094', 'ISIC_0557285', 'ISIC_3256190',\n 'ISIC_0596499', 'ISIC_8209167', 'ISIC_0631039', 'ISIC_8668157',\n 'ISIC_0635396', 'ISIC_3555756', 'ISIC_0674769', 'ISIC_4720030',\n 'ISIC_0704435', 'ISIC_2735323', 'ISIC_0735344', 'ISIC_9174152',\n 'ISIC_0762091', 'ISIC_4364994', 'ISIC_0904702', 'ISIC_4223719',\n 'ISIC_0936437', 'ISIC_9948221', 'ISIC_1026838', 'ISIC_4907927',\n 'ISIC_1115450', 'ISIC_6901140', 'ISIC_1142863', 'ISIC_9656943']
NAMES2 = ['ISIC_1171499', 'ISIC_2485658', 'ISIC_1240474', 'ISIC_2234584',\n 'ISIC_1270937', 'ISIC_8748932', 'ISIC_1425639', 'ISIC_5569800',\n 'ISIC_1437627', 'ISIC_5747282', 'ISIC_1706022', 'ISIC_9259972',\n 'ISIC_1808187', 'ISIC_2596120', 'ISIC_1848959', 'ISIC_5078133',\n 'ISIC_2131119', 'ISIC_9002552', 'ISIC_2190511', 'ISIC_5198297',\n 'ISIC_2192031', 'ISIC_8240408', 'ISIC_2200299', 'ISIC_4874456',\n 'ISIC_2258860', 'ISIC_4261613', 'ISIC_2313463', 'ISIC_3971415',\n 'ISIC_2376329', 'ISIC_3341347', 'ISIC_2394733', 'ISIC_9635083',\n 'ISIC_2468035', 'ISIC_6855256', 'ISIC_2507875', 'ISIC_8746376',\n 'ISIC_2689674', 'ISIC_5753616', 'ISIC_2722040', 'ISIC_8753442']
NAMES3 = ['ISIC_2757628', 'ISIC_8284060', 'ISIC_2817101', 'ISIC_5185780',\n 'ISIC_2862111', 'ISIC_3666478', 'ISIC_2890008', 'ISIC_3761540',\n 'ISIC_2964421', 'ISIC_8749647', 'ISIC_3004197', 'ISIC_5859817',\n 'ISIC_3046977', 'ISIC_3447990', 'ISIC_3083302', 'ISIC_8340210',\n 'ISIC_3276396', 'ISIC_6084895', 'ISIC_3508787', 'ISIC_3597521',\n 'ISIC_3588900', 'ISIC_4535526', 'ISIC_3656810', 'ISIC_5576295',\n 'ISIC_3659414', 'ISIC_6250185', 'ISIC_3716403', 'ISIC_8328232',\n 'ISIC_3796609', 'ISIC_3945172', 'ISIC_3819795', 'ISIC_7373877',\n 'ISIC_4019458', 'ISIC_9388406', 'ISIC_4061347', 'ISIC_4808024',\n 'ISIC_4133898', 'ISIC_9247289', 'ISIC_4156931', 'ISIC_9850411']
NAMES4 = ['ISIC_4247203', 'ISIC_4967356', 'ISIC_4294619', 'ISIC_6946895',\n 'ISIC_4308112', 'ISIC_4818658', 'ISIC_4443268', 'ISIC_7113586',\n 'ISIC_4495527', 'ISIC_7548504', 'ISIC_4537621', 'ISIC_4872245',\n 'ISIC_4545487', 'ISIC_8965291', 'ISIC_4819603', 'ISIC_6976452',\n 'ISIC_4829472', 'ISIC_6204751', 'ISIC_4892807', 'ISIC_5472411',\n 'ISIC_4990272', 'ISIC_8750817', 'ISIC_5006966', 'ISIC_9931481',\n 'ISIC_5193073', 'ISIC_5872962', 'ISIC_5265808', 'ISIC_7106452',\n 'ISIC_5269066', 'ISIC_9552129', 'ISIC_5467408', 'ISIC_8715911',\n 'ISIC_5735204', 'ISIC_7036799', 'ISIC_5746049', 'ISIC_7262158',\n 'ISIC_5788972', 'ISIC_7249001', 'ISIC_5866452', 'ISIC_6850984']
NAMES5 = ['ISIC_5995007', 'ISIC_7194273', 'ISIC_6170663', 'ISIC_8454848',\n 'ISIC_6209867', 'ISIC_7793652', 'ISIC_6248112', 'ISIC_8253521',\n 'ISIC_6249626', 'ISIC_8648368', 'ISIC_6275427', 'ISIC_9535767',\n 'ISIC_6668182', 'ISIC_9075192', 'ISIC_6717898', 'ISIC_8562507',\n 'ISIC_7005958', 'ISIC_7414621', 'ISIC_7270127', 'ISIC_9232173',\n 'ISIC_7480177', 'ISIC_8986079', 'ISIC_7556912', 'ISIC_9693172',\n 'ISIC_8061920', 'ISIC_8661785', 'ISIC_8166745', 'ISIC_9155599',\n 'ISIC_8683473', 'ISIC_9084283', 'ISIC_9008728', 'ISIC_9982577',\n 'ISIC_9640394', 'ISIC_9846473']
To participate, only submit one of the four that haven't been done already. Run and submit the following code. Change NAMES1 in two places to NAMES? where ? is 4, or 5.
sub = pd.read_csv('sample_submission.csv')\nsub.target = 0\nsub.loc[sub.image_name.isin(NAMES1),'target'] = 1\nsub.to_csv('submission_NAMES1.csv',index=False)\nWe are (probabilistically) expected to see a result of (A) LB 0.5113 or (B) LB 0.4981. These are the cases that out of 40 images, (A) 12 (= 0.3 * 40) are in public LB with 2 malignant and 10 benign, or (B) 12 are in public LB with 12 benign. If we see (C) LB 0.5047 that is a LEAK SITUATION where 12 are in public LB with 1 malignant and 11 benign. This can only happen if a pair is split between public and private.
Here is Sub1. This is weird, can someone check my math. With 3295 images in public test and an assumed 78 malignant, then an LB score of 0.4992 implies that 5 of these 40 are in public test and benign. That means that a pair was split !
\n\n(Of course no malignant were found, so either way this isn't a problematic leak. Let's check the other 4).
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1723677%2F4260c18884afced52c14a4cdd8700033%2FScreen%20Shot%202020-08-03%20at%208.33.31%20AM.png?generation=1596469225692765&alt=media\")
Sub2, 12 benign
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2370491%2Fc922be33feeb8a77453be269fbfcb220%2FScreenshot%202020-08-03%20at%2017.05.34.png?generation=1596470780736702&alt=media\")
Sub 3, ????? (looks like 6 but rounding is wrong) benign:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2370491%2F9b04879dc18e20810710dddb34a25196%2FScreenshot%202020-08-03%20at%2017.11.38.png?generation=1596471154550494&alt=media\")
Sub 4, 10 benign (rounding issue again) :![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2370491%2Fbbeba7ec65f725e999f200551aab27ac%2FScreenshot%202020-08-03%20at%2017.18.27.png?generation=1596471529005005&alt=media\")
Sub 5, 7 benign: ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2370491%2F7e32511fb32f3168cd74c9375f2b8be3%2FScreenshot%202020-08-03%20at%2017.20.26.png?generation=1596471645502541&alt=media\")
Here is Sub2.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3744443%2Fe4abb787c046de5fb2c3a9789bfb6024%2FNAMES2.png?generation=1596508220507501&alt=media\")
Fantastic. Thank you for your help @ucanmakeit175 Each benign decreases public LB by 0.000155 and each malignant increases public LB by 0.006410. Your submission is perfect probability. Out of 40 images, 40 * 0.3 = 12 are in public LB and all of them are benign. Great Work!
Does anyone want to submit NAMES3, NAMES4 or NAMES5 for us?
Sub3:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4042564%2Fff43506c363fe537d0434c8c31d192a1%2Fsub3.JPG?generation=1596517694242176&alt=media\")
Wonderful @benboren Thank you. Your result shows that NAMES3 has 6 images in public test that are benign. So far neither NAMES1, NAMES2, nor NAMES3 are problematic. We're checked 60 pairs! (plus the special 10 pairs) Only 37 more pairs to check. Who wants to check NAMES4 and NAMES5 ??
@cdeotte I posted them all - are they not visible? Perhaps I should condense them to one comment?
", "rawMarkdown": "@cdeotte I posted them all - are they not visible? Perhaps I should condense them to one comment?", "votes": 1 }, { "id": 966605, "postDate": "2020-08-11T15:01:30.283Z", "content": "Thanks @FChmiel . I'm just seeing your posts now.
", "rawMarkdown": "Thanks @FChmiel . I'm just seeing your posts now." }, { "id": 966620, "postDate": "2020-08-11T15:20:17.550Z", "content": "No problem - I think by posting them in multiple comments in quick succession they got suppressed by the spam filter. My mistake!
", "rawMarkdown": "No problem - I think by posting them in multiple comments in quick succession they got suppressed by the spam filter. My mistake!", "votes": 1 } ] }, { "id": 957225, "postDate": "2020-08-04T06:45:49.927Z", "content": "sub5!\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F830229%2F8c653a58ee0b2485ca8497d104de80e2%2Fsub5.png?generation=1596523532403847&alt=media\")
sub4!
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1454111%2F3d2da0406d78664a130c10246f4e31fb%2Fsub4.png?generation=1596524814489404&alt=media\")
@cdeotte To summarize, they are all <0.5 score i.e. no Malignant images present right?
", "rawMarkdown": "@cdeotte To summarize, they are all <0.5 score i.e. no Malignant images present right?", "votes": 1 }, { "id": 966598, "postDate": "2020-08-11T14:55:13.670Z", "content": "@sirishks I just saw your comment now. These results confirm that all 214 of these duplicate images are are (i) either not M or (ii) not in Public LB part. (I'm quoting what your wrote about your probe testing the 20 images).
So, we cannot use them to our advantage to increase private score, i.e. no leak. (But some of these 214 images may be (i) M and (ii) Private. So we cannot set their predictions to zero for our final submission.
", "rawMarkdown": "@sirishks I just saw your comment now. These results confirm that all 214 of these duplicate images are `are (i) either not M or (ii) not in Public LB part`. (I'm quoting what your wrote about your probe testing the 20 images).\n\nSo, we cannot use them to our advantage to increase private score, i.e. no leak. (But some of these 214 images may be (i) M and (ii) Private. So we cannot set their predictions to zero for our final submission." } ] }, { "id": 957711, "postDate": "2020-08-04T14:03:04.117Z", "content": "Thank you @ajtryt2 Thank you @coreacasa
\n\nIt would only be problematic if a pair contained a malignant in public test with the other image in private test. Then we would know a malignant in private test and could increase our private score by 0.00274 AUC. Knowing a few benign in private makes no significant advantage.
Thanks for clarifying this Chris, and thanks to all other contributors here!
", "rawMarkdown": "Thanks for clarifying this Chris, and thanks to all other contributors here!", "votes": 1 }, { "id": 966512, "postDate": "2020-08-11T13:35:16.053Z", "content": "Great Insight @cdeotte
", "rawMarkdown": "Great Insight @cdeotte ", "votes": 1 } ] }, { "id": 955872, "postDate": "2020-08-03T03:19:14.503Z", "content": "Its no wonder using metadata has been off... its not even consistent. ANATOM_SITE for the same image has differing values for the EXACT SAME IMAGE. Good work Chris.
", "rawMarkdown": "Its no wonder using metadata has been off... its not even consistent. ANATOM_SITE for the same image has differing values for the EXACT SAME IMAGE. Good work Chris.", "votes": 9, "replies": [ { "id": 956381, "postDate": "2020-08-03T13:02:30.157Z", "content": "This might be the issue with their deduplication process.
", "rawMarkdown": "This might be the issue with their deduplication process.", "votes": -1 } ] }, { "id": 955913, "postDate": "2020-08-03T04:37:32.180Z", "content": "Yep they should remove this observations from the private test.
", "rawMarkdown": "Yep they should remove this observations from the private test.", "votes": 3 }, { "id": 955850, "postDate": "2020-08-03T02:21:25.327Z", "content": "Thanks @cdeotte . This needs to be clarified!
", "rawMarkdown": "Thanks @cdeotte . This needs to be clarified!", "votes": 3 }, { "id": 961364, "postDate": "2020-08-07T05:54:35.057Z", "content": "It well be appreciated
", "rawMarkdown": "It well be appreciated ", "votes": 1 }, { "id": 959037, "postDate": "2020-08-05T09:48:17.970Z", "content": "Great work!
", "rawMarkdown": "Great work!", "votes": 1 }, { "id": 958483, "postDate": "2020-08-05T02:43:34.393Z", "content": "Nice！~
", "rawMarkdown": "Nice！~", "votes": 1 }, { "id": 958398, "postDate": "2020-08-05T00:49:54.043Z", "content": "gj
", "rawMarkdown": "gj", "votes": 1 }, { "id": 957855, "postDate": "2020-08-04T15:43:10.590Z", "content": "Great! Well Done!
", "rawMarkdown": "Great! Well Done!", "votes": 1 }, { "id": 957523, "postDate": "2020-08-04T11:27:44.823Z", "content": "nice job
", "rawMarkdown": "nice job", "votes": 1 }, { "id": 957234, "postDate": "2020-08-04T06:59:27.143Z", "content": "ok
", "rawMarkdown": "ok", "votes": 1 }, { "id": 957052, "postDate": "2020-08-04T02:53:09.920Z", "content": "Well done.
", "rawMarkdown": "Well done.", "votes": 1 }, { "id": 955981, "postDate": "2020-08-03T06:07:42.857Z", "content": "If the hosts don't provide a timely response, then I see a potential gold notebook or discussion, where the editor shares the results of LB probing.
", "rawMarkdown": "If the hosts don't provide a timely response, then I see a potential gold notebook or discussion, where the editor shares the results of LB probing.", "votes": 1 }, { "id": 962305, "postDate": "2020-08-08T03:14:02.010Z", "content": "Nice job!
", "rawMarkdown": "Nice job!", "votes": 2 }, { "id": 958557, "postDate": "2020-08-05T03:54:22.343Z", "content": "Well explained
", "rawMarkdown": "Well explained", "votes": 2 }, { "id": 958544, "postDate": "2020-08-05T03:43:06.117Z", "content": "Well done
", "rawMarkdown": "Well done", "votes": 2 }, { "id": 956346, "postDate": "2020-08-03T12:31:42.923Z", "content": "Wow... I assumed anatom sites might add some value when I started working with external metadata, but I see they're not even consistent, just wow... Thanks for sharing this Chris!
", "rawMarkdown": "Wow... I assumed anatom sites might add some value when I started working with external metadata, but I see they're not even consistent, just wow... Thanks for sharing this Chris!", "votes": 2, "replies": [ { "id": 956357, "postDate": "2020-08-03T12:40:47.253Z", "content": "agreed. Head/neck and Upper Extremity could be considered the same.........but torso and upper extremity is definitely an error somewhere.
", "rawMarkdown": "agreed. Head/neck and Upper Extremity could be considered the same.........but torso and upper extremity is definitely an error somewhere." } ] }, { "id": 957758, "postDate": "2020-08-04T14:47:11.483Z", "content": "ok
", "rawMarkdown": "ok" }, { "id": 957642, "postDate": "2020-08-04T13:10:37.283Z", "content": "Very well done!
", "rawMarkdown": "Very well done!" }, { "id": 957028, "postDate": "2020-08-04T02:29:26.973Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 955868, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-08-03T03:04:41.150000", "content": "I would like to strongly encourage Kaggle to remove the 10 problematic images from the private test set. This is a very good solution of the problem which does not have any obvious downsides. Why not do it for the sake of fairness? I would like to spend the remaining time of the competition refining my ML models instead of dealing with this unnecessary headache.
", "votes": 41, "replies": [ { "id": 955869, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-03T03:10:17.280000", "content": "Agree +1
", "votes": 6, "replies": [] }, { "id": 956365, "author_name": "Psi", "author_url": "", "post_date": "2020-08-03T12:48:20.397000", "content": "I dont understand why they dont do it. It is literally just a flag they need to set and they can keep public LB like it is.
", "votes": 1, "replies": [] } ] }, { "id": 958004, "author_name": "Jochen Weber", "author_url": "", "post_date": "2020-08-04T17:27:28.350000", "content": "Dear Chris and all other challenge participants,
\n\nWe have confirmed with Kaggle that these duplicates won't have a significant impact on the scoring of the private test set. And so, at this point, we believe that no changes are warranted or advisable, and that the impact is so small that we feel it would be more disruptive and create other problems that outweigh the benefit of removing this minimal bias.
\n\nAgain, we apologize for having these duplicates slip through the cracks and not being picked up! As mentioned in an earlier thread, we have put in place outlined procedures for next year's challenge that will run the entire dataset (train and test) through a thorough duplicate detection process.
\n\nAnd also again thank you for your diligence and sleuthing for looking into this!! As one of the organizers, I think I can speak for the team that we have been extremely impressed by the amount of work and commitment that has gone into delving into the depths of the dataset!
\n\n/jochen
", "votes": 19, "replies": [ { "id": 959525, "author_name": "Ananmay Suri", "author_url": "", "post_date": "2020-08-05T16:54:55.917000", "content": "A duplicate detection process would be appreciable👍
", "votes": 0, "replies": [] } ] }, { "id": 956437, "author_name": "Sirish Somanchi", "author_url": "", "post_date": "2020-08-03T13:55:12.427000", "content": "@cdeotte As far as I can tell, these 20 images are (i) either not M or (ii) not in Public LB part\ni.e. their labels cannot be known by LB probing!
\n\n['ISIC_0434890', 'ISIC_5421651', 'ISIC_0559101', 'ISIC_1437807', 'ISIC_0924293', 'ISIC_6844611', 'ISIC_1663074', 'ISIC_6192610', 'ISIC_1950045', 'ISIC_6614779', 'ISIC_2568903', 'ISIC_5541445', 'ISIC_3377854', 'ISIC_6827668', 'ISIC_3884657', 'ISIC_6014626', 'ISIC_5099850', 'ISIC_5176953', 'ISIC_6456904', 'ISIC_8200009']
\n\nMy proof: I have wasted 1-submission by setting above 20 images to 1 (and rest of test images to 0), and obtained score of 0.4987 (less than 0.5). If even one image was an M, I would expect to get a score greater than 0.5!
", "votes": 17, "replies": [ { "id": 956451, "author_name": "Optimo", "author_url": "", "post_date": "2020-08-03T14:11:31.373000", "content": "@sirishks thanks for sharing this! (I consider that you paid your debt towards me about my wasted submission last time ^^)
\n\nI agree with you that if one of them was malginant and in Public LB you would have ended up with something >0.5.
\n\nIt seems that what this means is at least one of them is in Public LB and is benign (otherwise you would have had a 0.5 AUC LB). Does not seem like a very dangerous leak!
", "votes": 1, "replies": [] }, { "id": 956481, "author_name": "UnknownRookie", "author_url": "", "post_date": "2020-08-03T14:34:20.163000", "content": "thanks for sharing. I can confirm that what you said is right. @sirishks
", "votes": 2, "replies": [] }, { "id": 956491, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-03T14:41:02.363000", "content": "@sirishks Thank you for your test. Your conclusions are correct. If you set target=1 for a set of images and target=0 for the rest the following holds. When one image is both malignant and in public LB, it will increase public LB AUC by 0.006410. When one image is both benign and in public LB, it will decrease public LB AUC by 0.000155. Your experiment concludes that 8 of the 20 images are in public test and those 8 images are benign. Thus their leak ability to affect private LB is insignificant.
We're grateful to Sirish for checking the potential 10 leaks for us and determining that they are not problems. And we're grateful to @ucanmakeit175 @benboren @coreacasa @ajtryt2 for determining the other 97 pairs are not problems either.
\n\nWe finished checking all 107 pairs. There are NO LEAKS that will give anyone an advantage on private LB.
\n\nThe lists below contain the 97 pairs of duplicates. (The other 10 pairs are in another comment below). I have submitted the first, @ucanmakeit175 submitted the second, and @benboren submitted the third, @coreacasa the fourth, and @ajtryt2 the fifth. Thank you . Our results confirmed that there are no leaks in private test.
\n\nNAMES1 = ['ISIC_0158761', 'ISIC_8874925', 'ISIC_0375431', 'ISIC_5921042',\n 'ISIC_0388802', 'ISIC_6082446', 'ISIC_0462336', 'ISIC_2866924',\n 'ISIC_0506082', 'ISIC_1259317', 'ISIC_0517531', 'ISIC_2403393',\n 'ISIC_0554819', 'ISIC_8315094', 'ISIC_0557285', 'ISIC_3256190',\n 'ISIC_0596499', 'ISIC_8209167', 'ISIC_0631039', 'ISIC_8668157',\n 'ISIC_0635396', 'ISIC_3555756', 'ISIC_0674769', 'ISIC_4720030',\n 'ISIC_0704435', 'ISIC_2735323', 'ISIC_0735344', 'ISIC_9174152',\n 'ISIC_0762091', 'ISIC_4364994', 'ISIC_0904702', 'ISIC_4223719',\n 'ISIC_0936437', 'ISIC_9948221', 'ISIC_1026838', 'ISIC_4907927',\n 'ISIC_1115450', 'ISIC_6901140', 'ISIC_1142863', 'ISIC_9656943']
NAMES2 = ['ISIC_1171499', 'ISIC_2485658', 'ISIC_1240474', 'ISIC_2234584',\n 'ISIC_1270937', 'ISIC_8748932', 'ISIC_1425639', 'ISIC_5569800',\n 'ISIC_1437627', 'ISIC_5747282', 'ISIC_1706022', 'ISIC_9259972',\n 'ISIC_1808187', 'ISIC_2596120', 'ISIC_1848959', 'ISIC_5078133',\n 'ISIC_2131119', 'ISIC_9002552', 'ISIC_2190511', 'ISIC_5198297',\n 'ISIC_2192031', 'ISIC_8240408', 'ISIC_2200299', 'ISIC_4874456',\n 'ISIC_2258860', 'ISIC_4261613', 'ISIC_2313463', 'ISIC_3971415',\n 'ISIC_2376329', 'ISIC_3341347', 'ISIC_2394733', 'ISIC_9635083',\n 'ISIC_2468035', 'ISIC_6855256', 'ISIC_2507875', 'ISIC_8746376',\n 'ISIC_2689674', 'ISIC_5753616', 'ISIC_2722040', 'ISIC_8753442']
NAMES3 = ['ISIC_2757628', 'ISIC_8284060', 'ISIC_2817101', 'ISIC_5185780',\n 'ISIC_2862111', 'ISIC_3666478', 'ISIC_2890008', 'ISIC_3761540',\n 'ISIC_2964421', 'ISIC_8749647', 'ISIC_3004197', 'ISIC_5859817',\n 'ISIC_3046977', 'ISIC_3447990', 'ISIC_3083302', 'ISIC_8340210',\n 'ISIC_3276396', 'ISIC_6084895', 'ISIC_3508787', 'ISIC_3597521',\n 'ISIC_3588900', 'ISIC_4535526', 'ISIC_3656810', 'ISIC_5576295',\n 'ISIC_3659414', 'ISIC_6250185', 'ISIC_3716403', 'ISIC_8328232',\n 'ISIC_3796609', 'ISIC_3945172', 'ISIC_3819795', 'ISIC_7373877',\n 'ISIC_4019458', 'ISIC_9388406', 'ISIC_4061347', 'ISIC_4808024',\n 'ISIC_4133898', 'ISIC_9247289', 'ISIC_4156931', 'ISIC_9850411']
NAMES4 = ['ISIC_4247203', 'ISIC_4967356', 'ISIC_4294619', 'ISIC_6946895',\n 'ISIC_4308112', 'ISIC_4818658', 'ISIC_4443268', 'ISIC_7113586',\n 'ISIC_4495527', 'ISIC_7548504', 'ISIC_4537621', 'ISIC_4872245',\n 'ISIC_4545487', 'ISIC_8965291', 'ISIC_4819603', 'ISIC_6976452',\n 'ISIC_4829472', 'ISIC_6204751', 'ISIC_4892807', 'ISIC_5472411',\n 'ISIC_4990272', 'ISIC_8750817', 'ISIC_5006966', 'ISIC_9931481',\n 'ISIC_5193073', 'ISIC_5872962', 'ISIC_5265808', 'ISIC_7106452',\n 'ISIC_5269066', 'ISIC_9552129', 'ISIC_5467408', 'ISIC_8715911',\n 'ISIC_5735204', 'ISIC_7036799', 'ISIC_5746049', 'ISIC_7262158',\n 'ISIC_5788972', 'ISIC_7249001', 'ISIC_5866452', 'ISIC_6850984']
NAMES5 = ['ISIC_5995007', 'ISIC_7194273', 'ISIC_6170663', 'ISIC_8454848',\n 'ISIC_6209867', 'ISIC_7793652', 'ISIC_6248112', 'ISIC_8253521',\n 'ISIC_6249626', 'ISIC_8648368', 'ISIC_6275427', 'ISIC_9535767',\n 'ISIC_6668182', 'ISIC_9075192', 'ISIC_6717898', 'ISIC_8562507',\n 'ISIC_7005958', 'ISIC_7414621', 'ISIC_7270127', 'ISIC_9232173',\n 'ISIC_7480177', 'ISIC_8986079', 'ISIC_7556912', 'ISIC_9693172',\n 'ISIC_8061920', 'ISIC_8661785', 'ISIC_8166745', 'ISIC_9155599',\n 'ISIC_8683473', 'ISIC_9084283', 'ISIC_9008728', 'ISIC_9982577',\n 'ISIC_9640394', 'ISIC_9846473']
To participate, only submit one of the four that haven't been done already. Run and submit the following code. Change NAMES1 in two places to NAMES? where ? is 4, or 5.
sub = pd.read_csv('sample_submission.csv')\nsub.target = 0\nsub.loc[sub.image_name.isin(NAMES1),'target'] = 1\nsub.to_csv('submission_NAMES1.csv',index=False)\nWe are (probabilistically) expected to see a result of (A) LB 0.5113 or (B) LB 0.4981. These are the cases that out of 40 images, (A) 12 (= 0.3 * 40) are in public LB with 2 malignant and 10 benign, or (B) 12 are in public LB with 12 benign. If we see (C) LB 0.5047 that is a LEAK SITUATION where 12 are in public LB with 1 malignant and 11 benign. This can only happen if a pair is split between public and private.
Here is Sub1. This is weird, can someone check my math. With 3295 images in public test and an assumed 78 malignant, then an LB score of 0.4992 implies that 5 of these 40 are in public test and benign. That means that a pair was split !
\n\n(Of course no malignant were found, so either way this isn't a problematic leak. Let's check the other 4).
\n\n
Sub2, 12 benign
\n\n
Sub 3, ????? (looks like 6 but rounding is wrong) benign:\n
Sub 4, 10 benign (rounding issue again) :
Sub 5, 7 benign:
Here is Sub2.\n
Fantastic. Thank you for your help @ucanmakeit175 Each benign decreases public LB by 0.000155 and each malignant increases public LB by 0.006410. Your submission is perfect probability. Out of 40 images, 40 * 0.3 = 12 are in public LB and all of them are benign. Great Work!
Does anyone want to submit NAMES3, NAMES4 or NAMES5 for us?
Sub3:\n
Wonderful @benboren Thank you. Your result shows that NAMES3 has 6 images in public test that are benign. So far neither NAMES1, NAMES2, nor NAMES3 are problematic. We're checked 60 pairs! (plus the special 10 pairs) Only 37 more pairs to check. Who wants to check NAMES4 and NAMES5 ??
@cdeotte I posted them all - are they not visible? Perhaps I should condense them to one comment?
", "votes": 1, "replies": [] }, { "id": 966605, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-11T15:01:30.283000", "content": "Thanks @FChmiel . I'm just seeing your posts now.
", "votes": 0, "replies": [] }, { "id": 966620, "author_name": "FChmiel", "author_url": "", "post_date": "2020-08-11T15:20:17.550000", "content": "No problem - I think by posting them in multiple comments in quick succession they got suppressed by the spam filter. My mistake!
", "votes": 1, "replies": [] } ] }, { "id": 957225, "author_name": "cp_t2", "author_url": "", "post_date": "2020-08-04T06:45:49.927000", "content": "sub5!\n
sub4!
\n\n
@cdeotte To summarize, they are all <0.5 score i.e. no Malignant images present right?
", "votes": 1, "replies": [] }, { "id": 966598, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-11T14:55:13.670000", "content": "@sirishks I just saw your comment now. These results confirm that all 214 of these duplicate images are are (i) either not M or (ii) not in Public LB part. (I'm quoting what your wrote about your probe testing the 20 images).
So, we cannot use them to our advantage to increase private score, i.e. no leak. (But some of these 214 images may be (i) M and (ii) Private. So we cannot set their predictions to zero for our final submission.
", "votes": 0, "replies": [] } ] }, { "id": 957711, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-04T14:03:04.117000", "content": "Thank you @ajtryt2 Thank you @coreacasa
\n\nIt would only be problematic if a pair contained a malignant in public test with the other image in private test. Then we would know a malignant in private test and could increase our private score by 0.00274 AUC. Knowing a few benign in private makes no significant advantage.
Thanks for clarifying this Chris, and thanks to all other contributors here!
", "votes": 1, "replies": [] }, { "id": 966512, "author_name": "Vikrant", "author_url": "", "post_date": "2020-08-11T13:35:16.053000", "content": "Great Insight @cdeotte
", "votes": 1, "replies": [] } ] }, { "id": 955872, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-03T03:19:14.503000", "content": "Its no wonder using metadata has been off... its not even consistent. ANATOM_SITE for the same image has differing values for the EXACT SAME IMAGE. Good work Chris.
", "votes": 9, "replies": [ { "id": 956381, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2020-08-03T13:02:30.157000", "content": "This might be the issue with their deduplication process.
", "votes": -1, "replies": [] } ] }, { "id": 955913, "author_name": "Martin Kovacevic Buvinic", "author_url": "", "post_date": "2020-08-03T04:37:32.180000", "content": "Yep they should remove this observations from the private test.
", "votes": 3, "replies": [] }, { "id": 955850, "author_name": "KhanhVD", "author_url": "", "post_date": "2020-08-03T02:21:25.327000", "content": "Thanks @cdeotte . This needs to be clarified!
", "votes": 3, "replies": [] }, { "id": 961364, "author_name": "Bhupendra Kumar Yadav", "author_url": "", "post_date": "2020-08-07T05:54:35.057000", "content": "It well be appreciated
", "votes": 1, "replies": [] }, { "id": 959037, "author_name": "Anders Ericsson Gnosco", "author_url": "", "post_date": "2020-08-05T09:48:17.970000", "content": "Great work!
", "votes": 1, "replies": [] }, { "id": 958483, "author_name": "Monster-H7", "author_url": "", "post_date": "2020-08-05T02:43:34.393000", "content": "Nice！~
", "votes": 1, "replies": [] }, { "id": 958398, "author_name": "Maksim Borisov", "author_url": "", "post_date": "2020-08-05T00:49:54.043000", "content": "gj
", "votes": 1, "replies": [] }, { "id": 957855, "author_name": "titanumm", "author_url": "", "post_date": "2020-08-04T15:43:10.590000", "content": "Great! Well Done!
", "votes": 1, "replies": [] }, { "id": 957523, "author_name": "Nancy", "author_url": "", "post_date": "2020-08-04T11:27:44.823000", "content": "nice job
", "votes": 1, "replies": [] }, { "id": 957234, "author_name": "Mohd Khurram", "author_url": "", "post_date": "2020-08-04T06:59:27.143000", "content": "ok
", "votes": 1, "replies": [] }, { "id": 957052, "author_name": "YALIN", "author_url": "", "post_date": "2020-08-04T02:53:09.920000", "content": "Well done.
", "votes": 1, "replies": [] }, { "id": 955981, "author_name": "Rohit Agarwal", "author_url": "", "post_date": "2020-08-03T06:07:42.857000", "content": "If the hosts don't provide a timely response, then I see a potential gold notebook or discussion, where the editor shares the results of LB probing.
", "votes": 1, "replies": [] }, { "id": 962305, "author_name": "Sanjay K", "author_url": "", "post_date": "2020-08-08T03:14:02.010000", "content": "Nice job!
", "votes": 2, "replies": [] }, { "id": 958557, "author_name": "", "author_url": "", "post_date": "2020-08-05T03:54:22.343000", "content": "Well explained
", "votes": 2, "replies": [] }, { "id": 958544, "author_name": "", "author_url": "", "post_date": "2020-08-05T03:43:06.117000", "content": "Well done
", "votes": 2, "replies": [] }, { "id": 956346, "author_name": "Ertuğrul Demir", "author_url": "", "post_date": "2020-08-03T12:31:42.923000", "content": "Wow... I assumed anatom sites might add some value when I started working with external metadata, but I see they're not even consistent, just wow... Thanks for sharing this Chris!
", "votes": 2, "replies": [ { "id": 956357, "author_name": "Signal", "author_url": "", "post_date": "2020-08-03T12:40:47.253000", "content": "agreed. Head/neck and Upper Extremity could be considered the same.........but torso and upper extremity is definitely an error somewhere.
", "votes": 0, "replies": [] } ] }, { "id": 957758, "author_name": "Gajesh Naik", "author_url": "", "post_date": "2020-08-04T14:47:11.483000", "content": "ok
", "votes": 0, "replies": [] }, { "id": 957642, "author_name": "Nitya Jain", "author_url": "", "post_date": "2020-08-04T13:10:37.283000", "content": "Very well done!
", "votes": 0, "replies": [] }, { "id": 957028, "author_name": "", "author_url": "", "post_date": "2020-08-04T02:29:26.973000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "955836": "This is a continuation of @zaharch discussion [here][3]\n\n# Test Data Duplicates\nIt has been published [here][1] by the host that there are 107 pairs of images in the test data that are identical images. Of these 107 pairs, 97 pairs have the same `patient_id`. Presumably ( @jwebermsk please correct me if I'm wrong), all images from one patient are either all in public test or all in private test and therefore these 97 pairs are not leaks.\n\n# Test Data Leaks\nThere are 10 pairs of test images that are duplicate images but each image has a different `patient_id`. That means that one image may be in public test and one image may be in private test. As such, we can probe public LB and determine the truth of the following 20 potential private test leaks:\n\n`['ISIC_0434890', 'ISIC_5421651', 'ISIC_0559101', 'ISIC_1437807',\n 'ISIC_0924293', 'ISIC_6844611', 'ISIC_1663074', 'ISIC_6192610',\n 'ISIC_1950045', 'ISIC_6614779', 'ISIC_2568903', 'ISIC_5541445',\n 'ISIC_3377854', 'ISIC_6827668', 'ISIC_3884657', 'ISIC_6014626',\n 'ISIC_5099850', 'ISIC_5176953', 'ISIC_6456904', 'ISIC_8200009']`\n\n# Suggestion\nInstead of everyone wasting 20 submissions, we could work together and post the results in this discussion. Or better yet, Kaggle can remove these 10 images from private test dataset. Does this sound fair @juliaelliott @philculliton to remove these leaks from private test? (Another discussion about these leaks is [here][3]).\n\n# UPDATE - Aug 3, 2020 - No Leaks in 10 Pairs\nIt has been confirmed by Sirish (in comments below) that none of these 20 images are both malignant and in public test. Therefore they do not have the ability to significantly affect private LB. No action is required regarding these images. (It would still be good to get confirmation that the other 97 pairs with same `patient_id` are not split between public and private @jwebermsk @juliaelliott @philculliton )\n\n# UPDATE - Aug 4, 2020 - No Leaks in 97 Pairs\nIt has been confirmed by @ucanmakeit175 @benboren @coreacasa @ajtryt2 that none of the 194 images (97 pairs with same patient id) contain any private test leaks.\n\n# Consequence\nEach potential private test leak, has the potential of increasing private LB AUC score by 0.0027! (because 7687 private images with [hypothesized][2] 2.367% malignant equals 182 malignant. Then `0.5 / 182 = 0.00274 AUC`) Thus maximum (although unlikely) increase in private LB AUC from using these leaks is `0.0274 = 10 * 0.00274` !\n\n\n\n\n\n\n\n\n\n\n\n[1]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/161943\n[2]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/167215\n[3]: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/171801", "955868": "I would like to strongly encourage Kaggle to remove the 10 problematic images from the private test set. This is a very good solution of the problem which does not have any obvious downsides. Why not do it for the sake of fairness? I would like to spend the remaining time of the competition refining my ML models instead of dealing with this unnecessary headache. ", "958004": "Dear Chris and all other challenge participants,\n\nWe have confirmed with Kaggle that these duplicates won't have a significant impact on the scoring of the private test set. And so, at this point, we believe that no changes are warranted or advisable, and that the impact is so small that we feel it would be more disruptive and create other problems that outweigh the benefit of removing this minimal bias.\n\nAgain, we apologize for having these duplicates slip through the cracks and not being picked up! As mentioned in an earlier thread, we have put in place outlined procedures for next year's challenge that will run the entire dataset (train and test) through a thorough duplicate detection process.\n\nAnd also again thank you for your diligence and sleuthing for looking into this!! As one of the organizers, I think I can speak for the team that we have been extremely impressed by the amount of work and commitment that has gone into delving into the depths of the dataset!\n\n/jochen", "956437": "@cdeotte As far as I can tell, these 20 images are (i) either **not M** or (ii) **not in Public LB** part\ni.e. their labels cannot be known by LB probing!\n\n['ISIC_0434890', 'ISIC_5421651', 'ISIC_0559101', 'ISIC_1437807', 'ISIC_0924293', 'ISIC_6844611', 'ISIC_1663074', 'ISIC_6192610', 'ISIC_1950045', 'ISIC_6614779', 'ISIC_2568903', 'ISIC_5541445', 'ISIC_3377854', 'ISIC_6827668', 'ISIC_3884657', 'ISIC_6014626', 'ISIC_5099850', 'ISIC_5176953', 'ISIC_6456904', 'ISIC_8200009']\n\n**My proof:** I have wasted 1-submission by setting above 20 images to 1 (and rest of test images to 0), and obtained score of 0.4987 (less than 0.5). If even one image was an M, I would expect to get a score greater than 0.5!", "956548": "# Leak Searching\nWe're grateful to Sirish for checking the potential 10 leaks for us and determining that they are not problems. And we're grateful to @ucanmakeit175 @benboren @coreacasa @ajtryt2 for determining the other 97 pairs are not problems either.\n\n## UPDATE - Aug 4, 2020 - NO LEAKS\nWe finished checking all 107 pairs. There are NO LEAKS that will give anyone an advantage on private LB.\n\nThe lists below contain the 97 pairs of duplicates. (The other 10 pairs are in another comment below). I have submitted the first, @ucanmakeit175 submitted the second, and @benboren submitted the third, @coreacasa the fourth, and @ajtryt2 the fifth. Thank you . Our results confirmed that there are no leaks in private test.\n \n## Sub 1\n`NAMES1 = ['ISIC_0158761', 'ISIC_8874925', 'ISIC_0375431', 'ISIC_5921042',\n 'ISIC_0388802', 'ISIC_6082446', 'ISIC_0462336', 'ISIC_2866924',\n 'ISIC_0506082', 'ISIC_1259317', 'ISIC_0517531', 'ISIC_2403393',\n 'ISIC_0554819', 'ISIC_8315094', 'ISIC_0557285', 'ISIC_3256190',\n 'ISIC_0596499', 'ISIC_8209167', 'ISIC_0631039', 'ISIC_8668157',\n 'ISIC_0635396', 'ISIC_3555756', 'ISIC_0674769', 'ISIC_4720030',\n 'ISIC_0704435', 'ISIC_2735323', 'ISIC_0735344', 'ISIC_9174152',\n 'ISIC_0762091', 'ISIC_4364994', 'ISIC_0904702', 'ISIC_4223719',\n 'ISIC_0936437', 'ISIC_9948221', 'ISIC_1026838', 'ISIC_4907927',\n 'ISIC_1115450', 'ISIC_6901140', 'ISIC_1142863', 'ISIC_9656943']`\n## Sub 2\n`NAMES2 = ['ISIC_1171499', 'ISIC_2485658', 'ISIC_1240474', 'ISIC_2234584',\n 'ISIC_1270937', 'ISIC_8748932', 'ISIC_1425639', 'ISIC_5569800',\n 'ISIC_1437627', 'ISIC_5747282', 'ISIC_1706022', 'ISIC_9259972',\n 'ISIC_1808187', 'ISIC_2596120', 'ISIC_1848959', 'ISIC_5078133',\n 'ISIC_2131119', 'ISIC_9002552', 'ISIC_2190511', 'ISIC_5198297',\n 'ISIC_2192031', 'ISIC_8240408', 'ISIC_2200299', 'ISIC_4874456',\n 'ISIC_2258860', 'ISIC_4261613', 'ISIC_2313463', 'ISIC_3971415',\n 'ISIC_2376329', 'ISIC_3341347', 'ISIC_2394733', 'ISIC_9635083',\n 'ISIC_2468035', 'ISIC_6855256', 'ISIC_2507875', 'ISIC_8746376',\n 'ISIC_2689674', 'ISIC_5753616', 'ISIC_2722040', 'ISIC_8753442']`\n## Sub 3\n`NAMES3 = ['ISIC_2757628', 'ISIC_8284060', 'ISIC_2817101', 'ISIC_5185780',\n 'ISIC_2862111', 'ISIC_3666478', 'ISIC_2890008', 'ISIC_3761540',\n 'ISIC_2964421', 'ISIC_8749647', 'ISIC_3004197', 'ISIC_5859817',\n 'ISIC_3046977', 'ISIC_3447990', 'ISIC_3083302', 'ISIC_8340210',\n 'ISIC_3276396', 'ISIC_6084895', 'ISIC_3508787', 'ISIC_3597521',\n 'ISIC_3588900', 'ISIC_4535526', 'ISIC_3656810', 'ISIC_5576295',\n 'ISIC_3659414', 'ISIC_6250185', 'ISIC_3716403', 'ISIC_8328232',\n 'ISIC_3796609', 'ISIC_3945172', 'ISIC_3819795', 'ISIC_7373877',\n 'ISIC_4019458', 'ISIC_9388406', 'ISIC_4061347', 'ISIC_4808024',\n 'ISIC_4133898', 'ISIC_9247289', 'ISIC_4156931', 'ISIC_9850411']`\n## Sub 4\n`NAMES4 = ['ISIC_4247203', 'ISIC_4967356', 'ISIC_4294619', 'ISIC_6946895',\n 'ISIC_4308112', 'ISIC_4818658', 'ISIC_4443268', 'ISIC_7113586',\n 'ISIC_4495527', 'ISIC_7548504', 'ISIC_4537621', 'ISIC_4872245',\n 'ISIC_4545487', 'ISIC_8965291', 'ISIC_4819603', 'ISIC_6976452',\n 'ISIC_4829472', 'ISIC_6204751', 'ISIC_4892807', 'ISIC_5472411',\n 'ISIC_4990272', 'ISIC_8750817', 'ISIC_5006966', 'ISIC_9931481',\n 'ISIC_5193073', 'ISIC_5872962', 'ISIC_5265808', 'ISIC_7106452',\n 'ISIC_5269066', 'ISIC_9552129', 'ISIC_5467408', 'ISIC_8715911',\n 'ISIC_5735204', 'ISIC_7036799', 'ISIC_5746049', 'ISIC_7262158',\n 'ISIC_5788972', 'ISIC_7249001', 'ISIC_5866452', 'ISIC_6850984']`\n## Sub 5\n`NAMES5 = ['ISIC_5995007', 'ISIC_7194273', 'ISIC_6170663', 'ISIC_8454848',\n 'ISIC_6209867', 'ISIC_7793652', 'ISIC_6248112', 'ISIC_8253521',\n 'ISIC_6249626', 'ISIC_8648368', 'ISIC_6275427', 'ISIC_9535767',\n 'ISIC_6668182', 'ISIC_9075192', 'ISIC_6717898', 'ISIC_8562507',\n 'ISIC_7005958', 'ISIC_7414621', 'ISIC_7270127', 'ISIC_9232173',\n 'ISIC_7480177', 'ISIC_8986079', 'ISIC_7556912', 'ISIC_9693172',\n 'ISIC_8061920', 'ISIC_8661785', 'ISIC_8166745', 'ISIC_9155599',\n 'ISIC_8683473', 'ISIC_9084283', 'ISIC_9008728', 'ISIC_9982577',\n 'ISIC_9640394', 'ISIC_9846473']`\n# How to Sub\nTo participate, only submit one of the four that haven't been done already. Run and submit the following code. Change `NAMES1` in two places to `NAMES?` where `?` is 4, or 5.\n\n sub = pd.read_csv('sample_submission.csv')\n sub.target = 0\n sub.loc[sub.image_name.isin(NAMES1),'target'] = 1\n sub.to_csv('submission_NAMES1.csv',index=False)\n\n# What to Expect\nWe are (probabilistically) expected to see a result of (A) `LB 0.5113` or (B) `LB 0.4981`. These are the cases that out of 40 images, (A) 12 (`= 0.3 * 40`) are in public LB with 2 malignant and 10 benign, or (B) 12 are in public LB with 12 benign. If we see (C) `LB 0.5047` that is a LEAK SITUATION where 12 are in public LB with 1 malignant and 11 benign. This can only happen if a pair is split between public and private.", "955872": "Its no wonder using metadata has been off... its not even consistent. ANATOM_SITE for the same image has differing values for the EXACT SAME IMAGE. Good work Chris.", "955913": "Yep they should remove this observations from the private test.", "955850": "Thanks @cdeotte . This needs to be clarified!", "961364": "It well be appreciated ", "959037": "Great work!", "958483": "Nice！~", "958398": "gj", "957855": "Great! Well Done!", "957523": "nice job", "957234": "ok", "957052": "Well done.", "955981": "If the hosts don't provide a timely response, then I see a potential gold notebook or discussion, where the editor shares the results of LB probing.", "962305": "Nice job!", "958557": "Well explained", "958544": "Well done", "956346": "Wow... I assumed anatom sites might add some value when I started working with external metadata, but I see they're not even consistent, just wow... Thanks for sharing this Chris!", "957758": "ok", "957642": "Very well done!", "957028": "" } }