Examples:
\n\nTrain data:\n - 115d3ca4-bbc9-11e8-b2bc-ac1f6b6435d0 and b927ef4a-bbb2-11e8-b2ba-ac1f6b6435d0
\n\nTest data:\n- 45b4b7b8-bad2-11e8-b2b8-ac1f6b6435d0 and 3f0d0104-bac8-11e8-b2b7-ac1f6b6435d0
", "messages": [ { "id": 409996, "postDate": "2018-10-25T07:14:55.180Z", "content": "Examples:
\n\nTrain data:\n - 115d3ca4-bbc9-11e8-b2bc-ac1f6b6435d0 and b927ef4a-bbb2-11e8-b2ba-ac1f6b6435d0
\n\nTest data:\n- 45b4b7b8-bad2-11e8-b2b8-ac1f6b6435d0 and 3f0d0104-bac8-11e8-b2b7-ac1f6b6435d0
", "rawMarkdown": "Examples:\n\nTrain data:\n - 115d3ca4-bbc9-11e8-b2bc-ac1f6b6435d0 and b927ef4a-bbb2-11e8-b2ba-ac1f6b6435d0\n\nTest data:\n- 45b4b7b8-bad2-11e8-b2b8-ac1f6b6435d0 and 3f0d0104-bac8-11e8-b2b7-ac1f6b6435d0\n\n ", "votes": 6 }, { "id": 427190, "postDate": "2018-11-24T20:29:14.540Z", "content": "Oh dear, this is interesting... Anyone got any idea how many actual dupes there are within the training set itself? If some are just darker, could we set anything above a specific threshold to 1, then do a dice coefficient or IoU overlap check?
\n\nEDIT: nevermind, looks like @Tilii sorted it.
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534
", "rawMarkdown": "Oh dear, this is interesting... Anyone got any idea how many actual dupes there are within the training set itself? If some are just darker, could we set anything above a specific threshold to 1, then do a dice coefficient or IoU overlap check?\n\nEDIT: nevermind, looks like @Tilii sorted it.\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534", "replies": [ { "id": 427271, "postDate": "2018-11-25T02:52:24.483Z", "content": "I came here to leave my duplicate image finding notebook and let you all know there are barely any duplicates before I saw this was settled.
\n\nI'll leave the link to it anyways and let the thread know I looked at @Tilii 's list of duplicates. Didn't look w/ fine tooth comb but the amount looks very similar to the possible duplicate numbers I found. About 70 to 80. Not worth removing and laboring over changing the csv training labels imo.
\n\nhttps://github.com/MotorCityCobra/face_ripper_9000/blob/master/duplicate_image_finder.ipynb
", "rawMarkdown": "I came here to leave my duplicate image finding notebook and let you all know there are barely any duplicates before I saw this was settled. \n\nI'll leave the link to it anyways and let the thread know I looked at @Tilii 's list of duplicates. Didn't look w/ fine tooth comb but the amount looks very similar to the possible duplicate numbers I found. About 70 to 80. Not worth removing and laboring over changing the csv training labels imo. \n\nhttps://github.com/MotorCityCobra/face_ripper_9000/blob/master/duplicate_image_finder.ipynb", "votes": 3 }, { "id": 427285, "postDate": "2018-11-25T04:50:05.957Z", "content": "Thanks for sharing. Once I already made such mistake (removed only part of duplicates) that caused some leak to val. The result: top 3 public -> top 38 private LB((( So, it is better to be safe.
", "rawMarkdown": "Thanks for sharing. Once I already made such mistake (removed only part of duplicates) that caused some leak to val. The result: top 3 public -> top 38 private LB((( So, it is better to be safe." }, { "id": 427493, "postDate": "2018-11-25T15:56:56.080Z", "content": "Thanks. I agree the numbers really are not worth worrying about. Different exposures may even add some information.
", "rawMarkdown": "Thanks. I agree the numbers really are not worth worrying about. Different exposures may even add some information." } ] }, { "id": 426888, "postDate": "2018-11-24T04:42:22.930Z", "content": "Looks like two issues. Brian sees duplicate overlap train/test in exactly identical images. Renan see thousands of duplicates in train set itself. For the former, I will see if I get the same and then do as DS says. For the latter, I will remove them from the train set. If they are VERY numerous, the distributions could be changed?
\n\n@brian: I just checked those images you mention. They are both there but in my data, they are pretty different.
", "rawMarkdown": "Looks like two issues. Brian sees duplicate overlap train/test in exactly identical images. Renan see thousands of duplicates in train set itself. For the former, I will see if I get the same and then do as DS says. For the latter, I will remove them from the train set. If they are VERY numerous, the distributions could be changed?\n\n@brian: I just checked those images you mention. They are both there but in my data, they are pretty different.", "replies": [ { "id": 426919, "postDate": "2018-11-24T05:44:56.007Z", "content": "I'm not sure if there is a straightforward way to identify pairs of images that Brian found. I tried MSE on resized images, but dark images have lower MSE between themselves than the pair of images suggested by Brian. They look similar, but pixel-wise they are different(
", "rawMarkdown": "I'm not sure if there is a straightforward way to identify pairs of images that Brian found. I tried MSE on resized images, but dark images have lower MSE between themselves than the pair of images suggested by Brian. They look similar, but pixel-wise they are different(" } ] }, { "id": 426709, "postDate": "2018-11-23T17:28:52.973Z", "content": ">What do you do
\n\nI guess they aren't going to comment.
\n\nYou're using something hand written script with PIL to find them? I have a script that does the same somewhere.
\n\nHow bad is the problem? How many duplicates have you found, (unless you don't want to answer for competitive purposes). Dozens? Hundreds? More?
", "rawMarkdown": ">What do you do\n\nI guess they aren't going to comment. \n\nYou're using something hand written script with PIL to find them? I have a script that does the same somewhere.\n\nHow bad is the problem? How many duplicates have you found, (unless you don't want to answer for competitive purposes). Dozens? Hundreds? More?", "replies": [ { "id": 426716, "postDate": "2018-11-23T17:39:30.990Z", "content": "Couple thousands so far
", "rawMarkdown": "Couple thousands so far", "votes": -1 }, { "id": 426735, "postDate": "2018-11-23T18:19:46.520Z", "rawMarkdown": "", "isDeleted": true }, { "id": 426746, "postDate": "2018-11-23T18:48:44.610Z", "content": "Since we're auditing the data we should get a biologists to check a few of the training set labels.
", "rawMarkdown": "Since we're auditing the data we should get a biologists to check a few of the training set labels. \n" }, { "id": 426873, "postDate": "2018-11-24T03:52:59.050Z", "content": "I think it would be really beneficial for others if you could provide the list of duplicates.
", "rawMarkdown": "I think it would be really beneficial for others if you could provide the list of duplicates." } ] }, { "id": 410500, "postDate": "2018-10-26T06:39:47.703Z", "content": "Do you mean we have a leakage? or there are multiple identical images in train and test but without overlap?
", "rawMarkdown": "Do you mean we have a leakage? or there are multiple identical images in train and test but without overlap?", "replies": [ { "id": 410569, "postDate": "2018-10-26T09:23:50.907Z", "content": "No leakage, there are multiple duplicates within each but without overlap.
", "rawMarkdown": "No leakage, there are multiple duplicates within each but without overlap.", "votes": 3 } ] }, { "id": 410488, "postDate": "2018-10-26T06:12:09.377Z", "content": "Just copy train results into test and enjoy your free lunch. :)
", "rawMarkdown": "Just copy train results into test and enjoy your free lunch. :)" } ], "comments": [ { "id": 427190, "author_name": "DStjhb", "author_url": "", "post_date": "2018-11-24T20:29:14.540000", "content": "Oh dear, this is interesting... Anyone got any idea how many actual dupes there are within the training set itself? If some are just darker, could we set anything above a specific threshold to 1, then do a dice coefficient or IoU overlap check?
\n\nEDIT: nevermind, looks like @Tilii sorted it.
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534
", "votes": 0, "replies": [ { "id": 427271, "author_name": "Telemachus", "author_url": "", "post_date": "2018-11-25T02:52:24.483000", "content": "I came here to leave my duplicate image finding notebook and let you all know there are barely any duplicates before I saw this was settled.
\n\nI'll leave the link to it anyways and let the thread know I looked at @Tilii 's list of duplicates. Didn't look w/ fine tooth comb but the amount looks very similar to the possible duplicate numbers I found. About 70 to 80. Not worth removing and laboring over changing the csv training labels imo.
\n\nhttps://github.com/MotorCityCobra/face_ripper_9000/blob/master/duplicate_image_finder.ipynb
", "votes": 3, "replies": [] }, { "id": 427285, "author_name": "Iafoss", "author_url": "", "post_date": "2018-11-25T04:50:05.957000", "content": "Thanks for sharing. Once I already made such mistake (removed only part of duplicates) that caused some leak to val. The result: top 3 public -> top 38 private LB((( So, it is better to be safe.
", "votes": 0, "replies": [] }, { "id": 427493, "author_name": "DStjhb", "author_url": "", "post_date": "2018-11-25T15:56:56.080000", "content": "Thanks. I agree the numbers really are not worth worrying about. Different exposures may even add some information.
", "votes": 0, "replies": [] } ] }, { "id": 426888, "author_name": "pete", "author_url": "", "post_date": "2018-11-24T04:42:22.930000", "content": "Looks like two issues. Brian sees duplicate overlap train/test in exactly identical images. Renan see thousands of duplicates in train set itself. For the former, I will see if I get the same and then do as DS says. For the latter, I will remove them from the train set. If they are VERY numerous, the distributions could be changed?
\n\n@brian: I just checked those images you mention. They are both there but in my data, they are pretty different.
", "votes": 0, "replies": [ { "id": 426919, "author_name": "Iafoss", "author_url": "", "post_date": "2018-11-24T05:44:56.007000", "content": "I'm not sure if there is a straightforward way to identify pairs of images that Brian found. I tried MSE on resized images, but dark images have lower MSE between themselves than the pair of images suggested by Brian. They look similar, but pixel-wise they are different(
", "votes": 0, "replies": [] } ] }, { "id": 426709, "author_name": "Telemachus", "author_url": "", "post_date": "2018-11-23T17:28:52.973000", "content": ">What do you do
\n\nI guess they aren't going to comment.
\n\nYou're using something hand written script with PIL to find them? I have a script that does the same somewhere.
\n\nHow bad is the problem? How many duplicates have you found, (unless you don't want to answer for competitive purposes). Dozens? Hundreds? More?
", "votes": 0, "replies": [ { "id": 426716, "author_name": "Renan Rodrigues dos Santos", "author_url": "", "post_date": "2018-11-23T17:39:30.990000", "content": "Couple thousands so far
", "votes": -1, "replies": [] }, { "id": 426735, "author_name": "", "author_url": "", "post_date": "2018-11-23T18:19:46.520000", "content": "", "votes": 0, "replies": [] }, { "id": 426746, "author_name": "Telemachus", "author_url": "", "post_date": "2018-11-23T18:48:44.610000", "content": "Since we're auditing the data we should get a biologists to check a few of the training set labels.
", "votes": 0, "replies": [] }, { "id": 426873, "author_name": "Iafoss", "author_url": "", "post_date": "2018-11-24T03:52:59.050000", "content": "I think it would be really beneficial for others if you could provide the list of duplicates.
", "votes": 0, "replies": [] } ] }, { "id": 410500, "author_name": "Iafoss", "author_url": "", "post_date": "2018-10-26T06:39:47.703000", "content": "Do you mean we have a leakage? or there are multiple identical images in train and test but without overlap?
", "votes": 0, "replies": [ { "id": 410569, "author_name": "Brian", "author_url": "", "post_date": "2018-10-26T09:23:50.907000", "content": "No leakage, there are multiple duplicates within each but without overlap.
", "votes": 3, "replies": [] } ] }, { "id": 410488, "author_name": "DennisSakva", "author_url": "", "post_date": "2018-10-26T06:12:09.377000", "content": "Just copy train results into test and enjoy your free lunch. :)
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "409996": "Examples:\n\nTrain data:\n - 115d3ca4-bbc9-11e8-b2bc-ac1f6b6435d0 and b927ef4a-bbb2-11e8-b2ba-ac1f6b6435d0\n\nTest data:\n- 45b4b7b8-bad2-11e8-b2b8-ac1f6b6435d0 and 3f0d0104-bac8-11e8-b2b7-ac1f6b6435d0\n\n ", "427190": "Oh dear, this is interesting... Anyone got any idea how many actual dupes there are within the training set itself? If some are just darker, could we set anything above a specific threshold to 1, then do a dice coefficient or IoU overlap check?\n\nEDIT: nevermind, looks like @Tilii sorted it.\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534", "426888": "Looks like two issues. Brian sees duplicate overlap train/test in exactly identical images. Renan see thousands of duplicates in train set itself. For the former, I will see if I get the same and then do as DS says. For the latter, I will remove them from the train set. If they are VERY numerous, the distributions could be changed?\n\n@brian: I just checked those images you mention. They are both there but in my data, they are pretty different.", "426709": ">What do you do\n\nI guess they aren't going to comment. \n\nYou're using something hand written script with PIL to find them? I have a script that does the same somewhere.\n\nHow bad is the problem? How many duplicates have you found, (unless you don't want to answer for competitive purposes). Dozens? Hundreds? More?", "410500": "Do you mean we have a leakage? or there are multiple identical images in train and test but without overlap?", "410488": "Just copy train results into test and enjoy your free lunch. :)" } }