\nlabel 1\nwsi camelyon16_train_tumor_007\nName: c448ff7117b8382151d115fb54a63987e0fb5c0d, dtype: object to label 1\nwsi camelyon16_train_tumor_006\nName: d547e4b7204266e4412887ad82ce59bae5852ba5, dtype: object\n\n![\"\"](\"https://i.postimg.cc/GpDTwnyS/download.png\")
This could explain why using WSI makes a difference!\nAlso, these duplicates images can inspire us about augmentations.\nWhat do you think?
", "messages": [ { "id": "495220", "postDate": "03/20/2019 20:42:07", "content": "\nlabel 1\nwsi camelyon16_train_tumor_007\nName: c448ff7117b8382151d115fb54a63987e0fb5c0d, dtype: object to label 1\nwsi camelyon16_train_tumor_006\nName: d547e4b7204266e4412887ad82ce59bae5852ba5, dtype: object\n\n
This could explain why using WSI makes a difference!\nAlso, these duplicates images can inspire us about augmentations.\nWhat do you think?
", "rawMarkdown": "```\nlabel 1\nwsi camelyon16_train_tumor_007\nName: c448ff7117b8382151d115fb54a63987e0fb5c0d, dtype: object to label 1\nwsi camelyon16_train_tumor_006\nName: d547e4b7204266e4412887ad82ce59bae5852ba5, dtype: object\n```\n\n\nThis could explain why using WSI makes a difference!\nAlso, these duplicates images can inspire us about augmentations.\nWhat do you think?", "votes": null }, { "id": "495731", "postDate": "03/21/2019 13:37:39", "content": "Yes you are right, if we don't split by WSI the model overfits to these duplicates
", "rawMarkdown": "Yes you are right, if we don't split by WSI the model overfits to these duplicates", "votes": null }, { "id": "495767", "postDate": "03/21/2019 14:37:24", "content": "The point is, even if we split according to wsi, we still cannot make a stable cv.
", "rawMarkdown": "The point is, even if we split according to wsi, we still cannot make a stable cv.", "votes": null }, { "id": "495886", "postDate": "03/21/2019 16:33:03", "content": "Good find! How did you discover this? I wonder how many more cross WSI duplicates there are.
", "rawMarkdown": "Good find! How did you discover this? I wonder how many more cross WSI duplicates there are.", "votes": null }, { "id": "496260", "postDate": "03/22/2019 02:25:32", "content": "Your trained model can tell you that! Looking at the probability I get from my model's prediction, I found some interesting patterns there. The playground competition is very boring compared to the Featured ones - so having some fun digging some golds is not a bad idea. (chat on twitter?)
Also, do you see the LB 1st there? with a huge increase in LB score than the rest of us. I bet he understands the data a lot better than we do.
", "rawMarkdown": "Your trained model can tell you that! Looking at the probability I get from my model's prediction, I found some interesting patterns there. The playground competition is very boring compared to the `Featured` ones - so having some fun digging some golds is not a bad idea. (chat on twitter?)\n\nAlso, do you see the LB 1st there? with a huge increase in LB score than the rest of us. I bet he understands the data a lot better than we do.", "votes": null }, { "id": "496429", "postDate": "03/22/2019 07:30:25", "content": "Domain knowledge is definitely useful here. Also, understanding how the WSIs were extracted could be helpful.
\n\nI extracted some feature vectors from the images and compared their distances. By just comparing the first 1000 training images to the rest, I found two duplicate sets. However, this is a very slow way of searching these and I'm probably not going to go through the whole set.
\n\n\na5e032f36bcf57c281c0d6c482ccfa7ef9251cf5\nlabel:1\nwsi:camelyon16_train_tumor_021\nbcbaa35feec3d65df42799179719ededa3daa2e5\nlabel:1\nwsi:not found\n06b546ead8e2fa3c423da707da163d31f1d954df\nlabel:1\nwsi:camelyon16_train_tumor_027\n474059a719a0810382a7ed3cf0b9bc3097e44ae0\nlabel:1\nwsi:not found\n
![\"Duplicates\"](\"https://i.imgur.com/cFC2gBt.png\")
Duplicates with the first 2000 training images (+ the ones above):
\n\n\ne41069001765d923f2865de764f70be74afc1307\nlabel:1\nwsi: not found\n41f1ae2f73a5fe6a8f6642dca5c6e14e7175a3d7\nlabel:1\nwsi: not found\nd8f32c46bf7557821cfa5cf114bbb1a347ac6438\nlabel:1\nwsi: not found\n9e17709ee5f13555ae5506a325af007b4752e51f\nlabel:1\nwsi: camelyon16_train_tumor_050\n48f19a11abb3b96bb3166182dbf9291b241b81cc\nlabel:1\nwsi: camelyon16_train_tumor_029\neccd0bbb88dfe5fd3fbce6db705581b4a72c035b\nlabel:1\nwsi: camelyon16_train_tumor_029\nee18d2d7a5549aa0a8624870594127be1a74b777\nlabel:1\nwsi: not found\n7727fd6be2163869b61f99d7233d46e9bc4cada9\nlabel:1\nwsi: not found\nf238c8d42363a347e43599c95fa70bfbdcacb151\nlabel:1\nwsi: not found\n9be2f92b143860ae6b49e67b526502152058d996\nlabel:1\nwsi: camelyon16_train_tumor_092\nc256b7c830b0c3027779f23b77843b4c4f341603\nlabel:1\nwsi: camelyon16_train_tumor_061\nc45586ea9a193578191d0f3b22548f061e83b084\nlabel:1\nwsi: camelyon16_train_tumor_061\n
![\"duplicates](\"https://i.imgur.com/l8mEQIv.png\")
The second id d547e4b7204266e4412887ad82ce59bae5852ba5 is not the same as the Image (2). This is reassuring because it means that there are yet no duplicates found with different WSIs.
The first 10,000 test images don't seem to match with any of the train images. So far no test-train leakage found.
", "rawMarkdown": "The first 10,000 test images don't seem to match with any of the train images. So far no test-train leakage found.", "votes": null }, { "id": "496591", "postDate": "03/22/2019 11:24:56", "content": "![\"\"](\"https://i.postimg.cc/3R2j4tCR/1.png\")
\nThis image will answer your question.
Do you know how @sermakarevich (SM) obtained the WSI id? I looked into the original dataset and did not find an efficient way to do so. If we can replicate his process (he only used means, but it would be better to also use stdev), we can find more WSI ids and that will help us to find more duplicates (because you would expect WSI to duplicate too)
", "rawMarkdown": "Do you know how @sermakarevich (SM) obtained the WSI id? I looked into the original dataset and did not find an efficient way to do so. If we can replicate his process (he only used means, but it would be better to also use stdev), we can find more WSI ids and that will help us to find more duplicates (because you would expect WSI to duplicate too)", "votes": null }, { "id": "496601", "postDate": "03/22/2019 11:35:38", "content": "Oh! Thank you for the catch. I will look for the id of Image (2) later.
Image (2) is 7a0c08fce92d57350add2b13e4c6e825122e7b1a\nInterestingly, they have the same WSI
Sorry, I don't know how to get the WSIs out of the original ds.
\n\nIf I have understood correctly, what's different in each WSI beside of the tissue sample, is the dye saturation. The coloring is not consistent and what affects the output are:\n- Staining mixture, the combination of hematoxylin & eosin. Different batches have different binding properties.\n- Staining procedure, dipping time. The longer each tissue sample on a glass slide is dipped in the mixture, the more color it gets.\n- Instruments, different light microscope.\n- The dyes tend to faint over time so if the images are captured immediately vs. an hour later, they will look a bit different.
\n\nPractically, all WSIs are different in the stain color intensities. If you could extract the amounts of stain in each WSI, perhaps it could be used to identify patch WSIs.\nThe color forming procedure in light microscopy is subtractive:
\n\nlight_that_gets_to_the_image = light_from_source - hematoxylin_amount*hematoxylin_absorption_factor - easin_amount*easin_absorption_factor
One way of doing this is color deconvolution. I tried this but didn't have much luck. I used HistomicsTK
", "rawMarkdown": "Sorry, I don't know how to get the WSIs out of the original ds.\n\nIf I have understood correctly, what's different in each WSI beside of the tissue sample, is the dye saturation. The coloring is not consistent and what affects the output are:\n- Staining mixture, the combination of hematoxylin & eosin. Different batches have different binding properties.\n- Staining procedure, dipping time. The longer each tissue sample on a glass slide is dipped in the mixture, the more color it gets.\n- Instruments, different light microscope.\n- The dyes tend to faint over time so if the images are captured immediately vs. an hour later, they will look a bit different.\n\nPractically, all WSIs are different in the stain color intensities. If you could extract the amounts of stain in each WSI, perhaps it could be used to identify patch WSIs.\nThe color forming procedure in light microscopy is subtractive: \n\n```light_that_gets_to_the_image = light_from_source - hematoxylin_amount*hematoxylin_absorption_factor - easin_amount*easin_absorption_factor```\n\nOne way of doing this is color deconvolution. I tried this but didn't have much luck. I used [HistomicsTK](https://digitalslidearchive.github.io/HistomicsTK/examples/color-deconvolution.html)", "votes": null }, { "id": "496715", "postDate": "03/22/2019 14:05:12", "content": "The patches from the dataset do not add up to a WSI, so using the RGB color to identify WSI would have a lot of noise. As far as I know, the network does a pretty good job interpreting the difference in dye saturation, but finding the leakage is more important.
", "rawMarkdown": "The patches from the dataset do not add up to a WSI, so using the RGB color to identify WSI would have a lot of noise. As far as I know, the network does a pretty good job interpreting the difference in dye saturation, but finding the leakage is more important.", "votes": null }, { "id": "496738", "postDate": "03/22/2019 14:27:12", "content": "I believe this was stated in the overview for the competition. The WSI labels are derived from PCam.
\n\nThe data for this competition is a slightly modified version of the PatchCamelyon (PCam) benchmark dataset (the original PCam dataset contains duplicate images due to its probabilistic sampling, however, the version presented on Kaggle does not contain duplicates).
@qitvision I have assigned all dataset images to their correct WSI so none remain as unknown. See my post https://www.kaggle.com/c/histopathologic-cancer-detection/discussion/85283
", "rawMarkdown": "qitvision I have assigned all dataset images to their correct WSI so none remain as unknown. See my post https://www.kaggle.com/c/histopathologic-cancer-detection/discussion/85283", "votes": null }, { "id": "496903", "postDate": "03/22/2019 18:17:15", "content": "After @markdesimone released patch_id_wsi_full (thanks), it seems like the duplicates are all from the same WSI id. 8/2000=0.4%(+-1%?) of the training images have duplicates. If the above is true, the expected increase in LB would be about 0.0005~0.0001... Not my priority to work on.
Nice, a clever solution!
", "rawMarkdown": "Nice, a clever solution!", "votes": null }, { "id": "497013", "postDate": "03/22/2019 20:38:02", "content": "Thanks, I just got started on this but your excellent kernel is a great starting point, thank you
", "rawMarkdown": "Thanks, I just got started on this but your excellent kernel is a great starting point, thank you", "votes": null }, { "id": "497538", "postDate": "03/23/2019 16:41:05", "content": "Duplicate search:\n- found 396 duplicates in training set (this list is not exhaustive)\n- all duplicates have the same WSI\n- found 0 duplicates between train and test sets
\n\nTraining set duplicate id's are in duplicates.csv
", "rawMarkdown": "**Duplicate search**:\n- found 396 duplicates in training set (this list is not exhaustive)\n- all duplicates have the same WSI\n- found 0 duplicates between train and test sets\n\nTraining set duplicate id's are in *duplicates.csv*", "votes": null }, { "id": "497549", "postDate": "03/23/2019 16:49:27", "content": "Thank you for sharing! How did you find them?
", "rawMarkdown": "Thank you for sharing! How did you find them?", "votes": null }, { "id": "497571", "postDate": "03/23/2019 17:08:52", "content": "I extracted color and texture features and then performed a nearest neighbor search between feature vectors. If the distance was less than a set threshold, it was a match. I set a quite low threshold to filter out false duplicates so this didn't find all the pairs. For example, this didn't find that duplicate pair you found.
", "rawMarkdown": "I extracted color and texture features and then performed a nearest neighbor search between feature vectors. If the distance was less than a set threshold, it was a match. I set a quite low threshold to filter out false duplicates so this didn't find all the pairs. For example, this didn't find that duplicate pair you found.", "votes": null } ], "comments": [ { "id": 495731, "author_name": "guntherthepenguin", "author_url": "", "post_date": "03/21/2019 13:37:39", "content": "Yes you are right, if we don't split by WSI the model overfits to these duplicates
", "votes": null, "replies": [] }, { "id": 495767, "author_name": "kokecacao", "author_url": "", "post_date": "03/21/2019 14:37:24", "content": "The point is, even if we split according to wsi, we still cannot make a stable cv.
", "votes": null, "replies": [] }, { "id": 495886, "author_name": "qitvision", "author_url": "", "post_date": "03/21/2019 16:33:03", "content": "Good find! How did you discover this? I wonder how many more cross WSI duplicates there are.
", "votes": null, "replies": [ { "id": 496260, "author_name": "kokecacao", "author_url": "", "post_date": "03/22/2019 02:25:32", "content": "Your trained model can tell you that! Looking at the probability I get from my model's prediction, I found some interesting patterns there. The playground competition is very boring compared to the Featured ones - so having some fun digging some golds is not a bad idea. (chat on twitter?)
Also, do you see the LB 1st there? with a huge increase in LB score than the rest of us. I bet he understands the data a lot better than we do.
", "votes": null, "replies": [] }, { "id": 496429, "author_name": "qitvision", "author_url": "", "post_date": "03/22/2019 07:30:25", "content": "Domain knowledge is definitely useful here. Also, understanding how the WSIs were extracted could be helpful.
\n\nI extracted some feature vectors from the images and compared their distances. By just comparing the first 1000 training images to the rest, I found two duplicate sets. However, this is a very slow way of searching these and I'm probably not going to go through the whole set.
\n\n\na5e032f36bcf57c281c0d6c482ccfa7ef9251cf5\nlabel:1\nwsi:camelyon16_train_tumor_021\nbcbaa35feec3d65df42799179719ededa3daa2e5\nlabel:1\nwsi:not found\n06b546ead8e2fa3c423da707da163d31f1d954df\nlabel:1\nwsi:camelyon16_train_tumor_027\n474059a719a0810382a7ed3cf0b9bc3097e44ae0\nlabel:1\nwsi:not found\n
Duplicates with the first 2000 training images (+ the ones above):
\n\n\ne41069001765d923f2865de764f70be74afc1307\nlabel:1\nwsi: not found\n41f1ae2f73a5fe6a8f6642dca5c6e14e7175a3d7\nlabel:1\nwsi: not found\nd8f32c46bf7557821cfa5cf114bbb1a347ac6438\nlabel:1\nwsi: not found\n9e17709ee5f13555ae5506a325af007b4752e51f\nlabel:1\nwsi: camelyon16_train_tumor_050\n48f19a11abb3b96bb3166182dbf9291b241b81cc\nlabel:1\nwsi: camelyon16_train_tumor_029\neccd0bbb88dfe5fd3fbce6db705581b4a72c035b\nlabel:1\nwsi: camelyon16_train_tumor_029\nee18d2d7a5549aa0a8624870594127be1a74b777\nlabel:1\nwsi: not found\n7727fd6be2163869b61f99d7233d46e9bc4cada9\nlabel:1\nwsi: not found\nf238c8d42363a347e43599c95fa70bfbdcacb151\nlabel:1\nwsi: not found\n9be2f92b143860ae6b49e67b526502152058d996\nlabel:1\nwsi: camelyon16_train_tumor_092\nc256b7c830b0c3027779f23b77843b4c4f341603\nlabel:1\nwsi: camelyon16_train_tumor_061\nc45586ea9a193578191d0f3b22548f061e83b084\nlabel:1\nwsi: camelyon16_train_tumor_061\n
The first 10,000 test images don't seem to match with any of the train images. So far no test-train leakage found.
", "votes": null, "replies": [] }, { "id": 496591, "author_name": "kokecacao", "author_url": "", "post_date": "03/22/2019 11:24:56", "content": "\nThis image will answer your question.
Do you know how @sermakarevich (SM) obtained the WSI id? I looked into the original dataset and did not find an efficient way to do so. If we can replicate his process (he only used means, but it would be better to also use stdev), we can find more WSI ids and that will help us to find more duplicates (because you would expect WSI to duplicate too)
", "votes": null, "replies": [] }, { "id": 496634, "author_name": "qitvision", "author_url": "", "post_date": "03/22/2019 12:15:49", "content": "Sorry, I don't know how to get the WSIs out of the original ds.
\n\nIf I have understood correctly, what's different in each WSI beside of the tissue sample, is the dye saturation. The coloring is not consistent and what affects the output are:\n- Staining mixture, the combination of hematoxylin & eosin. Different batches have different binding properties.\n- Staining procedure, dipping time. The longer each tissue sample on a glass slide is dipped in the mixture, the more color it gets.\n- Instruments, different light microscope.\n- The dyes tend to faint over time so if the images are captured immediately vs. an hour later, they will look a bit different.
\n\nPractically, all WSIs are different in the stain color intensities. If you could extract the amounts of stain in each WSI, perhaps it could be used to identify patch WSIs.\nThe color forming procedure in light microscopy is subtractive:
\n\nlight_that_gets_to_the_image = light_from_source - hematoxylin_amount*hematoxylin_absorption_factor - easin_amount*easin_absorption_factor
One way of doing this is color deconvolution. I tried this but didn't have much luck. I used HistomicsTK
", "votes": null, "replies": [] }, { "id": 496715, "author_name": "kokecacao", "author_url": "", "post_date": "03/22/2019 14:05:12", "content": "The patches from the dataset do not add up to a WSI, so using the RGB color to identify WSI would have a lot of noise. As far as I know, the network does a pretty good job interpreting the difference in dye saturation, but finding the leakage is more important.
", "votes": null, "replies": [] }, { "id": 496903, "author_name": "kokecacao", "author_url": "", "post_date": "03/22/2019 18:17:15", "content": "After @markdesimone released patch_id_wsi_full (thanks), it seems like the duplicates are all from the same WSI id. 8/2000=0.4%(+-1%?) of the training images have duplicates. If the above is true, the expected increase in LB would be about 0.0005~0.0001... Not my priority to work on.
The second id d547e4b7204266e4412887ad82ce59bae5852ba5 is not the same as the Image (2). This is reassuring because it means that there are yet no duplicates found with different WSIs.
Oh! Thank you for the catch. I will look for the id of Image (2) later.
Image (2) is 7a0c08fce92d57350add2b13e4c6e825122e7b1a\nInterestingly, they have the same WSI
I believe this was stated in the overview for the competition. The WSI labels are derived from PCam.
\n\nThe data for this competition is a slightly modified version of the PatchCamelyon (PCam) benchmark dataset (the original PCam dataset contains duplicate images due to its probabilistic sampling, however, the version presented on Kaggle does not contain duplicates).
@qitvision I have assigned all dataset images to their correct WSI so none remain as unknown. See my post https://www.kaggle.com/c/histopathologic-cancer-detection/discussion/85283
", "votes": null, "replies": [ { "id": 496937, "author_name": "qitvision", "author_url": "", "post_date": "03/22/2019 18:52:15", "content": "Nice, a clever solution!
", "votes": null, "replies": [] }, { "id": 497013, "author_name": "markdesimone", "author_url": "", "post_date": "03/22/2019 20:38:02", "content": "Thanks, I just got started on this but your excellent kernel is a great starting point, thank you
", "votes": null, "replies": [] } ] }, { "id": 497538, "author_name": "qitvision", "author_url": "", "post_date": "03/23/2019 16:41:05", "content": "Duplicate search:\n- found 396 duplicates in training set (this list is not exhaustive)\n- all duplicates have the same WSI\n- found 0 duplicates between train and test sets
\n\nTraining set duplicate id's are in duplicates.csv
", "votes": null, "replies": [ { "id": 497549, "author_name": "kokecacao", "author_url": "", "post_date": "03/23/2019 16:49:27", "content": "Thank you for sharing! How did you find them?
", "votes": null, "replies": [] }, { "id": 497571, "author_name": "qitvision", "author_url": "", "post_date": "03/23/2019 17:08:52", "content": "I extracted color and texture features and then performed a nearest neighbor search between feature vectors. If the distance was less than a set threshold, it was a match. I set a quite low threshold to filter out false duplicates so this didn't find all the pairs. For example, this didn't find that duplicate pair you found.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "495220": "```\nlabel 1\nwsi camelyon16_train_tumor_007\nName: c448ff7117b8382151d115fb54a63987e0fb5c0d, dtype: object to label 1\nwsi camelyon16_train_tumor_006\nName: d547e4b7204266e4412887ad82ce59bae5852ba5, dtype: object\n```\n\n\nThis could explain why using WSI makes a difference!\nAlso, these duplicates images can inspire us about augmentations.\nWhat do you think?", "495731": "Yes you are right, if we don't split by WSI the model overfits to these duplicates", "495767": "The point is, even if we split according to wsi, we still cannot make a stable cv.", "495886": "Good find! How did you discover this? I wonder how many more cross WSI duplicates there are.", "496260": "Your trained model can tell you that! Looking at the probability I get from my model's prediction, I found some interesting patterns there. The playground competition is very boring compared to the `Featured` ones - so having some fun digging some golds is not a bad idea. (chat on twitter?)\n\nAlso, do you see the LB 1st there? with a huge increase in LB score than the rest of us. I bet he understands the data a lot better than we do.", "496429": "Domain knowledge is definitely useful here. Also, understanding how the WSIs were extracted could be helpful.\n\nI extracted some feature vectors from the images and compared their distances. By just comparing the first 1000 training images to the rest, I found two duplicate sets. However, this is a very slow way of searching these and I'm probably not going to go through the whole set.\n\n```\na5e032f36bcf57c281c0d6c482ccfa7ef9251cf5\nlabel:1\nwsi:camelyon16_train_tumor_021\nbcbaa35feec3d65df42799179719ededa3daa2e5\nlabel:1\nwsi:not found\n06b546ead8e2fa3c423da707da163d31f1d954df\nlabel:1\nwsi:camelyon16_train_tumor_027\n474059a719a0810382a7ed3cf0b9bc3097e44ae0\nlabel:1\nwsi:not found\n```\n\n", "496477": "Duplicates with the first 2000 training images (+ the ones above):\n\n```\ne41069001765d923f2865de764f70be74afc1307\nlabel:1\nwsi: not found\n41f1ae2f73a5fe6a8f6642dca5c6e14e7175a3d7\nlabel:1\nwsi: not found\nd8f32c46bf7557821cfa5cf114bbb1a347ac6438\nlabel:1\nwsi: not found\n9e17709ee5f13555ae5506a325af007b4752e51f\nlabel:1\nwsi: camelyon16_train_tumor_050\n48f19a11abb3b96bb3166182dbf9291b241b81cc\nlabel:1\nwsi: camelyon16_train_tumor_029\neccd0bbb88dfe5fd3fbce6db705581b4a72c035b\nlabel:1\nwsi: camelyon16_train_tumor_029\nee18d2d7a5549aa0a8624870594127be1a74b777\nlabel:1\nwsi: not found\n7727fd6be2163869b61f99d7233d46e9bc4cada9\nlabel:1\nwsi: not found\nf238c8d42363a347e43599c95fa70bfbdcacb151\nlabel:1\nwsi: not found\n9be2f92b143860ae6b49e67b526502152058d996\nlabel:1\nwsi: camelyon16_train_tumor_092\nc256b7c830b0c3027779f23b77843b4c4f341603\nlabel:1\nwsi: camelyon16_train_tumor_061\nc45586ea9a193578191d0f3b22548f061e83b084\nlabel:1\nwsi: camelyon16_train_tumor_061\n```\n\n", "496482": "The second id `d547e4b7204266e4412887ad82ce59bae5852ba5` is not the same as the `Image (2)`. This is reassuring because it means that there are yet no duplicates found with different WSIs.", "496558": "The first 10,000 test images don't seem to match with any of the train images. So far no test-train leakage found.", "496591": "\nThis image will answer your question.", "496596": "Do you know how @sermakarevich (SM) obtained the WSI id? I looked into the original dataset and did not find an efficient way to do so. If we can replicate his process (he only used means, but it would be better to also use stdev), we can find more WSI ids and that will help us to find more duplicates (because you would expect WSI to duplicate too)", "496601": "Oh! Thank you for the catch. I will look for the id of `Image (2)` later.", "496620": "`Image (2)` is `7a0c08fce92d57350add2b13e4c6e825122e7b1a`\nInterestingly, they have the same WSI", "496634": "Sorry, I don't know how to get the WSIs out of the original ds.\n\nIf I have understood correctly, what's different in each WSI beside of the tissue sample, is the dye saturation. The coloring is not consistent and what affects the output are:\n- Staining mixture, the combination of hematoxylin & eosin. Different batches have different binding properties.\n- Staining procedure, dipping time. The longer each tissue sample on a glass slide is dipped in the mixture, the more color it gets.\n- Instruments, different light microscope.\n- The dyes tend to faint over time so if the images are captured immediately vs. an hour later, they will look a bit different.\n\nPractically, all WSIs are different in the stain color intensities. If you could extract the amounts of stain in each WSI, perhaps it could be used to identify patch WSIs.\nThe color forming procedure in light microscopy is subtractive: \n\n```light_that_gets_to_the_image = light_from_source - hematoxylin_amount*hematoxylin_absorption_factor - easin_amount*easin_absorption_factor```\n\nOne way of doing this is color deconvolution. I tried this but didn't have much luck. I used [HistomicsTK](https://digitalslidearchive.github.io/HistomicsTK/examples/color-deconvolution.html)", "496715": "The patches from the dataset do not add up to a WSI, so using the RGB color to identify WSI would have a lot of noise. As far as I know, the network does a pretty good job interpreting the difference in dye saturation, but finding the leakage is more important.", "496738": "I believe this was stated in the overview for the competition. The WSI labels are derived from PCam. \n\n` The data for this competition is a slightly modified version of the PatchCamelyon (PCam) benchmark dataset (the original PCam dataset contains duplicate images due to its probabilistic sampling, however, the version presented on Kaggle does not contain duplicates).`", "496880": "qitvision I have assigned all dataset images to their correct WSI so none remain as unknown. See my post https://www.kaggle.com/c/histopathologic-cancer-detection/discussion/85283", "496903": "After @markdesimone released `patch_id_wsi_full` (thanks), it seems like the duplicates are all from the same WSI id. 8/2000=0.4%(+-1%?) of the training images have duplicates. If the above is true, the expected increase in LB would be about 0.0005~0.0001... Not my priority to work on.", "496937": "Nice, a clever solution!", "497013": "Thanks, I just got started on this but your excellent kernel is a great starting point, thank you", "497538": "**Duplicate search**:\n- found 396 duplicates in training set (this list is not exhaustive)\n- all duplicates have the same WSI\n- found 0 duplicates between train and test sets\n\nTraining set duplicate id's are in *duplicates.csv*", "497549": "Thank you for sharing! How did you find them?", "497571": "I extracted color and texture features and then performed a nearest neighbor search between feature vectors. If the distance was less than a set threshold, it was a match. I set a quite low threshold to filter out false duplicates so this didn't find all the pairs. For example, this didn't find that duplicate pair you found." }, "source": "meta" }