![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F682501%2Faa9cd93cb1be379349c7465c1f01a528%2Flb_changes.png?generation=1565166644397459&alt=media\")
We want to make everyone aware of structure in the data that we have not previously disclosed, and thank team Double Strand for encouraging us to do so. As you may know by now, each experiment contains at most one well for each of the 1,108 treatment siRNAs (any treatment siRNA not appearing in an experiment is due to an operational error that occurred while attempting to transfer siRNA from “source” plates to “destination\" plates used in an experiment). What you may not have noticed yet is that these treatment siRNAs are divided into four groups of 277, and each group always occurs together in a single plate per experiment. While the plate the group is assigned within an experiment is randomly chosen, and the individual well each siRNA occupies is randomly assigned per plate, the plate will nevertheless contain the same 277 treatment siRNAs as a corresponding plate in another experiment. This structure occurs as the result of simple operational efficiency — Recursion stores the 1,108 treatment siRNAs in source plates containing 277 siRNA each, and transfers siRNA from these source plates to the destination plates used in an experiment by plate. Exploiting this non-essential structure in your models can provide some lift to your results, and thus we wanted all to be aware of it.
", "messages": [ { "id": "592853", "postDate": "08/05/2019 22:30:52", "content": "We want to make everyone aware of structure in the data that we have not previously disclosed, and thank team Double Strand for encouraging us to do so. As you may know by now, each experiment contains at most one well for each of the 1,108 treatment siRNAs (any treatment siRNA not appearing in an experiment is due to an operational error that occurred while attempting to transfer siRNA from “source” plates to “destination\" plates used in an experiment). What you may not have noticed yet is that these treatment siRNAs are divided into four groups of 277, and each group always occurs together in a single plate per experiment. While the plate the group is assigned within an experiment is randomly chosen, and the individual well each siRNA occupies is randomly assigned per plate, the plate will nevertheless contain the same 277 treatment siRNAs as a corresponding plate in another experiment. This structure occurs as the result of simple operational efficiency — Recursion stores the 1,108 treatment siRNAs in source plates containing 277 siRNA each, and transfers siRNA from these source plates to the destination plates used in an experiment by plate. Exploiting this non-essential structure in your models can provide some lift to your results, and thus we wanted all to be aware of it.
", "rawMarkdown": "We want to make everyone aware of structure in the data that we have not previously disclosed, and thank team Double Strand for encouraging us to do so. As you may know by now, each experiment contains at most one well for each of the 1,108 treatment siRNAs (any treatment siRNA not appearing in an experiment is due to an operational error that occurred while attempting to transfer siRNA from “source” plates to “destination\" plates used in an experiment). What you may not have noticed yet is that these treatment siRNAs are divided into four groups of 277, and each group always occurs together in a single plate per experiment. While the plate the group is assigned within an experiment is randomly chosen, and the individual well each siRNA occupies is randomly assigned per plate, the plate will nevertheless contain the same 277 treatment siRNAs as a corresponding plate in another experiment. This structure occurs as the result of simple operational efficiency — Recursion stores the 1,108 treatment siRNAs in source plates containing 277 siRNA each, and transfers siRNA from these source plates to the destination plates used in an experiment *by plate*. Exploiting this non-essential structure in your models can provide some lift to your results, and thus we wanted all to be aware of it.", "votes": null }, { "id": "592878", "postDate": "08/06/2019 00:05:18", "content": "Another point here is that out of 4*3*2=24 possible group assignments to plates there are only 3 in the train data (can be easily observed), and 4 in the test data (our speculation based on some evidence, see the kernel). It seems like Recursion uses some kind of rotation operation on plates, therefore only 4 options.
There is a kernel that I prepared to demonstrate the leak and show the way to exploit it, it improves the selected public kernel result from 0.113 to 0.207. I can also report that when we found it, it improved our LB 0.661 to 0.769, a similar boost.
\n\nThanks to Recursion for making the public announcement, - this pattern is somewhat tricky to spot, but the competition is not about that. So we now have a problem of assigning 277 sirnas to 277 wells 4*18=72 times, still a very difficult problem. Good luck to all the participants with the updated challenge!
A visualization here: https://www.kaggle.com/giuliasavorgnan/plates-leak-clear-visualization
", "rawMarkdown": "A visualization here: https://www.kaggle.com/giuliasavorgnan/plates-leak-clear-visualization", "votes": null }, { "id": "593315", "postDate": "08/06/2019 12:50:48", "content": "Thanks for the announcement, this is important to know and should save some people some frustration..
\n\nBut people are calling this a leak - is this really considered a data leak?\nI found this like half an hour into my initial EDA - it is more just the structure of the problem than a data leak.
", "rawMarkdown": "Thanks for the announcement, this is important to know and should save some people some frustration.. \n\nBut people are calling this a leak - is this really considered a data leak?\nI found this like half an hour into my initial EDA - it is more just the structure of the problem than a data leak.", "votes": null }, { "id": "593319", "postDate": "08/06/2019 12:56:17", "content": "How is this NOT a leak? Now that you know that the treatments always occur in groups of 277, you can filter your predictions knowing this important hidden point. Is this pattern occurring in the test set? I would almost surely expect so and @zaharch's kernel seems to confirm this.
", "rawMarkdown": "How is this NOT a leak? Now that you know that the treatments always occur in groups of 277, you can filter your predictions knowing this important hidden point. Is this pattern occurring in the test set? I would almost surely expect so and @zaharch's [kernel](https://www.kaggle.com/zaharch/keras-model-boosted-with-plates-leak) seems to confirm this.", "votes": null }, { "id": "593356", "postDate": "08/06/2019 13:41:37", "content": "Right, sorry I take that back haha I was a bit quick to write that without doing some reading.
\n\nYes it is definitely a leak and my half an hour of EDA definitely did not find this. Just found the pattern in the train set but not the test set :) So this definitely saved me a bit of time - in fact I just cancelled three training jobs :p
", "rawMarkdown": "Right, sorry I take that back haha I was a bit quick to write that without doing some reading. \n\nYes it is definitely a leak and my half an hour of EDA definitely did not find this. Just found the pattern in the train set but not the test set :) So this definitely saved me a bit of time - in fact I just cancelled three training jobs :p", "votes": null }, { "id": "593706", "postDate": "08/07/2019 02:18:18", "content": "Thanks for taking the time to find, explain, and demonstrate this aspect of the data.
", "rawMarkdown": "Thanks for taking the time to find, explain, and demonstrate this aspect of the data.", "votes": null }, { "id": "593721", "postDate": "08/07/2019 02:49:25", "content": "How many scores may be increased? For your reference, the figure shows the changes of LB scores before and after the \"277 leaks\" is disclosed:\n---update 0807---\n
Related Code is in the attach file.
", "rawMarkdown": "How many scores may be increased? For your reference, the figure shows the changes of LB scores before and after the \"277 leaks\" is disclosed:\n---update 0807---\n\n\nRelated Code is in the attach file.", "votes": null }, { "id": "593925", "postDate": "08/07/2019 09:47:32", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F455216%2F2369c4f4b0e07c1750d1ec76ea8569fe%2FScreenshot%202019-08-07%2010.04.36.png?generation=1565171003470494&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F455216%2F0a73b8c363433db395d7f7c3a2c237cf%2FScreenshot%202019-08-07%2010.04.42.png?generation=1565171028206946&alt=media\")
These two experiments have the same pattern of controls, but not only... also the treatments are in the same pattern, but the plates are rotated.
\n\nNow, there is an experiment in the test set that has the same pattern of controls... HUVEC-18. You can check what I said in this kernel
\n\nDoes HUVEC-18 also have the same pattern of treatments in some plate rotation?
", "rawMarkdown": "\n\n\nThese two experiments have the same pattern of controls, but not only... also the treatments are in the same pattern, but the plates are rotated. \n\nNow, there is an experiment in the test set that has the same pattern of controls... HUVEC-18. You can check what I said in [this kernel](https://www.kaggle.com/giuliasavorgnan/plates-leak-clear-visualization)\n\nDoes HUVEC-18 also have the same pattern of treatments in some plate rotation?", "votes": null }, { "id": "593933", "postDate": "08/07/2019 10:00:09", "content": "Wow, this is a crazy finding, I hope the competition is not in danger. Recursion has to take a serious look at their random number generator.
", "rawMarkdown": "Wow, this is a crazy finding, I hope the competition is not in danger. Recursion has to take a serious look at their random number generator.", "votes": null }, { "id": "593942", "postDate": "08/07/2019 10:12:52", "content": "I figured it would be better to disclose any leak if aware of it, before this comp becomes a combinatory problem instead of a data science one...
", "rawMarkdown": "I figured it would be better to disclose any leak if aware of it, before this comp becomes a combinatory problem instead of a data science one...", "votes": null }, { "id": "593965", "postDate": "08/07/2019 11:03:19", "content": "The good news is: this seems to be the only case in the train set.
", "rawMarkdown": "The good news is: this seems to be the only case in the train set.", "votes": null }, { "id": "594004", "postDate": "08/07/2019 12:23:07", "content": "Since HUVEC-18 is in private test set (according to this post: https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/98075#latest-567690), we can't check if the same treatment placement holds. If it is indeed the case, then everyone should basically modify 1 final sub just to account for this possibility. Maybe organisers should remove this experiment from scoring?
", "rawMarkdown": "Since HUVEC-18 is in private test set (according to this post: https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/98075#latest-567690), we can't check if the same treatment placement holds. If it is indeed the case, then everyone should basically modify 1 final sub just to account for this possibility. Maybe organisers should remove this experiment from scoring?", "votes": null }, { "id": "594008", "postDate": "08/07/2019 12:32:25", "content": "Good point, I had missed that post. Agreed on the removal.
", "rawMarkdown": "Good point, I had missed that post. Agreed on the removal.", "votes": null }, { "id": "594018", "postDate": "08/07/2019 12:54:26", "content": "Could we get a stage 2 testset without well&plate information?
", "rawMarkdown": "Could we get a stage 2 testset without well&plate information?", "votes": null }, { "id": "595252", "postDate": "08/09/2019 02:11:38", "content": "Can someone explain in layman terms how having 277 siRNA on a plate help increase model accuracy? Is it that you will have to train only for 277 siRNA per plate rather 1108?
", "rawMarkdown": "Can someone explain in layman terms how having 277 siRNA on a plate help increase model accuracy? Is it that you will have to train only for 277 siRNA per plate rather 1108?", "votes": null }, { "id": "595305", "postDate": "08/09/2019 04:54:23", "content": "Yes, you can train models that have only 277 classes. However it can benefit models that have already been trained as you can artificially set 3*277 of the predicted results to 0 and then just look at the strongest prediction for the remaining 277 predictions.
", "rawMarkdown": "Yes, you can train models that have only 277 classes. However it can benefit models that have already been trained as you can artificially set 3*277 of the predicted results to 0 and then just look at the strongest prediction for the remaining 277 predictions.", "votes": null }, { "id": "596369", "postDate": "08/10/2019 14:41:39", "content": "Thanks for the explanation!
", "rawMarkdown": "Thanks for the explanation!", "votes": null }, { "id": "602180", "postDate": "08/18/2019 17:56:10", "content": "FYI, It looks like this leakage has since been removed from the leaderboard scoring https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/104011#latest-598578 Great find @giuliasavorgnan!
", "rawMarkdown": "FYI, It looks like this leakage has since been removed from the leaderboard scoring https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/104011#latest-598578 Great find @giuliasavorgnan!", "votes": null }, { "id": "602240", "postDate": "08/18/2019 20:03:31", "content": "Hi @bearnshaw
\n\nWhat subcellular structure does each channel represent? and is the channel meaning constant? Because I'm pretty sure you're using DAPI. For example:
\n\n\n1. Nucleus \n2. Nucleoplasm\n3. Nucleoli\n4. Nucleoli fibrillar center\n5. Nuclear speckles\n6. Cytoskeleton\n
In this case I can see clearly:
\n\n\n1. Nucleus\n3. Cytoskeleton\n6. Cytoskeleton\n
Thanks!
", "rawMarkdown": "Hi @bearnshaw \n\nWhat subcellular structure does each **channel** represent? and is the channel meaning constant? Because I'm pretty sure you're using DAPI. For example:\n\n```\n1. Nucleus \n2. Nucleoplasm\n3. Nucleoli\n4. Nucleoli fibrillar center\n5. Nuclear speckles\n6. Cytoskeleton\n```\n\nIn this case I can see clearly:\n\n```\n1. Nucleus\n3. Cytoskeleton\n6. Cytoskeleton\n```\n\n\n\n\nThanks!", "votes": null }, { "id": "603270", "postDate": "08/20/2019 04:54:13", "content": "Just guessing, but according to the provided info,
\n\nit seems
\n\n@zaharch Thx for the detailed explanation! excuse me for not understanding, why are there 4*3*2 group assignments? Also 4*18
", "rawMarkdown": "zaharch Thx for the detailed explanation! excuse me for not understanding, why are there 4*3*2 group assignments? Also 4*18", "votes": null }, { "id": "624588", "postDate": "09/12/2019 07:40:41", "content": "@roguekk007 , there are 4 numbered groups of sirnas 1,2,3,4 and 4 numbered plates. It is just combinatorics, there are 24 different ways to match them. 4*18 is number of plates (4) and number of test experiments (18), for each we need to match 277 sirnas into 277 wells.
", "rawMarkdown": "roguekk007 , there are 4 numbered groups of sirnas 1,2,3,4 and 4 numbered plates. It is just combinatorics, there are 24 different ways to match them. 4*18 is number of plates (4) and number of test experiments (18), for each we need to match 277 sirnas into 277 wells.", "votes": null } ], "comments": [ { "id": 592878, "author_name": "zaharch", "author_url": "", "post_date": "08/06/2019 00:05:18", "content": "Another point here is that out of 4*3*2=24 possible group assignments to plates there are only 3 in the train data (can be easily observed), and 4 in the test data (our speculation based on some evidence, see the kernel). It seems like Recursion uses some kind of rotation operation on plates, therefore only 4 options.
There is a kernel that I prepared to demonstrate the leak and show the way to exploit it, it improves the selected public kernel result from 0.113 to 0.207. I can also report that when we found it, it improved our LB 0.661 to 0.769, a similar boost.
\n\nThanks to Recursion for making the public announcement, - this pattern is somewhat tricky to spot, but the competition is not about that. So we now have a problem of assigning 277 sirnas to 277 wells 4*18=72 times, still a very difficult problem. Good luck to all the participants with the updated challenge!
Thanks for taking the time to find, explain, and demonstrate this aspect of the data.
", "votes": null, "replies": [] }, { "id": 624515, "author_name": "roguekk007", "author_url": "", "post_date": "09/12/2019 06:45:17", "content": "@zaharch Thx for the detailed explanation! excuse me for not understanding, why are there 4*3*2 group assignments? Also 4*18
", "votes": null, "replies": [] }, { "id": 624588, "author_name": "zaharch", "author_url": "", "post_date": "09/12/2019 07:40:41", "content": "@roguekk007 , there are 4 numbered groups of sirnas 1,2,3,4 and 4 numbered plates. It is just combinatorics, there are 24 different ways to match them. 4*18 is number of plates (4) and number of test experiments (18), for each we need to match 277 sirnas into 277 wells.
", "votes": null, "replies": [] } ] }, { "id": 593239, "author_name": "giuliasavorgnan", "author_url": "", "post_date": "08/06/2019 10:38:39", "content": "A visualization here: https://www.kaggle.com/giuliasavorgnan/plates-leak-clear-visualization
", "votes": null, "replies": [] }, { "id": 593315, "author_name": "cherring", "author_url": "", "post_date": "08/06/2019 12:50:48", "content": "Thanks for the announcement, this is important to know and should save some people some frustration..
\n\nBut people are calling this a leak - is this really considered a data leak?\nI found this like half an hour into my initial EDA - it is more just the structure of the problem than a data leak.
", "votes": null, "replies": [ { "id": 593319, "author_name": "giuliasavorgnan", "author_url": "", "post_date": "08/06/2019 12:56:17", "content": "How is this NOT a leak? Now that you know that the treatments always occur in groups of 277, you can filter your predictions knowing this important hidden point. Is this pattern occurring in the test set? I would almost surely expect so and @zaharch's kernel seems to confirm this.
", "votes": null, "replies": [] }, { "id": 593356, "author_name": "cherring", "author_url": "", "post_date": "08/06/2019 13:41:37", "content": "Right, sorry I take that back haha I was a bit quick to write that without doing some reading.
\n\nYes it is definitely a leak and my half an hour of EDA definitely did not find this. Just found the pattern in the train set but not the test set :) So this definitely saved me a bit of time - in fact I just cancelled three training jobs :p
", "votes": null, "replies": [] } ] }, { "id": 593721, "author_name": "yiheng", "author_url": "", "post_date": "08/07/2019 02:49:25", "content": "How many scores may be increased? For your reference, the figure shows the changes of LB scores before and after the \"277 leaks\" is disclosed:\n---update 0807---\n
Related Code is in the attach file.
", "votes": null, "replies": [] }, { "id": 593925, "author_name": "giuliasavorgnan", "author_url": "", "post_date": "08/07/2019 09:47:32", "content": "\n
These two experiments have the same pattern of controls, but not only... also the treatments are in the same pattern, but the plates are rotated.
\n\nNow, there is an experiment in the test set that has the same pattern of controls... HUVEC-18. You can check what I said in this kernel
\n\nDoes HUVEC-18 also have the same pattern of treatments in some plate rotation?
", "votes": null, "replies": [ { "id": 593933, "author_name": "zaharch", "author_url": "", "post_date": "08/07/2019 10:00:09", "content": "Wow, this is a crazy finding, I hope the competition is not in danger. Recursion has to take a serious look at their random number generator.
", "votes": null, "replies": [] }, { "id": 593942, "author_name": "giuliasavorgnan", "author_url": "", "post_date": "08/07/2019 10:12:52", "content": "I figured it would be better to disclose any leak if aware of it, before this comp becomes a combinatory problem instead of a data science one...
", "votes": null, "replies": [] }, { "id": 593965, "author_name": "giuliasavorgnan", "author_url": "", "post_date": "08/07/2019 11:03:19", "content": "The good news is: this seems to be the only case in the train set.
", "votes": null, "replies": [] }, { "id": 594004, "author_name": "lukeeee", "author_url": "", "post_date": "08/07/2019 12:23:07", "content": "Since HUVEC-18 is in private test set (according to this post: https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/98075#latest-567690), we can't check if the same treatment placement holds. If it is indeed the case, then everyone should basically modify 1 final sub just to account for this possibility. Maybe organisers should remove this experiment from scoring?
", "votes": null, "replies": [] }, { "id": 594008, "author_name": "giuliasavorgnan", "author_url": "", "post_date": "08/07/2019 12:32:25", "content": "Good point, I had missed that post. Agreed on the removal.
", "votes": null, "replies": [] }, { "id": 602180, "author_name": "pflood314", "author_url": "", "post_date": "08/18/2019 17:56:10", "content": "FYI, It looks like this leakage has since been removed from the leaderboard scoring https://www.kaggle.com/c/recursion-cellular-image-classification/discussion/104011#latest-598578 Great find @giuliasavorgnan!
", "votes": null, "replies": [] } ] }, { "id": 594018, "author_name": "yiheng", "author_url": "", "post_date": "08/07/2019 12:54:26", "content": "Could we get a stage 2 testset without well&plate information?
", "votes": null, "replies": [] }, { "id": 595252, "author_name": "harshpatel1692", "author_url": "", "post_date": "08/09/2019 02:11:38", "content": "Can someone explain in layman terms how having 277 siRNA on a plate help increase model accuracy? Is it that you will have to train only for 277 siRNA per plate rather 1108?
", "votes": null, "replies": [ { "id": 595305, "author_name": "cherring", "author_url": "", "post_date": "08/09/2019 04:54:23", "content": "Yes, you can train models that have only 277 classes. However it can benefit models that have already been trained as you can artificially set 3*277 of the predicted results to 0 and then just look at the strongest prediction for the remaining 277 predictions.
", "votes": null, "replies": [] }, { "id": 596369, "author_name": "harshpatel1692", "author_url": "", "post_date": "08/10/2019 14:41:39", "content": "Thanks for the explanation!
", "votes": null, "replies": [] } ] }, { "id": 602240, "author_name": "jesucristo", "author_url": "", "post_date": "08/18/2019 20:03:31", "content": "Hi @bearnshaw
\n\nWhat subcellular structure does each channel represent? and is the channel meaning constant? Because I'm pretty sure you're using DAPI. For example:
\n\n\n1. Nucleus \n2. Nucleoplasm\n3. Nucleoli\n4. Nucleoli fibrillar center\n5. Nuclear speckles\n6. Cytoskeleton\n
In this case I can see clearly:
\n\n\n1. Nucleus\n3. Cytoskeleton\n6. Cytoskeleton\n
Thanks!
", "votes": null, "replies": [ { "id": 603270, "author_name": "harahelix", "author_url": "", "post_date": "08/20/2019 04:54:13", "content": "Just guessing, but according to the provided info,
\n\nit seems
\n\n![https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2779868%2Fbe1dd2074cf8cfd4d5caa2b67a5cbbab%2FScreenshot%20from%202019-08-18%2022-01-21.png?generation=1566158538027825&alt=media](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2779868%2Fbe1dd2074cf8cfd4d5caa2b67a5cbbab%2FScreenshot%20from%202019-08-18%2022-01-21.png?generation=1566158538027825&alt=media\"){kind=link}