For those who missed it, this post from @gwentea 8 days ago identifies a leak in the public test set: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/347890
\nTo get meaningful LB scores, you need to replace day 2 public test predictions with the corresponding target values from donor 32606. If you don't make this replacement, your LB scores will be misleading, since overfitted models will score higher. Note that this issue doesn't affect the private test set.
\nThis is an unfortunate data error and I'm surprised that it hasn't already been corrected, or at least announced. But then again, following the Foursquare disaster, I guess I shouldn't be surprised.
", "messages": [ { "id": 1924556, "postDate": "2022-09-03T06:49:32.003Z", "content": "For those who missed it, this post from @gwentea 8 days ago identifies a leak in the public test set: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/347890
\nTo get meaningful LB scores, you need to replace day 2 public test predictions with the corresponding target values from donor 32606. If you don't make this replacement, your LB scores will be misleading, since overfitted models will score higher. Note that this issue doesn't affect the private test set.
\nThis is an unfortunate data error and I'm surprised that it hasn't already been corrected, or at least announced. But then again, following the Foursquare disaster, I guess I shouldn't be surprised.
", "rawMarkdown": "For those who missed it, this post from @gwentea 8 days ago identifies a leak in the public test set: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/347890\n\nTo get meaningful LB scores, you need to replace day 2 public test predictions with the corresponding target values from donor 32606. If you don't make this replacement, your LB scores will be misleading, since overfitted models will score higher. Note that this issue doesn't affect the private test set.\n\nThis is an unfortunate data error and I'm surprised that it hasn't already been corrected, or at least announced. But then again, following the Foursquare disaster, I guess I shouldn't be surprised.", "votes": 94 }, { "id": 1926032, "postDate": "2022-09-04T14:01:39.347Z", "content": "We're aware of this issue and working with Kaggle to resolve it. It's actually not a leak of the test set, somehow the training data was duplicated for this time point when we created the training / public test split and over-wrote the data for day 2 public test.
\nNote, this does not affect the private LB or the private test.
\nThe current plan is to remove this time point from the public test and re-score all submissions, but I'll let you all know more when we finalize the solution with Kaggle.
", "rawMarkdown": "We're aware of this issue and working with Kaggle to resolve it. It's actually not a leak of the test set, somehow the training data was duplicated for this time point when we created the training / public test split and over-wrote the data for day 2 public test.\n\nNote, this does not affect the private LB or the private test.\n\nThe current plan is to remove this time point from the public test and re-score all submissions, but I'll let you all know more when we finalize the solution with Kaggle.", "votes": 25, "replies": [ { "id": 1927265, "postDate": "2022-09-05T13:42:14.947Z", "content": "Thank you for the prompt response. Great solution!
", "rawMarkdown": "Thank you for the prompt response. Great solution!" }, { "id": 1929726, "postDate": "2022-09-07T10:09:09.707Z", "content": "Are you aware that the scoring puts 10 times higher weight on the MULTI predictions than on the CITE, as 89% of the cells that are evaluated for the score are from that data set? (89% gene names match ENSG in evaluation_ids.csv). In case you go for re-scoring, it might be worth addressing this as well.
", "rawMarkdown": "Are you aware that the scoring puts 10 times higher weight on the MULTI predictions than on the CITE, as 89% of the cells that are evaluated for the score are from that data set? (89% gene names match ENSG in evaluation_ids.csv). In case you go for re-scoring, it might be worth addressing this as well." } ] }, { "id": 1924638, "postDate": "2022-09-03T09:00:41.203Z", "content": "With more and more such issues, I think soon kagglers will do a data duplication test as the first step in any competition. We expect the organizers to do these basic sanity checks before competition goes live. Otherwise just don't segregate train and test beforehand. In foursquare, it went even a step further where the leak happened with a code competition and participants didn't even get to know their scores if they were not in top 5. 😕
", "rawMarkdown": "With more and more such issues, I think soon kagglers will do a data duplication test as the first step in any competition. We expect the organizers to do these basic sanity checks before competition goes live. Otherwise just don't segregate train and test beforehand. In foursquare, it went even a step further where the leak happened with a code competition and participants didn't even get to know their scores if they were not in top 5. 😕", "votes": 11 }, { "id": 1997447, "postDate": "2022-10-20T22:18:11.553Z", "content": "nice work, ty
", "rawMarkdown": "nice work, ty" }, { "id": 1929103, "postDate": "2022-09-06T20:27:40.260Z", "content": "awesome work
", "rawMarkdown": "awesome work" }, { "id": 1928364, "postDate": "2022-09-06T12:41:13.297Z", "content": "Nice explanation..
", "rawMarkdown": "Nice explanation.." }, { "id": 1928354, "postDate": "2022-09-06T12:36:53.700Z", "content": "Great solution!
", "rawMarkdown": "Great solution!" }, { "id": 1928350, "postDate": "2022-09-06T12:35:39.210Z", "content": "Good work on leak public test set
", "rawMarkdown": "Good work on leak public test set" }, { "id": 1928347, "postDate": "2022-09-06T12:33:29.053Z", "content": "very nice work
", "rawMarkdown": "very nice work" }, { "id": 1928315, "postDate": "2022-09-06T12:14:57.223Z", "content": "awesome work
", "rawMarkdown": "awesome work" }, { "id": 1928307, "postDate": "2022-09-06T12:11:16.697Z", "content": "Good Job……….
", "rawMarkdown": "Good Job.........." }, { "id": 1927819, "postDate": "2022-09-06T02:25:07.380Z", "content": "great! Note, this does not affect the private LB or the private test.
", "rawMarkdown": "great! Note, this does not affect the private LB or the private test.\n\n" }, { "id": 2027475, "postDate": "2022-11-12T21:23:40.177Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1924615, "postDate": "2022-09-03T08:26:50.967Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2028658, "postDate": "2022-11-14T06:30:24.347Z", "content": "Thank you very much for sharing.
", "rawMarkdown": "Thank you very much for sharing." }, { "id": 1930577, "postDate": "2022-09-08T04:02:00.860Z", "content": "thank you sir
", "rawMarkdown": "thank you sir" }, { "id": 1927964, "postDate": "2022-09-06T05:59:16.870Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 1927890, "postDate": "2022-09-06T03:47:08.607Z", "content": "Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 1927417, "postDate": "2022-09-05T15:53:13.627Z", "content": "Thanks for the information.
", "rawMarkdown": "Thanks for the information.\n\n" }, { "id": 1926292, "postDate": "2022-09-04T18:00:48.603Z", "content": "Thanks for the information.
", "rawMarkdown": "Thanks for the information." } ], "comments": [ { "id": 1926032, "author_name": "Daniel Burkhardt", "author_url": "", "post_date": "2022-09-04T14:01:39.347000", "content": "We're aware of this issue and working with Kaggle to resolve it. It's actually not a leak of the test set, somehow the training data was duplicated for this time point when we created the training / public test split and over-wrote the data for day 2 public test.
\nNote, this does not affect the private LB or the private test.
\nThe current plan is to remove this time point from the public test and re-score all submissions, but I'll let you all know more when we finalize the solution with Kaggle.
", "votes": 25, "replies": [ { "id": 1927265, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-09-05T13:42:14.947000", "content": "Thank you for the prompt response. Great solution!
", "votes": 0, "replies": [] }, { "id": 1929726, "author_name": "Karsten", "author_url": "", "post_date": "2022-09-07T10:09:09.707000", "content": "Are you aware that the scoring puts 10 times higher weight on the MULTI predictions than on the CITE, as 89% of the cells that are evaluated for the score are from that data set? (89% gene names match ENSG in evaluation_ids.csv). In case you go for re-scoring, it might be worth addressing this as well.
", "votes": 0, "replies": [] } ] }, { "id": 1924638, "author_name": "tarick.morty", "author_url": "", "post_date": "2022-09-03T09:00:41.203000", "content": "With more and more such issues, I think soon kagglers will do a data duplication test as the first step in any competition. We expect the organizers to do these basic sanity checks before competition goes live. Otherwise just don't segregate train and test beforehand. In foursquare, it went even a step further where the leak happened with a code competition and participants didn't even get to know their scores if they were not in top 5. 😕
", "votes": 11, "replies": [] }, { "id": 1997447, "author_name": "GraC2H5", "author_url": "", "post_date": "2022-10-20T22:18:11.553000", "content": "nice work, ty
", "votes": 0, "replies": [] }, { "id": 1929103, "author_name": "Shahzaman Aftab", "author_url": "", "post_date": "2022-09-06T20:27:40.260000", "content": "awesome work
", "votes": 0, "replies": [] }, { "id": 1928364, "author_name": "Abu Usama Khan", "author_url": "", "post_date": "2022-09-06T12:41:13.297000", "content": "Nice explanation..
", "votes": 0, "replies": [] }, { "id": 1928354, "author_name": "sehrish mughal", "author_url": "", "post_date": "2022-09-06T12:36:53.700000", "content": "Great solution!
", "votes": 0, "replies": [] }, { "id": 1928350, "author_name": "Areeba Noor", "author_url": "", "post_date": "2022-09-06T12:35:39.210000", "content": "Good work on leak public test set
", "votes": 0, "replies": [] }, { "id": 1928347, "author_name": "Mustufa Jawaid", "author_url": "", "post_date": "2022-09-06T12:33:29.053000", "content": "very nice work
", "votes": 0, "replies": [] }, { "id": 1928315, "author_name": "Anas Khan", "author_url": "", "post_date": "2022-09-06T12:14:57.223000", "content": "awesome work
", "votes": 0, "replies": [] }, { "id": 1928307, "author_name": "Hira Mushtaq", "author_url": "", "post_date": "2022-09-06T12:11:16.697000", "content": "Good Job……….
", "votes": 0, "replies": [] }, { "id": 1927819, "author_name": "Melissa Aylas", "author_url": "", "post_date": "2022-09-06T02:25:07.380000", "content": "great! Note, this does not affect the private LB or the private test.
", "votes": 0, "replies": [] }, { "id": 2027475, "author_name": "", "author_url": "", "post_date": "2022-11-12T21:23:40.177000", "content": "", "votes": 0, "replies": [] }, { "id": 1924615, "author_name": "", "author_url": "", "post_date": "2022-09-03T08:26:50.967000", "content": "", "votes": 0, "replies": [] }, { "id": 2028658, "author_name": "", "author_url": "", "post_date": "2022-11-14T06:30:24.347000", "content": "Thank you very much for sharing.
", "votes": 0, "replies": [] }, { "id": 1930577, "author_name": "Bilal Suppal", "author_url": "", "post_date": "2022-09-08T04:02:00.860000", "content": "thank you sir
", "votes": 0, "replies": [] }, { "id": 1927964, "author_name": "HJ", "author_url": "", "post_date": "2022-09-06T05:59:16.870000", "content": "Thank you!
", "votes": 0, "replies": [] }, { "id": 1927890, "author_name": "Siddharth Sarai", "author_url": "", "post_date": "2022-09-06T03:47:08.607000", "content": "Thank you!
", "votes": 0, "replies": [] }, { "id": 1927417, "author_name": "Diptesh Banerjee", "author_url": "", "post_date": "2022-09-05T15:53:13.627000", "content": "Thanks for the information.
", "votes": 0, "replies": [] }, { "id": 1926292, "author_name": "Chidambaram Narayanan", "author_url": "", "post_date": "2022-09-04T18:00:48.603000", "content": "Thanks for the information.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1924556": "For those who missed it, this post from @gwentea 8 days ago identifies a leak in the public test set: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/347890\n\nTo get meaningful LB scores, you need to replace day 2 public test predictions with the corresponding target values from donor 32606. If you don't make this replacement, your LB scores will be misleading, since overfitted models will score higher. Note that this issue doesn't affect the private test set.\n\nThis is an unfortunate data error and I'm surprised that it hasn't already been corrected, or at least announced. But then again, following the Foursquare disaster, I guess I shouldn't be surprised.", "1926032": "We're aware of this issue and working with Kaggle to resolve it. It's actually not a leak of the test set, somehow the training data was duplicated for this time point when we created the training / public test split and over-wrote the data for day 2 public test.\n\nNote, this does not affect the private LB or the private test.\n\nThe current plan is to remove this time point from the public test and re-score all submissions, but I'll let you all know more when we finalize the solution with Kaggle.", "1924638": "With more and more such issues, I think soon kagglers will do a data duplication test as the first step in any competition. We expect the organizers to do these basic sanity checks before competition goes live. Otherwise just don't segregate train and test beforehand. In foursquare, it went even a step further where the leak happened with a code competition and participants didn't even get to know their scores if they were not in top 5. 😕", "1997447": "nice work, ty", "1929103": "awesome work", "1928364": "Nice explanation..", "1928354": "Great solution!", "1928350": "Good work on leak public test set", "1928347": "very nice work", "1928315": "awesome work", "1928307": "Good Job..........", "1927819": "great! Note, this does not affect the private LB or the private test.\n\n", "2027475": "", "1924615": "", "2028658": "Thank you very much for sharing.", "1930577": "thank you sir", "1927964": "Thank you!", "1927890": "Thank you!", "1927417": "Thanks for the information.\n\n", "1926292": "Thanks for the information." } }