To Kaggle team and Organisers,
\nIn the Data section, it said that there are some issues with the data and I guess there might be some leakage(maybe).
\nI know some mistakes can happen but why take any chance, I think making the test data hidden and making it a code only competition would be better.
To Kaggle team and Organisers,
\nIn the Data section, it said that there are some issues with the data and I guess there might be some leakage(maybe).
\nI know some mistakes can happen but why take any chance, I think making the test data hidden and making it a code only competition would be better.
Great question! Theoretically, we could make everything a Code competition, since that better simulates \"real life\" machine learning where you don't have access to the future unseen test data. But Code competitions are an extra friction point for some people who prefer to have the entire workflow on their own system, so we generally only make a Code competition when it really is necessary for technical reasons.
\nWhy not this competition because of the (possible) data issues? The main reason is that the possible data issues wouldn't be solved with a Code competition. The most likely issue is an individual who is only photographed during a single encounter with similar images. We've tried to be smart with how theses are split when we know they are from the same encounter, but for some of the species we did not have photograph dates, so there may be unintentional, e.g., background leakage, that a classifier learns but that is not solved by having the leaky image \"hidden\" from the competitors.
\nAnother consideration is that it actually helps marine mammal researchers when the Kaggle community discovers and reports weaknesses in the data setup. The scrutiny helps improve the dataset for future iterations.
", "rawMarkdown": "Great question! Theoretically, we could make everything a Code competition, since that better simulates \"real life\" machine learning where you don't have access to the future unseen test data. But Code competitions are an extra friction point for some people who prefer to have the entire workflow on their own system, so we generally only make a Code competition when it really is necessary for technical reasons.\n\nWhy not this competition because of the (possible) data issues? The main reason is that the possible data issues wouldn't be solved with a Code competition. The most likely issue is an individual who is only photographed during a single encounter with similar images. We've tried to be smart with how theses are split *when we know they are from the same encounter*, but for some of the species we did not have photograph dates, so there may be unintentional, e.g., background leakage, that a classifier learns but that is not solved by having the leaky image \"hidden\" from the competitors.\n\nAnother consideration is that it actually helps marine mammal researchers when the Kaggle community discovers and reports weaknesses in the data setup. The scrutiny helps improve the dataset for future iterations.", "votes": 14, "replies": [ { "id": 1675185, "postDate": "2022-02-04T04:58:15.407Z", "content": "Thanks for explaining it very thoughtfully,
\nbackground leakage, in my initial model I can see this, images with the almost the same background are clustered together irrespective of their species. Anyway that's something I have to solve
\nThanks for the reply
Yes, it is a good сomment, but I DON'T WANT TO WAIT FOR N HOURS! 😄 just kidding…
", "rawMarkdown": "Yes, it is a good сomment, but I DON'T WANT TO WAIT FOR N HOURS! 😄 just kidding...", "votes": 4 }, { "id": 1673107, "postDate": "2022-02-02T14:12:50.913Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1674595, "author_name": "inversion", "author_url": "", "post_date": "2022-02-03T16:11:06.923000", "content": "Great question! Theoretically, we could make everything a Code competition, since that better simulates \"real life\" machine learning where you don't have access to the future unseen test data. But Code competitions are an extra friction point for some people who prefer to have the entire workflow on their own system, so we generally only make a Code competition when it really is necessary for technical reasons.
\nWhy not this competition because of the (possible) data issues? The main reason is that the possible data issues wouldn't be solved with a Code competition. The most likely issue is an individual who is only photographed during a single encounter with similar images. We've tried to be smart with how theses are split when we know they are from the same encounter, but for some of the species we did not have photograph dates, so there may be unintentional, e.g., background leakage, that a classifier learns but that is not solved by having the leaky image \"hidden\" from the competitors.
\nAnother consideration is that it actually helps marine mammal researchers when the Kaggle community discovers and reports weaknesses in the data setup. The scrutiny helps improve the dataset for future iterations.
", "votes": 14, "replies": [ { "id": 1675185, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2022-02-04T04:58:15.407000", "content": "Thanks for explaining it very thoughtfully,
\nbackground leakage, in my initial model I can see this, images with the almost the same background are clustered together irrespective of their species. Anyway that's something I have to solve
\nThanks for the reply
Yes, it is a good сomment, but I DON'T WANT TO WAIT FOR N HOURS! 😄 just kidding…
", "votes": 4, "replies": [] }, { "id": 1673107, "author_name": "", "author_url": "", "post_date": "2022-02-02T14:12:50.913000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1673051": "To Kaggle team and Organisers,\n\nIn the Data section, it said that there are some issues with the data and I guess there might be some leakage(maybe).\nI know some mistakes can happen but why take any chance, I think making the test data hidden and making it a code only competition would be better.", "1674595": "Great question! Theoretically, we could make everything a Code competition, since that better simulates \"real life\" machine learning where you don't have access to the future unseen test data. But Code competitions are an extra friction point for some people who prefer to have the entire workflow on their own system, so we generally only make a Code competition when it really is necessary for technical reasons.\n\nWhy not this competition because of the (possible) data issues? The main reason is that the possible data issues wouldn't be solved with a Code competition. The most likely issue is an individual who is only photographed during a single encounter with similar images. We've tried to be smart with how theses are split *when we know they are from the same encounter*, but for some of the species we did not have photograph dates, so there may be unintentional, e.g., background leakage, that a classifier learns but that is not solved by having the leaky image \"hidden\" from the competitors.\n\nAnother consideration is that it actually helps marine mammal researchers when the Kaggle community discovers and reports weaknesses in the data setup. The scrutiny helps improve the dataset for future iterations.", "1673335": "Yes, it is a good сomment, but I DON'T WANT TO WAIT FOR N HOURS! 😄 just kidding...", "1673107": "" } }