As a contestant who spend a lot of time on this competition, could I know my exactly rank on the private leaderboard (filter out those who broke the rules and disqualified) after the competition?
\n\nI think lots of contestants want to improve their machine learning skills through the competitions. However, since there is no strict limitation on the training data, the problem of overlapping of training and testing data is really worse, and it results in lower discrimination of the testing score and leaderboard rank. The problem makes the contestant hard to compare their algorithms to the others', and lose the opportunities to learning from the other people
\n\nAs a matter of fact, I want to know whether Kaggle would do some efforts to filter out the disqualified final submission, and publish the \"clean\" final rank and score on the private leaderboard after the contest as a compensation to the hardworking contestants?
", "messages": [ { "id": "841848", "postDate": "05/11/2020 04:09:22", "content": "As a contestant who spend a lot of time on this competition, could I know my exactly rank on the private leaderboard (filter out those who broke the rules and disqualified) after the competition?
\n\nI think lots of contestants want to improve their machine learning skills through the competitions. However, since there is no strict limitation on the training data, the problem of overlapping of training and testing data is really worse, and it results in lower discrimination of the testing score and leaderboard rank. The problem makes the contestant hard to compare their algorithms to the others', and lose the opportunities to learning from the other people
\n\nAs a matter of fact, I want to know whether Kaggle would do some efforts to filter out the disqualified final submission, and publish the \"clean\" final rank and score on the private leaderboard after the contest as a compensation to the hardworking contestants?
", "rawMarkdown": "As a contestant who spend a lot of time on this competition, could I know my exactly rank on the private leaderboard (filter out those who broke the rules and disqualified) after the competition?\n\nI think lots of contestants want to improve their machine learning skills through the competitions. However, since there is no strict limitation on the training data, the problem of overlapping of training and testing data is really worse, and it results in lower discrimination of the testing score and leaderboard rank. The problem makes the contestant hard to compare their algorithms to the others', and lose the opportunities to learning from the other people\n\nAs a matter of fact, I want to know whether Kaggle would do some efforts to filter out the disqualified final submission, and publish the \"clean\" final rank and score on the private leaderboard after the contest as a compensation to the hardworking contestants?", "votes": null }, { "id": "842718", "postDate": "05/11/2020 15:41:49", "content": "I believe Kaggle will do the check only for the winners.\nAre you sure you did not violate the rules? I guess you have used pretrained imagenet weights to train the model. As far as I know Imagenet contains images which are in the test set.
", "rawMarkdown": "I believe Kaggle will do the check only for the winners.\nAre you sure you did not violate the rules? I guess you have used pretrained imagenet weights to train the model. As far as I know Imagenet contains images which are in the test set.", "votes": null }, { "id": "842731", "postDate": "05/11/2020 15:57:16", "content": "From this thread:
\n\n\"Using publicly-available pre-trained models is allowed, including those with components of the test set underlying the pre-trained models. To be super clear, this means you're not permitted to be \"pre-training\" models using the test set and then calling that \"pre-trained,\" but you can use the host of ImageNet or other canned publicly-available pre-trained models mentioned in preceding posts.\"
", "rawMarkdown": "From this [thread](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329):\n\n\"Using publicly-available pre-trained models is allowed, including those with components of the test set underlying the pre-trained models. To be super clear, this means you're not permitted to be \"pre-training\" models using the test set and then calling that \"pre-trained,\" but you can use the host of ImageNet or other canned publicly-available pre-trained models mentioned in preceding posts.\"", "votes": null }, { "id": "842816", "postDate": "05/11/2020 16:32:17", "content": "Is it stated in the rules somewhere? I did not see. It obviously contradicts with the previous statements(no using of test data in training). If it is an exception - why it is not added to the rules, why we get this comment a few days before the competition end?
", "rawMarkdown": "Is it stated in the rules somewhere? I did not see. It obviously contradicts with the previous statements(no using of test data in training). If it is an exception - why it is not added to the rules, why we get this comment a few days before the competition end?", "votes": null }, { "id": "842820", "postDate": "05/11/2020 16:33:25", "content": "Compare to the training with testing data, using imagenet's pretrained weight has very small impact to the score.
", "rawMarkdown": "Compare to the training with testing data, using imagenet's pretrained weight has very small impact to the score.", "votes": null }, { "id": "842851", "postDate": "05/11/2020 16:57:36", "content": "We do not have any notion about \"how much of test data you can have so you have small impact\" :)
\n\nLet me share with you my view please. It is subjective opinion, it can not be written in official tone. I think these rules are intended to disallow people intentional usage of test data to improve their models. Since we were given and shared the same datasets(imagenet, oxford etc.) - we all are in equal conditions. If someone intentionally picked images similar to those in external datasets and trained exactly on them - it is unfair. And I think 7K test images can be easily processed in this way in two months, so the guys could reach not 0.98, but 0.99 or even higher. I think so because I personally did not pick test images to get 0.984.
\n\nAlso, keep in mind - this is playground competition. I found many inconsistencies in classes - for example, there is heavy mislabeling of mallow and hibiscus in the given training data. The species are wrong, the way they are introduced is wrong as well. This is all about training and getting acquaintance with TPUs, I would hardly consider this competition as benchmark for some serious experiments.
", "rawMarkdown": "We do not have any notion about \"how much of test data you can have so you have small impact\" :)\n\nLet me share with you my view please. It is subjective opinion, it can not be written in official tone. I think these rules are intended to disallow people **intentional** usage of test data to improve their models. Since we were given and shared the same datasets(imagenet, oxford etc.) - we all are in equal conditions. If someone intentionally picked images similar to those in external datasets and trained exactly on them - it is unfair. And I think 7K test images can be easily processed in this way in two months, so the guys could reach not 0.98, but 0.99 or even higher. I think so because I personally did not pick test images to get 0.984. \n\nAlso, keep in mind - this is playground competition. I found many inconsistencies in classes - for example, there is heavy mislabeling of mallow and hibiscus in the given training data. The species are wrong, the way they are introduced is wrong as well. This is all about training and getting acquaintance with TPUs, I would hardly consider this competition as benchmark for some serious experiments.", "votes": null } ], "comments": [ { "id": 842718, "author_name": "vpaslay", "author_url": "", "post_date": "05/11/2020 15:41:49", "content": "I believe Kaggle will do the check only for the winners.\nAre you sure you did not violate the rules? I guess you have used pretrained imagenet weights to train the model. As far as I know Imagenet contains images which are in the test set.
", "votes": null, "replies": [ { "id": 842731, "author_name": "awanderingsoul", "author_url": "", "post_date": "05/11/2020 15:57:16", "content": "From this thread:
\n\n\"Using publicly-available pre-trained models is allowed, including those with components of the test set underlying the pre-trained models. To be super clear, this means you're not permitted to be \"pre-training\" models using the test set and then calling that \"pre-trained,\" but you can use the host of ImageNet or other canned publicly-available pre-trained models mentioned in preceding posts.\"
", "votes": null, "replies": [] }, { "id": 842816, "author_name": "vpaslay", "author_url": "", "post_date": "05/11/2020 16:32:17", "content": "Is it stated in the rules somewhere? I did not see. It obviously contradicts with the previous statements(no using of test data in training). If it is an exception - why it is not added to the rules, why we get this comment a few days before the competition end?
", "votes": null, "replies": [] }, { "id": 842820, "author_name": "hardworkingkaggler", "author_url": "", "post_date": "05/11/2020 16:33:25", "content": "Compare to the training with testing data, using imagenet's pretrained weight has very small impact to the score.
", "votes": null, "replies": [] }, { "id": 842851, "author_name": "vpaslay", "author_url": "", "post_date": "05/11/2020 16:57:36", "content": "We do not have any notion about \"how much of test data you can have so you have small impact\" :)
\n\nLet me share with you my view please. It is subjective opinion, it can not be written in official tone. I think these rules are intended to disallow people intentional usage of test data to improve their models. Since we were given and shared the same datasets(imagenet, oxford etc.) - we all are in equal conditions. If someone intentionally picked images similar to those in external datasets and trained exactly on them - it is unfair. And I think 7K test images can be easily processed in this way in two months, so the guys could reach not 0.98, but 0.99 or even higher. I think so because I personally did not pick test images to get 0.984.
\n\nAlso, keep in mind - this is playground competition. I found many inconsistencies in classes - for example, there is heavy mislabeling of mallow and hibiscus in the given training data. The species are wrong, the way they are introduced is wrong as well. This is all about training and getting acquaintance with TPUs, I would hardly consider this competition as benchmark for some serious experiments.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "841848": "As a contestant who spend a lot of time on this competition, could I know my exactly rank on the private leaderboard (filter out those who broke the rules and disqualified) after the competition?\n\nI think lots of contestants want to improve their machine learning skills through the competitions. However, since there is no strict limitation on the training data, the problem of overlapping of training and testing data is really worse, and it results in lower discrimination of the testing score and leaderboard rank. The problem makes the contestant hard to compare their algorithms to the others', and lose the opportunities to learning from the other people\n\nAs a matter of fact, I want to know whether Kaggle would do some efforts to filter out the disqualified final submission, and publish the \"clean\" final rank and score on the private leaderboard after the contest as a compensation to the hardworking contestants?", "842718": "I believe Kaggle will do the check only for the winners.\nAre you sure you did not violate the rules? I guess you have used pretrained imagenet weights to train the model. As far as I know Imagenet contains images which are in the test set.", "842731": "From this [thread](https://www.kaggle.com/c/flower-classification-with-tpus/discussion/148329):\n\n\"Using publicly-available pre-trained models is allowed, including those with components of the test set underlying the pre-trained models. To be super clear, this means you're not permitted to be \"pre-training\" models using the test set and then calling that \"pre-trained,\" but you can use the host of ImageNet or other canned publicly-available pre-trained models mentioned in preceding posts.\"", "842816": "Is it stated in the rules somewhere? I did not see. It obviously contradicts with the previous statements(no using of test data in training). If it is an exception - why it is not added to the rules, why we get this comment a few days before the competition end?", "842820": "Compare to the training with testing data, using imagenet's pretrained weight has very small impact to the score.", "842851": "We do not have any notion about \"how much of test data you can have so you have small impact\" :)\n\nLet me share with you my view please. It is subjective opinion, it can not be written in official tone. I think these rules are intended to disallow people **intentional** usage of test data to improve their models. Since we were given and shared the same datasets(imagenet, oxford etc.) - we all are in equal conditions. If someone intentionally picked images similar to those in external datasets and trained exactly on them - it is unfair. And I think 7K test images can be easily processed in this way in two months, so the guys could reach not 0.98, but 0.99 or even higher. I think so because I personally did not pick test images to get 0.984. \n\nAlso, keep in mind - this is playground competition. I found many inconsistencies in classes - for example, there is heavy mislabeling of mallow and hibiscus in the given training data. The species are wrong, the way they are introduced is wrong as well. This is all about training and getting acquaintance with TPUs, I would hardly consider this competition as benchmark for some serious experiments." }, "source": "meta" }