Thank you for this competition! Even without significant prizes, it looks very interesting. It refers to a very hot topic. But why there is no ranking points?\nAfter all, ranking points will attract more participants and community will be able to get a really SOTA solution in terms of accuracy.
", "messages": [ { "id": 309269, "postDate": "2018-04-04T23:08:01.773Z", "content": "Thank you for this competition! Even without significant prizes, it looks very interesting. It refers to a very hot topic. But why there is no ranking points?\nAfter all, ranking points will attract more participants and community will be able to get a really SOTA solution in terms of accuracy.
", "rawMarkdown": "Thank you for this competition! Even without significant prizes, it looks very interesting. It refers to a very hot topic. But why there is no ranking points?\nAfter all, ranking points will attract more participants and community will be able to get a really SOTA solution in terms of accuracy.", "votes": 60 }, { "id": 309442, "postDate": "2018-04-05T10:41:50.977Z", "content": "@n01z3 Great question! We also think it's a really hot topic and we're excited to host our first Autonomous Driving challenge!
\n\nWe often do not offer ranking tiers/points for research competitions. The decision to grant points/medals is a judgement call we make based on our confidence in a problem withstanding the onslaught of the Kaggle community. Lack of points does not mean a problem is low quality (as you can see in this problem it is in fact very high quality!). Rather, a number of other small questions play into it:
\n\nWe don't like the idea of adding or removing points mid competition. Adding makes people who didn't join feel like they're missing out. Removing makes participants frustrated and ruins trust. Therefore, we try to make a reasonable assessment of a problem before launching.
", "rawMarkdown": "@n01z3 Great question! We also think it's a really hot topic and we're excited to host our first Autonomous Driving challenge!\n\nWe often do not offer ranking tiers/points for research competitions. The decision to grant points/medals is a judgement call we make based on our confidence in a problem withstanding the onslaught of the Kaggle community. Lack of points does not mean a problem is low quality (as you can see in this problem it is in fact very high quality!). Rather, a number of other small questions play into it:\n\n - Is the data public? Semi-public? Mostly private but related to some data that is public?\n - How much oversight did we have on the competition prep? Did we prepare the competition files from A-Z? How much time did we have to review the preparation code and process? Did the preparer pay attention to ways competitions can be exploited?\n - Is the problem atypical? Are we sure ML can even work on it? Is it totally outside our realm of expertise?\n - Is the size of the data borderline? Is the size of the submission file/complexity of the evaluation metric borderline?\n - Is there too much luck at play? (we usually give 1/2 points for March ML Mania for this reason)\n\nWe don't like the idea of adding or removing points mid competition. Adding makes people who didn't join feel like they're missing out. Removing makes participants frustrated and ruins trust. Therefore, we try to make a reasonable assessment of a problem before launching.", "votes": 1, "replies": [ { "id": 309467, "postDate": "2018-04-05T12:14:03.580Z", "content": "Thank you for the detailed answer!
\n\nThis problem is very exciting. I really would like to participate. But It's very complex and challenging in terms of hardware. So I would like to receive kind of reward for efforts. Unfortunately there is no sufficient prize. But this can be disregarded if you add rank points.
\n\nI have reason to believe that strong kagglers from russian community ODS.ai agree with me. If there were points, it would be possible to repeat the situation on the leaderboard at the previous competition You could notice [ods.ai] in team names.
\n\nIf you are afraid that restarting the competition will cause discontent, then right now the very beginning. And a couple of days of difference will not give an advantage. There is also relevant discussion.
", "rawMarkdown": "Thank you for the detailed answer!\n\nThis problem is very exciting. I really would like to participate. But It's very complex and challenging in terms of hardware. So I would like to receive kind of reward for efforts. Unfortunately there is no sufficient prize. But this can be disregarded if you add rank points.\n\nI have reason to believe that strong kagglers from russian community ODS.ai agree with me. If there were points, it would be possible to repeat the situation on the leaderboard at the [previous competition][1] You could notice [ods.ai] in team names.\n\nIf you are afraid that restarting the competition will cause discontent, then right now the very beginning. And a couple of days of difference will not give an advantage. There is also relevant [discussion][2].\n\n\n [1]: https://www.kaggle.com/c/sp-society-camera-model-identification/leaderboard\n [2]: https://www.kaggle.com/c/inaturalist-2018/discussion/50415", "votes": 18 }, { "id": 310223, "postDate": "2018-04-06T20:58:58.533Z", "content": "Hey @n01z3,
\n\nWe discussed this again with our strategy team today and unfortunately will not be able to include points or tiers to this competition.
\n\nWe're excited to partner with research groups to push forward the frontier of machine learning. Research competitions make use of Kaggle's platform and experience, but are largely organized by the research group's data science team. Therefore, Kaggle does not always get to perform the same depth of analysis on the dataset for a research competition as we do for many of our other featured competitions. This means we can't always ensure there are no gotchas, leakage, or surprise twists which can be present in many supervised ML competitions, which would not only tarnish the points issued to each participant, but could risk providing winning solutions that simply exploit the problem statement, dataset, evaluation metric, etc., as opposed to providing a meaningful benefit to the research organization responsible. In order to protect the integrity of the points the community has already earned, we sometimes have to disable points on other competitions as a \"just in case\" condition.
\n\nThis does not mean we believe the competition is risky, or the dataset is not of high quality, but it means that in order to publish competitions in a timely manner (this one, for example, to conclude in time for CVPR), we often have to make a judgement call as to whether or not to A) Host the competition with points and risk a mid-competition surprise, B) Host the competition without points, C) Delay the launch of a competition - which could risk it not launching at all, or D) Not host the competition at all. For this specific challenge, we made the judgement call that hosting the competition without points was the appropriate decision having weighed all factors.
\n\nWe field scores of requests for research challenges and we have to turn many of them down because we don't have the resources, (wo)manpower, or because we do not think they would provide value to the Kaggle community, as well as a number of other factors that may result in us being unable to host the competition. Many of these organizations have hosted the competitions on their own, or on other platforms, but know the Kaggle community is the best of the best and wants to put their problem in front of you. We're glad we have the chance to host these problems, and hopefully their problem hosted on Kaggle without points still beats their problem hosted elsewhere without points :).
\n\nWe certainly understand that you have to weigh whether or not points are included as to whether or not you'll enter a competition and we hope that you still want to enter. We also hope to be able to provide more research competitions (hopefully with points, too!) in the future.
", "rawMarkdown": "Hey @n01z3,\n\nWe discussed this again with our strategy team today and unfortunately will not be able to include points or tiers to this competition. \n\nWe're excited to partner with research groups to push forward the frontier of machine learning. Research competitions make use of Kaggle's platform and experience, but are largely organized by the research group's data science team. Therefore, Kaggle does not always get to perform the same depth of analysis on the dataset for a research competition as we do for many of our other featured competitions. This means we can't always ensure there are no gotchas, leakage, or surprise twists which can be present in many supervised ML competitions, which would not only tarnish the points issued to each participant, but could risk providing winning solutions that simply exploit the problem statement, dataset, evaluation metric, etc., as opposed to providing a meaningful benefit to the research organization responsible. In order to protect the integrity of the points the community has already earned, we sometimes have to disable points on other competitions as a \"just in case\" condition.\n\nThis does not mean we believe the competition is risky, or the dataset is not of high quality, but it means that in order to publish competitions in a timely manner (this one, for example, to conclude in time for CVPR), we often have to make a judgement call as to whether or not to A) Host the competition with points and risk a mid-competition surprise, B) Host the competition without points, C) Delay the launch of a competition - which could risk it not launching at all, or D) Not host the competition at all. For this specific challenge, we made the judgement call that hosting the competition without points was the appropriate decision having weighed all factors.\n\nWe field scores of requests for research challenges and we have to turn many of them down because we don't have the resources, (wo)manpower, or because we do not think they would provide value to the Kaggle community, as well as a number of other factors that may result in us being unable to host the competition. Many of these organizations have hosted the competitions on their own, or on other platforms, but know the Kaggle community is the best of the best and wants to put their problem in front of you. We're glad we have the chance to host these problems, and hopefully their problem hosted on Kaggle without points still beats their problem hosted elsewhere without points :).\n\nWe certainly understand that you have to weigh whether or not points are included as to whether or not you'll enter a competition and we hope that you still want to enter. We also hope to be able to provide more research competitions (hopefully with points, too!) in the future.", "votes": 6 } ] }, { "id": 315203, "postDate": "2018-04-16T21:27:45.220Z", "content": "isn't this changed now
", "rawMarkdown": "isn't this changed now", "replies": [ { "id": 315850, "postDate": "2018-04-17T17:03:36.537Z", "content": "nope
", "rawMarkdown": "nope" } ] }, { "id": 310330, "postDate": "2018-04-07T06:08:09.343Z", "content": "I am a novice, I think this competition will attract more participants and community
", "rawMarkdown": "I am a novice, I think this competition will attract more participants and community", "replies": [ { "id": 311027, "postDate": "2018-04-09T08:12:51.047Z", "content": "so far only 8 teams...
", "rawMarkdown": "so far only 8 teams... ", "votes": 1 }, { "id": 315848, "postDate": "2018-04-17T17:03:15.783Z", "content": "21 teams...
", "rawMarkdown": "21 teams... " } ] }, { "id": 309352, "postDate": "2018-04-05T05:30:23.440Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 309442, "author_name": "Addison Howard", "author_url": "", "post_date": "2018-04-05T10:41:50.977000", "content": "@n01z3 Great question! We also think it's a really hot topic and we're excited to host our first Autonomous Driving challenge!
\n\nWe often do not offer ranking tiers/points for research competitions. The decision to grant points/medals is a judgement call we make based on our confidence in a problem withstanding the onslaught of the Kaggle community. Lack of points does not mean a problem is low quality (as you can see in this problem it is in fact very high quality!). Rather, a number of other small questions play into it:
\n\nWe don't like the idea of adding or removing points mid competition. Adding makes people who didn't join feel like they're missing out. Removing makes participants frustrated and ruins trust. Therefore, we try to make a reasonable assessment of a problem before launching.
", "votes": 1, "replies": [ { "id": 309467, "author_name": "n01z3", "author_url": "", "post_date": "2018-04-05T12:14:03.580000", "content": "Thank you for the detailed answer!
\n\nThis problem is very exciting. I really would like to participate. But It's very complex and challenging in terms of hardware. So I would like to receive kind of reward for efforts. Unfortunately there is no sufficient prize. But this can be disregarded if you add rank points.
\n\nI have reason to believe that strong kagglers from russian community ODS.ai agree with me. If there were points, it would be possible to repeat the situation on the leaderboard at the previous competition You could notice [ods.ai] in team names.
\n\nIf you are afraid that restarting the competition will cause discontent, then right now the very beginning. And a couple of days of difference will not give an advantage. There is also relevant discussion.
", "votes": 18, "replies": [] }, { "id": 310223, "author_name": "Addison Howard", "author_url": "", "post_date": "2018-04-06T20:58:58.533000", "content": "Hey @n01z3,
\n\nWe discussed this again with our strategy team today and unfortunately will not be able to include points or tiers to this competition.
\n\nWe're excited to partner with research groups to push forward the frontier of machine learning. Research competitions make use of Kaggle's platform and experience, but are largely organized by the research group's data science team. Therefore, Kaggle does not always get to perform the same depth of analysis on the dataset for a research competition as we do for many of our other featured competitions. This means we can't always ensure there are no gotchas, leakage, or surprise twists which can be present in many supervised ML competitions, which would not only tarnish the points issued to each participant, but could risk providing winning solutions that simply exploit the problem statement, dataset, evaluation metric, etc., as opposed to providing a meaningful benefit to the research organization responsible. In order to protect the integrity of the points the community has already earned, we sometimes have to disable points on other competitions as a \"just in case\" condition.
\n\nThis does not mean we believe the competition is risky, or the dataset is not of high quality, but it means that in order to publish competitions in a timely manner (this one, for example, to conclude in time for CVPR), we often have to make a judgement call as to whether or not to A) Host the competition with points and risk a mid-competition surprise, B) Host the competition without points, C) Delay the launch of a competition - which could risk it not launching at all, or D) Not host the competition at all. For this specific challenge, we made the judgement call that hosting the competition without points was the appropriate decision having weighed all factors.
\n\nWe field scores of requests for research challenges and we have to turn many of them down because we don't have the resources, (wo)manpower, or because we do not think they would provide value to the Kaggle community, as well as a number of other factors that may result in us being unable to host the competition. Many of these organizations have hosted the competitions on their own, or on other platforms, but know the Kaggle community is the best of the best and wants to put their problem in front of you. We're glad we have the chance to host these problems, and hopefully their problem hosted on Kaggle without points still beats their problem hosted elsewhere without points :).
\n\nWe certainly understand that you have to weigh whether or not points are included as to whether or not you'll enter a competition and we hope that you still want to enter. We also hope to be able to provide more research competitions (hopefully with points, too!) in the future.
", "votes": 6, "replies": [] } ] }, { "id": 315203, "author_name": "", "author_url": "", "post_date": "2018-04-16T21:27:45.220000", "content": "isn't this changed now
", "votes": 0, "replies": [ { "id": 315850, "author_name": "Vadim Borisov", "author_url": "", "post_date": "2018-04-17T17:03:36.537000", "content": "nope
", "votes": 0, "replies": [] } ] }, { "id": 310330, "author_name": "BaotingZZZ", "author_url": "", "post_date": "2018-04-07T06:08:09.343000", "content": "I am a novice, I think this competition will attract more participants and community
", "votes": 0, "replies": [ { "id": 311027, "author_name": "Vadim Borisov", "author_url": "", "post_date": "2018-04-09T08:12:51.047000", "content": "so far only 8 teams...
", "votes": 1, "replies": [] }, { "id": 315848, "author_name": "Vadim Borisov", "author_url": "", "post_date": "2018-04-17T17:03:15.783000", "content": "21 teams...
", "votes": 0, "replies": [] } ] }, { "id": 309352, "author_name": "", "author_url": "", "post_date": "2018-04-05T05:30:23.440000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "309269": "Thank you for this competition! Even without significant prizes, it looks very interesting. It refers to a very hot topic. But why there is no ranking points?\nAfter all, ranking points will attract more participants and community will be able to get a really SOTA solution in terms of accuracy.", "309442": "@n01z3 Great question! We also think it's a really hot topic and we're excited to host our first Autonomous Driving challenge!\n\nWe often do not offer ranking tiers/points for research competitions. The decision to grant points/medals is a judgement call we make based on our confidence in a problem withstanding the onslaught of the Kaggle community. Lack of points does not mean a problem is low quality (as you can see in this problem it is in fact very high quality!). Rather, a number of other small questions play into it:\n\n - Is the data public? Semi-public? Mostly private but related to some data that is public?\n - How much oversight did we have on the competition prep? Did we prepare the competition files from A-Z? How much time did we have to review the preparation code and process? Did the preparer pay attention to ways competitions can be exploited?\n - Is the problem atypical? Are we sure ML can even work on it? Is it totally outside our realm of expertise?\n - Is the size of the data borderline? Is the size of the submission file/complexity of the evaluation metric borderline?\n - Is there too much luck at play? (we usually give 1/2 points for March ML Mania for this reason)\n\nWe don't like the idea of adding or removing points mid competition. Adding makes people who didn't join feel like they're missing out. Removing makes participants frustrated and ruins trust. Therefore, we try to make a reasonable assessment of a problem before launching.", "315203": "isn't this changed now", "310330": "I am a novice, I think this competition will attract more participants and community", "309352": "" } }