{ "id": 5832, "title": "Using Live Wind Information", "url": "/competitions/flight2-milestone/discussion/5832", "author_name": "", "post_date": "2013-09-20T23:13:55.713Z", "votes": 6, "comment_count": 12, "views": 5113, "content": "

Hi Admins,

\n

As the milestone prize deadline is approaching, I'm thinking more and more about the wind data rules.

\n

The live wind data used for evaluating public leaderboard submissions can be extracted from the files available here, however contestants are not allowed to use this data in their solution (see this post).

\n

I have not used live wind information in any of my submissions, but my experiments on the OneDay data set show that using it could improve the score by >10$.

\n

Suppose that the candidate for the milestone prize submits his/her code to Kaggle for validation. It's just impossible to check if the code uses live wind information or not. For example, one could encode a piece of live wind information in an indirect way, e.g. as constants in the code. How can Kaggle detect if some constants were optimized for live wind?

\n

I think that the rules should be modified to eliminate this nasty and serious issue.
Here are two suggestions:

\n
    \n
  1. Contestants should be allowed to use live wind information in their milestone prize solution.
    2. \n
  2. The milestone prize evaluation should use something else instead of the publicly available live wind data, e.g. randomly perturb the wind vectors included in TestWeatherData-Partial.zip.
    3. \n
\n

Gabor

", "messages": [ { "id": "31282", "postDate": "09/20/2013 23:13:55", "content": "

Hi Admins,

\n

As the milestone prize deadline is approaching, I'm thinking more and more about the wind data rules.

\n

The live wind data used for evaluating public leaderboard submissions can be extracted from the files available here, however contestants are not allowed to use this data in their solution (see this post).

\n

I have not used live wind information in any of my submissions, but my experiments on the OneDay data set show that using it could improve the score by >10$.

\n

Suppose that the candidate for the milestone prize submits his/her code to Kaggle for validation. It's just impossible to check if the code uses live wind information or not. For example, one could encode a piece of live wind information in an indirect way, e.g. as constants in the code. How can Kaggle detect if some constants were optimized for live wind?

\n

I think that the rules should be modified to eliminate this nasty and serious issue.
Here are two suggestions:

\n
    \n
  1. Contestants should be allowed to use live wind information in their milestone prize solution.
    2. \n
  2. The milestone prize evaluation should use something else instead of the publicly available live wind data, e.g. randomly perturb the wind vectors included in TestWeatherData-Partial.zip.
    3. \n
\n

Gabor

", "rawMarkdown": "", "votes": null }, { "id": "31289", "postDate": "09/21/2013 01:28:41", "content": "

This a very important issue! I am waiting for answer too...

", "rawMarkdown": "", "votes": null }, { "id": "31297", "postDate": "09/21/2013 07:41:04", "content": "

[quote=Gábor Takács;31282]

\n

I think that the rules should be modified to eliminate this nasty and serious issue.
Here are two suggestions:

\n
    \n
  1. Contestants should be allowed to use live wind information in their milestone prize solution.
    2. \n
  2. The milestone prize evaluation should use something else instead of the publicly available live wind data, e.g. randomly perturb the wind vectors included in TestWeatherData-Partial.zip.
    3. \n
\n

[/quote]

\n

Using all data, using cutoff data and using random data it's three different tasks with three different solutions. I don't see the way to keep task and eliminate this issue except two phase validation as like final prize use.

", "rawMarkdown": "", "votes": null }, { "id": "31386", "postDate": "09/23/2013 09:05:22", "content": "

[quote=Gábor Takács;31282]

\n

I think that the rules should be modified to eliminate this nasty and serious issue.

\n

[/quote]

\n

I concur.

\n

[quote]

\n
    \n
  1. Contestants should be allowed to use live wind information in their milestone prize solution.
    2. \n
  2. The milestone prize evaluation should use something else instead of the publicly available live wind data, e.g. randomly perturb the wind vectors included in TestWeatherData-Partial.zip.
    3. \n
\n

[/quote]

\n

At first glance, #2 looks like an excellent solution, but it does not eliminate the edge that a contestant could gain by using live data. If the admins were to randomly perturb the wind vectors and re-score the leaderboard, one would expect the ranking to stay the same. Sure, the gaps between users and nonusers of live data would decrease. Depending on the drops in score, you can probably come up with some heuristic to detect usage of live data.

\n

If the admins go with #1, I would request that they provide the complete weather files instead of having the contestants crawl the data and reformat.

", "rawMarkdown": "", "votes": null }, { "id": "31387", "postDate": "09/23/2013 09:11:44", "content": "

[quote=Anil Thomas;31386]

\n

If the admins go with #1, I would request that they provide the complete weather files instead of having the contestants crawl the data and reformat.

\n

[/quote]

\n

yep - I guess we're all waiting for a clarification on this!

", "rawMarkdown": "", "votes": null }, { "id": "31389", "postDate": "09/23/2013 09:58:07", "content": "

[quote=Anil Thomas;31386]

\n

At first glance, #2 sounds like an excellent solution, but it does not eliminate the edge that a contestant could gain by using live data. If the admins were to randomly perturb the wind vectors and re-score the leaderboard, one would expect the ranking to stay the same. Sure, the gaps between users and nonusers of live data would decrease. Depending on the drops in score, you can probably come up with some heuristic to detect usage of live data.

\n

If the admins go with #1, I would request that they provide the complete weather files instead of having the contestants crawl the data and reformat.

\n

[/quote]

\n

I didn't mean perturbing the live wind vectors but creating the wind vectors of future hours by randomly perturbing the wind vectors of the initial hour.

\n

Anyway, I see no really good solution for the situation. It is also problematic to change rules in the last minute. Also, I don't know if it's really possible to gain significant advantage from live wind information encoded into a few parameters. I observed that the direct usage of live wind can help a lot, therefore there is a danger of undetectable cheating. Another point: Everybody is using live wind information to some small extent, by using public leaderboard scores as a feedback.

\n

No matter what will be the decision, Kaggle should take this issue seriously.

\n

Gabor

", "rawMarkdown": "", "votes": null }, { "id": "31408", "postDate": "09/23/2013 15:29:30", "content": "

[quote=Gábor Takács;31389]

\n

I didn't mean perturbing the live wind vectors but creating the wind vectors of future hours by randomly perturbing the wind vectors of the initial hour.

\n

[/quote]

\n

Ah... that would work.

", "rawMarkdown": "", "votes": null }, { "id": "31412", "postDate": "09/23/2013 16:08:12", "content": "

I don't know if that would work as well as intended. I assume that someone is trying to create some forecast from the "legal" data we've been given. Creating a random perturbation would probably mean disrupting the work of such person.

", "rawMarkdown": "", "votes": null }, { "id": "31501", "postDate": "09/24/2013 14:49:37", "content": "

Gabor and all: thank you for raising this issue and the thoughtful input. As was pointed out, it is too late to change the rules for the Milestone prize. So only the wind information up to the cutoff time for each day may be used. We will look very closely at the preliminary winner's solution for the potential use of live wind information, which would be grounds for disqualification.

\n

As for the rest of the competition, remember that this is a 2-phase competition that requires your models to be locked down at the end of the training period and before the release of the final evaluation set. Therefore you will not be able to incorporate live wind information on the final evaluation data. We are looking into providing the wind data to you in an easier-to-use format that does not tempt the use of future information.

\n

The limitation of randomly perturbing the wind data and using that in the evaluation is that moves the wind data away from a realistic scenario.

\n

Please continue to share your thoughts on the competition design on the forums.

", "rawMarkdown": "", "votes": null }, { "id": "31514", "postDate": "09/24/2013 19:30:10", "content": "

[quote=joycenv;31501]

\n

We will look very closely at the preliminary winner's solution...

\n

[/quote]

\n

Will the milestone winner's methods be published?

", "rawMarkdown": "", "votes": null }, { "id": "31521", "postDate": "09/25/2013 00:08:22", "content": "

I sure hope not. That would kill the point of the competition if everyone gets to use that code as a baseline.

", "rawMarkdown": "", "votes": null }, { "id": "31523", "postDate": "09/25/2013 01:02:38", "content": "

yes, the milestone winner's algorithms and methods will be published, as per the official rules:

\n

AS A CONDITION OF ACCEPTING THE MILESTONE PRIZE, THE MILESTONE PRIZE WINNER THEREBY AGREES TO PUBLISH THEIR IDEAS (AS DEFINED HEREIN TO INCLUDE NO SOURCE OR EXECUTABLE CODE), METHODOLOGY AND DOCUMENTATION ON THE COMPETITION FORUM WITHIN ONE WEEK OF THE MILESTONE PRIZE ANNOUNCEMENT AND IN ACCORDANCE WITH THE WINNING MODEL DOCUMENTATION TEMPLATE AVAILABLE ON THE KAGGLE WIKI AT http://www.kaggle.com/wiki. The Milestone Prize winner will be announced at the Minds & Machines conference scheduled for October 9, 2013.

", "rawMarkdown": "", "votes": null }, { "id": "31527", "postDate": "09/25/2013 05:25:42", "content": "

Wow, that's ridiculous that it has to be published on the forums.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 31289, "author_name": "superpilot", "author_url": "", "post_date": "09/21/2013 01:28:41", "content": "

This a very important issue! I am waiting for answer too...

", "votes": null, "replies": [] }, { "id": 31297, "author_name": "dmytrolystopad", "author_url": "", "post_date": "09/21/2013 07:41:04", "content": "

[quote=Gábor Takács;31282]

\n

I think that the rules should be modified to eliminate this nasty and serious issue.
Here are two suggestions:

\n
    \n
  1. Contestants should be allowed to use live wind information in their milestone prize solution.
    2. \n
  2. The milestone prize evaluation should use something else instead of the publicly available live wind data, e.g. randomly perturb the wind vectors included in TestWeatherData-Partial.zip.
    3. \n
\n

[/quote]

\n

Using all data, using cutoff data and using random data it's three different tasks with three different solutions. I don't see the way to keep task and eliminate this issue except two phase validation as like final prize use.

", "votes": null, "replies": [] }, { "id": 31386, "author_name": "anlthms", "author_url": "", "post_date": "09/23/2013 09:05:22", "content": "

[quote=Gábor Takács;31282]

\n

I think that the rules should be modified to eliminate this nasty and serious issue.

\n

[/quote]

\n

I concur.

\n

[quote]

\n
    \n
  1. Contestants should be allowed to use live wind information in their milestone prize solution.
    2. \n
  2. The milestone prize evaluation should use something else instead of the publicly available live wind data, e.g. randomly perturb the wind vectors included in TestWeatherData-Partial.zip.
    3. \n
\n

[/quote]

\n

At first glance, #2 looks like an excellent solution, but it does not eliminate the edge that a contestant could gain by using live data. If the admins were to randomly perturb the wind vectors and re-score the leaderboard, one would expect the ranking to stay the same. Sure, the gaps between users and nonusers of live data would decrease. Depending on the drops in score, you can probably come up with some heuristic to detect usage of live data.

\n

If the admins go with #1, I would request that they provide the complete weather files instead of having the contestants crawl the data and reformat.

", "votes": null, "replies": [] }, { "id": 31387, "author_name": "alzmcr", "author_url": "", "post_date": "09/23/2013 09:11:44", "content": "

[quote=Anil Thomas;31386]

\n

If the admins go with #1, I would request that they provide the complete weather files instead of having the contestants crawl the data and reformat.

\n

[/quote]

\n

yep - I guess we're all waiting for a clarification on this!

", "votes": null, "replies": [] }, { "id": 31389, "author_name": "gtakacs", "author_url": "", "post_date": "09/23/2013 09:58:07", "content": "

[quote=Anil Thomas;31386]

\n

At first glance, #2 sounds like an excellent solution, but it does not eliminate the edge that a contestant could gain by using live data. If the admins were to randomly perturb the wind vectors and re-score the leaderboard, one would expect the ranking to stay the same. Sure, the gaps between users and nonusers of live data would decrease. Depending on the drops in score, you can probably come up with some heuristic to detect usage of live data.

\n

If the admins go with #1, I would request that they provide the complete weather files instead of having the contestants crawl the data and reformat.

\n

[/quote]

\n

I didn't mean perturbing the live wind vectors but creating the wind vectors of future hours by randomly perturbing the wind vectors of the initial hour.

\n

Anyway, I see no really good solution for the situation. It is also problematic to change rules in the last minute. Also, I don't know if it's really possible to gain significant advantage from live wind information encoded into a few parameters. I observed that the direct usage of live wind can help a lot, therefore there is a danger of undetectable cheating. Another point: Everybody is using live wind information to some small extent, by using public leaderboard scores as a feedback.

\n

No matter what will be the decision, Kaggle should take this issue seriously.

\n

Gabor

", "votes": null, "replies": [] }, { "id": 31408, "author_name": "anlthms", "author_url": "", "post_date": "09/23/2013 15:29:30", "content": "

[quote=Gábor Takács;31389]

\n

I didn't mean perturbing the live wind vectors but creating the wind vectors of future hours by randomly perturbing the wind vectors of the initial hour.

\n

[/quote]

\n

Ah... that would work.

", "votes": null, "replies": [] }, { "id": 31412, "author_name": "gabrieleseppi", "author_url": "", "post_date": "09/23/2013 16:08:12", "content": "

I don't know if that would work as well as intended. I assume that someone is trying to create some forecast from the "legal" data we've been given. Creating a random perturbation would probably mean disrupting the work of such person.

", "votes": null, "replies": [] }, { "id": 31501, "author_name": "joycenv", "author_url": "", "post_date": "09/24/2013 14:49:37", "content": "

Gabor and all: thank you for raising this issue and the thoughtful input. As was pointed out, it is too late to change the rules for the Milestone prize. So only the wind information up to the cutoff time for each day may be used. We will look very closely at the preliminary winner's solution for the potential use of live wind information, which would be grounds for disqualification.

\n

As for the rest of the competition, remember that this is a 2-phase competition that requires your models to be locked down at the end of the training period and before the release of the final evaluation set. Therefore you will not be able to incorporate live wind information on the final evaluation data. We are looking into providing the wind data to you in an easier-to-use format that does not tempt the use of future information.

\n

The limitation of randomly perturbing the wind data and using that in the evaluation is that moves the wind data away from a realistic scenario.

\n

Please continue to share your thoughts on the competition design on the forums.

", "votes": null, "replies": [] }, { "id": 31514, "author_name": "anlthms", "author_url": "", "post_date": "09/24/2013 19:30:10", "content": "

[quote=joycenv;31501]

\n

We will look very closely at the preliminary winner's solution...

\n

[/quote]

\n

Will the milestone winner's methods be published?

", "votes": null, "replies": [] }, { "id": 31521, "author_name": "ericmartinez", "author_url": "", "post_date": "09/25/2013 00:08:22", "content": "

I sure hope not. That would kill the point of the competition if everyone gets to use that code as a baseline.

", "votes": null, "replies": [] }, { "id": 31523, "author_name": "dave40", "author_url": "", "post_date": "09/25/2013 01:02:38", "content": "

yes, the milestone winner's algorithms and methods will be published, as per the official rules:

\n

AS A CONDITION OF ACCEPTING THE MILESTONE PRIZE, THE MILESTONE PRIZE WINNER THEREBY AGREES TO PUBLISH THEIR IDEAS (AS DEFINED HEREIN TO INCLUDE NO SOURCE OR EXECUTABLE CODE), METHODOLOGY AND DOCUMENTATION ON THE COMPETITION FORUM WITHIN ONE WEEK OF THE MILESTONE PRIZE ANNOUNCEMENT AND IN ACCORDANCE WITH THE WINNING MODEL DOCUMENTATION TEMPLATE AVAILABLE ON THE KAGGLE WIKI AT http://www.kaggle.com/wiki. The Milestone Prize winner will be announced at the Minds & Machines conference scheduled for October 9, 2013.

", "votes": null, "replies": [] }, { "id": 31527, "author_name": "ericmartinez", "author_url": "", "post_date": "09/25/2013 05:25:42", "content": "

Wow, that's ridiculous that it has to be published on the forums.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "31282": "", "31289": "", "31297": "", "31386": "", "31387": "", "31389": "", "31408": "", "31412": "", "31501": "", "31514": "", "31521": "", "31523": "", "31527": "" }, "source": "meta" }