{ "id": 3413, "title": "Public Leaderboard Set Release + Leaderboard Activation", "url": "/competitions/flight/discussion/3413", "author_name": "", "post_date": "2012-12-20T08:48:25.257Z", "votes": 1, "comment_count": 21, "views": 17605, "content": "

Hi all,

\r\n

Thanks everyone for your participation in this quest so far, along with your patience with us on the data issues.

\r\n

I'm tremendously excited to announce the release of the Public Leaderboard Set, which contains 14 days of test data from November 26, 2012 through December 9, 2012. Each day is truncated at a randomly selected cutoff time. This data, along with sample benchmark\r\n submission files, can be downloaded from the \r\ndata page.

\r\n

This contest is far more complex than the typical Kaggle contest and currently it is relatively challenging to get started. We will be providing additional code and guides soon to help you with this, as well as clarifying and simplyifing some of the data\r\n descriptions.

\r\n

Please let us know if you have any questions, and good luck!

\r\n

Ben

", "messages": [ { "id": "18249", "postDate": "12/20/2012 08:48:25", "content": "

Hi all,

\r\n

Thanks everyone for your participation in this quest so far, along with your patience with us on the data issues.

\r\n

I'm tremendously excited to announce the release of the Public Leaderboard Set, which contains 14 days of test data from November 26, 2012 through December 9, 2012. Each day is truncated at a randomly selected cutoff time. This data, along with sample benchmark\r\n submission files, can be downloaded from the \r\ndata page.

\r\n

This contest is far more complex than the typical Kaggle contest and currently it is relatively challenging to get started. We will be providing additional code and guides soon to help you with this, as well as clarifying and simplyifing some of the data\r\n descriptions.

\r\n

Please let us know if you have any questions, and good luck!

\r\n

Ben

", "rawMarkdown": "", "votes": null }, { "id": "18260", "postDate": "12/20/2012 11:57:53", "content": "

Comparing the benchmark scores on the public leaderboard to scores of my reimplementations on the training set, I get ~95 for the cutoff benchmark, but only 10-12 for the estimated and sceduled arrival benchmarks. It would be really nice to have the sources\r\n of the benchmarks.

", "rawMarkdown": "", "votes": null }, { "id": "18272", "postDate": "12/20/2012 15:37:20", "content": "

Two unrelated things:

\r\n

The capitalization/case of some folder names (metar vs Metar, for instance) has changed. Of course, this is hardly fatal just inconvenient.

\r\n

Why does test_flights.csv contain anything else then flight_history_ids? Aren't those columns just duplicated from flighthistory.csv?

", "rawMarkdown": "", "votes": null }, { "id": "18275", "postDate": "12/20/2012 16:10:34", "content": "

[quote=Gábor Melis;18260]

\r\n

Comparing the benchmark scores on the public leaderboard to scores of my reimplementations on the training set, I get ~95 for the cutoff benchmark, but only 10-12 for the estimated and sceduled arrival benchmarks. It would be really nice to have the sources\r\n of the benchmarks.

\r\n

[/quote]Thanks for pointing this out. ]There was a bug in the estimated arrival benchmark and I've updated it accordingly. (Code is on github, and I'll publicize here once I get a chance to refactor & document it).

", "rawMarkdown": "", "votes": null }, { "id": "18276", "postDate": "12/20/2012 16:15:12", "content": "

[quote=Gábor Melis;18272]

\r\n

The capitalization/case of some folder names (metar vs Metar, for instance) has changed. Of course, this is hardly fatal just inconvenient.[/quote]This will be consistent moving forward ("Metar", etc.).

\r\n

[quote]Why does test_flights.csv contain anything else then flight_history_ids? Aren't those columns just duplicated from flighthistory.csv?[/quote]These are duplicated from flighthistory.csv. Added so a newcomer to the site could download a combined test_flights.csv\r\n file and run a quick script to make a basic submission without downloading a large zip file (haven't put this up yet).

", "rawMarkdown": "", "votes": null }, { "id": "18277", "postDate": "12/20/2012 16:55:25", "content": "

Thanks. One more thing: are non-integer minutes in the prediction ok? The rounding (that was apparently done with floor instead of the more correct round in the cutoff benchmark) could be an unncessary (?) source of noise.

", "rawMarkdown": "", "votes": null }, { "id": "18278", "postDate": "12/20/2012 17:41:43", "content": "

[quote=Gábor Melis;18277]

\r\n

Thanks. One more thing: are non-integer minutes in the prediction ok? The rounding (that was apparently done with floor instead of the more correct round in the cutoff benchmark) could be an unncessary (?) source of noise.

\r\n

[/quote]Real-valued predictions are fine

", "rawMarkdown": "", "votes": null }, { "id": "18281", "postDate": "12/20/2012 19:25:09", "content": "

Hi Ben,

\r\n

On the leaderboard I see only the entry above the top benchmark as being “in the money”, shouldn't this be the first 5 entries made by individuals/teams?

\r\n

Or is beating a benchmark in anyway conditional for winning a prize?

\r\n

Thanks,

\r\n

Jules

", "rawMarkdown": "", "votes": null }, { "id": "18283", "postDate": "12/20/2012 20:01:59", "content": "

Thanks again.

\r\n

Are all flights to be predicted included in the calculation of the public score on the leaderboard or is there a private component at this early stage too? I'm asking so that the noise level can be estimated.

", "rawMarkdown": "", "votes": null }, { "id": "18284", "postDate": "12/20/2012 20:28:30", "content": "

[quote=Gábor Melis;18283]

\r\n

Thanks again.

\r\n

Are all flights to be predicted included in the calculation of the public score on the leaderboard or is there a private component at this early stage too? I'm asking so that the noise level can be estimated.

\r\n

[/quote]No private component at this stage - that will only come with the final evaluation set.

", "rawMarkdown": "", "votes": null }, { "id": "18285", "postDate": "12/20/2012 20:29:24", "content": "

[quote=Jules;18281]

\r\n

Hi Ben,

\r\n

On the leaderboard I see only the entry above the top benchmark as being “in the money”, shouldn't this be the first 5 entries made by individuals/teams?

\r\n

Or is beating a benchmark in anyway conditional for winning a prize?

\r\n

Thanks,

\r\n

Jules

\r\n

[/quote]Thanks for pointing this out, this is fixed now.

", "rawMarkdown": "", "votes": null }, { "id": "18462", "postDate": "12/23/2012 09:24:00", "content": "

Ben,

\r\n

I was wondering what the status was for publishing the code needed to generate the benchmark.

\r\n

 

\r\n

rg,

\r\n

K

", "rawMarkdown": "", "votes": null }, { "id": "18502", "postDate": "12/23/2012 22:43:48", "content": "

Hi,
\r\ni've noticed in some FlightHistory.csv of the Leaderboard set that the field scheduled_aircraft_type is not filled for several rows. Is there any reason for this?

\r\n

I've also found out that always in the field airline_code sometimes there's the '1I' value, that's not a valid IATA code and there wasn't in the files of training set data.

\r\n

 

\r\n

Thank you,
\r\nPierluigi

", "rawMarkdown": "", "votes": null }, { "id": "19010", "postDate": "01/05/2013 22:13:22", "content": "

Hi,

\r\n

It is still a pain for a newcomer to get an idea on how to start. It takes roughly 4 hours of forum scanning and python parsing (once found on the forum) to figure out that days.csv, testflights.csv, and test-flightscombined.csv are crucial and\r\n what they contain.

\r\n

Further, the only clear explanation of what is the allowed training data and what is the test set is well hidden in one of the many 'clarification needed' threads (I have lost it already, I will have to dig again)

\r\n

Is it possible to put the description of the PublicLeaderboard files on the data-wiki?
\r\n* what is test-flights.csv + columns (or where they are copied from, where MISSING/HIDDEN values are added)
\r\n* what is testflightscombined.csv + columns
\r\n* what is days.csv
\r\n* how PublicLeaderboard set is different from the InitialTraining set (values missing after cutoff)
\r\n* that one should not use PublicLeaderboard set for training

\r\n

I can do it myslef if you open the front page for editing. Or please move 'DirectoryStructure' to a new, editable page.

\r\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "19026", "postDate": "01/06/2013 14:04:58", "content": "

Ben:
\r\nCan you confirm that the timezone of flighthistory.csv in both PublicLeaderboard and also in InitialRelease (training) folder is in UTC?
\r\nAnd that the rest are not in UTC

\r\n

Thanks
\r\nkiran

", "rawMarkdown": "", "votes": null }, { "id": "19031", "postDate": "01/06/2013 16:22:08", "content": "

Are there links to the wiki and github repository somewhere on the information page? If not, could you please provide them?

", "rawMarkdown": "", "votes": null }, { "id": "19125", "postDate": "01/09/2013 09:35:27", "content": "

Hi,

\r\n

The benchmark files have only three columns, each row corresponding to a distinct flight ID. The for which this prediction has been made is not mentioned.

\r\n

In the submission file for a prticular flight which day should the prediction of actual flight gate and arrival time correspond to ?

\r\n

Should there be 15 files for the 15 days of the test set ?

", "rawMarkdown": "", "votes": null }, { "id": "19207", "postDate": "01/11/2013 23:23:19", "content": "

Sorry wrong post

", "rawMarkdown": "", "votes": null }, { "id": "19904", "postDate": "01/31/2013 17:17:55", "content": "

Hi  Ben,

\r\n

 

\r\n

 

\r\n

What is the format of model submission. 

\r\n

Algorithm pseudocode or detailed algorithm ?

\r\n

 

\r\n

 

\r\n

 

\r\n

regards,

\r\n

I. Jaganjac

", "rawMarkdown": "", "votes": null }, { "id": "19906", "postDate": "01/31/2013 17:19:43", "content": "

[quote=Lamb;19904]

\r\n

Hi  Ben,

\r\n

 

\r\n

 

\r\n

What is the format of model submission. 

\r\n

Algorithm pseudocode or detailed algorithm ?

\r\n

 

\r\n

 

\r\n

 

\r\n

regards,

\r\n

I. Jaganjac

\r\n

[/quote]A hash of a zip archive containing everything necessary to make flight arrival time predictions on a new day's data.

", "rawMarkdown": "", "votes": null }, { "id": "19932", "postDate": "02/01/2013 17:00:15", "content": "

[quote=Ben Hamner;19906] A hash of a zip archive containing everything necessary to make flight arrival time predictions on a new day's data. [/quote]

\r\n

But.. how to calculate this hash?

\r\n

...I have found an utility:  http://support.microsoft.com/kb/841290

\r\n

It seems to work..

", "rawMarkdown": "", "votes": null }, { "id": "19968", "postDate": "02/03/2013 09:02:21", "content": "

[quote=Victor;19932]

\r\n

[quote=Ben Hamner;19906] A hash of a zip archive containing everything necessary to make flight arrival time predictions on a new day's data. [/quote]

\r\n

But.. how to calculate this hash?

\r\n

[/quote]The hash should be SHA-256. You should be able to calculate this on all common platforms.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 18260, "author_name": "melisgl", "author_url": "", "post_date": "12/20/2012 11:57:53", "content": "

Comparing the benchmark scores on the public leaderboard to scores of my reimplementations on the training set, I get ~95 for the cutoff benchmark, but only 10-12 for the estimated and sceduled arrival benchmarks. It would be really nice to have the sources\r\n of the benchmarks.

", "votes": null, "replies": [] }, { "id": 18272, "author_name": "melisgl", "author_url": "", "post_date": "12/20/2012 15:37:20", "content": "

Two unrelated things:

\r\n

The capitalization/case of some folder names (metar vs Metar, for instance) has changed. Of course, this is hardly fatal just inconvenient.

\r\n

Why does test_flights.csv contain anything else then flight_history_ids? Aren't those columns just duplicated from flighthistory.csv?

", "votes": null, "replies": [] }, { "id": 18275, "author_name": "benhamner", "author_url": "", "post_date": "12/20/2012 16:10:34", "content": "

[quote=Gábor Melis;18260]

\r\n

Comparing the benchmark scores on the public leaderboard to scores of my reimplementations on the training set, I get ~95 for the cutoff benchmark, but only 10-12 for the estimated and sceduled arrival benchmarks. It would be really nice to have the sources\r\n of the benchmarks.

\r\n

[/quote]Thanks for pointing this out. ]There was a bug in the estimated arrival benchmark and I've updated it accordingly. (Code is on github, and I'll publicize here once I get a chance to refactor & document it).

", "votes": null, "replies": [] }, { "id": 18276, "author_name": "benhamner", "author_url": "", "post_date": "12/20/2012 16:15:12", "content": "

[quote=Gábor Melis;18272]

\r\n

The capitalization/case of some folder names (metar vs Metar, for instance) has changed. Of course, this is hardly fatal just inconvenient.[/quote]This will be consistent moving forward ("Metar", etc.).

\r\n

[quote]Why does test_flights.csv contain anything else then flight_history_ids? Aren't those columns just duplicated from flighthistory.csv?[/quote]These are duplicated from flighthistory.csv. Added so a newcomer to the site could download a combined test_flights.csv\r\n file and run a quick script to make a basic submission without downloading a large zip file (haven't put this up yet).

", "votes": null, "replies": [] }, { "id": 18277, "author_name": "melisgl", "author_url": "", "post_date": "12/20/2012 16:55:25", "content": "

Thanks. One more thing: are non-integer minutes in the prediction ok? The rounding (that was apparently done with floor instead of the more correct round in the cutoff benchmark) could be an unncessary (?) source of noise.

", "votes": null, "replies": [] }, { "id": 18278, "author_name": "benhamner", "author_url": "", "post_date": "12/20/2012 17:41:43", "content": "

[quote=Gábor Melis;18277]

\r\n

Thanks. One more thing: are non-integer minutes in the prediction ok? The rounding (that was apparently done with floor instead of the more correct round in the cutoff benchmark) could be an unncessary (?) source of noise.

\r\n

[/quote]Real-valued predictions are fine

", "votes": null, "replies": [] }, { "id": 18281, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "12/20/2012 19:25:09", "content": "

Hi Ben,

\r\n

On the leaderboard I see only the entry above the top benchmark as being “in the money”, shouldn't this be the first 5 entries made by individuals/teams?

\r\n

Or is beating a benchmark in anyway conditional for winning a prize?

\r\n

Thanks,

\r\n

Jules

", "votes": null, "replies": [] }, { "id": 18283, "author_name": "melisgl", "author_url": "", "post_date": "12/20/2012 20:01:59", "content": "

Thanks again.

\r\n

Are all flights to be predicted included in the calculation of the public score on the leaderboard or is there a private component at this early stage too? I'm asking so that the noise level can be estimated.

", "votes": null, "replies": [] }, { "id": 18284, "author_name": "benhamner", "author_url": "", "post_date": "12/20/2012 20:28:30", "content": "

[quote=Gábor Melis;18283]

\r\n

Thanks again.

\r\n

Are all flights to be predicted included in the calculation of the public score on the leaderboard or is there a private component at this early stage too? I'm asking so that the noise level can be estimated.

\r\n

[/quote]No private component at this stage - that will only come with the final evaluation set.

", "votes": null, "replies": [] }, { "id": 18285, "author_name": "benhamner", "author_url": "", "post_date": "12/20/2012 20:29:24", "content": "

[quote=Jules;18281]

\r\n

Hi Ben,

\r\n

On the leaderboard I see only the entry above the top benchmark as being “in the money”, shouldn't this be the first 5 entries made by individuals/teams?

\r\n

Or is beating a benchmark in anyway conditional for winning a prize?

\r\n

Thanks,

\r\n

Jules

\r\n

[/quote]Thanks for pointing this out, this is fixed now.

", "votes": null, "replies": [] }, { "id": 18462, "author_name": "konradb", "author_url": "", "post_date": "12/23/2012 09:24:00", "content": "

Ben,

\r\n

I was wondering what the status was for publishing the code needed to generate the benchmark.

\r\n

 

\r\n

rg,

\r\n

K

", "votes": null, "replies": [] }, { "id": 18502, "author_name": "pierluigivinciguerra", "author_url": "", "post_date": "12/23/2012 22:43:48", "content": "

Hi,
\r\ni've noticed in some FlightHistory.csv of the Leaderboard set that the field scheduled_aircraft_type is not filled for several rows. Is there any reason for this?

\r\n

I've also found out that always in the field airline_code sometimes there's the '1I' value, that's not a valid IATA code and there wasn't in the files of training set data.

\r\n

 

\r\n

Thank you,
\r\nPierluigi

", "votes": null, "replies": [] }, { "id": 19010, "author_name": "vojtekb", "author_url": "", "post_date": "01/05/2013 22:13:22", "content": "

Hi,

\r\n

It is still a pain for a newcomer to get an idea on how to start. It takes roughly 4 hours of forum scanning and python parsing (once found on the forum) to figure out that days.csv, testflights.csv, and test-flightscombined.csv are crucial and\r\n what they contain.

\r\n

Further, the only clear explanation of what is the allowed training data and what is the test set is well hidden in one of the many 'clarification needed' threads (I have lost it already, I will have to dig again)

\r\n

Is it possible to put the description of the PublicLeaderboard files on the data-wiki?
\r\n* what is test-flights.csv + columns (or where they are copied from, where MISSING/HIDDEN values are added)
\r\n* what is testflightscombined.csv + columns
\r\n* what is days.csv
\r\n* how PublicLeaderboard set is different from the InitialTraining set (values missing after cutoff)
\r\n* that one should not use PublicLeaderboard set for training

\r\n

I can do it myslef if you open the front page for editing. Or please move 'DirectoryStructure' to a new, editable page.

\r\n

Thanks!

", "votes": null, "replies": [] }, { "id": 19026, "author_name": "rkirana", "author_url": "", "post_date": "01/06/2013 14:04:58", "content": "

Ben:
\r\nCan you confirm that the timezone of flighthistory.csv in both PublicLeaderboard and also in InitialRelease (training) folder is in UTC?
\r\nAnd that the rest are not in UTC

\r\n

Thanks
\r\nkiran

", "votes": null, "replies": [] }, { "id": 19031, "author_name": "albeam", "author_url": "", "post_date": "01/06/2013 16:22:08", "content": "

Are there links to the wiki and github repository somewhere on the information page? If not, could you please provide them?

", "votes": null, "replies": [] }, { "id": 19125, "author_name": "illuminanti", "author_url": "", "post_date": "01/09/2013 09:35:27", "content": "

Hi,

\r\n

The benchmark files have only three columns, each row corresponding to a distinct flight ID. The for which this prediction has been made is not mentioned.

\r\n

In the submission file for a prticular flight which day should the prediction of actual flight gate and arrival time correspond to ?

\r\n

Should there be 15 files for the 15 days of the test set ?

", "votes": null, "replies": [] }, { "id": 19207, "author_name": "ejlok1", "author_url": "", "post_date": "01/11/2013 23:23:19", "content": "

Sorry wrong post

", "votes": null, "replies": [] }, { "id": 19904, "author_name": "ijaganjac", "author_url": "", "post_date": "01/31/2013 17:17:55", "content": "

Hi  Ben,

\r\n

 

\r\n

 

\r\n

What is the format of model submission. 

\r\n

Algorithm pseudocode or detailed algorithm ?

\r\n

 

\r\n

 

\r\n

 

\r\n

regards,

\r\n

I. Jaganjac

", "votes": null, "replies": [] }, { "id": 19906, "author_name": "benhamner", "author_url": "", "post_date": "01/31/2013 17:19:43", "content": "

[quote=Lamb;19904]

\r\n

Hi  Ben,

\r\n

 

\r\n

 

\r\n

What is the format of model submission. 

\r\n

Algorithm pseudocode or detailed algorithm ?

\r\n

 

\r\n

 

\r\n

 

\r\n

regards,

\r\n

I. Jaganjac

\r\n

[/quote]A hash of a zip archive containing everything necessary to make flight arrival time predictions on a new day's data.

", "votes": null, "replies": [] }, { "id": 19932, "author_name": "nedelko", "author_url": "", "post_date": "02/01/2013 17:00:15", "content": "

[quote=Ben Hamner;19906] A hash of a zip archive containing everything necessary to make flight arrival time predictions on a new day's data. [/quote]

\r\n

But.. how to calculate this hash?

\r\n

...I have found an utility:  http://support.microsoft.com/kb/841290

\r\n

It seems to work..

", "votes": null, "replies": [] }, { "id": 19968, "author_name": "benhamner", "author_url": "", "post_date": "02/03/2013 09:02:21", "content": "

[quote=Victor;19932]

\r\n

[quote=Ben Hamner;19906] A hash of a zip archive containing everything necessary to make flight arrival time predictions on a new day's data. [/quote]

\r\n

But.. how to calculate this hash?

\r\n

[/quote]The hash should be SHA-256. You should be able to calculate this on all common platforms.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "18249": "", "18260": "", "18272": "", "18275": "", "18276": "", "18277": "", "18278": "", "18281": "", "18283": "", "18284": "", "18285": "", "18462": "", "18502": "", "19010": "", "19026": "", "19031": "", "19125": "", "19207": "", "19904": "", "19906": "", "19932": "", "19968": "" }, "source": "meta" }