{ "id": 6239, "title": "Which files are going to be provided ?", "url": "/competitions/flight2-main/discussion/6239", "author_name": "", "post_date": "2013-11-05T22:11:15.777Z", "votes": null, "comment_count": 36, "views": 18852, "content": "

Hello,

\n

can you please confirm which files are going to be provided for the final evaluation ?

\n

i understand that: "Configuration.json" ; "Projection.json" ; "Airports.csv" (both easting/northing and latitude/longitude version) ; Landing.csv ; Taxi.csv ; restrictedZones.csv and turbulentZones_{0}.csv are going to be provided. and also weather_{0}.txt.gz for the first hour.

\n

what about "flights_{0}.csv" ? you already provided them so why we need "TestFlights.csv" ?

\n

also i assume that you will give the relevant Aggregate Test Features.

\n

can you please also explain the format of Landing.csv ; Taxi.csv ?

\n

 

\n

thanks

\n

 

\n

 

", "messages": [ { "id": "33287", "postDate": "11/05/2013 22:11:15", "content": "

Hello,

\n

can you please confirm which files are going to be provided for the final evaluation ?

\n

i understand that: "Configuration.json" ; "Projection.json" ; "Airports.csv" (both easting/northing and latitude/longitude version) ; Landing.csv ; Taxi.csv ; restrictedZones.csv and turbulentZones_{0}.csv are going to be provided. and also weather_{0}.txt.gz for the first hour.

\n

what about "flights_{0}.csv" ? you already provided them so why we need "TestFlights.csv" ?

\n

also i assume that you will give the relevant Aggregate Test Features.

\n

can you please also explain the format of Landing.csv ; Taxi.csv ?

\n

 

\n

thanks

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "33288", "postDate": "11/05/2013 22:27:39", "content": "

It really would be nice to have everything very clearly and precisely described regarding the data. Currently it feels like a baffling labyrinth laid out across various pages and forum threads, with different bits of data released at different times, and the flight simulator source serving as the primary documentation for the format of the data files themselves.

", "rawMarkdown": "", "votes": null }, { "id": "33388", "postDate": "11/07/2013 22:04:36", "content": "

We've listed out the files used and needed by the Simulator on this webpage. Please take a look.

", "rawMarkdown": "", "votes": null }, { "id": "33393", "postDate": "11/08/2013 00:32:37", "content": "

That's extremely helpful, thanks very much!

", "rawMarkdown": "", "votes": null }, { "id": "33400", "postDate": "11/08/2013 04:33:45", "content": "

[quote=joycenv;33388]

\n

We've listed out the files used and needed by the Simulator on this webpage. Please take a look.

\n

[/quote]

\n

Thanks for the link joycenv, very useful! 

\n

I think I've raised this during phase one, about landing & takeoff events. I'm trying to recreate the actualLanding20130910_1803.csv which is provided as part of the oneday files.

\n

I not sure what I'm doing wrong, but I can't recreate this from the training2_flighthistory.csv. i'm short of 1,022 flights landing from 18:03 to 01:03 of the following day (+7 hours). I'm not only short, I have a significant variation of flights landing at each airports (mean 16, median 4, min -81, max 135), this grouping by airports and actual_runaway_arrivals.

\n

I do understand that the "actual" data is not provided for the test set, but I can't even recreate train data from the flighthistory file provided. Basically, I can't get an approximation of the congestion of an airports if I can't recreate a similar counts per each airports.

\n

Can you please have a look a this? Hopefully is just me doing something wrong, but I'm kinda spent enough time to have some concerns!

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "33410", "postDate": "11/08/2013 10:55:11", "content": "

thanks, indeed this is very helpful.

\n

And i join the question regarding the landing & takeoff events, i also found some inconsistence.

\n

please see my question:

\n

https://www.gequest.com/c/flight2-main/forums/t/6115/how-the-time-in-actuallandings-file-is-calculated

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "33554", "postDate": "11/11/2013 11:57:11", "content": "

Hi admins,

\n

Will TestFlights.csv also be provided for the validation period?

\n

If yes, is it possible to get also a version of TestFlights.csv for Sept 10th 2013?

\n

Thanks!

", "rawMarkdown": "", "votes": null }, { "id": "33643", "postDate": "11/12/2013 22:19:40", "content": "

[quote=Alessandro Mariani;33400]

\n

[quote=joycenv;33388]

\n

We've listed out the files used and needed by the Simulator on this webpage. Please take a look.

\n

[/quote]

\n

Thanks for the link joycenv, very useful! 

\n

I think I've raised this during phase one, about landing & takeoff events. I'm trying to recreate the actualLanding20130910_1803.csv which is provided as part of the oneday files.

\n

I not sure what I'm doing wrong, but I can't recreate this from the training2_flighthistory.csv. i'm short of 1,022 flights landing from 18:03 to 01:03 of the following day (+7 hours). I'm not only short, I have a significant variation of flights landing at each airports (mean 16, median 4, min -81, max 135), this grouping by airports and actual_runaway_arrivals.

\n

I do understand that the "actual" data is not provided for the test set, but I can't even recreate train data from the flighthistory file provided. Basically, I can't get an approximation of the congestion of an airports if I can't recreate a similar counts per each airports.

\n

Can you please have a look a this? Hopefully is just me doing something wrong, but I'm kinda spent enough time to have some concerns!

\n

Thanks!

\n

[/quote]

\n

Hi Alessandro,

\n

I'm not exactly sure what you are doing differently, but here are a couple things to consider.

\n\n

Does this information help you? There maybe is an "edge" effect because this the last day of the training set (although I'm not sure how), but the days previous to Sep 10 are completely available and usable for building a congestion model.

", "rawMarkdown": "", "votes": null }, { "id": "33645", "postDate": "11/12/2013 23:17:39", "content": "

[quote=Jules;33554]

\n

Hi admins,

\n

Will TestFlights.csv also be provided for the validation period?

\n

If yes, is it possible to get also a version of TestFlights.csv for Sept 10th 2013?

\n

[/quote]

\n

Jules, TestFlights.csv will always be provided.  I've added the Sept. 10th TestFlights to the zip download.  

", "rawMarkdown": "", "votes": null }, { "id": "33649", "postDate": "11/13/2013 00:37:34", "content": "

[quote=joycenv;33643]

\n

Hi Alessandro,

\n

I'm not exactly sure what you are doing differently, but here are a couple things to consider.

\n\n

Does this information help you? There maybe is an "edge" effect because this the last day of the training set (although I'm not sure how), but the days previous to Sep 10 are completely available and usable for building a congestion model.

\n

[/quote]

\n

Thanks Joyce! Unfortunately doesn't help me... :(

\n

I'll try to make it as simple as possibile. If I open your training2_flighthistory.csv file, then I use a filter on actual_runaway_arrival between 2013/09/10 18:03 (UTC) and 2013/09/11 01:03 (UTC), this will return 10,353 flights. Now, if I apply a filter on arrival_airport_icao_code to include only the 63 US aiports I end up with 7,743 flights.

\n

actualLandings_20130910_1803.csv has 8,639 flights landing between the cutoff time and the following 7 hours (2013/09/10 18:03 to 2013/09/11 01:03).

\n

Shouldn't be the counts be quite similar? Is not just similarity, slicing down by airport, the counts are in excess or deficit which is not a close rapresentation of what happen at each airports.

\n

What am I doing wrong?

", "rawMarkdown": "", "votes": null }, { "id": "33689", "postDate": "11/13/2013 21:21:18", "content": "

Hi,

\n

I assume that you are going to provide TestFeatures files (like test2_flighthistory.csv) in order to calculate "actualTakeoffs" , "groundConditions" , "actualLandings". am i right ?

\n

In this case i see you filter out flights (form flighthistory) which "published_departure" is aftrer the cutoff time. i think this is too aggressive filter since "published_departure" and even other features such as "scheduled_*" should be available, as information,  before the cutoff time to flights that should depature after the cutoff time. in other words, we should have more information to calcualte "actualTakeoffs". Am i wrong?

", "rawMarkdown": "", "votes": null }, { "id": "33780", "postDate": "11/15/2013 22:49:26", "content": "

[quote=Alessandro Mariani;33649]

\n

I'll try to make it as simple as possibile. If I open your training2_flighthistory.csv file, then I use a filter on actual_runaway_arrival between 2013/09/10 18:03 (UTC) and 2013/09/11 01:03 (UTC), this will return 10,353 flights. Now, if I apply a filter on arrival_airport_icao_code to include only the 63 US aiports I end up with 7,743 flights.

\n

actualLandings_20130910_1803.csv has 8,639 flights landing between the cutoff time and the following 7 hours (2013/09/10 18:03 to 2013/09/11 01:03).

\n

Shouldn't be the counts be quite similar? Is not just similarity, slicing down by airport, the counts are in excess or deficit which is not a close rapresentation of what happen at each airports.

\n

What am I doing wrong?

\n

[/quote]

\n

Can you upload a list of the flight_history_ids for your 7743 flights? That'll help us narrow down the sources of any discrepancy.

", "rawMarkdown": "", "votes": null }, { "id": "34806", "postDate": "11/16/2013 21:39:11", "content": "

here is attached - thanks for having a look into this!

\n

Is any of my steps above incorrect?

", "rawMarkdown": "", "votes": null }, { "id": "34813", "postDate": "11/17/2013 02:20:35", "content": "

I'd like official word on sparrow's assumption at the top of this thread that weather_{0}.txt.gz for the first hour will be provided by Kaggle for the final evaluation. Is this correct?

\n

More generally, I am having doubts about the status of RAP data. There is  wealth of high resolution information in those files, but is it off limits after the final model submission deadline on December 18? In other words, can I train a model using RAP data up to December 18, but not use new RAP data after December 18?

\n

Obvious example: RAP includes hourly weather predictions produced by a very sophisticated physical model which we can have no hope to match. It seems very likely that route optimization results could be improved by incorporating the last RAP predictions available at each day's cutoff. But we will not be able to use those predictions in the final evaluation, because they do not yet exist on December 18, and Kaggle will not include them in the final evaluation data. Is this correct?

", "rawMarkdown": "", "votes": null }, { "id": "34995", "postDate": "11/21/2013 13:16:28", "content": "

short of 1000 landing when i recreated the actual landing file. and i carefully followed the python code.  So i think we really need the something like the ids of these 7743 flights to solve this. Thanks 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "34997", "postDate": "11/21/2013 13:52:25", "content": "

I haven't tried the exercise myself, but I wonder if it's not just a matter of the flight history file only listing flights between airports on the US mainland, while actual landings include flights originating anywhere (Alaska, Hawaii, Europe...).

", "rawMarkdown": "", "votes": null }, { "id": "35001", "postDate": "11/21/2013 15:35:10", "content": "

Has someone confirmed which, if any, weather files (simulator format) will be available?  There were indications that the first weather file for each day would be available, but that's not confirmed on the webpage joycenv indicated.  Either way, the rest of the day the simulator will run with actual weather patterns that we'd, I assume, have to predict (or ignore or something).  Is that correct?

\n

It seems, then, that weather is the only non-deterministic factor we have in the simulator and files for the final submission... yes?  The other three files listed as "not provided" come with code that can be used to create them, and of course those rely on the actual predicted flight data so that's understandable (and deterministic, if a bit recursive).

\n

The main reason I want to be certain is that it seems the weather could really screw up an otherwise good solution, for example by causing collisions in an algorithm that actually has good collision avoidance.  If, for example, an eastbound and westbound flight encounter a windy day they may collide when there was no reason to expect them to when the flight paths were created.  It also affects concerns in the simulator bugs forum about crossing boundaries of restricted areas during a simulator timeslice... with the unknown airspeed component change, it becomes impossible to be that precise on a final submission, correct?

\n

Or am I missing something?  Will we be able to generate the actual weather files used in the final simulator?  Thanks for any advice; right now my wind model is my achilles heel (right along side the 50 other things that don't work, but at least I think I grok).

", "rawMarkdown": "", "votes": null }, { "id": "35049", "postDate": "11/22/2013 17:40:40", "content": "

I'm bumping this because I think it's important.

\n

ChipMonkey, collisions between planes are not an issue; the simulator handles one flight at the time, and does so as if it were the only one. But winds can certainly affect the choice of optimal route substantially.

\n

Besides winds, takeoffs and landings after the cutoff are not provided and must be predicted. Or at least congestion must be predicted. That too can be affected by the weather.

\n

It bothers me that we still have no clear statement on which data we can use for the final evaluation. Let me sketch a few possible scenarios:

\n

- Team A decides to rely on the latest RAP data available at each day's cutoff. They lead the public board and fully expect to place well in the final evaluation. Instead, when it's too late to change approach, Kaggle declares that RAP data can not be used this way. Team A spends the next month telling everybody who listens how they feel about Kaggle.

\n

- Team B decides not to rely on RAP, since it's not guaranteed to be allowed. They spend most of their time creating a heroic weather predictor which does miracles with low quality METAR input. In the final evaluation, they are beaten by somebody who simply used the latest RAP data available at each day's cutoff. You can guess what Team B spends the next month doing.

\n

- Team C has a bunch of ideas which it could try, but is unwilling to spend days coding and testing them if Kaggle is unwilling to spend a few minutes to spell out which data they will be able to rely on for the final evaluation. When somebody else eventually takes the big prize, they go "we could esily have done that too, but it's hard to shoot for a goal before the goal posts have been put down", and spend the next month doing you know what.

\n

This is fun, but I think I'd better stop here...

", "rawMarkdown": "", "votes": null }, { "id": "35050", "postDate": "11/22/2013 17:58:27", "content": "

For each day, you may use the RAP data up to that day's cutoff. Additionally, we will be providing wind data files in the simulator format containing the last hour of wind data prior to the cutoff for each day in the final evaluation set, as we have for the first two data releases.

", "rawMarkdown": "", "votes": null }, { "id": "35055", "postDate": "11/22/2013 18:16:25", "content": "

Thanks, now I can leave Team C. :)

", "rawMarkdown": "", "votes": null }, { "id": "35196", "postDate": "11/25/2013 15:31:15", "content": "

Where can we find the RAP files before cut-off ?

\n

It would be nice if the organizers could provide us with a pointer to the last RAP data that is allowed to use in the agent.

\n

Also, is there any place where we could find a list of scheduled take-offs (as known at cutoff time)?

", "rawMarkdown": "", "votes": null }, { "id": "35328", "postDate": "11/27/2013 02:37:50", "content": "

http://soostrc.comet.ucar.edu/data/grib/rap/  here are the 'rap' documents. But it seems very difficult for me to understand the context of such data. Such data is too professional

", "rawMarkdown": "", "votes": null }, { "id": "36155", "postDate": "12/13/2013 18:34:16", "content": "

[quote=Alessandro Mariani;34806]

\n

here is attached - thanks for having a look into this!

\n

Is any of my steps above incorrect?

\n

[/quote]

\n

Hi!

\n

Do we have more details about this? I still think is quite a significant aspect which requires some clarification please! I'm still catching up with the last 4 weeks posts but I couldn't find an answer :(

", "rawMarkdown": "", "votes": null }, { "id": "36254", "postDate": "12/17/2013 14:52:47", "content": "

@Alessandro and admins,

\n

I too wasn't able to recreate actualLandings_20130910_1803.csv

\n

I used the SQL from

\n

https://github.com/benhamner/GEFlightQuest/blob/master/PythonModule/geflight/postgres_ingest/schema.sql#L377

\n

and the training2_flighthistory.csv file.

\n

I found 19 flights landing at 2013-09-10 18:04 (time = 0.001758 in the actuallandings file)

\n

instead of the 23 according to actualLandings_20130910_1803.csv.

\n

I found a total of 7734 flights landing between

\n

2013-09-10 18:03:53.6712

\n

and

\n

2013-09-11 01:03:53.6712

\n

using one of the 63 airports defined.

\n

So my question for the admins is: What is the exact procedure to create actualLandings_20130910_1803.csv from training2_flighthistory.csv?

\n

Thanks in advance

", "rawMarkdown": "", "votes": null }, { "id": "36465", "postDate": "12/21/2013 13:44:40", "content": "

[quote=sparrow ;33689]

\n

Hi,

\n

I assume that you are going to provide TestFeatures files (like test2_flighthistory.csv) in order to calculate "actualTakeoffs" , "groundConditions" , "actualLandings". am i right ?

\n

In this case i see you filter out flights (form flighthistory) which "published_departure" is aftrer the cutoff time. i think this is too aggressive filter since "published_departure" and even other features such as "scheduled_*" should be available, as information,  before the cutoff time to flights that should depature after the cutoff time. in other words, we should have more information to calcualte "actualTakeoffs". Am i wrong?

\n

[/quote]

\n

I'm bumping this question, because it appears it hasn't been answered: are TestFeatures files going to be included in the final set?

\n

And what about those flights whose published departure is after the cutoff?

", "rawMarkdown": "", "votes": null }, { "id": "36534", "postDate": "12/22/2013 21:33:30", "content": "

[quote=Jules;36254]

\n

So my question for the admins is: What is the exact procedure to create actualLandings_20130910_1803.csv from training2_flighthistory.csv?

\n

[/quote]

\n

Admins,

\n

As reported by Alessandro and Jules, there are discrepancies between training2_flighthistory.csv and actualLandings_20130910_1803.csv.

\n

Want proof? On a bash shell, cd to the directory that has training2_flighthistory.csv and run this command:

\n

cat training2_flighthistory.csv | grep KCVG | cut -f18 -d',' |  grep "2013-09-10 18:04" | wc -l

\n

This will output the number of flights that either took off or landed at KCVG AND has an actual_runway_arrival time of "2013-09-10 18:04". One would expect this to be a non-zero value. Why? Because KCVG appears in the first time slot within actualLandings_20130910_1803.csv.

\n

The command, however returns 0. Maybe I got the time wrong? The exact cutoff time is 18:03:53 after all. So let's try this:

\n

cat training2_flighthistory.csv | grep KCVG | cut -f18 -d',' |  grep "2013-09-10 18:03" | wc -l

\n

Nada.

\n

Maybe this will work?

\n

cat training2_flighthistory.csv | grep KCVG | cut -f18 -d',' |  grep "2013-09-10 18:05" | wc -l

\n

Still nada.

\n

So, what gives?

\n

Curiously enough, there is a flight that lands at KCVG with an actual_runway_departure of 18:04. Makes you wonder...

\n

Please look into it. It is also possible that you are using  actualLanding files with incorrect values to score the leaderboard.

", "rawMarkdown": "", "votes": null }, { "id": "36751", "postDate": "12/28/2013 12:42:41", "content": "

Hi

\n

I guess we have a filtered version of the complete flighthistory table.

\n

training2_flighthistory.csv and training3_flighthistory.csv only includes continental domestic flights, i.e. flights whose departure AND arrival airport are in the continental United States, but actualLandings and actualTakeoffs include also international flights (I guess).

\n

Admins, Am I right? Can you provide the complete flighthistory table?  We would need all the flights whose departure OR arrival airport are in the continental United States. Don`t we?

", "rawMarkdown": "", "votes": null }, { "id": "36756", "postDate": "12/28/2013 14:37:13", "content": "

I'm sure this has already been asked and answered several times, but since I can't find it, just to check: the hour of weather data that we'll be provided for the test day, does that hour END at the cutoff time or START at the cutoff time?

", "rawMarkdown": "", "votes": null }, { "id": "36964", "postDate": "01/03/2014 20:29:09", "content": "

[quote=Anil Thomas;36534]

\n

Admins,

\n

As reported by Alessandro and Jules, there are discrepancies between training2_flighthistory.csv and actualLandings_20130910_1803.csv.

\n

...

\n

[/quote]

\n

Still waiting for an answer on this. May I point out that this issue was originally raised almost two months ago?

", "rawMarkdown": "", "votes": null }, { "id": "37254", "postDate": "01/10/2014 20:19:14", "content": "

Thanks for pointing the discrepancy between flighthistory.csv and actualLandings*.csv out. We've investigated it and got to the bottom of the discrepancy.

\n

As José pointed out, the counts do not agree since the flighthistory file provided only contained flights both departing from and arriving to US airports. This was incorrect, and we apologize for the trouble that it's called. That file should have also contained flights with only one endpoint at a US airport as well (since international flights departing from or arriving to the US also naturally cause congestion at US airports). The actualLandings*.csv and actualTakeoffs*.csv files were built off the complete dataset (including international flights), as was correct.

\n

To address this, we have uploaded a corrected version of the flighthistory files for both the augmented training data sets and the current leaderboard data set as FQ2_DataRelease_FHWithUSArrivalOrDeparture.zip. If you are estimating actualLandings*.csv and actualTakeoffs*.csv as part of your solution, make sure to use these updated files.

", "rawMarkdown": "", "votes": null }, { "id": "37257", "postDate": "01/10/2014 21:23:57", "content": "

Hi Ben, thanks for this!

\n

training2_flighthistory_USARRDEP.csv now contains new flights, but test2_flighthistory_USARRDEP.csv is the same file as the original provided test2_flighthistory.csv :-/

", "rawMarkdown": "", "votes": null }, { "id": "37260", "postDate": "01/10/2014 22:13:04", "content": "

Corrected test2 is now up on the Data page.

", "rawMarkdown": "", "votes": null }, { "id": "37282", "postDate": "01/11/2014 14:34:48", "content": "

Gentlemen, this is not acceptable. It's inconceivable that training datasets released more than 4 (four) months ago are being changed 1 (one) week before the final deadline. Either the final deadline is going to be extended, or actualLandings and actualTakeoffs for both the current leaderboard and the final one must not contain international flights.

\n

We've spent the last 2 (two) months trying to develop a model for producing actualLandings and actualTakeoffs as close as possible as the real ones, not to mention the effects that these files had on the development of the complete route model. Now we have 1 (one) week to redo/rethink our entire approach. Do you think this is fair? Actually, I think we don't even have enough time to re-calculate 14 (fourteen) days of routes from scratch with our simple computers. 8 (eight) days are less than the time allowed from the release of the final dataset to the end of the competition. I think only a bunch of people here have access to big data center-like machines.

\n

I checked the differences in the old and new datasets. As an example, take test2_flighthistory.csv: we're moving from 148017 to 173621 flights. That's 25604 flight more than before: +17.3%! Are you kidding me? Come on, it's almost 1/5 more flights than before! Not to mention the fact that current datasets are still not complete: as an example, take test2_flighthistoryevents.csv. I see no international flights there...

", "rawMarkdown": "", "votes": null }, { "id": "37286", "postDate": "01/11/2014 18:10:36", "content": "

Ouch. FWIW, I predict takeoffs and landings too, and it takes days to train the models. I thought that was done and I could spend the last few days before the deadline polishing the optimization. Guess not. :(

", "rawMarkdown": "", "votes": null }, { "id": "37288", "postDate": "01/11/2014 18:23:00", "content": "

Is training1_flighthistory.csv from the initial data release also incomplete? If so, will it be updated too? (Yes, you guessed, I use that file.)

", "rawMarkdown": "", "votes": null }, { "id": "37291", "postDate": "01/11/2014 18:46:27", "content": "

I don't have it with me at the moment, but you can easily check if either source or destination ICAO codes contain 4-letter codes that don't start by K. In that case there's a high chance that the dataset is complete.

", "rawMarkdown": "", "votes": null }, { "id": "37296", "postDate": "01/11/2014 21:27:25", "content": "

Good point. And wouldn't you know it, it's K all the way. So I'm about to find out what happens when I drop one third of my training data.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 33288, "author_name": "stereotomy", "author_url": "", "post_date": "11/05/2013 22:27:39", "content": "

It really would be nice to have everything very clearly and precisely described regarding the data. Currently it feels like a baffling labyrinth laid out across various pages and forum threads, with different bits of data released at different times, and the flight simulator source serving as the primary documentation for the format of the data files themselves.

", "votes": null, "replies": [] }, { "id": 33388, "author_name": "joycenv", "author_url": "", "post_date": "11/07/2013 22:04:36", "content": "

We've listed out the files used and needed by the Simulator on this webpage. Please take a look.

", "votes": null, "replies": [] }, { "id": 33393, "author_name": "stereotomy", "author_url": "", "post_date": "11/08/2013 00:32:37", "content": "

That's extremely helpful, thanks very much!

", "votes": null, "replies": [] }, { "id": 33400, "author_name": "alzmcr", "author_url": "", "post_date": "11/08/2013 04:33:45", "content": "

[quote=joycenv;33388]

\n

We've listed out the files used and needed by the Simulator on this webpage. Please take a look.

\n

[/quote]

\n

Thanks for the link joycenv, very useful! 

\n

I think I've raised this during phase one, about landing & takeoff events. I'm trying to recreate the actualLanding20130910_1803.csv which is provided as part of the oneday files.

\n

I not sure what I'm doing wrong, but I can't recreate this from the training2_flighthistory.csv. i'm short of 1,022 flights landing from 18:03 to 01:03 of the following day (+7 hours). I'm not only short, I have a significant variation of flights landing at each airports (mean 16, median 4, min -81, max 135), this grouping by airports and actual_runaway_arrivals.

\n

I do understand that the "actual" data is not provided for the test set, but I can't even recreate train data from the flighthistory file provided. Basically, I can't get an approximation of the congestion of an airports if I can't recreate a similar counts per each airports.

\n

Can you please have a look a this? Hopefully is just me doing something wrong, but I'm kinda spent enough time to have some concerns!

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 33410, "author_name": "drorrr", "author_url": "", "post_date": "11/08/2013 10:55:11", "content": "

thanks, indeed this is very helpful.

\n

And i join the question regarding the landing & takeoff events, i also found some inconsistence.

\n

please see my question:

\n

https://www.gequest.com/c/flight2-main/forums/t/6115/how-the-time-in-actuallandings-file-is-calculated

\n

 

", "votes": null, "replies": [] }, { "id": 33554, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "11/11/2013 11:57:11", "content": "

Hi admins,

\n

Will TestFlights.csv also be provided for the validation period?

\n

If yes, is it possible to get also a version of TestFlights.csv for Sept 10th 2013?

\n

Thanks!

", "votes": null, "replies": [] }, { "id": 33643, "author_name": "joycenv", "author_url": "", "post_date": "11/12/2013 22:19:40", "content": "

[quote=Alessandro Mariani;33400]

\n

[quote=joycenv;33388]

\n

We've listed out the files used and needed by the Simulator on this webpage. Please take a look.

\n

[/quote]

\n

Thanks for the link joycenv, very useful! 

\n

I think I've raised this during phase one, about landing & takeoff events. I'm trying to recreate the actualLanding20130910_1803.csv which is provided as part of the oneday files.

\n

I not sure what I'm doing wrong, but I can't recreate this from the training2_flighthistory.csv. i'm short of 1,022 flights landing from 18:03 to 01:03 of the following day (+7 hours). I'm not only short, I have a significant variation of flights landing at each airports (mean 16, median 4, min -81, max 135), this grouping by airports and actual_runaway_arrivals.

\n

I do understand that the "actual" data is not provided for the test set, but I can't even recreate train data from the flighthistory file provided. Basically, I can't get an approximation of the congestion of an airports if I can't recreate a similar counts per each airports.

\n

Can you please have a look a this? Hopefully is just me doing something wrong, but I'm kinda spent enough time to have some concerns!

\n

Thanks!

\n

[/quote]

\n

Hi Alessandro,

\n

I'm not exactly sure what you are doing differently, but here are a couple things to consider.

\n\n

Does this information help you? There maybe is an "edge" effect because this the last day of the training set (although I'm not sure how), but the days previous to Sep 10 are completely available and usable for building a congestion model.

", "votes": null, "replies": [] }, { "id": 33645, "author_name": "nicksardo", "author_url": "", "post_date": "11/12/2013 23:17:39", "content": "

[quote=Jules;33554]

\n

Hi admins,

\n

Will TestFlights.csv also be provided for the validation period?

\n

If yes, is it possible to get also a version of TestFlights.csv for Sept 10th 2013?

\n

[/quote]

\n

Jules, TestFlights.csv will always be provided.  I've added the Sept. 10th TestFlights to the zip download.  

", "votes": null, "replies": [] }, { "id": 33649, "author_name": "alzmcr", "author_url": "", "post_date": "11/13/2013 00:37:34", "content": "

[quote=joycenv;33643]

\n

Hi Alessandro,

\n

I'm not exactly sure what you are doing differently, but here are a couple things to consider.

\n\n

Does this information help you? There maybe is an "edge" effect because this the last day of the training set (although I'm not sure how), but the days previous to Sep 10 are completely available and usable for building a congestion model.

\n

[/quote]

\n

Thanks Joyce! Unfortunately doesn't help me... :(

\n

I'll try to make it as simple as possibile. If I open your training2_flighthistory.csv file, then I use a filter on actual_runaway_arrival between 2013/09/10 18:03 (UTC) and 2013/09/11 01:03 (UTC), this will return 10,353 flights. Now, if I apply a filter on arrival_airport_icao_code to include only the 63 US aiports I end up with 7,743 flights.

\n

actualLandings_20130910_1803.csv has 8,639 flights landing between the cutoff time and the following 7 hours (2013/09/10 18:03 to 2013/09/11 01:03).

\n

Shouldn't be the counts be quite similar? Is not just similarity, slicing down by airport, the counts are in excess or deficit which is not a close rapresentation of what happen at each airports.

\n

What am I doing wrong?

", "votes": null, "replies": [] }, { "id": 33689, "author_name": "drorrr", "author_url": "", "post_date": "11/13/2013 21:21:18", "content": "

Hi,

\n

I assume that you are going to provide TestFeatures files (like test2_flighthistory.csv) in order to calculate "actualTakeoffs" , "groundConditions" , "actualLandings". am i right ?

\n

In this case i see you filter out flights (form flighthistory) which "published_departure" is aftrer the cutoff time. i think this is too aggressive filter since "published_departure" and even other features such as "scheduled_*" should be available, as information,  before the cutoff time to flights that should depature after the cutoff time. in other words, we should have more information to calcualte "actualTakeoffs". Am i wrong?

", "votes": null, "replies": [] }, { "id": 33780, "author_name": "joycenv", "author_url": "", "post_date": "11/15/2013 22:49:26", "content": "

[quote=Alessandro Mariani;33649]

\n

I'll try to make it as simple as possibile. If I open your training2_flighthistory.csv file, then I use a filter on actual_runaway_arrival between 2013/09/10 18:03 (UTC) and 2013/09/11 01:03 (UTC), this will return 10,353 flights. Now, if I apply a filter on arrival_airport_icao_code to include only the 63 US aiports I end up with 7,743 flights.

\n

actualLandings_20130910_1803.csv has 8,639 flights landing between the cutoff time and the following 7 hours (2013/09/10 18:03 to 2013/09/11 01:03).

\n

Shouldn't be the counts be quite similar? Is not just similarity, slicing down by airport, the counts are in excess or deficit which is not a close rapresentation of what happen at each airports.

\n

What am I doing wrong?

\n

[/quote]

\n

Can you upload a list of the flight_history_ids for your 7743 flights? That'll help us narrow down the sources of any discrepancy.

", "votes": null, "replies": [] }, { "id": 34806, "author_name": "alzmcr", "author_url": "", "post_date": "11/16/2013 21:39:11", "content": "

here is attached - thanks for having a look into this!

\n

Is any of my steps above incorrect?

", "votes": null, "replies": [] }, { "id": 34813, "author_name": "hyperborean", "author_url": "", "post_date": "11/17/2013 02:20:35", "content": "

I'd like official word on sparrow's assumption at the top of this thread that weather_{0}.txt.gz for the first hour will be provided by Kaggle for the final evaluation. Is this correct?

\n

More generally, I am having doubts about the status of RAP data. There is  wealth of high resolution information in those files, but is it off limits after the final model submission deadline on December 18? In other words, can I train a model using RAP data up to December 18, but not use new RAP data after December 18?

\n

Obvious example: RAP includes hourly weather predictions produced by a very sophisticated physical model which we can have no hope to match. It seems very likely that route optimization results could be improved by incorporating the last RAP predictions available at each day's cutoff. But we will not be able to use those predictions in the final evaluation, because they do not yet exist on December 18, and Kaggle will not include them in the final evaluation data. Is this correct?

", "votes": null, "replies": [] }, { "id": 34995, "author_name": "hightechfarmer", "author_url": "", "post_date": "11/21/2013 13:16:28", "content": "

short of 1000 landing when i recreated the actual landing file. and i carefully followed the python code.  So i think we really need the something like the ids of these 7743 flights to solve this. Thanks 

\n

 

", "votes": null, "replies": [] }, { "id": 34997, "author_name": "hyperborean", "author_url": "", "post_date": "11/21/2013 13:52:25", "content": "

I haven't tried the exercise myself, but I wonder if it's not just a matter of the flight history file only listing flights between airports on the US mainland, while actual landings include flights originating anywhere (Alaska, Hawaii, Europe...).

", "votes": null, "replies": [] }, { "id": 35001, "author_name": "chipmonkey", "author_url": "", "post_date": "11/21/2013 15:35:10", "content": "

Has someone confirmed which, if any, weather files (simulator format) will be available?  There were indications that the first weather file for each day would be available, but that's not confirmed on the webpage joycenv indicated.  Either way, the rest of the day the simulator will run with actual weather patterns that we'd, I assume, have to predict (or ignore or something).  Is that correct?

\n

It seems, then, that weather is the only non-deterministic factor we have in the simulator and files for the final submission... yes?  The other three files listed as "not provided" come with code that can be used to create them, and of course those rely on the actual predicted flight data so that's understandable (and deterministic, if a bit recursive).

\n

The main reason I want to be certain is that it seems the weather could really screw up an otherwise good solution, for example by causing collisions in an algorithm that actually has good collision avoidance.  If, for example, an eastbound and westbound flight encounter a windy day they may collide when there was no reason to expect them to when the flight paths were created.  It also affects concerns in the simulator bugs forum about crossing boundaries of restricted areas during a simulator timeslice... with the unknown airspeed component change, it becomes impossible to be that precise on a final submission, correct?

\n

Or am I missing something?  Will we be able to generate the actual weather files used in the final simulator?  Thanks for any advice; right now my wind model is my achilles heel (right along side the 50 other things that don't work, but at least I think I grok).

", "votes": null, "replies": [] }, { "id": 35049, "author_name": "hyperborean", "author_url": "", "post_date": "11/22/2013 17:40:40", "content": "

I'm bumping this because I think it's important.

\n

ChipMonkey, collisions between planes are not an issue; the simulator handles one flight at the time, and does so as if it were the only one. But winds can certainly affect the choice of optimal route substantially.

\n

Besides winds, takeoffs and landings after the cutoff are not provided and must be predicted. Or at least congestion must be predicted. That too can be affected by the weather.

\n

It bothers me that we still have no clear statement on which data we can use for the final evaluation. Let me sketch a few possible scenarios:

\n

- Team A decides to rely on the latest RAP data available at each day's cutoff. They lead the public board and fully expect to place well in the final evaluation. Instead, when it's too late to change approach, Kaggle declares that RAP data can not be used this way. Team A spends the next month telling everybody who listens how they feel about Kaggle.

\n

- Team B decides not to rely on RAP, since it's not guaranteed to be allowed. They spend most of their time creating a heroic weather predictor which does miracles with low quality METAR input. In the final evaluation, they are beaten by somebody who simply used the latest RAP data available at each day's cutoff. You can guess what Team B spends the next month doing.

\n

- Team C has a bunch of ideas which it could try, but is unwilling to spend days coding and testing them if Kaggle is unwilling to spend a few minutes to spell out which data they will be able to rely on for the final evaluation. When somebody else eventually takes the big prize, they go "we could esily have done that too, but it's hard to shoot for a goal before the goal posts have been put down", and spend the next month doing you know what.

\n

This is fun, but I think I'd better stop here...

", "votes": null, "replies": [] }, { "id": 35050, "author_name": "benhamner", "author_url": "", "post_date": "11/22/2013 17:58:27", "content": "

For each day, you may use the RAP data up to that day's cutoff. Additionally, we will be providing wind data files in the simulator format containing the last hour of wind data prior to the cutoff for each day in the final evaluation set, as we have for the first two data releases.

", "votes": null, "replies": [] }, { "id": 35055, "author_name": "hyperborean", "author_url": "", "post_date": "11/22/2013 18:16:25", "content": "

Thanks, now I can leave Team C. :)

", "votes": null, "replies": [] }, { "id": 35196, "author_name": "arieli", "author_url": "", "post_date": "11/25/2013 15:31:15", "content": "

Where can we find the RAP files before cut-off ?

\n

It would be nice if the organizers could provide us with a pointer to the last RAP data that is allowed to use in the agent.

\n

Also, is there any place where we could find a list of scheduled take-offs (as known at cutoff time)?

", "votes": null, "replies": [] }, { "id": 35328, "author_name": "hightechfarmer", "author_url": "", "post_date": "11/27/2013 02:37:50", "content": "

http://soostrc.comet.ucar.edu/data/grib/rap/  here are the 'rap' documents. But it seems very difficult for me to understand the context of such data. Such data is too professional

", "votes": null, "replies": [] }, { "id": 36155, "author_name": "alzmcr", "author_url": "", "post_date": "12/13/2013 18:34:16", "content": "

[quote=Alessandro Mariani;34806]

\n

here is attached - thanks for having a look into this!

\n

Is any of my steps above incorrect?

\n

[/quote]

\n

Hi!

\n

Do we have more details about this? I still think is quite a significant aspect which requires some clarification please! I'm still catching up with the last 4 weeks posts but I couldn't find an answer :(

", "votes": null, "replies": [] }, { "id": 36254, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "12/17/2013 14:52:47", "content": "

@Alessandro and admins,

\n

I too wasn't able to recreate actualLandings_20130910_1803.csv

\n

I used the SQL from

\n

https://github.com/benhamner/GEFlightQuest/blob/master/PythonModule/geflight/postgres_ingest/schema.sql#L377

\n

and the training2_flighthistory.csv file.

\n

I found 19 flights landing at 2013-09-10 18:04 (time = 0.001758 in the actuallandings file)

\n

instead of the 23 according to actualLandings_20130910_1803.csv.

\n

I found a total of 7734 flights landing between

\n

2013-09-10 18:03:53.6712

\n

and

\n

2013-09-11 01:03:53.6712

\n

using one of the 63 airports defined.

\n

So my question for the admins is: What is the exact procedure to create actualLandings_20130910_1803.csv from training2_flighthistory.csv?

\n

Thanks in advance

", "votes": null, "replies": [] }, { "id": 36465, "author_name": "gabrieleseppi", "author_url": "", "post_date": "12/21/2013 13:44:40", "content": "

[quote=sparrow ;33689]

\n

Hi,

\n

I assume that you are going to provide TestFeatures files (like test2_flighthistory.csv) in order to calculate "actualTakeoffs" , "groundConditions" , "actualLandings". am i right ?

\n

In this case i see you filter out flights (form flighthistory) which "published_departure" is aftrer the cutoff time. i think this is too aggressive filter since "published_departure" and even other features such as "scheduled_*" should be available, as information,  before the cutoff time to flights that should depature after the cutoff time. in other words, we should have more information to calcualte "actualTakeoffs". Am i wrong?

\n

[/quote]

\n

I'm bumping this question, because it appears it hasn't been answered: are TestFeatures files going to be included in the final set?

\n

And what about those flights whose published departure is after the cutoff?

", "votes": null, "replies": [] }, { "id": 36534, "author_name": "anlthms", "author_url": "", "post_date": "12/22/2013 21:33:30", "content": "

[quote=Jules;36254]

\n

So my question for the admins is: What is the exact procedure to create actualLandings_20130910_1803.csv from training2_flighthistory.csv?

\n

[/quote]

\n

Admins,

\n

As reported by Alessandro and Jules, there are discrepancies between training2_flighthistory.csv and actualLandings_20130910_1803.csv.

\n

Want proof? On a bash shell, cd to the directory that has training2_flighthistory.csv and run this command:

\n

cat training2_flighthistory.csv | grep KCVG | cut -f18 -d',' |  grep "2013-09-10 18:04" | wc -l

\n

This will output the number of flights that either took off or landed at KCVG AND has an actual_runway_arrival time of "2013-09-10 18:04". One would expect this to be a non-zero value. Why? Because KCVG appears in the first time slot within actualLandings_20130910_1803.csv.

\n

The command, however returns 0. Maybe I got the time wrong? The exact cutoff time is 18:03:53 after all. So let's try this:

\n

cat training2_flighthistory.csv | grep KCVG | cut -f18 -d',' |  grep "2013-09-10 18:03" | wc -l

\n

Nada.

\n

Maybe this will work?

\n

cat training2_flighthistory.csv | grep KCVG | cut -f18 -d',' |  grep "2013-09-10 18:05" | wc -l

\n

Still nada.

\n

So, what gives?

\n

Curiously enough, there is a flight that lands at KCVG with an actual_runway_departure of 18:04. Makes you wonder...

\n

Please look into it. It is also possible that you are using  actualLanding files with incorrect values to score the leaderboard.

", "votes": null, "replies": [] }, { "id": 36751, "author_name": "jarfo1", "author_url": "", "post_date": "12/28/2013 12:42:41", "content": "

Hi

\n

I guess we have a filtered version of the complete flighthistory table.

\n

training2_flighthistory.csv and training3_flighthistory.csv only includes continental domestic flights, i.e. flights whose departure AND arrival airport are in the continental United States, but actualLandings and actualTakeoffs include also international flights (I guess).

\n

Admins, Am I right? Can you provide the complete flighthistory table?  We would need all the flights whose departure OR arrival airport are in the continental United States. Don`t we?

", "votes": null, "replies": [] }, { "id": 36756, "author_name": "stereotomy", "author_url": "", "post_date": "12/28/2013 14:37:13", "content": "

I'm sure this has already been asked and answered several times, but since I can't find it, just to check: the hour of weather data that we'll be provided for the test day, does that hour END at the cutoff time or START at the cutoff time?

", "votes": null, "replies": [] }, { "id": 36964, "author_name": "anlthms", "author_url": "", "post_date": "01/03/2014 20:29:09", "content": "

[quote=Anil Thomas;36534]

\n

Admins,

\n

As reported by Alessandro and Jules, there are discrepancies between training2_flighthistory.csv and actualLandings_20130910_1803.csv.

\n

...

\n

[/quote]

\n

Still waiting for an answer on this. May I point out that this issue was originally raised almost two months ago?

", "votes": null, "replies": [] }, { "id": 37254, "author_name": "benhamner", "author_url": "", "post_date": "01/10/2014 20:19:14", "content": "

Thanks for pointing the discrepancy between flighthistory.csv and actualLandings*.csv out. We've investigated it and got to the bottom of the discrepancy.

\n

As José pointed out, the counts do not agree since the flighthistory file provided only contained flights both departing from and arriving to US airports. This was incorrect, and we apologize for the trouble that it's called. That file should have also contained flights with only one endpoint at a US airport as well (since international flights departing from or arriving to the US also naturally cause congestion at US airports). The actualLandings*.csv and actualTakeoffs*.csv files were built off the complete dataset (including international flights), as was correct.

\n

To address this, we have uploaded a corrected version of the flighthistory files for both the augmented training data sets and the current leaderboard data set as FQ2_DataRelease_FHWithUSArrivalOrDeparture.zip. If you are estimating actualLandings*.csv and actualTakeoffs*.csv as part of your solution, make sure to use these updated files.

", "votes": null, "replies": [] }, { "id": 37257, "author_name": "alzmcr", "author_url": "", "post_date": "01/10/2014 21:23:57", "content": "

Hi Ben, thanks for this!

\n

training2_flighthistory_USARRDEP.csv now contains new flights, but test2_flighthistory_USARRDEP.csv is the same file as the original provided test2_flighthistory.csv :-/

", "votes": null, "replies": [] }, { "id": 37260, "author_name": "joycenv", "author_url": "", "post_date": "01/10/2014 22:13:04", "content": "

Corrected test2 is now up on the Data page.

", "votes": null, "replies": [] }, { "id": 37282, "author_name": "gabrieleseppi", "author_url": "", "post_date": "01/11/2014 14:34:48", "content": "

Gentlemen, this is not acceptable. It's inconceivable that training datasets released more than 4 (four) months ago are being changed 1 (one) week before the final deadline. Either the final deadline is going to be extended, or actualLandings and actualTakeoffs for both the current leaderboard and the final one must not contain international flights.

\n

We've spent the last 2 (two) months trying to develop a model for producing actualLandings and actualTakeoffs as close as possible as the real ones, not to mention the effects that these files had on the development of the complete route model. Now we have 1 (one) week to redo/rethink our entire approach. Do you think this is fair? Actually, I think we don't even have enough time to re-calculate 14 (fourteen) days of routes from scratch with our simple computers. 8 (eight) days are less than the time allowed from the release of the final dataset to the end of the competition. I think only a bunch of people here have access to big data center-like machines.

\n

I checked the differences in the old and new datasets. As an example, take test2_flighthistory.csv: we're moving from 148017 to 173621 flights. That's 25604 flight more than before: +17.3%! Are you kidding me? Come on, it's almost 1/5 more flights than before! Not to mention the fact that current datasets are still not complete: as an example, take test2_flighthistoryevents.csv. I see no international flights there...

", "votes": null, "replies": [] }, { "id": 37286, "author_name": "hyperborean", "author_url": "", "post_date": "01/11/2014 18:10:36", "content": "

Ouch. FWIW, I predict takeoffs and landings too, and it takes days to train the models. I thought that was done and I could spend the last few days before the deadline polishing the optimization. Guess not. :(

", "votes": null, "replies": [] }, { "id": 37288, "author_name": "hyperborean", "author_url": "", "post_date": "01/11/2014 18:23:00", "content": "

Is training1_flighthistory.csv from the initial data release also incomplete? If so, will it be updated too? (Yes, you guessed, I use that file.)

", "votes": null, "replies": [] }, { "id": 37291, "author_name": "gabrieleseppi", "author_url": "", "post_date": "01/11/2014 18:46:27", "content": "

I don't have it with me at the moment, but you can easily check if either source or destination ICAO codes contain 4-letter codes that don't start by K. In that case there's a high chance that the dataset is complete.

", "votes": null, "replies": [] }, { "id": 37296, "author_name": "hyperborean", "author_url": "", "post_date": "01/11/2014 21:27:25", "content": "

Good point. And wouldn't you know it, it's K all the way. So I'm about to find out what happens when I drop one third of my training data.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "33287": "", "33288": "", "33388": "", "33393": "", "33400": "", "33410": "", "33554": "", "33643": "", "33645": "", "33649": "", "33689": "", "33780": "", "34806": "", "34813": "", "34995": "", "34997": "", "35001": "", "35049": "", "35050": "", "35055": "", "35196": "", "35328": "", "36155": "", "36254": "", "36465": "", "36534": "", "36751": "", "36756": "", "36964": "", "37254": "", "37257": "", "37260": "", "37282": "", "37286": "", "37288": "", "37291": "", "37296": "" }, "source": "meta" }