{ "id": 3302, "title": "Training Set Revision 1 Release + Sample Test Set", "url": "/competitions/flight/discussion/3302", "author_name": "", "post_date": "2012-12-12T02:10:50.547Z", "votes": null, "comment_count": 20, "views": 13728, "content": "

I just uploaded InitialTrainingSet_rev1, which accounts for \r\nthe data issues that have been pointed out so far. Most significantly, the times in flighthistory.csv should now be correct. The initial data format we received had identified the times in this file as being Pacific time in each timestamp, even though they\r\n were local to the departure and arrival airports. Correcting this impacted many other files as well, since other files were split into days based on the departure time in flighthistory.csv. The  updated data may be downloaded from the\r\ndata page.

\r\n

Additionally, we have released SampleTestSet, a sample of the test set format generated with random cutoff times from the last week of the training set, and SampleSolution, which shows the solution file for SampleTestSet. (The submission file will be in\r\n the same format). The runway arrival and gate arrival times you predict will be measured in minutes from midnight UTC on the day associated with the flight.

\r\n

We have released the code for transforming the data files as well. You can use this code to create days in the test format at specific cutoff times from any day in the training set.

\r\n

Thanks for your input and participation so far, and please let us know if you see any other potential data leakage or data quality issues.

\r\n

Good luck on the contest!

", "messages": [ { "id": "17669", "postDate": "12/12/2012 02:10:50", "content": "

I just uploaded InitialTrainingSet_rev1, which accounts for \r\nthe data issues that have been pointed out so far. Most significantly, the times in flighthistory.csv should now be correct. The initial data format we received had identified the times in this file as being Pacific time in each timestamp, even though they\r\n were local to the departure and arrival airports. Correcting this impacted many other files as well, since other files were split into days based on the departure time in flighthistory.csv. The  updated data may be downloaded from the\r\ndata page.

\r\n

Additionally, we have released SampleTestSet, a sample of the test set format generated with random cutoff times from the last week of the training set, and SampleSolution, which shows the solution file for SampleTestSet. (The submission file will be in\r\n the same format). The runway arrival and gate arrival times you predict will be measured in minutes from midnight UTC on the day associated with the flight.

\r\n

We have released the code for transforming the data files as well. You can use this code to create days in the test format at specific cutoff times from any day in the training set.

\r\n

Thanks for your input and participation so far, and please let us know if you see any other potential data leakage or data quality issues.

\r\n

Good luck on the contest!

", "rawMarkdown": "", "votes": null }, { "id": "17672", "postDate": "12/12/2012 02:43:17", "content": "

One common question involved the logic used to identify flights for the test set. This code is available in\r\n\r\nflighhistory.py on Github.

\r\n

The conditions for a flight to be included in the test set are as follows:

\r\n", "rawMarkdown": "", "votes": null }, { "id": "17700", "postDate": "12/12/2012 11:13:51", "content": "

Hi,

\r\n

Just wanted to mention there still seem to be some problems with the training data actual arrival / departure times in flight history. Example:

\r\n

2012-11-12, flight_history_id: 280154062

\r\n
 actual_gate_departure actual_gate_arrival departure_airport_timezone_offset arrival_airport_timezone_offset
 2012-11-12 16:02:00+00:00 2012-11-12 16:59:00+00:00 -5 -6
\r\n
which, unless I'm mistaken, translates (after normalizing to the same timezone) into
\r\n
 (12-11-12 11:02:00) (12-11-12 10:59:00)
\r\n
I am reasonably confident a flight can not take -3 minutes, so I would appreciate some help - is it a data glitch, or am I missing something obvious
\r\n
 (misinterpreting the columns?).
\r\n
 
\r\n
rg,
\r\n
K
", "rawMarkdown": "", "votes": null }, { "id": "17747", "postDate": "12/12/2012 19:19:00", "content": "

The flighthistory.csv time columns are now all in UTC as specified in each timestamp, so the times are correct. The flight took 57 minutes to go from the departure gate to the arrival gate.

", "rawMarkdown": "", "votes": null }, { "id": "17754", "postDate": "12/12/2012 21:05:34", "content": "

Ben, in SampleTestSet the cutoff time for the first day is 2012-12-19 17:44 UTC. However, there are no events in flighthistoryevents.csv for that day later than 2012-11-19 10:00:00-08:00 (= 2012-11-19 02:00:00 UTC). Seems like the -8 was treated as +8?

", "rawMarkdown": "", "votes": null }, { "id": "17760", "postDate": "12/12/2012 22:21:28", "content": "

[quote=Gábor Melis;17754]

\r\n

Ben, in SampleTestSet the cutoff time for the first day is 2012-12-19 17:44 UTC. However, there are no events in flighthistoryevents.csv for that day later than 2012-11-19 10:00:00-08:00 (= 2012-11-19 02:00:00 UTC). Seems like the -8 was treated as +8?

\r\n

[/quote]

\r\n

Just looked into this. The latestevent in flighthistory.csv for that day is "2012-11-19 09:44:13.358000-08:00", which corresponds to "2012-11-19 17:44:13.358000+00:00" (UTC), which is correct. Keep in mind that the timezone offset -08 refers to Pacific Time,\r\n which is 8 hours before UTC (not after). You get UTC from the time zone offset by subtracting the offset, not adding it.

", "rawMarkdown": "", "votes": null }, { "id": "17761", "postDate": "12/12/2012 23:03:05", "content": "

[quote=Gábor Melis;17754]

\r\n

Ben, in SampleTestSet the cutoff time for the first day is 2012-12-19 17:44 UTC. However, there are no events in flighthistoryevents.csv for that day later than 2012-11-19 10:00:00-08:00 (= 2012-11-19 02:00:00 UTC). Seems like the -8 was treated as +8?

\r\n

[/quote]Thanks for checking this though. Let me know if you see any other inconsistencies or potential ways we could be leaking information - this is far easier to correct now than after December 18, when we will be releasing the Public Leaderboard Set.

", "rawMarkdown": "", "votes": null }, { "id": "17774", "postDate": "12/13/2012 08:48:00", "content": "

Thanks. Indeed, it was me who confused the signs.

", "rawMarkdown": "", "votes": null }, { "id": "17776", "postDate": "12/13/2012 12:41:36", "content": "

Ben,

\r\n

Thanks for the clarification, two more points where I could use an explanation:

\r\n

1. has the normalization to utc been applied to other files (e.g. flighthistoryevents.csv) as well?

\r\n

2. that sort makes the departure_airport_timezone_offset and arrival_airport_timezone_offset columns redundant, right?

\r\n

 

\r\n

rg,

\r\n

K

", "rawMarkdown": "", "votes": null }, { "id": "17778", "postDate": "12/13/2012 14:49:29", "content": "

[quote=Konrad Banachewicz;17776]

\r\n

2. that sort makes the departure_airport_timezone_offset and arrival_airport_timezone_offset columns redundant, right?

\r\n

[/quote]

\r\n

I don't think it is redundant.  I believe the offsets are needed if one wanted to account for the local daily cycle, i.e., morning/evening rush hours.

", "rawMarkdown": "", "votes": null }, { "id": "17785", "postDate": "12/13/2012 15:49:18", "content": "

Fair point, haven't thought about that...

", "rawMarkdown": "", "votes": null }, { "id": "17802", "postDate": "12/13/2012 19:29:40", "content": "

[quote=Konrad Banachewicz;17776]

\r\n

Ben,

\r\n

Thanks for the clarification, two more points where I could use an explanation:

\r\n

1. has the normalization to utc been applied to other files (e.g. flighthistoryevents.csv) as well?

\r\n

2. that sort makes the departure_airport_timezone_offset and arrival_airport_timezone_offset columns redundant, right?

\r\n

 

\r\n

rg,

\r\n

K

\r\n

[/quote]Normalization hasn't ben applied to flighthistoryevents - times in the data_updated column are local to the corresponding airports, and the time the event was recorded is in pacific time.

", "rawMarkdown": "", "votes": null }, { "id": "17879", "postDate": "12/15/2012 17:06:09", "content": "

[quote=Ben Hamner;17672]

\r\n

One common question involved the logic used to identify flights for the test set. This code is available in\r\n\r\nflighhistory.py on Github.

\r\n

The conditions for a flight to be included in the test set are as follows:

\r\n\r\n

[/quote]

\r\n

There may be one more condition necessary: that actual_gate_departure is less than actual_runway_departure. There are flights that are considered to be in flight because the cutoff is in between that actual_runway_departure and actual_runway_arrival, but\r\n the actual_runway_arrival is wildly wrong. This flight made it to TestSampleSet:

\r\n
  HISTORY-ID = 281197158
  CREATOR-CODE = "I"
  DEPARTURE-AIRPORT = #<AIRPORT SFO/KSFO/-8>
  ARRIVAL-AIRPORT = #<AIRPORT FLL/KFLL/-5>
  PUBLISHED-DEPARTURE = "2012-11-24 07:55:00"
  PUBLISHED-ARRIVAL = "2012-11-24 13:10:00"
  ACTUAL-GATE-DEPARTURE = "2012-11-24 07:53:00"
  ACTUAL-RUNWAY-DEPARTURE = "2012-11-23 20:04:00"
  ACTUAL-RUNWAY-ARRIVAL = "2012-11-24 13:17:00"
  ACTUAL-GATE-ARRIVAL = "2012-11-24 13:24:00"
  EVENTS = (#<EVENT 2012-11-23 08:00:57 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-23 08:00:57 SRA 2012-11-24 13:15:00>
            #<EVENT 2012-11-23 08:00:57 ERD 2012-11-24 08:04:00>
            #<EVENT 2012-11-23 08:00:57 SRD 2012-11-24 08:04:00>
            #<EVENT 2012-11-24 00:00:01 AGATE C9>
            #<EVENT 2012-11-24 00:52:03 DGATE 50>
            #<EVENT 2012-11-24 05:52:34 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 05:53:13 EGA 2012-11-24 13:20:00>
            #<EVENT 2012-11-24 07:14:33 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 08:10:24 AEQP A319>
            #<EVENT 2012-11-24 08:10:24 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 08:10:24 ERD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 08:10:24 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 08:10:24 STATUS A>
            #<EVENT 2012-11-24 08:10:24 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 08:10:24 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 08:10:28 EGA 2012-11-24 13:22:00>
            #<EVENT --- 2012-11-24 08:10:28 ARD 2012-11-23 20:04:00>
            #<EVENT 2012-11-24 08:10:28 AGD 2012-11-24 07:53:00>
            #<EVENT 2012-11-24 08:18:38 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-24 08:18:38 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 08:23:29 EGA 2012-11-24 13:21:00>
            #<EVENT --- 2012-11-24 08:23:29 ARD 2012-11-23 20:04:00>
            #<EVENT 2012-11-24 08:25:42 ERA 2012-11-24 13:14:00>
            #<EVENT 2012-11-24 08:25:42 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 09:13:29 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-24 10:28:13 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 10:39:15 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-24 10:40:15 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 11:08:48 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 11:33:08 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 11:45:09 ERA 2012-11-24 13:18:00>
            #<EVENT 2012-11-24 11:53:17 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 12:17:30 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 12:40:33 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 12:48:05 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-24 12:51:30 EGA 2012-11-24 13:22:00>
            #<EVENT --- 2012-11-24 12:51:30 ARD 2012-11-23 20:04:00>
            #<EVENT 2012-11-24 12:53:55 ERA 2012-11-24 13:11:00>
            #<EVENT --- 2012-11-24 12:53:55 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 12:54:34 ERA 2012-11-24 13:12:00>
            #<EVENT 2012-11-24 13:05:31 EGA 2012-11-24 13:23:00>
            #<EVENT --- 2012-11-24 13:05:31 ARD 2012-11-23 20:04:00>
            #<EVENT 2012-11-24 13:18:32 STATUS L>
            #<EVENT 2012-11-24 13:18:32 ARA 2012-11-24 13:17:00>
            #<EVENT --- 2012-11-24 13:25:18 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 13:34:33 AGA 2012-11-24 13:24:00>
            #<EVENT --- 2012-11-24 13:34:33 ARD 2012-11-23 20:04:00>)
\r\n
 
\r\n

Suspicious events are marked with "---". The important part is this:

\r\n
  ACTUAL-GATE-DEPARTURE = "2012-11-24 07:53:00"
  ACTUAL-RUNWAY-DEPARTURE = "2012-11-23 20:04:00"
\r\n

If the cutoff is between 2012-11-23 20:04:00 and 2012-11-24 07:53:00, then there is almost no information on which to base the prediction.

", "rawMarkdown": "", "votes": null }, { "id": "17889", "postDate": "12/15/2012 20:14:41", "content": "

Even if the problem in the previous post is fixed, then we aren't golden yet. It seems that basing the quantities to be predicted on the unreliable flighthistoryevents.csv files could be problematic.

\r\n
  HISTORY-ID = 280480834
  CREATOR-CODE = "I"
  DEPARTURE-AIRPORT = #<AIRPORT PDX/KPDX/-8>
  ARRIVAL-AIRPORT = #<AIRPORT PHX/KPHX/-7>
  PUBLISHED-DEPARTURE = "2012-11-15 14:55:00"
  PUBLISHED-ARRIVAL = "2012-11-15 17:29:00"
  ACTUAL-GATE-DEPARTURE = "2012-11-15 16:03:00"
  ACTUAL-RUNWAY-DEPARTURE = "2012-11-15 16:13:00"
  ACTUAL-RUNWAY-ARRIVAL = "2012-11-16 01:23:00"
  ACTUAL-GATE-ARRIVAL = "2012-11-16 01:27:00"
  EVENTS = (#<EVENT 2012-11-14 03:34:51 TAIL N786AS>
            #<EVENT 2012-11-14 03:34:51 AGATE 11>
            #<EVENT 2012-11-14 03:34:51 EGA 2012-11-15 17:29:00>
            #<EVENT 2012-11-14 03:34:51 DGATE C5>
            #<EVENT 2012-11-14 03:34:51 EGD 2012-11-15 14:55:00>
            #<EVENT 2012-11-14 03:38:50 DGATE C21>
            #<EVENT 2012-11-14 03:57:03 DGATE B2>
            #<EVENT 2012-11-14 03:57:37 DGATE C21>
            #<EVENT 2012-11-14 15:01:05 ERA 2012-11-15 17:19:00>
            #<EVENT 2012-11-14 15:01:05 SRA 2012-11-15 17:19:00>
            #<EVENT 2012-11-14 15:01:05 ERD 2012-11-15 15:04:00>
            #<EVENT 2012-11-14 15:01:05 SRD 2012-11-15 15:04:00>
            #<EVENT 2012-11-14 21:20:44 TAIL N703AS>
            #<EVENT 2012-11-14 21:20:44 DGATE C7>
            #<EVENT 2012-11-15 08:35:31 EGA 2012-11-15 17:28:00>
            #<EVENT 2012-11-15 13:37:00 ERA 2012-11-15 17:21:00>
            #<EVENT 2012-11-15 13:37:00 ERD 2012-11-15 15:05:00>
            #<EVENT 2012-11-15 15:00:05 STATUS A>
            #<EVENT 2012-11-15 15:00:05 AGD 2012-11-15 14:48:00>
            #<EVENT 2012-11-15 15:06:27 EGA 2012-11-15 18:18:00>
            #<EVENT 2012-11-15 15:06:43 EGD 2012-11-15 15:45:00>
            #<EVENT 2012-11-15 15:13:05 STATUS R>
            #<EVENT 2012-11-15 15:44:00 EGA 2012-11-15 18:53:00>
            #<EVENT 2012-11-15 15:44:00 EGD 2012-11-15 16:20:00>
            #<EVENT 2012-11-15 15:53:48 STATUS S>
            #<EVENT 2012-11-15 15:53:48 ERA 2012-11-15 18:46:00>
            #<EVENT 2012-11-15 15:53:48 ERD 2012-11-15 16:30:00>
            #<EVENT 2012-11-15 16:00:06 STATUS R>
            #<EVENT 2012-11-15 16:13:07 STATUS A>
            #<EVENT 2012-11-15 16:13:07 AGD 2012-11-15 16:03:00>
            #<EVENT 2012-11-15 16:13:49 EGA 2012-11-15 18:34:00>
            #<EVENT 2012-11-15 16:18:49 AEQP B734>
            #<EVENT 2012-11-15 16:18:49 ERD 2012-11-15 16:13:00>
            #<EVENT 2012-11-15 16:18:49 ERA 2012-11-15 18:29:00>
            #<EVENT 2012-11-15 16:18:51 ARD 2012-11-15 16:13:00>
            #<EVENT 2012-11-15 16:43:56 ERA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 16:49:55 ERA 2012-11-15 18:26:00>
            #<EVENT 2012-11-15 17:03:28 ERA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 17:10:38 ERA 2012-11-15 18:28:00>
            #<EVENT 2012-11-15 17:45:11 ERA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 17:47:24 ERA 2012-11-15 18:28:00>
            #<EVENT 2012-11-15 17:48:10 EGA 2012-11-15 18:30:00>
            #<EVENT 2012-11-15 17:48:10 ERA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 18:23:04 ERA 2012-11-15 18:29:00>
            #<EVENT 2012-11-15 18:29:01 STATUS L>
            #<EVENT 2012-11-15 18:29:01 AGA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 18:31:28 ERA 2012-11-15 18:26:00>
            #<EVENT 2012-11-15 18:33:12 ARA 2012-11-15 18:25:00>
            #<EVENT --- 2012-11-16 06:56:09 AGA 2012-11-16 01:27:00>
            #<EVENT --- 2012-11-16 06:56:09 ARA 2012-11-16 01:23:00>)
\r\n

Apparently the actual arrivals times are from the last two events. Unfortunately those happen to be garbage as evidenced by the landing event (STATUS L) and the asdiposition information. I'm not sure what the right solution is at the moment.

", "rawMarkdown": "", "votes": null }, { "id": "17919", "postDate": "12/16/2012 09:55:30", "content": "

Gabor,

\r\n

 

\r\n

I second that - found some unreliable lines in flighthistory.csv myself. Assuming all time columns are indeed in UTC (i.e. no need to normalize anymore),

\r\n

we have the following on 12.11.2012:

\r\n\r\n
Unless I'm missing sth obvious, I'm quite sure planes arrive at a runway first, and only then proceed to the gate?
\r\n
rg,
\r\n
K
", "rawMarkdown": "", "votes": null }, { "id": "17921", "postDate": "12/16/2012 10:19:52", "content": "

[quote=Konrad Banachewicz;17919]

\r\n

I second that - found some unreliable lines in flighthistory.csv myself.

\r\n

[/quote]

\r\n

Right. These are more benign though, because there is already a check in the test set generator to drop flights with actual_runway_arrival > actual_gate_arrival.

", "rawMarkdown": "", "votes": null }, { "id": "18071", "postDate": "12/17/2012 20:52:38", "content": "

There's nothing that we can do about the reliability of the data, since we are very far removed from the point at which the data was created. Any real world machine learning system needs to handle cases where the "ground truth" may be noisy or unreliable.\r\n Fortunately in this case, it appears that only a small portion of the samples have obvious issues.

\r\n

To address this, we are not asking you to predict any runway or gate arrival times that are either missing or clearly inconsistent (with gate arrival coming before runway arrival). Additionally, we are following Gabor's suggestion of filtering out flights\r\n where the runway departure occurred before the gate departure.

", "rawMarkdown": "", "votes": null }, { "id": "18487", "postDate": "12/23/2012 18:17:42", "content": "

so we just subtract 8 from every datetime in the flighthistoryevents to get the CST time. Correct?

\r\n

 

\r\n

 

\r\n

[quote=Ben Hamner;17802]

\r\n

[quote=Konrad Banachewicz;17776]

\r\n

Ben,

\r\n

Thanks for the clarification, two more points where I could use an explanation:

\r\n

1. has the normalization to utc been applied to other files (e.g. flighthistoryevents.csv) as well?

\r\n

2. that sort makes the departure_airport_timezone_offset and arrival_airport_timezone_offset columns redundant, right?

\r\n

 

\r\n

rg,

\r\n

K

\r\n

[/quote]Normalization hasn't ben applied to flighthistoryevents - times in the data_updated column are local to the corresponding airports, and the time the event was recorded is in pacific time.

\r\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "19936", "postDate": "02/01/2013 17:45:23", "content": "

[quote=Ben Hamner;18071]

\r\n

To address this, we are not asking you to predict any runway or gate arrival times that are either missing or clearly inconsistent (with gate arrival coming before runway arrival). Additionally, we are following Gabor's suggestion of filtering out flights\r\n where the runway departure occurred before the gate departure.

\r\n

[/quote]

\r\n

But what about the cases when gate arrival coming many hours later runway arrival?

\r\n

For example history 280181745 on 2012_11_12: gate arrival is 262 minutes later runway arrival.

\r\n

I can't imagine a company that kept passengers in the airplane for four hours after landing.

", "rawMarkdown": "", "votes": null }, { "id": "19967", "postDate": "02/03/2013 08:58:15", "content": "

[quote=Victor;19936]

\r\n

[quote=Ben Hamner;18071]

\r\n

To address this, we are not asking you to predict any runway or gate arrival times that are either missing or clearly inconsistent (with gate arrival coming before runway arrival). Additionally, we are following Gabor's suggestion of filtering out flights\r\n where the runway departure occurred before the gate departure.

\r\n

[/quote]

\r\n

But what about the cases when gate arrival coming many hours later runway arrival?

\r\n

For example history 280181745 on 2012_11_12: gate arrival is 262 minutes later runway arrival.

\r\n

I can't imagine a company that kept passengers in the airplane for four hours after landing.

\r\n

[/quote]We'll handle data points in the test set that appear to be inconsistent or extreme outliers on a case-by-case basis.

", "rawMarkdown": "", "votes": null }, { "id": "19985", "postDate": "02/04/2013 00:18:33", "content": "

[quote=Ben Hamner;19967]
\r\n
\r\nWe'll handle data points in the test set that appear to be inconsistent or extreme outliers on a case-by-case basis.
\r\n
\r\n[/quote]
\r\n
\r\nDo you plan to clean the data by hand? If so, this induces an additional layer
\r\nof uncertainty. Simpler models that are fit on such "error free" data will have
\r\nan advantage over more complex error-detection models. Hence, it is important to know how the
\r\ndata is cleaned and how it is generated. And it better be automatic and
\r\nreproducible.

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 17672, "author_name": "benhamner", "author_url": "", "post_date": "12/12/2012 02:43:17", "content": "

One common question involved the logic used to identify flights for the test set. This code is available in\r\n\r\nflighhistory.py on Github.

\r\n

The conditions for a flight to be included in the test set are as follows:

\r\n", "votes": null, "replies": [] }, { "id": 17700, "author_name": "konradb", "author_url": "", "post_date": "12/12/2012 11:13:51", "content": "

Hi,

\r\n

Just wanted to mention there still seem to be some problems with the training data actual arrival / departure times in flight history. Example:

\r\n

2012-11-12, flight_history_id: 280154062

\r\n
 actual_gate_departure actual_gate_arrival departure_airport_timezone_offset arrival_airport_timezone_offset
 2012-11-12 16:02:00+00:00 2012-11-12 16:59:00+00:00 -5 -6
\r\n
which, unless I'm mistaken, translates (after normalizing to the same timezone) into
\r\n
 (12-11-12 11:02:00) (12-11-12 10:59:00)
\r\n
I am reasonably confident a flight can not take -3 minutes, so I would appreciate some help - is it a data glitch, or am I missing something obvious
\r\n
 (misinterpreting the columns?).
\r\n
 
\r\n
rg,
\r\n
K
", "votes": null, "replies": [] }, { "id": 17747, "author_name": "benhamner", "author_url": "", "post_date": "12/12/2012 19:19:00", "content": "

The flighthistory.csv time columns are now all in UTC as specified in each timestamp, so the times are correct. The flight took 57 minutes to go from the departure gate to the arrival gate.

", "votes": null, "replies": [] }, { "id": 17754, "author_name": "melisgl", "author_url": "", "post_date": "12/12/2012 21:05:34", "content": "

Ben, in SampleTestSet the cutoff time for the first day is 2012-12-19 17:44 UTC. However, there are no events in flighthistoryevents.csv for that day later than 2012-11-19 10:00:00-08:00 (= 2012-11-19 02:00:00 UTC). Seems like the -8 was treated as +8?

", "votes": null, "replies": [] }, { "id": 17760, "author_name": "benhamner", "author_url": "", "post_date": "12/12/2012 22:21:28", "content": "

[quote=Gábor Melis;17754]

\r\n

Ben, in SampleTestSet the cutoff time for the first day is 2012-12-19 17:44 UTC. However, there are no events in flighthistoryevents.csv for that day later than 2012-11-19 10:00:00-08:00 (= 2012-11-19 02:00:00 UTC). Seems like the -8 was treated as +8?

\r\n

[/quote]

\r\n

Just looked into this. The latestevent in flighthistory.csv for that day is "2012-11-19 09:44:13.358000-08:00", which corresponds to "2012-11-19 17:44:13.358000+00:00" (UTC), which is correct. Keep in mind that the timezone offset -08 refers to Pacific Time,\r\n which is 8 hours before UTC (not after). You get UTC from the time zone offset by subtracting the offset, not adding it.

", "votes": null, "replies": [] }, { "id": 17761, "author_name": "benhamner", "author_url": "", "post_date": "12/12/2012 23:03:05", "content": "

[quote=Gábor Melis;17754]

\r\n

Ben, in SampleTestSet the cutoff time for the first day is 2012-12-19 17:44 UTC. However, there are no events in flighthistoryevents.csv for that day later than 2012-11-19 10:00:00-08:00 (= 2012-11-19 02:00:00 UTC). Seems like the -8 was treated as +8?

\r\n

[/quote]Thanks for checking this though. Let me know if you see any other inconsistencies or potential ways we could be leaking information - this is far easier to correct now than after December 18, when we will be releasing the Public Leaderboard Set.

", "votes": null, "replies": [] }, { "id": 17774, "author_name": "melisgl", "author_url": "", "post_date": "12/13/2012 08:48:00", "content": "

Thanks. Indeed, it was me who confused the signs.

", "votes": null, "replies": [] }, { "id": 17776, "author_name": "konradb", "author_url": "", "post_date": "12/13/2012 12:41:36", "content": "

Ben,

\r\n

Thanks for the clarification, two more points where I could use an explanation:

\r\n

1. has the normalization to utc been applied to other files (e.g. flighthistoryevents.csv) as well?

\r\n

2. that sort makes the departure_airport_timezone_offset and arrival_airport_timezone_offset columns redundant, right?

\r\n

 

\r\n

rg,

\r\n

K

", "votes": null, "replies": [] }, { "id": 17778, "author_name": "jimthompson", "author_url": "", "post_date": "12/13/2012 14:49:29", "content": "

[quote=Konrad Banachewicz;17776]

\r\n

2. that sort makes the departure_airport_timezone_offset and arrival_airport_timezone_offset columns redundant, right?

\r\n

[/quote]

\r\n

I don't think it is redundant.  I believe the offsets are needed if one wanted to account for the local daily cycle, i.e., morning/evening rush hours.

", "votes": null, "replies": [] }, { "id": 17785, "author_name": "konradb", "author_url": "", "post_date": "12/13/2012 15:49:18", "content": "

Fair point, haven't thought about that...

", "votes": null, "replies": [] }, { "id": 17802, "author_name": "benhamner", "author_url": "", "post_date": "12/13/2012 19:29:40", "content": "

[quote=Konrad Banachewicz;17776]

\r\n

Ben,

\r\n

Thanks for the clarification, two more points where I could use an explanation:

\r\n

1. has the normalization to utc been applied to other files (e.g. flighthistoryevents.csv) as well?

\r\n

2. that sort makes the departure_airport_timezone_offset and arrival_airport_timezone_offset columns redundant, right?

\r\n

 

\r\n

rg,

\r\n

K

\r\n

[/quote]Normalization hasn't ben applied to flighthistoryevents - times in the data_updated column are local to the corresponding airports, and the time the event was recorded is in pacific time.

", "votes": null, "replies": [] }, { "id": 17879, "author_name": "melisgl", "author_url": "", "post_date": "12/15/2012 17:06:09", "content": "

[quote=Ben Hamner;17672]

\r\n

One common question involved the logic used to identify flights for the test set. This code is available in\r\n\r\nflighhistory.py on Github.

\r\n

The conditions for a flight to be included in the test set are as follows:

\r\n\r\n

[/quote]

\r\n

There may be one more condition necessary: that actual_gate_departure is less than actual_runway_departure. There are flights that are considered to be in flight because the cutoff is in between that actual_runway_departure and actual_runway_arrival, but\r\n the actual_runway_arrival is wildly wrong. This flight made it to TestSampleSet:

\r\n
  HISTORY-ID = 281197158
  CREATOR-CODE = "I"
  DEPARTURE-AIRPORT = #<AIRPORT SFO/KSFO/-8>
  ARRIVAL-AIRPORT = #<AIRPORT FLL/KFLL/-5>
  PUBLISHED-DEPARTURE = "2012-11-24 07:55:00"
  PUBLISHED-ARRIVAL = "2012-11-24 13:10:00"
  ACTUAL-GATE-DEPARTURE = "2012-11-24 07:53:00"
  ACTUAL-RUNWAY-DEPARTURE = "2012-11-23 20:04:00"
  ACTUAL-RUNWAY-ARRIVAL = "2012-11-24 13:17:00"
  ACTUAL-GATE-ARRIVAL = "2012-11-24 13:24:00"
  EVENTS = (#<EVENT 2012-11-23 08:00:57 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-23 08:00:57 SRA 2012-11-24 13:15:00>
            #<EVENT 2012-11-23 08:00:57 ERD 2012-11-24 08:04:00>
            #<EVENT 2012-11-23 08:00:57 SRD 2012-11-24 08:04:00>
            #<EVENT 2012-11-24 00:00:01 AGATE C9>
            #<EVENT 2012-11-24 00:52:03 DGATE 50>
            #<EVENT 2012-11-24 05:52:34 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 05:53:13 EGA 2012-11-24 13:20:00>
            #<EVENT 2012-11-24 07:14:33 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 08:10:24 AEQP A319>
            #<EVENT 2012-11-24 08:10:24 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 08:10:24 ERD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 08:10:24 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 08:10:24 STATUS A>
            #<EVENT 2012-11-24 08:10:24 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 08:10:24 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 08:10:28 EGA 2012-11-24 13:22:00>
            #<EVENT --- 2012-11-24 08:10:28 ARD 2012-11-23 20:04:00>
            #<EVENT 2012-11-24 08:10:28 AGD 2012-11-24 07:53:00>
            #<EVENT 2012-11-24 08:18:38 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-24 08:18:38 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 08:23:29 EGA 2012-11-24 13:21:00>
            #<EVENT --- 2012-11-24 08:23:29 ARD 2012-11-23 20:04:00>
            #<EVENT 2012-11-24 08:25:42 ERA 2012-11-24 13:14:00>
            #<EVENT 2012-11-24 08:25:42 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 09:13:29 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-24 10:28:13 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 10:39:15 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-24 10:40:15 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 11:08:48 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 11:33:08 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 11:45:09 ERA 2012-11-24 13:18:00>
            #<EVENT 2012-11-24 11:53:17 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 12:17:30 ERA 2012-11-24 13:16:00>
            #<EVENT 2012-11-24 12:40:33 ERA 2012-11-24 13:17:00>
            #<EVENT 2012-11-24 12:48:05 ERA 2012-11-24 13:15:00>
            #<EVENT 2012-11-24 12:51:30 EGA 2012-11-24 13:22:00>
            #<EVENT --- 2012-11-24 12:51:30 ARD 2012-11-23 20:04:00>
            #<EVENT 2012-11-24 12:53:55 ERA 2012-11-24 13:11:00>
            #<EVENT --- 2012-11-24 12:53:55 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 12:54:34 ERA 2012-11-24 13:12:00>
            #<EVENT 2012-11-24 13:05:31 EGA 2012-11-24 13:23:00>
            #<EVENT --- 2012-11-24 13:05:31 ARD 2012-11-23 20:04:00>
            #<EVENT 2012-11-24 13:18:32 STATUS L>
            #<EVENT 2012-11-24 13:18:32 ARA 2012-11-24 13:17:00>
            #<EVENT --- 2012-11-24 13:25:18 ARD 2012-11-24 08:05:00>
            #<EVENT 2012-11-24 13:34:33 AGA 2012-11-24 13:24:00>
            #<EVENT --- 2012-11-24 13:34:33 ARD 2012-11-23 20:04:00>)
\r\n
 
\r\n

Suspicious events are marked with "---". The important part is this:

\r\n
  ACTUAL-GATE-DEPARTURE = "2012-11-24 07:53:00"
  ACTUAL-RUNWAY-DEPARTURE = "2012-11-23 20:04:00"
\r\n

If the cutoff is between 2012-11-23 20:04:00 and 2012-11-24 07:53:00, then there is almost no information on which to base the prediction.

", "votes": null, "replies": [] }, { "id": 17889, "author_name": "melisgl", "author_url": "", "post_date": "12/15/2012 20:14:41", "content": "

Even if the problem in the previous post is fixed, then we aren't golden yet. It seems that basing the quantities to be predicted on the unreliable flighthistoryevents.csv files could be problematic.

\r\n
  HISTORY-ID = 280480834
  CREATOR-CODE = "I"
  DEPARTURE-AIRPORT = #<AIRPORT PDX/KPDX/-8>
  ARRIVAL-AIRPORT = #<AIRPORT PHX/KPHX/-7>
  PUBLISHED-DEPARTURE = "2012-11-15 14:55:00"
  PUBLISHED-ARRIVAL = "2012-11-15 17:29:00"
  ACTUAL-GATE-DEPARTURE = "2012-11-15 16:03:00"
  ACTUAL-RUNWAY-DEPARTURE = "2012-11-15 16:13:00"
  ACTUAL-RUNWAY-ARRIVAL = "2012-11-16 01:23:00"
  ACTUAL-GATE-ARRIVAL = "2012-11-16 01:27:00"
  EVENTS = (#<EVENT 2012-11-14 03:34:51 TAIL N786AS>
            #<EVENT 2012-11-14 03:34:51 AGATE 11>
            #<EVENT 2012-11-14 03:34:51 EGA 2012-11-15 17:29:00>
            #<EVENT 2012-11-14 03:34:51 DGATE C5>
            #<EVENT 2012-11-14 03:34:51 EGD 2012-11-15 14:55:00>
            #<EVENT 2012-11-14 03:38:50 DGATE C21>
            #<EVENT 2012-11-14 03:57:03 DGATE B2>
            #<EVENT 2012-11-14 03:57:37 DGATE C21>
            #<EVENT 2012-11-14 15:01:05 ERA 2012-11-15 17:19:00>
            #<EVENT 2012-11-14 15:01:05 SRA 2012-11-15 17:19:00>
            #<EVENT 2012-11-14 15:01:05 ERD 2012-11-15 15:04:00>
            #<EVENT 2012-11-14 15:01:05 SRD 2012-11-15 15:04:00>
            #<EVENT 2012-11-14 21:20:44 TAIL N703AS>
            #<EVENT 2012-11-14 21:20:44 DGATE C7>
            #<EVENT 2012-11-15 08:35:31 EGA 2012-11-15 17:28:00>
            #<EVENT 2012-11-15 13:37:00 ERA 2012-11-15 17:21:00>
            #<EVENT 2012-11-15 13:37:00 ERD 2012-11-15 15:05:00>
            #<EVENT 2012-11-15 15:00:05 STATUS A>
            #<EVENT 2012-11-15 15:00:05 AGD 2012-11-15 14:48:00>
            #<EVENT 2012-11-15 15:06:27 EGA 2012-11-15 18:18:00>
            #<EVENT 2012-11-15 15:06:43 EGD 2012-11-15 15:45:00>
            #<EVENT 2012-11-15 15:13:05 STATUS R>
            #<EVENT 2012-11-15 15:44:00 EGA 2012-11-15 18:53:00>
            #<EVENT 2012-11-15 15:44:00 EGD 2012-11-15 16:20:00>
            #<EVENT 2012-11-15 15:53:48 STATUS S>
            #<EVENT 2012-11-15 15:53:48 ERA 2012-11-15 18:46:00>
            #<EVENT 2012-11-15 15:53:48 ERD 2012-11-15 16:30:00>
            #<EVENT 2012-11-15 16:00:06 STATUS R>
            #<EVENT 2012-11-15 16:13:07 STATUS A>
            #<EVENT 2012-11-15 16:13:07 AGD 2012-11-15 16:03:00>
            #<EVENT 2012-11-15 16:13:49 EGA 2012-11-15 18:34:00>
            #<EVENT 2012-11-15 16:18:49 AEQP B734>
            #<EVENT 2012-11-15 16:18:49 ERD 2012-11-15 16:13:00>
            #<EVENT 2012-11-15 16:18:49 ERA 2012-11-15 18:29:00>
            #<EVENT 2012-11-15 16:18:51 ARD 2012-11-15 16:13:00>
            #<EVENT 2012-11-15 16:43:56 ERA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 16:49:55 ERA 2012-11-15 18:26:00>
            #<EVENT 2012-11-15 17:03:28 ERA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 17:10:38 ERA 2012-11-15 18:28:00>
            #<EVENT 2012-11-15 17:45:11 ERA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 17:47:24 ERA 2012-11-15 18:28:00>
            #<EVENT 2012-11-15 17:48:10 EGA 2012-11-15 18:30:00>
            #<EVENT 2012-11-15 17:48:10 ERA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 18:23:04 ERA 2012-11-15 18:29:00>
            #<EVENT 2012-11-15 18:29:01 STATUS L>
            #<EVENT 2012-11-15 18:29:01 AGA 2012-11-15 18:27:00>
            #<EVENT 2012-11-15 18:31:28 ERA 2012-11-15 18:26:00>
            #<EVENT 2012-11-15 18:33:12 ARA 2012-11-15 18:25:00>
            #<EVENT --- 2012-11-16 06:56:09 AGA 2012-11-16 01:27:00>
            #<EVENT --- 2012-11-16 06:56:09 ARA 2012-11-16 01:23:00>)
\r\n

Apparently the actual arrivals times are from the last two events. Unfortunately those happen to be garbage as evidenced by the landing event (STATUS L) and the asdiposition information. I'm not sure what the right solution is at the moment.

", "votes": null, "replies": [] }, { "id": 17919, "author_name": "konradb", "author_url": "", "post_date": "12/16/2012 09:55:30", "content": "

Gabor,

\r\n

 

\r\n

I second that - found some unreliable lines in flighthistory.csv myself. Assuming all time columns are indeed in UTC (i.e. no need to normalize anymore),

\r\n

we have the following on 12.11.2012:

\r\n\r\n
Unless I'm missing sth obvious, I'm quite sure planes arrive at a runway first, and only then proceed to the gate?
\r\n
rg,
\r\n
K
", "votes": null, "replies": [] }, { "id": 17921, "author_name": "melisgl", "author_url": "", "post_date": "12/16/2012 10:19:52", "content": "

[quote=Konrad Banachewicz;17919]

\r\n

I second that - found some unreliable lines in flighthistory.csv myself.

\r\n

[/quote]

\r\n

Right. These are more benign though, because there is already a check in the test set generator to drop flights with actual_runway_arrival > actual_gate_arrival.

", "votes": null, "replies": [] }, { "id": 18071, "author_name": "benhamner", "author_url": "", "post_date": "12/17/2012 20:52:38", "content": "

There's nothing that we can do about the reliability of the data, since we are very far removed from the point at which the data was created. Any real world machine learning system needs to handle cases where the "ground truth" may be noisy or unreliable.\r\n Fortunately in this case, it appears that only a small portion of the samples have obvious issues.

\r\n

To address this, we are not asking you to predict any runway or gate arrival times that are either missing or clearly inconsistent (with gate arrival coming before runway arrival). Additionally, we are following Gabor's suggestion of filtering out flights\r\n where the runway departure occurred before the gate departure.

", "votes": null, "replies": [] }, { "id": 18487, "author_name": "rkirana", "author_url": "", "post_date": "12/23/2012 18:17:42", "content": "

so we just subtract 8 from every datetime in the flighthistoryevents to get the CST time. Correct?

\r\n

 

\r\n

 

\r\n

[quote=Ben Hamner;17802]

\r\n

[quote=Konrad Banachewicz;17776]

\r\n

Ben,

\r\n

Thanks for the clarification, two more points where I could use an explanation:

\r\n

1. has the normalization to utc been applied to other files (e.g. flighthistoryevents.csv) as well?

\r\n

2. that sort makes the departure_airport_timezone_offset and arrival_airport_timezone_offset columns redundant, right?

\r\n

 

\r\n

rg,

\r\n

K

\r\n

[/quote]Normalization hasn't ben applied to flighthistoryevents - times in the data_updated column are local to the corresponding airports, and the time the event was recorded is in pacific time.

\r\n

[/quote]

", "votes": null, "replies": [] }, { "id": 19936, "author_name": "nedelko", "author_url": "", "post_date": "02/01/2013 17:45:23", "content": "

[quote=Ben Hamner;18071]

\r\n

To address this, we are not asking you to predict any runway or gate arrival times that are either missing or clearly inconsistent (with gate arrival coming before runway arrival). Additionally, we are following Gabor's suggestion of filtering out flights\r\n where the runway departure occurred before the gate departure.

\r\n

[/quote]

\r\n

But what about the cases when gate arrival coming many hours later runway arrival?

\r\n

For example history 280181745 on 2012_11_12: gate arrival is 262 minutes later runway arrival.

\r\n

I can't imagine a company that kept passengers in the airplane for four hours after landing.

", "votes": null, "replies": [] }, { "id": 19967, "author_name": "benhamner", "author_url": "", "post_date": "02/03/2013 08:58:15", "content": "

[quote=Victor;19936]

\r\n

[quote=Ben Hamner;18071]

\r\n

To address this, we are not asking you to predict any runway or gate arrival times that are either missing or clearly inconsistent (with gate arrival coming before runway arrival). Additionally, we are following Gabor's suggestion of filtering out flights\r\n where the runway departure occurred before the gate departure.

\r\n

[/quote]

\r\n

But what about the cases when gate arrival coming many hours later runway arrival?

\r\n

For example history 280181745 on 2012_11_12: gate arrival is 262 minutes later runway arrival.

\r\n

I can't imagine a company that kept passengers in the airplane for four hours after landing.

\r\n

[/quote]We'll handle data points in the test set that appear to be inconsistent or extreme outliers on a case-by-case basis.

", "votes": null, "replies": [] }, { "id": 19985, "author_name": "vspinu", "author_url": "", "post_date": "02/04/2013 00:18:33", "content": "

[quote=Ben Hamner;19967]
\r\n
\r\nWe'll handle data points in the test set that appear to be inconsistent or extreme outliers on a case-by-case basis.
\r\n
\r\n[/quote]
\r\n
\r\nDo you plan to clean the data by hand? If so, this induces an additional layer
\r\nof uncertainty. Simpler models that are fit on such "error free" data will have
\r\nan advantage over more complex error-detection models. Hence, it is important to know how the
\r\ndata is cleaned and how it is generated. And it better be automatic and
\r\nreproducible.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "17669": "", "17672": "", "17700": "", "17747": "", "17754": "", "17760": "", "17761": "", "17774": "", "17776": "", "17778": "", "17785": "", "17802": "", "17879": "", "17889": "", "17919": "", "17921": "", "18071": "", "18487": "", "19936": "", "19967": "", "19985": "" }, "source": "meta" }