{ "id": 5661, "title": "PAcific time & time zone of training1_asdiposition.csv", "url": "/competitions/flight2-milestone/discussion/5661", "author_name": "", "post_date": "2013-09-05T13:56:09.503Z", "votes": null, "comment_count": 3, "views": 1844, "content": "

Hi,

\n

I am wondering about the different times, also because it sees to me that FlightStatsDataDictionary.xlsx does not describe the zones right.

\n

 

\n

Question1) First of all what is pacific time, wikipedia says pacific time zone is UTC-8 and in daylight-saving UTC-7 (since the data is in June, it would be UTC-7 AND UTC-8, since if I remember correctly daylight-saving starts the 21. of June)

\n

 

\n

Question2) Short version: the times in training1_asdiposition.csv are said to be pacific, but they seem to rather UTC. Is this a mistake in the description?

\n

Long version and motivation of question2:

\n

Here is an example from one plane throughout the different files from FlightQuest2DataRelease1_AggregateTrain.7z:

\n

 

\n

You may SKIP all this and just look at my conclusion at the end, if you have not much time.

\n

 

\n

I pick a random entry from traning1_flighthistory.csv:

\n

299112639,DL,DAL,55,ATL,KATL,IAH,KIAH,2013-06-08 11:25:00+00,2013-06-08 13:28:00+00,2013-06-08 11:25:00+00,,2013-06-08 13:28:00+00,,2013-06-08 11:34:00+00,2013-06-08 11:33:00+00,2013-06-08 13:14:00+00,2013-06-08 13:10:00+00,I,100,123,-4,-5,319,,A319

\n

 

\n

the xlsx-file does not say anything about the time format here, but one can see that the airports are at UTC-4 and UTC-5 respectively.

\n

Then I look at the entries in training1_flighthistoryevents, xlsx says recorded times are in pacific time, it says nothing though about the format in data_updated field (NB:I truncated the full list of events for this flight):

\n

2558186,299112639,2013-06-05 01:04:00.672+00,RecordCreated,5246859,299112639,2013-06-07 11:30:59.193+00,Time Adjustment,"SRD- New=06/08/13 07:34, ERD- New=06/08/13 07:34, SRA- New=06/08/13 08:14, ERA- New=06/08/13 08:14"

\n

[...]

\n

6460097,299112639,2013-06-08 13:18:19.602+00,STATUS-Wheels Down,"ARA- New=06/08/13 08:10, STATUS- Old=A New=L"

\n

6840821,299112639,2013-06-08 21:26:20.525+00,Gate Adjustment,AGATE- New=15

\n

Then I compare to ScheduleRunwayDeparture from training1_flighthistory.csv (i.e. 2013-06-08 11:33:00+00) to the (first) one in data_update fields (SRD- New=06/08/13 07:34). Knowing (also from training1_flighthistory.csv) that the departure airport is at UTC-4 implies that datetime in training1_flighthistory.csv is UTC time and hence data_update_fields in training1_flighthistoryevents is local time of the airport (one can double check this by checking that the arrival times have indeed a UTC-5 shift, like entry in training1_flighthistory.csv says).

\n

 

\n

So far so good, then I look at the asdiflightplan (xlsx says everything a part recorded time (2nd col) is UTC, recorded time beeing pacific):

\n

81358634,2013-06-07 11:25:44+00,299112639,ATL,IAH,DAL55,KATL.JOGOR5.GARND..JYU..SJI.WOLDE3.KIAH,2013-06-08 11:25:00+00,2013-06-08 11:34:00+00,2013-06-08 13:28:00+00,2013-06-08 13:14:00+00

\n

Hence estimateddeparture_utc is 2013-06-08 11:34:00+00, which is in agreement with the preceeding conclusions.

\n

Then I look at training1_asdiposition.csv, xlsx says time received is in pacific time (NB: truncated output):

\n

6113801,2013-06-08 11:34:36+00,DAL55,3500,463,33.6300010681152,-84.4800033569336,299112639

\n

6114280,2013-06-08 11:34:52+00,DAL55,4000,210,33.6500015258789,-84.5,299112639

\n

6115739,2013-06-08 11:35:36+00,DAL55,6900,349,33.6500015258789,-84.5699996948242,299112639

\n

6116442,2013-06-08 11:35:58+00,DAL55,8500,267,33.6699981689453,-84.5999984741211,299112639

\n

6117670,2013-06-08 11:36:36+00,DAL55,10700,273,33.6699981689453,-84.6500015258789,299112639

\n

[...]

\n

 

\n

By looking at the altitude evolution (and also that it corresponds to the first 5 entries) it is clear that this is during the ascent. But comparing to all times in the other files, the time here smells a lot like UTC and not like pacific time. First entry is 2013-06-08 11:34:36+00, which I assume to be right after takeoff.

\n

CONCLUSION:

\n\n

Regards,

\n

Sandro

", "messages": [ { "id": "30229", "postDate": "09/05/2013 13:56:09", "content": "

Hi,

\n

I am wondering about the different times, also because it sees to me that FlightStatsDataDictionary.xlsx does not describe the zones right.

\n

 

\n

Question1) First of all what is pacific time, wikipedia says pacific time zone is UTC-8 and in daylight-saving UTC-7 (since the data is in June, it would be UTC-7 AND UTC-8, since if I remember correctly daylight-saving starts the 21. of June)

\n

 

\n

Question2) Short version: the times in training1_asdiposition.csv are said to be pacific, but they seem to rather UTC. Is this a mistake in the description?

\n

Long version and motivation of question2:

\n

Here is an example from one plane throughout the different files from FlightQuest2DataRelease1_AggregateTrain.7z:

\n

 

\n

You may SKIP all this and just look at my conclusion at the end, if you have not much time.

\n

 

\n

I pick a random entry from traning1_flighthistory.csv:

\n

299112639,DL,DAL,55,ATL,KATL,IAH,KIAH,2013-06-08 11:25:00+00,2013-06-08 13:28:00+00,2013-06-08 11:25:00+00,,2013-06-08 13:28:00+00,,2013-06-08 11:34:00+00,2013-06-08 11:33:00+00,2013-06-08 13:14:00+00,2013-06-08 13:10:00+00,I,100,123,-4,-5,319,,A319

\n

 

\n

the xlsx-file does not say anything about the time format here, but one can see that the airports are at UTC-4 and UTC-5 respectively.

\n

Then I look at the entries in training1_flighthistoryevents, xlsx says recorded times are in pacific time, it says nothing though about the format in data_updated field (NB:I truncated the full list of events for this flight):

\n

2558186,299112639,2013-06-05 01:04:00.672+00,RecordCreated,5246859,299112639,2013-06-07 11:30:59.193+00,Time Adjustment,"SRD- New=06/08/13 07:34, ERD- New=06/08/13 07:34, SRA- New=06/08/13 08:14, ERA- New=06/08/13 08:14"

\n

[...]

\n

6460097,299112639,2013-06-08 13:18:19.602+00,STATUS-Wheels Down,"ARA- New=06/08/13 08:10, STATUS- Old=A New=L"

\n

6840821,299112639,2013-06-08 21:26:20.525+00,Gate Adjustment,AGATE- New=15

\n

Then I compare to ScheduleRunwayDeparture from training1_flighthistory.csv (i.e. 2013-06-08 11:33:00+00) to the (first) one in data_update fields (SRD- New=06/08/13 07:34). Knowing (also from training1_flighthistory.csv) that the departure airport is at UTC-4 implies that datetime in training1_flighthistory.csv is UTC time and hence data_update_fields in training1_flighthistoryevents is local time of the airport (one can double check this by checking that the arrival times have indeed a UTC-5 shift, like entry in training1_flighthistory.csv says).

\n

 

\n

So far so good, then I look at the asdiflightplan (xlsx says everything a part recorded time (2nd col) is UTC, recorded time beeing pacific):

\n

81358634,2013-06-07 11:25:44+00,299112639,ATL,IAH,DAL55,KATL.JOGOR5.GARND..JYU..SJI.WOLDE3.KIAH,2013-06-08 11:25:00+00,2013-06-08 11:34:00+00,2013-06-08 13:28:00+00,2013-06-08 13:14:00+00

\n

Hence estimateddeparture_utc is 2013-06-08 11:34:00+00, which is in agreement with the preceeding conclusions.

\n

Then I look at training1_asdiposition.csv, xlsx says time received is in pacific time (NB: truncated output):

\n

6113801,2013-06-08 11:34:36+00,DAL55,3500,463,33.6300010681152,-84.4800033569336,299112639

\n

6114280,2013-06-08 11:34:52+00,DAL55,4000,210,33.6500015258789,-84.5,299112639

\n

6115739,2013-06-08 11:35:36+00,DAL55,6900,349,33.6500015258789,-84.5699996948242,299112639

\n

6116442,2013-06-08 11:35:58+00,DAL55,8500,267,33.6699981689453,-84.5999984741211,299112639

\n

6117670,2013-06-08 11:36:36+00,DAL55,10700,273,33.6699981689453,-84.6500015258789,299112639

\n

[...]

\n

 

\n

By looking at the altitude evolution (and also that it corresponds to the first 5 entries) it is clear that this is during the ascent. But comparing to all times in the other files, the time here smells a lot like UTC and not like pacific time. First entry is 2013-06-08 11:34:36+00, which I assume to be right after takeoff.

\n

CONCLUSION:

\n\n

Regards,

\n

Sandro

", "rawMarkdown": "", "votes": null }, { "id": "30836", "postDate": "09/13/2013 06:51:45", "content": "

112 views, but no one noticed the same (or tells me I am wrong, which would be fine too) :-)

", "rawMarkdown": "", "votes": null }, { "id": "31103", "postDate": "09/18/2013 17:26:17", "content": "

Hi sandros,

\n

I work for FlightStats so I can answer your questions with authority.  Thanks for the details which helped me track down the problem.  First, the short version:

\n

1) yes, PDT or PST

\n

2) no, training1_asdiposition.csv 'received' is in pacific time

\n

It looks like your problems stem from your import of flighthistory.csv into a database, and it exposes a problem in how we exported the data.  If you look at the raw data for the example that you used:

\n

299112639,DL,DAL,55,,ATL,KATL,IAH,KIAH,,,2013-06-08 07:25:00-07,2013-06-08 08:28:00-07,2013-06-08 07:25:00-07,,[...]

\n

There are incorrect timezone offsets on the times in that file.  The first datetime field claims an offset of -07 but ATL is -4.  Same thing with the next field (published_arrival) which claims -07 for IAH which should be -5.  Your import method needs to ignore or strip off those timezone offsets.  The times are actually in airport local time.  The correct offsets are available in the fields 'departure_airport_timezone_offset', 'arrival_airport_timezone_offset', and 'diverted_airport_timezone_offset'.  You can use the correct offsets to convert to UTC if you want.

\n

I think that if you make that change to your import procedure then everything will shake out correctly.

", "rawMarkdown": "", "votes": null }, { "id": "31117", "postDate": "09/18/2013 18:23:42", "content": "

Hi sandro and Luana,

\n

Sorry for the slow reply, and thanks for looking in to this.

\n

All of times you mention in flighthistory, asdiposition and asdiflightplan are in UTC. When a +00 is present, then that means UTC. We at Kaggle processed the raw FlightStats files so that it would be easier to work with by doing the time conversion for you. So the timezone_offset columns are not needed in the flighthistory table. (Yes, we should probably remove those columns to make it more clear.)

\n

Flighthistoryevents has local time since that is a steam of text and notes as the flight plan was being updated while in flight. So those times are local.

\n

Hope this helps, and sorry for the confusion!

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30836, "author_name": "sandros", "author_url": "", "post_date": "09/13/2013 06:51:45", "content": "

112 views, but no one noticed the same (or tells me I am wrong, which would be fine too) :-)

", "votes": null, "replies": [] }, { "id": 31103, "author_name": "luana126774", "author_url": "", "post_date": "09/18/2013 17:26:17", "content": "

Hi sandros,

\n

I work for FlightStats so I can answer your questions with authority.  Thanks for the details which helped me track down the problem.  First, the short version:

\n

1) yes, PDT or PST

\n

2) no, training1_asdiposition.csv 'received' is in pacific time

\n

It looks like your problems stem from your import of flighthistory.csv into a database, and it exposes a problem in how we exported the data.  If you look at the raw data for the example that you used:

\n

299112639,DL,DAL,55,,ATL,KATL,IAH,KIAH,,,2013-06-08 07:25:00-07,2013-06-08 08:28:00-07,2013-06-08 07:25:00-07,,[...]

\n

There are incorrect timezone offsets on the times in that file.  The first datetime field claims an offset of -07 but ATL is -4.  Same thing with the next field (published_arrival) which claims -07 for IAH which should be -5.  Your import method needs to ignore or strip off those timezone offsets.  The times are actually in airport local time.  The correct offsets are available in the fields 'departure_airport_timezone_offset', 'arrival_airport_timezone_offset', and 'diverted_airport_timezone_offset'.  You can use the correct offsets to convert to UTC if you want.

\n

I think that if you make that change to your import procedure then everything will shake out correctly.

", "votes": null, "replies": [] }, { "id": 31117, "author_name": "joycenv", "author_url": "", "post_date": "09/18/2013 18:23:42", "content": "

Hi sandro and Luana,

\n

Sorry for the slow reply, and thanks for looking in to this.

\n

All of times you mention in flighthistory, asdiposition and asdiflightplan are in UTC. When a +00 is present, then that means UTC. We at Kaggle processed the raw FlightStats files so that it would be easier to work with by doing the time conversion for you. So the timezone_offset columns are not needed in the flighthistory table. (Yes, we should probably remove those columns to make it more clear.)

\n

Flighthistoryevents has local time since that is a steam of text and notes as the flight plan was being updated while in flight. So those times are local.

\n

Hope this helps, and sorry for the confusion!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30229": "", "30836": "", "31103": "", "31117": "" }, "source": "meta" }