I want to confirm my interpretation of the date/time values in the flighthistory.csv dataset.
\r\nValues are given as "2012-11-21 17:40:00+00:00" I believe this is UTC.
\r\nTo determine the locale time for the departure airport or arrival airport, I add the values from the "departure_airport_timezone_offset" or "arrival_airport_timezone_offset" columns, respectively, to the UTC time.
\r\nIs this correct?
", "messages": [ { "id": "17278", "postDate": "11/30/2012 17:54:29", "content": "I want to confirm my interpretation of the date/time values in the flighthistory.csv dataset.
\r\nValues are given as "2012-11-21 17:40:00+00:00" I believe this is UTC.
\r\nTo determine the locale time for the departure airport or arrival airport, I add the values from the "departure_airport_timezone_offset" or "arrival_airport_timezone_offset" columns, respectively, to the UTC time.
\r\nIs this correct?
", "rawMarkdown": "", "votes": null }, { "id": "17282", "postDate": "11/30/2012 19:33:20", "content": "I'm guessing here, but based on the data it looks to me like the date/time values are in local time.\r\n
\r\nYou can back into the scheduled air time by taking the difference between the scheduled runway departure and the scheduled runway arrival. That difference will NOT match up with the scheduled air time on flights where the departure timezone offset is different\r\n from the arrival timezone offset. If you adjust the departure and arrival times by the timezone offsets first, then the scheduled air time will always match the difference between those adjusted times.
", "rawMarkdown": "", "votes": null }, { "id": "17286", "postDate": "11/30/2012 19:53:16", "content": "Ben, if that's the case, can you explain why the 'scheduled air time' does not match the difference between the scheduled departure and arrival times on flights that cross time zones?
", "rawMarkdown": "", "votes": null }, { "id": "17290", "postDate": "11/30/2012 20:36:40", "content": "There does seem to be an error. Compare the flight number with that found in asdiflightplan.csv, which lists the departure and arrival times in UTC. I just looked at a couple entries, but if the UTC time in asdiflightplan.csv is to be trusted, then the values\r\n in flighthistory.csv are off. It appears that the departure times are incremented by the number of hours' difference in timezone from departure to arrival.
\r\nFor instance, flight 280190244 from 11/12/2012 has a UTC departure listed from asdiflightplan of 10:59PM on 11/12/2012. The departure time listed in flighthistory is 1:59AM on 11/13/2012, and the timezone at departure is -5 and at arrival -8. So the flighthistory\r\n departure time is increased by 3 hours (the difference between -5 and -8) over the UTC time. I suspect that this adjustment was made because they assumed that the departure and arrival times were at local times, and they wanted to make sure everything was\r\n in UTC. In fact, they were already at UTC, so the adjustment threw them off.
", "rawMarkdown": "", "votes": null }, { "id": "17291", "postDate": "11/30/2012 21:10:04", "content": "Yes -- that looks plausible. Unfortunately that would mean that the fields we are trying to predict are being misrepresented. Should we try to predict the arrival times in UTC, in local-time, or in nonsense-time that is consistent with the data set?
", "rawMarkdown": "", "votes": null }, { "id": "17298", "postDate": "11/30/2012 21:52:34", "content": "[quote=Daniel Lupton;17290]
\r\nThere does seem to be an error. Compare the flight number with that found in asdiflightplan.csv, which lists the departure and arrival times in UTC. I just looked at a couple entries, but if the UTC time in asdiflightplan.csv is to be trusted, then the values\r\n in flighthistory.csv are off. It appears that the departure times are incremented by the number of hours' difference in timezone from departure to arrival.
\r\n[/quote]Thanks for pointing this out and we apologize for the issue.
\r\nI can confirm that they are in local time (not UTC). We will be releasing an updated version of the dataset that accounts for this issue and includes a sample of the test set format between now and the release of the test set. Please let us know if you come\r\n across any other discrepancies in the data as well.
", "rawMarkdown": "", "votes": null }, { "id": "17330", "postDate": "12/02/2012 02:51:25", "content": "There seems to be duplicate values in "\\InitialTrainingSet\\2012_11_12\\ASDI\\asdiposition.csv" (There are duplicates in other days too)\r\n
\r\n
\tLine 534: 2012-11-12 04:23:26-08,ASH2648,14500,356,34.4000015258789,-79.9499969482422,280206428\r\n
\tLine 545: 2012-11-12 04:23:26-08,ASH2648,14300,367,34.4199981689453,-79.9700012207031,280206428
\r\n
I'm claiming these are duplicates because they are reporting the same flighthistoryid, time, and tailnumber.\r\n
Is this intentional? Should we take the average of the two values?\r\n
\r\n
Also, what does the '-08' at the end of the timestamp mean? Are we talking about UTC-08?\r\n
\r\n
2012-11-12 04:23:26-08,ASH2648,14500,356,34.4000015258789,-79.9499969482422,280206428",
"rawMarkdown": "",
"votes": null
},
{
"id": "17335",
"postDate": "12/02/2012 07:39:02",
"content": "I still confused by different date/time values in different files.
\r\nFor example, flight_history_id 280845188 in file FlightHistory/flighthistory.csv from 11/18/2012, the published departure and arrival time are 14:05:00 and15:25:00(this two are local time). Timezone offset at departure is -5 and at arrival is -6. but in file\r\n ASDI/asdiflightplan.csv the departure and arrival time are 11:05:00,13:25:00(UTC time as described in file).\r\n
\r\nThe question is how to convert the local time to UTC time?
\r\nDoes other date/time values in other files(e.g, the start and end time in flightstats_atsccdelay.csv) are UTC format or they are local time?
Flight with flight historyid = 280818201 (DAL120 from JFK to LAX)
\r\nIn asdiflightplan is given as: estimatedarrivalutc = 20/11/2012 7:38:00 PM
\r\nIn asdiposition is given as: received = 21/11/2012 6:38:16 AM
\r\n- the last entry for this flight at an altitide of 100, which I'm guessing is when it touched down (and at a lat and long which puts it approx 600m south of the southern most runway at LAX - at 1221 E Acacia Ave, El\r\n Segundo, CA according to google maps!!).
\r\nHow do the times in asdiposition relate to those in asdiflightplan and/or flighthistory?
", "rawMarkdown": "", "votes": null }, { "id": "17406", "postDate": "12/05/2012 14:36:33", "content": "[quote=Ben Hamner;17298]
\r\nI can confirm that they are in local time (not UTC). We will be releasing an updated version of the dataset that accounts for this issue and includes a sample of the test set format between now and the release of the test set. Please let us know if you come\r\n across any other discrepancies in the data as well.
\r\n[/quote]
\r\nBen...any ETA for the "updated version of the dataset"?
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 17282, "author_name": "jonrodney", "author_url": "", "post_date": "11/30/2012 19:33:20", "content": "I'm guessing here, but based on the data it looks to me like the date/time values are in local time.\r\n
\r\nYou can back into the scheduled air time by taking the difference between the scheduled runway departure and the scheduled runway arrival. That difference will NOT match up with the scheduled air time on flights where the departure timezone offset is different\r\n from the arrival timezone offset. If you adjust the departure and arrival times by the timezone offsets first, then the scheduled air time will always match the difference between those adjusted times.
", "votes": null, "replies": [] }, { "id": 17286, "author_name": "jonrodney", "author_url": "", "post_date": "11/30/2012 19:53:16", "content": "Ben, if that's the case, can you explain why the 'scheduled air time' does not match the difference between the scheduled departure and arrival times on flights that cross time zones?
", "votes": null, "replies": [] }, { "id": 17290, "author_name": "daniellupton", "author_url": "", "post_date": "11/30/2012 20:36:40", "content": "There does seem to be an error. Compare the flight number with that found in asdiflightplan.csv, which lists the departure and arrival times in UTC. I just looked at a couple entries, but if the UTC time in asdiflightplan.csv is to be trusted, then the values\r\n in flighthistory.csv are off. It appears that the departure times are incremented by the number of hours' difference in timezone from departure to arrival.
\r\nFor instance, flight 280190244 from 11/12/2012 has a UTC departure listed from asdiflightplan of 10:59PM on 11/12/2012. The departure time listed in flighthistory is 1:59AM on 11/13/2012, and the timezone at departure is -5 and at arrival -8. So the flighthistory\r\n departure time is increased by 3 hours (the difference between -5 and -8) over the UTC time. I suspect that this adjustment was made because they assumed that the departure and arrival times were at local times, and they wanted to make sure everything was\r\n in UTC. In fact, they were already at UTC, so the adjustment threw them off.
", "votes": null, "replies": [] }, { "id": 17291, "author_name": "jonrodney", "author_url": "", "post_date": "11/30/2012 21:10:04", "content": "Yes -- that looks plausible. Unfortunately that would mean that the fields we are trying to predict are being misrepresented. Should we try to predict the arrival times in UTC, in local-time, or in nonsense-time that is consistent with the data set?
", "votes": null, "replies": [] }, { "id": 17298, "author_name": "benhamner", "author_url": "", "post_date": "11/30/2012 21:52:34", "content": "[quote=Daniel Lupton;17290]
\r\nThere does seem to be an error. Compare the flight number with that found in asdiflightplan.csv, which lists the departure and arrival times in UTC. I just looked at a couple entries, but if the UTC time in asdiflightplan.csv is to be trusted, then the values\r\n in flighthistory.csv are off. It appears that the departure times are incremented by the number of hours' difference in timezone from departure to arrival.
\r\n[/quote]Thanks for pointing this out and we apologize for the issue.
\r\nI can confirm that they are in local time (not UTC). We will be releasing an updated version of the dataset that accounts for this issue and includes a sample of the test set format between now and the release of the test set. Please let us know if you come\r\n across any other discrepancies in the data as well.
", "votes": null, "replies": [] }, { "id": 17330, "author_name": "rabeti", "author_url": "", "post_date": "12/02/2012 02:51:25", "content": "There seems to be duplicate values in "\\InitialTrainingSet\\2012_11_12\\ASDI\\asdiposition.csv" (There are duplicates in other days too)\r\n
\r\n
\tLine 534: 2012-11-12 04:23:26-08,ASH2648,14500,356,34.4000015258789,-79.9499969482422,280206428\r\n
\tLine 545: 2012-11-12 04:23:26-08,ASH2648,14300,367,34.4199981689453,-79.9700012207031,280206428
\r\n
I'm claiming these are duplicates because they are reporting the same flighthistoryid, time, and tailnumber.\r\n
Is this intentional? Should we take the average of the two values?\r\n
\r\n
Also, what does the '-08' at the end of the timestamp mean? Are we talking about UTC-08?\r\n
\r\n
2012-11-12 04:23:26-08,ASH2648,14500,356,34.4000015258789,-79.9499969482422,280206428",
"votes": null,
"replies": []
},
{
"id": 17335,
"author_name": "rongkaixia",
"author_url": "",
"post_date": "12/02/2012 07:39:02",
"content": "I still confused by different date/time values in different files.
\r\nFor example, flight_history_id 280845188 in file FlightHistory/flighthistory.csv from 11/18/2012, the published departure and arrival time are 14:05:00 and15:25:00(this two are local time). Timezone offset at departure is -5 and at arrival is -6. but in file\r\n ASDI/asdiflightplan.csv the departure and arrival time are 11:05:00,13:25:00(UTC time as described in file).\r\n
\r\nThe question is how to convert the local time to UTC time?
\r\nDoes other date/time values in other files(e.g, the start and end time in flightstats_atsccdelay.csv) are UTC format or they are local time?
Flight with flight historyid = 280818201 (DAL120 from JFK to LAX)
\r\nIn asdiflightplan is given as: estimatedarrivalutc = 20/11/2012 7:38:00 PM
\r\nIn asdiposition is given as: received = 21/11/2012 6:38:16 AM
\r\n- the last entry for this flight at an altitide of 100, which I'm guessing is when it touched down (and at a lat and long which puts it approx 600m south of the southern most runway at LAX - at 1221 E Acacia Ave, El\r\n Segundo, CA according to google maps!!).
\r\nHow do the times in asdiposition relate to those in asdiflightplan and/or flighthistory?
", "votes": null, "replies": [] }, { "id": 17406, "author_name": "jimthompson", "author_url": "", "post_date": "12/05/2012 14:36:33", "content": "[quote=Ben Hamner;17298]
\r\nI can confirm that they are in local time (not UTC). We will be releasing an updated version of the dataset that accounts for this issue and includes a sample of the test set format between now and the release of the test set. Please let us know if you come\r\n across any other discrepancies in the data as well.
\r\n[/quote]
\r\nBen...any ETA for the "updated version of the dataset"?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "17278": "", "17282": "", "17286": "", "17290": "", "17291": "", "17298": "", "17330": "", "17335": "", "17399": "", "17406": "" }, "source": "meta" }