{
  "id": 6648,
  "title": "Third data release now available",
  "url": "/competitions/flight2-main/discussion/6648",
  "author_name": "",
  "post_date": "2013-12-20T22:40:19.277Z",
  "votes": null,
  "comment_count": 11,
  "views": 5206,
  "content": "<p>The third Flight Stats data release (Augmented Train Set 2) is now available for download from the <a href=\"https://www.gequest.com/c/flight2-main/data\">Data </a>page. The purpose of this data set is to provide information on flights closer in time to the final test data set due out in January. There is no leaderboard change associated with this data release.</p>",
  "messages": [
    {
      "id": "36431",
      "postDate": "12/20/2013 22:40:19",
      "content": "<p>The third Flight Stats data release (Augmented Train Set 2) is now available for download from the <a href=\"https://www.gequest.com/c/flight2-main/data\">Data </a>page. The purpose of this data set is to provide information on flights closer in time to the final test data set due out in January. There is no leaderboard change associated with this data release.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36732",
      "postDate": "12/27/2013 20:39:24",
      "content": "<p>Hi.<br>training3_metar_reports.csv seems to have all times wrong. Apparently UTC time (column 2) has been mis-translated into strange &quot;inverse&quot; time zones. All airports on the east coast apparently report times UTC-8 (even though they're reported +00), all airports on the west coast report times UTC-5 (again, written as +00), and of course the same happens for other time zones in between.</p>\n<p>I don't know if this is the only problem with this dataset (actually, no:&nbsp;https://www.gequest.com/c/flight2-main/forums/t/6055/simulator-issues-and-bug-reports/36622#post36622 and&nbsp;https://www.gequest.com/c/flight2-main/forums/t/6055/simulator-issues-and-bug-reports/36728#post36728), but it seems like its release was rushed in order to meet deadlines... Since the final milestone has already been postponed once, what about doing it again so we can find all the problems in this dataset and fix it?</p>\n<p>Thank you.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36736",
      "postDate": "12/27/2013 22:00:39",
      "content": "<p>[quote=Gabriele Seppi;36732]</p>\n<p>&nbsp;Since the final milestone has already been postponed once, what about doing it again so we can find all the problems in this dataset and fix it?</p>\n<p>[/quote]</p>\n<p>I would approve to this. It's somehow disappointing if we cannot be sure to interpret the input date correctly.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36762",
      "postDate": "12/28/2013 15:52:24",
      "content": "<p>Sorry everyone for the errors. These problems stem from the data provider. We will upload a corrected data set after the holidays and as soon as we can. We'll also consider moving dates around to account for this as well.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36902",
      "postDate": "01/02/2014 04:01:44",
      "content": "<p>I would also prefer some extra time to figure out all the problems with the new dataset. Thanks!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36930",
      "postDate": "01/02/2014 23:56:08",
      "content": "<p>[quote=Gabriele Seppi;36732]</p>\n<p>Hi.<br>training3_metar_reports.csv seems to have all times wrong. Apparently UTC time (column 2) has been mis-translated into strange &quot;inverse&quot; time zones. All airports on the east coast apparently report times UTC-8 (even though they're reported +00), all airports on the west coast report times UTC-5 (again, written as +00), and of course the same happens for other time zones in between.</p>\n<p>I don't know if this is the only problem with this dataset (actually, no:&nbsp;https://www.gequest.com/c/flight2-main/forums/t/6055/simulator-issues-and-bug-reports/36622#post36622 and&nbsp;https://www.gequest.com/c/flight2-main/forums/t/6055/simulator-issues-and-bug-reports/36728#post36728), but it seems like its release was rushed in order to meet deadlines... Since the final milestone has already been postponed once, what about doing it again so we can find all the problems in this dataset and fix it?</p>\n<p>Thank you.</p>\n<p>[/quote]</p>\n<p>Hi Gabriele,</p>\n<p>Sorry for the delay in getting back to you. We're now back in the office. Could you explain how you determined the time zone issue with metar_reports? I'm not sure if I see it so if you could take me through your analysis that would help me figure out where in the processing the error could have come up either on our end or with the data provider.</p>\n<p>Thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36939",
      "postDate": "01/03/2014 08:08:48",
      "content": "<p>Well, I didn't do a complete analysis, but I filtered the view for Atlanta and compared the&nbsp;date_time_issued field with the actual time in the raw metar report. When I noticed the discrepancy, I did the same thing for New York and then for Los Angeles in order to check the behavior on the west coast and that's when I realized that the time difference was &quot;reversed&quot;. To be more precise, it looks like raw metar times (which are GMT) are all converted into GMT-8, regardless of the actual time zone (which should not be taken into account, anyway).</p>\n<p>As an example, here are 3 metar reports:</p>\n<p>30.2,2013-11-14 09:51:00+00,17,F,F,245297290,KJFK 141751Z 24015KT 10SM FEW220 BKN270 10/M08 A3020 RMK AO2 SLP225 T01001083 10100 20011 58022,T01001083 10100 20011 58022,,30.19,AO2,50,,10,KJFK,240,,17.26</p>\n<p>30.4,2013-11-14 09:52:00+00,17,F,F,245297492,KATL 141752Z 10005KT 10SM FEW250 12/M08 A3040 RMK AO2 SLP301 T01221078 10122 21006 58024,T01221078 10122 21006 58024,,30.42,AO2,53,,10,KATL,100,,5.75</p>\n<p>29.91,2013-11-14 09:53:00+00,28,F,F,245297460,KLAX 141753Z 05004KT 10SM FEW180 30/M02 A2991 RMK AO2 SLP128 T03001017 10300 20167 50009 ,T03001017 10300 20167 50009,,29.91,AO2,86,,10,KLAX,50,,4.6</p>\n<p>I don't know if you generate the metar_reports.csv file starting from raw metars or you get complete data in this form, but to me it looks like a time conversion error somewhere along the way.</p>\n<p>Let me know if you need more details.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36954",
      "postDate": "01/03/2014 16:55:38",
      "content": "<p>[quote=Gabriele Seppi;36939]</p>\n<p>Well, I didn't do a complete analysis, but I filtered the view for Atlanta and compared the&nbsp;date_time_issued field with the actual time in the raw metar report.&nbsp;</p>\n<p>[/quote]</p>\n<p>What do you mean by &quot;the actual time in the raw metar report&quot;? What is the &quot;raw metar report&quot;? I understand that you are saying there is a time conversion issue, and we've seen the issue with this data set before and we thought we've caught most of them. I just need to know where you are finding the reference time to determine that date_time_issued in training3_metar_reports is off. Thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36958",
      "postDate": "01/03/2014 18:57:27",
      "content": "<p>I'll take the metar for KATL I reported above to explain better:</p>\n<p>30.4,2013-11-14 09:52:00+00,17,F,F,245297492,KATL 141752Z 10005KT 10SM FEW250 12/M08 A3040 RMK AO2 SLP301 T01221078 10122 21006 58024,T01221078 10122 21006 58024,,30.42,AO2,53,,10,KATL,100,,5.75</p>\n<p>date_time_issued is the second column here, so&nbsp;2013-11-14 09:52:00+00, that is 9:52AM GMT of November 14th.</p>\n<p>The raw metar report is the seventh column. Parsing that we get, after airport ICAO code,&nbsp;141752Z. That is 17:52 (5:52 PM) GMT of the 14th day of the month, November in this case.</p>\n<p>Hence, date_time_issued is equal to the metar report time - 8 hours.</p>\n<p>I hope it's clearer now.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36959",
      "postDate": "01/03/2014 19:18:45",
      "content": "<p>Ah, got it. The error needs to be fixed by the data provider. I have contacted them.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "36967",
      "postDate": "01/03/2014 21:11:16",
      "content": "<p>I hope this get corrected soon, because there's a real risk we get wrong data on the final leaderboard here...</p>\n<p>Thanks for the help.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "37060",
      "postDate": "01/06/2014 21:48:15",
      "content": "<p>The corrected METAR_REPORTS file is now available for download. Thanks for your patience while we obtained the correct version of the data from the provider.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 36732,
      "author_name": "gabrieleseppi",
      "author_url": "",
      "post_date": "12/27/2013 20:39:24",
      "content": "<p>Hi.<br>training3_metar_reports.csv seems to have all times wrong. Apparently UTC time (column 2) has been mis-translated into strange &quot;inverse&quot; time zones. All airports on the east coast apparently report times UTC-8 (even though they're reported +00), all airports on the west coast report times UTC-5 (again, written as +00), and of course the same happens for other time zones in between.</p>\n<p>I don't know if this is the only problem with this dataset (actually, no:&nbsp;https://www.gequest.com/c/flight2-main/forums/t/6055/simulator-issues-and-bug-reports/36622#post36622 and&nbsp;https://www.gequest.com/c/flight2-main/forums/t/6055/simulator-issues-and-bug-reports/36728#post36728), but it seems like its release was rushed in order to meet deadlines... Since the final milestone has already been postponed once, what about doing it again so we can find all the problems in this dataset and fix it?</p>\n<p>Thank you.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36736,
      "author_name": "derdasmachenmuss",
      "author_url": "",
      "post_date": "12/27/2013 22:00:39",
      "content": "<p>[quote=Gabriele Seppi;36732]</p>\n<p>&nbsp;Since the final milestone has already been postponed once, what about doing it again so we can find all the problems in this dataset and fix it?</p>\n<p>[/quote]</p>\n<p>I would approve to this. It's somehow disappointing if we cannot be sure to interpret the input date correctly.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36762,
      "author_name": "joycenv",
      "author_url": "",
      "post_date": "12/28/2013 15:52:24",
      "content": "<p>Sorry everyone for the errors. These problems stem from the data provider. We will upload a corrected data set after the holidays and as soon as we can. We'll also consider moving dates around to account for this as well.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36902,
      "author_name": "mlsword",
      "author_url": "",
      "post_date": "01/02/2014 04:01:44",
      "content": "<p>I would also prefer some extra time to figure out all the problems with the new dataset. Thanks!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36930,
      "author_name": "joycenv",
      "author_url": "",
      "post_date": "01/02/2014 23:56:08",
      "content": "<p>[quote=Gabriele Seppi;36732]</p>\n<p>Hi.<br>training3_metar_reports.csv seems to have all times wrong. Apparently UTC time (column 2) has been mis-translated into strange &quot;inverse&quot; time zones. All airports on the east coast apparently report times UTC-8 (even though they're reported +00), all airports on the west coast report times UTC-5 (again, written as +00), and of course the same happens for other time zones in between.</p>\n<p>I don't know if this is the only problem with this dataset (actually, no:&nbsp;https://www.gequest.com/c/flight2-main/forums/t/6055/simulator-issues-and-bug-reports/36622#post36622 and&nbsp;https://www.gequest.com/c/flight2-main/forums/t/6055/simulator-issues-and-bug-reports/36728#post36728), but it seems like its release was rushed in order to meet deadlines... Since the final milestone has already been postponed once, what about doing it again so we can find all the problems in this dataset and fix it?</p>\n<p>Thank you.</p>\n<p>[/quote]</p>\n<p>Hi Gabriele,</p>\n<p>Sorry for the delay in getting back to you. We're now back in the office. Could you explain how you determined the time zone issue with metar_reports? I'm not sure if I see it so if you could take me through your analysis that would help me figure out where in the processing the error could have come up either on our end or with the data provider.</p>\n<p>Thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36939,
      "author_name": "gabrieleseppi",
      "author_url": "",
      "post_date": "01/03/2014 08:08:48",
      "content": "<p>Well, I didn't do a complete analysis, but I filtered the view for Atlanta and compared the&nbsp;date_time_issued field with the actual time in the raw metar report. When I noticed the discrepancy, I did the same thing for New York and then for Los Angeles in order to check the behavior on the west coast and that's when I realized that the time difference was &quot;reversed&quot;. To be more precise, it looks like raw metar times (which are GMT) are all converted into GMT-8, regardless of the actual time zone (which should not be taken into account, anyway).</p>\n<p>As an example, here are 3 metar reports:</p>\n<p>30.2,2013-11-14 09:51:00+00,17,F,F,245297290,KJFK 141751Z 24015KT 10SM FEW220 BKN270 10/M08 A3020 RMK AO2 SLP225 T01001083 10100 20011 58022,T01001083 10100 20011 58022,,30.19,AO2,50,,10,KJFK,240,,17.26</p>\n<p>30.4,2013-11-14 09:52:00+00,17,F,F,245297492,KATL 141752Z 10005KT 10SM FEW250 12/M08 A3040 RMK AO2 SLP301 T01221078 10122 21006 58024,T01221078 10122 21006 58024,,30.42,AO2,53,,10,KATL,100,,5.75</p>\n<p>29.91,2013-11-14 09:53:00+00,28,F,F,245297460,KLAX 141753Z 05004KT 10SM FEW180 30/M02 A2991 RMK AO2 SLP128 T03001017 10300 20167 50009 ,T03001017 10300 20167 50009,,29.91,AO2,86,,10,KLAX,50,,4.6</p>\n<p>I don't know if you generate the metar_reports.csv file starting from raw metars or you get complete data in this form, but to me it looks like a time conversion error somewhere along the way.</p>\n<p>Let me know if you need more details.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36954,
      "author_name": "joycenv",
      "author_url": "",
      "post_date": "01/03/2014 16:55:38",
      "content": "<p>[quote=Gabriele Seppi;36939]</p>\n<p>Well, I didn't do a complete analysis, but I filtered the view for Atlanta and compared the&nbsp;date_time_issued field with the actual time in the raw metar report.&nbsp;</p>\n<p>[/quote]</p>\n<p>What do you mean by &quot;the actual time in the raw metar report&quot;? What is the &quot;raw metar report&quot;? I understand that you are saying there is a time conversion issue, and we've seen the issue with this data set before and we thought we've caught most of them. I just need to know where you are finding the reference time to determine that date_time_issued in training3_metar_reports is off. Thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36958,
      "author_name": "gabrieleseppi",
      "author_url": "",
      "post_date": "01/03/2014 18:57:27",
      "content": "<p>I'll take the metar for KATL I reported above to explain better:</p>\n<p>30.4,2013-11-14 09:52:00+00,17,F,F,245297492,KATL 141752Z 10005KT 10SM FEW250 12/M08 A3040 RMK AO2 SLP301 T01221078 10122 21006 58024,T01221078 10122 21006 58024,,30.42,AO2,53,,10,KATL,100,,5.75</p>\n<p>date_time_issued is the second column here, so&nbsp;2013-11-14 09:52:00+00, that is 9:52AM GMT of November 14th.</p>\n<p>The raw metar report is the seventh column. Parsing that we get, after airport ICAO code,&nbsp;141752Z. That is 17:52 (5:52 PM) GMT of the 14th day of the month, November in this case.</p>\n<p>Hence, date_time_issued is equal to the metar report time - 8 hours.</p>\n<p>I hope it's clearer now.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36959,
      "author_name": "joycenv",
      "author_url": "",
      "post_date": "01/03/2014 19:18:45",
      "content": "<p>Ah, got it. The error needs to be fixed by the data provider. I have contacted them.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 36967,
      "author_name": "gabrieleseppi",
      "author_url": "",
      "post_date": "01/03/2014 21:11:16",
      "content": "<p>I hope this get corrected soon, because there's a real risk we get wrong data on the final leaderboard here...</p>\n<p>Thanks for the help.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 37060,
      "author_name": "joycenv",
      "author_url": "",
      "post_date": "01/06/2014 21:48:15",
      "content": "<p>The corrected METAR_REPORTS file is now available for download. Thanks for your patience while we obtained the correct version of the data from the provider.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "36431": "",
    "36732": "",
    "36736": "",
    "36762": "",
    "36902": "",
    "36930": "",
    "36939": "",
    "36954": "",
    "36958": "",
    "36959": "",
    "36967": "",
    "37060": ""
  },
  "source": "meta"
}