{
  "id": 3368,
  "title": "Problems and Potential Problems with the Data",
  "url": "/competitions/flight/discussion/3368",
  "author_name": "",
  "post_date": "2012-12-17T09:07:01.937Z",
  "votes": null,
  "comment_count": 6,
  "views": 4524,
  "content": "<p>Instead of creating a new thread for each issue I suspect with the data, I'm going to keep all the posts under one thread.</p>\r\n<p>&nbsp;</p>\r\n<p>Please keep in mind that I'm going to post some of the issues which may not be actual bugs but patterns that might signal underlying issues.</p>",
  "messages": [
    {
      "id": "17996",
      "postDate": "12/17/2012 09:07:01",
      "content": "<p>Instead of creating a new thread for each issue I suspect with the data, I'm going to keep all the posts under one thread.</p>\r\n<p>&nbsp;</p>\r\n<p>Please keep in mind that I'm going to post some of the issues which may not be actual bugs but patterns that might signal underlying issues.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "17998",
      "postDate": "12/17/2012 09:17:44",
      "content": "<p><strong>Issue:</strong>&nbsp;Some ASDI Flight Plans entries indicate 1970-01-01 00:00:00 (unix time 0) for some timestamps. Is this indicative of &quot;Unavailable Data&quot;?</p>\r\n<p><strong>Solution:</strong> Awaiting response from admininstrators</p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;asdiflightplan.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_15\\ASDI\\asdiflightplan.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>311083</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>1239422,2012-11-16 13:09:52,280771891,KSDF,KGEG,UPS982,KSDF./.SEA089036..ZOOMR.ZOOMR1.KGEG,2012-11-16 08:23:00,2012-11-16 09:12:00,<strong>1970-01-01 00:00:00</strong>,2012-11-16 13:28:00</pre>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18000",
      "postDate": "12/17/2012 09:24:46",
      "content": "<p><strong>SOLVED</strong></p>\r\n<p><strong>Issue:</strong>&nbsp;A very small number (About 35)&nbsp;of Flight History Events have a timezone of -07. Is there an issue with the timezone adjustment script?</p>\r\n<p><strong>Solution:</strong>&nbsp;<span style=\"text-decoration:underline\">No timezones were modified in this file on Kaggle's side, so this simply reflects daylight savings time.</span></p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;flighthistoryevents.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_12\\FlightHistory\\flighthistoryevents.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>53176</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>279521153,2012-11-02 10:24:45.552<strong>-07</strong>,RecordCreated,</pre>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18007",
      "postDate": "12/17/2012 12:17:19",
      "content": "<p>[quote=Hesam Rabeti;18000]</p>\r\n<p><strong>Issue:</strong>&nbsp;A very small number (About 35)&nbsp;of Flight History Events have a timezone of -07. Is there an issue with the timezone adjustment script?</p>\r\n<p><strong>Solution:</strong>&nbsp;Awaiting response from admininstrators</p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;flighthistoryevents.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_12\\FlightHistory\\flighthistoryevents.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>53176</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>279521153,2012-11-02 10:24:45.552<strong>-07</strong>,RecordCreated,</pre>\r\n<p>[/quote]</p>\r\n<p>All those dates are under a different time saving regime, it's still PST. It's another question whether the dates are correct.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18021",
      "postDate": "12/17/2012 15:00:36",
      "content": "<p>[quote=Gábor Melis;18007]</p>\r\n<p>[quote=Hesam Rabeti;18000]</p>\r\n<p><strong>Issue:</strong>&nbsp;A very small number (About 35)&nbsp;of Flight History Events have a timezone of -07. Is there an issue with the timezone adjustment script?</p>\r\n<p><strong>Solution:</strong>&nbsp;Awaiting response from admininstrators</p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;flighthistoryevents.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_12\\FlightHistory\\flighthistoryevents.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>53176</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>279521153,2012-11-02 10:24:45.552<strong>-07</strong>,RecordCreated,</pre>\r\n<p>[/quote]</p>\r\n<p>All those dates are under a different time saving regime, it's still PST. It's another question whether the dates are correct.</p>\r\n<p>[/quote]</p>\r\n<p>I'm not an administrator but I suspect this issue and the one before it are due to Daylight savings time in the US.&nbsp; The date in the record, 2012-11-02, is in the period the US would have been observing Daylight savings time.&nbsp; If all the records with issues\r\n were created before 2012-11-04, then I think the times reflect Daylight savings time.</p>\r\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18073",
      "postDate": "12/17/2012 20:57:32",
      "content": "<p>[quote=Gábor Melis;18007]</p>\r\n<p>[quote=Hesam Rabeti;18000]</p>\r\n<p><strong>Issue:</strong>&nbsp;A very small number (About 35)&nbsp;of Flight History Events have a timezone of -07. Is there an issue with the timezone adjustment script?</p>\r\n<p><strong>Solution:</strong>&nbsp;Awaiting response from admininstrators</p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;flighthistoryevents.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_12\\FlightHistory\\flighthistoryevents.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>53176</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>279521153,2012-11-02 10:24:45.552<strong>-07</strong>,RecordCreated,</pre>\r\n<p>[/quote]</p>\r\n<p>All those dates are under a different time saving regime, it's still PST. It's another question whether the dates are correct.</p>\r\n<p>[/quote]No timezones were modified in this file on Kaggle's side, so this simply reflects daylight savings time.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18075",
      "postDate": "12/17/2012 21:06:15",
      "content": "<p>[quote=Ben Hamner;18073]</p>\r\n<p>No timezones were modified in this file on Kaggle's side, so this simply reflects daylight savings time.</p>\r\n<p>[/quote]</p>\r\n<p>Yup, that's what I intended to convey too. I certainly wasn't at my best phrasing that sentence.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 17998,
      "author_name": "rabeti",
      "author_url": "",
      "post_date": "12/17/2012 09:17:44",
      "content": "<p><strong>Issue:</strong>&nbsp;Some ASDI Flight Plans entries indicate 1970-01-01 00:00:00 (unix time 0) for some timestamps. Is this indicative of &quot;Unavailable Data&quot;?</p>\r\n<p><strong>Solution:</strong> Awaiting response from admininstrators</p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;asdiflightplan.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_15\\ASDI\\asdiflightplan.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>311083</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>1239422,2012-11-16 13:09:52,280771891,KSDF,KGEG,UPS982,KSDF./.SEA089036..ZOOMR.ZOOMR1.KGEG,2012-11-16 08:23:00,2012-11-16 09:12:00,<strong>1970-01-01 00:00:00</strong>,2012-11-16 13:28:00</pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18000,
      "author_name": "rabeti",
      "author_url": "",
      "post_date": "12/17/2012 09:24:46",
      "content": "<p><strong>SOLVED</strong></p>\r\n<p><strong>Issue:</strong>&nbsp;A very small number (About 35)&nbsp;of Flight History Events have a timezone of -07. Is there an issue with the timezone adjustment script?</p>\r\n<p><strong>Solution:</strong>&nbsp;<span style=\"text-decoration:underline\">No timezones were modified in this file on Kaggle's side, so this simply reflects daylight savings time.</span></p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;flighthistoryevents.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_12\\FlightHistory\\flighthistoryevents.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>53176</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>279521153,2012-11-02 10:24:45.552<strong>-07</strong>,RecordCreated,</pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18007,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "12/17/2012 12:17:19",
      "content": "<p>[quote=Hesam Rabeti;18000]</p>\r\n<p><strong>Issue:</strong>&nbsp;A very small number (About 35)&nbsp;of Flight History Events have a timezone of -07. Is there an issue with the timezone adjustment script?</p>\r\n<p><strong>Solution:</strong>&nbsp;Awaiting response from admininstrators</p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;flighthistoryevents.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_12\\FlightHistory\\flighthistoryevents.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>53176</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>279521153,2012-11-02 10:24:45.552<strong>-07</strong>,RecordCreated,</pre>\r\n<p>[/quote]</p>\r\n<p>All those dates are under a different time saving regime, it's still PST. It's another question whether the dates are correct.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18021,
      "author_name": "jimthompson",
      "author_url": "",
      "post_date": "12/17/2012 15:00:36",
      "content": "<p>[quote=Gábor Melis;18007]</p>\r\n<p>[quote=Hesam Rabeti;18000]</p>\r\n<p><strong>Issue:</strong>&nbsp;A very small number (About 35)&nbsp;of Flight History Events have a timezone of -07. Is there an issue with the timezone adjustment script?</p>\r\n<p><strong>Solution:</strong>&nbsp;Awaiting response from admininstrators</p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;flighthistoryevents.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_12\\FlightHistory\\flighthistoryevents.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>53176</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>279521153,2012-11-02 10:24:45.552<strong>-07</strong>,RecordCreated,</pre>\r\n<p>[/quote]</p>\r\n<p>All those dates are under a different time saving regime, it's still PST. It's another question whether the dates are correct.</p>\r\n<p>[/quote]</p>\r\n<p>I'm not an administrator but I suspect this issue and the one before it are due to Daylight savings time in the US.&nbsp; The date in the record, 2012-11-02, is in the period the US would have been observing Daylight savings time.&nbsp; If all the records with issues\r\n were created before 2012-11-04, then I think the times reflect Daylight savings time.</p>\r\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18073,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "12/17/2012 20:57:32",
      "content": "<p>[quote=Gábor Melis;18007]</p>\r\n<p>[quote=Hesam Rabeti;18000]</p>\r\n<p><strong>Issue:</strong>&nbsp;A very small number (About 35)&nbsp;of Flight History Events have a timezone of -07. Is there an issue with the timezone adjustment script?</p>\r\n<p><strong>Solution:</strong>&nbsp;Awaiting response from admininstrators</p>\r\n<p>&nbsp;</p>\r\n<p><strong>Filename:</strong>&nbsp;flighthistoryevents.csv</p>\r\n<p><strong>Example File:</strong>&nbsp;\\InitialTrainingSet_rev1\\2012_11_12\\FlightHistory\\flighthistoryevents.csv</p>\r\n<p><strong>Example Line:&nbsp;</strong>53176</p>\r\n<p><strong>Example Line Content:&nbsp;</strong></p>\r\n<pre>279521153,2012-11-02 10:24:45.552<strong>-07</strong>,RecordCreated,</pre>\r\n<p>[/quote]</p>\r\n<p>All those dates are under a different time saving regime, it's still PST. It's another question whether the dates are correct.</p>\r\n<p>[/quote]No timezones were modified in this file on Kaggle's side, so this simply reflects daylight savings time.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18075,
      "author_name": "melisgl",
      "author_url": "",
      "post_date": "12/17/2012 21:06:15",
      "content": "<p>[quote=Ben Hamner;18073]</p>\r\n<p>No timezones were modified in this file on Kaggle's side, so this simply reflects daylight savings time.</p>\r\n<p>[/quote]</p>\r\n<p>Yup, that's what I intended to convey too. I certainly wasn't at my best phrasing that sentence.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "17996": "",
    "17998": "",
    "18000": "",
    "18007": "",
    "18021": "",
    "18073": "",
    "18075": ""
  },
  "source": "meta"
}