{
  "id": 3444,
  "title": "Useful data",
  "url": "/competitions/flight/discussion/3444",
  "author_name": "",
  "post_date": "2012-12-22T18:07:35.280Z",
  "votes": 9,
  "comment_count": 13,
  "views": 9561,
  "content": "<p>Hi everyone,</p>\r\n<p>There's a lot of data in this competition, and I thought it would be useful if people could post exactly what data they were using in their models. &nbsp;It would be helpful for those just starting out, as well as those looking to add new features to their models.</p>\r\n<p>&nbsp;</p>\r\n<p>I'm currently using data from the files:</p>\r\n<p>flighthistory.csv &nbsp;(Pretty much everything here is useful)</p>\r\n<p>flighthistoryevents.csv &nbsp;(Extract most recent estimated runway arrival, estimated gate arrival, and arrival gate)</p>\r\n<p>asdiflightplan.csv &nbsp;(Extract most recent estimated arrival)</p>\r\n<p>weather_stations.csv (Extract airport locations)</p>\r\n<p>&nbsp;</p>\r\n<p>- updated 12/27&nbsp;</p>",
  "messages": [
    {
      "id": "18434",
      "postDate": "12/22/2012 18:07:35",
      "content": "<p>Hi everyone,</p>\r\n<p>There's a lot of data in this competition, and I thought it would be useful if people could post exactly what data they were using in their models. &nbsp;It would be helpful for those just starting out, as well as those looking to add new features to their models.</p>\r\n<p>&nbsp;</p>\r\n<p>I'm currently using data from the files:</p>\r\n<p>flighthistory.csv &nbsp;(Pretty much everything here is useful)</p>\r\n<p>flighthistoryevents.csv &nbsp;(Extract most recent estimated runway arrival, estimated gate arrival, and arrival gate)</p>\r\n<p>asdiflightplan.csv &nbsp;(Extract most recent estimated arrival)</p>\r\n<p>weather_stations.csv (Extract airport locations)</p>\r\n<p>&nbsp;</p>\r\n<p>- updated 12/27&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18435",
      "postDate": "12/22/2012 18:40:49",
      "content": "Same files (with tweaking to account for flights absent from asdi files), only test data - until the rules conflict thing is resolved by the admins, i am leaving the training stuff aside so that each day is based on the pre/post cutoff (= train/ test split)\r\n for this day only.",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18442",
      "postDate": "12/22/2012 22:20:29",
      "content": "<p>Those are the files I'm using as well.&nbsp;</p>\r\n<p>I have another, somewhat related question. How are you dealing with records in the training set where the actual gate arrival occurs before the actual runway arrival?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18449",
      "postDate": "12/23/2012 00:06:05",
      "content": "<p>ricardo sachez, you should remove these records as&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Ben Hamner;17672]</p>\r\n<p>One common question involved the logic used to identify flights for the test set. This code is available in&nbsp;<a href=\"https://github.com/benhamner/GEFlightQuest/blob/master/PythonModule/geflight/transform/flighthistory.py#L288\">flighhistory.py on Github</a>.</p>\r\n<p>The conditions for a flight to be included in the test set are as follows:</p>\r\n<ul>\r\n<li>It took off and landed at a continental US airport </li><li>We received data for the actual gate and runway departure and arrival times </li><li>The runway arrival time was less than the gate arrival time </li><li>The flight was in the air at the cutoff time for the day </li></ul>\r\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18467",
      "postDate": "12/23/2012 13:39:04",
      "content": "<p>Thanks, Julien. I guess their script isn't working exactly as advertised.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18475",
      "postDate": "12/23/2012 16:24:57",
      "content": "<p>Decyphering the Rosetta stone would be easier than trying to interpret the structure of the provided data and hence this competition. Also did I read things correctly that for a submission to be considered as being valid, code and instructions needs to be\r\n made available upfront?</p>\r\n<p>MODEL SUBMISSION:<br>\r\nWhen you submit an Entry, you also will be able to upload your models to the Website.<br>\r\nYour final model submission must contain all code and parameter settings necessary to<br>\r\nevaluate your models on new data, and include a README file with instructions on how<br>\r\nto do so.</p>\r\n<p>WTF.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18489",
      "postDate": "12/23/2012 18:48:38",
      "content": "<p>so you are using only these events from flighthistoryevents</p>\r\n<p>#STATUS-Landed</p>\r\n<p>#STATUS-Wheels Down&nbsp;</p>\r\n<p># RecordCreated&nbsp;</p>\r\n<p># &nbsp;STATUS-Active&nbsp;</p>\r\n<p># Gate Adjustment&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>Which of the above indicate arrival and departure - especially runway and gate?</p>\r\n<p>Again poor documentation is not helping anyone</p>\r\n<p>[quote=Lonely at the Top;18434]</p>\r\n<p>Hi everyone,</p>\r\n<p>There's a lot of data in this competition, and I thought it would be useful if people could post exactly what data they were using in their models. &nbsp;It would be helpful for those just starting out, as well as those looking to add new features to their models.</p>\r\n<p>&nbsp;</p>\r\n<p>I'm currently using data from the files:</p>\r\n<p>flighthistory.csv &nbsp;(Pretty much everything here is useful)</p>\r\n<p>flighthistoryevents.csv &nbsp;(Extract most recent estimated runway arrival, estimated gate arrival, and arrival gate)</p>\r\n<p>asdiflightplan.csv &nbsp;(Extract most recent estimated arrival)</p>\r\n<p>The most useful entries to me right now are the estimated runway/gate arrival times.</p>\r\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18490",
      "postDate": "12/23/2012 18:49:51",
      "content": "<p>where are the splits available in training data?&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Konrad Banachewicz;18435]</p>\r\n<p>Same files (with tweaking to account for flights absent from asdi files), only test data - until the rules conflict thing is resolved by the admins, i am leaving the training stuff aside so that each day is based on the pre/post cutoff (= train/ test split)\r\n for this day only.</p>\r\n<p>[/quote]</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18493",
      "postDate": "12/23/2012 19:22:06",
      "content": "<p>test flights = clearly defined, training flights = all other flights in the publicLeaderboard data folder for this day.&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18495",
      "postDate": "12/23/2012 19:37:11",
      "content": "<p>[quote=Black Magic;18489]</p>\r\n<p>so you are using only these events from flighthistoryevents</p>\r\n<p>#STATUS-Landed</p>\r\n<p>#STATUS-Wheels Down&nbsp;</p>\r\n<p># RecordCreated&nbsp;</p>\r\n<p># &nbsp;STATUS-Active&nbsp;</p>\r\n<p># Gate Adjustment&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>Which of the above indicate arrival and departure - especially runway and gate?</p>\r\n<p>Again poor documentation is not helping anyone</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>For the flighthistoryevents file, I scan through looking for:</p>\r\n<p>ERA</p>\r\n<p>EGA</p>\r\n<p>AGATE</p>\r\n<p>After I find one of these, I look for &quot;New&quot; after it, then pull out the time or gate which appears after it. &nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>So far I've spent most of my time on this project working on data input ...</p>\r\n<p>&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18510",
      "postDate": "12/24/2012 00:19:57",
      "content": "<p>In addition I join the latest available Metar data (flightstats<em>_</em>metarreports_combined.csv) to each flight by the arrival airport's icao code. (That helped a little, but not much.)</p>\r\n<p>Still working on joining the asdiflightplan, and I have a question there:<br>\r\nIn the asdiflightplan.csv for 2012-11-12 I see only date/time values from the next day.<br>\r\n(estimatedarrivalutc is between 2012-11-13 02:50:00 and 2012-11-13 14:43:00)</p>\r\n<p>What am I missing?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18515",
      "postDate": "12/24/2012 03:43:52",
      "content": "<p>also looks like the times are different:</p>\r\n<p>data<em>updated date</em>time<em>recorded</em>old<br>\r\nSTATUS- Old=S New=U 2012-11-13 10:33:02.358-08 2012-11-12 16:20:20.573-08<br>\r\nARA- New=11/12/12 09:41, STATUS- Old=A New=L 2012-11-12 09:49:29.343-08<br>\r\nERA- Old=11/12/12 10:27 New=11/12/12 09:41 2012-11-12 09:46:57.819-08 <br>\r\nERA- Old=11/12/12 09:38 New=11/12/12 10:27 2012-11-12 09:46:22.529-08</p>\r\n<p>Looks like the time recorded column has a different date than what is in data updated. What is the right one?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18574",
      "postDate": "12/25/2012 10:50:46",
      "content": "<p>[quote=Lonely at the Top;18434]</p>\r\n<p>Hi everyone,</p>\r\n<p>There's a lot of data in this competition, and I thought it would be useful if people could post exactly what data they were using in their models. &nbsp;It would be helpful for those just starting out, as well as those looking to add new features to their models.</p>\r\n<p>&nbsp;</p>\r\n<p>I'm currently using data from the files:</p>\r\n<p>flighthistory.csv &nbsp;(Pretty much everything here is useful)</p>\r\n<p>flighthistoryevents.csv &nbsp;(Extract most recent estimated runway arrival, estimated gate arrival, and arrival gate)</p>\r\n<p>asdiflightplan.csv &nbsp;(Extract most recent estimated arrival)</p>\r\n<p>The most useful entries to me right now are the estimated runway/gate arrival times.</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>So am i assuming that you actually don't use the other parts of the data? They gave us 2 GB of data, but I am still unsure what it is they want to do with all this data and I question whether all of this data will be used by me. Based on your post, I am\r\n guessing that not a lot of people use all of the data that they've given us?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "18613",
      "postDate": "12/26/2012 06:47:03",
      "content": "<p>[quote=ngoctranfire;18574]</p>\r\n<p>So am i assuming that you actually don't use the other parts of the data? They gave us 2 GB of data, but I am still unsure what it is they want to do with all this data and I question whether all of this data will be used by me. Based on your post, I am\r\n guessing that not a lot of people use all of the data that they've given us?</p>\r\n<p>[/quote]We provided a bunch of data that could potentially be useful, and want you to use as much of it or as little of it as is necessary to accurately predict flight arrival times. Many of the data sources and files will likely be useful, and many others\r\n may not be.</p>\r\n<p>To get started, the most relevant files (flighthistory.csv and flighthistoryevents.csv) are in the FlightHistory subdirectory for each day.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 18435,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "12/22/2012 18:40:49",
      "content": "Same files (with tweaking to account for flights absent from asdi files), only test data - until the rules conflict thing is resolved by the admins, i am leaving the training stuff aside so that each day is based on the pre/post cutoff (= train/ test split)\r\n for this day only.",
      "votes": null,
      "replies": []
    },
    {
      "id": 18442,
      "author_name": "rickyars",
      "author_url": "",
      "post_date": "12/22/2012 22:20:29",
      "content": "<p>Those are the files I'm using as well.&nbsp;</p>\r\n<p>I have another, somewhat related question. How are you dealing with records in the training set where the actual gate arrival occurs before the actual runway arrival?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18449,
      "author_name": "jay670790",
      "author_url": "",
      "post_date": "12/23/2012 00:06:05",
      "content": "<p>ricardo sachez, you should remove these records as&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Ben Hamner;17672]</p>\r\n<p>One common question involved the logic used to identify flights for the test set. This code is available in&nbsp;<a href=\"https://github.com/benhamner/GEFlightQuest/blob/master/PythonModule/geflight/transform/flighthistory.py#L288\">flighhistory.py on Github</a>.</p>\r\n<p>The conditions for a flight to be included in the test set are as follows:</p>\r\n<ul>\r\n<li>It took off and landed at a continental US airport </li><li>We received data for the actual gate and runway departure and arrival times </li><li>The runway arrival time was less than the gate arrival time </li><li>The flight was in the air at the cutoff time for the day </li></ul>\r\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18467,
      "author_name": "rickyars",
      "author_url": "",
      "post_date": "12/23/2012 13:39:04",
      "content": "<p>Thanks, Julien. I guess their script isn't working exactly as advertised.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18475,
      "author_name": "adpowers",
      "author_url": "",
      "post_date": "12/23/2012 16:24:57",
      "content": "<p>Decyphering the Rosetta stone would be easier than trying to interpret the structure of the provided data and hence this competition. Also did I read things correctly that for a submission to be considered as being valid, code and instructions needs to be\r\n made available upfront?</p>\r\n<p>MODEL SUBMISSION:<br>\r\nWhen you submit an Entry, you also will be able to upload your models to the Website.<br>\r\nYour final model submission must contain all code and parameter settings necessary to<br>\r\nevaluate your models on new data, and include a README file with instructions on how<br>\r\nto do so.</p>\r\n<p>WTF.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18489,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "12/23/2012 18:48:38",
      "content": "<p>so you are using only these events from flighthistoryevents</p>\r\n<p>#STATUS-Landed</p>\r\n<p>#STATUS-Wheels Down&nbsp;</p>\r\n<p># RecordCreated&nbsp;</p>\r\n<p># &nbsp;STATUS-Active&nbsp;</p>\r\n<p># Gate Adjustment&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>Which of the above indicate arrival and departure - especially runway and gate?</p>\r\n<p>Again poor documentation is not helping anyone</p>\r\n<p>[quote=Lonely at the Top;18434]</p>\r\n<p>Hi everyone,</p>\r\n<p>There's a lot of data in this competition, and I thought it would be useful if people could post exactly what data they were using in their models. &nbsp;It would be helpful for those just starting out, as well as those looking to add new features to their models.</p>\r\n<p>&nbsp;</p>\r\n<p>I'm currently using data from the files:</p>\r\n<p>flighthistory.csv &nbsp;(Pretty much everything here is useful)</p>\r\n<p>flighthistoryevents.csv &nbsp;(Extract most recent estimated runway arrival, estimated gate arrival, and arrival gate)</p>\r\n<p>asdiflightplan.csv &nbsp;(Extract most recent estimated arrival)</p>\r\n<p>The most useful entries to me right now are the estimated runway/gate arrival times.</p>\r\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18490,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "12/23/2012 18:49:51",
      "content": "<p>where are the splits available in training data?&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>[quote=Konrad Banachewicz;18435]</p>\r\n<p>Same files (with tweaking to account for flights absent from asdi files), only test data - until the rules conflict thing is resolved by the admins, i am leaving the training stuff aside so that each day is based on the pre/post cutoff (= train/ test split)\r\n for this day only.</p>\r\n<p>[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18493,
      "author_name": "konradb",
      "author_url": "",
      "post_date": "12/23/2012 19:22:06",
      "content": "<p>test flights = clearly defined, training flights = all other flights in the publicLeaderboard data folder for this day.&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18495,
      "author_name": "lonelyatthetop",
      "author_url": "",
      "post_date": "12/23/2012 19:37:11",
      "content": "<p>[quote=Black Magic;18489]</p>\r\n<p>so you are using only these events from flighthistoryevents</p>\r\n<p>#STATUS-Landed</p>\r\n<p>#STATUS-Wheels Down&nbsp;</p>\r\n<p># RecordCreated&nbsp;</p>\r\n<p># &nbsp;STATUS-Active&nbsp;</p>\r\n<p># Gate Adjustment&nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>Which of the above indicate arrival and departure - especially runway and gate?</p>\r\n<p>Again poor documentation is not helping anyone</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>For the flighthistoryevents file, I scan through looking for:</p>\r\n<p>ERA</p>\r\n<p>EGA</p>\r\n<p>AGATE</p>\r\n<p>After I find one of these, I look for &quot;New&quot; after it, then pull out the time or gate which appears after it. &nbsp;</p>\r\n<p>&nbsp;</p>\r\n<p>So far I've spent most of my time on this project working on data input ...</p>\r\n<p>&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18510,
      "author_name": "ansgar",
      "author_url": "",
      "post_date": "12/24/2012 00:19:57",
      "content": "<p>In addition I join the latest available Metar data (flightstats<em>_</em>metarreports_combined.csv) to each flight by the arrival airport's icao code. (That helped a little, but not much.)</p>\r\n<p>Still working on joining the asdiflightplan, and I have a question there:<br>\r\nIn the asdiflightplan.csv for 2012-11-12 I see only date/time values from the next day.<br>\r\n(estimatedarrivalutc is between 2012-11-13 02:50:00 and 2012-11-13 14:43:00)</p>\r\n<p>What am I missing?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18515,
      "author_name": "rkirana",
      "author_url": "",
      "post_date": "12/24/2012 03:43:52",
      "content": "<p>also looks like the times are different:</p>\r\n<p>data<em>updated date</em>time<em>recorded</em>old<br>\r\nSTATUS- Old=S New=U 2012-11-13 10:33:02.358-08 2012-11-12 16:20:20.573-08<br>\r\nARA- New=11/12/12 09:41, STATUS- Old=A New=L 2012-11-12 09:49:29.343-08<br>\r\nERA- Old=11/12/12 10:27 New=11/12/12 09:41 2012-11-12 09:46:57.819-08 <br>\r\nERA- Old=11/12/12 09:38 New=11/12/12 10:27 2012-11-12 09:46:22.529-08</p>\r\n<p>Looks like the time recorded column has a different date than what is in data updated. What is the right one?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18574,
      "author_name": "ngoctranfire",
      "author_url": "",
      "post_date": "12/25/2012 10:50:46",
      "content": "<p>[quote=Lonely at the Top;18434]</p>\r\n<p>Hi everyone,</p>\r\n<p>There's a lot of data in this competition, and I thought it would be useful if people could post exactly what data they were using in their models. &nbsp;It would be helpful for those just starting out, as well as those looking to add new features to their models.</p>\r\n<p>&nbsp;</p>\r\n<p>I'm currently using data from the files:</p>\r\n<p>flighthistory.csv &nbsp;(Pretty much everything here is useful)</p>\r\n<p>flighthistoryevents.csv &nbsp;(Extract most recent estimated runway arrival, estimated gate arrival, and arrival gate)</p>\r\n<p>asdiflightplan.csv &nbsp;(Extract most recent estimated arrival)</p>\r\n<p>The most useful entries to me right now are the estimated runway/gate arrival times.</p>\r\n<p>[/quote]</p>\r\n<p>&nbsp;</p>\r\n<p>So am i assuming that you actually don't use the other parts of the data? They gave us 2 GB of data, but I am still unsure what it is they want to do with all this data and I question whether all of this data will be used by me. Based on your post, I am\r\n guessing that not a lot of people use all of the data that they've given us?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 18613,
      "author_name": "benhamner",
      "author_url": "",
      "post_date": "12/26/2012 06:47:03",
      "content": "<p>[quote=ngoctranfire;18574]</p>\r\n<p>So am i assuming that you actually don't use the other parts of the data? They gave us 2 GB of data, but I am still unsure what it is they want to do with all this data and I question whether all of this data will be used by me. Based on your post, I am\r\n guessing that not a lot of people use all of the data that they've given us?</p>\r\n<p>[/quote]We provided a bunch of data that could potentially be useful, and want you to use as much of it or as little of it as is necessary to accurately predict flight arrival times. Many of the data sources and files will likely be useful, and many others\r\n may not be.</p>\r\n<p>To get started, the most relevant files (flighthistory.csv and flighthistoryevents.csv) are in the FlightHistory subdirectory for each day.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "18434": "",
    "18435": "",
    "18442": "",
    "18449": "",
    "18467": "",
    "18475": "",
    "18489": "",
    "18490": "",
    "18493": "",
    "18495": "",
    "18510": "",
    "18515": "",
    "18574": "",
    "18613": ""
  },
  "source": "meta"
}