If your model happens to include the estimated arrival time in the asdiflightplan.csv, there's a reason your leaderboard submissions are terrible:
\r\nOn Dec 5, of the 4 entries for flight 282233030, 3 have an eta near the end of Dec 5, the last (and most recent) however, has an eta near the end of Dec 6. This is wrong and should be changed to Dec 5.
\r\n\r\n
To the admins: I hope there will be some leeway in the final test set submissions if isolated errors like this pop up in the final data. This one wrong data point increased my score by nearly two minutes. I've now put in some sanity checks on the data,\r\n but it's hard to account for all possibilities. I feel that if your final submission is unexpectedly bad, and you can fix it by making a small change to the data (like Dec 6 --> Dec 5), this should be allowed.
", "messages": [ { "id": "18597", "postDate": "12/26/2012 00:41:25", "content": "If your model happens to include the estimated arrival time in the asdiflightplan.csv, there's a reason your leaderboard submissions are terrible:
\r\nOn Dec 5, of the 4 entries for flight 282233030, 3 have an eta near the end of Dec 5, the last (and most recent) however, has an eta near the end of Dec 6. This is wrong and should be changed to Dec 5.
\r\n\r\n
To the admins: I hope there will be some leeway in the final test set submissions if isolated errors like this pop up in the final data. This one wrong data point increased my score by nearly two minutes. I've now put in some sanity checks on the data,\r\n but it's hard to account for all possibilities. I feel that if your final submission is unexpectedly bad, and you can fix it by making a small change to the data (like Dec 6 --> Dec 5), this should be allowed.
", "rawMarkdown": "", "votes": null }, { "id": "18606", "postDate": "12/26/2012 05:13:42", "content": "I've seen the other cases where the flight plans were filled incorrectly (e.g. a flight planned to fly at over Mach speed), so these kind of errors are to be expected.
\r\nYou'll have to deal with those errors as a part of competition, I suppose.
", "rawMarkdown": "", "votes": null }, { "id": "18609", "postDate": "12/26/2012 06:31:18", "content": "[quote=charango;18606]
\r\nI've seen the other cases where the flight plans were filled incorrectly (e.g. a flight planned to fly at over Mach speed), so these kind of errors are to be expected.
\r\nYou'll have to deal with those errors as a part of competition, I suppose.
\r\n[/quote]Yes, practically no real-world data is 100% correct. Your models should be robust to inconsistencies in the data that affect only a small percentage of the flights and you shouldn't be making manual changes to the test data.
\r\nWe'll handle any extreme inconsistencies on a case-by-case basis in the event that they arise.
", "rawMarkdown": "", "votes": null }, { "id": "19227", "postDate": "01/12/2013 10:28:12", "content": "What about a case like this?\r\n
\r\n
FLIGHT ID 280977327\r\n
Airport departure: DCA (WASHINGTON NATIONAL AIRPORT )
Airport arrival: MCI (KANSAS CITY INTERNATIONAL AIRPORT)
actual_gate_departure=11/22 00:00:00
actual_runway_departure=11/22 00:20:00
scheduled_runway_arrival=11/22 02:41:00
scheduled_gate_arrival=11/22 02:59:00
actual_runway_arrival=11/22 00:27:00
actual_gate_arrival=11/22 02:35:00
\r\n
actual_runway_arrival is clearly wrong by 2 hours. Are we expected to minimize the error against the wrong data? That could affect our models and make them less useful for their intended purpose.", "rawMarkdown": "", "votes": null }, { "id": "19228", "postDate": "01/12/2013 12:25:55", "content": "
I guess errors like these are unpredictable, and will affect all competitors, so you shouldn't worry too much about this.
", "rawMarkdown": "", "votes": null }, { "id": "19287", "postDate": "01/14/2013 11:32:29", "content": "Hi, I am not worrying. The issue is that there may in fact exist patterns in those errors which one could include in the model to minimize the overall error (but increasing the error against the actual times in the process). Is this what GE wants, or does\r\n this qualify as an extreme inconsistency? It is not an isolated case.
", "rawMarkdown": "", "votes": null }, { "id": "19292", "postDate": "01/14/2013 14:06:26", "content": "If you could identify this kind of patterns (that create errors), it could be useful to the client, as this will allow them to filter out flights with incorrect data.
\r\nFor example, in initial training set I've seen a case where the landing (runway arrival) time was registered incorrectly, like 30 minutes after the actual landing. As evidenced by asdiposition.csv, where the plane speed and altitude went to zero at, say,\r\n 2:00pm, but the runway arrival time in flighthistory.csv was stating 2:30pm. This looks to me like an operator error, where someone has forgotten to register the plane landing.
\r\nIf you could identify the cases like the above, the client could (theoretically) benefit by eliminating the source of these problems, if that is possible. Or, at least, the final software could identify the obvious erroneous flights, and do not publicly\r\n display the arrival times that are not correct.
\r\nAs I see it, if airline customers see that a plane is due to arrive in 15 minutes, and the flight is late by 5 minutes, that will be fine. However, if (due to an error in flightstats software, or anything else), the customers sees ETA in 15 hours, that will\r\n create a negative image of the airline. This explains the reasoning of selecting RMSE for scoring.
", "rawMarkdown": "", "votes": null }, { "id": "19321", "postDate": "01/15/2013 08:08:09", "content": "It seems to me this would be a different problem than computing an expected value for future records which may or may not be erroneous.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 18606, "author_name": "sergeykozub", "author_url": "", "post_date": "12/26/2012 05:13:42", "content": "I've seen the other cases where the flight plans were filled incorrectly (e.g. a flight planned to fly at over Mach speed), so these kind of errors are to be expected.
\r\nYou'll have to deal with those errors as a part of competition, I suppose.
", "votes": null, "replies": [] }, { "id": 18609, "author_name": "benhamner", "author_url": "", "post_date": "12/26/2012 06:31:18", "content": "[quote=charango;18606]
\r\nI've seen the other cases where the flight plans were filled incorrectly (e.g. a flight planned to fly at over Mach speed), so these kind of errors are to be expected.
\r\nYou'll have to deal with those errors as a part of competition, I suppose.
\r\n[/quote]Yes, practically no real-world data is 100% correct. Your models should be robust to inconsistencies in the data that affect only a small percentage of the flights and you shouldn't be making manual changes to the test data.
\r\nWe'll handle any extreme inconsistencies on a case-by-case basis in the event that they arise.
", "votes": null, "replies": [] }, { "id": 19227, "author_name": "elloco", "author_url": "", "post_date": "01/12/2013 10:28:12", "content": "What about a case like this?\r\n
\r\n
FLIGHT ID 280977327\r\n
Airport departure: DCA (WASHINGTON NATIONAL AIRPORT )
Airport arrival: MCI (KANSAS CITY INTERNATIONAL AIRPORT)
actual_gate_departure=11/22 00:00:00
actual_runway_departure=11/22 00:20:00
scheduled_runway_arrival=11/22 02:41:00
scheduled_gate_arrival=11/22 02:59:00
actual_runway_arrival=11/22 00:27:00
actual_gate_arrival=11/22 02:35:00
\r\n
actual_runway_arrival is clearly wrong by 2 hours. Are we expected to minimize the error against the wrong data? That could affect our models and make them less useful for their intended purpose.", "votes": null, "replies": [] }, { "id": 19228, "author_name": "sergeykozub", "author_url": "", "post_date": "01/12/2013 12:25:55", "content": "
I guess errors like these are unpredictable, and will affect all competitors, so you shouldn't worry too much about this.
", "votes": null, "replies": [] }, { "id": 19287, "author_name": "elloco", "author_url": "", "post_date": "01/14/2013 11:32:29", "content": "Hi, I am not worrying. The issue is that there may in fact exist patterns in those errors which one could include in the model to minimize the overall error (but increasing the error against the actual times in the process). Is this what GE wants, or does\r\n this qualify as an extreme inconsistency? It is not an isolated case.
", "votes": null, "replies": [] }, { "id": 19292, "author_name": "sergeykozub", "author_url": "", "post_date": "01/14/2013 14:06:26", "content": "If you could identify this kind of patterns (that create errors), it could be useful to the client, as this will allow them to filter out flights with incorrect data.
\r\nFor example, in initial training set I've seen a case where the landing (runway arrival) time was registered incorrectly, like 30 minutes after the actual landing. As evidenced by asdiposition.csv, where the plane speed and altitude went to zero at, say,\r\n 2:00pm, but the runway arrival time in flighthistory.csv was stating 2:30pm. This looks to me like an operator error, where someone has forgotten to register the plane landing.
\r\nIf you could identify the cases like the above, the client could (theoretically) benefit by eliminating the source of these problems, if that is possible. Or, at least, the final software could identify the obvious erroneous flights, and do not publicly\r\n display the arrival times that are not correct.
\r\nAs I see it, if airline customers see that a plane is due to arrive in 15 minutes, and the flight is late by 5 minutes, that will be fine. However, if (due to an error in flightstats software, or anything else), the customers sees ETA in 15 hours, that will\r\n create a negative image of the airline. This explains the reasoning of selecting RMSE for scoring.
", "votes": null, "replies": [] }, { "id": 19321, "author_name": "elloco", "author_url": "", "post_date": "01/15/2013 08:08:09", "content": "It seems to me this would be a different problem than computing an expected value for future records which may or may not be erroneous.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "18597": "", "18606": "", "18609": "", "19227": "", "19228": "", "19287": "", "19292": "", "19321": "" }, "source": "meta" }