As a warning to anyone planning on training with weather features, note that the training sets take place in midsummer but the final evaluation will be based on winter data. Experience shows that weather patterns in the US are very different in those different seasons, especially in relation to some conditions that may appear in the test set but not the training set (e. g. icing conditions at a large number of airports). There are also different density and structure of holidays, different passenger cost functions (which the simulator may or may not reflect; it's not released yet), and other important distinguishing factors.
\nThe leaderboards are based on different seasons yet (summer & early autumn), so there is not really any way to know how one's algorithm is going to perform on the final evaluation set. The agent training leaderboard also covers Sept. 11, though I'm not sure if the simulator will reflect any flukes in US aviation on that date (e. g. cost function for routes changed to be closer to major urban centers, especially at lower altitudes, possibly being higher than on other dates).
\n
I'd look for some pretty big changes between leaderboard scores and final evaluation scores. #100 on the leaderboard at the close of competition...you could win!
Personally, I think it would be better to have training data on similar conditions as the test data, either having both in the midsummer (maybe August, without holidays) or both as year-round samples.
As this competition is not set up like that, unless the admins change it, you will either have to rely on external weather sources for other time periods (though you don't have matching flight data to pass through the simulator) or simply not include a lot of detail on weather-related features, since they're likely to overfit on the summer data (match the details of summer much better than the test data, which is in winter).
As a warning to anyone planning on training with weather features, note that the training sets take place in midsummer but the final evaluation will be based on winter data. Experience shows that weather patterns in the US are very different in those different seasons, especially in relation to some conditions that may appear in the test set but not the training set (e. g. icing conditions at a large number of airports). There are also different density and structure of holidays, different passenger cost functions (which the simulator may or may not reflect; it's not released yet), and other important distinguishing factors.
\nThe leaderboards are based on different seasons yet (summer & early autumn), so there is not really any way to know how one's algorithm is going to perform on the final evaluation set. The agent training leaderboard also covers Sept. 11, though I'm not sure if the simulator will reflect any flukes in US aviation on that date (e. g. cost function for routes changed to be closer to major urban centers, especially at lower altitudes, possibly being higher than on other dates).
\n
I'd look for some pretty big changes between leaderboard scores and final evaluation scores. #100 on the leaderboard at the close of competition...you could win!
Personally, I think it would be better to have training data on similar conditions as the test data, either having both in the midsummer (maybe August, without holidays) or both as year-round samples.
As this competition is not set up like that, unless the admins change it, you will either have to rely on external weather sources for other time periods (though you don't have matching flight data to pass through the simulator) or simply not include a lot of detail on weather-related features, since they're likely to overfit on the summer data (match the details of summer much better than the test data, which is in winter).
Aircraft schedules in winter and summer are very different not only due to weather but due to passenger preferences and demand (people on summer vacation etc...). Congestion related delays in the real world are very seasonal at many airports.
", "rawMarkdown": "", "votes": null }, { "id": "28712", "postDate": "08/15/2013 00:14:36", "content": "I don't think it will matter much to final ranking. These kind of errors are systematic - they should touch everyone to the same extent.
", "rawMarkdown": "", "votes": null }, { "id": "28733", "postDate": "08/15/2013 13:37:01", "content": "If everybody had the same algorithm, that would be the case.
These kinds of errors will affect everyone differently, depending on the degree to which the training and test features *that are prominent in the model* differ between training and test.
Here, I recommend that people pick features that are not as likely to have big differences between training and test, but between my post and Noam's, there isn't a whole lot left, and we should just expect big swings and reshufflings among ranked performance order.
Aircraft schedules in winter and summer are very different not only due to weather but due to passenger preferences and demand (people on summer vacation etc...). Congestion related delays in the real world are very seasonal at many airports.
", "votes": null, "replies": [] }, { "id": 28712, "author_name": "paweljankiewicz", "author_url": "", "post_date": "08/15/2013 00:14:36", "content": "I don't think it will matter much to final ranking. These kind of errors are systematic - they should touch everyone to the same extent.
", "votes": null, "replies": [] }, { "id": 28733, "author_name": "wbtthefrog", "author_url": "", "post_date": "08/15/2013 13:37:01", "content": "If everybody had the same algorithm, that would be the case.
These kinds of errors will affect everyone differently, depending on the degree to which the training and test features *that are prominent in the model* differ between training and test.
Here, I recommend that people pick features that are not as likely to have big differences between training and test, but between my post and Noam's, there isn't a whole lot left, and we should just expect big swings and reshufflings among ranked performance order.