The final evaluation set is now up on the data page.
\r\nYou may make a submission on the final evaluation set regardless of whether you've submitted a final model or hash. However, you are only eligible for prize money if you submitted a final model and use the same model to make predictions on this new set.
\r\nWhen you make a submission on this new set, a public score of "0.0000" without errors means the submission was processed correctly.
\r\nPlease let us know if you find any issues with the data on this thread.
\r\nGood luck!
\r\n", "messages": [ { "id": "21060", "postDate": "03/05/2013 11:05:50", "content": "
The final evaluation set is now up on the data page.
\r\nYou may make a submission on the final evaluation set regardless of whether you've submitted a final model or hash. However, you are only eligible for prize money if you submitted a final model and use the same model to make predictions on this new set.
\r\nWhen you make a submission on this new set, a public score of "0.0000" without errors means the submission was processed correctly.
\r\nPlease let us know if you find any issues with the data on this thread.
\r\nGood luck!
\r\n", "rawMarkdown": "", "votes": null }, { "id": "21080", "postDate": "03/06/2013 07:01:08", "content": "
Hi Ben,
\r\nI did some sanity checks on my submission (just to stay sane until the deadline) - I found that there is one flight for which predicted gate and runway arrival are totally off (differ by more than 200 minutes). I investigated the issue: it was the flight\r\n number 3670 from DTW to GSO on 28.2. I checked flight stats - the flight took off 2 minutes ahead of scheduled (gate) departure (10:20 am; runway is about 20min later) but the actual gate arrival is 02:43 on 01.3, thus the actual block time is\r\n more than 16 hours - the scheduled block time is 113 minutes!
\r\nhere is the entry on flight stats: http://www.flightstats.com/go/FlightStatus/flightStatusByFlight.do?id=289751327&airlineCode=9E&flightNumber=3670
\r\nIs this flight still valid?
", "rawMarkdown": "", "votes": null }, { "id": "21083", "postDate": "03/06/2013 07:20:39", "content": "When you're making your final predictions, you should be applying your previously submitted model to the flights that we've asked you to make predictions on and assuming that all flights in the test set are valid.
\r\nI can't provide guidance on individual cases at this point in time (and you shouldn't be manually modifying any of your predictions). However, keep posting any potential inconsistiencies you find and we'll go through them in detail after the March 11 deadline\r\n (which may result in dropping a very small number of obvious errors from the final evaluation set).
", "rawMarkdown": "", "votes": null }, { "id": "21086", "postDate": "03/06/2013 08:38:17", "content": "[quote=Ben Hamner;21083]
\r\nWhen you're making your final predictions, you should be applying your previously submitted model to the flights that we've asked you to make predictions on and assuming that all flights in the test set are valid.
\r\nI can't provide guidance on individual cases at this point in time (and you shouldn't be manually modifying any of your predictions).
\r\n[/quote]
\r\nI'm absolutely aware about that (I've already submitted my predictions).
", "rawMarkdown": "", "votes": null }, { "id": "21089", "postDate": "03/06/2013 10:46:57", "content": "Kaggle, admins and contestants,
\r\nI don’t find it easy to write this post and maybe I’m the only one who thinks this way (in that case, just ignore me).
\r\nIn my opinion in an ideal world all the rules of which flights to include in the validation set and which not should be clear before the start of the competition.
\r\nIf that isn’t possible they should be clear well before the model submission deadline.
\r\nWhat I see happening now is that we have rules e.g:
\r\n- no diverted flights
\r\n- no redirected flights
\r\n- no flights with actual_gate_departure after actual_runway_departure
\r\n- etc.
\r\n\r\n
But that next to the rules there is also still room for discussion.
\r\nI don’t find this a good development because:
\r\na. It is an open invitation to everybody who thinks he/she makes a chance of winning to discuss flights which are possible harmful for there predictions. And this would make it a challenge of predictive modeling AND Public Relations instead of only predictive\r\n modeling.
\r\nb. It takes away the possibility for Kaggle to demonstrate that they work in an unbiased and transparent way, because in theory they could leave out flights (after March 11) which benefit one contestant more than others.
\r\nI’m NOT saying any contestant is thinking in such a way or that Kaggle or any admins are biased, we are just making it harder to proof that we’re not.
\r\nI propose we only filter the validation set according to rules which were clear before February 15th
\r\n", "rawMarkdown": "", "votes": null }, { "id": "21099", "postDate": "03/06/2013 16:28:59", "content": "
Jules
\r\nI can agree with your opinion. But... imagine a situation when you work for 3 months on the solution and unexpected error / inconsitency in the data appears only in the FinalValidationSet. The expected performance on such observations is random in my opinion.\r\n When you revealed the diverted flights solutions the gains of the competitors varied between 0.20 and 0.40. And you can see what 0.2 difference can mean on the leaderboard.
\r\nIn real life you could correct the issue with the data and it wouldn't diminish the model value. Here you start with the assumption that users will cheat (which is not true for 99.9% of the us ;)). For future competitions I would think on a formula that\r\n isn't impacted by an anticheater methodology to such extent.
", "rawMarkdown": "", "votes": null }, { "id": "21112", "postDate": "03/06/2013 19:39:30", "content": "My background is signal processing, and I just started to learn machine learning last year. From a point of view of time series, the compition is well designed in terms of causality. However, it dosn't take care of data consistence too much (techincally\r\n speaking, stationarity). All information are recorded by software, which are updated pierortically for sure. Espaically, engineer s in airline managment also want to impove the expected arrivals that are important for most models\r\n here. Software updates definitely make features unstationary. In fact, it is fine since it is the problem for us to face in the real word. However, we should to able to observe time-varying changes in the train set in order to provide more useful models for\r\n the hosts.
\r\nAnyway, we do gambling on guessing if their system (including data processing flows) was changed too much or not recently;)
", "rawMarkdown": "", "votes": null }, { "id": "21116", "postDate": "03/06/2013 21:02:48", "content": "Thanks Pawel. Jules, most likely no additional flights will be filtered from the final evaluation set.
\r\nHowever, there may be errors and inconsistencies in this data. If there's an extreme one (e.g. one that's 24 hours off), we don't want that skewing everyone's results and the error measurements (RMSE is especially\r\n sensitive to outliers) and would remove that observation.
", "rawMarkdown": "", "votes": null }, { "id": "21140", "postDate": "03/07/2013 12:14:10", "content": "Hi Ben,
\r\nWe have a query regarding the submissions. We have used the times data from the Flight History file in our work. However, due to some particular values being MISSING in the data of the Final Evaluation Set, our predictions do not show for some flights. We\r\n can fix this issue with a small amount of code (around 5-6 lines) which:
\r\nThis modification only comes in when the actual time is being calculated using the delays. Please clarify if this is an acceptable deviation so that we can submit accordingly. Thanks.
", "rawMarkdown": "", "votes": null }, { "id": "21151", "postDate": "03/07/2013 18:51:37", "content": "@Harish,
\r\nMy practice was to submit not only the work dir also the git repository. If I need to fix bugs, they could easily track and judge the changes.
", "rawMarkdown": "", "votes": null }, { "id": "21152", "postDate": "03/07/2013 18:55:24", "content": "[quote=Harish Krishnamurthy;21140]
\r\nHi Ben,
\r\nWe have a query regarding the submissions. We have used the times data from the Flight History file in our work. However, due to some particular values being MISSING in the data of the Final Evaluation Set, our predictions do not show for some flights. We\r\n can fix this issue with a small amount of code (around 5-6 lines) which:
\r\nThis modification only comes in when the actual time is being calculated using the delays. Please clarify if this is an acceptable deviation so that we can submit accordingly. Thanks.
\r\n[/quote]This is fine
", "rawMarkdown": "", "votes": null }, { "id": "21201", "postDate": "03/09/2013 19:29:57", "content": "Sorry for repeating the same question again, but what is the exact deadline for the final submission?
\r\nThe details page says 11 march, but what time, what timezone?
\r\nEven the 'make a submission' page does not give the details...
\r\nActually, why don't you just specify the UNIX time?
\r\n", "rawMarkdown": "", "votes": null }, { "id": "21212", "postDate": "03/10/2013 13:14:13", "content": "
Hi,
\r\n
\r\nWe have an issue similar to Harish.
\r\nDue to a bug in our code, two of the final test days resulted in an extra column as compared to all the training days. It's most probably caused by our insufficient cleaning of the ASDI data. This happens only for two days (2013_02_22,2013_02_26) and generates\r\n one extra column (nof_Linked.Airport.Closure.s.).
\r\n
\r\nThis caused a call to rbind() to fail, due to column mismatch. To fix this, we had to delete this extra column. This did not involve tweaking any of the model parameters, just correcting the pre-processing script.
\r\n
\r\nThe Error:
\r\n[2013-03-07 00:44:38] test day: 2013_02_22
Error in rbind(deparse.level, ...) :
numbers of columns of arguments do not match
Calls: rbind -> rbind
Execution halted
\r\n
\r\nOur fix (in the pre-processing script):
\r\n[2013-03-10 11:10:57] test day: 2013_02_22
\r\n[2013-03-10 11:10:57] 51 52
\r\n[2013-03-10 11:10:57] nof_Linked.Airport.Closure.s. in xtr not in xtrain, removing column
\r\n[…]
\r\n[2013-03-10 11:13:40] test day: 2013_02_26
\r\n[2013-03-10 11:13:40] 51 52
\r\n[2013-03-10 11:13:40] nof_Linked.Airport.Closure.s. in xtr not in xtrain, removing column
\r\n
\r\n
\r\nAgain, please clarify if this is an acceptable deviation so that we can submit accordingly. Thanks!
\r\n
\r\n
[quote=vojtekb;21212]
\r\nHi,
\r\n
\r\nWe have an issue similar to Harish.
\r\nDue to a bug in our code, two of the final test days resulted in an extra column as compared to all the training days. It's most probably caused by our insufficient cleaning of the ASDI data. This happens only for two days (2013_02_22,2013_02_26) and generates\r\n one extra column (nof_Linked.Airport.Closure.s.).
\r\n
\r\nThis caused a call to rbind() to fail, due to column mismatch. To fix this, we had to delete this extra column. This did not involve tweaking any of the model parameters, just correcting the pre-processing script.
\r\n
\r\nThe Error:
\r\n[2013-03-07 00:44:38] test day: 2013_02_22
Error in rbind(deparse.level, ...) :
numbers of columns of arguments do not match
Calls: rbind -> rbind
Execution halted
\r\n
\r\nOur fix (in the pre-processing script):
\r\n[2013-03-10 11:10:57] test day: 2013_02_22
\r\n[2013-03-10 11:10:57] 51 52
\r\n[2013-03-10 11:10:57] nof_Linked.Airport.Closure.s. in xtr not in xtrain, removing column
\r\n[…]
\r\n[2013-03-10 11:13:40] test day: 2013_02_26
\r\n[2013-03-10 11:13:40] 51 52
\r\n[2013-03-10 11:13:40] nof_Linked.Airport.Closure.s. in xtr not in xtrain, removing column
\r\n
\r\n
\r\nAgain, please clarify if this is an acceptable deviation so that we can submit accordingly. Thanks!
\r\n
[/quote]This is fine
", "rawMarkdown": "", "votes": null }, { "id": "21234", "postDate": "03/11/2013 17:53:06", "content": "[quote=vojtekb;21201]
\r\nSorry for repeating the same question again, but what is the exact deadline for the final submission?
\r\nThe details page says 11 march, but what time, what timezone?
\r\nEven the 'make a submission' page does not give the details...
\r\nActually, why don't you just specify the UNIX time?
\r\n[/quote]Deadline is 11:59 PM UTC
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 21080, "author_name": "pprett", "author_url": "", "post_date": "03/06/2013 07:01:08", "content": "Hi Ben,
\r\nI did some sanity checks on my submission (just to stay sane until the deadline) - I found that there is one flight for which predicted gate and runway arrival are totally off (differ by more than 200 minutes). I investigated the issue: it was the flight\r\n number 3670 from DTW to GSO on 28.2. I checked flight stats - the flight took off 2 minutes ahead of scheduled (gate) departure (10:20 am; runway is about 20min later) but the actual gate arrival is 02:43 on 01.3, thus the actual block time is\r\n more than 16 hours - the scheduled block time is 113 minutes!
\r\nhere is the entry on flight stats: http://www.flightstats.com/go/FlightStatus/flightStatusByFlight.do?id=289751327&airlineCode=9E&flightNumber=3670
\r\nIs this flight still valid?
", "votes": null, "replies": [] }, { "id": 21083, "author_name": "benhamner", "author_url": "", "post_date": "03/06/2013 07:20:39", "content": "When you're making your final predictions, you should be applying your previously submitted model to the flights that we've asked you to make predictions on and assuming that all flights in the test set are valid.
\r\nI can't provide guidance on individual cases at this point in time (and you shouldn't be manually modifying any of your predictions). However, keep posting any potential inconsistiencies you find and we'll go through them in detail after the March 11 deadline\r\n (which may result in dropping a very small number of obvious errors from the final evaluation set).
", "votes": null, "replies": [] }, { "id": 21086, "author_name": "pprett", "author_url": "", "post_date": "03/06/2013 08:38:17", "content": "[quote=Ben Hamner;21083]
\r\nWhen you're making your final predictions, you should be applying your previously submitted model to the flights that we've asked you to make predictions on and assuming that all flights in the test set are valid.
\r\nI can't provide guidance on individual cases at this point in time (and you shouldn't be manually modifying any of your predictions).
\r\n[/quote]
\r\nI'm absolutely aware about that (I've already submitted my predictions).
", "votes": null, "replies": [] }, { "id": 21089, "author_name": "julesvanligtenberg", "author_url": "", "post_date": "03/06/2013 10:46:57", "content": "Kaggle, admins and contestants,
\r\nI don’t find it easy to write this post and maybe I’m the only one who thinks this way (in that case, just ignore me).
\r\nIn my opinion in an ideal world all the rules of which flights to include in the validation set and which not should be clear before the start of the competition.
\r\nIf that isn’t possible they should be clear well before the model submission deadline.
\r\nWhat I see happening now is that we have rules e.g:
\r\n- no diverted flights
\r\n- no redirected flights
\r\n- no flights with actual_gate_departure after actual_runway_departure
\r\n- etc.
\r\n\r\n
But that next to the rules there is also still room for discussion.
\r\nI don’t find this a good development because:
\r\na. It is an open invitation to everybody who thinks he/she makes a chance of winning to discuss flights which are possible harmful for there predictions. And this would make it a challenge of predictive modeling AND Public Relations instead of only predictive\r\n modeling.
\r\nb. It takes away the possibility for Kaggle to demonstrate that they work in an unbiased and transparent way, because in theory they could leave out flights (after March 11) which benefit one contestant more than others.
\r\nI’m NOT saying any contestant is thinking in such a way or that Kaggle or any admins are biased, we are just making it harder to proof that we’re not.
\r\nI propose we only filter the validation set according to rules which were clear before February 15th
\r\n", "votes": null, "replies": [] }, { "id": 21099, "author_name": "paweljankiewicz", "author_url": "", "post_date": "03/06/2013 16:28:59", "content": "
Jules
\r\nI can agree with your opinion. But... imagine a situation when you work for 3 months on the solution and unexpected error / inconsitency in the data appears only in the FinalValidationSet. The expected performance on such observations is random in my opinion.\r\n When you revealed the diverted flights solutions the gains of the competitors varied between 0.20 and 0.40. And you can see what 0.2 difference can mean on the leaderboard.
\r\nIn real life you could correct the issue with the data and it wouldn't diminish the model value. Here you start with the assumption that users will cheat (which is not true for 99.9% of the us ;)). For future competitions I would think on a formula that\r\n isn't impacted by an anticheater methodology to such extent.
", "votes": null, "replies": [] }, { "id": 21112, "author_name": "songgc", "author_url": "", "post_date": "03/06/2013 19:39:30", "content": "My background is signal processing, and I just started to learn machine learning last year. From a point of view of time series, the compition is well designed in terms of causality. However, it dosn't take care of data consistence too much (techincally\r\n speaking, stationarity). All information are recorded by software, which are updated pierortically for sure. Espaically, engineer s in airline managment also want to impove the expected arrivals that are important for most models\r\n here. Software updates definitely make features unstationary. In fact, it is fine since it is the problem for us to face in the real word. However, we should to able to observe time-varying changes in the train set in order to provide more useful models for\r\n the hosts.
\r\nAnyway, we do gambling on guessing if their system (including data processing flows) was changed too much or not recently;)
", "votes": null, "replies": [] }, { "id": 21116, "author_name": "benhamner", "author_url": "", "post_date": "03/06/2013 21:02:48", "content": "Thanks Pawel. Jules, most likely no additional flights will be filtered from the final evaluation set.
\r\nHowever, there may be errors and inconsistencies in this data. If there's an extreme one (e.g. one that's 24 hours off), we don't want that skewing everyone's results and the error measurements (RMSE is especially\r\n sensitive to outliers) and would remove that observation.
", "votes": null, "replies": [] }, { "id": 21140, "author_name": "harishkrishnamurthy", "author_url": "", "post_date": "03/07/2013 12:14:10", "content": "Hi Ben,
\r\nWe have a query regarding the submissions. We have used the times data from the Flight History file in our work. However, due to some particular values being MISSING in the data of the Final Evaluation Set, our predictions do not show for some flights. We\r\n can fix this issue with a small amount of code (around 5-6 lines) which:
\r\nThis modification only comes in when the actual time is being calculated using the delays. Please clarify if this is an acceptable deviation so that we can submit accordingly. Thanks.
", "votes": null, "replies": [] }, { "id": 21151, "author_name": "songgc", "author_url": "", "post_date": "03/07/2013 18:51:37", "content": "@Harish,
\r\nMy practice was to submit not only the work dir also the git repository. If I need to fix bugs, they could easily track and judge the changes.
", "votes": null, "replies": [] }, { "id": 21152, "author_name": "benhamner", "author_url": "", "post_date": "03/07/2013 18:55:24", "content": "[quote=Harish Krishnamurthy;21140]
\r\nHi Ben,
\r\nWe have a query regarding the submissions. We have used the times data from the Flight History file in our work. However, due to some particular values being MISSING in the data of the Final Evaluation Set, our predictions do not show for some flights. We\r\n can fix this issue with a small amount of code (around 5-6 lines) which:
\r\nThis modification only comes in when the actual time is being calculated using the delays. Please clarify if this is an acceptable deviation so that we can submit accordingly. Thanks.
\r\n[/quote]This is fine
", "votes": null, "replies": [] }, { "id": 21201, "author_name": "vojtekb", "author_url": "", "post_date": "03/09/2013 19:29:57", "content": "Sorry for repeating the same question again, but what is the exact deadline for the final submission?
\r\nThe details page says 11 march, but what time, what timezone?
\r\nEven the 'make a submission' page does not give the details...
\r\nActually, why don't you just specify the UNIX time?
\r\n", "votes": null, "replies": [] }, { "id": 21212, "author_name": "vojtekb", "author_url": "", "post_date": "03/10/2013 13:14:13", "content": "
Hi,
\r\n
\r\nWe have an issue similar to Harish.
\r\nDue to a bug in our code, two of the final test days resulted in an extra column as compared to all the training days. It's most probably caused by our insufficient cleaning of the ASDI data. This happens only for two days (2013_02_22,2013_02_26) and generates\r\n one extra column (nof_Linked.Airport.Closure.s.).
\r\n
\r\nThis caused a call to rbind() to fail, due to column mismatch. To fix this, we had to delete this extra column. This did not involve tweaking any of the model parameters, just correcting the pre-processing script.
\r\n
\r\nThe Error:
\r\n[2013-03-07 00:44:38] test day: 2013_02_22
Error in rbind(deparse.level, ...) :
numbers of columns of arguments do not match
Calls: rbind -> rbind
Execution halted
\r\n
\r\nOur fix (in the pre-processing script):
\r\n[2013-03-10 11:10:57] test day: 2013_02_22
\r\n[2013-03-10 11:10:57] 51 52
\r\n[2013-03-10 11:10:57] nof_Linked.Airport.Closure.s. in xtr not in xtrain, removing column
\r\n[…]
\r\n[2013-03-10 11:13:40] test day: 2013_02_26
\r\n[2013-03-10 11:13:40] 51 52
\r\n[2013-03-10 11:13:40] nof_Linked.Airport.Closure.s. in xtr not in xtrain, removing column
\r\n
\r\n
\r\nAgain, please clarify if this is an acceptable deviation so that we can submit accordingly. Thanks!
\r\n
\r\n
[quote=vojtekb;21212]
\r\nHi,
\r\n
\r\nWe have an issue similar to Harish.
\r\nDue to a bug in our code, two of the final test days resulted in an extra column as compared to all the training days. It's most probably caused by our insufficient cleaning of the ASDI data. This happens only for two days (2013_02_22,2013_02_26) and generates\r\n one extra column (nof_Linked.Airport.Closure.s.).
\r\n
\r\nThis caused a call to rbind() to fail, due to column mismatch. To fix this, we had to delete this extra column. This did not involve tweaking any of the model parameters, just correcting the pre-processing script.
\r\n
\r\nThe Error:
\r\n[2013-03-07 00:44:38] test day: 2013_02_22
Error in rbind(deparse.level, ...) :
numbers of columns of arguments do not match
Calls: rbind -> rbind
Execution halted
\r\n
\r\nOur fix (in the pre-processing script):
\r\n[2013-03-10 11:10:57] test day: 2013_02_22
\r\n[2013-03-10 11:10:57] 51 52
\r\n[2013-03-10 11:10:57] nof_Linked.Airport.Closure.s. in xtr not in xtrain, removing column
\r\n[…]
\r\n[2013-03-10 11:13:40] test day: 2013_02_26
\r\n[2013-03-10 11:13:40] 51 52
\r\n[2013-03-10 11:13:40] nof_Linked.Airport.Closure.s. in xtr not in xtrain, removing column
\r\n
\r\n
\r\nAgain, please clarify if this is an acceptable deviation so that we can submit accordingly. Thanks!
\r\n
[/quote]This is fine
", "votes": null, "replies": [] }, { "id": 21234, "author_name": "benhamner", "author_url": "", "post_date": "03/11/2013 17:53:06", "content": "[quote=vojtekb;21201]
\r\nSorry for repeating the same question again, but what is the exact deadline for the final submission?
\r\nThe details page says 11 march, but what time, what timezone?
\r\nEven the 'make a submission' page does not give the details...
\r\nActually, why don't you just specify the UNIX time?
\r\n[/quote]Deadline is 11:59 PM UTC
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "21060": "", "21080": "", "21083": "", "21086": "", "21089": "", "21099": "", "21112": "", "21116": "", "21140": "", "21151": "", "21152": "", "21201": "", "21212": "", "21233": "", "21234": "" }, "source": "meta" }