As a reminder, you have until 6:00pm UTC on February 15 to upload a hash or archive of your final model to the site. Basically, the final model should consist of a single archive (or SHA-256 hash thereof) that\r\n contains all the data and code necessary to make predictions on the final evaluation set (which will be released on March 4). At that point, you will then have a week to run your model on the final evaluation set and submit these predictions.
\r\nTo submit your model to the site, go to the "My Submissions" section of this competition and then click "add model hash" or "add attachment" as appropriate on your best-scoring public submission.
\r\nAlso, we've started drafting a \r\nbest practices guide for model submissions. We don't expect models submitted for this competition to conform precisely to these best practices, but wanted to go ahead and release them to get early feedback and as rough guidelines.
\r\nPlease let us know in this forum if you have any questions/comments on the model submission process or the "best practices" guide.
\r\nThanks for your participation in this contest so far, and good luck as it enters its final stages!
", "messages": [ { "id": "20402", "postDate": "02/13/2013 06:33:42", "content": "As a reminder, you have until 6:00pm UTC on February 15 to upload a hash or archive of your final model to the site. Basically, the final model should consist of a single archive (or SHA-256 hash thereof) that\r\n contains all the data and code necessary to make predictions on the final evaluation set (which will be released on March 4). At that point, you will then have a week to run your model on the final evaluation set and submit these predictions.
\r\nTo submit your model to the site, go to the "My Submissions" section of this competition and then click "add model hash" or "add attachment" as appropriate on your best-scoring public submission.
\r\nAlso, we've started drafting a \r\nbest practices guide for model submissions. We don't expect models submitted for this competition to conform precisely to these best practices, but wanted to go ahead and release them to get early feedback and as rough guidelines.
\r\nPlease let us know in this forum if you have any questions/comments on the model submission process or the "best practices" guide.
\r\nThanks for your participation in this contest so far, and good luck as it enters its final stages!
", "rawMarkdown": "", "votes": null }, { "id": "20410", "postDate": "02/13/2013 09:07:01", "content": "Will the general structure of the test days be the same as the PublicLeaderboardSet archive? Can You tell me a bit about the format of the file that contains the cutoffs for days to be predicted? Will it be the same as the\r\n days.csv as in PublicLeaderboardSet? Would there be a test_flights.csv in the folder of the test days just like there were in the PublicLeaderboardSet?
\r\nAre we supposed to provide preprocessing code for the training set or just include the data generated that we train our model on?\r\n
\r\nFor the yet unknown test set what kind of assumptions could be made? For example, my code would fail - and many other's as well I presume - if there are no flighthistoryevents.csv table (or it is called flighthistory_events.csv\r\n or whatever)? How should one overcome this problem or uncertainty?
", "rawMarkdown": "", "votes": null }, { "id": "20432", "postDate": "02/13/2013 17:23:46", "content": "Ben,
\r\nAccording to the countdown timer, the deadline would be 0AM UTC on February 15 rather than 6PM, right?
", "rawMarkdown": "", "votes": null }, { "id": "20433", "postDate": "02/13/2013 18:30:07", "content": "The rules as posted at https://www.gequest.com/c/flight/details/submission-instructions state:
\r\nYour final model must contain all code and parameter settings necessary to evaluate your models on new data, and include a README file with instructions on how to do so.
\r\nThey do not say that the software used to generate the parameters (i.e., for training the model) need to be included.
\r\nThe top post, however, says the we must upload an archive or hash, "... that contains all the code necessary to train your model and make predictions on the final evaluation set."
\r\nThis is ambiguous. Does it mean
\r\n1) Upload the code to train your model on the training data. And upload the code to make predictions on the final evaluation set.
\r\nor does it mean
\r\n2) Upload the code to train your model on the final evaluation set and make predictions on the final evaluation set.
\r\nIn other words, is this a NEW requirement that we must also upload the software to generate our parameters from the training data? In my case, that's a much larger collection of code. Or do we just upload the code and (pre-defined) tables of parameters\r\n that can apply to the new test set?
", "rawMarkdown": "", "votes": null }, { "id": "20440", "postDate": "02/13/2013 20:59:39", "content": "[quote=Guocong Song;20432]
\r\nBen,
\r\nAccording to the countdown timer, the deadline would be 0AM UTC on February 15 rather than 6PM, right?
\r\n[/quote]Thanks for pointing this out, the deadline has been updated to correspond to the model submission deadline (6pm UTC on February 15).
", "rawMarkdown": "", "votes": null }, { "id": "20447", "postDate": "02/13/2013 23:04:31", "content": "[quote=ivo;20410]
\r\nWill the general structure of the test days be the same as the PublicLeaderboardSet archive? Can You tell me a bit about the format of the file that contains the cutoffs for days to be predicted? Will it be the same as the\r\n days.csv as in PublicLeaderboardSet? Would there be a test_flights.csv in the folder of the test days just like there were in the PublicLeaderboardSet?
\r\n[/quote]The structure of the test days will be the same as that of the PublicLeaderboardSet.
\r\n[quote=ivo;20410]Are we supposed to provide preprocessing code for the training set or just include the data generated that we train our model on?[/quote]If it is straightforward for you to include this, you should go ahead and include it. However,\r\n this is not mandatory.
\r\n[quote=ivo;20410]For the yet unknown test set what kind of assumptions could be made? For example, my code would fail - and many other's as well I presume - if there are no flighthistoryevents.csv table (or it is called flighthistory_events.csv or\r\n whatever)? How should one overcome this problem or uncertainty?[/quote]Bug fixes to correct for data formatting / IO issues that arise will be permitted (you will need to make the case that this isn't a meangingful change to the model if this arises, simply\r\n fixing a data formatting bug).
", "rawMarkdown": "", "votes": null }, { "id": "20448", "postDate": "02/13/2013 23:12:56", "content": "[quote=Michael 7402;20433]
\r\nThe rules as posted at https://www.gequest.com/c/flight/details/submission-instructions state:
\r\nYour final model must contain all code and parameter settings necessary to evaluate your models on new data, and include a README file with instructions on how to do so.
\r\nThey do not say that the software used to generate the parameters (i.e., for training the model) need to be included.
\r\nThe top post, however, says the we must upload an archive or hash, "... that contains all the code necessary to train your model and make predictions on the final evaluation set."
\r\nThis is ambiguous. Does it mean
\r\n1) Upload the code to train your model on the training data. And upload the code to make predictions on the final evaluation set.
\r\nor does it mean
\r\n2) Upload the code to train your model on the final evaluation set and make predictions on the final evaluation set.
\r\nIn other words, is this a NEW requirement that we must also upload the software to generate our parameters from the training data? In my case, that's a much larger collection of code. Or do we just upload the code and (pre-defined) tables of parameters\r\n that can apply to the new test set?
\r\n[/quote]Sorry for the confusion - there are no new requirements. The submission instructions you linked have it correct, and I've edited my original post accordingly. It is not required to include the code to train your model on the training data in the\r\n model submission, only the model itself and the code to make predictions on a new day. However, if it straightforward to including the training code as well, I recommend doing so (this could make things easier to debug in the event that something doesn't work).
", "rawMarkdown": "", "votes": null }, { "id": "20506", "postDate": "02/15/2013 18:07:59", "content": "sigh... According to the uber time clock in the house the time was 5:58pm UTC when I went to find my submissions to attach a last code archive containing a little polish and a few tweaks. I was unable to access 'My Submissions' to attach the final code\r\n archive. Did the model submission deadline end early? Did anyone else run into this? Luckily I had attached a rougher version earlier...
\r\n\r\n
ps. The uber time clock is made by EndRun Technologies and is accurate to 10 micro seconds.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 20410, "author_name": "iamivo", "author_url": "", "post_date": "02/13/2013 09:07:01", "content": "Will the general structure of the test days be the same as the PublicLeaderboardSet archive? Can You tell me a bit about the format of the file that contains the cutoffs for days to be predicted? Will it be the same as the\r\n days.csv as in PublicLeaderboardSet? Would there be a test_flights.csv in the folder of the test days just like there were in the PublicLeaderboardSet?
\r\nAre we supposed to provide preprocessing code for the training set or just include the data generated that we train our model on?\r\n
\r\nFor the yet unknown test set what kind of assumptions could be made? For example, my code would fail - and many other's as well I presume - if there are no flighthistoryevents.csv table (or it is called flighthistory_events.csv\r\n or whatever)? How should one overcome this problem or uncertainty?
", "votes": null, "replies": [] }, { "id": 20432, "author_name": "songgc", "author_url": "", "post_date": "02/13/2013 17:23:46", "content": "Ben,
\r\nAccording to the countdown timer, the deadline would be 0AM UTC on February 15 rather than 6PM, right?
", "votes": null, "replies": [] }, { "id": 20433, "author_name": "budiansky", "author_url": "", "post_date": "02/13/2013 18:30:07", "content": "The rules as posted at https://www.gequest.com/c/flight/details/submission-instructions state:
\r\nYour final model must contain all code and parameter settings necessary to evaluate your models on new data, and include a README file with instructions on how to do so.
\r\nThey do not say that the software used to generate the parameters (i.e., for training the model) need to be included.
\r\nThe top post, however, says the we must upload an archive or hash, "... that contains all the code necessary to train your model and make predictions on the final evaluation set."
\r\nThis is ambiguous. Does it mean
\r\n1) Upload the code to train your model on the training data. And upload the code to make predictions on the final evaluation set.
\r\nor does it mean
\r\n2) Upload the code to train your model on the final evaluation set and make predictions on the final evaluation set.
\r\nIn other words, is this a NEW requirement that we must also upload the software to generate our parameters from the training data? In my case, that's a much larger collection of code. Or do we just upload the code and (pre-defined) tables of parameters\r\n that can apply to the new test set?
", "votes": null, "replies": [] }, { "id": 20440, "author_name": "benhamner", "author_url": "", "post_date": "02/13/2013 20:59:39", "content": "[quote=Guocong Song;20432]
\r\nBen,
\r\nAccording to the countdown timer, the deadline would be 0AM UTC on February 15 rather than 6PM, right?
\r\n[/quote]Thanks for pointing this out, the deadline has been updated to correspond to the model submission deadline (6pm UTC on February 15).
", "votes": null, "replies": [] }, { "id": 20447, "author_name": "benhamner", "author_url": "", "post_date": "02/13/2013 23:04:31", "content": "[quote=ivo;20410]
\r\nWill the general structure of the test days be the same as the PublicLeaderboardSet archive? Can You tell me a bit about the format of the file that contains the cutoffs for days to be predicted? Will it be the same as the\r\n days.csv as in PublicLeaderboardSet? Would there be a test_flights.csv in the folder of the test days just like there were in the PublicLeaderboardSet?
\r\n[/quote]The structure of the test days will be the same as that of the PublicLeaderboardSet.
\r\n[quote=ivo;20410]Are we supposed to provide preprocessing code for the training set or just include the data generated that we train our model on?[/quote]If it is straightforward for you to include this, you should go ahead and include it. However,\r\n this is not mandatory.
\r\n[quote=ivo;20410]For the yet unknown test set what kind of assumptions could be made? For example, my code would fail - and many other's as well I presume - if there are no flighthistoryevents.csv table (or it is called flighthistory_events.csv or\r\n whatever)? How should one overcome this problem or uncertainty?[/quote]Bug fixes to correct for data formatting / IO issues that arise will be permitted (you will need to make the case that this isn't a meangingful change to the model if this arises, simply\r\n fixing a data formatting bug).
", "votes": null, "replies": [] }, { "id": 20448, "author_name": "benhamner", "author_url": "", "post_date": "02/13/2013 23:12:56", "content": "[quote=Michael 7402;20433]
\r\nThe rules as posted at https://www.gequest.com/c/flight/details/submission-instructions state:
\r\nYour final model must contain all code and parameter settings necessary to evaluate your models on new data, and include a README file with instructions on how to do so.
\r\nThey do not say that the software used to generate the parameters (i.e., for training the model) need to be included.
\r\nThe top post, however, says the we must upload an archive or hash, "... that contains all the code necessary to train your model and make predictions on the final evaluation set."
\r\nThis is ambiguous. Does it mean
\r\n1) Upload the code to train your model on the training data. And upload the code to make predictions on the final evaluation set.
\r\nor does it mean
\r\n2) Upload the code to train your model on the final evaluation set and make predictions on the final evaluation set.
\r\nIn other words, is this a NEW requirement that we must also upload the software to generate our parameters from the training data? In my case, that's a much larger collection of code. Or do we just upload the code and (pre-defined) tables of parameters\r\n that can apply to the new test set?
\r\n[/quote]Sorry for the confusion - there are no new requirements. The submission instructions you linked have it correct, and I've edited my original post accordingly. It is not required to include the code to train your model on the training data in the\r\n model submission, only the model itself and the code to make predictions on a new day. However, if it straightforward to including the training code as well, I recommend doing so (this could make things easier to debug in the event that something doesn't work).
", "votes": null, "replies": [] }, { "id": 20506, "author_name": "hurlburt", "author_url": "", "post_date": "02/15/2013 18:07:59", "content": "sigh... According to the uber time clock in the house the time was 5:58pm UTC when I went to find my submissions to attach a last code archive containing a little polish and a few tweaks. I was unable to access 'My Submissions' to attach the final code\r\n archive. Did the model submission deadline end early? Did anyone else run into this? Luckily I had attached a rougher version earlier...
\r\n\r\n
ps. The uber time clock is made by EndRun Technologies and is accurate to 10 micro seconds.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "20402": "", "20410": "", "20432": "", "20433": "", "20440": "", "20447": "", "20448": "", "20506": "" }, "source": "meta" }