A question: is it OK to train models for our ensemble until the final 7 days are over? We were planning to have a priority list of models to train, and train as many of them as possible in the 7 day time-frame. The ensembling is automatic and can deal with any number of models.
\n\nTherefore, our final result would be exactly reproducible, but only after the end of the competition, not at the end of the first round.
\n\nWould this strategy be allowed? There are no parameters which will change in the last week using this approach, the final code will only be dependent on the speed of our machines. The less GPU's break down, the more models will be included in our ensemble. Yet, it will be fully reproducible after the final deadline.
", "messages": [ { "id": "110554", "postDate": "03/06/2016 15:47:13", "content": "A question: is it OK to train models for our ensemble until the final 7 days are over? We were planning to have a priority list of models to train, and train as many of them as possible in the 7 day time-frame. The ensembling is automatic and can deal with any number of models.
\n\nTherefore, our final result would be exactly reproducible, but only after the end of the competition, not at the end of the first round.
\n\nWould this strategy be allowed? There are no parameters which will change in the last week using this approach, the final code will only be dependent on the speed of our machines. The less GPU's break down, the more models will be included in our ensemble. Yet, it will be fully reproducible after the final deadline.
", "rawMarkdown": "A question: is it OK to train models for our ensemble until the final 7 days are over? We were planning to have a priority list of models to train, and train as many of them as possible in the 7 day time-frame. The ensembling is automatic and can deal with any number of models.\r\n\r\nTherefore, our final result would be exactly reproducible, but only after the end of the competition, not at the end of the first round.\r\n\r\nWould this strategy be allowed? There are no parameters which will change in the last week using this approach, the final code will only be dependent on the speed of our machines. The less GPU's break down, the more models will be included in our ensemble. Yet, it will be fully reproducible after the final deadline.", "votes": null }, { "id": "110568", "postDate": "03/06/2016 18:14:29", "content": "The model you upload needs to be the model that generates your selected submissions.
", "rawMarkdown": "The model you upload needs to be the model that generates your selected submissions.", "votes": null }, { "id": "110572", "postDate": "03/06/2016 18:56:16", "content": "I'm sorry, but I don't understand. Aren't the selected submissions the ones you generate in the second round? What is the relevance of the selected submissions in the first round?
\n\nI thought I understood, but apparently I'm missing something. As I understand it now:
\n\n1: Why mark 2 submissions which are not relevant? Or are they there to limit your model's behaviour?
\n\n2: Is a hard coded computing time limit a valid part of a model?
", "rawMarkdown": "I'm sorry, but I don't understand. Aren't the selected submissions the ones you generate in the second round? What is the relevance of the selected submissions in the first round?\r\n\r\nI thought I understood, but apparently I'm missing something. As I understand it now:\r\n\r\n - We submit our final model and need to select 2 final submissions (which will not be relevant to the final leaderboard) tomorrow.\r\n - We get more data, re-train and predict with the same model we submitted, and make 2 new submissions (which do count towards the final leaderboard).\r\n\r\n1: Why mark 2 submissions which are not relevant? Or are they there to limit your model's behaviour?\r\n\r\n2: Is a hard coded computing time limit a valid part of a model?", "votes": null }, { "id": "110575", "postDate": "03/06/2016 19:19:29", "content": "The selected submissions are for the second stage. The flow should go:
\n\nDoes this make sense? Perhaps I'm not fully understanding your initial question?
", "rawMarkdown": "The selected submissions are for the second stage. The flow should go:\r\n\r\n 1. Upload your code before deadline tomorrow\r\n 2. Test set data is released, validation set answers released, leaderboard cleared\r\n 3. (Optionally) retrain your model using validation answers\r\n 4. Predict for the test set using the code you uploaded for (1)\r\n 5. Select up to 2 submissions\r\n\r\nDoes this make sense? Perhaps I'm not fully understanding your initial question?", "votes": null }, { "id": "110579", "postDate": "03/06/2016 19:36:24", "content": "Yes, that's indeed my first question! Thank you for answering! :)
\n\nI hate to push this, but can you also answer my second question? We reckon it's in the grey area of the rules, and we would like to make it black or white before we submit our final model.
\n\nQ: Is a hard coded computing time limit a valid part of a model? So, is it okay to have a model which stops computing after x days?\nCode like this:
\n\nif time.time() - start_time > SIX_DAYS:\n sys.exit(0)\nDoes it count as both reproducible and non-parametric?
", "rawMarkdown": "Yes, that's indeed my first question! Thank you for answering! :)\r\n\r\nI hate to push this, but can you also answer my second question? We reckon it's in the grey area of the rules, and we would like to make it black or white before we submit our final model.\r\n\r\nQ: Is a hard coded computing time limit a valid part of a model? So, is it okay to have a model which stops computing after x days?\r\nCode like this:\r\n\r\n if time.time() - start_time > SIX_DAYS:\r\n sys.exit(0)\r\n\r\nDoes it count as both reproducible and non-parametric?", "votes": null }, { "id": "110590", "postDate": "03/06/2016 21:34:53", "content": "William, \nBased on your explanation of the flow for the second stage,\nCould you please clarify the following questions
\n\nQ1 - Will you release the new test set and the validation set answers after the deadline on Monday ?\nQ2 - In the case of using the validation dataset to re-train the model(s), if we are not allowed to change the workflow or the parameters of the model(s) in stage 1, how we can/should generate two or more different submission during the stage 2\n a - without changing either the workflow of the data pre-processing or \n b - changing/updating (optimizing) the hyper-parameters of the model(s) submitted
\n in the first stage
Thanks
", "rawMarkdown": "William, \r\nBased on your explanation of the flow for the second stage,\r\nCould you please clarify the following questions\r\n\r\nQ1 - Will you release the new test set and the validation set answers after the deadline on Monday ?\r\nQ2 - In the case of using the validation dataset to re-train the model(s), if we are not allowed to change the workflow or the parameters of the model(s) in stage 1, how we can/should generate two or more different submission during the stage 2\r\n a - without changing either the workflow of the data pre-processing or \r\n b - changing/updating (optimizing) the hyper-parameters of the model(s) submitted \r\n in the first stage\r\n\r\nThanks", "votes": null }, { "id": "110591", "postDate": "03/06/2016 21:35:34", "content": "The only real time limit is that your code must create test set predictions in the (approximately) 7 day window of stage 2.
\n\nYour code should be deterministic* (per the rules, "The delivered software code must be capable of generating the winning Submission"). It's ultimately your decision whether such code would return the same submission file if run again, potentially on different hardware. Generally this means specifying a number of iterations, as opposed to time limits.
\n\n* We will apply a reasonable definition of reproducibility (think floating point errors or very very minor decimal place differences), but the smart and safe thing to do is aim for true reproducibility.
", "rawMarkdown": "The only real time limit is that your code must create test set predictions in the (approximately) 7 day window of stage 2.\r\n\r\nYour code should be deterministic* (per the rules, \"The delivered software code must be capable of generating the winning Submission\"). It's ultimately your decision whether such code would return the same submission file if run again, potentially on different hardware. Generally this means specifying a number of iterations, as opposed to time limits.\r\n\r\n\\* We will apply a reasonable definition of reproducibility (think floating point errors or very very minor decimal place differences), but the smart and safe thing to do is aim for true reproducibility.", "votes": null }, { "id": "110593", "postDate": "03/06/2016 21:42:02", "content": "@ELOCAN yes, the new test set and validation labels are released after the model upload deadline.
\n\nI think I covered your other question here - https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/19148/1st-deadline-model-upload/109368#post109368
\n\nLet me know if that doesn't answer your question.
", "rawMarkdown": "ELOCAN yes, the new test set and validation labels are released after the model upload deadline.\r\n\r\nI think I covered your other question here - https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/19148/1st-deadline-model-upload/109368#post109368\r\n\r\nLet me know if that doesn't answer your question.", "votes": null }, { "id": "110642", "postDate": "03/07/2016 08:14:28", "content": "@William Sorry, the rules are still not clear for me. Are we or are we not allowed to modify any parameter once our model submitted? If there are parameters that we can modify, could you please specify which ones? A number of iteration is a parameter, so can we modify the number of iterations?\nFor example, if we follow your guideline, can we specify in the file SETTINGS.json a number of training iterations for each model (in the case our solution combines different models)?
\n\nFurthermore, you said previously that "You will upload a model for each of your two selected submissions". I don't understand, can we upload two models? On the upload section, it is indicated that we can upload one model.
", "rawMarkdown": "William Sorry, the rules are still not clear for me. Are we or are we not allowed to modify any parameter once our model submitted? If there are parameters that we can modify, could you please specify which ones? A number of iteration is a parameter, so can we modify the number of iterations?\r\nFor example, if we follow your guideline, can we specify in the file SETTINGS.json a number of training iterations for each model (in the case our solution combines different models)?\r\n\r\nFurthermore, you said previously that \"You will upload a model for each of your two selected submissions\". I don't understand, can we upload two models? On the upload section, it is indicated that we can upload one model.", "votes": null }, { "id": "110669", "postDate": "03/07/2016 13:05:49", "content": "Any modifications should be to accommodate new data and retrain on the validation set. You should not modify aspects like algorithm parameters, number of iterations, training times, etc. The idea is that you are merely applying your already uploaded model to a new set of data.
\n\nYes, you should upload two models for two different submissions. You zip both in the same archive. You may either isolate them as two separate codebases, or keep them as one codebase and make clear how each submission is generated.
", "rawMarkdown": "Any modifications should be to accommodate new data and retrain on the validation set. You should not modify aspects like algorithm parameters, number of iterations, training times, etc. The idea is that you are merely applying your already uploaded model to a new set of data.\r\n\r\nYes, you should upload two models for two different submissions. You zip both in the same archive. You may either isolate them as two separate codebases, or keep them as one codebase and make clear how each submission is generated.", "votes": null }, { "id": "110680", "postDate": "03/07/2016 14:21:09", "content": "Thank you for your reply, it's more clear now. I am using a low-end computer without GPU, so I was hoping to be able to adapt the number of iterations depending on how the computer withstand the load. Since it seems it is not possible, I will just go for a low amount of iterations.
", "rawMarkdown": "Thank you for your reply, it's more clear now. I am using a low-end computer without GPU, so I was hoping to be able to adapt the number of iterations depending on how the computer withstand the load. Since it seems it is not possible, I will just go for a low amount of iterations.", "votes": null }, { "id": "110811", "postDate": "03/08/2016 12:47:53", "content": "So i understand that its not ok to change parameters like learning rate, training/validation split ratio and batch size.
", "rawMarkdown": "So i understand that its not ok to change parameters like learning rate, training/validation split ratio and batch size.", "votes": null } ], "comments": [ { "id": 110568, "author_name": "wcukierski", "author_url": "", "post_date": "03/06/2016 18:14:29", "content": "The model you upload needs to be the model that generates your selected submissions.
", "votes": null, "replies": [] }, { "id": 110572, "author_name": "de317070", "author_url": "", "post_date": "03/06/2016 18:56:16", "content": "I'm sorry, but I don't understand. Aren't the selected submissions the ones you generate in the second round? What is the relevance of the selected submissions in the first round?
\n\nI thought I understood, but apparently I'm missing something. As I understand it now:
\n\n1: Why mark 2 submissions which are not relevant? Or are they there to limit your model's behaviour?
\n\n2: Is a hard coded computing time limit a valid part of a model?
", "votes": null, "replies": [] }, { "id": 110575, "author_name": "wcukierski", "author_url": "", "post_date": "03/06/2016 19:19:29", "content": "The selected submissions are for the second stage. The flow should go:
\n\nDoes this make sense? Perhaps I'm not fully understanding your initial question?
", "votes": null, "replies": [] }, { "id": 110579, "author_name": "de317070", "author_url": "", "post_date": "03/06/2016 19:36:24", "content": "Yes, that's indeed my first question! Thank you for answering! :)
\n\nI hate to push this, but can you also answer my second question? We reckon it's in the grey area of the rules, and we would like to make it black or white before we submit our final model.
\n\nQ: Is a hard coded computing time limit a valid part of a model? So, is it okay to have a model which stops computing after x days?\nCode like this:
\n\nif time.time() - start_time > SIX_DAYS:\n sys.exit(0)\nDoes it count as both reproducible and non-parametric?
", "votes": null, "replies": [] }, { "id": 110590, "author_name": "data2015", "author_url": "", "post_date": "03/06/2016 21:34:53", "content": "William, \nBased on your explanation of the flow for the second stage,\nCould you please clarify the following questions
\n\nQ1 - Will you release the new test set and the validation set answers after the deadline on Monday ?\nQ2 - In the case of using the validation dataset to re-train the model(s), if we are not allowed to change the workflow or the parameters of the model(s) in stage 1, how we can/should generate two or more different submission during the stage 2\n a - without changing either the workflow of the data pre-processing or \n b - changing/updating (optimizing) the hyper-parameters of the model(s) submitted
\n in the first stage
Thanks
", "votes": null, "replies": [] }, { "id": 110591, "author_name": "wcukierski", "author_url": "", "post_date": "03/06/2016 21:35:34", "content": "The only real time limit is that your code must create test set predictions in the (approximately) 7 day window of stage 2.
\n\nYour code should be deterministic* (per the rules, "The delivered software code must be capable of generating the winning Submission"). It's ultimately your decision whether such code would return the same submission file if run again, potentially on different hardware. Generally this means specifying a number of iterations, as opposed to time limits.
\n\n* We will apply a reasonable definition of reproducibility (think floating point errors or very very minor decimal place differences), but the smart and safe thing to do is aim for true reproducibility.
", "votes": null, "replies": [] }, { "id": 110593, "author_name": "wcukierski", "author_url": "", "post_date": "03/06/2016 21:42:02", "content": "@ELOCAN yes, the new test set and validation labels are released after the model upload deadline.
\n\nI think I covered your other question here - https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/19148/1st-deadline-model-upload/109368#post109368
\n\nLet me know if that doesn't answer your question.
", "votes": null, "replies": [] }, { "id": 110642, "author_name": "vincentl", "author_url": "", "post_date": "03/07/2016 08:14:28", "content": "@William Sorry, the rules are still not clear for me. Are we or are we not allowed to modify any parameter once our model submitted? If there are parameters that we can modify, could you please specify which ones? A number of iteration is a parameter, so can we modify the number of iterations?\nFor example, if we follow your guideline, can we specify in the file SETTINGS.json a number of training iterations for each model (in the case our solution combines different models)?
\n\nFurthermore, you said previously that "You will upload a model for each of your two selected submissions". I don't understand, can we upload two models? On the upload section, it is indicated that we can upload one model.
", "votes": null, "replies": [] }, { "id": 110669, "author_name": "wcukierski", "author_url": "", "post_date": "03/07/2016 13:05:49", "content": "Any modifications should be to accommodate new data and retrain on the validation set. You should not modify aspects like algorithm parameters, number of iterations, training times, etc. The idea is that you are merely applying your already uploaded model to a new set of data.
\n\nYes, you should upload two models for two different submissions. You zip both in the same archive. You may either isolate them as two separate codebases, or keep them as one codebase and make clear how each submission is generated.
", "votes": null, "replies": [] }, { "id": 110680, "author_name": "vincentl", "author_url": "", "post_date": "03/07/2016 14:21:09", "content": "Thank you for your reply, it's more clear now. I am using a low-end computer without GPU, so I was hoping to be able to adapt the number of iterations depending on how the computer withstand the load. Since it seems it is not possible, I will just go for a low amount of iterations.
", "votes": null, "replies": [] }, { "id": 110811, "author_name": "rajgot", "author_url": "", "post_date": "03/08/2016 12:47:53", "content": "So i understand that its not ok to change parameters like learning rate, training/validation split ratio and batch size.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "110554": "A question: is it OK to train models for our ensemble until the final 7 days are over? We were planning to have a priority list of models to train, and train as many of them as possible in the 7 day time-frame. The ensembling is automatic and can deal with any number of models.\r\n\r\nTherefore, our final result would be exactly reproducible, but only after the end of the competition, not at the end of the first round.\r\n\r\nWould this strategy be allowed? There are no parameters which will change in the last week using this approach, the final code will only be dependent on the speed of our machines. The less GPU's break down, the more models will be included in our ensemble. Yet, it will be fully reproducible after the final deadline.", "110568": "The model you upload needs to be the model that generates your selected submissions.", "110572": "I'm sorry, but I don't understand. Aren't the selected submissions the ones you generate in the second round? What is the relevance of the selected submissions in the first round?\r\n\r\nI thought I understood, but apparently I'm missing something. As I understand it now:\r\n\r\n - We submit our final model and need to select 2 final submissions (which will not be relevant to the final leaderboard) tomorrow.\r\n - We get more data, re-train and predict with the same model we submitted, and make 2 new submissions (which do count towards the final leaderboard).\r\n\r\n1: Why mark 2 submissions which are not relevant? Or are they there to limit your model's behaviour?\r\n\r\n2: Is a hard coded computing time limit a valid part of a model?", "110575": "The selected submissions are for the second stage. The flow should go:\r\n\r\n 1. Upload your code before deadline tomorrow\r\n 2. Test set data is released, validation set answers released, leaderboard cleared\r\n 3. (Optionally) retrain your model using validation answers\r\n 4. Predict for the test set using the code you uploaded for (1)\r\n 5. Select up to 2 submissions\r\n\r\nDoes this make sense? Perhaps I'm not fully understanding your initial question?", "110579": "Yes, that's indeed my first question! Thank you for answering! :)\r\n\r\nI hate to push this, but can you also answer my second question? We reckon it's in the grey area of the rules, and we would like to make it black or white before we submit our final model.\r\n\r\nQ: Is a hard coded computing time limit a valid part of a model? So, is it okay to have a model which stops computing after x days?\r\nCode like this:\r\n\r\n if time.time() - start_time > SIX_DAYS:\r\n sys.exit(0)\r\n\r\nDoes it count as both reproducible and non-parametric?", "110590": "William, \r\nBased on your explanation of the flow for the second stage,\r\nCould you please clarify the following questions\r\n\r\nQ1 - Will you release the new test set and the validation set answers after the deadline on Monday ?\r\nQ2 - In the case of using the validation dataset to re-train the model(s), if we are not allowed to change the workflow or the parameters of the model(s) in stage 1, how we can/should generate two or more different submission during the stage 2\r\n a - without changing either the workflow of the data pre-processing or \r\n b - changing/updating (optimizing) the hyper-parameters of the model(s) submitted \r\n in the first stage\r\n\r\nThanks", "110591": "The only real time limit is that your code must create test set predictions in the (approximately) 7 day window of stage 2.\r\n\r\nYour code should be deterministic* (per the rules, \"The delivered software code must be capable of generating the winning Submission\"). It's ultimately your decision whether such code would return the same submission file if run again, potentially on different hardware. Generally this means specifying a number of iterations, as opposed to time limits.\r\n\r\n\\* We will apply a reasonable definition of reproducibility (think floating point errors or very very minor decimal place differences), but the smart and safe thing to do is aim for true reproducibility.", "110593": "ELOCAN yes, the new test set and validation labels are released after the model upload deadline.\r\n\r\nI think I covered your other question here - https://www.kaggle.com/c/second-annual-data-science-bowl/forums/t/19148/1st-deadline-model-upload/109368#post109368\r\n\r\nLet me know if that doesn't answer your question.", "110642": "William Sorry, the rules are still not clear for me. Are we or are we not allowed to modify any parameter once our model submitted? If there are parameters that we can modify, could you please specify which ones? A number of iteration is a parameter, so can we modify the number of iterations?\r\nFor example, if we follow your guideline, can we specify in the file SETTINGS.json a number of training iterations for each model (in the case our solution combines different models)?\r\n\r\nFurthermore, you said previously that \"You will upload a model for each of your two selected submissions\". I don't understand, can we upload two models? On the upload section, it is indicated that we can upload one model.", "110669": "Any modifications should be to accommodate new data and retrain on the validation set. You should not modify aspects like algorithm parameters, number of iterations, training times, etc. The idea is that you are merely applying your already uploaded model to a new set of data.\r\n\r\nYes, you should upload two models for two different submissions. You zip both in the same archive. You may either isolate them as two separate codebases, or keep them as one codebase and make clear how each submission is generated.", "110680": "Thank you for your reply, it's more clear now. I am using a low-end computer without GPU, so I was hoping to be able to adapt the number of iterations depending on how the computer withstand the load. Since it seems it is not possible, I will just go for a low amount of iterations.", "110811": "So i understand that its not ok to change parameters like learning rate, training/validation split ratio and batch size." }, "source": "meta" }