I will assume tuning ensemble weights based on the release of stage 1 labels has to be allowed. Admins please acknowledge!!!! if I had (0.4 * model1 + 0.6 * model2) then I retrained my models and find (0.5 * model1 + 0.5 * model2) is optimal I should be allowed to manually change this. Note model architecture does not change just ensemble weights based on released stage 1 labels. Prompt response will be greatly appreciated.
", "messages": [ { "id": "193564", "postDate": "06/16/2017 22:32:28", "content": "I will assume tuning ensemble weights based on the release of stage 1 labels has to be allowed. Admins please acknowledge!!!! if I had (0.4 * model1 + 0.6 * model2) then I retrained my models and find (0.5 * model1 + 0.5 * model2) is optimal I should be allowed to manually change this. Note model architecture does not change just ensemble weights based on released stage 1 labels. Prompt response will be greatly appreciated.
", "rawMarkdown": "I will assume tuning ensemble weights based on the release of stage 1 labels has to be allowed. Admins please acknowledge!!!! if I had (0.4 * model1 + 0.6 * model2) then I retrained my models and find (0.5 * model1 + 0.5 * model2) is optimal I should be allowed to manually change this. Note model architecture does not change just ensemble weights based on released stage 1 labels. Prompt response will be greatly appreciated.", "votes": null }, { "id": "193570", "postDate": "06/16/2017 23:26:33", "content": "The whole pipeline of training and making inferences has to be visible in the model you uploaded. You can't tune anything that affects the predicted probabilities once stage2 begins.
\n\nAs for the release of stage1 labels, I don't really see how they are relevant here. In any case, the fact that they will be released was announced before stage2 began.
", "rawMarkdown": "The whole pipeline of training and making inferences has to be visible in the model you uploaded. You can't tune anything that affects the predicted probabilities once stage2 begins. \n\nAs for the release of stage1 labels, I don't really see how they are relevant here. In any case, the fact that they will be released was announced before stage2 began.", "votes": null }, { "id": "193581", "postDate": "06/17/2017 00:38:15", "content": "I respectfully disagree, with the release of stage1 labels we are allowed to re-train our models with the additional stage 1 test data, there is no debating that. Now, in stage 1 lets say you created a 80/20 train validation split and trained 2 models on the 80/20 split. you then created a weighted ensemble using those 2 models and used the 20 % holdout to find optimal ensemble weights. You upload your model saying 0.6 model1 + 0.4 model2. Now stage2 begins they release stage1 labels and say you can retrain your models with the additional stage 1 test data. You now create a new 87/13 split where the 13 % is the stage 1 test data or however you want to shuffle and split, you are now effectively training on 100% of stage1 train data and using the stage 1 test data as your new validation data. Are you telling me that the ensemble weights you used on your 80/20 split are optimal for this new 87/13 split? Allowing retraining implies that ensemble weights also have to be readjusted.
", "rawMarkdown": "I respectfully disagree, with the release of stage1 labels we are allowed to re-train our models with the additional stage 1 test data, there is no debating that. Now, in stage 1 lets say you created a 80/20 train validation split and trained 2 models on the 80/20 split. you then created a weighted ensemble using those 2 models and used the 20 % holdout to find optimal ensemble weights. You upload your model saying 0.6 model1 + 0.4 model2. Now stage2 begins they release stage1 labels and say you can retrain your models with the additional stage 1 test data. You now create a new 87/13 split where the 13 % is the stage 1 test data or however you want to shuffle and split, you are now effectively training on 100% of stage1 train data and using the stage 1 test data as your new validation data. Are you telling me that the ensemble weights you used on your 80/20 split are optimal for this new 87/13 split? Allowing retraining implies that ensemble weights also have to be readjusted.", "votes": null }, { "id": "193593", "postDate": "06/17/2017 01:43:55", "content": "I do agree that when training with more data different parameter values become better.
\n\nHowever, any parameter tuning is prohibited during stage 2 as you already see the new testing data and can tune the parameters to fit it, which removes the whole point of having the second stage.
\n\nAllowing retraining doesn't imply anything, it only means the instructions included in your uploaded model could state that you are going to retrain with the test data appended. All the details (number of iterations, learning rates..) had to be chosen before stage 2.
", "rawMarkdown": "I do agree that when training with more data different parameter values become better.\n\nHowever, any parameter tuning is prohibited during stage 2 as you already see the new testing data and can tune the parameters to fit it, which removes the whole point of having the second stage. \n\nAllowing retraining doesn't imply anything, it only means the instructions included in your uploaded model could state that you are going to retrain with the test data appended. All the details (number of iterations, learning rates..) had to be chosen before stage 2.", "votes": null }, { "id": "193596", "postDate": "06/17/2017 02:00:01", "content": "hyper-parameter tuning (model architecture e.t.c) is prohibited not parameter tuning, I will simply refer you to the 2 stage competition FAQ. The question here for me to the admins given the scenario I have described is plain and simple. Are we allowed to adjust the ensemble weights when retraining is allowed. Its a simple yes or no! it not a big deal!
", "rawMarkdown": "hyper-parameter tuning (model architecture e.t.c) is prohibited not parameter tuning, I will simply refer you to the 2 stage competition [FAQ][1]. The question here for me to the admins given the scenario I have described is plain and simple. Are we allowed to adjust the ensemble weights when retraining is allowed. Its a simple yes or no! it not a big deal!\n\n\n [1]: https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580", "votes": null }, { "id": "193601", "postDate": "06/17/2017 03:02:34", "content": "I don't think you can adjust the weights according to stage 1 test data. All your training/validation data should be fixed before the end of stage 1. Otherwise the competition will be meaningless.
", "rawMarkdown": "I don't think you can adjust the weights according to stage 1 test data. All your training/validation data should be fixed before the end of stage 1. Otherwise the competition will be meaningless.", "votes": null }, { "id": "193614", "postDate": "06/17/2017 05:13:59", "content": "I guess you should have uploaded code to find optimal ensemble weights in automated way. I think you can not do this manually to claim a prize.
", "rawMarkdown": "I guess you should have uploaded code to find optimal ensemble weights in automated way. I think you can not do this manually to claim a prize.", "votes": null }, { "id": "193629", "postDate": "06/17/2017 08:10:17", "content": "It is not allowed. Just “non scientific” alterations are allowed. Just read the rules in the FAQ
\n\nWhat happens if I want to change something in my code in the second stage?\nWe expect you may need to make some “non scientific” alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.
", "rawMarkdown": "It is **not** allowed. Just “non scientific” alterations are allowed. Just read the rules in the [FAQ][1]\n\n\n [1]: https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580\n\n**What happens if I want to change something in my code in the second stage?**\nWe expect you may need to make some “non scientific” alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.", "votes": null }, { "id": "193635", "postDate": "06/17/2017 08:57:27", "content": "It's plainly stated in the FAQ: no manual parameter tuning is allowed, only (fully) automatic. I fail to see how that doesn't answer your question. If you have an automatic way to adjust those weights w.r.t. the amount of training data (and it's explained in the instructions you've uploaded with your code) then go ahead.
", "rawMarkdown": "It's plainly stated in the FAQ: no manual parameter tuning is allowed, only (fully) automatic. I fail to see how that doesn't answer your question. If you have an automatic way to adjust those weights w.r.t. the amount of training data (and it's explained in the instructions you've uploaded with your code) then go ahead.", "votes": null }, { "id": "193641", "postDate": "06/17/2017 10:02:19", "content": "\"You are allowed to re-train your model (including the stage one data), but your code should not change.\". You can't change hard-coded parameter values without changing the code.
\n\nI do like the idea of changing \"parameters\" when only \"meta-parameter\" tuning is prohibited. You could also have parameters that control how the learning rate changes, for example, they would be \"meta-meta-parameters\" so you could change them too.
", "rawMarkdown": "\"You are allowed to re-train your model (including the stage one data), but your code should not change.\". You can't change hard-coded parameter values without changing the code.\n\nI do like the idea of changing \"parameters\" when only \"meta-parameter\" tuning is prohibited. You could also have parameters that control how the learning rate changes, for example, they would be \"meta-meta-parameters\" so you could change them too.", "votes": null }, { "id": "193659", "postDate": "06/17/2017 11:55:16", "content": "@ZadrraS it does fail to answer my question. When you retrain your models are you automatically adjusting initial learning rate like @bobutis said? are you automatically adjusting initial data augmentation parameters e.t.c? Are your feet going to be held to the fire because you didn't? There are grey areas, all I am looking for is clarification on this specific case. Its not a big deal for us, the gains we see are small, we will keep them as our uploaded code if we have to, but these are practical things to think about for future 2 stage competitions, that's all.
", "rawMarkdown": "ZadrraS it does fail to answer my question. When you retrain your models are you automatically adjusting initial learning rate like @bobutis said? are you automatically adjusting initial data augmentation parameters e.t.c? Are your feet going to be held to the fire because you didn't? There are grey areas, all I am looking for is clarification on this specific case. Its not a big deal for us, the gains we see are small, we will keep them as our uploaded code if we have to, but these are practical things to think about for future 2 stage competitions, that's all.", "votes": null }, { "id": "193666", "postDate": "06/17/2017 12:44:14", "content": "I don't see this as a gray area at all. I don't adjust those things automatically, because I don't know a good way of doing it in the context of this competition. I don't adjust them by hand in stage2 either, because that would be against the rules. What changes have to be made to the whole pipeline, when retraining, were described in the instructions our team uploaded at the end of stage1. Some of the steps required some guesswork on how our models would react to more data, but all of the steps are laid out concretely and can be reproduced by anybody who picks up our code.
", "rawMarkdown": "I don't see this as a gray area at all. I don't adjust those things automatically, because I don't know a good way of doing it in the context of this competition. I don't adjust them by hand in stage2 either, because that would be against the rules. What changes have to be made to the whole pipeline, when retraining, were described in the instructions our team uploaded at the end of stage1. Some of the steps required some guesswork on how our models would react to more data, but all of the steps are laid out concretely and can be reproduced by anybody who picks up our code.", "votes": null }, { "id": "193667", "postDate": "06/17/2017 12:54:01", "content": "adjusting your learning rate in stage 2 would be against the rules?
", "rawMarkdown": "adjusting your learning rate in stage 2 would be against the rules?", "votes": null }, { "id": "193691", "postDate": "06/17/2017 15:25:45", "content": "If you didn't describe in the instructions you've uploaded how exactly that was going to be done, then yes.
", "rawMarkdown": "If you didn't describe in the instructions you've uploaded how exactly that was going to be done, then yes.", "votes": null }, { "id": "194203", "postDate": "06/19/2017 16:52:26", "content": "I think the whole point is that anything you do should be automated. If it isn't automated, it can't be changed.
\n\nI have a question about the two submissions feature. We are allowed to mark two submissions as our submissions to be included. Does this mean that if we uploaded two different scripts in our model submission we can use each script to produce a different prediction file, submit both of them, and our final score will be the min score between the two files?
", "rawMarkdown": "I think the whole point is that anything you do should be automated. If it isn't automated, it can't be changed.\n\nI have a question about the two submissions feature. We are allowed to mark two submissions as our submissions to be included. Does this mean that if we uploaded two different scripts in our model submission we can use each script to produce a different prediction file, submit both of them, and our final score will be the min score between the two files?", "votes": null } ], "comments": [ { "id": 193570, "author_name": "bobutis", "author_url": "", "post_date": "06/16/2017 23:26:33", "content": "The whole pipeline of training and making inferences has to be visible in the model you uploaded. You can't tune anything that affects the predicted probabilities once stage2 begins.
\n\nAs for the release of stage1 labels, I don't really see how they are relevant here. In any case, the fact that they will be released was announced before stage2 began.
", "votes": null, "replies": [ { "id": 193581, "author_name": "godaibo", "author_url": "", "post_date": "06/17/2017 00:38:15", "content": "I respectfully disagree, with the release of stage1 labels we are allowed to re-train our models with the additional stage 1 test data, there is no debating that. Now, in stage 1 lets say you created a 80/20 train validation split and trained 2 models on the 80/20 split. you then created a weighted ensemble using those 2 models and used the 20 % holdout to find optimal ensemble weights. You upload your model saying 0.6 model1 + 0.4 model2. Now stage2 begins they release stage1 labels and say you can retrain your models with the additional stage 1 test data. You now create a new 87/13 split where the 13 % is the stage 1 test data or however you want to shuffle and split, you are now effectively training on 100% of stage1 train data and using the stage 1 test data as your new validation data. Are you telling me that the ensemble weights you used on your 80/20 split are optimal for this new 87/13 split? Allowing retraining implies that ensemble weights also have to be readjusted.
", "votes": null, "replies": [] }, { "id": 193593, "author_name": "bobutis", "author_url": "", "post_date": "06/17/2017 01:43:55", "content": "I do agree that when training with more data different parameter values become better.
\n\nHowever, any parameter tuning is prohibited during stage 2 as you already see the new testing data and can tune the parameters to fit it, which removes the whole point of having the second stage.
\n\nAllowing retraining doesn't imply anything, it only means the instructions included in your uploaded model could state that you are going to retrain with the test data appended. All the details (number of iterations, learning rates..) had to be chosen before stage 2.
", "votes": null, "replies": [] }, { "id": 193596, "author_name": "godaibo", "author_url": "", "post_date": "06/17/2017 02:00:01", "content": "hyper-parameter tuning (model architecture e.t.c) is prohibited not parameter tuning, I will simply refer you to the 2 stage competition FAQ. The question here for me to the admins given the scenario I have described is plain and simple. Are we allowed to adjust the ensemble weights when retraining is allowed. Its a simple yes or no! it not a big deal!
", "votes": null, "replies": [] }, { "id": 193635, "author_name": "zadrras", "author_url": "", "post_date": "06/17/2017 08:57:27", "content": "It's plainly stated in the FAQ: no manual parameter tuning is allowed, only (fully) automatic. I fail to see how that doesn't answer your question. If you have an automatic way to adjust those weights w.r.t. the amount of training data (and it's explained in the instructions you've uploaded with your code) then go ahead.
", "votes": null, "replies": [] }, { "id": 193641, "author_name": "bobutis", "author_url": "", "post_date": "06/17/2017 10:02:19", "content": "\"You are allowed to re-train your model (including the stage one data), but your code should not change.\". You can't change hard-coded parameter values without changing the code.
\n\nI do like the idea of changing \"parameters\" when only \"meta-parameter\" tuning is prohibited. You could also have parameters that control how the learning rate changes, for example, they would be \"meta-meta-parameters\" so you could change them too.
", "votes": null, "replies": [] }, { "id": 193659, "author_name": "godaibo", "author_url": "", "post_date": "06/17/2017 11:55:16", "content": "@ZadrraS it does fail to answer my question. When you retrain your models are you automatically adjusting initial learning rate like @bobutis said? are you automatically adjusting initial data augmentation parameters e.t.c? Are your feet going to be held to the fire because you didn't? There are grey areas, all I am looking for is clarification on this specific case. Its not a big deal for us, the gains we see are small, we will keep them as our uploaded code if we have to, but these are practical things to think about for future 2 stage competitions, that's all.
", "votes": null, "replies": [] }, { "id": 193666, "author_name": "zadrras", "author_url": "", "post_date": "06/17/2017 12:44:14", "content": "I don't see this as a gray area at all. I don't adjust those things automatically, because I don't know a good way of doing it in the context of this competition. I don't adjust them by hand in stage2 either, because that would be against the rules. What changes have to be made to the whole pipeline, when retraining, were described in the instructions our team uploaded at the end of stage1. Some of the steps required some guesswork on how our models would react to more data, but all of the steps are laid out concretely and can be reproduced by anybody who picks up our code.
", "votes": null, "replies": [] }, { "id": 193667, "author_name": "godaibo", "author_url": "", "post_date": "06/17/2017 12:54:01", "content": "adjusting your learning rate in stage 2 would be against the rules?
", "votes": null, "replies": [] }, { "id": 193691, "author_name": "zadrras", "author_url": "", "post_date": "06/17/2017 15:25:45", "content": "If you didn't describe in the instructions you've uploaded how exactly that was going to be done, then yes.
", "votes": null, "replies": [] } ] }, { "id": 193601, "author_name": "wangg12", "author_url": "", "post_date": "06/17/2017 03:02:34", "content": "I don't think you can adjust the weights according to stage 1 test data. All your training/validation data should be fixed before the end of stage 1. Otherwise the competition will be meaningless.
", "votes": null, "replies": [] }, { "id": 193614, "author_name": "victorsd", "author_url": "", "post_date": "06/17/2017 05:13:59", "content": "I guess you should have uploaded code to find optimal ensemble weights in automated way. I think you can not do this manually to claim a prize.
", "votes": null, "replies": [] }, { "id": 193629, "author_name": "titericz", "author_url": "", "post_date": "06/17/2017 08:10:17", "content": "It is not allowed. Just “non scientific” alterations are allowed. Just read the rules in the FAQ
\n\nWhat happens if I want to change something in my code in the second stage?\nWe expect you may need to make some “non scientific” alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.
", "votes": null, "replies": [] }, { "id": 194203, "author_name": "gkericks", "author_url": "", "post_date": "06/19/2017 16:52:26", "content": "I think the whole point is that anything you do should be automated. If it isn't automated, it can't be changed.
\n\nI have a question about the two submissions feature. We are allowed to mark two submissions as our submissions to be included. Does this mean that if we uploaded two different scripts in our model submission we can use each script to produce a different prediction file, submit both of them, and our final score will be the min score between the two files?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "193564": "I will assume tuning ensemble weights based on the release of stage 1 labels has to be allowed. Admins please acknowledge!!!! if I had (0.4 * model1 + 0.6 * model2) then I retrained my models and find (0.5 * model1 + 0.5 * model2) is optimal I should be allowed to manually change this. Note model architecture does not change just ensemble weights based on released stage 1 labels. Prompt response will be greatly appreciated.", "193570": "The whole pipeline of training and making inferences has to be visible in the model you uploaded. You can't tune anything that affects the predicted probabilities once stage2 begins. \n\nAs for the release of stage1 labels, I don't really see how they are relevant here. In any case, the fact that they will be released was announced before stage2 began.", "193581": "I respectfully disagree, with the release of stage1 labels we are allowed to re-train our models with the additional stage 1 test data, there is no debating that. Now, in stage 1 lets say you created a 80/20 train validation split and trained 2 models on the 80/20 split. you then created a weighted ensemble using those 2 models and used the 20 % holdout to find optimal ensemble weights. You upload your model saying 0.6 model1 + 0.4 model2. Now stage2 begins they release stage1 labels and say you can retrain your models with the additional stage 1 test data. You now create a new 87/13 split where the 13 % is the stage 1 test data or however you want to shuffle and split, you are now effectively training on 100% of stage1 train data and using the stage 1 test data as your new validation data. Are you telling me that the ensemble weights you used on your 80/20 split are optimal for this new 87/13 split? Allowing retraining implies that ensemble weights also have to be readjusted.", "193593": "I do agree that when training with more data different parameter values become better.\n\nHowever, any parameter tuning is prohibited during stage 2 as you already see the new testing data and can tune the parameters to fit it, which removes the whole point of having the second stage. \n\nAllowing retraining doesn't imply anything, it only means the instructions included in your uploaded model could state that you are going to retrain with the test data appended. All the details (number of iterations, learning rates..) had to be chosen before stage 2.", "193596": "hyper-parameter tuning (model architecture e.t.c) is prohibited not parameter tuning, I will simply refer you to the 2 stage competition [FAQ][1]. The question here for me to the admins given the scenario I have described is plain and simple. Are we allowed to adjust the ensemble weights when retraining is allowed. Its a simple yes or no! it not a big deal!\n\n\n [1]: https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580", "193601": "I don't think you can adjust the weights according to stage 1 test data. All your training/validation data should be fixed before the end of stage 1. Otherwise the competition will be meaningless.", "193614": "I guess you should have uploaded code to find optimal ensemble weights in automated way. I think you can not do this manually to claim a prize.", "193629": "It is **not** allowed. Just “non scientific” alterations are allowed. Just read the rules in the [FAQ][1]\n\n\n [1]: https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580\n\n**What happens if I want to change something in my code in the second stage?**\nWe expect you may need to make some “non scientific” alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.", "193635": "It's plainly stated in the FAQ: no manual parameter tuning is allowed, only (fully) automatic. I fail to see how that doesn't answer your question. If you have an automatic way to adjust those weights w.r.t. the amount of training data (and it's explained in the instructions you've uploaded with your code) then go ahead.", "193641": "\"You are allowed to re-train your model (including the stage one data), but your code should not change.\". You can't change hard-coded parameter values without changing the code.\n\nI do like the idea of changing \"parameters\" when only \"meta-parameter\" tuning is prohibited. You could also have parameters that control how the learning rate changes, for example, they would be \"meta-meta-parameters\" so you could change them too.", "193659": "ZadrraS it does fail to answer my question. When you retrain your models are you automatically adjusting initial learning rate like @bobutis said? are you automatically adjusting initial data augmentation parameters e.t.c? Are your feet going to be held to the fire because you didn't? There are grey areas, all I am looking for is clarification on this specific case. Its not a big deal for us, the gains we see are small, we will keep them as our uploaded code if we have to, but these are practical things to think about for future 2 stage competitions, that's all.", "193666": "I don't see this as a gray area at all. I don't adjust those things automatically, because I don't know a good way of doing it in the context of this competition. I don't adjust them by hand in stage2 either, because that would be against the rules. What changes have to be made to the whole pipeline, when retraining, were described in the instructions our team uploaded at the end of stage1. Some of the steps required some guesswork on how our models would react to more data, but all of the steps are laid out concretely and can be reproduced by anybody who picks up our code.", "193667": "adjusting your learning rate in stage 2 would be against the rules?", "193691": "If you didn't describe in the instructions you've uploaded how exactly that was going to be done, then yes.", "194203": "I think the whole point is that anything you do should be automated. If it isn't automated, it can't be changed.\n\nI have a question about the two submissions feature. We are allowed to mark two submissions as our submissions to be included. Does this mean that if we uploaded two different scripts in our model submission we can use each script to produce a different prediction file, submit both of them, and our final score will be the min score between the two files?" }, "source": "meta" }