December 10, 2017 - Model upload & first stage deadline. This is the last day you may upload your model to be eligible for a prize.
\n\nBut I can't find anything in the Timeline about stage 2 model upload. I expect official answer here.
\n\nPersonally, I'm 100% sure that nobody must have a right to update weights for the stage 2. There is a little sense in the competition if they can.
", "messages": [ { "id": 255692, "postDate": "2017-12-09T20:34:42.613Z", "content": "December 10, 2017 - Model upload & first stage deadline. This is the last day you may upload your model to be eligible for a prize.
\n\nBut I can't find anything in the Timeline about stage 2 model upload. I expect official answer here.
\n\nPersonally, I'm 100% sure that nobody must have a right to update weights for the stage 2. There is a little sense in the competition if they can.
", "rawMarkdown": "*December 10, 2017 - Model upload & first stage deadline. This is the last day you may upload your model to be eligible for a prize.*\n\nBut I can't find anything in the Timeline about stage 2 model upload. I expect official answer here.\n\n1. Should we upload a new model (the same code, but updated weights) for the stage 2?\n2. Can we upload a new model (the same code, but updated weights) for the stage 2?\n\nPersonally, I'm 100% sure that nobody must have a right to update weights for the stage 2. There is a little sense in the competition if they can.", "votes": -1 }, { "id": 258379, "postDate": "2017-12-16T02:28:59.843Z", "content": "It would be interesting to see the results for the stage 2 based on the uploaded models and weighs after the stage 1. I'm 100% sure that those results would be more fair and completely different!
", "rawMarkdown": "It would be interesting to see the results for the stage 2 based on the uploaded models and weighs after the stage 1. I'm 100% sure that those results would be more fair and completely different!", "replies": [ { "id": 258433, "postDate": "2017-12-16T04:32:24.063Z", "content": "My understanding of the rules was that training + prediction should be automatic: The organizers are supposed to be able to run a script that makes the final predictions (and does about as well as you did).
\n\nWhether there are weights that are produced as an intermediate step is unimportant (Especially since not all ML algorithms separate training and prediction, like NNs do).
\n\nIt's possible that my understanding of the rules is incorrect though: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/discussion/45361 seems to imply that manual fiddling is acceptable. I don't know how that would work with retraining for stage2 or semi-supervised training.
", "rawMarkdown": "My understanding of the rules was that training + prediction should be automatic: The organizers are supposed to be able to run a script that makes the final predictions (and does about as well as you did). \n\nWhether there are weights that are produced as an intermediate step is unimportant (Especially since not all ML algorithms separate training and prediction, like NNs do).\n\nIt's possible that my understanding of the rules is incorrect though: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/discussion/45361 seems to imply that manual fiddling is acceptable. I don't know how that would work with retraining for stage2 or semi-supervised training." }, { "id": 258444, "postDate": "2017-12-16T05:22:22.060Z", "content": "Oleg, will your solution work for unseen people? In other words, will it be useful in practice at airport?
", "rawMarkdown": "Oleg, will your solution work for unseen people? In other words, will it be useful in practice at airport?" }, { "id": 258464, "postDate": "2017-12-16T06:08:28.257Z", "content": "\n\n\nOleg, will your solution work for unseen people?
\n
My model didn't change at all after I saw Stage2 data (and they are all new subjects), if this is what you are asking.
\n\n\n\n\nIn other words, will it be useful in practice at airport?
\n
It's Homeland's decision what to do with any of the models, as far as airports go. One of the competition's announcements said they just wanted exposure to new algorithms/talent at this point, as I recall. It's not like your code goes straight from the leaderboard to deciding who flies to Hawaii or Guantanamo.
", "rawMarkdown": "> Oleg, will your solution work for unseen people?\n\nMy model didn't change at all after I saw Stage2 data (and they are all new subjects), if this is what you are asking.\n\n> In other words, will it be useful in practice at airport?\n\nIt's Homeland's decision what to do with any of the models, as far as airports go. One of the competition's announcements said they just wanted exposure to new algorithms/talent at this point, as I recall. It's not like your code goes straight from the leaderboard to deciding who flies to Hawaii or Guantanamo." }, { "id": 258465, "postDate": "2017-12-16T06:11:05.190Z", "content": "My question was about predictions of unique people (like in real life). So you can't analyze multiple scans of the same person before you make your predictions. Will your solution workr with the same accuracy for unseen unique people?
", "rawMarkdown": "My question was about predictions of unique people (like in real life). So you can't analyze multiple scans of the same person before you make your predictions. Will your solution workr with the same accuracy for unseen unique people? " }, { "id": 258473, "postDate": "2017-12-16T07:11:14.397Z", "content": "I see. My best model was fully i.i.d. with respect to the test set (i.e. if you remove some of the test set, it will predict the same values for the rest of it).
\n\nI suspect that the top 3 are doing semi-supervised stuff more (as the rules allow).
\n\nOne easy heuristic that would have worked was: \"If the same subject has a bomb in the same zone in all of the dataset, it's probably a false positive\". I didn't rely on anything like that though. (Regrets will kill you, so I try not to dwell on it)
", "rawMarkdown": "I see. My best model was fully i.i.d. with respect to the test set (*i.e.* if you remove some of the test set, it will predict the same values for the rest of it). \n\nI suspect that the top 3 are doing semi-supervised stuff more (as the rules allow).\n\nOne easy heuristic that would have worked was: \"If the same subject has a bomb in the same zone in all of the dataset, it's probably a false positive\". I didn't rely on anything like that though. (Regrets will kill you, so I try not to dwell on it)\n\n" } ] }, { "id": 255724, "postDate": "2017-12-09T22:32:38.843Z", "content": "Dmitry,
\n\nAs noted in the posted FAQ:
\n\nWhat should I upload?\nWhen you upload a model, you pack all the code that you are eventually going to use to generate your submission csv file. If your models generate some output files containing the weights, for example, ‘.caffemodel’ or ‘.tfmodel’ files, you are NOT required to submit those. However, you should submit the code used to generate those files. You can typically select two submissions for final scoring, so don't forget to include the code/instructions for reproducing both! It can be totally different code, or it can be the same code with instructions about the modifications you would make to generate each.
\n\nWhat happens to my pre-trained model?\nIf you are using a pre-trained for which you don't have the source code, you must include the model as part of your upload.
\n\nWhat happens if I want to change something in my code in the second stage?\nWe expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.
", "rawMarkdown": "Dmitry,\n\nAs noted in the posted FAQ:\n\n**What should I upload?**\nWhen you upload a model, you pack all the code that you are eventually going to use to generate your submission csv file. If your models generate some output files containing the weights, for example, ‘.caffemodel’ or ‘.tfmodel’ files, you are NOT required to submit those. However, you should submit the code used to generate those files. You can typically select two submissions for final scoring, so don't forget to include the code/instructions for reproducing both! It can be totally different code, or it can be the same code with instructions about the modifications you would make to generate each.\n\n**What happens to my pre-trained model?**\nIf you are using a pre-trained for which you don't have the source code, you must include the model as part of your upload.\n\n**What happens if I want to change something in my code in the second stage?**\nWe expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.", "replies": [ { "id": 255726, "postDate": "2017-12-09T22:50:36.340Z", "content": "It's just FAQ - it's not the answer to the questions. I still don't understand should we upload updated weights or we just can do that but better to avoid that. It's also unclear how can we upload retrained model for the stage 2. Will it replace already uploaded model for the stage 1? In this case you won't be able to check that the stage 1 model was uploaded on time and you won't be able to check differences between stage 1 and stage 2 models.
\n\nAnd why do you even allow to retrain models for the stage 2? Anybody can retrain model for the stage 2, so it will be overfitted on the stage 2 data. As result you can get the worst overfitted models on the winner places and actually the best models won't win at all. Personally, I'm greatly disappointed in that fact and I'm not going to participate in any competitions on Kaggle anymore until you fix that problem.
", "rawMarkdown": "It's just FAQ - it's not the answer to the questions. I still don't understand should we upload updated weights or we just can do that but better to avoid that. It's also unclear how can we upload retrained model for the stage 2. Will it replace already uploaded model for the stage 1? In this case you won't be able to check that the stage 1 model was uploaded on time and you won't be able to check differences between stage 1 and stage 2 models.\n\nAnd why do you even allow to retrain models for the stage 2? Anybody can retrain model for the stage 2, so it will be overfitted on the stage 2 data. As result you can get the worst overfitted models on the winner places and actually the best models won't win at all. Personally, I'm greatly disappointed in that fact and I'm not going to participate in any competitions on Kaggle anymore until you fix that problem.", "votes": -2 }, { "id": 255727, "postDate": "2017-12-09T22:55:27.137Z", "content": "Dmitry,
\n\nLike Stage 1, you will submit a solution.csv file with the Stage 2 data - not your model.
\n\nWe will only request the stage 2 models from those who have won the competition, which will be after the conclusion of the competition.
\n\nWe have measurements in place to protect against additional overfitting for the final week of competition.
", "rawMarkdown": "Dmitry, \n\nLike Stage 1, you will submit a solution.csv file with the Stage 2 data - not your model. \n\nWe will only request the stage 2 models from those who have won the competition, which will be after the conclusion of the competition.\n\nWe have measurements in place to protect against additional overfitting for the final week of competition." }, { "id": 255729, "postDate": "2017-12-09T23:02:06.770Z", "content": "I'm glad to hear that you have some \"measurements\" to protect against additional overfitting. But nobody here can be sure that these measurements are sufficient and that fair models won't violate them as well. The only way that can allow everybody to be sure that the stage 2 if fair is to forbid model retraining on the stage 2. It's so obvious and simple that I don't understand at all why you allow to retrain models for the stage 2.
", "rawMarkdown": "I'm glad to hear that you have some \"measurements\" to protect against additional overfitting. But nobody here can be sure that these measurements are sufficient and that fair models won't violate them as well. The only way that can allow everybody to be sure that the stage 2 if fair is to forbid model retraining on the stage 2. It's so obvious and simple that I don't understand at all why you allow to retrain models for the stage 2.", "votes": -3 }, { "id": 255755, "postDate": "2017-12-10T01:21:06.553Z", "content": "Assuming we don't get any additional images with ground-truth Y/N labels for stage-2, I'm not sure how one would retrain their model on additional data given that hand-labeling is strictly forbidden and there's no possibility to \"probe\" the score using submissions to Kaggle as was possible during stage-1?
\n\nBased on this, I've focused my efforts on creating a model that (hopefully :-)) generalizes well to never seen data, rather than a model that can quickly and easily be retrained on (and hence optimized for) a new/additional dataset.
", "rawMarkdown": "Assuming we don't get any additional images with ground-truth Y/N labels for stage-2, I'm not sure how one would retrain their model on additional data given that hand-labeling is strictly forbidden and there's no possibility to \"probe\" the score using submissions to Kaggle as was possible during stage-1?\n\nBased on this, I've focused my efforts on creating a model that (hopefully :-)) generalizes well to never seen data, rather than a model that can quickly and easily be retrained on (and hence optimized for) a new/additional dataset." }, { "id": 255758, "postDate": "2017-12-10T01:24:37.073Z", "content": "Me too. But we can't be sure that other competitors will do the same while retraining is permitted. Do you agree?
", "rawMarkdown": "Me too. But we can't be sure that other competitors will do the same while retraining is permitted. Do you agree?", "votes": -2 }, { "id": 255761, "postDate": "2017-12-10T01:34:38.147Z", "content": "It was my understanding that \"retraining\" models for stage 2 was basically semi-supervised methods (pseudo-labeling, etc.), since we aren't going to have the stage 2 labels for stage 2, and the retraining process has to be automated (so you can't \"overfit\" to the public LB, if the stage 2 public LB uses the actual stage 2 data, which I'm not sure it does). What sort of overfitting are you concerned about?
", "rawMarkdown": "It was my understanding that \"retraining\" models for stage 2 was basically semi-supervised methods (pseudo-labeling, etc.), since we aren't going to have the stage 2 labels for stage 2, and the retraining process has to be automated (so you can't \"overfit\" to the public LB, if the stage 2 public LB uses the actual stage 2 data, which I'm not sure it does). What sort of overfitting are you concerned about?" }, { "id": 255762, "postDate": "2017-12-10T01:39:19.187Z", "content": "If you follow the rules, then don't need to retrain your model. So, anybody who's going to retrain his model concerns me.
", "rawMarkdown": "If you follow the rules, then don't need to retrain your model. So, anybody who's going to retrain his model concerns me.", "votes": -2 }, { "id": 255763, "postDate": "2017-12-10T01:40:47.493Z", "content": "I agree. I trust the organizers @ Kaggle will guard against this and will very carefully check that that the predictions submitted in the final stage2 .csv-file are indeed the result of a 100% automated process. I.e. no code changes whatsoever (other than \"non-scientific\" changes like e.g. adjusting paths).
", "rawMarkdown": "I agree. I trust the organizers @ Kaggle will guard against this and will very carefully check that that the predictions submitted in the final stage2 .csv-file are indeed the result of a 100% automated process. I.e. no code changes whatsoever (other than \"non-scientific\" changes like e.g. adjusting paths)." }, { "id": 255764, "postDate": "2017-12-10T01:42:36.387Z", "content": "Not only the code determines the final scores. But also the weights. And you can't be sure that they are obtained correctly when retraining is permitted. That is the problem.
", "rawMarkdown": "Not only the code determines the final scores. But also the weights. And you can't be sure that they are obtained correctly when retraining is permitted. That is the problem.", "votes": -2 }, { "id": 255766, "postDate": "2017-12-10T01:51:41.257Z", "content": "As far as reproducibility is concerned, I don't think Kaggle actually cares about your weights, which is why you aren't required to upload them. I think they will just use your uploaded code to train their own model, which is expected to produce predictions that are the same (or extremely similar) to the ones that you submitted yourself.
", "rawMarkdown": "As far as reproducibility is concerned, I don't think Kaggle actually cares about your weights, which is why you aren't required to upload them. I think they will just use your uploaded code to train their own model, which is expected to produce predictions that are the same (or extremely similar) to the ones that you submitted yourself.", "votes": -1 }, { "id": 255767, "postDate": "2017-12-10T02:03:15.197Z", "content": "Model training can't be reproduced with the same precision as predictions.
", "rawMarkdown": "Model training can't be reproduced with the same precision as predictions.", "votes": -2 }, { "id": 255769, "postDate": "2017-12-10T02:15:02.550Z", "rawMarkdown": "", "isDeleted": true }, { "id": 255800, "postDate": "2017-12-10T06:08:04.043Z", "content": "Having only have those who win a prize upload their models for the hosts to download from our platform has been a standard procedure and established precedent across all Kaggle competitions.
\n\nWe do not disclose any information on the methods employed for our anti-cheating process (which I hope you can understand).
", "rawMarkdown": "Having only have those who win a prize upload their models for the hosts to download from our platform has been a standard procedure and established precedent across all Kaggle competitions. \n\nWe do not disclose any information on the methods employed for our anti-cheating process (which I hope you can understand)." }, { "id": 255873, "postDate": "2017-12-10T12:35:22.683Z", "rawMarkdown": "", "isDeleted": true }, { "id": 255912, "postDate": "2017-12-10T15:08:42.997Z", "content": "Hi Bo:
\n\nPer the FAQ:
\n\nWhat happens if I want to change something in my code in the second stage? We expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.
", "rawMarkdown": "Hi Bo:\n\nPer the FAQ:\n\n**What happens if I want to change something in my code in the second stage?** We expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated." } ] } ], "comments": [ { "id": 258379, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-16T02:28:59.843000", "content": "It would be interesting to see the results for the stage 2 based on the uploaded models and weighs after the stage 1. I'm 100% sure that those results would be more fair and completely different!
", "votes": 0, "replies": [ { "id": 258433, "author_name": "Oleg Trott", "author_url": "", "post_date": "2017-12-16T04:32:24.063000", "content": "My understanding of the rules was that training + prediction should be automatic: The organizers are supposed to be able to run a script that makes the final predictions (and does about as well as you did).
\n\nWhether there are weights that are produced as an intermediate step is unimportant (Especially since not all ML algorithms separate training and prediction, like NNs do).
\n\nIt's possible that my understanding of the rules is incorrect though: https://www.kaggle.com/c/passenger-screening-algorithm-challenge/discussion/45361 seems to imply that manual fiddling is acceptable. I don't know how that would work with retraining for stage2 or semi-supervised training.
", "votes": 0, "replies": [] }, { "id": 258444, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-16T05:22:22.060000", "content": "Oleg, will your solution work for unseen people? In other words, will it be useful in practice at airport?
", "votes": 0, "replies": [] }, { "id": 258464, "author_name": "Oleg Trott", "author_url": "", "post_date": "2017-12-16T06:08:28.257000", "content": "\n\n\nOleg, will your solution work for unseen people?
\n
My model didn't change at all after I saw Stage2 data (and they are all new subjects), if this is what you are asking.
\n\n\n\n\nIn other words, will it be useful in practice at airport?
\n
It's Homeland's decision what to do with any of the models, as far as airports go. One of the competition's announcements said they just wanted exposure to new algorithms/talent at this point, as I recall. It's not like your code goes straight from the leaderboard to deciding who flies to Hawaii or Guantanamo.
", "votes": 0, "replies": [] }, { "id": 258465, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-16T06:11:05.190000", "content": "My question was about predictions of unique people (like in real life). So you can't analyze multiple scans of the same person before you make your predictions. Will your solution workr with the same accuracy for unseen unique people?
", "votes": 0, "replies": [] }, { "id": 258473, "author_name": "Oleg Trott", "author_url": "", "post_date": "2017-12-16T07:11:14.397000", "content": "I see. My best model was fully i.i.d. with respect to the test set (i.e. if you remove some of the test set, it will predict the same values for the rest of it).
\n\nI suspect that the top 3 are doing semi-supervised stuff more (as the rules allow).
\n\nOne easy heuristic that would have worked was: \"If the same subject has a bomb in the same zone in all of the dataset, it's probably a false positive\". I didn't rely on anything like that though. (Regrets will kill you, so I try not to dwell on it)
", "votes": 0, "replies": [] } ] }, { "id": 255724, "author_name": "Addison Howard", "author_url": "", "post_date": "2017-12-09T22:32:38.843000", "content": "Dmitry,
\n\nAs noted in the posted FAQ:
\n\nWhat should I upload?\nWhen you upload a model, you pack all the code that you are eventually going to use to generate your submission csv file. If your models generate some output files containing the weights, for example, ‘.caffemodel’ or ‘.tfmodel’ files, you are NOT required to submit those. However, you should submit the code used to generate those files. You can typically select two submissions for final scoring, so don't forget to include the code/instructions for reproducing both! It can be totally different code, or it can be the same code with instructions about the modifications you would make to generate each.
\n\nWhat happens to my pre-trained model?\nIf you are using a pre-trained for which you don't have the source code, you must include the model as part of your upload.
\n\nWhat happens if I want to change something in my code in the second stage?\nWe expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.
", "votes": 0, "replies": [ { "id": 255726, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-09T22:50:36.340000", "content": "It's just FAQ - it's not the answer to the questions. I still don't understand should we upload updated weights or we just can do that but better to avoid that. It's also unclear how can we upload retrained model for the stage 2. Will it replace already uploaded model for the stage 1? In this case you won't be able to check that the stage 1 model was uploaded on time and you won't be able to check differences between stage 1 and stage 2 models.
\n\nAnd why do you even allow to retrain models for the stage 2? Anybody can retrain model for the stage 2, so it will be overfitted on the stage 2 data. As result you can get the worst overfitted models on the winner places and actually the best models won't win at all. Personally, I'm greatly disappointed in that fact and I'm not going to participate in any competitions on Kaggle anymore until you fix that problem.
", "votes": -2, "replies": [] }, { "id": 255727, "author_name": "Addison Howard", "author_url": "", "post_date": "2017-12-09T22:55:27.137000", "content": "Dmitry,
\n\nLike Stage 1, you will submit a solution.csv file with the Stage 2 data - not your model.
\n\nWe will only request the stage 2 models from those who have won the competition, which will be after the conclusion of the competition.
\n\nWe have measurements in place to protect against additional overfitting for the final week of competition.
", "votes": 0, "replies": [] }, { "id": 255729, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-09T23:02:06.770000", "content": "I'm glad to hear that you have some \"measurements\" to protect against additional overfitting. But nobody here can be sure that these measurements are sufficient and that fair models won't violate them as well. The only way that can allow everybody to be sure that the stage 2 if fair is to forbid model retraining on the stage 2. It's so obvious and simple that I don't understand at all why you allow to retrain models for the stage 2.
", "votes": -3, "replies": [] }, { "id": 255755, "author_name": "Hans Bouwmeester", "author_url": "", "post_date": "2017-12-10T01:21:06.553000", "content": "Assuming we don't get any additional images with ground-truth Y/N labels for stage-2, I'm not sure how one would retrain their model on additional data given that hand-labeling is strictly forbidden and there's no possibility to \"probe\" the score using submissions to Kaggle as was possible during stage-1?
\n\nBased on this, I've focused my efforts on creating a model that (hopefully :-)) generalizes well to never seen data, rather than a model that can quickly and easily be retrained on (and hence optimized for) a new/additional dataset.
", "votes": 0, "replies": [] }, { "id": 255758, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-10T01:24:37.073000", "content": "Me too. But we can't be sure that other competitors will do the same while retraining is permitted. Do you agree?
", "votes": -2, "replies": [] }, { "id": 255761, "author_name": "Suchir Balaji", "author_url": "", "post_date": "2017-12-10T01:34:38.147000", "content": "It was my understanding that \"retraining\" models for stage 2 was basically semi-supervised methods (pseudo-labeling, etc.), since we aren't going to have the stage 2 labels for stage 2, and the retraining process has to be automated (so you can't \"overfit\" to the public LB, if the stage 2 public LB uses the actual stage 2 data, which I'm not sure it does). What sort of overfitting are you concerned about?
", "votes": 0, "replies": [] }, { "id": 255762, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-10T01:39:19.187000", "content": "If you follow the rules, then don't need to retrain your model. So, anybody who's going to retrain his model concerns me.
", "votes": -2, "replies": [] }, { "id": 255763, "author_name": "Hans Bouwmeester", "author_url": "", "post_date": "2017-12-10T01:40:47.493000", "content": "I agree. I trust the organizers @ Kaggle will guard against this and will very carefully check that that the predictions submitted in the final stage2 .csv-file are indeed the result of a 100% automated process. I.e. no code changes whatsoever (other than \"non-scientific\" changes like e.g. adjusting paths).
", "votes": 0, "replies": [] }, { "id": 255764, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-10T01:42:36.387000", "content": "Not only the code determines the final scores. But also the weights. And you can't be sure that they are obtained correctly when retraining is permitted. That is the problem.
", "votes": -2, "replies": [] }, { "id": 255766, "author_name": "Suchir Balaji", "author_url": "", "post_date": "2017-12-10T01:51:41.257000", "content": "As far as reproducibility is concerned, I don't think Kaggle actually cares about your weights, which is why you aren't required to upload them. I think they will just use your uploaded code to train their own model, which is expected to produce predictions that are the same (or extremely similar) to the ones that you submitted yourself.
", "votes": -1, "replies": [] }, { "id": 255767, "author_name": "Dmitry Kovba", "author_url": "", "post_date": "2017-12-10T02:03:15.197000", "content": "Model training can't be reproduced with the same precision as predictions.
", "votes": -2, "replies": [] }, { "id": 255769, "author_name": "", "author_url": "", "post_date": "2017-12-10T02:15:02.550000", "content": "", "votes": 0, "replies": [] }, { "id": 255800, "author_name": "Addison Howard", "author_url": "", "post_date": "2017-12-10T06:08:04.043000", "content": "Having only have those who win a prize upload their models for the hosts to download from our platform has been a standard procedure and established precedent across all Kaggle competitions.
\n\nWe do not disclose any information on the methods employed for our anti-cheating process (which I hope you can understand).
", "votes": 0, "replies": [] }, { "id": 255873, "author_name": "", "author_url": "", "post_date": "2017-12-10T12:35:22.683000", "content": "", "votes": 0, "replies": [] }, { "id": 255912, "author_name": "Addison Howard", "author_url": "", "post_date": "2017-12-10T15:08:42.997000", "content": "Hi Bo:
\n\nPer the FAQ:
\n\nWhat happens if I want to change something in my code in the second stage? We expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "255692": "*December 10, 2017 - Model upload & first stage deadline. This is the last day you may upload your model to be eligible for a prize.*\n\nBut I can't find anything in the Timeline about stage 2 model upload. I expect official answer here.\n\n1. Should we upload a new model (the same code, but updated weights) for the stage 2?\n2. Can we upload a new model (the same code, but updated weights) for the stage 2?\n\nPersonally, I'm 100% sure that nobody must have a right to update weights for the stage 2. There is a little sense in the competition if they can.", "258379": "It would be interesting to see the results for the stage 2 based on the uploaded models and weighs after the stage 1. I'm 100% sure that those results would be more fair and completely different!", "255724": "Dmitry,\n\nAs noted in the posted FAQ:\n\n**What should I upload?**\nWhen you upload a model, you pack all the code that you are eventually going to use to generate your submission csv file. If your models generate some output files containing the weights, for example, ‘.caffemodel’ or ‘.tfmodel’ files, you are NOT required to submit those. However, you should submit the code used to generate those files. You can typically select two submissions for final scoring, so don't forget to include the code/instructions for reproducing both! It can be totally different code, or it can be the same code with instructions about the modifications you would make to generate each.\n\n**What happens to my pre-trained model?**\nIf you are using a pre-trained for which you don't have the source code, you must include the model as part of your upload.\n\n**What happens if I want to change something in my code in the second stage?**\nWe expect you may need to make some \"non scientific\" alterations, such as changes to path names, in order to create your submissions for the second stage. You are allowed to re-train your model (including the stage one data), but your code should not change. You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated." } }