{ "id": 69745, "title": "Questions about stage 2", "url": "/competitions/rsna-pneumonia-detection-challenge/discussion/69745", "author_name": "Andy Harless", "post_date": "2018-10-26T17:24:18.330000", "votes": 14, "comment_count": 13, "views": 0, "content": "

The FAQ is very helpful (and would have been even more helpful if we'd seen it earlier), but some questions remain:

\n\n
    \n
  1. I gather it is not considered a rule violation if we experiment with new models in stage 2 submissions, as long as we choose the submission that uses the stage 1 model?
    2. \n
  2. I gather also that someone could, in violation of the rules, choose such an experiment as a final submission, and they could expect not to get caught unless they ended up in a winning position. So, for an unscrupulous Kaggler that doesn't feel the need to play by the rules and also doesn't see much chance of ending up in a winning position, the optimal strategy would be to choose the submission most likely to win a medal.
    3. \n
  3. In consequence of #1, stage 2 public LB scores (which aren't very useful anyhow, since they're based on a very small sample) may reflect teams' best-scoring experiments rather than their actual final models that they will choose (assuming they are following the rules)?
    4. \n
  4. In some cases there are multiple possibilities for what you could consider to be a run of your uploaded model. For example, if you uploaded the weights and also the training code, you could submit results from both a retrained and an unretrained model and make a final decision after submitting both. Is making such a choice a violation of the rules? Is it one that would (or could) be enforced on a prize-winning team?
    5. \n
  5. When the FAQ says, \"You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.\" I take it the second sentence supersedes the first? IOW the first sentence only referred to tuning that was not automated in the uploaded model?
    6. \n
  6. Do you have to select your stage 2 submission explicitly? (I'm assuming the answer is yes, since the now invalid stage 1 submission seems to remain checked after submitting for stage 2, but this ought to be spelled out somewhere. Or will an invalid selection trigger an automatic re-selection of the best-scoring model?)
    7. \n
  7. Medal positions on current LB appear to reflect only stage 2 submissions, but the FAQ says medal positions are determined by the number of stage 1 submissions. I take it this is just a glitch in the presentation, and we should ignore the medal ranges as presented?
    8. \n
", "messages": [ { "id": 410798, "postDate": "2018-10-26T17:24:18.330Z", "content": "

The FAQ is very helpful (and would have been even more helpful if we'd seen it earlier), but some questions remain:

\n\n
    \n
  1. I gather it is not considered a rule violation if we experiment with new models in stage 2 submissions, as long as we choose the submission that uses the stage 1 model?
    2. \n
  2. I gather also that someone could, in violation of the rules, choose such an experiment as a final submission, and they could expect not to get caught unless they ended up in a winning position. So, for an unscrupulous Kaggler that doesn't feel the need to play by the rules and also doesn't see much chance of ending up in a winning position, the optimal strategy would be to choose the submission most likely to win a medal.
    3. \n
  3. In consequence of #1, stage 2 public LB scores (which aren't very useful anyhow, since they're based on a very small sample) may reflect teams' best-scoring experiments rather than their actual final models that they will choose (assuming they are following the rules)?
    4. \n
  4. In some cases there are multiple possibilities for what you could consider to be a run of your uploaded model. For example, if you uploaded the weights and also the training code, you could submit results from both a retrained and an unretrained model and make a final decision after submitting both. Is making such a choice a violation of the rules? Is it one that would (or could) be enforced on a prize-winning team?
    5. \n
  5. When the FAQ says, \"You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.\" I take it the second sentence supersedes the first? IOW the first sentence only referred to tuning that was not automated in the uploaded model?
    6. \n
  6. Do you have to select your stage 2 submission explicitly? (I'm assuming the answer is yes, since the now invalid stage 1 submission seems to remain checked after submitting for stage 2, but this ought to be spelled out somewhere. Or will an invalid selection trigger an automatic re-selection of the best-scoring model?)
    7. \n
  7. Medal positions on current LB appear to reflect only stage 2 submissions, but the FAQ says medal positions are determined by the number of stage 1 submissions. I take it this is just a glitch in the presentation, and we should ignore the medal ranges as presented?
    8. \n
", "rawMarkdown": "The [FAQ][1] is very helpful (and would have been even more helpful if we'd seen it earlier), but some questions remain:\n\n 1. I gather it is not considered a rule violation if we experiment with new models in stage 2 submissions, as long as we choose the submission that uses the stage 1 model?\n 2. I gather also that someone could, in violation of the rules, choose such an experiment as a final submission, and they could expect not to get caught unless they ended up in a winning position. So, for an unscrupulous Kaggler that doesn't feel the need to play by the rules and also doesn't see much chance of ending up in a winning position, the optimal strategy would be to choose the submission most likely to win a medal.\n 3. In consequence of #1, stage 2 public LB scores (which aren't very useful anyhow, since they're based on a very small sample) may reflect teams' best-scoring experiments rather than their actual final models that they will choose (assuming they are following the rules)?\n 4. In some cases there are multiple possibilities for what you could consider to be a run of your uploaded model. For example, if you uploaded the weights and also the training code, you could submit results from both a retrained and an unretrained model and make a final decision after submitting both. Is making such a choice a violation of the rules? Is it one that would (or could) be enforced on a prize-winning team?\n 5. When the FAQ says, \"You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.\" I take it the second sentence supersedes the first? IOW the first sentence only referred to tuning that was not automated in the uploaded model?\n 6. Do you have to select your stage 2 submission explicitly? (I'm assuming the answer is yes, since the now invalid stage 1 submission seems to remain checked after submitting for stage 2, but this ought to be spelled out somewhere. Or will an invalid selection trigger an automatic re-selection of the best-scoring model?)\n 7. Medal positions on current LB appear to reflect only stage 2 submissions, but the FAQ says medal positions are determined by the number of stage 1 submissions. I take it this is just a glitch in the presentation, and we should ignore the medal ranges as presented?\n\n [1]: https://www.kaggle.com/two-stage-frequently-asked-questions", "votes": 14 }, { "id": 410882, "postDate": "2018-10-26T20:28:15.490Z", "content": "

Thank you for your detailed questions. Some responses:

\n\n", "rawMarkdown": "Thank you for your detailed questions. Some responses:\n\n - The public leaderboard for Stage 2 is intentionally based on a very small number of images. Its purpose is to give people a basis for testing that their submissions don't error out and some small signal as to how your submission is performing on the Stage 2 data. How participants decide to use their daily submissions is up to them.\n - Ultimately, you are permitted 1 final submission selection. That submission is required to match the model that was uploaded in Stage 1, in adherence to the rules. You can expect the host to validate this, most certainly for those in prize standing.\n - Yes, participants should explicitly select their Stage 2 final submission.\n - Regarding the FAQ statement: \"You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.\" >> Yes, only fully automated parameter tuning is permitted.\n - The medal positions and rankings are awarded based on the stage 2 standing. **However**, the total number of participants in the competition is based on the Stage 1 volume of submitters.\n - In any contest, there are an unbound number of ways participants might try to game an advantage. We've seen many, and as such do our very best to construct an environment that (a) fosters learning and sharing among people who share a passion for data science & (b) gives our hosts a basis for advancing solutions to their very real-life problems. In particular, this competition's topic has the potential to make a tremendous impact on radiology and healthcare. We implore competitors to direct their energy in the spirit of that mission.", "votes": 5, "replies": [ { "id": 410904, "postDate": "2018-10-26T21:22:33.013Z", "content": "

@Julia since there may possibly be significantly lower number of submissions in stage-2. Suppose we end up with only 100 total number of teams in stage-2, and based on stage-1 volume, all 100 teams will get a medal? Is that correct?

\n\n
\n

The medal positions and rankings are awarded based on the stage 2 standing. However, the total number of participants in the competition is based on the Stage 1 volume of submitters.

\n
", "rawMarkdown": "\n@Julia since there may possibly be significantly lower number of submissions in stage-2. Suppose we end up with only 100 total number of teams in stage-2, and based on stage-1 volume, all 100 teams will get a medal? Is that correct?\n\n>The medal positions and rankings are awarded based on the stage 2 standing. However, the total number of participants in the competition is based on the Stage 1 volume of submitters.\n" }, { "id": 410953, "postDate": "2018-10-27T02:20:22.393Z", "content": "

Hi Julia, Can you please give an example for this statement \"only fully automated parameter tuning\"? Thank you!

", "rawMarkdown": "Hi Julia, Can you please give an example for this statement \"only fully automated parameter tuning\"? Thank you!", "votes": 1 }, { "id": 410962, "postDate": "2018-10-27T03:03:08.927Z", "content": "

Hi,

\n\n

Is it allowed to do fine tuning with stage1 test set (partial stage2 train set)? It does not change any hyper parameters. Changes are only followings.

\n\n\n\n

Thanks,

", "rawMarkdown": "Hi,\n\nIs it allowed to do fine tuning with stage1 test set (partial stage2 train set)? It does not change any hyper parameters. Changes are only followings.\n\n* Change the way to load train set.\n* Change the way to load pre-trained weight (from ImageNet to last model I used.)\n\nThanks," }, { "id": 410969, "postDate": "2018-10-27T03:17:19.780Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 411480, "postDate": "2018-10-28T08:53:53.847Z", "content": "

Hi Julia, Could you please confirm if we can possibly perform the tasks like AkiraSosa stated in his/her previous statement? Thanks!

", "rawMarkdown": "Hi Julia, Could you please confirm if we can possibly perform the tasks like AkiraSosa stated in his/her previous statement? Thanks!", "votes": 1 }, { "id": 411571, "postDate": "2018-10-28T13:39:25.023Z", "content": "
Will the number of participants change in the second stage? Will I get a medal? Will I get points?\n\nYes, the number of participants will be smaller since some people won't submit in the second stage. For the purposes of medals and points calculations, we will use the number of participants in the first stage to calculate points and medals.
\n\n

From the quote above in the two-stage-FAQ, clearly both medals and points are calculated with the number of participants in the first stage. Not that I am saying this is more fair but rules should be consistent, especially when you post it in a FAQ!

", "rawMarkdown": "
Will the number of participants change in the second stage? Will I get a medal? Will I get points?\n\nYes, the number of participants will be smaller since some people won't submit in the second stage. For the purposes of medals and points calculations, we will use the number of participants in the first stage to calculate points and medals.
\n\nFrom the quote above in the two-stage-FAQ, clearly both medals and points are calculated with the number of participants in the first stage. Not that I am saying this is more fair but rules should be consistent, especially when you post it in a FAQ!" }, { "id": 411786, "postDate": "2018-10-29T02:52:52.490Z", "content": "

I guess it comes down to the difference between hyperparameter and parameter. So, automated parameter tuning is the same as training your model. But you can't change the hyperparameters (lr, number of neurons and so on).

", "rawMarkdown": "I guess it comes down to the difference between hyperparameter and parameter. So, automated parameter tuning is the same as training your model. But you can't change the hyperparameters (lr, number of neurons and so on)." }, { "id": 412260, "postDate": "2018-10-29T21:52:51.290Z", "content": "

@YaGana - Yes, the total number of participants in stage 1 was > 1,000 - so the points/medals are awarded based on the \"1000+\" column on the progression page's \"Competition Medals\" section (See: https://www.kaggle.com/progression). But your score in stage 2 is what determines your final ranking/placement.

\n\n

@AkiraSosa - You are permitted to make changes that are necessary to run and load the stage 2 test set. But, as already stated, you should not be doing any non-automated tuning.

", "rawMarkdown": "@YaGana - Yes, the total number of participants in stage 1 was > 1,000 - so the points/medals are awarded based on the \"1000+\" column on the progression page's \"Competition Medals\" section (See: https://www.kaggle.com/progression). But your score in stage 2 is what determines your final ranking/placement.\n\n@AkiraSosa - You are permitted to make changes that are necessary to run and load the stage 2 test set. But, as already stated, you should not be doing any non-automated tuning." }, { "id": 412264, "postDate": "2018-10-29T21:59:59.100Z", "content": "

@Julia

\n\n

I believe AkiraSosa was asking if it is permitted to load the new stage 2 train set in a particular manner. That is, selectively loading a part of the stage 2 train set (stage 1 test) and fine-tuning a pretrained stage 1 network on that data. Is this allowed?

", "rawMarkdown": "@Julia\n\nI believe AkiraSosa was asking if it is permitted to load the new stage 2 train set in a particular manner. That is, selectively loading a part of the stage 2 train set (stage 1 test) and fine-tuning a pretrained stage 1 network on that data. Is this allowed?", "votes": 1 }, { "id": 412267, "postDate": "2018-10-29T22:01:13.617Z", "content": "

@Julia, thanks for the confirmation.

", "rawMarkdown": "@Julia, thanks for the confirmation." }, { "id": 412914, "postDate": "2018-10-31T01:39:31.757Z", "content": "

@Julia Thanks for your answer, but still I have same questions with @Ian Pan.

", "rawMarkdown": "@Julia Thanks for your answer, but still I have same questions with @Ian Pan." } ] }, { "id": 412557, "postDate": "2018-10-30T11:30:41.860Z", "content": "

Could you please give an example of fully automated parameter?

", "rawMarkdown": "Could you please give an example of fully automated parameter?" } ], "comments": [ { "id": 410882, "author_name": "Julia Elliott", "author_url": "", "post_date": "2018-10-26T20:28:15.490000", "content": "

Thank you for your detailed questions. Some responses:

\n\n", "votes": 5, "replies": [ { "id": 410904, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-10-26T21:22:33.013000", "content": "

@Julia since there may possibly be significantly lower number of submissions in stage-2. Suppose we end up with only 100 total number of teams in stage-2, and based on stage-1 volume, all 100 teams will get a medal? Is that correct?

\n\n
\n

The medal positions and rankings are awarded based on the stage 2 standing. However, the total number of participants in the competition is based on the Stage 1 volume of submitters.

\n
", "votes": 0, "replies": [] }, { "id": 410953, "author_name": "", "author_url": "", "post_date": "2018-10-27T02:20:22.393000", "content": "

Hi Julia, Can you please give an example for this statement \"only fully automated parameter tuning\"? Thank you!

", "votes": 1, "replies": [] }, { "id": 410962, "author_name": "AkiraSosa", "author_url": "", "post_date": "2018-10-27T03:03:08.927000", "content": "

Hi,

\n\n

Is it allowed to do fine tuning with stage1 test set (partial stage2 train set)? It does not change any hyper parameters. Changes are only followings.

\n\n\n\n

Thanks,

", "votes": 0, "replies": [] }, { "id": 410969, "author_name": "", "author_url": "", "post_date": "2018-10-27T03:17:19.780000", "content": "", "votes": 2, "replies": [] }, { "id": 411480, "author_name": "", "author_url": "", "post_date": "2018-10-28T08:53:53.847000", "content": "

Hi Julia, Could you please confirm if we can possibly perform the tasks like AkiraSosa stated in his/her previous statement? Thanks!

", "votes": 1, "replies": [] }, { "id": 411571, "author_name": "amigd23", "author_url": "", "post_date": "2018-10-28T13:39:25.023000", "content": "
Will the number of participants change in the second stage? Will I get a medal? Will I get points?\n\nYes, the number of participants will be smaller since some people won't submit in the second stage. For the purposes of medals and points calculations, we will use the number of participants in the first stage to calculate points and medals.
\n\n

From the quote above in the two-stage-FAQ, clearly both medals and points are calculated with the number of participants in the first stage. Not that I am saying this is more fair but rules should be consistent, especially when you post it in a FAQ!

", "votes": 0, "replies": [] }, { "id": 411786, "author_name": "Fernando Camargo", "author_url": "", "post_date": "2018-10-29T02:52:52.490000", "content": "

I guess it comes down to the difference between hyperparameter and parameter. So, automated parameter tuning is the same as training your model. But you can't change the hyperparameters (lr, number of neurons and so on).

", "votes": 0, "replies": [] }, { "id": 412260, "author_name": "Julia Elliott", "author_url": "", "post_date": "2018-10-29T21:52:51.290000", "content": "

@YaGana - Yes, the total number of participants in stage 1 was > 1,000 - so the points/medals are awarded based on the \"1000+\" column on the progression page's \"Competition Medals\" section (See: https://www.kaggle.com/progression). But your score in stage 2 is what determines your final ranking/placement.

\n\n

@AkiraSosa - You are permitted to make changes that are necessary to run and load the stage 2 test set. But, as already stated, you should not be doing any non-automated tuning.

", "votes": 0, "replies": [] }, { "id": 412264, "author_name": "Ian Pan", "author_url": "", "post_date": "2018-10-29T21:59:59.100000", "content": "

@Julia

\n\n

I believe AkiraSosa was asking if it is permitted to load the new stage 2 train set in a particular manner. That is, selectively loading a part of the stage 2 train set (stage 1 test) and fine-tuning a pretrained stage 1 network on that data. Is this allowed?

", "votes": 1, "replies": [] }, { "id": 412267, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2018-10-29T22:01:13.617000", "content": "

@Julia, thanks for the confirmation.

", "votes": 0, "replies": [] }, { "id": 412914, "author_name": "AkiraSosa", "author_url": "", "post_date": "2018-10-31T01:39:31.757000", "content": "

@Julia Thanks for your answer, but still I have same questions with @Ian Pan.

", "votes": 0, "replies": [] } ] }, { "id": 412557, "author_name": "Marcelo Piovan", "author_url": "", "post_date": "2018-10-30T11:30:41.860000", "content": "

Could you please give an example of fully automated parameter?

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "410798": "The [FAQ][1] is very helpful (and would have been even more helpful if we'd seen it earlier), but some questions remain:\n\n 1. I gather it is not considered a rule violation if we experiment with new models in stage 2 submissions, as long as we choose the submission that uses the stage 1 model?\n 2. I gather also that someone could, in violation of the rules, choose such an experiment as a final submission, and they could expect not to get caught unless they ended up in a winning position. So, for an unscrupulous Kaggler that doesn't feel the need to play by the rules and also doesn't see much chance of ending up in a winning position, the optimal strategy would be to choose the submission most likely to win a medal.\n 3. In consequence of #1, stage 2 public LB scores (which aren't very useful anyhow, since they're based on a very small sample) may reflect teams' best-scoring experiments rather than their actual final models that they will choose (assuming they are following the rules)?\n 4. In some cases there are multiple possibilities for what you could consider to be a run of your uploaded model. For example, if you uploaded the weights and also the training code, you could submit results from both a retrained and an unretrained model and make a final decision after submitting both. Is making such a choice a violation of the rules? Is it one that would (or could) be enforced on a prize-winning team?\n 5. When the FAQ says, \"You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.\" I take it the second sentence supersedes the first? IOW the first sentence only referred to tuning that was not automated in the uploaded model?\n 6. Do you have to select your stage 2 submission explicitly? (I'm assuming the answer is yes, since the now invalid stage 1 submission seems to remain checked after submitting for stage 2, but this ought to be spelled out somewhere. Or will an invalid selection trigger an automatic re-selection of the best-scoring model?)\n 7. Medal positions on current LB appear to reflect only stage 2 submissions, but the FAQ says medal positions are determined by the number of stage 1 submissions. I take it this is just a glitch in the presentation, and we should ignore the medal ranges as presented?\n\n [1]: https://www.kaggle.com/two-stage-frequently-asked-questions", "410882": "Thank you for your detailed questions. Some responses:\n\n - The public leaderboard for Stage 2 is intentionally based on a very small number of images. Its purpose is to give people a basis for testing that their submissions don't error out and some small signal as to how your submission is performing on the Stage 2 data. How participants decide to use their daily submissions is up to them.\n - Ultimately, you are permitted 1 final submission selection. That submission is required to match the model that was uploaded in Stage 1, in adherence to the rules. You can expect the host to validate this, most certainly for those in prize standing.\n - Yes, participants should explicitly select their Stage 2 final submission.\n - Regarding the FAQ statement: \"You should not be doing any hyper parameter tuning in the second stage. Parameter tuning is permitted as long as it is fully automated.\" >> Yes, only fully automated parameter tuning is permitted.\n - The medal positions and rankings are awarded based on the stage 2 standing. **However**, the total number of participants in the competition is based on the Stage 1 volume of submitters.\n - In any contest, there are an unbound number of ways participants might try to game an advantage. We've seen many, and as such do our very best to construct an environment that (a) fosters learning and sharing among people who share a passion for data science & (b) gives our hosts a basis for advancing solutions to their very real-life problems. In particular, this competition's topic has the potential to make a tremendous impact on radiology and healthcare. We implore competitors to direct their energy in the spirit of that mission.", "412557": "Could you please give an example of fully automated parameter?" } }