{
  "id": 34790,
  "title": "@Wendy Kan - clarification on allowed changes",
  "url": "/competitions/intel-mobileodt-cervical-cancer-screening/discussion/34790",
  "author_name": "",
  "post_date": "2017-06-15T17:03:35.798665400Z",
  "votes": 3,
  "comment_count": 7,
  "views": 0,
  "content": "<p>@Wendy Kan</p>\n\n<p>Hi Wendy, admins,</p>\n\n<p>hopefully we can get some clarification (on some \"real-life examples\" of what's allowed and what not), so let me start by some points which I would normally do but I'm not clear if I can:\n1. train an unlimited number of new models (for ensembling) -&gt; with \"new model\" I mean a new instance of a NN with the same architecture (layers, activations, final layer etc.)\n2. change number of epochs I train my models for (each model separately)\n3. change learning rate for each trained model\n4. use a different train/validation split than I have in the submitted code\n5. change the way I create the final ensemble</p>\n\n<p>I think it will be helpful for everyone to know what is \"safe\" to do and what's a define no-no</p>\n\n<p>thanks</p>",
  "messages": [
    {
      "id": "193124",
      "postDate": "06/15/2017 17:03:35",
      "content": "<p>@Wendy Kan</p>\n\n<p>Hi Wendy, admins,</p>\n\n<p>hopefully we can get some clarification (on some \"real-life examples\" of what's allowed and what not), so let me start by some points which I would normally do but I'm not clear if I can:\n1. train an unlimited number of new models (for ensembling) -&gt; with \"new model\" I mean a new instance of a NN with the same architecture (layers, activations, final layer etc.)\n2. change number of epochs I train my models for (each model separately)\n3. change learning rate for each trained model\n4. use a different train/validation split than I have in the submitted code\n5. change the way I create the final ensemble</p>\n\n<p>I think it will be helpful for everyone to know what is \"safe\" to do and what's a define no-no</p>\n\n<p>thanks</p>",
      "rawMarkdown": "Wendy Kan\n\nHi Wendy, admins,\n\nhopefully we can get some clarification (on some \"real-life examples\" of what's allowed and what not), so let me start by some points which I would normally do but I'm not clear if I can:\n1. train an unlimited number of new models (for ensembling) -&gt; with \"new model\" I mean a new instance of a NN with the same architecture (layers, activations, final layer etc.)\n2. change number of epochs I train my models for (each model separately)\n3. change learning rate for each trained model\n4. use a different train/validation split than I have in the submitted code\n5. change the way I create the final ensemble\n\nI think it will be helpful for everyone to know what is \"safe\" to do and what's a define no-no\n\nthanks",
      "votes": null
    },
    {
      "id": "193134",
      "postDate": "06/15/2017 17:21:12",
      "content": "<p>It should be the same as in other competitions with two stages: absolutely zero tuning of parameters or algorithms. This is stated quite clearly in the FAQ (<a href=\"https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580\">https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580</a>). Only run what you have described in your instructions when uploading your model at the end of stage 1.</p>",
      "rawMarkdown": "It should be the same as in other competitions with two stages: absolutely zero tuning of parameters or algorithms. This is stated quite clearly in the FAQ (https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580). Only run what you have described in your instructions when uploading your model at the end of stage 1.",
      "votes": null
    },
    {
      "id": "193143",
      "postDate": "06/15/2017 17:30:46",
      "content": "<p>well, I guess I get it but doesn't make sense - the size of the train data has increased (quite significantly, if we consider that we have the best quality images with correct labels, and on top of that each from a different patient, unlike the data in additional), so training with the same number of epochs, same learning rate etc. can be suboptimal</p>\n\n<p>I always assumed that the purpose of the code submission is to \"freeze\" the models as such - i.e. you don't create a new model with a completely new architecture, new ensembling logic etc.</p>",
      "rawMarkdown": "well, I guess I get it but doesn't make sense - the size of the train data has increased (quite significantly, if we consider that we have the best quality images with correct labels, and on top of that each from a different patient, unlike the data in additional), so training with the same number of epochs, same learning rate etc. can be suboptimal\n\nI always assumed that the purpose of the code submission is to \"freeze\" the models as such - i.e. you don't create a new model with a completely new architecture, new ensembling logic etc.",
      "votes": null
    },
    {
      "id": "193147",
      "postDate": "06/15/2017 17:39:43",
      "content": "<p>Yeah, the whole situation with leaderboard mining is quite unfortunate. There simply is no good way out for the admins if they allow any sort of tuning - deciding what counts as a \"new model\" and what is a minor adjustment isn't trivial. Our team tried to work having in mind, that the algorithm will have to be retrained on more data. We'll see how it pans out...</p>",
      "rawMarkdown": "Yeah, the whole situation with leaderboard mining is quite unfortunate. There simply is no good way out for the admins if they allow any sort of tuning - deciding what counts as a \"new model\" and what is a minor adjustment isn't trivial. Our team tried to work having in mind, that the algorithm will have to be retrained on more data. We'll see how it pans out...",
      "votes": null
    },
    {
      "id": "193370",
      "postDate": "06/16/2017 06:55:19",
      "content": "<p>@ Admins: <br>\nI would like to add two additional questions: <br>\n 6) is it allowed to re-train the \"frozen\" model (uploaded one, unchanged) with ANY kind of NEW augmentation/transformation applied to the original images? <br>\n 7) regarding point 2. from @steelrose, in case changing the number of epochs in the code is not allowed is it possible to run the unchanged code repeatedly (assuming it would always start from the last saved status)?  </p>\n\n<p>Thanks</p>",
      "rawMarkdown": "Admins:  \nI would like to add two additional questions:  \n 6) is it allowed to re-train the \"frozen\" model (uploaded one, unchanged) with ANY kind of NEW augmentation/transformation applied to the original images?  \n 7) regarding point 2. from @steelrose, in case changing the number of epochs in the code is not allowed is it possible to run the unchanged code repeatedly (assuming it would always start from the last saved status)?  \n\nThanks",
      "votes": null
    },
    {
      "id": "193387",
      "postDate": "06/16/2017 08:53:00",
      "content": "<p>You would be allowed to tune learning rate etc, but it has to be in a fully automated manner. If you have to manually change these parameters, it is no longer allowed, afaik.</p>",
      "rawMarkdown": "You would be allowed to tune learning rate etc, but it has to be in a fully automated manner. If you have to manually change these parameters, it is no longer allowed, afaik.",
      "votes": null
    },
    {
      "id": "193394",
      "postDate": "06/16/2017 09:42:16",
      "content": "<p>yeah, anyway most likely no chance to finish in top 3 so will still  do it the \"normal\" way imo \nmy understanding is, that it's more about proving that you don't \"manually\" create resp. modify the submission file, i.e. the submission itself is really done by a computer and not a person (and also don't train on manually prepared labels etc.)</p>",
      "rawMarkdown": "yeah, anyway most likely no chance to finish in top 3 so will still  do it the \"normal\" way imo \nmy understanding is, that it's more about proving that you don't \"manually\" create resp. modify the submission file, i.e. the submission itself is really done by a computer and not a person (and also don't train on manually prepared labels etc.)",
      "votes": null
    },
    {
      "id": "193563",
      "postDate": "06/16/2017 22:30:56",
      "content": "<p>I will assume tuning ensemble weights based on the release of stage 1 labels has to be allowed. Admins please acknowledge!!!! if I had (0.4 * model1 + 0.6 * model2) then I retrained my models and find (0.5 * model1 + 0.5 * model2) is optimal I should be allowed to manually change this. </p>",
      "rawMarkdown": "I will assume tuning ensemble weights based on the release of stage 1 labels has to be allowed. Admins please acknowledge!!!! if I had (0.4 * model1 + 0.6 * model2) then I retrained my models and find (0.5 * model1 + 0.5 * model2) is optimal I should be allowed to manually change this.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 193134,
      "author_name": "zadrras",
      "author_url": "",
      "post_date": "06/15/2017 17:21:12",
      "content": "<p>It should be the same as in other competitions with two stages: absolutely zero tuning of parameters or algorithms. This is stated quite clearly in the FAQ (<a href=\"https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580\">https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580</a>). Only run what you have described in your instructions when uploading your model at the end of stage 1.</p>",
      "votes": null,
      "replies": [
        {
          "id": 193143,
          "author_name": "steelrose",
          "author_url": "",
          "post_date": "06/15/2017 17:30:46",
          "content": "<p>well, I guess I get it but doesn't make sense - the size of the train data has increased (quite significantly, if we consider that we have the best quality images with correct labels, and on top of that each from a different patient, unlike the data in additional), so training with the same number of epochs, same learning rate etc. can be suboptimal</p>\n\n<p>I always assumed that the purpose of the code submission is to \"freeze\" the models as such - i.e. you don't create a new model with a completely new architecture, new ensembling logic etc.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 193147,
          "author_name": "zadrras",
          "author_url": "",
          "post_date": "06/15/2017 17:39:43",
          "content": "<p>Yeah, the whole situation with leaderboard mining is quite unfortunate. There simply is no good way out for the admins if they allow any sort of tuning - deciding what counts as a \"new model\" and what is a minor adjustment isn't trivial. Our team tried to work having in mind, that the algorithm will have to be retrained on more data. We'll see how it pans out...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 193387,
          "author_name": "ankasor",
          "author_url": "",
          "post_date": "06/16/2017 08:53:00",
          "content": "<p>You would be allowed to tune learning rate etc, but it has to be in a fully automated manner. If you have to manually change these parameters, it is no longer allowed, afaik.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 193394,
          "author_name": "steelrose",
          "author_url": "",
          "post_date": "06/16/2017 09:42:16",
          "content": "<p>yeah, anyway most likely no chance to finish in top 3 so will still  do it the \"normal\" way imo \nmy understanding is, that it's more about proving that you don't \"manually\" create resp. modify the submission file, i.e. the submission itself is really done by a computer and not a person (and also don't train on manually prepared labels etc.)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 193563,
          "author_name": "godaibo",
          "author_url": "",
          "post_date": "06/16/2017 22:30:56",
          "content": "<p>I will assume tuning ensemble weights based on the release of stage 1 labels has to be allowed. Admins please acknowledge!!!! if I had (0.4 * model1 + 0.6 * model2) then I retrained my models and find (0.5 * model1 + 0.5 * model2) is optimal I should be allowed to manually change this. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 193370,
      "author_name": "roebius",
      "author_url": "",
      "post_date": "06/16/2017 06:55:19",
      "content": "<p>@ Admins: <br>\nI would like to add two additional questions: <br>\n 6) is it allowed to re-train the \"frozen\" model (uploaded one, unchanged) with ANY kind of NEW augmentation/transformation applied to the original images? <br>\n 7) regarding point 2. from @steelrose, in case changing the number of epochs in the code is not allowed is it possible to run the unchanged code repeatedly (assuming it would always start from the last saved status)?  </p>\n\n<p>Thanks</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "193124": "Wendy Kan\n\nHi Wendy, admins,\n\nhopefully we can get some clarification (on some \"real-life examples\" of what's allowed and what not), so let me start by some points which I would normally do but I'm not clear if I can:\n1. train an unlimited number of new models (for ensembling) -&gt; with \"new model\" I mean a new instance of a NN with the same architecture (layers, activations, final layer etc.)\n2. change number of epochs I train my models for (each model separately)\n3. change learning rate for each trained model\n4. use a different train/validation split than I have in the submitted code\n5. change the way I create the final ensemble\n\nI think it will be helpful for everyone to know what is \"safe\" to do and what's a define no-no\n\nthanks",
    "193134": "It should be the same as in other competitions with two stages: absolutely zero tuning of parameters or algorithms. This is stated quite clearly in the FAQ (https://www.kaggle.com/c/intel-mobileodt-cervical-cancer-screening/discussion/32580). Only run what you have described in your instructions when uploading your model at the end of stage 1.",
    "193143": "well, I guess I get it but doesn't make sense - the size of the train data has increased (quite significantly, if we consider that we have the best quality images with correct labels, and on top of that each from a different patient, unlike the data in additional), so training with the same number of epochs, same learning rate etc. can be suboptimal\n\nI always assumed that the purpose of the code submission is to \"freeze\" the models as such - i.e. you don't create a new model with a completely new architecture, new ensembling logic etc.",
    "193147": "Yeah, the whole situation with leaderboard mining is quite unfortunate. There simply is no good way out for the admins if they allow any sort of tuning - deciding what counts as a \"new model\" and what is a minor adjustment isn't trivial. Our team tried to work having in mind, that the algorithm will have to be retrained on more data. We'll see how it pans out...",
    "193370": "Admins:  \nI would like to add two additional questions:  \n 6) is it allowed to re-train the \"frozen\" model (uploaded one, unchanged) with ANY kind of NEW augmentation/transformation applied to the original images?  \n 7) regarding point 2. from @steelrose, in case changing the number of epochs in the code is not allowed is it possible to run the unchanged code repeatedly (assuming it would always start from the last saved status)?  \n\nThanks",
    "193387": "You would be allowed to tune learning rate etc, but it has to be in a fully automated manner. If you have to manually change these parameters, it is no longer allowed, afaik.",
    "193394": "yeah, anyway most likely no chance to finish in top 3 so will still  do it the \"normal\" way imo \nmy understanding is, that it's more about proving that you don't \"manually\" create resp. modify the submission file, i.e. the submission itself is really done by a computer and not a person (and also don't train on manually prepared labels etc.)",
    "193563": "I will assume tuning ensemble weights based on the release of stage 1 labels has to be allowed. Admins please acknowledge!!!! if I had (0.4 * model1 + 0.6 * model2) then I retrained my models and find (0.5 * model1 + 0.5 * model2) is optimal I should be allowed to manually change this."
  },
  "source": "meta"
}