{
  "id": 360658,
  "title": "Federated Learning to extend Training Time",
  "url": "/competitions/rsna-2022-cervical-spine-fracture-detection/discussion/360658",
  "author_name": "",
  "post_date": "2022-10-17T16:30:37.411086100Z",
  "votes": 15,
  "comment_count": 5,
  "views": 0,
  "content": "<p>In running a model on Kaggle, I exceeded the time limit, so in order to train the entire dataset I implemented a Federated Learning approach.</p>\n<ol>\n<li>First, the model was trained separately on the first and second half of the data.</li>\n<li>The two model weights were averaged using code from <a href=\"https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008\" target=\"_blank\">https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008</a></li>\n<li>The new model was trained on the mid one half of the data.</li>\n</ol>\n<h4>Fastai load_learner from exported models</h4>\n<p>learnA = load_learner('../input/models/model_first_ half.pkl', cpu = False)<br>\nlearnB = load_learner('../input/models/model_second_ half.pkl', cpu = False)</p>\n<p>modelA = learnA.model<br>\nmodelB = learnB.model</p>\n<p>sdA = modelA.state_dict()<br>\nsdB = modelB.state_dict()</p>\n<h4>Average all parameters</h4>\n<p>for key in sdA:<br>\n    sdB[key] = (sdB[key] + sdA[key]) / 2.</p>\n<h4>Recreate model and load averaged state_dict (or use modelA/B)</h4>\n<p>model_fed = MyModel()<br>\nmodel_fed.load_state_dict(sdB)</p>\n<h4>Train new model on middle of  dataset eg. train_df.iloc[500:1500]</h4>",
  "messages": [
    {
      "id": "1992330",
      "postDate": "10/17/2022 16:30:37",
      "content": "<p>In running a model on Kaggle, I exceeded the time limit, so in order to train the entire dataset I implemented a Federated Learning approach.</p>\n<ol>\n<li>First, the model was trained separately on the first and second half of the data.</li>\n<li>The two model weights were averaged using code from <a href=\"https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008\" target=\"_blank\">https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008</a></li>\n<li>The new model was trained on the mid one half of the data.</li>\n</ol>\n<h4>Fastai load_learner from exported models</h4>\n<p>learnA = load_learner('../input/models/model_first_ half.pkl', cpu = False)<br>\nlearnB = load_learner('../input/models/model_second_ half.pkl', cpu = False)</p>\n<p>modelA = learnA.model<br>\nmodelB = learnB.model</p>\n<p>sdA = modelA.state_dict()<br>\nsdB = modelB.state_dict()</p>\n<h4>Average all parameters</h4>\n<p>for key in sdA:<br>\n    sdB[key] = (sdB[key] + sdA[key]) / 2.</p>\n<h4>Recreate model and load averaged state_dict (or use modelA/B)</h4>\n<p>model_fed = MyModel()<br>\nmodel_fed.load_state_dict(sdB)</p>\n<h4>Train new model on middle of  dataset eg. train_df.iloc[500:1500]</h4>",
      "rawMarkdown": "In running a model on Kaggle, I exceeded the time limit, so in order to train the entire dataset I implemented a Federated Learning approach.\n\n1. First, the model was trained separately on the first and second half of the data.\n2. The two model weights were averaged using code from https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008\n3. The new model was trained on the mid one half of the data.\n\n#### Fastai load_learner from exported models\nlearnA = load_learner('../input/models/model_first_ half.pkl', cpu = False)\nlearnB = load_learner('../input/models/model_second_ half.pkl', cpu = False)\n\nmodelA = learnA.model\nmodelB = learnB.model\n\nsdA = modelA.state_dict()\nsdB = modelB.state_dict()\n\n#### Average all parameters\nfor key in sdA:\n    sdB[key] = (sdB[key] + sdA[key]) / 2.\n\n#### Recreate model and load averaged state_dict (or use modelA/B)\nmodel_fed = MyModel()\nmodel_fed.load_state_dict(sdB)\n\n#### Train new model on middle of  dataset eg. train_df.iloc[500:1500]",
      "votes": null
    },
    {
      "id": "1992831",
      "postDate": "10/18/2022 01:04:00",
      "content": "<p>I think your approach is amazing and could be deployed for real-time data! Please do review my work and give your feedback to whenever you get time 😊</p>",
      "rawMarkdown": "I think your approach is amazing and could be deployed for real-time data! Please do review my work and give your feedback to whenever you get time 😊",
      "votes": null
    },
    {
      "id": "1993054",
      "postDate": "10/18/2022 05:12:58",
      "content": "<p>I may not choose to manipulate with weights directly. Have you ever tried ensembled models or training with teacher-student networks?</p>",
      "rawMarkdown": "I may not choose to manipulate with weights directly. Have you ever tried ensembled models or training with teacher-student networks?",
      "votes": null
    },
    {
      "id": "1993800",
      "postDate": "10/18/2022 15:43:30",
      "content": "<p>I have not tried either and I'm actually struggling to get past Submission Not Found errors not matter what I try. Thank you for your comment!</p>",
      "rawMarkdown": "I have not tried either and I'm actually struggling to get past Submission Not Found errors not matter what I try. Thank you for your comment!",
      "votes": null
    },
    {
      "id": "1997487",
      "postDate": "10/21/2022 00:17:09",
      "content": "<p>You may read <a href=\"https://www.kaggle.com/code/fx6300/kaggle-error\" target=\"_blank\">this post</a>. It shows a proper way to how to generate a submission file.</p>",
      "rawMarkdown": "You may read [this post](https://www.kaggle.com/code/fx6300/kaggle-error). It shows a proper way to how to generate a submission file.",
      "votes": null
    },
    {
      "id": "1997500",
      "postDate": "10/21/2022 00:37:59",
      "content": "<p>Yes that notebook was very helpful!! I tried saving the jpg to kaggle/ rather than kaggle/working and it got past the sticking point :-)</p>",
      "rawMarkdown": "Yes that notebook was very helpful!! I tried saving the jpg to kaggle/ rather than kaggle/working and it got past the sticking point :-)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1992831,
      "author_name": "muhammadammarjamshed",
      "author_url": "",
      "post_date": "10/18/2022 01:04:00",
      "content": "<p>I think your approach is amazing and could be deployed for real-time data! Please do review my work and give your feedback to whenever you get time 😊</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1993054,
      "author_name": "chenjiexu",
      "author_url": "",
      "post_date": "10/18/2022 05:12:58",
      "content": "<p>I may not choose to manipulate with weights directly. Have you ever tried ensembled models or training with teacher-student networks?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1993800,
          "author_name": "anoukstein",
          "author_url": "",
          "post_date": "10/18/2022 15:43:30",
          "content": "<p>I have not tried either and I'm actually struggling to get past Submission Not Found errors not matter what I try. Thank you for your comment!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1997487,
          "author_name": "chenjiexu",
          "author_url": "",
          "post_date": "10/21/2022 00:17:09",
          "content": "<p>You may read <a href=\"https://www.kaggle.com/code/fx6300/kaggle-error\" target=\"_blank\">this post</a>. It shows a proper way to how to generate a submission file.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1997500,
          "author_name": "anoukstein",
          "author_url": "",
          "post_date": "10/21/2022 00:37:59",
          "content": "<p>Yes that notebook was very helpful!! I tried saving the jpg to kaggle/ rather than kaggle/working and it got past the sticking point :-)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1992330": "In running a model on Kaggle, I exceeded the time limit, so in order to train the entire dataset I implemented a Federated Learning approach.\n\n1. First, the model was trained separately on the first and second half of the data.\n2. The two model weights were averaged using code from https://discuss.pytorch.org/t/average-each-weight-of-two-models/77008\n3. The new model was trained on the mid one half of the data.\n\n#### Fastai load_learner from exported models\nlearnA = load_learner('../input/models/model_first_ half.pkl', cpu = False)\nlearnB = load_learner('../input/models/model_second_ half.pkl', cpu = False)\n\nmodelA = learnA.model\nmodelB = learnB.model\n\nsdA = modelA.state_dict()\nsdB = modelB.state_dict()\n\n#### Average all parameters\nfor key in sdA:\n    sdB[key] = (sdB[key] + sdA[key]) / 2.\n\n#### Recreate model and load averaged state_dict (or use modelA/B)\nmodel_fed = MyModel()\nmodel_fed.load_state_dict(sdB)\n\n#### Train new model on middle of  dataset eg. train_df.iloc[500:1500]",
    "1992831": "I think your approach is amazing and could be deployed for real-time data! Please do review my work and give your feedback to whenever you get time 😊",
    "1993054": "I may not choose to manipulate with weights directly. Have you ever tried ensembled models or training with teacher-student networks?",
    "1993800": "I have not tried either and I'm actually struggling to get past Submission Not Found errors not matter what I try. Thank you for your comment!",
    "1997487": "You may read [this post](https://www.kaggle.com/code/fx6300/kaggle-error). It shows a proper way to how to generate a submission file.",
    "1997500": "Yes that notebook was very helpful!! I tried saving the jpg to kaggle/ rather than kaggle/working and it got past the sticking point :-)"
  },
  "source": "meta"
}