{
  "id": 202154,
  "title": "How to save model efficiently",
  "url": "/competitions/riiid-test-answer-prediction/discussion/202154",
  "author_name": "",
  "post_date": "2020-12-08T16:16:55.398526500Z",
  "votes": 2,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi Guys,<br>\nI have run a model on my local machine, and I save the model using Pickle or joblib. But both method result in a large file, which it could not be loaded in Kaggle notebook due to memory error. Anyone has a better idea to save the model?</p>\n<p>Thanks in advanced.</p>",
  "messages": [
    {
      "id": "1106233",
      "postDate": "12/08/2020 16:16:55",
      "content": "<p>Hi Guys,<br>\nI have run a model on my local machine, and I save the model using Pickle or joblib. But both method result in a large file, which it could not be loaded in Kaggle notebook due to memory error. Anyone has a better idea to save the model?</p>\n<p>Thanks in advanced.</p>",
      "rawMarkdown": "Hi Guys,\nI have run a model on my local machine, and I save the model using Pickle or joblib. But both method result in a large file, which it could not be loaded in Kaggle notebook due to memory error. Anyone has a better idea to save the model?\n\nThanks in advanced.",
      "votes": null
    },
    {
      "id": "1106692",
      "postDate": "12/09/2020 03:26:31",
      "content": "<p>Have you tried to increase the compression of joblib?<br>\nYou can pass the **compress ** param wich will result in a lower size file, but will take a little longer to load.</p>",
      "rawMarkdown": "Have you tried to increase the compression of joblib?\nYou can pass the **compress ** param wich will result in a lower size file, but will take a little longer to load.",
      "votes": null
    },
    {
      "id": "1108418",
      "postDate": "12/10/2020 16:14:33",
      "content": "<p>What type of model do you want to save? Neural Network? XGBoost/LightGBM or TabNet?</p>\n<p>GBM models by default result in small file size. So it should be one of NN or TabNet. One way to reduce space is to only save the weights of the models instead of the whole model. Would be able to comment if we know the model you want to save. </p>",
      "rawMarkdown": "What type of model do you want to save? Neural Network? XGBoost/LightGBM or TabNet?\n\nGBM models by default result in small file size. So it should be one of NN or TabNet. One way to reduce space is to only save the weights of the models instead of the whole model. Would be able to comment if we know the model you want to save.",
      "votes": null
    },
    {
      "id": "1121288",
      "postDate": "12/21/2020 14:23:16",
      "content": "<p>It is not NN. It is LightGBM. But It is a large file with both methods, and cannot be opened by Kaggle Notebooks.</p>",
      "rawMarkdown": "It is not NN. It is LightGBM. But It is a large file with both methods, and cannot be opened by Kaggle Notebooks.",
      "votes": null
    },
    {
      "id": "1121289",
      "postDate": "12/21/2020 14:23:58",
      "content": "<p>Yes, I did. It did not work too. I am looking for a method other than Pickle or Joblib.</p>",
      "rawMarkdown": "Yes, I did. It did not work too. I am looking for a method other than Pickle or Joblib.",
      "votes": null
    },
    {
      "id": "1121891",
      "postDate": "12/22/2020 02:16:10",
      "content": "<p>Even the own lgb's save_model method generates a large size?<br>\nex:<br>\n`<br>\nmodel = lgb.train(params, lgb_train)</p>\n<p>model.save_model('model_train.pkl')<br>\n`<br>\nMy models usualy generates around 875.92 KB only with this method</p>",
      "rawMarkdown": "Even the own lgb's save_model method generates a large size?\nex:\n`\nmodel = lgb.train(params, lgb_train)\n\nmodel.save_model('model_train.pkl')\n`\nMy models usualy generates around 875.92 KB only with this method",
      "votes": null
    },
    {
      "id": "1122102",
      "postDate": "12/22/2020 07:35:11",
      "content": "<p>Thanks. I did not know about this one. I will give it a try.</p>",
      "rawMarkdown": "Thanks. I did not know about this one. I will give it a try.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1106692,
      "author_name": "eduardorenz",
      "author_url": "",
      "post_date": "12/09/2020 03:26:31",
      "content": "<p>Have you tried to increase the compression of joblib?<br>\nYou can pass the **compress ** param wich will result in a lower size file, but will take a little longer to load.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1121289,
          "author_name": "bahramimaryam",
          "author_url": "",
          "post_date": "12/21/2020 14:23:58",
          "content": "<p>Yes, I did. It did not work too. I am looking for a method other than Pickle or Joblib.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1121891,
          "author_name": "eduardorenz",
          "author_url": "",
          "post_date": "12/22/2020 02:16:10",
          "content": "<p>Even the own lgb's save_model method generates a large size?<br>\nex:<br>\n`<br>\nmodel = lgb.train(params, lgb_train)</p>\n<p>model.save_model('model_train.pkl')<br>\n`<br>\nMy models usualy generates around 875.92 KB only with this method</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1122102,
          "author_name": "bahramimaryam",
          "author_url": "",
          "post_date": "12/22/2020 07:35:11",
          "content": "<p>Thanks. I did not know about this one. I will give it a try.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1108418,
      "author_name": "manikanthr5",
      "author_url": "",
      "post_date": "12/10/2020 16:14:33",
      "content": "<p>What type of model do you want to save? Neural Network? XGBoost/LightGBM or TabNet?</p>\n<p>GBM models by default result in small file size. So it should be one of NN or TabNet. One way to reduce space is to only save the weights of the models instead of the whole model. Would be able to comment if we know the model you want to save. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1121288,
          "author_name": "bahramimaryam",
          "author_url": "",
          "post_date": "12/21/2020 14:23:16",
          "content": "<p>It is not NN. It is LightGBM. But It is a large file with both methods, and cannot be opened by Kaggle Notebooks.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1106233": "Hi Guys,\nI have run a model on my local machine, and I save the model using Pickle or joblib. But both method result in a large file, which it could not be loaded in Kaggle notebook due to memory error. Anyone has a better idea to save the model?\n\nThanks in advanced.",
    "1106692": "Have you tried to increase the compression of joblib?\nYou can pass the **compress ** param wich will result in a lower size file, but will take a little longer to load.",
    "1108418": "What type of model do you want to save? Neural Network? XGBoost/LightGBM or TabNet?\n\nGBM models by default result in small file size. So it should be one of NN or TabNet. One way to reduce space is to only save the weights of the models instead of the whole model. Would be able to comment if we know the model you want to save.",
    "1121288": "It is not NN. It is LightGBM. But It is a large file with both methods, and cannot be opened by Kaggle Notebooks.",
    "1121289": "Yes, I did. It did not work too. I am looking for a method other than Pickle or Joblib.",
    "1121891": "Even the own lgb's save_model method generates a large size?\nex:\n`\nmodel = lgb.train(params, lgb_train)\n\nmodel.save_model('model_train.pkl')\n`\nMy models usualy generates around 875.92 KB only with this method",
    "1122102": "Thanks. I did not know about this one. I will give it a try."
  },
  "source": "meta"
}