{
  "id": 316530,
  "title": "Understanding num_workers in pytorch",
  "url": "/competitions/ultra-mnist/discussion/316530",
  "author_name": "",
  "post_date": "2022-04-02T11:47:34.216813800Z",
  "votes": 4,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi, <br>\nI use pytorch on kaggle  and sometimes google colab to train my models. Recently I figured out about numworkers, which greatly reduced my training time. I tried digging more about it and learned that usually it works best if keep num workers = 4 x number of cpu cores. Kaggle has 4 cpu cores and thus 16 num workers sometimes work, but the other times it wouldn't and I am forces to reduce my num_workers to 4 (sometimes 8 works too). Once, the training paused after around error \"Insufficient shared memory\". Can anyone explain the mechanism of how this num_workers thing works, so that I can utilize the resources in the best possible manner.</p>\n<p>Thank you.</p>",
  "messages": [
    {
      "id": "1742907",
      "postDate": "04/02/2022 11:47:34",
      "content": "<p>Hi, <br>\nI use pytorch on kaggle  and sometimes google colab to train my models. Recently I figured out about numworkers, which greatly reduced my training time. I tried digging more about it and learned that usually it works best if keep num workers = 4 x number of cpu cores. Kaggle has 4 cpu cores and thus 16 num workers sometimes work, but the other times it wouldn't and I am forces to reduce my num_workers to 4 (sometimes 8 works too). Once, the training paused after around error \"Insufficient shared memory\". Can anyone explain the mechanism of how this num_workers thing works, so that I can utilize the resources in the best possible manner.</p>\n<p>Thank you.</p>",
      "rawMarkdown": "Hi, \nI use pytorch on kaggle  and sometimes google colab to train my models. Recently I figured out about numworkers, which greatly reduced my training time. I tried digging more about it and learned that usually it works best if keep num workers = 4 x number of cpu cores. Kaggle has 4 cpu cores and thus 16 num workers sometimes work, but the other times it wouldn't and I am forces to reduce my num_workers to 4 (sometimes 8 works too). Once, the training paused after around error \"Insufficient shared memory\". Can anyone explain the mechanism of how this num_workers thing works, so that I can utilize the resources in the best possible manner.\n\nThank you.",
      "votes": null
    },
    {
      "id": "1743189",
      "postDate": "04/02/2022 17:41:41",
      "content": "<p>kaggle cpu kernels only have 2 virtual cores and 4 threads. gpu kernels only have 1 virtual core and 2 threads. from my own experience setting num_workers &gt; than number of cpu cores does nothing for speedup.</p>",
      "rawMarkdown": "kaggle cpu kernels only have 2 virtual cores and 4 threads. gpu kernels only have 1 virtual core and 2 threads. from my own experience setting num_workers > than number of cpu cores does nothing for speedup.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1743189,
      "author_name": "teeyee314",
      "author_url": "",
      "post_date": "04/02/2022 17:41:41",
      "content": "<p>kaggle cpu kernels only have 2 virtual cores and 4 threads. gpu kernels only have 1 virtual core and 2 threads. from my own experience setting num_workers &gt; than number of cpu cores does nothing for speedup.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1742907": "Hi, \nI use pytorch on kaggle  and sometimes google colab to train my models. Recently I figured out about numworkers, which greatly reduced my training time. I tried digging more about it and learned that usually it works best if keep num workers = 4 x number of cpu cores. Kaggle has 4 cpu cores and thus 16 num workers sometimes work, but the other times it wouldn't and I am forces to reduce my num_workers to 4 (sometimes 8 works too). Once, the training paused after around error \"Insufficient shared memory\". Can anyone explain the mechanism of how this num_workers thing works, so that I can utilize the resources in the best possible manner.\n\nThank you.",
    "1743189": "kaggle cpu kernels only have 2 virtual cores and 4 threads. gpu kernels only have 1 virtual core and 2 threads. from my own experience setting num_workers > than number of cpu cores does nothing for speedup."
  },
  "source": "meta"
}