{
  "id": 132090,
  "title": "How much is your computer's memory？",
  "url": "/competitions/bengaliai-cv19/discussion/132090",
  "author_name": "",
  "post_date": "2020-02-24T03:03:32.363982600Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>My cpu is i9 9900k, gpu is 2080ti, current memory is 32G, do I need to increase memory to improve training efficiency? \nThanks.</p>",
  "messages": [
    {
      "id": "754764",
      "postDate": "02/24/2020 03:03:32",
      "content": "<p>My cpu is i9 9900k, gpu is 2080ti, current memory is 32G, do I need to increase memory to improve training efficiency? \nThanks.</p>",
      "rawMarkdown": "My cpu is i9 9900k, gpu is 2080ti, current memory is 32G, do I need to increase memory to improve training efficiency? \nThanks.",
      "votes": null
    },
    {
      "id": "754912",
      "postDate": "02/24/2020 07:56:25",
      "content": "<p>The whole dataset fits in the 16Gb of the Kaggle notebooks' RAM, so going to 32 and over isn't going to change anything. Your 2080Ti has 11Gb, which is already quite big, and that's what you'd want to improve if you had money to through.</p>\n\n<p>But honestly, a 2080Ti is already providing diminishing returns in terms of price / performance IMHO, and being clever about the way you make your model will go a long way to improve your performances.</p>\n\n<p>This dataset isn't particularly big (~6Gb of training images if I remember correctly) so there's a lot of room to tinker with some pre-processing, batch sizes, architectures, etc.</p>",
      "rawMarkdown": "The whole dataset fits in the 16Gb of the Kaggle notebooks' RAM, so going to 32 and over isn't going to change anything. Your 2080Ti has 11Gb, which is already quite big, and that's what you'd want to improve if you had money to through.\n\nBut honestly, a 2080Ti is already providing diminishing returns in terms of price / performance IMHO, and being clever about the way you make your model will go a long way to improve your performances.\n\nThis dataset isn't particularly big (~6Gb of training images if I remember correctly) so there's a lot of room to tinker with some pre-processing, batch sizes, architectures, etc.",
      "votes": null
    },
    {
      "id": "754926",
      "postDate": "02/24/2020 08:18:49",
      "content": "<p>when i set num_workers &gt;=3, there has a MemoryError:\nTraceback (most recent call last):\n......\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\spawn.py\", line 115, in _main\n    self = reduction.pickle.load(from_parent)\nMemoryError</p>",
      "rawMarkdown": "when i set num_workers &gt;=3, there has a MemoryError:\nTraceback (most recent call last):\n......\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\spawn.py\", line 115, in _main\n    self = reduction.pickle.load(from_parent)\nMemoryError",
      "votes": null
    },
    {
      "id": "754927",
      "postDate": "02/24/2020 08:20:11",
      "content": "<p>Traceback (most recent call last):\n......\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\process.py\", line 112, in start\n    self._popen = self._Popen(self)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\context.py\", line 223, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\context.py\", line 322, in _Popen\n    return Popen(process_obj)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\popen_spawn_win32.py\", line 89, in <strong>init</strong>\n    reduction.dump(process_obj, to_child)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\reduction.py\", line 60, in dump\n    ForkingPickler(file, protocol).dump(obj)\nBrokenPipeError: [Errno 32] Broken pipe</p>",
      "rawMarkdown": "Traceback (most recent call last):\n......\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\process.py\", line 112, in start\n    self._popen = self._Popen(self)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\context.py\", line 223, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\context.py\", line 322, in _Popen\n    return Popen(process_obj)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\popen_spawn_win32.py\", line 89, in __init__\n    reduction.dump(process_obj, to_child)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\reduction.py\", line 60, in dump\n    ForkingPickler(file, protocol).dump(obj)\nBrokenPipeError: [Errno 32] Broken pipe",
      "votes": null
    },
    {
      "id": "755037",
      "postDate": "02/24/2020 11:19:31",
      "content": "<p>Nice Information</p>",
      "rawMarkdown": "Nice Information",
      "votes": null
    },
    {
      "id": "755101",
      "postDate": "02/24/2020 12:40:54",
      "content": "<p>Experimental results:\nmemory = 32G, num_workers = 3(just train_loader)\nmemory = 48G, num_workers = 5(just train_loader)</p>",
      "rawMarkdown": "Experimental results:\nmemory = 32G, num_workers = 3(just train_loader)\nmemory = 48G, num_workers = 5(just train_loader)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 754912,
      "author_name": "maxlenormand",
      "author_url": "",
      "post_date": "02/24/2020 07:56:25",
      "content": "<p>The whole dataset fits in the 16Gb of the Kaggle notebooks' RAM, so going to 32 and over isn't going to change anything. Your 2080Ti has 11Gb, which is already quite big, and that's what you'd want to improve if you had money to through.</p>\n\n<p>But honestly, a 2080Ti is already providing diminishing returns in terms of price / performance IMHO, and being clever about the way you make your model will go a long way to improve your performances.</p>\n\n<p>This dataset isn't particularly big (~6Gb of training images if I remember correctly) so there's a lot of room to tinker with some pre-processing, batch sizes, architectures, etc.</p>",
      "votes": null,
      "replies": [
        {
          "id": 754926,
          "author_name": "ludongliang",
          "author_url": "",
          "post_date": "02/24/2020 08:18:49",
          "content": "<p>when i set num_workers &gt;=3, there has a MemoryError:\nTraceback (most recent call last):\n......\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\spawn.py\", line 115, in _main\n    self = reduction.pickle.load(from_parent)\nMemoryError</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 754927,
          "author_name": "ludongliang",
          "author_url": "",
          "post_date": "02/24/2020 08:20:11",
          "content": "<p>Traceback (most recent call last):\n......\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\process.py\", line 112, in start\n    self._popen = self._Popen(self)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\context.py\", line 223, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\context.py\", line 322, in _Popen\n    return Popen(process_obj)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\popen_spawn_win32.py\", line 89, in <strong>init</strong>\n    reduction.dump(process_obj, to_child)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\reduction.py\", line 60, in dump\n    ForkingPickler(file, protocol).dump(obj)\nBrokenPipeError: [Errno 32] Broken pipe</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 755037,
      "author_name": "alexandrann",
      "author_url": "",
      "post_date": "02/24/2020 11:19:31",
      "content": "<p>Nice Information</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 755101,
      "author_name": "ludongliang",
      "author_url": "",
      "post_date": "02/24/2020 12:40:54",
      "content": "<p>Experimental results:\nmemory = 32G, num_workers = 3(just train_loader)\nmemory = 48G, num_workers = 5(just train_loader)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "754764": "My cpu is i9 9900k, gpu is 2080ti, current memory is 32G, do I need to increase memory to improve training efficiency? \nThanks.",
    "754912": "The whole dataset fits in the 16Gb of the Kaggle notebooks' RAM, so going to 32 and over isn't going to change anything. Your 2080Ti has 11Gb, which is already quite big, and that's what you'd want to improve if you had money to through.\n\nBut honestly, a 2080Ti is already providing diminishing returns in terms of price / performance IMHO, and being clever about the way you make your model will go a long way to improve your performances.\n\nThis dataset isn't particularly big (~6Gb of training images if I remember correctly) so there's a lot of room to tinker with some pre-processing, batch sizes, architectures, etc.",
    "754926": "when i set num_workers &gt;=3, there has a MemoryError:\nTraceback (most recent call last):\n......\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\spawn.py\", line 115, in _main\n    self = reduction.pickle.load(from_parent)\nMemoryError",
    "754927": "Traceback (most recent call last):\n......\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\process.py\", line 112, in start\n    self._popen = self._Popen(self)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\context.py\", line 223, in _Popen\n    return _default_context.get_context().Process._Popen(process_obj)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\context.py\", line 322, in _Popen\n    return Popen(process_obj)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\popen_spawn_win32.py\", line 89, in __init__\n    reduction.dump(process_obj, to_child)\n  File \"D:\\pTools\\Anaconda3\\lib\\multiprocessing\\reduction.py\", line 60, in dump\n    ForkingPickler(file, protocol).dump(obj)\nBrokenPipeError: [Errno 32] Broken pipe",
    "755037": "Nice Information",
    "755101": "Experimental results:\nmemory = 32G, num_workers = 3(just train_loader)\nmemory = 48G, num_workers = 5(just train_loader)"
  },
  "source": "meta"
}