{
  "id": 74360,
  "title": "One trick to speed up model training (For Rookie like me = =)",
  "url": "/competitions/human-protein-atlas-image-classification/discussion/74360",
  "author_name": "",
  "post_date": "2018-12-11T13:58:00.456723700Z",
  "votes": 5,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Just share a little trick I just discovered to speed up the iteration .\nIf you have a SSD (Solid-state disk) , you may  put your data in that disk .Then the speed of loading data will be greatly improved.Because I am a newbie, I didn’t think of this before :).\nMy model's previous iteration time is mainly consumed on loading data.And because the size of the dataset is too large.I can't load it to my RAM totally.\nMy disk C is a SSD. After I put my data in disk C.My epoch training speed is more than twice that of the previous one.\nSo if you have a SSD ,just try to put your data there!\nI hope this advice can help you.If you have any ideas to speed up the iteration,just share it here, thanks!~</p>",
  "messages": [
    {
      "id": "437184",
      "postDate": "12/11/2018 13:58:00",
      "content": "<p>Just share a little trick I just discovered to speed up the iteration .\nIf you have a SSD (Solid-state disk) , you may  put your data in that disk .Then the speed of loading data will be greatly improved.Because I am a newbie, I didn’t think of this before :).\nMy model's previous iteration time is mainly consumed on loading data.And because the size of the dataset is too large.I can't load it to my RAM totally.\nMy disk C is a SSD. After I put my data in disk C.My epoch training speed is more than twice that of the previous one.\nSo if you have a SSD ,just try to put your data there!\nI hope this advice can help you.If you have any ideas to speed up the iteration,just share it here, thanks!~</p>",
      "rawMarkdown": "Just share a little trick I just discovered to speed up the iteration .\nIf you have a SSD (Solid-state disk) , you may  put your data in that disk .Then the speed of loading data will be greatly improved.Because I am a newbie, I didn’t think of this before :).\nMy model's previous iteration time is mainly consumed on loading data.And because the size of the dataset is too large.I can't load it to my RAM totally.\nMy disk C is a SSD. After I put my data in disk C.My epoch training speed is more than twice that of the previous one.\nSo if you have a SSD ,just try to put your data there!\nI hope this advice can help you.If you have any ideas to speed up the iteration,just share it here, thanks!~",
      "votes": null
    },
    {
      "id": "437349",
      "postDate": "12/11/2018 18:47:52",
      "content": "<p>Good summarized, Basically for serious machine learning an SSD drive is a 'must have'. You will benefit even more if you have a dedicated SSD drive for all your data and models.</p>\n\n<p>There are offcourse different types of SSD's but even a cheap one will give you enormous speedup over an HDD. </p>\n\n<p>If you do have the ability to use memory then preloading  your data into memory from SSD will almost be you're golden egg ;-)</p>\n\n<p>Good luck with the competition.</p>",
      "rawMarkdown": "Good summarized, Basically for serious machine learning an SSD drive is a 'must have'. You will benefit even more if you have a dedicated SSD drive for all your data and models.\n\nThere are offcourse different types of SSD's but even a cheap one will give you enormous speedup over an HDD. \n\nIf you do have the ability to use memory then preloading  your data into memory from SSD will almost be you're golden egg ;-)\n\nGood luck with the competition.",
      "votes": null
    },
    {
      "id": "437488",
      "postDate": "12/12/2018 02:33:42",
      "content": "<p>Thanks for your advice !</p>",
      "rawMarkdown": "Thanks for your advice !",
      "votes": null
    },
    {
      "id": "440637",
      "postDate": "12/17/2018 20:11:08",
      "content": "<p>Have you ever tried nvme SSD which is faster than basic SATA SSD. If the bottleneck is still the disk, you can improve the speed further.</p>",
      "rawMarkdown": "Have you ever tried nvme SSD which is faster than basic SATA SSD. If the bottleneck is still the disk, you can improve the speed further.",
      "votes": null
    },
    {
      "id": "440684",
      "postDate": "12/17/2018 22:17:06",
      "content": "<p>Caching in memory and multithreaded prefetch (although tricky to make) is probably something to try </p>",
      "rawMarkdown": "Caching in memory and multithreaded prefetch (although tricky to make) is probably something to try",
      "votes": null
    },
    {
      "id": "440685",
      "postDate": "12/17/2018 22:17:06",
      "content": "<p>You can also put your data into an h5 file. I put all the images in an h5 using the filename as the key for each array, epoch time went from 45mins loading images from SSD to 25 mins loading with h5. </p>",
      "rawMarkdown": "You can also put your data into an h5 file. I put all the images in an h5 using the filename as the key for each array, epoch time went from 45mins loading images from SSD to 25 mins loading with h5.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 437349,
      "author_name": "rsmits",
      "author_url": "",
      "post_date": "12/11/2018 18:47:52",
      "content": "<p>Good summarized, Basically for serious machine learning an SSD drive is a 'must have'. You will benefit even more if you have a dedicated SSD drive for all your data and models.</p>\n\n<p>There are offcourse different types of SSD's but even a cheap one will give you enormous speedup over an HDD. </p>\n\n<p>If you do have the ability to use memory then preloading  your data into memory from SSD will almost be you're golden egg ;-)</p>\n\n<p>Good luck with the competition.</p>",
      "votes": null,
      "replies": [
        {
          "id": 437488,
          "author_name": "lrs1353281004",
          "author_url": "",
          "post_date": "12/12/2018 02:33:42",
          "content": "<p>Thanks for your advice !</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 440637,
      "author_name": "appian",
      "author_url": "",
      "post_date": "12/17/2018 20:11:08",
      "content": "<p>Have you ever tried nvme SSD which is faster than basic SATA SSD. If the bottleneck is still the disk, you can improve the speed further.</p>",
      "votes": null,
      "replies": [
        {
          "id": 440684,
          "author_name": "moshel",
          "author_url": "",
          "post_date": "12/17/2018 22:17:06",
          "content": "<p>Caching in memory and multithreaded prefetch (although tricky to make) is probably something to try </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 440685,
      "author_name": "florianm",
      "author_url": "",
      "post_date": "12/17/2018 22:17:06",
      "content": "<p>You can also put your data into an h5 file. I put all the images in an h5 using the filename as the key for each array, epoch time went from 45mins loading images from SSD to 25 mins loading with h5. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "437184": "Just share a little trick I just discovered to speed up the iteration .\nIf you have a SSD (Solid-state disk) , you may  put your data in that disk .Then the speed of loading data will be greatly improved.Because I am a newbie, I didn’t think of this before :).\nMy model's previous iteration time is mainly consumed on loading data.And because the size of the dataset is too large.I can't load it to my RAM totally.\nMy disk C is a SSD. After I put my data in disk C.My epoch training speed is more than twice that of the previous one.\nSo if you have a SSD ,just try to put your data there!\nI hope this advice can help you.If you have any ideas to speed up the iteration,just share it here, thanks!~",
    "437349": "Good summarized, Basically for serious machine learning an SSD drive is a 'must have'. You will benefit even more if you have a dedicated SSD drive for all your data and models.\n\nThere are offcourse different types of SSD's but even a cheap one will give you enormous speedup over an HDD. \n\nIf you do have the ability to use memory then preloading  your data into memory from SSD will almost be you're golden egg ;-)\n\nGood luck with the competition.",
    "437488": "Thanks for your advice !",
    "440637": "Have you ever tried nvme SSD which is faster than basic SATA SSD. If the bottleneck is still the disk, you can improve the speed further.",
    "440684": "Caching in memory and multithreaded prefetch (although tricky to make) is probably something to try",
    "440685": "You can also put your data into an h5 file. I put all the images in an h5 using the filename as the key for each array, epoch time went from 45mins loading images from SSD to 25 mins loading with h5."
  },
  "source": "meta"
}