{
  "id": 41030,
  "title": "Read bson file directly will be faster than read images?",
  "url": "/competitions/cdiscount-image-classification-challenge/discussion/41030",
  "author_name": "",
  "post_date": "2017-10-11T19:13:33.209391900Z",
  "votes": null,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Right now I am using the strategy that processing bson files and save images on disk, and read those images while training. However, I think in the hard disk, the IO will be the bottleneck (see discussion here <a href=\"https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/40498\">https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/40498</a> @zhangsongwei).</p>\n\n<p>So, I am wondering if I use bson file directly, can I get better IO performance?</p>",
  "messages": [
    {
      "id": "230346",
      "postDate": "10/11/2017 19:13:33",
      "content": "<p>Right now I am using the strategy that processing bson files and save images on disk, and read those images while training. However, I think in the hard disk, the IO will be the bottleneck (see discussion here <a href=\"https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/40498\">https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/40498</a> @zhangsongwei).</p>\n\n<p>So, I am wondering if I use bson file directly, can I get better IO performance?</p>",
      "rawMarkdown": "Right now I am using the strategy that processing bson files and save images on disk, and read those images while training. However, I think in the hard disk, the IO will be the bottleneck (see discussion here https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/40498 @zhangsongwei).\n\nSo, I am wondering if I use bson file directly, can I get better IO performance?",
      "votes": null
    },
    {
      "id": "230883",
      "postDate": "10/13/2017 02:25:08",
      "content": "<p>I have implemented a <a href=\"https://www.kaggle.com/theblackcat/loading-bson-data-for-keras-fit-generator\">bson generator</a> which only requires 118 ms to read from each images compared to 315ms to <a href=\"https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson\">Human Analog implementaion</a>. However, there's a serious drawback as this generator can support single worker ( not thread safe ) and no random shuffle for the data. </p>",
      "rawMarkdown": "I have implemented a [bson generator][1] which only requires 118 ms to read from each images compared to 315ms to [Human Analog implementaion][2]. However, there's a serious drawback as this generator can support single worker ( not thread safe ) and no random shuffle for the data. \n\n  [1]: https://www.kaggle.com/theblackcat/loading-bson-data-for-keras-fit-generator\n  [2]: https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson",
      "votes": null
    },
    {
      "id": "232389",
      "postDate": "10/17/2017 12:39:57",
      "content": "<p>Take a look at <a href=\"https://www.kaggle.com/aloisiodn/fast-thread-safe-keras-generator-from-bin-files\">this kernel</a>. It uses auxiliary binary files and is fast and thread safe.</p>",
      "rawMarkdown": "Take a look at [this kernel][1]. It uses auxiliary binary files and is fast and thread safe.\n\n\n  [1]: https://www.kaggle.com/aloisiodn/fast-thread-safe-keras-generator-from-bin-files",
      "votes": null
    },
    {
      "id": "233619",
      "postDate": "10/20/2017 17:55:19",
      "content": "<p>I will definitelt try this</p>",
      "rawMarkdown": "I will definitelt try this",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 230883,
      "author_name": "theblackcat",
      "author_url": "",
      "post_date": "10/13/2017 02:25:08",
      "content": "<p>I have implemented a <a href=\"https://www.kaggle.com/theblackcat/loading-bson-data-for-keras-fit-generator\">bson generator</a> which only requires 118 ms to read from each images compared to 315ms to <a href=\"https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson\">Human Analog implementaion</a>. However, there's a serious drawback as this generator can support single worker ( not thread safe ) and no random shuffle for the data. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 232389,
      "author_name": "aloisiodn",
      "author_url": "",
      "post_date": "10/17/2017 12:39:57",
      "content": "<p>Take a look at <a href=\"https://www.kaggle.com/aloisiodn/fast-thread-safe-keras-generator-from-bin-files\">this kernel</a>. It uses auxiliary binary files and is fast and thread safe.</p>",
      "votes": null,
      "replies": [
        {
          "id": 233619,
          "author_name": "strideradu",
          "author_url": "",
          "post_date": "10/20/2017 17:55:19",
          "content": "<p>I will definitelt try this</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "230346": "Right now I am using the strategy that processing bson files and save images on disk, and read those images while training. However, I think in the hard disk, the IO will be the bottleneck (see discussion here https://www.kaggle.com/c/cdiscount-image-classification-challenge/discussion/40498 @zhangsongwei).\n\nSo, I am wondering if I use bson file directly, can I get better IO performance?",
    "230883": "I have implemented a [bson generator][1] which only requires 118 ms to read from each images compared to 315ms to [Human Analog implementaion][2]. However, there's a serious drawback as this generator can support single worker ( not thread safe ) and no random shuffle for the data. \n\n  [1]: https://www.kaggle.com/theblackcat/loading-bson-data-for-keras-fit-generator\n  [2]: https://www.kaggle.com/humananalog/keras-generator-for-reading-directly-from-bson",
    "232389": "Take a look at [this kernel][1]. It uses auxiliary binary files and is fast and thread safe.\n\n\n  [1]: https://www.kaggle.com/aloisiodn/fast-thread-safe-keras-generator-from-bin-files",
    "233619": "I will definitelt try this"
  },
  "source": "meta"
}