{
  "id": 40569,
  "title": "Dealing with JSON files instead of BSON",
  "url": "/competitions/cdiscount-image-classification-challenge/discussion/40569",
  "author_name": "battafordata",
  "post_date": "2017-10-04T15:10:16.846000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>I've used <em>bsondump</em> (MongoDB) to dump BSON files to JSON. It's the fastest I've seen, Entire test.bson (~16GB) converted in 22 mins to a ~40GB file. So must have disk space!</p>\n\n<p>The JSON looks like follows (truncated image base64 string for brevity):</p>\n\n<p>{\"_id\":0,\"imgs\":[{\"picture\":{\"$binary\":\"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLD......\",\"$type\":\"00\"}}],\"category_id\":1000010653}\n{\"_id\":1,\"imgs\":[{\"picture\":{\"$binary\":\"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLD......\",\"$type\":\"00\"}}],\"category_id\":1000010653}</p>\n\n<p>So each line in JSON can be converted to a dictionary in any language.\nImage Base64 can be converted to bytes and written to disk as an image file.</p>",
  "messages": [
    {
      "id": 227524,
      "postDate": "2017-10-04T15:10:16.847Z",
      "content": "<p>I've used <em>bsondump</em> (MongoDB) to dump BSON files to JSON. It's the fastest I've seen, Entire test.bson (~16GB) converted in 22 mins to a ~40GB file. So must have disk space!</p>\n\n<p>The JSON looks like follows (truncated image base64 string for brevity):</p>\n\n<p>{\"_id\":0,\"imgs\":[{\"picture\":{\"$binary\":\"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLD......\",\"$type\":\"00\"}}],\"category_id\":1000010653}\n{\"_id\":1,\"imgs\":[{\"picture\":{\"$binary\":\"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLD......\",\"$type\":\"00\"}}],\"category_id\":1000010653}</p>\n\n<p>So each line in JSON can be converted to a dictionary in any language.\nImage Base64 can be converted to bytes and written to disk as an image file.</p>",
      "rawMarkdown": "I've used *bsondump* (MongoDB) to dump BSON files to JSON. It's the fastest I've seen, Entire test.bson (~16GB) converted in 22 mins to a ~40GB file. So must have disk space!\n\nThe JSON looks like follows (truncated image base64 string for brevity):\n\n{\"_id\":0,\"imgs\":[{\"picture\":{\"$binary\":\"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLD......\",\"$type\":\"00\"}}],\"category_id\":1000010653}\n{\"_id\":1,\"imgs\":[{\"picture\":{\"$binary\":\"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLD......\",\"$type\":\"00\"}}],\"category_id\":1000010653}\n\nSo each line in JSON can be converted to a dictionary in any language.\nImage Base64 can be converted to bytes and written to disk as an image file.",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "227524": "I've used *bsondump* (MongoDB) to dump BSON files to JSON. It's the fastest I've seen, Entire test.bson (~16GB) converted in 22 mins to a ~40GB file. So must have disk space!\n\nThe JSON looks like follows (truncated image base64 string for brevity):\n\n{\"_id\":0,\"imgs\":[{\"picture\":{\"$binary\":\"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLD......\",\"$type\":\"00\"}}],\"category_id\":1000010653}\n{\"_id\":1,\"imgs\":[{\"picture\":{\"$binary\":\"/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLD......\",\"$type\":\"00\"}}],\"category_id\":1000010653}\n\nSo each line in JSON can be converted to a dictionary in any language.\nImage Base64 can be converted to bytes and written to disk as an image file."
  }
}