{
  "id": 116268,
  "title": "Memory efficient JSON file reader",
  "url": "/competitions/tensorflow2-question-answering/discussion/116268",
  "author_name": "",
  "post_date": "2019-11-08T03:01:48.598655500Z",
  "votes": 13,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I made a class that can read jsonl file with lower memory.</p>\n\n<p>```\nimport itertools\nimport json</p>\n\n<p>import pandas as pd\nfrom pandas.io.json._json import JsonReader</p>\n\n<p>class JsonChunkReader(JsonReader):\n    \"\"\"JsonReader provides an interface for reading in a JSON file.\n    \"\"\"</p>\n\n<pre><code>@classmethod\ndef chunk_reader(cls, filepath_or_buffer, chunksize):\n    return cls(\n        filepath_or_buffer,\n        orient=None,\n        typ='frame',\n        dtype=None,\n        convert_axes=None,\n        convert_dates=True,\n        keep_default_dates=True,\n        numpy=False,\n        precise_float=False,\n        date_unit=None,\n        encoding=None,\n        lines=True,\n        chunksize=chunksize,\n        compression=None,\n    )\n\ndef __next__(self):\n    lines = list(itertools.islice(self.data, self.chunksize))\n    if lines:\n        obj = pd.DataFrame([\n            json.loads(line)\n            for line in lines\n        ])\n        return obj\n\n    self.close()\n    raise StopIteration\n</code></pre>\n\n<p>```</p>\n\n<p>usage example:</p>\n\n<p><code>\nreader = JsonChunkReader.chunk_reader('../input/tensorflow2-question-answering/simplified-nq-train.jsonl', chunksize=5000)\ntrain = next(reader)\n</code></p>",
  "messages": [
    {
      "id": "668146",
      "postDate": "11/08/2019 03:01:48",
      "content": "<p>I made a class that can read jsonl file with lower memory.</p>\n\n<p>```\nimport itertools\nimport json</p>\n\n<p>import pandas as pd\nfrom pandas.io.json._json import JsonReader</p>\n\n<p>class JsonChunkReader(JsonReader):\n    \"\"\"JsonReader provides an interface for reading in a JSON file.\n    \"\"\"</p>\n\n<pre><code>@classmethod\ndef chunk_reader(cls, filepath_or_buffer, chunksize):\n    return cls(\n        filepath_or_buffer,\n        orient=None,\n        typ='frame',\n        dtype=None,\n        convert_axes=None,\n        convert_dates=True,\n        keep_default_dates=True,\n        numpy=False,\n        precise_float=False,\n        date_unit=None,\n        encoding=None,\n        lines=True,\n        chunksize=chunksize,\n        compression=None,\n    )\n\ndef __next__(self):\n    lines = list(itertools.islice(self.data, self.chunksize))\n    if lines:\n        obj = pd.DataFrame([\n            json.loads(line)\n            for line in lines\n        ])\n        return obj\n\n    self.close()\n    raise StopIteration\n</code></pre>\n\n<p>```</p>\n\n<p>usage example:</p>\n\n<p><code>\nreader = JsonChunkReader.chunk_reader('../input/tensorflow2-question-answering/simplified-nq-train.jsonl', chunksize=5000)\ntrain = next(reader)\n</code></p>",
      "rawMarkdown": "I made a class that can read jsonl file with lower memory.\n\n```\nimport itertools\nimport json\n\nimport pandas as pd\nfrom pandas.io.json._json import JsonReader\n\n\nclass JsonChunkReader(JsonReader):\n    \"\"\"JsonReader provides an interface for reading in a JSON file.\n    \"\"\"\n    \n    @classmethod\n    def chunk_reader(cls, filepath_or_buffer, chunksize):\n        return cls(\n            filepath_or_buffer,\n            orient=None,\n            typ='frame',\n            dtype=None,\n            convert_axes=None,\n            convert_dates=True,\n            keep_default_dates=True,\n            numpy=False,\n            precise_float=False,\n            date_unit=None,\n            encoding=None,\n            lines=True,\n            chunksize=chunksize,\n            compression=None,\n        )\n    \n    def __next__(self):\n        lines = list(itertools.islice(self.data, self.chunksize))\n        if lines:\n            obj = pd.DataFrame([\n                json.loads(line)\n                for line in lines\n            ])\n            return obj\n\n        self.close()\n        raise StopIteration\n```\n\nusage example:\n\n```\nreader = JsonChunkReader.chunk_reader('../input/tensorflow2-question-answering/simplified-nq-train.jsonl', chunksize=5000)\ntrain = next(reader)\n```",
      "votes": null
    },
    {
      "id": "675532",
      "postDate": "11/18/2019 07:28:40",
      "content": "<p>Thanks for sharing, didn't even know about that JsonReader in pandas. </p>",
      "rawMarkdown": "Thanks for sharing, didn't even know about that JsonReader in pandas.",
      "votes": null
    },
    {
      "id": "681688",
      "postDate": "11/26/2019 12:25:35",
      "content": "<p>Thanks for sharing!</p>",
      "rawMarkdown": "Thanks for sharing!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 675532,
      "author_name": "swayson",
      "author_url": "",
      "post_date": "11/18/2019 07:28:40",
      "content": "<p>Thanks for sharing, didn't even know about that JsonReader in pandas. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 681688,
      "author_name": "vishalchauhan",
      "author_url": "",
      "post_date": "11/26/2019 12:25:35",
      "content": "<p>Thanks for sharing!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "668146": "I made a class that can read jsonl file with lower memory.\n\n```\nimport itertools\nimport json\n\nimport pandas as pd\nfrom pandas.io.json._json import JsonReader\n\n\nclass JsonChunkReader(JsonReader):\n    \"\"\"JsonReader provides an interface for reading in a JSON file.\n    \"\"\"\n    \n    @classmethod\n    def chunk_reader(cls, filepath_or_buffer, chunksize):\n        return cls(\n            filepath_or_buffer,\n            orient=None,\n            typ='frame',\n            dtype=None,\n            convert_axes=None,\n            convert_dates=True,\n            keep_default_dates=True,\n            numpy=False,\n            precise_float=False,\n            date_unit=None,\n            encoding=None,\n            lines=True,\n            chunksize=chunksize,\n            compression=None,\n        )\n    \n    def __next__(self):\n        lines = list(itertools.islice(self.data, self.chunksize))\n        if lines:\n            obj = pd.DataFrame([\n                json.loads(line)\n                for line in lines\n            ])\n            return obj\n\n        self.close()\n        raise StopIteration\n```\n\nusage example:\n\n```\nreader = JsonChunkReader.chunk_reader('../input/tensorflow2-question-answering/simplified-nq-train.jsonl', chunksize=5000)\ntrain = next(reader)\n```",
    "675532": "Thanks for sharing, didn't even know about that JsonReader in pandas.",
    "681688": "Thanks for sharing!"
  },
  "source": "meta"
}