{
  "id": 76742,
  "title": "Code to read parquet data",
  "url": "/competitions/vsb-power-line-fault-detection/discussion/76742",
  "author_name": "simon",
  "post_date": "2019-01-06T09:39:30.155000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi guys, try this, you will get a numpy array, each row connect to a sample.</p>\n\n<p>```python\nimport os\nimport numpy as np\nimport pandas as pd\nimport pyarrow.parquet as pq</p>\n\n<p>def read_table(sPath):\n    # Read parquet data, and return a numpy array\n    pdData = pq.read_table(sPath).to_pandas()\n    pdData.info()\n    return pdData.values.T</p>\n\n<p>def main():\n    sTrainPath = \"../data/train.parquet\"\n    sTestPath = \"../data/test.parquet\"\n    sSubsetPath = \"../data/subset.parquet\"\n    npData = read_table(sSubsetPath)\n    print(npData[0,0:100])</p>\n\n<p>if <strong>name</strong> == \"<strong>main</strong>\":\n    main()\n```</p>",
  "messages": [
    {
      "id": 451021,
      "postDate": "2019-01-06T09:39:30.157Z",
      "content": "<p>Hi guys, try this, you will get a numpy array, each row connect to a sample.</p>\n\n<p>```python\nimport os\nimport numpy as np\nimport pandas as pd\nimport pyarrow.parquet as pq</p>\n\n<p>def read_table(sPath):\n    # Read parquet data, and return a numpy array\n    pdData = pq.read_table(sPath).to_pandas()\n    pdData.info()\n    return pdData.values.T</p>\n\n<p>def main():\n    sTrainPath = \"../data/train.parquet\"\n    sTestPath = \"../data/test.parquet\"\n    sSubsetPath = \"../data/subset.parquet\"\n    npData = read_table(sSubsetPath)\n    print(npData[0,0:100])</p>\n\n<p>if <strong>name</strong> == \"<strong>main</strong>\":\n    main()\n```</p>",
      "rawMarkdown": "Hi guys, try this, you will get a numpy array, each row connect to a sample.\n\n```python\nimport os\nimport numpy as np\nimport pandas as pd\nimport pyarrow.parquet as pq\n\ndef read_table(sPath):\n    # Read parquet data, and return a numpy array\n    pdData = pq.read_table(sPath).to_pandas()\n    pdData.info()\n    return pdData.values.T\n\ndef main():\n    sTrainPath = \"../data/train.parquet\"\n    sTestPath = \"../data/test.parquet\"\n    sSubsetPath = \"../data/subset.parquet\"\n    npData = read_table(sSubsetPath)\n    print(npData[0,0:100])\n\nif __name__ == \"__main__\":\n    main()\n```"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "451021": "Hi guys, try this, you will get a numpy array, each row connect to a sample.\n\n```python\nimport os\nimport numpy as np\nimport pandas as pd\nimport pyarrow.parquet as pq\n\ndef read_table(sPath):\n    # Read parquet data, and return a numpy array\n    pdData = pq.read_table(sPath).to_pandas()\n    pdData.info()\n    return pdData.values.T\n\ndef main():\n    sTrainPath = \"../data/train.parquet\"\n    sTestPath = \"../data/test.parquet\"\n    sSubsetPath = \"../data/subset.parquet\"\n    npData = read_table(sSubsetPath)\n    print(npData[0,0:100])\n\nif __name__ == \"__main__\":\n    main()\n```"
  }
}