{
  "id": 332094,
  "title": " How to read the csv files , without getting memory issues. ",
  "url": "/competitions/amex-default-prediction/discussion/332094",
  "author_name": "",
  "post_date": "2022-06-20T11:01:27.046890600Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi,<br>\nI am new to this competition , I am trying to read the train, and test csv files. I'm getting the memory issues .<br>\nCan I use parquet file for this competition which is already preprocessed data by kaggle grand masters.<br>\nIs it allowed for this competition, Nothing will wrong for the model building.</p>",
  "messages": [
    {
      "id": "1826455",
      "postDate": "06/20/2022 11:01:27",
      "content": "<p>Hi,<br>\nI am new to this competition , I am trying to read the train, and test csv files. I'm getting the memory issues .<br>\nCan I use parquet file for this competition which is already preprocessed data by kaggle grand masters.<br>\nIs it allowed for this competition, Nothing will wrong for the model building.</p>",
      "rawMarkdown": "Hi,\nI am new to this competition , I am trying to read the train, and test csv files. I'm getting the memory issues .\nCan I use parquet file for this competition which is already preprocessed data by kaggle grand masters.\nIs it allowed for this competition, Nothing will wrong for the model building.",
      "votes": null
    },
    {
      "id": "1826465",
      "postDate": "06/20/2022 11:12:36",
      "content": "<p>Yes you can use the parquet or feather files which are smaller in size e.g.:</p>\n<p><a href=\"https://www.kaggle.com/datasets/odins0n/amex-parquet\" target=\"_blank\">https://www.kaggle.com/datasets/odins0n/amex-parquet</a></p>",
      "rawMarkdown": "Yes you can use the parquet or feather files which are smaller in size e.g.:\n\nhttps://www.kaggle.com/datasets/odins0n/amex-parquet",
      "votes": null
    },
    {
      "id": "1826876",
      "postDate": "06/20/2022 16:55:54",
      "content": "<p>Use parquet/ feather files. You may also use chunk_size to split the file into several chunks. pd.read_csv has this option</p>",
      "rawMarkdown": "Use parquet/ feather files. You may also use chunk_size to split the file into several chunks. pd.read_csv has this option",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1826465,
      "author_name": "aliabdin1",
      "author_url": "",
      "post_date": "06/20/2022 11:12:36",
      "content": "<p>Yes you can use the parquet or feather files which are smaller in size e.g.:</p>\n<p><a href=\"https://www.kaggle.com/datasets/odins0n/amex-parquet\" target=\"_blank\">https://www.kaggle.com/datasets/odins0n/amex-parquet</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1826876,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "06/20/2022 16:55:54",
      "content": "<p>Use parquet/ feather files. You may also use chunk_size to split the file into several chunks. pd.read_csv has this option</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1826455": "Hi,\nI am new to this competition , I am trying to read the train, and test csv files. I'm getting the memory issues .\nCan I use parquet file for this competition which is already preprocessed data by kaggle grand masters.\nIs it allowed for this competition, Nothing will wrong for the model building.",
    "1826465": "Yes you can use the parquet or feather files which are smaller in size e.g.:\n\nhttps://www.kaggle.com/datasets/odins0n/amex-parquet",
    "1826876": "Use parquet/ feather files. You may also use chunk_size to split the file into several chunks. pd.read_csv has this option"
  },
  "source": "meta"
}