{
  "id": 501440,
  "title": "Processing whole training data in kaggle ",
  "url": "/competitions/leash-BELKA/discussion/501440",
  "author_name": "",
  "post_date": "2024-05-09T08:14:17.209181500Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Dear all, I have created a notebook for parsing whole training data in Kaggle with minimal CPU and RAM usage. I performed some EDA tasks however I believe many other EDA tasks can also be performed. Your reviews and comments are highly appreciated. Here is the link of the notebook<br>\n<a href=\"https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data\" target=\"_blank\">https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data</a> </p>",
  "messages": [
    {
      "id": "2802902",
      "postDate": "05/09/2024 08:14:17",
      "content": "<p>Dear all, I have created a notebook for parsing whole training data in Kaggle with minimal CPU and RAM usage. I performed some EDA tasks however I believe many other EDA tasks can also be performed. Your reviews and comments are highly appreciated. Here is the link of the notebook<br>\n<a href=\"https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data\" target=\"_blank\">https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data</a> </p>",
      "rawMarkdown": "Dear all, I have created a notebook for parsing whole training data in Kaggle with minimal CPU and RAM usage. I performed some EDA tasks however I believe many other EDA tasks can also be performed. Your reviews and comments are highly appreciated. Here is the link of the notebook\nhttps://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data",
      "votes": null
    },
    {
      "id": "2809327",
      "postDate": "05/12/2024 17:11:27",
      "content": "<p>Thank you for notebook link</p>",
      "rawMarkdown": "Thank you for notebook link",
      "votes": null
    },
    {
      "id": "2812074",
      "postDate": "05/14/2024 05:15:37",
      "content": "<p>Hello, I am Rajiv from Mumbai. I am not able to open the xl input as it is very large. Can anyone suggest solution? </p>",
      "rawMarkdown": "Hello, I am Rajiv from Mumbai. I am not able to open the xl input as it is very large. Can anyone suggest solution?",
      "votes": null
    },
    {
      "id": "2812621",
      "postDate": "05/14/2024 10:31:00",
      "content": "<p><a href=\"https://www.kaggle.com/profrajiviyer\" target=\"_blank\">@profrajiviyer</a>, If you have large data files like in this competition and you have limited RAM, for example, even if you have 30GM RAM (I have test on kaggle as well), you can not open so big file at once. The trick is to read/ process this data in batches or chunks as I have explained the notebook above. For your convenience, the link is being reproduced. <a href=\"https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data\" target=\"_blank\">https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data</a></p>",
      "rawMarkdown": "profrajiviyer, If you have large data files like in this competition and you have limited RAM, for example, even if you have 30GM RAM (I have test on kaggle as well), you can not open so big file at once. The trick is to read/ process this data in batches or chunks as I have explained the notebook above. For your convenience, the link is being reproduced. https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2809327,
      "author_name": "",
      "author_url": "",
      "post_date": "05/12/2024 17:11:27",
      "content": "<p>Thank you for notebook link</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2812074,
      "author_name": "profrajiviyer",
      "author_url": "",
      "post_date": "05/14/2024 05:15:37",
      "content": "<p>Hello, I am Rajiv from Mumbai. I am not able to open the xl input as it is very large. Can anyone suggest solution? </p>",
      "votes": null,
      "replies": [
        {
          "id": 2812621,
          "author_name": "tariqcp",
          "author_url": "",
          "post_date": "05/14/2024 10:31:00",
          "content": "<p><a href=\"https://www.kaggle.com/profrajiviyer\" target=\"_blank\">@profrajiviyer</a>, If you have large data files like in this competition and you have limited RAM, for example, even if you have 30GM RAM (I have test on kaggle as well), you can not open so big file at once. The trick is to read/ process this data in batches or chunks as I have explained the notebook above. For your convenience, the link is being reproduced. <a href=\"https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data\" target=\"_blank\">https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2802902": "Dear all, I have created a notebook for parsing whole training data in Kaggle with minimal CPU and RAM usage. I performed some EDA tasks however I believe many other EDA tasks can also be performed. Your reviews and comments are highly appreciated. Here is the link of the notebook\nhttps://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data",
    "2809327": "Thank you for notebook link",
    "2812074": "Hello, I am Rajiv from Mumbai. I am not able to open the xl input as it is very large. Can anyone suggest solution?",
    "2812621": "profrajiviyer, If you have large data files like in this competition and you have limited RAM, for example, even if you have 30GM RAM (I have test on kaggle as well), you can not open so big file at once. The trick is to read/ process this data in batches or chunks as I have explained the notebook above. For your convenience, the link is being reproduced. https://www.kaggle.com/code/tariqcp/simple-yet-powerful-approach-to-read-whole-data"
  },
  "source": "meta"
}