{ "id": 327104, "title": "The tabular data is soooooo huge that reading is a problem! Sad but exciting.", "url": "/competitions/amex-default-prediction/discussion/327104", "author_name": "", "post_date": "2022-05-25T15:47:29.619577800Z", "votes": 6, "comment_count": 5, "views": 0, "content": "

Any good suggestions for reading this data? :)

", "messages": [ { "id": "1801300", "postDate": "05/25/2022 15:47:29", "content": "

Any good suggestions for reading this data? :)

", "rawMarkdown": "Any good suggestions for reading this data? :)", "votes": null }, { "id": "1801326", "postDate": "05/25/2022 16:03:47", "content": "

A number of years ago, back in the days when memory was often a limiter, there were some cool techniques to read and process data line-by-line, e.g., FTRL.

", "rawMarkdown": "A number of years ago, back in the days when memory was often a limiter, there were some cool techniques to read and process data line-by-line, e.g., [FTRL](https://www.kaggle.com/competitions/avazu-ctr-prediction/discussion/10927).", "votes": null }, { "id": "1801328", "postDate": "05/25/2022 16:04:06", "content": "

How do you plan to import this data to any other ecosystem? Is there a ZIP link?

", "rawMarkdown": "How do you plan to import this data to any other ecosystem? Is there a ZIP link?", "votes": null }, { "id": "1801334", "postDate": "05/25/2022 16:10:47", "content": "

I recommend @rohanrao 's post
\nhttps://www.kaggle.com/competitions/riiid-test-answer-prediction/discussion/191751

\n

found it very useful when I attempted Riiid Answer Correctness Prediction last year

", "rawMarkdown": "I recommend @rohanrao 's post \nhttps://www.kaggle.com/competitions/riiid-test-answer-prediction/discussion/191751\n\nfound it very useful when I attempted Riiid Answer Correctness Prediction last year", "votes": null }, { "id": "1801536", "postDate": "05/25/2022 20:57:59", "content": "

You can explore the data using the first 1 million lines, for example (hoping your conclusions don't get biased) and then when you train and test your model by reading the CSVs in chunks

", "rawMarkdown": "You can explore the data using the first 1 million lines, for example (hoping your conclusions don't get biased) and then when you train and test your model by reading the CSVs in [chunks](https://stackoverflow.com/questions/25962114/how-do-i-read-a-large-csv-file-with-pandas)", "votes": null }, { "id": "1801660", "postDate": "05/26/2022 02:38:36", "content": "

Thank u guys. These are very helpful to me -v-

", "rawMarkdown": "Thank u guys. These are very helpful to me -v-", "votes": null } ], "comments": [ { "id": 1801326, "author_name": "inversion", "author_url": "", "post_date": "05/25/2022 16:03:47", "content": "

A number of years ago, back in the days when memory was often a limiter, there were some cool techniques to read and process data line-by-line, e.g., FTRL.

", "votes": null, "replies": [ { "id": 1801334, "author_name": "kmldas", "author_url": "", "post_date": "05/25/2022 16:10:47", "content": "

I recommend @rohanrao 's post
\nhttps://www.kaggle.com/competitions/riiid-test-answer-prediction/discussion/191751

\n

found it very useful when I attempted Riiid Answer Correctness Prediction last year

", "votes": null, "replies": [] }, { "id": 1801660, "author_name": "carlmaxace", "author_url": "", "post_date": "05/26/2022 02:38:36", "content": "

Thank u guys. These are very helpful to me -v-

", "votes": null, "replies": [] } ] }, { "id": 1801328, "author_name": "pratyush", "author_url": "", "post_date": "05/25/2022 16:04:06", "content": "

How do you plan to import this data to any other ecosystem? Is there a ZIP link?

", "votes": null, "replies": [] }, { "id": 1801536, "author_name": "bgmello", "author_url": "", "post_date": "05/25/2022 20:57:59", "content": "

You can explore the data using the first 1 million lines, for example (hoping your conclusions don't get biased) and then when you train and test your model by reading the CSVs in chunks

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1801300": "Any good suggestions for reading this data? :)", "1801326": "A number of years ago, back in the days when memory was often a limiter, there were some cool techniques to read and process data line-by-line, e.g., [FTRL](https://www.kaggle.com/competitions/avazu-ctr-prediction/discussion/10927).", "1801328": "How do you plan to import this data to any other ecosystem? Is there a ZIP link?", "1801334": "I recommend @rohanrao 's post \nhttps://www.kaggle.com/competitions/riiid-test-answer-prediction/discussion/191751\n\nfound it very useful when I attempted Riiid Answer Correctness Prediction last year", "1801536": "You can explore the data using the first 1 million lines, for example (hoping your conclusions don't get biased) and then when you train and test your model by reading the CSVs in [chunks](https://stackoverflow.com/questions/25962114/how-do-i-read-a-large-csv-file-with-pandas)", "1801660": "Thank u guys. These are very helpful to me -v-" }, "source": "meta" }