{ "id": 329808, "title": "Too big to fit in memory", "url": "/competitions/amex-default-prediction/discussion/329808", "author_name": "", "post_date": "2022-06-08T20:28:20.051997300Z", "votes": 3, "comment_count": 3, "views": 0, "content": "

I'm new to Kaggle.
\nMy question is should I work on my machine (or in Colab) or do I have to work on the Kaggle notebook? The available memory space in the Kaggle notebook is not sufficient for loading the dataset.

", "messages": [ { "id": "1815244", "postDate": "06/08/2022 20:28:20", "content": "

I'm new to Kaggle.
\nMy question is should I work on my machine (or in Colab) or do I have to work on the Kaggle notebook? The available memory space in the Kaggle notebook is not sufficient for loading the dataset.

", "rawMarkdown": "I'm new to Kaggle.\nMy question is should I work on my machine (or in Colab) or do I have to work on the Kaggle notebook? The available memory space in the Kaggle notebook is not sufficient for loading the dataset.", "votes": null }, { "id": "1815254", "postDate": "06/08/2022 21:00:16", "content": "

You can work on any environment you want, no restriction.

\n

Kaggle notebooks have the advantage that they can easily be shared and forked by other kagglers. But, as you suggest, they have limited resources.

", "rawMarkdown": "You can work on any environment you want, no restriction.\n\nKaggle notebooks have the advantage that they can easily be shared and forked by other kagglers. But, as you suggest, they have limited resources.", "votes": null }, { "id": "1815299", "postDate": "06/08/2022 22:43:42", "content": "

You can use kaggle, colab or local notebook with memory optimize techniques.
\nBy the wya, I am using kaggle and colab.

\n

Some posts that may help you.

\n

How To Reduce Data Size - Chris Deotte

\n

Handling large datasets with Dask - pestipeti

", "rawMarkdown": "You can use kaggle, colab or local notebook with memory optimize techniques.\nBy the wya, I am using kaggle and colab.\n\nSome posts that may help you.\n\n[How To Reduce Data Size](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328054) - Chris Deotte\n\n[Handling large datasets with Dask](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327110) - pestipeti", "votes": null }, { "id": "1816063", "postDate": "06/09/2022 19:11:28", "content": "

Feel free to use the datasets below:

\n

https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format
\nhttps://www.kaggle.com/datasets/ruchi798/parquet-files-amexdefault-prediction

\n

They are cleaned and shrunk without losing much information.

\n

Dont forget to thank the contributors. @raddar @ruchi798

", "rawMarkdown": "Feel free to use the datasets below:\n\nhttps://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format\nhttps://www.kaggle.com/datasets/ruchi798/parquet-files-amexdefault-prediction\n\nThey are cleaned and shrunk without losing much information.\n\nDont forget to thank the contributors. @raddar @ruchi798", "votes": null } ], "comments": [ { "id": 1815254, "author_name": "crayola", "author_url": "", "post_date": "06/08/2022 21:00:16", "content": "

You can work on any environment you want, no restriction.

\n

Kaggle notebooks have the advantage that they can easily be shared and forked by other kagglers. But, as you suggest, they have limited resources.

", "votes": null, "replies": [] }, { "id": 1815299, "author_name": "jingwora1", "author_url": "", "post_date": "06/08/2022 22:43:42", "content": "

You can use kaggle, colab or local notebook with memory optimize techniques.
\nBy the wya, I am using kaggle and colab.

\n

Some posts that may help you.

\n

How To Reduce Data Size - Chris Deotte

\n

Handling large datasets with Dask - pestipeti

", "votes": null, "replies": [] }, { "id": 1816063, "author_name": "karandora", "author_url": "", "post_date": "06/09/2022 19:11:28", "content": "

Feel free to use the datasets below:

\n

https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format
\nhttps://www.kaggle.com/datasets/ruchi798/parquet-files-amexdefault-prediction

\n

They are cleaned and shrunk without losing much information.

\n

Dont forget to thank the contributors. @raddar @ruchi798

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1815244": "I'm new to Kaggle.\nMy question is should I work on my machine (or in Colab) or do I have to work on the Kaggle notebook? The available memory space in the Kaggle notebook is not sufficient for loading the dataset.", "1815254": "You can work on any environment you want, no restriction.\n\nKaggle notebooks have the advantage that they can easily be shared and forked by other kagglers. But, as you suggest, they have limited resources.", "1815299": "You can use kaggle, colab or local notebook with memory optimize techniques.\nBy the wya, I am using kaggle and colab.\n\nSome posts that may help you.\n\n[How To Reduce Data Size](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328054) - Chris Deotte\n\n[Handling large datasets with Dask](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327110) - pestipeti", "1816063": "Feel free to use the datasets below:\n\nhttps://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format\nhttps://www.kaggle.com/datasets/ruchi798/parquet-files-amexdefault-prediction\n\nThey are cleaned and shrunk without losing much information.\n\nDont forget to thank the contributors. @raddar @ruchi798" }, "source": "meta" }