{ "id": 327195, "title": "Reading & Working with Large Dataset", "url": "/competitions/amex-default-prediction/discussion/327195", "author_name": "Dev Khant", "post_date": "2022-05-26T04:41:39.502000", "votes": 30, "comment_count": 0, "views": 0, "content": "

As this competition has large dataset simply using pd.read_csv will result into in an out-of-memory error on Kaggle Notebooks. So here we will see different methods and libraries to work with huge data.

\n

Libraries

\n

There are few libraries which can be used to read Large Dataset. Here we will see 3 such libraries.

\n

1. Dask

\n

Dask is an open-sourced Python library that provides multi-core and distributed parallel execution of larger-than-memory datasets. It provides parallelized NumPy array and Pandas DataFrame objects. And it also provides API similar to Pandas and Numpy.

\n

import dask.dataframe as dd
\ndf_dask = dd.read_csv(\"your_dataset.csv\")

\n

Documentation : https://docs.dask.org/en/latest/

\n

2. Modin

\n

Modin uses Ray or Dask to provide an effortless way to speed up your pandas notebooks and utilizes all the cores available in the system, only requiring users to change a single line of code in their notebooks.

\n

import modin.pandas as md
\nmodin_df = pd.read_csv(\"your_dataset.csv\")

\n

Documentation : https://modin.readthedocs.io/en/stable/

\n

3. Rapids

\n

The RAPIDS data science framework is a collection of libraries for executing end-to-end data science pipelines completely in the GPU. It includes libraries like CuML, CuDF, Xgboost etc…

\n

import cudf
\ndf = cudf.read_csv('your_dataset.csv')

\n

Documentation : https://rapids.ai/start.html

\n

Methods

\n

We can convert our csv to other formats for faster loading.

\n

1. Parquet

\n

In the Hadoop ecosystem, parquet was popularly used as the primary file format for tabular datasets and is now extensively used with Spark. It has become more available and efficient over the years and is also supported by pandas.

\n

In order to convert csv file to parquet file simply do
\ndf.to_parquet('output.parquet')

\n

To read parquet file in pandas
\ndata = pd.read_parquet(\"your_dataset.parquet\")

\n

Documentation : https://parquet.apache.org/

\n

2. Pickle

\n

Python objects can be stored in the form of pickle files and pandas has inbuilt functions to read and write dataframes as pickle objects.

\n

In order to convert csv file to pickle file simply do
\ndf.to_parquet('output.pkl')

\n

To read pickle file in pandas
\ndata = pd.read_parquet(\"your_dataset.pkl\")

\n

I hope you learnt something new. So keep Learning & keep Kaggling :)

", "messages": [ { "id": 1801715, "postDate": "2022-05-26T04:41:39.503Z", "content": "

As this competition has large dataset simply using pd.read_csv will result into in an out-of-memory error on Kaggle Notebooks. So here we will see different methods and libraries to work with huge data.

\n

Libraries

\n

There are few libraries which can be used to read Large Dataset. Here we will see 3 such libraries.

\n

1. Dask

\n

Dask is an open-sourced Python library that provides multi-core and distributed parallel execution of larger-than-memory datasets. It provides parallelized NumPy array and Pandas DataFrame objects. And it also provides API similar to Pandas and Numpy.

\n

import dask.dataframe as dd
\ndf_dask = dd.read_csv(\"your_dataset.csv\")

\n

Documentation : https://docs.dask.org/en/latest/

\n

2. Modin

\n

Modin uses Ray or Dask to provide an effortless way to speed up your pandas notebooks and utilizes all the cores available in the system, only requiring users to change a single line of code in their notebooks.

\n

import modin.pandas as md
\nmodin_df = pd.read_csv(\"your_dataset.csv\")

\n

Documentation : https://modin.readthedocs.io/en/stable/

\n

3. Rapids

\n

The RAPIDS data science framework is a collection of libraries for executing end-to-end data science pipelines completely in the GPU. It includes libraries like CuML, CuDF, Xgboost etc…

\n

import cudf
\ndf = cudf.read_csv('your_dataset.csv')

\n

Documentation : https://rapids.ai/start.html

\n

Methods

\n

We can convert our csv to other formats for faster loading.

\n

1. Parquet

\n

In the Hadoop ecosystem, parquet was popularly used as the primary file format for tabular datasets and is now extensively used with Spark. It has become more available and efficient over the years and is also supported by pandas.

\n

In order to convert csv file to parquet file simply do
\ndf.to_parquet('output.parquet')

\n

To read parquet file in pandas
\ndata = pd.read_parquet(\"your_dataset.parquet\")

\n

Documentation : https://parquet.apache.org/

\n

2. Pickle

\n

Python objects can be stored in the form of pickle files and pandas has inbuilt functions to read and write dataframes as pickle objects.

\n

In order to convert csv file to pickle file simply do
\ndf.to_parquet('output.pkl')

\n

To read pickle file in pandas
\ndata = pd.read_parquet(\"your_dataset.pkl\")

\n

I hope you learnt something new. So keep Learning & keep Kaggling :)

", "rawMarkdown": "As this competition has large dataset simply using ***pd.read_csv*** will result into in an out-of-memory error on Kaggle Notebooks. So here we will see different methods and libraries to work with huge data.\n\n## **Libraries**\nThere are few libraries which can be used to read Large Dataset. Here we will see 3 such libraries.\n\n### 1. Dask\nDask is an open-sourced Python library that provides multi-core and distributed parallel execution of larger-than-memory datasets. It provides parallelized NumPy array and Pandas DataFrame objects. And it also provides API similar to Pandas and Numpy.\n\n`import dask.dataframe as dd`\n`df_dask = dd.read_csv(\"your_dataset.csv\")`\n\nDocumentation : https://docs.dask.org/en/latest/\n\n### 2. Modin\nModin uses Ray or Dask to provide an effortless way to speed up your pandas notebooks and utilizes all the cores available in the system, only requiring users to change a single line of code in their notebooks.\n\n`import modin.pandas as md`\n`modin_df = pd.read_csv(\"your_dataset.csv\")`\n\nDocumentation : https://modin.readthedocs.io/en/stable/\n\n### 3. Rapids\nThe RAPIDS data science framework is a collection of libraries for executing end-to-end data science pipelines completely in the GPU. It includes libraries like CuML, CuDF, Xgboost etc...\n\n`import cudf`\n`df = cudf.read_csv('your_dataset.csv')`\n\nDocumentation : https://rapids.ai/start.html\n\n## **Methods**\nWe can convert our csv to other formats for faster loading.\n\n### 1. Parquet\nIn the Hadoop ecosystem, parquet was popularly used as the primary file format for tabular datasets and is now extensively used with Spark. It has become more available and efficient over the years and is also supported by pandas.\n\nIn order to convert csv file to parquet file simply do\n`df.to_parquet('output.parquet')`\n\nTo read parquet file in pandas\n`data = pd.read_parquet(\"your_dataset.parquet\")`\n\nDocumentation : https://parquet.apache.org/\n\n### 2. Pickle\nPython objects can be stored in the form of pickle files and pandas has inbuilt functions to read and write dataframes as pickle objects.\n\nIn order to convert csv file to pickle file simply do\n`df.to_parquet('output.pkl')`\n\nTo read pickle file in pandas\n`data = pd.read_parquet(\"your_dataset.pkl\")`\n\n\n\n**I hope you learnt something new. So keep Learning & keep Kaggling :)**", "votes": 30 } ], "comments": [], "raw_markdown_by_id": { "1801715": "As this competition has large dataset simply using ***pd.read_csv*** will result into in an out-of-memory error on Kaggle Notebooks. So here we will see different methods and libraries to work with huge data.\n\n## **Libraries**\nThere are few libraries which can be used to read Large Dataset. Here we will see 3 such libraries.\n\n### 1. Dask\nDask is an open-sourced Python library that provides multi-core and distributed parallel execution of larger-than-memory datasets. It provides parallelized NumPy array and Pandas DataFrame objects. And it also provides API similar to Pandas and Numpy.\n\n`import dask.dataframe as dd`\n`df_dask = dd.read_csv(\"your_dataset.csv\")`\n\nDocumentation : https://docs.dask.org/en/latest/\n\n### 2. Modin\nModin uses Ray or Dask to provide an effortless way to speed up your pandas notebooks and utilizes all the cores available in the system, only requiring users to change a single line of code in their notebooks.\n\n`import modin.pandas as md`\n`modin_df = pd.read_csv(\"your_dataset.csv\")`\n\nDocumentation : https://modin.readthedocs.io/en/stable/\n\n### 3. Rapids\nThe RAPIDS data science framework is a collection of libraries for executing end-to-end data science pipelines completely in the GPU. It includes libraries like CuML, CuDF, Xgboost etc...\n\n`import cudf`\n`df = cudf.read_csv('your_dataset.csv')`\n\nDocumentation : https://rapids.ai/start.html\n\n## **Methods**\nWe can convert our csv to other formats for faster loading.\n\n### 1. Parquet\nIn the Hadoop ecosystem, parquet was popularly used as the primary file format for tabular datasets and is now extensively used with Spark. It has become more available and efficient over the years and is also supported by pandas.\n\nIn order to convert csv file to parquet file simply do\n`df.to_parquet('output.parquet')`\n\nTo read parquet file in pandas\n`data = pd.read_parquet(\"your_dataset.parquet\")`\n\nDocumentation : https://parquet.apache.org/\n\n### 2. Pickle\nPython objects can be stored in the form of pickle files and pandas has inbuilt functions to read and write dataframes as pickle objects.\n\nIn order to convert csv file to pickle file simply do\n`df.to_parquet('output.pkl')`\n\nTo read pickle file in pandas\n`data = pd.read_parquet(\"your_dataset.pkl\")`\n\n\n\n**I hope you learnt something new. So keep Learning & keep Kaggling :)**" } }