A few days ago I was facing issues working with huge datasets. Sometimes the notebooks stopped unexpectedly, sometimes it showed memory outage, and so on. I tried to work with Google colab as well. But there also I faced similar issues. So, I was searching for solutions regarding this matter.
\n\n\nHere I have compiled all the discussions, comments, and notebooks on working with huge/large datasets.
\n
\n Here you will find all the tips regarding,\n
\n- How to download large datasets,
\n- How to work with google colab for large datasets,
\n- How to do EDA,model building and training on those.
\n
I will request everyone to go through the comments of all the discussions.\nDifferent Kaggle users have shared their experiences and tips regarding the matter in the comments.
\n\n\nDon't forget to upvote the real discussions. They have put a lot of effort into it.
\n
\n", "messages": [ { "id": "1640346", "postDate": "01/06/2022 12:04:48", "content": "[If you want to add more tips regarding this issue, then you can add your comments here. I will attach those in the main post]
\n
A few days ago I was facing issues working with huge datasets. Sometimes the notebooks stopped unexpectedly, sometimes it showed memory outage, and so on. I tried to work with Google colab as well. But there also I faced similar issues. So, I was searching for solutions regarding this matter.
\n\n\nHere I have compiled all the discussions, comments, and notebooks on working with huge/large datasets.
\n
\n Here you will find all the tips regarding,\n
\n- How to download large datasets,
\n- How to work with google colab for large datasets,
\n- How to do EDA,model building and training on those.
\n
I will request everyone to go through the comments of all the discussions.\nDifferent Kaggle users have shared their experiences and tips regarding the matter in the comments.
\n\n\nDon't forget to upvote the real discussions. They have put a lot of effort into it.
\n
\n", "rawMarkdown": "A few days ago I was facing issues **working with huge datasets**. Sometimes the notebooks stopped unexpectedly, sometimes it showed memory outage, and so on. I tried to work with Google colab as well. But there also I faced similar issues. So, I was searching for solutions regarding this matter.\n> Here I have compiled all the discussions, comments, and notebooks on working with huge/large datasets.\nHere you will find all the tips regarding,\n- How to download large datasets,\n- How to work with google colab for large datasets,\n- How to do EDA,model building and training on those.\n\n\n**I will request everyone to go through the comments of all the discussions.\nDifferent Kaggle users have shared their experiences and tips regarding the matter in the comments.**\n\n\n\n> ### Don't forget to upvote the real discussions. They have put a lot of effort into it.\n\n- [Issues with loading large datasets in pandas](https://www.kaggle.com/general/234366)\n- [Tips & Tricks for Working with Large Datasets](https://www.kaggle.com/frankherfert/tips-tricks-for-working-with-large-datasets/notebook)\n- [Large CSV datasets with pandas (use less memory!)](https://www.kaggle.com/marcmuc/large-csv-datasets-with-pandas-use-less-memory/notebook)\n- [Upload huge data on google storage account](https://www.kaggle.com/general/158217)\n- [Handling huge datasets(example, >500GB or >1TB)](https://www.kaggle.com/questions-and-answers/63798)\n- [modin.pandas for faster processing dataframes than pandas](https://www.kaggle.com/getting-started/146047)\n- [Ideas to handle this huge dataset](https://www.kaggle.com/c/data-science-bowl-2017/discussion/27935)\n- [Loading huge datasets using pandas](https://www.kaggle.com/questions-and-answers/63209)\n- [Tutorial on reading large datasets](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets)\n- [Now, Load huge datasets within no time using Vaex!! β‘](https://www.kaggle.com/getting-started/156662)\n- [How To Read And Process Huge Datasets in Seconds Using Vaex Library](https://www.kaggle.com/getting-started/156308)\n- [Large DataSet - can get started in Notebooks](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182801)\n- [Dealing with large dataset](https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/41574)\n- [How to do hyperparameter tuning on large dataset?](https://www.kaggle.com/questions-and-answers/235402)\n- [βοΈπ€π How to tackle limited memory issue for large tabular dataset ?](https://www.kaggle.com/c/tabular-playground-series-oct-2021/discussion/275892)\n- [How to Work w. Million-row Datasets Like a Pro](https://www.kaggle.com/bextuychiev/how-to-work-w-million-row-datasets-like-a-pro)\n- [How to do exploratory analysis of large dataset?](https://www.kaggle.com/questions-and-answers/219507)\n- [Dataset too large for Google Colab](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/227062)\n- [How can I downlaod this large Datasets?](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/186652)'\n- [Run out of RAM - Large Dataset](https://www.kaggle.com/c/predict-volcanic-eruptions-ingv-oe/discussion/204972)\n- [Modin - for Large Dataset](https://www.kaggle.com/getting-started/157419)\n- [How to use a large a very dataset (csv) for training?](https://www.kaggle.com/questions-and-answers/150449)\n- [Extracting large dataset and working with that](https://www.kaggle.com/questions-and-answers/190813)\n- [How to handle Large datasets](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/221421)\n- [What is the way to upload large dataset more than 4 gb while working on a local machine.](https://www.kaggle.com/questions-and-answers/47505)\n- [How to work with large datasets in a notebook?](https://www.kaggle.com/questions-and-answers/297384)\n> \n**[If you want to add more tips regarding this issue, then you can add your comments here. I will attach those in the main post]**", "votes": null }, { "id": "1640354", "postDate": "01/06/2022 12:14:06", "content": "[If you want to add more tips regarding this issue, then you can add your comments here. I will attach those in the main post]
\n
Eager Loading >> Lazy Loading but it is more complicated. You can eg store picture data in chunks in mongodb :)
", "rawMarkdown": "Eager Loading >> Lazy Loading but it is more complicated. You can eg store picture data in chunks in mongodb :)", "votes": null }, { "id": "1641139", "postDate": "01/07/2022 06:12:11", "content": "Hi @towhidultonmoy, this is so helpful. Thanks for sharing. Upvoted π
", "rawMarkdown": "Hi @towhidultonmoy, this is so helpful. Thanks for sharing. Upvoted π", "votes": null }, { "id": "1641274", "postDate": "01/07/2022 08:57:36", "content": "Glad you find it useful!
", "rawMarkdown": "Glad you find it useful!", "votes": null } ], "comments": [ { "id": 1640354, "author_name": "marcinstasko", "author_url": "", "post_date": "01/06/2022 12:14:06", "content": "Eager Loading >> Lazy Loading but it is more complicated. You can eg store picture data in chunks in mongodb :)
", "votes": null, "replies": [] }, { "id": 1641139, "author_name": "balabaskar", "author_url": "", "post_date": "01/07/2022 06:12:11", "content": "Hi @towhidultonmoy, this is so helpful. Thanks for sharing. Upvoted π
", "votes": null, "replies": [ { "id": 1641274, "author_name": "towhidultonmoy", "author_url": "", "post_date": "01/07/2022 08:57:36", "content": "Glad you find it useful!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1640346": "A few days ago I was facing issues **working with huge datasets**. Sometimes the notebooks stopped unexpectedly, sometimes it showed memory outage, and so on. I tried to work with Google colab as well. But there also I faced similar issues. So, I was searching for solutions regarding this matter.\n> Here I have compiled all the discussions, comments, and notebooks on working with huge/large datasets.\nHere you will find all the tips regarding,\n- How to download large datasets,\n- How to work with google colab for large datasets,\n- How to do EDA,model building and training on those.\n\n\n**I will request everyone to go through the comments of all the discussions.\nDifferent Kaggle users have shared their experiences and tips regarding the matter in the comments.**\n\n\n\n> ### Don't forget to upvote the real discussions. They have put a lot of effort into it.\n\n- [Issues with loading large datasets in pandas](https://www.kaggle.com/general/234366)\n- [Tips & Tricks for Working with Large Datasets](https://www.kaggle.com/frankherfert/tips-tricks-for-working-with-large-datasets/notebook)\n- [Large CSV datasets with pandas (use less memory!)](https://www.kaggle.com/marcmuc/large-csv-datasets-with-pandas-use-less-memory/notebook)\n- [Upload huge data on google storage account](https://www.kaggle.com/general/158217)\n- [Handling huge datasets(example, >500GB or >1TB)](https://www.kaggle.com/questions-and-answers/63798)\n- [modin.pandas for faster processing dataframes than pandas](https://www.kaggle.com/getting-started/146047)\n- [Ideas to handle this huge dataset](https://www.kaggle.com/c/data-science-bowl-2017/discussion/27935)\n- [Loading huge datasets using pandas](https://www.kaggle.com/questions-and-answers/63209)\n- [Tutorial on reading large datasets](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets)\n- [Now, Load huge datasets within no time using Vaex!! β‘](https://www.kaggle.com/getting-started/156662)\n- [How To Read And Process Huge Datasets in Seconds Using Vaex Library](https://www.kaggle.com/getting-started/156308)\n- [Large DataSet - can get started in Notebooks](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/182801)\n- [Dealing with large dataset](https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/41574)\n- [How to do hyperparameter tuning on large dataset?](https://www.kaggle.com/questions-and-answers/235402)\n- [βοΈπ€π How to tackle limited memory issue for large tabular dataset ?](https://www.kaggle.com/c/tabular-playground-series-oct-2021/discussion/275892)\n- [How to Work w. Million-row Datasets Like a Pro](https://www.kaggle.com/bextuychiev/how-to-work-w-million-row-datasets-like-a-pro)\n- [How to do exploratory analysis of large dataset?](https://www.kaggle.com/questions-and-answers/219507)\n- [Dataset too large for Google Colab](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/227062)\n- [How can I downlaod this large Datasets?](https://www.kaggle.com/c/rsna-str-pulmonary-embolism-detection/discussion/186652)'\n- [Run out of RAM - Large Dataset](https://www.kaggle.com/c/predict-volcanic-eruptions-ingv-oe/discussion/204972)\n- [Modin - for Large Dataset](https://www.kaggle.com/getting-started/157419)\n- [How to use a large a very dataset (csv) for training?](https://www.kaggle.com/questions-and-answers/150449)\n- [Extracting large dataset and working with that](https://www.kaggle.com/questions-and-answers/190813)\n- [How to handle Large datasets](https://www.kaggle.com/c/hpa-single-cell-image-classification/discussion/221421)\n- [What is the way to upload large dataset more than 4 gb while working on a local machine.](https://www.kaggle.com/questions-and-answers/47505)\n- [How to work with large datasets in a notebook?](https://www.kaggle.com/questions-and-answers/297384)\n> \n**[If you want to add more tips regarding this issue, then you can add your comments here. I will attach those in the main post]**", "1640354": "Eager Loading >> Lazy Loading but it is more complicated. You can eg store picture data in chunks in mongodb :)", "1641139": "Hi @towhidultonmoy, this is so helpful. Thanks for sharing. Upvoted π", "1641274": "Glad you find it useful!" }, "source": "meta" }