\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2875376%2Fc000f8d6bb1243da7eb235de17cbdbd9%2FScreen%20Shot%202024-10-15%20at%209.03.16%20AM.png?generation=1728963235861626&alt=media\")
Given, how large the dataset is, it's obvious to encounter OOM error when trying to load the entire dataset in one go. If you're familiar with LeashBio competition that was hosted few months back, I opened a discussion thread on using dask to deal with larger than life datasets. And you can see the code implementation in the same discussion thread:
\nhttps://www.kaggle.com/competitions/leash-BELKA/discussion/496954
For this competition, it took me only 5.31 sec to load the entire dataset.
\n
Given, how large the dataset is, it's obvious to encounter OOM error when trying to load the entire dataset in one go. If you're familiar with LeashBio competition that was hosted few months back, I opened a discussion thread on using dask to deal with larger than life datasets. And you can see the code implementation in the same discussion thread:
\nhttps://www.kaggle.com/competitions/leash-BELKA/discussion/496954
For this competition, it took me only 5.31 sec to load the entire dataset.
\n
It took 5.31s to load the first few rows. Dask is similar to spark in the sense that it does lazy compute - it doesn't load anything into memory until it's needed.
", "rawMarkdown": "It took 5.31s to load the first few rows. Dask is similar to spark in the sense that it does lazy compute - it doesn't load anything into memory until it's needed.", "votes": null }, { "id": "3018727", "postDate": "10/16/2024 02:45:19", "content": "Great work!
", "rawMarkdown": "Great work!", "votes": null }, { "id": "3020354", "postDate": "10/17/2024 12:53:45", "content": "It sounds better to use polars its written based on Rust.
", "rawMarkdown": "It sounds better to use polars its written based on Rust.", "votes": null }, { "id": "3073012", "postDate": "12/15/2024 23:11:20", "content": "however, when you load it into the memory, the memory explodes again.
", "rawMarkdown": "however, when you load it into the memory, the memory explodes again.", "votes": null } ], "comments": [ { "id": 3018328, "author_name": "madcarrot", "author_url": "", "post_date": "10/15/2024 17:14:21", "content": "It took 5.31s to load the first few rows. Dask is similar to spark in the sense that it does lazy compute - it doesn't load anything into memory until it's needed.
", "votes": null, "replies": [] }, { "id": 3018727, "author_name": "zeminglin", "author_url": "", "post_date": "10/16/2024 02:45:19", "content": "Great work!
", "votes": null, "replies": [] }, { "id": 3020354, "author_name": "farhankardan", "author_url": "", "post_date": "10/17/2024 12:53:45", "content": "It sounds better to use polars its written based on Rust.
", "votes": null, "replies": [] }, { "id": 3073012, "author_name": "billyrobert", "author_url": "", "post_date": "12/15/2024 23:11:20", "content": "however, when you load it into the memory, the memory explodes again.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3017608": "Given, how large the dataset is, it's obvious to encounter OOM error when trying to load the entire dataset in one go. If you're familiar with LeashBio competition that was hosted few months back, I opened a discussion thread on using dask to deal with larger than life datasets. And you can see the code implementation in the same discussion thread: \nhttps://www.kaggle.com/competitions/leash-BELKA/discussion/496954\n\nFor this competition, it took me only 5.31 sec to load the entire dataset.\n", "3018328": "It took 5.31s to load the first few rows. Dask is similar to spark in the sense that it does lazy compute - it doesn't load anything into memory until it's needed.", "3018727": "Great work!", "3020354": "It sounds better to use polars its written based on Rust.", "3073012": "however, when you load it into the memory, the memory explodes again." }, "source": "meta" }