{ "id": 335054, "title": "Large Datasets from Kaggle to Google CoLab", "url": "/competitions/amex-default-prediction/discussion/335054", "author_name": "", "post_date": "2022-07-04T13:38:52.055221500Z", "votes": 21, "comment_count": 16, "views": 0, "content": "

If you are running out of storage and memory in Kaggle notebooks and want to use Google CoLab then below steps will help you and save a lot of time required to move large datasets. 🤓

\n

I am using Google CoLab paid membership. So your memory and storage requirements may require it as well. So check your requirements properly beforehand. 👀

\n

1. Get Kaggle API Key

\n

Go to Kaggle profile page
\nClick Accounts tab
\nUnder API section, click \"Create New API Token\". This will download your credentials in a JSON file.

\n

2. Create New Notebook in Google CoLab

\n

3. Download Kaggle dataset into Google CoLab Notebook using below code.

\n

!pip install -U -q kaggle
\n!mkdir -p ~/.kaggle
\n!echo '{\"username\":\"YOUR_USER_ID\",\"key\":\"YOUR_KAGGLE_KEY\"}' > ~/.kaggle/kaggle.json
\n!chmod 600 ~/.kaggle/kaggle.json
\n!kaggle competitions download -c amex-default-prediction

\n

Kaggle dataset will be copied into Google CoLab in compressed format.

\n

4. Uncompress Dataset

\n

!unzip '/content/amex-default-prediction.zip' -d '/content'

\n

Remove compressed dataset to free storage in Google CoLab.

", "messages": [ { "id": "1843020", "postDate": "07/04/2022 13:38:52", "content": "

If you are running out of storage and memory in Kaggle notebooks and want to use Google CoLab then below steps will help you and save a lot of time required to move large datasets. 🤓

\n

I am using Google CoLab paid membership. So your memory and storage requirements may require it as well. So check your requirements properly beforehand. 👀

\n

1. Get Kaggle API Key

\n

Go to Kaggle profile page
\nClick Accounts tab
\nUnder API section, click \"Create New API Token\". This will download your credentials in a JSON file.

\n

2. Create New Notebook in Google CoLab

\n

3. Download Kaggle dataset into Google CoLab Notebook using below code.

\n

!pip install -U -q kaggle
\n!mkdir -p ~/.kaggle
\n!echo '{\"username\":\"YOUR_USER_ID\",\"key\":\"YOUR_KAGGLE_KEY\"}' > ~/.kaggle/kaggle.json
\n!chmod 600 ~/.kaggle/kaggle.json
\n!kaggle competitions download -c amex-default-prediction

\n

Kaggle dataset will be copied into Google CoLab in compressed format.

\n

4. Uncompress Dataset

\n

!unzip '/content/amex-default-prediction.zip' -d '/content'

\n

Remove compressed dataset to free storage in Google CoLab.

", "rawMarkdown": "If you are running out of storage and memory in Kaggle notebooks and want to use Google CoLab then below steps will help you and save a lot of time required to move large datasets. 🤓\n\nI am using Google CoLab paid membership. So your memory and storage requirements may require it as well. So check your requirements properly beforehand. 👀\n\n**1. Get Kaggle API Key**\n \nGo to Kaggle profile page\nClick Accounts tab\nUnder API section, click \"Create New API Token\". This will download your credentials in a JSON file.\n \n\n**2. Create New Notebook in Google CoLab**\n \n\n**3. Download Kaggle dataset into Google CoLab Notebook using below code.**\n \n\n`!pip install -U -q kaggle `\n`!mkdir -p ~/.kaggle `\n`!echo '{\"username\":\"YOUR_USER_ID\",\"key\":\"YOUR_KAGGLE_KEY\"}' > ~/.kaggle/kaggle.json `\n`!chmod 600 ~/.kaggle/kaggle.json `\n`!kaggle competitions download -c amex-default-prediction`\n \n\nKaggle dataset will be copied into Google CoLab in compressed format.\n \n\n**4. Uncompress Dataset** \n \n\n`!unzip '/content/amex-default-prediction.zip' -d '/content'`\n \n\nRemove compressed dataset to free storage in Google CoLab.", "votes": null }, { "id": "1843209", "postDate": "07/04/2022 16:17:09", "content": "

This is quite useful, colab is a pretty economical yet powerful supplement to the usual Kaggle kernels

", "rawMarkdown": "This is quite useful, colab is a pretty economical yet powerful supplement to the usual Kaggle kernels", "votes": null }, { "id": "1843334", "postDate": "07/04/2022 17:57:09", "content": "

Hi Muhammad, another option is paperspace gradient, which even in its free version offers 30 GB of ram.

", "rawMarkdown": "Hi Muhammad, another option is paperspace gradient, which even in its free version offers 30 GB of ram.", "votes": null }, { "id": "1843397", "postDate": "07/04/2022 19:10:07", "content": "

Thanks for sharing very useful for using in case of large data

", "rawMarkdown": "Thanks for sharing very useful for using in case of large data", "votes": null }, { "id": "1843436", "postDate": "07/04/2022 20:07:44", "content": "

@maxdiazbattan lovely suggestion 💥☀️

", "rawMarkdown": "maxdiazbattan lovely suggestion 💥☀️", "votes": null }, { "id": "1843810", "postDate": "07/05/2022 06:24:38", "content": "

Cool thing, i tried this with Colab Pro+, got 51gb RAM and still have trouble running the whole thing. I guess i have to really do feature importance ahead

", "rawMarkdown": "Cool thing, i tried this with Colab Pro+, got 51gb RAM and still have trouble running the whole thing. I guess i have to really do feature importance ahead", "votes": null }, { "id": "1843823", "postDate": "07/05/2022 06:36:26", "content": "

Actually moving datasets to Google CoLab is the first challenge we face 😄.

\n

Reading such big data is still remains a challenge. I will post another topic on this when i am done with my experiments. 🙉🙈🙊

", "rawMarkdown": "Actually moving datasets to Google CoLab is the first challenge we face 😄.\n\nReading such big data is still remains a challenge. I will post another topic on this when i am done with my experiments. 🙉🙈🙊", "votes": null }, { "id": "1844021", "postDate": "07/05/2022 09:38:09", "content": "

Unfortunately, It only allow users in USA to sign up.

", "rawMarkdown": "Unfortunately, It only allow users in USA to sign up.", "votes": null }, { "id": "1844052", "postDate": "07/05/2022 10:13:51", "content": "

@dragonzhang Use some dragon fire 🔥🔥🔥. May be they will allow you 🙊 .

", "rawMarkdown": "dragonzhang Use some dragon fire 🔥🔥🔥. May be they will allow you 🙊 .", "votes": null }, { "id": "1844205", "postDate": "07/05/2022 12:17:07", "content": "

I think the best way to move them is to use the stripped down versions that some Kagglers have shared (I use Raddar's) or make your own. With the original datasets as they are, it' not easy to work. Maybe using Dask.

", "rawMarkdown": "I think the best way to move them is to use the stripped down versions that some Kagglers have shared (I use Raddar's) or make your own. With the original datasets as they are, it' not easy to work. Maybe using Dask.", "votes": null }, { "id": "1844220", "postDate": "07/05/2022 12:23:51", "content": "

Thanks @mirfanazam. Hi @dragonzhang paperspace works globally, you can try its free version if you want. I'm from Argentina and I had no problems, right now I'm with the pro version. I don't know of another similar service, the others offer you GPUs but you have to pay for the time of use.

", "rawMarkdown": "Thanks @mirfanazam. Hi @dragonzhang paperspace works globally, you can try its free version if you want. I'm from Argentina and I had no problems, right now I'm with the pro version. I don't know of another similar service, the others offer you GPUs but you have to pay for the time of use.", "votes": null }, { "id": "1845502", "postDate": "07/06/2022 11:35:57", "content": "

@dragonzhang just enter your phone number properly with your country calling code, and it automatically switchs.

", "rawMarkdown": "dragonzhang just enter your phone number properly with your country calling code, and it automatically switchs.", "votes": null }, { "id": "1845523", "postDate": "07/06/2022 11:52:11", "content": "

@mirfanazam Thanks for sharing very useful for Large Datasets from Kaggle to Google CoLab

", "rawMarkdown": "mirfanazam Thanks for sharing very useful for Large Datasets from Kaggle to Google CoLab", "votes": null }, { "id": "1845542", "postDate": "07/06/2022 12:05:28", "content": "

16 gigs of memory is insufficient for this data.
\nI am working with parquet files for that reason.
\nDid you guys have any other ideas or any suggestions?
\nCause this is absolutely mental.

", "rawMarkdown": "16 gigs of memory is insufficient for this data.\nI am working with parquet files for that reason.\nDid you guys have any other ideas or any suggestions?\nCause this is absolutely mental.", "votes": null }, { "id": "1845594", "postDate": "07/06/2022 12:54:46", "content": "

@sarang210 Parquet is the best option. It reduces size to almost 50%. I have created another notebook \"Large Dataset - CSV - DASK - Parquet\". You can have a look at it.

", "rawMarkdown": "sarang210 Parquet is the best option. It reduces size to almost 50%. I have created another notebook \"Large Dataset - CSV - DASK - Parquet\". You can have a look at it.", "votes": null }, { "id": "1845629", "postDate": "07/06/2022 13:25:23", "content": "

Sure,Thanks @mirfanazam , i will check it.

", "rawMarkdown": "Sure,Thanks @mirfanazam , i will check it.", "votes": null }, { "id": "1846304", "postDate": "07/07/2022 02:29:12", "content": "

Thanks for helpful suggestion when using large data.

", "rawMarkdown": "Thanks for helpful suggestion when using large data.", "votes": null } ], "comments": [ { "id": 1843209, "author_name": "ravi20076", "author_url": "", "post_date": "07/04/2022 16:17:09", "content": "

This is quite useful, colab is a pretty economical yet powerful supplement to the usual Kaggle kernels

", "votes": null, "replies": [] }, { "id": 1843334, "author_name": "maxdiazbattan", "author_url": "", "post_date": "07/04/2022 17:57:09", "content": "

Hi Muhammad, another option is paperspace gradient, which even in its free version offers 30 GB of ram.

", "votes": null, "replies": [ { "id": 1843436, "author_name": "mirfanazam", "author_url": "", "post_date": "07/04/2022 20:07:44", "content": "

@maxdiazbattan lovely suggestion 💥☀️

", "votes": null, "replies": [] }, { "id": 1844021, "author_name": "dragonzhang", "author_url": "", "post_date": "07/05/2022 09:38:09", "content": "

Unfortunately, It only allow users in USA to sign up.

", "votes": null, "replies": [] }, { "id": 1844052, "author_name": "mirfanazam", "author_url": "", "post_date": "07/05/2022 10:13:51", "content": "

@dragonzhang Use some dragon fire 🔥🔥🔥. May be they will allow you 🙊 .

", "votes": null, "replies": [] }, { "id": 1844220, "author_name": "maxdiazbattan", "author_url": "", "post_date": "07/05/2022 12:23:51", "content": "

Thanks @mirfanazam. Hi @dragonzhang paperspace works globally, you can try its free version if you want. I'm from Argentina and I had no problems, right now I'm with the pro version. I don't know of another similar service, the others offer you GPUs but you have to pay for the time of use.

", "votes": null, "replies": [] }, { "id": 1845502, "author_name": "wouldyoujustfocus", "author_url": "", "post_date": "07/06/2022 11:35:57", "content": "

@dragonzhang just enter your phone number properly with your country calling code, and it automatically switchs.

", "votes": null, "replies": [] } ] }, { "id": 1843397, "author_name": "faseeh001", "author_url": "", "post_date": "07/04/2022 19:10:07", "content": "

Thanks for sharing very useful for using in case of large data

", "votes": null, "replies": [] }, { "id": 1843810, "author_name": "markusdegen", "author_url": "", "post_date": "07/05/2022 06:24:38", "content": "

Cool thing, i tried this with Colab Pro+, got 51gb RAM and still have trouble running the whole thing. I guess i have to really do feature importance ahead

", "votes": null, "replies": [ { "id": 1843823, "author_name": "mirfanazam", "author_url": "", "post_date": "07/05/2022 06:36:26", "content": "

Actually moving datasets to Google CoLab is the first challenge we face 😄.

\n

Reading such big data is still remains a challenge. I will post another topic on this when i am done with my experiments. 🙉🙈🙊

", "votes": null, "replies": [] }, { "id": 1844205, "author_name": "maxdiazbattan", "author_url": "", "post_date": "07/05/2022 12:17:07", "content": "

I think the best way to move them is to use the stripped down versions that some Kagglers have shared (I use Raddar's) or make your own. With the original datasets as they are, it' not easy to work. Maybe using Dask.

", "votes": null, "replies": [] } ] }, { "id": 1845523, "author_name": "mumeryasin", "author_url": "", "post_date": "07/06/2022 11:52:11", "content": "

@mirfanazam Thanks for sharing very useful for Large Datasets from Kaggle to Google CoLab

", "votes": null, "replies": [] }, { "id": 1845542, "author_name": "sarang210", "author_url": "", "post_date": "07/06/2022 12:05:28", "content": "

16 gigs of memory is insufficient for this data.
\nI am working with parquet files for that reason.
\nDid you guys have any other ideas or any suggestions?
\nCause this is absolutely mental.

", "votes": null, "replies": [ { "id": 1845594, "author_name": "mirfanazam", "author_url": "", "post_date": "07/06/2022 12:54:46", "content": "

@sarang210 Parquet is the best option. It reduces size to almost 50%. I have created another notebook \"Large Dataset - CSV - DASK - Parquet\". You can have a look at it.

", "votes": null, "replies": [] }, { "id": 1845629, "author_name": "sarang210", "author_url": "", "post_date": "07/06/2022 13:25:23", "content": "

Sure,Thanks @mirfanazam , i will check it.

", "votes": null, "replies": [] } ] }, { "id": 1846304, "author_name": "takatoueno", "author_url": "", "post_date": "07/07/2022 02:29:12", "content": "

Thanks for helpful suggestion when using large data.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1843020": "If you are running out of storage and memory in Kaggle notebooks and want to use Google CoLab then below steps will help you and save a lot of time required to move large datasets. 🤓\n\nI am using Google CoLab paid membership. So your memory and storage requirements may require it as well. So check your requirements properly beforehand. 👀\n\n**1. Get Kaggle API Key**\n \nGo to Kaggle profile page\nClick Accounts tab\nUnder API section, click \"Create New API Token\". This will download your credentials in a JSON file.\n \n\n**2. Create New Notebook in Google CoLab**\n \n\n**3. Download Kaggle dataset into Google CoLab Notebook using below code.**\n \n\n`!pip install -U -q kaggle `\n`!mkdir -p ~/.kaggle `\n`!echo '{\"username\":\"YOUR_USER_ID\",\"key\":\"YOUR_KAGGLE_KEY\"}' > ~/.kaggle/kaggle.json `\n`!chmod 600 ~/.kaggle/kaggle.json `\n`!kaggle competitions download -c amex-default-prediction`\n \n\nKaggle dataset will be copied into Google CoLab in compressed format.\n \n\n**4. Uncompress Dataset** \n \n\n`!unzip '/content/amex-default-prediction.zip' -d '/content'`\n \n\nRemove compressed dataset to free storage in Google CoLab.", "1843209": "This is quite useful, colab is a pretty economical yet powerful supplement to the usual Kaggle kernels", "1843334": "Hi Muhammad, another option is paperspace gradient, which even in its free version offers 30 GB of ram.", "1843397": "Thanks for sharing very useful for using in case of large data", "1843436": "maxdiazbattan lovely suggestion 💥☀️", "1843810": "Cool thing, i tried this with Colab Pro+, got 51gb RAM and still have trouble running the whole thing. I guess i have to really do feature importance ahead", "1843823": "Actually moving datasets to Google CoLab is the first challenge we face 😄.\n\nReading such big data is still remains a challenge. I will post another topic on this when i am done with my experiments. 🙉🙈🙊", "1844021": "Unfortunately, It only allow users in USA to sign up.", "1844052": "dragonzhang Use some dragon fire 🔥🔥🔥. May be they will allow you 🙊 .", "1844205": "I think the best way to move them is to use the stripped down versions that some Kagglers have shared (I use Raddar's) or make your own. With the original datasets as they are, it' not easy to work. Maybe using Dask.", "1844220": "Thanks @mirfanazam. Hi @dragonzhang paperspace works globally, you can try its free version if you want. I'm from Argentina and I had no problems, right now I'm with the pro version. I don't know of another similar service, the others offer you GPUs but you have to pay for the time of use.", "1845502": "dragonzhang just enter your phone number properly with your country calling code, and it automatically switchs.", "1845523": "mirfanazam Thanks for sharing very useful for Large Datasets from Kaggle to Google CoLab", "1845542": "16 gigs of memory is insufficient for this data.\nI am working with parquet files for that reason.\nDid you guys have any other ideas or any suggestions?\nCause this is absolutely mental.", "1845594": "sarang210 Parquet is the best option. It reduces size to almost 50%. I have created another notebook \"Large Dataset - CSV - DASK - Parquet\". You can have a look at it.", "1845629": "Sure,Thanks @mirfanazam , i will check it.", "1846304": "Thanks for helpful suggestion when using large data." }, "source": "meta" }