Hi, Kagglers. As many of you might know, Google Colab can provide you with free GPU/TPU. This is really helpful when your Kaggle GPU/TPU quota runs out, you can run your models on Google Colab. However, for me, it's always a pain setting the environment up and downloading the data, configuring the GCS_PATH for TPU usage. Here, I will show you how to set up and download the data for Google Colab and how to automatically save your models to your Google Drive so that you won't loose the model when your notebook turns off.
After you open a new notebook on Google Colab, you'll have to install the Kaggle API. However, when you install it, it will not install the newest version on Colab. So After you type in ! pip install kaggle, run !pip install --upgrade --force-reinstall --no-deps kaggle to force reinstall and update it.
Getting the API key. Go to your Kaggle profile, under accounts, there's an API section. Press Expire API Token first, then create a new one by pressing Create New API Token. It'll download a json file to your computer. In Google Colab type in:
\nfrom google.colab import files then files.upload()
\nUp load your json file.
Type in these three lines which makes a directory for the json file then gives you permission to it.
\n! mkdir ~/.kaggle
\n! cp kaggle.json ~/.kaggle/
\n! chmod 600 ~/.kaggle/kaggle.json
Getting the Competition data. Download the data by ! kaggle competitions download -c ranzcr-clip-catheter-line-classification. After downloading the data, unzip the file by ! unzip /content/ranzcr-clip-catheter-line-classification.zip
For File paths, instead of ../input/ranzcr-clip-catheter-line-classification/competition_file it's just /content/competition_file.
For the TPU's GCS_PATH, just use the file paths returned by KaggleDatasets().get_gcs_path(COMPETITION_NAME). You don't have to configure anything on the Google Colab end.
In an empty cell, type in:
\nfrom google.colab import drive
\ndrive.mount('/content/gdrive', force_remount=True)
\ngdrive_file_path = '/content/gdrive/MyDrive/the_name_of_directory_that_you_want'
Follow the prompt that it gives you and allow your account to be connected to Google Drive.
\nNow, you can save things to the gdrive_file_path and it will appear in your Google Drive.
For example, now your Keras Model Checkpoint Callback would be:
\ncheckpoint = tf.keras.callbacks.ModelCheckpoint(\n gdrive_file_path + f'/model{fold}.h5', save_best_only=True, monitor='val_auc', mode='max')
I really hope this post helped you, for me Colab can give me from anywhere of 7 hrs of extra GPU/TPU to upwards of 20+ hrs depending on the availability. If there's any questions or I have wrote something wrong please comment on the post and let me know! Happy Kaggling!
\nPS: I really should write this in a notebook but Im too lazy to.
", "messages": [ { "id": "1144933", "postDate": "01/08/2021 18:50:56", "content": "Hi, Kagglers. As many of you might know, Google Colab can provide you with free GPU/TPU. This is really helpful when your Kaggle GPU/TPU quota runs out, you can run your models on Google Colab. However, for me, it's always a pain setting the environment up and downloading the data, configuring the GCS_PATH for TPU usage. Here, I will show you how to set up and download the data for Google Colab and how to automatically save your models to your Google Drive so that you won't loose the model when your notebook turns off.
After you open a new notebook on Google Colab, you'll have to install the Kaggle API. However, when you install it, it will not install the newest version on Colab. So After you type in ! pip install kaggle, run !pip install --upgrade --force-reinstall --no-deps kaggle to force reinstall and update it.
Getting the API key. Go to your Kaggle profile, under accounts, there's an API section. Press Expire API Token first, then create a new one by pressing Create New API Token. It'll download a json file to your computer. In Google Colab type in:
\nfrom google.colab import files then files.upload()
\nUp load your json file.
Type in these three lines which makes a directory for the json file then gives you permission to it.
\n! mkdir ~/.kaggle
\n! cp kaggle.json ~/.kaggle/
\n! chmod 600 ~/.kaggle/kaggle.json
Getting the Competition data. Download the data by ! kaggle competitions download -c ranzcr-clip-catheter-line-classification. After downloading the data, unzip the file by ! unzip /content/ranzcr-clip-catheter-line-classification.zip
For File paths, instead of ../input/ranzcr-clip-catheter-line-classification/competition_file it's just /content/competition_file.
For the TPU's GCS_PATH, just use the file paths returned by KaggleDatasets().get_gcs_path(COMPETITION_NAME). You don't have to configure anything on the Google Colab end.
In an empty cell, type in:
\nfrom google.colab import drive
\ndrive.mount('/content/gdrive', force_remount=True)
\ngdrive_file_path = '/content/gdrive/MyDrive/the_name_of_directory_that_you_want'
Follow the prompt that it gives you and allow your account to be connected to Google Drive.
\nNow, you can save things to the gdrive_file_path and it will appear in your Google Drive.
For example, now your Keras Model Checkpoint Callback would be:
\ncheckpoint = tf.keras.callbacks.ModelCheckpoint(\n gdrive_file_path + f'/model{fold}.h5', save_best_only=True, monitor='val_auc', mode='max')
I really hope this post helped you, for me Colab can give me from anywhere of 7 hrs of extra GPU/TPU to upwards of 20+ hrs depending on the availability. If there's any questions or I have wrote something wrong please comment on the post and let me know! Happy Kaggling!
\nPS: I really should write this in a notebook but Im too lazy to.
", "rawMarkdown": "Hi, Kagglers. As many of you might know, [Google Colab](https://colab.research.google.com/) can provide you with free GPU/TPU. This is really helpful when your Kaggle GPU/TPU quota runs out, you can run your models on Google Colab. However, for me, it's always a pain setting the environment up and downloading the data, configuring the `GCS_PATH` for TPU usage. Here, I will show you how to set up and download the data for Google Colab and how to automatically save your models to your Google Drive so that you won't loose the model when your notebook turns off.\n\n### Downloading the Data\n\n1. After you open a new notebook on Google Colab, you'll have to install the Kaggle API. However, when you install it, it will not install the newest version on Colab. So After you type in `! pip install kaggle`, run `!pip install --upgrade --force-reinstall --no-deps kaggle` to force reinstall and update it.\n\n2. Getting the API key. Go to your Kaggle profile, under accounts, there's an API section. Press `Expire API Token` first, then create a new one by pressing `Create New API Token`. It'll download a `json` file to your computer. In Google Colab type in: \n`from google.colab import files` then `files.upload()` \nUp load your `json` file.\n\n3. Type in these three lines which makes a directory for the `json` file then gives you permission to it.\n`! mkdir ~/.kaggle`\n`! cp kaggle.json ~/.kaggle/`\n`! chmod 600 ~/.kaggle/kaggle.json`\n\n4. Getting the Competition data. Download the data by `! kaggle competitions download -c ranzcr-clip-catheter-line-classification`. After downloading the data, unzip the file by `! unzip /content/ranzcr-clip-catheter-line-classification.zip`\n\n### File Paths and TPU\n\nFor File paths, instead of `../input/ranzcr-clip-catheter-line-classification/competition_file` it's just `/content/competition_file`.\n\nFor the TPU's `GCS_PATH`, just use the file paths returned by `KaggleDatasets().get_gcs_path(COMPETITION_NAME)`. You don't have to configure anything on the Google Colab end.\n\n### Saving Models Automatically to Drive\n\nIn an empty cell, type in: \n`from google.colab import drive`\n`drive.mount('/content/gdrive', force_remount=True)`\n`gdrive_file_path = '/content/gdrive/MyDrive/the_name_of_directory_that_you_want'`\n\nFollow the prompt that it gives you and allow your account to be connected to Google Drive.\n\nNow, you can save things to the `gdrive_file_path` and it will appear in your Google Drive.\n\nFor example, now your Keras Model Checkpoint Callback would be:\n`checkpoint = tf.keras.callbacks.ModelCheckpoint(\n gdrive_file_path + f'/model{fold}.h5', save_best_only=True, monitor='val_auc', mode='max')`\n\nI really hope this post helped you, for me Colab can give me from anywhere of 7 hrs of extra GPU/TPU to upwards of 20+ hrs depending on the availability. If there's any questions or I have wrote something wrong please comment on the post and let me know! Happy Kaggling! \n\nPS: I really should write this in a notebook but Im too lazy to.", "votes": null }, { "id": "1145063", "postDate": "01/08/2021 21:09:36", "content": "to simplify the process, save your kaggle json key in your drive and pull it from there automatically. removes one annoying manual step.
", "rawMarkdown": "to simplify the process, save your kaggle json key in your drive and pull it from there automatically. removes one annoying manual step.", "votes": null }, { "id": "1145082", "postDate": "01/08/2021 21:39:53", "content": "Yes, good point because every time I have to start a new session I'd have to manually upload the file myself.
", "rawMarkdown": "Yes, good point because every time I have to start a new session I'd have to manually upload the file myself.", "votes": null }, { "id": "1145339", "postDate": "01/09/2021 04:40:26", "content": "I took the liberty of making it into a notebook 😛
\nThanks for the discussion - very useful for newcomers like myself.
Thank you so much! upvoted!
", "rawMarkdown": "Thank you so much! upvoted!", "votes": null }, { "id": "1149813", "postDate": "01/12/2021 07:03:03", "content": "a good way to handle the data. I will try it. Thanks to @andy1010 and @moshel
", "rawMarkdown": "a good way to handle the data. I will try it. Thanks to @andy1010 and @moshel", "votes": null }, { "id": "1150653", "postDate": "01/12/2021 18:17:19", "content": "This worked for me thanks @andy1010
", "rawMarkdown": "This worked for me thanks @andy1010", "votes": null }, { "id": "1187612", "postDate": "02/05/2021 15:14:59", "content": "Your suggestion is very useful to me, I am a kaggle beginner, I really need this kind of useful skills, thank you
", "rawMarkdown": "Your suggestion is very useful to me, I am a kaggle beginner, I really need this kind of useful skills, thank you", "votes": null }, { "id": "1188309", "postDate": "02/06/2021 06:06:39", "content": "You may consider download and save dataset to google drive, avoid download data from kaggle every time.
", "rawMarkdown": "You may consider download and save dataset to google drive, avoid download data from kaggle every time.", "votes": null }, { "id": "1199387", "postDate": "02/13/2021 19:47:41", "content": "I have a question that if I need to use TPU and I am using gas path then do I need to download data to google drive using colab? Thanks in advance.
", "rawMarkdown": "I have a question that if I need to use TPU and I am using gas path then do I need to download data to google drive using colab? Thanks in advance.", "votes": null }, { "id": "1200570", "postDate": "02/14/2021 18:53:36", "content": "Thanks @moshel and @andy1010 !
", "rawMarkdown": "Thanks @moshel and @andy1010 !", "votes": null }, { "id": "1217031", "postDate": "02/24/2021 18:02:15", "content": "Hey @andy1010 appreciate your post. This is a handy information. Thanks for sharing.
", "rawMarkdown": "Hey @andy1010 appreciate your post. This is a handy information. Thanks for sharing.", "votes": null }, { "id": "2007821", "postDate": "10/28/2022 14:12:49", "content": "Your suggestion is very useful, Thanks.
", "rawMarkdown": "Your suggestion is very useful, Thanks.", "votes": null }, { "id": "2060692", "postDate": "12/10/2022 09:58:19", "content": "it's demonstrated here:
\nhttps://www.kaggle.com/discussions/general/371462#2060661
This is usefull for me. Thank you.
", "rawMarkdown": "This is usefull for me. Thank you.", "votes": null } ], "comments": [ { "id": 1145063, "author_name": "moshel", "author_url": "", "post_date": "01/08/2021 21:09:36", "content": "to simplify the process, save your kaggle json key in your drive and pull it from there automatically. removes one annoying manual step.
", "votes": null, "replies": [ { "id": 1145082, "author_name": "andy1010", "author_url": "", "post_date": "01/08/2021 21:39:53", "content": "Yes, good point because every time I have to start a new session I'd have to manually upload the file myself.
", "votes": null, "replies": [] }, { "id": 1200570, "author_name": "saurabhbagchi", "author_url": "", "post_date": "02/14/2021 18:53:36", "content": "Thanks @moshel and @andy1010 !
", "votes": null, "replies": [] } ] }, { "id": 1145339, "author_name": "reubenschmidt", "author_url": "", "post_date": "01/09/2021 04:40:26", "content": "I took the liberty of making it into a notebook 😛
\nThanks for the discussion - very useful for newcomers like myself.
Thank you so much! upvoted!
", "votes": null, "replies": [] } ] }, { "id": 1149813, "author_name": "arunrathi", "author_url": "", "post_date": "01/12/2021 07:03:03", "content": "a good way to handle the data. I will try it. Thanks to @andy1010 and @moshel
", "votes": null, "replies": [] }, { "id": 1150653, "author_name": "digvijayyadav", "author_url": "", "post_date": "01/12/2021 18:17:19", "content": "This worked for me thanks @andy1010
", "votes": null, "replies": [] }, { "id": 1187612, "author_name": "kingeee22", "author_url": "", "post_date": "02/05/2021 15:14:59", "content": "Your suggestion is very useful to me, I am a kaggle beginner, I really need this kind of useful skills, thank you
", "votes": null, "replies": [] }, { "id": 1188309, "author_name": "steamedsheep", "author_url": "", "post_date": "02/06/2021 06:06:39", "content": "You may consider download and save dataset to google drive, avoid download data from kaggle every time.
", "votes": null, "replies": [] }, { "id": 1199387, "author_name": "yashchoksi16", "author_url": "", "post_date": "02/13/2021 19:47:41", "content": "I have a question that if I need to use TPU and I am using gas path then do I need to download data to google drive using colab? Thanks in advance.
", "votes": null, "replies": [] }, { "id": 1217031, "author_name": "riyadhar", "author_url": "", "post_date": "02/24/2021 18:02:15", "content": "Hey @andy1010 appreciate your post. This is a handy information. Thanks for sharing.
", "votes": null, "replies": [] }, { "id": 2007821, "author_name": "redaghanem99", "author_url": "", "post_date": "10/28/2022 14:12:49", "content": "Your suggestion is very useful, Thanks.
", "votes": null, "replies": [] }, { "id": 2060692, "author_name": "drkhaledmamar", "author_url": "", "post_date": "12/10/2022 09:58:19", "content": "it's demonstrated here:
\nhttps://www.kaggle.com/discussions/general/371462#2060661
This is usefull for me. Thank you.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1144933": "Hi, Kagglers. As many of you might know, [Google Colab](https://colab.research.google.com/) can provide you with free GPU/TPU. This is really helpful when your Kaggle GPU/TPU quota runs out, you can run your models on Google Colab. However, for me, it's always a pain setting the environment up and downloading the data, configuring the `GCS_PATH` for TPU usage. Here, I will show you how to set up and download the data for Google Colab and how to automatically save your models to your Google Drive so that you won't loose the model when your notebook turns off.\n\n### Downloading the Data\n\n1. After you open a new notebook on Google Colab, you'll have to install the Kaggle API. However, when you install it, it will not install the newest version on Colab. So After you type in `! pip install kaggle`, run `!pip install --upgrade --force-reinstall --no-deps kaggle` to force reinstall and update it.\n\n2. Getting the API key. Go to your Kaggle profile, under accounts, there's an API section. Press `Expire API Token` first, then create a new one by pressing `Create New API Token`. It'll download a `json` file to your computer. In Google Colab type in: \n`from google.colab import files` then `files.upload()` \nUp load your `json` file.\n\n3. Type in these three lines which makes a directory for the `json` file then gives you permission to it.\n`! mkdir ~/.kaggle`\n`! cp kaggle.json ~/.kaggle/`\n`! chmod 600 ~/.kaggle/kaggle.json`\n\n4. Getting the Competition data. Download the data by `! kaggle competitions download -c ranzcr-clip-catheter-line-classification`. After downloading the data, unzip the file by `! unzip /content/ranzcr-clip-catheter-line-classification.zip`\n\n### File Paths and TPU\n\nFor File paths, instead of `../input/ranzcr-clip-catheter-line-classification/competition_file` it's just `/content/competition_file`.\n\nFor the TPU's `GCS_PATH`, just use the file paths returned by `KaggleDatasets().get_gcs_path(COMPETITION_NAME)`. You don't have to configure anything on the Google Colab end.\n\n### Saving Models Automatically to Drive\n\nIn an empty cell, type in: \n`from google.colab import drive`\n`drive.mount('/content/gdrive', force_remount=True)`\n`gdrive_file_path = '/content/gdrive/MyDrive/the_name_of_directory_that_you_want'`\n\nFollow the prompt that it gives you and allow your account to be connected to Google Drive.\n\nNow, you can save things to the `gdrive_file_path` and it will appear in your Google Drive.\n\nFor example, now your Keras Model Checkpoint Callback would be:\n`checkpoint = tf.keras.callbacks.ModelCheckpoint(\n gdrive_file_path + f'/model{fold}.h5', save_best_only=True, monitor='val_auc', mode='max')`\n\nI really hope this post helped you, for me Colab can give me from anywhere of 7 hrs of extra GPU/TPU to upwards of 20+ hrs depending on the availability. If there's any questions or I have wrote something wrong please comment on the post and let me know! Happy Kaggling! \n\nPS: I really should write this in a notebook but Im too lazy to.", "1145063": "to simplify the process, save your kaggle json key in your drive and pull it from there automatically. removes one annoying manual step.", "1145082": "Yes, good point because every time I have to start a new session I'd have to manually upload the file myself.", "1145339": "I took the liberty of making it into a [notebook](https://www.kaggle.com/reubenschmidt/how-to-use-google-colab-for-extra-tpu-gpu) 😛\nThanks for the discussion - very useful for newcomers like myself.", "1145363": "Thank you so much! upvoted!", "1149813": "a good way to handle the data. I will try it. Thanks to @andy1010 and @moshel", "1150653": "This worked for me thanks @andy1010", "1187612": "Your suggestion is very useful to me, I am a kaggle beginner, I really need this kind of useful skills, thank you", "1188309": "You may consider download and save dataset to google drive, avoid download data from kaggle every time.", "1199387": "I have a question that if I need to use TPU and I am using gas path then do I need to download data to google drive using colab? Thanks in advance.", "1200570": "Thanks @moshel and @andy1010 !", "1217031": "Hey @andy1010 appreciate your post. This is a handy information. Thanks for sharing.", "2007821": "Your suggestion is very useful, Thanks.", "2060692": "it's demonstrated here:\nhttps://www.kaggle.com/discussions/general/371462#2060661", "2908231": "This is usefull for me. Thank you." }, "source": "meta" }