{ "id": 289262, "title": "Kaggling on Colab: How to set everything up", "url": "/competitions/sartorius-cell-instance-segmentation/discussion/289262", "author_name": "Andreas Horlbeck", "post_date": "2021-11-19T12:54:20.769000", "votes": 13, "comment_count": 4, "views": 0, "content": "

Hey,

\n

in this post I want to introduce Google Colab to you and how you can set up the environment (meaning Google Colab and Google Drive)
\nto make kaggle competitions on Colab.
\nThe reason why I switched to Colab can be found in this post (better perfomance as a spoiler):
\nhttps://www.kaggle.com/general/287193

\n

To give a summarize:
\n0. Assuming you have a Google Account
\n1. Downloading Kaggle data sets to Google Drive via Google Colab (Drive has to be connected to the Colab Session)
\n2. Copy the data (which is usually zipped which is much better, so faster and stable ) from Drive to a contemporary storage in the Google Colab session (which is deleted after closing the session)

\n

Starting point:
\nYou start a competition and want to get the data to colab.

\n

1.
\nThe following code pieces should be run in yor colab session:

\n

the imports you need:
\nfrom google.colab import drive
\nimport os
\nimport shutil

\n

You start a Colab Session and mount it to your Google drive
\ndrive.mount('/content/gdrive')
\nThen you are aksed to connect Colab via your Google Account to your Google drive - just follow the few steps.

\n

To download the kaggle data you first have to install the kaggle API so Colab can have access to kaggle.
\nTherefore go to kaggle -> your profile -> Account -> scroll a bit to \"create a new token\".
\nIf you push this button a txt-file gets automatically downloaded.
\nOpen this and copy the content to the following line of code (just an example):

\n

os.environ['KAGGLE_USERNAME']  =  \"harrypotter\"                # username from the json file
\nos.environ['KAGGLE_KEY']       =  \"7821caeb282251e58c265b3bfa9ab7c1\"      # key from the json file

\n

Then install the kaggle API:
\n!pip install --upgrade --force-reinstall --no-deps kaggle

\n

Then download the kaggle dataset via the command that can be found in the data section of competition.
\nDont forget the \"!\" In front of the prewritten command on kaggle.
\nThe second part of the command '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train' is the destination where the data should be stored (in your Google Drive)

\n

!kaggle competitions download -c rsna-miccai-brain-tumor-radiogenomic-classification  '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train'

\n

Of course you can also download the data from a kaggle kernel.
\nWhat you have to consider if the data is too large for your drive or your Colab depending which memberhip you have, it can be necessary to preprocess it directly on kaggle and then download it to your drive/Colab.
\nWhen you process your data on kaggle and write it in the output folder, don't forget to zip it:

\n

shutil.make_archive('./name_of_zip_folder', 'zip', './folder_that gets_zipped')

\n

Another tip is if you have a huge amount of images, store them in subfolders, that is easier for Colab to deal with, especially when you encounter
\ntimeouts in your session.

\n

2.
\nNow move the zipped data to your Colab session (a own storage not the drive storage).
\nI would advise you to use the \"/content\" - folder where also your drive is located
\nMake a new folder for you data there, here I call it colab/train:

\n

os.mkdir('/content/colab/train')

\n

Then copy the data (here called train.zip) from your drive (first input) to this new folder (last input)

\n

shutil.copyfile( '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train.zip', /content/colab/train/train.zip')

\n

Now change directory, to the new folder '/content/colab/train.
\nos.chdir( '/content/colab/train')
\n!unzip -q train.zip

\n

The reason for moving the data to the own Colab storage as instead leave it in the drive, is that
\nwhile training (eg. With data generators) the images have to be downloaded from Drive to Colab which costs a lot of time!

\n

You can find further information here:
\nhttps://medium.com/swlh/setting-up-google-colab-for-cnn-modeling-55b5208599c4

\n

If there is any problem please write me immediately, I know the struggles with this topic.
\nHope it helps:)

", "messages": [ { "id": 1588532, "postDate": "2021-11-19T12:54:20.770Z", "content": "

Hey,

\n

in this post I want to introduce Google Colab to you and how you can set up the environment (meaning Google Colab and Google Drive)
\nto make kaggle competitions on Colab.
\nThe reason why I switched to Colab can be found in this post (better perfomance as a spoiler):
\nhttps://www.kaggle.com/general/287193

\n

To give a summarize:
\n0. Assuming you have a Google Account
\n1. Downloading Kaggle data sets to Google Drive via Google Colab (Drive has to be connected to the Colab Session)
\n2. Copy the data (which is usually zipped which is much better, so faster and stable ) from Drive to a contemporary storage in the Google Colab session (which is deleted after closing the session)

\n

Starting point:
\nYou start a competition and want to get the data to colab.

\n

1.
\nThe following code pieces should be run in yor colab session:

\n

the imports you need:
\nfrom google.colab import drive
\nimport os
\nimport shutil

\n

You start a Colab Session and mount it to your Google drive
\ndrive.mount('/content/gdrive')
\nThen you are aksed to connect Colab via your Google Account to your Google drive - just follow the few steps.

\n

To download the kaggle data you first have to install the kaggle API so Colab can have access to kaggle.
\nTherefore go to kaggle -> your profile -> Account -> scroll a bit to \"create a new token\".
\nIf you push this button a txt-file gets automatically downloaded.
\nOpen this and copy the content to the following line of code (just an example):

\n

os.environ['KAGGLE_USERNAME']  =  \"harrypotter\"                # username from the json file
\nos.environ['KAGGLE_KEY']       =  \"7821caeb282251e58c265b3bfa9ab7c1\"      # key from the json file

\n

Then install the kaggle API:
\n!pip install --upgrade --force-reinstall --no-deps kaggle

\n

Then download the kaggle dataset via the command that can be found in the data section of competition.
\nDont forget the \"!\" In front of the prewritten command on kaggle.
\nThe second part of the command '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train' is the destination where the data should be stored (in your Google Drive)

\n

!kaggle competitions download -c rsna-miccai-brain-tumor-radiogenomic-classification  '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train'

\n

Of course you can also download the data from a kaggle kernel.
\nWhat you have to consider if the data is too large for your drive or your Colab depending which memberhip you have, it can be necessary to preprocess it directly on kaggle and then download it to your drive/Colab.
\nWhen you process your data on kaggle and write it in the output folder, don't forget to zip it:

\n

shutil.make_archive('./name_of_zip_folder', 'zip', './folder_that gets_zipped')

\n

Another tip is if you have a huge amount of images, store them in subfolders, that is easier for Colab to deal with, especially when you encounter
\ntimeouts in your session.

\n

2.
\nNow move the zipped data to your Colab session (a own storage not the drive storage).
\nI would advise you to use the \"/content\" - folder where also your drive is located
\nMake a new folder for you data there, here I call it colab/train:

\n

os.mkdir('/content/colab/train')

\n

Then copy the data (here called train.zip) from your drive (first input) to this new folder (last input)

\n

shutil.copyfile( '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train.zip', /content/colab/train/train.zip')

\n

Now change directory, to the new folder '/content/colab/train.
\nos.chdir( '/content/colab/train')
\n!unzip -q train.zip

\n

The reason for moving the data to the own Colab storage as instead leave it in the drive, is that
\nwhile training (eg. With data generators) the images have to be downloaded from Drive to Colab which costs a lot of time!

\n

You can find further information here:
\nhttps://medium.com/swlh/setting-up-google-colab-for-cnn-modeling-55b5208599c4

\n

If there is any problem please write me immediately, I know the struggles with this topic.
\nHope it helps:)

", "rawMarkdown": "Hey,\n\nin this post I want to introduce Google Colab to you and how you can set up the environment (meaning Google Colab and Google Drive)\nto make kaggle competitions on Colab.\nThe reason why I switched to Colab can be found in this post (better perfomance as a spoiler):\nhttps://www.kaggle.com/general/287193\n\n**To give a summarize:**\n**0. Assuming you have a Google Account**\n**1. Downloading Kaggle data sets to Google Drive via Google Colab (Drive has to be connected to the Colab Session)**\n**2. Copy the data (which is usually zipped which is much better, so faster and stable ) from Drive to a contemporary storage in the Google Colab session (which is deleted after closing the session)**\n\nStarting point:\nYou start a competition and want to get the data to colab.\n\n\n**1.**\nThe following code pieces should be run in yor colab session:\n\nthe imports you need:\n**from google.colab import drive**\n**import os**\n**import shutil**\n\nYou start a Colab Session and mount it to your Google drive\n**drive.mount('/content/gdrive')**\nThen you are aksed to connect Colab via your Google Account to your Google drive - just follow the few steps.\n\nTo download the kaggle data you first have to install the kaggle API so Colab can have access to kaggle.\nTherefore go to kaggle -> your profile -> Account -> scroll a bit to \"create a new token\".\nIf you push this button a txt-file gets automatically downloaded.\nOpen this and copy the content to the following line of code (just an example):\n\n**os.environ['KAGGLE_USERNAME']  =  \"harrypotter\"                # username from the json file**\n**os.environ['KAGGLE_KEY']       =  \"7821caeb282251e58c265b3bfa9ab7c1\"      # key from the json file**\n\nThen install the kaggle API:\n**!pip install --upgrade --force-reinstall --no-deps kaggle**\n\nThen download the kaggle dataset via the command that can be found in the data section of competition.\nDont forget the \"!\" In front of the prewritten command on kaggle.\nThe second part of the command '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train' is the destination where the data should be stored (in your Google Drive)\n\n**!kaggle competitions download -c rsna-miccai-brain-tumor-radiogenomic-classification  '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train'**\n\nOf course you can also download the data from a kaggle kernel.\nWhat you have to consider if the data is too large for your drive or your Colab depending which memberhip you have, it can be necessary to preprocess it directly on kaggle and then download it to your drive/Colab.\nWhen you process your data on kaggle and write it in the output folder, don't forget to zip it:\n\n**shutil.make_archive('./name_of_zip_folder', 'zip', './folder_that gets_zipped')**\n\nAnother tip is if you have a huge amount of images, store them in subfolders, that is easier for Colab to deal with, especially when you encounter\ntimeouts in your session.\n\n**2.**\nNow move the zipped data to your Colab session (a own storage not the drive storage).\nI would advise you to use the \"/content\" - folder where also your drive is located\nMake a new folder for you data there, here I call it colab/train:\n\n**os.mkdir('/content/colab/train')**\n\nThen copy the data (here called train.zip) from your drive (first input) to this new folder (last input)\n\n**shutil.copyfile( '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train.zip', /content/colab/train/train.zip')**\n\nNow change directory, to the new folder '/content/colab/train.\n**os.chdir( '/content/colab/train')**\n**!unzip -q train.zip**\n\nThe reason for moving the data to the own Colab storage as instead leave it in the drive, is that \nwhile training (eg. With data generators) the images have to be downloaded from Drive to Colab which costs a lot of time!\n\nYou can find further information here:\nhttps://medium.com/swlh/setting-up-google-colab-for-cnn-modeling-55b5208599c4\n\n\nIf there is any problem please write me immediately, I know the struggles with this topic.\nHope it helps:)\n", "votes": 12 }, { "id": 1589075, "postDate": "2021-11-19T22:37:31.140Z", "content": "

Auto competition Submitting from Google Colab via Kaggle API is FUN :)

\n

https://www.kaggle.com/c/tabular-playground-series-nov-2021/discussion/286475

", "rawMarkdown": "Auto competition Submitting from Google Colab via Kaggle API is FUN :)\n\nhttps://www.kaggle.com/c/tabular-playground-series-nov-2021/discussion/286475\n", "votes": 1 }, { "id": 1589063, "postDate": "2021-11-19T22:23:29.683Z", "content": "

I am glad to see your post, I am using Google Colab a lot.

\n

for Kaggle I found that the best way is to avoid using Google Drive for Dataset, but it is perfect for saving the training outcomes.

\n

Kaggle API is fast, so it is ok to use it every time you train your model.

\n

for this competition, This is my code Google Colab

\n
!mkdir ~/.kaggle\n!cp /content/gdrive/MyDrive/Kaggle_all/kaggle.json ~/.kaggle/\n!pip install --upgrade --force-reinstall --no-deps kaggle -q\n!mkdir input/\n!mkdir input/sartorius_cell_instance_segmentation\n!kaggle competitions download -c sartorius-cell-instance-segmentation -p /content/input/sartorius_cell_instance_segmentation\n!unzip /content/input/sartorius_cell_instance_segmentation/*.zip -d /content/input/sartorius_cell_instance_segmentation\n!rm /content/input/sartorius_cell_instance_segmentation/*.zip\n....\ndataDir=Path('/content/input/sartorius_cell_instance_segmentation/')\n....\n!cp -R /content/output /content/gdrive/MyDrive/Kaggle_all/sartorius/\n
", "rawMarkdown": "I am glad to see your post, I am using Google Colab a lot. \n\nfor Kaggle I found that the best way is to avoid using Google Drive for Dataset, but it is perfect for saving the training outcomes. \n\nKaggle API is fast, so it is ok to use it every time you train your model. \n\nfor this competition, This is my code Google Colab \n\n```\n!mkdir ~/.kaggle\n!cp /content/gdrive/MyDrive/Kaggle_all/kaggle.json ~/.kaggle/\n!pip install --upgrade --force-reinstall --no-deps kaggle -q\n!mkdir input/\n!mkdir input/sartorius_cell_instance_segmentation\n!kaggle competitions download -c sartorius-cell-instance-segmentation -p /content/input/sartorius_cell_instance_segmentation\n!unzip /content/input/sartorius_cell_instance_segmentation/*.zip -d /content/input/sartorius_cell_instance_segmentation\n!rm /content/input/sartorius_cell_instance_segmentation/*.zip\n....\ndataDir=Path('/content/input/sartorius_cell_instance_segmentation/')\n....\n!cp -R /content/output /content/gdrive/MyDrive/Kaggle_all/sartorius/\n```", "votes": 2 }, { "id": 1589067, "postDate": "2021-11-19T22:26:24.927Z", "content": "

check this out
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/241414

", "rawMarkdown": "check this out \nhttps://www.kaggle.com/c/birdclef-2021/discussion/241414" }, { "id": 1588884, "postDate": "2021-11-19T18:18:23.207Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1589075, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-11-19T22:37:31.140000", "content": "

Auto competition Submitting from Google Colab via Kaggle API is FUN :)

\n

https://www.kaggle.com/c/tabular-playground-series-nov-2021/discussion/286475

", "votes": 1, "replies": [] }, { "id": 1589063, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-11-19T22:23:29.683000", "content": "

I am glad to see your post, I am using Google Colab a lot.

\n

for Kaggle I found that the best way is to avoid using Google Drive for Dataset, but it is perfect for saving the training outcomes.

\n

Kaggle API is fast, so it is ok to use it every time you train your model.

\n

for this competition, This is my code Google Colab

\n
!mkdir ~/.kaggle\n!cp /content/gdrive/MyDrive/Kaggle_all/kaggle.json ~/.kaggle/\n!pip install --upgrade --force-reinstall --no-deps kaggle -q\n!mkdir input/\n!mkdir input/sartorius_cell_instance_segmentation\n!kaggle competitions download -c sartorius-cell-instance-segmentation -p /content/input/sartorius_cell_instance_segmentation\n!unzip /content/input/sartorius_cell_instance_segmentation/*.zip -d /content/input/sartorius_cell_instance_segmentation\n!rm /content/input/sartorius_cell_instance_segmentation/*.zip\n....\ndataDir=Path('/content/input/sartorius_cell_instance_segmentation/')\n....\n!cp -R /content/output /content/gdrive/MyDrive/Kaggle_all/sartorius/\n
", "votes": 2, "replies": [] }, { "id": 1589067, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-11-19T22:26:24.927000", "content": "

check this out
\nhttps://www.kaggle.com/c/birdclef-2021/discussion/241414

", "votes": 0, "replies": [] }, { "id": 1588884, "author_name": "", "author_url": "", "post_date": "2021-11-19T18:18:23.207000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1588532": "Hey,\n\nin this post I want to introduce Google Colab to you and how you can set up the environment (meaning Google Colab and Google Drive)\nto make kaggle competitions on Colab.\nThe reason why I switched to Colab can be found in this post (better perfomance as a spoiler):\nhttps://www.kaggle.com/general/287193\n\n**To give a summarize:**\n**0. Assuming you have a Google Account**\n**1. Downloading Kaggle data sets to Google Drive via Google Colab (Drive has to be connected to the Colab Session)**\n**2. Copy the data (which is usually zipped which is much better, so faster and stable ) from Drive to a contemporary storage in the Google Colab session (which is deleted after closing the session)**\n\nStarting point:\nYou start a competition and want to get the data to colab.\n\n\n**1.**\nThe following code pieces should be run in yor colab session:\n\nthe imports you need:\n**from google.colab import drive**\n**import os**\n**import shutil**\n\nYou start a Colab Session and mount it to your Google drive\n**drive.mount('/content/gdrive')**\nThen you are aksed to connect Colab via your Google Account to your Google drive - just follow the few steps.\n\nTo download the kaggle data you first have to install the kaggle API so Colab can have access to kaggle.\nTherefore go to kaggle -> your profile -> Account -> scroll a bit to \"create a new token\".\nIf you push this button a txt-file gets automatically downloaded.\nOpen this and copy the content to the following line of code (just an example):\n\n**os.environ['KAGGLE_USERNAME']  =  \"harrypotter\"                # username from the json file**\n**os.environ['KAGGLE_KEY']       =  \"7821caeb282251e58c265b3bfa9ab7c1\"      # key from the json file**\n\nThen install the kaggle API:\n**!pip install --upgrade --force-reinstall --no-deps kaggle**\n\nThen download the kaggle dataset via the command that can be found in the data section of competition.\nDont forget the \"!\" In front of the prewritten command on kaggle.\nThe second part of the command '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train' is the destination where the data should be stored (in your Google Drive)\n\n**!kaggle competitions download -c rsna-miccai-brain-tumor-radiogenomic-classification  '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train'**\n\nOf course you can also download the data from a kaggle kernel.\nWhat you have to consider if the data is too large for your drive or your Colab depending which memberhip you have, it can be necessary to preprocess it directly on kaggle and then download it to your drive/Colab.\nWhen you process your data on kaggle and write it in the output folder, don't forget to zip it:\n\n**shutil.make_archive('./name_of_zip_folder', 'zip', './folder_that gets_zipped')**\n\nAnother tip is if you have a huge amount of images, store them in subfolders, that is easier for Colab to deal with, especially when you encounter\ntimeouts in your session.\n\n**2.**\nNow move the zipped data to your Colab session (a own storage not the drive storage).\nI would advise you to use the \"/content\" - folder where also your drive is located\nMake a new folder for you data there, here I call it colab/train:\n\n**os.mkdir('/content/colab/train')**\n\nThen copy the data (here called train.zip) from your drive (first input) to this new folder (last input)\n\n**shutil.copyfile( '/content/gdrive/MyDrive/KAGGLE/Competition_RSNA/original_data/train.zip', /content/colab/train/train.zip')**\n\nNow change directory, to the new folder '/content/colab/train.\n**os.chdir( '/content/colab/train')**\n**!unzip -q train.zip**\n\nThe reason for moving the data to the own Colab storage as instead leave it in the drive, is that \nwhile training (eg. With data generators) the images have to be downloaded from Drive to Colab which costs a lot of time!\n\nYou can find further information here:\nhttps://medium.com/swlh/setting-up-google-colab-for-cnn-modeling-55b5208599c4\n\n\nIf there is any problem please write me immediately, I know the struggles with this topic.\nHope it helps:)\n", "1589075": "Auto competition Submitting from Google Colab via Kaggle API is FUN :)\n\nhttps://www.kaggle.com/c/tabular-playground-series-nov-2021/discussion/286475\n", "1589063": "I am glad to see your post, I am using Google Colab a lot. \n\nfor Kaggle I found that the best way is to avoid using Google Drive for Dataset, but it is perfect for saving the training outcomes. \n\nKaggle API is fast, so it is ok to use it every time you train your model. \n\nfor this competition, This is my code Google Colab \n\n```\n!mkdir ~/.kaggle\n!cp /content/gdrive/MyDrive/Kaggle_all/kaggle.json ~/.kaggle/\n!pip install --upgrade --force-reinstall --no-deps kaggle -q\n!mkdir input/\n!mkdir input/sartorius_cell_instance_segmentation\n!kaggle competitions download -c sartorius-cell-instance-segmentation -p /content/input/sartorius_cell_instance_segmentation\n!unzip /content/input/sartorius_cell_instance_segmentation/*.zip -d /content/input/sartorius_cell_instance_segmentation\n!rm /content/input/sartorius_cell_instance_segmentation/*.zip\n....\ndataDir=Path('/content/input/sartorius_cell_instance_segmentation/')\n....\n!cp -R /content/output /content/gdrive/MyDrive/Kaggle_all/sartorius/\n```", "1589067": "check this out \nhttps://www.kaggle.com/c/birdclef-2021/discussion/241414", "1588884": "" } }