{
  "id": 238314,
  "title": "Running on Kaggle or Locally?",
  "url": "/competitions/birdclef-2021/discussion/238314",
  "author_name": "",
  "post_date": "2021-05-11T21:50:45.605129900Z",
  "votes": 3,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Just curious, but how many of you guys run on local machines vs using Kaggle's cloud resources?</p>",
  "messages": [
    {
      "id": "1303053",
      "postDate": "05/11/2021 21:50:45",
      "content": "<p>Just curious, but how many of you guys run on local machines vs using Kaggle's cloud resources?</p>",
      "rawMarkdown": "Just curious, but how many of you guys run on local machines vs using Kaggle's cloud resources?",
      "votes": null
    },
    {
      "id": "1303549",
      "postDate": "05/12/2021 06:09:52",
      "content": "<p>training locally (not very successful though…)</p>",
      "rawMarkdown": "training locally (not very successful though...)",
      "votes": null
    },
    {
      "id": "1303878",
      "postDate": "05/12/2021 10:01:09",
      "content": "<p>Google Colab Pro :) </p>\n<p><strong>what I like the most about Colab pro:</strong></p>\n<p>1- You can use Kaggle datasets via using the google cloud storage (GCS) bucket link without downloading it and for free. since all Kaggle datasets in GCS.</p>\n<p>2- with changing the working directory in Colab and using training callbacks you can keep saving your models in your Google Drive.</p>\n<p>3- Using google cloud/Colab is a skill I need to master because from next year I will start using tensorflow cloud carefully, I feel that it needed to be Kaggle competition grandmaster.</p>\n<p>What is TensorFlow Cloud?</p>\n<blockquote>\n  <p>TensorFlow Cloud is a Python package that provides APIs for a seamless transition from local debugging to <strong>distributed training</strong> in Google Cloud. It simplifies the process of training TensorFlow models on the cloud into a single, simple function call, requiring minimal setup and no changes to your model. TensorFlow Cloud handles cloud-specific tasks such as creating VM instances and distribution strategies for your models automatically.</p>\n</blockquote>\n<p><a href=\"https://www.tensorflow.org/guide/keras/training_keras_models_on_cloud\" target=\"_blank\">https://www.tensorflow.org/guide/keras/training_keras_models_on_cloud</a></p>",
      "rawMarkdown": "Google Colab Pro :) \n\n**what I like the most about Colab pro:**\n\n1- You can use Kaggle datasets via using the google cloud storage (GCS) bucket link without downloading it and for free. since all Kaggle datasets in GCS.\n\n2- with changing the working directory in Colab and using training callbacks you can keep saving your models in your Google Drive.\n\n3- Using google cloud/Colab is a skill I need to master because from next year I will start using tensorflow cloud carefully, I feel that it needed to be Kaggle competition grandmaster.\n\nWhat is TensorFlow Cloud?\n\n> TensorFlow Cloud is a Python package that provides APIs for a seamless transition from local debugging to **distributed training** in Google Cloud. It simplifies the process of training TensorFlow models on the cloud into a single, simple function call, requiring minimal setup and no changes to your model. TensorFlow Cloud handles cloud-specific tasks such as creating VM instances and distribution strategies for your models automatically.\n\nhttps://www.tensorflow.org/guide/keras/training_keras_models_on_cloud",
      "votes": null
    },
    {
      "id": "1310313",
      "postDate": "05/16/2021 15:35:35",
      "content": "<p>Training on Google Colab. I didn't realize you could copy Kaggle datasets through GCD as Faisal recommended, but I also don't have the Pro version of Colab.</p>\n<p>Instead, I converted all the audio data to iunt8 numpy array and then converted them to flac files, so that the data is roughly ~5 GB. Uploaded it to Colab overnight and voila!</p>",
      "rawMarkdown": "Training on Google Colab. I didn't realize you could copy Kaggle datasets through GCD as Faisal recommended, but I also don't have the Pro version of Colab.\n\nInstead, I converted all the audio data to iunt8 numpy array and then converted them to flac files, so that the data is roughly ~5 GB. Uploaded it to Colab overnight and voila!",
      "votes": null
    },
    {
      "id": "1313917",
      "postDate": "05/18/2021 19:48:12",
      "content": "<blockquote>\n  <p>1- You can use Kaggle datasets via using the google cloud storage (GCS) bucket link without downloading it and for free. since all Kaggle datasets in GCS.</p>\n</blockquote>\n<p>You are talking about using kaggle datasets on colab without donwloading them?</p>",
      "rawMarkdown": "> 1- You can use Kaggle datasets via using the google cloud storage (GCS) bucket link without downloading it and for free. since all Kaggle datasets in GCS.\n\nYou are talking about using kaggle datasets on colab without donwloading them?",
      "votes": null
    },
    {
      "id": "1313936",
      "postDate": "05/18/2021 20:08:01",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/victorasso\" target=\"_blank\">@victorasso</a> </p>\n<p>Yes, you can get the (Google Cloud Storage) path for any Kaggle datasets. like this </p>\n<pre><code>from kaggle_datasets import KaggleDatasets\nCOMPETITION_NAME = \"birdclef-2021\"\nGCS_DS_PATH = KaggleDatasets().get_gcs_path(COMPETITION_NAME)\n</code></pre>",
      "rawMarkdown": "Hi @victorasso \n\nYes, you can get the (Google Cloud Storage) path for any Kaggle datasets. like this \n\n```\nfrom kaggle_datasets import KaggleDatasets\nCOMPETITION_NAME = \"birdclef-2021\"\nGCS_DS_PATH = KaggleDatasets().get_gcs_path(COMPETITION_NAME)\n```",
      "votes": null
    },
    {
      "id": "1313991",
      "postDate": "05/18/2021 21:18:01",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/victorasso\" target=\"_blank\">@victorasso</a> <a href=\"https://www.kaggle.com/dryanfurman\" target=\"_blank\">@dryanfurman</a> </p>\n<p>I have made this code to show you how</p>\n<p><a href=\"https://www.kaggle.com/faisalalsrheed/birdclef-2021-kaggledatasets-get-gcs-path-demo\" target=\"_blank\">https://www.kaggle.com/faisalalsrheed/birdclef-2021-kaggledatasets-get-gcs-path-demo</a></p>\n<p>Let me know if you have any questions.</p>",
      "rawMarkdown": "Hi @victorasso @dryanfurman \n\nI have made this code to show you how\n\nhttps://www.kaggle.com/faisalalsrheed/birdclef-2021-kaggledatasets-get-gcs-path-demo\n\nLet me know if you have any questions.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1303549,
      "author_name": "botkop",
      "author_url": "",
      "post_date": "05/12/2021 06:09:52",
      "content": "<p>training locally (not very successful though…)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1303878,
      "author_name": "faisalalsrheed",
      "author_url": "",
      "post_date": "05/12/2021 10:01:09",
      "content": "<p>Google Colab Pro :) </p>\n<p><strong>what I like the most about Colab pro:</strong></p>\n<p>1- You can use Kaggle datasets via using the google cloud storage (GCS) bucket link without downloading it and for free. since all Kaggle datasets in GCS.</p>\n<p>2- with changing the working directory in Colab and using training callbacks you can keep saving your models in your Google Drive.</p>\n<p>3- Using google cloud/Colab is a skill I need to master because from next year I will start using tensorflow cloud carefully, I feel that it needed to be Kaggle competition grandmaster.</p>\n<p>What is TensorFlow Cloud?</p>\n<blockquote>\n  <p>TensorFlow Cloud is a Python package that provides APIs for a seamless transition from local debugging to <strong>distributed training</strong> in Google Cloud. It simplifies the process of training TensorFlow models on the cloud into a single, simple function call, requiring minimal setup and no changes to your model. TensorFlow Cloud handles cloud-specific tasks such as creating VM instances and distribution strategies for your models automatically.</p>\n</blockquote>\n<p><a href=\"https://www.tensorflow.org/guide/keras/training_keras_models_on_cloud\" target=\"_blank\">https://www.tensorflow.org/guide/keras/training_keras_models_on_cloud</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1313917,
          "author_name": "victorasso",
          "author_url": "",
          "post_date": "05/18/2021 19:48:12",
          "content": "<blockquote>\n  <p>1- You can use Kaggle datasets via using the google cloud storage (GCS) bucket link without downloading it and for free. since all Kaggle datasets in GCS.</p>\n</blockquote>\n<p>You are talking about using kaggle datasets on colab without donwloading them?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1313936,
          "author_name": "faisalalsrheed",
          "author_url": "",
          "post_date": "05/18/2021 20:08:01",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/victorasso\" target=\"_blank\">@victorasso</a> </p>\n<p>Yes, you can get the (Google Cloud Storage) path for any Kaggle datasets. like this </p>\n<pre><code>from kaggle_datasets import KaggleDatasets\nCOMPETITION_NAME = \"birdclef-2021\"\nGCS_DS_PATH = KaggleDatasets().get_gcs_path(COMPETITION_NAME)\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1313991,
          "author_name": "faisalalsrheed",
          "author_url": "",
          "post_date": "05/18/2021 21:18:01",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/victorasso\" target=\"_blank\">@victorasso</a> <a href=\"https://www.kaggle.com/dryanfurman\" target=\"_blank\">@dryanfurman</a> </p>\n<p>I have made this code to show you how</p>\n<p><a href=\"https://www.kaggle.com/faisalalsrheed/birdclef-2021-kaggledatasets-get-gcs-path-demo\" target=\"_blank\">https://www.kaggle.com/faisalalsrheed/birdclef-2021-kaggledatasets-get-gcs-path-demo</a></p>\n<p>Let me know if you have any questions.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1310313,
      "author_name": "dryanfurman",
      "author_url": "",
      "post_date": "05/16/2021 15:35:35",
      "content": "<p>Training on Google Colab. I didn't realize you could copy Kaggle datasets through GCD as Faisal recommended, but I also don't have the Pro version of Colab.</p>\n<p>Instead, I converted all the audio data to iunt8 numpy array and then converted them to flac files, so that the data is roughly ~5 GB. Uploaded it to Colab overnight and voila!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1303053": "Just curious, but how many of you guys run on local machines vs using Kaggle's cloud resources?",
    "1303549": "training locally (not very successful though...)",
    "1303878": "Google Colab Pro :) \n\n**what I like the most about Colab pro:**\n\n1- You can use Kaggle datasets via using the google cloud storage (GCS) bucket link without downloading it and for free. since all Kaggle datasets in GCS.\n\n2- with changing the working directory in Colab and using training callbacks you can keep saving your models in your Google Drive.\n\n3- Using google cloud/Colab is a skill I need to master because from next year I will start using tensorflow cloud carefully, I feel that it needed to be Kaggle competition grandmaster.\n\nWhat is TensorFlow Cloud?\n\n> TensorFlow Cloud is a Python package that provides APIs for a seamless transition from local debugging to **distributed training** in Google Cloud. It simplifies the process of training TensorFlow models on the cloud into a single, simple function call, requiring minimal setup and no changes to your model. TensorFlow Cloud handles cloud-specific tasks such as creating VM instances and distribution strategies for your models automatically.\n\nhttps://www.tensorflow.org/guide/keras/training_keras_models_on_cloud",
    "1310313": "Training on Google Colab. I didn't realize you could copy Kaggle datasets through GCD as Faisal recommended, but I also don't have the Pro version of Colab.\n\nInstead, I converted all the audio data to iunt8 numpy array and then converted them to flac files, so that the data is roughly ~5 GB. Uploaded it to Colab overnight and voila!",
    "1313917": "> 1- You can use Kaggle datasets via using the google cloud storage (GCS) bucket link without downloading it and for free. since all Kaggle datasets in GCS.\n\nYou are talking about using kaggle datasets on colab without donwloading them?",
    "1313936": "Hi @victorasso \n\nYes, you can get the (Google Cloud Storage) path for any Kaggle datasets. like this \n\n```\nfrom kaggle_datasets import KaggleDatasets\nCOMPETITION_NAME = \"birdclef-2021\"\nGCS_DS_PATH = KaggleDatasets().get_gcs_path(COMPETITION_NAME)\n```",
    "1313991": "Hi @victorasso @dryanfurman \n\nI have made this code to show you how\n\nhttps://www.kaggle.com/faisalalsrheed/birdclef-2021-kaggledatasets-get-gcs-path-demo\n\nLet me know if you have any questions."
  },
  "source": "meta"
}