{
  "id": 148986,
  "title": "How do I download dataset to Colab?",
  "url": "/competitions/alaska2-image-steganalysis/discussion/148986",
  "author_name": "",
  "post_date": "2020-05-06T10:50:09.372410400Z",
  "votes": 6,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n\n<p>I am new to Kaggle, I recently used all of my GPU trying to train my model, so I tried switching to Colab. <br>\n I am having problems while downloading the Kaggle dataset for <a href=\"https://www.kaggle.com/c/alaska2-image-steganalysis\">Steganalysis Challenge</a> in Google Colab.</p>\n\n<p>Here's what I did: \n1. I downloaded the <code>kaggle.json</code> file from my account, and uploaded it in Colab.\n2. <code>!mkdir -p ~/.kaggle</code>\n3. <code>!cp kaggle.json ~/.kaggle/</code>\n4. <code>!kaggle competitions download -c alaska2-image-steganalysis</code></p>\n\n<p>This 4th command starts to download all the images as <code>00001.jpg</code>, <code>00002.jpg</code>, and so on.</p>\n\n<p>Issues:\n1. It does not download all the images in the dataset.\n2. BIGGER ISSUE: It does not maintain the directory structure, all images are downloaded without the parent directory.</p>\n\n<p>Ques: <strong>How can  I download my entire dataset to Google Colab when the dataset is not in zipped format?</strong>\nAlt Ques: How can I maintain the directory structure of the dataset while downloading using kaggle api cli?</p>",
  "messages": [
    {
      "id": "835559",
      "postDate": "05/06/2020 10:50:09",
      "content": "<p>Hi everyone,</p>\n\n<p>I am new to Kaggle, I recently used all of my GPU trying to train my model, so I tried switching to Colab. <br>\n I am having problems while downloading the Kaggle dataset for <a href=\"https://www.kaggle.com/c/alaska2-image-steganalysis\">Steganalysis Challenge</a> in Google Colab.</p>\n\n<p>Here's what I did: \n1. I downloaded the <code>kaggle.json</code> file from my account, and uploaded it in Colab.\n2. <code>!mkdir -p ~/.kaggle</code>\n3. <code>!cp kaggle.json ~/.kaggle/</code>\n4. <code>!kaggle competitions download -c alaska2-image-steganalysis</code></p>\n\n<p>This 4th command starts to download all the images as <code>00001.jpg</code>, <code>00002.jpg</code>, and so on.</p>\n\n<p>Issues:\n1. It does not download all the images in the dataset.\n2. BIGGER ISSUE: It does not maintain the directory structure, all images are downloaded without the parent directory.</p>\n\n<p>Ques: <strong>How can  I download my entire dataset to Google Colab when the dataset is not in zipped format?</strong>\nAlt Ques: How can I maintain the directory structure of the dataset while downloading using kaggle api cli?</p>",
      "rawMarkdown": "Hi everyone,\n\nI am new to Kaggle, I recently used all of my GPU trying to train my model, so I tried switching to Colab.  \n I am having problems while downloading the Kaggle dataset for [Steganalysis Challenge](https://www.kaggle.com/c/alaska2-image-steganalysis) in Google Colab.\n\nHere's what I did: \n1. I downloaded the `kaggle.json` file from my account, and uploaded it in Colab.\n2. `!mkdir -p ~/.kaggle`\n3. `!cp kaggle.json ~/.kaggle/`\n4. `!kaggle competitions download -c alaska2-image-steganalysis`\n\nThis 4th command starts to download all the images as `00001.jpg`, `00002.jpg`, and so on.\n\nIssues:\n1. It does not download all the images in the dataset.\n2. BIGGER ISSUE: It does not maintain the directory structure, all images are downloaded without the parent directory.\n\nQues: **How can  I download my entire dataset to Google Colab when the dataset is not in zipped format?**\nAlt Ques: How can I maintain the directory structure of the dataset while downloading using kaggle api cli?",
      "votes": null
    },
    {
      "id": "835581",
      "postDate": "05/06/2020 11:09:22",
      "content": "<p>Seems like an earlier version of Kaggle API works by default on Colab.\n<code>!kaggle -v</code> prints: Kaggle API 1.5.4</p>\n\n<p>So I updated the api\n<code>\n!pip uninstall -y kaggle\n!pip install --upgrade pip\n!pip install kaggle==1.5.6\n</code>\nNow <code>!kaggle -v</code> prints Kaggle API 1.5.6\nThis allows me to download the dataset in zipped format.</p>",
      "rawMarkdown": "Seems like an earlier version of Kaggle API works by default on Colab.\n`!kaggle -v` prints: Kaggle API 1.5.4\n\nSo I updated the api\n```\n!pip uninstall -y kaggle\n!pip install --upgrade pip\n!pip install kaggle==1.5.6\n```\nNow `!kaggle -v` prints Kaggle API 1.5.6\nThis allows me to download the dataset in zipped format.",
      "votes": null
    },
    {
      "id": "912285",
      "postDate": "07/02/2020 11:39:58",
      "content": "<p>Hi, have you been able to use Google Colab?\nIt seems disk storage is a big issue there.</p>\n\n<p>Only ~70 GB avaible in total. 32 GB usable</p>",
      "rawMarkdown": "Hi, have you been able to use Google Colab?\nIt seems disk storage is a big issue there.\n\nOnly ~70 GB avaible in total. 32 GB usable",
      "votes": null
    },
    {
      "id": "912429",
      "postDate": "07/02/2020 13:21:01",
      "content": "<p>No.\nIf you use GPU on colab, the disk space is very less (32gb). It is not possible to download and unzip the entire dataset directly.\nAs a workaround, I saved the zipped file of the dataset on Google Drive, and tried to unzip it into the colab session. Unfortunately, while unzipping the zipped dataset file, there is an error because colab cannot handle the unzipping of very large files (greater than 4-5 GB in my experience). <br>\nSo I do not know if it is even possible to use colab for this competition.</p>",
      "rawMarkdown": "No.\nIf you use GPU on colab, the disk space is very less (32gb). It is not possible to download and unzip the entire dataset directly.\nAs a workaround, I saved the zipped file of the dataset on Google Drive, and tried to unzip it into the colab session. Unfortunately, while unzipping the zipped dataset file, there is an error because colab cannot handle the unzipping of very large files (greater than 4-5 GB in my experience).  \nSo I do not know if it is even possible to use colab for this competition.",
      "votes": null
    },
    {
      "id": "912432",
      "postDate": "07/02/2020 13:22:44",
      "content": "<p>Same here.\nDon't know where people are training their compute heavy models.</p>",
      "rawMarkdown": "Same here.\nDon't know where people are training their compute heavy models.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 835581,
      "author_name": "abhinavtripathi9",
      "author_url": "",
      "post_date": "05/06/2020 11:09:22",
      "content": "<p>Seems like an earlier version of Kaggle API works by default on Colab.\n<code>!kaggle -v</code> prints: Kaggle API 1.5.4</p>\n\n<p>So I updated the api\n<code>\n!pip uninstall -y kaggle\n!pip install --upgrade pip\n!pip install kaggle==1.5.6\n</code>\nNow <code>!kaggle -v</code> prints Kaggle API 1.5.6\nThis allows me to download the dataset in zipped format.</p>",
      "votes": null,
      "replies": [
        {
          "id": 912285,
          "author_name": "prashantkikani",
          "author_url": "",
          "post_date": "07/02/2020 11:39:58",
          "content": "<p>Hi, have you been able to use Google Colab?\nIt seems disk storage is a big issue there.</p>\n\n<p>Only ~70 GB avaible in total. 32 GB usable</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 912429,
          "author_name": "abhinavtripathi9",
          "author_url": "",
          "post_date": "07/02/2020 13:21:01",
          "content": "<p>No.\nIf you use GPU on colab, the disk space is very less (32gb). It is not possible to download and unzip the entire dataset directly.\nAs a workaround, I saved the zipped file of the dataset on Google Drive, and tried to unzip it into the colab session. Unfortunately, while unzipping the zipped dataset file, there is an error because colab cannot handle the unzipping of very large files (greater than 4-5 GB in my experience). <br>\nSo I do not know if it is even possible to use colab for this competition.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 912432,
          "author_name": "prashantkikani",
          "author_url": "",
          "post_date": "07/02/2020 13:22:44",
          "content": "<p>Same here.\nDon't know where people are training their compute heavy models.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "835559": "Hi everyone,\n\nI am new to Kaggle, I recently used all of my GPU trying to train my model, so I tried switching to Colab.  \n I am having problems while downloading the Kaggle dataset for [Steganalysis Challenge](https://www.kaggle.com/c/alaska2-image-steganalysis) in Google Colab.\n\nHere's what I did: \n1. I downloaded the `kaggle.json` file from my account, and uploaded it in Colab.\n2. `!mkdir -p ~/.kaggle`\n3. `!cp kaggle.json ~/.kaggle/`\n4. `!kaggle competitions download -c alaska2-image-steganalysis`\n\nThis 4th command starts to download all the images as `00001.jpg`, `00002.jpg`, and so on.\n\nIssues:\n1. It does not download all the images in the dataset.\n2. BIGGER ISSUE: It does not maintain the directory structure, all images are downloaded without the parent directory.\n\nQues: **How can  I download my entire dataset to Google Colab when the dataset is not in zipped format?**\nAlt Ques: How can I maintain the directory structure of the dataset while downloading using kaggle api cli?",
    "835581": "Seems like an earlier version of Kaggle API works by default on Colab.\n`!kaggle -v` prints: Kaggle API 1.5.4\n\nSo I updated the api\n```\n!pip uninstall -y kaggle\n!pip install --upgrade pip\n!pip install kaggle==1.5.6\n```\nNow `!kaggle -v` prints Kaggle API 1.5.6\nThis allows me to download the dataset in zipped format.",
    "912285": "Hi, have you been able to use Google Colab?\nIt seems disk storage is a big issue there.\n\nOnly ~70 GB avaible in total. 32 GB usable",
    "912429": "No.\nIf you use GPU on colab, the disk space is very less (32gb). It is not possible to download and unzip the entire dataset directly.\nAs a workaround, I saved the zipped file of the dataset on Google Drive, and tried to unzip it into the colab session. Unfortunately, while unzipping the zipped dataset file, there is an error because colab cannot handle the unzipping of very large files (greater than 4-5 GB in my experience).  \nSo I do not know if it is even possible to use colab for this competition.",
    "912432": "Same here.\nDon't know where people are training their compute heavy models."
  },
  "source": "meta"
}