{
  "id": 570681,
  "title": "Downloading the Data locally: running for 7 hours and hitting memory errors",
  "url": "/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/570681",
  "author_name": "",
  "post_date": "2025-03-30T00:37:55.243836200Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I've attempted to download the data locally onto my machine via kaggle APIs, unfortunately after 7 hours, I got a memory error:</p>\n<pre><code>!kaggle competitions download -c byu-locating-bacterial-flagellar-motors- -p data/  \n</code></pre>\n<pre><code>File \"C:312-packages.py\", line 902, in content\n    self.content = b\"\".join(self.itercontent(CONTENTCHUNKSIZE)) or b\"\"\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nMemoryError\n</code></pre>\n<p>I have more than enough RAM, a disk space to download…</p>\n<p>How has everyone else gone with downloading the competition data &amp; other external data? </p>\n<p>I am thinking of linking a notebook directly to a GCS bucket instead.</p>",
  "messages": [
    {
      "id": "3162901",
      "postDate": "03/30/2025 00:37:55",
      "content": "<p>I've attempted to download the data locally onto my machine via kaggle APIs, unfortunately after 7 hours, I got a memory error:</p>\n<pre><code>!kaggle competitions download -c byu-locating-bacterial-flagellar-motors- -p data/  \n</code></pre>\n<pre><code>File \"C:312-packages.py\", line 902, in content\n    self.content = b\"\".join(self.itercontent(CONTENTCHUNKSIZE)) or b\"\"\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nMemoryError\n</code></pre>\n<p>I have more than enough RAM, a disk space to download…</p>\n<p>How has everyone else gone with downloading the competition data &amp; other external data? </p>\n<p>I am thinking of linking a notebook directly to a GCS bucket instead.</p>",
      "rawMarkdown": "I've attempted to download the data locally onto my machine via kaggle APIs, unfortunately after 7 hours, I got a memory error:\n\n```python\n!kaggle competitions download -c byu-locating-bacterial-flagellar-motors-2025 -p data/  \n```\n```\nFile \"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\requests\\models.py\", line 902, in content\n    self._content = b\"\".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b\"\"\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nMemoryError\n```\n\nI have more than enough RAM, a disk space to download...\n\nHow has everyone else gone with downloading the competition data & other external data? \n\nI am thinking of linking a notebook directly to a GCS bucket instead.",
      "votes": null
    },
    {
      "id": "3162946",
      "postDate": "03/30/2025 04:21:28",
      "content": "<p>Using the <code>kagglehub</code> package, looks like it'll take anywhere from 10-24 hours thanks to Australian WiFi 🇦🇺 👌</p>",
      "rawMarkdown": "Using the `kagglehub` package, looks like it'll take anywhere from 10-24 hours thanks to Australian WiFi 🇦🇺 👌",
      "votes": null
    },
    {
      "id": "3162969",
      "postDate": "03/30/2025 05:40:58",
      "content": "<p>Downgrade your Kaggle package to 1.6.17 (pip install kaggle==1.6.17).</p>",
      "rawMarkdown": "Downgrade your Kaggle package to 1.6.17 (pip install kaggle==1.6.17).",
      "votes": null
    },
    {
      "id": "3163161",
      "postDate": "03/30/2025 11:20:06",
      "content": "<p>Thank you for this: )  I spent almost 2 hours on Colab and was about to offer a sacrifice to python gods.</p>",
      "rawMarkdown": "Thank you for this: )  I spent almost 2 hours on Colab and was about to offer a sacrifice to python gods.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3162946,
      "author_name": "homiecal",
      "author_url": "",
      "post_date": "03/30/2025 04:21:28",
      "content": "<p>Using the <code>kagglehub</code> package, looks like it'll take anywhere from 10-24 hours thanks to Australian WiFi 🇦🇺 👌</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3162969,
      "author_name": "jeroencottaar",
      "author_url": "",
      "post_date": "03/30/2025 05:40:58",
      "content": "<p>Downgrade your Kaggle package to 1.6.17 (pip install kaggle==1.6.17).</p>",
      "votes": null,
      "replies": [
        {
          "id": 3163161,
          "author_name": "rzatemizel",
          "author_url": "",
          "post_date": "03/30/2025 11:20:06",
          "content": "<p>Thank you for this: )  I spent almost 2 hours on Colab and was about to offer a sacrifice to python gods.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3162901": "I've attempted to download the data locally onto my machine via kaggle APIs, unfortunately after 7 hours, I got a memory error:\n\n```python\n!kaggle competitions download -c byu-locating-bacterial-flagellar-motors-2025 -p data/  \n```\n```\nFile \"C:\\Users\\user\\AppData\\Local\\Programs\\Python\\Python312\\Lib\\site-packages\\requests\\models.py\", line 902, in content\n    self._content = b\"\".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b\"\"\n                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nMemoryError\n```\n\nI have more than enough RAM, a disk space to download...\n\nHow has everyone else gone with downloading the competition data & other external data? \n\nI am thinking of linking a notebook directly to a GCS bucket instead.",
    "3162946": "Using the `kagglehub` package, looks like it'll take anywhere from 10-24 hours thanks to Australian WiFi 🇦🇺 👌",
    "3162969": "Downgrade your Kaggle package to 1.6.17 (pip install kaggle==1.6.17).",
    "3163161": "Thank you for this: )  I spent almost 2 hours on Colab and was about to offer a sacrifice to python gods."
  },
  "source": "meta"
}