{
  "id": 201840,
  "title": "`get_gcs_path` for large dataset",
  "url": "/competitions/riiid-test-answer-prediction/discussion/201840",
  "author_name": "Hoang Nguyen",
  "post_date": "2020-12-07T03:22:26.897000",
  "votes": 0,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hey everyone,</p>\n<p>I'm training my model on TPU using 3 relatively big datasets, each ~ 50GBs. While trying to get GCS path for them, I encounter the following error<br>\n`Making a copy of a large dataset may take several minutes. If internet is turned on, please wait several minutes and try again.</p>\n<p>ConnectionError: Connection error trying to communicate with service.`</p>\n<p>If I understand correctly, Kaggle tries to upload the dataset to Google Cloud and keeps it there for a few days, so when I first run get_gcs_path, it's going to take some time for the uploading, which was too long and resulted in the error above. Is that correct? If so,<br>\n(1) how long does it typically take to upload a dataset such big (~50GB)<br>\nand/or<br>\n(2) is there any workaround beside opening a notebook and calling get_gcs_path beforehand?</p>\n<p>Edit 1: Somehow I could not paste the screenshot of the error in the post so I have attached it below.</p>",
  "messages": [
    {
      "id": 1104530,
      "postDate": "2020-12-07T03:22:26.897Z",
      "content": "<p>Hey everyone,</p>\n<p>I'm training my model on TPU using 3 relatively big datasets, each ~ 50GBs. While trying to get GCS path for them, I encounter the following error<br>\n`Making a copy of a large dataset may take several minutes. If internet is turned on, please wait several minutes and try again.</p>\n<p>ConnectionError: Connection error trying to communicate with service.`</p>\n<p>If I understand correctly, Kaggle tries to upload the dataset to Google Cloud and keeps it there for a few days, so when I first run get_gcs_path, it's going to take some time for the uploading, which was too long and resulted in the error above. Is that correct? If so,<br>\n(1) how long does it typically take to upload a dataset such big (~50GB)<br>\nand/or<br>\n(2) is there any workaround beside opening a notebook and calling get_gcs_path beforehand?</p>\n<p>Edit 1: Somehow I could not paste the screenshot of the error in the post so I have attached it below.</p>",
      "rawMarkdown": "Hey everyone,\n\nI'm training my model on TPU using 3 relatively big datasets, each ~ 50GBs. While trying to get GCS path for them, I encounter the following error\n`Making a copy of a large dataset may take several minutes. If internet is turned on, please wait several minutes and try again.\n\nConnectionError: Connection error trying to communicate with service.`\n\nIf I understand correctly, Kaggle tries to upload the dataset to Google Cloud and keeps it there for a few days, so when I first run get_gcs_path, it's going to take some time for the uploading, which was too long and resulted in the error above. Is that correct? If so,\n(1) how long does it typically take to upload a dataset such big (~50GB)\nand/or\n(2) is there any workaround beside opening a notebook and calling get_gcs_path beforehand?\n\nEdit 1: Somehow I could not paste the screenshot of the error in the post so I have attached it below."
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1104530": "Hey everyone,\n\nI'm training my model on TPU using 3 relatively big datasets, each ~ 50GBs. While trying to get GCS path for them, I encounter the following error\n`Making a copy of a large dataset may take several minutes. If internet is turned on, please wait several minutes and try again.\n\nConnectionError: Connection error trying to communicate with service.`\n\nIf I understand correctly, Kaggle tries to upload the dataset to Google Cloud and keeps it there for a few days, so when I first run get_gcs_path, it's going to take some time for the uploading, which was too long and resulted in the error above. Is that correct? If so,\n(1) how long does it typically take to upload a dataset such big (~50GB)\nand/or\n(2) is there any workaround beside opening a notebook and calling get_gcs_path beforehand?\n\nEdit 1: Somehow I could not paste the screenshot of the error in the post so I have attached it below."
  }
}