{
  "id": 261300,
  "title": "how to upload data set to gcp buket? ",
  "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/261300",
  "author_name": "Simon Alerdic",
  "post_date": "2021-08-04T09:50:45.435000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p>How to upload or transfer data to gcp bucket. I've followed this <a href=\"https://www.kaggle.com/paultimothymooney/how-to-move-data-from-kaggle-to-gcs-and-back/notebook\" target=\"_blank\">tutorials </a> but I'm not sure how to handle the following structure data set. </p>\n<pre><code>storage_client = storage.Client(project='carimageimprovement')\n\ndef create_bucket(bucket_name):\n    bucket = storage_client.create_bucket(bucket_name)\n\ndef upload_files(bucket_name, source_folder):\n    bucket = storage_client.get_bucket(bucket_name)\n\n  # my try \n   for dirname, _, filenames in os.walk(source_folder):\n           for filename in filenames:\n                blob = bucket.blob(filename)\n                blob.upload_from_filename(os.path.join(dirname, filename))\n\n  #  for filename in os.listdir(source_folder):\n  #         blob = bucket.blob(filename)\n  #         blob.upload_from_filename(source_folder + filename)\n</code></pre>\n<p>As the data set is in the following structure, it saves all files without ignoring the following structure. How to fix this? </p>\n<pre><code>Training/Validation/Testing\n│\n└─── 00000\n│   │\n│   └─── FLAIR\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T1w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T1wCE\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T2w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ .....\n│   \n└─── 00001\n│   │ ...\n│   \n│ ...   \n│   \n└─── 00002\n│   │ ...\n</code></pre>",
  "messages": [
    {
      "id": 1445709,
      "postDate": "2021-08-04T09:50:45.437Z",
      "content": "<p>How to upload or transfer data to gcp bucket. I've followed this <a href=\"https://www.kaggle.com/paultimothymooney/how-to-move-data-from-kaggle-to-gcs-and-back/notebook\" target=\"_blank\">tutorials </a> but I'm not sure how to handle the following structure data set. </p>\n<pre><code>storage_client = storage.Client(project='carimageimprovement')\n\ndef create_bucket(bucket_name):\n    bucket = storage_client.create_bucket(bucket_name)\n\ndef upload_files(bucket_name, source_folder):\n    bucket = storage_client.get_bucket(bucket_name)\n\n  # my try \n   for dirname, _, filenames in os.walk(source_folder):\n           for filename in filenames:\n                blob = bucket.blob(filename)\n                blob.upload_from_filename(os.path.join(dirname, filename))\n\n  #  for filename in os.listdir(source_folder):\n  #         blob = bucket.blob(filename)\n  #         blob.upload_from_filename(source_folder + filename)\n</code></pre>\n<p>As the data set is in the following structure, it saves all files without ignoring the following structure. How to fix this? </p>\n<pre><code>Training/Validation/Testing\n│\n└─── 00000\n│   │\n│   └─── FLAIR\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T1w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T1wCE\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T2w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ .....\n│   \n└─── 00001\n│   │ ...\n│   \n│ ...   \n│   \n└─── 00002\n│   │ ...\n</code></pre>",
      "rawMarkdown": "How to upload or transfer data to gcp bucket. I've followed this [tutorials ](https://www.kaggle.com/paultimothymooney/how-to-move-data-from-kaggle-to-gcs-and-back/notebook) but I'm not sure how to handle the following structure data set. \n\n```\nstorage_client = storage.Client(project='carimageimprovement')\n\ndef create_bucket(bucket_name):\n    bucket = storage_client.create_bucket(bucket_name)\n    \ndef upload_files(bucket_name, source_folder):\n    bucket = storage_client.get_bucket(bucket_name)\n   \n  # my try \n   for dirname, _, filenames in os.walk(source_folder):\n           for filename in filenames:\n                blob = bucket.blob(filename)\n                blob.upload_from_filename(os.path.join(dirname, filename))\n            \n  #  for filename in os.listdir(source_folder):\n  #         blob = bucket.blob(filename)\n  #         blob.upload_from_filename(source_folder + filename)\n```\n\nAs the data set is in the following structure, it saves all files without ignoring the following structure. How to fix this? \n\n```\nTraining/Validation/Testing\n│\n└─── 00000\n│   │\n│   └─── FLAIR\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T1w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T1wCE\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T2w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ .....\n│   \n└─── 00001\n│   │ ...\n│   \n│ ...   \n│   \n└─── 00002\n│   │ ...\n```",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1445709": "How to upload or transfer data to gcp bucket. I've followed this [tutorials ](https://www.kaggle.com/paultimothymooney/how-to-move-data-from-kaggle-to-gcs-and-back/notebook) but I'm not sure how to handle the following structure data set. \n\n```\nstorage_client = storage.Client(project='carimageimprovement')\n\ndef create_bucket(bucket_name):\n    bucket = storage_client.create_bucket(bucket_name)\n    \ndef upload_files(bucket_name, source_folder):\n    bucket = storage_client.get_bucket(bucket_name)\n   \n  # my try \n   for dirname, _, filenames in os.walk(source_folder):\n           for filename in filenames:\n                blob = bucket.blob(filename)\n                blob.upload_from_filename(os.path.join(dirname, filename))\n            \n  #  for filename in os.listdir(source_folder):\n  #         blob = bucket.blob(filename)\n  #         blob.upload_from_filename(source_folder + filename)\n```\n\nAs the data set is in the following structure, it saves all files without ignoring the following structure. How to fix this? \n\n```\nTraining/Validation/Testing\n│\n└─── 00000\n│   │\n│   └─── FLAIR\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T1w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T1wCE\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ ...\n│   │   \n│   └─── T2w\n│   │   │ Image-1.dcm\n│   │   │ Image-2.dcm\n│   │   │ .....\n│   \n└─── 00001\n│   │ ...\n│   \n│ ...   \n│   \n└─── 00002\n│   │ ...\n```"
  }
}