I have been trying to use TPU to train , and getting GCS Paths of datasets isn't working only for a few SPECIFIC Datasets. It's working fine for the rest. For example ,
\nBLUE_TILE_DIR = KaggleDatasets().get_gcs_path(\"human-protein-atlas-blue-cell-tile-dataset\")
works perfectly fine. But in the same notebook ,
\nYELLOW_TILE_DIR = KaggleDatasets().get_gcs_path('human-protein-atlas-yellow-cell-tile-dataset')
gives the following error -
\n---------------------------------------------------------------------------\nHTTPError Traceback (most recent call last)\n~/.local/lib/python3.7/site-packages/kaggle_web_client.py in make_post_request(self, data, endpoint, timeout)\n 45 try:\n---> 46 with urllib.request.urlopen(req, timeout=timeout) as response:\n 47 response_json = json.loads(response.read())\n\n/opt/conda/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)\n 221 opener = _opener\n--> 222 return opener.open(url, data, timeout)\n 223 \n\n/opt/conda/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout)\n 530 meth = getattr(processor, meth_name)\n--> 531 response = meth(req, response)\n 532 \n\n/opt/conda/lib/python3.7/urllib/request.py in http_response(self, request, response)\n 640 response = self.parent.error(\n--> 641 'http', request, response, code, msg, hdrs)\n 642 \n\n/opt/conda/lib/python3.7/urllib/request.py in error(self, proto, *args)\n 568 args = (dict, 'default', 'http_error_default') + orig_args\n--> 569 return self._call_chain(*args)\n 570 \n\n/opt/conda/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)\n 502 func = getattr(handler, meth_name)\n--> 503 result = func(*args)\n 504 if result is not None:\n\n/opt/conda/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)\n 648 def http_error_default(self, req, fp, code, msg, hdrs):\n--> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp)\n 650 \n\nHTTPError: HTTP Error 502: Bad Gateway\n\nThe above exception was the direct cause of the following exception:\n\nConnectionError Traceback (most recent call last)\n<ipython-input-11-85cfb3a1672e> in <module>\n----> 1 YELLOW_TILE_DIR = KaggleDatasets().get_gcs_path('human-protein-atlas-yellow-cell-tile-dataset')\n\n~/.local/lib/python3.7/site-packages/kaggle_datasets.py in get_gcs_path(self, dataset_dir)\n 22 'IntegrationType': integration_type,\n 23 }\n---> 24 result = self.web_client.make_post_request(data, self.GET_GCS_PATH_ENDPOINT, self.TIMEOUT_SECS)\n 25 return result['destinationBucket']\n\n~/.local/lib/python3.7/site-packages/kaggle_web_client.py in make_post_request(self, data, endpoint, timeout)\n 57 'Timeout error trying to communicate with service. Please ensure internet is on.') from e\n 58 raise ConnectionError(\n---> 59 'Connection error trying to communicate with service.') from e\n 60 except HTTPError as e:\n 61 if e.code == 401 or e.code == 403:\n\nConnectionError: Connection error trying to communicate with service.\nInternet is on , TPU is enabled. Datasets are added to the notebook's input. This is really a huge block for me , please let me know if there are any workarounds to this. Thank you!
", "messages": [ { "id": 1267071, "postDate": "2021-04-08T09:50:58.397Z", "content": "Update - Restarting internet on the notebook managed to make it work. However now , I am not able to open any files in the GS Dataset. I double checked the path , it is correct. Tried using gsutil ls on the GS path , it is listing all the files present in the directory properly. But when I try to open a single file from the path , it says file not found. :(
\nUpdate 2 - Fixed the issue. I was reading the files from the GCS bucket, using PIL to open the files. Which is not really a good way to read files from GCS , once I used tf.io.read_file(), it was working properly.
", "rawMarkdown": "Update - Restarting internet on the notebook managed to make it work. However now , I am not able to open any files in the GS Dataset. I double checked the path , it is correct. Tried using gsutil ls on the GS path , it is listing all the files present in the directory properly. But when I try to open a single file from the path , it says file not found. :(\n\nUpdate 2 - Fixed the issue. I was reading the files from the GCS bucket, using PIL to open the files. Which is not really a good way to read files from GCS , once I used tf.io.read_file(), it was working properly.", "votes": 2 }, { "id": 1266969, "postDate": "2021-04-08T08:01:12.887Z", "content": "I have been trying to use TPU to train , and getting GCS Paths of datasets isn't working only for a few SPECIFIC Datasets. It's working fine for the rest. For example ,
\nBLUE_TILE_DIR = KaggleDatasets().get_gcs_path(\"human-protein-atlas-blue-cell-tile-dataset\")
works perfectly fine. But in the same notebook ,
\nYELLOW_TILE_DIR = KaggleDatasets().get_gcs_path('human-protein-atlas-yellow-cell-tile-dataset')
gives the following error -
\n---------------------------------------------------------------------------\nHTTPError Traceback (most recent call last)\n~/.local/lib/python3.7/site-packages/kaggle_web_client.py in make_post_request(self, data, endpoint, timeout)\n 45 try:\n---> 46 with urllib.request.urlopen(req, timeout=timeout) as response:\n 47 response_json = json.loads(response.read())\n\n/opt/conda/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)\n 221 opener = _opener\n--> 222 return opener.open(url, data, timeout)\n 223 \n\n/opt/conda/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout)\n 530 meth = getattr(processor, meth_name)\n--> 531 response = meth(req, response)\n 532 \n\n/opt/conda/lib/python3.7/urllib/request.py in http_response(self, request, response)\n 640 response = self.parent.error(\n--> 641 'http', request, response, code, msg, hdrs)\n 642 \n\n/opt/conda/lib/python3.7/urllib/request.py in error(self, proto, *args)\n 568 args = (dict, 'default', 'http_error_default') + orig_args\n--> 569 return self._call_chain(*args)\n 570 \n\n/opt/conda/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)\n 502 func = getattr(handler, meth_name)\n--> 503 result = func(*args)\n 504 if result is not None:\n\n/opt/conda/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)\n 648 def http_error_default(self, req, fp, code, msg, hdrs):\n--> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp)\n 650 \n\nHTTPError: HTTP Error 502: Bad Gateway\n\nThe above exception was the direct cause of the following exception:\n\nConnectionError Traceback (most recent call last)\n<ipython-input-11-85cfb3a1672e> in <module>\n----> 1 YELLOW_TILE_DIR = KaggleDatasets().get_gcs_path('human-protein-atlas-yellow-cell-tile-dataset')\n\n~/.local/lib/python3.7/site-packages/kaggle_datasets.py in get_gcs_path(self, dataset_dir)\n 22 'IntegrationType': integration_type,\n 23 }\n---> 24 result = self.web_client.make_post_request(data, self.GET_GCS_PATH_ENDPOINT, self.TIMEOUT_SECS)\n 25 return result['destinationBucket']\n\n~/.local/lib/python3.7/site-packages/kaggle_web_client.py in make_post_request(self, data, endpoint, timeout)\n 57 'Timeout error trying to communicate with service. Please ensure internet is on.') from e\n 58 raise ConnectionError(\n---> 59 'Connection error trying to communicate with service.') from e\n 60 except HTTPError as e:\n 61 if e.code == 401 or e.code == 403:\n\nConnectionError: Connection error trying to communicate with service.\nInternet is on , TPU is enabled. Datasets are added to the notebook's input. This is really a huge block for me , please let me know if there are any workarounds to this. Thank you!
", "rawMarkdown": "I have been trying to use TPU to train , and getting GCS Paths of datasets isn't working only for a few SPECIFIC Datasets. It's working fine for the rest. For example , \n\n`BLUE_TILE_DIR = KaggleDatasets().get_gcs_path(\"human-protein-atlas-blue-cell-tile-dataset\")`\n\nworks perfectly fine. But in the same notebook , \n\n`YELLOW_TILE_DIR = KaggleDatasets().get_gcs_path('human-protein-atlas-yellow-cell-tile-dataset')`\n\ngives the following error -\n\n```\n---------------------------------------------------------------------------\nHTTPError Traceback (most recent call last)\n~/.local/lib/python3.7/site-packages/kaggle_web_client.py in make_post_request(self, data, endpoint, timeout)\n 45 try:\n---> 46 with urllib.request.urlopen(req, timeout=timeout) as response:\n 47 response_json = json.loads(response.read())\n\n/opt/conda/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)\n 221 opener = _opener\n--> 222 return opener.open(url, data, timeout)\n 223 \n\n/opt/conda/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout)\n 530 meth = getattr(processor, meth_name)\n--> 531 response = meth(req, response)\n 532 \n\n/opt/conda/lib/python3.7/urllib/request.py in http_response(self, request, response)\n 640 response = self.parent.error(\n--> 641 'http', request, response, code, msg, hdrs)\n 642 \n\n/opt/conda/lib/python3.7/urllib/request.py in error(self, proto, *args)\n 568 args = (dict, 'default', 'http_error_default') + orig_args\n--> 569 return self._call_chain(*args)\n 570 \n\n/opt/conda/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)\n 502 func = getattr(handler, meth_name)\n--> 503 result = func(*args)\n 504 if result is not None:\n\n/opt/conda/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)\n 648 def http_error_default(self, req, fp, code, msg, hdrs):\n--> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp)\n 650 \n\nHTTPError: HTTP Error 502: Bad Gateway\n\nThe above exception was the direct cause of the following exception:\n\nConnectionError Traceback (most recent call last)\nUpdate - Restarting internet on the notebook managed to make it work. However now , I am not able to open any files in the GS Dataset. I double checked the path , it is correct. Tried using gsutil ls on the GS path , it is listing all the files present in the directory properly. But when I try to open a single file from the path , it says file not found. :(
\nUpdate 2 - Fixed the issue. I was reading the files from the GCS bucket, using PIL to open the files. Which is not really a good way to read files from GCS , once I used tf.io.read_file(), it was working properly.
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1267071": "Update - Restarting internet on the notebook managed to make it work. However now , I am not able to open any files in the GS Dataset. I double checked the path , it is correct. Tried using gsutil ls on the GS path , it is listing all the files present in the directory properly. But when I try to open a single file from the path , it says file not found. :(\n\nUpdate 2 - Fixed the issue. I was reading the files from the GCS bucket, using PIL to open the files. Which is not really a good way to read files from GCS , once I used tf.io.read_file(), it was working properly.", "1266969": "I have been trying to use TPU to train , and getting GCS Paths of datasets isn't working only for a few SPECIFIC Datasets. It's working fine for the rest. For example , \n\n`BLUE_TILE_DIR = KaggleDatasets().get_gcs_path(\"human-protein-atlas-blue-cell-tile-dataset\")`\n\nworks perfectly fine. But in the same notebook , \n\n`YELLOW_TILE_DIR = KaggleDatasets().get_gcs_path('human-protein-atlas-yellow-cell-tile-dataset')`\n\ngives the following error -\n\n```\n---------------------------------------------------------------------------\nHTTPError Traceback (most recent call last)\n~/.local/lib/python3.7/site-packages/kaggle_web_client.py in make_post_request(self, data, endpoint, timeout)\n 45 try:\n---> 46 with urllib.request.urlopen(req, timeout=timeout) as response:\n 47 response_json = json.loads(response.read())\n\n/opt/conda/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)\n 221 opener = _opener\n--> 222 return opener.open(url, data, timeout)\n 223 \n\n/opt/conda/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout)\n 530 meth = getattr(processor, meth_name)\n--> 531 response = meth(req, response)\n 532 \n\n/opt/conda/lib/python3.7/urllib/request.py in http_response(self, request, response)\n 640 response = self.parent.error(\n--> 641 'http', request, response, code, msg, hdrs)\n 642 \n\n/opt/conda/lib/python3.7/urllib/request.py in error(self, proto, *args)\n 568 args = (dict, 'default', 'http_error_default') + orig_args\n--> 569 return self._call_chain(*args)\n 570 \n\n/opt/conda/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)\n 502 func = getattr(handler, meth_name)\n--> 503 result = func(*args)\n 504 if result is not None:\n\n/opt/conda/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)\n 648 def http_error_default(self, req, fp, code, msg, hdrs):\n--> 649 raise HTTPError(req.full_url, code, msg, hdrs, fp)\n 650 \n\nHTTPError: HTTP Error 502: Bad Gateway\n\nThe above exception was the direct cause of the following exception:\n\nConnectionError Traceback (most recent call last)\n