Hello all,
\nI'm struggling trying to run the great TPU notebook on Colab: https://www.kaggle.com/code/shlomoron/srrf-transformer-tpu-training
\nHas anyone managed to so?
\n(If you made it run on GCP I'm also interested)
\n0 - To load the data I'm using the code found on GCP:
\n os\n sys\n tempfile NamedTemporaryFile\n urllib.request urlopen\n urllib.parse unquote\n urllib.error HTTPError\n zipfile ZipFile\n\nCHUNK_SIZE = \nDATASET_MAPPING = \nKAGGLE_INPUT_PATH=\nKAGGLE_INPUT_SYMLINK=\n\n\nos.makedirs(KAGGLE_INPUT_PATH, exist_ok=)\n os.path.exists(os.path.join(, )):\n os.remove(os.path.join(, ))\nos.symlink(KAGGLE_INPUT_PATH, os.path.join(, ), target_is_directory=)\n\n\nos.makedirs(KAGGLE_INPUT_SYMLINK, exist_ok=)\nos.symlink(KAGGLE_INPUT_PATH, os.path.join(KAGGLE_INPUT_SYMLINK, ), target_is_directory=)\n\n dataset_mapping DATASET_MAPPING.split():\n directory, download_url_encoded = dataset_mapping.split()\n download_url = unquote(download_url_encoded)\n destination_path = os.path.join(KAGGLE_INPUT_PATH, directory)\n :\n urlopen(download_url) zipfileres, NamedTemporaryFile() tfile:\n total_length = zipfileres.headers[]\n ()\n dl = \n data = zipfileres.read(CHUNK_SIZE)\n (data) > :\n dl += (data)\n tfile.write(data)\n done = ( * dl / (total_length))\n sys.stdout.write()\n sys.stdout.flush()\n data = zipfileres.read(CHUNK_SIZE)\n ()\n ZipFile(tfile) zfile:\n zfile.extractall(destination_path)\n HTTPError e:\n ()\n \n OSError e:\n ()\n \n()\n1 - the TPU were not found by strategy, GPT told me to change the code to :
tensorflow tf\n\n\n:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.TPUStrategy(tpu)\n ()\n (, strategy.num_replicas_in_sync)\n ValueError:\n \n strategy = tf.distribute.get_strategy()\n ()\n\n\n2 - I'm getting \"File system scheme '[local]' not implemented in Google Colab TPU\" error and I can't fix.
\nIt fails while loading data in Dataset:
\n ():\n ds = tf.data.TFRecordDataset(\n tffiles, num_parallel_reads=tf.data.AUTOTUNE, compression_type = ).prefetch(tf.data.AUTOTUNE)\n\n ds = ds.(decode_tfrec, tf.data.AUTOTUNE)\n to_filter == :\n ds = ds.(filter_function_1)\n to_filter == :\n ds = ds.(filter_function_2)\n ds = ds.(nan_below_filter, tf.data.AUTOTUNE)\n ds = ds.(concat_target, tf.data.AUTOTUNE)\n\n DEBUG:\n ds = ds.take()\n\n cache:\n ds = ds.cache()\n\n samples_num = \n calculate_sample_num:\n samples_num = ds.reduce(, x,_: x+).numpy() \n\n shuffle:\n shuffle == -:\n ds = ds.shuffle(samples_num, reshuffle_each_iteration = )\n :\n ds = ds.shuffle(shuffle, reshuffle_each_iteration = )\n\n to_repeat:\n ds = ds.repeat()\n\n batch_size:\n ds = ds.padded_batch(\n batch_size, padding_values=(PAD_x, PAD_y), padded_shapes=([X_max_len],[X_max_len, ]), drop_remainder=)\n ds = ds.prefetch(tf.data.AUTOTUNE)\n ds, samples_num\nHello all,
\nI'm struggling trying to run the great TPU notebook on Colab: https://www.kaggle.com/code/shlomoron/srrf-transformer-tpu-training
\nHas anyone managed to so?
\n(If you made it run on GCP I'm also interested)
\n0 - To load the data I'm using the code found on GCP:
\n os\n sys\n tempfile NamedTemporaryFile\n urllib.request urlopen\n urllib.parse unquote\n urllib.error HTTPError\n zipfile ZipFile\n\nCHUNK_SIZE = \nDATASET_MAPPING = \nKAGGLE_INPUT_PATH=\nKAGGLE_INPUT_SYMLINK=\n\n\nos.makedirs(KAGGLE_INPUT_PATH, exist_ok=)\n os.path.exists(os.path.join(, )):\n os.remove(os.path.join(, ))\nos.symlink(KAGGLE_INPUT_PATH, os.path.join(, ), target_is_directory=)\n\n\nos.makedirs(KAGGLE_INPUT_SYMLINK, exist_ok=)\nos.symlink(KAGGLE_INPUT_PATH, os.path.join(KAGGLE_INPUT_SYMLINK, ), target_is_directory=)\n\n dataset_mapping DATASET_MAPPING.split():\n directory, download_url_encoded = dataset_mapping.split()\n download_url = unquote(download_url_encoded)\n destination_path = os.path.join(KAGGLE_INPUT_PATH, directory)\n :\n urlopen(download_url) zipfileres, NamedTemporaryFile() tfile:\n total_length = zipfileres.headers[]\n ()\n dl = \n data = zipfileres.read(CHUNK_SIZE)\n (data) > :\n dl += (data)\n tfile.write(data)\n done = ( * dl / (total_length))\n sys.stdout.write()\n sys.stdout.flush()\n data = zipfileres.read(CHUNK_SIZE)\n ()\n ZipFile(tfile) zfile:\n zfile.extractall(destination_path)\n HTTPError e:\n ()\n \n OSError e:\n ()\n \n()\n1 - the TPU were not found by strategy, GPT told me to change the code to :
tensorflow tf\n\n\n:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.TPUStrategy(tpu)\n ()\n (, strategy.num_replicas_in_sync)\n ValueError:\n \n strategy = tf.distribute.get_strategy()\n ()\n\n\n2 - I'm getting \"File system scheme '[local]' not implemented in Google Colab TPU\" error and I can't fix.
\nIt fails while loading data in Dataset:
\n ():\n ds = tf.data.TFRecordDataset(\n tffiles, num_parallel_reads=tf.data.AUTOTUNE, compression_type = ).prefetch(tf.data.AUTOTUNE)\n\n ds = ds.(decode_tfrec, tf.data.AUTOTUNE)\n to_filter == :\n ds = ds.(filter_function_1)\n to_filter == :\n ds = ds.(filter_function_2)\n ds = ds.(nan_below_filter, tf.data.AUTOTUNE)\n ds = ds.(concat_target, tf.data.AUTOTUNE)\n\n DEBUG:\n ds = ds.take()\n\n cache:\n ds = ds.cache()\n\n samples_num = \n calculate_sample_num:\n samples_num = ds.reduce(, x,_: x+).numpy() \n\n shuffle:\n shuffle == -:\n ds = ds.shuffle(samples_num, reshuffle_each_iteration = )\n :\n ds = ds.shuffle(shuffle, reshuffle_each_iteration = )\n\n to_repeat:\n ds = ds.repeat()\n\n batch_size:\n ds = ds.padded_batch(\n batch_size, padding_values=(PAD_x, PAD_y), padded_shapes=([X_max_len],[X_max_len, ]), drop_remainder=)\n ds = ds.prefetch(tf.data.AUTOTUNE)\n ds, samples_num\nOf course, I develop and train on colab; this notebook is an adaptation of my colab notebook for Kaggle :)
\nThe problem is that you try to load your files on the local vm of colab/GCP. But TPU in colab is accessed as a separate, distant vm. To put it in simple terms, TPU in colab can only access data that is stored on GCP buckets.
\nLuckily, Kaggle datasets are stored on GCP buckets and can be accessed directly from colab TPU.
\nUsing TPU on colab is a bit of an art, but once you grasp it, it's a very powerful tool. I won't go into details on how to do it. Instead, I recommend that you look at my GitHub solution to the Kaggle Sign Language Fingerspelling Recognition competition. Following the two first steps would show you how to work with Kaggle datasets on colab TPU.
Thank you for your insights!
", "rawMarkdown": "Thank you for your insights!", "votes": null } ], "comments": [ { "id": 2505881, "author_name": "shlomoron", "author_url": "", "post_date": "10/30/2023 23:17:06", "content": "Of course, I develop and train on colab; this notebook is an adaptation of my colab notebook for Kaggle :)
\nThe problem is that you try to load your files on the local vm of colab/GCP. But TPU in colab is accessed as a separate, distant vm. To put it in simple terms, TPU in colab can only access data that is stored on GCP buckets.
\nLuckily, Kaggle datasets are stored on GCP buckets and can be accessed directly from colab TPU.
\nUsing TPU on colab is a bit of an art, but once you grasp it, it's a very powerful tool. I won't go into details on how to do it. Instead, I recommend that you look at my GitHub solution to the Kaggle Sign Language Fingerspelling Recognition competition. Following the two first steps would show you how to work with Kaggle datasets on colab TPU.
Thank you for your insights!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2505853": "Hello all, \n\nI'm struggling trying to run the great TPU notebook on Colab: https://www.kaggle.com/code/shlomoron/srrf-transformer-tpu-training\n\nHas anyone managed to so? \n\n(If you made it run on GCP I'm also interested)\n\n0 - To load the data I'm using the code found on GCP:\n\n```python\nimport os\nimport sys\nfrom tempfile import NamedTemporaryFile\nfrom urllib.request import urlopen\nfrom urllib.parse import unquote\nfrom urllib.error import HTTPError\nfrom zipfile import ZipFile\n\nCHUNK_SIZE = 40960\nDATASET_MAPPING = 'stanford-ribonanza-rna-folding:https%3A%2F%2Fstorage.googleapis.com%2Fkaggle-competitions-data%2Fkaggle-v2%2F51294%2F6900467%2Fbundle%2Farchive.zip%3FX-Goog-Algorithm%3DGOOG4-RSA-SHA256%26X-Goog-Credential%3Dgcp-kaggle-com%2540kaggle-161607.iam.gserviceaccount.com%252F20231029%252Fauto%252Fstorage%252Fgoog4_request%26X-Goog-Date%3D20231029T133314Z%26X-Goog-Expires%3D259200%26X-Goog-SignedHeaders%3Dhost%26X-Goog-Signature%3D4081aca3016cb00f6a253499f127b4c4ecbca6b35c645e2c586742525402830f66502d150e7162e8777e19e15f2d93d374520cd1baa75c9981c24d07e7834fe64d887b4d39c26d7372098d7d2acf695d4c0dab9575bfdc1ed23989193f28f66fa6fe6dec8ecf7742cba794d8048addc32edc22223719d8de36de134d78e1b47a8facd7797fcb74540305a23e8cfc9df3f8d192d9a34db18faf54a69bcaf04207d42fbe264008365d3edea94a51b830101a2bd0c58d51d42d210c80ed3f71fe15af27d5d69cee00aa3369de46b99055f730315c705d338bd6820a896c6697a30b4dd81b421ad42e79337c749d1d93489693472dd827a5fea113d5770b20067876,srrf-tfrecords-ds:https%3A%2F%2Fstorage.googleapis.com%2Fkaggle-data-sets%2F3860063%2F6817039%2Fbundle%2Farchive.zip%3FX-Goog-Algorithm%3DGOOG4-RSA-SHA256%26X-Goog-Credential%3Dgcp-kaggle-com%2540kaggle-161607.iam.gserviceaccount.com%252F20231029%252Fauto%252Fstorage%252Fgoog4_request%26X-Goog-Date%3D20231029T133314Z%26X-Goog-Expires%3D259200%26X-Goog-SignedHeaders%3Dhost%26X-Goog-Signature%3D2c4c0c7a2c19ca51a210c82e60c4ea6bf76f9fc67dc21c608f9b32df713d8df77fb36efaf1df087b01b927a4a911477c634a198e5bfff1a326095420a7b834153430e245897fc58e533ae163b31c707131ac3eb90b4f2a4ddb94e6c59011a794d22805e071826598935f7350e3e4b0e5abecc31091204f32687d13a94d9d167ae060670d909a85f5d4b217bc636f1dd0d3d710428d1e06e85ade88db7957ba745832ffbc8bd206c51fbd48bc7bf52b8af24c692a56407cba367ffc237720f9e113e2deae9939becc5078d9779136ba84ca245517b61397a02fa53366db0e2a27bf130c44847a464f578db3a3413bd01ece9587590fc98adf1b1e71bd6b824c12'\nKAGGLE_INPUT_PATH='/home/kaggle/input'\nKAGGLE_INPUT_SYMLINK='/kaggle'\n\n# If directory already exists, this will do nothing, otherwise will create it\nos.makedirs(KAGGLE_INPUT_PATH, exist_ok=True)\nif os.path.exists(os.path.join('..', 'input')):\n os.remove(os.path.join('..', 'input'))\nos.symlink(KAGGLE_INPUT_PATH, os.path.join('..', 'input'), target_is_directory=True)\n\n# Same here\nos.makedirs(KAGGLE_INPUT_SYMLINK, exist_ok=True)\nos.symlink(KAGGLE_INPUT_PATH, os.path.join(KAGGLE_INPUT_SYMLINK, 'input'), target_is_directory=True)\n\nfor dataset_mapping in DATASET_MAPPING.split(','):\n directory, download_url_encoded = dataset_mapping.split(':')\n download_url = unquote(download_url_encoded)\n destination_path = os.path.join(KAGGLE_INPUT_PATH, directory)\n try:\n with urlopen(download_url) as zipfileres, NamedTemporaryFile() as tfile:\n total_length = zipfileres.headers['content-length']\n print(f'Downloading {directory}, {total_length} bytes zipped')\n dl = 0\n data = zipfileres.read(CHUNK_SIZE)\n while len(data) > 0:\n dl += len(data)\n tfile.write(data)\n done = int(50 * dl / int(total_length))\n sys.stdout.write(f\"\\r[{'=' * done}{' ' * (50-done)}] {dl} bytes downloaded\")\n sys.stdout.flush()\n data = zipfileres.read(CHUNK_SIZE)\n print(f'\\nUnzipping {directory}')\n with ZipFile(tfile) as zfile:\n zfile.extractall(destination_path)\n except HTTPError as e:\n print(f'Failed to load (likely expired) {download_url} to path {destination_path}')\n continue\n except OSError as e:\n print(f'Failed to load {download_url} to path {destination_path}')\n continue\nprint('Dataset import complete.')\n```\n\n1 - the TPU were not found by `strategy`, GPT told me to change the code to :\n\n```python\nimport tensorflow as tf\n\n# Define the TPU cluster resolver\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.TPUStrategy(tpu)\n print(\"Using TPU\")\n print(\"Number of replicas:\", strategy.num_replicas_in_sync)\nexcept ValueError:\n # TPU not found, fall back to GPU/CPU strategy\n strategy = tf.distribute.get_strategy()\n print(\"Using default strategy (GPU/CPU)\")\n\n# Now you can build and train your model using the 'strategy' object.\n```\n\n2 - I'm getting \"File system scheme '[local]' not implemented in Google Colab TPU\" error and I can't fix.\n\nIt fails while loading data in Dataset:\n\n```python\ndef get_tfrec_dataset(tffiles, shuffle, batch_size, cache = False, to_filter = False,\n calculate_sample_num = True, to_repeat = False):\n ds = tf.data.TFRecordDataset(\n tffiles, num_parallel_reads=tf.data.AUTOTUNE, compression_type = 'GZIP').prefetch(tf.data.AUTOTUNE)\n\n ds = ds.map(decode_tfrec, tf.data.AUTOTUNE)\n if to_filter == 'filter_1':\n ds = ds.filter(filter_function_1)\n elif to_filter == 'filter_2':\n ds = ds.filter(filter_function_2)\n ds = ds.map(nan_below_filter, tf.data.AUTOTUNE)\n ds = ds.map(concat_target, tf.data.AUTOTUNE)\n\n if DEBUG:\n ds = ds.take(8)\n\n if cache:\n ds = ds.cache()\n\n samples_num = 0\n if calculate_sample_num:\n samples_num = ds.reduce(0, lambda x,_: x+1).numpy() \n\n if shuffle:\n if shuffle == -1:\n ds = ds.shuffle(samples_num, reshuffle_each_iteration = True)\n else:\n ds = ds.shuffle(shuffle, reshuffle_each_iteration = True)\n\n if to_repeat:\n ds = ds.repeat()\n \n if batch_size:\n ds = ds.padded_batch(\n batch_size, padding_values=(PAD_x, PAD_y), padded_shapes=([X_max_len],[X_max_len, 2]), drop_remainder=True)\n ds = ds.prefetch(tf.data.AUTOTUNE)\n return ds, samples_num\n```\n\n\nsee SO: https://stackoverflow.com/questions/62870656/file-system-scheme-local-not-implemented-in-google-colab-tpu", "2505881": "Of course, I develop and train on colab; this notebook is an adaptation of my colab notebook for Kaggle :)\nThe problem is that you try to load your files on the local vm of colab/GCP. But TPU in colab is accessed as a separate, distant vm. To put it in simple terms, TPU in colab can only access data that is stored on GCP buckets.\nLuckily, Kaggle datasets are stored on GCP buckets and can be accessed directly from colab TPU.\nUsing TPU on colab is a bit of an art, but once you grasp it, it's a very powerful tool. I won't go into details on how to do it. Instead, I recommend that you look at my GitHub solution to the Kaggle [Sign Language Fingerspelling Recognition competition](https://github.com/shlomoron/Google---American-Sign-Language-Fingerspelling-Recognition-12th-place-solution). Following the two first steps would show you how to work with Kaggle datasets on colab TPU.", "2506357": "Thank you for your insights!" }, "source": "meta" }