Hi!\nI'm trying to use Tpus in my own dataset. I took the kaggle dataset (nih x rays) and i pass it to TFRecords in my own gcs storage bucket.\nIm avialable to read it and get access to the images and labels from my gcs bucket.\nThe problem comes after i create the model, in fit process. Give me the next error.
\n\n\n\n\n{{function_node __inference_distributed_function_20816}} Compilation failure: Asked to propagate a dynamic dimension from hlo dot.397@{}@1 to hlo %all-reduce.402 = f32[50,2]{1,0} all-reduce(f32[50,2]{1,0} %dot.397), replica_groups={{0,1,2,3,4,5,6,7}}, to_apply=%sum.398, metadata={op_type=\"CrossReplicaSum\" op_name=\"RMSprop/CrossReplicaSum_4\"}, which is not implemented.\n TPU compilation failed\n [[{{node tpu_compile_succeeded_assert/_10007542197504102739/_7}}]]
\n
I'm trying a small net to try to solve it. \nI had the same error in a kaggle notebook and a jupyter notebook from GC AI notebooks .
\n\nMy Tpu connection works fine.
\n\nThis is the net example:
\n\n\n\n\nwith strategy.scope():
\n \nmodel = tf.keras.Sequential([\n tf.keras.layers.Flatten(input_shape=[512, 512, 1]),\n tf.keras.layers.Dense(500, activation=\"relu\"),\n tf.keras.layers.Dense(50, activation=\"relu\"),\n tf.keras.layers.Dense(2, activation='softmax') # classifying into 5 classes\n ])
\n \nmodel.compile(\n optimizer='adam',\n loss= 'categorical_crossentropy',\n metrics=['accuracy']) # % of correct answers
\n \nmodel.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS)
\n
Any tips?\nThanks
", "messages": [ { "id": "756136", "postDate": "02/25/2020 12:49:12", "content": "Hi!\nI'm trying to use Tpus in my own dataset. I took the kaggle dataset (nih x rays) and i pass it to TFRecords in my own gcs storage bucket.\nIm avialable to read it and get access to the images and labels from my gcs bucket.\nThe problem comes after i create the model, in fit process. Give me the next error.
\n\n\n\n\n{{function_node __inference_distributed_function_20816}} Compilation failure: Asked to propagate a dynamic dimension from hlo dot.397@{}@1 to hlo %all-reduce.402 = f32[50,2]{1,0} all-reduce(f32[50,2]{1,0} %dot.397), replica_groups={{0,1,2,3,4,5,6,7}}, to_apply=%sum.398, metadata={op_type=\"CrossReplicaSum\" op_name=\"RMSprop/CrossReplicaSum_4\"}, which is not implemented.\n TPU compilation failed\n [[{{node tpu_compile_succeeded_assert/_10007542197504102739/_7}}]]
\n
I'm trying a small net to try to solve it. \nI had the same error in a kaggle notebook and a jupyter notebook from GC AI notebooks .
\n\nMy Tpu connection works fine.
\n\nThis is the net example:
\n\n\n\n\nwith strategy.scope():
\n \nmodel = tf.keras.Sequential([\n tf.keras.layers.Flatten(input_shape=[512, 512, 1]),\n tf.keras.layers.Dense(500, activation=\"relu\"),\n tf.keras.layers.Dense(50, activation=\"relu\"),\n tf.keras.layers.Dense(2, activation='softmax') # classifying into 5 classes\n ])
\n \nmodel.compile(\n optimizer='adam',\n loss= 'categorical_crossentropy',\n metrics=['accuracy']) # % of correct answers
\n \nmodel.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS)
\n
Any tips?\nThanks
", "rawMarkdown": "Hi!\nI'm trying to use Tpus in my own dataset. I took the kaggle dataset (nih x rays) and i pass it to TFRecords in my own gcs storage bucket.\nIm avialable to read it and get access to the images and labels from my gcs bucket.\nThe problem comes after i create the model, in fit process. Give me the next error.\n\n\n> {{function_node __inference_distributed_function_20816}} Compilation failure: Asked to propagate a dynamic dimension from hlo dot.397@{}@1 to hlo %all-reduce.402 = f32[50,2]{1,0} all-reduce(f32[50,2]{1,0} %dot.397), replica_groups={{0,1,2,3,4,5,6,7}}, to_apply=%sum.398, metadata={op_type=\"CrossReplicaSum\" op_name=\"RMSprop/CrossReplicaSum_4\"}, which is not implemented.\nTPU compilation failed\n[[{{node tpu_compile_succeeded_assert/_10007542197504102739/_7}}]]\n\nI'm trying a small net to try to solve it. \nI had the same error in a kaggle notebook and a jupyter notebook from GC AI notebooks .\n\nMy Tpu connection works fine.\n\nThis is the net example: \n\n\n>with strategy.scope():\n\n> model = tf.keras.Sequential([\ntf.keras.layers.Flatten(input_shape=[512, 512, 1]),\ntf.keras.layers.Dense(500, activation=\"relu\"),\ntf.keras.layers.Dense(50, activation=\"relu\"),\ntf.keras.layers.Dense(2, activation='softmax') # classifying into 5 classes\n ])\n\n>model.compile(\noptimizer='adam',\nloss= 'categorical_crossentropy',\nmetrics=['accuracy']) # % of correct answers\n\n>model.fit(get_training_dataset(), steps_per_epoch=STEPS_PER_EPOCH, epochs=EPOCHS)\n\n\nAny tips?\nThanks", "votes": null }, { "id": "756327", "postDate": "02/25/2020 16:03:54", "content": "I wonder it can be the channel number, is your own dataset are grey scale ?
", "rawMarkdown": "I wonder it can be the channel number, is your own dataset are grey scale ?", "votes": null }, { "id": "756636", "postDate": "02/25/2020 22:59:06", "content": "How did you set up your dataset ? Can you share the code ?\nI concur with @swordfaith below, it might be worth checking into which format your images are decoded.
", "rawMarkdown": "How did you set up your dataset ? Can you share the code ?\nI concur with @swordfaith below, it might be worth checking into which format your images are decoded.", "votes": null }, { "id": "757013", "postDate": "02/26/2020 10:44:02", "content": "I already fix it. Was a problem with the labels. My problem is binary classification, but i would like to use labels [0,1] and [1,0] with a softmax. the reason of that is see the heatmaps of each class after train.\nIf i use binary labels (not one hot) i can train and all its okey.\nI still working to try train with the labels as one hot.
\n\nThe images were not the problem,a i code and decode as pngs and works fine.
", "rawMarkdown": "I already fix it. Was a problem with the labels. My problem is binary classification, but i would like to use labels [0,1] and [1,0] with a softmax. the reason of that is see the heatmaps of each class after train.\nIf i use binary labels (not one hot) i can train and all its okey.\nI still working to try train with the labels as one hot. \n\nThe images were not the problem,a i code and decode as pngs and works fine.", "votes": null }, { "id": "757457", "postDate": "02/26/2020 19:29:51", "content": "In Keras with one-hot labels, use categorical_crossentropy.\nWith numerical labels, use sparse_categorical_crossentropy
Title: 🚨 Urgent: TPU Initialization Fails (OpKernel Missing) Despite Active TPU Runtime
\nHello Kaggle Community,
\nI am facing a persistent issue when trying to initialize the TPU in my notebook. I have already confirmed that:
\nMy account is Verified.\n The Notebook Accelerator is set to TPU.\n My TPU quota is currently available.
\nHowever, the standard initialization code consistently throws a NotFoundError because the required OpKernel is missing. I suspect this is an environment configuration issue on the platform itself.
Has anyone encountered this specific OpKernel not registered error recently while using the TPU runtime and found a workaround?
Code and Error Details\nCode Used:
\n tensorflow tf\n\n\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=)\ntf.tpu.experimental.initialize_tpu_system(tpu)\n\n\nstrategy = tf.distribute.TPUStrategy(tpu)\n()\nTraceback Snippet:
\nInvalidArgumentError: OpKernel was registered support Op used by {{node ConfigureDistributedTPU}}\n.\nRegistered devices: [CPU]\nRegistered kernels:\n < registered kernels>\n\nDuring handling of the above exception, another exception occurred:\nNotFoundError: TPUs found the cluster. Failed initialization: OpKernel was registered support Op .\nKey Observation:\nThe output shows Registered devices: [CPU], confirming that the environment is not detecting the active TPU accelerator at the TensorFlow software level.
Any assistance or known workarounds would be greatly appreciated! Thank you.
", "rawMarkdown": "Title: 🚨 Urgent: TPU Initialization Fails (OpKernel Missing) Despite Active TPU Runtime\n\nHello Kaggle Community,\n\nI am facing a persistent issue when trying to initialize the TPU in my notebook. I have already confirmed that:\n\n My account is Verified.\n The Notebook Accelerator is set to TPU.\n My TPU quota is currently available.\n\nHowever, the standard initialization code consistently throws a `NotFoundError` because the required OpKernel is missing. I suspect this is an environment configuration issue on the platform itself.\n\nHas anyone encountered this specific `OpKernel not registered` error recently while using the TPU runtime and found a workaround?\n\n-----\n\nCode and Error Details\nCode Used:\n```python\nimport tensorflow as tf\n\n# Detect and initialize TPU\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='local')\ntf.tpu.experimental.initialize_tpu_system(tpu)\n\n# Create TPU distribution strategy\nstrategy = tf.distribute.TPUStrategy(tpu)\nprint(\"TPU initialized successfully.\")\n```\n\nTraceback Snippet:\n\n```\nInvalidArgumentError: No OpKernel was registered to support Op 'ConfigureDistributedTPU' used by {{node ConfigureDistributedTPU}}\n...\nRegistered devices: [CPU]\nRegistered kernels:\nTitle: 🚨 Urgent: TPU Initialization Fails (OpKernel Missing) Despite Active TPU Runtime
\nHello Kaggle Community,
\nI am facing a persistent issue when trying to initialize the TPU in my notebook. I have already confirmed that:
\nMy account is Verified.\n The Notebook Accelerator is set to TPU.\n My TPU quota is currently available.
\nHowever, the standard initialization code consistently throws a NotFoundError because the required OpKernel is missing. I suspect this is an environment configuration issue on the platform itself.
Has anyone encountered this specific OpKernel not registered error recently while using the TPU runtime and found a workaround?
Code and Error Details\nCode Used:
\n tensorflow tf\n\n\ntpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=)\ntf.tpu.experimental.initialize_tpu_system(tpu)\n\n\nstrategy = tf.distribute.TPUStrategy(tpu)\n()\nTraceback Snippet:
\nInvalidArgumentError: OpKernel was registered support Op used by {{node ConfigureDistributedTPU}}\n.\nRegistered devices: [CPU]\nRegistered kernels:\n < registered kernels>\n\nDuring handling of the above exception, another exception occurred:\nNotFoundError: TPUs found the cluster. Failed initialization: OpKernel was registered support Op .\nKey Observation:\nThe output shows Registered devices: [CPU], confirming that the environment is not detecting the active TPU accelerator at the TensorFlow software level.
Any assistance or known workarounds would be greatly appreciated! Thank you.
", "votes": null, "replies": [] }, { "id": 756327, "author_name": "swordfaith", "author_url": "", "post_date": "02/25/2020 16:03:54", "content": "I wonder it can be the channel number, is your own dataset are grey scale ?
", "votes": null, "replies": [] }, { "id": 756636, "author_name": "mgorner", "author_url": "", "post_date": "02/25/2020 22:59:06", "content": "How did you set up your dataset ? Can you share the code ?\nI concur with @swordfaith below, it might be worth checking into which format your images are decoded.
", "votes": null, "replies": [] }, { "id": 757013, "author_name": "ianwilkinson", "author_url": "", "post_date": "02/26/2020 10:44:02", "content": "I already fix it. Was a problem with the labels. My problem is binary classification, but i would like to use labels [0,1] and [1,0] with a softmax. the reason of that is see the heatmaps of each class after train.\nIf i use binary labels (not one hot) i can train and all its okey.\nI still working to try train with the labels as one hot.
\n\nThe images were not the problem,a i code and decode as pngs and works fine.
", "votes": null, "replies": [ { "id": 757457, "author_name": "mgorner", "author_url": "", "post_date": "02/26/2020 19:29:51", "content": "In Keras with one-hot labels, use categorical_crossentropy.\nWith numerical labels, use sparse_categorical_crossentropy