When I runing my model on TPU, machine, it gets an error in the function fit() , which seems to indicate that a device does not exist, but my TPU has already been initialized normally. How can I solve this problem? Thanks!!
ValueError Traceback (most recent last)\nCell In[], line \n model = NewDCNN_model()\n model.summary()\n---> history = model.fit(\n X_train,\n y_train,\n validation_data=(X_val, y_val),\n epochs=CFG.EPOCHS,\n callbacks=[checkpoint, reduce_lr_loss, estimate],\n batch_size=CFG.BATCH_SIZE,\n verbose=\n )\n model.load_weights(f)\n oof = model.predict(X_val, batch_size = *CFG.BATCH_SIZE)\n\n localpython3.kerastils/traceback_utils.py:, in filter_traceback.<locals>.error_handler(*args, **kwargs)\n filtered_tb = _process_traceback_frames(e.__traceback__)\n # To get the full stack trace, :\n # `keras.config.disable_traceback_filtering()`\n--> raise e.with_traceback(filtered_tb) None\n :\n del filtered_tb\n\n localpython3.tensorflowdistribute/packed_distributed_variable.py:, in PackedDistributedVariable.get_var_on_device(self, device)\n d == device:\n self._distributed_variables[i]\n---> raise ValueError( % device)\n\nValueError: Device replica:device:CPU: is not found\nWhen I runing my model on TPU, machine, it gets an error in the function fit() , which seems to indicate that a device does not exist, but my TPU has already been initialized normally. How can I solve this problem? Thanks!!
ValueError Traceback (most recent last)\nCell In[], line \n model = NewDCNN_model()\n model.summary()\n---> history = model.fit(\n X_train,\n y_train,\n validation_data=(X_val, y_val),\n epochs=CFG.EPOCHS,\n callbacks=[checkpoint, reduce_lr_loss, estimate],\n batch_size=CFG.BATCH_SIZE,\n verbose=\n )\n model.load_weights(f)\n oof = model.predict(X_val, batch_size = *CFG.BATCH_SIZE)\n\n localpython3.kerastils/traceback_utils.py:, in filter_traceback.<locals>.error_handler(*args, **kwargs)\n filtered_tb = _process_traceback_frames(e.__traceback__)\n # To get the full stack trace, :\n # `keras.config.disable_traceback_filtering()`\n--> raise e.with_traceback(filtered_tb) None\n :\n del filtered_tb\n\n localpython3.tensorflowdistribute/packed_distributed_variable.py:, in PackedDistributedVariable.get_var_on_device(self, device)\n d == device:\n self._distributed_variables[i]\n---> raise ValueError( % device)\n\nValueError: Device replica:device:CPU: is not found\nhaving the same issue, one epoch runs just fine after which I get errors. I am now running multiple epochs like this, not ideal so do let me know if you find the proper solution
\nfor _ in range(5):
\n try:
\n history = model.fit(
\n X_train, y_train,
\n validation_data=(X_val, y_val),
\n epochs=1,
\n callbacks=[reduce_lr_loss],
\n batch_size=CFG.BATCH_SIZE,
\n verbose=1,
\n )
\n except:
\n print()
Yes, I am also facing the same problem, but so far I have not been able to find an effective solution.😞
", "rawMarkdown": "Yes, I am also facing the same problem, but so far I have not been able to find an effective solution.😞", "votes": null }, { "id": "2898535", "postDate": "07/01/2024 07:52:50", "content": "Have you tried this? :
\n:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu=) \n \n \n \n strategy = tf.distribute.TPUStrategy(tpu)\n ()\n (, strategy.num_replicas_in_sync)\n:\n strategy = tf.distribute.get_strategy()\nEasy solution: copy public notebook, and load your notebook into it, so you'll be able to use old environment (not latest). For me, 2 gpus works only in latest, while tpu in this old environment.
\nMost likely something with tensorflow or keras versions.
Okay, I will try. Thx! 🙏🏼
", "rawMarkdown": "Okay, I will try. Thx! 🙏🏼", "votes": null } ], "comments": [ { "id": 2892109, "author_name": "mayankdeshwal", "author_url": "", "post_date": "06/27/2024 05:15:46", "content": "having the same issue, one epoch runs just fine after which I get errors. I am now running multiple epochs like this, not ideal so do let me know if you find the proper solution
\nfor _ in range(5):
\n try:
\n history = model.fit(
\n X_train, y_train,
\n validation_data=(X_val, y_val),
\n epochs=1,
\n callbacks=[reduce_lr_loss],
\n batch_size=CFG.BATCH_SIZE,
\n verbose=1,
\n )
\n except:
\n print()
Yes, I am also facing the same problem, but so far I have not been able to find an effective solution.😞
", "votes": null, "replies": [] }, { "id": 2898535, "author_name": "lau01b", "author_url": "", "post_date": "07/01/2024 07:52:50", "content": "Have you tried this? :
\n:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect(tpu=) \n \n \n \n strategy = tf.distribute.TPUStrategy(tpu)\n ()\n (, strategy.num_replicas_in_sync)\n:\n strategy = tf.distribute.get_strategy()\nEasy solution: copy public notebook, and load your notebook into it, so you'll be able to use old environment (not latest). For me, 2 gpus works only in latest, while tpu in this old environment.
\nMost likely something with tensorflow or keras versions.
Okay, I will try. Thx! 🙏🏼
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2887538": "When I runing my model on TPU, machine, it gets an error in the function `fit()` , which seems to indicate that a device does not exist, but my TPU has already been initialized normally. How can I solve this problem? Thanks!!\n\n```\nValueError Traceback (most recent call last)\nCell In[12], line 47\n 44 model = NewDCNN_model()\n 45 model.summary()\n---> 47 history = model.fit(\n 48 X_train,\n 49 y_train,\n 50 validation_data=(X_val, y_val),\n 51 epochs=CFG.EPOCHS,\n 52 callbacks=[checkpoint, reduce_lr_loss, estimate],\n 53 batch_size=CFG.BATCH_SIZE,\n 54 verbose=1\n 55 )\n 56 model.load_weights(f\"model-{fold}.weights.h5\")\n 57 oof = model.predict(X_val, batch_size = 2*CFG.BATCH_SIZE)\n\nFile /usr/local/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py:122, in filter_traceback.