Error when I switch to bi_binary_tempered_logistic_loss
\nThere is no problem using CPU and GPU. The following problems occur when using TPU. What is the problem?
UnavailableError Traceback (most recent call last)
\n in
\n 90 callbacks=[es,model_checkpoint0,model_checkpoint1,model_checkpoint2,model_checkpoint3,model_checkpoint4,reduce],#,swa],
\n 91 epochs=EPOCHS,
\n---> 92 verbose=1)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
\n 64 def _method_wrapper(self, *args, **kwargs):
\n 65 if not self._in_multi_worker_mode(): # pylint: disable=protected-access
\n---> 66 return method(self, *args, **kwargs)
\n 67
\n 68 # Running inside run_distribute_coordinator already.
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
\n 853 context.async_wait()
\n 854 logs = tmp_logs # No error, now safe to assign to logs.
\n--> 855 callbacks.on_train_batch_end(step, logs)
\n 856 epoch_logs = copy.copy(logs)
\n 857
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_train_batch_end(self, batch, logs)
\n 387 \"\"\"
\n 388 if self._should_call_train_batch_hooks:
\n--> 389 logs = self._process_logs(logs)
\n 390 self._call_batch_hook(ModeKeys.TRAIN, 'end', batch, logs=logs)
\n 391
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _process_logs(self, logs)
\n 263 \"\"\"Turns tensors into numpy arrays or Python scalars.\"\"\"
\n 264 if logs:
\n--> 265 return tf_utils.to_numpy_or_python_type(logs)
\n 266 return {}
\n 267
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in to_numpy_or_python_type(tensors)
\n 521 return t # Don't turn ragged or sparse tensors to NumPy.
\n 522
\n--> 523 return nest.map_structure(_to_single_numpy_or_python_type, tensors)
\n 524
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, *kwargs)\n 615 \n 616 return pack_sequence_as(\n--> 617 structure[0], [func(x) for x in entries],
\n 618 expand_composites=expand_composites)
\n 619
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in (.0)
\n 615
\n 616 return pack_sequence_as(
\n--> 617 structure[0], [func(*x) for x in entries],
\n 618 expand_composites=expand_composites)
\n 619
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in _to_single_numpy_or_python_type(t)
\n 517 def _to_single_numpy_or_python_type(t):
\n 518 if isinstance(t, ops.Tensor):
\n--> 519 x = t.numpy()
\n 520 return x.item() if np.ndim(x) == 0 else x
\n 521 return t # Don't turn ragged or sparse tensors to NumPy.
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in numpy(self)
\n 959 \"\"\"
\n 960 # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.
\n--> 961 maybe_arr = self._numpy() # pylint: disable=protected-access
\n 962 return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr
\n 963
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _numpy(self)
\n 927 return self._numpy_internal()
\n 928 except core._NotOkStatusException as e:
\n--> 929 six.raise_from(core._status_to_exception(e.code, e.message), None)
\n 930
\n 931 @property
/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)
\nUnavailableError: Socket closed
\nAdditional GRPC error information:
\n{\"created\":\"@1610252392.294793236\",\"description\":\"Error received from peer ipv4:10.0.0.2:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}
Error when I switch to bi_binary_tempered_logistic_loss
\nThere is no problem using CPU and GPU. The following problems occur when using TPU. What is the problem?
UnavailableError Traceback (most recent call last)
\n in
\n 90 callbacks=[es,model_checkpoint0,model_checkpoint1,model_checkpoint2,model_checkpoint3,model_checkpoint4,reduce],#,swa],
\n 91 epochs=EPOCHS,
\n---> 92 verbose=1)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
\n 64 def _method_wrapper(self, *args, **kwargs):
\n 65 if not self._in_multi_worker_mode(): # pylint: disable=protected-access
\n---> 66 return method(self, *args, **kwargs)
\n 67
\n 68 # Running inside run_distribute_coordinator already.
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
\n 853 context.async_wait()
\n 854 logs = tmp_logs # No error, now safe to assign to logs.
\n--> 855 callbacks.on_train_batch_end(step, logs)
\n 856 epoch_logs = copy.copy(logs)
\n 857
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_train_batch_end(self, batch, logs)
\n 387 \"\"\"
\n 388 if self._should_call_train_batch_hooks:
\n--> 389 logs = self._process_logs(logs)
\n 390 self._call_batch_hook(ModeKeys.TRAIN, 'end', batch, logs=logs)
\n 391
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _process_logs(self, logs)
\n 263 \"\"\"Turns tensors into numpy arrays or Python scalars.\"\"\"
\n 264 if logs:
\n--> 265 return tf_utils.to_numpy_or_python_type(logs)
\n 266 return {}
\n 267
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in to_numpy_or_python_type(tensors)
\n 521 return t # Don't turn ragged or sparse tensors to NumPy.
\n 522
\n--> 523 return nest.map_structure(_to_single_numpy_or_python_type, tensors)
\n 524
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, *kwargs)\n 615 \n 616 return pack_sequence_as(\n--> 617 structure[0], [func(x) for x in entries],
\n 618 expand_composites=expand_composites)
\n 619
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in (.0)
\n 615
\n 616 return pack_sequence_as(
\n--> 617 structure[0], [func(*x) for x in entries],
\n 618 expand_composites=expand_composites)
\n 619
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in _to_single_numpy_or_python_type(t)
\n 517 def _to_single_numpy_or_python_type(t):
\n 518 if isinstance(t, ops.Tensor):
\n--> 519 x = t.numpy()
\n 520 return x.item() if np.ndim(x) == 0 else x
\n 521 return t # Don't turn ragged or sparse tensors to NumPy.
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in numpy(self)
\n 959 \"\"\"
\n 960 # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.
\n--> 961 maybe_arr = self._numpy() # pylint: disable=protected-access
\n 962 return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr
\n 963
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _numpy(self)
\n 927 return self._numpy_internal()
\n 928 except core._NotOkStatusException as e:
\n--> 929 six.raise_from(core._status_to_exception(e.code, e.message), None)
\n 930
\n 931 @property
/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)
\nUnavailableError: Socket closed
\nAdditional GRPC error information:
\n{\"created\":\"@1610252392.294793236\",\"description\":\"Error received from peer ipv4:10.0.0.2:8470\",\"file\":\"external/com_github_grpc_grpc/src/core/lib/surface/call.cc\",\"file_line\":1056,\"grpc_message\":\"Socket closed\",\"grpc_status\":14}
Check if there is something wrong with your callbacks. I got a similar error when using a callback for tensorboard which isn't supported on Kaggle
", "rawMarkdown": "Check if there is something wrong with your callbacks. I got a similar error when using a callback for tensorboard which isn't supported on Kaggle", "votes": null }, { "id": "1148095", "postDate": "01/10/2021 22:15:28", "content": "Oh no!! TPU is so hard to debug👀
", "rawMarkdown": "Oh no!! TPU is so hard to debug👀", "votes": null }, { "id": "1149552", "postDate": "01/11/2021 23:59:26", "content": "It is a Lookahead problem. When using TPU, you may not be able to use Lookahead that comes with tf
", "rawMarkdown": "It is a Lookahead problem. When using TPU, you may not be able to use Lookahead that comes with tf", "votes": null } ], "comments": [ { "id": 1147831, "author_name": "mohitve", "author_url": "", "post_date": "01/10/2021 18:24:37", "content": "Check if there is something wrong with your callbacks. I got a similar error when using a callback for tensorboard which isn't supported on Kaggle
", "votes": null, "replies": [ { "id": 1149552, "author_name": "zhangeng", "author_url": "", "post_date": "01/11/2021 23:59:26", "content": "It is a Lookahead problem. When using TPU, you may not be able to use Lookahead that comes with tf
", "votes": null, "replies": [] } ] }, { "id": 1148095, "author_name": "underwearfitting", "author_url": "", "post_date": "01/10/2021 22:15:28", "content": "Oh no!! TPU is so hard to debug👀
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1146807": "Error when I switch to bi_binary_tempered_logistic_loss\nThere is no problem using CPU and GPU. The following problems occur when using TPU. What is the problem?\n\n\n\n\n---------------------------------------------------------------------------\nUnavailableError Traceback (most recent call last)\n