In the past, It show the waiting queue number.
\nNow just traffic ahead, and then show committing.
\nIs the committing time counted for TPU quota or not?
", "messages": [ { "id": "1712829", "postDate": "03/05/2022 11:56:00", "content": "In the past, It show the waiting queue number.
\nNow just traffic ahead, and then show committing.
\nIs the committing time counted for TPU quota or not?
", "rawMarkdown": "In the past, It show the waiting queue number.\n\nNow just traffic ahead, and then show committing.\n\nIs the committing time counted for TPU quota or not?", "votes": null }, { "id": "1712873", "postDate": "03/05/2022 12:57:00", "content": "\n\nIs the committing time counted for TPU quota or not?
\n
No.
\nBy the way, TPU is hot today.
", "rawMarkdown": "> Is the committing time counted for TPU quota or not?\n\nNo. \n\nBy the way, TPU is hot today.", "votes": null }, { "id": "1712965", "postDate": "03/05/2022 14:26:52", "content": "I've had my notebook in queue for 3 hours. 😂
", "rawMarkdown": "I've had my notebook in queue for 3 hours. 😂", "votes": null }, { "id": "1713044", "postDate": "03/05/2022 16:05:30", "content": "interactive notebook:
\nalmost finish, then socket closed??
\n&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
\nccuracy: 0.9998 - val_loss: 9.1697 - val_sparse_categorical_accuracy: 0.4211 - val_sparse_top_k_categorical_accuracy: 0.4925
\nEpoch 28/30
UnavailableError Traceback (most recent call last)
\n/tmp/ipykernel_64/3737550666.py in
\n 4 epochs = config.EPOCHS,
\n 5 callbacks = [snap,get_lr_callback(),train_logger,sv_loss],
\n----> 6 verbose = VERBOSE)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
\n 1139 workers=workers,
\n 1140 use_multiprocessing=use_multiprocessing,
\n-> 1141 return_dict=True)
\n 1142 val_logs = {'val_' + name: val for name, val in val_logs.items()}
\n 1143 epoch_logs.update(val_logs)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in evaluate(self, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, return_dict)
\n 1392 logs = tmp_logs # No error, now safe to assign to logs.
\n 1393 end_step = step + data_handler.step_increment
\n-> 1394 callbacks.on_test_batch_end(end_step, logs)
\n 1395 logs = tf_utils.to_numpy_or_python_type(logs)
\n 1396 callbacks.on_test_end(logs=logs)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_test_batch_end(self, batch, logs)
\n 474 \"\"\"
\n 475 if self._should_call_test_batch_hooks:
\n--> 476 self._call_batch_hook(ModeKeys.TEST, 'end', batch, logs=logs)
\n 477
\n 478 def on_predict_batch_begin(self, batch, logs=None):
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _call_batch_hook(self, mode, hook, batch, logs)
\n 294 self._call_batch_begin_hook(mode, batch, logs)
\n 295 elif hook == 'end':
\n--> 296 self._call_batch_end_hook(mode, batch, logs)
\n 297 else:
\n 298 raise ValueError('Unrecognized hook: {}'.format(hook))
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _call_batch_end_hook(self, mode, batch, logs)
\n 314 self._batch_times.append(batch_time)
\n 315
\n--> 316 self._call_batch_hook_helper(hook_name, batch, logs)
\n 317
\n 318 if len(self._batch_times) >= self._num_batches_for_timing_check:
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _call_batch_hook_helper(self, hook_name, batch, logs)
\n 357 else:
\n 358 if numpy_logs is None: # Only convert once.
\n--> 359 numpy_logs = tf_utils.to_numpy_or_python_type(logs)
\n 360 hook(batch, numpy_logs)
\n 361
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in to_numpy_or_python_type(tensors)
\n 512 return t # Don't turn ragged or sparse tensors to NumPy.
\n 513
\n--> 514 return nest.map_structure(_to_single_numpy_or_python_type, tensors)
\n 515
\n 516
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, *kwargs)\n 657 \n 658 return pack_sequence_as(\n--> 659 structure[0], [func(x) for x in entries],
\n 660 expand_composites=expand_composites)
\n 661
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in (.0)
\n 657
\n 658 return pack_sequence_as(
\n--> 659 structure[0], [func(*x) for x in entries],
\n 660 expand_composites=expand_composites)
\n 661
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in _to_single_numpy_or_python_type(t)
\n 508 def _to_single_numpy_or_python_type(t):
\n 509 if isinstance(t, ops.Tensor):
\n--> 510 x = t.numpy()
\n 511 return x.item() if np.ndim(x) == 0 else x
\n 512 return t # Don't turn ragged or sparse tensors to NumPy.
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in numpy(self)
\n 1069 \"\"\"
\n 1070 # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.
\n-> 1071 maybe_arr = self._numpy() # pylint: disable=protected-access
\n 1072 return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr
\n 1073
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _numpy(self)
\n 1037 return self._numpy_internal()
\n 1038 except core._NotOkStatusException as e: # pylint: disable=protected-access
\n-> 1039 six.raise_from(core._status_to_exception(e.code, e.message), None) # pylint: disable=protected-access
\n 1040
\n 1041 @property
/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)
\nUnavailableError: Socket closed
", "rawMarkdown": "interactive notebook:\n\nalmost finish, then socket closed??\n\n&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&\n\nccuracy: 0.9998 - val_loss: 9.1697 - val_sparse_categorical_accuracy: 0.4211 - val_sparse_top_k_categorical_accuracy: 0.4925\nEpoch 28/30\n259/259 [==============================] - ETA: 0s - loss: 0.0492 - sparse_categorical_accuracy: 0.9972 - sparse_top_k_categorical_accuracy: 0.9999\n---------------------------------------------------------------------------\nUnavailableError Traceback (most recent call last)\n/tmp/ipykernel_64/3737550666.py in\n\nIs the committing time counted for TPU quota or not?
\n
No.
\nBy the way, TPU is hot today.
", "votes": null, "replies": [] }, { "id": 1712965, "author_name": "shawntung", "author_url": "", "post_date": "03/05/2022 14:26:52", "content": "I've had my notebook in queue for 3 hours. 😂
", "votes": null, "replies": [] }, { "id": 1713044, "author_name": "dragonzhang", "author_url": "", "post_date": "03/05/2022 16:05:30", "content": "interactive notebook:
\nalmost finish, then socket closed??
\n&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&
\nccuracy: 0.9998 - val_loss: 9.1697 - val_sparse_categorical_accuracy: 0.4211 - val_sparse_top_k_categorical_accuracy: 0.4925
\nEpoch 28/30
UnavailableError Traceback (most recent call last)
\n/tmp/ipykernel_64/3737550666.py in
\n 4 epochs = config.EPOCHS,
\n 5 callbacks = [snap,get_lr_callback(),train_logger,sv_loss],
\n----> 6 verbose = VERBOSE)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
\n 1139 workers=workers,
\n 1140 use_multiprocessing=use_multiprocessing,
\n-> 1141 return_dict=True)
\n 1142 val_logs = {'val_' + name: val for name, val in val_logs.items()}
\n 1143 epoch_logs.update(val_logs)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in evaluate(self, x, y, batch_size, verbose, sample_weight, steps, callbacks, max_queue_size, workers, use_multiprocessing, return_dict)
\n 1392 logs = tmp_logs # No error, now safe to assign to logs.
\n 1393 end_step = step + data_handler.step_increment
\n-> 1394 callbacks.on_test_batch_end(end_step, logs)
\n 1395 logs = tf_utils.to_numpy_or_python_type(logs)
\n 1396 callbacks.on_test_end(logs=logs)
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_test_batch_end(self, batch, logs)
\n 474 \"\"\"
\n 475 if self._should_call_test_batch_hooks:
\n--> 476 self._call_batch_hook(ModeKeys.TEST, 'end', batch, logs=logs)
\n 477
\n 478 def on_predict_batch_begin(self, batch, logs=None):
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _call_batch_hook(self, mode, hook, batch, logs)
\n 294 self._call_batch_begin_hook(mode, batch, logs)
\n 295 elif hook == 'end':
\n--> 296 self._call_batch_end_hook(mode, batch, logs)
\n 297 else:
\n 298 raise ValueError('Unrecognized hook: {}'.format(hook))
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _call_batch_end_hook(self, mode, batch, logs)
\n 314 self._batch_times.append(batch_time)
\n 315
\n--> 316 self._call_batch_hook_helper(hook_name, batch, logs)
\n 317
\n 318 if len(self._batch_times) >= self._num_batches_for_timing_check:
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _call_batch_hook_helper(self, hook_name, batch, logs)
\n 357 else:
\n 358 if numpy_logs is None: # Only convert once.
\n--> 359 numpy_logs = tf_utils.to_numpy_or_python_type(logs)
\n 360 hook(batch, numpy_logs)
\n 361
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in to_numpy_or_python_type(tensors)
\n 512 return t # Don't turn ragged or sparse tensors to NumPy.
\n 513
\n--> 514 return nest.map_structure(_to_single_numpy_or_python_type, tensors)
\n 515
\n 516
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, *kwargs)\n 657 \n 658 return pack_sequence_as(\n--> 659 structure[0], [func(x) for x in entries],
\n 660 expand_composites=expand_composites)
\n 661
/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in (.0)
\n 657
\n 658 return pack_sequence_as(
\n--> 659 structure[0], [func(*x) for x in entries],
\n 660 expand_composites=expand_composites)
\n 661
/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in _to_single_numpy_or_python_type(t)
\n 508 def _to_single_numpy_or_python_type(t):
\n 509 if isinstance(t, ops.Tensor):
\n--> 510 x = t.numpy()
\n 511 return x.item() if np.ndim(x) == 0 else x
\n 512 return t # Don't turn ragged or sparse tensors to NumPy.
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in numpy(self)
\n 1069 \"\"\"
\n 1070 # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.
\n-> 1071 maybe_arr = self._numpy() # pylint: disable=protected-access
\n 1072 return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr
\n 1073
/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _numpy(self)
\n 1037 return self._numpy_internal()
\n 1038 except core._NotOkStatusException as e: # pylint: disable=protected-access
\n-> 1039 six.raise_from(core._status_to_exception(e.code, e.message), None) # pylint: disable=protected-access
\n 1040
\n 1041 @property
/opt/conda/lib/python3.7/site-packages/six.py in raise_from(value, from_value)
\nUnavailableError: Socket closed
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1712829": "In the past, It show the waiting queue number.\n\nNow just traffic ahead, and then show committing.\n\nIs the committing time counted for TPU quota or not?", "1712873": "> Is the committing time counted for TPU quota or not?\n\nNo. \n\nBy the way, TPU is hot today.", "1712965": "I've had my notebook in queue for 3 hours. 😂", "1713044": "interactive notebook:\n\nalmost finish, then socket closed??\n\n&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&&\n\nccuracy: 0.9998 - val_loss: 9.1697 - val_sparse_categorical_accuracy: 0.4211 - val_sparse_top_k_categorical_accuracy: 0.4925\nEpoch 28/30\n259/259 [==============================] - ETA: 0s - loss: 0.0492 - sparse_categorical_accuracy: 0.9972 - sparse_top_k_categorical_accuracy: 0.9999\n---------------------------------------------------------------------------\nUnavailableError Traceback (most recent call last)\n/tmp/ipykernel_64/3737550666.py in