Hello everyone.\nI am trying TPU for the first time, but I stumbled.
\n\nIn TPU, can't I use class_weight inmodel.fit()?\nI get an error with the code below (I don't understand what the error is).
It runs fine on the GPU instead of the TPU. It can also be run on TPU by commenting out class_weight.
Error Code\n\nhistory = model.fit(\n train_dataset, \n epochs = EPOCHS, \n steps_per_epoch = y_train.shape[0] // BATCH_SIZE,\n validation_data = valid_dataset,\n class_weight = {0:1., 1:1./3} # <===== ?????\n)\n\nError message\n\nInvalidArgumentError: {{function_node __inference_distributed_function_363200}} Compilation failure: Detected unsupported operations when trying to compile graph has_valid_nonscalar_shape_true_357243_const_0[] on XLA_TPU_JIT: DenseToDenseSetOperation (No registered 'DenseToDenseSetOperation' OpKernel for XLA_TPU_JIT devices compatible with node {{node has_invalid_dims/DenseToDenseSetOperation}}\n . Registered: device='CPU'; T in [DT_INT8]\n device='CPU'; T in [DT_INT16]\n device='CPU'; T in [DT_INT32]\n device='CPU'; T in [DT_INT64]\n device='CPU'; T in [DT_UINT8]\n device='CPU'; T in [DT_UINT16]\n device='CPU'; T in [DT_STRING]\n){{node has_invalid_dims/DenseToDenseSetOperation}}\n [[has_valid_nonscalar_shape]]\n [[loss/dense_17_loss/weighted_loss/broadcast_weights/assert_broadcastable/is_valid_shape]]\n TPU compilation failed\n [[tpu_compile_succeeded_assert/_12217194444665189181/_6]]\n\nPlease give me some advice. Thank you.
Hello everyone.\nI am trying TPU for the first time, but I stumbled.
\n\nIn TPU, can't I use class_weight inmodel.fit()?\nI get an error with the code below (I don't understand what the error is).
It runs fine on the GPU instead of the TPU. It can also be run on TPU by commenting out class_weight.
Error Code\n\nhistory = model.fit(\n train_dataset, \n epochs = EPOCHS, \n steps_per_epoch = y_train.shape[0] // BATCH_SIZE,\n validation_data = valid_dataset,\n class_weight = {0:1., 1:1./3} # <===== ?????\n)\n\nError message\n\nInvalidArgumentError: {{function_node __inference_distributed_function_363200}} Compilation failure: Detected unsupported operations when trying to compile graph has_valid_nonscalar_shape_true_357243_const_0[] on XLA_TPU_JIT: DenseToDenseSetOperation (No registered 'DenseToDenseSetOperation' OpKernel for XLA_TPU_JIT devices compatible with node {{node has_invalid_dims/DenseToDenseSetOperation}}\n . Registered: device='CPU'; T in [DT_INT8]\n device='CPU'; T in [DT_INT16]\n device='CPU'; T in [DT_INT32]\n device='CPU'; T in [DT_INT64]\n device='CPU'; T in [DT_UINT8]\n device='CPU'; T in [DT_UINT16]\n device='CPU'; T in [DT_STRING]\n){{node has_invalid_dims/DenseToDenseSetOperation}}\n [[has_valid_nonscalar_shape]]\n [[loss/dense_17_loss/weighted_loss/broadcast_weights/assert_broadcastable/is_valid_shape]]\n TPU compilation failed\n [[tpu_compile_succeeded_assert/_12217194444665189181/_6]]\n\nPlease give me some advice. Thank you.
I've run into this same issue on TPUs before. Unfortunately, class weighting doesn't work on TPUs. You'll need to create a weighted loss function.
\n\nSee helpful discussion here: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130272
", "rawMarkdown": "I've run into this same issue on TPUs before. Unfortunately, class weighting doesn't work on TPUs. You'll need to create a weighted loss function.\n\nSee helpful discussion here: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130272", "votes": null }, { "id": "840664", "postDate": "05/10/2020 08:57:15", "content": "Hi Dave!\nThank you for the information.\nI implemented the weighted loss function with reference to the following, and it works without error on TPU.\n* https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130272\n* https://github.com/keras-team/keras/issues/2115
\n\nI'm checking if it's working properly ...\nIn any case, thank you.
", "rawMarkdown": "Hi Dave!\nThank you for the information.\nI implemented the weighted loss function with reference to the following, and it works without error on TPU.\n* https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130272\n* https://github.com/keras-team/keras/issues/2115\n\nI'm checking if it's working properly ...\nIn any case, thank you.", "votes": null }, { "id": "844993", "postDate": "05/13/2020 01:49:33", "content": "Thanks for posting this! Found this helpful 👍
", "rawMarkdown": "Thanks for posting this! Found this helpful 👍", "votes": null }, { "id": "847625", "postDate": "05/14/2020 14:27:32", "content": "Hi Matt!\nIt was able to run without error. However, the expected effect has not been obtained. My model may not be good, but originally, the data of this competition has a poor identification rate, so I think it may be better to use the same number of data for each class.
", "rawMarkdown": "Hi Matt!\nIt was able to run without error. However, the expected effect has not been obtained. My model may not be good, but originally, the data of this competition has a poor identification rate, so I think it may be better to use the same number of data for each class.", "votes": null } ], "comments": [ { "id": 840257, "author_name": "davelo12", "author_url": "", "post_date": "05/09/2020 21:35:57", "content": "I've run into this same issue on TPUs before. Unfortunately, class weighting doesn't work on TPUs. You'll need to create a weighted loss function.
\n\nSee helpful discussion here: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130272
", "votes": null, "replies": [ { "id": 840664, "author_name": "amanooo", "author_url": "", "post_date": "05/10/2020 08:57:15", "content": "Hi Dave!\nThank you for the information.\nI implemented the weighted loss function with reference to the following, and it works without error on TPU.\n* https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130272\n* https://github.com/keras-team/keras/issues/2115
\n\nI'm checking if it's working properly ...\nIn any case, thank you.
", "votes": null, "replies": [] } ] }, { "id": 844993, "author_name": "yeayates21", "author_url": "", "post_date": "05/13/2020 01:49:33", "content": "Thanks for posting this! Found this helpful 👍
", "votes": null, "replies": [ { "id": 847625, "author_name": "amanooo", "author_url": "", "post_date": "05/14/2020 14:27:32", "content": "Hi Matt!\nIt was able to run without error. However, the expected effect has not been obtained. My model may not be good, but originally, the data of this competition has a poor identification rate, so I think it may be better to use the same number of data for each class.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "839560": "Hello everyone.\nI am trying TPU for the first time, but I stumbled.\n\nIn TPU, can't I use `class_weight` in`model.fit()`?\nI get an error with the code below (I don't understand what the error is).\n\nIt **runs fine on the GPU instead of the TPU**. It can also be **run on TPU by commenting out** `class_weight`.\n\n**Error Code**\n```\nhistory = model.fit(\n train_dataset, \n epochs = EPOCHS, \n steps_per_epoch = y_train.shape[0] // BATCH_SIZE,\n validation_data = valid_dataset,\n class_weight = {0:1., 1:1./3} # <===== ?????\n)\n```\n**Error message**\n```\nInvalidArgumentError: {{function_node __inference_distributed_function_363200}} Compilation failure: Detected unsupported operations when trying to compile graph has_valid_nonscalar_shape_true_357243_const_0[] on XLA_TPU_JIT: DenseToDenseSetOperation (No registered 'DenseToDenseSetOperation' OpKernel for XLA_TPU_JIT devices compatible with node {{node has_invalid_dims/DenseToDenseSetOperation}}\n\t. Registered: device='CPU'; T in [DT_INT8]\n device='CPU'; T in [DT_INT16]\n device='CPU'; T in [DT_INT32]\n device='CPU'; T in [DT_INT64]\n device='CPU'; T in [DT_UINT8]\n device='CPU'; T in [DT_UINT16]\n device='CPU'; T in [DT_STRING]\n){{node has_invalid_dims/DenseToDenseSetOperation}}\n\t [[has_valid_nonscalar_shape]]\n\t [[loss/dense_17_loss/weighted_loss/broadcast_weights/assert_broadcastable/is_valid_shape]]\n\tTPU compilation failed\n\t [[tpu_compile_succeeded_assert/_12217194444665189181/_6]]\n```\nPlease give me some advice. Thank you.", "840257": "I've run into this same issue on TPUs before. Unfortunately, class weighting doesn't work on TPUs. You'll need to create a weighted loss function.\n\nSee helpful discussion here: https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130272", "840664": "Hi Dave!\nThank you for the information.\nI implemented the weighted loss function with reference to the following, and it works without error on TPU.\n* https://www.kaggle.com/c/flower-classification-with-tpus/discussion/130272\n* https://github.com/keras-team/keras/issues/2115\n\nI'm checking if it's working properly ...\nIn any case, thank you.", "844993": "Thanks for posting this! Found this helpful 👍", "847625": "Hi Matt!\nIt was able to run without error. However, the expected effect has not been obtained. My model may not be good, but originally, the data of this competition has a poor identification rate, so I think it may be better to use the same number of data for each class." }, "source": "meta" }