I am building the pipeline to train model on tpu but end up getting this error please help.
\nException in device=TPU:0: /pytorch/xla/third_party/tensorflow/bazel-tensorflow/tensorflow/compiler/xla/xla_client/debug_macros.h:27 : Check failed: status.status() == ::tensorflow::Status::OK() (Invalid argument: Automatic shape inference not supported: bf16[64,81313] and bf16[64,81313,1] vs. OK)
\n*** Begin stack trace ***
\n tensorflow::CurrentStackTrace()
\n xla::Shape const* ConsumeValue(stream_executor::port::StatusOr&&)
\n torch_xla::XlaHelpers::ShapeOfXlaOp(xla::XlaOp)
torch_xla::BuildNllLoss(xla::XlaOp, xla::XlaOp, xla::XlaOp, int, torch_xla::ReductionMode)\n\ntorch_xla::ir::ops::InferOutputShape(absl::lts_2020_02_25::Span<xla::Shape const>, std::function<xla::XlaOp (absl::lts_2020_02_25::Span<xla::XlaOp const>)> const&)\n\ntorch_xla::ir::Node::GetOpShape(std::function<xla::Shape ()> const&) const\ntorch_xla::ir::Node::Node(torch_xla::ir::OpKind, absl::lts_2020_02_25::Span<torch_xla::ir::Value const>, std::function<xla::Shape ()> const&, unsigned long, absl::lts_2020_02_25::uint128)\ntorch_xla::ir::ops::NllLoss::NllLoss(torch_xla::ir::Value const&, torch_xla::ir::Value const&, absl::lts_2020_02_25::optional<torch_xla::ir::Value> const&, torch_xla::ReductionMode, int)\ntorch_xla::XLATensor::nll_loss(torch_xla::XLATensor const&, torch_xla::XLATensor const&, torch_xla::XLATensor const&, long, int)\ntorch_xla::AtenXlaType::nll_loss_forward(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long)\nc10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<std::tuple<at::Tensor, at::Tensor> (*)(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long), std::tuple<at::Tensor, at::Tensor>, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long> >, std::tuple<at::Tensor, at::Tensor> (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long)\nI am building the pipeline to train model on tpu but end up getting this error please help.
\nException in device=TPU:0: /pytorch/xla/third_party/tensorflow/bazel-tensorflow/tensorflow/compiler/xla/xla_client/debug_macros.h:27 : Check failed: status.status() == ::tensorflow::Status::OK() (Invalid argument: Automatic shape inference not supported: bf16[64,81313] and bf16[64,81313,1] vs. OK)
\n*** Begin stack trace ***
\n tensorflow::CurrentStackTrace()
\n xla::Shape const* ConsumeValue(stream_executor::port::StatusOr&&)
\n torch_xla::XlaHelpers::ShapeOfXlaOp(xla::XlaOp)
torch_xla::BuildNllLoss(xla::XlaOp, xla::XlaOp, xla::XlaOp, int, torch_xla::ReductionMode)\n\ntorch_xla::ir::ops::InferOutputShape(absl::lts_2020_02_25::Span<xla::Shape const>, std::function<xla::XlaOp (absl::lts_2020_02_25::Span<xla::XlaOp const>)> const&)\n\ntorch_xla::ir::Node::GetOpShape(std::function<xla::Shape ()> const&) const\ntorch_xla::ir::Node::Node(torch_xla::ir::OpKind, absl::lts_2020_02_25::Span<torch_xla::ir::Value const>, std::function<xla::Shape ()> const&, unsigned long, absl::lts_2020_02_25::uint128)\ntorch_xla::ir::ops::NllLoss::NllLoss(torch_xla::ir::Value const&, torch_xla::ir::Value const&, absl::lts_2020_02_25::optional<torch_xla::ir::Value> const&, torch_xla::ReductionMode, int)\ntorch_xla::XLATensor::nll_loss(torch_xla::XLATensor const&, torch_xla::XLATensor const&, torch_xla::XLATensor const&, long, int)\ntorch_xla::AtenXlaType::nll_loss_forward(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long)\nc10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoRuntimeFunctor_<std::tuple<at::Tensor, at::Tensor> (*)(at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long), std::tuple<at::Tensor, at::Tensor>, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long> >, std::tuple<at::Tensor, at::Tensor> (at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long)\nThe most eye-catching part of the error for me is
\nAutomatic shape inference not supported\nTPU's generally require explicit shapes. Without seeing your code, I would suggest looking at parts of your code where images are read and call a reshape on it to explicitly state the shape. I had roughly the same error once and fixed it this way.
\nThis would look roughly like this:
image = tf.io.decode_jpeg(file_path) # You know this image is 256x256, but the TPU doesn't!\nimage = tf.reshape(image, [256, 256, 3]) # Now the TPU knows it is 256x256!\nclass Dataset:
\n def init(self, df):
\n self.df = df
\n self.transforms = transforms.Compose(transforms.Resize((256,256)))
def __len__(self):\n return len(self.df)\n\ndef __getitem__(self, index):\n label = self.df.iloc[index].landmark_id\n filename = self.df.iloc[index].path\n image = Image.open(filename).convert('L')\n image = cv2.resize(np.array(image), dsize=(256,256)) \n data_tensor = torch.tensor(image).float().unsqueeze(0)\n #data_tensor = data_tensor.permute(2, 0, 1)\n\n return (data_tensor, torch.tensor(label))\nThis is my Dataset class. Please have a look.
", "rawMarkdown": "class Dataset:\n def __init__(self, df):\n self.df = df\n self.transforms = transforms.Compose(transforms.Resize((256,256)))\n \n def __len__(self):\n return len(self.df)\n\n def __getitem__(self, index):\n label = self.df.iloc[index].landmark_id\n filename = self.df.iloc[index].path\n image = Image.open(filename).convert('L')\n image = cv2.resize(np.array(image), dsize=(256,256)) \n data_tensor = torch.tensor(image).float().unsqueeze(0)\n #data_tensor = data_tensor.permute(2, 0, 1)\n \n return (data_tensor, torch.tensor(label))\n\n\nThis is my Dataset class. Please have a look.", "votes": null }, { "id": "1484866", "postDate": "08/21/2021 16:14:08", "content": "I have little experience with Pytorch, but you can try the following:
\ndef __getitem__(self, index):\n label = self.df.iloc[index].landmark_id\n filename = self.df.iloc[index].path\n image = Image.open(filename).convert('L')\n image = cv2.resize(np.array(image), dsize=(256,256)) \n data_tensor = torch.tensor(image).float()\n # Explicitly state the shape so the compiler can inference it\n data_tensor = torch.reshape(data_tensor, (256,256,3)).unsqueeze(0)\n #data_tensor = data_tensor.permute(2, 0, 1)\n\n return (data_tensor, torch.tensor(label))\nHowever, looking more closely at your code, I think the error is in your label. Your output layer has 81313 neurons, meaning the maximum label value is 81312 (zero based). The landmark_id is however not continuous, resulting in label values far larger than 81312. You should create continuous label values as follows.
train['label'] = train['landmark_id'].astype('category').cat.codes\nAnother quick way to check if that's the error is to hardcode to label to 0, thus replacing label = self.df.iloc[index].landmark_id with label = 0
What also surprises me is the combination of Pytorch and Tensorflow, you seem to use both frameworks. This is possible, but its asking for weird errors in practice.
\nHope this helps!
\nP.S. I am working on a training notebook as well, planning on publishing it somewhere next week.
", "rawMarkdown": "I have little experience with Pytorch, but you can try the following:\n\n```\ndef __getitem__(self, index):\n label = self.df.iloc[index].landmark_id\n filename = self.df.iloc[index].path\n image = Image.open(filename).convert('L')\n image = cv2.resize(np.array(image), dsize=(256,256)) \n data_tensor = torch.tensor(image).float()\n # Explicitly state the shape so the compiler can inference it\n data_tensor = torch.reshape(data_tensor, (256,256,3)).unsqueeze(0)\n #data_tensor = data_tensor.permute(2, 0, 1)\n\n return (data_tensor, torch.tensor(label))\n```\n\nHowever, looking more closely at your code, I think the error is in your label. Your output layer has 81313 neurons, meaning the maximum label value is 81312 (zero based). The `landmark_id` is however not continuous, resulting in label values far larger than 81312. You should create continuous label values as follows.\n\n```\ntrain['label'] = train['landmark_id'].astype('category').cat.codes\n```\n\nAnother quick way to check if that's the error is to hardcode to label to 0, thus replacing `label = self.df.iloc[index].landmark_id` with `label = 0`\n\nWhat also surprises me is the combination of Pytorch and Tensorflow, you seem to use both frameworks. This is possible, but its asking for weird errors in practice.\n\nHope this helps!\n\nP.S. I am working on a training notebook as well, planning on publishing it somewhere next week.", "votes": null }, { "id": "1485101", "postDate": "08/21/2021 19:30:11", "content": "One last thought, try to print the labels in your dataset using the following:
\nimages, labels = next(iter(your_dataset))\nprint(labels[0])\nIt looks like your labels are wrapped as the error message states bf16[64,81313] and bf16[64,81313,1]. Thus your labels might be [42] instead of just 42, unsqueezing the label in the dataloader should do the trick if this is the case.
The most eye-catching part of the error for me is
\nAutomatic shape inference not supported\nTPU's generally require explicit shapes. Without seeing your code, I would suggest looking at parts of your code where images are read and call a reshape on it to explicitly state the shape. I had roughly the same error once and fixed it this way.
\nThis would look roughly like this:
image = tf.io.decode_jpeg(file_path) # You know this image is 256x256, but the TPU doesn't!\nimage = tf.reshape(image, [256, 256, 3]) # Now the TPU knows it is 256x256!\nclass Dataset:
\n def init(self, df):
\n self.df = df
\n self.transforms = transforms.Compose(transforms.Resize((256,256)))
def __len__(self):\n return len(self.df)\n\ndef __getitem__(self, index):\n label = self.df.iloc[index].landmark_id\n filename = self.df.iloc[index].path\n image = Image.open(filename).convert('L')\n image = cv2.resize(np.array(image), dsize=(256,256)) \n data_tensor = torch.tensor(image).float().unsqueeze(0)\n #data_tensor = data_tensor.permute(2, 0, 1)\n\n return (data_tensor, torch.tensor(label))\nThis is my Dataset class. Please have a look.
", "votes": null, "replies": [] }, { "id": 1484866, "author_name": "markwijkhuizen", "author_url": "", "post_date": "08/21/2021 16:14:08", "content": "I have little experience with Pytorch, but you can try the following:
\ndef __getitem__(self, index):\n label = self.df.iloc[index].landmark_id\n filename = self.df.iloc[index].path\n image = Image.open(filename).convert('L')\n image = cv2.resize(np.array(image), dsize=(256,256)) \n data_tensor = torch.tensor(image).float()\n # Explicitly state the shape so the compiler can inference it\n data_tensor = torch.reshape(data_tensor, (256,256,3)).unsqueeze(0)\n #data_tensor = data_tensor.permute(2, 0, 1)\n\n return (data_tensor, torch.tensor(label))\nHowever, looking more closely at your code, I think the error is in your label. Your output layer has 81313 neurons, meaning the maximum label value is 81312 (zero based). The landmark_id is however not continuous, resulting in label values far larger than 81312. You should create continuous label values as follows.
train['label'] = train['landmark_id'].astype('category').cat.codes\nAnother quick way to check if that's the error is to hardcode to label to 0, thus replacing label = self.df.iloc[index].landmark_id with label = 0
What also surprises me is the combination of Pytorch and Tensorflow, you seem to use both frameworks. This is possible, but its asking for weird errors in practice.
\nHope this helps!
\nP.S. I am working on a training notebook as well, planning on publishing it somewhere next week.
", "votes": null, "replies": [] }, { "id": 1485101, "author_name": "markwijkhuizen", "author_url": "", "post_date": "08/21/2021 19:30:11", "content": "One last thought, try to print the labels in your dataset using the following:
\nimages, labels = next(iter(your_dataset))\nprint(labels[0])\nIt looks like your labels are wrapped as the error message states bf16[64,81313] and bf16[64,81313,1]. Thus your labels might be [42] instead of just 42, unsqueezing the label in the dataloader should do the trick if this is the case.