{
  "id": 163535,
  "title": "How to Implement Custom Metric on TPU?",
  "url": "/competitions/alaska2-image-steganalysis/discussion/163535",
  "author_name": "",
  "post_date": "2020-07-02T12:34:55.214662400Z",
  "votes": 2,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I tried to add custom metric function, but it was impossible to convert a symbolic tensor to numpy array.</p>\n\n<p>I also tried to use a custom metric callback, but when I use self.validation_data, it outputted NoneType. I then tried to add an <strong>init</strong>() in the custom callback class to pass in validation dataset. with self.validation_data = get_validation_data(), but it seems that get_validation_data()'s return is too large that crashes the progress with too much RAM usage, although this seems shouldn't happen because the return is a iterater.</p>\n\n<p>I was wondering what should I do to incorporate the custom metric on TPU?</p>\n\n<p>Thanks!</p>",
  "messages": [
    {
      "id": "912362",
      "postDate": "07/02/2020 12:34:55",
      "content": "<p>I tried to add custom metric function, but it was impossible to convert a symbolic tensor to numpy array.</p>\n\n<p>I also tried to use a custom metric callback, but when I use self.validation_data, it outputted NoneType. I then tried to add an <strong>init</strong>() in the custom callback class to pass in validation dataset. with self.validation_data = get_validation_data(), but it seems that get_validation_data()'s return is too large that crashes the progress with too much RAM usage, although this seems shouldn't happen because the return is a iterater.</p>\n\n<p>I was wondering what should I do to incorporate the custom metric on TPU?</p>\n\n<p>Thanks!</p>",
      "rawMarkdown": "I tried to add custom metric function, but it was impossible to convert a symbolic tensor to numpy array.\n\nI also tried to use a custom metric callback, but when I use self.validation_data, it outputted NoneType. I then tried to add an __init__() in the custom callback class to pass in validation dataset. with self.validation_data = get_validation_data(), but it seems that get_validation_data()'s return is too large that crashes the progress with too much RAM usage, although this seems shouldn't happen because the return is a iterater.\n\nI was wondering what should I do to incorporate the custom metric on TPU?\n\nThanks!",
      "votes": null
    },
    {
      "id": "920595",
      "postDate": "07/08/2020 17:41:55",
      "content": "<p>I looked into this and metrics are really difficult in tf keras.</p>\n\n<p>I recommend finding the metric that behaves the closest to what you need, then hijack it. That is what I did for the competition metric to get it to work for TPUs <a href=\"https://www.kaggle.com/hooong/tpu-weighted-auc-metric\">https://www.kaggle.com/hooong/tpu-weighted-auc-metric</a></p>",
      "rawMarkdown": "I looked into this and metrics are really difficult in tf keras.\n\nI recommend finding the metric that behaves the closest to what you need, then hijack it. That is what I did for the competition metric to get it to work for TPUs https://www.kaggle.com/hooong/tpu-weighted-auc-metric",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 920595,
      "author_name": "hooong",
      "author_url": "",
      "post_date": "07/08/2020 17:41:55",
      "content": "<p>I looked into this and metrics are really difficult in tf keras.</p>\n\n<p>I recommend finding the metric that behaves the closest to what you need, then hijack it. That is what I did for the competition metric to get it to work for TPUs <a href=\"https://www.kaggle.com/hooong/tpu-weighted-auc-metric\">https://www.kaggle.com/hooong/tpu-weighted-auc-metric</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "912362": "I tried to add custom metric function, but it was impossible to convert a symbolic tensor to numpy array.\n\nI also tried to use a custom metric callback, but when I use self.validation_data, it outputted NoneType. I then tried to add an __init__() in the custom callback class to pass in validation dataset. with self.validation_data = get_validation_data(), but it seems that get_validation_data()'s return is too large that crashes the progress with too much RAM usage, although this seems shouldn't happen because the return is a iterater.\n\nI was wondering what should I do to incorporate the custom metric on TPU?\n\nThanks!",
    "920595": "I looked into this and metrics are really difficult in tf keras.\n\nI recommend finding the metric that behaves the closest to what you need, then hijack it. That is what I did for the competition metric to get it to work for TPUs https://www.kaggle.com/hooong/tpu-weighted-auc-metric"
  },
  "source": "meta"
}