{
  "id": 101934,
  "title": "Keras - use kappa metric callback for Imagegenerator",
  "url": "/competitions/aptos2019-blindness-detection/discussion/101934",
  "author_name": "",
  "post_date": "2019-07-29T21:13:42.739703400Z",
  "votes": 7,
  "comment_count": 1,
  "views": 0,
  "content": "<h3>Issue</h3>\n\n<p>The original kappa metric callback works in xhlulu's kernel. But, if you use ImageGenerator to generate validation data and then try to callback this function, it pops an error saying validation_data is NoneType.</p>\n\n<p>The original kappa metric is from APTOS 2019: DenseNet Keras Starter <a href=\"https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter\">https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter</a>\n<code>``\nclassMetrics(Callback):\n  def on_train_begin(self, logs={}):\n    self.val_kappas = []\n  def on_epoch_end(self, epoch, logs={}):\n        X_val, y_val = self.validation_data[:2]\n        y_val = y_val.sum(axis=1) - 1\n</code>\n        y_pred = self.model.predict(X_val) &gt; 0.5\n        y_pred = y_pred.astype(int).sum(axis=1) - 1</p>\n\n<pre><code>    _val_kappa = cohen_kappa_score(\n        y_val,\n        y_pred, \n        weights='quadratic'\n    )\n\n    self.val_kappas.append(_val_kappa)\n\n    print(f\"val_kappa: {_val_kappa:.4f}\")\n\n    if _val_kappa == max(self.val_kappas):\n        print(\"Validation Kappa has improved. Saving model.\")\n        self.model.save('model.h5')\n    return\n</code></pre>\n\n<p>```</p>\n\n<h3>The issue was discussed here:</h3>\n\n<p><a href=\"https://github.com/keras-team/keras/issues/10472#issuecomment-472543538\">https://github.com/keras-team/keras/issues/10472#issuecomment-472543538</a>\nThanks for EzioA's exploring. In fit_generator(), if validation_data argument is a generator, it won't be set until callbacks executed.</p>\n\n<h3>How to solve it?</h3>\n\n<p>ref from EzioA's code:\nYou need to initialise the validation_data by yourselves.\n<code>\n    def __init__(self, val_data, validation_steps = 15, batch_size = 32):\n        super().__init__()\n        self.validation_data = val_data\n        self.batch_size = batch_size\n        self.validation_steps = validation_steps\n</code>\nThen you can modify the metric calculation as following:  (PS: I also added an early stopping here)\n```\n    def on_epoch_end(self, epoch, logs={}):\n        bs = self.batch_size \n        y_pred = np.empty((self.validation_steps * bs, 1), dtype = np.uint8)\n        y_val = np.empty((self.validation_steps * bs, 1), dtype = np.uint8)\n        w_size = 0;</p>\n\n<pre><code>    for idx in range(self.validation_steps):\n        l = self.validation_data.next()\n        l_len = len(l[1])\n        y_val[w_size:(w_size + l_len),0] = l[1].astype(int).sum(axis=1) - 1\n        y_pred[w_size:(w_size + l_len),0] = (model.predict(l[0]) &amp;gt; 0.5).astype(int).sum(axis=1) - 1\n        w_size = w_size + l_len\n    y_val = y_val[:w_size]\n    y_pred = y_pred[:w_size]\n\n    _val_kappa = cohen_kappa_score(\n        y_val,\n        y_pred, \n        weights='quadratic'\n    )\n\n    self.val_kappas.append(_val_kappa)\n\n    print(f\"val_kappa: {_val_kappa:.4f}\")\n\n    if _val_kappa == max(self.val_kappas):\n        print(\"Validation Kappa has improved. Saving model.\")\n        self.model.save('model.h5')\n        self.val_stop_p_count = 0\n    else:\n    ## stop the training\n        self.val_stop_p_count = self.val_stop_p_count + 1;\n        if(self.val_stop_p_count &amp;gt; self.val_stop_patience):\n            print(\"Epoch %05d: early stopping \" % epoch)\n            self.model.stop_training = True\n\n    return\n</code></pre>\n\n<p>```\nIt does work for me, hope it can help you all!</p>",
  "messages": [
    {
      "id": "586903",
      "postDate": "07/29/2019 21:13:42",
      "content": "<h3>Issue</h3>\n\n<p>The original kappa metric callback works in xhlulu's kernel. But, if you use ImageGenerator to generate validation data and then try to callback this function, it pops an error saying validation_data is NoneType.</p>\n\n<p>The original kappa metric is from APTOS 2019: DenseNet Keras Starter <a href=\"https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter\">https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter</a>\n<code>``\nclassMetrics(Callback):\n  def on_train_begin(self, logs={}):\n    self.val_kappas = []\n  def on_epoch_end(self, epoch, logs={}):\n        X_val, y_val = self.validation_data[:2]\n        y_val = y_val.sum(axis=1) - 1\n</code>\n        y_pred = self.model.predict(X_val) &gt; 0.5\n        y_pred = y_pred.astype(int).sum(axis=1) - 1</p>\n\n<pre><code>    _val_kappa = cohen_kappa_score(\n        y_val,\n        y_pred, \n        weights='quadratic'\n    )\n\n    self.val_kappas.append(_val_kappa)\n\n    print(f\"val_kappa: {_val_kappa:.4f}\")\n\n    if _val_kappa == max(self.val_kappas):\n        print(\"Validation Kappa has improved. Saving model.\")\n        self.model.save('model.h5')\n    return\n</code></pre>\n\n<p>```</p>\n\n<h3>The issue was discussed here:</h3>\n\n<p><a href=\"https://github.com/keras-team/keras/issues/10472#issuecomment-472543538\">https://github.com/keras-team/keras/issues/10472#issuecomment-472543538</a>\nThanks for EzioA's exploring. In fit_generator(), if validation_data argument is a generator, it won't be set until callbacks executed.</p>\n\n<h3>How to solve it?</h3>\n\n<p>ref from EzioA's code:\nYou need to initialise the validation_data by yourselves.\n<code>\n    def __init__(self, val_data, validation_steps = 15, batch_size = 32):\n        super().__init__()\n        self.validation_data = val_data\n        self.batch_size = batch_size\n        self.validation_steps = validation_steps\n</code>\nThen you can modify the metric calculation as following:  (PS: I also added an early stopping here)\n```\n    def on_epoch_end(self, epoch, logs={}):\n        bs = self.batch_size \n        y_pred = np.empty((self.validation_steps * bs, 1), dtype = np.uint8)\n        y_val = np.empty((self.validation_steps * bs, 1), dtype = np.uint8)\n        w_size = 0;</p>\n\n<pre><code>    for idx in range(self.validation_steps):\n        l = self.validation_data.next()\n        l_len = len(l[1])\n        y_val[w_size:(w_size + l_len),0] = l[1].astype(int).sum(axis=1) - 1\n        y_pred[w_size:(w_size + l_len),0] = (model.predict(l[0]) &amp;gt; 0.5).astype(int).sum(axis=1) - 1\n        w_size = w_size + l_len\n    y_val = y_val[:w_size]\n    y_pred = y_pred[:w_size]\n\n    _val_kappa = cohen_kappa_score(\n        y_val,\n        y_pred, \n        weights='quadratic'\n    )\n\n    self.val_kappas.append(_val_kappa)\n\n    print(f\"val_kappa: {_val_kappa:.4f}\")\n\n    if _val_kappa == max(self.val_kappas):\n        print(\"Validation Kappa has improved. Saving model.\")\n        self.model.save('model.h5')\n        self.val_stop_p_count = 0\n    else:\n    ## stop the training\n        self.val_stop_p_count = self.val_stop_p_count + 1;\n        if(self.val_stop_p_count &amp;gt; self.val_stop_patience):\n            print(\"Epoch %05d: early stopping \" % epoch)\n            self.model.stop_training = True\n\n    return\n</code></pre>\n\n<p>```\nIt does work for me, hope it can help you all!</p>",
      "rawMarkdown": "### Issue\nThe original kappa metric callback works in xhlulu's kernel. But, if you use ImageGenerator to generate validation data and then try to callback this function, it pops an error saying validation_data is NoneType.\n\nThe original kappa metric is from APTOS 2019: DenseNet Keras Starter https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter\n```\nclassMetrics(Callback):\n  def on_train_begin(self, logs={}):\n    self.val_kappas = []\n  def on_epoch_end(self, epoch, logs={}):\n        X_val, y_val = self.validation_data[:2]\n        y_val = y_val.sum(axis=1) - 1\n`\n        y_pred = self.model.predict(X_val) &gt; 0.5\n        y_pred = y_pred.astype(int).sum(axis=1) - 1\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n        \n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n        return\n```\n\n### The issue was discussed here:\nhttps://github.com/keras-team/keras/issues/10472#issuecomment-472543538\nThanks for EzioA's exploring. In fit_generator(), if validation_data argument is a generator, it won't be set until callbacks executed.\n\n### How to solve it?\nref from EzioA's code:\nYou need to initialise the validation_data by yourselves.\n```\n    def __init__(self, val_data, validation_steps = 15, batch_size = 32):\n        super().__init__()\n        self.validation_data = val_data\n        self.batch_size = batch_size\n        self.validation_steps = validation_steps\n```\nThen you can modify the metric calculation as following:  (PS: I also added an early stopping here)\n```\n    def on_epoch_end(self, epoch, logs={}):\n        bs = self.batch_size \n        y_pred = np.empty((self.validation_steps * bs, 1), dtype = np.uint8)\n        y_val = np.empty((self.validation_steps * bs, 1), dtype = np.uint8)\n        w_size = 0;\n        \n        for idx in range(self.validation_steps):\n            l = self.validation_data.next()\n            l_len = len(l[1])\n            y_val[w_size:(w_size + l_len),0] = l[1].astype(int).sum(axis=1) - 1\n            y_pred[w_size:(w_size + l_len),0] = (model.predict(l[0]) &gt; 0.5).astype(int).sum(axis=1) - 1\n            w_size = w_size + l_len\n        y_val = y_val[:w_size]\n        y_pred = y_pred[:w_size]\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n        \n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n            self.val_stop_p_count = 0\n        else:\n        ## stop the training\n            self.val_stop_p_count = self.val_stop_p_count + 1;\n            if(self.val_stop_p_count &gt; self.val_stop_patience):\n                print(\"Epoch %05d: early stopping \" % epoch)\n                self.model.stop_training = True\n\n        return\n```\nIt does work for me, hope it can help you all!",
      "votes": null
    },
    {
      "id": "589261",
      "postDate": "07/31/2019 17:14:51",
      "content": "<p>I had indeed issues with the various WQK implementations when using an ImageDataGenerator as validation data. And while I couldn't get your implementation to work as well, I inspired heavily from it to get something working. I'll copy it in here, in case it can help someone : </p>\n\n<p>```\nclass QWKEvaluation(Callback):\n    def <strong>init</strong>(self, val_data, val_stop_patience=15):\n        super(Callback, self).<strong>init</strong>()\n        self.validation_data = val_data\n        self.val_kappas = []\n        self.val_stop_patience = val_stop_patience\n        self.length = len(val_data)</p>\n\n<pre><code>def on_epoch_end(self, epoch, logs={}):\n    y_pred = np.empty((self.length, 1), dtype = np.uint8)\n    y_val = np.empty((self.length, 1), dtype = np.uint8)\n    w_size = 0\n    for idx in range(self.length):\n        x, y = self.validation_data.next()\n        y_val[w_size:(w_size + 1),0] = y\n        y_pred[w_size:(w_size + 1),0] = model.predict(x)\n        w_size = w_size + 1\n\n    _val_kappa = cohen_kappa_score(\n        y_val,\n        y_pred, \n        weights='quadratic')\n\n    self.val_kappas.append(_val_kappa)\n    print(\"val_kappa:\", np.round(_val_kappa, 3))\n\n    if _val_kappa == max(self.val_kappas):\n        print(\"Validation Kappa has improved. Saving model.\")\n        self.model.save('model.h5')\n        self.val_stop_p_count = 0\n    else:\n        # stop the training\n        self.val_stop_p_count = self.val_stop_p_count + 1;\n        if(self.val_stop_p_count &amp;gt; self.val_stop_patience):\n            print(\"Epoch %05d: early stopping \" % epoch)\n            self.model.stop_training = True\n</code></pre>\n\n<p>```</p>",
      "rawMarkdown": "I had indeed issues with the various WQK implementations when using an ImageDataGenerator as validation data. And while I couldn't get your implementation to work as well, I inspired heavily from it to get something working. I'll copy it in here, in case it can help someone : \n\n```\nclass QWKEvaluation(Callback):\n    def __init__(self, val_data, val_stop_patience=15):\n        super(Callback, self).__init__()\n        self.validation_data = val_data\n        self.val_kappas = []\n        self.val_stop_patience = val_stop_patience\n        self.length = len(val_data)\n\n    def on_epoch_end(self, epoch, logs={}):\n        y_pred = np.empty((self.length, 1), dtype = np.uint8)\n        y_val = np.empty((self.length, 1), dtype = np.uint8)\n        w_size = 0\n        for idx in range(self.length):\n            x, y = self.validation_data.next()\n            y_val[w_size:(w_size + 1),0] = y\n            y_pred[w_size:(w_size + 1),0] = model.predict(x)\n            w_size = w_size + 1\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic')\n\n        self.val_kappas.append(_val_kappa)\n        print(\"val_kappa:\", np.round(_val_kappa, 3))\n\n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n            self.val_stop_p_count = 0\n        else:\n            # stop the training\n            self.val_stop_p_count = self.val_stop_p_count + 1;\n            if(self.val_stop_p_count &gt; self.val_stop_patience):\n                print(\"Epoch %05d: early stopping \" % epoch)\n                self.model.stop_training = True\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 589261,
      "author_name": "juliencs",
      "author_url": "",
      "post_date": "07/31/2019 17:14:51",
      "content": "<p>I had indeed issues with the various WQK implementations when using an ImageDataGenerator as validation data. And while I couldn't get your implementation to work as well, I inspired heavily from it to get something working. I'll copy it in here, in case it can help someone : </p>\n\n<p>```\nclass QWKEvaluation(Callback):\n    def <strong>init</strong>(self, val_data, val_stop_patience=15):\n        super(Callback, self).<strong>init</strong>()\n        self.validation_data = val_data\n        self.val_kappas = []\n        self.val_stop_patience = val_stop_patience\n        self.length = len(val_data)</p>\n\n<pre><code>def on_epoch_end(self, epoch, logs={}):\n    y_pred = np.empty((self.length, 1), dtype = np.uint8)\n    y_val = np.empty((self.length, 1), dtype = np.uint8)\n    w_size = 0\n    for idx in range(self.length):\n        x, y = self.validation_data.next()\n        y_val[w_size:(w_size + 1),0] = y\n        y_pred[w_size:(w_size + 1),0] = model.predict(x)\n        w_size = w_size + 1\n\n    _val_kappa = cohen_kappa_score(\n        y_val,\n        y_pred, \n        weights='quadratic')\n\n    self.val_kappas.append(_val_kappa)\n    print(\"val_kappa:\", np.round(_val_kappa, 3))\n\n    if _val_kappa == max(self.val_kappas):\n        print(\"Validation Kappa has improved. Saving model.\")\n        self.model.save('model.h5')\n        self.val_stop_p_count = 0\n    else:\n        # stop the training\n        self.val_stop_p_count = self.val_stop_p_count + 1;\n        if(self.val_stop_p_count &amp;gt; self.val_stop_patience):\n            print(\"Epoch %05d: early stopping \" % epoch)\n            self.model.stop_training = True\n</code></pre>\n\n<p>```</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "586903": "### Issue\nThe original kappa metric callback works in xhlulu's kernel. But, if you use ImageGenerator to generate validation data and then try to callback this function, it pops an error saying validation_data is NoneType.\n\nThe original kappa metric is from APTOS 2019: DenseNet Keras Starter https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter\n```\nclassMetrics(Callback):\n  def on_train_begin(self, logs={}):\n    self.val_kappas = []\n  def on_epoch_end(self, epoch, logs={}):\n        X_val, y_val = self.validation_data[:2]\n        y_val = y_val.sum(axis=1) - 1\n`\n        y_pred = self.model.predict(X_val) &gt; 0.5\n        y_pred = y_pred.astype(int).sum(axis=1) - 1\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n        \n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n        return\n```\n\n### The issue was discussed here:\nhttps://github.com/keras-team/keras/issues/10472#issuecomment-472543538\nThanks for EzioA's exploring. In fit_generator(), if validation_data argument is a generator, it won't be set until callbacks executed.\n\n### How to solve it?\nref from EzioA's code:\nYou need to initialise the validation_data by yourselves.\n```\n    def __init__(self, val_data, validation_steps = 15, batch_size = 32):\n        super().__init__()\n        self.validation_data = val_data\n        self.batch_size = batch_size\n        self.validation_steps = validation_steps\n```\nThen you can modify the metric calculation as following:  (PS: I also added an early stopping here)\n```\n    def on_epoch_end(self, epoch, logs={}):\n        bs = self.batch_size \n        y_pred = np.empty((self.validation_steps * bs, 1), dtype = np.uint8)\n        y_val = np.empty((self.validation_steps * bs, 1), dtype = np.uint8)\n        w_size = 0;\n        \n        for idx in range(self.validation_steps):\n            l = self.validation_data.next()\n            l_len = len(l[1])\n            y_val[w_size:(w_size + l_len),0] = l[1].astype(int).sum(axis=1) - 1\n            y_pred[w_size:(w_size + l_len),0] = (model.predict(l[0]) &gt; 0.5).astype(int).sum(axis=1) - 1\n            w_size = w_size + l_len\n        y_val = y_val[:w_size]\n        y_pred = y_pred[:w_size]\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n        \n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n            self.val_stop_p_count = 0\n        else:\n        ## stop the training\n            self.val_stop_p_count = self.val_stop_p_count + 1;\n            if(self.val_stop_p_count &gt; self.val_stop_patience):\n                print(\"Epoch %05d: early stopping \" % epoch)\n                self.model.stop_training = True\n\n        return\n```\nIt does work for me, hope it can help you all!",
    "589261": "I had indeed issues with the various WQK implementations when using an ImageDataGenerator as validation data. And while I couldn't get your implementation to work as well, I inspired heavily from it to get something working. I'll copy it in here, in case it can help someone : \n\n```\nclass QWKEvaluation(Callback):\n    def __init__(self, val_data, val_stop_patience=15):\n        super(Callback, self).__init__()\n        self.validation_data = val_data\n        self.val_kappas = []\n        self.val_stop_patience = val_stop_patience\n        self.length = len(val_data)\n\n    def on_epoch_end(self, epoch, logs={}):\n        y_pred = np.empty((self.length, 1), dtype = np.uint8)\n        y_val = np.empty((self.length, 1), dtype = np.uint8)\n        w_size = 0\n        for idx in range(self.length):\n            x, y = self.validation_data.next()\n            y_val[w_size:(w_size + 1),0] = y\n            y_pred[w_size:(w_size + 1),0] = model.predict(x)\n            w_size = w_size + 1\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic')\n\n        self.val_kappas.append(_val_kappa)\n        print(\"val_kappa:\", np.round(_val_kappa, 3))\n\n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n            self.val_stop_p_count = 0\n        else:\n            # stop the training\n            self.val_stop_p_count = self.val_stop_p_count + 1;\n            if(self.val_stop_p_count &gt; self.val_stop_patience):\n                print(\"Epoch %05d: early stopping \" % epoch)\n                self.model.stop_training = True\n```"
  },
  "source": "meta"
}