{
  "id": 128765,
  "title": "[Keras] Earlystopping use recall score",
  "url": "/competitions/bengaliai-cv19/discussion/128765",
  "author_name": "Orion",
  "post_date": "2020-02-03T07:03:19.313000",
  "votes": 8,
  "comment_count": 12,
  "views": 0,
  "content": "<p>Based on this kernel :<a href=\"https://www.kaggle.com/kaushal2896/bengali-graphemes-starter-eda-multi-output-cnn\">Bengali Graphemes: Starter EDA+ Multi Output CNN</a></p>\n\n<p>Definition:\n```\nclass Metric(Callback):\n    def <strong>init</strong>(self, model, callbacks, data):\n        super().<strong>init</strong>()\n        self.model = model\n        self.callbacks = callbacks\n        self.data = data</p>\n\n<pre><code>def on_train_begin(self, logs=None):\n    for callback in self.callbacks:\n        callback.on_train_begin(logs)\n\ndef on_train_end(self, logs=None):\n    for callback in self.callbacks:\n        callback.on_train_end(logs)\n\ndef on_epoch_end(self, batch, logs=None):\n\n    X_train, y_1, y_2, y_3 = self.data[0][0], self.data[0][1][0], self.data[0][1][1], self.data[0][1][2]\n\n    y_pred = self.model.predict(X_train)\n    y_pred1, y_pred2, y_pred3 = y_pred[0], y_pred[1], y_pred[2]\n    y_1 = [np.argmax(py) for py in y_1]\n    y_2 = [np.argmax(py) for py in y_2]\n    y_3 = [np.argmax(py) for py in y_3]\n    y_pred1 = [np.argmax(py) for py in y_pred1]\n    y_pred2 = [np.argmax(py) for py in y_pred2]\n    y_pred3 = [np.argmax(py) for py in y_pred3]\n    recall_grapheme = sklearn.metrics.recall_score(y_pred1, y_1, average='macro')\n    recall_vowel = sklearn.metrics.recall_score(y_pred2, y_2, average='macro')\n    recall_consonant = sklearn.metrics.recall_score(y_pred3, y_3, average='macro')\n    scores = [recall_grapheme, recall_vowel, recall_consonant]\n    tr_recall = np.average(scores, weights=[2, 1, 1])\n    logs['tr_recall'] = tr_recall\n\n\n    X_valid, y_1, y_2, y_3 = self.data[1][0], self.data[1][1][0], self.data[1][1][1], self.data[1][1][2]\n\n    y_pred = self.model.predict(X_valid)\n    y_pred1, y_pred2, y_pred3 = y_pred[0], y_pred[1], y_pred[2]\n    y_1 = [np.argmax(py) for py in y_1]\n    y_2 = [np.argmax(py) for py in y_2]\n    y_3 = [np.argmax(py) for py in y_3]\n    y_pred1 = [np.argmax(py) for py in y_pred1]\n    y_pred2 = [np.argmax(py) for py in y_pred2]\n    y_pred3 = [np.argmax(py) for py in y_pred3]\n    recall_grapheme = sklearn.metrics.recall_score(y_pred1, y_1, average='macro')\n    recall_vowel = sklearn.metrics.recall_score(y_pred2, y_2, average='macro')\n    recall_consonant = sklearn.metrics.recall_score(y_pred3, y_3, average='macro')\n    scores = [recall_grapheme, recall_vowel, recall_consonant]\n    val_recall = np.average(scores, weights=[2, 1, 1])\n    logs['val_recall'] = val_recall\n\n    print('tr recall', tr_recall, 'val recall', val_recall)\n\n    for callback in self.callbacks:\n        callback.on_epoch_end(batch, logs)\n</code></pre>\n\n<p>```</p>\n\n<p>```\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[])\nearlyStop = EarlyStopping(monitor='val_recall',\n                          mode = 'max',\n                          patience = 13,\n                          restore_best_weights=True,\n                          min_delta = 0,\n                          verbose = 1)\nearlyStop.set_model(model)</p>\n\n<p>chkPoint = ModelCheckpoint(ROOT + 'cnn.h5',\n                           monitor = 'val_recall',\n                           save_best_only = True,\n                           save_weights_only = False,\n                           mode = 'max',\n                           period = 1,\n                           verbose = 1)\nchkPoint.set_model(model)\n```</p>\n\n<p>Then call it before your training\n<code>metric = Metric(model, [earlyStop], [(x_train, [y_train_root, y_train_vowel, y_train_consonant]), (x_test, [y_test_root, y_test_vowel, y_test_consonant])])</code></p>",
  "messages": [
    {
      "id": 735532,
      "postDate": "2020-02-03T07:03:19.313Z",
      "content": "<p>Based on this kernel :<a href=\"https://www.kaggle.com/kaushal2896/bengali-graphemes-starter-eda-multi-output-cnn\">Bengali Graphemes: Starter EDA+ Multi Output CNN</a></p>\n\n<p>Definition:\n```\nclass Metric(Callback):\n    def <strong>init</strong>(self, model, callbacks, data):\n        super().<strong>init</strong>()\n        self.model = model\n        self.callbacks = callbacks\n        self.data = data</p>\n\n<pre><code>def on_train_begin(self, logs=None):\n    for callback in self.callbacks:\n        callback.on_train_begin(logs)\n\ndef on_train_end(self, logs=None):\n    for callback in self.callbacks:\n        callback.on_train_end(logs)\n\ndef on_epoch_end(self, batch, logs=None):\n\n    X_train, y_1, y_2, y_3 = self.data[0][0], self.data[0][1][0], self.data[0][1][1], self.data[0][1][2]\n\n    y_pred = self.model.predict(X_train)\n    y_pred1, y_pred2, y_pred3 = y_pred[0], y_pred[1], y_pred[2]\n    y_1 = [np.argmax(py) for py in y_1]\n    y_2 = [np.argmax(py) for py in y_2]\n    y_3 = [np.argmax(py) for py in y_3]\n    y_pred1 = [np.argmax(py) for py in y_pred1]\n    y_pred2 = [np.argmax(py) for py in y_pred2]\n    y_pred3 = [np.argmax(py) for py in y_pred3]\n    recall_grapheme = sklearn.metrics.recall_score(y_pred1, y_1, average='macro')\n    recall_vowel = sklearn.metrics.recall_score(y_pred2, y_2, average='macro')\n    recall_consonant = sklearn.metrics.recall_score(y_pred3, y_3, average='macro')\n    scores = [recall_grapheme, recall_vowel, recall_consonant]\n    tr_recall = np.average(scores, weights=[2, 1, 1])\n    logs['tr_recall'] = tr_recall\n\n\n    X_valid, y_1, y_2, y_3 = self.data[1][0], self.data[1][1][0], self.data[1][1][1], self.data[1][1][2]\n\n    y_pred = self.model.predict(X_valid)\n    y_pred1, y_pred2, y_pred3 = y_pred[0], y_pred[1], y_pred[2]\n    y_1 = [np.argmax(py) for py in y_1]\n    y_2 = [np.argmax(py) for py in y_2]\n    y_3 = [np.argmax(py) for py in y_3]\n    y_pred1 = [np.argmax(py) for py in y_pred1]\n    y_pred2 = [np.argmax(py) for py in y_pred2]\n    y_pred3 = [np.argmax(py) for py in y_pred3]\n    recall_grapheme = sklearn.metrics.recall_score(y_pred1, y_1, average='macro')\n    recall_vowel = sklearn.metrics.recall_score(y_pred2, y_2, average='macro')\n    recall_consonant = sklearn.metrics.recall_score(y_pred3, y_3, average='macro')\n    scores = [recall_grapheme, recall_vowel, recall_consonant]\n    val_recall = np.average(scores, weights=[2, 1, 1])\n    logs['val_recall'] = val_recall\n\n    print('tr recall', tr_recall, 'val recall', val_recall)\n\n    for callback in self.callbacks:\n        callback.on_epoch_end(batch, logs)\n</code></pre>\n\n<p>```</p>\n\n<p>```\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[])\nearlyStop = EarlyStopping(monitor='val_recall',\n                          mode = 'max',\n                          patience = 13,\n                          restore_best_weights=True,\n                          min_delta = 0,\n                          verbose = 1)\nearlyStop.set_model(model)</p>\n\n<p>chkPoint = ModelCheckpoint(ROOT + 'cnn.h5',\n                           monitor = 'val_recall',\n                           save_best_only = True,\n                           save_weights_only = False,\n                           mode = 'max',\n                           period = 1,\n                           verbose = 1)\nchkPoint.set_model(model)\n```</p>\n\n<p>Then call it before your training\n<code>metric = Metric(model, [earlyStop], [(x_train, [y_train_root, y_train_vowel, y_train_consonant]), (x_test, [y_test_root, y_test_vowel, y_test_consonant])])</code></p>",
      "rawMarkdown": "Based on this kernel :[Bengali Graphemes: Starter EDA+ Multi Output CNN](https://www.kaggle.com/kaushal2896/bengali-graphemes-starter-eda-multi-output-cnn)\n\nDefinition:\n```\nclass Metric(Callback):\n    def __init__(self, model, callbacks, data):\n        super().__init__()\n        self.model = model\n        self.callbacks = callbacks\n        self.data = data\n\n    def on_train_begin(self, logs=None):\n        for callback in self.callbacks:\n            callback.on_train_begin(logs)\n\n    def on_train_end(self, logs=None):\n        for callback in self.callbacks:\n            callback.on_train_end(logs)\n\n    def on_epoch_end(self, batch, logs=None):\n        \n        X_train, y_1, y_2, y_3 = self.data[0][0], self.data[0][1][0], self.data[0][1][1], self.data[0][1][2]\n        \n        y_pred = self.model.predict(X_train)\n        y_pred1, y_pred2, y_pred3 = y_pred[0], y_pred[1], y_pred[2]\n        y_1 = [np.argmax(py) for py in y_1]\n        y_2 = [np.argmax(py) for py in y_2]\n        y_3 = [np.argmax(py) for py in y_3]\n        y_pred1 = [np.argmax(py) for py in y_pred1]\n        y_pred2 = [np.argmax(py) for py in y_pred2]\n        y_pred3 = [np.argmax(py) for py in y_pred3]\n        recall_grapheme = sklearn.metrics.recall_score(y_pred1, y_1, average='macro')\n        recall_vowel = sklearn.metrics.recall_score(y_pred2, y_2, average='macro')\n        recall_consonant = sklearn.metrics.recall_score(y_pred3, y_3, average='macro')\n        scores = [recall_grapheme, recall_vowel, recall_consonant]\n        tr_recall = np.average(scores, weights=[2, 1, 1])\n        logs['tr_recall'] = tr_recall\n\n\n        X_valid, y_1, y_2, y_3 = self.data[1][0], self.data[1][1][0], self.data[1][1][1], self.data[1][1][2]\n\n        y_pred = self.model.predict(X_valid)\n        y_pred1, y_pred2, y_pred3 = y_pred[0], y_pred[1], y_pred[2]\n        y_1 = [np.argmax(py) for py in y_1]\n        y_2 = [np.argmax(py) for py in y_2]\n        y_3 = [np.argmax(py) for py in y_3]\n        y_pred1 = [np.argmax(py) for py in y_pred1]\n        y_pred2 = [np.argmax(py) for py in y_pred2]\n        y_pred3 = [np.argmax(py) for py in y_pred3]\n        recall_grapheme = sklearn.metrics.recall_score(y_pred1, y_1, average='macro')\n        recall_vowel = sklearn.metrics.recall_score(y_pred2, y_2, average='macro')\n        recall_consonant = sklearn.metrics.recall_score(y_pred3, y_3, average='macro')\n        scores = [recall_grapheme, recall_vowel, recall_consonant]\n        val_recall = np.average(scores, weights=[2, 1, 1])\n        logs['val_recall'] = val_recall\n\n        print('tr recall', tr_recall, 'val recall', val_recall)\n\n        for callback in self.callbacks:\n            callback.on_epoch_end(batch, logs)\n```\n\n```\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[])\nearlyStop = EarlyStopping(monitor='val_recall',\n                          mode = 'max',\n                          patience = 13,\n                          restore_best_weights=True,\n                          min_delta = 0,\n                          verbose = 1)\nearlyStop.set_model(model)\n\nchkPoint = ModelCheckpoint(ROOT + 'cnn.h5',\n                           monitor = 'val_recall',\n                           save_best_only = True,\n                           save_weights_only = False,\n                           mode = 'max',\n                           period = 1,\n                           verbose = 1)\nchkPoint.set_model(model)\n```\n\nThen call it before your training\n`metric = Metric(model, [earlyStop], [(x_train, [y_train_root, y_train_vowel, y_train_consonant]), (x_test, [y_test_root, y_test_vowel, y_test_consonant])])`",
      "votes": 8
    },
    {
      "id": 736082,
      "postDate": "2020-02-03T19:42:13.193Z",
      "content": "<p>Great. Thanks for sharing. Much appreciated 💚 \n<a href=\"/kaushal2896\">@kaushal2896</a> kernel is really awesome, great insight. </p>",
      "rawMarkdown": "Great. Thanks for sharing. Much appreciated 💚 \n@kaushal2896 kernel is really awesome, great insight. ",
      "votes": 1
    },
    {
      "id": 735827,
      "postDate": "2020-02-03T13:50:34.950Z",
      "content": "<p>That's pretty interesting!</p>\n\n<p>I feel like you'd loose a lot of time doing that for each epoch though? I have a 3 fold CV and it takes already a lot of time to do the validation CV. I feel like doing it per epoch would take a lot more time, for probably marginal improvements?</p>\n\n<p>I'm also saying this because I'm limited to Kaggle kernels / Colab, so compute time is the biggest resource I'm trying to optimize. Otherwise, this looks great! I'd be curious on having your feedback on it though.</p>",
      "rawMarkdown": "That's pretty interesting!\n\nI feel like you'd loose a lot of time doing that for each epoch though? I have a 3 fold CV and it takes already a lot of time to do the validation CV. I feel like doing it per epoch would take a lot more time, for probably marginal improvements?\n\nI'm also saying this because I'm limited to Kaggle kernels / Colab, so compute time is the biggest resource I'm trying to optimize. Otherwise, this looks great! I'd be curious on having your feedback on it though.",
      "votes": 1,
      "replies": [
        {
          "id": 735924,
          "postDate": "2020-02-03T16:03:47.040Z",
          "content": "<p>Doing this on every epoch end is a real little waste of computing resource, but I'think the model feedback is worth it. And actually it will reduce the risk of overfitting.</p>\n\n<p>If you still want to cut computing time, you can set it on every 5 or 10 epochs.</p>",
          "rawMarkdown": "Doing this on every epoch end is a real little waste of computing resource, but I'think the model feedback is worth it. And actually it will reduce the risk of overfitting.\n\nIf you still want to cut computing time, you can set it on every 5 or 10 epochs."
        },
        {
          "id": 735930,
          "postDate": "2020-02-03T16:09:46.400Z",
          "content": "<p>Doesn't the <code>model.predict</code> take up a huge amount of time?</p>\n\n<p>That takes so long for me when evaluating my validation recall score on an out-of-fold prediction. I would naively think you'd have it worse when doing it per epoch?</p>",
          "rawMarkdown": "Doesn't the `model.predict` take up a huge amount of time?\n\nThat takes so long for me when evaluating my validation recall score on an out-of-fold prediction. I would naively think you'd have it worse when doing it per epoch?"
        },
        {
          "id": 735945,
          "postDate": "2020-02-03T16:26:02.813Z",
          "content": "<p>I split train data into 4 parts, everytime I just train my model on one part and use 10% of this part for validation. So <code>model.predcit</code> is fast enough. </p>\n\n<p>Actually the most of time is wasted on training, not validation.</p>",
          "rawMarkdown": "I split train data into 4 parts, everytime I just train my model on one part and use 10% of this part for validation. So `model.predcit` is fast enough. \n\nActually the most of time is wasted on training, not validation."
        },
        {
          "id": 736081,
          "postDate": "2020-02-03T19:40:02.857Z",
          "content": "<p>Hmm, interesting</p>\n\n<p>Sure, training is still the biggest time spend. That doesn't mean that when GPU resources are limited validation doesn't take a big chunk of it! ;)</p>\n\n<p>But again, really nice implementation!</p>",
          "rawMarkdown": "Hmm, interesting\n\nSure, training is still the biggest time spend. That doesn't mean that when GPU resources are limited validation doesn't take a big chunk of it! ;)\n\nBut again, really nice implementation!"
        }
      ]
    },
    {
      "id": 741442,
      "postDate": "2020-02-10T16:27:36.750Z",
      "content": "<p>Hey <a href=\"/xiaohuhayou\">@xiaohuhayou</a> Nice end-to-end example.!\nHave you also tried or taken a look at using the official Metric 'tensorflow.keras.metrics.Recall()' in combination with your code? I'am not using the EarlyStopping myself but the metric is very usefull.\nIf I am not mistaken it is also calculated on the GPU..so it is fast.</p>\n\n<p>May'be you can combine it. Or make a custom loop around it and check the values for recall after every epoch based on model.history.\nThe loop and early stopping would be custom then..but the recall calculation would be fast and may'be outweigh the additional calculatioins done by an sklearn.</p>\n\n<p>Hope you can use it to your benefit. Goodluck.</p>",
      "rawMarkdown": "Hey @xiaohuhayou Nice end-to-end example.!\nHave you also tried or taken a look at using the official Metric 'tensorflow.keras.metrics.Recall()' in combination with your code? I'am not using the EarlyStopping myself but the metric is very usefull.\nIf I am not mistaken it is also calculated on the GPU..so it is fast.\n\nMay'be you can combine it. Or make a custom loop around it and check the values for recall after every epoch based on model.history.\nThe loop and early stopping would be custom then..but the recall calculation would be fast and may'be outweigh the additional calculatioins done by an sklearn.\n\nHope you can use it to your benefit. Goodluck."
    },
    {
      "id": 735823,
      "postDate": "2020-02-03T13:45:14.927Z",
      "content": "<p>This is fantastic !! I tried developing a loss including the recall in order to optimize it but I kind of failed to make it work and this uses the same spirit that motivated to try so !! Awesome share</p>",
      "rawMarkdown": "This is fantastic !! I tried developing a loss including the recall in order to optimize it but I kind of failed to make it work and this uses the same spirit that motivated to try so !! Awesome share",
      "replies": [
        {
          "id": 735925,
          "postDate": "2020-02-03T16:04:33.033Z",
          "content": "<p>Glad to see that!</p>",
          "rawMarkdown": "Glad to see that!"
        }
      ]
    },
    {
      "id": 739661,
      "postDate": "2020-02-08T06:10:31.650Z",
      "rawMarkdown": "",
      "votes": 2,
      "isDeleted": true,
      "replies": [
        {
          "id": 739680,
          "postDate": "2020-02-08T07:28:18.803Z",
          "content": "<p>That a simple yet really interesting idea</p>",
          "rawMarkdown": "That a simple yet really interesting idea"
        }
      ]
    },
    {
      "id": 739354,
      "postDate": "2020-02-07T18:37:10.167Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 736082,
      "author_name": "Innat",
      "author_url": "",
      "post_date": "2020-02-03T19:42:13.193000",
      "content": "<p>Great. Thanks for sharing. Much appreciated 💚 \n<a href=\"/kaushal2896\">@kaushal2896</a> kernel is really awesome, great insight. </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 735827,
      "author_name": "Maxime Lenormand",
      "author_url": "",
      "post_date": "2020-02-03T13:50:34.950000",
      "content": "<p>That's pretty interesting!</p>\n\n<p>I feel like you'd loose a lot of time doing that for each epoch though? I have a 3 fold CV and it takes already a lot of time to do the validation CV. I feel like doing it per epoch would take a lot more time, for probably marginal improvements?</p>\n\n<p>I'm also saying this because I'm limited to Kaggle kernels / Colab, so compute time is the biggest resource I'm trying to optimize. Otherwise, this looks great! I'd be curious on having your feedback on it though.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 735924,
          "author_name": "Orion",
          "author_url": "",
          "post_date": "2020-02-03T16:03:47.040000",
          "content": "<p>Doing this on every epoch end is a real little waste of computing resource, but I'think the model feedback is worth it. And actually it will reduce the risk of overfitting.</p>\n\n<p>If you still want to cut computing time, you can set it on every 5 or 10 epochs.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 735930,
          "author_name": "Maxime Lenormand",
          "author_url": "",
          "post_date": "2020-02-03T16:09:46.400000",
          "content": "<p>Doesn't the <code>model.predict</code> take up a huge amount of time?</p>\n\n<p>That takes so long for me when evaluating my validation recall score on an out-of-fold prediction. I would naively think you'd have it worse when doing it per epoch?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 735945,
          "author_name": "Orion",
          "author_url": "",
          "post_date": "2020-02-03T16:26:02.813000",
          "content": "<p>I split train data into 4 parts, everytime I just train my model on one part and use 10% of this part for validation. So <code>model.predcit</code> is fast enough. </p>\n\n<p>Actually the most of time is wasted on training, not validation.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 736081,
          "author_name": "Maxime Lenormand",
          "author_url": "",
          "post_date": "2020-02-03T19:40:02.857000",
          "content": "<p>Hmm, interesting</p>\n\n<p>Sure, training is still the biggest time spend. That doesn't mean that when GPU resources are limited validation doesn't take a big chunk of it! ;)</p>\n\n<p>But again, really nice implementation!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 741442,
      "author_name": "Robin Smits",
      "author_url": "",
      "post_date": "2020-02-10T16:27:36.750000",
      "content": "<p>Hey <a href=\"/xiaohuhayou\">@xiaohuhayou</a> Nice end-to-end example.!\nHave you also tried or taken a look at using the official Metric 'tensorflow.keras.metrics.Recall()' in combination with your code? I'am not using the EarlyStopping myself but the metric is very usefull.\nIf I am not mistaken it is also calculated on the GPU..so it is fast.</p>\n\n<p>May'be you can combine it. Or make a custom loop around it and check the values for recall after every epoch based on model.history.\nThe loop and early stopping would be custom then..but the recall calculation would be fast and may'be outweigh the additional calculatioins done by an sklearn.</p>\n\n<p>Hope you can use it to your benefit. Goodluck.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 735823,
      "author_name": "Thomas Di Martino",
      "author_url": "",
      "post_date": "2020-02-03T13:45:14.927000",
      "content": "<p>This is fantastic !! I tried developing a loss including the recall in order to optimize it but I kind of failed to make it work and this uses the same spirit that motivated to try so !! Awesome share</p>",
      "votes": 0,
      "replies": [
        {
          "id": 735925,
          "author_name": "Orion",
          "author_url": "",
          "post_date": "2020-02-03T16:04:33.033000",
          "content": "<p>Glad to see that!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 739661,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-08T06:10:31.650000",
      "content": "",
      "votes": 2,
      "replies": [
        {
          "id": 739680,
          "author_name": "Maxime Lenormand",
          "author_url": "",
          "post_date": "2020-02-08T07:28:18.803000",
          "content": "<p>That a simple yet really interesting idea</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 739354,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-07T18:37:10.167000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "735532": "Based on this kernel :[Bengali Graphemes: Starter EDA+ Multi Output CNN](https://www.kaggle.com/kaushal2896/bengali-graphemes-starter-eda-multi-output-cnn)\n\nDefinition:\n```\nclass Metric(Callback):\n    def __init__(self, model, callbacks, data):\n        super().__init__()\n        self.model = model\n        self.callbacks = callbacks\n        self.data = data\n\n    def on_train_begin(self, logs=None):\n        for callback in self.callbacks:\n            callback.on_train_begin(logs)\n\n    def on_train_end(self, logs=None):\n        for callback in self.callbacks:\n            callback.on_train_end(logs)\n\n    def on_epoch_end(self, batch, logs=None):\n        \n        X_train, y_1, y_2, y_3 = self.data[0][0], self.data[0][1][0], self.data[0][1][1], self.data[0][1][2]\n        \n        y_pred = self.model.predict(X_train)\n        y_pred1, y_pred2, y_pred3 = y_pred[0], y_pred[1], y_pred[2]\n        y_1 = [np.argmax(py) for py in y_1]\n        y_2 = [np.argmax(py) for py in y_2]\n        y_3 = [np.argmax(py) for py in y_3]\n        y_pred1 = [np.argmax(py) for py in y_pred1]\n        y_pred2 = [np.argmax(py) for py in y_pred2]\n        y_pred3 = [np.argmax(py) for py in y_pred3]\n        recall_grapheme = sklearn.metrics.recall_score(y_pred1, y_1, average='macro')\n        recall_vowel = sklearn.metrics.recall_score(y_pred2, y_2, average='macro')\n        recall_consonant = sklearn.metrics.recall_score(y_pred3, y_3, average='macro')\n        scores = [recall_grapheme, recall_vowel, recall_consonant]\n        tr_recall = np.average(scores, weights=[2, 1, 1])\n        logs['tr_recall'] = tr_recall\n\n\n        X_valid, y_1, y_2, y_3 = self.data[1][0], self.data[1][1][0], self.data[1][1][1], self.data[1][1][2]\n\n        y_pred = self.model.predict(X_valid)\n        y_pred1, y_pred2, y_pred3 = y_pred[0], y_pred[1], y_pred[2]\n        y_1 = [np.argmax(py) for py in y_1]\n        y_2 = [np.argmax(py) for py in y_2]\n        y_3 = [np.argmax(py) for py in y_3]\n        y_pred1 = [np.argmax(py) for py in y_pred1]\n        y_pred2 = [np.argmax(py) for py in y_pred2]\n        y_pred3 = [np.argmax(py) for py in y_pred3]\n        recall_grapheme = sklearn.metrics.recall_score(y_pred1, y_1, average='macro')\n        recall_vowel = sklearn.metrics.recall_score(y_pred2, y_2, average='macro')\n        recall_consonant = sklearn.metrics.recall_score(y_pred3, y_3, average='macro')\n        scores = [recall_grapheme, recall_vowel, recall_consonant]\n        val_recall = np.average(scores, weights=[2, 1, 1])\n        logs['val_recall'] = val_recall\n\n        print('tr recall', tr_recall, 'val recall', val_recall)\n\n        for callback in self.callbacks:\n            callback.on_epoch_end(batch, logs)\n```\n\n```\nmodel.compile(optimizer='adam', loss='categorical_crossentropy', metrics=[])\nearlyStop = EarlyStopping(monitor='val_recall',\n                          mode = 'max',\n                          patience = 13,\n                          restore_best_weights=True,\n                          min_delta = 0,\n                          verbose = 1)\nearlyStop.set_model(model)\n\nchkPoint = ModelCheckpoint(ROOT + 'cnn.h5',\n                           monitor = 'val_recall',\n                           save_best_only = True,\n                           save_weights_only = False,\n                           mode = 'max',\n                           period = 1,\n                           verbose = 1)\nchkPoint.set_model(model)\n```\n\nThen call it before your training\n`metric = Metric(model, [earlyStop], [(x_train, [y_train_root, y_train_vowel, y_train_consonant]), (x_test, [y_test_root, y_test_vowel, y_test_consonant])])`",
    "736082": "Great. Thanks for sharing. Much appreciated 💚 \n@kaushal2896 kernel is really awesome, great insight. ",
    "735827": "That's pretty interesting!\n\nI feel like you'd loose a lot of time doing that for each epoch though? I have a 3 fold CV and it takes already a lot of time to do the validation CV. I feel like doing it per epoch would take a lot more time, for probably marginal improvements?\n\nI'm also saying this because I'm limited to Kaggle kernels / Colab, so compute time is the biggest resource I'm trying to optimize. Otherwise, this looks great! I'd be curious on having your feedback on it though.",
    "741442": "Hey @xiaohuhayou Nice end-to-end example.!\nHave you also tried or taken a look at using the official Metric 'tensorflow.keras.metrics.Recall()' in combination with your code? I'am not using the EarlyStopping myself but the metric is very usefull.\nIf I am not mistaken it is also calculated on the GPU..so it is fast.\n\nMay'be you can combine it. Or make a custom loop around it and check the values for recall after every epoch based on model.history.\nThe loop and early stopping would be custom then..but the recall calculation would be fast and may'be outweigh the additional calculatioins done by an sklearn.\n\nHope you can use it to your benefit. Goodluck.",
    "735823": "This is fantastic !! I tried developing a loss including the recall in order to optimize it but I kind of failed to make it work and this uses the same spirit that motivated to try so !! Awesome share",
    "739661": "",
    "739354": ""
  }
}