Hi,\nWould it possible to use ModelCheckPiont with AUC score (or define a custom callsback) to save the best model with the highest AUC score on validation data?
", "messages": [ { "id": 452799, "postDate": "2019-01-09T07:01:19.880Z", "content": "Yes, you can do it.
\n\nBut before using ModelCheckpoint, you need to compute the AUC score on your validation data. It can be done by implement your custom callback to help you to compute it. The following is the simple example.
\n\n```python\nfrom keras.callbacks import Callback\nfrom sklearn.metrics import roc_auc_score
\n\nclass RocAucMetric(Callback):
\n\ndef on_train_begin(self, logs={}):\n # By default, self.params['metrics'] contains loss and the metric assigned in `model.compile()`\n if not 'val_roc_auc' in self.params['metrics']:\n self.params['metrics'].append('val_roc_auc')\n\n logs['val_roc_auc'] = float('-inf')\n\ndef on_epoch_end(self, epoch, logs={}):\n y_true = self.validation_data[1]\n y_pred = self.model.predict(self.validation_data[0])\n score = roc_auc_score(y_true, y_pred)\n\n logs['val_roc_auc'] = score\n```
\n\nThen you can set your callbacks and pass it to model.fit(). Remember to put RocAucMetric ahead of ModelCheckpoint so that val_roc_auc exists and ModelCheckpoint can find it.
python\ncallbacks = [RocAucMetric(),\n ModelCheckpoint(path, monitor='val_roc_auc', save_best_only=True, mode='max')\n ]\nmodel.fit(X, y,\n batch_size= batch_size,\n epochs= epochs\n callbacks= callbacks,\n validation_data= validation_data)\n
Hi,\nIt seems it doesn't work with model.fit_generator . I got an error as follows: \n```\n14 def on_epoch_end(self, epoch, logs={}):\n---> 15 y_true = self.validation_data[1]'\n 16 y_pred = self.model.predict(self.validation_data[0])\n 17 score = roc_auc_score(y_true, y_pred)
TypeError: 'NoneType' object is not subscriptable\n```
\n\nAny idea how to solve this?
", "rawMarkdown": "Hi,\nIt seems it doesn't work with ` model.fit_generator` . I got an error as follows: \n```\n14 def on_epoch_end(self, epoch, logs={}):\n---> 15 y_true = self.validation_data[1]'\n 16 y_pred = self.model.predict(self.validation_data[0])\n 17 score = roc_auc_score(y_true, y_pred)\n\nTypeError: 'NoneType' object is not subscriptable\n```\n\nAny idea how to solve this? " } ] }, { "id": 449680, "postDate": "2019-01-03T15:19:48.077Z", "content": "Hi,\nWould it possible to use ModelCheckPiont with AUC score (or define a custom callsback) to save the best model with the highest AUC score on validation data?
", "rawMarkdown": "Hi,\nWould it possible to use ModelCheckPiont with AUC score (or define a custom callsback) to save the best model with the highest AUC score on validation data? ", "votes": 2 } ], "comments": [ { "id": 452799, "author_name": "William Huang", "author_url": "", "post_date": "2019-01-09T07:01:19.880000", "content": "Yes, you can do it.
\n\nBut before using ModelCheckpoint, you need to compute the AUC score on your validation data. It can be done by implement your custom callback to help you to compute it. The following is the simple example.
\n\n```python\nfrom keras.callbacks import Callback\nfrom sklearn.metrics import roc_auc_score
\n\nclass RocAucMetric(Callback):
\n\ndef on_train_begin(self, logs={}):\n # By default, self.params['metrics'] contains loss and the metric assigned in `model.compile()`\n if not 'val_roc_auc' in self.params['metrics']:\n self.params['metrics'].append('val_roc_auc')\n\n logs['val_roc_auc'] = float('-inf')\n\ndef on_epoch_end(self, epoch, logs={}):\n y_true = self.validation_data[1]\n y_pred = self.model.predict(self.validation_data[0])\n score = roc_auc_score(y_true, y_pred)\n\n logs['val_roc_auc'] = score\n```
\n\nThen you can set your callbacks and pass it to model.fit(). Remember to put RocAucMetric ahead of ModelCheckpoint so that val_roc_auc exists and ModelCheckpoint can find it.
python\ncallbacks = [RocAucMetric(),\n ModelCheckpoint(path, monitor='val_roc_auc', save_best_only=True, mode='max')\n ]\nmodel.fit(X, y,\n batch_size= batch_size,\n epochs= epochs\n callbacks= callbacks,\n validation_data= validation_data)\n
Hi,\nIt seems it doesn't work with model.fit_generator . I got an error as follows: \n```\n14 def on_epoch_end(self, epoch, logs={}):\n---> 15 y_true = self.validation_data[1]'\n 16 y_pred = self.model.predict(self.validation_data[0])\n 17 score = roc_auc_score(y_true, y_pred)
TypeError: 'NoneType' object is not subscriptable\n```
\n\nAny idea how to solve this?
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "452799": "Yes, you can do it.\n\nBut before using ModelCheckpoint, you need to compute the AUC score on your validation data. It can be done by implement your custom callback to help you to compute it. The following is the simple example.\n\n```python\nfrom keras.callbacks import Callback\nfrom sklearn.metrics import roc_auc_score\n\nclass RocAucMetric(Callback):\n\n def on_train_begin(self, logs={}):\n # By default, self.params['metrics'] contains loss and the metric assigned in `model.compile()`\n if not 'val_roc_auc' in self.params['metrics']:\n self.params['metrics'].append('val_roc_auc')\n\n logs['val_roc_auc'] = float('-inf')\n\n def on_epoch_end(self, epoch, logs={}):\n y_true = self.validation_data[1]\n y_pred = self.model.predict(self.validation_data[0])\n score = roc_auc_score(y_true, y_pred)\n\n logs['val_roc_auc'] = score\n```\n\nThen you can set your `callbacks` and pass it to `model.fit()`. Remember to put `RocAucMetric` ahead of `ModelCheckpoint` so that `val_roc_auc` exists and `ModelCheckpoint ` can find it.\n\n```python\ncallbacks = [RocAucMetric(),\n ModelCheckpoint(path, monitor='val_roc_auc', save_best_only=True, mode='max')\n ]\nmodel.fit(X, y,\n batch_size= batch_size,\n epochs= epochs\n callbacks= callbacks,\n validation_data= validation_data)\n```", "449680": "Hi,\nWould it possible to use ModelCheckPiont with AUC score (or define a custom callsback) to save the best model with the highest AUC score on validation data? " } }