{"cells":[{"metadata":{},"cell_type":"markdown","source":"# [micro tip]Control weight save(Keras callback)"},{"metadata":{},"cell_type":"markdown","source":"When use Kaggle kernel, file I/O takes lots of time to save model weights. In this, I make custom `ModelCheckpoint` callbaks to control the point save model weights using `epoch` or `accuracy` etc...\n\nI apply APTOS model and work without error. In the kerenl, just make simple Resnet model and training mnist dataset to check fast.\n\nReference code is @Keeplearning's [[APTOS] resnet50 baseline](https://www.kaggle.com/mathormad/aptos-resnet50-baseline/output). "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\n\nimport numpy as np\nimport pandas as pd\nimport os\nimport matplotlib.pyplot as plt\n\nfrom keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau, CSVLogger\n\nprint(os.listdir('../input'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make custom ModelCheckpoint callback"},{"metadata":{},"cell_type":"markdown","source":"Basic `ModelCheckpoint` callback make using some parameters like that([Keras callback doc](https://keras.io/callbacks/#modelcheckpoint)):\n    \n    keras.callbacks.ModelCheckpoint(filepath, monitor='val_loss', verbose=0, save_best_only=False, save_weights_only=False, mode='auto', period=1)"},{"metadata":{},"cell_type":"markdown","source":"Let make custom `ModelCheckpoint` callback. I override `on_epoch_end` function simply adding 'epoch condition'. I use some `print` to check `epoch` and `logs`. Delete this part after using.\n\nOriginal Keras `callback` code is in [keras.callback.ModelCheckpoint](https://github.com/tensorflow/tensorflow/blob/r1.14/tensorflow/python/keras/callbacks.py#L805-L1139)"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import ModelCheckpoint","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Custom_checkpoint(ModelCheckpoint):\n    def on_epoch_end(self, epoch, logs=None):\n        print('epoch : ', epoch) #Just check \n        print('logs : ', logs)  #Just check\n        #Set this epoch\n        if epoch <= 2: \n            return print('Low epoch')\n        super().on_epoch_end(epoch, logs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cp = Custom_checkpoint('../working/Resnet50.h5', monitor='val_loss', verbose=1, save_best_only=True, mode='min', save_weights_only=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Create other callbacks.\nreduceLROnPlat = ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=4, \n                                   verbose=1, mode='min', epsilon=0.0001)\nearly = EarlyStopping(monitor=\"val_loss\", \n                      mode=\"min\", \n                      patience=9)\n\ncsv_logger = CSVLogger(filename='../working/training_log.csv',\n                       separator=',',\n                       append=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model training"},{"metadata":{},"cell_type":"markdown","source":"mnist model training code is in [Keras mnist cnn](https://keras.io/examples/mnist_cnn/)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from __future__ import print_function\nimport keras\nfrom keras.datasets import mnist\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Dropout, Flatten\nfrom keras.layers import Conv2D, MaxPooling2D\nfrom keras import backend as K\n\nbatch_size = 128\nnum_classes = 10\nepochs = 12\n\n# input image dimensions\nimg_rows, img_cols = 28, 28\n\n# the data, split between train and test sets\n(x_train, y_train), (x_test, y_test) = mnist.load_data()\n\nif K.image_data_format() == 'channels_first':\n    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n    input_shape = (1, img_rows, img_cols)\nelse:\n    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n    input_shape = (img_rows, img_cols, 1)\n\nx_train = x_train.astype('float32')\nx_test = x_test.astype('float32')\nx_train /= 255\nx_test /= 255\nprint('x_train shape:', x_train.shape)\nprint(x_train.shape[0], 'train samples')\nprint(x_test.shape[0], 'test samples')\n\n# convert class vectors to binary class matrices\ny_train = keras.utils.to_categorical(y_train, num_classes)\ny_test = keras.utils.to_categorical(y_test, num_classes)\n\nmodel = Sequential()\nmodel.add(Conv2D(32, kernel_size=(3, 3),\n                 activation='relu',\n                 input_shape=input_shape))\nmodel.add(Conv2D(64, (3, 3), activation='relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\nmodel.add(Flatten())\nmodel.add(Dense(128, activation='relu'))\nmodel.add(Dropout(0.5))\nmodel.add(Dense(num_classes, activation='softmax'))\n\nmodel.compile(loss=keras.losses.categorical_crossentropy,\n              optimizer=keras.optimizers.Adadelta(),\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(x_train, y_train,\n          batch_size=batch_size,\n          epochs=5,\n          verbose=1,\n          validation_data=(x_test, y_test),\n          callbacks=[cp, reduceLROnPlat])\nscore = model.evaluate(x_test, y_test, verbose=0)\nprint('Test loss:', score[0])\nprint('Test accuracy:', score[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Model weights saved after `epoch` over 2(setting number). We can see that `epoch` start 0 and `logs` is a dict writen 'val_loss', 'val_acc' etc.\n\nI make other `ModelCheckpoint` to use 'val_loss'."},{"metadata":{"trusted":true},"cell_type":"code","source":"class Custom_checkpoint(ModelCheckpoint):\n    def on_epoch_end(self, epoch, logs=None):\n        if logs['val_acc'] <= 0.99:\n            return print('Low val_acc')\n        super().on_epoch_end(epoch, logs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cp2 = Custom_checkpoint('../working/Resnet50.h5', monitor='val_loss', verbose=1, save_best_only=True, mode='min', save_weights_only=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.fit(x_train, y_train,\n          batch_size=batch_size,\n          epochs=5,\n          verbose=1,\n          validation_data=(x_test, y_test),\n          callbacks=[cp2, reduceLROnPlat])\nscore = model.evaluate(x_test, y_test, verbose=0)\nprint('Test loss:', score[0])\nprint('Test accuracy:', score[1])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In [APTOS] resnet50 baseline, make `QWKEvaluation` callback option. We can apply control option like above.\n\nI make `QWKEvaluation` after epoch is over 20."},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.callbacks import Callback\nclass QWKEvaluation(Callback):\n    def __init__(self, validation_data=(), batch_size=64, interval=1):\n        super(Callback, self).__init__()\n\n        self.interval = interval\n        self.batch_size = batch_size\n        self.valid_generator, self.y_val = validation_data\n        self.history = []\n\n    def on_epoch_end(self, epoch, logs={}):\n        ###Add epoch condition\n        if epoch <= 20:\n            return print('Low epoch : ', epoch)\n        \n        ###Org code###\n        if epoch % self.interval == 0:\n            y_pred = self.model.predict_generator(generator=self.valid_generator,\n                                                  steps=np.ceil(float(len(self.y_val)) / float(self.batch_size)),\n                                                  workers=1, use_multiprocessing=True,\n                                                  verbose=1)\n            def flatten(y):\n                return np.argmax(y, axis=1).reshape(-1)\n                # return np.sum(y.astype(int), axis=1) - 1\n            \n            score = cohen_kappa_score(flatten(self.y_val),\n                                      flatten(y_pred),\n                                      labels=[0,1,2,3,4],\n                                      weights='quadratic')\n#             print(flatten(self.y_val)[:5])\n#             print(flatten(y_pred)[:5])\n            print(\"\\n epoch: %d - QWK_score: %.6f \\n\" % (epoch+1, score))\n            self.history.append(score)\n            if score >= max(self.history):\n                print('save checkpoint: ', score)\n                self.model.save('../working/Resnet50_bestqwk.h5')\n\nqwk = QWKEvaluation(validation_data=(valid_generator, valid_y),\n                    batch_size=batch_size, interval=1)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}