{"cells":[{"metadata":{},"cell_type":"markdown","source":"Objective:\n\nObjective of this kernel is to introduce transfer learning to beginners. I have taken the following deep neural network applications \n- ResNet50\n- InceptionV3\n- Xception\n- NASNet\n\n**Accuracy versus Computational Demand (Left) and Number of Parameters (Right)**\n![Accuracy versus Computational Demand (Left) and Number of Parameters (Right)\n](https://miro.medium.com/max/1000/1*jQZ_oJ3VZbVWiBWuc5SMMg.png)\n\n### Transfer Learning\nTransfer learning is an ML methodology that enables to reuse a model developed for one task to another task.\nThe applications are predominantly in Deep Learning for computer vision and natural language processing.\nThis kernel introduces one on how to use Keras transfer learning applications.\n\n\n## ResNet50 (APTOS Accuracy: 0.396)\nCreated By: Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun  \nLiterature: [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385)  \nTopological Depth: **152 Layers**  \nImageNet Validation Accuracy: **Top-1 Accuracy**: 0.749 **Top-5 Accuracy**: 0.921  \n\n## InceptionV3 (APTOS Accuracy: 0.559)\nCreated By: Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna  \nLiterature: [Rethinking the Inception Architecture for Computer Vision](https://arxiv.org/abs/1512.00567)  \nTopological Depth: **159 Layers**  \nImageNet Validation Accuracy: **Top-1 Accuracy**: 0.779 **Top-5 Accuracy**: 0.937 \n\n## Xception (APTOS Accuracy: 0.509)\nCreated By: François Chollet  \nLiterature: [Xception: Deep Learning with Depthwise Separable Convolutions](https://arxiv.org/abs/1610.02357)  \nTopological Depth: **126 Layers**  \nImageNet Validation Accuracy: **Top-1 Accuracy**: 0.790 **Top-5 Accuracy**: 0.945 \n\n## NASNet (APTOS Accuracy: TBD)\nCreated By: Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le  \nLiterature: [Learning Transferable Architectures for Scalable Image Recognition](https://arxiv.org/abs/1707.07012)  \nTopological Depth: **~1040**  \nImageNet Validation Accuracy: **Top-1 Accuracy**: 0.825 **Top-5 Accuracy**: 0.960 \n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preprocess"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\ntrain_df = pd.read_csv(\"../input/aptos2019-blindness-detection/train.csv\")\nprint(\"Shape of train data: {0}\".format(train_df.shape))\ntest_df = pd.read_csv(\"../input/aptos2019-blindness-detection/test.csv\")\nprint(\"Shape of test data: {0}\".format(test_df.shape))\n\ndiagnosis_df = pd.DataFrame({\n    'diagnosis': [0, 1, 2, 3, 4],\n    'diagnosis_label': ['No DR', 'Mild', 'Moderate', 'Severe', 'Proliferative DR']\n})\n\ntrain_df = train_df.merge(diagnosis_df, how=\"left\", on=\"diagnosis\")\n\ntrain_image_files = [os.path.join(dp, f) for dp, dn, fn in os.walk(os.path.expanduser(\"../input/aptos2019-blindness-detection/train_images\")) for f in fn]\ntrain_images_df = pd.DataFrame({\n    'files': train_image_files,\n    'id_code': [file.split('/')[4].split('.')[0] for file in train_image_files],\n})\ntrain_df = train_df.merge(train_images_df, how=\"left\", on=\"id_code\")\ndel train_images_df\nprint(\"Shape of train data: {0}\".format(train_df.shape))\n\ntest_image_files = [os.path.join(dp, f) for dp, dn, fn in os.walk(os.path.expanduser(\"../input/aptos2019-blindness-detection/test_images\")) for f in fn]\ntest_images_df = pd.DataFrame({\n    'files': test_image_files,\n    'id_code': [file.split('/')[4].split('.')[0] for file in test_image_files],\n})\n\n\ntest_df = test_df.merge(test_images_df, how=\"left\", on=\"id_code\")\ndel test_images_df\nprint(\"Shape of test data: {0}\".format(test_df.shape))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_SIZE = 150\nN_CLASSES = train_df.diagnosis.nunique()\nCLASSES = list(map(str, range(N_CLASSES)))\nBATCH_SIZE = 32\nEPOCH_STEPS = 10\nEPOCHS = 25\nNB_FILTERS = 32\nKERNEL_SIZE = 4\nCHANNELS = 3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Generator: Train, Validation and Test Datasets"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nprint(tf.__version__)\n\nfrom keras.preprocessing.image import ImageDataGenerator\n\ntrain_df[\"diagnosis\"] = train_df[\"diagnosis\"].astype(str)\n\ntrain_data_gen = ImageDataGenerator(\n    # featurewise_center = True, # Set input mean to 0 over the dataset\n    samplewise_center = True, # set each sample mean to 0\n    featurewise_std_normalization = True, # Divide inputs by std of the dataset\n    samplewise_std_normalization = True, # Divide each input by its std\n    # zca_whitening = True, # Apply ZCA whitening\n    zca_epsilon = 1e-06, # Epsilon for ZCA whitening,\n    rotation_range = 30, # randomly rotate imges in the range (degrees, 0 to 189)\n    width_shift_range = 0.1, # randomly shift images horizontally (fraction of total width)\n    height_shift_range = 0.1, # Randomly shift images vertically (fraction of total height)\n    shear_range = 0, # set range for random shear\n    zoom_range = [0.75, 1.25], # set range for random zoom\n    channel_shift_range = 0.05, # set range for random channel shifts\n    fill_mode = 'constant', # set mode for filling points outside the input boundaries\n    cval = 0, # value used for fill_mode\n    horizontal_flip = True,\n    vertical_flip = True,\n    rescale = 1/255.,\n    preprocessing_function = None,\n    validation_split=0.1\n)\ntrain_data = train_data_gen.flow_from_dataframe(\n    dataframe=train_df, \n    x_col=\"files\",\n    y_col=\"diagnosis\",\n    batch_size=BATCH_SIZE,\n    shuffle=True,\n    classes=CLASSES,\n    class_mode=\"categorical\",\n    target_size=(IMG_SIZE, IMG_SIZE),\n    subset=\"training\"\n)\n\nvalidation_data = train_data_gen.flow_from_dataframe(\n    dataframe=train_df, \n    x_col=\"files\",\n    y_col=\"diagnosis\",\n    batch_size=BATCH_SIZE,\n    shuffle=True,\n    classes=CLASSES,\n    class_mode=\"categorical\",\n    target_size=(IMG_SIZE, IMG_SIZE),\n    subset=\"validation\"\n)\n\ntest_data_gen = ImageDataGenerator(rescale=1./255)\ntest_data = test_data_gen.flow_from_dataframe(\n    dataframe=test_df,\n    x_col=\"files\",\n    target_size=(IMG_SIZE, IMG_SIZE),\n    batch_size = 1,\n    shuffle=False,\n    class_mode=None\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Transfer Learning Assets"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.python.keras.applications import ResNet50, InceptionV3, Xception, NASNetLarge\nprint(os.listdir((\"../input/keras-pretrained-models/\")))\nprint(os.listdir((\"../input/nasnetlarge/\")))\n\nmodel_resnet50 = ResNet50(\n    weights=\"../input/keras-pretrained-models/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5\", \n    include_top=False, \n    input_tensor=tf.keras.layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))\n)\n\n\nmodel_inception_v3 = InceptionV3(\n    weights=\"../input/keras-pretrained-models/inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5\", \n    include_top=False, \n    input_tensor=tf.keras.layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))\n)\n\nmodel_xception = Xception(\n    weights=\"../input/keras-pretrained-models/xception_weights_tf_dim_ordering_tf_kernels_notop.h5\", \n    include_top=False, \n    input_tensor=tf.keras.layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))\n)\n\nmodel_nasnet_large = NASNetLarge(\n    weights=\"../input/nasnetlarge/NASNet-large-no-top.h5\", \n    include_top=False, \n    input_tensor=tf.keras.layers.Input(shape=(IMG_SIZE, IMG_SIZE, 3))\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\ndef create_model():\n    model = tf.keras.models.Sequential([\n        tf.keras.layers.Conv2D(NB_FILTERS, (KERNEL_SIZE, KERNEL_SIZE), padding=\"valid\", strides=1, input_shape=(IMG_SIZE, IMG_SIZE, CHANNELS), activation=\"relu\"),\n        tf.keras.layers.Conv2D(NB_FILTERS, (KERNEL_SIZE, KERNEL_SIZE), activation=\"relu\"),\n        tf.keras.layers.Conv2D(NB_FILTERS, (KERNEL_SIZE, KERNEL_SIZE), activation=\"relu\"),\n        tf.keras.layers.MaxPooling2D(pool_size=(8, 8)),\n        tf.keras.layers.Flatten(),\n        tf.keras.layers.Dense(2048, activation=\"relu\"),\n        tf.keras.layers.Dropout(0.25),\n        tf.keras.layers.Dense(2048, activation=\"relu\"),\n        tf.keras.layers.Dropout(0.25),\n        tf.keras.layers.Dense(N_CLASSES, activation=\"softmax\")\n        \n    ])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Resnet50: 0.396\n# create_model: 0.152\n# InceptionV3: 0.559\n# Xception: 0.509\n\ndef get_model(model):\n    X = model.output\n\n    X = tf.keras.layers.GlobalAveragePooling2D()(X)\n    X = tf.keras.layers.Dense(2048, activation='relu')(X)\n    X = tf.keras.layers.Dropout(0.25)(X)\n    X = tf.keras.layers.Dense(1024, activation='relu')(X)\n    X = tf.keras.layers.Dropout(0.25)(X)\n    X = tf.keras.layers.Dense(512, activation='relu')(X)\n    X = tf.keras.layers.Dropout(0.25)(X)\n    X = tf.keras.layers.Dense(256, activation='relu')(X)\n    X = tf.keras.layers.Dropout(0.25)(X)\n    X = tf.keras.layers.Dense(128, activation='relu')(X)\n    predictions = tf.keras.layers.Dense(N_CLASSES, activation='softmax')(X)\n    model = tf.keras.Model(inputs=model.input, outputs=predictions)\n    \n#     for layer in model.layers:\n#         layer.trainable = True\n        \n#     for layer in model.layers[15:]:\n#         layer.trainable = False\n    \n    return model\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Optimize, Compile, Train and Predict"},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer=tf.keras.optimizers.Nadam(lr=2*1e-3, schedule_decay=1e-5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"algo = \"inception_v3\"\nklass = \"basics\"\n\nmodel = get_model(model_inception_v3)\n\nopt = tf.keras.optimizers.Adam(lr=0.001, epsilon=1e-6)\nmodel.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])\nmodel.summary()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import cohen_kappa_score\nclass QWKCallback(tf.keras.callbacks.Callback):\n    def __init__(self, validation_data):\n        super(tf.keras.callbacks.Callback, self).__init__()\n        self.X = validation_data[0]\n        self.Y = validation_data[1]\n        self.history = []\n        \n    def on_epoch_end(self, epoch, logs={}):\n        pred = self.model.predict(self.X)\n        score = cohen_kappa_score(\n            np.argmax(self.Y, axis=1), np.argmax(pred, axis=1), labels=[0, 1, 2, 3, 4], weights=\"quadratic\"\n        )\n        print((\"Epoch {0} : QWK : {1}\".format(epoch, score)))\n        self.history.append(score)\n        if(score >= max(self.history)):\n            print(\"Saving Checkpoint: {0}\".format(score))\n            self.model.save(\"../Resnet50_bestQWK.h5\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"early_stopping = tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\", \n    min_delta=0.0001, patience=3, verbose=1, mode=\"auto\")\nreduce_lr = tf.keras.callbacks.ReduceLROnPlateau(monitor=\"val_loss\", \n    min_delta=0.0004, patience=2, factor=0.1, min_lr=1e-6, mode=\"auto\", verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"qwk = QWKCallback(validation_data)\nmodel.fit_generator(\n    generator=train_data,\n    #steps_per_epochs=EPOCH_STEPS,\n    #batch_size=BATCH_SIZE,\n    epochs=EPOCHS,\n    validation_data=validation_data,\n    validation_steps=30#,\n    #callbacks=[early_stopping, reduce_lr]\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filenames = test_data.filenames\nclassifications = model.predict_generator(test_data, steps=len(filenames))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results = pd.DataFrame({\n    \"id_code\": filenames,\n    \"diagnosis\": np.argmax(classifications, axis=1)\n})\nresults[\"id_code\"] = results[\"id_code\"].map(lambda x: str(x)[:-4].split(\"/\")[4])\nresults.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"file_name = \"{0}_{1}.csv\".format(algo, klass)\nresults.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"results.diagnosis.value_counts()\n\nlen(model.layers)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}