{"cells":[{"metadata":{},"cell_type":"markdown","source":"added dataset: https://www.kaggle.com/srg9000/cassava-plant-disease-merged-20192020 (Merged data from 2019 and 2020 competition)"},{"metadata":{"id":"dTuh4Q4pQXiz"},"cell_type":"markdown","source":"# Using the combined and unlabelled data to pretrain a model\nOne of the methods to make the network learn important image features is training it with autoencoders.\n\nAutoencoders are pair of Encoder and Decoder networks which try to recreate the source image (or some variant, such as denoised image). The encoder encodes the image in a smaller dimension, whereas the decoder tries to recreate the image from the encodings produced by decoder.\n\nMore on Autoencoders: https://en.wikipedia.org/wiki/Autoencoder"},{"metadata":{"id":"bLyHjSQivh45","trusted":true},"cell_type":"code","source":"import numpy as np\nimport tensorflow as tf\nimport tensorflow.keras as keras","execution_count":null,"outputs":[]},{"metadata":{"id":"UrrHp06-wsMP","trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [512, 512]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Image augmentation and creating generators"},{"metadata":{"id":"devh_lLxyF9m","trusted":true},"cell_type":"code","source":"def preprocess_func(image):\n    image = tf.image.random_saturation(image, 0.9, 2)\n    image = tf.image.random_contrast(image, lower=0.8, upper=1.2)\n    return image\n\ndatagen = tf.keras.preprocessing.image.ImageDataGenerator(\n    rescale=1./255,\n    rotation_range=20,\n    brightness_range=(0.8, 1.2),\n    shear_range=5.0,\n    zoom_range=0.2, \n    fill_mode=\"constant\",\n    validation_split=0.3,\n    preprocessing_function=preprocess_func\n)\n","execution_count":null,"outputs":[]},{"metadata":{"id":"3eMbWKYszBsL","outputId":"b7804650-71f8-452b-be75-26d0e8e7dbaa","trusted":true},"cell_type":"code","source":"train_gen = datagen.flow_from_directory('../input/cassava-plant-disease-merged-20192020/extra_images', target_size=(512, 512), batch_size=3, class_mode='input', subset='training')\nval_gen = datagen.flow_from_directory('../input/cassava-plant-disease-merged-20192020/extra_images', target_size=(512, 512), batch_size=3, class_mode='input', subset='validation')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualize a batch\nimport matplotlib.pyplot as plt\nx = next(iter(train_gen))\nplt.imshow(x[0][0])\nplt.show()\nplt.imshow(x[0][1])\nplt.show()\nplt.imshow(x[0][2])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Creating the model"},{"metadata":{},"cell_type":"markdown","source":"Here, I have used an EfficientNetB5 as the base model and instead of flattening and reshaping the outputs, I have taken the upscaling from 2D activations only.\n\nI have used Conv2D layers along with Transposed convolutions as this sometimes improves the results by not giving the Mosaic like output of transposed convs. (It is more of a personal choice and I have not checked the outputs in this case)\n\nInstead of dilation = 2, dilation and stride of 1 worked better for me in TransposedConvolutions"},{"metadata":{"id":"Caj9w_YGlNla","trusted":true},"cell_type":"code","source":"def get_upsample_layers(x, filters, tx_size, block_name):\n    with keras.backend.name_scope('decode_'+block_name):\n        x = tf.keras.layers.Conv2DTranspose(filters, tx_size, strides=(1, 1), padding='valid',dilation_rate=(1, 1))(x)\n        x = tf.keras.layers.BatchNormalization()(x)\n        x = tf.keras.layers.Dropout(0.1)(x)\n        x = tf.keras.layers.UpSampling2D((2,2))(x)\n        x = tf.keras.layers.Conv2D(filters, (3,3), activation='relu')(x)\n        # x = tf.keras.layers.LeakyReLU(alpha=0.3)(x)\n    return x\n\ndef make_model():\n    base_model = tf.keras.applications.EfficientNetB5(\n        input_shape=(*IMAGE_SIZE, 3), include_top=False, weights='imagenet'\n    )\n\n    base_model.trainable = True\n    for layer in base_model.layers:\n        layer.trainable = True\n        layer._trainable = True\n\n    inputs = tf.keras.layers.Input([*IMAGE_SIZE, 3])\n    # x = tf.keras.applications.densenet.preprocess_input(inputs)\n    x = base_model(inputs)\n    x = tf.keras.layers.Conv2D(512, (1,1))(x)\n    x = tf.keras.layers.BatchNormalization()(x)\n    x = tf.keras.layers.Conv2D(256, (1,1))(x)\n    x = get_upsample_layers(x, 256, (2,2), '1')\n    x = get_upsample_layers(x, 256, (2,2), '2')\n    x = get_upsample_layers(x, 128, (2,2), '3')\n    x = get_upsample_layers(x, 128, (2,2), '4')\n    x = tf.keras.layers.MaxPool2D((2,2))(x)\n    x = tf.keras.layers.Conv2D(64, (3,3))(x)\n    x = get_upsample_layers(x, 32, (4,4), '5')\n    x = get_upsample_layers(x, 32, (4,4), '6')\n    x = tf.keras.layers.Conv2D(16, (3,3), activation='relu')(x)\n    x = tf.keras.layers.Conv2D(3, (3,3), activation='sigmoid')(x)\n    model = tf.keras.Model(inputs=inputs, outputs=x)      \n\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{"id":"rDATAnfUvdnM","trusted":true},"cell_type":"code","source":"model = make_model()","execution_count":null,"outputs":[]},{"metadata":{"id":"s4IzKiyMxnKw","trusted":true},"cell_type":"code","source":"model.summary()","execution_count":null,"outputs":[]},{"metadata":{"id":"84TqbLZ6pBOb","trusted":true},"cell_type":"code","source":"initial_learning_rate = 0.0001\nlr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(\n    initial_learning_rate, decay_steps=1000, decay_rate=0.95, staircase=True\n)\nmodel.compile(\n    optimizer=tf.keras.optimizers.Adam(learning_rate=lr_schedule),\n    loss=keras.losses.MSE\n)","execution_count":null,"outputs":[]},{"metadata":{"id":"LEUihQMOu2uX","trusted":true},"cell_type":"code","source":"callbacks = [\n             tf.keras.callbacks.ModelCheckpoint('./pretrained2_cp{epoch:.2f}.h5', save_best_only=True, monitor='loss'),\n             tf.keras.callbacks.EarlyStopping(patience=10),\n]","execution_count":null,"outputs":[]},{"metadata":{"id":"23NRIfwAyEl_","outputId":"0c97c585-6239-4170-f7a0-dcd4b21c3721","trusted":true},"cell_type":"code","source":"# Check if model architecture works on properly as per inputs and outputs provided\nmodel.fit(np.zeros((1, 512, 512, 3)), np.zeros((1, 512, 512, 3)), verbose=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Fit the model"},{"metadata":{"id":"7zOoXf3BxjAh","trusted":true},"cell_type":"code","source":"tf.keras.backend.clear_session()","execution_count":null,"outputs":[]},{"metadata":{"id":"rEklIoxSBv8X","outputId":"d65d7c5a-5177-47ae-a6e8-1b79d9567fce","trusted":true},"cell_type":"code","source":"model.fit(train_gen, verbose=1, epochs=1, callbacks=callbacks, validation_data=val_gen)","execution_count":null,"outputs":[]},{"metadata":{"id":"lifT-RK5S3XF","trusted":false},"cell_type":"code","source":"# model.save('pretrained3.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check results"},{"metadata":{"id":"zI4LDmfyC500","outputId":"a523ee1f-3494-4b06-a929-44a28fe5da7a","trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nx = next(iter(train_gen))\nplt.imshow(x[0][0])\nplt.show()\nplt.imshow(x[0][1])\nplt.show()\nplt.imshow(x[0][2])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"id":"tvlVX660bS5e","outputId":"93c7e53d-31d8-4a29-e9d4-0044c5407207","trusted":true},"cell_type":"code","source":"# x = next(iter(train_gen))\n# model.fit(x[0],x[0], epochs=10, verbose=0)\nxx = model.predict(x[0])\nplt.imshow(xx[0])\nplt.show()\nplt.imshow(xx[1])\nplt.show()\nplt.imshow(xx[2])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After one epoch, the results seem like they will improve (they did for me), and we can add training data images for pretraining as well."},{"metadata":{},"cell_type":"markdown","source":"## Separate the encoder"},{"metadata":{"trusted":true},"cell_type":"code","source":"model_encoder = keras.models.Model(model.layers[1].input, model.layers[1].outputs) # separate the base model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Thank you"},{"metadata":{},"cell_type":"markdown","source":"### Using autoencoders, we might be able to learn general shape of plants but when it comes to diseases, it comes down to smaller patches on leaves, which don't seem like major components for reconstructing the source image. In this case, we can use something like deep-metric learning where we can learn differentiating features among classes (Will try to publish a notebook for that soon)"},{"metadata":{},"cell_type":"markdown","source":"If you find this kernel helpful, do upvote it and also comment any suggestions."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}