{"cells":[{"metadata":{},"cell_type":"markdown","source":"If you are korean, please check out this notebook.\nhttps://www.kaggle.com/vkehfdl1/for-korean-cassava"},{"metadata":{},"cell_type":"markdown","source":"# **Cassava Competition Baseline**\n\nThis competition is classification problem that classify cassava leaves. There are 5 classes, 4 each diseases and 1 healthy class. Given images are 600x800 resolution, and test set does not reveal. \n\nPlease look up this EDA for characteristic of each disease. Thank you for @ihelon (Yaroslav Isaienkov).\nhttps://www.kaggle.com/ihelon/cassava-leaf-disease-exploratory-data-analysis\n\nThis baseline code refers to this notebook. Thank you for @frlemarchand (Francois Lemarchand).\nhttps://www.kaggle.com/frlemarchand/efficientnet-aug-tf-keras-for-cassava-diseases\n\nThere are detailed explanation at annotation.\n\nFeel free to leave comments below for questions. You can find accurate information looking up official documnets of libraries like tensorflow, numpy, pandas, scikit-learn.\n* Tensorflow - https://www.tensorflow.org/api_docs/python/tf\n* numpy - https://numpy.org/doc/1.19/\n* pandas - https://pandas.pydata.org/pandas-docs/stable/reference/index.html\n* scikit-learn - https://scikit-learn.org/stable/modules/classes.html"},{"metadata":{},"cell_type":"markdown","source":"# Import Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nfrom PIL import Image\nimport os\nimport matplotlib.pyplot as plt\nfrom sklearn.utils import shuffle\nfrom sklearn.utils import class_weight\nfrom sklearn.preprocessing import minmax_scale\nimport random\nimport cv2\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential, Model\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import Dense, Dropout, Activation, Input, BatchNormalization, GlobalAveragePooling2D\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\nfrom tensorflow.keras.experimental import CosineDecay\nfrom tensorflow.keras.applications import EfficientNetB0\nfrom tensorflow.keras.layers.experimental.preprocessing import RandomCrop,CenterCrop, RandomRotation\nfrom tensorflow.keras.optimizers import Adam","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Preprocessing (Image Load)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"training_folder = '../input/cassava-leaf-disease-classification/train_images/' # Folder that train images located\nsamples_df = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv') # Load train image file names and each label data\nsamples_df[\"filepath\"] = training_folder+samples_df[\"image_id\"] # Create path by adding folder name and image name for load images easily\nsamples_df = samples_df.drop(['image_id'],axis=1) # Drop image names which is useless.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"samples_df = shuffle(samples_df, random_state=42) # Shuffle all data randomly\ntrain_size = int(len(samples_df)*0.8) # Define data set size for training\ntraining_df = samples_df[:train_size] # Make training dataset\nvalidation_df = samples_df[train_size:] # Make validation dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 8 # Set batch size\nimage_size = 512 # Set image size\ninput_shape = (image_size, image_size, 3) # Set image shape (Require 3 numbers per pixel becuase it is color images)\ndropout_rate = 0.4 # Set dropout rate\nclasses_to_predict = sorted(training_df.label.unique()) # Set number of classes which is 5 in this competition","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nWhen we use train dataset and validation dataset, we applicate tensorflow dataset.\nTensorflow dataset can allocate data to device dynamically, so you can improve performance because it prevents overloading.\nFor details, please refer to below link.\nhttps://www.tensorflow.org/guide/data_performance?hl=en\n\"\"\"\ntraining_data = tf.data.Dataset.from_tensor_slices((training_df.filepath.values, training_df.label.values))\nvalidation_data = tf.data.Dataset.from_tensor_slices((validation_df.filepath.values, validation_df.label.values))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_image_and_label_from_path(image_path, label): # Function : Load image data and transform to tensor (simillar with array)\n    img = tf.io.read_file(image_path) # Read file from image path\n    img = tf.image.decode_jpeg(img, channels=3) # Transform image to array and save it\n    img = tf.image.random_crop(img, size=[image_size,image_size,3]) # Crop images to desired size. You can use \"central_crop\" if you want to crop middle part of the images, not randomly. \n    return img, label\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE # AUTOTUNE for dynamic memory allocation\ntraining_data = training_data.map(load_image_and_label_from_path, num_parallel_calls=AUTOTUNE) # Load train data\nvalidation_data = validation_data.map(load_image_and_label_from_path,num_parallel_calls=AUTOTUNE) # Load validation data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Cut train and validation data for training easily.\ntraining_data_batches = training_data.shuffle(buffer_size=1000).batch(batch_size).prefetch(buffer_size=AUTOTUNE)\nvalidation_data_batches = validation_data.shuffle(buffer_size=1000).batch(batch_size).prefetch(buffer_size=AUTOTUNE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nCreate augmentation layer for augmentation. When you put augmentation layer to your model, it will transform images automatically. \nIf you want to adapt more augmentations, please refer to this link :  https://www.tensorflow.org/api_docs/python/tf/keras/layers/experimental/preprocessing\nAlso, refer to imgaug or albumentation, one of the most powerful augmentation libraries.\n\"\"\"\ndata_augmentation_layers = tf.keras.Sequential(\n    [\n        layers.experimental.preprocessing.RandomFlip(\"horizontal_and_vertical\"), #Flip images randomly\n        layers.experimental.preprocessing.RandomRotation(0.25), #Rotate images randomly\n        layers.experimental.preprocessing.RandomZoom((-0.2, 0)), #Zoom out images randomly\n        \n    ]\n)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build model & train model"},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nIn this baseline, we use transfer learning. Transfer learninng is method that load pre-trained large model, add custom layer at the end of it, and train it. \nWe use transfer learning because design numerous layers on our own is very hard.\nWe use EfficientNetB0. For details, please refer to this article : https://arxiv.org/pdf/1905.11946.pdf\n\nCaution!! You must turn on the Internet with pressing |< at the right top for downloading imagenet weights. \n\"\"\"\nefficientnet = EfficientNetB0(weights=\"imagenet\", #Download imagenet weights\n                              include_top=False, \n                              input_shape=input_shape, \n                              drop_connect_rate=dropout_rate) #Load EfficientNetB0 model\nefficientnet.trainable=True # Enable training EfficientNetB0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nIt's okay to build your own CNN model\n\"\"\"\nmodel = Sequential() #Build new Sequential model \nmodel.add(Input(shape=input_shape)) #Set input to image size\nmodel.add(data_augmentation_layers) #Add image augmentation layer\nmodel.add(efficientnet) # Add EfficientNetB0\nmodel.add(layers.GlobalAveragePooling2D()) # Add pooling layer\nmodel.add(layers.Dropout(dropout_rate)) # Add dropout layer for avoiding overfitting\nmodel.add(Dense(len(classes_to_predict), activation=\"softmax\")) #Add last Dense layer. Classes to predict becomes output size\nmodel.summary() #Check model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 30 #Set epochs\ndecay_steps = int(round(len(training_df)/batch_size))*epochs\ncosine_decay = CosineDecay(initial_learning_rate=1e-4, decay_steps=decay_steps, alpha=0.3) #Use cosien decay : Decaying learning rate per epochs.\ncallbacks = [ModelCheckpoint(filepath='mymodel.h5', monitor='val_loss', save_best_only=True), # Save best model (the lowest validation loss) to .h5 format.\n            EarlyStopping(monitor='val_loss', patience = 5, verbose=1)] #Stop training when validation loss doesn't improve while 5 epochs.\n\nmodel.compile(loss=\"sparse_categorical_crossentropy\", optimizer=Adam(cosine_decay), metrics=[\"accuracy\"]) #Loss is sparse_categorical_crossentropy and optimizer is Adam. Use accuracy for monitoring model's performance. ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model.fit(training_data_batches, #Train model\n                  epochs = epochs, \n                  validation_data=validation_data_batches,\n                  callbacks=callbacks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Predict test data (Submission)"},{"metadata":{},"cell_type":"markdown","source":"Below, there are codes for submission. The notebook must run whole things again when you submit it, so if you train your models again, it is really inefficient. So you can save lots of time by splitting train and submission notebooks. Below is explanation about it. \n\n1. Find saved .h5 file at 'output -> /kaggle/working'. And press three dots at right side and download it. \n2. Go to the kaggle 'Data' tab and press '+New Dataset'.\n3. Enter dataset name, and upload your .h5 file. Next, press 'Create' button. \n4. Make new notebook for submission, and press 'Add data' at the upper right. Go to the 'Your Dataset', and add dataset you just made by pressing blue 'Add' button.\n5. Load your model using 'model = tf.keras.models.load_model('filepath')'\nFYI : You can easily copy file path by pressing little button called 'Copy File path'."},{"metadata":{"trusted":true},"cell_type":"code","source":"\"\"\"\nmodel = tf.keras.models.load_model('filepath')\n\"\"\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Load test data. Same way with loading trian data.\ntest_folder = '../input/cassava-leaf-disease-classification/test_images/' \nsubmission_df = pd.DataFrame(columns={\"image_id\",\"label\",\"filepath\"})\nsubmission_df[\"image_id\"] =  os.listdir(test_folder) #Put image names in test foler to \"image_id\"\nsubmission_df[\"label\"] = 0\nsubmission_df[\"filepath\"] = test_folder+submission_df['image_id']\nsubmission_df = submission_df.drop(['image_id'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_image(image_path): #Function : Load image data and transform to tensor (simillar with array)\n    img = tf.io.read_file(image_path) #Read file from image path\n    img = tf.image.decode_jpeg(img, channels=3) #Transform image to array and save it\n    img = tf.image.random_crop(img, size=[image_size,image_size,3]) # Crop images to desired size. You can use \"central_crop\" if you want to crop middle part of the images, not randomly.\n    img = np.reshape(img, [-1,512,512,3]) #Transform images 3-d tensor to 4-d tensor\n    return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def test_predict(filepath):\n    local_image = load_image(filepath) #load image using 'load_image' function\n    predictions = model.predict(local_image) #Predict each class probabilities\n    final_prediction = np.argmax(predictions) #Return final result (the highest probability in classes)\n    return final_prediction #Return prediction result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def predictions_over_image(filepath):\n    predictions = [] #List that save predictions\n    for path in filepath:   \n        predictions.append(test_predict(path)) # Save predictions for each images\n    return predictions #Return prediction values","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df[\"label\"] = predictions_over_image(submission_df[\"filepath\"]) # Put predictions to submission DataFrame","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.to_csv(\"submission.csv\", index=False) # Make submission csv file","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Things you can try for improving score\n\n1. Use large models ex) EfflicientNetB4\n2. Use heavy augmentation ex) CoarseDropout, Cutmix, Mixup\n3. Use TTA (Test Time Augmentation)  \n4. Use Stratified Kfold : Because data is unbalanced.\n5. Use another callbacks strategy\n6. Use another loss function like Symmetric Cross Entropy    FYI : https://arxiv.org/abs/1908.06112\n7. Use another optimizers like Lookahead or RAdam\n8. Use more datasets    https://www.kaggle.com/tahsin/cassava-leaf-disease-merged\n9. Use various ensemble methods ex) Bagging, voting, stacking, etc..\n10. Design new models"},{"metadata":{},"cell_type":"markdown","source":"**If this notebook is helpful, please upvote :) Thank you!!**"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}