{"cells":[{"metadata":{},"cell_type":"markdown","source":"Hey kagglers,\n\nThis is my first competition in kaggle, I would like to share my approach on the Cassava leaf disease competition.\nMy approach involoves:\n* Using keras library\n* Using GPU to train the model\n* Using albumentations to augment the dataset to prevent overfitting\n* Using the StratifiedKFold as the dataset is skewed.\n\nThis notebook is for beginner who would like to get started with this competition\n\nReferences:\n1. Approaching (Almost) Any Machine Learning Problem - Book by Abhishek Thakur\n2. https://www.kaggle.com/junyingsg/end-to-end-cassava-disease-classification-in-keras#Image-Augmentation-(Albumentations)\n3. https://www.kaggle.com/tuckerarrants/cassava-tensorflow-starter-training\n\nPlease leave a upvote if you like this notebook"},{"metadata":{},"cell_type":"markdown","source":"# **Importing libraries**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport os \nfrom sklearn.model_selection import StratifiedKFold\nimport seaborn as sns\nimport json\nimport cv2\n\nimport tensorflow as tf\nfrom tensorflow.keras import models, layers,Sequential,regularizers\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau,EarlyStopping, ModelCheckpoint\nfrom keras.layers import Input, Flatten, Dense, Conv2D, MaxPooling2D, GlobalAveragePooling2D, Dropout\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.applications import Xception\nfrom tensorflow.keras.losses import CategoricalCrossentropy \nfrom tensorflow.keras.mixed_precision import experimental as mixed_precision\npolicy = mixed_precision.Policy('mixed_float16')\nmixed_precision.set_policy(policy) #shortens training time by 2x\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Cassava leaves are a rich source of protein, minerals, and vitamins. However, the presence of antinutrients and cyanogenic glucosides are the major drawbacks in cassava leaves which limit its human consumption. These antinutrients and toxic compounds of cassava leaves cause various diseases depending on the consumption level. But viral diseases are major sources of poor yields. With the help of data science, it may be possible to identify common diseases so they can be treated.\n\n![](https://www.rural21.com/fileadmin/_processed_/8/2/csm_Seite30_96bc157e8d.jpg)\n\nIn this competition, we are introduced with a dataset of 21,367 labeled images collected during a regular survey in Uganda. Our task is to classify each cassava image into four disease categories or a fifth category indicating a healthy leaf.\nThe label in the dataset is as follows:\n* 0 - CBB - Cassava Bacterial Blight\n* 1 - CBSD - Cassava Brown Streak Disease\n* 2 - CGM - Cassava Green Mottle\n* 3 - CMD - Cassava Mosaic Disease\n* 4 - Healthy"},{"metadata":{},"cell_type":"markdown","source":"# **Dataset and Kfold**"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_image_path=\"../input/cassava-leaf-disease-classification/train_images/\"\ntrain_df_path=\"../input/cassava-leaf-disease-classification/train.csv\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df=pd.read_csv(train_df_path)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(\"../input/cassava-leaf-disease-classification/label_num_to_disease_map.json\") as file:\n    map_classes = json.loads(file.read())\n    map_classes = {int(k) : v for k, v in map_classes.items()}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[\"Class\"]=train_df[\"label\"].map(map_classes)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set(rc={'figure.figsize':(8,4)})\nsns.set_style('whitegrid')\n\nva=sns.countplot(y=\"Class\",data=train_df,palette='rainbow')\nplt.xlabel(\"Classes of leaves\",fontsize=20)\nplt.ylabel(\"Count\",fontsize=20)\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize(image_ids, labels):\n    plt.figure(figsize=(16, 12))\n    for ind, (image_id, label) in enumerate(zip(image_ids, labels)):\n        plt.subplot(4, 4, ind + 1)\n        image = cv2.imread(os.path.join(\"../input/cassava-leaf-disease-classification/train_images\", image_id))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        plt.imshow(image)\n        plt.title(f\"Class: {label}\", fontsize=12)\n        plt.axis(\"off\")\n        plt.tight_layout()\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df1=train_df.sample(8)\nimage_ids = train_df1[\"image_id\"].values\nlabels = train_df1[\"Class\"].values\nvisualize(image_ids, labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.label.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can observe, The data set is skewed with the largest number of samples for label 3, Cassava Mosiac Disease (CMD), and the fewest number of samples for label 0, Cassava Bacterial Blight (CBB). To over come this we use StratifiedKfold to split the dataset for training and validation to check the performance of the model.\n\nThe general procedure is as follows:\n1. Shuffle the dataset randomly.\n2. Split the dataset into k groups\n3. For each unique group:\n    * Take the group as a holdout or test data set\n    * Take the remaining groups as a training data set\n    * Fit a model on the training set and evaluate it on the test set\n    * Retain the evaluation score and discard the model\n    * Summarize the skill of the model using the sample of model evaluation scores.\n    \nStratifiedKFold is a variation of KFold. First, StratifiedKFold shuffles your data, after that splits the data into n_splits parts and Done. Now, it will use each part as a test set. Note that it only and always shuffles data one time before splitting.\n\n![](https://miro.medium.com/max/703/0*QKJTHrcriSx2ZNYr.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def creat_nfold(train_df,n_split):\n    train_df.loc[:,\"kfold\"]=-1\n    train_df=train_df.sample(frac=1).reset_index(drop=True)\n    SS=StratifiedKFold(n_splits=n_split)\n    y=train_df.label.values\n    for fold,(t_,v_) in enumerate((SS.split(X=train_df,y=y))):\n        train_df.loc[v_,\"kfold\"]=fold\n    return train_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.label=train_df.label.astype(\"str\") \n#converting the label to str as we will be using categorical cross entropy as loss function to train the model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df=creat_nfold(train_df,5)\ntrain_df.head(5)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Augmentation**"},{"metadata":{},"cell_type":"markdown","source":"Image augmentations is done with the help of the library Albumentation through both ImageDataGenerator. We will use a tool called ImageDataAugmentor (thanks to mjkvaak at github) that allows us to do this. "},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install git+https://github.com/mjkvaak/ImageDataAugmentor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from ImageDataAugmentor.image_data_augmentor import *\nimport albumentations as A\n\n# augmentations referred from: https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug\ntrain_aug = albumentations.Compose([\n            albumentations.RandomResizedCrop(300, 300),\n            albumentations.Transpose(p=0.5),\n            albumentations.HorizontalFlip(p=0.5),\n            albumentations.VerticalFlip(p=0.5),\n            albumentations.ShiftScaleRotate(p=0.5),\n            albumentations.HueSaturationValue(\n                hue_shift_limit=0.2, \n                sat_shift_limit=0.2, \n                val_shift_limit=0.2, \n                p=0.5\n            ),\n            albumentations.RandomBrightnessContrast(\n                brightness_limit=(-0.2,0.2), \n                contrast_limit=(-0.2, 0.2), \n                p=0.5\n            ),\n            albumentations.Normalize(\n                mean=[0.485, 0.456, 0.406], \n                std=[0.229, 0.224, 0.225], \n                max_pixel_value=255.0, \n                p=1.0\n            ),\n            albumentations.CoarseDropout(p=0.5),\n            albumentations.Cutout(p=0.5),albumentations.ToFloat()], p=1.)\n  \n        \nvalid_aug = albumentations.Compose([\n            albumentations.CenterCrop(300, 300, p=1.),\n            albumentations.Normalize(\n                mean=[0.485, 0.456, 0.406], \n                std=[0.229, 0.224, 0.225], \n                max_pixel_value=255.0, \n                p=1.0\n            ),albumentations.ToFloat()], p=1.)\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_generator(train,valid,batch_size,image_size):\n    \n        datagen_train = ImageDataAugmentor(augment=train_aug)\n        datagen_val = ImageDataAugmentor(augment=valid_aug)\n        train_generator = datagen_train.flow_from_dataframe(dataframe=train,\n                                                    directory=train_image_path,\n                                                    x_col=\"image_id\",\n                                                    y_col=\"label\",\n                                                    subset=\"training\",\n                                                    batch_size=batch_size,\n                                                    seed=42,\n                                                    shuffle=True,\n                                                    class_mode=\"categorical\",\n                                                    target_size=(image_size,image_size))\n        val_generator = datagen_val.flow_from_dataframe(dataframe=valid,\n                                                    directory=train_image_path,\n                                                    x_col=\"image_id\",\n                                                    y_col=\"label\",\n                                                    subset=\"training\",\n                                                    batch_size=batch_size,\n                                                    seed=42,\n                                                    shuffle=False,\n                                                    class_mode=\"categorical\",\n                                                    target_size=(image_size,image_size))\n        \n        return train_generator, val_generator","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Define the model**"},{"metadata":{},"cell_type":"markdown","source":"We will use the Xception architecture to train the model. To read more about this architecture refer https://arxiv.org/abs/1610.02357"},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_model(IMG_SIZE):\n    base_model = Xception(input_shape = (IMG_SIZE, IMG_SIZE, 3), include_top = False,\n                             weights = 'imagenet')\n    x = base_model.output\n    x = GlobalAveragePooling2D()(x)\n    x = Dropout(0.3)(x)\n    predictions = Dense(5, activation='softmax',name='Final', dtype='float32')(x)\n\n    model = Model(inputs=base_model.input, outputs=predictions)\n\n    model.compile(optimizer =tf.keras.optimizers.SGD(learning_rate=0.01, momentum=0.9, nesterov=True),\n                  loss = CategoricalCrossentropy(from_logits = True,\n                                                   label_smoothing=0.2,\n                                                   name='categorical_crossentropy'),\n                  metrics = ['categorical_accuracy']) \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Training the model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def run_train(df,batch_size,image_size,fold):\n    \n    train=df[df.kfold!=fold].reset_index(drop=True)\n    valid=df[df.kfold==fold].reset_index(drop=True)\n    \n    train_gen,val_gen= train_generator(train,valid,batch_size,image_size)\n    \n    my_callbacks = [EarlyStopping(monitor = 'val_loss', min_delta = 0.001, \n                                  patience = 3, mode = 'min', verbose = 1,\n                                  restore_best_weights = True),\n                    ModelCheckpoint(filepath=f'model{fold}.h5', \n                                    save_best_only = True, \n                                    monitor = 'val_loss', \n                                    mode = 'min', verbose = 1),\n                    ReduceLROnPlateau(monitor='val_loss',\n                                      factor=0.1,\n                                      patience=2, \n                                      min_lr=0.00001,\n                                      mode='min',\n                                      verbose=1)]\n    \n    steps_per_epoch = train_gen.n//train_gen.batch_size\n    validation_steps = val_gen.n//val_gen.batch_size\n    \n    model=make_model(image_size)\n    \n    history = model.fit_generator(train_gen,\n                                  steps_per_epoch=steps_per_epoch,\n                                  validation_steps=validation_steps,\n                                  validation_data = val_gen,\n                                  epochs = 10, \n                                  callbacks =my_callbacks)\n    return model, history,train_gen, val_gen","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"oof_acc=[]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(5):\n        print(25*\"-\")    \n        print(f'{i}-fold training')\n        print(25*\"-\")\n        \n        model,history,train_gen, val_gen = run_train(train_df,16,300,i)\n\n        train_acc = history.history['categorical_accuracy']\n        val_acc = history.history['val_categorical_accuracy']\n        loss = history.history['loss']\n        val_loss = history.history['val_loss']\n        \n        oof_acc.append(val_acc)\n        \n        epochs = range(1, len(train_acc) + 1)\n        fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))\n        fig.set_size_inches(20,10)\n\n        ax1.plot(epochs , train_acc , 'go-' , label = 'Training Accuracy')\n        ax1.plot(epochs , val_acc , 'ro-' , label = 'Validation Accuracy')\n        ax1.set_title('Training & Validation Accuracy')\n        ax1.legend()\n        ax1.set_xlabel(\"Epochs\")\n        ax1.set_ylabel(\"Accuracy\")\n\n        ax2.plot(epochs , loss , 'g-o' , label = 'Training Loss')\n        ax2.plot(epochs , val_loss , 'r-o' , label = 'Validation Loss')\n        ax2.set_title('Testing Accuracy & Loss')\n        ax2.legend()\n        ax2.set_xlabel(\"Epochs\")\n        ax2.set_ylabel(\"Training & Validation Loss\")\n       \n        fig.tight_layout()\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(np.mean(oof_acc))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, classification_report\n        \npred = model.predict_generator(val_gen) # Gives class probabilities\npred = np.round(pred) # Gives one-hot encoded classes\npred = np.argmax(pred, axis = 1) # Gives class labels\n\n# Obtain actual labels\nactual = val_gen.classes\n    \n# Now plot matrix\nsns.set(rc={'figure.figsize':(10,10)})\nsns.set_style('whitegrid')\ncm = confusion_matrix(actual, pred, labels = [0,1,2,3,4])\nsns.heatmap(\n    cm, \n    cmap=\"Blues\",\n    annot = True, \n    fmt = \"d\"\n)\nplt.title(\"Confusion Matrix\", fontsize=12)\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(classification_report(actual,pred))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Work Under progress**"},{"metadata":{},"cell_type":"markdown","source":"# **Improvements**"},{"metadata":{},"cell_type":"markdown","source":"Some improvements that could possibly be made:\n\n1. Image augmentation (using cutmix etc.)\n2. Different learning rate and learning rate schedule\n3. Using TPU to decrease the training time\n4. Increased input size\n5. Add more dense layers and regularization\n6. Using other architectures such as EfficientNet\n\nIf this notebook helped you, please leave an upvote!"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}