{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h1><center> Classify foliar diseases in durain trees</center></h1>","metadata":{}},{"cell_type":"markdown","source":"#  1. Problem Statement ？\nDurain are one of the most important temperate fruit crops in Thailand. Foliar (leaf) diseases pose a major threat to the overall productivity and quality of Durain orchards. The current process for disease diagnosis in durain orchards is based on manual scouting by humans, which is time-consuming and expensive.\n\nThe main objective of the competition is to develop machine learning-based models to accurately classify a given leaf image from the test dataset to a particular disease category, and to identify an individual disease from multiple disease symptoms on a single leaf image.","metadata":{}},{"cell_type":"markdown","source":"## libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n%matplotlib inline\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport cv2\nimport os\nimport warnings\nwarnings.filterwarnings('ignore')\nimport tensorflow as tf\nimport random\nimport albumentations as A\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.layers import Dense,Activation,Flatten, Conv2D, MaxPooling2D\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.callbacks import ModelCheckpoint,EarlyStopping","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:09:46.72787Z","iopub.execute_input":"2022-02-17T13:09:46.728304Z","iopub.status.idle":"2022-02-17T13:09:54.883721Z","shell.execute_reply.started":"2022-02-17T13:09:46.728198Z","shell.execute_reply":"2022-02-17T13:09:54.882863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. About Dataset","metadata":{}},{"cell_type":"code","source":"train_image_path = '../input/plant-pathology-2021-fgvc8/train_images'\ntest_image_path = '../input/plant-pathology-2021-fgvc8/test_images'\ntrain_df_path = '../input/plant-pathology-2021-fgvc8/train.csv'\ntest_df_path = '../input/plant-pathology-2021-fgvc8/sample_submission.csv'","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-02-17T13:09:54.885451Z","iopub.execute_input":"2022-02-17T13:09:54.88573Z","iopub.status.idle":"2022-02-17T13:09:54.890372Z","shell.execute_reply.started":"2022-02-17T13:09:54.885691Z","shell.execute_reply":"2022-02-17T13:09:54.889431Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌**Note**:\n* `train.csv` contains information about the image files available in `train_images`. It contains 18632 rows(images) with 2 columns i.e (image , labels )\n* `test.csv` The test set images. This competition has a hidden test set: only three images are provided here as samples while the remaining 5,000 images will be available to your notebook once it is submitted.","metadata":{}},{"cell_type":"code","source":"df_train = pd.read_csv(train_df_path)\ndf_test=pd.read_csv(test_df_path)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:09:54.891573Z","iopub.execute_input":"2022-02-17T13:09:54.891854Z","iopub.status.idle":"2022-02-17T13:09:54.945663Z","shell.execute_reply.started":"2022-02-17T13:09:54.891815Z","shell.execute_reply":"2022-02-17T13:09:54.944971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_test","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:09:54.947721Z","iopub.execute_input":"2022-02-17T13:09:54.94814Z","iopub.status.idle":"2022-02-17T13:09:54.964858Z","shell.execute_reply.started":"2022-02-17T13:09:54.948103Z","shell.execute_reply":"2022-02-17T13:09:54.964046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train.labels.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:09:54.967424Z","iopub.execute_input":"2022-02-17T13:09:54.967827Z","iopub.status.idle":"2022-02-17T13:09:54.9823Z","shell.execute_reply.started":"2022-02-17T13:09:54.967796Z","shell.execute_reply":"2022-02-17T13:09:54.981485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(14,10))\nlabels = sns.barplot(df_train.labels.value_counts().index,df_train.labels.value_counts())\nfor item in labels.get_xticklabels():\n    item.set_rotation(40)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:09:54.983847Z","iopub.execute_input":"2022-02-17T13:09:54.98483Z","iopub.status.idle":"2022-02-17T13:09:55.33287Z","shell.execute_reply.started":"2022-02-17T13:09:54.984785Z","shell.execute_reply":"2022-02-17T13:09:55.332047Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 📌**Note**:\n* We have multiple labels for eg. label can be **scab** or **scab and rust**\n* Main labels are - **scab** , **healthy** , **frog_eye_leaf_spot** , **rust** , **complex** and **powdery_mildew**\n","metadata":{}},{"cell_type":"markdown","source":"## Batch Visualisation of Images ","metadata":{}},{"cell_type":"code","source":"def batch_visualize(df,batch_size,path):\n    sample_df = df_train.sample(9)\n    image_names = sample_df[\"image\"].values\n    labels = sample_df[\"labels\"].values\n    plt.figure(figsize=(13, 12))\n    \n    for image_ind, (image_name, label) in enumerate(zip(image_names, labels)):\n        plt.subplot(3, 3, image_ind + 1)\n        image = cv2.imread(os.path.join(path, image_name))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        plt.imshow(image)\n        plt.title(f\"{label}\", fontsize=13)\n        plt.axis(\"off\")\n    plt.show()\n    \nbatch_visualize(df_train,9,train_image_path)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:09:55.333944Z","iopub.execute_input":"2022-02-17T13:09:55.334216Z","iopub.status.idle":"2022-02-17T13:10:03.580185Z","shell.execute_reply.started":"2022-02-17T13:09:55.334188Z","shell.execute_reply":"2022-02-17T13:10:03.577807Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualise healthy leaves","metadata":{}},{"cell_type":"code","source":"batch_visualize_with_label(df_train,9,train_image_path,'healthy')","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.581421Z","iopub.status.idle":"2022-02-17T13:10:03.582229Z","shell.execute_reply.started":"2022-02-17T13:10:03.582015Z","shell.execute_reply":"2022-02-17T13:10:03.582039Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualise scab leaves","metadata":{}},{"cell_type":"code","source":"batch_visualize_with_label(df_train,9,train_image_path,'scab')","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.583068Z","iopub.status.idle":"2022-02-17T13:10:03.583621Z","shell.execute_reply.started":"2022-02-17T13:10:03.583447Z","shell.execute_reply":"2022-02-17T13:10:03.583467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualise frog_eye_leaf_spot leaves","metadata":{}},{"cell_type":"code","source":"batch_visualize_with_label(df_train,9,train_image_path,'frog_eye_leaf_spot')","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.584533Z","iopub.status.idle":"2022-02-17T13:10:03.585046Z","shell.execute_reply.started":"2022-02-17T13:10:03.584852Z","shell.execute_reply":"2022-02-17T13:10:03.584872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualise rust leaves","metadata":{}},{"cell_type":"code","source":"batch_visualize_with_label(df_train,9,train_image_path,'rust')","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.585977Z","iopub.status.idle":"2022-02-17T13:10:03.586312Z","shell.execute_reply.started":"2022-02-17T13:10:03.586135Z","shell.execute_reply":"2022-02-17T13:10:03.586151Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualise complex leaves","metadata":{}},{"cell_type":"code","source":"batch_visualize_with_label(df_train,9,train_image_path,'complex')","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.587278Z","iopub.status.idle":"2022-02-17T13:10:03.587564Z","shell.execute_reply.started":"2022-02-17T13:10:03.587411Z","shell.execute_reply":"2022-02-17T13:10:03.587427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualise powdery_mildew leaves","metadata":{}},{"cell_type":"code","source":"batch_visualize_with_label(df_train,9,train_image_path,'powdery_mildew')","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.588321Z","iopub.status.idle":"2022-02-17T13:10:03.588605Z","shell.execute_reply.started":"2022-02-17T13:10:03.588452Z","shell.execute_reply":"2022-02-17T13:10:03.588468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"3. Tensorflow Dataset Generation","metadata":{}},{"cell_type":"code","source":"HEIGHT = 128\nWIDTH=128\nSEED = 45\nBATCH_SIZE= 64\n\ntrain_datagen = ImageDataGenerator(rescale = 1/255.,\n    rotation_range=20,\n    width_shift_range=0.2,\n    height_shift_range=0.2,\n    horizontal_flip=True,\n    validation_split = 0.2,\n    zoom_range = 0.2,\n    shear_range = 0.2,\n    vertical_flip = False)\n\ntrain_dataset = train_datagen.flow_from_dataframe(\n    df_train,\n    directory = train_image_path,\n    x_col = \"image\",\n    y_col = \"labels\",\n    target_size = (HEIGHT,WIDTH),\n    class_mode='categorical',\n    batch_size = BATCH_SIZE,\n    subset = \"training\",\n    shuffle = True,\n    seed = SEED,\n    validate_filenames = False\n)\n\n\nvalidation_dataset = train_datagen.flow_from_dataframe(\n    df_train,\n    directory = train_image_path,\n    x_col = \"image\",\n    y_col = \"labels\",\n    target_size = (HEIGHT,WIDTH),\n    class_mode='categorical',\n    batch_size = BATCH_SIZE,\n    subset = \"validation\",\n    shuffle = True,\n    seed = SEED,\n    validate_filenames = False\n)\n\ntest_datagen = ImageDataGenerator(\n    rescale = 1./255\n)\nINPUT_SIZE = (HEIGHT,WIDTH,3)\ntest_dataset=test_datagen.flow_from_dataframe(\n    df_test,\n    directory=test_image_path,\n    x_col='image',\n    y_col=None,\n    class_mode=None,\n    target_size=INPUT_SIZE[:2]\n)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.5895Z","iopub.status.idle":"2022-02-17T13:10:03.589861Z","shell.execute_reply.started":"2022-02-17T13:10:03.589637Z","shell.execute_reply":"2022-02-17T13:10:03.589652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Found 14906 non-validated image filenames belonging to 12 classes.\nFound 3726 non-validated image filenames belonging to 12 classes.\nFound 3 validated image filenames.","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.5909Z","iopub.status.idle":"2022-02-17T13:10:03.591227Z","shell.execute_reply.started":"2022-02-17T13:10:03.591061Z","shell.execute_reply":"2022-02-17T13:10:03.591077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"4. Convolutional Neural Networks","metadata":{}},{"cell_type":"code","source":"model=Sequential()\nmodel.add(Conv2D(32,(3,3),activation='relu',padding='same',input_shape=(HEIGHT,WIDTH,3)))\nmodel.add(MaxPooling2D(2,2))\nmodel.add(Conv2D(64,(3,3),activation='relu',padding='same'))\nmodel.add(MaxPooling2D(2,2))\nmodel.add(Conv2D(64,(3,3),activation='relu',padding='same'))\nmodel.add(MaxPooling2D(2,2))\nmodel.add(Conv2D(128,(3,3),activation='relu',padding='same'))\nmodel.add(MaxPooling2D(2,2))\nmodel.add(Flatten())\nmodel.add(Dense(12,activation='softmax'))","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.592094Z","iopub.status.idle":"2022-02-17T13:10:03.592395Z","shell.execute_reply.started":"2022-02-17T13:10:03.592246Z","shell.execute_reply":"2022-02-17T13:10:03.592261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Prediction","metadata":{}},{"cell_type":"code","source":"train_dataset.class_indices.items()","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.59314Z","iopub.status.idle":"2022-02-17T13:10:03.593431Z","shell.execute_reply.started":"2022-02-17T13:10:03.593278Z","shell.execute_reply":"2022-02-17T13:10:03.593294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"dict_items([('complex', 0), ('frog_eye_leaf_spot', 1), ('frog_eye_leaf_spot complex', 2), ('healthy', 3), ('powdery_mildew', 4), ('powdery_mildew complex', 5), ('rust', 6), ('rust complex', 7), ('rust frog_eye_leaf_spot', 8), ('scab', 9), ('scab frog_eye_leaf_spot', 10), ('scab frog_eye_leaf_spot complex', 11)])","metadata":{}},{"cell_type":"code","source":"preds = model.predict(test_dataset)\nprint(preds)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.594217Z","iopub.status.idle":"2022-02-17T13:10:03.594515Z","shell.execute_reply.started":"2022-02-17T13:10:03.594361Z","shell.execute_reply":"2022-02-17T13:10:03.594377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"[[3.15346941e-02 4.21638459e-01 1.09025370e-03 1.72932621e-03\n  1.84729125e-03 9.81649573e-05 4.89124656e-01 1.07697621e-02\n  1.69256907e-02 1.15041025e-02 1.31795211e-02 5.58156404e-04]\n [1.72777532e-03 8.16138228e-04 2.62255389e-06 3.26147936e-02\n  1.11416727e-03 4.17654564e-06 9.13539171e-01 2.25775689e-03\n  1.37027148e-02 3.40761654e-02 1.41656958e-04 2.95801260e-06]\n [4.16421682e-01 1.59591630e-01 1.68289524e-02 1.07063204e-02\n  1.17181502e-01 9.84419230e-03 4.98372242e-02 2.66466383e-02\n  5.04306890e-02 8.48022178e-02 3.84849906e-02 1.92240179e-02]]","metadata":{}},{"cell_type":"code","source":"preds_disease_ind=np.argmax(preds, axis=-1)","metadata":{"execution":{"iopub.status.busy":"2022-02-17T13:10:03.595473Z","iopub.status.idle":"2022-02-17T13:10:03.595766Z","shell.execute_reply.started":"2022-02-17T13:10:03.595605Z","shell.execute_reply":"2022-02-17T13:10:03.595621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"preds_disease_ind","metadata":{}},{"cell_type":"markdown","source":"array([6, 6, 0])","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}