{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cassava Leaf Disease Exploratory Data Analysis"},{"metadata":{},"cell_type":"markdown","source":"This is an EDA to explore the dataset for the Cassava Leaf Disease Classification challenge\n\nManihot esculenta or the cassava plant is the most widely grown root crop in the world and the third largest source of carbohydrates for human food. However, it is vunerable to a number of viral diseases.\n\nThe aim of the challenge is the classify images of cassava leaves into 1 of 5 categories: 4 types of disease or a healthy leaf.\n\nHere we will look at a dataset of labelled images to use in training a classification model."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Imports\n!pip install -q imutils\nimport numpy as np\nimport pandas as pd\n\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nimport os\nfrom pathlib import Path\nimport cv2,imutils\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Filepath\nbase_path = Path('../input/cassava-leaf-disease-classification')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Read in train.csv and labels mapped to disease names\ntrain_df = pd.read_csv(base_path/'train.csv')\ndisease_map = pd.read_json(base_path/'label_num_to_disease_map.json', typ='series')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Map disease names to labels in dataframe\nmap_dict = disease_map.to_dict()\nshort_map_dict = {0:'CBB', 1:'CBSD', 2:'CGM', 3:'CMD', 4:'Healthy'}\ntrain_df['class'] = train_df['label']\ntrain_df = train_df.replace({'class':map_dict})\ntrain_df['class_short'] = train_df['label']\ntrain_df = train_df.replace({'class_short':short_map_dict})\ntrain_df = train_df.rename(columns={'label':'label_number'})\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now each image has a number and a disease name to classify it"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of total images\nlen_df = len(train_df)\nprint(f\"Number of train images: {len_df}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Number of images in each class\ntrain_df['class'].value_counts().sort_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# No. of images in each class plotted\ntrain_df['class'].value_counts().sort_index().plot(kind='bar')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see there is a huge class imbalance."},{"metadata":{},"cell_type":"markdown","source":"## Resolution of images"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Finding height and width of each image\nfrom PIL import Image\nwidth_list = []\nheight_list = []\nfor i in range(0,len_df):  \n    im = Image.open(base_path/'train_images'/train_df['image_id'][i])\n    width, height = im.size\n    width_list.append(width)\n    height_list.append(height)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"width_list.count(width_list[0]) == len(width_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"height_list.count(height_list[0]) == len(height_list)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since both the cells above result in True, the width and height of all the images are the same, 800 and 600 pixels respectively"},{"metadata":{},"cell_type":"markdown","source":"## Detecting outliers"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_path = '../input/cassava-leaf-disease-classification/train_images/'\n\nfor i in range(0,len_df):\n    img_name = train_df.loc[i,'image_id']\n    img = cv2.cvtColor(cv2.imread(train_path+img_name), cv2.COLOR_BGR2RGB)\n    #normalize each image in the range [0,1]\n    norm_image = cv2.normalize(img, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)\n    \n    #add columns for width and height\n    width,height,depth = img.shape\n    train_df.loc[i,'width'] = width\n    train_df.loc[i,'height'] = height\n    \n    #calculate the mean for each original and normalized image\n    train_df.loc[i,'mean'] = img.mean()\n    train_df.loc[i,'normalized_mean'] = norm_image.mean()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean_images = [0,0,0]\nnorm_mean_images = [0,0,0]\nfor i in range(0,len_df):\n    img_name = train_df.loc[i,'image_id']\n    img = cv2.cvtColor(cv2.imread(train_path+img_name), cv2.COLOR_BGR2RGB)\n    #normalize each image in the range [0,1]\n    norm_img = cv2.normalize(img, None, alpha=0, beta=1, norm_type=cv2.NORM_MINMAX, dtype=cv2.CV_32F)\n    \n    mean_images += np.mean(img, axis=(0,1))\n    norm_mean_images += np.mean(norm_img, axis=(0,1))\n    \nmean_images = mean_images/len_df/255\nnorm_mean_images = norm_mean_images/len_df/255    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"std = []\n\ntrain_path = '../input/cassava-leaf-disease-classification/train_images/'\nfiles = os.listdir(train_path)\nx = np.array([np.array(Image.open(train_path + fname)) for fname in files])\n\nfor image in x:\n    c = 0\n    r_std = image[:,:,c].flatten()\n    c+=1\n    g_std = image[:,:,c].flatten()\n    c+=1\n    b_std = image[:,:,c].flatten()\n    std.append(np.array([r_std,g_std,b_std]))\n\nnp_std = np.array(std)\n\nr_channel = np_std[:,0].flatten()\ng_channel = np_std[:,1].flatten()\nb_channel = np_std[:,2].flatten()\n\nr_norm = np.std(r_channel) / 255\ng_norm = np.std(g_channel) / 255\nb_norm = np.std(b_channel) / 255\nimgs_std = np.array([r_norm,g_norm,b_norm])\nprint(imgs_std)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#create box plots\nfig = make_subplots(rows=1,cols=2,\n                    subplot_titles=['Mean','Normalized Mean'])\ncolours = ['rgb(69, 130, 191)','rgb(102, 179, 70)','rgb(237, 232, 71)','rgb(240, 158, 50)','rgb(222, 55, 29)']\n\nfor i,class_name in enumerate(train_df['class_short'].unique()):\n    \n    #create subplot for mean\n    fig.add_trace(go.Box(y=train_df[train_df['class_short'] == class_name]['mean'],\n                        name=class_name,showlegend=False,\n                        fillcolor=colours[i],\n                        marker=dict(size=6,\n                        color='black',\n                        outliercolor = colours[i]),\n                        line=dict(width=1)),1,1)\n    \n    #create subplot for normalized mean\n    fig.add_trace(go.Box(y=train_df[train_df['class_short'] == class_name]['normalized_mean'],\n                        name=class_name,showlegend=False,\n                        fillcolor=colours[i],\n                        marker=dict(size=6,\n                        color='black',\n                        outliercolor = colours[i]),\n                        line=dict(width=1)),1,2)\n\nfig.update_layout(title='Outlier Detection Box Plots')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Put each class into a list to display them\nhealthy_images = train_df[train_df['class_short'] == 'Healthy']['image_id'].to_list()\ncbb_images = train_df[train_df['class_short'] == 'CBB']['image_id'].to_list()\ncbsd_images = train_df[train_df['class_short'] == 'CBSD']['image_id'].to_list()\ncgm_images = train_df[train_df['class_short'] == 'CGM']['image_id'].to_list()\ncmd_images = train_df[train_df['class_short'] == 'CMD']['image_id'].to_list()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#display a random sample of 9 images for each class\ndef showImages(images):\n    random_images = [np.random.choice(images) for i in range(9)]\n    plt.figure(figsize=(10,9))\n    for i in range(9):\n        plt.subplot(3, 3, i + 1)\n        img = plt.imread(base_path/'train_images'/random_images[i])\n        plt.imshow(img, cmap='gray')\n        plt.axis('off')\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def findOutliers(df):\n    Q1 = df['normalized_mean'].quantile(0.25)\n    Q3 = df['normalized_mean'].quantile(0.75)\n    IQR = Q3 - Q1\n    len_df = len(df)\n    # label each image as True if it is an outlier, False is it isn't\n    for i in range(0,len_df):\n        if df.loc[i,'normalized_mean'] < (Q1 - 1.5 * IQR) or df.loc[i,'normalized_mean'] > (Q3 + 1.5 * IQR):\n            df.loc[i,'outlier'] = True\n        else:\n            df.loc[i,'outlier'] = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def displayOutliers(df, class_name):\n    # Get only outlier images\n    outlier_df = df.loc[df['outlier'] == True]\n    outlier_images = outlier_df['image_id'].to_list()\n    print(f'Number of Outlier Images in {class_name} Class: '+str(len(outlier_images))+'\\n')\n    # Plot outlier images\n    plt.figure(figsize=(10,len(outlier_images)))\n    for i in range(0,len(outlier_images)):\n        plt.subplot(len(outlier_images)/3, 3, i + 1)\n        img = plt.imread(base_path/'train_images'/outlier_images[i])\n        plt.imshow(img, cmap='gray')\n        plt.axis('off')\n    plt.tight_layout()\n    \n    # return outlier_df as class_name_outlier_df\n    return outlier_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **Healthy leaf images**"},{"metadata":{"trusted":true},"cell_type":"code","source":"healthy_df  = train_df[train_df['class_short'] ==  'Healthy']\nhealthy_df.reset_index(inplace = True)\nprint('Number of Images in the Healthy Class: '+str(len(healthy_df)))\nshowImages(healthy_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Healthy leaf outliers"},{"metadata":{"trusted":true},"cell_type":"code","source":"findOutliers(healthy_df)\nhealthy_outlier_df = displayOutliers(healthy_df, 'Healthy')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **CBB (Cassava Bacterial Blight) images**"},{"metadata":{"trusted":true},"cell_type":"code","source":"CBB_df  = train_df[train_df['class_short'] ==  'CBB']\nCBB_df.reset_index(inplace = True)\nprint('Number of Images in the CBB Class: '+str(len(CBB_df)))\nshowImages(cbb_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CBB outliers"},{"metadata":{"trusted":true},"cell_type":"code","source":"findOutliers(CBB_df)\nCBB_outlier_df = displayOutliers(CBB_df, 'CBB')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CBB_outlier_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **CBSD (Cassava Brown Streak Disease) images**"},{"metadata":{"trusted":true},"cell_type":"code","source":"CBSD_df  = train_df[train_df['class_short'] ==  'CBSD']\nCBSD_df.reset_index(inplace = True)\nprint('Number of Images in the CBSD Class: '+str(len(CBSD_df)))\nshowImages(cbsd_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CBSD outliers"},{"metadata":{"trusted":true},"cell_type":"code","source":"findOutliers(CBSD_df)\nCBSD_outlier_df = displayOutliers(CBSD_df, 'CBSD')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **CGM (Cassava Green Mottle) images**"},{"metadata":{"trusted":true},"cell_type":"code","source":"CGM_df  = train_df[train_df['class_short'] ==  'CGM']\nCGM_df.reset_index(inplace = True)\nprint('Number of Images in the CGM Class: '+str(len(CGM_df)))\nshowImages(cgm_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CGM outliers"},{"metadata":{"trusted":true},"cell_type":"code","source":"findOutliers(CGM_df)\nCGM_outlier_df displayOutliers(CGM_df, 'CGM')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## **CMD (Cassava Mosiac Disease) images**"},{"metadata":{"trusted":true},"cell_type":"code","source":"CMD_df  = train_df[train_df['class_short'] ==  'CMD']\nCMD_df.reset_index(inplace = True)\nprint('Number of Images in the CMD Class: '+str(len(CMD_df)))\nshowImages(cmd_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### CMD outliers"},{"metadata":{"trusted":true},"cell_type":"code","source":"findOutliers(CMD_df)\nCMD_outlier_df = displayOutliers(CMD_df, 'CMD')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Remove outlier from train_df\ncond = train_df['image_id'].isin(CBB_outlier_df['image_id'])\ntrain_df.drop(train_df[cond].index, inplace = True)\ntrain_df = train_df.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}