{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from IPython.display import Image\nImage(\"../input/cassava-leaf-disease-classification/train_images/100042118.jpg\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Credits**\n\n* [Notebook](https://www.kaggle.com/kretes/eda-distributions-images-and-no-duplicates)"},{"metadata":{},"cell_type":"markdown","source":"As the second-largest provider of carbohydrates in Africa, cassava is a key food security crop grown by smallholder farmers because it can withstand harsh conditions. At least 80% of household farms in Sub-Saharan Africa grow this starchy root, but viral diseases are major sources of poor yields. With the help of data science, it may be possible to identify common diseases so they can be treated.\n\nExisting methods of disease detection require farmers to solicit the help of government-funded agricultural experts to visually inspect and diagnose the plants. This suffers from being labor-intensive, low-supply and costly. As an added challenge, effective solutions for farmers must perform well under significant constraints, since African farmers may only have access to mobile-quality cameras with low-bandwidth.\n\nIn this competition, we introduce a dataset of 21,367 labeled images collected during a regular survey in Uganda. Most images were crowdsourced from farmers taking photos of their gardens, and annotated by experts at the National Crops Resources Research Institute (NaCRRI) in collaboration with the AI lab at Makerere University, Kampala. This is in a format that most realistically represents what farmers would need to diagnose in real life.\n\nYour task is to classify each cassava image into four disease categories or a fifth category indicating a healthy leaf. With your help, farmers may be able to quickly identify diseased plants, potentially saving their crops before they inflict irreparable damage."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport cv2\nimport json\nimport sys\nfrom glob import glob\nimport gc\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n%matplotlib inline\nimport matplotlib.pyplot as plt\nfrom IPython.display import Image,display\nimport seaborn as sns\nimport matplotlib.image as mpimg\nimport scipy.spatial.distance as dist\nfrom sklearn.model_selection import train_test_split\nfrom skimage.measure import compare_ssim\nimport os\nfrom sklearn.preprocessing import LabelBinarizer,LabelEncoder\nfrom sklearn.utils.class_weight import compute_class_weight\n\nimport math, re\nimport tensorflow as tf\nfrom kaggle_datasets import KaggleDatasets\nfrom tensorflow import keras\nfrom functools import partial\nfrom tensorflow.keras.layers.experimental import preprocessing\nfrom tensorflow.keras.models import Sequential\nfrom sklearn.model_selection import train_test_split\nprint(\"Tensorflow version \" + tf.__version__)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\ntrain.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"with open(\"../input/cassava-leaf-disease-classification/label_num_to_disease_map.json\") as f:\n    map_dis = json.loads(f.read())\nprint(\"Image Class Labels:\")\nprint(json.dumps(map_dis, indent=4))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's check for any missing values in the train labels df**"},{"metadata":{"trusted":true},"cell_type":"code","source":"missing = train.isnull().sum()\nall_val = train.count()\n\nmissing_train = pd.concat([missing, all_val], axis=1, keys=['Missing', 'AllObservations'])\nmissing_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Looks good. Let's review the class distributions**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"## Distinct classes\nprint(\"Distinct number of classes is :\" + str(train['label'].nunique()))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (8, 8))\nplt.title('Leaf Classes Density plot')\nsns.kdeplot(train['label'], color=\"blue\", shade=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**There is a good amount of class imbalance where a large section of images belong to class 3 only**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"leaf_id_count = pd.DataFrame(train.groupby(['label'])['label'].count())\nleaf_id_count.rename(columns={'label': 'Count_Images'}, inplace=True)\nleaf_id_count.reset_index(inplace=True)\nleaf_id_count.sort_values(by=['Count_Images'],ascending=False, inplace=True)\nleaf_id_count['Cummulative_Count'] = leaf_id_count['Count_Images'].cumsum()\nleaf_id_count['Cummulative_Pctg']= leaf_id_count['Cummulative_Count']/leaf_id_count['Count_Images'].sum()\nleaf_id_count['Row_id'] = np.arange(len(leaf_id_count))\nfig = plt.figure()\nax = plt.axes()\nax.plot(leaf_id_count['Row_id'], leaf_id_count['Cummulative_Pctg']);\nax.set(xlabel='Count of Classes', ylabel='Cummulative %',\n       title='Cummulative distribution of images by class count');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Let's compare the images of majority class vs. minority classes**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"mainPath = '../input/cassava-leaf-disease-classification/train_images/'\nall_img_paths = [y for x in os.walk(mainPath) for y in glob(os.path.join(x[0], '*.jpg'))]\nall_filenames = []\nfor filepath in all_img_paths:\n    FileName = os.path.basename(filepath)\n    all_filenames.append(FileName)\npath_dict = dict(zip(all_filenames,all_img_paths))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"all_cats= train.label.unique().tolist()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for img_cat in all_cats:\n    process_img_lst = train[train['label']==img_cat].image_id.tolist()[0:4]\n    full_img_paths = [path_dict[x] for x in process_img_lst]\n    print(\"Sample image in Category:\" + str(map_dis[str(img_cat)]))\n    img0 = mpimg.imread(full_img_paths[0])\n    img1 = mpimg.imread(full_img_paths[1])\n    img2 = mpimg.imread(full_img_paths[2])\n    img3 = mpimg.imread(full_img_paths[3])\n    \n    fig, ((ax0,ax1),(ax2,ax3)) = plt.subplots(nrows=2,ncols=2,figsize=(15,10))\n    ax0.imshow(img0)\n    ax0.set_title(\"Image 1\")\n    ax1.imshow(img1)\n    ax1.set_title(\"Image 2\")\n    ax2.imshow(img2)\n    ax2.set_title(\"Image 3\")\n    ax3.imshow(img3)\n    ax3.set_title(\"Image 4\")\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Looking beyond what meets the eye in the pictures**\n\n\nFor this excercise we will use intensity histograms"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"img0 = cv2.imread('../input/cassava-leaf-disease-classification/train_images/1000015157.jpg')\n\ntrain_hist = plt.hist(img0.ravel(), bins = 256, color = 'orange', )\ntrain_hist = plt.hist(img0[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\ntrain_hist = plt.hist(img0[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\ntrain_hist = plt.hist(img0[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\ntrain_hist = plt.xlabel('Intensity Value')\ntrain_hist = plt.ylabel('Count')\ntrain_hist = plt.legend(['Total', 'Red Channel', 'Green Channel', 'Blue Channel'])\nprint('Intensity Histogram of Sample Image for Cassava Bacterial Blight (CBB)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"img0 = cv2.imread('../input/cassava-leaf-disease-classification/train_images/1000201771.jpg')\n\ntrain_hist = plt.hist(img0.ravel(), bins = 256, color = 'orange', )\ntrain_hist = plt.hist(img0[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\ntrain_hist = plt.hist(img0[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\ntrain_hist = plt.hist(img0[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\ntrain_hist = plt.xlabel('Intensity Value')\ntrain_hist = plt.ylabel('Count')\ntrain_hist = plt.legend(['Total', 'Red Channel', 'Green Channel', 'Blue Channel'])\nprint('Intensity Histogram of Sample image for Cassava Mosaic Disease (CMD)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"img0 = cv2.imread('../input/cassava-leaf-disease-classification/train_images/1001723730.jpg')\n\ntrain_hist = plt.hist(img0.ravel(), bins = 256, color = 'orange', )\ntrain_hist = plt.hist(img0[:, :, 0].ravel(), bins = 256, color = 'red', alpha = 0.5)\ntrain_hist = plt.hist(img0[:, :, 1].ravel(), bins = 256, color = 'Green', alpha = 0.5)\ntrain_hist = plt.hist(img0[:, :, 2].ravel(), bins = 256, color = 'Blue', alpha = 0.5)\ntrain_hist = plt.xlabel('Intensity Value')\ntrain_hist = plt.ylabel('Count')\ntrain_hist = plt.legend(['Total', 'Red Channel', 'Green Channel', 'Blue Channel'])\nprint('Intensity Histogram of Sample image for Healthy Leaf')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**There could be interesting insights to how the diseases get manifested in the images in the underlying RGB channels**"},{"metadata":{},"cell_type":"markdown","source":"**Let's look at Image resolutions provided in the training set**"},{"metadata":{"trusted":true},"cell_type":"code","source":"import collections\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"DIR = '../input/cassava-leaf-disease-classification/train_images'\nimageSizes_train = collections.Counter([Image.open(f'{DIR}/{filename}').size\n                        for filename in os.listdir(f\"{DIR}/\")])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def isdf(imageSizes):\n    imageSizeFrame = pd.DataFrame(list(imageSizes.most_common()),columns = [\"imageDim\",\"count\"])\n    imageSizeFrame['fraction'] = imageSizeFrame['count'] / sum(imageSizes.values())\n    imageSizeFrame['count_cum'] = imageSizeFrame['count'].cumsum()\n    imageSizeFrame['count_cum_fraction'] = imageSizeFrame['count_cum'] / sum(imageSizes.values())\n    return imageSizeFrame\n\ntrain_isdf = isdf(imageSizes_train)\ntrain_isdf['set'] = 'train'\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_isdf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**All the images are of the size 800X600. For image classification case we need to change this to square format like 512X512 or 224X224. We will take care of this in our Data generator**"},{"metadata":{},"cell_type":"markdown","source":"**Let's build a Baseline model on GPU using any Pre trained networks**\n\nFor this demonstration we will use EfficientNet"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport skimage.io\nfrom skimage.transform import resize\nfrom imgaug import augmenters as iaa\nfrom sklearn import preprocessing \nfrom sklearn.preprocessing import LabelBinarizer,LabelEncoder\nfrom sklearn.utils.class_weight import compute_class_weight\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.layers import Input, Add, Dense, Dropout, Activation, ZeroPadding2D, BatchNormalization, Flatten, Conv2D, AveragePooling2D, MaxPooling2D, GlobalMaxPooling2D,GlobalAveragePooling2D,Concatenate, ReLU, LeakyReLU,Reshape, Lambda\nfrom tensorflow.keras.callbacks import ModelCheckpoint, LearningRateScheduler, EarlyStopping, ReduceLROnPlateau\nfrom tensorflow.keras.optimizers import Adam,SGD\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras.models import Sequential, load_model, Model\nfrom tensorflow.keras.callbacks import LearningRateScheduler\nfrom tensorflow.keras.applications import EfficientNetB0 \nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.applications.vgg16 import preprocess_input\nfrom tensorflow.keras.applications.vgg16 import decode_predictions\nfrom tensorflow.keras.utils import to_categorical\nfrom tensorflow.keras import metrics\nfrom tensorflow.keras.preprocessing import image\nfrom tensorflow.keras.applications.imagenet_utils import preprocess_input\nfrom tqdm import tqdm\nimport imgaug as ia\nfrom imgaug import augmenters as iaa\nimport keras.backend as K\nK.set_image_data_format('channels_last')\nK.set_learning_phase(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##Basic Model params\nbatch_size = 64\nseed = 42\nshape = (224, 224, 3) ##desired shape of the image for resizing purposes\nval_sample = 0.2 # 10 % as validation sample","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path_df = pd.DataFrame(path_dict.items())\npath_df.rename(columns={ path_df.columns[0]: \"image_id\" }, inplace = True)\npath_df.rename(columns={ path_df.columns[1]: \"path\" }, inplace = True)\ntrain_all = pd.merge(train, path_df, on='image_id')\n#del path_df,path_dict,all_img_paths,all_filenames\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_all.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**UDFs**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def getTrainParams():\n    data = train_all.copy()\n    le = preprocessing.LabelEncoder()\n    data['label'] = le.fit_transform(data['label'])\n    lbls = data['label'].tolist()\n    lb = LabelBinarizer()\n    labels = lb.fit_transform(lbls)\n    \n    return np.array(train_all['path'].tolist()),np.array(labels),le","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Leaf_DataGenerator(tf.keras.utils.Sequence):\n    def __init__(self, paths, labels, batch_size, shape, shuffle = False, use_cache = False, augment = False):\n        self.paths, self.labels = paths, labels\n        self.batch_size = batch_size\n        self.shape = shape\n        self.shuffle = shuffle\n        self.use_cache = use_cache\n        self.augment = augment\n        if use_cache == True:\n            self.cache = np.zeros((paths.shape[0], shape[0], shape[1], shape[2]), dtype=np.float16)\n            self.is_cached = np.zeros((paths.shape[0]))\n        self.on_epoch_end()\n    \n    def __len__(self):\n        return int(np.ceil(len(self.paths) / float(self.batch_size)))\n    \n    def __getitem__(self, idx):\n        indexes = self.indexes[idx * self.batch_size : (idx+1) * self.batch_size]\n\n        paths = self.paths[indexes]\n        X = np.zeros((paths.shape[0], self.shape[0], self.shape[1], self.shape[2]))\n        # Generate data\n        if self.use_cache == True:\n            X = self.cache[indexes]\n            for i, path in enumerate(paths[np.where(self.is_cached[indexes] == 0)]):\n                image = self.__load_image(path)\n                self.is_cached[indexes[i]] = 1\n                self.cache[indexes[i]] = image\n                X[i] = image\n        else:\n            for i, path in enumerate(paths):\n                X[i] = self.__load_image(path)\n\n        y = self.labels[indexes]\n                \n        if self.augment == True:\n            seq = iaa.Sequential([\n                iaa.OneOf([\n                    iaa.Fliplr(0.5), # horizontal flips\n                    iaa.Crop(percent=(0, 0.1)), # random crops\n                    #iaa.ContrastNormalization((0.75, 1.5)),\n                    #iaa.AdditiveGaussianNoise(loc=0, scale=(0.0, 0.05*255), per_channel=0.5),\n                    #iaa.Multiply((0.8, 1.2), per_channel=0.2),\n                    #iaa.Affine(rotate=90),\n                    #iaa.Affine(rotate=180),\n                    #iaa.Affine(rotate=270),\n                    iaa.Flipud(0.5),\n                ])], random_order=True)\n\n            X = np.concatenate((X, seq.augment_images(X), seq.augment_images(X), seq.augment_images(X)), 0)\n            X = X.astype('float32')\n            y = np.concatenate((y, y, y, y), 0)\n            y = y.astype('float32')\n        \n        return X, y\n    \n    def on_epoch_end(self):\n        \n        # Updates indexes after each epoch\n        self.indexes = np.arange(len(self.paths))\n        if self.shuffle == True:\n            np.random.shuffle(self.indexes)\n\n    def __iter__(self):\n        \"\"\"Create a generator that iterate over the Sequence.\"\"\"\n        for item in (self[i] for i in range(len(self))):\n            yield item\n            \n    def __load_image(self, path):\n        image_norm = skimage.io.imread(path)/255.0\n        \n\n        im = resize(image_norm, (shape[0], shape[1],shape[2]), mode='reflect')\n        im = im.astype('float32')\n        return im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Test data generator to generate predictions\nclass Leaf_TestDataGenerator(tf.keras.utils.Sequence):\n    def __init__(self, paths, batch_size, shape, shuffle = False, use_cache = False, augment = False):\n        self.paths= paths\n        self.batch_size = batch_size\n        self.shape = shape\n        self.shuffle = shuffle\n        self.use_cache = use_cache\n        self.augment = augment\n        if use_cache == True:\n            self.cache = np.zeros((paths.shape[0], shape[0], shape[1], shape[2]), dtype=np.float16)\n            self.is_cached = np.zeros((paths.shape[0]))\n        self.on_epoch_end()\n    \n    def __len__(self):\n        return int(np.ceil(len(self.paths) / float(self.batch_size)))\n    \n    def __getitem__(self, idx):\n        indexes = self.indexes[idx * self.batch_size : (idx+1) * self.batch_size]\n\n        paths = self.paths[indexes]\n        X = np.zeros((paths.shape[0], self.shape[0], self.shape[1], self.shape[2]))\n        # Generate data\n        if self.use_cache == True:\n            X = self.cache[indexes]\n            for i, path in enumerate(paths[np.where(self.is_cached[indexes] == 0)]):\n                image = self.__load_image(path)\n                self.is_cached[indexes[i]] = 1\n                self.cache[indexes[i]] = image\n                X[i] = image\n        else:\n            for i, path in enumerate(paths):\n                X[i] = self.__load_image(path)\n\n        #y = self.labels[indexes]\n                \n        if self.augment == True:\n            seq = iaa.Sequential([\n                iaa.OneOf([\n                    iaa.Fliplr(0.5), # horizontal flips\n                    iaa.Crop(percent=(0, 0.1)), # random crops\n                    #iaa.ContrastNormalization((0.75, 1.5)),\n                    #iaa.AdditiveGaussianNoise(loc=0, scale=(0.0, 0.05*255), per_channel=0.5),\n                    #iaa.Multiply((0.8, 1.2), per_channel=0.2),\n                    #iaa.Affine(rotate=90),\n                    #iaa.Affine(rotate=180),\n                    #iaa.Affine(rotate=270),\n                    iaa.Flipud(0.5),\n                ])], random_order=True)\n\n            X = np.concatenate((X, seq.augment_images(X), seq.augment_images(X), seq.augment_images(X)), 0)\n            X = X.astype('float32')\n            #y = np.concatenate((y, y, y, y), 0)\n            #y = y.astype('float32')\n        \n        return X\n    \n    def on_epoch_end(self):\n        \n        # Updates indexes after each epoch\n        self.indexes = np.arange(len(self.paths))\n        if self.shuffle == True:\n            np.random.shuffle(self.indexes)\n\n    def __iter__(self):\n        \"\"\"Create a generator that iterate over the Sequence.\"\"\"\n        for item in (self[i] for i in range(len(self))):\n            yield item\n            \n    def __load_image(self, path):\n        image_norm = skimage.io.imread(path)/255.0\n        \n\n        im = resize(image_norm, (shape[0], shape[1],shape[2]), mode='reflect')\n        im = im.astype('float32')\n        return im","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Define the model**"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model(input_shape, n_out):\n    inp = Input(input_shape)\n    #x = img_augmentation(inp)\n    pretrain_model = EfficientNetB0(include_top=False, weights='imagenet', input_tensor=inp)\n    x = pretrain_model.output\n    x = GlobalAveragePooling2D()(x)\n    x = Dropout(0.25)(x)\n    x = Dense(n_out, activation=\"sigmoid\")(x)\n    ##Uncomment if you want to train few more layers before the head\n    #for layer in pretrain_model.layers[:160]:\n        #layer.trainable = False\n    \n    for layer in pretrain_model.layers:\n        layer.trainable = False\n        \n    return Model(inp, x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from tensorflow.keras.metrics import categorical_accuracy,top_k_categorical_accuracy\ndef top_5_accuracy(y_true, y_pred):\n    return top_k_categorical_accuracy(y_true, y_pred, k=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nlabls = train['label'].nunique()\nmodel = create_model(input_shape=(224,224,3), n_out=nlabls)\nmodel.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['categorical_accuracy'])\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"paths, labels,_ = getTrainParams()\nkeys = np.arange(paths.shape[0], dtype=np.int)  \nnp.random.seed(seed)\nnp.random.shuffle(keys)\nlastTrainIndex = int((1-val_sample) * paths.shape[0])\n\npathsTrain = paths[0:lastTrainIndex]\nlabelsTrain = labels[0:lastTrainIndex]\n\npathsVal = paths[lastTrainIndex:]\nlabelsVal = labels[lastTrainIndex:]\n\nprint(paths.shape, labels.shape)\nprint(pathsTrain.shape, labelsTrain.shape, pathsVal.shape, labelsVal.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator = Leaf_DataGenerator(pathsTrain, labelsTrain, batch_size, shape, use_cache=False, augment = False, shuffle = True)\nval_generator = Leaf_DataGenerator(pathsVal, labelsVal, batch_size, shape, use_cache=False, shuffle = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##Calculate class weights to help the model compensate for some of the class imbalance\nclass_wt = pd.DataFrame(train[['image_id', 'label']].groupby(['label']).agg(['count']))\nclass_wt.reset_index(inplace=True)\nclass_wt['weight']= class_wt.iloc[:,1]/train.shape[0]\nclass_wt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_weight = {0: 0.050802, 1: 0.102304,2:0.111511,3:0.614946,4:0.120437}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"epochs = 2\nuse_multiprocessing = False \nbase_cnn = model.fit_generator(\n    train_generator,\n    steps_per_epoch=len(train_generator),\n    validation_data=val_generator,\n    validation_steps=24,\n    class_weight = class_weight,\n    epochs=epochs,\n    #callbacks = [clr],\n    verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.save('EfficientNetB0_2epochs.h5')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Generate Submission**"},{"metadata":{"trusted":true},"cell_type":"code","source":"TestPath = '../input/cassava-leaf-disease-classification/test_images/'\nall_testimg_paths = [y for x in os.walk(TestPath) for y in glob(os.path.join(x[0], '*.jpg'))]\nall_testfilenames = []\nfor filepath in all_testimg_paths:\n    FileName = os.path.basename(filepath)\n    all_testfilenames.append(FileName)\ntest_path_dict = dict(zip(all_testfilenames,all_testimg_paths))\ntestPaths = np.array(list(test_path_dict.values()))\ntest_generator = Leaf_TestDataGenerator(testPaths, batch_size, shape, use_cache=False, shuffle = False)\npred_prob = model.predict(test_generator)\npred = np.argmax(pred_prob, axis=-1)\ndata_items = test_path_dict.items()\ndata_list = list(data_items)\nsub_df = pd.DataFrame(data_list)\nsub_df.drop(sub_df.columns[1], axis=1, inplace=True)\nsub_df['label']= pred\nsub_df.columns = ['image_id','label']\nsub_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Work in progress....**\n\nIf you found this helpful, please upvote!![](http://)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}