{"cells":[{"metadata":{},"cell_type":"markdown","source":"---------------------------------------\n1. [Import Required Libraries](#1)\n1. [Loading Data ](#2)\n1. [Data Visualization](#3)\n1. [Train and Test dataset](#4)\n1. [Data Pre-Processing](#6)\n1. [Image Data Generator](#7)\n1. [Keras Callback Funcations](#8)\n1. [Transfer Learning](#9)\n1. [Validation Accuracy & Loss](#10)\n1. [Validation Accuracy](#11)\n1. [Test-Time Augmentation](#12)\n1. [Visualization Test Result](#13)\n------------------------------------\n- Design CNN from Scratch\n- Use pre-train model for Blindness Detection\n \n Stages Of Diabetic Retinopathy\n- NO DR\n- Mild\n- Moderate \n- Servere\n- Proliferative DR"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a> \n# Import Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport PIL\nimport gc\nimport psutil\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import GridSearchCV\nfrom tensorflow import set_random_seed\nfrom tqdm import tqdm\nfrom math import ceil\nimport math\nimport sys\nimport gc\n\nimport keras\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import array_to_img\nfrom keras.preprocessing.image import img_to_array\nfrom keras.applications.resnet50 import ResNet50\nfrom keras.applications.resnet50 import preprocess_input\nfrom keras.models import Model\nfrom keras.models import Sequential\nfrom keras.layers.convolutional import Conv2D\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.layers.pooling import GlobalAveragePooling2D\nfrom keras.layers import Input\nfrom keras.layers.core import Dropout\nfrom keras.layers.core import Flatten\nfrom keras.layers.core import Dense\nfrom keras.callbacks import ModelCheckpoint\nfrom keras.callbacks import ReduceLROnPlateau\nfrom keras.callbacks import EarlyStopping\nfrom keras.wrappers.scikit_learn import KerasClassifier\n\nfrom keras.activations import softmax\nfrom keras.activations import elu\nfrom keras.activations import relu\nfrom keras.optimizers import Adam\nfrom keras.optimizers import RMSprop\nfrom keras.optimizers import SGD\nfrom keras.layers.normalization import BatchNormalization\nfrom tqdm import tqdm\n\ngc.enable()\n\nprint(os.listdir(\"../input/\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n#### Exploratory Data Analysis\n- Loading Data \n- Data Disribution\n- Data Visualization\n\nSetup all the param, which we will use in model\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED = 2019\nnp.random.seed(SEED)\nset_random_seed(SEED)\ndir_path = \"../input/aptos2019-blindness-detection/\"\nIMG_DIM = 299  # 224 399 #\nBATCH_SIZE = 42\nCHANNEL_SIZE = 3\n#NUM_EPOCHS = 50\nTRAIN_DIR = 'train_images'\nTEST_DIR = 'test_images'\nFREEZE_LAYERS = 2  # freeze the first this many layers for training\nCLASSS = {0: \"No DR\", 1: \"Mild\", 2: \"Moderate\", 3: \"Severe\", 4: \"Proliferative DR\"}\nNUM_CLASS = len(CLASSS.keys())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n## Loading Data "},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join(dir_path, \"train.csv\"))\ndf_test = pd.read_csv(os.path.join(dir_path, \"test.csv\"))\nNUM_CLASSES = df_train['diagnosis'].nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Training set has {} samples and {} classes.\".format(df_train.shape[0], df_train.shape[1]))\nprint(\"Testing set has {} samples and {} classes.\".format(df_test.shape[0], df_test.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a>\n# Data Visualization and EDA\n> Data distrubution per class\n\nas per below bar chart, it clearly showing that data set is quite imbalance. And even it's expected in medical domain."},{"metadata":{"trusted":true},"cell_type":"code","source":"chat_data = df_train.diagnosis.value_counts()\nchat_data.plot(kind='bar');\nplt.title('Sample Per Class');\nplt.show()\nplt.pie(chat_data, autopct='%1.1f%%', shadow=True, labels=[\"No DR\", \"Mild\", \"Moderate\", \"Severe\", \"Proliferative DR\"])\nplt.title('Per class sample Percentage');\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Histogram is clearing showing that training data is Imbalanced. Because in class ‘No DR’ records are approx. 1750 while in class ‘Severe’ very less. So, may be for balancing data set, we would be requiring data augmentation. \n\nThere are couple of ways to do image data augmentation. We will see down in this kernel.\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a>\n### Train and Test dataset \n- We will use pie chart for showing the size of dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train & Test samples ratio\n# Plot Data\nlabels = 'Train', 'Test'\nsizes = df_train.shape[0], df_test.shape[0]\ncolors = 'lightskyblue', 'lightcoral'\n# Plot\nplt.figure(figsize=(7, 5))\nplt.pie(sizes, labels=labels, autopct='%1.1f%%', shadow=True)\nplt.axis('equal')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's showing train and testing data are in 2:1 ratio. Both are quite small data set."},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a>\n#### Split DataSet"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train, x_test, y_train, y_test = train_test_split(df_train.id_code, df_train.diagnosis, test_size=0.2,\n                                                    random_state=SEED, stratify=df_train.diagnosis)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nData visualization is a process in  AI, which will give you better insight of data."},{"metadata":{"trusted":true},"cell_type":"code","source":"def draw_img(imgs, target_dir, class_label='0'):\n    fig, axis = plt.subplots(2, 6, figsize=(15, 6))\n    for idnx, (idx, row) in enumerate(imgs.iterrows()):\n        imgPath = os.path.join(dir_path, f\"{target_dir}/{row['id_code']}.png\")\n        img = cv2.imread(imgPath)\n        row = idnx // 6\n        col = idnx % 6\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        axis[row, col].imshow(img)\n    plt.suptitle(class_label)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 0\ndraw_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASSS[CLASS_ID])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 1\ndraw_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASSS[CLASS_ID])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 2\ndraw_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASSS[CLASS_ID])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 3\ndraw_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASSS[CLASS_ID])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 4\ndraw_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASSS[CLASS_ID])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sample images of dataset.\n- As we can see the image shape is not in standard shape, we need to resize data set image.\n- Some images are very small, and some are very large they are not in same standard.\n- Some are having large black area like image Proliferative[1,2] has lot of black area. Which is not relevant for your problem? May we would be requiring doing the image cropping.\n- Some image light is very dark.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 'Test DataSet'\ndraw_img(df_test.sample(12, random_state=SEED), 'test_images', CLASS_ID)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- In Test data, there are some image are bigger and some are having black area. So, testing images also require doing image pre-processing.  \n- May be would be require creating our image Generator."},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a>\n\n### Max Min Height and Width"},{"metadata":{"trusted":true},"cell_type":"code","source":"def check_max_min_img_height_width(df, img_dir):\n    max_Height , max_Width =0 ,0\n    min_Height , min_Width =sys.maxsize ,sys.maxsize \n    for idx, row in df.iterrows():\n        imgPath=os.path.join(dir_path,f\"{img_dir}/{row['id_code']}.png\") \n        img=cv2.imread(imgPath)\n        H,W=img.shape[:2]\n        max_Height=max(H,max_Height)\n        max_Width =max(W,max_Width)\n        min_Height=min(H,min_Height)\n        min_Width =min(W,min_Width)\n    return max_Height, max_Width, min_Height, min_Width","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"check_max_min_img_height_width(df_train, TRAIN_DIR)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"check_max_min_img_height_width(df_test, TEST_DIR)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a>\n### GrayScale Images\nConverting the Ratina Images into Grayscale. So, we can usnderstand the regin or intest ."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display some random images from Data Set with class categories ing gray\nfigure = plt.figure(figsize=(20, 16))\nfor target_class in (y_train.unique()):\n    for i, (idx, row) in enumerate(\n            df_train.loc[df_train.diagnosis == target_class].sample(5, random_state=SEED).iterrows()):\n        ax = figure.add_subplot(5, 5, target_class * 5 + i + 1)\n        imagefile = f\"../input/aptos2019-blindness-detection/train_images/{row['id_code']}.png\"\n        img = cv2.imread(imagefile)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        img = cv2.resize(img, (IMG_DIM, IMG_DIM))\n        plt.imshow(img, cmap='gray')\n        ax.set_title(CLASSS[target_class])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's clearly showing, that the image [0,1] has give regin  black around the EYE ball. Which is ust noise, that will not add any value fo model. We need to remove this black area. in my next iteration will work on that to crop black are from image. "},{"metadata":{},"cell_type":"markdown","source":"## Image Cropping\nSome images has big blank space. they will take only computation power and add noise to model.\nSo better will will crop the blank spaces from images. \n\n#### References\nI have followed the below kaggle kernal for it. \n\nhttps://www.kaggle.com/ratthachat/aptos-updatedv14-preprocessing-ben-s-cropping\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Add Lighting to the images for improving the visibility \n\ndef draw_img_light(imgs, target_dir, class_label='0'):\n    fig, axis = plt.subplots(2, 6, figsize=(15, 6))\n    for idnx, (idx, row) in enumerate(imgs.iterrows()):\n        imgPath = os.path.join(dir_path, f\"{target_dir}/{row['id_code']}.png\")\n        img = cv2.imread(imgPath)\n        row = idnx // 6\n        col = idnx % 6\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (IMG_DIM, IMG_DIM))\n        img=cv2.addWeighted ( img,4, cv2.GaussianBlur( img , (0,0) , IMG_DIM/10) ,-4 ,128) # the trick is to add this line\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        axis[row, col].imshow(img, cmap='gray')\n    plt.suptitle(class_label)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 3\ndraw_img_light(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASSS[CLASS_ID])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Image Croping\ndef crop_image1(img,tol=7):\n    # img is image data\n    # tol  is tolerance\n        \n    mask = img>tol\n    return img[np.ix_(mask.any(1),mask.any(0))]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_image_from_gray(img,tol=7):\n    if img.ndim== 2:\n        mask=img>tol\n    elif img.ndim==3:\n        gray_img=cv2.cvtColor(img,cv2.COLOR_RGB2GRAY)\n        mask=gray_img>tol\n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n#         check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if check_shape ==0: # Image was full dark and may be cropout everything.\n            return img # Return original Image\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n            print(img1.shape,img2.shape,img3.shape)            \n            img=np.stack([img1,img2,img3],axis=1)\n            print(img.shape)\n            return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_ben_color(path, sigmaX=10):\n    image = cv2.imread(path)\n#     image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)S\n    image = crop_image_from_gray(image)\n    image = cv2.resize(image, (IMG_DIM, IMG_DIM))\n    image=cv2.addWeighted ( image,4, cv2.GaussianBlur( image , (0,0) , sigmaX) ,-4 ,128)\n        \n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#%%time\n#NUM_SAMP=7\n#fig = plt.figure(figsize=(25, 16))\n#for class_id in sorted(y_train.unique()):\n #  for i, (idx, row) in enumerate(df_train.loc[df_train['diagnosis'] == class_id].sample(NUM_SAMP, random_state=SEED).iterrows()):\n  #      ax = fig.add_subplot(5, NUM_SAMP, class_id * NUM_SAMP + i + 1, xticks=[], yticks=[])\n   #     path=f\"../input/aptos2019-blindness-detection/train_images/{row['id_code']}.png\"\n    #    image = load_ben_color(path,sigmaX=30)\n\n  #      plt.imshow(image)\n   #    ax.set_title('%d-%d-%s' % (class_id, idx, row['id_code']) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_image(img,tol=7):\n    w, h = img.shape[1],img.shape[0]\n    gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n    gray_img = cv2.blur(gray_img,(5,5))\n    shape = gray_img.shape \n    gray_img = gray_img.reshape(-1,1)\n    quant = quantile_transform(gray_img, n_quantiles=256, random_state=0, copy=True)\n    quant = (quant*256).astype(int)\n    gray_img = quant.reshape(shape)\n    xp = (gray_img.mean(axis=0)>tol)\n    yp = (gray_img.mean(axis=1)>tol)\n    x1, x2 = np.argmax(xp), w-np.argmax(np.flip(xp))\n    y1, y2 = np.argmax(yp), h-np.argmax(np.flip(yp))\n    if x1 >= x2 or y1 >= y2 : # something wrong with the crop\n        return img # return original image\n    else:\n        img1=img[y1:y2,x1:x2,0]\n        img2=img[y1:y2,x1:x2,1]\n        img3=img[y1:y2,x1:x2,2]\n        img = np.stack([img1,img2,img3],axis=-1)\n    return img\n\ndef process_image(image, size=512):\n    image = cv2.resize(image, (size,int(size*image.shape[0]/image.shape[1])))\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    try:\n        image = crop_image(image, tol=15)\n    except Exception as e:\n        image = image\n        print( str(e) )\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#178412895d5e\n #process_image\n #imagefile=f\"../input/aptos2019-blindness-detection/train_images/{row['id_code']}.png\" \n #img=cv2.imread(imagefile)\n #process_image(img, size=IMG_DIM)\n #plt.imshow('Crop',process_image(img, size=IMG_DIM))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display some random images from Data Set with class categories. showig Gray image removing other channel and adding lighting to image.\nfigure = plt.figure(figsize=(20, 16))\nfor target_class in (y_train.unique()):\n    #     print(CLASSS[target_class],target_class)\n    for i, (idx, row) in enumerate(\n            df_train.loc[df_train.diagnosis == target_class].sample(5, random_state=SEED).iterrows()):\n        ax = figure.add_subplot(5, 5, target_class * 5 + i + 1)\n        imagefile = f\"../input/aptos2019-blindness-detection/train_images/{row['id_code']}.png\"\n        img = cv2.imread(imagefile)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        img = cv2.resize(img, (IMG_DIM, IMG_DIM))\n        img = cv2.addWeighted(img, 4, cv2.GaussianBlur(img, (0, 0), IMG_DIM / 10), -4, 128)\n        plt.imshow(img, cmap='gray')\n        ax.set_title('%s-%d-%s' % (CLASSS[target_class], idx, row['id_code']))\n#         print(row['id_code'])\n#     plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"8\"></a>\n# Data Pre-Processing\n\n #### Croping Images\n\nhttps://stackoverflow.com/questions/13538748/crop-black-edges-with-opencv"},{"metadata":{"trusted":true},"cell_type":"code","source":"imgPath = f\"../input/aptos2019-blindness-detection/train_images/cd54d022e37d.png\"\nimg = cv2.imread(imgPath)\ngray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n_, thresh = cv2.threshold(gray, 1, 255, cv2.THRESH_BINARY)\ncontours, hierarchy = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\ncnt = contours[0]\nx, y, w, h = cv2.boundingRect(cnt)\nimg = img[y:y + h, x:x + w]\nplt.imshow(img)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Croping Images randomly for resizing."},{"metadata":{"trusted":true},"cell_type":"code","source":"def random_crop(img, random_crop_size):\n    # Note: image_data_format is 'channel_last'\n    assert img.shape[2] == 3\n    height, width = img.shape[0], img.shape[1]\n    dy, dx = random_crop_size\n    x = np.random.randint(0, width - dx + 1)\n    y = np.random.randint(0, height - dy + 1)\n    img = img[y:(y + dy), x:(x + dx), :]\n    return img\n\n\n\"\"\"Take as input a Keras ImageGen (Iterator) and generate random\n    crops from the image batches generated by the original iterator.\n    \"\"\"\n\n\ndef crop_generator(batches, crop_length):\n    while True:\n        batch_x, batch_y = next(batches)\n        batch_crops = np.zeros((batch_x.shape[0], crop_length, 3))\n        for i in range(batch_x.shape[0]):\n            batch_crops[0] = random_crop(batch_x[i], (crop_length, crop_length))\n        yield (batch_crops, batch_y)\n\n            ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Adding image type with image in dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"available RAM:\", psutil.virtual_memory())\ngc.collect()\n# print(\"available RAM:\", psutil.virtual_memory())\n\ndf_train.id_code = df_train.id_code.apply(lambda x: x + \".png\")\ndf_test.id_code = df_test.id_code.apply(lambda x: x + \".png\")\ndf_train['diagnosis'] = df_train['diagnosis'].astype('str')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a>\n# Image Data Generator\nIn this section willl use Keras ImageDataGenerator class for generating data for Keras model. It is used for data generation, increasing the data size. with the help of ImageDataGenerator we will do image \"augment\" via a number of random transformations, so that our model would never see twice the exact same picture. \n\nTraining Deep Learning model can perform better with more data, and augementation technique can create variations of data that can increase the ababiliy of fit model to gene\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the imageDatagenerator Instance \ndatagenerator=ImageDataGenerator(#rescale=1./255,\n#                                       validation_split=0.15, \n                                         horizontal_flip=True,\n                                         vertical_flip=True, \n                                         rotation_range=40, \n                                         zoom_range=0.2, \n                                         shear_range=0.1,\n                                        fill_mode='nearest')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imgPath = f\"../input/aptos2019-blindness-detection/train_images/cd54d022e37d.png\"\n# Loading image\nimg = load_img(imgPath)\ndata = img_to_array(img)\nsamples =np.expand_dims(data, 0)\ni=5\nit=datagenerator.flow(samples , batch_size=1)\nfor i in range(5):\n    plt.subplot(230 + 1 + i)\n    batch = it.next()\n    image = batch[0].astype('uint8')\n    plt.imshow(image)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale=1. / 255, \n                                         validation_split=0.15, \n                                         horizontal_flip=True,\n                                         vertical_flip=True, \n                                         rotation_range=40, \n                                         zoom_range=0.2, \n                                         shear_range=0.1,\n                                        fill_mode='nearest')\n# valid_datagen=image.ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator = train_datagen.flow_from_dataframe(dataframe=df_train,\n                                                    directory=\"../input/aptos2019-blindness-detection/train_images/\",\n                                                    x_col=\"id_code\",\n                                                    y_col=\"diagnosis\",\n                                                    batch_size=BATCH_SIZE,\n                                                    class_mode=\"categorical\",\n                                                    target_size=(IMG_DIM, IMG_DIM),\n                                                    subset='training',\n                                                    shuffle=True,\n                                                    seed=SEED,\n                                                    )\nvalid_generator = train_datagen.flow_from_dataframe(dataframe=df_train,\n                                                    directory=\"../input/aptos2019-blindness-detection/train_images/\",\n                                                    x_col=\"id_code\",\n                                                    y_col=\"diagnosis\",\n                                                    batch_size=BATCH_SIZE,\n                                                    class_mode=\"categorical\",\n                                                    target_size=(IMG_DIM, IMG_DIM),\n                                                    subset='validation',\n                                                    shuffle=True,\n                                                    seed=SEED\n                                                    )\ndel x_train\n# # del x_test\ndel y_train\n# del y_test\ngc.collect()\n#  color_mode= \"grayscale\",\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"9\"></a>\n# Keras Callback Funcations\n- Call Back functions Eraly Stoping and Learning Rate Reducing"},{"metadata":{"trusted":true},"cell_type":"code","source":"eraly_stop = EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=3, verbose=1, mode='auto')\n# Reducing the Learning Rate if result is not improving. \nreduce_lr = ReduceLROnPlateau(monitor='val_loss', min_delta=0.0004, patience=2, factor=0.1, min_lr=1e-6, mode='auto',\n                              verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NUB_TRAIN_STEPS = train_generator.n // train_generator.batch_size\nNUB_VALID_STEPS = valid_generator.n // valid_generator.batch_size\n\nNUB_TRAIN_STEPS, NUB_VALID_STEPS","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.fit_generator(generator=train_generator,\n#                     validation_data=valid_generator,\n#                     steps_per_epoch=STEP_SIZE_TRAIN,\n#                     validation_steps=STEP_SIZE_TRAIN,\n#                     verbose=1,\n#                     callbacks=[checkpoint],\n#                     use_multiprocessing=True,\n#                     workers=3,\n#                     shuffle=True,\n#                     max_queue_size=16,\n#                     epochs=NB_EPOCHS)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"10\"></a>\n# Transfer Learning "},{"metadata":{"trusted":true},"cell_type":"code","source":"img_dim, CHANNEL, n_class = 224, 3, NUM_CLASS\n#TODO: inicializar as variaveis\n\ndef create_resnet(dropout_rate=0.0):\n    input_tensor = Input(shape=(img_dim, img_dim, CHANNEL))\n    base_model = ResNet50(weights=None, include_top=False, input_tensor=input_tensor)\n    base_model.load_weights('../input/resnet50weightsfile/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5')\n\n\n    \n    x = GlobalAveragePooling2D()(base_model.output)\n    x = BatchNormalization()(x)\n    x = Dropout(dropout_rate)(x)\n    x = Dense(2048, activation=elu)(x)\n    x = BatchNormalization()(x)\n    x = Dropout(dropout_rate)(x)\n    x = Dense(1024, activation=elu)(x)\n    x = BatchNormalization()(x)\n    x = Dropout(dropout_rate)(x)\n    x = Dense(512, activation=elu)(x)\n    x = BatchNormalization()(x)\n    x = Dropout(dropout_rate)(x)  \n    \n    output_layer = Dense(n_class, activation='softmax', name=\"Output_Layer\")(x)   \n     \n    model_resnet = Model(input_tensor, output_layer)\n\n    return model_resnet\n\na = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\nX = a['diagnosis']\n#b = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\nY = a['diagnosis']\n\nx_train, x_teste = train_test_split(X, test_size = 0.2, random_state = 0)\nmean_values = x_train.mean()\nstd_values = x_train.std()    \n#x_train, x_value = train_test_split (x_train, test_size = 0.2, random_state = 0)\n    \nbase_resnet = KerasClassifier(build_fn=create_resnet, verbose=0)\nbatch_size = [32, 40, 60]\nepochs = [1]\ndropout_rate = [0.3]\n\nparam_grid = dict(batch_size=batch_size, epochs=epochs, dropout_rate=dropout_rate)\ngrid = GridSearchCV(estimator=base_resnet, param_grid=param_grid, n_jobs=-1, cv=3)\ngrid_result = grid.fit(X, Y)\n\nprint(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\nmeans = grid_result.cv_results_['mean_test_score']\nstds = grid_result.cv_results_['std_test_score']\nparams = grid_result.cv_results_['params']    \nfor mean, stdev, param in zip(means, stds, params):\n    print(\"%f (%f) with: %r\" % (mean, stdev, param))\n\n#model_resnet = create_resnet(IMG_DIM, CHANNEL_SIZE, NUM_CLASSES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Layers \n# for i, lay in enumerate(model_resnet.layers):\n#     print(i,lay.name)\n# Training All Layers\n\nfor layers in model_resnet.layers:\n    layers.trainable = True\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lr = 1e-3\noptimizer = SGD(lr=lr, decay=1e-6, momentum=0.9, nesterov=True) # Adam(lr=lr, decay=0.01) \nmodel_resnet.compile(optimizer=optimizer, loss=keras.losses.categorical_crossentropy, metrics=['accuracy'])\n# model.summary()\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model_resnet.fit_generator(generator=train_generator,\n                                     steps_per_epoch=NUB_TRAIN_STEPS,\n                                     validation_data=valid_generator,\n                                     validation_steps=NUB_VALID_STEPS,\n                                     epochs=NUM_EPOCHS,\n                                     #                            shuffle=True,  \n                                     callbacks=[eraly_stop, reduce_lr],\n                                     verbose=2)\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"11\"></a>\n# Display Validation Accuracy & Loss\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"history.history.keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"accu = history.history['acc']\nval_acc = history.history['val_acc']\n\nplt.plot(accu, label=\"Accuracy\")\nplt.plot(val_acc)\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Accuracy\")\nplt.legend(['Acc', 'val_acc'])\nplt.plot(np.argmax(history.history[\"val_acc\"]), np.max(history.history[\"val_acc\"]), marker=\"x\", color=\"r\",\n         label=\"best model\")\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8, 8))\nplt.title(\"Learning curve\")\nplt.plot(history.history[\"loss\"], label=\"loss\")\nplt.plot(history.history[\"val_loss\"], label=\"val_loss\")\nplt.plot(np.argmin(history.history[\"val_loss\"]), np.min(history.history[\"val_loss\"]), marker=\"x\", color=\"r\",\n         label=\"best model\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"log_loss\")\nplt.legend();\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"12\"></a>\n## Validation Accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"# STEP_SIZE_TEST=test_generator.n//test_generator.batch_size\n(eval_loss, eval_accuracy) = tqdm(\n    model_resnet.evaluate_generator(generator=valid_generator, steps=NUB_VALID_STEPS, pickle_safe=False))\nprint(\"[INFO] accuracy: {:.2f}%\".format(eval_accuracy * 100))\nprint(\"[INFO] Loss: {}\".format(eval_loss))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_datagen = ImageDataGenerator(rescale=1. / 255, validation_split=0.2, horizontal_flip=True)\n\ntest_generator = test_datagen.flow_from_dataframe(dataframe=df_test,\n                                                  directory=\"../input/aptos2019-blindness-detection/test_images/\",\n                                                  x_col=\"id_code\",\n                                                  target_size=(IMG_DIM, IMG_DIM),\n                                                  batch_size=1,\n                                                  shuffle=False,\n                                                  class_mode=None,\n                                                  seed=SEED)\n# del df_test\nprint(df_test.shape[0])\n# del train_datagen\n# del traabsin_generator\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Kapkaha"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"13\"></a>\n# Test-Time Augmentation\nIn the below section, we are doning TTA imporving the prediction accuracy. It will transform image and predict "},{"metadata":{"trusted":true},"cell_type":"code","source":"tta_steps = 5\npreds_tta = []\nfor i in tqdm(range(tta_steps)):\n    test_generator.reset()\n    preds = model_resnet.predict_generator(generator=test_generator, steps=ceil(df_test.shape[0]))\n    #     print('Before ', preds.shape)\n    preds_tta.append(preds)\n#     print(i,  len(preds_tta))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_pred = np.mean(preds_tta, axis=0)\npredicted_class_indices = np.argmax(final_pred, axis=1)\nlen(predicted_class_indices)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# del valid_generator\n# gc.collect()\n# test_generator.reset()\n\n# pred=model.predict_generator(test_generator, verbose=0, steps=STEP_SIZE_TEST)\n# predicted_class_indices=np.argmax(pred,axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_generator.filenames.apply(lambda x: x[-4])\nresults = pd.DataFrame({\"id_code\": test_generator.filenames, \"diagnosis\": predicted_class_indices})\nresults.id_code = results.id_code.apply(lambda x: x[:-4])  # results.head()\nresults.to_csv(\"submission.csv\", index=False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"  <a id=\"14\"></a>\n # Visualization Test Result\n- this section will visualize the predicted classes of test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"results['diagnosis'].value_counts().plot(kind='bar')\nplt.title('Test Samples Per Class')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"References:\n\n1. https://medium.com/@vijayabhaskar96/tutorial-on-keras-imagedatagenerator-with-flow-from-dataframe-8bd5776e45c1\n1. https://medium.com/@vijayabhaskar96/tutorial-on-keras-flow-from-dataframe-1fd4493d237c\n1. https://www.pyimagesearch.com/2017/03/20/imagenet-vggnet-resnet-inception-xception-keras/\n1. https://jkjung-avt.github.io/keras-image-cropping/\n1. https://www.kaggle.com/aleksandradeis/aptos2019-blindness-detection-eda"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}