{"cells":[{"metadata":{},"cell_type":"markdown","source":"## APTOS 2019 Blindness Detection challeng\n---------------------------------------\nwe are taking this basic 14 steps.\n1. [Import Libraries](#1)\n1. [Loading Data ](#2)\n1. [Data Visualization](#3)\n1. [Train and Test dataset](#4)\n1. [Data Pre-Processing](#6)\n1. [Image Data Generator](#7)\n1. [Model Architecture Design](#8)\n1. [Keras Callback Funcations](#9)\n1. [Transfer Learning](#10)\n1. [Validation Accuracy & Loss](#11)\n1. [Validation Accuracy](#12)\n1. [Test-Time Augmentation](#13)\n1. [Visualization Test Result](#14)\n------------------------------------\n- Design CNN from Scratch\n- Use pre-train model for Blindness Detection\n \n diagnosis Of Diabetic Retinopathy in these five stages\n- NO DR\n- Mild\n- Moderate \n- Servere\n- Proliferative DR"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a> \n# Import Libraries"},{"metadata":{},"cell_type":"markdown","source":"**psutil** library is useful mainly for system monitoring.\n\n**GC** is use for garbage collection.\n\n**tqdm** is use for showing progress.\n\n**math** is for mathemetical operation.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport PIL\nimport gc\nimport psutil\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\nfrom tensorflow import set_random_seed\nfrom tqdm import tqdm\nfrom math import ceil\nimport math\nimport sys\nprint('all data processing libraries imported')\n\nimport keras\nfrom keras.preprocessing.image import ImageDataGenerator, load_img, array_to_img, img_to_array\nfrom keras.models import Sequential, Model\nfrom keras.applications.resnet50 import ResNet50, preprocess_input\nfrom keras.layers import Conv2D, MaxPooling2D, GlobalAveragePooling2D, Input\nfrom keras.layers import Dropout, Flatten, Dense\nfrom keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\nfrom keras.activations import softmax, relu, elu\nfrom keras.optimizers import Adam, rmsprop, RMSprop   ,SGD\nfrom keras.layers import BatchNormalization, LeakyReLU\nfrom tqdm import tqdm\ngc.enable()\nprint('all Deep Learning processing libraries imported')\nprint(os.listdir(\"../input/\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n#### Exploratory Data Analysis\n- Load the  Data \n- Data plotting\n- Data analysis\n\nSetup all the model\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED = 7\nnp.random.seed(SEED)\nset_random_seed(SEED)\ndir_path = \"../input/aptos2019-blindness-detection/\"\nIMG_DIM = 250  # 224 399 #\nBATCH_SIZE = 36\nCHANNEL_SIZE = 3\nNUM_EPOCHS = 1\nTRAIN_DIR = 'train_images'\nTEST_DIR = 'test_images'\nFREEZE_LAYERS = 2  # freeze the first this many layers for training\nCLASS = {0: \"No DR\", 1: \"Mild\", 2: \"Moderate\", 3: \"Severe\", 4: \"Proliferative DR\"}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"2\"></a>\n## Loading Data "},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join(dir_path, \"train.csv\"))\ndf_test = pd.read_csv(os.path.join(dir_path, \"test.csv\"))\nNUM_CLASSES = df_train['diagnosis'].nunique()\nprint(np.shape(df_train))\nprint(np.shape(df_test))\nprint(\"Training set has {} samples and {} classes.\".format(df_train.shape[0], df_train.shape[1]))\nprint(\"Testing set has {} samples and {} classes.\".format(df_test.shape[0], df_test.shape[1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a>\n# Data Visualization and Exploratory Data Analysis.\n> Data distrubution per class\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_data = df_train.diagnosis.value_counts()\nprint(sample_data)\nsample_data.plot(kind='bar');\nplt.title('number sample per class');\nplt.show()\n#plt.pie(sample_data,shadow=0, labels=[\"No DR\", \"Mild\", \"Moderate\", \"Severe\", \"Proliferative DR\"])\n#plt.title('sample in pie chart');\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Histogram is clearing showing that training data is Imbalanced. Because in class ‘No DR’ records are approx. 1750 while in class ‘Severe’ less than 250. So, we found that very  imbalancing data set, because of this we have to augmented the data. \n\nThere are many ways to do image data augmentation.\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a>\n### Train and Test dataset \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Train & Test samples ratio\n# Plot Data\nlabels = 'Train', 'Test'\nsizes = df_train.shape[0], df_test.shape[0]\ncolors = 'red', 'blue'\nplt.pie(sizes, labels=labels, autopct='%1.1f%%', shadow=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a>\n#### Split DataSet by using train_test_split model"},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train, x_test, y_train, y_test = train_test_split(df_train.id_code, df_train.diagnosis, test_size=0.3,\n                                                    random_state=SEED, stratify=df_train.diagnosis)\n#you can better idea from printing the data.\nprint(x_train.head())\nprint(y_train.head())\nprint(x_test.head())\nprint(y_test.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nData visualization is a process in  AI, which will give you better insight of data.\n\nin the ***draw_img** function (**img, target_dir and class_label** are variables)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_img(img, target_dir, class_label='0'):\n    fig, axis = plt.subplots(2, 6, figsize=(15, 6))\n    #enumerate for getting row or column line by line.\n    for idnx, (idx, row) in enumerate(img.iterrows()):\n        imgPath = os.path.join(dir_path, f\"{target_dir}/{row['id_code']}.png\")\n        img = cv2.imread(imgPath)\n        row = idnx // 6\n        col = idnx % 6\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        axis[row, col].imshow(img)\n    plt.suptitle(class_label)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 0\nplot_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASS[CLASS_ID])\n\nCLASS_ID = 1\nplot_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASS[CLASS_ID])\n\nCLASS_ID = 2\nplot_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASS[CLASS_ID])\n\nCLASS_ID = 3\nplot_img(df_train[df_train.diagnosis == CLASS_ID].head(12), 'train_images', CLASS[CLASS_ID])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sample images of dataset.\n- As we can see the image shape is not in standard shape, we need to resize data set image.\n- Some images are very small, and some are very large they are not in same standard.\n- Some are having large black area like image Proliferative[1,2] has lot of black area. Which is not relevant for your problem? May we would be requiring doing the image cropping.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"CLASS_ID = 'Test DataSet'\nplot_img(df_test.sample(12, random_state=SEED), 'test_images', CLASS_ID)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- In Test data, there are some image are bigger and some are having black area. So, testing images also require doing image pre-processing.  \n- May be would be require creating our image Generator."},{"metadata":{},"cell_type":"markdown","source":"sys.maxsize predict the maximum number of array here image."},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a>\n### GrayScale Images\nConverting the Ratina Images into Grayscale. So, we can usnderstand the ROI ."},{"metadata":{},"cell_type":"markdown","source":"unique : ndarray...The sorted unique values.\n.loc : Access a group of rows and columns by label(s) or a boolean array."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display some random images from Data Set with class categories ing gray\nfigure = plt.figure(figsize=(20, 16))\nfor target_class in (y_train.unique()):\n    for i, (idx, row) in enumerate(\n            df_train.loc[df_train.diagnosis == target_class].sample(5, random_state=SEED).iterrows()):\n        axis = figure.add_subplot(5, 5, target_class * 5 + i + 1)\n        image = f\"../input/aptos2019-blindness-detection/train_images/{row['id_code']}.png\"\n        img = cv2.imread(image)\n        img=cv2.addWeighted ( img,4, cv2.GaussianBlur( img , (0,0) , IMG_DIM/10) ,-4 ,128) # the trick is to add this line\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        img = cv2.resize(img, (IMG_DIM, IMG_DIM))\n        img = cv2.addWeighted(img, 4, cv2.GaussianBlur(img, (0, 0), IMG_DIM / 10), -4, 128)\n        plt.imshow(img, cmap='gray')\n        axis.set_title(CLASS[target_class])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's clearly showing, that the image [0,1] has give regin  black around the EYE ball. Which is ust noise, that will not add any value fo model. We need to remove this black area. in my next iteration will work on that to crop black are from image. "},{"metadata":{},"cell_type":"markdown","source":"## Image Cropping\nSome images has big blank space. they will take only computation power and add noise to model.\nSo better will will crop the blank spaces from images. \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Display some random images from Data Set with class categories. showig Gray image removing other channel and adding lighting to image.\nfigure = plt.figure(figsize=(20, 16))\nfor target_class in (y_train.unique()):\n    #     print(CLASSS[target_class],target_class)\n    for i, (idx, row) in enumerate(\n            df_train.loc[df_train.diagnosis == target_class].sample(5, random_state=SEED).iterrows()):\n        ax = figure.add_subplot(5, 5, target_class * 5 + i + 1)\n        imagefile = f\"../input/aptos2019-blindness-detection/train_images/{row['id_code']}.png\"\n        img = cv2.imread(imagefile)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        img = cv2.resize(img, (IMG_DIM, IMG_DIM))\n        img = cv2.addWeighted(img, 4, cv2.GaussianBlur(img, (0, 0), IMG_DIM / 10), -4, 128)\n        plt.imshow(img, cmap='gray')\n        ax.set_title('%s-%d-%s' % (CLASS[target_class], idx, row['id_code']))\n#         print(row['id_code'])\n#     plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"available RAM:\", psutil.virtual_memory())\ngc.collect()\n# print(\"available RAM:\", psutil.virtual_memory())\n\ndf_train.id_code = df_train.id_code.apply(lambda x: x + \".png\")\ndf_test.id_code = df_test.id_code.apply(lambda x: x + \".png\")\ndf_train['diagnosis'] = df_train['diagnosis'].astype('str')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a>\n# Image Data Generator\nIn this section willl use Keras ImageDataGenerator class for generating data for Keras model. It is used for data generation, increasing the data size. with the help of ImageDataGenerator we will do image \"augment\" via a number of random transformations, so that our model would never see twice the exact same picture. \n\nTraining Deep Learning model can perform better with more data, and augementation technique can create variations of data that can increase the ababiliy of fit model to gene\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the imageDatagenerator Instance \ndatagenerator=ImageDataGenerator(#rescale=1./255,\n#                                       validation_split=0.15, \n                                         horizontal_flip=True,\n                                         vertical_flip=True, \n                                         rotation_range=40, \n                                         zoom_range=0.2, \n                                         shear_range=0.1,\n                                        fill_mode='nearest')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imgPath = f\"../input/aptos2019-blindness-detection/train_images/cd54d022e37d.png\"\n# Loading image\nimg = load_img(imgPath)\ndata = img_to_array(img)\nsamples =np.expand_dims(data, 0)\ni=5\nit=datagenerator.flow(samples , batch_size=1)\nfor i in range(5):\n    plt.subplot(230 + 1 + i)\n    batch = it.next()\n    image = batch[0].astype('uint8')\n    plt.imshow(image)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_datagen = ImageDataGenerator(rescale=1. / 255, \n                                         validation_split=0.15, \n                                         horizontal_flip=True,\n                                         vertical_flip=True, \n                                         rotation_range=40, \n                                         zoom_range=0.2, \n                                         shear_range=0.1,\n                                        fill_mode='nearest')\n# valid_datagen=image.ImageDataGenerator(rescale=1./255)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_generator = train_datagen.flow_from_dataframe(dataframe=df_train,\n                                                    directory=\"../input/aptos2019-blindness-detection/train_images/\",\n                                                    x_col=\"id_code\",\n                                                    y_col=\"diagnosis\",\n                                                    batch_size=BATCH_SIZE,\n                                                    class_mode=\"categorical\",\n                                                    target_size=(IMG_DIM, IMG_DIM),\n                                                    subset='training',\n                                                    shaffle=True,\n                                                    seed=SEED,\n                                                    )\nvalid_generator = train_datagen.flow_from_dataframe(dataframe=df_train,\n                                                    directory=\"../input/aptos2019-blindness-detection/train_images/\",\n                                                    x_col=\"id_code\",\n                                                    y_col=\"diagnosis\",\n                                                    batch_size=BATCH_SIZE,\n                                                    class_mode=\"categorical\",\n                                                    target_size=(IMG_DIM, IMG_DIM),\n                                                    subset='validation',\n                                                    shaffle=True,\n                                                    seed=SEED\n                                                    )\ndel x_train\n# # del x_test\ndel y_train\n# del y_test\ngc.collect()\n#  color_mode= \"grayscale\",\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"8\"></a>\n# Model Architecture Design"},{"metadata":{"trusted":true},"cell_type":"code","source":"def design_model():\n    model = Sequential()\n    model.add(Conv2D(filters=16, kernel_size=(2, 2), input_shape=[IMG_DIM, IMG_DIM, CHANNEL_SIZE], activation=relu))\n    model.add(MaxPooling2D(pool_size=(2, 2)))\n    model.add(Conv2D(filters=32, kernel_size=(2, 2), activation=relu))\n    model.add(MaxPooling2D(pool_size=(2, 2)))\n    model.add(Conv2D(filters=64, kernel_size=(2, 2), activation=relu))\n    model.add(MaxPooling2D(pool_size=(2, 2)))\n    model.add(GlobalAveragePooling2D())\n    model.add(Dense(units=1000, activation=relu))\n    model.add(Dropout(rate=0.2))\n    model.add(Dense(units=1000, activation=relu))\n    model.add(Dropout(rate=0.2))\n    model.add(Dense(5, activation='softmax'))\n    return model\n\n\nmodel = design_model()\n# model.summary()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Compile model"},{"metadata":{"trusted":true},"cell_type":"code","source":"sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)\nmodel.compile(loss='categorical_crossentropy', optimizer=sgd,metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"9\"></a>\n# Keras Callback Funcations\n- Call Back functions Eraly Stoping and Learning Rate Reducing"},{"metadata":{"trusted":true},"cell_type":"code","source":"eraly_stop = EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=3, verbose=1, mode='auto')\n# Reducing the Learning Rate if result is not improving. \nreduce_lr = ReduceLROnPlateau(monitor='val_loss', min_delta=0.0004, patience=2, factor=0.1, min_lr=1e-6, mode='auto',\n                              verbose=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"NUB_TRAIN_STEPS = train_generator.n // train_generator.batch_size\nNUB_VALID_STEPS = valid_generator.n // valid_generator.batch_size\n\nNUB_TRAIN_STEPS, NUB_VALID_STEPS","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# model.fit_generator(generator=train_generator,\n#                     validation_data=valid_generator,\n#                     steps_per_epoch=STEP_SIZE_TRAIN,\n#                     validation_steps=STEP_SIZE_TRAIN,\n#                     verbose=1,\n#                     callbacks=[checkpoint],\n#                     use_multiprocessing=True,\n#                     workers=3,\n#                     shuffle=True,\n#                     max_queue_size=16,\n#                     epochs=NB_EPOCHS)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"10\"></a>\n# Transfer Learning "},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_resnet(img_dim, CHANNEL, n_class):\n    input_tensor = Input(shape=(img_dim, img_dim, CHANNEL))\n\n    base_model = ResNet50(weights=None, include_top=False, input_tensor=input_tensor)\n    base_model.load_weights('../input/resnet50weightsfile/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5')\n#     base_model.load_weights('../input/restnet101/resnet101_weights_tf.h5')\n\n    \n\n    x = GlobalAveragePooling2D()(base_model.output)\n    x = Dropout(0.4)(x)\n    x = Dense(2048, activation=elu)(x)\n    x = Dropout(0.4)(x)\n    x = BatchNormalization()(x)\n    x = Dense(1024, activation=elu)(x)\n    x = Dropout(0.4)(x)\n    x = BatchNormalization()(x)\n    x = Dense(512, activation=elu)(x)\n    x = Dropout(0.4)(x)\n    x = BatchNormalization()(x)\n    output_layer = Dense(n_class, activation='softmax', name=\"Output_Layer\")(x)\n    model_resnet = Model(input_tensor, output_layer)\n\n    return model_resnet\n\nmodel_resnet = create_resnet(IMG_DIM, CHANNEL_SIZE, NUM_CLASSES)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Layers \n# for i, lay in enumerate(model_resnet.layers):\n#     print(i,lay.name)\n# Training All Layers\n\nfor layers in model_resnet.layers:\n    layers.trainable = True\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lr = 1e-3\noptimizer = SGD(lr=lr, decay=1e-6, momentum=0.9, nesterov=True) # Adam(lr=lr, decay=0.01) \nmodel_resnet.compile(optimizer=optimizer, loss=keras.losses.categorical_crossentropy, metrics=['accuracy'])\n# model.summary()\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history = model_resnet.fit_generator(generator=train_generator,\n                                     steps_per_epoch=NUB_TRAIN_STEPS,\n                                     validation_data=valid_generator,\n                                     validation_steps=NUB_VALID_STEPS,\n                                     epochs=NUM_EPOCHS,\n                                     #                            shuffle=True,  \n                                     callbacks=[eraly_stop, reduce_lr],\n                                     verbose=2)\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"11\"></a>\n# Display Validation Accuracy & Loss\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"history.history.keys()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"accu = history.history['acc']\nval_acc = history.history['val_acc']\n\nplt.plot(accu, label=\"Accuracy\")\nplt.plot(val_acc)\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Accuracy\")\nplt.legend(['Acc', 'val_acc'])\nplt.plot(np.argmax(history.history[\"val_acc\"]), np.max(history.history[\"val_acc\"]), marker=\"x\", color=\"r\",\n         label=\"best model\")\nplt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8, 8))\nplt.title(\"Learning curve\")\nplt.plot(history.history[\"loss\"], label=\"loss\")\nplt.plot(history.history[\"val_loss\"], label=\"val_loss\")\nplt.plot(np.argmin(history.history[\"val_loss\"]), np.min(history.history[\"val_loss\"]), marker=\"x\", color=\"r\",\n         label=\"best model\")\nplt.xlabel(\"Epochs\")\nplt.ylabel(\"log_loss\")\nplt.legend();\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"12\"></a>\n## Validation Accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"# STEP_SIZE_TEST=test_generator.n//test_generator.batch_size\n(eval_loss, eval_accuracy) = tqdm(\n    model_resnet.evaluate_generator(generator=valid_generator, steps=NUB_VALID_STEPS, pickle_safe=False))\nprint(\"[INFO] accuracy: {:.2f}%\".format(eval_accuracy * 100))\nprint(\"[INFO] Loss: {}\".format(eval_loss))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_datagen = ImageDataGenerator(rescale=1. / 255, validation_split=0.2, horizontal_flip=True)\n\ntest_generator = test_datagen.flow_from_dataframe(dataframe=df_test,\n                                                  directory=\"../input/aptos2019-blindness-detection/test_images/\",\n                                                  x_col=\"id_code\",\n                                                  target_size=(IMG_DIM, IMG_DIM),\n                                                  batch_size=1,\n                                                  shuffle=False,\n                                                  class_mode=None,\n                                                  seed=SEED)\n# del df_test\nprint(df_test.shape[0])\n# del train_datagen\n# del traabsin_generator\ngc.collect()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Kapkaha"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"13\"></a>\n# Test-Time Augmentation\nIn the below section, we are doning TTA imporving the prediction accuracy. It will transform image and predict "},{"metadata":{"trusted":true},"cell_type":"code","source":"tta_steps = 5\npreds_tta = []\nfor i in tqdm(range(tta_steps)):\n    test_generator.reset()\n    preds = model_resnet.predict_generator(generator=test_generator, steps=ceil(df_test.shape[0]))\n    #     print('Before ', preds.shape)\n    preds_tta.append(preds)\n#     print(i,  len(preds_tta))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"final_pred = np.mean(preds_tta, axis=0)\npredicted_class_indices = np.argmax(final_pred, axis=1)\nlen(predicted_class_indices)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_generator.filenames.apply(lambda x: x[-4])\nresults = pd.DataFrame({\"id_code\": test_generator.filenames, \"diagnosis\": predicted_class_indices})\nresults.id_code = results.id_code.apply(lambda x: x[:-4])  # results.head()\nresults.to_csv(\"submission.csv\", index=False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"  <a id=\"14\"></a>\n # Visualization Test Result\n- this section will visualize the predicted classes of test data."},{"metadata":{"trusted":true},"cell_type":"code","source":"results['diagnosis'].value_counts().plot(kind='bar')\nplt.title('Test Samples Per Class')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"References:\n\n1. https://medium.com/@vijayabhaskar96/tutorial-on-keras-imagedatagenerator-with-flow-from-dataframe-8bd5776e45c1\n1. https://medium.com/@vijayabhaskar96/tutorial-on-keras-flow-from-dataframe-1fd4493d237c\n1. https://www.pyimagesearch.com/2017/03/20/imagenet-vggnet-resnet-inception-xception-keras/\n1. https://jkjung-avt.github.io/keras-image-cropping/\n1. https://www.kaggle.com/aleksandradeis/aptos2019-blindness-detection-eda"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}