{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Winning Team - what are we trying to do in this kernel?\n\nIn this kernel, we will attempt to bring together a series of ideas. We attempt to: \n\n1. [X] Use Ben Graham's image preprocessing method\n2. [X] Dynamically zoom to crop out all black edges\n3. [X] Apply an additional random zoom on the zoomed-out photos of 20-40% percent. (our current models are very good at predicting class 1-4, but the training images in class 0 are different, we apply a selective zoom in order to improve our training on the 0's.\n4. [X] Bring in external data as supplied by Brandon, found here: https://ieee-dataport.org/open-access/indian-diabetic-retinopathy-image-dataset-idrid \n5. [X] Use a callback to monitor and optimize for the Quadratic Weighted Kappa\n6. [X] Use preprocessing to introduce flips and turns, rotation, other, to further enrich the training dataset\n7. [X] Build a set of binary classifiers and compare\n8. [X] Add in early stopping if the model begins to overfit\n9. [X] Save the best models as they occur\n10. [X] Test out various CNN architectures (EfficientNetB5, Fast-R CNN)\n11. [ ] Build a multiple-input model using Nicholas' Zernike Moments\n12. [ ] Linear Output on the NN and compare\n13. [ ] Fix over-fitting problem\n![](http://)14. [ ] Find additional external images? "},{"metadata":{},"cell_type":"markdown","source":"SUmmary of changes this iteration: V20:\n* epochs to 15\n* changed zoom to [1, 1.25]\n* changed validation split from .1 to .2\n\nV21:\n* Made the Neural net Deeper\n\nV22 or V23: \n* Added some level of rotation to the preprocessing set. \n\nV24: \n* To prevent overfitting and high variance on our newer, deeper, neural net, I'm adding in a small regularization factor = (l2 = 0.0005) as well as a second dropout layer (p=.25) to see if this will make a difference.\n    ** Note: if this works well, it may make sense to do a GridSearch as described in this tutorial to find the optimal regularization factor.   https://machinelearningmastery.com/how-to-reduce-overfitting-in-deep-learning-with-weight-regularization/\n    \n    \nV25: \n* Trying out EfficientnetB5!! Whoo hoo!    Never got this to work\n\n\n\n\n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"#!pip install -U '../input/efficientnetwhl/keras_efficientnet-0.1.4-py3-none-any.whl'","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import json\nimport math\nimport os\n\nimport cv2\nfrom PIL import Image\nimport numpy as np\nfrom keras import layers\nfrom keras.applications import DenseNet121\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\nfrom keras.callbacks import Callback, EarlyStopping, ReduceLROnPlateau\nfrom keras.regularizers import l2\n\nimport matplotlib.pyplot as plt\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score\nimport scipy\nimport tensorflow as tf\nfrom tqdm import tqdm\nimport cv2\nfrom sklearn.utils import class_weight, shuffle\nfrom random import *\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nSEED = 42\nIMG_SIZE = 256\n# Specify title of our final model\nSAVED_MODEL_NAME = 'efficient_netB7.h5'\n\n%matplotlib inline\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#!pip install -U git+https://github.com/qubvel/efficientnet\n#!pip install efficientnet-master\n#import efficientnet\n\n#!pip freeze","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import sys\nimport os\n#sys.path.append(os.path.abspath('../input/efficientnet2ndattempt/efficientnet-master/'))\n#from efficientnet.model import EfficientNet\n\n\n###\n\nsys.path.append(os.path.abspath('../input/efficientnet/efficientnet-master/efficientnet-master/'))\nfrom efficientnet import EfficientNetB5\n\nefficientnet_model = EfficientNetB5(weights=None, include_top=False, input_shape=(224, 224, 3))\n\n\n#/kaggle/input/efficientnet2ndattempt/efficientnet-master/efficientnet/model.py\n#/kaggle/input/efficientnet-keras-weights-b0b5/efficientnet-b1_imagenet_1000_notop.h5\n\n\n\n\n#import EffienctNet\n\n\n#import keras_efficientnet as efn\n#import efficientnet.keras as efn \n\n#model = EfficientNetB5(weights='imagenet')\n#efn.show\n#import efficientnet.keras as efn \n\n#model = efn.EfficientNetB0(weights='imagenet')\n# credit to Jenessa\n#from keras.applications import MobileNet\n#from keras.applications.mobilenet import preprocess_input \n\n#model = efn.EfficientNetB5(weights='imagenet')\n\n\n#efn.show\n#import efficientnet.keras as efn \n# Import the model weights pretrained on Imagenet available in the Keras framework.\n# mobileNet_base = MobileNet(weights='imagenet', include_top=False, input_shape=(256, 256, 3))\n# Specify title of our final model\n#SAVED_MODEL_NAME = 'x.h5'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#!pip install -U --pre efficientnet\n#EfficientNetB5(weights='imagenet')\n#model = .EfficientNetB5(weights='imagenet')\n#efn.show\n#import efficientnet.keras as efn \n\n#/kaggle/input/efficientnet2ndattempt/efficientnet-master/efficientnet/model.py\n#/kaggle/input/efficientnet-keras-weights-b0b5/efficientnet-b1_imagenet_1000_notop.h5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#for dirname, _, filenames in os.walk('/kaggle/input'):\n#    for filename in filenames:\n#        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Set random seed for reproducibility."},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(2019)\ntf.set_random_seed(2019)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading & Exploration"},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_df = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\ntest_df = pd.read_csv('../input/aptos2019-blindness-detection/test.csv')\n###################### Adding in external dataset which has been combined with the internal data####################\ntrain_df = pd.read_csv('/kaggle/input/ext-data/train_combined.csv')\n\n\nprint(train_df.shape)\nprint(test_df.shape)\ntest_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['diagnosis'].hist()\ntrain_df['diagnosis'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Shown below are the Class 0's, compared with other classes\nOur current best model is good at predicting Classes 1, 2, 3, and 4, but is missing many of the Class 0's. This is because many of the Class 0's are zoomed too far out, and so the CNN is training on the surrounding black space instead of features of the eye itself. We hope to improve the differences on the 0's through preprocessing to make them ultimately more similar to the prediction dataset. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#Show me the 0's\ndef display_samples(df, columns=4, rows=3):\n    fig=plt.figure(figsize=(5*columns, 4*rows))\n\n    for i in range(columns*rows):\n        image_path = df.loc[i,'id_code']\n        image_id = df.loc[i,'diagnosis']\n        img = cv2.imread(f'../input/aptos2019-blindness-detection/train_images/{image_path}.png')\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        fig.add_subplot(rows, columns, i+1)\n        plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(train_df[train_df['diagnosis']==0].reset_index())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Show me the other classes\ndef display_samples(df, columns=4, rows=3):\n    fig=plt.figure(figsize=(5*columns, 4*rows))\n\n    for i in range(columns*rows):\n        image_path = df.loc[i,'id_code']\n        image_id = df.loc[i,'diagnosis']\n        img = cv2.imread(f'../input/aptos2019-blindness-detection/train_images/{image_path}.png')\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        fig.add_subplot(rows, columns, i+1)\n        plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(train_df[train_df['diagnosis']!=0].reset_index())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ben Graham's preprocessing method.\nBen Graham (last competition's winner) has an insightful way to improve lighting condition. Here, we apply his idea, and can see many important details in the eyes much better. \n* Credit to Neuron Engineer at https://www.kaggle.com/ratthachat/aptos-updatedv14-preprocessing-ben-s-cropping\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_image1(img,tol=7):\n    # img is image data\n    # tol  is tolerance\n        \n    mask = img>tol\n    return img[np.ix_(mask.any(1),mask.any(0))]\n\ndef crop_image_from_gray(img,tol=7):\n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n        mask = gray_img>tol\n        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        #print(check_shape)\n        if (check_shape == 0): # image is too dark so that we crop out everything,\n            return img # return original image\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n        #    print(img1.shape,img2.shape,img3.shape)\n            img = np.stack([img1,img2,img3],axis=-1)\n        #    print(img.shape)\n        return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Credit to Stack Overflow\ndef cv2_clipped_zoom(img, zoom_factor):\n    \"\"\"\n    Center zoom in/out of the given image and returning an enlarged/shrinked view of \n    the image without changing dimensions\n    Args:\n        img : Image array\n        zoom_factor : amount of zoom as a ratio (0 to Inf)\n    \"\"\"\n    height, width = img.shape[:2] # It's also the final desired shape\n    new_height, new_width = int(height * zoom_factor), int(width * zoom_factor)\n\n    ### Crop only the part that will remain in the result (more efficient)\n    # Centered bbox of the final desired size in resized (larger/smaller) image coordinates\n    y1, x1 = max(0, new_height - height) // 2, max(0, new_width - width) // 2\n    y2, x2 = y1 + height, x1 + width\n    bbox = np.array([y1,x1,y2,x2])\n    # Map back to original image coordinates\n    bbox = (bbox / zoom_factor).astype(np.int)\n    y1, x1, y2, x2 = bbox\n    cropped_img = img[y1:y2, x1:x2]\n\n    # Handle padding when downscaling\n    resize_height, resize_width = min(new_height, height), min(new_width, width)\n    pad_height1, pad_width1 = (height - resize_height) // 2, (width - resize_width) //2\n    pad_height2, pad_width2 = (height - resize_height) - pad_height1, (width - resize_width) - pad_width1\n    pad_spec = [(pad_height1, pad_height2), (pad_width1, pad_width2)] + [(0,0)] * (img.ndim - 2)\n\n    result = cv2.resize(cropped_img, (resize_width, resize_height))\n    result = np.pad(result, pad_spec, mode='constant')\n    assert result.shape[0] == height and result.shape[1] == width\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Introducing a dynamic zoom on images that are too zoomed out (mostly class 0's)"},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_dir = '/kaggle/input/aptos2019-blindness-detection/train_images/'\n#######################################################################################\n# Pointing train_dir to include external and internal data combined\n#######################################################################################\ntrain_dir = '/kaggle/input/ext-data/train_combined/train_combined/'\nimgpath = train_df[train_df['diagnosis']==0]['id_code'][4]\n\npath = train_dir+imgpath+'.png'\nimage = cv2.imread(path)\nplt.imshow(image)\nprint(image.shape[1])\norig_shape = image.shape[1]\nplt.show()\nplt.imshow(crop_image_from_gray(image,tol=7))\nprint(crop_image_from_gray(image,tol=7).shape)\ncropped_shape = crop_image_from_gray(image,tol=7).shape[1]\ncropping_threshold = 0.05\nprint((cropped_shape - orig_shape)/orig_shape)\nplt.imshow(cv2_clipped_zoom(image, 1.4))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_ben_color_for_image(image, sigmaX=10):\n   # image = cv2.imread(path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    \n    orig_shape = image.shape[1]\n    image = crop_image_from_gray(image,tol=7)\n    cropped_shape = image.shape[1]\n    # Places an additional zoom on way-zoomed out eyes\n    cropping_threshold = -0.01\n    zoom_level  = random()*.2+1.0\n    if ((cropped_shape - orig_shape)/orig_shape < cropping_threshold):\n        image = cv2_clipped_zoom(image, zoom_level) \n    \n    image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))\n    image=cv2.addWeighted ( image,4, cv2.GaussianBlur( image , (0,0) , sigmaX) ,-4 ,128)\n        \n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Do it all preprocessing\ndef do_it_all(image):\n    #print(image)\n    # recolors, crops black space, adds the GaussianBlur, resizes\n    benned_image = load_ben_color_for_image(image)\n    benned_image = benned_image.astype(\"float32\")*(1.)/255\n    return np.array(benned_image)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Example of preprocessed images from every label\n\nfig, ax = plt.subplots(1, 5, figsize=(15, 6))\nfor i in range(5):\n    sample = train_df[train_df['diagnosis'] == i].sample(1)\n    image_name = sample['id_code'].item()\n    image_png = f\"{image_name}.png\"\n    X = do_it_all(cv2.imread(f\"{train_dir}{image_png}\"))\n\n    ax[i].set_title(f\"Image: {image_name}\\n Label = {sample['diagnosis'].item()}\", \n                    weight='bold', fontsize=10)\n    ax[i].axis('off')\n    ax[i].imshow(X);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare the dataframe for the flow_from_dataframe function"},{"metadata":{"trusted":true},"cell_type":"code","source":"training_df = pd.DataFrame()\ntesting_df = pd.DataFrame()\ntraining_df['diagnosis'] = train_df['diagnosis'].astype(str)\n\ntraining_df['id'] = train_df.id_code.apply(lambda x: x + '.png')\ntesting_df['id'] = test_df.id_code.apply(lambda x: x + '.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_train = pd.get_dummies(training_df['diagnosis']).values\ny_train_multi = np.empty(y_train.shape, dtype=y_train.dtype)\ny_train_multi[:, 4] = y_train[:, 4]\n\nfor i in range(3, -1, -1):\n    y_train_multi[:, i] = np.logical_or(y_train[:, i], y_train_multi[:, i+1])\ntesting = pd.DataFrame()\n#testing['0', '1', '2', '3', '4'] = y_train_multi\nprint(\"Original y_train:\\n\", y_train[:3])\nprint(\"Multilabel version:\\n\", y_train_multi[:3])\n\ny_train_multi_df = pd.DataFrame(data = y_train_multi[0:, 0:], index = y_train_multi[0:, 0], columns = ['0', '1', '2', '3', '4'])\ny_train_multi_df = y_train_multi_df.reset_index(drop = True)\nappendedTrainDF = training_df.merge(y_train_multi_df, left_index = True, right_index = True)\ntraining_df = appendedTrainDF\nprint(appendedTrainDF.head())\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfrom tensorflow.python.keras.utils.data_utils import Sequence\n\n#train_dir = '/kaggle/input/aptos2019-blindness-detection/train_images/'\n\n##########\n#  Ext data\n##############\n\ntrain_dir = '/kaggle/input/ext-data/train_combined/train_combined/'\ndatagen=ImageDataGenerator(rescale=1./255, validation_split = 0.20)\n\ntrain_generator=datagen.flow_from_dataframe(\n    dataframe=training_df, \n    directory=train_dir, \n    x_col=\"id\", \n    y_col=['0', '1', '2', '3', '4'],#\"diagnosis\", \n    subset = 'training', \n    class_mode='raw',#\"categorical\", \n    target_size=(224,224), \n    batch_size=20, \n    horizontal_flip = True,\n    zoom_range = [1,1.20],#0.25,\n    vertical_flip = True,\n    #just added\n    shuffle=True,\n    #Added in V29\n #   rotation_range = 120,\n    preprocessing_function = do_it_all)\n\nvalid_generator=datagen.flow_from_dataframe(\n    dataframe=training_df, \n    directory = train_dir, \n    x_col=\"id\", \n    y_col=['0', '1', '2', '3', '4'],#\"diagnosis\", \n    subset = 'validation', \n    target_size=(224, 224), \n    batch_size=20, \n    class_mode='raw',#'categorical', \n    horizontal_flip = True,\n    zoom_range = [1,1.20],#0.25,\n    vertical_flip = True,\n    #just added\n    shuffle=True,\n    #Added in V29\n#    rotation_range = 120,\n\n    preprocessing_function = do_it_all)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Creating keras callback for Quadratic Weighted Kappa"},{"metadata":{"trusted":true},"cell_type":"code","source":"#new\n\ndef get_preds_and_labels(model, generator):\n    \"\"\"\n    Get predictions and labels from the generator\n    \"\"\"\n    preds = []\n    labels = []\n    for _ in range(int(np.ceil(generator.samples / BATCH_SIZE))):\n        x, y = next(generator)\n        preds.append(model.predict(x))\n        labels.append(y)\n    # Flatten list of numpy arrays\n    return np.concatenate(preds).ravel(), np.concatenate(labels).ravel()\n\n\nclass Metrics(Callback):\n    \"\"\"\n    A custom Keras callback for saving the best model\n    according to the Quadratic Weighted Kappa (QWK) metric\n    \"\"\"\n    def on_train_begin(self, logs={}):\n        \"\"\"\n        Initialize list of QWK scores on validation data\n        \"\"\"\n        self.val_kappas = []\n\n    def on_epoch_end(self, epoch, logs={}):\n        \"\"\"\n        Gets QWK score on the validation data\n        \"\"\"\n        # Get predictions and convert to integers\n        y_pred, labels = get_preds_and_labels(model, valid_generator)\n        y_pred = np.rint(y_pred).astype(np.uint8).clip(0, 4)\n        # We can use sklearns implementation of QWK straight out of the box\n        # as long as we specify weights as 'quadratic'\n        _val_kappa = cohen_kappa_score(labels, y_pred, weights='quadratic')\n        self.val_kappas.append(_val_kappa)\n        print(f\"val_kappa: {round(_val_kappa, 4)}\")\n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save(SAVED_MODEL_NAME)\n        return","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model: DenseNet-121"},{"metadata":{"trusted":true},"cell_type":"code","source":"#densenet = DenseNet121(\n#    weights='../input/densenet-keras/DenseNet-BC-121-32-no-top.h5',\n#    include_top=False,\n#    input_shape=(224,224,3)\n#)\n\n#import effecientnet\n#efficientnetb5 = efn.EfficientNetB5(weights='imagenet', include_top=False, input_shape=(224, 224, 3))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model():\n    model = Sequential()\n    model.add(efficientnet_model)\n    model.add(layers.Dropout(0.25))\n    model.add(layers.Dense(2048))\n    model.add(layers.LeakyReLU())\n    model.add(layers.GlobalAveragePooling2D())\n    model.add(layers.Dropout(0.5))\n    #model.add(layers.Dense(1, activation='linear'))\n    \n    \n    \n    \n    \n    #model.add(efficientnetb5)    \n #   model.add(layers.GlobalAveragePooling2D())\n ##   model.add(layers.Dropout(0.5))\n #   model.add(layers.Dense(1024, activation = 'relu'))#, kernel_regularizer=l2(0.0005)))\n    #model.add(layers.Dense(1024, activation = 'relu'))#, kernel_regularizer=l2(0.0005)))\n #   model.add(layers.Dense(512, activation = 'relu'))#, kernel_regularizer=l2(0.001)))\n\n    model.add(layers.Dense(5, activation='sigmoid'))\n    \n    model.compile(\n        loss='binary_crossentropy',\n#        optimizer=Adam(lr=0.00005),\n        optimizer=Adam(lr=0.0001),\n        metrics=['accuracy']\n    )\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = build_model()\nmodel.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training & Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"kappa_metrics = Metrics()\n\n# Monitor MSE to avoid overfitting and save best model\nes = EarlyStopping(monitor='val_loss', mode='auto', verbose=1, patience=12)\nrlr = ReduceLROnPlateau(monitor='val_loss', \n                        factor=0.5, \n                        patience=1, \n                        verbose=1, \n                        mode='auto', \n                        epsilon=0.0001)\n\nBATCH_SIZE = valid_generator.batch_size\nSTEP_SIZE_TRAIN=train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID=valid_generator.n//valid_generator.batch_size\n\n\n#from keras.utils import to_categorical\n\nhistory = model.fit_generator(generator=train_generator,\n                              steps_per_epoch=STEP_SIZE_TRAIN, \n                              validation_data=valid_generator, \n                              validation_steps=STEP_SIZE_VALID, \n                              epochs=15,\n                              callbacks=[kappa_metrics, es, rlr]\n)\n\n#After 3 epochs, Epoch 3/3 With selective zoom in place\n#164/164 [==============================] - 352s 2s/step - loss: 0.1890 - acc: 0.9243 - val_loss: 0.1732 - val_acc: 0.9266\n\n#After 3 epochs Epoch 3/3 With multiple labels addition\n#164/164 [==============================] - 353s 2s/step - loss: 0.1270 - acc: 0.9525 - val_loss: 0.1328 - val_acc: 0.9526\n\n#AFter 3 epochs with random zoom set at rand()*0.2+1.3\n##Epoch 3/4\n#164/164 [==============================] - 388s 2s/step - loss: 0.1262 - acc: 0.9524 - val_loss: 0.1346 - val_acc: 0.9462\n\n#Added model saving capabilities based on the QWK, added preprocessing to the test step\n#164/164 [==============================] - 347s 2s/step - loss: 0.1244 - acc: 0.9535 - val_loss: 0.1288 - val_acc: 0.9506\n#val_kappa: 0.8877\n\n#Introducing a much larger range on the zoom, and from 1 to 1.4 instead of .75-.25   --> \n\n\n# Adding in the external data!!!\n\n\n# Solved the overfitting problem and made neural net deeper\n#Epoch 15/15\n#215/215 [==============================] - 621s 3s/step - loss: 0.1470 - acc: 0.9375 - val_loss: 0.1345 - val_acc: 0.9418\n#val_kappa: 0.8841x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"acc = history.history['acc']\nval_acc = history.history['val_acc']\nloss = history.history['loss']\nval_loss = history.history['val_loss']\n\nepochs = range(len(acc))\n\n\nplt.plot(epochs, acc, 'r', label='Training accuracy')\nplt.plot(epochs, val_acc, 'b', label='Validation accuracy')\nplt.title('Training and validation accuracy')\nplt.legend(loc=0)\nplt.figure()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load best weights according to MSE\nmodel.load_weights(SAVED_MODEL_NAME)\n\ntest_datagen=ImageDataGenerator(rescale=1./255.)\n\ntest_generator=test_datagen.flow_from_dataframe(\n    dataframe=testing_df,\n    directory=\"/kaggle/input/aptos2019-blindness-detection/test_images/\",\n    x_col=\"id\",\n    y_col=None,\n    batch_size=1,\n    seed=42,\n    shuffle=False,\n    class_mode=None,\n    target_size=(224,224), \n    preprocessing_function = do_it_all\n    )\n\nmodel.evaluate_generator(generator=valid_generator,steps=STEP_SIZE_VALID)\n\nSTEP_SIZE_TEST=test_generator.n//test_generator.batch_size\n\ntest_generator.reset()\npred=model.predict_generator(test_generator, steps=STEP_SIZE_TEST, verbose=1)\n\npredicted_class_indices=np.argmax(pred,axis=1)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submit"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_test = pred > 0.5\ny_test = y_test.astype(int).sum(axis=1) - 1\n\ntest_df['diagnosis'] = y_test\ntest_df.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label distribution\ntrain_df['diagnosis'].value_counts().sort_index().plot(kind=\"bar\", \n                                                       figsize=(15,5), \n                                                       rot=0)\nplt.title(\"Diagnosis Distribution in the Training Set\", \n          weight='bold', \n          fontsize=15)\nplt.xticks(fontsize=15)\nplt.yticks(fontsize=15)\nplt.xlabel(\"Label\", fontsize=17)\nplt.ylabel(\"Frequency\", fontsize=17);\nplt.show()\n\n# Label distribution\ntest_df['diagnosis'].value_counts().sort_index().plot(kind=\"bar\", \n                                                       figsize=(15,5), \n                                                       rot=0)\nplt.title(\"Label Distribution (Training Set)\", \n          weight='bold', \n          fontsize=15)\nplt.xticks(fontsize=15)\nplt.yticks(fontsize=15)\nplt.xlabel(\"Label\", fontsize=17)\nplt.ylabel(\"Frequency\", fontsize=17);\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}