{"cells":[{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#from IPython.core.display import display, HTML\n#toggle_code_str = '''\n#
\n#'''\n#\n#toggle_code_prepare_str = '''\n# \n#\n#'''\n#\n#display(HTML(toggle_code_prepare_str + toggle_code_str))\n#\n#def toggle_code():\n# display(HTML(toggle_code_str))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# APTOS 2019 Blindness Detection\n\n## Detect diabetic retinopathy to stop blindness before it's too late\n\nMillions of people suffer from diabetic retinopathy, the leading cause of blindness among working aged adults. [Aravind Eye Hospital](https://aravind.org/) in India hopes to detect and prevent this disease among people living in rural areas where medical screening is difficult to conduct. Successful entries in this competition will improve the hospital’s ability to identify potential patients. Further, the solutions will be spread to other Ophthalmologists through the [4th Asia Pacific Tele-Ophthalmology Society (APTOS) Symposium](http://2019.asiateleophth.org/).\n\nCurrently, Aravind technicians travel to these rural areas to capture images and then rely on highly trained doctors to review the images and provide diagnosis. Their goal is to scale their efforts through technology; to gain the ability to automatically screen images for disease and provide information on how severe the condition may be.\n\nIn this synchronous Kernels-only competition, you'll build a machine learning model to speed up disease detection. You’ll work with thousands of images collected in rural areas to help identify diabetic retinopathy automatically. If successful, you will not only help to prevent lifelong blindness, but these models may be used to detect other sorts of diseases in the future, like glaucoma and macular degeneration.\n\nLet's go!","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"First, I'll import some necessary modules. **The following code is ready to work under Keras 2.3.0 and Tensorflow 1.14.0...**","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"from warnings import filterwarnings\nfilterwarnings(\"ignore\")\n\nimport os\nimport sys\nimport glob\nfrom time import time\nimport numpy as np\nimport pandas as pd\nimport imageio as io\nimport cv2 as cv\nimport matplotlib.pyplot as plt\nfrom IPython.display import SVG, HTML\nfrom keras.applications.resnet50 import ResNet50, preprocess_input\nimport tensorflow as tf\nimport keras.backend as K\nfrom keras.callbacks import Callback, EarlyStopping, ReduceLROnPlateau\nfrom keras.layers import Dense, Dropout, Activation, BatchNormalization, GlobalAveragePooling2D\nfrom keras.models import Sequential, load_model\nfrom keras.optimizers import Adam, SGD, RMSprop, Nadam, Optimizer\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.regularizers import l2\nfrom keras.utils import plot_model, to_categorical\nfrom keras.utils.generic_utils import get_custom_objects\nfrom keras.utils.vis_utils import model_to_dot\nfrom keras.wrappers.scikit_learn import KerasClassifier\nfrom sklearn.model_selection import GridSearchCV, train_test_split\nfrom sklearn.metrics import roc_auc_score, confusion_matrix, precision_score, recall_score, f1_score, cohen_kappa_score\n\n#!pip install -U graphviz\n#!pip install -U --pre efficientnet\n#!pip install -U git+https://github.com/qubvel/efficientnet\n#from efficientnet.keras import EfficientNetB7, preprocess_input\nsys.path.append(os.path.abspath('/kaggle/input/efficientnetb7-keras-model-weights'))\nsys.path.append(os.path.abspath('/kaggle/input/ranger-optimizer-rectified-adam-lookahead'))\n#from efficientnet import EfficientNetB7, preprocess_input\nfrom RAdam import RAdam\nfrom Lookahead import Lookahead\n\n#sys.path.append(os.path.abspath('../input/ranger-optimizer-rectified-adam-lookahead'))\n#import ranger_optimizer\n\nplt.style.use('seaborn-paper')\n\n%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\n\nprint('Tensorflow version:', tf.__version__)\n\nprint('\\nSetup complete!')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#%load '../input/efficientnetb7-keras-model-weights/efficientnet.py'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's define some useful functions...","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def convert_seconds_to_time(seconds):\n \"\"\"\n (float -> str)\n \n Converts seconds (float) in days, hours, minutes and seconds and returns a string with the result. \n \"\"\"\n if seconds < float(86400) and seconds >= float(3600):\n h, sec = divmod(int(round(seconds)), 3600)\n m, sec = divmod(int(sec), 60)\n return f'{int(h)} hours, {int(m)} minutes and {round(sec)} seconds'\n \n elif seconds < float(86400) and seconds < float(3600):\n if seconds >= float(60):\n m, sec = divmod(int(round(seconds)), 60)\n return f'{int(m)} minutes and {round(sec)} seconds'\n else:\n return f'{round(seconds)} seconds'\n else:\n d, sec = divmod(int(round(seconds)), 86400)\n return f'{int(d)} days, {convert_seconds(float(sec))}'\n \ndef diab_retin(prediction):\n \"\"\"\n (int -> str)\n \n Returns a string with information of the type of diabetic retinopathy, if present, \n according to an integer which is the prediction given by the model.\n \"\"\"\n if prediction == 0:\n return 'No diabetic retinopathy'\n elif prediction == 1:\n return 'Mild nonproliferative diabetic retinopathy'\n elif prediction == 2:\n return 'Moderate nonproliferative diabetic retinopathy'\n elif prediction == 3:\n return 'Severe nonproliferative diabetic retinopathy'\n elif prediction == 4:\n return 'Proliferative diabetic retinopathy'\n else:\n raise ValueError('The argument should be an integer from 0 to 4, both included.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Although submissions are scored based on the quadratic weighted kappa, I'll use [Cohen's kappa coefficient](https://en.wikipedia.org/wiki/Cohen%27s_kappa) ($\\kappa$) instead for convenience. It measures the agreement between two raters who each classify N items into C mutually exclusive categories.\n\n$\\kappa$ is defined to be:\n\n$$\\kappa := \\frac{p_o - p_e}{1 - p_e}$$\n\nwhere $p_o$ is the relative observed agreement among raters (identical to accuracy), and $p_e$ is the hypothetical probability of chance agreement, using the observed data to calculate the probabilities of each observer randomly seeing each category. If the raters are in complete agreement then $\\kappa = 1$. If there is no agreement among the raters other than what would be expected by chance (as given by $p_e$) $\\kappa = 0$. It is possible for the statistic to be negative, which implies that there is no effective agreement between the two raters or the agreement is worse than random.","execution_count":null},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# PREPARING CUSTOM METRICS (Cohen's kappa coefficient): \n#def cohens_kappa(y_true, y_pred):\n# y_true_classes = K.argmax(y_true, axis = 1)\n# y_pred_classes = K.argmax(y_pred, axis = 1)\n# return tf.contrib.metrics.cohen_kappa(y_true_classes, y_pred_classes, 5)[1] # Returns update_op: Operation that increments po, pe_row and pe_col variables appropriately and whose value matches kappa.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'll will use a new state-of-the-art activation function: **Mish**\n\n$$Mish(x) = x \\, \\tanh{(\\varsigma(x))} \\\\\n\\text{where } \\varsigma(x) = \\text{softplus(x)} = \\ln{(1+e^x)}$$\n\n\n\nReference: [Mish: A Self Regularized Non-Monotonic Neural Activation Function](https://arxiv.org/abs/1908.08681) (Diganta Misra)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# PREPARING CUSTOM ACTIVATION FUNCTION (MISH(x) = x * tanh(ln(1+e^x))) \n#-------------------------------------------------------------------\nclass Mish(Activation):\n \n def __init__(self, activation, **kwargs):\n super(Mish, self).__init__(activation, **kwargs)\n self.__name__ = 'mish'\n \ndef mish_activation(z):\n \"\"\"\n Returns new Mish activation of z (Mish = z * tanh(ln(1+e^z))) as a tensor\n \"\"\"\n return z * K.tanh(K.softplus(z))\n\nget_custom_objects().update({'mish': Mish(mish_activation)})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To create a powerful model I'll use transfer learning and fine tune of a pre-trained model ([ResNet50](https://ethereon.github.io/netscope/#/gist/db945b393d40bfa26006) or [EfficientNet](https://arxiv.org/abs/1905.11946))","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"url_resnet50 = '../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5' \nbase_model = ResNet50(include_top = False, input_shape = (256,256,3), pooling = 'avg', weights = url_resnet50)\n\n#base_model = EfficientNetB7(weights = None, include_top = False, input_shape = (256, 256, 3))\n#base_model.load_weights('../input/efficientnetb7-keras-model-weights/efficientnet-b7_weights_tf_dim_ordering_tf_kernels_autoaugment_notop.h5')\n\n\nhidden_layers = 1\ndropout = 0.05\nlambda2 = 0.\nunits = 250","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#for idx, layer in enumerate(base_model.layers):\n# print(idx, layer.name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### RAdam (rectified adaptive moment estimation) optimizer\n\nRAdam is a state-of-the-art optimizer which is a variation of the classic Adam that provides an automated, dynamic adjustment to the adaptive learning rate based on their detailed study into the effects of variance and momentum during training. RAdam, compared to RMSProp and Adam, does not need to warm the learning rate up. This is the process:\n\n$$\\begin{cases}\nv_0, s_0 = 0, 0 \\qquad \\text{(Initialize moving 1st and 2nd moment)}\\\\\np_\\infty = \\frac {2}{1-\\beta_2} - 1 \\qquad \\text{(maximum length of the approximated simple moving average)}\\\\\np_t = p_\\infty - 2t\\frac{(\\beta_2)^t}{1-(\\beta_2)^t} \\qquad \\text{(length of the approximated simple moving average at iteration $t$)}\\\\\nv_l = \\beta_1 v_{l-1} + (1 - \\beta_1) \\frac{\\partial \\mathcal{J} }{ \\partial W^{[l]}} \\qquad \\text{(updates the exponential moving 1st moment)}\\\\\n\\hat{v_l} = \\frac{v_l}{1 - (\\beta_1)^t} \\qquad \\text{(bias-corrected $v_l$)}\\\\\ns_l = \\beta_2 s_{l-1} + (1 - \\beta_2) (\\frac{\\partial \\mathcal{J} }{\\partial W^{[l]} })^2 \\qquad \\text{(updates the exponential moving 2st moment)}\n\\end{cases}$$\n\n$\\qquad \\qquad$ if the variance is tractable, i.e., $p_t > 4$:\n\n$$\\begin{cases}\n\\hat{s_l} = \\sqrt{\\frac{s_l}{1 - (\\beta_2)^t}} \\qquad \\text{(bias-corrected $s_l$)}\\\\\nr_t = \\sqrt{\\frac{(p_t-4) (p_t-2) p_\\infty}{(p_\\infty-4) (p_\\infty-2) p_t}} \\qquad \\text{(rectification term at iteration $t$)}\\\\\nW^{[l]} = W^{[l]} - \\alpha r_t \\frac{\\hat{v_l}}{\\hat{s_l}} \\qquad \\text{(updates parameters with adaptive momentum)}\n\\end{cases}$$\n\n$\\qquad \\qquad$ else:\n\n$$W^{[l]} = W^{[l]} - \\alpha \\hat{v_l} \\qquad \\text{(updates parameters with un-adapted momentum)}$$\n\nwhere:\n- $t$ counts the number of steps taken of RAdam (iterations)\n- $l$ is the current layer\n- $\\beta_1$ and $\\beta_2$ are hyperparameters which control the two exponentially weighted averages. \n- $\\alpha$ is the learning rate\n\nReference: [“On the Variance of the Adaptive Learning Rate and Beyond”](https://arxiv.org/abs/1908.03265v1) (Liyuan Liu, Haoming Jiang, et *al*.)\n\n### LookAhead optimizer\n\nLookAhead chooses a search direction by *looking ahead* at the sequence of “fast weights\" generated by another optimizer (RAdam in this case). This process improves the learning stability and lowers the variance of its inner optimizer with negligible computation and memory cost. \n\n\n\nReference: [“Lookahead Optimizer: k steps forward, 1 step back”](https://arxiv.org/pdf/1907.08610v1.pdf) (Michael R. Zhang, James Lucas, Geoffrey Hinton)","execution_count":null},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"def my_model(hidden_layers = hidden_layers,\n dropout = dropout,\n lambda2 = lambda2,\n base_model = base_model,\n units = units, \n lr = 0.03, \n pool = 'avg', \n classes = 5):\n \n model = Sequential(name = 'APTOS_model')\n model.add(base_model)\n \n #model.add(GlobalAveragePooling2D())\n\n for num in range(hidden_layers):\n #dense_layer_name = 'FC_' + str(num + 1)\n model.add(Dense(units, kernel_regularizer = l2(lambda2)))\n model.add(BatchNormalization())\n model.add(Activation('mish'))\n if dropout > 0:\n model.add(Dropout(dropout))\n \n model.add(Dense(classes, activation = 'softmax', name = 'Predictions', kernel_regularizer = l2(lambda2)))\n \n for layer in base_model.layers:\n layer.trainable = False\n \n optim = RAdam(beta_1 = 0.95, beta_2 = 0.999, learning_rate = lr)\n Ranger = Lookahead(optimizer = optim, k = 5, alpha = 0.5) # # Implement RAdam with LookAhead\n model.compile(optimizer = Ranger,\n loss = 'categorical_crossentropy',\n metrics = ['accuracy'])\n \n return model","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# In case we do not use model.fit_generator but model.fit with validation_data: (x_val, y_val) or model.evaluate. Create an instance of the class as a callback.\n\n#class Metrics(Callback):\n# \n# def __init__(self, classification, classes):\n# self.classification = classification\n# if self.classification.lower() == 'binary':\n# classes = 2\n# self.classes = classes\n# \n# def on_train_begin(self, logs={}):\n# #self.confusion = []\n# #self.precision = []\n# #self.recall = []\n# #self.f1s = []\n# #self.auc = []\n# self.kappa = []\n#\n# def on_epoch_end(self, epoch, logs={}):\n# if self.classification.lower() == 'binary':\n# score = np.asarray(self.model.predict(self.validation_data[0]))\n# predict = np.round(score).astype(int)\n# targ = self.validation_data[1]\n# elif self.classification.lower() == 'multilabel' or self.classification.lower() == 'categorical':\n# score = np.asarray(self.model.predict(self.validation_data[0]))\n# predict = np.argmax(score, axis = 1)\n# predict_categ = to_categorical(predict, self.classes)\n# # we assume here that we have one-hot encoded labels:\n# targ_categ = self.validation_data[1]\n# targ = np.argmax(targ_categ, axis = 1)\n# # in case we have ground truth labels, not one-hot encoded labels:\n# #targ = self.validation_data[1]\n# #targ_categ = to_categorical(targ, self.classes)\n# \n# #val_auc = roc_auc_score(targ, score)\n# #val_confusion = confusion_matrix(targ, predict)\n# #val_precision = precision_score(targ, predict)\n# #val_recall = recall_score(targ, predict)\n# #val_f1 = f1_score(targ, predict, average = 'macro')\n# val_kappa = cohen_kappa_score(targ, predict)\n# \n# #self.auc.append(val_auc)\n# #self.confusion.append(val_confusion)\n# #self.precision.append(val_precision)\n# #self.recall.append(val_recall)\n# #self.f1s.append(val_f1)\n# self.kappa.append(val_kappa)\n# \n# print(f'Validation kappa score: {val_kappa:.4f}\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = my_model()\nprint('MODEL SUMMARY BEFORE FINE-TUNING', '\\n')\nprint(model.summary(), '\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_model(model, to_file = 'model_APTOS.png', show_shapes = False)\n#SVG(model_to_dot(model).create(prog = 'dot', format = 'svg'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"#%%time\ntimea = time()\n \nwith np.load('../input/kernel-aptos-img-to-arrays-one-hot-and-split/train_set.npz') as traindata:\n print('Data in train_set.npz:', traindata.files, '.....', end = '')\n X_train = traindata['X_train']\n Y_train = traindata['Y_train']\n print('Retrieved!')\n\nwith np.load('../input/kernel-aptos-img-to-arrays-one-hot-and-split/val_set.npz') as valdata:\n print('Data in val_set.npz:', valdata.files, '.....', end = '')\n X_val = valdata['X_val']\n Y_val = valdata['Y_val']\n print('Retrieved!')\n\ntimeb = time()\ntotal_time = timeb - timea\n\nprint(f'All preprocessed data retrieved in {convert_seconds_to_time(total_time)}\\n')\n#\nprint('Shape of the array containing preprocessed training images', X_train.shape)\nprint('Shape of the array containing one-hot encoded labels of the training subset', Y_train.shape)\nprint('Shape of the array containing preprocessed cross-validation images', X_val.shape)\nprint('Shape of the array containing one-hot encoded labels of the cross-validation subset', Y_val.shape,'\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#GRID SEARCH CV\n#--------------\n\n#model = KerasClassifier(build_fn = my_model)\n#params = {'dropout': [0, 0.15, 0.3],\n# 'hidden_layers': [1, 2, 3]}\n#\n#grid = GridSearchCV(estimator = model,\n# param_grid = params,\n# cv = 2)\n#\n#timea = time()\n#grid = grid.fit(X_train, Y_train)\n#best_params = grid.best_params_\n#best_score = grid.best_score_\n#timeb = time()\n#\n#total_time = timeb - timea\n#\n#print(f'\\nGrid Search completed in {convert_seconds_to_time(total_time)}')\n#print(best_params)\n#print(best_score)\n#print(f'\\nThe best score is {best_score:.4f} with the following parameters: {best_params}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#DATA AUGMENTATION - TRANSFER LEARNING\n#---------------------------------------\n#timea = time()\n#\n#df_train = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\n#df_train['id_code'] = df_train['id_code'].apply(lambda x: x + '.png')\n#df_train['diagnosis'] = df_train['diagnosis'].astype('str')\n#\n#url_train = '../input/aptos2019-blindness-detection/train_images/'\n#train_datagen = ImageDataGenerator(rescale = 1./255,\n# preprocessing_function = preprocess_input,\n# horizontal_flip = True,\n# vertical_flip = True,\n# rotation_range = 8,\n# shear_range = 0.3,\n# zoom_range = 0.3,\n# validation_split = 0.2,\n# channel_shift_range = 0.3)\n#\n#train_generator = train_datagen.flow_from_dataframe(dataframe = df_train, \n# directory = url_train, \n# x_col = 'id_code',\n# y_col = 'diagnosis',\n# target_size = (600, 600),\n# subset = 'training',\n# class_mode = 'categorical',\n# batch_size = 16, \n# shuffle = True)\n#\n#val_generator = train_datagen.flow_from_dataframe(dataframe = df_train, \n# directory = url_train, \n# x_col = 'id_code',\n# y_col = 'diagnosis',\n# target_size = (600, 600),\n# subset = 'validation',\n# class_mode = 'categorical',\n# batch_size = 16,\n# shuffle = True)\n#\n#num_minibatches_train = train_generator.samples // train_generator.batch_size # number of training images // batch size (16 in this case)\n#num_minibatches_cv = val_generator.samples // val_generator.batch_size # number of cross-validation images // batch size (16 in this case)\n#\n#K.get_session().run(tf.local_variables_initializer()) # We need to initialize tf variables before training to use cohens_kappa function we defined before.\n#\n#epochs_pre = 2\n##my_metrics = Metrics(classification = 'categorical', classes = 5)\n#modelAPTOS_pretuned_hist = model.fit_generator(train_generator, \n# steps_per_epoch = num_minibatches_train, \n# epochs = epochs_pre,\n# verbose = 1, \n# validation_data = val_generator,\n# validation_steps = num_minibatches_cv)\n#\n#timeb = time()\n#total_time = timeb - timea","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#print(f'\\nData augmentation and training with cross-validation after {epochs_pre} epochs completed in {convert_seconds_to_time(total_time)}', '\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"#DATA AUGMENTATION - TRANSFER LEARNING\n#---------------------------------------\n\ngen_train = ImageDataGenerator(horizontal_flip = True,\n vertical_flip = True,\n rotation_range = 10,\n shear_range = 0.3,\n zoom_range = 0.3,\n preprocessing_function = preprocess_input,\n channel_shift_range = 0.3)\n#\ngen_cv = ImageDataGenerator(preprocessing_function = preprocess_input)\n#\ntrain_generator = gen_train.flow(X_train, Y_train, batch_size = 16, shuffle = True)\nval_generator = gen_cv.flow(X_val, Y_val, batch_size = 16, shuffle = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"#%%time\ntimea = time()\n\n#num_minibatches_train = train_generator.n // train_generator.batch_size # number of training images // batch size (16 in this case)\n#num_minibatches_cv = val_generator.n // val_generator.batch_size # number of cross-validation images // batch size (16 in this case)\n\n#K.get_session().run(tf.local_variables_initializer()) # We need to initialize tf variables before training to use cohens_kappa function we defined before.\n\nepochs_pre = 2\n#my_metrics = Metrics(classification = 'categorical', classes = 5)\nmodelAPTOS_pretuned_hist = model.fit_generator(train_generator, \n steps_per_epoch = train_generator.n, \n epochs = epochs_pre,\n verbose = 1, \n validation_data = val_generator,\n validation_steps = val_generator.n)\ntimeb = time()\ntotal_time = timeb - timea\n#\n#pre_tune_history = modelAPTOS_pretuned_hist.history\n#model.save('modelAPTOS_PRETUNED.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'\\nData augmentation and pre-training with cross-validation after {epochs_pre} epochs completed in {convert_seconds_to_time(total_time)}', '\\n')\n#print('Accuracy of the pre-tuned model on the training subset:', pre_tune_history['accuracy'][-1])\n#print('Accuracy of the pre-tuned model on the cross-validation subset:', pre_tune_history['val_accuracy'][-1], '\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#FINE TUNING\n#---------\n\ntimea = time()\n\n#-------------------------\n#FINE TUNING FOR RESNET50:\n\nfor layer in base_model.layers[:143]:\n layer.trainable = False\nfor layer in base_model.layers[143:]:\n layer.trainable = True\n\n#--------------------------\n#FINE TUNING FOR EFFICIENTNETB7\n\n#841\n\n#for layer in base_model.layers[:841]:\n# layer.trainable = False\n#for layer in base_model.layers[841:]:\n# layer.trainable = True \n#-----------------------------\n\noptim = RAdam(beta_1 = 0.95, beta_2 = 0.999, learning_rate = 3e-5) # Low learning rate for RAdam for fine tuning...\nRanger = Lookahead(optimizer = optim, k = 5, alpha = 0.5) # # Implement RAdam with LookAhead again.\n\nmodel.compile(optimizer = Ranger,\n loss = 'categorical_crossentropy', \n metrics = ['accuracy'])\n\n#K.get_session().run(tf.local_variables_initializer()) # We need to initialize tf variables before training to use cohens_kappa function we defined before.\n\nprint('MODEL SUMMARY AFTER FINE-TUNING', '\\n') \nprint(model.summary(), '\\n')\n\nepochs_post = 42\n#my_metrics = Metrics(classification = 'categorical', classes = 5)\nearly_stop = EarlyStopping(monitor = 'val_accuracy', min_delta = 0, patience = 10, verbose = 1, mode = 'max', baseline = None, restore_best_weights = True)\nreduce_lr = ReduceLROnPlateau(monitor = 'val_accuracy', factor = 0.1, patience = 8, mode = 'max', min_lr = 1e-8)\n\nmodelAPTOS_tuned_hist = model.fit_generator(train_generator, \n steps_per_epoch = train_generator.n, \n epochs = epochs_post,\n verbose = 2, \n validation_data = val_generator,\n validation_steps = val_generator.n,\n callbacks = [early_stop, reduce_lr])\n\ntimeb = time()\ntotal_time = timeb - timea\n\npost_tune_history = modelAPTOS_tuned_hist.history\nind_best_epoch = post_tune_history['val_accuracy'].index(early_stop.best)\nmodel.save('modelAPTOS_TUNED.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'\\nFine tuning with early stopping of the model after {early_stop.stopped_epoch +1} epochs completed in {convert_seconds_to_time(total_time)}', '\\n')\nprint(f'Accuracy of the tuned model on the training subset (from epoch {ind_best_epoch + 1}):', post_tune_history['accuracy'][ind_best_epoch])\nprint(f'Accuracy of the tuned model on the cross-validation subset (from epoch {ind_best_epoch + 1}):', early_stop.best, '\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#print(f'\\nFine tuning of the model after {epochs_post} epochs completed in {convert_seconds_to_time(total_time)}', '\\n')\n#print('Accuracy of the tuned model on the training subset:', post_tune_history['accuracy'][-1])\n#print('Accuracy of the tuned model on the cross-validation subset:', post_tune_history['val_accuracy'][-1], '\\n')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"#plot_model(model, to_file = 'model_APTOS.png', show_shapes = True)\n#SVG(model_to_dot(model).set_size('4x48').create(prog = 'dot', format = 'svg'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#model.evaluate_generator(batch_cv, steps = batch_cv.n, verbose = 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"%reset_selective -f X_train\n%reset_selective -f Y_train\n%reset_selective -f X_val\n%reset_selective -f Y_val","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#timea = time()\n#with np.load('../input/kernel-aptos-img-to-arrays-one-hot-and-split/test_set.npz') as testdata:\n# print('Data in test_set.npz:', testdata.files, '.....', end = '')\n# X_test = testdata['test_im']\n# print('Retrieved!')\n# \n#timeb = time()\n#total_time = timeb - timea\n#\n#print(f'Array containing test images retrieved in {convert_seconds_to_time(total_time)}\\n')\n# \n#print('Shape of the array containing preprocessed test images', X_test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#gen_test = ImageDataGenerator(preprocessing_function = preprocess_input)\n#\n#batch_test = gen_test.flow(X_test, y = None, batch_size = 1, shuffle = False)\n\n#batch_test.reset()\n\n#STEP_SIZE_TEST = batch_test.n // batch_test.batch_size # number of test images // batch size (1 in this case)\n#predictions = model.predict_generator(batch_test, \n# steps = batch_test.n, \n# verbose = 2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#predictions = np.argmax(predictions, axis = 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#url = r'../input/aptos2019-blindness-detection/test.csv'\n#test = pd.read_csv(url)\n#test_ids = test['id_code'].tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#submission = pd.DataFrame({'id_code': test_ids, 'diagnosis': predictions})\n#submission.to_csv('submission.csv', index = False)\n#\n#print('PREDICTIONS READY!')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#epochs_used = len(post_tune_history['acc'])\n#\n#plt.figure(figsize = (12,6))\n\n#ax[0].plot(range(1, epochs_pre + 1), [x * 100 for x in pre_tune_history['acc']], 'o-b', label = 'Training')\n#ax[0].plot(range(1, epochs_pre + 1), [x * 100 for x in pre_tune_history['val_acc']], 'o-r', label = 'Cross validation')\n#ax[0].set_xlabel('Epochs')\n#ax[0].set_ylabel('Accuracy (%)')\n#ax[0].set_yticks(range(40,105,10))\n#ax[0].set_xticks(range(1, epochs_pre + 1, 1))\n#ax[0].set_title(f'PERFORMANCE OF TRANSFER LEARNING\\n WITH RESNET-50 AFTER {epochs_pre} EPOCHS', loc = 'center')\n#ax[0].legend(loc = 'upper left')\n\n#plt.plot(range(1, epochs_used + 1), [x * 100 for x in post_tune_history['acc']], 'o-b', label = 'Training')\n#plt.plot(range(1, epochs_used + 1), [x * 100 for x in post_tune_history['val_acc']], 'o-r', label = 'Cross validation')\n#plt.xlabel('Epochs')\n#plt.ylabel('Accuracy (%)')\n#plt.yticks(range(40,110,10))\n#plt.xticks(range(1, epochs_used + 1))\n#plt.title(f'PERFORMANCE OF FINE TUNING ON RESNET-50 AFTER {epochs_used} EPOCHS WITH EARLY STOPPING')\n#plt.legend(loc = 'upper left')\n#\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#plt.figure(figsize = (12,6))\n\n#plt.plot(range(1, epochs_used + 1), my_metrics.kappa, 'o-b')\n#plt.xlabel('Epochs')\n#plt.ylabel('Kappa score for the cross validation set')\n#plt.yticks(range(-1, 2))\n#plt.xticks(range(1, epochs_used + 1))\n#plt.title(f'PERFORMANCE OF FINE TUNING ON RESNET-50 AFTER {epochs_used} EPOCHS WITH EARLY STOPPING')\n#\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#WITH EARLYSTOPPING:\n\nepochs_used = ind_best_epoch + 1\nplt.figure(figsize = (12,6))\n\nplt.plot(range(1, epochs_used + 1), [x * 100 for x in post_tune_history['accuracy'][:ind_best_epoch + 1]], 'o-b', label = 'Training')\nplt.plot(range(1, epochs_used + 1), [x * 100 for x in post_tune_history['val_accuracy'][:ind_best_epoch + 1]], 'o-r', label = 'Cross validation')\nplt.xlabel('Epochs')\nplt.ylabel('Accuracy (%)')\nplt.yticks(range(40,110,10))\nplt.xticks(range(1, epochs_used + 1))\nplt.title(f'PERFORMANCE OF FINE TUNING WITH EARLY STOPPING ON RESNET-50 AFTER {epochs_used} EPOCHS')\nplt.legend(loc = 'upper left')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"#WITHOUT EARLYSTOPPING:\n\n#epochs_used = len(post_tune_history['accuracy'])\n#plt.figure(figsize = (12,6))\n#\n#plt.plot(range(1, epochs_used + 1), [x * 100 for x in post_tune_history['accuracy']], 'o-b', label = 'Training')\n#plt.plot(range(1, epochs_used + 1), [x * 100 for x in post_tune_history['val_accuracy']], 'o-r', label = 'Cross validation')\n#plt.xlabel('Epochs')\n#plt.ylabel('Accuracy (%)')\n#plt.yticks(range(40,110,10))\n#plt.xticks(range(1, epochs_used + 1))\n#plt.title(f'PERFORMANCE OF FINE TUNING ON RESNET-50 AFTER {epochs_used} EPOCHS')\n#plt.legend(loc = 'upper left')\n#\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#epochs_used = len(post_tune_history['accuracy'])\n#\n#fig, ax = plt.subplots(1, 2, figsize = (24, 6))\n#fig.suptitle(f'PERFORMANCE OF TRANSFER LEARNING WITH RESNET-50 AFTER {epochs_used} EPOCHS', fontsize = 18)\n#fig.suptitle(f'PERFORMANCE OF TRANSFER LEARNING WITH EFFICIENTNET AFTER {epochs_used} EPOCHS', fontsize = 18)\n#\n#ax[0].plot(range(1, epochs_used + 1), [x * 100 for x in post_tune_history['accuracy']], 'o-b', label = 'Training')\n#ax[0].plot(range(1, epochs_used + 1), [x * 100 for x in post_tune_history['val_accuracy']], 'o-r', label = 'Cross validation')\n#ax[0].set_xlabel('Epochs')\n#ax[0].set_ylabel('Accuracy (%)')\n#ax[0].set_yticks(range(40,105,10))\n#ax[0].set_xticks(range(1, epochs_used + 1))\n#ax[0].set_title('ACCURACY FOR THE TRAINING AND CROSS VALIDATIONS SETS', loc = 'center')\n#ax[0].legend(loc = 'upper left')\n#\n#ax[1].plot(range(1, epochs_used + 1), post_tune_history['cohens_kappa'], 'o-b', label = 'Training')\n#ax[1].plot(range(1, epochs_used + 1), post_tune_history['val_cohens_kappa'], 'o-r', label = 'Cross validation')\n#ax[1].set_xlabel('Epochs')\n#ax[1].set_ylabel(\"Cohen's kappa coefficient\")\n#ax[1].set_yticks(range(-1, 2))\n#ax[1].set_xticks(range(1, epochs_used + 1))\n#ax[1].set_title(\"COHEN'S KAPPA COEFFICIENT FOR THE TRAINING AND CROSS VALIDATIONS SETS\", loc = 'center')\n#ax[1].legend(loc = 'upper left')\n\n#ax[1].plot(range(1, epochs_used + 1), my_metrics.kappa, 'o-g')\n#ax[1].set_xlabel('Epochs')\n#ax[1].set_ylabel('Quadratic kappa score')\n#ax[1].set_yticks(range(-1, 2))\n#ax[1].set_xticks(range(1, epochs_used + 1))\n#ax[1].set_title(\"COHEN'S KAPPA SCORE FOR THE CROSS VALIDATION SET\", loc = 'center')\n \n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"url_test = r\"/kaggle/input/aptos2019-blindness-detection/test.csv\"\nurl_sample = r\"/kaggle/input/aptos2019-blindness-detection/sample_submission.csv\"\ntest = pd.read_csv(url_test)\nsample = pd.read_csv(url_sample)\n\ntest_id_codes = test[\"id_code\"].tolist()\n#%reset_selective -f test\n\nindex = 0\npredictions_list = []\n#gen_test = ImageDataGenerator(preprocessing_function = preprocess_input)\n\ntimec = time()\nprint(\"Processing and resizing images from the test set...\")\nfor im_test in test_id_codes:\n uri = glob.glob(\"/kaggle/input/aptos2019-blindness-detection/test_images/\" + im_test + \".*\")\n image = io.imread(uri[0])\n image = cv.resize(image, (256, 256), interpolation = cv.INTER_AREA) / 255 #Normalising...\n image = np.expand_dims(image, axis = 0)\n image = preprocess_input(image)\n prediction = model.predict(image)\n prediction = np.argmax(prediction)\n predictions_list.append(prediction)\n %reset_selective -f image\n index += 1\n if index % 500 == 0:\n print(f\"\\t{index} images\") \n\n%reset_selective -f test_id_codes, test\ntimed = time()\ntotal_time = timed - timec \nprint(f\"All images from the test set have been processed in {convert_seconds_to_time(total_time)}\\n\")\nprint(\"Predictions ready!\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Distribution of the predicted diagnoses')\nprint('---------------------------------------\\n')\n\nnum_diagnosis_0 = predictions_list.count(0)\nnum_diagnosis_1 = predictions_list.count(1)\nnum_diagnosis_2 = predictions_list.count(2)\nnum_diagnosis_3 = predictions_list.count(3)\nnum_diagnosis_4 = predictions_list.count(4)\n\nsample['diagnosis'] = predictions_list\n#%reset_selective -f predictions_list\n\nprint(f\" 0 - No DR: {num_diagnosis_0} examples\")\nprint(f\" 1 - Mild: {num_diagnosis_1} examples\")\nprint(f\" 2 - Moderate: {num_diagnosis_2} examples\")\nprint(f\" 3 - Severe: {num_diagnosis_3} examples\")\nprint(f\" 4 - Proliferative DR: {num_diagnosis_4} examples\\n\")\n\nprint(f\" Total number of test examples = {num_diagnosis_0 + num_diagnosis_1 + num_diagnosis_2 + num_diagnosis_3 + num_diagnosis_4}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with np.load('../input/kernel-aptos-img-to-arrays-one-hot-and-split/test_set.npz') as testdata:\n print('Data in test_set.npz:', testdata.files, '.....', end = '')\n X_test = testdata['test_im']\n print('Retrieved!')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rand_samples = np.random.randint(0, X_test.shape[0] + 1, size = 3)\n\nfig, ax = plt.subplots(1,len(rand_samples), figsize = (18, 9))\nplt.suptitle('Predictions for 3 random examples from the provided test set of images', fontsize = 20)\n\nfor i in range(len(rand_samples)):\n ax[i].imshow(X_test[rand_samples[i]])\n ax[i].set_title(diab_retin(predictions_list[rand_samples[i]]))\n ax[i].axis('off')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"%reset_selective -f X_test","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample.to_csv('submission.csv', index = False)\nprint('Submission ready!')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}