{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h1><center>APTOS 2019 Blindness Detection</center></h1>\n<h2><center>Detect diabetic retinopathy to stop blindness before it's too late</center></h2>\n<center><img src=\"https://raw.githubusercontent.com/dimitreOliveira/MachineLearning/master/Kaggle/APTOS%202019%20Blindness%20Detection/aux_img.png\"></center>\n\nIn this synchronous Kernels-only competition, you'll build a machine learning model to speed up disease detection. You’ll work with thousands of images collected in rural areas to help identify diabetic retinopathy automatically. If successful, you will not only help to prevent lifelong blindness, but these models may be used to detect other sorts of diseases in the future, like glaucoma and macular degeneration.\n\nIn this notebook, I will be using basic deep learning and transfer learning (ResNet50) to create a baseline.\n##### Image source: http://cceyemd.com/diabetes-and-eye-exams/\n\n### Dependencies","metadata":{}},{"cell_type":"code","source":"import os\nimport cv2\nimport random\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom sklearn.metrics import confusion_matrix, cohen_kappa_score,classification_report\nfrom keras.models import Model,load_model\nfrom keras import optimizers, applications\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import EarlyStopping, ReduceLROnPlateau\nfrom keras.layers import Dense, Dropout, GlobalAveragePooling2D, Input, BatchNormalization\nfrom tensorflow.keras.applications import DenseNet121\nfrom tensorflow.keras.applications.densenet import preprocess_input\n# Set seeds to make the experiment more reproducible.\nimport tensorflow as tf\ndef seed_everything(seed=0):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(0)\nseed_everything()\n\n%matplotlib inline\nsns.set(style=\"whitegrid\")\nwarnings.filterwarnings(\"ignore\")","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load data","metadata":{"_kg_hide-output":true}},{"cell_type":"code","source":"train = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\ntest = pd.read_csv('../input/aptos2019-blindness-detection/test.csv')","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":false,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# EDA\n\n## Data overview","metadata":{}},{"cell_type":"code","source":"print('Number of train samples: ', train.shape[0])\nprint('Number of test samples: ', test.shape[0])\ndisplay(train.head())","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Label class distribution\n\nAs we can see we have an unbalanced database, we have two times more class 0 than 2, and classes 1, 2 and 4 each have less than half of the class 2 data.","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(14, 8.7))\nax = sns.countplot(x=\"diagnosis\", data=train, palette=\"GnBu_d\")\nsns.despine()\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##### Legend\n- 0 - No DR\n- 1 - Mild\n- 2 - Moderate\n- 3 - Severe\n- 4 - Proliferative DR ","metadata":{}},{"cell_type":"markdown","source":"### Now let's see some of the images\n\nThe images have different sizes, they may need resizing or some padding.","metadata":{}},{"cell_type":"code","source":"sns.set_style(\"white\")\ncount = 1\nplt.figure(figsize=[20, 20])\nfor img_name in train['id_code'][:15]:\n    img = cv2.imread(\"../input/aptos2019-blindness-detection/train_images/%s.png\" % img_name)[...,[2, 1, 0]]\n    plt.subplot(5, 5, count)\n    plt.imshow(img)\n    plt.title(\"Image %s\" % count)\n    count += 1\n    \nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model parameters","metadata":{}},{"cell_type":"code","source":"# Model parameters\nBATCH_SIZE = 32\nEPOCHS = 20\nWARMUP_EPOCHS = 2\nLEARNING_RATE = 1e-4\nWARMUP_LEARNING_RATE = 1e-3\nHEIGHT = 512\nWIDTH = 512\nCANAL = 3\nN_CLASSES = train['diagnosis'].nunique()\nES_PATIENCE = 5\nRLROP_PATIENCE = 3\nDECAY_DROP = 0.5","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Preprocecss data\ntrain[\"id_code\"] = train[\"id_code\"].apply(lambda x: x + \".png\")\ntest[\"id_code\"] = test[\"id_code\"].apply(lambda x: x + \".png\")\ntrain['diagnosis'] = train['diagnosis'].astype('str')\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data generator","metadata":{}},{"cell_type":"code","source":"train_datagen=ImageDataGenerator(rescale=1./255, \n                                 validation_split=0.3,\n                                 horizontal_flip=True)\n\ntrain_generator=train_datagen.flow_from_dataframe(\n    dataframe=train,\n    directory=\"../input/aptos2019-blindness-detection/train_images/\",\n    x_col=\"id_code\",\n    y_col=\"diagnosis\",\n    batch_size=BATCH_SIZE,\n    class_mode=\"categorical\",\n    target_size=(HEIGHT, WIDTH),\n    subset='training')\n\nvalid_generator=train_datagen.flow_from_dataframe(\n    dataframe=train,\n    directory=\"../input/aptos2019-blindness-detection/train_images/\",\n    x_col=\"id_code\",\n    y_col=\"diagnosis\",\n    batch_size=BATCH_SIZE,\n    class_mode=\"categorical\",    \n    target_size=(HEIGHT, WIDTH),\n    subset='validation')\n\ntest_datagen = ImageDataGenerator(rescale=1./255)\n\ntest_generator = test_datagen.flow_from_dataframe(  \n        dataframe=test,\n        directory = \"../input/aptos2019-blindness-detection/test_images/\",\n        x_col=\"id_code\",\n        target_size=(HEIGHT, WIDTH),\n        batch_size=1,\n        shuffle=False,\n        class_mode=None)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"code","source":"def create_model(input_shape, n_out):\n    input_tensor = Input(shape=input_shape)\n    base_model = applications.DenseNet121(weights='imagenet', \n                                       include_top=False,\n                                       input_tensor=input_tensor)\n    #base_model.load_weights('../input/resnet50/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5')\n\n    x = GlobalAveragePooling2D()(base_model.output)\n    x= BatchNormalization()(x)\n    x= Dropout(0.5)(x)\n    x= Dense(1024,activation='relu')(x) \n    x= Dense(512,activation='relu')(x) \n    x= BatchNormalization()(x)\n    x= Dropout(0.5)(x)\n    final_output = Dense(n_out, activation='softmax', name='final_output')(x)\n    model = Model(input_tensor, final_output)\n    \n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = create_model(input_shape=(HEIGHT, WIDTH, CANAL), n_out=N_CLASSES)\n\nfor layer in model.layers:\n    layer.trainable = False\n\nfor i in range(-5, 0):\n    model.layers[i].trainable = True\n\nmetric_list = [\"accuracy\"]\noptimizer = optimizers.Adam(lr=WARMUP_LEARNING_RATE)\nmodel.compile(optimizer=optimizer, loss=\"categorical_crossentropy\",  metrics=metric_list)\nmodel.summary()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train top layers","metadata":{}},{"cell_type":"code","source":"STEP_SIZE_TRAIN = train_generator.n//train_generator.batch_size\nSTEP_SIZE_VALID = valid_generator.n//valid_generator.batch_size\n\nhistory_warmup = model.fit_generator(generator=train_generator,\n                                     steps_per_epoch=STEP_SIZE_TRAIN,\n                                     validation_data=valid_generator,\n                                     validation_steps=STEP_SIZE_VALID,\n                                     epochs=WARMUP_EPOCHS,\n                                     verbose=1).history","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fine-tune the complete model","metadata":{}},{"cell_type":"code","source":"for layer in model.layers:\n    layer.trainable = True\n\nes = EarlyStopping(monitor='val_loss', mode='min', patience=ES_PATIENCE, restore_best_weights=True, verbose=1)\nrlrop = ReduceLROnPlateau(monitor='val_loss', mode='min', patience=RLROP_PATIENCE, factor=DECAY_DROP, min_lr=1e-6, verbose=1)\n\ncallback_list = [es, rlrop]\noptimizer = optimizers.Adam(lr=LEARNING_RATE)\nmodel.compile(optimizer=optimizer, loss=\"categorical_crossentropy\",  metrics=metric_list)\nmodel.summary()","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history_finetunning = model.fit_generator(generator=train_generator,\n                                          steps_per_epoch=STEP_SIZE_TRAIN,\n                                          validation_data=valid_generator,\n                                          validation_steps=STEP_SIZE_VALID,\n                                          epochs=EPOCHS,\n                                          callbacks=callback_list,\n                                          verbose=1).history","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model loss graph ","metadata":{}},{"cell_type":"code","source":"history = {'loss': history_warmup['loss'] + history_finetunning['loss'], \n           'val_loss': history_warmup['val_loss'] + history_finetunning['val_loss'], \n           'accuracy': history_warmup['accuracy'] + history_finetunning['accuracy'], \n           'val_accuracy': history_warmup['val_accuracy'] + history_finetunning['val_accuracy']}\n\nsns.set_style(\"whitegrid\")\nfig, (ax1, ax2) = plt.subplots(2, 1, sharex='col', figsize=(20, 14))\n\nax1.plot(history['loss'], label='Train loss')\nax1.plot(history['val_loss'], label='Validation loss')\nax1.legend(loc='best')\nax1.set_title('Loss')\n\nax2.plot(history['accuracy'], label='Train Accuracy')\nax2.plot(history['val_accuracy'], label='Validation accuracy')\nax2.legend(loc='best')\nax2.set_title('Accuracy')\n\nplt.xlabel('Epochs')\nsns.despine()\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.save(\"./my_model.h5\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Evaluation","metadata":{}},{"cell_type":"code","source":"complete_datagen = ImageDataGenerator(rescale=1./255)\ncomplete_generator = complete_datagen.flow_from_dataframe(  \n        dataframe=train,\n        directory = \"../input/aptos2019-blindness-detection/train_images/\",\n        x_col=\"id_code\",\n        target_size=(HEIGHT, WIDTH),\n        batch_size=1,\n        shuffle=False,\n        class_mode=None)\n\nSTEP_SIZE_COMPLETE = complete_generator.n//complete_generator.batch_size\ntrain_preds = model.predict_generator(complete_generator, steps=STEP_SIZE_COMPLETE)\ntrain_preds = [np.argmax(pred) for pred in train_preds]","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Confusion Matrix","metadata":{}},{"cell_type":"code","source":"labels = ['0 - No DR', '1 - Mild', '2 - Moderate', '3 - Severe', '4 - Proliferative DR']\ncnf_matrix = confusion_matrix(train['diagnosis'].astype('int'), train_preds)\ncnf_matrix_norm = cnf_matrix.astype('float') / cnf_matrix.sum(axis=1)[:, np.newaxis]\ndf_cm = pd.DataFrame(cnf_matrix_norm, index=labels, columns=labels)\nplt.figure(figsize=(16, 7))\nsns.heatmap(df_cm, annot=True, fmt='.2f', cmap=\"Blues\")\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(classification_report(train['diagnosis'].astype('int'), train_preds))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Quadratic Weighted Kappa","metadata":{}},{"cell_type":"code","source":"print(\"Train Cohen Kappa score: %.3f\" % cohen_kappa_score(train_preds, train['diagnosis'].astype('int'), weights='quadratic'))","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Apply model to test set and output predictions","metadata":{}},{"cell_type":"code","source":"test_generator.reset()\nSTEP_SIZE_TEST = test_generator.n//test_generator.batch_size\npreds = model.predict_generator(test_generator, steps=STEP_SIZE_TEST,verbose =1)\npredictions = [np.argmax(pred) for pred in preds]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filenames = test_generator.filenames\nresults = pd.DataFrame({'id_code':filenames, 'diagnosis':predictions})\nresults['id_code'] = results['id_code'].map(lambda x: str(x)[:-4])\nresults.to_csv('submission.csv',index=False)\nresults.head(10)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Predictions class distribution","metadata":{}},{"cell_type":"code","source":"f, ax = plt.subplots(figsize=(14, 8.7))\nax = sns.countplot(x=\"diagnosis\", data=results, palette=\"GnBu_d\")\nsns.despine()\nplt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"my_model = load_model(\"./my_model.h5\")\nmy_model.summary()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}