{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# ResNet50\n## Trained on APTOS19 + EyePACS15 \n\n#### Quadratic Weighted Kappa (QWK):\nQuadratic Weighted Kappa (QWK, the greek letter $\\kappa$), also known as Cohen's Kappa, is the official evaluation metric in both 2015 and 2019 competitions. \nAccording to the [wikipedia article](https://en.wikipedia.org/wiki/Cohen%27s_kappa), we have\n> The definition of $\\kappa$ is:\n> $$\\kappa \\equiv \\frac{p_o - p_e}{1 - p_e}$$\n> where $p_o$ is the relative observed agreement among raters (identical to accuracy), and $p_e$ is the hypothetical probability of chance agreement, using the observed data to calculate the probabilities of each observer randomly seeing each category.\n\nThis metric is used because if we just using accuracy as the metric, it will give spurious results (because the data is unbalanced). The QWK is more fit to the problem.\n\n#### Ordinal Regression (OR):\n> In DR classification, the disease classes are not independent, meaning that the severity of the disease is not necessarily limited to a specific set of discrete classes. Therefore, an ordinal regression model is more appropriate because it can predict a continuous severity score, allowing for a more precise prediction of disease severity.\n---\n\nWe treat this as an Ordinal Regression Problem based on this litterature: \n- Mean absolute error as training metric instead of accuracy; \n> For an ordinal regression model, accuracy is not an appropriate metric for ordinal regression because it only measures the percentage of correctly classified samples and does not take into account the ordinal structure of the output. Instead, you can use metrics such as mean absolute error (MAE) or mean squared error (MSE) of the predicted scores compared to the true scores. \n- Mean Squared Error as loss function instead of Cross Entropy;  [This](https://ieeexplore.ieee.org/iel7/9342628/9342629/09342711.pdf) \nand [this (Presentation video)](https://www.youtube.com/watch?v=IvhzwhjFyFQ)\n\n\n### Credits\n#### This series of notebooks is heavily inspired by FEDERICO RAIMONDI *brilliant* kernels (DenseNet and EfficientNet).\nPlease check his [Inference Kernel](https://www.kaggle.com/raimonds1993/aptos19-densenet-inference-old-new-data/data?scriptVersionId=17252732)!\n\nThank you!","metadata":{}},{"cell_type":"markdown","source":"#  ResNet Notebook \n","metadata":{}},{"cell_type":"code","source":"# To have reproducible results and compare them\nnr_seed = 2019\nimport numpy as np \nnp.random.seed(nr_seed)\nimport tensorflow as tf\ntf.set_random_seed(nr_seed)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:39:06.960123Z","iopub.execute_input":"2023-08-04T11:39:06.960469Z","iopub.status.idle":"2023-08-04T11:39:08.313426Z","shell.execute_reply.started":"2023-08-04T11:39:06.960409Z","shell.execute_reply":"2023-08-04T11:39:08.312526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import libraries\nimport json\nimport math\nfrom tqdm import tqdm, tqdm_notebook\nimport gc\nimport warnings\nimport os\n\nimport cv2\nfrom PIL import Image\n\nimport pandas as pd\nimport scipy\nimport matplotlib.pyplot as plt\n\nfrom keras import backend as K\nfrom keras import layers\nfrom keras.applications.resnet50 import ResNet50\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score\n\nwarnings.filterwarnings(\"ignore\")\n\n%matplotlib inline","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-08-04T11:39:11.220498Z","iopub.execute_input":"2023-08-04T11:39:11.220860Z","iopub.status.idle":"2023-08-04T11:39:12.145741Z","shell.execute_reply.started":"2023-08-04T11:39:11.220802Z","shell.execute_reply":"2023-08-04T11:39:12.144575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Image size\nim_size = 224\n# Batch size\nBATCH_SIZE = 32","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:39:16.816636Z","iopub.execute_input":"2023-08-04T11:39:16.816989Z","iopub.status.idle":"2023-08-04T11:39:16.821714Z","shell.execute_reply.started":"2023-08-04T11:39:16.816922Z","shell.execute_reply":"2023-08-04T11:39:16.820833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading & Merging","metadata":{}},{"cell_type":"code","source":"# Load the datasets\nnew_train = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\nold_train = pd.read_csv('../input/diabetic-retinopathy-resized/trainLabels.csv')\ntest_df = pd.read_csv('/kaggle/input/messidor2preprocess/messidor_data.csv')\n# Drop the columns adjudicated_dme and adjudicated_gradable\ntest_df.drop(['adjudicated_dme', 'adjudicated_gradable'], axis=1, inplace=True)\n# Display the shape of the test dataset\nprint(test_df.shape)\nprint(new_train.shape)\nprint(old_train.shape)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:41:33.302485Z","iopub.execute_input":"2023-08-04T11:41:33.302875Z","iopub.status.idle":"2023-08-04T11:41:33.380355Z","shell.execute_reply.started":"2023-08-04T11:41:33.302816Z","shell.execute_reply":"2023-08-04T11:41:33.379608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"old_train = old_train[['image','level']]\nold_train.columns = new_train.columns\nold_train.diagnosis.value_counts()\n\n# Combine the image filenames with their respective paths\nnew_train['id_code'] = '../input/aptos2019-blindness-detection/train_images/' + new_train['id_code'].astype(str) + '.png'\nold_train['id_code'] = '../input/diabetic-retinopathy-resized/resized_train/resized_train/' + old_train['id_code'].astype(str) + '.jpeg'\ntest_df['id_code'] = '../input/messidor2preprocess/messidor-2/messidor-2/preprocess/' + test_df['id_code'].astype(str)\n\ntrain_df = old_train.copy()\nval_df = new_train.copy()\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:42:05.056301Z","iopub.execute_input":"2023-08-04T11:42:05.056656Z","iopub.status.idle":"2023-08-04T11:42:05.173018Z","shell.execute_reply.started":"2023-08-04T11:42:05.056585Z","shell.execute_reply":"2023-08-04T11:42:05.172187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# TRAIN DF","metadata":{}},{"cell_type":"code","source":"# Create a histogram of the diabetic retinopathy grades for train_df\nplt.figure(figsize=(8, 6))\nplt.hist(train_df['diagnosis'], bins=5, color='blue', alpha=0.7, rwidth=0.85)\nplt.xlabel('Diabetic Retinopathy Grade')\nplt.ylabel('Number of images')\nplt.title('Distribution of Diabetic Retinopathy Grades in EyePACS 2015 (Train Dataset)')\nplt.xticks(range(5), ['0', '1', '2', '3', '4'])\nplt.show()\n\n# Create a table to show grade distribution in train_df\ntrain_grade_distribution = train_df['diagnosis'].value_counts().sort_index()\ntrain_grade_distribution_table = pd.DataFrame({'Diagnosis Grade': train_grade_distribution.index, 'Frequency': train_grade_distribution.values})\nprint(train_grade_distribution_table)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:42:44.002639Z","iopub.execute_input":"2023-08-04T11:42:44.002973Z","iopub.status.idle":"2023-08-04T11:42:44.285049Z","shell.execute_reply.started":"2023-08-04T11:42:44.002916Z","shell.execute_reply":"2023-08-04T11:42:44.282449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# VALIDATION DF\nThe APTOS 2019 Kaggle dataset [46] consists of 3662 retina images with different image sizes. Only the ground truths of the training images are publicly available. The dataset is classified into five DR stages. In addition, 1805 of the images are normal and 1857 are DR images. The distribution of the dataset is imbalanced, with most of the images normal.","metadata":{}},{"cell_type":"code","source":"# Create a histogram of the diabetic retinopathy grades for val_df\nplt.figure(figsize=(8, 6))\nplt.hist(val_df['diagnosis'], bins=5, color='salmon', alpha=0.7, rwidth=0.85)\nplt.xlabel('Diabetic Retinopathy Grade')\nplt.ylabel('Number of Images')\nplt.title('Distribution of Diabetic Retinopathy Grades in APTOS 2019 (Validation Dataset)')\nplt.xticks(range(5), ['0', '1', '2', '3', '4'])\nplt.show()\n\n# Create a table to show grade distribution in val_df\nval_grade_distribution = val_df['diagnosis'].value_counts().sort_index()\nval_grade_distribution_table = pd.DataFrame({'Diagnosis Grade': val_grade_distribution.index, 'Frequency': val_grade_distribution.values})\nprint(val_grade_distribution_table)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:42:49.840300Z","iopub.execute_input":"2023-08-04T11:42:49.840698Z","iopub.status.idle":"2023-08-04T11:42:50.126621Z","shell.execute_reply.started":"2023-08-04T11:42:49.840635Z","shell.execute_reply":"2023-08-04T11:42:50.125464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **TEST DF**\n\nThe Messidor-2 dataset is a collection of Diabetic Retinopathy (DR) examinations, each consisting of two macula-centered eye fundus images (one per eye). Only macula-centered images were included in the dataset. Messidor-2 contains 874 examinations (1748 images). The dataset comes with a spreadsheet containing image pairing.\nThis is a preprocessed version of Messidor-2 dataset available at Messidor-2 in order to delete the excess of black background. The Diabetic Retinopathy grades (as well as DME and Gradability) are from MESSIDOR-2 DR Grades.\n\nPlease cite the following paper if you use this data in your research/study:\n\nKrause, J. et al. Grader variability and the importance of reference standards\nfor evaluating machine learning models for diabetic retinopathy.\nOphthalmology (2018). doi:10.1016/j.ophtha.2018.01.034\n\nAbout the data:\n\nThe csv file contains adjudicated 5 point ICDR grades and Referable DME grades\nfor the Messidor 2 dataset [1]. Each row of the csv corresponds to a single\nimage, with 4 columns labeled as:\n\nimage_id, adjudicated_dr_grade, adjudicated_dme, adjudicated_gradable\n\nColumn descriptions:\n\nimage_id: Filename as provided in the Messidor 2 dataset.\n\nadjudicated_dr_grade: 5 point ICDR grade\n  0=None\n  1=Mild DR\n  2=Moderate DR\n  3=Severe DR\n  4=PDR\n\nadjudicated_dme: Referable DME defined by Hard exudates within 1DD\n  0=No Referable DME\n  1=Referable DME\n\nadjudicated_gradable: Image quality grade.\n  0=Ungradable, no DR or DME grade is provided\n  1=Gradable, both DR and DME were graded.\nPlease note that adjudicated_dr_grade and adjudicated_dme columns will be empty\nfor images where adjudicated_gradable=0.\n\nThe grading was done according to the adjudication protocol described in\nKrause et al. [2], which demonstrated improvements in data and model quality\nover Gulshan et al. [3] using adjudicated data.\n\nReferences:\n\n[1] Decencière  E, Etienne  D, Xiwei  Z,  et al.  Feedback on a publicly distributed image database: the Messidor database.  Image Anal Stereol. 2014;33(3):231-234. doi:10.5566/ias.1155\n\n[2] Krause, J. et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology (2018). doi:10.1016/j.ophtha.2018.01.034\n\n[3] Gulshan, V. et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA 316, 2402–2410 (2016)","metadata":{}},{"cell_type":"code","source":"# Create a histogram of the diabetic retinopathy grades for test_df\nplt.figure(figsize=(8, 6))\nplt.hist(test_df['diagnosis'], bins=5, color='green', alpha=0.7, rwidth=0.85)\nplt.xlabel('Diabetic Retinopathy Grade')\nplt.ylabel('Number of images')\nplt.title('Distribution of Diabetic Retinopathy Grades in Messidor-2 (Test Dataset)')\nplt.xticks(range(5), ['0', '1', '2', '3', '4'])\nplt.show()\n\n# Create a table to show grade distribution in test_df\ntest_grade_distribution = test_df['diagnosis'].value_counts().sort_index()\ntest_grade_distribution_table = pd.DataFrame({'Diagnosis Grade': test_grade_distribution.index, 'Frequency': test_grade_distribution.values})\nprint(test_grade_distribution_table)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:42:57.953733Z","iopub.execute_input":"2023-08-04T11:42:57.954055Z","iopub.status.idle":"2023-08-04T11:42:58.221126Z","shell.execute_reply.started":"2023-08-04T11:42:57.953999Z","shell.execute_reply":"2023-08-04T11:42:58.217797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's shuffle the datasets\ntrain_df = train_df.sample(frac=1).reset_index(drop=True)\nval_df = val_df.sample(frac=1).reset_index(drop=True)\ntest_df = test_df.sample(frac=1).reset_index(drop=True)\n\n# Display the shapes of the shuffled datasets\nprint(train_df.shape)\nprint(val_df.shape)\nprint(test_df.shape)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:43:14.475174Z","iopub.execute_input":"2023-08-04T11:43:14.475528Z","iopub.status.idle":"2023-08-04T11:43:14.495145Z","shell.execute_reply.started":"2023-08-04T11:43:14.475464Z","shell.execute_reply":"2023-08-04T11:43:14.494304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## PREPROCESSING","metadata":{}},{"cell_type":"code","source":"def display_samples(df, columns=4, rows=3):\n    fig=plt.figure(figsize=(5*columns, 4*rows))\n\n    for i in range(columns*rows):\n        image_path = df.loc[i,'id_code']\n        image_id = df.loc[i,'diagnosis']\n        img = cv2.imread(f'{image_path}')\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        #img = crop_image_from_gray(img)\n        img = cv2.resize(img, (im_size,im_size))\n        img = cv2.addWeighted(img,4,cv2.GaussianBlur(img, (0,0), im_size/40) ,-4 ,128)\n        \n        fig.add_subplot(rows, columns, i+1)\n        plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-08-04T11:43:18.655482Z","iopub.execute_input":"2023-08-04T11:43:18.655849Z","iopub.status.idle":"2023-08-04T11:43:18.666018Z","shell.execute_reply.started":"2023-08-04T11:43:18.655790Z","shell.execute_reply":"2023-08-04T11:43:18.664556Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_samples(train_df)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:43:39.979843Z","iopub.execute_input":"2023-08-04T11:43:39.980176Z","iopub.status.idle":"2023-08-04T11:43:43.984032Z","shell.execute_reply.started":"2023-08-04T11:43:39.980118Z","shell.execute_reply":"2023-08-04T11:43:43.981501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_samples(val_df)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:43:53.764016Z","iopub.execute_input":"2023-08-04T11:43:53.764345Z","iopub.status.idle":"2023-08-04T11:44:00.090677Z","shell.execute_reply.started":"2023-08-04T11:43:53.764289Z","shell.execute_reply":"2023-08-04T11:44:00.089573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_samples(test_df)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:44:09.864587Z","iopub.execute_input":"2023-08-04T11:44:09.864925Z","iopub.status.idle":"2023-08-04T11:44:14.106969Z","shell.execute_reply.started":"2023-08-04T11:44:09.864868Z","shell.execute_reply":"2023-08-04T11:44:14.104710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Processing Images","metadata":{}},{"cell_type":"markdown","source":"**Updated Ben Grahm preprocessing method by Chatpatanasiri (crop function: : https://www.kaggle.com/ratthachat/aptos-updated-preprocessing-ben-s-cropping )**\n\ncrop_image1(img, tol=7): This function takes an input image img and a tolerance parameter tol. It creates a binary mask based on pixel intensity values greater than the tolerance value. It then crops the input image to include only the non-zero intensity regions based on the mask. The resulting cropped image is returned.\n\ncrop_image_from_gray(img, tol=7): This function performs a similar operation as crop_image1, but it can handle both grayscale (ndim==2) and color (ndim==3) images. For grayscale images, it directly applies the cropping operation. For color images, it first converts the image to grayscale, applies the cropping, and then extracts the corresponding color channels from the cropped region. If the cropped region is completely dark (all pixels below the tolerance threshold), the original image is returned.\n\npreprocess_image(image_path, desired_size=224): This function reads an image from the specified image_path. It converts the image to RGB color space, then applies the crop_image_from_gray function to remove dark or unimportant areas. The image is then resized to the desired size using bilinear interpolation. A weighted combination of the image and a Gaussian-blurred version is performed, enhancing edges and details. The resulting preprocessed image is returned.\n\npreprocess_image_old(image_path, desired_size=224): This is a simplified version of the previous preprocessing function. It doesn't include the cropping step based on pixel intensity. Instead, it directly resizes the image to the desired size and applies a similar weighted combination of the image and Gaussian blur.\n\n@article{Chatpatanasiri2019_APTOS_Preprocess,\ntitle = \"APTOS : Eye Preprocessing in Diabetic Retinopathy\",\nauthor = \"Chatpatanasiri, Ratthachat\",\njournal = \"https://www.kaggle.com/\",\nyear = \"2019\",\nurl = \"\\url{https://www.kaggle.com/ratthachat/aptos-eye-preprocessing-in-diabetic-retinopathy}\",\nnote = \"[Online; accessed DD-MM-YYYY]\"\n}","metadata":{}},{"cell_type":"code","source":"def crop_image1(img,tol=7):\n    # img is image data\n    # tol  is tolerance\n        \n    mask = img>tol\n    return img[np.ix_(mask.any(1),mask.any(0))]\n\ndef crop_image_from_gray(img,tol=7):\n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n        mask = gray_img>tol\n        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0): # image is too dark so that we crop out everything,\n            return img # return original image\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n            img = np.stack([img1,img2,img3],axis=-1)\n        return img\n\ndef preprocess_image(image_path, desired_size=224):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = crop_image_from_gray(img)\n    img = cv2.resize(img, (desired_size,desired_size))\n    img = cv2.addWeighted(img,4,cv2.GaussianBlur(img, (0,0), desired_size/30) ,-4 ,128)\n    \n    return img\n\ndef preprocess_image_old(image_path, desired_size=224):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    #img = crop_image_from_gray(img)\n    img = cv2.resize(img, (desired_size,desired_size))\n    img = cv2.addWeighted(img,4,cv2.GaussianBlur(img, (0,0), desired_size/40) ,-4 ,128)\n    \n    return img","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:44:38.756251Z","iopub.execute_input":"2023-08-04T11:44:38.756600Z","iopub.status.idle":"2023-08-04T11:44:38.776675Z","shell.execute_reply.started":"2023-08-04T11:44:38.756519Z","shell.execute_reply":"2023-08-04T11:44:38.775428Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"__UPDATE:__ Here we are reading just the validation set. In order to use 224x224 images, we are going to load one bucket at a time only when needed. This will let our code run without memory-related errors.","metadata":{}},{"cell_type":"code","source":"# validation set\nN = val_df.shape[0]\nx_val = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n\nfor i, image_id in enumerate(tqdm_notebook(val_df['id_code'])):\n    x_val[i, :, :, :] = preprocess_image(\n        f'{image_id}',\n        desired_size = im_size\n    )","metadata":{"execution":{"iopub.status.busy":"2023-08-04T11:48:35.959133Z","iopub.execute_input":"2023-08-04T11:48:35.959564Z","iopub.status.idle":"2023-08-04T11:59:51.432437Z","shell.execute_reply.started":"2023-08-04T11:48:35.959486Z","shell.execute_reply":"2023-08-04T11:59:51.431604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a directory to store the processed validation images\noutput_dir = '/kaggle/working/processed_images'\nos.makedirs(output_dir, exist_ok=True)\n\n# Save the processed validation images\nfor i, image_id in enumerate(tqdm_notebook(val_df['id_code'])):\n    output_path = os.path.join(output_dir, f'{image_id}.png')\n    cv2.imwrite(output_path, x_val[i])","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:04:18.972738Z","iopub.execute_input":"2023-08-04T12:04:18.973071Z","iopub.status.idle":"2023-08-04T12:04:19.054437Z","shell.execute_reply.started":"2023-08-04T12:04:18.973015Z","shell.execute_reply":"2023-08-04T12:04:19.053681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load the labels for the datasets\ny_train = train_df['diagnosis'].values\ny_val = val_df['diagnosis'].values\ny_test = test_df['diagnosis'].values\n\n# Display the shapes of the target arrays\nprint(\"Train Labels Shape:\", y_train.shape)\nprint(\"Validation Labels Shape:\", y_val.shape)\nprint(\"Test Labels Shape:\", y_test.shape)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:06:15.486204Z","iopub.execute_input":"2023-08-04T12:06:15.486767Z","iopub.status.idle":"2023-08-04T12:06:15.495470Z","shell.execute_reply.started":"2023-08-04T12:06:15.486512Z","shell.execute_reply":"2023-08-04T12:06:15.493945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# delete the uneeded df\ndel new_train\ndel old_train\ndel val_df\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:06:33.666191Z","iopub.execute_input":"2023-08-04T12:06:33.666523Z","iopub.status.idle":"2023-08-04T12:06:33.781841Z","shell.execute_reply.started":"2023-08-04T12:06:33.666461Z","shell.execute_reply":"2023-08-04T12:06:33.780927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating keras callback for QWK\n---\nBecause our main metric in this problem is QWK, we have to make a function that calculate the QWK for every epoch and also saving the model that got the highest score. \nI had to change this function, in order to consider the best kappa score among all the buckets.","metadata":{}},{"cell_type":"code","source":"class Metrics(Callback):\n\n    def on_epoch_end(self, epoch, logs={}):\n        X_val, y_val = self.validation_data[:2]\n        \n        y_pred = self.model.predict(X_val)\n        \n        coef = [0.5, 1.5, 2.5, 3.5]\n\n        for i, pred in enumerate(y_pred):\n            if pred < coef[0]:\n                y_pred[i] = 0\n            elif pred >= coef[0] and pred < coef[1]:\n                y_pred[i] = 1\n            elif pred >= coef[1] and pred < coef[2]:\n                y_pred[i] = 2\n            elif pred >= coef[2] and pred < coef[3]:\n                y_pred[i] = 3\n            else:\n                y_pred[i] = 4\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n        \n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n\n        return","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:06:50.118237Z","iopub.execute_input":"2023-08-04T12:06:50.118572Z","iopub.status.idle":"2023-08-04T12:06:50.130519Z","shell.execute_reply.started":"2023-08-04T12:06:50.118502Z","shell.execute_reply":"2023-08-04T12:06:50.129279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Generator","metadata":{}},{"cell_type":"code","source":"def create_datagen():\n    return ImageDataGenerator(\n        horizontal_flip = True,\n        vertical_flip = True,\n        rotation_range = 160,\n        zoom_range=0.35\n    )","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:06:59.041208Z","iopub.execute_input":"2023-08-04T12:06:59.041529Z","iopub.status.idle":"2023-08-04T12:06:59.045945Z","shell.execute_reply.started":"2023-08-04T12:06:59.041471Z","shell.execute_reply":"2023-08-04T12:06:59.045138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model: ResNet50","metadata":{}},{"cell_type":"code","source":"resnet = ResNet50(\n    weights=\"imagenet\",\n    include_top=False,\n    input_shape=(im_size,im_size,3)\n)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:07:03.321527Z","iopub.execute_input":"2023-08-04T12:07:03.321920Z","iopub.status.idle":"2023-08-04T12:07:20.015650Z","shell.execute_reply.started":"2023-08-04T12:07:03.321858Z","shell.execute_reply":"2023-08-04T12:07:20.014683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Model Architecture:**\nThe model uses a Sequential architecture, where layers are stacked sequentially.\nA pre-trained ResNet model is added as the initial layer. This leverages the power of transfer learning, enabling the model to learn relevant features from a large dataset.\nA Global Average Pooling 2D layer is added to reduce spatial dimensions and aggregate feature information.\nA Dropout layer with a dropout rate of 0.5 is added to prevent overfitting by randomly deactivating a fraction of neurons during training.\nA Dense (fully connected) layer with a single neuron and linear activation is added as the output layer, suitable for regression tasks where the goal is to predict a continuous output value.\n\n**Compilation:**\nThe model is compiled with the 'mean_squared_error' loss function, which is appropriate for regression problems. It measures the squared difference between predicted and actual values.\nThe Adam optimizer is used with a learning rate of 0.0001 and a small decay rate of 1e-6. Adam is an efficient optimizer for training neural networks.\n'mae' (Mean Absolute Error) is chosen as the evaluation metric, which provides a measure of the average absolute difference between predictions and ground truth.\n\n**Return:**\nThe built model is returned, ready for training and evaluation.\nOverall, this model architecture leverages the strengths of a pre-trained ResNet for feature extraction and introduces necessary components like Global Average Pooling and Dropout to enhance generalization and prevent overfitting. The choice of loss function, optimizer, and evaluation metric aligns well with the regression problem's nature, aiming to produce accurate continuous predictions.\n\n**Commentary:**\nOverall, this model architecture leverages the strengths of a pre-trained ResNet for feature extraction and introduces necessary components like Global Average Pooling and Dropout to enhance generalization and prevent overfitting. The choice of loss function, optimizer, and evaluation metric aligns well with the regression problem's nature, aiming to produce accurate continuous predictions.","metadata":{}},{"cell_type":"code","source":"def build_model():\n    model = Sequential()\n    model.add(resnet)\n    model.add(layers.GlobalAveragePooling2D())\n    model.add(layers.Dropout(0.5))\n    model.add(layers.Dense(1, activation='linear'))\n    \n    model.compile(\n        loss='mean_squared_error',\n        optimizer=Adam(lr=0.0001,decay=1e-6),\n        metrics=['mae']\n    )\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:07:36.944879Z","iopub.execute_input":"2023-08-04T12:07:36.945208Z","iopub.status.idle":"2023-08-04T12:07:36.960408Z","shell.execute_reply.started":"2023-08-04T12:07:36.945146Z","shell.execute_reply":"2023-08-04T12:07:36.959306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = build_model()\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:11:51.041985Z","iopub.execute_input":"2023-08-04T12:11:51.042328Z","iopub.status.idle":"2023-08-04T12:11:55.126316Z","shell.execute_reply.started":"2023-08-04T12:11:51.042272Z","shell.execute_reply":"2023-08-04T12:11:55.125341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pip install pydot graphviz\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:12:40.697230Z","iopub.execute_input":"2023-08-04T12:12:40.697566Z","iopub.status.idle":"2023-08-04T12:12:48.627793Z","shell.execute_reply.started":"2023-08-04T12:12:40.697497Z","shell.execute_reply":"2023-08-04T12:12:48.626900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.utils.vis_utils import plot_model\nfrom IPython.display import SVG\n\n# Plot the model architecture with shapes\nplot_model(model, show_shapes=True)\n# Saved as a PNG image\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:18:13.310162Z","iopub.execute_input":"2023-08-04T12:18:13.310507Z","iopub.status.idle":"2023-08-04T12:18:13.470529Z","shell.execute_reply.started":"2023-08-04T12:18:13.310448Z","shell.execute_reply":"2023-08-04T12:18:13.469477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training & Evaluation","metadata":{}},{"cell_type":"code","source":"#train_df = train_df.reset_index(drop=True)\nnum_bucket = 8\ndiv = round(train_df.shape[0]/num_bucket)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:18:34.071302Z","iopub.execute_input":"2023-08-04T12:18:34.071668Z","iopub.status.idle":"2023-08-04T12:18:34.077474Z","shell.execute_reply.started":"2023-08-04T12:18:34.071605Z","shell.execute_reply":"2023-08-04T12:18:34.076695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = pd.DataFrame({\n                        'val_loss': [0.0],\n                        'val_mean_absolute_error': [0.0],\n                        'loss': [0.0], \n                        'mean_absolute_error': [0.0],\n                        'bucket': [0.0]\n                        })","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:18:43.562298Z","iopub.execute_input":"2023-08-04T12:18:43.562651Z","iopub.status.idle":"2023-08-04T12:18:43.570669Z","shell.execute_reply.started":"2023-08-04T12:18:43.562590Z","shell.execute_reply":"2023-08-04T12:18:43.569635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I found that changing the nr. of epochs for each bucket helped in terms of performances\nepochs = [10,10,10,15,15,20,20,25]\nkappa_metrics = Metrics()\nkappa_metrics.val_kappas = []","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:19:01.535996Z","iopub.execute_input":"2023-08-04T12:19:01.536324Z","iopub.status.idle":"2023-08-04T12:19:01.541492Z","shell.execute_reply.started":"2023-08-04T12:19:01.536267Z","shell.execute_reply":"2023-08-04T12:19:01.540384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Update:** I decided to process images in buckets, save them then load them from storage to try and free more memory.","metadata":{}},{"cell_type":"code","source":"import gc\n\n# Preprocess and save images to working space\nfor i in range(0, num_bucket):\n    print(\"Bucket Nr: {}\".format(i))\n    \n    if i != (num_bucket - 1):\n        N = train_df.iloc[i * div:(1 + i) * div].shape[0]\n    else:\n        N = train_df.iloc[i * div:].shape[0]\n    \n    x_train = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n    \n    for j, image_id in enumerate(tqdm_notebook(train_df.iloc[i * div:(1 + i) * div, 0])):\n        x_train[j, :, :, :] = preprocess_image_old(f'{image_id}', desired_size=im_size)\n    \n    # Save the preprocessed images to working space\n    np.save(f'/kaggle/working/preprocessed_images_bucket_{i}.npy', x_train)\n    \n    del x_train\n    gc.collect()\n    print('-' * 40)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:26:29.235352Z","iopub.execute_input":"2023-08-04T12:26:29.235718Z","iopub.status.idle":"2023-08-04T12:38:25.376252Z","shell.execute_reply.started":"2023-08-04T12:26:29.235650Z","shell.execute_reply":"2023-08-04T12:38:25.375326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load and train model using preprocessed images\nfor i in range(0, num_bucket):\n    print(\"Bucket Nr: {}\".format(i))\n    \n    x_train = np.load(f'/kaggle/working/preprocessed_images_bucket_{i}.npy')\n    \n    if i != (num_bucket - 1):\n        data_generator = create_datagen().flow(\n            x_train, y_train[i * div:(1 + i) * div], batch_size=BATCH_SIZE\n        )\n    else:\n        data_generator = create_datagen().flow(x_train, y_train[i * div:], batch_size=BATCH_SIZE)\n    \n    history = model.fit_generator(\n        data_generator,\n        steps_per_epoch=x_train.shape[0] / BATCH_SIZE,\n        epochs=epochs[i],\n        validation_data=(x_val, y_val),\n        callbacks=[kappa_metrics]\n    )\n    \n    dic = history.history\n    df_model = pd.DataFrame(dic)\n    df_model['bucket'] = i\n    \n    results = results.append(df_model)\n    \n    del data_generator\n    del x_train\n    gc.collect()\n    \n    print('-' * 40)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T12:43:09.802910Z","iopub.execute_input":"2023-08-04T12:43:09.803288Z","iopub.status.idle":"2023-08-04T15:21:54.274274Z","shell.execute_reply.started":"2023-08-04T12:43:09.803227Z","shell.execute_reply":"2023-08-04T15:21:54.273342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(0,num_bucket):\n    if i != (num_bucket-1):\n        print(\"Bucket Nr: {}\".format(i))\n        \n        N = train_df.iloc[i*div:(1+i)*div].shape[0]\n        x_train = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n        for j, image_id in enumerate(tqdm_notebook(train_df.iloc[i*div:(1+i)*div,0])):\n            x_train[j, :, :, :] = preprocess_image_old(f'{image_id}', desired_size = im_size)\n\n        data_generator = create_datagen().flow(x_train, y_train[i*div:(1+i)*div], batch_size=BATCH_SIZE)\n        history = model.fit_generator(\n                        data_generator,\n                        steps_per_epoch=x_train.shape[0] / BATCH_SIZE,\n                        epochs=epochs[i],\n                        validation_data=(x_val, y_val),\n                        callbacks=[kappa_metrics]\n                        )\n        \n        dic = history.history\n        df_model = pd.DataFrame(dic)\n        df_model['bucket'] = i\n    else:\n        print(\"Bucket Nr: {}\".format(i))\n        \n        N = train_df.iloc[i*div:].shape[0]\n        x_train = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n        for j, image_id in enumerate(tqdm_notebook(train_df.iloc[i*div:,0])):\n            x_train[j, :, :, :] = preprocess_image_old(f'{image_id}', desired_size = im_size)\n        data_generator = create_datagen().flow(x_train, y_train[i*div:], batch_size=BATCH_SIZE)\n        \n        history = model.fit_generator(\n                        data_generator,\n                        steps_per_epoch=x_train.shape[0] / BATCH_SIZE,\n                        epochs=epochs[i],\n                        validation_data=(x_val, y_val),\n                        callbacks=[kappa_metrics]\n                        )\n        \n        dic = history.history\n        df_model = pd.DataFrame(dic)\n        df_model['bucket'] = i\n\n    results = results.append(df_model)\n    \n    del data_generator\n    del x_train\n    gc.collect()\n    \n    print('-'*40)","metadata":{"execution":{"iopub.status.busy":"2023-03-07T16:11:16.044460Z","iopub.execute_input":"2023-03-07T16:11:16.045038Z","iopub.status.idle":"2023-03-07T18:50:23.845805Z","shell.execute_reply.started":"2023-03-07T16:11:16.044729Z","shell.execute_reply":"2023-03-07T18:50:23.844942Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = results.iloc[1:]\nresults['kappa'] = kappa_metrics.val_kappas\nresults = results.reset_index()\nresults = results.rename(index=str, columns={\"index\": \"epoch\"})\nprint(max(results.kappa))","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:21:54.277081Z","iopub.execute_input":"2023-08-04T15:21:54.277720Z","iopub.status.idle":"2023-08-04T15:21:54.560219Z","shell.execute_reply.started":"2023-08-04T15:21:54.277664Z","shell.execute_reply":"2023-08-04T15:21:54.559005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results[['loss', 'val_loss']].plot()\nresults[['mean_absolute_error', 'val_mean_absolute_error']].plot()\nresults[['kappa']].plot()\nresults.to_csv('model_results.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:21:54.561814Z","iopub.execute_input":"2023-08-04T15:21:54.562301Z","iopub.status.idle":"2023-08-04T15:21:55.566469Z","shell.execute_reply.started":"2023-08-04T15:21:54.562086Z","shell.execute_reply":"2023-08-04T15:21:55.565672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# QWK Optimizer\nSee Abhishek optimizer [here](https://www.kaggle.com/abhishek/optimizer-for-quadratic-weighted-kappa)","metadata":{}},{"cell_type":"code","source":"class OptimizedRounder(object):\n    def __init__(self):\n        self.coef_ = 0\n\n    def _kappa_loss(self, coef, X, y):\n        X_p = np.copy(X)\n        for i, pred in enumerate(X_p):\n            if pred < coef[0]:\n                X_p[i] = 0\n            elif pred >= coef[0] and pred < coef[1]:\n                X_p[i] = 1\n            elif pred >= coef[1] and pred < coef[2]:\n                X_p[i] = 2\n            elif pred >= coef[2] and pred < coef[3]:\n                X_p[i] = 3\n            else:\n                X_p[i] = 4\n\n        ll = cohen_kappa_score(y, X_p, weights='quadratic')\n        return -ll\n\n    def fit(self, X, y):\n        loss_partial = partial(self._kappa_loss, X=X, y=y)\n        initial_coef = [0.5, 1.5, 2.5, 3.5]\n        self.coef_ = scipy.optimize.minimize(loss_partial, initial_coef, method='nelder-mead')\n        print(-loss_partial(self.coef_['x']))\n\n    def predict(self, X, coef):\n        X_p = np.copy(X)\n        for i, pred in enumerate(X_p):\n            if pred < coef[0]:\n                X_p[i] = 0\n            elif pred >= coef[0] and pred < coef[1]:\n                X_p[i] = 1\n            elif pred >= coef[1] and pred < coef[2]:\n                X_p[i] = 2\n            elif pred >= coef[2] and pred < coef[3]:\n                X_p[i] = 3\n            else:\n                X_p[i] = 4\n        return X_p\n\n    def coefficients(self):\n        return self.coef_['x']","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:23:05.274516Z","iopub.execute_input":"2023-08-04T15:23:05.274879Z","iopub.status.idle":"2023-08-04T15:23:05.292015Z","shell.execute_reply.started":"2023-08-04T15:23:05.274821Z","shell.execute_reply":"2023-08-04T15:23:05.291032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from functools import partial","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:23:12.067357Z","iopub.execute_input":"2023-08-04T15:23:12.067795Z","iopub.status.idle":"2023-08-04T15:23:12.073404Z","shell.execute_reply.started":"2023-08-04T15:23:12.067736Z","shell.execute_reply":"2023-08-04T15:23:12.072520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.load_weights('model.h5')\ny_val_pred = model.predict(x_val)\n\noptR = OptimizedRounder()\noptR.fit(y_val_pred, y_val)\ncoefficients = optR.coefficients()\nprint(f'Coefficients: {coefficients}')\ny_val_pred = optR.predict(y_val_pred, coefficients)\n\nscore = cohen_kappa_score(y_val_pred, y_val, weights='quadratic')\n\nprint('Optimized Validation QWK score: {}'.format(score))\nprint('Not Optimized Validation QWK score: {}'.format(max(results.kappa)))","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:23:26.813303Z","iopub.execute_input":"2023-08-04T15:23:26.813668Z","iopub.status.idle":"2023-08-04T15:23:43.141774Z","shell.execute_reply.started":"2023-08-04T15:23:26.813601Z","shell.execute_reply":"2023-08-04T15:23:43.140675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Save final model","metadata":{}},{"cell_type":"code","source":"model.save('ResNet50_model.h5')","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:23:53.662239Z","iopub.execute_input":"2023-08-04T15:23:53.662572Z","iopub.status.idle":"2023-08-04T15:23:54.445394Z","shell.execute_reply.started":"2023-08-04T15:23:53.662495Z","shell.execute_reply":"2023-08-04T15:23:54.444510Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, roc_curve, auc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport statsmodels.api as sm\n\n# Load the model and predict on the validation data\nmodel.load_weights('model.h5')\ny_val_pred = model.predict(x_val)\n\n# Instantiate an OptimizedRounder object and fit on the validation data\noptR = OptimizedRounder()\noptR.fit(y_val_pred, y_val)\ncoefficients = optR.coefficients()\nprint(f'Coefficients: {coefficients}')\n\n# Get the predicted labels using the optimized coefficients\ny_val_pred_rounded = optR.predict(y_val_pred, coefficients)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:24:04.513088Z","iopub.execute_input":"2023-08-04T15:24:04.513426Z","iopub.status.idle":"2023-08-04T15:24:20.984104Z","shell.execute_reply.started":"2023-08-04T15:24:04.513368Z","shell.execute_reply":"2023-08-04T15:24:20.982476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compute confusion matrix\ncm = confusion_matrix(y_val, y_val_pred_rounded)\n\n# Plot confusion matrix\nfig, ax = plt.subplots(figsize=(8, 6))\nsns.heatmap(cm, annot=True, cmap='Blues', fmt='g', ax=ax)\n\n# Add labels, title, and axis ticks\nax.set_xlabel('Predicted labels')\nax.set_ylabel('True labels')\nax.set_title('Confusion Matrix')\nax.xaxis.set_ticklabels(['No-DR', 'Mild NPDR', 'Moderate NPDR', 'Severe NPDR', 'PDR'])\nax.yaxis.set_ticklabels(['No-DR', 'Mild NPDR', 'Moderate NPDR', 'Severe NPDR', 'PDR'])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:24:26.756970Z","iopub.execute_input":"2023-08-04T15:24:26.757302Z","iopub.status.idle":"2023-08-04T15:24:27.209427Z","shell.execute_reply.started":"2023-08-04T15:24:26.757244Z","shell.execute_reply":"2023-08-04T15:24:27.208554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, classification_report\n\n# assuming y_val and y_val_pred_rounded are the true and predicted labels respectively\ncm = confusion_matrix(y_val, y_val_pred_rounded)\n\n# Calculate precision, recall, and accuracy\nreport = classification_report(y_val, y_val_pred_rounded, target_names=['No-DR', 'Mild NPDR', 'Moderate NPDR', 'Severe NPDR', 'PDR'])\n\n# Print confusion matrix, precision, recall, and accuracy\nprint('Confusion Matrix:\\n', cm)\nprint('Classification Report:\\n', report)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:24:37.864096Z","iopub.execute_input":"2023-08-04T15:24:37.864430Z","iopub.status.idle":"2023-08-04T15:24:37.902258Z","shell.execute_reply.started":"2023-08-04T15:24:37.864373Z","shell.execute_reply":"2023-08-04T15:24:37.901301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate Quadratic Weighted Kappa (QWK)\nqwk = cohen_kappa_score(y_val, y_val_pred_rounded, weights='quadratic')\n# Calculate accuracy for binary classification\nacc = accuracy_score(y_val, y_val_pred_rounded)\n\n# Print QWK score\nprint('Quadratic Weighted Kappa (QWK) for multiple classification:', qwk)\n# Print accuracy\nprint('Accuracy for multiple classification:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:36:10.195400Z","iopub.execute_input":"2023-08-04T15:36:10.195742Z","iopub.status.idle":"2023-08-04T15:36:10.223617Z","shell.execute_reply.started":"2023-08-04T15:36:10.195684Z","shell.execute_reply":"2023-08-04T15:36:10.222694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Map 0 to 'No DR' and 1,2,3,4 to 'DR' in y_true\ny_true_binary = np.where(y_val == 0, 0, 1)\ny_val_pred_rounded_binary = np.where(y_val_pred_rounded == 0, 0, 1)\n\n# Compute ROC curve and AUC\nfpr, tpr, thresholds = roc_curve(y_true_binary, y_val_pred_rounded_binary, pos_label=1)\nroc_auc = auc(fpr, tpr)\n\nplt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('Receiver operating characteristic curve')\nplt.legend(loc=\"lower right\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:24:46.351106Z","iopub.execute_input":"2023-08-04T15:24:46.354588Z","iopub.status.idle":"2023-08-04T15:24:46.723099Z","shell.execute_reply.started":"2023-08-04T15:24:46.351469Z","shell.execute_reply":"2023-08-04T15:24:46.722051Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate Quadratic Weighted Kappa (QWK)\nqwk = cohen_kappa_score(y_true_binary, y_val_pred_rounded_binary, weights='quadratic')\n# Calculate accuracy for binary classification\nacc = accuracy_score(y_true_binary, y_val_pred_rounded_binary)\n\n# Print QWK score\nprint('Quadratic Weighted Kappa (QWK) for binary classification:', qwk)\n# Print accuracy\nprint('Accuracy for binary classification:', acc)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:32:56.124437Z","iopub.execute_input":"2023-08-04T15:32:56.124815Z","iopub.status.idle":"2023-08-04T15:32:56.140875Z","shell.execute_reply.started":"2023-08-04T15:32:56.124754Z","shell.execute_reply":"2023-08-04T15:32:56.140158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compute confusion matrix\ncm = confusion_matrix(y_true_binary, y_val_pred_rounded_binary)\n\n# Calculate specificity and sensitivity\ntn, fp, fn, tp = cm.ravel()\nspecificity = tn / (tn + fp)\nsensitivity = tp / (tp + fn)\n\n# Display confusion matrix with percentages\nfig, ax = plt.subplots(figsize=(8, 6))\nsns.heatmap(cm, annot=True, cmap='Blues', fmt='g', ax=ax)\n\n# Add labels, title, and axis ticks\nax.set_xlabel('Predicted labels')\nax.set_ylabel('True labels')\nax.set_title('Confusion Matrix (Specificity={:.2f}, Sensitivity={:.2f})'.format(specificity, sensitivity))\nax.xaxis.set_ticklabels(['No-DR', 'DR'])\nax.yaxis.set_ticklabels(['No-DR', 'DR'])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:24:57.099365Z","iopub.execute_input":"2023-08-04T15:24:57.099704Z","iopub.status.idle":"2023-08-04T15:24:57.407793Z","shell.execute_reply.started":"2023-08-04T15:24:57.099645Z","shell.execute_reply":"2023-08-04T15:24:57.406676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"report_binary = classification_report(y_true_binary, y_val_pred_rounded_binary, target_names=['No-DR', 'DR'])\nprint('Classification Report:\\n', report_binary)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:25:02.130926Z","iopub.execute_input":"2023-08-04T15:25:02.131260Z","iopub.status.idle":"2023-08-04T15:25:02.148551Z","shell.execute_reply.started":"2023-08-04T15:25:02.131201Z","shell.execute_reply":"2023-08-04T15:25:02.147526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluation on Test Dataset (Messidor)","metadata":{}},{"cell_type":"code","source":"# Preprocess and store test images\nN_test = test_df.shape[0]\nx_test = np.empty((N_test, im_size, im_size, 3), dtype=np.uint8)\n\nfor i, image_id in enumerate(tqdm_notebook(test_df['id_code'])):\n    x_test[i, :, :, :] = preprocess_image(\n        f'{image_id}',\n        desired_size=im_size\n    )\n\n# Save the preprocessed test images to working space\nnp.save('/kaggle/working/preprocessed_test_images.npy', x_test)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:25:17.452441Z","iopub.execute_input":"2023-08-04T15:25:17.452817Z","iopub.status.idle":"2023-08-04T15:26:05.770499Z","shell.execute_reply.started":"2023-08-04T15:25:17.452755Z","shell.execute_reply":"2023-08-04T15:26:05.769582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load preprocessed test images\nx_test = np.load('/kaggle/working/preprocessed_test_images.npy')\n\n# Make predictions on the preprocessed test images\ny_pred = model.predict(x_test)\n\n# Perform any necessary post-processing on the predictions if needed\n\n# Evaluate the model's performance on the test set\ntest_loss, test_mae = model.evaluate(x_test, y_test)  # Replace y_test with your actual test labels\n\n# Print or display the evaluation results\nprint(\"Test Loss:\", test_loss)\nprint(\"Test Mean Absolute Error:\", test_mae)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:26:12.841514Z","iopub.execute_input":"2023-08-04T15:26:12.841870Z","iopub.status.idle":"2023-08-04T15:26:21.452864Z","shell.execute_reply.started":"2023-08-04T15:26:12.841814Z","shell.execute_reply":"2023-08-04T15:26:21.451902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import confusion_matrix, roc_curve, auc\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport statsmodels.api as sm\n\n# Load the model and predict on the test data\nmodel.load_weights('model.h5')\ny_test_pred = model.predict(x_test)\n\n# Instantiate an OptimizedRounder object and fit on the test data predictions\noptR = OptimizedRounder()\noptR.fit(y_test_pred, y_test)  # Assuming y_test is your ground truth test labels\ncoefficients = optR.coefficients()\nprint(f'Coefficients: {coefficients}')\n\n# Get the predicted labels using the optimized coefficients\ny_test_pred_rounded = optR.predict(y_test_pred, coefficients)\n\n# Compute confusion matrix\ncm = confusion_matrix(y_test, y_test_pred_rounded)\n\n# Plot confusion matrix\nfig, ax = plt.subplots(figsize=(8, 6))\nsns.heatmap(cm, annot=True, cmap='Blues', fmt='g', ax=ax)\n\n# Add labels, title, and axis ticks\nax.set_xlabel('Predicted labels')\nax.set_ylabel('True labels')\nax.set_title('Confusion Matrix')\nax.xaxis.set_ticklabels(['No-DR', 'Mild NPDR', 'Moderate NPDR', 'Severe NPDR', 'PDR'])\nax.yaxis.set_ticklabels(['No-DR', 'Mild NPDR', 'Moderate NPDR', 'Severe NPDR', 'PDR'])\n\n# Show the plot\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:26:33.301181Z","iopub.execute_input":"2023-08-04T15:26:33.301518Z","iopub.status.idle":"2023-08-04T15:26:40.287562Z","shell.execute_reply.started":"2023-08-04T15:26:33.301452Z","shell.execute_reply":"2023-08-04T15:26:40.286473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import classification_report\n\n# Calculate precision, recall, and accuracy\nreport = classification_report(y_test, y_test_pred_rounded, target_names=['No-DR', 'Mild NPDR', 'Moderate NPDR', 'Severe NPDR', 'PDR'])\n\n# Print confusion matrix, precision, recall, and accuracy\nprint('Confusion Matrix:\\n', cm)\nprint('Classification Report:\\n', report)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:26:50.117608Z","iopub.execute_input":"2023-08-04T15:26:50.117939Z","iopub.status.idle":"2023-08-04T15:26:50.134567Z","shell.execute_reply.started":"2023-08-04T15:26:50.117883Z","shell.execute_reply":"2023-08-04T15:26:50.133690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate Quadratic Weighted Kappa (QWK)\nqwk = cohen_kappa_score(y_test, y_test_pred_rounded, weights='quadratic')\n# Calculate accuracy for binary classification\nacc = accuracy_score(y_test, y_test_pred_rounded)\n\n# Print QWK score\nprint('Quadratic Weighted Kappa (QWK) for multiple classification:', qwk)\n# Print accuracy\nprint('Accuracy for multiple classification:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:46:35.695786Z","iopub.execute_input":"2023-08-04T15:46:35.696127Z","iopub.status.idle":"2023-08-04T15:46:35.717508Z","shell.execute_reply.started":"2023-08-04T15:46:35.696068Z","shell.execute_reply":"2023-08-04T15:46:35.716585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Map 0 to 'No DR' and 1, 2, 3, 4 to 'DR' in y_test\ny_test_binary = np.where(y_test == 0, 0, 1)\ny_test_pred_rounded_binary = np.where(y_test_pred_rounded == 0, 0, 1)\n\n# Compute ROC curve and AUC\nfpr, tpr, thresholds = roc_curve(y_test_binary, y_test_pred_rounded_binary, pos_label=1)\nroc_auc = auc(fpr, tpr)\n\nplt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)\nplt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\nplt.xlim([0.0, 1.0])\nplt.ylim([0.0, 1.05])\nplt.xlabel('False Positive Rate')\nplt.ylabel('True Positive Rate')\nplt.title('Receiver Operating Characteristic (ROC) Curve')\nplt.legend(loc=\"lower right\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:29:17.913205Z","iopub.execute_input":"2023-08-04T15:29:17.913555Z","iopub.status.idle":"2023-08-04T15:29:18.174580Z","shell.execute_reply.started":"2023-08-04T15:29:17.913486Z","shell.execute_reply":"2023-08-04T15:29:18.173607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compute confusion matrix\ncm = confusion_matrix(y_test_binary, y_test_pred_rounded_binary)\n\n# Calculate specificity and sensitivity\ntn, fp, fn, tp = cm.ravel()\nspecificity = tn / (tn + fp)\nsensitivity = tp / (tp + fn)\n\n# Display confusion matrix with percentages\nfig, ax = plt.subplots(figsize=(8, 6))\nsns.heatmap(cm, annot=True, cmap='Blues', fmt='g', ax=ax)\n\n# Add labels, title, and axis ticks\nax.set_xlabel('Predicted labels')\nax.set_ylabel('True labels')\nax.set_title('Confusion Matrix (Specificity={:.2f}, Sensitivity={:.2f})'.format(specificity, sensitivity))\nax.xaxis.set_ticklabels(['No-DR', 'DR'])\nax.yaxis.set_ticklabels(['No-DR', 'DR'])\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:38:12.243985Z","iopub.execute_input":"2023-08-04T15:38:12.244348Z","iopub.status.idle":"2023-08-04T15:38:12.555950Z","shell.execute_reply.started":"2023-08-04T15:38:12.244285Z","shell.execute_reply":"2023-08-04T15:38:12.555052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate Quadratic Weighted Kappa (QWK)\nqwk = cohen_kappa_score(y_test_binary, y_test_pred_rounded_binary, weights='quadratic')\n# Calculate accuracy for binary classification\nacc = accuracy_score(y_test_binary, y_test_pred_rounded_binary)\n\n# Print QWK score\nprint('Quadratic Weighted Kappa (QWK) for binary classification:', qwk)\n# Print accuracy\nprint('Accuracy for binary classification:', acc)","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:31:29.733805Z","iopub.execute_input":"2023-08-04T15:31:29.734161Z","iopub.status.idle":"2023-08-04T15:31:29.746933Z","shell.execute_reply.started":"2023-08-04T15:31:29.734099Z","shell.execute_reply":"2023-08-04T15:31:29.745829Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Calculate precision, recall, and accuracy\nreport = classification_report(y_test_binary, y_test_pred_rounded_binary, target_names=['No-DR', 'DR'])\n\n# Print confusion matrix, precision, recall, and accuracy\nprint('Confusion Matrix:\\n', cm)\nprint('Classification Report:\\n', report)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-04T15:40:04.542022Z","iopub.execute_input":"2023-08-04T15:40:04.542435Z","iopub.status.idle":"2023-08-04T15:40:04.560147Z","shell.execute_reply.started":"2023-08-04T15:40:04.542365Z","shell.execute_reply":"2023-08-04T15:40:04.558978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Fine Tuning for Messidor","metadata":{}}]}