{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# About this kernel\n\nIn this kernel, we will explore the complete workflow for the APTOS 2019 competition. We will go through:\n\n1. Loading & Exploration: A quick overview of the dataset\n2. Resize Images: We will resize both the training and test images to 224x224, so that it matches the ImageNet format.\n3. Mixup & Data Generator: We show how to create a data generator that will perform random transformation to our datasets (flip vertically/horizontally, rotation, zooming). This will help our model generalize better to the data, since it is fairly small (only ~3000 images).\n4. Quadratic Weighted Kappa: A thorough overview of the metric used for this competition, with an intuitive example. Check it out!\n5. Model: We will use a DenseNet-121 pre-trained on ImageNet. We will finetune it using Adam for 15 epochs, and evaluate it on an unseen validation set.\n6. Training & Evaluation: We take a look at the change in loss and QWK score through the epochs.\n\n### Unused Methods\n\nThroughout V15-V18 of this kernel, I ablated a few methods that I presented in this kernel. The highest LB score was achieved after I removed:\n* Mixup\n* Optimized Threshold\n\nI decided to keep them in the kernel if it ever becomes useful for you.\n\n### Citations & Resources\n\n* I had the idea of using mixup from [KeepLearning's ResNet50 baseline](https://www.kaggle.com/mathormad/aptos-resnet50-baseline). Since the implementation was in PyTorch, I instead used an [open-sourced keras implementation](https://github.com/yu4u/mixup-generator).\n* The transfer learning procedure is mostly inspired from my [previous kernel for iWildCam](https://www.kaggle.com/xhlulu/densenet-transfer-learning-iwildcam-2019). The workflow was however heavily modified since then.\n* Used similar [method as Abhishek](https://www.kaggle.com/abhishek/optimizer-for-quadratic-weighted-kappa) to find the optimal threshold.\n* [Lex's kernel](https://www.kaggle.com/lextoumbourou/blindness-detection-resnet34-ordinal-targets) prompted me to try using Multilabel instead of multiclass classification, which slightly improved the kappa score.","metadata":{}},{"cell_type":"code","source":"import json\nimport math\nimport os\n\nimport cv2\nfrom PIL import Image\nimport numpy as np\nfrom keras import layers\nfrom keras.applications import DenseNet121\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\nimport matplotlib.pyplot as plt\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score\nimport scipy\nimport tensorflow as tf\nfrom tqdm import tqdm\nimport keras\nfrom keras.datasets import mnist\n\nimport datetime\n\n%matplotlib inline","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.status.busy":"2022-06-27T21:28:08.990825Z","iopub.execute_input":"2022-06-27T21:28:08.991140Z","iopub.status.idle":"2022-06-27T21:28:09.003380Z","shell.execute_reply.started":"2022-06-27T21:28:08.991087Z","shell.execute_reply":"2022-06-27T21:28:09.002616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Set random seed for reproducibility.","metadata":{}},{"cell_type":"code","source":"np.random.seed(2019)\ntf.set_random_seed(2019)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:28:09.102788Z","iopub.execute_input":"2022-06-27T21:28:09.103536Z","iopub.status.idle":"2022-06-27T21:28:09.108205Z","shell.execute_reply.started":"2022-06-27T21:28:09.103095Z","shell.execute_reply":"2022-06-27T21:28:09.107467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading & Exploration","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\nprint(train_df.shape)\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:28:09.110052Z","iopub.execute_input":"2022-06-27T21:28:09.110714Z","iopub.status.idle":"2022-06-27T21:28:09.137642Z","shell.execute_reply.started":"2022-06-27T21:28:09.110483Z","shell.execute_reply":"2022-06-27T21:28:09.136846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df['diagnosis'].hist()\ntrain_df['diagnosis'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:28:09.139163Z","iopub.execute_input":"2022-06-27T21:28:09.139413Z","iopub.status.idle":"2022-06-27T21:28:09.332089Z","shell.execute_reply.started":"2022-06-27T21:28:09.139353Z","shell.execute_reply":"2022-06-27T21:28:09.331429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Displaying some Sample Images","metadata":{}},{"cell_type":"code","source":"def display_samples(df, columns=4, rows=3):\n    fig=plt.figure(figsize=(5*columns, 4*rows))\n\n    for i in range(columns*rows):\n        image_path = df.loc[i,'id_code']\n        image_id = df.loc[i,'diagnosis']\n        img = cv2.imread(f'../input/aptos2019-blindness-detection/train_images/{image_path}.png')\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        fig.add_subplot(rows, columns, i+1)\n        plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(train_df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-27T21:28:09.334067Z","iopub.execute_input":"2022-06-27T21:28:09.334604Z","iopub.status.idle":"2022-06-27T21:28:17.601846Z","shell.execute_reply.started":"2022-06-27T21:28:09.334329Z","shell.execute_reply":"2022-06-27T21:28:17.600998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Resize Images\n\nWe will resize the images to 224x224, then create a single numpy array to hold the data.","metadata":{}},{"cell_type":"code","source":"def load_ben_color(path, sigmaX=10):\n    image = cv2.imread(path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    image = cv2.resize(image, (224, 224))\n    image=cv2.addWeighted ( image,4, cv2.GaussianBlur( image , (0,0) , sigmaX) ,-4 ,128)\n    return image\n\ndef preprocess_image(image_path, desired_size=224):\n    # Add Lighting, to improve quality\n    im = load_ben_color(image_path)\n    return im\n    \n\n# Trail-1 Under sampling by deleting oversized classes (Class:0)\ndef under_sample_make_all_same(df, categories, max_per_category):\n    df = pd.concat([df[df['diagnosis'] == c][:max_per_category] for c in categories])\n    df = df.sample(n=(max_per_category)*len(categories), replace=False, random_state=20031976)\n    df.index = np.arange(len(df))\n    return df\n# train_df = under_sample_make_all_same(train_df,[0,1,2,3,4], 193 ) \n#Under-sample class-0 (1805-805=1000) and Over-sample other classes so each class has 1000 entries\nprint('Train DF Shape:',train_df.shape)\ntrain_df = train_df.drop(train_df[train_df['diagnosis'] == 0].sample(n=805, replace=False).index)\n\nN = train_df.shape[0]\nx_train = np.empty((N, 224, 224, 3), dtype=np.uint8)\n#tqdm\nfor i, image_id in enumerate((train_df['id_code'])):\n    x_train[i, :, :, :] = preprocess_image(\n        f'/kaggle/input/aptos2019-blindness-detection/train_images/{image_id}.png'\n    )\n#     print('Preprocessing Image:',i)\ny_train = pd.get_dummies(train_df['diagnosis']).values","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:28:17.603095Z","iopub.execute_input":"2022-06-27T21:28:17.603501Z","iopub.status.idle":"2022-06-27T21:34:29.452699Z","shell.execute_reply.started":"2022-06-27T21:28:17.603456Z","shell.execute_reply":"2022-06-27T21:34:29.451982Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# View the sample pre-processed images here\ndef display_single_image(img):\n    fig=plt.figure(figsize=(10, 10))\n    plt.title('Sample Img')\n    plt.imshow(img)\n\n# Training Images\nprint('X Train Shape:', x_train.shape)\ndisplay_single_image(x_train[0])\ndisplay_single_image(x_train[1])\n\n\n# Testing Images\n#print('X Test Shape:', x_test.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:34:29.456441Z","iopub.execute_input":"2022-06-27T21:34:29.456709Z","iopub.status.idle":"2022-06-27T21:34:29.979115Z","shell.execute_reply.started":"2022-06-27T21:34:29.456659Z","shell.execute_reply":"2022-06-27T21:34:29.978394Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's crop the images, to their region of interest\n# Credits to https://www.kaggle.com/taindow/pre-processing-train-and-test-images\ndef crop_image_from_gray(img,tol=7):\n    \n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        mask = gray_img>tol        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0):\n            return img\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n            img = np.stack([img1,img2,img3],axis=-1)\n        return img\n\ndef circle_crop(img):   \n    height, width, depth = img.shape    \n    \n    x = int(width/2)\n    y = int(height/2)\n    r = np.amin((x,y))\n    \n    circle_img = np.zeros((height, width), np.uint8)\n    cv2.circle(circle_img, (x,y), int(r), 1, thickness=-1)\n    img = cv2.bitwise_and(img, img, mask=circle_img)\n    img = crop_image_from_gray(img)\n    \n    return img \n\ncircle_crop_img = circle_crop(x_train[1])\n\ndisplay_single_image(circle_crop_img)\n\n# This has some work, to do. Not continuing any more :)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:34:29.981529Z","iopub.execute_input":"2022-06-27T21:34:29.982066Z","iopub.status.idle":"2022-06-27T21:34:30.242155Z","shell.execute_reply.started":"2022-06-27T21:34:29.982012Z","shell.execute_reply":"2022-06-27T21:34:30.241368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"x_train.shape=\",x_train.shape)\nprint(\"y_train.shape=\",y_train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:34:30.243660Z","iopub.execute_input":"2022-06-27T21:34:30.244189Z","iopub.status.idle":"2022-06-27T21:34:30.250369Z","shell.execute_reply.started":"2022-06-27T21:34:30.244136Z","shell.execute_reply":"2022-06-27T21:34:30.249492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"SEED=2019","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:34:30.251965Z","iopub.execute_input":"2022-06-27T21:34:30.252540Z","iopub.status.idle":"2022-06-27T21:34:30.257772Z","shell.execute_reply.started":"2022-06-27T21:34:30.252484Z","shell.execute_reply":"2022-06-27T21:34:30.256668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Trail-2 Over sampling by increasing undersized classes\nfrom imblearn.over_sampling import SMOTE, ADASYN\nx_resampled, y_resampled = SMOTE(random_state=SEED).fit_sample(x_train.reshape(x_train.shape[0], -1), train_df['diagnosis'].ravel())\n\nprint(\"x_resampled.shape=\",x_resampled.shape)\nprint(\"y_resampled.shape=\",y_resampled.shape)\n\nx_train = x_resampled.reshape(x_resampled.shape[0], 224, 224, 3)\ny_train = pd.get_dummies(y_resampled).values\n\nprint(\"x_train.shape=\",x_train.shape)\nprint(\"y_train.shape=\",y_train.shape)","metadata":{"jupyter":{"source_hidden":true},"execution":{"iopub.status.busy":"2022-06-27T21:34:30.259547Z","iopub.execute_input":"2022-06-27T21:34:30.260267Z","iopub.status.idle":"2022-06-27T21:41:03.908717Z","shell.execute_reply.started":"2022-06-27T21:34:30.260201Z","shell.execute_reply":"2022-06-27T21:41:03.907717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train_multi = np.empty(y_train.shape, dtype=y_train.dtype)\ny_train_multi[:, 4] = y_train[:, 4]\n\nfor i in range(3, -1, -1):\n    y_train_multi[:, i] = np.logical_or(y_train[:, i], y_train_multi[:, i+1])\n\nprint(\"Original y_train:\", y_train.sum(axis=0))\nprint(\"Multilabel version:\", y_train_multi.sum(axis=0))","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:03.910657Z","iopub.execute_input":"2022-06-27T21:41:03.910952Z","iopub.status.idle":"2022-06-27T21:41:03.922546Z","shell.execute_reply.started":"2022-06-27T21:41:03.910904Z","shell.execute_reply":"2022-06-27T21:41:03.921680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Creating multilabels\n\nInstead of predicting a single label, we will change our target to be a multilabel problem; i.e., if the target is a certain class, then it encompasses all the classes before it. E.g. encoding a class 4 retinopathy would usually be `[0, 0, 0, 1]`, but in our case we will predict `[1, 1, 1, 1]`. For more details, please check out [Lex's kernel](https://www.kaggle.com/lextoumbourou/blindness-detection-resnet34-ordinal-targets).","metadata":{}},{"cell_type":"code","source":"#y_train_multi = np.empty(y_train.shape, dtype=y_train.dtype)\n#y_train_multi[:, 4] = y_train[:, 4]\n\n#for i in range(3, -1, -1):\n #   y_train_multi[:, i] = np.logical_or(y_train[:, i], y_train_multi[:, i+1])\n\n#print(\"Original y_train:\", y_train.sum(axis=0))\n#print(\"Multilabel version:\", y_train_multi.sum(axis=0))","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:03.924164Z","iopub.execute_input":"2022-06-27T21:41:03.924875Z","iopub.status.idle":"2022-06-27T21:41:03.934550Z","shell.execute_reply.started":"2022-06-27T21:41:03.924782Z","shell.execute_reply":"2022-06-27T21:41:03.933612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can split it into a training and validation set.","metadata":{}},{"cell_type":"code","source":"x_train_val, x_test, y_train_val, y_test = train_test_split(\n    x_train, y_train_multi, \n    test_size=0.3, \n    random_state=2019\n)\nx_train, x_val, y_train, y_val = train_test_split(\n    x_train_val, y_train_val, \n    test_size=0.2, \n    random_state=2019\n)\nprint(\"shape of train images is :\",x_train.shape)\nprint(\"shape of test images is :\",x_test.shape)\nprint(\"shape of valid images is :\",x_val.shape)\nprint(\"shape of train labels is :\",y_train.shape)\nprint(\"shape of test labels is :\",y_test.shape)\nprint(\"shape of valid labels is :\",y_val.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:03.936335Z","iopub.execute_input":"2022-06-27T21:41:03.937113Z","iopub.status.idle":"2022-06-27T21:41:04.744183Z","shell.execute_reply.started":"2022-06-27T21:41:03.936795Z","shell.execute_reply":"2022-06-27T21:41:04.743304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Mixup & Data Generator\n\nPlease Note: Although I show how to construct Mixup, **it is currently unused**. Please see notice at the top of the kernel.","metadata":{}},{"cell_type":"code","source":"class MixupGenerator():\n    def __init__(self, X_train, y_train, batch_size=32, alpha=0.2, shuffle=True, datagen=None):\n        self.X_train = X_train\n        self.y_train = y_train\n        self.batch_size = batch_size\n        self.alpha = alpha\n        self.shuffle = shuffle\n        self.sample_num = len(X_train)\n        self.datagen = datagen\n\n    def __call__(self):\n        while True:\n            indexes = self.__get_exploration_order()\n            itr_num = int(len(indexes) // (self.batch_size * 2))\n\n            for i in range(itr_num):\n                batch_ids = indexes[i * self.batch_size * 2:(i + 1) * self.batch_size * 2]\n                X, y = self.__data_generation(batch_ids)\n\n                yield X, y\n\n    def __get_exploration_order(self):\n        indexes = np.arange(self.sample_num)\n\n        if self.shuffle:\n            np.random.shuffle(indexes)\n\n        return indexes\n\n    def __data_generation(self, batch_ids):\n        _, h, w, c = self.X_train.shape\n        l = np.random.beta(self.alpha, self.alpha, self.batch_size)\n        X_l = l.reshape(self.batch_size, 1, 1, 1)\n        y_l = l.reshape(self.batch_size, 1)\n\n        X1 = self.X_train[batch_ids[:self.batch_size]]\n        X2 = self.X_train[batch_ids[self.batch_size:]]\n        X = X1 * X_l + X2 * (1 - X_l)\n\n        if self.datagen:\n            for i in range(self.batch_size):\n                X[i] = self.datagen.random_transform(X[i])\n                X[i] = self.datagen.standardize(X[i])\n\n        if isinstance(self.y_train, list):\n            y = []\n\n            for y_train_ in self.y_train:\n                y1 = y_train_[batch_ids[:self.batch_size]]\n                y2 = y_train_[batch_ids[self.batch_size:]]\n                y.append(y1 * y_l + y2 * (1 - y_l))\n        else:\n            y1 = self.y_train[batch_ids[:self.batch_size]]\n            y2 = self.y_train[batch_ids[self.batch_size:]]\n            y = y1 * y_l + y2 * (1 - y_l)\n\n        return X, y","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:04.746432Z","iopub.execute_input":"2022-06-27T21:41:04.746906Z","iopub.status.idle":"2022-06-27T21:41:04.766251Z","shell.execute_reply.started":"2022-06-27T21:41:04.746801Z","shell.execute_reply":"2022-06-27T21:41:04.765472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BATCH_SIZE = 32\n\ndef create_datagen():\n    return ImageDataGenerator(\n        zoom_range=0.15,  # set range for random zoom\n        # set mode for filling points outside the input boundaries\n        fill_mode='constant',\n        cval=0.,  # value used for fill_mode = \"constant\"\n        horizontal_flip=True,  # randomly flip images\n        vertical_flip=True,  # randomly flip images\n    )\n\n# Using original generator\ndata_generator = create_datagen().flow(x_train, y_train, batch_size=BATCH_SIZE, seed=2019)\n# Using Mixup\nmixup_generator = MixupGenerator(x_train, y_train, batch_size=BATCH_SIZE, alpha=0.2, datagen=create_datagen())()","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:04.767986Z","iopub.execute_input":"2022-06-27T21:41:04.768846Z","iopub.status.idle":"2022-06-27T21:41:05.792067Z","shell.execute_reply.started":"2022-06-27T21:41:04.768323Z","shell.execute_reply":"2022-06-27T21:41:05.791200Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Quadratic Weighted Kappa\n\nQuadratic Weighted Kappa (QWK, the greek letter $\\kappa$), also known as Cohen's Kappa, is the official evaluation metric. For our kernel, we will use a custom callback to monitor the score, and plot it at the end.\n\n### What is Cohen Kappa?\n\nAccording to the [wikipedia article](https://en.wikipedia.org/wiki/Cohen%27s_kappa), we have\n> The definition of $\\kappa$ is:\n> $$\\kappa \\equiv \\frac{p_o - p_e}{1 - p_e}$$\n> where $p_o$ is the relative observed agreement among raters (identical to accuracy), and $p_e$ is the hypothetical probability of chance agreement, using the observed data to calculate the probabilities of each observer randomly seeing each category.\n\n### How is it computed?\n\nLet's take the example of a binary classification problem. Say we have:","metadata":{}},{"cell_type":"code","source":"true_labels = np.array([1, 0, 1, 1, 0, 1])\npred_labels = np.array([1, 0, 0, 0, 0, 1])","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:05.793397Z","iopub.execute_input":"2022-06-27T21:41:05.793711Z","iopub.status.idle":"2022-06-27T21:41:05.799261Z","shell.execute_reply.started":"2022-06-27T21:41:05.793662Z","shell.execute_reply":"2022-06-27T21:41:05.798330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can construct the following table:\n\n| true | pred | agreement      |\n|------|------|----------------|\n| 1    | 1    | true positive  |\n| 0    | 0    | true negative  |\n| 1    | 0    | false negative |\n| 1    | 0    | false negative |\n| 0    | 0    | true negative  |\n| 1    | 1    | true positive  |\n\n\nThen the \"observed proportionate agreement\" is calculated exactly the same way as accuracy:\n\n$$\np_o = acc = \\frac{tp + tn}{all} = {2 + 2}{6} = 0.66\n$$\n\nThis can be confirmed using scikit-learn:","metadata":{}},{"cell_type":"code","source":"accuracy_score(true_labels, pred_labels)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:05.800624Z","iopub.execute_input":"2022-06-27T21:41:05.801165Z","iopub.status.idle":"2022-06-27T21:41:05.811302Z","shell.execute_reply.started":"2022-06-27T21:41:05.801113Z","shell.execute_reply":"2022-06-27T21:41:05.810304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Additionally, we also need to compute `p_e`:\n\n$$p_{yes} = \\frac{tp + fp}{all} \\frac{tp + fn}{all} = \\frac{2}{6} \\frac{4}{6} = 0.222$$\n\n$$p_{no} = \\frac{fn + tn}{all} \\frac{fp + tn}{all} = \\frac{4}{6} \\frac{2}{6} = 0.222$$\n\n$$p_{e} = p_{yes} + p_{no} = 0.222 + 0.222 = 0.444$$\n\nFinally,\n\n$$\n\\kappa = \\frac{p_o - p_e}{1-p_e} = \\frac{0.666 - 0.444}{1 - 0.444} = 0.4\n$$\n\nLet's verify with scikit-learn:","metadata":{}},{"cell_type":"code","source":"cohen_kappa_score(true_labels, pred_labels)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:05.812917Z","iopub.execute_input":"2022-06-27T21:41:05.813626Z","iopub.status.idle":"2022-06-27T21:41:05.825802Z","shell.execute_reply.started":"2022-06-27T21:41:05.813271Z","shell.execute_reply":"2022-06-27T21:41:05.824709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### What is the weighted kappa?\n\nThe wikipedia page offer a very concise explanation: \n> The weighted kappa allows disagreements to be weighted differently and is especially useful when **codes are ordered**. Three matrices are involved, the matrix of observed scores, the matrix of expected scores based on chance agreement, and the weight matrix. Weight matrix cells located on the diagonal (upper-left to bottom-right) represent agreement and thus contain zeros. Off-diagonal cells contain weights indicating the seriousness of that disagreement.\n\nSimply put, if two scores disagree, then the penalty will depend on how far they are apart. That means that our score will be higher if (a) the real value is 4 but the model predicts a 3, and the score will be lower if (b) the model instead predicts a 0. This metric makes sense for this competition, since the labels 0-4 indicates how severe the illness is. Intuitively, a model that predicts a severe retinopathy (3) when it is in reality a proliferative retinopathy (4) is probably better than a model that predicts a mild retinopathy (1).","metadata":{}},{"cell_type":"markdown","source":"### Creating keras callback for QWK","metadata":{}},{"cell_type":"code","source":"class Metrics(Callback):\n    def on_train_begin(self, logs={}):\n        self.val_kappas = []\n\n    def on_epoch_end(self, epoch, logs={}):\n        X_val, y_val = self.validation_data[:2]\n        y_val = y_val.sum(axis=1) - 1\n        \n        y_pred = self.model.predict(X_val) > 0.5\n        y_pred = y_pred.astype(int).sum(axis=1) - 1\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n        \n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n\n        return","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:05.827226Z","iopub.execute_input":"2022-06-27T21:41:05.827695Z","iopub.status.idle":"2022-06-27T21:41:05.837598Z","shell.execute_reply.started":"2022-06-27T21:41:05.827504Z","shell.execute_reply":"2022-06-27T21:41:05.836725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model: DenseNet-121","metadata":{}},{"cell_type":"code","source":"densenet = DenseNet121(\n    weights='../input/densenet-keras/DenseNet-BC-121-32-no-top.h5',\n    include_top=False,\n    input_shape=(224,224,3)\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:05.839093Z","iopub.execute_input":"2022-06-27T21:41:05.839570Z","iopub.status.idle":"2022-06-27T21:41:32.684072Z","shell.execute_reply.started":"2022-06-27T21:41:05.839517Z","shell.execute_reply":"2022-06-27T21:41:32.683338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model():\n    model = Sequential()\n    model.add(densenet)\n    model.add(layers.GlobalAveragePooling2D())\n    model.add(layers.Dropout(0.5))\n    model.add(layers.Dense(5, activation='sigmoid'))\n    \n    model.compile(\n        loss='binary_crossentropy',\n        optimizer=Adam(lr=0.00005),\n        metrics=['accuracy']\n    )\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:32.689662Z","iopub.execute_input":"2022-06-27T21:41:32.689952Z","iopub.status.idle":"2022-06-27T21:41:32.696350Z","shell.execute_reply.started":"2022-06-27T21:41:32.689896Z","shell.execute_reply":"2022-06-27T21:41:32.695244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = build_model()\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:32.698091Z","iopub.execute_input":"2022-06-27T21:41:32.698578Z","iopub.status.idle":"2022-06-27T21:41:44.560329Z","shell.execute_reply.started":"2022-06-27T21:41:32.698353Z","shell.execute_reply":"2022-06-27T21:41:44.559661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training & Evaluation","metadata":{}},{"cell_type":"code","source":"def subtime(date1, date2):\n    date1 = datetime.datetime.strptime(date1, \"%Y-%m-%d %H:%M:%S\")\n    date2 = datetime.datetime.strptime(date2, \"%Y-%m-%d %H:%M:%S\")\n    return date2 - date1\n","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:44.563310Z","iopub.execute_input":"2022-06-27T21:41:44.563566Z","iopub.status.idle":"2022-06-27T21:41:44.568993Z","shell.execute_reply.started":"2022-06-27T21:41:44.563519Z","shell.execute_reply":"2022-06-27T21:41:44.568006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"startdate = datetime.datetime.now() \nstartdate = startdate.strftime(\"%Y-%m-%d %H:%M:%S\") ","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:44.570667Z","iopub.execute_input":"2022-06-27T21:41:44.571488Z","iopub.status.idle":"2022-06-27T21:41:44.578655Z","shell.execute_reply.started":"2022-06-27T21:41:44.571324Z","shell.execute_reply":"2022-06-27T21:41:44.577512Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kappa_metrics = Metrics()\n\nhistory = model.fit_generator(\n    data_generator,\n    steps_per_epoch=x_train.shape[0] / BATCH_SIZE,\n    epochs=15,\n    validation_data=(x_val, y_val),\n    callbacks=[kappa_metrics]\n   \n)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:41:44.580204Z","iopub.execute_input":"2022-06-27T21:41:44.580742Z","iopub.status.idle":"2022-06-27T21:52:48.637055Z","shell.execute_reply.started":"2022-06-27T21:41:44.580581Z","shell.execute_reply":"2022-06-27T21:52:48.636153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"enddate = datetime.datetime.now() \nenddate = enddate.strftime(\"%Y-%m-%d %H:%M:%S\") ","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:52:48.641468Z","iopub.execute_input":"2022-06-27T21:52:48.643663Z","iopub.status.idle":"2022-06-27T21:52:48.648259Z","shell.execute_reply.started":"2022-06-27T21:52:48.643610Z","shell.execute_reply":"2022-06-27T21:52:48.646934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('start date ',startdate)\nprint('end date ',enddate)\nprint('Time ',subtime(startdate,enddate)) ","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:52:48.652223Z","iopub.execute_input":"2022-06-27T21:52:48.653517Z","iopub.status.idle":"2022-06-27T21:52:48.665608Z","shell.execute_reply.started":"2022-06-27T21:52:48.652916Z","shell.execute_reply":"2022-06-27T21:52:48.664724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with open('history.json', 'w') as f:\n    json.dump(history.history, f)\n\nhistory_df = pd.DataFrame(history.history)\nhistory_df[['loss', 'val_loss']].plot()\nhistory_df[['acc', 'val_acc']].plot()","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:52:48.667486Z","iopub.execute_input":"2022-06-27T21:52:48.672432Z","iopub.status.idle":"2022-06-27T21:52:49.196624Z","shell.execute_reply.started":"2022-06-27T21:52:48.672353Z","shell.execute_reply":"2022-06-27T21:52:49.195888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(kappa_metrics.val_kappas)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:52:49.198010Z","iopub.execute_input":"2022-06-27T21:52:49.198539Z","iopub.status.idle":"2022-06-27T21:52:49.352312Z","shell.execute_reply.started":"2022-06-27T21:52:49.198489Z","shell.execute_reply":"2022-06-27T21:52:49.351640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.load_weights('model.h5')\ny_val_pred = model.predict(x_val)\n\ndef compute_score_inv(threshold):\n    y1 = y_val_pred > threshold\n    y1 = y1.astype(int).sum(axis=1) - 1\n    y2 = y_val.sum(axis=1) - 1\n    score = cohen_kappa_score(y1, y2, weights='quadratic')\n    \n    return 1 - score\n\nsimplex = scipy.optimize.minimize(\n    compute_score_inv, 0.5, method='nelder-mead'\n)\n\nbest_threshold = simplex['x'][0]\nbest_threshold","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:54:11.550626Z","iopub.execute_input":"2022-06-27T21:54:11.550939Z","iopub.status.idle":"2022-06-27T21:54:13.793498Z","shell.execute_reply.started":"2022-06-27T21:54:11.550884Z","shell.execute_reply":"2022-06-27T21:54:13.791992Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.load_weights('model.h5')\nresults = model.evaluate(x_test,  y_test, verbose=1)\n\nprint(\"test loss:\",results[0])\nprint(\"test accuracy:\",results[1])\n","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:52:49.355596Z","iopub.execute_input":"2022-06-27T21:52:49.355827Z","iopub.status.idle":"2022-06-27T21:52:58.440579Z","shell.execute_reply.started":"2022-06-27T21:52:49.355783Z","shell.execute_reply":"2022-06-27T21:52:58.439952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = model.predict( x_test) > 0.5\ny_pred = y_pred.astype(int).sum(axis=1) - 1\ny_pred\n        ","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:52:58.443741Z","iopub.execute_input":"2022-06-27T21:52:58.443987Z","iopub.status.idle":"2022-06-27T21:53:02.111579Z","shell.execute_reply.started":"2022-06-27T21:52:58.443941Z","shell.execute_reply":"2022-06-27T21:53:02.110914Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_test = y_test.astype(int).sum(axis=1) - 1\ny_test","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:53:02.114781Z","iopub.execute_input":"2022-06-27T21:53:02.115025Z","iopub.status.idle":"2022-06-27T21:53:02.124156Z","shell.execute_reply.started":"2022-06-27T21:53:02.114979Z","shell.execute_reply":"2022-06-27T21:53:02.123368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn import metrics\nmetrics.confusion_matrix(y_test, y_pred)","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:53:02.125955Z","iopub.execute_input":"2022-06-27T21:53:02.126679Z","iopub.status.idle":"2022-06-27T21:53:02.137972Z","shell.execute_reply.started":"2022-06-27T21:53:02.126322Z","shell.execute_reply":"2022-06-27T21:53:02.137041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import (mean_squared_error,confusion_matrix, f1_score)\nfrom sklearn.metrics import classification_report\n\ntarget_names = ['class 0', 'class 1','class 2','class 3','class 4']\nprint(classification_report(y_test, y_pred, target_names=target_names))","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:53:02.139677Z","iopub.execute_input":"2022-06-27T21:53:02.140209Z","iopub.status.idle":"2022-06-27T21:53:02.155259Z","shell.execute_reply.started":"2022-06-27T21:53:02.139974Z","shell.execute_reply":"2022-06-27T21:53:02.154326Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\nprint(torch.version.cuda)\n\n!nvidia-smi ","metadata":{"execution":{"iopub.status.busy":"2022-06-27T21:53:02.156303Z","iopub.execute_input":"2022-06-27T21:53:02.156545Z","iopub.status.idle":"2022-06-27T21:53:03.018478Z","shell.execute_reply.started":"2022-06-27T21:53:02.156502Z","shell.execute_reply":"2022-06-27T21:53:03.017623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Find best threshold\n\nPlease Note: Although I show how to construct a threshold optimizer, **it is currently unused**. Please see notice at the top of the kernel.","metadata":{}},{"cell_type":"markdown","source":"## Submit","metadata":{}}]}