{"cells":[{"metadata":{},"cell_type":"markdown","source":"> # Introduction\n\n* This is a very basic kernel and is a great starter for anyone who is new to the world of deep learning. I myself am a starter and hence I personally think that this would be a great place for anyone who is new to this field to start from. I have used pre-trained weights on the ImageNet of the EfficientNet Architecture.\n\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Importing the libraries\n\nimport json\nimport math\nimport os\n\nimport cv2\nfrom PIL import Image\nimport numpy as np\nfrom keras import layers\nfrom keras.layers import Flatten\nfrom keras.applications import DenseNet121\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\nimport matplotlib.pyplot as plt\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score\nimport scipy\nimport tensorflow as tf\nfrom tqdm import tqdm\nimport glob\n\nIMG_SIZE = 224\nNUM_CLASSES = 5\nSEED = 77\nTRAIN_NUM = -1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input/efficientnet/efficientnet-master/efficientnet-master/efficientnet\"))\nimport sys\nsys.path.append(os.path.abspath('../input/efficientnet/efficientnet-master/efficientnet-master/'))\nfrom efficientnet import EfficientNetB5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Loading the dataframe\n\ntrain_df = pd.read_csv('../input/aptos2019-blindness-detection/train.csv') \ntest_df = pd.read_csv('../input/aptos2019-blindness-detection/test.csv')\nprint(train_df.shape)\nprint(test_df.shape)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Seeing the number of labels we have for different classes in our distribution\n\ntrain_df['diagnosis'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting the distribution of different labels we have\n\nsize_NoDR = 0\nsize_Mild = 0\nsize_Moderate = 0\nsize_Severe = 0\nsize_ProliferativeDR = 0\nfor i in range(len(train_df)):\n    if train_df['diagnosis'][i] == 0:\n        size_NoDR = size_NoDR + 1\n    if train_df['diagnosis'][i] == 1:\n        size_Mild = size_Mild + 1\n    if train_df['diagnosis'][i] == 2:\n        size_Moderate = size_Moderate + 1\n    if train_df['diagnosis'][i] == 3:\n        size_Severe = size_Severe + 1 \n    if train_df['diagnosis'][i] == 4:\n        size_ProliferativeDR = size_ProliferativeDR + 1\nexplode = [0.1, 0, 0, 0, 0]\nlabels = 'No DR', 'Mild','Moderate','Severe','Proliferative DR'        \nsizes = [size_NoDR, size_Mild, size_Moderate, size_Severe, size_ProliferativeDR]        \nfig1, ax1 = plt.subplots()\nax1.pie(sizes, explode = explode, labels=labels, autopct='%1.1f%%', shadow=True, startangle=90)\nax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us display some sample images to see how our input images are\n\ndef display_samples(df, columns = 4, rows = 3):\n    fig = plt.figure(figsize = (5 * columns, 4 * rows))\n\n    for i in range(columns * rows):\n        image_path = df.loc[i,'id_code']\n        image_id = df.loc[i,'diagnosis']\n        img = cv2.imread(f'../input/aptos2019-blindness-detection/train_images/{image_path}.png')\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        \n        fig.add_subplot(rows, columns, i+1)\n        plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_y = train_df['diagnosis']\ntrain_y.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> # Pre-Processing:\n\n* For pre-processing the images, I have used the method which has suggested by Ben Graham, who was the winner of the earlier version of this competition. This method turns out to be really good at bringing out the details of the images and also excluding the outer edges that don't really add much to out training data.\n* We also use DataGenerator function to generate certain random samples for training as we do not have much data for our training set and data generation through random processes like rotating etc mostly helps us in getting a better accuracy."},{"metadata":{"trusted":true},"cell_type":"code","source":"def crop_image_from_gray(img, tol = 7):\n    if img.ndim == 2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1), mask.any(0))]\n    elif img.ndim == 3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)\n        mask = gray_img>tol\n        \n        check_shape = img[:,:,0][np.ix_(mask.any(1), mask.any(0))].shape[0]\n        if (check_shape == 0): # image is too dark so that we crop out everything,\n            return img # return original image\n        else:\n            img1 = img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2 = img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3 = img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n    #         print(img1.shape,img2.shape,img3.shape)\n            img = np.stack([img1,img2,img3],axis=-1)\n    #         print(img.shape)\n        return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def circle_crop_v2(img):\n\n        #img = cv2.imread(img)\n        #img = crop_image_from_gray(img)\n\n        height, width, depth = img.shape\n        largest_side = np.max((height, width))\n        img = cv2.resize(img, (largest_side, largest_side))\n\n        height, width, depth = img.shape\n\n        x = int(width / 2)\n        y = int(height / 2)\n        r = np.amin((x, y))\n\n        circle_img = np.zeros((height, width), np.uint8)\n        cv2.circle(circle_img, (x, y), int(r), 1, thickness=-1)\n        img = cv2.bitwise_and(img, img, mask=circle_img)\n        #img = crop_image_from_gray(img)\n\n        return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":" def ben_color2(image_path, sigmaX = 10, scale = 270):\n    #image = cv2.imread(image_path)   \n    img = cv2.imread(image_path)\n    #gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n    #clahe = cv2.createCLAHE(clipLimit=1.0, tileGridSize=(10,10))\n    #gray_img = clahe.apply(gray_img)\n    #image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))\n    bgr = cv2.imread(image_path)\n\n    lab = cv2.cvtColor(bgr, cv2.COLOR_BGR2LAB)\n    \n    lab_planes = cv2.split(lab)\n\n    clahe = cv2.createCLAHE(clipLimit=2.0,tileGridSize=(10,10))\n\n    lab_planes[0] = clahe.apply(lab_planes[0])\n\n    lab = cv2.merge(lab_planes)\n\n    image = cv2.cvtColor(lab, cv2.COLOR_LAB2RGB)\n\n    x = image[image.shape[0]//2 ,: ,:].sum(1)\n    r = (x > x.mean()/10).sum()//2\n    s = scale * 1.0/ r\n    image = cv2.resize(image,(0,0), fx = s, fy = s)\n    #image = crop_image_from_gray(image)\n    #image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))\n    image = cv2.addWeighted (image,4, cv2.GaussianBlur(image, (0,0) ,sigmaX), -4, 128)\n    image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))\n    image = circle_crop_v2(image)\n    #image = cv2.fastNlMeansDenoisingColored(image,None,20,10,7,21)\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining the function to resize the images\n\ndef preprocess_image(image_path, desired_size = 300):\n    image = ben_color2(image_path,sigmaX= 10)\n    return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let us display some processed sample images to see how our input images are\n\ndef display_samples(df, columns = 4, rows = 3):\n    fig = plt.figure(figsize = (5 * columns, 4 * rows))\n\n    for i in range(columns * rows):\n        image_path = df.loc[i,'id_code']\n        image_id = df.loc[i,'diagnosis']\n        img = cv2.imread(f'../input/aptos2019-blindness-detection/train_images/{image_path}.png')\n        #img = load_ben_color(f'../input/aptos2019-blindness-detection/train_images/{image_path}.png', 10)\n        img = ben_color2(f'../input/aptos2019-blindness-detection/train_images/{image_path}.png', 10)\n       \n        fig.add_subplot(rows, columns, i + 1)\n        plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#### Processing the training images\n\nN = train_df.shape[0]\nx_train = np.empty((N, 224, 224, 3), dtype = np.uint8)\n\nfor i, image_id in enumerate(tqdm(train_df['id_code'])):\n    x_train[i, :, :, :] = preprocess_image(\n        f'../input/aptos2019-blindness-detection/train_images/{image_id}.png'\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Resizing the test images\n\nN = test_df.shape[0]\nx_test = np.empty((N, 224, 224, 3), dtype = np.uint8)\n\nfor i, image_id in enumerate(tqdm(test_df['id_code'])):\n    x_test[i, :, :, :] = preprocess_image(\n        f'../input/aptos2019-blindness-detection/test_images/{image_id}.png'\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Forming the target labels\n\ny_train = pd.get_dummies(train_df['diagnosis']).values\n\nprint(x_train.shape)\nprint(y_train.shape)\nprint(x_test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We use multi-label classification instead of multi-class classification because for this particular problem, multi-label classification approach has turned out to give much better results as compared to the multi-class classification approach. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Converting our target labels into multi-labels\n\ny_train_multi = np.empty(y_train.shape, dtype = y_train.dtype)\ny_train_multi[:, 4] = y_train[:, 4]\n\nfor i in range(3, -1, -1):\n    y_train_multi[:, i] = np.logical_or(y_train[:, i], y_train_multi[:, i+1])\n\nprint(\"Original y_train:\", y_train.sum(axis = 0))\nprint(\"Multilabel version:\", y_train_multi.sum(axis = 0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.utils import class_weight\nwts = class_weight.compute_class_weight('balanced', np.unique(train_y),train_y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Splitting our data into training and cross-validations sets\n\nx_train_NN, x_val_NN, y_train_NN, y_val_NN = train_test_split(x_train, y_train_multi, test_size = 0.15, random_state = 77) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining the data generator function\n\nBATCH_SIZE = 32\n\ndef create_datagen():\n            return ImageDataGenerator(\n               zoom_range = 0.15,  # set range for random zoom\n               rotation_range = 360,\n               # set mode for filling points outside the input boundaries\n               fill_mode = 'constant',\n               cval = 0, \n               horizontal_flip = True,  # randomly flip images\n               vertical_flip = True,  # randomly flip images\n               #rescale = 1 / 256\n              )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Using original generator\n\ndata_generator = create_datagen().flow(x_train_NN, y_train_NN, batch_size = BATCH_SIZE, seed = 77)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nimport keras\nclass RAdam(keras.optimizers.Optimizer):\n    \"\"\"RAdam optimizer.\n    # Arguments\n        lr: float >= 0. Learning rate.\n        beta_1: float, 0 < beta < 1. Generally close to 1.\n        beta_2: float, 0 < beta < 1. Generally close to 1.\n        epsilon: float >= 0. Fuzz factor. If `None`, defaults to `K.epsilon()`.\n        decay: float >= 0. Learning rate decay over each update.\n        weight_decay: float >= 0. Weight decay for each param.\n        amsgrad: boolean. Whether to apply the AMSGrad variant of this\n            algorithm from the paper \"On the Convergence of Adam and\n            Beyond\".\n        total_steps: int >= 0. Total number of training steps. Enable warmup by setting a positive value.\n        warmup_proportion: 0 < warmup_proportion < 1. The proportion of increasing steps.\n        min_lr: float >= 0. Minimum learning rate after warmup.\n    # References\n        - [Adam - A Method for Stochastic Optimization](https://arxiv.org/abs/1412.6980v8)\n        - [On the Convergence of Adam and Beyond](https://openreview.net/forum?id=ryQu7f-RZ)\n        - [On The Variance Of The Adaptive Learning Rate And Beyond](https://arxiv.org/pdf/1908.03265v1.pdf)\n    \"\"\"\n\n    def __init__(self, lr=0.001, beta_1=0.9, beta_2=0.999,\n                 epsilon=None, decay=0., weight_decay=0., amsgrad=False,\n                 total_steps=0, warmup_proportion=0.1, min_lr=0., **kwargs):\n        super(RAdam, self).__init__(**kwargs)\n        with K.name_scope(self.__class__.__name__):\n            self.iterations = K.variable(0, dtype='int64', name='iterations')\n            self.lr = K.variable(lr, name='lr')\n            self.beta_1 = K.variable(beta_1, name='beta_1')\n            self.beta_2 = K.variable(beta_2, name='beta_2')\n            self.decay = K.variable(decay, name='decay')\n            self.weight_decay = K.variable(weight_decay, name='weight_decay')\n            self.total_steps = K.variable(total_steps, name='total_steps')\n            self.warmup_proportion = K.variable(warmup_proportion, name='warmup_proportion')\n            self.min_lr = K.variable(lr, name='min_lr')\n        if epsilon is None:\n            epsilon = K.epsilon()\n        self.epsilon = epsilon\n        self.initial_decay = decay\n        self.initial_weight_decay = weight_decay\n        self.initial_total_steps = total_steps\n        self.amsgrad = amsgrad\n\n    def get_updates(self, loss, params):\n        grads = self.get_gradients(loss, params)\n        self.updates = [K.update_add(self.iterations, 1)]\n\n        lr = self.lr\n\n        if self.initial_decay > 0:\n            lr = lr * (1. / (1. + self.decay * K.cast(self.iterations, K.dtype(self.decay))))\n\n        t = K.cast(self.iterations, K.floatx()) + 1\n\n        if self.initial_total_steps > 0:\n            warmup_steps = self.total_steps * self.warmup_proportion\n            decay_steps = self.total_steps - warmup_steps\n            lr = K.switch(\n                t <= warmup_steps,\n                lr * (t / warmup_steps),\n                lr * (1.0 - K.minimum(t, decay_steps) / decay_steps),\n            )\n\n        ms = [K.zeros(K.int_shape(p), dtype=K.dtype(p), name='m_' + str(i)) for (i, p) in enumerate(params)]\n        vs = [K.zeros(K.int_shape(p), dtype=K.dtype(p), name='v_' + str(i)) for (i, p) in enumerate(params)]\n\n        if self.amsgrad:\n            vhats = [K.zeros(K.int_shape(p), dtype=K.dtype(p), name='vhat_' + str(i)) for (i, p) in enumerate(params)]\n        else:\n            vhats = [K.zeros(1, name='vhat_' + str(i)) for i in range(len(params))]\n\n        self.weights = [self.iterations] + ms + vs + vhats\n\n        beta_1_t = K.pow(self.beta_1, t)\n        beta_2_t = K.pow(self.beta_2, t)\n\n        sma_inf = 2.0 / (1.0 - self.beta_2) - 1.0\n        sma_t = sma_inf - 2.0 * t * beta_2_t / (1.0 - beta_2_t)\n\n        for p, g, m, v, vhat in zip(params, grads, ms, vs, vhats):\n            m_t = (self.beta_1 * m) + (1. - self.beta_1) * g\n            v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(g)\n\n            m_corr_t = m_t / (1.0 - beta_1_t)\n            if self.amsgrad:\n                vhat_t = K.maximum(vhat, v_t)\n                v_corr_t = K.sqrt(vhat_t / (1.0 - beta_2_t) + self.epsilon)\n                self.updates.append(K.update(vhat, vhat_t))\n            else:\n                v_corr_t = K.sqrt(v_t / (1.0 - beta_2_t) + self.epsilon)\n\n            r_t = K.sqrt((sma_t - 4.0) / (sma_inf - 4.0) *\n                         (sma_t - 2.0) / (sma_inf - 2.0) *\n                         sma_inf / sma_t)\n\n            p_t = K.switch(sma_t > 5, r_t * m_corr_t / v_corr_t, m_corr_t)\n\n            if self.initial_weight_decay > 0:\n                p_t += self.weight_decay * p\n\n            p_t = p - lr * p_t\n\n            self.updates.append(K.update(m, m_t))\n            self.updates.append(K.update(v, v_t))\n            new_p = p_t\n\n            # Apply constraints.\n            if getattr(p, 'constraint', None) is not None:\n                new_p = p.constraint(new_p)\n\n            self.updates.append(K.update(p, new_p))\n        return self.updates\n\n    def get_config(self):\n        config = {\n            'lr': float(K.get_value(self.lr)),\n            'beta_1': float(K.get_value(self.beta_1)),\n            'beta_2': float(K.get_value(self.beta_2)),\n            'decay': float(K.get_value(self.decay)),\n            'weight_decay': float(K.get_value(self.weight_decay)),\n            'epsilon': self.epsilon,\n            'amsgrad': self.amsgrad,\n            'total_steps': float(K.get_value(self.total_steps)),\n            'warmup_proportion': float(K.get_value(self.warmup_proportion)),\n            'min_lr': float(K.get_value(self.min_lr)),\n        }\n        base_config = super(RAdam, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from __future__ import absolute_import\nimport random\nimport numpy as np\nfrom keras.layers import *\nimport tensorflow as tf\n\nclass FractionalPooling2D(Layer):\n\tdef __init__(self, pool_ratio = None, pseudo_random = True, overlap = False, name ='FractionPooling2D', **kwargs):\n\t\tself.pool_ratio = pool_ratio\n\t\tself.input_spec = [InputSpec(ndim=4)]\n\t\tself.pseudo_random = pseudo_random\n\t\tself.overlap = overlap\n\t\tself.name = name\n\t\tsuper(FractionalPooling2D, self).__init__(**kwargs)\n\t\t\n\tdef call(self, input):\n\t\t[batch_tensor,row_pooling,col_pooling] = tf.nn.fractional_max_pool(input, pooling_ratio = self.pool_ratio, pseudo_random = self.pseudo_random, overlapping = self.overlap, seed2 = 0, seed = 0)\n\t\treturn(batch_tensor)\n\t\t\n\tdef compute_output_shape(self, input_shape):\n\t\n\t\tif(K.image_dim_ordering() == 'channels_last' or K.image_dim_ordering() == 'tf'):\n\t\t\tif(input_shape[0] != None):\n\t\t\t\tbatch_size = int(input_shape[0]/self.pool_ratio[0])\n\t\t\telse:\n\t\t\t\tbatch_size = input_shape[0]\n\t\t\twidth = int(input_shape[1]/self.pool_ratio[1])\n\t\t\theight = int(input_shape[2]/self.pool_ratio[2])\n\t\t\tchannels = int(input_shape[3]/self.pool_ratio[3])\n\t\t\treturn(batch_size, width, height, channels)\n\t\t\t\n\t\telif(K.image_dim_ordering() == 'channels_first' or K.image_dim_ordering() == 'th'):\n\t\t\tif(input_shape[0] != None):\n\t\t\t\tbatch_size = int(input_shape[0]/self.pool_ratio[0])\n\t\t\telse:\n\t\t\t\tbatch_size = input_shape[0]\n\t\t\tchannels = int(input_shape[1]/self.pool_ratio[1])\n\t\t\twidth = int(input_shape[2]/self.pool_ratio[2])\n\t\t\theight = int(input_shape[3]/self.pool_ratio[3])\n\t\t\treturn(batch_size, channels, width, height)\n\t\t\n\tdef get_config(self):\n\t\tconfig = {'pooling_ratio': self.pool_ratio, 'pseudo_random': self.pseudo_random, 'overlap': self.overlap, 'name':self.name}\n\t\tbase_config = super(FractionalPooling2D, self).get_config()\n\t\treturn dict(list(base_config.items()) + list(config.items()))\n\t\t\n\tdef build(self, input_shape):\n\t\tself.input_spec = [InputSpec(shape=input_shape)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from efficientnet import EfficientNetB5\n\neffnet = EfficientNetB5(\n    weights= None, \n    include_top=False,\n    input_shape=(224,224,3)\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def build_model2():\n    model = Sequential()\n    model.add(effnet)\n    model.add(layers.GlobalAveragePooling2D())\n    model.add(layers.Dropout(0.3))\n    model.add(layers.Dense(2048, activation = 'relu'))\n    model.add(layers.Dropout(0.3))\n    model.add(layers.Dense(1024, activation = 'relu'))\n    model.add(layers.Dropout(0.3))\n    model.add(layers.Dense(512, activation = 'relu'))\n    model.add(layers.Dropout(0.3))\n    model.add(layers.Dense(256, activation = 'relu'))\n    model.add(layers.Dropout(0.3))\n    model.add(layers.Dense(128, activation = 'relu'))\n    model.add(layers.Dense(5, activation = 'sigmoid'))\n    effnet.load_weights('../input/efficientnet-keras-weights-b0b5/efficientnet-b5_imagenet_1000_notop.h5')\n    model.compile(\n        loss = 'binary_crossentropy',\n        optimizer = RAdam(lr=0.0005),\n        metrics = ['accuracy']\n    )\n    \n    return model\nmodel2 = build_model2()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the Metrics class\n\nclass Metrics2(Callback):\n    def on_train_begin(self, logs = {}):\n        self.val_kappas = []\n\n    def on_epoch_end(self, epoch, logs = {}):\n        X_val, y_val = self.validation_data[:2]\n        y_val = y_val.sum(axis = 1) - 1\n        \n        y_pred = self.model.predict(X_val) > 0.5\n        y_pred = y_pred.astype(int).sum(axis = 1) - 1\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights = 'quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n        \n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model2.h5')\n\n        return","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Training our model\n\nkappa_metrics2 = Metrics2()\n\nhistory2 = model2.fit_generator(\n    data_generator,\n    steps_per_epoch = x_train_NN.shape[0] / BATCH_SIZE,\n    epochs = 25,\n    validation_data = (x_val_NN, y_val_NN),\n    callbacks = [kappa_metrics2],\n    class_weight = wts,\n    validation_steps = x_val_NN.shape[0] / BATCH_SIZE\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2.load_weights('model2.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting the graph to show our training and cross-validation loss\n\nwith open('history2.json', 'w') as f:\n    json.dump(history2.history, f)\n\nhistory_df = pd.DataFrame(history2.history)\nhistory_df[['loss', 'val_loss']].plot()\nhistory_df[['acc', 'val_acc']].plot()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predict = model2.predict(x_val_NN)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plotting the trend for our kappa values\n\nplt.plot(kappa_metrics2.val_kappas)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2.load_weights('model2.h5')\n#y_val_pred_lol = model2.predict(x_val_NN)\n\ndef compute_score_inv(threshold):\n    y1 = predict > 0.5\n    y1 = y1.astype(int).sum(axis=1) - 1\n    y2 = y_val_NN.sum(axis=1) - 1\n    score = cohen_kappa_score(y1, y2, weights='quadratic')\n    \n    return 1 - score\n\nsimplex = scipy.optimize.minimize(\n    compute_score_inv, 0.5, method='nelder-mead'\n)\n\nbest_threshold = simplex['x'][0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y1 = predict > 0.5\ny1 = y1.astype(int).sum(axis=1) - 1\ny2 = y_val_NN.sum(axis=1) - 1\nscore = cohen_kappa_score(y1, y2, weights='quadratic')\nprint(score)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Finding the predictions:\n\ny_test = model2.predict(x_test) > 0.5\ny_test = y_test.astype(int).sum(axis=1) - 1\n\ntest_df['diagnosis'] = y_test\ntest_df.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}