{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Comparing DenseNet121, EfficientNetB5, InceptionV3, ResNet50 and VGG16 (Part 3/6)\n## Trained on APTOS19 + EyePACS15 ; Validation on Messidor-2 for generalization\n\n#### Quadratic Weighted Kappa (QWK):\nQuadratic Weighted Kappa (QWK, the greek letter $\\kappa$), also known as Cohen's Kappa, is the official evaluation metric in both 2015 and 2019 competitions. \nAccording to the [wikipedia article](https://en.wikipedia.org/wiki/Cohen%27s_kappa), we have\n> The definition of $\\kappa$ is:\n> $$\\kappa \\equiv \\frac{p_o - p_e}{1 - p_e}$$\n> where $p_o$ is the relative observed agreement among raters (identical to accuracy), and $p_e$ is the hypothetical probability of chance agreement, using the observed data to calculate the probabilities of each observer randomly seeing each category.\n\nThis metric is used because if we just using accuracy as the metric, it will give spurious results (because the data is unbalanced). The QWK is more fit to the problem.\n\n#### Ordinal Regression (OR):\nWe also treat this as an Ordinal Regression Problem based on this litterature: \n- Mean absolute error as training metric instead of accuracy;  [This](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9955015/)\n- Mean Squared Error as loss function instead of Cross Entropy;  [This](https://ieeexplore.ieee.org/iel7/9342628/9342629/09342711.pdf) \nand [this (Presentation video)](https://www.youtube.com/watch?v=IvhzwhjFyFQ)\n> In DR classification, the disease classes are not independent, meaning that the severity of the disease is not necessarily limited to a specific set of discrete classes. Therefore, an ordinal regression model is more appropriate because it can predict a continuous severity score, allowing for a more precise prediction of disease severity.\n---\n\n#### Note#1: \nUnlike Binary Regression, we cannot use ROC, AUC, Spe, Sen nor CM :(\nTherefore we will attempt to use these plots to compare the models: \n- Learning curve: training and validation loss (e.g., MAE or MSE) as a function of the number of epochs. By comparing the learning curves of your models, you can assess which one is better at minimizing the loss function and which one is less prone to overfitting.\n- Scatter plot: A scatter plot shows the predicted severity level vs. the actual severity level for each sample in the validation set. By comparing the scatter plots of your models, you can assess which one has better predictions and how consistent the predictions are across the severity levels.\n- Box plot: A box plot shows the distribution of the predicted severity levels across different severity levels in the validation set. By comparing the box plots of your models, you can assess how well each model predicts the different severity levels and whether there are any systematic biases in the predictions.\n- Cumulative distribution function (CDF) plot: A CDF plot shows the cumulative distribution of the predicted severity levels across different severity levels in the validation set. By comparing the CDF plots of your models, you can assess how well each model predicts the different severity levels and how the predictions are distributed across the severity levels.\n\n#### Note#2:\nAlthough ROC curves are not appropriate for ordinal regression, you can use Cumulative Net Reclassification Improvement (NRI) or Receiver Operating Characteristic Integrated with Decision Analytic (ROC-DA) curves that are appropriate for ordinal regression. These curves allow you to evaluate how well each model performs at classifying samples across different severity levels.\n\n### Credits\n#### This series of notebooks is heavily inspired by FEDERICO RAIMONDI *brilliant* kernels (DenseNet and EfficientNet).\nPlease check his [Inference Kernel](https://www.kaggle.com/raimonds1993/aptos19-densenet-inference-old-new-data/data?scriptVersionId=17252732)!\n\nThank you!","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"#  InceptionV3 Notebook \n","metadata":{}},{"cell_type":"code","source":"# To have reproducible results and compare them\nnr_seed = 2019\nimport numpy as np \nnp.random.seed(nr_seed)\nimport tensorflow as tf\ntf.set_random_seed(nr_seed)","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:13:15.233983Z","iopub.execute_input":"2023-03-06T15:13:15.234438Z","iopub.status.idle":"2023-03-06T15:13:16.476336Z","shell.execute_reply.started":"2023-03-06T15:13:15.234231Z","shell.execute_reply":"2023-03-06T15:13:16.475426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import libraries\nimport json\nimport math\nfrom tqdm import tqdm, tqdm_notebook\nimport gc\nimport warnings\nimport os\n\nimport cv2\nfrom PIL import Image\n\nimport pandas as pd\nimport scipy\nimport matplotlib.pyplot as plt\n\nfrom keras import backend as K\nfrom keras import layers\nfrom tensorflow.keras.applications.inception_v3 import InceptionV3\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score\n\nwarnings.filterwarnings(\"ignore\")\n\n%matplotlib inline","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-06T15:16:59.394450Z","iopub.execute_input":"2023-03-06T15:16:59.394893Z","iopub.status.idle":"2023-03-06T15:17:00.093443Z","shell.execute_reply.started":"2023-03-06T15:16:59.394830Z","shell.execute_reply":"2023-03-06T15:17:00.092432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Image size\nim_size = 224\n# Batch size\nBATCH_SIZE = 32","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:17:08.066158Z","iopub.execute_input":"2023-03-06T15:17:08.066492Z","iopub.status.idle":"2023-03-06T15:17:08.073007Z","shell.execute_reply.started":"2023-03-06T15:17:08.066434Z","shell.execute_reply":"2023-03-06T15:17:08.072227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading & Merging","metadata":{}},{"cell_type":"code","source":"new_train = pd.read_csv('../input/aptos2019-blindness-detection/train.csv')\nold_train = pd.read_csv('../input/diabetic-retinopathy-resized/trainLabels.csv')\nprint(new_train.shape)\nprint(old_train.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:17:11.510346Z","iopub.execute_input":"2023-03-06T15:17:11.510648Z","iopub.status.idle":"2023-03-06T15:17:11.570572Z","shell.execute_reply.started":"2023-03-06T15:17:11.510597Z","shell.execute_reply":"2023-03-06T15:17:11.569698Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"old_train = old_train[['image','level']]\nold_train.columns = new_train.columns\nold_train.diagnosis.value_counts()\n\n# path columns\nnew_train['id_code'] = '../input/aptos2019-blindness-detection/train_images/' + new_train['id_code'].astype(str) + '.png'\nold_train['id_code'] = '../input/diabetic-retinopathy-resized/resized_train/resized_train/' + old_train['id_code'].astype(str) + '.jpeg'\n\ntrain_df = old_train.copy()\nval_df = new_train.copy()\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:17:15.899792Z","iopub.execute_input":"2023-03-06T15:17:15.900093Z","iopub.status.idle":"2023-03-06T15:17:16.009071Z","shell.execute_reply.started":"2023-03-06T15:17:15.900039Z","shell.execute_reply":"2023-03-06T15:17:16.008231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Let's shuffle the datasets\ntrain_df = train_df.sample(frac=1).reset_index(drop=True)\nval_df = val_df.sample(frac=1).reset_index(drop=True)\nprint(train_df.shape)\nprint(val_df.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:17:21.128330Z","iopub.execute_input":"2023-03-06T15:17:21.128652Z","iopub.status.idle":"2023-03-06T15:17:21.145213Z","shell.execute_reply.started":"2023-03-06T15:17:21.128596Z","shell.execute_reply":"2023-03-06T15:17:21.144165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train - Valid split","metadata":{}},{"cell_type":"code","source":"# Not used in version 5 train_df, val_df = train_test_split(train_df, shuffle=True, stratify=train_df.diagnosis, test_size=0.2, random_state=2019)","metadata":{"execution":{"iopub.status.busy":"2023-02-28T18:41:37.653469Z","iopub.execute_input":"2023-02-28T18:41:37.658127Z","iopub.status.idle":"2023-02-28T18:41:37.70021Z","shell.execute_reply.started":"2023-02-28T18:41:37.653816Z","shell.execute_reply":"2023-02-28T18:41:37.699432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_samples(df, columns=4, rows=3):\n    fig=plt.figure(figsize=(5*columns, 4*rows))\n\n    for i in range(columns*rows):\n        image_path = df.loc[i,'id_code']\n        image_id = df.loc[i,'diagnosis']\n        img = cv2.imread(f'{image_path}')\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        #img = crop_image_from_gray(img)\n        img = cv2.resize(img, (im_size,im_size))\n        img = cv2.addWeighted(img,4,cv2.GaussianBlur(img, (0,0), im_size/40) ,-4 ,128)\n        \n        fig.add_subplot(rows, columns, i+1)\n        plt.title(image_id)\n        plt.imshow(img)\n    \n    plt.tight_layout()\n\ndisplay_samples(train_df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-06T15:17:25.565104Z","iopub.execute_input":"2023-03-06T15:17:25.565438Z","iopub.status.idle":"2023-03-06T15:17:29.358707Z","shell.execute_reply.started":"2023-03-06T15:17:25.565381Z","shell.execute_reply":"2023-03-06T15:17:29.353448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Processing Images","metadata":{}},{"cell_type":"markdown","source":"Crop function: https://www.kaggle.com/ratthachat/aptos-updated-preprocessing-ben-s-cropping ","metadata":{}},{"cell_type":"code","source":"def crop_image1(img,tol=7):\n    # img is image data\n    # tol  is tolerance\n        \n    mask = img>tol\n    return img[np.ix_(mask.any(1),mask.any(0))]\n\ndef crop_image_from_gray(img,tol=7):\n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)\n        mask = gray_img>tol\n        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0): # image is too dark so that we crop out everything,\n            return img # return original image\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n            img = np.stack([img1,img2,img3],axis=-1)\n        return img\n\ndef preprocess_image(image_path, desired_size=224):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = crop_image_from_gray(img)\n    img = cv2.resize(img, (desired_size,desired_size))\n    img = cv2.addWeighted(img,4,cv2.GaussianBlur(img, (0,0), desired_size/30) ,-4 ,128)\n    \n    return img\n\ndef preprocess_image_old(image_path, desired_size=224):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    #img = crop_image_from_gray(img)\n    img = cv2.resize(img, (desired_size,desired_size))\n    img = cv2.addWeighted(img,4,cv2.GaussianBlur(img, (0,0), desired_size/40) ,-4 ,128)\n    \n    return img","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:17:32.234877Z","iopub.execute_input":"2023-03-06T15:17:32.235219Z","iopub.status.idle":"2023-03-06T15:17:32.251302Z","shell.execute_reply.started":"2023-03-06T15:17:32.235140Z","shell.execute_reply":"2023-03-06T15:17:32.250386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"__UPDATE:__ Here we are reading just the validation set. In order to use 224x224 images, we are going to load one bucket at a time only when needed. This will let our code run without memory-related errors.","metadata":{}},{"cell_type":"code","source":"# validation set\nN = val_df.shape[0]\nx_val = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n\nfor i, image_id in enumerate(tqdm_notebook(val_df['id_code'])):\n    x_val[i, :, :, :] = preprocess_image(\n        f'{image_id}',\n        desired_size = im_size\n    )","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:17:42.925124Z","iopub.execute_input":"2023-03-06T15:17:42.925458Z","iopub.status.idle":"2023-03-06T15:29:25.010083Z","shell.execute_reply.started":"2023-03-06T15:17:42.925400Z","shell.execute_reply":"2023-03-06T15:29:25.009280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_train = train_df['diagnosis'].values\ny_val = val_df['diagnosis'].values\n\nprint(y_train.shape)\nprint(x_val.shape)\nprint(y_val.shape)","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:29:25.012068Z","iopub.execute_input":"2023-03-06T15:29:25.012619Z","iopub.status.idle":"2023-03-06T15:29:25.020023Z","shell.execute_reply.started":"2023-03-06T15:29:25.012566Z","shell.execute_reply":"2023-03-06T15:29:25.019113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating multilabels\n\nInstead of predicting a single label, we will change our target to be a multilabel problem; i.e., if the target is a certain class, then it encompasses all the classes before it. E.g. encoding a class 4 retinopathy would usually be `[0, 0, 0, 1]`, but in our case we will predict `[1, 1, 1, 1]`. For more details, please check out [Lex's kernel](https://www.kaggle.com/lextoumbourou/blindness-detection-resnet34-ordinal-targets).","metadata":{}},{"cell_type":"markdown","source":"y_train_multi = np.empty(y_train.shape, dtype=y_train.dtype)\ny_train_multi[:, 4] = y_train[:, 4]\n\nfor i in range(3, -1, -1):\n    y_train_multi[:, i] = np.logical_or(y_train[:, i], y_train_multi[:, i+1])\n\ny_val_multi = np.empty(y_val.shape, dtype=y_val.dtype)\ny_val_multi[:, 4] = y_val[:, 4]\n\nfor i in range(3, -1, -1):\n    y_val_multi[:, i] = np.logical_or(y_val[:, i], y_val_multi[:, i+1])\n\nprint(\"Y_train multi: {}\".format(y_train_multi.shape))\nprint(\"Y_val multi: {}\".format(y_val_multi.shape))","metadata":{"execution":{"iopub.status.busy":"2023-02-28T18:46:33.825579Z","iopub.execute_input":"2023-02-28T18:46:33.826046Z","iopub.status.idle":"2023-02-28T18:46:33.83561Z","shell.execute_reply.started":"2023-02-28T18:46:33.825983Z","shell.execute_reply":"2023-02-28T18:46:33.834857Z"}}},{"cell_type":"markdown","source":"y_train = y_train_multi\ny_val = y_val_multi","metadata":{"execution":{"iopub.status.busy":"2023-02-28T18:46:33.83678Z","iopub.execute_input":"2023-02-28T18:46:33.837242Z","iopub.status.idle":"2023-02-28T18:46:33.844853Z","shell.execute_reply.started":"2023-02-28T18:46:33.837193Z","shell.execute_reply":"2023-02-28T18:46:33.8441Z"}}},{"cell_type":"code","source":"# delete the uneeded df\ndel new_train\ndel old_train\ndel val_df\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:29:25.021592Z","iopub.execute_input":"2023-03-06T15:29:25.022113Z","iopub.status.idle":"2023-03-06T15:29:25.131374Z","shell.execute_reply.started":"2023-03-06T15:29:25.021870Z","shell.execute_reply":"2023-03-06T15:29:25.130370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating keras callback for QWK\n---\nBecause our main metric in this problem is QWK, we have to make a function that calculate the QWK for every epoch and also saving the model that got the highest score. \nI had to change this function, in order to consider the best kappa score among all the buckets.","metadata":{}},{"cell_type":"code","source":"class Metrics(Callback):\n\n    def on_epoch_end(self, epoch, logs={}):\n        X_val, y_val = self.validation_data[:2]\n        \n        y_pred = self.model.predict(X_val)\n        \n        coef = [0.5, 1.5, 2.5, 3.5]\n\n        for i, pred in enumerate(y_pred):\n            if pred < coef[0]:\n                y_pred[i] = 0\n            elif pred >= coef[0] and pred < coef[1]:\n                y_pred[i] = 1\n            elif pred >= coef[1] and pred < coef[2]:\n                y_pred[i] = 2\n            elif pred >= coef[2] and pred < coef[3]:\n                y_pred[i] = 3\n            else:\n                y_pred[i] = 4\n\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred, \n            weights='quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n        \n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n\n        return","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:29:25.132883Z","iopub.execute_input":"2023-03-06T15:29:25.134314Z","iopub.status.idle":"2023-03-06T15:29:25.145231Z","shell.execute_reply.started":"2023-03-06T15:29:25.133157Z","shell.execute_reply":"2023-03-06T15:29:25.144196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Generator","metadata":{}},{"cell_type":"code","source":"def create_datagen():\n    return ImageDataGenerator(\n        horizontal_flip = True,\n        vertical_flip = True,\n        rotation_range = 160,\n        zoom_range=0.35\n    )","metadata":{"execution":{"iopub.status.busy":"2023-03-06T15:29:25.147729Z","iopub.execute_input":"2023-03-06T15:29:25.148177Z","iopub.status.idle":"2023-03-06T15:29:25.156981Z","shell.execute_reply.started":"2023-03-06T15:29:25.147989Z","shell.execute_reply":"2023-03-06T15:29:25.156184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model: InceptionV3","metadata":{}},{"cell_type":"code","source":"inception= InceptionV3(\n    weights=\"imagenet\",\n    include_top=False,\n    input_shape=(im_size,im_size,3))","metadata":{"execution":{"iopub.status.busy":"2023-03-06T16:52:53.991860Z","iopub.execute_input":"2023-03-06T16:52:53.992161Z","iopub.status.idle":"2023-03-06T16:53:10.634483Z","shell.execute_reply.started":"2023-03-06T16:52:53.992107Z","shell.execute_reply":"2023-03-06T16:53:10.633623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model():\n    model = Sequential(InceptionV3(\n    weights=\"imagenet\",\n    include_top=False,\n    input_shape=(im_size,im_size,3)).layers)\n    model.add(layers.GlobalAveragePooling2D())\n    model.add(layers.Dropout(0.5))\n    model.add(layers.Dense(1, activation='linear'))\n    \n    model.compile(\n        loss='mean_squared_error',\n        optimizer=Adam(lr=0.0001,decay=1e-6),\n        metrics=['mae']\n    )\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-03-06T16:53:59.501202Z","iopub.execute_input":"2023-03-06T16:53:59.501528Z","iopub.status.idle":"2023-03-06T16:53:59.507839Z","shell.execute_reply.started":"2023-03-06T16:53:59.501474Z","shell.execute_reply":"2023-03-06T16:53:59.506832Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = build_model()\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2023-03-06T16:54:04.417729Z","iopub.execute_input":"2023-03-06T16:54:04.418102Z","iopub.status.idle":"2023-03-06T16:54:22.576390Z","shell.execute_reply.started":"2023-03-06T16:54:04.418046Z","shell.execute_reply":"2023-03-06T16:54:22.575352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training & Evaluation","metadata":{}},{"cell_type":"code","source":"#train_df = train_df.reset_index(drop=True)\nnum_bucket = 8\ndiv = round(train_df.shape[0]/bucket_num)","metadata":{"execution":{"iopub.status.busy":"2023-03-06T11:10:53.875656Z","iopub.execute_input":"2023-03-06T11:10:53.87599Z","iopub.status.idle":"2023-03-06T11:10:53.880677Z","shell.execute_reply.started":"2023-03-06T11:10:53.875913Z","shell.execute_reply":"2023-03-06T11:10:53.879609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_init = {\n    'val_loss': [0.0],\n    'val_acc': [0.0],\n    'loss': [0.0], \n    'acc': [0.0],\n    'bucket': [0.0]\n}\nresults = pd.DataFrame(df_init)","metadata":{"execution":{"iopub.status.busy":"2023-03-06T11:10:55.918069Z","iopub.execute_input":"2023-03-06T11:10:55.91837Z","iopub.status.idle":"2023-03-06T11:10:55.926472Z","shell.execute_reply.started":"2023-03-06T11:10:55.918317Z","shell.execute_reply":"2023-03-06T11:10:55.92575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I found that changing the nr. of epochs for each bucket helped in terms of performances\nepochs = [10,10,10,15,15,20,20,25]\nkappa_metrics = Metrics()\nkappa_metrics.val_kappas = []","metadata":{"execution":{"iopub.status.busy":"2023-03-06T11:10:58.356392Z","iopub.execute_input":"2023-03-06T11:10:58.356699Z","iopub.status.idle":"2023-03-06T11:10:58.361329Z","shell.execute_reply.started":"2023-03-06T11:10:58.356647Z","shell.execute_reply":"2023-03-06T11:10:58.360532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(0,num_bucket):\n    if i != (num_bucket-1):\n        print(\"Bucket Nr: {}\".format(i))\n        \n        N = train_df.iloc[i*div:(1+i)*div].shape[0]\n        x_train = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n        for j, image_id in enumerate(tqdm_notebook(train_df.iloc[i*div:(1+i)*div,0])):\n            x_train[j, :, :, :] = preprocess_image_old(f'{image_id}', desired_size = im_size)\n\n        data_generator = create_datagen().flow(x_train, y_train[i*div:(1+i)*div], batch_size=BATCH_SIZE)\n        history = model.fit_generator(\n                        data_generator,\n                        steps_per_epoch=x_train.shape[0] / BATCH_SIZE,\n                        epochs=epochs[i],\n                        validation_data=(x_val, y_val),\n                        callbacks=[kappa_metrics]\n                        )\n        \n        dic = history.history\n        df_model = pd.DataFrame(dic)\n        df_model['bucket'] = i\n    else:\n        print(\"Bucket Nr: {}\".format(i))\n        \n        N = train_df.iloc[i*div:].shape[0]\n        x_train = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n        for j, image_id in enumerate(tqdm_notebook(train_df.iloc[i*div:,0])):\n            x_train[j, :, :, :] = preprocess_image_old(f'{image_id}', desired_size = im_size)\n        data_generator = create_datagen().flow(x_train, y_train[i*div:], batch_size=BATCH_SIZE)\n        \n        history = model.fit_generator(\n                        data_generator,\n                        steps_per_epoch=x_train.shape[0] / BATCH_SIZE,\n                        epochs=epochs[i],\n                        validation_data=(x_val, y_val),\n                        callbacks=[kappa_metrics]\n                        )\n        \n        dic = history.history\n        df_model = pd.DataFrame(dic)\n        df_model['bucket'] = i\n\n    results = results.append(df_model)\n    \n    del data_generator\n    del x_train\n    gc.collect()\n    \n    print('-'*40)","metadata":{"execution":{"iopub.status.busy":"2023-03-06T11:11:00.172566Z","iopub.execute_input":"2023-03-06T11:11:00.172873Z","iopub.status.idle":"2023-03-06T14:11:59.770409Z","shell.execute_reply.started":"2023-03-06T11:11:00.172819Z","shell.execute_reply":"2023-03-06T14:11:59.769586Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = results.iloc[1:]\nresults['kappa'] = kappa_metrics.val_kappas\nresults = results.reset_index()\nresults = results.rename(index=str, columns={\"index\": \"epoch\"})\nprint(max(results.kappa))","metadata":{"execution":{"iopub.status.busy":"2023-03-06T14:14:17.322076Z","iopub.execute_input":"2023-03-06T14:14:17.322385Z","iopub.status.idle":"2023-03-06T14:14:18.274958Z","shell.execute_reply.started":"2023-03-06T14:14:17.322336Z","shell.execute_reply":"2023-03-06T14:14:18.273781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results[['loss', 'val_loss']].plot()\nresults[['mean_absolute_error', 'val_mean_absolute_error']].plot()\nresults[['kappa']].plot()\nresults.to_csv('model_results.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2023-03-06T14:14:31.962774Z","iopub.execute_input":"2023-03-06T14:14:31.963115Z","iopub.status.idle":"2023-03-06T14:14:32.882931Z","shell.execute_reply.started":"2023-03-06T14:14:31.963056Z","shell.execute_reply":"2023-03-06T14:14:32.881884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# QWK Optimizer\nSee Abhishek optimizer [here](https://www.kaggle.com/abhishek/optimizer-for-quadratic-weighted-kappa)","metadata":{}},{"cell_type":"code","source":"class OptimizedRounder(object):\n    def __init__(self):\n        self.coef_ = 0\n\n    def _kappa_loss(self, coef, X, y):\n        X_p = np.copy(X)\n        for i, pred in enumerate(X_p):\n            if pred < coef[0]:\n                X_p[i] = 0\n            elif pred >= coef[0] and pred < coef[1]:\n                X_p[i] = 1\n            elif pred >= coef[1] and pred < coef[2]:\n                X_p[i] = 2\n            elif pred >= coef[2] and pred < coef[3]:\n                X_p[i] = 3\n            else:\n                X_p[i] = 4\n\n        ll = cohen_kappa_score(y, X_p, weights='quadratic')\n        return -ll\n\n    def fit(self, X, y):\n        loss_partial = partial(self._kappa_loss, X=X, y=y)\n        initial_coef = [0.5, 1.5, 2.5, 3.5]\n        self.coef_ = scipy.optimize.minimize(loss_partial, initial_coef, method='nelder-mead')\n        print(-loss_partial(self.coef_['x']))\n\n    def predict(self, X, coef):\n        X_p = np.copy(X)\n        for i, pred in enumerate(X_p):\n            if pred < coef[0]:\n                X_p[i] = 0\n            elif pred >= coef[0] and pred < coef[1]:\n                X_p[i] = 1\n            elif pred >= coef[1] and pred < coef[2]:\n                X_p[i] = 2\n            elif pred >= coef[2] and pred < coef[3]:\n                X_p[i] = 3\n            else:\n                X_p[i] = 4\n        return X_p\n\n    def coefficients(self):\n        return self.coef_['x']","metadata":{"execution":{"iopub.status.busy":"2023-03-06T14:22:10.894529Z","iopub.execute_input":"2023-03-06T14:22:10.894832Z","iopub.status.idle":"2023-03-06T14:22:10.90723Z","shell.execute_reply.started":"2023-03-06T14:22:10.894779Z","shell.execute_reply":"2023-03-06T14:22:10.906414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from functools import partial","metadata":{"execution":{"iopub.status.busy":"2023-03-06T14:21:46.931107Z","iopub.execute_input":"2023-03-06T14:21:46.931417Z","iopub.status.idle":"2023-03-06T14:21:46.935336Z","shell.execute_reply.started":"2023-03-06T14:21:46.931366Z","shell.execute_reply":"2023-03-06T14:21:46.934531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.load_weights('model.h5')\ny_val_pred = model.predict(x_val)\n\noptR = OptimizedRounder()\noptR.fit(y_val_pred, y_val)\ncoefficients = optR.coefficients()\nprint(f'Coefficients: {coefficients}')\ny_val_pred = optR.predict(y_val_pred, coefficients)\n\nscore = cohen_kappa_score(y_val_pred, y_val, weights='quadratic')\n\nprint('Optimized Validation QWK score: {}'.format(score))\nprint('Not Optimized Validation QWK score: {}'.format(max(results.kappa)))","metadata":{"execution":{"iopub.status.busy":"2023-03-06T14:22:20.362984Z","iopub.execute_input":"2023-03-06T14:22:20.363291Z","iopub.status.idle":"2023-03-06T14:22:34.114199Z","shell.execute_reply.started":"2023-03-06T14:22:20.36324Z","shell.execute_reply":"2023-03-06T14:22:34.11277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Save final model","metadata":{}},{"cell_type":"code","source":"model.save('InceptionV3_model.h5')","metadata":{"execution":{"iopub.status.busy":"2023-03-06T14:22:45.961389Z","iopub.execute_input":"2023-03-06T14:22:45.961699Z","iopub.status.idle":"2023-03-06T14:22:46.933859Z","shell.execute_reply.started":"2023-03-06T14:22:45.961645Z","shell.execute_reply":"2023-03-06T14:22:46.933025Z"},"trusted":true},"execution_count":null,"outputs":[]}]}