{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# DenseNet Trained with Old and New Data\n\n---\n\nDue to the size of previous competition's data, I faced few memory-related problems while training my model. In this kernel I would like to show the approach I used.\n\nI basically splitted the training set in buckets and trained the model for each bucket.\n\nI'd truly interested to further discuss how it could has been solved. So, if you faced the same issues, please comment with your ideas :)\n\n#### Here you can find the [Inference Kernel](https://www.kaggle.com/raimonds1993/aptos19-densenet-inference-old-new-data/data?scriptVersionId=17252732)!\n\n---","metadata":{}},{"cell_type":"markdown","source":"### Credits\nI started this kernel by forking [APTOS 2019: DenseNet Keras Starter](https://www.kaggle.com/xhlulu/aptos-2019-densenet-keras-starter), by [Xhlulu](https://www.kaggle.com/xhlulu).\n\nI also used [previous competition's data](https://www.kaggle.com/tanlikesmath/diabetic-retinopathy-resized) uploaded by [ilovescience](https://www.kaggle.com/tanlikesmath).\n\nThank you both guys!","metadata":{}},{"cell_type":"markdown","source":"### Changes\n\n*Version 3:*\n- This is the first completed version to consider. (Still without seed)\n- Inference -> LB: 0.719\n\n*Version 4:*\n- Updated image size (320). In order to process the whole dataset, I load just one bucket at a time and trained the model on that.\n- Added seed to better evaluate the results.\n\n*Version 5:*\n- Changed train - val split: Now let's take previous comp data as train and the new comp data as validation\n\n*Version 9:*\n- Changed preprosessing filter in old (train) data\n- Changed Imagegenerator , more augmentation\n\n*Version 10:*\n- No Cropping in old comp data","metadata":{}},{"cell_type":"code","source":"# To have reproducible results and compare them\nnr_seed = 2019\nimport numpy as np \nnp.random.seed(nr_seed)\nimport tensorflow as tf\ntf.set_random_seed(nr_seed)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:25.024382Z","iopub.execute_input":"2022-03-06T10:04:25.02468Z","iopub.status.idle":"2022-03-06T10:04:26.34155Z","shell.execute_reply.started":"2022-03-06T10:04:25.024625Z","shell.execute_reply":"2022-03-06T10:04:26.340798Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import libraries\nimport json\nimport math\nfrom tqdm import tqdm, tqdm_notebook\nimport gc\nimport warnings\nimport os\n\nimport cv2\nfrom PIL import Image\n\nimport pandas as pd\nimport scipy\nimport matplotlib.pyplot as plt\n\nfrom keras import backend as K\nfrom keras import layers\nfrom keras.applications.densenet import DenseNet121\nfrom keras.callbacks import Callback, ModelCheckpoint\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.models import Sequential\nfrom keras.optimizers import Adam\n\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import cohen_kappa_score, accuracy_score\n\nwarnings.filterwarnings(\"ignore\")\n\n%matplotlib inline","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-06T10:04:26.343532Z","iopub.execute_input":"2022-03-06T10:04:26.343837Z","iopub.status.idle":"2022-03-06T10:04:27.27684Z","shell.execute_reply.started":"2022-03-06T10:04:26.34379Z","shell.execute_reply":"2022-03-06T10:04:27.275983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Image size\nim_size = 320\n# Batch size\nBATCH_SIZE = 32","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:27.282074Z","iopub.execute_input":"2022-03-06T10:04:27.28642Z","iopub.status.idle":"2022-03-06T10:04:27.29275Z","shell.execute_reply.started":"2022-03-06T10:04:27.286366Z","shell.execute_reply":"2022-03-06T10:04:27.291891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading & Merging","metadata":{}},{"cell_type":"code","source":"#read csv files\naptos = pd.read_csv('../input/aptos2019-blindness-detection/train.csv') #aptos\neyepacs= pd.read_csv('../input/diabetic-retinopathy-resized/trainLabels.csv')#kaggle\napt_csv=pd.read_csv('../input/aptos2019-blindness-detection/test.csv')#aptos test\nmessidor2=pd.read_csv('../input/messidor2preprocess/messidor_data.csv')#messidor\n\nprint(aptos.shape)\nprint(eyepacs.shape)\nprint(messidor2.shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:27.298498Z","iopub.execute_input":"2022-03-06T10:04:27.299033Z","iopub.status.idle":"2022-03-06T10:04:27.417215Z","shell.execute_reply.started":"2022-03-06T10:04:27.298983Z","shell.execute_reply":"2022-03-06T10:04:27.41603Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eyepacs= eyepacs[['image','level']]\neyepacs.columns = aptos.columns\neyepacs.diagnosis.value_counts()\nmessidor2=messidor2[['id_code','diagnosis']]\n\n\n# path columns\n#aptos['id_code'] = '../input/aptos2019-blindness-detection/train_images/' + aptos['id_code'].astype(str) + '.png'\neyepacs['id_code'] = '../input/diabetic-retinopathy-resized/resized_train/resized_train/' + eyepacs['id_code'].astype(str) + '.jpeg'\nmessidor2['id_code']='../input/messidor2preprocess/messidor-2/messidor-2/preprocess/' + messidor2['id_code'].astype(str)\n\npacs_df = eyepacs.copy()#train\ntest_df=messidor2.copy()#test\n\nprint(pacs_df .head())\nprint(pacs_df .shape)\nprint(test_df.shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:27.421044Z","iopub.execute_input":"2022-03-06T10:04:27.423342Z","iopub.status.idle":"2022-03-06T10:04:27.74605Z","shell.execute_reply.started":"2022-03-06T10:04:27.423278Z","shell.execute_reply":"2022-03-06T10:04:27.745084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train - Valid split","metadata":{}},{"cell_type":"code","source":"# Let's shuffle the datasets\npacs_df = pacs_df.sample(frac=1).reset_index(drop=True)\nprint(pacs_df.shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:27.748819Z","iopub.execute_input":"2022-03-06T10:04:27.749303Z","iopub.status.idle":"2022-03-06T10:04:27.766612Z","shell.execute_reply.started":"2022-03-06T10:04:27.749247Z","shell.execute_reply":"2022-03-06T10:04:27.765891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Process Images","metadata":{}},{"cell_type":"markdown","source":"Crop function: https://www.kaggle.com/ratthachat/aptos-updated-preprocessing-ben-s-cropping ","metadata":{}},{"cell_type":"code","source":"\"\"\"\ndef crop_image1(img,tol=7):\n    # img is image data\n    # tol  is tolerance\n        \n    mask = img>tol\n    return img[np.ix_(mask.any(1),mask.any(0))] #np.ix_ selects certain data from image(cropping)\n\n\ndef crop_image_from_gray(img,tol=7):\n    if img.ndim ==2: # 2d image (greyScale)\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3: # RGB colored image\n        gray_img = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY) # turn to grey scale\n        mask = gray_img>tol\n        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0): # image is too dark so that we crop out everything,\n            return img # return original image\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]#red channel\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]#green channel\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]#blue channel\n            img = np.stack([img1,img2,img3],axis=-1) #stack three channels\n        return img\n        \"\"\"\nimport os\nimport glob\nimport cv2\nimport numpy as np\ndef crop_image_from_gray(img,tol=7):\n    \"\"\"\n    Crop out black borders\n    https://www.kaggle.com/ratthachat/aptos-updated-preprocessing-ben-s-cropping\n    \"\"\"  \n    \n    if img.ndim ==2:\n        mask = img>tol\n        return img[np.ix_(mask.any(1),mask.any(0))]\n    elif img.ndim==3:\n        gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n        mask = gray_img>tol        \n        check_shape = img[:,:,0][np.ix_(mask.any(1),mask.any(0))].shape[0]\n        if (check_shape == 0):\n            return img\n        else:\n            img1=img[:,:,0][np.ix_(mask.any(1),mask.any(0))]\n            img2=img[:,:,1][np.ix_(mask.any(1),mask.any(0))]\n            img3=img[:,:,2][np.ix_(mask.any(1),mask.any(0))]\n            img = np.stack([img1,img2,img3],axis=-1)\n        return img\n\n\ndef circle_crop(img):   \n    \"\"\"\n    Create circular crop around image centre    \n    \"\"\"    \n    \n    img = cv2.imread(img)\n    img = crop_image_from_gray(img)    \n    \n    height, width, depth = img.shape    \n    \n    x = int(width/2)\n    y = int(height/2)\n    r = np.amin((x,y))\n    \n    circle_img = np.zeros((height, width), np.uint8)\n    cv2.circle(circle_img, (x,y), int(r), 1, thickness=-1)\n    img = cv2.bitwise_and(img, img, mask=circle_img)\n    img = crop_image_from_gray(img)\n    \n    return img \n\ndef circle_crop_v2(img):\n    \"\"\"\n    Create circular crop around image centre\n    \"\"\"\n    img = cv2.imread(img)\n    img = crop_image_from_gray(img)\n\n    height, width, depth = img.shape\n    largest_side = np.max((height, width))\n    img = cv2.resize(img, (largest_side, largest_side))\n\n    height, width, depth = img.shape\n\n    x = int(width / 2)\n    y = int(height / 2)\n    r = np.amin((x, y))\n\n    circle_img = np.zeros((height, width), np.uint8)\n    cv2.circle(circle_img, (x, y), int(r), 1, thickness=-1)\n    img = cv2.bitwise_and(img, img, mask=circle_img)\n    img = crop_image_from_gray(img)\n\n    return img    \n\ndef preprocess_image(image_path, desired_size=224):\n    img = cv2.imread(image_path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) #turn to greyscale\n    img = crop_image_from_gray(img)\n    img = cv2.resize(img, (desired_size,desired_size))\n    img = cv2.addWeighted(img,4,cv2.GaussianBlur(img, (0,0), desired_size/30) ,-4 ,128)\n    \n    return img\n\ndef preprocess_image_old(image_path, desired_size=im_size):\n    #img = cv2.imread(image_path)\n    #img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = circle_crop_v2(image_path) #already cropped\n    img = cv2.resize(img, (desired_size,desired_size))\n    img = cv2.addWeighted(img,4,cv2.GaussianBlur(img, (0,0), desired_size/40) ,-4 ,128)#blend two images\n    \n    return img\n\n\n","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:27.770804Z","iopub.execute_input":"2022-03-06T10:04:27.77306Z","iopub.status.idle":"2022-03-06T10:04:27.811675Z","shell.execute_reply.started":"2022-03-06T10:04:27.773008Z","shell.execute_reply":"2022-03-06T10:04:27.810444Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_samples(df, columns=4, rows=3):\n    fig = plt.figure(figsize=(5 * columns, 4 * rows))\n\n    for i in range(columns * rows):\n        image_path = df.loc[i, 'id_code']\n        image_id = df.loc[i, 'diagnosis']\n        #img = cv2.imread(f'{image_path}')\n        #img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = preprocess_image_old(image_path)\n        #img = cv2.resize(img, (im_size, im_size))\n        #img = cv2.addWeighted(img, 4, cv2.GaussianBlur(img, (0, 0), im_size / 40), -4, 128)\n\n        fig.add_subplot(rows, columns, i + 1)\n        plt.title(image_id)\n        plt.imshow(img)\n\n    plt.tight_layout()\n\ndisplay_samples(pacs_df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-03-06T10:04:27.815502Z","iopub.execute_input":"2022-03-06T10:04:27.817738Z","iopub.status.idle":"2022-03-06T10:04:31.594324Z","shell.execute_reply.started":"2022-03-06T10:04:27.815768Z","shell.execute_reply":"2022-03-06T10:04:31.592018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Processing Images","metadata":{}},{"cell_type":"markdown","source":"__UPDATE:__ Here we are reading just the validation set. In order to use 320x320 images, we are going to load one bucket at a time only when needed. This will let our code run without memory-related errors.","metadata":{}},{"cell_type":"code","source":" \"\"\"\n# validation set\nN = val_df.shape[0]\nx_val = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n\nfor i, image_id in enumerate(tqdm_notebook(val_df['id_code'])):\n    x_val[i, :, :, :] = preprocess_image(  #returns preprocessed image\n        f'{image_id}',\n        desired_size = im_size\n    )\n   \"\"\"","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:31.595457Z","iopub.execute_input":"2022-03-06T10:04:31.595709Z","iopub.status.idle":"2022-03-06T10:04:31.602137Z","shell.execute_reply.started":"2022-03-06T10:04:31.595672Z","shell.execute_reply":"2022-03-06T10:04:31.601278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x_npz = np.load(\"../input/fork-of-preprocess-aptos/images_array.npz\")\nx_val = x_npz['arr_0']\n# Load binary encoded labels for Lung Infiltrations: 0=Not_infiltration 1=Infiltration\ny_val = pd.read_csv('../input/fork-of-preprocess-aptos/mycsvfile.csv')\nprint(x_val.shape)\nprint(y_val.shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:31.604098Z","iopub.execute_input":"2022-03-06T10:04:31.60459Z","iopub.status.idle":"2022-03-06T10:04:42.02512Z","shell.execute_reply.started":"2022-03-06T10:04:31.604498Z","shell.execute_reply":"2022-03-06T10:04:42.024275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(y_val.head())\nprint(pacs_df.head())","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.026511Z","iopub.execute_input":"2022-03-06T10:04:42.026796Z","iopub.status.idle":"2022-03-06T10:04:42.041771Z","shell.execute_reply.started":"2022-03-06T10:04:42.026749Z","shell.execute_reply":"2022-03-06T10:04:42.040995Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Binary Classification\n####convert y values to 0 for no dr and 1 for dr# \ntemp=pacs_df.copy()\ntemp[\"diagnosis\"].replace({2:1, 3:1,4:1}, inplace=True)\ny_train = pd.get_dummies(temp['diagnosis']).values\ny_train[:,0]=1\n###\ntemp=test_df.copy()\ntemp[\"diagnosis\"].replace({2:1, 3:1,4:1}, inplace=True)\ny_tests = pd.get_dummies(temp['diagnosis']).values\ny_tests[:,0]=1\n###\n\ny_val[\"diagnosis\"].replace({2:1, 3:1,4:1}, inplace=True)\ny_val= pd.get_dummies(y_val['diagnosis']).values\ny_val[:,0]=1\n###\nprint(y_train.shape)\nprint(y_tests.shape)\nprint(y_val.shape)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.043083Z","iopub.execute_input":"2022-03-06T10:04:42.043595Z","iopub.status.idle":"2022-03-06T10:04:42.062417Z","shell.execute_reply.started":"2022-03-06T10:04:42.043543Z","shell.execute_reply":"2022-03-06T10:04:42.061731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(y_val[0]) #class 1 0 no dr\nprint(y_val[1]) #class 1 1 dr exists\nprint(y_train[0])#class 1 1 dr\nprint(y_train[3]) # class 10 no dr\n","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.064041Z","iopub.execute_input":"2022-03-06T10:04:42.064469Z","iopub.status.idle":"2022-03-06T10:04:42.070723Z","shell.execute_reply.started":"2022-03-06T10:04:42.064309Z","shell.execute_reply":"2022-03-06T10:04:42.069993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating multilabels\n\nInstead of predicting a single label, we will change our target to be a multilabel problem; i.e., if the target is a certain class, then it encompasses all the classes before it. E.g. encoding a class 4 retinopathy would usually be `[0, 0, 0, 1]`, but in our case we will predict `[1, 1, 1, 1]`. For more details, please check out [Lex's kernel](https://www.kaggle.com/lextoumbourou/blindness-detection-resnet34-ordinal-targets).","metadata":{}},{"cell_type":"code","source":"\"\"\"\ny_train_multi = np.empty(y_train.shape, dtype=y_train.dtype)\ny_train_multi[:, 4] = y_train[:, 4]\ny_tests_multi=np.empty(y_tests.shape, dtype=y_tests.dtype)\ny_tests_multi[:, 4] = y_tests[:,4]\nfor i in range(3, -1, -1):\n    y_train_multi[:, i] = np.logical_or(y_train[:, i], y_train_multi[:, i+1])\n    y_tests_multi[:, i] = np.logical_or(y_tests[:, i], y_tests_multi[:, i+1])\n\ny_val_multi = np.empty(y_val.shape, dtype=y_val.dtype)\ny_val_multi[:, 4] = y_val[:, 4]\n\nfor i in range(3, -1, -1):\n    y_val_multi[:, i] = np.logical_or(y_val[:, i], y_val_multi[:, i+1])\n\nprint(\"Y_train multi: {}\".format(y_train_multi.shape))\nprint(\"Y_tests multi: {}\".format(y_tests_multi.shape))\nprint(\"Y_val multi: {}\".format(y_val_multi.shape))\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.072018Z","iopub.execute_input":"2022-03-06T10:04:42.072495Z","iopub.status.idle":"2022-03-06T10:04:42.080548Z","shell.execute_reply.started":"2022-03-06T10:04:42.072446Z","shell.execute_reply":"2022-03-06T10:04:42.079616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\"\"\"y_train = y_train_multi\ny_tests=y_tests_multi\ny_val = y_val_multi\n\"\"\"","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.082011Z","iopub.execute_input":"2022-03-06T10:04:42.082655Z","iopub.status.idle":"2022-03-06T10:04:42.091621Z","shell.execute_reply.started":"2022-03-06T10:04:42.082604Z","shell.execute_reply":"2022-03-06T10:04:42.090825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# delete the uneeded df\ndel temp\ndel aptos\ndel eyepacs\ndel messidor2\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.093235Z","iopub.execute_input":"2022-03-06T10:04:42.093725Z","iopub.status.idle":"2022-03-06T10:04:42.203535Z","shell.execute_reply.started":"2022-03-06T10:04:42.093675Z","shell.execute_reply":"2022-03-06T10:04:42.20274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Creating keras callback for QWK\n\n---\n\nI had to change this function, in order to consider the best kappa score among all the buckets.","metadata":{}},{"cell_type":"code","source":"class Metrics(Callback):\n\n   def on_epoch_end(self, epoch, logs={}): # logs contains the loss value, and all the metrics at the end of a batch or epoch. Example includes the loss and mean absolute error.\n        X_val, y_val = self.validation_data[:2] #self.validation_data[0]:x val #self.validation_data[1]:y val\n        y_val = y_val.sum(axis=1)  - 1\n\n        y_pred = self.model.predict(X_val) > 0.5\n        y_pred = y_pred.astype(int).sum(axis=1) - 1\n        _val_kappa = cohen_kappa_score(\n            y_val,\n            y_pred,\n            weights='quadratic'\n        )\n\n        self.val_kappas.append(_val_kappa)\n\n        print(f\"val_kappa: {_val_kappa:.4f}\")\n\n        if _val_kappa == max(self.val_kappas):\n            print(\"Validation Kappa has improved. Saving model.\")\n            self.model.save('model.h5')\n\n        return\n","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.204909Z","iopub.execute_input":"2022-03-06T10:04:42.205374Z","iopub.status.idle":"2022-03-06T10:04:42.214569Z","shell.execute_reply.started":"2022-03-06T10:04:42.205326Z","shell.execute_reply":"2022-03-06T10:04:42.213843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Generator","metadata":{}},{"cell_type":"code","source":"def create_datagen():\n    return ImageDataGenerator(\n        featurewise_std_normalization = True,\n        horizontal_flip = True,\n        vertical_flip = True,\n        rotation_range = 360\n    )","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.21642Z","iopub.execute_input":"2022-03-06T10:04:42.216892Z","iopub.status.idle":"2022-03-06T10:04:42.223498Z","shell.execute_reply.started":"2022-03-06T10:04:42.216688Z","shell.execute_reply":"2022-03-06T10:04:42.222336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model: DenseNet-121","metadata":{}},{"cell_type":"code","source":"densenet = DenseNet121(\n    weights='../input/densenet-keras/DenseNet-BC-121-32-no-top.h5',\n    include_top=False,\n    input_shape=(im_size,im_size,3)\n)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:04:42.22513Z","iopub.execute_input":"2022-03-06T10:04:42.225604Z","iopub.status.idle":"2022-03-06T10:05:04.334743Z","shell.execute_reply.started":"2022-03-06T10:04:42.225542Z","shell.execute_reply":"2022-03-06T10:05:04.333998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_model():\n    model = Sequential()\n    model.add(densenet)\n    model.add(layers.GlobalAveragePooling2D())\n    model.add(layers.Dropout(0.5))\n    model.add(layers.Dense(2, activation='sigmoid'))\n    \n    model.compile(\n        loss='binary_crossentropy',\n        optimizer=Adam(lr=0.0001,decay=1e-6),\n        metrics=['accuracy']\n    )\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:04.338252Z","iopub.execute_input":"2022-03-06T10:05:04.338497Z","iopub.status.idle":"2022-03-06T10:05:04.352796Z","shell.execute_reply.started":"2022-03-06T10:05:04.338453Z","shell.execute_reply":"2022-03-06T10:05:04.352143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = build_model()\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:04.356453Z","iopub.execute_input":"2022-03-06T10:05:04.356732Z","iopub.status.idle":"2022-03-06T10:05:14.424187Z","shell.execute_reply.started":"2022-03-06T10:05:04.356665Z","shell.execute_reply":"2022-03-06T10:05:14.422595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training & Evaluation","metadata":{}},{"cell_type":"code","source":"#train_df = train_df.reset_index(drop=True)\nbucket_num = 8\ndiv = round(pacs_df.shape[0]/bucket_num)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:14.425558Z","iopub.execute_input":"2022-03-06T10:05:14.425846Z","iopub.status.idle":"2022-03-06T10:05:14.432521Z","shell.execute_reply.started":"2022-03-06T10:05:14.4258Z","shell.execute_reply":"2022-03-06T10:05:14.431516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_init = {\n    'val_loss': [0.0],\n    'val_acc': [0.0],\n    'loss': [0.0], \n    'acc': [0.0],\n    'bucket': [0.0]\n}\nresults = pd.DataFrame(df_init)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:14.434005Z","iopub.execute_input":"2022-03-06T10:05:14.434581Z","iopub.status.idle":"2022-03-06T10:05:14.441823Z","shell.execute_reply.started":"2022-03-06T10:05:14.43451Z","shell.execute_reply":"2022-03-06T10:05:14.440784Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I found that changing the nr. of epochs for each bucket helped in terms of performances\nepochs = [5,5,10,15,15,20,20,30]\nkappa_metrics = Metrics()\nkappa_metrics.val_kappas = []","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:14.443318Z","iopub.execute_input":"2022-03-06T10:05:14.443866Z","iopub.status.idle":"2022-03-06T10:05:14.450766Z","shell.execute_reply.started":"2022-03-06T10:05:14.443686Z","shell.execute_reply":"2022-03-06T10:05:14.449815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(0, bucket_num):\n    if i != (bucket_num - 1):\n        print(\"Bucket Nr: {}\".format(i))\n\n        N = pacs_df.iloc[i * div:(1 + i) * div].shape[0]\n        x_train = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n        for j, image_id in enumerate(tqdm_notebook(pacs_df.iloc[i * div:(1 + i) * div, 0])):\n            x_train[j, :, :, :] = preprocess_image_old(f'{image_id}', desired_size=im_size)\n\n        data_generator = create_datagen().flow(x_train, y_train[i * div:(1 + i) * div, :], batch_size=BATCH_SIZE)\n        history = model.fit_generator(\n            data_generator,\n            steps_per_epoch=x_train.shape[0] / BATCH_SIZE,\n            epochs=epochs[i],\n            validation_data=(x_val, y_val),\n            callbacks=[kappa_metrics]\n        )\n\n        dic = history.history\n        df_model = pd.DataFrame(dic)\n        df_model['bucket'] = i\n    else:\n        print(\"Bucket Nr: {}\".format(i))\n\n        N = pacs_df.iloc[i * div:].shape[0]\n        x_train = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n        for j, image_id in enumerate(tqdm_notebook(pacs_df.iloc[i * div:, 0])):\n            x_train[j, :, :, :] = preprocess_image_old(f'{image_id}', desired_size=im_size)\n        data_generator = create_datagen().flow(x_train, y_train[i * div:, :], batch_size=BATCH_SIZE)\n\n        history = model.fit_generator(\n            data_generator,\n            steps_per_epoch=x_train.shape[0] / BATCH_SIZE,\n            epochs=epochs[i],\n            validation_data=(x_val, y_val),\n            callbacks=[kappa_metrics]\n        )\n\n        dic = history.history\n        df_model = pd.DataFrame(dic)\n        df_model['bucket'] = i\n\n    results = results.append(df_model)\n    \n    del data_generator\n    del x_train\n    gc.collect()\n\n    print('-' * 40)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:14.452836Z","iopub.execute_input":"2022-03-06T10:05:14.453108Z","iopub.status.idle":"2022-03-06T10:05:39.05519Z","shell.execute_reply.started":"2022-03-06T10:05:14.453047Z","shell.execute_reply":"2022-03-06T10:05:39.053651Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results = results.iloc[1:]\nresults['kappa'] = kappa_metrics.val_kappas\nresults = results.reset_index()\nresults = results.rename(index=str, columns={\"index\": \"epoch\"})\nresults","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:39.056417Z","iopub.status.idle":"2022-03-06T10:05:39.057107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"results[['loss', 'val_loss']].plot()\nresults[['acc', 'val_acc']].plot()\nresults[['kappa']].plot()\nresults.to_csv('model_results.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:39.058327Z","iopub.status.idle":"2022-03-06T10:05:39.059022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Find best threshold","metadata":{}},{"cell_type":"code","source":"model.load_weights('model.h5')\ny_val_pred = model.predict(x_val)\n\ndef compute_score_inv(threshold):\n    y1 = y_val_pred > threshold\n    y1 = y1.astype(int).sum(axis=1) - 1\n    y2 = y_val.sum(axis=1) - 1\n    score = cohen_kappa_score(y1, y2, weights='quadratic')\n    \n    return 1 - score\n\nsimplex = scipy.optimize.minimize(\n    compute_score_inv, 0.5, method='nelder-mead'\n)\n\nbest_threshold = simplex['x'][0]\n\ny1 = y_val_pred > best_threshold\ny1 = y1.astype(int).sum(axis=1) - 1\ny2 = y_val.sum(axis=1) - 1\nscore = cohen_kappa_score(y1, y2, weights='quadratic')\nprint('Threshold: {}'.format(best_threshold))\nprint('Validation QWK score with best_threshold: {}'.format(score))\n\ny1 = y_val_pred > .5\ny1 = y1.astype(int).sum(axis=1) - 1\nscore = cohen_kappa_score(y1, y2, weights='quadratic')\nprint('Validation QWK score with .5 threshold: {}'.format(score))","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:39.060296Z","iopub.status.idle":"2022-03-06T10:05:39.060975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"testing phase","metadata":{}},{"cell_type":"code","source":"\"\"\"\nN = test_df.shape[0]\nx_tests = np.empty((N, im_size, im_size, 3), dtype=np.uint8)\n\nfor i, image_id in enumerate(tqdm_notebook(test_df['id_code'])):\n    print(image_id)\n    x_tests[i, :, :, :] = preprocess_image(  # returns preprocessed image\n        f'{image_id}',\n        desired_size = im_size\n    )\n    \"\"\"","metadata":{"execution":{"iopub.status.busy":"2022-03-06T10:05:39.062183Z","iopub.status.idle":"2022-03-06T10:05:39.062847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## [Inference Kernel](https://www.kaggle.com/raimonds1993/aptos19-densenet-inference-old-new-data/data?scriptVersionId=17252732)\n\n**Thanks for reading it all! Please let me know if you have any ideas to improve this process.**\n\n**Hope you liked it.**","metadata":{}}]}