{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Trying Out Baseline with EfficentNetB1\n\n\n## This notebook uses NIH Chest X-Ray dataset to learn and Predict Multiple Cateogry Dignosis from chest x-ray images. Please upvote if you like it.\n\n## To learn more about diagnosing chest x-rays follow the video below. ","metadata":{"papermill":{"duration":0.040849,"end_time":"2023-06-03T10:48:14.812685","exception":false,"start_time":"2023-06-03T10:48:14.771836","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Outline\nUse these links to jump to specific sections of this notebook!\n\n- [1. Import Packages and Function](#1)\n- [2. Load the Datasets](#2)\n    - [2.1 Preventing Data Leakage](#2-1)\n    - [2.2 Preparing Images](#2-2)\n    - [2.3 Visualizing Images](#2-3)\n- [3. Class Imbalence](#3)\n    - [3.1 Addressing Class Imbalance](#3-1)\n    - [3.2 Computing and Visualizing Class Imbalance](#3-2)\n    - [3.3 Dealing with Class Imbalance](#3-3)\n    - [3.4 Calculating the weighted loss](#3-4)\n- [4. Model Creation](#4)\n    - [4.1 Evaluating Pretrained Weights](#4-1)\n    - [4.2 Training i.e. Fine Tuning](#4-2)\n- [5. Prediction and Evaluation](#5)\n    - [5.1 ROC Curve and AUROC](#5-1)\n    - [5.2 Comparing ROC Curves](#5-2)\n    - [5.3 Comparing AUROC Values](#5-3)\n- [6. Saving Weights and Training History](#6)","metadata":{"papermill":{"duration":0.036393,"end_time":"2023-06-03T10:48:14.885483","exception":false,"start_time":"2023-06-03T10:48:14.84909","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<a name='1'></a>\n## 1. Import Packages and Functions¶\n\nWe'll make use of the following packages:\n- `numpy` and `pandas` is what we'll use to manipulate our data\n- `matplotlib.pyplot` and `seaborn` will be used to produce plots for visualization\n- `util` will provide the locally defined utility functions that have been provided for this assignment\n\nWe will also use several modules from the `keras` framework for building deep learning models.\n\nRun the next cell to import all the necessary packages.","metadata":{"papermill":{"duration":0.036391,"end_time":"2023-06-03T10:48:14.957766","exception":false,"start_time":"2023-06-03T10:48:14.921375","status":"completed"},"tags":[]}},{"cell_type":"code","source":"\nimport random\nimport cv2\nfrom keras import backend as K\nfrom keras.preprocessing import image\nfrom sklearn.metrics import roc_auc_score, roc_curve\nfrom tensorflow.compat.v1.logging import INFO, set_verbosity\n\nrandom.seed(a=None, version=2)\n\nset_verbosity(INFO)\n\n\ndef get_mean_std_per_batch(image_path, df, H=320, W=320):\n    sample_data = []\n    for idx, img in enumerate(df.sample(100)[\"Image\"].values):\n        # path = image_dir + img\n        sample_data.append(\n            np.array(image.load_img(image_path, target_size=(H, W))))\n\n    mean = np.mean(sample_data[0])\n    std = np.std(sample_data[0])\n    return mean, std\n\n\ndef load_image(img, image_dir, df, preprocess=True, H=320, W=320):\n    \"\"\"Load and preprocess image.\"\"\"\n    img_path = image_dir + img\n    mean, std = get_mean_std_per_batch(img_path, df, H=H, W=W)\n    x = image.load_img(img_path, target_size=(H, W))\n    if preprocess:\n        x -= mean\n        x /= std\n        x = np.expand_dims(x, axis=0)\n    return x\n\n\ndef grad_cam(input_model, image, cls, layer_name, H=320, W=320):\n    \"\"\"GradCAM method for visualizing input saliency.\"\"\"\n    y_c = input_model.output[0, cls]\n    conv_output = input_model.get_layer(layer_name).output\n    grads = K.gradients(y_c, conv_output)[0]\n\n    gradient_function = K.function([input_model.input], [conv_output, grads])\n\n    output, grads_val = gradient_function([image])\n    output, grads_val = output[0, :], grads_val[0, :, :, :]\n\n    weights = np.mean(grads_val, axis=(0, 1))\n    cam = np.dot(output, weights)\n\n    # Process CAM\n    cam = cv2.resize(cam, (W, H), cv2.INTER_LINEAR)\n    cam = np.maximum(cam, 0)\n    cam = cam / cam.max()\n    return cam\n\n\ndef compute_gradcam(model, img, image_dir, df, labels, selected_labels,\n                    layer_name='bn'):\n    preprocessed_input = load_image(img, image_dir, df)\n    predictions = model.predict(preprocessed_input)\n\n    print(\"Loading original image\")\n    plt.figure(figsize=(15, 10))\n    plt.subplot(151)\n    plt.title(\"Original\")\n    plt.axis('off')\n    plt.imshow(load_image(img, image_dir, df, preprocess=False), cmap='gray')\n\n    j = 1\n    for i in range(len(labels)):\n        if labels[i] in selected_labels:\n            print(f\"Generating gradcam for class {labels[i]}\")\n            gradcam = grad_cam(model, preprocessed_input, i, layer_name)\n            plt.subplot(151 + j)\n            plt.title(f\"{labels[i]}: p={predictions[0][i]:.3f}\")\n            plt.axis('off')\n            plt.imshow(load_image(img, image_dir, df, preprocess=False),\n                       cmap='gray')\n            plt.imshow(gradcam, cmap='jet', alpha=min(0.5, predictions[0][i]))\n            j += 1\n\n\ndef get_roc_curve(labels, predicted_vals, generator, when = ''):\n    auc_roc_vals = []\n    for i in range(len(labels)):\n        try:\n            gt = generator.labels[:, i]\n            pred = predicted_vals[:, i]\n            auc_roc = roc_auc_score(gt, pred)\n            auc_roc_vals.append(auc_roc)\n            fpr_rf, tpr_rf, _ = roc_curve(gt, pred)\n            plt.figure(1, figsize=(10, 10))\n            plt.plot([0, 1], [0, 1], 'k--')\n            plt.plot(fpr_rf, tpr_rf,\n                     label=labels[i] + \" (\" + str(round(auc_roc, 3)) + \")\")\n            plt.xlabel('False positive rate')\n            plt.ylabel('True positive rate')\n            plt.title('ROC curve ' + when)\n            plt.legend(loc='best')\n        except:\n            print(\n                f\"Error in generating ROC curve for {labels[i]}. \"\n                f\"Dataset lacks enough examples.\"\n            )\n    plt.show()\n    return auc_roc_vals\n","metadata":{"_kg_hide-input":true,"papermill":{"duration":5.974988,"end_time":"2023-06-03T10:48:20.969839","exception":false,"start_time":"2023-06-03T10:48:14.994851","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:27:49.387141Z","iopub.execute_input":"2023-06-06T05:27:49.387782Z","iopub.status.idle":"2023-06-06T05:27:57.886316Z","shell.execute_reply.started":"2023-06-06T05:27:49.387743Z","shell.execute_reply":"2023-06-06T05:27:57.885196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n!pip install -q efficientnet\nimport efficientnet.tfkeras as efn\n\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\n\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.applications.densenet import DenseNet121\nfrom keras.layers import Dense, GlobalAveragePooling2D\nfrom keras.models import Model\n\nfrom keras.models import load_model\n\n\n# from tensorflow.keras.applications import DenseNet121\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\n# import tensorflow.keras.layers as Layers","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":8.95104,"end_time":"2023-06-03T10:48:29.961801","exception":false,"start_time":"2023-06-03T10:48:21.010761","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:27:57.888221Z","iopub.execute_input":"2023-06-06T05:27:57.888975Z","iopub.status.idle":"2023-06-06T05:28:10.736315Z","shell.execute_reply.started":"2023-06-06T05:27:57.888939Z","shell.execute_reply":"2023-06-06T05:28:10.735242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tensorflow.keras.optimizers import Adam","metadata":{"execution":{"iopub.status.busy":"2023-06-06T05:31:25.008509Z","iopub.execute_input":"2023-06-06T05:31:25.00888Z","iopub.status.idle":"2023-06-06T05:31:25.014699Z","shell.execute_reply.started":"2023-06-06T05:31:25.008851Z","shell.execute_reply":"2023-06-06T05:31:25.013584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n    # TPU detection. No parameters necessary if TPU_NAME environment variable is\n    # set: this is always the case on Kaggle.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Running on TPU ', tpu.master())\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    # Default distribution strategy in Tensorflow. Works on CPU and single GPU.\n    strategy = tf.distribute.get_strategy()\n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"papermill":{"duration":0.054488,"end_time":"2023-06-03T10:48:30.053688","exception":false,"start_time":"2023-06-03T10:48:29.9992","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:28:10.738053Z","iopub.execute_input":"2023-06-06T05:28:10.738396Z","iopub.status.idle":"2023-06-06T05:28:10.759452Z","shell.execute_reply.started":"2023-06-06T05:28:10.73836Z","shell.execute_reply":"2023-06-06T05:28:10.758579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMAGE_SIZE=[128, 128]\nEPOCHS = 20\n# BATCH_SIZE = 8 * strategy.num_replicas_in_sync\nBATCH_SIZE = 64\n","metadata":{"papermill":{"duration":0.046258,"end_time":"2023-06-03T10:48:30.137368","exception":false,"start_time":"2023-06-03T10:48:30.09111","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:28:10.762731Z","iopub.execute_input":"2023-06-06T05:28:10.76405Z","iopub.status.idle":"2023-06-06T05:28:10.769099Z","shell.execute_reply.started":"2023-06-06T05:28:10.764016Z","shell.execute_reply":"2023-06-06T05:28:10.767943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='2'></a>\n## 2. Load the Datasets\n\nFor this assignment, we will be using the [ChestX-ray8 dataset](https://arxiv.org/abs/1705.02315) which contains 108,948 frontal-view X-ray images of 32,717 unique patients. \n- Each image in the data set contains multiple text-mined labels identifying 14 different pathological conditions. \n- These in turn can be used by physicians to diagnose 8 different diseases. \n- We will use this data to develop a single model that will provide binary classification predictions for each of the 14 labeled pathologies. \n- In other words it will predict 'positive' or 'negative' for each of the pathologies.\n \nThis dataset has been annotated by consensus among four different radiologists for 5 of our 14 pathologies:\n- `Consolidation`\n- `Edema`\n- `Effusion`\n- `Cardiomegaly`\n- `Atelectasis`","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","papermill":{"duration":0.03685,"end_time":"2023-06-03T10:48:30.211383","exception":false,"start_time":"2023-06-03T10:48:30.174533","status":"completed"},"tags":[]}},{"cell_type":"code","source":"train_df_main = pd.read_csv('../input/chestxray8-dataframe/train_df.csv')\n# valid_df = pd.read_csv(\"nih/valid-small.csv\")\n# test_df = pd.read_csv(\"nih/test.csv\")\n\ntrain_df_main.drop(['No Finding'], axis = 1, inplace = True)\nlabels = train_df_main.columns[2:-1]\nlabels","metadata":{"papermill":{"duration":0.512719,"end_time":"2023-06-03T10:48:30.76087","exception":false,"start_time":"2023-06-03T10:48:30.248151","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:28:10.770812Z","iopub.execute_input":"2023-06-06T05:28:10.771419Z","iopub.status.idle":"2023-06-06T05:28:11.145239Z","shell.execute_reply.started":"2023-06-06T05:28:10.771341Z","shell.execute_reply":"2023-06-06T05:28:11.14434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\ntrain_df, discard = train_test_split(train_df_main, test_size = 0.7, random_state = 1993)\n\ntrain_and_valid_set, test_set = train_test_split(train_df, test_size = 0.2, random_state = 1993)\ntrain_set, valid_set = train_test_split(train_and_valid_set, test_size = 0.2, random_state = 1993)","metadata":{"papermill":{"duration":0.109282,"end_time":"2023-06-03T10:48:30.9081","exception":false,"start_time":"2023-06-03T10:48:30.798818","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:28:11.146685Z","iopub.execute_input":"2023-06-06T05:28:11.147281Z","iopub.status.idle":"2023-06-06T05:28:11.207147Z","shell.execute_reply.started":"2023-06-06T05:28:11.147248Z","shell.execute_reply":"2023-06-06T05:28:11.206206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='2-1'></a>\n### 2.1 Preventing Data Leakage\nIt is worth noting that our dataset contains multiple images for each patient. This could be the case, for example, when a patient has taken multiple X-ray images at different times during their hospital visits. In our data splitting, we have ensured that the split is done on the patient level so that there is no data \"leakage\" between the train, validation, and test datasets.\n","metadata":{"papermill":{"duration":0.039314,"end_time":"2023-06-03T10:48:30.987714","exception":false,"start_time":"2023-06-03T10:48:30.9484","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<a name='Ex-1'></a>\n### Exercise 1 - Checking Data Leakage\nIn the cell below, write a function to check whether there is leakage between two datasets. We'll use this to make sure there are no patients in the test set that are also present in either the train or validation sets.","metadata":{"papermill":{"duration":0.039791,"end_time":"2023-06-03T10:48:31.068144","exception":false,"start_time":"2023-06-03T10:48:31.028353","status":"completed"},"tags":[]}},{"cell_type":"code","source":"\ndef check_for_leakage(df1, df2, patient_col):\n    \"\"\"\n    Return True if there any patients are in both df1 and df2.\n\n    Args:\n        df1 (dataframe): dataframe describing first dataset\n        df2 (dataframe): dataframe describing second dataset\n        patient_col (str): string name of column with patient IDs\n    \n    Returns:\n        leakage (bool): True if there is leakage, otherwise False\n    \"\"\"\n    \n    df1_patients_unique = set(df1[patient_col].values)\n    df2_patients_unique = set(df2[patient_col].values)\n    patients_in_both_groups = df1_patients_unique.intersection(df2_patients_unique)\n    # leakage contains true if there is patient overlap, otherwise false.\n    leakage = len(patients_in_both_groups)>0 \n    return leakage","metadata":{"papermill":{"duration":0.074131,"end_time":"2023-06-03T10:48:31.182388","exception":false,"start_time":"2023-06-03T10:48:31.108257","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:28:11.208588Z","iopub.execute_input":"2023-06-06T05:28:11.208954Z","iopub.status.idle":"2023-06-06T05:28:11.215252Z","shell.execute_reply.started":"2023-06-06T05:28:11.208923Z","shell.execute_reply":"2023-06-06T05:28:11.21429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='2-2'></a>\n### 2.2 Preparing Images\nWith our dataset splits ready, we can now proceed with setting up our model to consume them. \n- For this we will use the off-the-shelf [ImageDataGenerator](https://keras.io/preprocessing/image/) class from the Keras framework, which allows us to build a \"generator\" for images specified in a dataframe. \n- This class also provides support for basic data augmentation such as random horizontal flipping of images.\n- We also use the generator to transform the values in each batch so that their mean is $0$ and their standard deviation is 1. \n    - This will facilitate model training by standardizing the input distribution. \n- The generator also converts our single channel X-ray images (gray-scale) to a three-channel format by repeating the values in the image across all channels.\n    - We will want this because the pre-trained model that we'll use requires three-channel inputs.\n\nSince it is mainly a matter of reading and understanding Keras documentation, we have implemented the generator for you. There are a few things to note: \n1. We normalize the mean and standard deviation of the data\n3. We shuffle the input after each epoch.\n4. We set the image size to be **320px by 320px","metadata":{"papermill":{"duration":0.05902,"end_time":"2023-06-03T10:48:31.312019","exception":false,"start_time":"2023-06-03T10:48:31.252999","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_train_generator(df, image_dir, x_col, y_cols, shuffle=True, batch_size=8, seed=1, target_w = 320, target_h = 320):\n    \"\"\"\n    Return generator for training set, normalizing using batch\n    statistics.\n\n    Args:\n      train_df (dataframe): dataframe specifying training data.\n      image_dir (str): directory where image files are held.\n      x_col (str): name of column in df that holds filenames.\n      y_cols (list): list of strings that hold y labels for images.\n      batch_size (int): images per batch to be fed into model during training.\n      seed (int): random seed.\n      target_w (int): final width of input images.\n      target_h (int): final height of input images.\n    \n    Returns:\n        train_generator (DataFrameIterator): iterator over training set\n    \"\"\"        \n    print(\"getting train generator...\")\n    # normalize images\n    image_generator = ImageDataGenerator(\n        samplewise_center=True,\n        samplewise_std_normalization= True, \n        shear_range=0.1,\n        zoom_range=0.15,\n        rotation_range=5,\n        width_shift_range=0.1,\n        height_shift_range=0.05,\n        horizontal_flip=True, \n        vertical_flip = False, \n        fill_mode = 'reflect')\n    \n    \n    # flow from directory with specified batch size\n    # and target image size\n    generator = image_generator.flow_from_dataframe(\n            dataframe=df,\n            directory=None,\n            x_col=x_col,\n            y_col=y_cols,\n            class_mode=\"raw\",\n            batch_size=batch_size,\n            shuffle=shuffle,\n            seed=seed,\n            target_size=(target_w,target_h))\n    \n    return generator\n\n","metadata":{"papermill":{"duration":0.050297,"end_time":"2023-06-03T10:48:31.403499","exception":false,"start_time":"2023-06-03T10:48:31.353202","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:28:11.216972Z","iopub.execute_input":"2023-06-06T05:28:11.217624Z","iopub.status.idle":"2023-06-06T05:28:11.228024Z","shell.execute_reply.started":"2023-06-06T05:28:11.217593Z","shell.execute_reply":"2023-06-06T05:28:11.226921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Build a separate generator for valid and test sets\n\nNow we need to build a new generator for validation and testing data. \n\n**Why can't we use the same generator as for the training data?**\n\nLook back at the generator we wrote for the training data. \n- It normalizes each image **per batch**, meaning that it uses batch statistics. \n- We should not do this with the test and validation data, since in a real life scenario we don't process incoming images a batch at a time (we process one image at a time). \n- Knowing the average per batch of test data would effectively give our model an advantage.  \n    - The model should not have any information about the test data.\n\nWhat we need to do is normalize incoming test data using the statistics **computed from the training set**. \n* We implement this in the function below. \n* There is one technical note. Ideally, we would want to compute our sample mean and standard deviation using the entire training set. \n* However, since this is extremely large, that would be very time consuming. \n* In the interest of time, we'll take a random sample of the dataset and calcualte the sample mean and sample standard deviation.","metadata":{"papermill":{"duration":0.036858,"end_time":"2023-06-03T10:48:31.477278","exception":false,"start_time":"2023-06-03T10:48:31.44042","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_test_and_valid_generator(valid_df, test_df, train_df, image_dir, x_col, y_cols, sample_size=100, batch_size=8, seed=1, target_w = 320, target_h = 320):\n    \"\"\"\n    Return generator for validation set and test test set using \n    normalization statistics from training set.\n\n    Args:\n      valid_df (dataframe): dataframe specifying validation data.\n      test_df (dataframe): dataframe specifying test data.\n      train_df (dataframe): dataframe specifying training data.\n      image_dir (str): directory where image files are held.\n      x_col (str): name of column in df that holds filenames.\n      y_cols (list): list of strings that hold y labels for images.\n      sample_size (int): size of sample to use for normalization statistics.\n      batch_size (int): images per batch to be fed into model during training.\n      seed (int): random seed.\n      target_w (int): final width of input images.\n      target_h (int): final height of input images.\n    \n    Returns:\n        test_generator (DataFrameIterator) and valid_generator: iterators over test set and validation set respectively\n    \"\"\"\n    print(\"getting train and valid generators...\")\n    # get generator to sample dataset\n    raw_train_generator = ImageDataGenerator().flow_from_dataframe(\n        dataframe=train_df, \n        directory=image_dir, \n        x_col=\"FilePath\", \n        y_col=labels, \n        class_mode=\"raw\", \n        batch_size=sample_size, \n        shuffle=True, \n        target_size=(target_w, target_h))\n    \n    # get data sample\n    batch = raw_train_generator.next()\n    data_sample = batch[0]\n\n    # use sample to fit mean and std for test set generator\n    image_generator = ImageDataGenerator(\n        featurewise_center=True,\n        featurewise_std_normalization= True)\n    \n    # fit generator to sample from training data\n    image_generator.fit(data_sample)\n\n    # get test generator\n    valid_generator = image_generator.flow_from_dataframe(\n            dataframe=valid_df,\n            directory=image_dir,\n            x_col=x_col,\n            y_col=y_cols,\n            class_mode=\"raw\",\n            batch_size=batch_size,\n            shuffle=False,\n            seed=seed,\n            target_size=(target_w,target_h))\n\n    test_generator = image_generator.flow_from_dataframe(\n            dataframe=test_df,\n            directory=image_dir,\n            x_col=x_col,\n            y_col=y_cols,\n            class_mode=\"raw\",\n            batch_size=batch_size,\n            shuffle=False,\n            seed=seed,\n            target_size=(target_w,target_h))\n    return valid_generator, test_generator","metadata":{"papermill":{"duration":0.052803,"end_time":"2023-06-03T10:48:31.567284","exception":false,"start_time":"2023-06-03T10:48:31.514481","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:28:11.229591Z","iopub.execute_input":"2023-06-06T05:28:11.230003Z","iopub.status.idle":"2023-06-06T05:28:11.242137Z","shell.execute_reply.started":"2023-06-06T05:28:11.229972Z","shell.execute_reply":"2023-06-06T05:28:11.241272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_generator = get_train_generator(df = train_set,\n                                      image_dir = None, \n                                      x_col = \"FilePath\",\n                                      y_cols = labels, \n                                      batch_size=BATCH_SIZE,\n                                      target_w = IMAGE_SIZE[0], \n                                      target_h = IMAGE_SIZE[1] \n                                      )\n\nvalid_generator, test_generator= get_test_and_valid_generator(valid_df = valid_set, \n                                                              test_df = test_set, \n                                                              train_df = train_set,\n                                                              image_dir = None, \n                                                              x_col = \"FilePath\", \n                                                              y_cols = labels,\n                                                              batch_size = BATCH_SIZE,\n                                                              target_w = IMAGE_SIZE[0], \n                                                              target_h = IMAGE_SIZE[1])","metadata":{"papermill":{"duration":81.948281,"end_time":"2023-06-03T10:49:53.553064","exception":false,"start_time":"2023-06-03T10:48:31.604783","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:28:11.247975Z","iopub.execute_input":"2023-06-06T05:28:11.248285Z","iopub.status.idle":"2023-06-06T05:30:26.435921Z","shell.execute_reply.started":"2023-06-06T05:28:11.248262Z","shell.execute_reply":"2023-06-06T05:30:26.434933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='2-3'></a>\n### 2.3 Visualizing Images\n\nLet's write a small function `get_label()` that will produce the class of the image as the appended string of the labels. ","metadata":{"papermill":{"duration":0.038794,"end_time":"2023-06-03T10:49:53.630789","exception":false,"start_time":"2023-06-03T10:49:53.591995","status":"completed"},"tags":[]}},{"cell_type":"code","source":"\ndef get_label(y):\n    \"\"\"\n    Returns the appended label list of the given set. \n    \n    y(list) the one hot vector list containing the label encoding. \n    \"\"\"\n    ret_labels = []\n    i = 0\n    for idx in y:\n        if idx:\n            ret_labels.append(labels[i])\n        i += 1\n    if not ret_labels:\n        return 'No Label'\n    else:\n        return '|'.join(ret_labels)\n\n#get one batch of images from the imageset    \nx, y = train_generator.__getitem__(0)\n\n\n\n#show a set of images along with the labels appended at the top as title.\nfig=plt.figure(figsize=(20, 10))\ncolumns = 4; rows =2 \nfor i in tqdm(range(1, columns*rows +1)):\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(x[i-1], cmap = 'gray')\n    plt.title(get_label(y[i-1]))\n    plt.axis(False)\n    fig.add_subplot","metadata":{"papermill":{"duration":2.684151,"end_time":"2023-06-03T10:49:56.354638","exception":false,"start_time":"2023-06-03T10:49:53.670487","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:30:26.437293Z","iopub.execute_input":"2023-06-06T05:30:26.437762Z","iopub.status.idle":"2023-06-06T05:30:29.676691Z","shell.execute_reply.started":"2023-06-06T05:30:26.437726Z","shell.execute_reply":"2023-06-06T05:30:29.67585Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='3'></a>\n## 3. Class Imbalence\n\nNow we'll move on to model training and development. We have a few practical challenges to deal with before actually training a neural network, though. The first is class imbalance.\n\n<a name='3-1'></a>\n### 3.1 Addressing Class Imbalance\nOne of the challenges with working with medical diagnostic datasets is the large class imbalance present in such datasets. Let's plot the frequency of each of the labels in our dataset:","metadata":{"papermill":{"duration":0.044711,"end_time":"2023-06-03T10:49:56.445324","exception":false,"start_time":"2023-06-03T10:49:56.400613","status":"completed"},"tags":[]}},{"cell_type":"code","source":"plt.figure(figsize=(8,4))\nplt.xticks(rotation = 90)\nplt.bar(labels, train_generator.labels.sum(axis = 0)/train_generator.n * 100)\nplt.title('Percentage ofdifferent conditions in train dataset')\nplt.xlabel('Conditions')\nplt.ylabel('Percentage')\nplt.show()","metadata":{"papermill":{"duration":0.237786,"end_time":"2023-06-03T10:49:56.728161","exception":false,"start_time":"2023-06-03T10:49:56.490375","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:30:29.677638Z","iopub.execute_input":"2023-06-06T05:30:29.677947Z","iopub.status.idle":"2023-06-06T05:30:30.000886Z","shell.execute_reply.started":"2023-06-06T05:30:29.67792Z","shell.execute_reply":"2023-06-06T05:30:29.999856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see from this plot that the prevalance of positive cases varies significantly across the different pathologies. (These trends mirror the ones in the full dataset as well.) \n* The `Hernia` pathology has the greatest imbalance with the proportion of positive training cases being about 0.2%. \n* But even the `Infiltration` pathology, which has the least amount of imbalance, has only 17.5% of the training cases labelled positive.\n\nIdeally, we would train our model using an evenly balanced dataset so that the positive and negative training cases would contribute equally to the loss. \n\nIf we use a normal cross-entropy loss function with a highly unbalanced dataset, as we are seeing here, then the algorithm will be incentivized to prioritize the majority class (i.e negative in our case), since it contributes more to the loss. \n\n\n#### Impact of class imbalance on loss function\n\nLet's take a closer look at this. Assume we would have used a normal cross-entropy loss for each pathology. We recall that the cross-entropy loss contribution from the $i^{th}$ training data case is:\n\n$$\\mathcal{L}_{cross-entropy}(x_i) = -(y_i \\log(f(x_i)) + (1-y_i) \\log(1-f(x_i))),$$\n\nwhere $x_i$ and $y_i$ are the input features and the label, and $f(x_i)$ is the output of the model, i.e. the probability that it is positive. \n\nNote that for any training case, either $y_i=0$ or else $(1-y_i)=0$, so only one of these terms contributes to the loss (the other term is multiplied by zero, and becomes zero). \n\nWe can rewrite the overall average cross-entropy loss over the entire training set $\\mathcal{D}$ of size $N$ as follows: \n\n$$\\mathcal{L}_{cross-entropy}(\\mathcal{D}) = - \\frac{1}{N}\\big( \\sum_{\\text{positive examples}} \\log (f(x_i)) + \\sum_{\\text{negative examples}} \\log(1-f(x_i)) \\big).$$\n\nUsing this formulation, we can see that if there is a large imbalance with very few positive training cases, for example, then the loss will be dominated by the negative class. Summing the contribution over all the training cases for each class (i.e. pathological condition), we see that the contribution of each class (i.e. positive or negative) is: \n\n$$freq_{p} = \\frac{\\text{number of positive examples}}{N} $$\n\n$$\\text{and}$$\n\n$$freq_{n} = \\frac{\\text{number of negative examples}}{N}.$$","metadata":{"papermill":{"duration":0.045922,"end_time":"2023-06-03T10:49:56.822213","exception":false,"start_time":"2023-06-03T10:49:56.776291","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<a name='3-2'></a>\n### 3.2 Computing and Visualizing Class Imbalance\nLet's take the whole dataset and check the number of the different conditions identified among the x-rays. After that we plot the relative percentage of existance of different conditions in the image. ","metadata":{"papermill":{"duration":0.046596,"end_time":"2023-06-03T10:49:56.915632","exception":false,"start_time":"2023-06-03T10:49:56.869036","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def compute_class_freqs(labels):\n    \"\"\"\n    Compute positive and negative frequences for each class.\n\n    Args:\n        labels (np.array): matrix of labels, size (num_examples, num_classes)\n    Returns:\n        positive_frequencies (np.array): array of positive frequences for each\n                                         class, size (num_classes)\n        negative_frequencies (np.array): array of negative frequences for each\n                                         class, size (num_classes)\n    \"\"\"    \n    # total number of patients (rows)\n    N = labels.shape[0]\n    positive_frequencies = (labels.sum(axis = 0))/N\n    negative_frequencies = 1.0 - positive_frequencies\n    \n    return positive_frequencies, negative_frequencies\n\n\n# calulating and plotting the imbalanced classes\nfreq_pos, freq_neg = compute_class_freqs(train_generator.labels)\ndata = pd.DataFrame({\"Class\": labels, \"Label\": \"Positive\", \"Value\": freq_pos})\ndata = data.append([{\"Class\": labels[l], \"Label\": \"Negative\", \"Value\": v} for l,v in enumerate(freq_neg)], ignore_index=True)\nplt.xticks(rotation=90)\nf = sns.barplot(x=\"Class\", y=\"Value\", hue=\"Label\" ,data=data)","metadata":{"papermill":{"duration":0.389392,"end_time":"2023-06-03T10:49:57.351591","exception":false,"start_time":"2023-06-03T10:49:56.962199","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:30:30.002286Z","iopub.execute_input":"2023-06-06T05:30:30.002738Z","iopub.status.idle":"2023-06-06T05:30:30.564476Z","shell.execute_reply.started":"2023-06-06T05:30:30.002702Z","shell.execute_reply":"2023-06-06T05:30:30.563429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we see in the above plot, the contributions of positive cases is significantly lower than that of the negative ones. However, we want the contributions to be equal. One way of doing this is by multiplying each example from each class by a class-specific weight factor, $w_{pos}$ and $w_{neg}$, so that the overall contribution of each class is the same. \n\nTo have this, we want \n\n$$w_{pos} \\times freq_{p} = w_{neg} \\times freq_{n},$$\n\nwhich we can do simply by taking \n\n$$w_{pos} = freq_{neg}$$\n$$w_{neg} = freq_{pos}$$\n\nThis way, we will be balancing the contribution of positive and negative labels.\n\nNow lets balance the classes  and then use them for visualization \n\n<a name='3-3'></a>\n### 3.3 Dealing with Class Imbalance","metadata":{"papermill":{"duration":0.046921,"end_time":"2023-06-03T10:49:57.447151","exception":false,"start_time":"2023-06-03T10:49:57.40023","status":"completed"},"tags":[]}},{"cell_type":"code","source":"pos_weights = freq_neg\nneg_weights = freq_pos\npos_contribution = freq_pos * pos_weights \nneg_contribution = freq_neg * neg_weights\npos_weights\n\n\ndata = pd.DataFrame({\"Class\": labels, \"Label\": \"Positive\", \"Value\": pos_contribution})\ndata = data.append([{\"Class\": labels[l], \"Label\": \"Negative\", \"Value\": v} \n                        for l,v in enumerate(neg_contribution)], ignore_index=True)\nplt.xticks(rotation=90)\nsns.barplot(x=\"Class\", y=\"Value\", hue=\"Label\" ,data=data);","metadata":{"papermill":{"duration":0.355945,"end_time":"2023-06-03T10:49:57.85023","exception":false,"start_time":"2023-06-03T10:49:57.494285","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:30:30.56619Z","iopub.execute_input":"2023-06-06T05:30:30.56653Z","iopub.status.idle":"2023-06-06T05:30:31.050673Z","shell.execute_reply.started":"2023-06-06T05:30:30.566494Z","shell.execute_reply":"2023-06-06T05:30:31.049785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='3-4'></a>\n### 3.4 Calculating the weighted loss\n\nAs the above figure shows, by applying these weightings the positive and negative labels within each class would have the same aggregate contribution to the loss function. Now let's implement such a loss function. \n\nAfter computing the weights, our final weighted loss for each training case will be \n\n$$\\mathcal{L}_{cross-entropy}^{w}(x) = - (w_{p} y \\log(f(x)) + w_{n}(1-y) \\log( 1 - f(x) ) ).$$","metadata":{"papermill":{"duration":0.047641,"end_time":"2023-06-03T10:49:57.946275","exception":false,"start_time":"2023-06-03T10:49:57.898634","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def get_weighted_loss(pos_weights, neg_weights, epsilon=1e-7):\n    \"\"\"\n    Return weighted loss function given negative weights and positive weights.\n\n    Args:\n      pos_weights (np.array): array of positive weights for each class, size (num_classes)\n      neg_weights (np.array): array of negative weights for each class, size (num_classes)\n    \n    Returns:\n      weighted_loss (function): weighted loss function\n    \"\"\"\n    def weighted_loss(y_true, y_pred):\n        \"\"\"\n        Return weighted loss value. \n\n        Args:\n            y_true (Tensor): Tensor of true labels, size is (num_examples, num_classes)\n            y_pred (Tensor): Tensor of predicted labels, size is (num_examples, num_classes)\n        Returns:\n            loss (Float): overall scalar loss summed across all classes\n        \"\"\"\n        # initialize loss to zero\n        loss = 0.0\n        y_true = tf.cast(y_true, dtype=tf.float32)\n        \n        for i in range(len(pos_weights)):\n            # for each class, add average weighted loss for that class \n            loss_pos = -1 * K.mean(pos_weights[i] * y_true[:, i] * K.log(y_pred[:, i] + epsilon))\n            loss_neg = -1 * K.mean(neg_weights[i] * (1 - y_true[:, i]) * K.log(1 - y_pred[:, i] + epsilon))\n            loss += loss_pos + loss_neg\n        return loss\n\n    return weighted_loss","metadata":{"papermill":{"duration":0.059662,"end_time":"2023-06-03T10:49:58.053896","exception":false,"start_time":"2023-06-03T10:49:57.994234","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:44:45.254113Z","iopub.execute_input":"2023-06-06T05:44:45.25449Z","iopub.status.idle":"2023-06-06T05:44:45.263061Z","shell.execute_reply.started":"2023-06-06T05:44:45.254461Z","shell.execute_reply":"2023-06-06T05:44:45.261872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name=''>","metadata":{"papermill":{"duration":0.057823,"end_time":"2023-06-03T10:49:58.159031","exception":false,"start_time":"2023-06-03T10:49:58.101208","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"<a name='4'> </a>\n# 4. Model Creation\nFor the purpose of simplicity, we will use DenseNet121 model from the `keras.application` package. Then we will load the pretrained weights and have a look at the **ROC** curve and **AUROC** values for the different diagonostic categories. ","metadata":{"papermill":{"duration":0.047445,"end_time":"2023-06-03T10:49:58.25477","exception":false,"start_time":"2023-06-03T10:49:58.207325","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# with strategy.scope():\n#     dnet121 = DenseNet121(input_shape=(*IMAGE_SIZE, 3),\n#                           weights='imagenet',\n#                           include_top=False )\n#     dnet121.trainable = True\n\n#     model_dnet121 = tf.keras.Sequential([ dnet121, \n#                                          Layers.GlobalAveragePooling2D(), \n#                                          Layers.Dense(len(labels), activation ='sigmoid') ])\n\n#     model_dnet121.compile(optimizer='adam',\n#                            loss = get_weighted_loss(pos_weights, neg_weights), \n#                            metrics = ['accuracy'] )`\n\n#     model_dnet121.summary()\n\n# history = model_dnet121.fit_generator(train_generator, \n#                               validation_data=valid_generator,\n#                               steps_per_epoch=100, \n#                               validation_steps=25, \n#                               epochs = 3)","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"papermill":{"duration":0.054329,"end_time":"2023-06-03T10:49:58.356769","exception":false,"start_time":"2023-06-03T10:49:58.30244","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:30:31.063117Z","iopub.execute_input":"2023-06-06T05:30:31.063551Z","iopub.status.idle":"2023-06-06T05:30:31.074263Z","shell.execute_reply.started":"2023-06-06T05:30:31.063519Z","shell.execute_reply":"2023-06-06T05:30:31.073286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras.layers import Input, Conv2D, BatchNormalization, ReLU, Concatenate, GlobalAveragePooling2D, Dense\nfrom tensorflow.keras.models import Model\n","metadata":{"execution":{"iopub.status.busy":"2023-06-06T05:32:54.868686Z","iopub.execute_input":"2023-06-06T05:32:54.869453Z","iopub.status.idle":"2023-06-06T05:32:54.874611Z","shell.execute_reply.started":"2023-06-06T05:32:54.869387Z","shell.execute_reply":"2023-06-06T05:32:54.873697Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n  \n    def conv_block(x, growth_rate):\n        x1 = BatchNormalization()(x)\n        x1 = ReLU()(x1)\n        x1 = Conv2D(filters=growth_rate, kernel_size=(3, 3), padding='same')(x1)\n        x = Concatenate()([x, x1])\n        return x\n\n    def dense_block(x, num_layers, growth_rate):\n        for _ in range(num_layers):\n            x = conv_block(x, growth_rate)\n        return x\n\n    def transition_block(x, reduction):\n        x = BatchNormalization()(x)\n        x = ReLU()(x)\n        x = Conv2D(int(tf.keras.backend.int_shape(x)[-1] * reduction), kernel_size=(1, 1), padding='same')(x)\n        x = tf.keras.layers.AveragePooling2D((2, 2), strides=(2, 2))(x)\n        return x\n\n    def CustomNet121(input_shape=(224, 224, 3), num_classes=1000, growth_rate=32, num_blocks=[6, 12, 24, 16], reduction=0.5):\n        inputs = Input(shape=input_shape)\n        x = Conv2D(64, kernel_size=(7, 7), strides=(2, 2), padding='same')(inputs)\n        x = BatchNormalization()(x)\n        x = ReLU()(x)\n        x = tf.keras.layers.MaxPooling2D(pool_size=(3, 3), strides=(2, 2), padding='same')(x)\n\n        num_features = 64\n        for i, num_layers in enumerate(num_blocks):\n            x = dense_block(x, num_layers, growth_rate)\n            num_features += num_layers * growth_rate\n            if i != len(num_blocks) - 1:\n                x = transition_block(x, reduction)\n\n        x = BatchNormalization()(x)\n        x = ReLU()(x)\n        x = GlobalAveragePooling2D()(x)\n        x = Dense(num_classes, activation='softmax')(x)\n\n        model = Model(inputs, x, name='CustomNet-121')\n        return model\n    model = CustomNet121(input_shape = (128,128,3),num_classes=len(labels))\n    \n    \nmodel.compile(\n    optimizer=tf.keras.optimizers.Adam( learning_rate=1e-4, amsgrad=False), \n    #loss = 'binary_crossentropy',\n    loss = get_weighted_loss(pos_weights, neg_weights),\n    metrics = ['binary_accuracy']\n)\nmodel.summary()\n","metadata":{"papermill":{"duration":11.706531,"end_time":"2023-06-03T10:50:10.111158","exception":false,"start_time":"2023-06-03T10:49:58.404627","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:45:04.686293Z","iopub.execute_input":"2023-06-06T05:45:04.686646Z","iopub.status.idle":"2023-06-06T05:45:06.747339Z","shell.execute_reply.started":"2023-06-06T05:45:04.686618Z","shell.execute_reply":"2023-06-06T05:45:06.746543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"weights_path = \"../input/weights2/den121.h5\"  \nmodel.load_weights(weights_path, by_name=True, skip_mismatch=True)","metadata":{"execution":{"iopub.status.busy":"2023-06-06T05:45:11.554364Z","iopub.execute_input":"2023-06-06T05:45:11.554749Z","iopub.status.idle":"2023-06-06T05:45:12.289783Z","shell.execute_reply.started":"2023-06-06T05:45:11.554718Z","shell.execute_reply":"2023-06-06T05:45:12.288812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def build_lrfn(lr_start=0.000002, lr_max=0.00010, \n               lr_min=0, lr_rampup_epochs=8, \n               lr_sustain_epochs=0, lr_exp_decay=.8):\n\n    def lrfn(epoch):\n        if epoch < lr_rampup_epochs:\n            lr = (lr_max - lr_start) / lr_rampup_epochs * epoch + lr_start\n        elif epoch < lr_rampup_epochs + lr_sustain_epochs:\n            lr = lr_max\n        else:\n            lr = (lr_max - lr_min) *\\\n                 lr_exp_decay**(epoch - lr_rampup_epochs\\\n                                - lr_sustain_epochs) + lr_min\n        return lr\n    return lrfn\n\nlrfn = build_lrfn()\nlr_schedule = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)","metadata":{"papermill":{"duration":0.104206,"end_time":"2023-06-03T10:50:10.309377","exception":false,"start_time":"2023-06-03T10:50:10.205171","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:45:12.29396Z","iopub.execute_input":"2023-06-06T05:45:12.294264Z","iopub.status.idle":"2023-06-06T05:45:12.302423Z","shell.execute_reply.started":"2023-06-06T05:45:12.294238Z","shell.execute_reply":"2023-06-06T05:45:12.301314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = model.fit_generator(train_generator, \n                              validation_data=valid_generator,\n                              steps_per_epoch=len(train_generator), \n                              validation_steps=len(valid_generator), \n                              epochs = EPOCHS,\n                              callbacks=[lr_schedule]\n                             )","metadata":{"papermill":{"duration":10936.111579,"end_time":"2023-06-03T13:52:26.472074","exception":false,"start_time":"2023-06-03T10:50:10.360495","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2023-06-06T05:45:13.053804Z","iopub.execute_input":"2023-06-06T05:45:13.054639Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_vals_before = model.predict_generator(test_generator, steps = len(test_generator))","metadata":{"papermill":{"duration":189.309771,"end_time":"2023-06-03T13:55:38.138698","exception":false,"start_time":"2023-06-03T13:52:28.828927","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize_training(history, lw = 3):\n    plt.figure(figsize=(10,6))\n    plt.plot(history.history['binary_accuracy'], label = 'training', marker = '*', linewidth = lw)\n    plt.plot(history.history['val_binary_accuracy'], label = 'validation', marker = 'o', linewidth = lw)\n    plt.title('Training Accuracy vs Validation Accuracy')\n    plt.xlabel('Epochs')\n    plt.ylabel('Accuracy')\n    plt.legend(fontsize = 'x-large')\n    plt.show()\n\n    plt.figure(figsize=(10,6))\n    plt.plot(history.history['loss'], label = 'training', marker = '*', linewidth = lw)\n    plt.plot(history.history['val_loss'], label = 'validation', marker = 'o', linewidth = lw)\n    plt.title('Training Loss vs Validation Loss')\n    plt.xlabel('Epochs')\n    plt.ylabel('Loss')\n    plt.legend(fontsize = 'x-large')\n    plt.show()\nvisualize_training(history)    ","metadata":{"papermill":{"duration":2.801094,"end_time":"2023-06-03T13:55:43.233748","exception":false,"start_time":"2023-06-03T13:55:40.432654","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\nhistory = model.fit_generator(train_generator, \n                              validation_data=valid_generator,\n                              steps_per_epoch=len(train_generator), \n                              validation_steps=len(valid_generator), \n                              epochs = 9,\n                              callbacks=[lr_schedule]\n                             )","metadata":{"papermill":{"duration":4992.007794,"end_time":"2023-06-03T15:18:58.010157","exception":false,"start_time":"2023-06-03T13:55:46.002363","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_training(history)","metadata":{"papermill":{"duration":3.639285,"end_time":"2023-06-03T15:19:04.968804","exception":false,"start_time":"2023-06-03T15:19:01.329519","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_vals_after = model.predict_generator(test_generator, steps = len(test_generator))\nauc_rocs_before =get_roc_curve(labels, predicted_vals_before, test_generator, when = 'before training')","metadata":{"papermill":{"duration":194.437246,"end_time":"2023-06-03T15:22:23.28265","exception":false,"start_time":"2023-06-03T15:19:08.845404","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\nhistory = model.fit_generator(train_generator, \n                              validation_data=valid_generator,\n                              steps_per_epoch=len(train_generator), \n                              validation_steps=len(valid_generator),\n                              initial_epoch = 9,\n                              epochs = 20,\n                              callbacks=[lr_schedule]\n                             )","metadata":{"papermill":{"duration":6443.150961,"end_time":"2023-06-03T17:09:49.888018","exception":false,"start_time":"2023-06-03T15:22:26.737057","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_training(history)","metadata":{"papermill":{"duration":4.879039,"end_time":"2023-06-03T17:09:59.946948","exception":false,"start_time":"2023-06-03T17:09:55.067909","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predicted_vals_after = model.predict_generator(test_generator, steps = len(test_generator))\nauc_rocs_before =get_roc_curve(labels, predicted_vals_before, test_generator, when = 'before training')","metadata":{"papermill":{"duration":183.935027,"end_time":"2023-06-03T17:13:08.571199","exception":false,"start_time":"2023-06-03T17:10:04.636172","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# # create the base pre-trained model\n# base_model = DenseNet121(weights='imagenet', include_top=False)\n# # base_model = DenseNet121(weights='../input/chestxray8-dataframe/pretrained_model.h5', include_top=False)\n\n# x = base_model.output\n# # add a global spatial average pooling layer\n# x = GlobalAveragePooling2D()(x)\n\n# # and a logistic layer\n# predictions = Dense(len(labels), activation=\"sigmoid\")(x)\n\n# model = Model(inputs=base_model.input, outputs=predictions)\n# model.compile(optimizer='adam', loss=get_weighted_loss(pos_weights, neg_weights), metrics = ['accuracy'])\n# #Loading the pretrained weights from the custom dataset\n# model.load_weights('../input/chestxray8-dataframe/pretrained_model.h5')\n# predicted_vals_before = model.predict_generator(test_generator, steps = len(test_generator))\n\n\n\n# history = model.fit_generator(train_generator, \n#                               validation_data=valid_generator,\n#                               steps_per_epoch=len(train_generator), \n#                               validation_steps=len(valid_generator), \n#                               epochs = 5)","metadata":{"_kg_hide-input":true,"papermill":{"duration":4.595732,"end_time":"2023-06-03T17:13:17.92039","exception":false,"start_time":"2023-06-03T17:13:13.324658","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's train the model for a few epochs. ","metadata":{"papermill":{"duration":4.583197,"end_time":"2023-06-03T17:13:27.205997","exception":false,"start_time":"2023-06-03T17:13:22.6228","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"## Training Summary","metadata":{"papermill":{"duration":4.994331,"end_time":"2023-06-03T17:13:36.915931","exception":false,"start_time":"2023-06-03T17:13:31.9216","status":"completed"},"tags":[]}},{"cell_type":"code","source":"def visualize_training(history, lw = 3):\n    plt.figure(figsize=(10,6))\n    plt.plot(history.history['binary_accuracy'], label = 'training', marker = '*', linewidth = lw)\n    plt.plot(history.history['val_binary_accuracy'], label = 'validation', marker = 'o', linewidth = lw)\n    plt.title('Training Accuracy vs Validation Accuracy')\n    plt.xlabel('Epochs')\n    plt.ylabel('Accuracy')\n    plt.legend(fontsize = 'x-large')\n    plt.show()\n\n    plt.figure(figsize=(10,6))\n    plt.plot(history.history['loss'], label = 'training', marker = '*', linewidth = lw)\n    plt.plot(history.history['val_loss'], label = 'validation', marker = 'o', linewidth = lw)\n    plt.title('Training Loss vs Validation Loss')\n    plt.xlabel('Epochs')\n    plt.ylabel('Loss')\n    plt.legend(fontsize = 'x-large')\n    plt.show()","metadata":{"_kg_hide-input":true,"papermill":{"duration":4.749587,"end_time":"2023-06-03T17:13:46.233213","exception":false,"start_time":"2023-06-03T17:13:41.483626","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_training(history)","metadata":{"papermill":{"duration":4.931272,"end_time":"2023-06-03T17:13:55.700283","exception":false,"start_time":"2023-06-03T17:13:50.769011","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='5'></a>\n## 5. Prediction and Evaluation\nNow that we have a model, let's evaluate it using our test set. We can conveniently use the `predict_generator` function to generate the predictions for the images in our test set.\n\n<a name='5-1'></a>\n### 5.1 ROC Curve and AUROC\nWe'll cover topic of model evaluation in much more detail in later weeks, but for now we'll walk through computing a metric called the AUC (Area Under the Curve) from the ROC ([Receiver Operating Characteristic](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)) curve. This is also referred to as the AUROC value, but you will see all three terms in reference to the technique, and often used almost interchangeably. \n\nFor now, what you need to know in order to interpret the plot is that a curve that is more to the left and the top has more \"area\" under it, and indicates that the model is performing better.\n\nWe will use the `util.get_roc_curve()` function which has been provided for you in `util.py`. Look through this function and note the use of the `sklearn` library functions to generate the ROC curves and AUROC values for our model. \n\n- [roc_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_curve.html)\n- [roc_auc_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html)\n\n<a name='5-2'></a>\n### 5.2 Comparing ROC Curves","metadata":{"papermill":{"duration":4.813944,"end_time":"2023-06-03T17:14:05.169788","exception":false,"start_time":"2023-06-03T17:14:00.355844","status":"completed"},"tags":[]}},{"cell_type":"code","source":"predicted_vals_after = model.predict_generator(test_generator, steps = len(test_generator))\nauc_rocs_before =get_roc_curve(labels, predicted_vals_before, test_generator, when = 'before training')\nauc_rocs_after = get_roc_curve(labels, predicted_vals_after, test_generator, when = 'after training')","metadata":{"papermill":{"duration":116.920435,"end_time":"2023-06-03T17:16:06.969152","exception":false,"start_time":"2023-06-03T17:14:10.048717","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='5-3'></a>\n## 5.3 Comparing AUROC Values\nNow lets have a look at the change of individual **ROCAUC** change after training the network. ","metadata":{"papermill":{"duration":5.125453,"end_time":"2023-06-03T17:16:16.840664","exception":false,"start_time":"2023-06-03T17:16:11.715211","status":"completed"},"tags":[]}},{"cell_type":"code","source":"ind = np.arange(len(labels))\nplt.figure(figsize=(15,7))\nwidth = 0.2       \nplt.bar(ind, auc_rocs_before , width, label='Before')\nplt.bar(ind + width, auc_rocs_after, width, label='After')\nplt.ylabel('AUROC value', fontsize = 16)\nplt.title('AUROC of each diagnosis before and after training', fontsize = 18)\nplt.xticks(ind + width / 2, labels, rotation = 90, fontsize = 14)\nplt.yticks(fontsize = 14)\nplt.legend(loc='best')\nplt.grid(True)\nplt.show()","metadata":{"papermill":{"duration":4.824989,"end_time":"2023-06-03T17:16:26.250558","exception":false,"start_time":"2023-06-03T17:16:21.425569","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a name='6'></a>\n# 6. Saving Weights and Training History\nLet's save the newly trained weights and the trainigng history for the future use. ","metadata":{"papermill":{"duration":4.551755,"end_time":"2023-06-03T17:16:35.529325","exception":false,"start_time":"2023-06-03T17:16:30.97757","status":"completed"},"tags":[]}},{"cell_type":"code","source":"model.save_weights('efficent_net_b1_trained_weights.h5')\npd.DataFrame.from_dict(history.history).to_csv('efficent_net_b1_training_hisotry.csv', index = False)\n","metadata":{"papermill":{"duration":5.746107,"end_time":"2023-06-03T17:16:45.912362","exception":false,"start_time":"2023-06-03T17:16:40.166255","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df","metadata":{"papermill":{"duration":4.59971,"end_time":"2023-06-03T17:16:55.222166","exception":false,"start_time":"2023-06-03T17:16:50.622456","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"papermill":{"duration":4.63474,"end_time":"2023-06-03T17:17:04.501913","exception":false,"start_time":"2023-06-03T17:16:59.867173","status":"completed"},"tags":[]},"execution_count":null,"outputs":[]}]}