{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"# Intro\nIn this notebook I'd explore some data quality issues of the dataset. For that, I'd use images which appeared to be among the most challenging for the classifier. To let the classifier make excuses for its mistakes (and to use the excuses as clues to data quality), we'd use Guided GradCAM."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"# Plan\n1. [Preparing everything necessary](#Preparing-everything-necessary)\n2. [Inspecting False Positives](#Inspecting-False-Positives)\n  * [Confusion matrix](#Confusion-matrix)\n  * [Visualizations](#Visualizations)\n    * [Helper functions](#Helper-functions)\n      * [Guided Backprop routines](#Guided-Backprop-routines)\n      * [Functions checking distribution of labels for challenging images](#Functions-checking-distribution-of-labels-for-challenging-images)\n      * [Functions for plotting mistakes alongside with correct ones for comparison](#Functions-for-plotting-mistakes-alongside-with-correct-ones-for-comparison)\n    * [Fish](#Fish)\n      * [Visual comparison of class representatives with challenging images](#Visual-comparison-of-class-representatives-with-challenging-images)\n      * [Distribution of labels in Fish-false-positive images](#Distribution-of-labels-in-Fish-false-positive-images)\n    * [Exploring dark images](#Exploring-dark-images)\n    * [Flower](#Flower)\n      * [Visual comparison of class representatives with challenging images](#Visual-comparison-of-class-representatives-with-challenging-images)\n      * [Distribution of labels in Flower-false-positive images](#Distribution-of-labels-in-Flower-false-positive-images)\n    * [Sugar](#Sugar)\n      * [Visual comparison of class representatives with challenging images](#Visual-comparison-of-class-representatives-with-challenging-images)\n      * [Distribution of labels in Sugar-false-positive images](#Distribution-of-labels-in-Sugar-false-positive-images)\n    * [Gravel](#Gravel)\n      * [Visual comparison of class representatives with challenging images](#Visual-comparison-of-class-representatives-with-challenging-images)\n      * [Distribution of labels in Gravel-false-positive images](#Distribution-of-labels-in-Gravel-false-positive-images)\n3. [Conclusion](#Conclusion)"},{"metadata":{},"cell_type":"markdown","source":"# Preparing everything necessary"},{"metadata":{},"cell_type":"markdown","source":"## Libraries"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os, glob\nimport random\nfrom sklearn.model_selection import train_test_split\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport multiprocessing\nfrom copy import deepcopy\nfrom sklearn.metrics import precision_recall_curve, auc, multilabel_confusion_matrix\nimport tensorflow.keras as keras\nfrom tensorflow.keras.models import Model, load_model\nfrom tensorflow.keras.utils import Sequence\nimport matplotlib.pyplot as plt\nfrom IPython.display import Image\nfrom tqdm import tqdm_notebook as tqdm\nimport itertools\nimport tensorflow as tf\nfrom tensorflow.python.framework import ops\nfrom tensorflow.keras import backend as K\nfrom numpy.random import seed\nseed(10)\nfrom tensorflow import set_random_seed\nset_random_seed(10)\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test_imgs_folder = '../input/understanding_cloud_organization/test_images/'\ntrain_imgs_folder = '../input/understanding_cloud_organization/train_images/'\nnum_cores = multiprocessing.cpu_count()\nimage_width, image_height = 224, 224","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Generators"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain_df_orig = pd.read_csv('../input/understanding_cloud_organization/train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_df = train_df[~train_df['EncodedPixels'].isnull()]\ntrain_df['Image'] = train_df['Image_Label'].map(lambda x: x.split('_')[0])\ntrain_df['Class'] = train_df['Image_Label'].map(lambda x: x.split('_')[1])\nclasses = train_df['Class'].unique()\ntrain_df = train_df.groupby('Image')['Class'].agg(set).reset_index()\nfor class_name in classes:\n    train_df[class_name] = train_df['Class'].map(lambda x: 1 if class_name in x else 0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# dictionary for fast access to ohe vectors\nimg_2_ohe_vector = {img:vec for img, vec in zip(train_df['Image'], train_df.iloc[:, 2:].values)}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Stratified split into train/val"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_imgs, val_imgs = train_test_split(train_df['Image'].values, \n                                        test_size=0.2, \n                                        stratify=train_df['Class'].map(lambda x: str(sorted(list(x)))), # sorting present classes in lexicographical order, just to be sure\n                                        random_state=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generator class"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class DataGenenerator(Sequence):\n    def __init__(self, images_list=None, folder_imgs=train_imgs_folder, \n                 batch_size=32, shuffle=True, augmentation=None,\n                 resized_height=224, resized_width=224, num_channels=3):\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.augmentation = augmentation\n        if images_list is None:\n            self.images_list = os.listdir(folder_imgs)\n        else:\n            self.images_list = deepcopy(images_list)\n        self.folder_imgs = folder_imgs\n        self.len = len(self.images_list) // self.batch_size\n        self.resized_height = resized_height\n        self.resized_width = resized_width\n        self.num_channels = num_channels\n        self.num_classes = 4\n        self.is_test = not 'train' in folder_imgs\n        if not shuffle and not self.is_test:\n            self.labels = [img_2_ohe_vector[img] for img in self.images_list[:self.len*self.batch_size]]\n\n    def __len__(self):\n        return self.len\n    \n    def on_epoch_start(self):\n        if self.shuffle:\n            random.shuffle(self.images_list)\n\n    def __getitem__(self, idx):\n        current_batch = self.images_list[idx * self.batch_size: (idx + 1) * self.batch_size]\n        X = np.empty((self.batch_size, self.resized_height, self.resized_width, self.num_channels))\n        y = np.empty((self.batch_size, self.num_classes))\n\n        for i, image_name in enumerate(current_batch):\n            path = os.path.join(self.folder_imgs, image_name)\n            img = cv2.resize(cv2.imread(path), (self.resized_height, self.resized_width)).astype(np.float32)\n            if not self.augmentation is None:\n                augmented = self.augmentation(image=img)\n                img = augmented['image']\n            X[i, :, :, :] = img/255.0\n            if not self.is_test:\n                y[i, :] = img_2_ohe_vector[image_name]\n        return X, y\n\n    def get_labels(self):\n        if self.shuffle:\n            images_current = self.images_list[:self.len*self.batch_size]\n            labels = [img_2_ohe_vector[img] for img in images_current]\n        else:\n            labels = self.labels\n        return np.array(labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generator instances"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"data_generator_val = DataGenenerator(val_imgs, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Val predictions and ground truth"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"model = load_model('../input/clouds-classifier-files/classifier_epoch_45_val_pr_auc_0.8344173287108075.h5')\ny_pred = model.predict_generator(data_generator_val, workers=num_cores)\ny_true = data_generator_val.get_labels()\nmodel_class_names =  ['Fish', 'Flower', 'Sugar', 'Gravel']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Mask routines"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# helper functions\n# credits: https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools?scriptVersionId=20202006\ndef rle_decode(mask_rle: str = '', shape: tuple = (1400, 2100)):\n    '''\n    Decode rle encoded mask.\n    \n    :param mask_rle: run-length as string formatted (start length)\n    :param shape: (height, width) of array to return \n    Returns numpy array, 1 - mask, 0 - background\n    '''\n    s = mask_rle.split()\n    starts, lengths = [np.asarray(x, dtype=int) for x in (s[0:][::2], s[1:][::2])]\n    starts -= 1\n    ends = starts + lengths\n    img = np.zeros(shape[0] * shape[1], dtype=np.uint8)\n    for lo, hi in zip(starts, ends):\n        img[lo:hi] = 1\n    return img.reshape(shape, order='F')\n\ndef make_mask(df, image_label, shape: tuple = (1400, 2100)):\n    \"\"\"\n    Create mask based on df, image name and shape.\n    \"\"\"\n    df = df.set_index('Image_Label')\n    encoded_mask = df.loc[image_label, 'EncodedPixels']\n    mask = np.zeros((shape[0], shape[1]), dtype=np.float32)\n    if encoded_mask is not np.nan:\n        mask = rle_decode(encoded_mask)\n            \n    return cv2.resize(mask, (image_height, image_width))\n\nval_masks_np = np.empty((len(model_class_names), len(val_imgs), image_height, image_width))\nfor class_i, class_name in enumerate(tqdm(model_class_names)):\n    for img_i, img_name in enumerate(val_imgs):\n        mask = make_mask(train_df_orig, img_name + '_' + class_name)\n        val_masks_np[class_i][img_i] = mask","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Inspecting False Positives"},{"metadata":{},"cell_type":"markdown","source":"## Confusion matrix"},{"metadata":{},"cell_type":"markdown","source":"To better understand which classes are difficult for the classifier, let's create a confusion matrix. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# inspired by similar plots in PLAsTiCC Astronomical Classification\ndef plot_confusion_matrix(y_true, y_pred,\n                          cmap=plt.cm.Blues,\n                          normalize=False\n                         ):\n    confusion_matrices = multilabel_confusion_matrix(y_true, y_pred)\n    for class_name, conf_matrix in zip(model_class_names, confusion_matrices):\n        plt.figure()\n        if normalize:\n            conf_matrix = np.divide(conf_matrix, conf_matrix.sum(axis=1).repeat(2).reshape(2, 2))\n        \n        plt.imshow(conf_matrix, cmap=plt.cm.Blues)\n        plt.colorbar()\n        tick_marks = np.arange(-.5, 1.51, 0.5)\n        plt.xticks(tick_marks, ['', 'the rest', '', class_name])\n        plt.yticks(tick_marks, ['', 'the rest', '', class_name])\n        plt.title(f'Confusion matrix: {class_name} vs. All')\n\n        fmt = '.2f'\n        thresh = conf_matrix.max() / 2.\n        for i, j in itertools.product(range(conf_matrix.shape[0]), range(conf_matrix.shape[1])):\n            plt.text(j, i, format(conf_matrix[i, j], fmt),\n                     horizontalalignment=\"center\",\n                     color=\"white\" if conf_matrix[i, j] > thresh else \"black\")\n\n        plt.ylabel('True label')\n        plt.xlabel('Predicted label')\n        plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_probability_for_precision_threshold(precision, thresholds, precision_threshold):\n    # consice, even though unnecessary passing through all the values\n    probability_threshold = [thres for prec, thres in zip(precision, thresholds) if prec >= precision_threshold][0]\n    return probability_threshold\n\ndef get_probability_for_recall_threshold(recall, thresholds, recall_threshold):\n    i = len(thresholds) - 1\n    probability_threshold_recall = None\n    while probability_threshold_recall is None:\n        next_threshold = thresholds[i]\n        next_recall = recall[i]\n        if next_recall >= recall_threshold:\n            probability_threshold_recall = next_threshold \n        i -= 1\n    return probability_threshold_recall\n\nclass_precision_2_probability = [dict() for _ in range(len(model_class_names))]\nclass_recall_2_probability = [dict() for _ in range(len(model_class_names))]\nprecision, recall, thresholds = precision_recall_curve(y_true[:, class_i], y_pred[:, class_i])\nfor class_i in tqdm(range(len(model_class_names))):\n    for thres in np.arange(10, 100):\n        class_prob_prec = get_probability_for_precision_threshold(precision, thresholds, thres/100)\n        class_precision_2_probability[class_i][thres] = class_prob_prec\n        class_prob_recall = get_probability_for_recall_threshold(recall, thresholds, thres/100)\n        class_recall_2_probability[class_i][thres] = class_prob_recall","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"y_pred_prec_90 = y_pred.copy()\nfor class_i in range(len(model_class_names)):\n    y_pred_prec_90[:, class_i] = np.where(y_pred_prec_90[:, class_i] >= class_precision_2_probability[class_i][90], 1.0, 0.0)\nplot_confusion_matrix(y_true, y_pred_prec_90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For classes Flower and Sugar the classifier retrieves more than half of all instances while remaining 90% precision. It's not the case for Gravel and Fish. In other words, Gravel is more easily confused with other classes, the same goes for Fish."},{"metadata":{},"cell_type":"markdown","source":"## Visualizations"},{"metadata":{},"cell_type":"markdown","source":"### Helper functions"},{"metadata":{},"cell_type":"markdown","source":"#### Guided Backprop routines"},{"metadata":{},"cell_type":"markdown","source":"False Positive might be due to wrong labels or bias of the classifier. To check potential classifier bias, I'd use Guided GradCAM, which is simply GradCAM multiplied by the result of guided backprop (i.e. focusing only on pixels which stimulate the net)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"layer_name='conv5_block16_concat'\ndef build_guided_model():\n    \"\"\"Function returning modified model.\n    \n    Changes gradient function for all ReLu activations\n    according to Guided Backpropagation.\n    \"\"\"\n    if \"GuidedBackProp\" not in ops._gradient_registry._registry:\n        @ops.RegisterGradient(\"GuidedBackProp\")\n        def _GuidedBackProp(op, grad):\n            dtype = op.inputs[0].dtype\n            return grad * tf.cast(grad > 0., dtype) * \\\n                   tf.cast(op.inputs[0] > 0., dtype)\n\n    g = tf.get_default_graph()\n    with g.gradient_override_map({'Relu': 'GuidedBackProp'}):\n        new_model = load_model('../input/clouds-classifier-files/classifier_epoch_45_val_pr_auc_0.8344173287108075.h5')\n    return new_model\n\nguided_model = build_guided_model()\n\ndef get_guided_backprop_fn(input_model, layer_name):\n    \"\"\"Guided Backpropagation method for visualizing input saliency.\"\"\"\n    input_imgs = input_model.input\n    layer_output = input_model.get_layer(layer_name).output\n    grads = K.gradients(layer_output, input_imgs)[0]\n    backprop_fn = K.function([input_imgs, K.learning_phase()], [grads])\n    def guided_backprop(images):\n        gb = backprop_fn([images, 0])[0]\n        \"\"\"Same normalization as in:\n        https://github.com/fchollet/keras/blob/master/examples/conv_filter_visualization.py\n        \"\"\"\n        gb = np.mean(gb, axis=3)\n        img_means = np.mean(gb, axis=(1, 2))\n        img_means = np.repeat(img_means, gb.shape[1]*gb.shape[2]).reshape(gb.shape)\n        img_std = np.std(gb, axis=(1, 2))\n        img_std = np.repeat(img_std, gb.shape[1]*gb.shape[2]).reshape(gb.shape) + 1e-5\n        gb = np.divide((gb - img_means), img_std)\n        # ensure std is 0.1\n        gb *= 0.1\n        gb += 0.5\n        # clip to [0, 1]\n        gb = np.clip(gb, 0, 1)\n        return gb\n        \n    return guided_backprop\n\nguided_backprop_fn = get_guided_backprop_fn(guided_model, layer_name)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# gradcam functions source/inspiration: https://github.com/eclique/keras-gradcam/blob/master/grad_cam.py\n\n# for per image grad_cam\ngradient_functions = []\nfor class_i in range(len(model_class_names)):\n    y_c = model.output[0, class_i]\n    conv_output = model.get_layer(layer_name).output\n    grads = K.gradients(y_c, conv_output)[0]\n    gradient_functions.append(K.function([model.input], [conv_output, grads]))\n\n# for batch gradcam\ngradient_fns = []\nfor class_i in range(len(model_class_names)):\n    class_predictions = tf.slice(model.output, [0, class_i], [-1, 1])\n    conv_layer_output = model.get_layer(layer_name).output\n    grads = K.gradients(class_predictions, conv_layer_output)[0]\n    gradient_fns.append(K.function([model.input, K.learning_phase()], [conv_layer_output, grads]))\n    \ndef grad_cam(image, class_i):\n    \"\"\"GradCAM method for visualizing input saliency.\"\"\"\n    output, grads_val = gradient_functions[class_i](image)\n    output, grads_val = output[0, :], grads_val[0, :, :, :]\n    weights = np.mean(grads_val, axis=(0, 1))\n    cam = np.dot(output, weights)\n    # Process CAM\n    cam = cv2.resize(cam, (image_height, image_width), cv2.INTER_LINEAR)\n    cam = np.maximum(cam, 0)\n    cam = cam / cam.max()\n    return cam\n\n\ndef grad_cam_batch(images, class_i):\n    \"\"\"GradCAM method for visualizing input saliency.\n    Same as grad_cam but processes multiple images in one run.\"\"\"\n    conv_output, grads_val = gradient_fns[class_i]([images, 0])    \n    weights = np.mean(grads_val, axis=(1, 2))\n    cams = np.einsum('ijkl,il->ijk', conv_output, weights)\n    \n    # Process CAMs\n    new_cams = np.empty((images.shape[0], image_height, image_width))\n    for i in range(images.shape[0]):\n        cam_i = cams[i] - cams[i].mean()\n        cam_i = (cam_i + 1e-10) / (np.linalg.norm(cam_i, 2) + 1e-10)\n        new_cams[i] = cv2.resize(cam_i, (image_height, image_width), cv2.INTER_LINEAR)\n        new_cams[i] = np.maximum(new_cams[i], 0)\n        new_cams[i] = new_cams[i] / new_cams[i].max()    \n    return new_cams","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def guided_grad_cam_batch(imgs_batch, class_i):\n    guided_backprop = guided_backprop_fn(imgs_batch)\n    gradcams_batch = grad_cam_batch(imgs_batch, class_i)\n    return gradcams_batch*guided_backprop","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def generate_guided_gradcam_masks(model=model, model_class_names=model_class_names):\n    print('Reading images..')\n    imgs_np = np.empty((len(val_imgs), image_height, image_width, 3))\n    for img_i, img_name in enumerate(tqdm(val_imgs)):\n        img_path = os.path.join(train_imgs_folder, img_name)\n        imgs_np[img_i, :, :, :] = cv2.resize(cv2.imread(img_path), (image_height, image_width)).astype(np.float32)/255.0\n    \n    gradcam_masks_np = np.empty((len(model_class_names),) + imgs_np.shape[:3], np.float32)\n    zero_mask = np.zeros((image_height, image_width), np.float32)\n    batch_size = 32\n    num_batches = imgs_np.shape[0]//batch_size + 1\n    print('Generating Guided GradCAMs')\n    for batch_i in tqdm(range(num_batches)):\n        imgs_batch = imgs_np[batch_i*batch_size: (batch_i + 1)*batch_size]\n        predictions = model.predict(imgs_batch)\n        for class_i, class_name in enumerate(model_class_names):\n            guided_gradcams_batch = guided_grad_cam_batch(imgs_batch, class_i)\n            gradcam_masks_np[class_i][batch_i*batch_size: (batch_i + 1)*batch_size] = guided_gradcams_batch\n    return gradcam_masks_np\n\nmin_size = 2000\nguided_gradcam_masks_np = generate_guided_gradcam_masks()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Functions checking distribution of labels for challenging images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def barplot_per_classes(imgs_indices, class_i, img_list=val_imgs, fp=True):\n    labels_np = np.array([img_2_ohe_vector[img_list[img_idx]] for img_idx in imgs_indices])\n    class_counts = labels_np.sum(axis=0)\n    plt.figure(figsize=(10,7))\n    plt.bar(model_class_names, class_counts)\n    plt.xlabel('Class', fontsize=12)\n    plt.ylabel('Count of images', fontsize=12)\n    plt.title(f\"Distribution of classes for {model_class_names[class_i]} false {'positives' if fp else 'negatives'}\", fontsize=17)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Functions for plotting mistakes alongside with correct ones for comparison"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def get_img(img_idx, imgs_list=val_imgs, folder=train_imgs_folder):\n    img_name = imgs_list[img_idx]\n    img_path = os.path.join(folder, img_name)\n    return cv2.resize(cv2.imread(img_path), (image_height, image_width)).astype(np.float32)/255., img_name\n\ndef visualize_and_compare_to_base(img_indices_base, img_indices_wrong, masks_val_class, ggrad_cam_class, class_name=None, fp_classes=[]):\n    assert(len(img_indices_base) == 8)\n    fig, axes = plt.subplots(4, 4, figsize=(15,15))\n    for subplot_i, img_idx_good in enumerate(img_indices_base):\n        ax = axes[subplot_i//4, subplot_i%4]\n        ax.axis('off')\n        img, img_name = get_img(img_idx_good)\n        ax.imshow(img)\n        ax.imshow(masks_val_class[img_idx_good], alpha=0.2)\n        ax.set_title(f'{img_name} ({class_name})')\n        \n    for subplot_ii, img_idx_bad in enumerate(img_indices_wrong):\n        fp_ax = axes[(subplot_i + subplot_ii + 1)//4, (subplot_i + subplot_ii + 1)%4]\n        img, img_name = get_img(img_idx_bad)\n        fp_ax.set_title(f\"{img_name} ({', '.join(fp_classes[subplot_ii])})\")\n        fp_ax.imshow(img)\n        fp_ax.imshow(ggrad_cam_class[img_idx_bad], cmap='jet', alpha=0.2)\n        plt.setp(fp_ax.spines.values(), color='red') \n    plt.suptitle('Two rows of reference class images followed by challenging images', fontsize=18)\n        \ndef get_fp_tp_fn_indices(class_i, threshold_probability):\n    y_pred_prec = y_pred.copy()\n    y_pred_prec[:, class_i] = np.where(y_pred_prec[:, class_i] >= threshold_probability,\n                                       1.0, 0.0)\n    fp_idx = [idx for idx, (label, pred) in enumerate(zip(y_true[:, class_i], y_pred_prec[:, class_i])) if label == 0 and pred == 1]\n    tp_idx = [idx for idx, (label, pred) in enumerate(zip(y_true[:, class_i], y_pred_prec[:, class_i])) if label == 1 and pred == 1]\n    fn_idx = [idx for idx, (label, pred) in enumerate(zip(y_true[:, class_i], y_pred_prec[:, class_i])) if label == 1 and pred == 0]\n    return tp_idx, fp_idx, fn_idx\n\ndef visualize_and_compare_fp(class_i, precision_level=90):\n    threshold_probability = class_precision_2_probability[class_i][precision_level]\n    tp_idx, fp_idx, _ = get_fp_tp_fn_indices(class_i, threshold_probability)\n    indices_fp = random.sample(fp_idx, 8)\n    fp_true_classes = [np.array(model_class_names)[y_true[idx_fp] == 1] for idx_fp in indices_fp]\n    visualize_and_compare_to_base(random.sample(tp_idx, 8), indices_fp,\n                                  val_masks_np[class_i], guided_gradcam_masks_np[class_i],\n                                  model_class_names[class_i], fp_true_classes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Fish"},{"metadata":{},"cell_type":"markdown","source":"#### Visual comparison of class representatives with challenging images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class_i = model_class_names.index('Fish')\nvisualize_and_compare_fp(class_i=class_i, precision_level=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2 interesting observations:\n   1. it seems that some Fish-like pattern are not labeled,\n   2. Fish might have higher ration of black images (based on the 5264e81.jpg)"},{"metadata":{},"cell_type":"markdown","source":"#### Distribution of labels in Fish-false-positive images\n\nLet's now check if some label prevails in images, where the classifier \"saw\" Fish."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"threshold_probability = class_precision_2_probability[class_i][90]\ntp_idx, fp_idx, _ = get_fp_tp_fn_indices(class_i, threshold_probability)\nbarplot_per_classes(fp_idx, class_i)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, probably Fish might get lost in Gravel more easily. Not too many FP images for reliable conclusion though."},{"metadata":{},"cell_type":"markdown","source":"### Exploring dark images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#let's define a dark image as one with at least 33% its pixels being black\ndark_imgs_count = [0] * len(model_class_names)\nimg_threshold_area = image_width*image_height/3\n\ndef count_dark_ones(imgs_list):\n    global dark_imgs_count\n    for img_idx, img_name in enumerate(tqdm(imgs_list)):\n        img, img_name = get_img(img_idx, imgs_list=imgs_list)\n        if np.sum(img[:, :, 0] == 0) >= img_threshold_area:\n            labels = img_2_ohe_vector[img_name]\n            for class_i in labels.nonzero()[0]:\n                dark_imgs_count[class_i] += 1\n\nprint('Processing train..')\ncount_dark_ones(train_imgs)\nprint('Processing val..')\ncount_dark_ones(val_imgs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(7, 7))\n# credits: https://stackoverflow.com/questions/6170246/how-do-i-use-matplotlib-autopct\ndef make_autopct(values):\n    def my_autopct(pct):\n        total = sum(values)\n        val = int(round(pct*total/100.0))\n        return '{p:.0f}%  ({v:d})'.format(p=pct,v=val)\n    return my_autopct\nax.pie(dark_imgs_count, labels=model_class_names, autopct=make_autopct(dark_imgs_count), shadow=True, startangle=90)\nax.axis('equal')\nax.set_title('Dark Images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is bias in training data indeed, as half of dark images contain Fish. However, there are only 12 such dark images in the initial train dataset. Let's check how many test dark images are there."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"dark_imgs_test_count = 0\nprint('Processing test..')\n\nfor img_idx, img_name in enumerate(tqdm(os.listdir(test_imgs_folder))):\n    img, img_name = get_img(img_idx, imgs_list=os.listdir(test_imgs_folder), folder=test_imgs_folder)\n    if np.sum(img[:, :, 0] == 0) >= img_threshold_area:\n        dark_imgs_test_count += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'There are {dark_imgs_test_count} dark images in test.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As there are just 7 dark images, I'd ignore the issue for the time being."},{"metadata":{},"cell_type":"markdown","source":"### Flower"},{"metadata":{},"cell_type":"markdown","source":"#### Visual comparison of class representatives with challenging images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class_i = model_class_names.index('Flower')\nvisualize_and_compare_fp(class_i=class_i, precision_level=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I'd consider some of the unlabeled images a Flower pattern."},{"metadata":{},"cell_type":"markdown","source":"#### Distribution of labels in Flower-false-positive images\n\nNow, let's check distribution of class in Flower FPs."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"threshold_probability = class_precision_2_probability[class_i][90]\ntp_idx, fp_idx, _ = get_fp_tp_fn_indices(class_i, threshold_probability)\nbarplot_per_classes(fp_idx, class_i)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, it seems like Flower is usually just missed, on contrary to Fish which seemed to be confused with Gravel in addition to misses."},{"metadata":{},"cell_type":"markdown","source":"### Sugar"},{"metadata":{},"cell_type":"markdown","source":"#### Visual comparison of class representatives with challenging images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class_i = model_class_names.index('Sugar')\nvisualize_and_compare_fp(class_i=class_i, precision_level=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Distribution of labels in Flower-false-positive images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"threshold_probability = class_precision_2_probability[class_i][90]\ntp_idx, fp_idx, _ = get_fp_tp_fn_indices(class_i, threshold_probability)\nbarplot_per_classes(fp_idx, class_i)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sugar seems to be sometimes confused with Gravel and Fish. If our theory about subjective confusion of class pair holds, then in the distribution of Gravel false positives there would be more Fish and Sugar."},{"metadata":{},"cell_type":"markdown","source":"### Gravel"},{"metadata":{},"cell_type":"markdown","source":"#### Visual comparison of class representatives with challenging images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class_i = model_class_names.index('Gravel')\nvisualize_and_compare_fp(class_i=class_i, precision_level=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Distribution of labels in Gravel-false-positive images"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"threshold_probability = class_precision_2_probability[class_i][90]\ntp_idx, fp_idx, _ = get_fp_tp_fn_indices(class_i, threshold_probability)\nbarplot_per_classes(fp_idx, class_i)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Indeed! Gravel seems to be more easily confused with Fish and also Sugar. The amount of inspected Gravel FPs images is rather small. Yet, for Fish FPs we saw a larger count of Gravel-labeled images as well."},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\nIn later stages it might be worth investing time in cleaning data. It might be worth considering pseudo labeling the data and/or manually check some labels, especially for Gravel class as it seems to be more confusing to distinguish Sugar from Gravel, and likely Fish from Gravel.\n\nBlack images doesn't seem to be of significant concern as there are not too many of them.  "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.7"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"05b0faa46dbd45f690ac3111c7f18bdb":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"5eecc03a0ffa4e18b51f60c40b961d52":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"5f9c63a634644cb29147ebfe9914fc1c":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"IntProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"IntProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_5eecc03a0ffa4e18b51f60c40b961d52","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_a7e7829f8eaa4e0bb8a3e6b926283b2f","value":1}},"7d06bbec7c0a4b44a37c58ab18ee19df":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"a7e7829f8eaa4e0bb8a3e6b926283b2f":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"aa497e6fb874450ea0aa5ecb3d403e62":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_5f9c63a634644cb29147ebfe9914fc1c","IPY_MODEL_dc6612b2c0d54d198db223a2aaaf8b3b"],"layout":"IPY_MODEL_7d06bbec7c0a4b44a37c58ab18ee19df"}},"dc6612b2c0d54d198db223a2aaaf8b3b":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_05b0faa46dbd45f690ac3111c7f18bdb","placeholder":"​","style":"IPY_MODEL_ea64221693df44dd9bb2d0da6ac15bf3","value":"4/|/| 4/? [00:00<00:00, 14.00it/s]"}},"ea64221693df44dd9bb2d0da6ac15bf3":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}}},"version_major":2,"version_minor":0}}},"nbformat":4,"nbformat_minor":1}