{"cells":[{"metadata":{},"cell_type":"markdown","source":"**this kernel was forked from here : https://www.kaggle.com/samusram/cloud-classifier-for-post-processing?scriptVersionId=20265194\n\ni got 0.657 using densenet201(in version 3) **\n\ni tried  efficientnetb4 but it failed for large image size,so i will try efficientnetb3 now with bce dice loss\nif you find this kernel useful,please upvote,your upvote motivates kagglers like us to share things publicly,thanks"},{"metadata":{},"cell_type":"markdown","source":"**Version 13**\n\n- updating previously used submission.csv with version 3's generated submission file which achieved 0.657\n\n- threshold = 0.9\n\n- efficientnetb2\n\n- loss='categorical_crossentropy', metrics=['accuracy']"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"# Intro\nIn this notebook I'd create a classifier to distinguish types of cloud formations. Using this classifier I'd check if it improves currently the best LB score from the great [public notebook by Jan](https://www.kaggle.com/jpbremer/efficient-net-b4-unet-clouds). "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true},"cell_type":"markdown","source":"# Plan\n1. [Libraries](#Libraries)\n2. [Data Generators](#Data-Generators)\n  * [One-hot encoding classes](#One-hot-encoding-classes)\n  * [Stratified split into train/val](#Stratified-split-into-train/val)\n  * [Generator class](#Generator-class)\n3. [PR-AUC-based Callback](#PR-AUC-based-Callback)\n4. [Classifier](#Classifier)\n  * [Defining a model](#Defining-a-model)\n  * [Initial tuning of the added fully-connected layer](#Initial-tuning-of-the-added-fully-connected-layer)\n  * [Fine-tuning the whole model](#Fine-tuning-the-whole-model)\n  * [Visualizing train and val PR AUC](#Visualizing-train-and-val-PR-AUC)\n5. [Selecting postprocessing thresholds](#Selecting-postprocessing-thresholds)\n6. [Post-processing segmentation submission](#Post-processing-segmentation-submission)\n7. [Future work](#Future-work)"},{"metadata":{},"cell_type":"markdown","source":"# Libraries"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import os, glob\nfrom IPython.display import FileLink\nimport random\nfrom sklearn.model_selection import train_test_split\nimport cv2\nimport numpy as np\nimport pandas as pd\nimport multiprocessing\nfrom copy import deepcopy\nfrom sklearn.metrics import precision_recall_curve, auc\nimport keras\nimport tensorflow as tf\nimport keras.backend as K\nfrom keras.optimizers import Adam\nfrom keras.callbacks import Callback\nfrom keras.applications.densenet import DenseNet201\nfrom keras.callbacks import ModelCheckpoint, Callback, EarlyStopping\nfrom keras.layers import Dense, Flatten, Conv2D, BatchNormalization, GlobalMaxPooling2D,MaxPooling2D, Dropout\nfrom keras.models import Model, load_model\nfrom keras.utils import Sequence\nfrom albumentations import Compose, VerticalFlip, HorizontalFlip, Rotate, GridDistortion\nimport matplotlib.pyplot as plt\nfrom IPython.display import Image\nfrom tqdm import tqdm_notebook as tqdm\nfrom numpy.random import seed\nseed(10)\n# from tensorflow import set_random_seed\n# set_random_seed(10)\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"test_imgs_folder = '../input/understanding_cloud_organization/test_images/'\ntrain_imgs_folder = '../input/understanding_cloud_organization/train_images/'\nnum_cores = multiprocessing.cpu_count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data Generators"},{"metadata":{},"cell_type":"markdown","source":"## One-hot encoding classes"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"train_df = train_df[~train_df['EncodedPixels'].isnull()]\ntrain_df['Image'] = train_df['Image_Label'].map(lambda x: x.split('_')[0])\ntrain_df['Class'] = train_df['Image_Label'].map(lambda x: x.split('_')[1])\nclasses = train_df['Class'].unique()\ntrain_df = train_df.groupby('Image')['Class'].agg(set).reset_index()\nfor class_name in classes:\n    train_df[class_name] = train_df['Class'].map(lambda x: 1 if class_name in x else 0)\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# dictionary for fast access to ohe vectors\nimg_2_ohe_vector = {img:vec for img, vec in zip(train_df['Image'], train_df.iloc[:, 2:].values)}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* ## Stratified split into train/val"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_imgs, val_imgs = train_test_split(train_df['Image'].values, \n                                        test_size=0.25, \n                                        stratify=train_df['Class'].map(lambda x: str(sorted(list(x)))), # sorting present classes in lexicographical order, just to be sure\n                                        random_state=2019)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generator class"},{"metadata":{"trusted":true},"cell_type":"code","source":"class DataGenenerator(Sequence):\n    def __init__(self, images_list=None, folder_imgs=train_imgs_folder, \n                 batch_size=16, shuffle=True, augmentation=False,\n                 resized_height=260, resized_width=260, num_channels=3):\n        self.batch_size = batch_size\n        self.shuffle = shuffle\n        self.augmentation = augmentation\n        if images_list is None:\n            self.images_list = os.listdir(folder_imgs)\n        else:\n            self.images_list = deepcopy(images_list)\n        self.folder_imgs = folder_imgs\n        self.len = len(self.images_list) // self.batch_size\n        self.resized_height = resized_height\n        self.resized_width = resized_width\n        self.num_channels = num_channels\n        self.num_classes = 4\n        self.is_test = not 'train' in folder_imgs\n        if not shuffle and not self.is_test:\n            self.labels = [img_2_ohe_vector[img] for img in self.images_list[:self.len*self.batch_size]]\n\n    def __len__(self):\n        return self.len\n    \n    def on_epoch_start(self):\n        if self.shuffle:\n            random.shuffle(self.images_list)\n\n    def __getitem__(self, idx):\n        current_batch = self.images_list[idx * self.batch_size: (idx + 1) * self.batch_size]\n        X = np.empty((self.batch_size, self.resized_height, self.resized_width, self.num_channels))\n        y = np.empty((self.batch_size, self.num_classes))\n\n        for i, image_name in enumerate(current_batch):\n            path = os.path.join(self.folder_imgs, image_name)\n            img = cv2.resize(cv2.imread(path), (self.resized_height, self.resized_width)).astype(np.float32)\n#             if not self.augmentation is None:\n#                 augmented = self.augmentation(image=img)\n#                 img = augmented['image']\n            X[i, :, :, :] = img/255.0\n            if not self.is_test:\n                y[i, :] = img_2_ohe_vector[image_name]\n        return X, y\n\n    def get_labels(self):\n        if self.shuffle:\n            images_current = self.images_list[:self.len*self.batch_size]\n            labels = [img_2_ohe_vector[img] for img in images_current]\n        else:\n            labels = self.labels\n        return np.array(labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"albumentations_train = Compose([\n    VerticalFlip(), HorizontalFlip(), Rotate(limit=20), GridDistortion()\n], p=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generator instances"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_generator_train = DataGenenerator(train_imgs, augmentation=albumentations_train)\ndata_generator_train_eval = DataGenenerator(train_imgs, shuffle=True)\ndata_generator_val = DataGenenerator(val_imgs, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# PR-AUC-based Callback"},{"metadata":{},"cell_type":"markdown","source":"The callback would be used:\n1. to estimate AUC under precision recall curve for each class,\n2. to early stop after 5 epochs of no improvement in mean PR AUC,\n3. save a model with the best PR AUC in validation,\n4. to reduce learning rate on PR AUC plateau."},{"metadata":{"trusted":true},"cell_type":"code","source":"class PrAucCallback(Callback):\n    def __init__(self, data_generator, num_workers=num_cores, \n                 early_stopping_patience=5, \n                 plateau_patience=3, reduction_rate=0.5,\n                 stage='train', checkpoints_path='checkpoints/'):\n        super(Callback, self).__init__()\n        self.data_generator = data_generator\n        self.num_workers = num_workers\n        self.class_names = ['Fish', 'Flower', 'Sugar', 'Gravel']\n        self.history = [[] for _ in range(len(self.class_names) + 1)] # to store per each class and also mean PR AUC\n        self.early_stopping_patience = early_stopping_patience\n        self.plateau_patience = plateau_patience\n        self.reduction_rate = reduction_rate\n        self.stage = stage\n        self.best_pr_auc = -float('inf')\n        if not os.path.exists(checkpoints_path):\n            os.makedirs(checkpoints_path)\n        self.checkpoints_path = checkpoints_path\n        \n    def compute_pr_auc(self, y_true, y_pred):\n        pr_auc_mean = 0\n        print(f\"\\n{'#'*30}\\n\")\n        for class_i in range(len(self.class_names)):\n            precision, recall, _ = precision_recall_curve(y_true[:, class_i], y_pred[:, class_i])\n            pr_auc = auc(recall, precision)\n            pr_auc_mean += pr_auc/len(self.class_names)\n            print(f\"PR AUC {self.class_names[class_i]}, {self.stage}: {pr_auc:.3f}\\n\")\n            self.history[class_i].append(pr_auc)        \n        print(f\"\\n{'#'*20}\\n PR AUC mean, {self.stage}: {pr_auc_mean:.3f}\\n{'#'*20}\\n\")\n        self.history[-1].append(pr_auc_mean)\n        return pr_auc_mean\n              \n    def is_patience_lost(self, patience):\n        if len(self.history[-1]) > patience:\n            best_performance = max(self.history[-1][-(patience + 1):-1])\n            return best_performance == self.history[-1][-(patience + 1)] and best_performance >= self.history[-1][-1]    \n              \n    def early_stopping_check(self, pr_auc_mean):\n        if self.is_patience_lost(self.early_stopping_patience):\n            self.model.stop_training = True    \n              \n    def model_checkpoint(self, pr_auc_mean, epoch):\n        if pr_auc_mean > self.best_pr_auc:\n            # remove previous checkpoints to save space\n            for checkpoint in glob.glob(os.path.join(self.checkpoints_path, 'classifier_densenet169_epoch_*')):\n                os.remove(checkpoint)\n            self.best_pr_auc = pr_auc_mean\n            self.model.save(os.path.join(self.checkpoints_path, f'classifier_densenet169_epoch_{epoch}_val_pr_auc_{pr_auc_mean}.h5'))              \n            print(f\"\\n{'#'*20}\\nSaved new checkpoint\\n{'#'*20}\\n\")\n              \n    def reduce_lr_on_plateau(self):\n        if self.is_patience_lost(self.plateau_patience):\n            new_lr = float(keras.backend.get_value(self.model.optimizer.lr)) * self.reduction_rate\n            keras.backend.set_value(self.model.optimizer.lr, new_lr)\n            print(f\"\\n{'#'*20}\\nReduced learning rate to {new_lr}.\\n{'#'*20}\\n\")\n        \n    def on_epoch_end(self, epoch, logs={}):\n        y_pred = self.model.predict_generator(self.data_generator, workers=self.num_workers)\n        y_true = self.data_generator.get_labels()\n        # estimate AUC under precision recall curve for each class\n        pr_auc_mean = self.compute_pr_auc(y_true, y_pred)\n              \n        if self.stage == 'val':\n            # early stop after early_stopping_patience=4 epochs of no improvement in mean PR AUC\n            self.early_stopping_check(pr_auc_mean)\n\n            # save a model with the best PR AUC in validation\n            self.model_checkpoint(pr_auc_mean, epoch)\n\n            # reduce learning rate on PR AUC plateau\n            self.reduce_lr_on_plateau()            \n        \n    def get_pr_auc_history(self):\n        return self.history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Callback instances"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_metric_callback = PrAucCallback(data_generator_train_eval)\nval_callback = PrAucCallback(data_generator_val, stage='val')\npath_checkpoint = \"clouds_13-acc-{accuracy:.4f}.h5\"\ncallbacks_list = [\n    EarlyStopping(monitor='accuracy', min_delta=0.001, patience=5, verbose=0, mode='max', baseline=None,\n                  restore_best_weights=True)] #ModelCheckpoint(path_checkpoint, monitor='accuracy', verbose=1, save_best_only=True, mode='max')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Classifier"},{"metadata":{},"cell_type":"markdown","source":"## Defining a model"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.losses import binary_crossentropy\ndef dice_coef(y_true, y_pred, smooth=1):\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = K.sum(y_true_f * y_pred_f)\n    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n\ndef dice_loss(y_true, y_pred):\n    smooth = 1.\n    y_true_f = K.flatten(y_true)\n    y_pred_f = K.flatten(y_pred)\n    intersection = y_true_f * y_pred_f\n    score = (2. * K.sum(intersection) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)\n    return 1. - score\n\ndef bce_dice_loss(y_true, y_pred):\n    return binary_crossentropy(y_true, y_pred) + dice_loss(y_true, y_pred)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pip install efficientnet","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nos.listdir('/kaggle/input/cloudsefficientnet01')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import efficientnet.keras as efn   \n# modle has locked, Do not make any changes\nall_val_acc = []\ndef get_model():\n    K.clear_session()\n    base_model =  efn.EfficientNetB4(weights=None, include_top=False, pooling='avg', input_shape=(260, 260, 3))\n    x = base_model.output\n\n    y_pred = Dense(4, activation='sigmoid')(x)\n    return Model(inputs=base_model.input, outputs=y_pred)\n\nmodel = get_model()\nmodel = load_model('/kaggle/input/cloudsefficientnet01/clouds13-EffiecientNet-val_acc-0.4833.h5')\nmodel.compile(optimizer=Adam(lr=0.00011),  loss='categorical_crossentropy', metrics=['accuracy'])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Initial tuning of the added fully-connected layer"},{"metadata":{"trusted":true},"cell_type":"code","source":"# for base_layer in model.layers[:-1]:\n#     base_layer.trainable = False\n    \n# model.compile(optimizer=Adam(lr=0.00011),  loss='categorical_crossentropy', metrics=['accuracy'])\n# for i in range(3):\n#     history_0 = model.fit_generator(generator=data_generator_train,\n#                                   validation_data=data_generator_val,\n#                                   epochs=10,\n#                                   callbacks=callbacks_list,\n#                                   workers=num_cores,\n#                                   verbose=1\n#                                  )\n    \n#     os.chdir('/kaggle/working')\n#     val_acc = round(history_0.history['val_accuracy'][-1], 4)\n#     all_val_acc.append(val_acc)\n#     model.save('clouds13-EffiecientNet-val_acc-{}.h5'.format(val_acc))\n#     model_file = FileLink('clouds13-EffiecientNet-val_acc-{}.h5'.format(val_acc))\n#     display(model_file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for base_layer in model.layers:\n    base_layer.trainable = True\n    \nmodel.compile(optimizer=Adam(lr=0.00011),  loss='categorical_crossentropy', metrics=['accuracy'])\nfor i in range(6):\n    history = model.fit_generator(generator=data_generator_train,\n                                  validation_data=data_generator_val,\n                                  epochs=10,\n                                  callbacks=callbacks_list,\n                                  workers=num_cores,\n                                  verbose=1\n                                 )\n    os.chdir('/kaggle/working')\n    val_acc = round(history.history['val_accuracy'][-1], 4)\n    all_val_acc.append(val_acc)\n    model.save('clouds13-EffiecientNet-val_acc-{}.h5'.format(val_acc))\n    model_file = FileLink('clouds13-EffiecientNet-val_acc-{}.h5'.format(val_acc))\n    display(model_file)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualizing train and val PR AUC"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_with_dots(ax, np_array):\n    ax.scatter(list(range(1, len(np_array) + 1)), np_array, s=50)\n    ax.plot(list(range(1, len(np_array) + 1)), np_array)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# pr_auc_history_train = train_metric_callback.get_pr_auc_history()\n# pr_auc_history_val = val_callback.get_pr_auc_history()\n\n# plt.figure(figsize=(10, 7))\n# plot_with_dots(plt, pr_auc_history_train[-1])\n# plot_with_dots(plt, pr_auc_history_val[-1])\n\n# plt.xlabel('Epoch', fontsize=15)\n# plt.ylabel('Mean PR AUC', fontsize=15)\n# plt.legend(['Train', 'Val'])\n# plt.title('Training and Validation PR AUC', fontsize=20)\n# plt.savefig('pr_auc_hist.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.figure(figsize=(10, 7))\n# plot_with_dots(plt, history.history_0['loss']+history.history['loss'])\n# plot_with_dots(plt, history.history_0['val_loss']+history.history['val_loss'])\n\n# plt.xlabel('Epoch', fontsize=15)\n# plt.ylabel('Binary Crossentropy', fontsize=15)\n# plt.legend(['Train', 'Val'])\n# plt.title('Training and Validation Loss', fontsize=20)\n# plt.savefig('loss_hist.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I left the model to train longer on my local GPU. I then upload the best model and plots from the model training."},{"metadata":{},"cell_type":"markdown","source":"# Selecting postprocessing thresholds"},{"metadata":{"trusted":true},"cell_type":"code","source":"class_names = ['Fish', 'Flower', 'Sugar', 'Gravel']\ndef get_threshold_for_recall(y_true, y_pred, class_i, recall_threshold=0.94, precision_threshold=0.90, plot=False):\n    precision, recall, thresholds = precision_recall_curve(y_true[:, class_i], y_pred[:, class_i])\n    i = len(thresholds) - 1\n    best_recall_threshold = None\n    while best_recall_threshold is None:\n        next_threshold = thresholds[i]\n        next_recall = recall[i]\n        if next_recall >= recall_threshold:\n            best_recall_threshold = next_threshold\n        i -= 1\n        \n    # consice, even though unnecessary passing through all the values\n    best_precision_threshold = [thres for prec, thres in zip(precision, thresholds) if prec >= precision_threshold][0]\n    \n    if plot:\n        plt.figure(figsize=(10, 7))\n        plt.step(recall, precision, color='r', alpha=0.3, where='post')\n        plt.fill_between(recall, precision, alpha=0.3, color='r')\n        plt.axhline(y=precision[i + 1])\n        recall_for_prec_thres = [rec for rec, thres in zip(recall, thresholds) \n                                 if thres == best_precision_threshold][0]\n        plt.axvline(x=recall_for_prec_thres, color='g')\n        plt.xlabel('Recall')\n        plt.ylabel('Precision')\n        plt.ylim([0.0, 1.05])\n        plt.xlim([0.0, 1.0])\n        plt.legend(['PR curve', \n                    f'Precision {precision[i + 1]: .2f} corresponding to selected recall threshold',\n                    f'Recall {recall_for_prec_thres: .2f} corresponding to selected precision threshold'])\n        plt.title(f'Precision-Recall curve for Class {class_names[class_i]}')\n    return best_recall_threshold, best_precision_threshold\n\ny_pred_mean = []\nall_val_acc.sort(reverse=True)\nmodel_path = '/kaggle/input/clouds12weights'\n# model_path = '/kaggle/working'\n# model_names = ['clouds_12-acc-0.5593.pth', 'clouds_12-acc-0.5500.pth']\nfor val_acc in tqdm(all_val_acc[:10]):\n# for model_name in tqdm(model_names):\n    K.clear_session()  # clearing the session for decreasing the loading time or load the models by weight. save it to json_yaml and something like taht and finally imprort them.\n    model_name = 'clouds13-EffiecientNet-val_acc-{}.h5'.format(val_acc)\n    model = get_model()\n\n    print(model_name)\n    model = load_model(os.path.join(model_path, model_name))\n    model.compile(optimizer=Adam(lr=0.00011),  loss='categorical_crossentropy', metrics=['accuracy'])\n    y_pred_mean.append(model.predict_generator(data_generator_val, workers=num_cores))\n\nfor y in tqdm(y_pred_mean):\ny_pred_mean = pd.concat(pred for y in y_pred_mean)\ny_pred_mean = y_pred_concat.groupby(y_pred_mean).mean()\n\ny_true = data_generator_val.get_labels()\nrecall_thresholds = dict()\nprecision_thresholds = dict()\nfor i, class_name in tqdm(enumerate(class_names)):\n    recall_thresholds[class_name], precision_thresholds[class_name] = get_threshold_for_recall(y_true, y_pred_mean, i, plot=True)\ndel y_pred_mean","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Post-processing segmentation submission"},{"metadata":{},"cell_type":"markdown","source":"Predicting cloud classes for test."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_generator_test = DataGenenerator(folder_imgs=test_imgs_folder, shuffle=False)\n\ntest_pred_mean = []\n# model_path = '/kaggle/input/clouds12weights'\nmodel_path = '/kaggle/working'\nmodel_names = [  'clouds_12-acc-0.5593.pth' , 'clouds_12-acc-0.5500.pth']\n# model = get_model()\nfor val_acc in tqdm(all_val_acc[:10]):\n# for model_name in tqdm(model_names):\n    model_name = 'clouds13-EffiecientNet-val_acc-{}.h5'.format(val_acc)\n    model = get_model()\n\n    print(model_name)\n    model = load_model(os.path.join(model_path, model_name))\n    model.compile(optimizer=Adam(lr=0.00011),  loss='categorical_crossentropy', metrics=['accuracy'])\n    test_pred_mean.append(model.predict_generator(data_generator_test, workers=num_cores))\n    del model\n\n\ntest_pred_mean = pd.concat(pred for y in test_pred_mean)\ntest_pred_mean = y_pred_concat.groupby(test_pred_mean).mean()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Estimating set of images without masks."},{"metadata":{"trusted":true},"cell_type":"code","source":"image_labels_empty = set()\nfor i, (img, predictions) in enumerate(zip(os.listdir(test_imgs_folder), test_pred_mean)):\n    for class_i, class_name in enumerate(class_names):\n        if predictions[class_i] < recall_thresholds[class_name]:\n            image_labels_empty.add(f'{img}_{class_name}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Segmentation results:"},{"metadata":{"trusted":true},"cell_type":"code","source":"submission = pd.read_csv('../input/clouds13densenet/clouds13-densenet.csv')\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions_nonempty = set(submission.loc[~submission['EncodedPixels'].isnull(), 'Image_Label'].values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'{len(image_labels_empty.intersection(predictions_nonempty))} masks would be removed')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#removing masks\nsubmission.loc[submission['Image_Label'].isin(image_labels_empty), 'EncodedPixels'] = np.nan\nsubmission.to_csv('submission_final_segmentation_and_classifier.csv', index=None)\nmodel_file = FileLink('submission_final_segmentation_and_classifier.csv')\ndisplay(model_file)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Future work\n1. estimate distribution of classes in test set using the classifier. Then, if necessary and doable, modify val set accordingly,\n2. use the classifier with explainability technique [Gradient-weighted Class Activation Mapping](http://gradcam.cloudcv.org/) to generate a baseline, (please see [GradCAM: extracting masks from classifier](https://www.kaggle.com/samusram/gradcam-extracting-masks-from-classifier)),\n3. improve the classifier,\n4. use the classifier as backbone for UNet-like solution."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"0a53fa055d2b43358bafff0a453d2c8b":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"25064ebbb12a483690e0fd0feb52560b":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":"initial"}},"38ff85bb09144a9aaebf4dd59f000c69":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_7640235bcae6457c80e41c51a06677d5","IPY_MODEL_dd1866cdda2742debd15613963dd4759"],"layout":"IPY_MODEL_0a53fa055d2b43358bafff0a453d2c8b"}},"4fcf62cbaf71492f85b8e7e6f69a9105":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"7640235bcae6457c80e41c51a06677d5":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"IntProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"IntProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_eac971125b6e4705b4db95f6c2ab6318","max":1,"min":0,"orientation":"horizontal","style":"IPY_MODEL_25064ebbb12a483690e0fd0feb52560b","value":1}},"ca8b02376bfc46859708c5f3069f7ca5":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"dd1866cdda2742debd15613963dd4759":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_ca8b02376bfc46859708c5f3069f7ca5","placeholder":"​","style":"IPY_MODEL_4fcf62cbaf71492f85b8e7e6f69a9105","value":" 4/? [00:00&lt;00:00, 13.35it/s]"}},"eac971125b6e4705b4db95f6c2ab6318":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}}},"version_major":2,"version_minor":0}}},"nbformat":4,"nbformat_minor":1}