{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## General information\n\nIt is usually a good idea to write modular code, if we want reproducibility, ability to easily change the code and conveniently reuse the same code for multiple purposes.\n\nCurrently there is[ a utility scripts competition](https://www.kaggle.com/general/109651) and have decided to contribute.\n\nThis kernel is inspired by this work: https://www.kaggle.com/samusram/cloud-classifier-for-post-processing\n\nThe main idea is also training a classifier for postprocessing. I combine it with models trained in my previous kernel: https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools/\n\nIn this kernel I once again use the following libraries (or ideas from them):\n\n* [albumentations](https://github.com/albu/albumentations): this is a great library for image augmentation which makes it easier and more convenient\n* [catalyst](https://github.com/catalyst-team/catalyst): this is a great library which makes using PyTorch easier, helps with reprodicibility and contains a lot of useful utils\n* [segmentation_models_pytorch](https://github.com/qubvel/segmentation_models.pytorch): this is a great library with convenient wrappers for models, losses and other useful things\n* [pytorch-toolbelt](https://github.com/BloodAxe/pytorch-toolbelt): this is a great library with many useful shortcuts for building pytorch models\n\nBut in this case all the functions and classes are imported from my utility script: https://www.kaggle.com/artgor/pytorch-utils-for-images\n\nIt has functions to:\n- get models, optimizers and other things necessary for training;\n- get dataloaders;\n- train model;\n- make visualizations;\n- and so on.\n\nThis code isn't ideal and I continue working on improving it, but I actually use it to train models locally.\n\nAlso I want to say, that this code is split into several scripts: for models, augmentations, dataset and so on separately. But I think it is better to keep it in a single kaggle utility script.\n\nP. S. This code can be also used for severstal competition, you would only need to modify `prepare_loaders` function ;)\n![](https://cdn.technologynetworks.com/tn/images/thumbs/jpeg/640_360/repeatability-vs-reproducibility-317157.jpg)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# imports\nimport torch\nimport gc\nfrom pytorch_utils_for_images import *","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Setting up parameters\n\nAt first it is a good idea to set up random seeds."},{"metadata":{"trusted":true},"cell_type":"code","source":"SEED = 42\n\nset_global_seed(SEED)\nprepare_cudnn(deterministic=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now I'm setting various parameters for functions. Description for most of these parameters can be found in my utility script.\n\nWhen I train models locally, I prefer to either run the code in command line and pass parameters there or to set up config files.\n\nWith this approach it is easy to switch between classification and segmentation tasks, between model architectures and encoders."},{"metadata":{"trusted":true},"cell_type":"code","source":"task = 'classification'\nencoder = 'densenet169'\nencoder_weights = 'imagenet'\nbatch_size = 8\npath = '../input/understanding_cloud_organization'\nnum_workers = 0\nsegm_type = 'Unet'\nactivation = None\nn_classes = 4\nloss = 'BCE'\ngradient_accumulation = 'False'\nnum_epochs = 20\nlr = 1e-4","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing for training\n\nNow we need to get everything necessary for training the model:\n- get data loaders with training and validation datasets;\n- get the model itself. It is densenet169 with custom head right now (several linear layers with batch norm and dropout);\n- get optimizer, scheduler and criterion;\n- get catalyst callbacks;\n\nWe have the masks, but for multilabel classification we would need labels instead, I do it as in this great kernel: https://www.kaggle.com/samusram/cloud-classifier-for-post-processing (in my function `prepare_loaders`)\n```python\n    train = pd.read_csv(f'{path}/train.csv')\n    train['label'] = train['Image_Label'].apply(lambda x: x.split('_')[1])\n    train['im_id'] = train['Image_Label'].apply(lambda x: x.split('_')[0])\n    if task == 'classification':\n        train_df = train[~train['EncodedPixels'].isnull()]\n        classes = train_df['label'].unique()\n        train_df = train_df.groupby('im_id')['label'].agg(set).reset_index()\n        for class_name in classes:\n            train_df[class_name] = train_df['label'].map(lambda x: 1 if class_name in x else 0)\n\n        img_2_ohe_vector = {img: np.float32(vec) for img, vec in zip(train_df['im_id'], train_df.iloc[:, 2:].values)}\n````"},{"metadata":{"trusted":true},"cell_type":"code","source":"preprocessing_fn = smp.encoders.get_preprocessing_fn(encoder, encoder_weights)\nloaders = prepare_loaders(path=path, bs=batch_size,\n                          num_workers=num_workers, preprocessing_fn=preprocessing_fn, preload=False, task=task,\n                          image_size=(224, 224))\ntest_loader = loaders['test']\ndel loaders['test']\n\nmodel = get_model(model_type=segm_type, encoder=encoder, encoder_weights=encoder_weights,\n                  activation=activation, task=task, n_classes=n_classes, head='simple')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = get_optimizer(optimizer='RAdam', lookahead=False, model=model, separate_decoder=False, lr=lr, lr_e=lr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scheduler = ReduceLROnPlateau(optimizer, factor=0.7, patience=2)\ncriterion = get_loss(loss)\ncriterion","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if task == 'segmentation':\n    callbacks = [DiceCallback(), EarlyStoppingCallback(patience=5, min_delta=0.001), CriterionCallback()]\nelif task == 'classification':\n    callbacks = [AUCCallback(class_names=['Fish', 'Flower', 'Gravel', 'Sugar'], num_classes=4), EarlyStoppingCallback(patience=5, min_delta=0.001), CriterionCallback(), CustomCheckpointCallback()]\n\nrunner = SupervisedRunner()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring augmentations with albumentations\n\nOne of important things while working with images is choosing good augmentations. There are a lot of them, let's have a look at augmentations from albumentations!\n\nFor classification the position of clouds is less important, so we could use more agressive augmentations."},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/understanding_cloud_organization/train.csv')\ntrain['label'] = train['Image_Label'].apply(lambda x: x.split('_')[1])\ntrain['im_id'] = train['Image_Label'].apply(lambda x: x.split('_')[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"image_name = '8242ba0.jpg'\nimage = get_img(image_name, '../input/understanding_cloud_organization/train_images')\nmask = make_mask(train, image_name)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.HorizontalFlip(p=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.Blur(p=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.CLAHE(p=1))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_with_augmentation(image, mask, albu.ShiftScaleRotate(p=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training the model\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i, param in list((model.named_parameters()))[:-10]:\n    param.requires_grad = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"logdir = './logs/classification'\n\n#num_epochs = 1\nrunner.train(\n            model=model,\n            criterion=criterion,\n            optimizer=optimizer,\n            scheduler=scheduler,\n            loaders=loaders,\n            callbacks=callbacks,\n            logdir=logdir,\n            num_epochs=num_epochs,\n            verbose=True\n        )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see not only mean AUC, but also AUC for each class."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_metrics(\n    logdir='../input/cloud-segmentation-model', \n    # specify which metrics we want to plot\n    metrics=[\"loss\", \"auc/_mean\", \"auc/class_Fish\", \"auc/class_Flower\", \"auc/class_Gravel\", \"auc/class_Sugar\", \"_base/lr\"]\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's get predictions and the original labels."},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_predictions = runner.predict_loader(\n    model, loaders[\"valid\"],\n    resume=f\"{logdir}/checkpoints/best.pth\", verbose=True\n)\n\ny_valid = []\nfor img, label in loaders[\"valid\"].dataset:\n    y_valid.append(label)\ny_valid = np.array(y_valid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_predictions = sigmoid(valid_predictions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's have a look at precision-recall curves!"},{"metadata":{"trusted":true},"cell_type":"code","source":"class_dict = {0: 'Fish', 1: 'Flower', 2: 'Gravel', 3: 'Sugar'}\nfig, ax = plt.subplots(figsize = (16, 12))\nfor i in range(4):\n    plt.subplot(2, 2, i + 1)\n    plot_precision_recall(y_valid[:, i], valid_predictions[:, i], title=class_dict[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks quite good for a first attempt.\n\nFor classification we would need to find some thresholds, so that we have labels and not probabilites."},{"metadata":{"trusted":true},"cell_type":"code","source":"class_thresholds = {}\nfor i in range(4):\n    print(f\"Class: {class_dict[i]}\")\n    t = find_threshold(y_valid[:, i], valid_predictions[:, i])\n    print()\n    class_thresholds[i] = t","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For comparison I also want to try a function from this kernel: https://www.kaggle.com/samusram/cloud-classifier-for-post-processing"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_threshold_for_recall(y_true, y_pred, class_i, recall_threshold=0.95, precision_threshold=0.94, plot=False):\n    precision, recall, thresholds = metrics.precision_recall_curve(y_true[:, class_i], y_pred[:, class_i])\n    i = len(thresholds) - 1\n    best_recall_threshold = None\n    while best_recall_threshold is None:\n        next_threshold = thresholds[i]\n        next_recall = recall[i]\n        if next_recall >= recall_threshold:\n            best_recall_threshold = next_threshold\n        i -= 1\n        \n    # consice, even though unnecessary passing through all the values\n    best_precision_threshold = [thres for prec, thres in zip(precision, thresholds) if prec >= precision_threshold][0]\n    \n    if plot:\n        plt.figure(figsize=(10, 7))\n        plt.step(recall, precision, color='r', alpha=0.3, where='post')\n        plt.fill_between(recall, precision, alpha=0.3, color='r')\n        plt.axhline(y=precision[i + 1])\n        recall_for_prec_thres = [rec for rec, thres in zip(recall, thresholds) \n                                 if thres == best_precision_threshold][0]\n        plt.axvline(x=recall_for_prec_thres, color='g')\n        plt.xlabel('Recall')\n        plt.ylabel('Precision')\n        plt.ylim([0.0, 1.05])\n        plt.xlim([0.0, 1.0])\n        plt.legend(['PR curve', \n                    f'Precision {precision[i + 1]: .2f} corresponding to selected recall threshold',\n                    f'Recall {recall_for_prec_thres: .2f} corresponding to selected precision threshold'])\n        plt.title(f'Precision-Recall curve for Class {class_dict[class_i]}')\n    return best_recall_threshold, best_precision_threshold\n\nrecall_thresholds = dict()\nprecision_thresholds = dict()\nfor i, class_name in tqdm.tqdm(enumerate(class_dict.items())):\n    recall_thresholds[class_name], precision_thresholds[class_name] = get_threshold_for_recall(y_valid, valid_predictions, i, plot=True)\n    \nprint('recall_thresholds', recall_thresholds)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"recall_thresholds = {k[0]: v for k, v in recall_thresholds.items()}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Predicting masks\n\nNow we will need to load a segmentation model to make predictions. This can be easily done."},{"metadata":{"trusted":true},"cell_type":"code","source":"# freeing memory\ntorch.cuda.empty_cache()\ngc.collect()\ndel runner","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"task = 'segmentation'\nencoder = 'resnet50'\nbatch_size = 8\n\nmodel = get_model(model_type=segm_type, encoder=encoder, encoder_weights=encoder_weights,\n                  activation=activation, task=task, n_classes=n_classes, head='custom')\nloaders = prepare_loaders(path=path, bs=batch_size,\n                          num_workers=num_workers, preprocessing_fn=preprocessing_fn, preload=False, task=task)\n\ndel loaders['train']\ndel loaders['test']\ncheckpoint_path = '../input/cloud-segmentation-model/best_full.pth'\ncheckpoint = utils.load_checkpoint(checkpoint_path)\nmodel.cuda()\nutils.unpack_checkpoint(checkpoint, model=model)\nrunner = SupervisedRunner(model=model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loaders = {\"infer\": loaders['valid']}\nrunner.infer(\n    model=runner.model,\n    loaders=loaders,\n    callbacks=[\n        CheckpointCallback(\n            resume=checkpoint_path),\n        InferCallback()\n    ],\n)\n\nvalid_masks = []\nprobabilities = np.zeros((len(runner.callbacks[0].predictions['logits']) * 4, 350, 525))\nfor i, (batch, output) in enumerate(zip(\n    loaders['infer'].dataset, runner.callbacks[0].predictions[\"logits\"])):\n    image, mask = batch\n    for m in mask:\n        if m.shape != (350, 525):\n            m = cv2.resize(m, dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n        valid_masks.append(m)\n\n    for j, probability in enumerate(output):\n        if probability.shape != (350, 525):\n            probability = cv2.resize(probability, dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n        probabilities[i * 4 + j, :, :] = probability","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dice of raw predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_ids = loaders['infer'].dataset.img_ids","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"original_dices = []\nd = []\nfor ind, (i, j) in enumerate(zip(probabilities, valid_masks)):\n    if (i.sum() == 0) & (j.sum() == 0):\n        d.append(1)\n    else:\n        d.append(dice(i, j))\n    if len(d) == 4:\n        d = [valid_ids[ind // 4]] + d\n        original_dices.append(d)\n        d = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"original_dices = pd.DataFrame(original_dices)\noriginal_dices.columns = ['img', 'Fish', 'Flower', 'Gravel', 'Sugar']\noriginal_dices['total_dice'] = original_dices[['Fish', 'Flower', 'Gravel', 'Sugar']].sum(1)\noriginal_dices['mean_dice'] = original_dices['total_dice'] / 4\noriginal_dices.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for c in ['Fish', 'Flower', 'Gravel', 'Sugar']:\n    print(f\"Mean dice for {c} is {original_dices[c].mean():.4f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the raw predictions give quite a bad dice, which is to be expected. Let's try post-processing!"},{"metadata":{},"cell_type":"markdown","source":"## Post processing"},{"metadata":{"trusted":true},"cell_type":"code","source":"class_params = {}\nfor class_id in range(4):\n    print(class_id)\n    attempts = []\n    for t in range(0, 100, 10):\n        t /= 100\n        for ms in [0, 100, 1000, 5000, 10000, 11000, 14000, 15000, 16000, 18000, 19000, 20000, 21000, 23000, 25000, 27000, 30000, 50000]:\n            masks = []\n            for i in range(class_id, len(probabilities), 4):\n                probability = probabilities[i]\n                predict, num_predict = post_process(sigmoid(probability), t, ms)\n                masks.append(predict)\n\n            d = []\n            for i, j in zip(masks, valid_masks[class_id::4]):\n                if (i.sum() == 0) & (j.sum() == 0):\n                    d.append(1)\n                else:\n                    d.append(dice(i, j))\n\n            attempts.append((t, ms, np.mean(d)))\n\n    attempts_df = pd.DataFrame(attempts, columns=['threshold', 'size', 'dice'])\n\n    attempts_df = attempts_df.sort_values('dice', ascending=False)\n    print(attempts_df.head())\n    best_threshold = attempts_df['threshold'].values[0]\n    best_size = attempts_df['size'].values[0]\n\n    class_params[class_id] = (best_threshold, best_size)\n\nprint(class_params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"processed_dices = []\nd = []\nfor ind, (i, j) in enumerate(zip(probabilities, valid_masks)):\n    i, num_predict = post_process(sigmoid(i), class_params[ind % 4][0],\n                                                       class_params[ind % 4][1])\n    if (i.sum() == 0) & (j.sum() == 0):\n        d.append(1)\n    else:\n        d.append(dice(i, j))\n    if len(d) == 4:\n        d = [valid_ids[ind // 4]] + d\n        processed_dices.append(d)\n        d = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"processed_dices = pd.DataFrame(processed_dices)\nprocessed_dices.columns = ['img', 'Fish', 'Flower', 'Gravel', 'Sugar']\nprocessed_dices['total_dice'] = processed_dices[['Fish', 'Flower', 'Gravel', 'Sugar']].sum(1)\nprocessed_dices['mean_dice'] = processed_dices['total_dice'] / 4\nprocessed_dices.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for c in ['Fish', 'Flower', 'Gravel', 'Sugar']:\n    print(f\"Mean dice for {c} is {processed_dices[c].mean():.4f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Of course, now dice is much better."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (12, 6))\nplt.subplot(1, 2, 1)\nplt.hist(original_dices['Fish'], label='Fish');\nplt.hist(original_dices['Flower'], label='Flower');\nplt.hist(original_dices['Gravel'], label='Gravel');\nplt.hist(original_dices['Sugar'], label='Sugar');\nplt.title('Original dices');\n\nplt.subplot(1, 2, 2)\nplt.hist(processed_dices['Fish'], label='Fish');\nplt.hist(processed_dices['Flower'], label='Flower');\nplt.hist(processed_dices['Gravel'], label='Gravel');\nplt.hist(processed_dices['Sugar'], label='Sugar');\nplt.title('Processed dices');\nplt.legend();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that the main improvement is thanks to making some masks empty correctly! But is this always so? Let's have a look at images which had mean dice increased/decreased the most."},{"metadata":{"trusted":true},"cell_type":"code","source":"processed_dices.columns = [f'{col}_processed' for col in processed_dices.columns]\ndices = pd.merge(original_dices, processed_dices, left_on='img', right_on='img_processed')\ndices['dice_diff'] = dices['mean_dice_processed'] - dices['mean_dice']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dices.sort_values('dice_diff').head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dices.sort_values('dice_diff', ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that our post processing sometimes created additional problems. One of the ways to fix it is to apply a classifier for post-processing."},{"metadata":{},"cell_type":"markdown","source":"## Using classifier"},{"metadata":{"trusted":true},"cell_type":"code","source":"# converting probabilities from classifier to labels\nbinary_classifier_predictions = np.zeros_like(valid_predictions)\nfor i in range(4):\n    binary_classifier_predictions[:, i] = (valid_predictions[:, i] > class_thresholds[i]) * 1\n    \nbinary_classifier_predictions = binary_classifier_predictions.reshape(-1, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"classified_dices = []\nd = []\nfor ind, (i, j) in enumerate(zip(probabilities, valid_masks)):\n    i, num_predict = post_process(sigmoid(i), class_params[ind % 4][0],\n                                                       class_params[ind % 4][1])\n    i = i if binary_classifier_predictions[ind] == 1 else i * 0\n    if (i.sum() == 0) & (j.sum() == 0):\n        d.append(1)\n    else:\n        d.append(dice(i, j))\n    if len(d) == 4:\n        d = [valid_ids[ind // 4]] + d\n        classified_dices.append(d)\n        d = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"classified_dices = pd.DataFrame(classified_dices)\nclassified_dices.columns = ['img', 'Fish', 'Flower', 'Gravel', 'Sugar']\nclassified_dices['total_dice'] = classified_dices[['Fish', 'Flower', 'Gravel', 'Sugar']].sum(1)\nclassified_dices['mean_dice'] = classified_dices['total_dice'] / 4\nclassified_dices.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for c in ['Fish', 'Flower', 'Gravel', 'Sugar']:\n    print(f\"Mean dice for {c} is {classified_dices[c].mean():.4f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Using different classifier"},{"metadata":{"trusted":true},"cell_type":"code","source":"# converting probabilities from classifier to labels\nbinary_classifier_predictions = np.zeros_like(valid_predictions)\nfor i in range(4):\n    binary_classifier_predictions[:, i] = (valid_predictions[:, i] > recall_thresholds[i]) * 1\n    \nbinary_classifier_predictions = binary_classifier_predictions.reshape(-1, 1)\n\nclassified_dices = []\nd = []\nfor ind, (i, j) in enumerate(zip(probabilities, valid_masks)):\n    i, num_predict = post_process(sigmoid(i), class_params[ind % 4][0],\n                                                       class_params[ind % 4][1])\n    i = i if binary_classifier_predictions[ind] == 1 else i * 0\n    if (i.sum() == 0) & (j.sum() == 0):\n        d.append(1)\n    else:\n        d.append(dice(i, j))\n    if len(d) == 4:\n        d = [valid_ids[ind // 4]] + d\n        classified_dices.append(d)\n        d = []\n        \nclassified_dices = pd.DataFrame(classified_dices)\nclassified_dices.columns = ['img', 'Fish', 'Flower', 'Gravel', 'Sugar']\nclassified_dices['total_dice'] = classified_dices[['Fish', 'Flower', 'Gravel', 'Sugar']].sum(1)\nclassified_dices['mean_dice'] = classified_dices['total_dice'] / 4\nclassified_dices.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for c in ['Fish', 'Flower', 'Gravel', 'Sugar']:\n    print(f\"Mean dice for {c} is {classified_dices[c].mean():.4f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ideas for improvement\n\nThis was only one of the ways of improving the score, here are some more ideas:\n- try applying classifier at first and then finding thresholds for post-processing;\n- train segmentation only on images on masks;\n- while optimizing classifier thresholds and post processing, do it at the same time, trying to maximize dice;\n- use different way to find classifier thresholds;\n- use some ideas from severstal competition;"},{"metadata":{},"cell_type":"markdown","source":"## Making predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# predict with classifier\ntorch.cuda.empty_cache()\ndel runner\n\ntask = 'classification'\nencoder = 'densenet169'\nbatch_size = 8\n\nmodel = get_model(model_type=segm_type, encoder=encoder, encoder_weights=encoder_weights,\n                  activation=activation, task=task, n_classes=n_classes)\nloaders = prepare_loaders(path=path, bs=batch_size,\n                          num_workers=num_workers, preprocessing_fn=preprocessing_fn, preload=False, task=task,\n                          image_size=(224, 224))\ndel loaders['train']\ndel loaders['valid']\ncheckpoint_path = f\"{logdir}/checkpoints/best.pth\"\nrunner = SupervisedRunner()\ntest_predictions = runner.predict_loader(\n    model, loaders[\"test\"],\n    resume=checkpoint_path, verbose=True\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# convert classifier predictions into labels\nbinary_classifier_predictions = np.zeros_like(test_predictions)\nfor i in range(4):\n    binary_classifier_predictions[:, i] = (test_predictions[:, i] > class_thresholds[i]) * 1\n    \nbinary_classifier_predictions = binary_classifier_predictions.reshape(-1, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"binary_classifier_predictions1 = np.zeros_like(test_predictions)\nfor i in range(4):\n    binary_classifier_predictions1[:, i] = (test_predictions[:, i] > recall_thresholds[i]) * 1\n    \nbinary_classifier_predictions1 = binary_classifier_predictions1.reshape(-1, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.cuda.empty_cache()\ndel runner\ndel test_predictions\ndel probabilities\ndel model\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"task = 'segmentation'\nencoder = 'resnet50'\nbatch_size = 8\n\nmodel = get_model(model_type=segm_type, encoder=encoder, encoder_weights=encoder_weights,\n                  activation=activation, task=task, n_classes=n_classes)\nloaders = prepare_loaders(path=path, bs=batch_size,\n                          num_workers=num_workers, preprocessing_fn=preprocessing_fn, preload=False, task=task)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loaders['test'] = test_loader\ncheckpoint_path = '../input/cloud-segmentation-model/best_full.pth'\ncheckpoint = utils.load_checkpoint(checkpoint_path)\nmodel.cuda()\nutils.unpack_checkpoint(checkpoint, model=model)\nrunner = SupervisedRunner(model=model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"encoded_pixels = []\nencoded_pixels1 = []\nimage_id = 0\nfor _, test_batch in enumerate(loaders['test']):\n    runner_out = runner.predict_batch({\"features\": test_batch[0].cuda()})['logits']\n    for _, batch in enumerate(runner_out):\n        for probability in batch:\n\n            probability = probability.cpu().detach().numpy()\n            if probability.shape != (350, 525):\n                probability = cv2.resize(probability, dsize=(525, 350), interpolation=cv2.INTER_LINEAR)\n                prediction, num_predict = post_process(sigmoid(probability), class_params[image_id % 4][0],\n                                                   class_params[image_id % 4][1])\n                prediction = prediction if binary_classifier_predictions[image_id] == 1 else prediction * 0\n                prediction1 = prediction if binary_classifier_predictions1[image_id] == 1 else prediction * 0\n            if num_predict == 0:\n                encoded_pixels.append('')\n                encoded_pixels1.append('')\n            else:\n                r = mask2rle(prediction)\n                encoded_pixels.append(r)\n                r1 = mask2rle(prediction1)\n                encoded_pixels1.append(r1)\n            image_id += 1\n\nsub = pd.read_csv(f'{path}/sample_submission.csv')\nsub['EncodedPixels'] = encoded_pixels\nsub.to_csv(f'submission.csv', columns=['Image_Label', 'EncodedPixels'], index=False)\nsub['EncodedPixels'] = encoded_pixels1\nsub.to_csv(f'submission1.csv', columns=['Image_Label', 'EncodedPixels'], index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}