{"cells":[{"metadata":{},"cell_type":"markdown","source":"version 5. Generate predictions of hold-out set in this code + 2019 dataset"},{"metadata":{},"cell_type":"markdown","source":"# 1 Introduction"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!cp -r ../input/cassava-public-ckpt/filter.jpeg ./ ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![filtering](filter.jpeg)\n> [Image Source](https://www.star-spain.com/en/blog/transittermstar-nxt-tooltips/filtering-data-records-termstar-nxt)"},{"metadata":{},"cell_type":"markdown","source":"This is an analysis of the noisy label of the Cassava dataset using CleanLab.\nIt should be able to give you **the explanation and inspiration of noisy label.**\n\nThe goal of this competition is to help increase crop yields by quickly identifying cassava plant diseases.\n\nThis is a **multi-class classification task**.\nAll images contain Cassava and there is a label for the disease status of the plant.\nThe evaluation metric is Accuracy.\n\n*train_images/* and *train.csv* are in familiar form containing images and correct answers. \nSince this competition is a code competition, only one sample image exists for provided *test_images/*.\nIn addition, the 2019 competition was held, so you can use the 2019 competition data.\n\nThe information that is actively shared on Discussion and Notebook is that the labels on this dataset are noisy.\nThis notebok would like to provide an in-depth analysis of noisy label using [CleanLab](https://github.com/cgnorthcutt/cleanlab).\n\nThanks [@tmhrkt](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/210557) for sharing CleanLab and I've learnt many things from this community.\nI refered to [@jacoporepossi's discussion](https://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/198143) for recognizing the disease's characteristics.\nIf this notebook helps, please leave an upvote."},{"metadata":{},"cell_type":"markdown","source":"# 2 Preparation"},{"metadata":{},"cell_type":"markdown","source":"Here loads the required libraries and data."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"!mkdir -p /tmp/pip/cache/\n!cp ../input/omegaconf/PyYAML-5.4b2-cp38-cp38-manylinux1_x86_64.whl /tmp/pip/cache/\n!cp ../input/omegaconf/omegaconf-2.0.5-py3-none-any.whl /tmp/pip/cache/\n!cp ../input/omegaconf/typing_extensions-3.7.4.3-py3-none-any.whl /tmp/pip/cache/\n!pip install --no-index --find-links /tmp/pip/cache/ omegaconf > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import sys\nsys.path.append('../input/timm-pytorch-image-models/pytorch-image-models-master')\nsys.path.append(\"../input/cleanlab/\")\n\nimport os\nimport sys\nfrom glob import glob\nimport numpy as np\nimport pandas as pd\nfrom omegaconf import DictConfig, OmegaConf\nimport cleanlab\nimport cv2\nfrom ast import literal_eval\nfrom collections import OrderedDict\nfrom sklearn.model_selection import StratifiedKFold\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport torch\nfrom torch.utils.data import Dataset, DataLoader\nimport pytorch_lightning as pl\nfrom pytorch_lightning.utilities.seed import seed_everything\nfrom pytorch_lightning.core.decorators import auto_move_data\nfrom pytorch_lightning.utilities.cloud_io import load as pl_load\nimport timm\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2-1 Generate Predictions"},{"metadata":{},"cell_type":"markdown","source":"Generate predictions for hold-out set with EfficientNet-b0"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"from albumentations import *\nfrom albumentations.pytorch import ToTensorV2\n\n\nmean, std = (0.485, 0.456, 0.406), (0.229, 0.224, 0.225)\n\ndef get_transforms(img_size=(512, 512)):\n    transformations = Compose([\n        PadIfNeeded(min_height=img_size[0], min_width=img_size[1]),\n        CenterCrop(img_size[0], img_size[1]),\n        Normalize(mean=mean, std=std, max_pixel_value=255.0, p=1.0),\n        ToTensorV2(p=1.0),\n    ], p=1.0)\n    return transformations","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"class TestDataset(Dataset):\n    def __init__(self, img_dir, df, img_size=384):\n        self.img_dir = img_dir\n        self.df = df\n        self.file_names = df['image_id'].values\n        self.transform = get_transforms(img_size=img_size)\n        \n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        file_name = self.file_names[idx]\n        file_path = os.path.join(self.img_dir, file_name)\n        image = cv2.imread(file_path)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        if self.transform:\n            augmented = self.transform(image=image)\n            image = augmented['image']\n        return image","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport timm\n\n\ndef create_model(model_name: str,\n                 pretrained: bool,\n                 num_classes: int,\n                 in_chans: int):\n    model = timm.create_model(model_name=model_name,\n                              pretrained=pretrained,\n                              num_classes=num_classes,\n                              in_chans=in_chans)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import os\nfrom glob import glob\nfrom collections import OrderedDict\nfrom pytorch_lightning.utilities.cloud_io import load as pl_load\n\n\ndef get_state_dict_from_checkpoint(log_dir, fold_num):\n    ckpt_path = glob(os.path.join(log_dir, f'checkpoints/*fold{fold_num}*.ckpt'))[0]\n    state_dict = pl_load(ckpt_path, map_location='cpu')\n    if 'state_dict' in state_dict:\n        state_dict = state_dict['state_dict']\n    did_distillation = False\n    state_dict = OrderedDict((k.replace('model.', '')\n                              if 'model.' in k else k, v) for k, v in\n                             state_dict.items())\n    return state_dict","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"class LitTester(pl.LightningModule):\n    def __init__(self, network_cfg, state_dict):\n        super(LitTester, self).__init__()\n        self.model = create_model(**network_cfg)\n        self.model.load_state_dict(state_dict)\n        self.model.eval()\n        \n    def forward(self, x):\n        x = self.model(x)\n        return x\n    \n    def test_step(self, batch, batch_idx):\n        score = torch.nn.functional.softmax(self(batch), dim=1)\n        score2 = torch.nn.functional.softmax(self(torch.flip(batch, [-1])), dim=1)\n        score3 = torch.nn.functional.softmax(self(torch.flip(batch, [-2])), dim=1)\n\n        out = (score + score2 + score3) / 3.0\n        return {\"pred\": out}\n\n    def test_epoch_end(self, output_results):\n        all_outputs = torch.cat([out[\"pred\"] for out in output_results], dim=0)\n        all_outputs = all_outputs.cpu().numpy()\n        return {'prob': all_outputs}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"eff_b0_cfg_s = \"\"\"\nbatch_size: 160\nimg_size: [512, 512]\nnetwork:\n  model_name: tf_efficientnet_b0_ns\n  pretrained: False\n  num_classes: 5\n  in_chans: 3\n\"\"\"\neff_b0_cfg = OmegaConf.create(eff_b0_cfg_s)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### Configurations\n\n# Checkpoint\nname = '14-10-36'\ncfg = eff_b0_cfg\n\n# Generate Predictions (GPU needed)\ndo_predict = True\n# Generate Submission (GPU needed)\ndo_submit = False\n\nimg_dir = '../input/cassava-leaf-disease-merged/train/'\nlabel_path = '../input/cassava-leaf-disease-merged/merged.csv'\nlog_dir = os.path.join('../input/cassava-public-ckpt', name)\nn_folds = len(glob(os.path.join(log_dir, 'checkpoints/*.ckpt')))\n\nnum2class = [\"Cassava Bacterial Blight (CBB)\", \"Cassava Brown Streak Disease (CBSD)\", \"Cassava Green Mottle (CGM)\", \"Cassava Mosaic Disease (CMD)\", \"Healthy\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set seed for reproducing\nseed_everything(42)\n\n# If there is no fold in label dataframe, generate with Stratified K-fold\nlabel_df = pd.read_csv(label_path)\nif 'fold' not in label_df.columns:\n    skf = StratifiedKFold(n_splits=5, shuffle=True)\n    label_df.loc[:, 'fold'] = 0\n    for fold_num, (train_index, val_index) in enumerate(skf.split(X=label_df.index, y=label_df.label.values)):\n        label_df.loc[label_df.iloc[val_index].index, 'fold'] = fold_num\n\n# Generate predictions for hold-out sets (True) or load a saved prediction (False).\nif do_predict:\n    infer = pl.Trainer(gpus=1)\n    oof_dict = {'image_id': [], 'label': [], 'fold': []}\n    for fold_num in range(n_folds):\n        val_df = label_df[label_df.fold == fold_num]\n        test_dataset = TestDataset(img_dir, val_df, img_size=cfg.img_size)\n        test_dataloader = DataLoader(test_dataset,\n                                     batch_size=cfg.batch_size,\n                                     num_workers=4,\n                                     shuffle=False)\n\n        state_dict = get_state_dict_from_checkpoint(log_dir, fold_num)\n        model = LitTester(cfg.network, state_dict)\n        pred = infer.test(model, test_dataloaders=test_dataloader, verbose=False)[0]\n        oof_dict['image_id'].extend(val_df.image_id.values)\n        oof_dict['label'].extend(pred['prob'].tolist())\n        oof_dict['fold'].extend([fold_num] * len(pred['prob']))\n    pred_df = pd.DataFrame(oof_dict)\n    pred_df.to_csv('oof.csv', index=False)\nelse:\n    pred_df = pd.read_csv(os.path.join(log_dir, 'oof.csv'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2-2 Generate Submission (Optional)"},{"metadata":{},"cell_type":"markdown","source":"If you want to submit the result from EfficientNet-b0(baseline), you can set *do_submit = True*"},{"metadata":{"trusted":true},"cell_type":"code","source":"if do_submit:\n    sub = pd.read_csv('../input/cassava-leaf-disease-classification/sample_submission.csv')\n    infer = pl.Trainer(gpus=1)\n    test_dataset = TestDataset('../input/cassava-leaf-disease-classification/test_images',\n                               sub,\n                               img_size=cfg.img_size)\n    test_dataloader = DataLoader(test_dataset,\n                                 batch_size=cfg.batch_size,\n                                 num_workers=4,\n                                 shuffle=False)\n    preds = []\n    for fold_num in range(n_folds):\n        state_dict = get_state_dict_from_checkpoint(log_dir, fold_num)\n        model = LitTester(cfg.network, state_dict)\n        pred = infer.test(model, test_dataloaders=test_dataloader, verbose=False)[0]\n        preds.append(pred['prob'])\n    sub['label'] = np.argmax(np.mean(preds, axis=0), axis=1)\n    sub.to_csv(os.path.join(os.getcwd(), 'submission.csv'), index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"label_df = label_df.sort_values(by='image_id', ascending=1)\npred_df = pred_df.sort_values(by='image_id', ascending=1)\n\nids, labels = label_df.image_id.values, label_df.label.values\npreds = np.array([literal_eval(pred) if isinstance(pred, str) else pred for pred in pred_df.label.values])\n\nprint(f'total {len(ids)} images')\nprint(f'prediction shape: {preds.shape}, label shape: {labels.shape}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3 CleanLab: Denoise Dataset Labels"},{"metadata":{},"cell_type":"markdown","source":"Here, we use **CleanLab** to guess the data with Noisy Label.\n\nThe data utilized here is the label given in the competition and the prediction probability of the trained model.\nThe model used is *EfficientNet-b0* and the probabilty was only gotten from the holdout set(5-folds)."},{"metadata":{},"cell_type":"markdown","source":"**(Note: This below code is taken from [cleanlab](https://github.com/cgnorthcutt/cleanlab) tutorial)**"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"# STEP 1 - Compute confident joint\n\n# Verify inputs\ns = labels\npsx = preds\n\n# Find the number of unique classes if K is not given\nK = len(np.unique(s))\n\n# Estimate the probability thresholds for confident counting\n# You can specify these thresholds yourself if you want\n# as you may want to optimize them using a validation set.\n# By default (and provably so) they are set to the average class prob.\nthresholds = [np.mean(psx[:,k][s == k]) for k in range(K)] # P(s^=k|s=k)\nthresholds = np.asarray(thresholds)\n\n# Compute confident joint\nconfident_joint = np.zeros((K, K), dtype = int)\nfor i, row in enumerate(psx):\n    s_label = s[i]\n    # Find out how many classes each example is confidently labeled as\n    confident_bins = row >= thresholds - 1e-6\n    num_confident_bins = sum(confident_bins)\n    # If more than one conf class, inc the count of the max prob class\n    if num_confident_bins == 1:\n        confident_joint[s_label][np.argmax(confident_bins)] += 1\n    elif num_confident_bins > 1:\n        confident_joint[s_label][np.argmax(row)] += 1\n\n# Normalize confident joint (use cleanlab, trust me on this)\nconfident_joint = cleanlab.latent_estimation.calibrate_confident_joint(\n    confident_joint, s)\n\ncleanlab.util.print_joint_matrix(confident_joint)\n\n# STEP 2 - Find label errors\n\n# We arbitrarily choose at least 5 examples left in every class.\n# Regardless of whether some of them might be label errors.\nMIN_NUM_PER_CLASS = 5\n# Leave at least MIN_NUM_PER_CLASS examples per class.\n# NOTE prune_count_matrix is transposed (relative to confident_joint)\nprune_count_matrix = cleanlab.pruning.keep_at_least_n_per_class(\n    prune_count_matrix=confident_joint.T,\n    n=MIN_NUM_PER_CLASS,\n)\n\ns_counts = np.bincount(s)\nnoise_masks_per_class = []\n# For each row in the transposed confident joint\nfor k in range(K):\n    noise_mask = np.zeros(len(psx), dtype=bool)\n    psx_k = psx[:, k]\n    if s_counts[k] > MIN_NUM_PER_CLASS:  # Don't prune if not MIN_NUM_PER_CLASS\n        for j in range(K):  # noisy label index (k is the true label index)\n            if k != j:  # Only prune for noise rates, not diagonal entries\n                num2prune = prune_count_matrix[k][j]\n                if num2prune > 0:\n                    # num2prune'th largest p(classk) - p(class j)\n                    # for x with noisy label j\n                    margin = psx_k - psx[:, j]\n                    s_filter = s == j\n                    threshold = -np.partition(\n                        -margin[s_filter], num2prune - 1\n                    )[num2prune - 1]\n                    noise_mask = noise_mask | (s_filter & (margin >= threshold))\n        noise_masks_per_class.append(noise_mask)\n    else:\n        noise_masks_per_class.append(np.zeros(len(s), dtype=bool))\n\n# Boolean label error mask\nlabel_errors_bool = np.stack(noise_masks_per_class).any(axis=0)\n\n # Remove label errors if given label == model prediction\nfor i, pred_label in enumerate(psx.argmax(axis=1)):\n    # np.all let's this work for multi_label and single label\n    if label_errors_bool[i] and np.all(pred_label == s[i]):\n        label_errors_bool[i] = False\n\n# Convert boolean mask to an ordered list of indices for label errors\nlabel_errors_idx = np.arange(len(s))[label_errors_bool]\n# self confidence is the holdout probability that an example\n# belongs to its given class label\nself_confidence = np.array(\n    [np.mean(psx[i][s[i]]) for i in label_errors_idx]\n)\nmargin = self_confidence - psx[label_errors_bool].max(axis=1)\nlabel_errors_idx = label_errors_idx[np.argsort(margin)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"total_idx = np.arange(len(ids))\nclean_idx = np.array([idx for idx in total_idx if idx not in label_errors_idx])\n\nguesses = np.stack(noise_masks_per_class).argmax(axis=0)\nguesses[clean_idx] = labels[clean_idx]\n\nclean_ids = ids[clean_idx]\nclean_labels = labels[clean_idx]\nclean_guesses = guesses[clean_idx]\n\nnoisy_ids = ids[label_errors_idx]\nnoisy_labels = labels[label_errors_idx]\nnoisy_guesses = guesses[label_errors_idx]\n\nprint(f'[clean ratio] \\t {len(clean_idx) / len(total_idx) * 100:.2f}%')\nprint(f'[noise ratio] \\t {len(noisy_ids) / len(total_idx) * 100:.2f}%')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4 Visualize Clean or Noisy Data"},{"metadata":{},"cell_type":"markdown","source":"Before analyzing the distribution, let's take a visual look at the results of CleanLab."},{"metadata":{},"cell_type":"markdown","source":"<span style=\"color:blue; font-size:14pt\">*Blue: a 'guess' class*</span>\n\n<span style=\"color:red; font-size:14pt\">*Red: a 'given' class*</span>"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"def visualize_images(ids, labels, guesses, target_class,\n                     n_rows=4, n_cols=4):\n    c_ids, c_guess, c_labels = ids[guesses == target_class], \\\n                               guesses[guesses == target_class], \\\n                               labels[guesses == target_class]\n\n    fig, axes = plt.subplots(n_rows, n_cols, sharex=True, sharey=True, figsize=(15, 12))\n    for i in range(n_rows*n_cols):\n        img = cv2.imread(os.path.join(img_dir, c_ids[i]))[...,::-1]\n        axes[i//n_rows][i%n_cols].imshow(img)\n        axes[i//n_rows][i%n_cols].set_title(f'{num2class[c_labels[i]]}\\n{c_ids[i]}', color='r')\n    plt.suptitle(f'Guess: {num2class[c_guess[i]]}', y=1.03, color='b', fontsize=20, fontweight='bold')\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4-1 Clean Label"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"visualize_images(clean_ids, clean_labels, clean_guesses, target_class=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- At first, angular, water-soaked spots occur on the leaves which are restricted by the veins; the spots are more clearly seen on the lower leaf surface. The spots expand rapidly, join together, especially along the margins of the leaves, and turn brown with yellow borders. Droplets of a creamy-white ooze occur at the centre of the spots; later, they turn yellow.\n- Stem infections block the flow of water and food and the leaves above wilt, die and fall, and branches die back.\n\n- Main characteristics to leverage: **angular spots, brown spots with yellow borders, yellow leaves, leaves wilting**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"visualize_images(clean_ids, clean_labels, clean_guesses, target_class=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Symptoms of cassava brown streak disease appear as patches of yellow areas mixed with normal green colour. The characteristic yellow or necrotic vein banding may enlarge and coalesce to form a large yellow patches.\n- The infected leaves do not become distorted in shape as occurs with leaves infected by Cassava mosaic disease.\n\n- Main characteristics to leverage: **yellow spots**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"visualize_images(clean_ids, clean_labels, clean_guesses, target_class=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Young leaves are puckered with faint to distinct yellow spots, green patterns (mosaics), and twisted margins. Occasionally, plants become severely stunted.\n\n- Main characteristics to leverage: **yellow patterns, irregular patches of yellow and green, leaf margins distortion, stunted**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"visualize_images(clean_ids, clean_labels, clean_guesses, target_class=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Cassava Mosaic Disease is characterized by sever mosaic symptoms on leaves, with affected leaved showing mottling and light-green, yellow or white spots. Discoloration, malformation and puckering of the leaf blade occur.\n\n- Main characteristics to leverage: **severe shape distortion, mosaic patterns**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"visualize_images(clean_ids, clean_labels, clean_guesses, target_class=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4-2 Noisy Label"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"visualize_images(noisy_ids, noisy_labels, noisy_guesses, target_class=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- At first, angular, water-soaked spots occur on the leaves which are restricted by the veins; the spots are more clearly seen on the lower leaf surface. The spots expand rapidly, join together, especially along the margins of the leaves, and turn brown with yellow borders. Droplets of a creamy-white ooze occur at the centre of the spots; later, they turn yellow.\n- Stem infections block the flow of water and food and the leaves above wilt, die and fall, and branches die back.\n\n- Main characteristics to leverage: **angular spots, brown spots with yellow borders, yellow leaves, leaves wilting**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"visualize_images(noisy_ids, noisy_labels, noisy_guesses, target_class=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Symptoms of cassava brown streak disease appear as patches of yellow areas mixed with normal green colour. The characteristic yellow or necrotic vein banding may enlarge and coalesce to form a large yellow patches.\n- The infected leaves do not become distorted in shape as occurs with leaves infected by Cassava mosaic disease.\n\n- Main characteristics to leverage: **yellow spots**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"visualize_images(noisy_ids, noisy_labels, noisy_guesses, target_class=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Young leaves are puckered with faint to distinct yellow spots, green patterns (mosaics), and twisted margins. Occasionally, plants become severely stunted.\n\n- Main characteristics to leverage: **yellow patterns, irregular patches of yellow and green, leaf margins distortion, stunted**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"visualize_images(noisy_ids, noisy_labels, noisy_guesses, target_class=3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Cassava Mosaic Disease is characterized by sever mosaic symptoms on leaves, with affected leaved showing mottling and light-green, yellow or white spots. Discoloration, malformation and puckering of the leaf blade occur.\n\n- Main characteristics to leverage: **severe shape distortion, mosaic patterns**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"visualize_images(noisy_ids, noisy_labels, noisy_guesses, target_class=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5 Noisy Label Analysis"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"all_data = pd.DataFrame({'image_id': ids,\n                         'given_label': labels,\n                         'guess_label': guesses})\nall_data['is_noisy'] = (all_data.given_label != all_data.guess_label)\nall_data['max_prob'] = preds.max(axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5-1 Noise Distribution"},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"plt.figure(figsize=(6, 4.5)) \nax = sns.countplot(x = 'is_noisy', data = all_data, palette=[\"#55967e\", \"#263959\"])\n\nplt.xticks( np.arange(2), ['Clean', 'Noise'] )\nplt.title('Noise Distribution (hold-out set)',fontsize= 14)\nplt.xlabel('')\nplt.ylabel('Number of images')\n\ncounts = all_data['is_noisy'].value_counts()\ncounts_pct = [f'{elem * 100:.2f}%' for elem in counts / counts.sum()]\nfor i, v in enumerate(counts_pct):\n    ax.text(i, 0, v, horizontalalignment = 'center', size = 14, color = 'w', fontweight = 'bold')\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Noisy Label isn't as much as expected (8.02%)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"noise_by_class = all_data.groupby('given_label')['is_noisy'].value_counts(normalize = True)\nnoise_by_class = noise_by_class.unstack()\nnoise_by_class.plot(kind='bar', stacked='True', color = [\"#55967e\", \"#263959\"]) # '#6d819c', '#e4e7ec'\nplt.legend(loc=(1.04,0))\n_ = plt.xticks(\n    rotation = 45, \n    horizontalalignment = 'right',\n    fontweight = 'light'  \n)\n\nplt.title('Noise Distribution by class',fontsize= 14)\nplt.xlabel('Class of the images')\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- There are many noisy labels from 0-Cassava Bacterial Blight (CBB) and 4-Healthy classes.\n- 3-Cassava Mosaic Disease (CMD) includes very small amount of noisy labels (1.7%)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(20, 5))\npalette = \"Set3\"\n\nplt.subplot(1, 3, 1)\nsns.boxplot(x = 'guess_label', y = 'max_prob', data = all_data,\n     palette = palette, fliersize = 0)\nsns.stripplot(x = 'guess_label', y = 'max_prob', data = all_data,\n     linewidth = 0.6, palette = palette)\nplt.title('Prediction Probability Distribution by Class',fontsize= 14)\nplt.ylim(0.2, 1.05)\nplt.ylabel('Prediction Probability')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- 0-Cassava Bacterial Blight (CBB) and 4-Healthy results a little confidence(probability).\n- 3-Cassava Mosaic Disease (CMD) results strong confidence."},{"metadata":{},"cell_type":"markdown","source":"## 5-2 How does a label change?"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"class_colors = np.array(['#fe4365', '#fc9d9a', '#f9cdad', '#c8c8a9', '#aacfd0'])\nnum2class = [f'{idx}-{elem}' for idx, elem in enumerate(num2class)]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class_num = 0\nlabel_shift = all_data[all_data.given_label==class_num]['guess_label'].value_counts(normalize=True)\n\nsns.set_style(\"whitegrid\")\nbar,ax = plt.subplots(figsize=(10,6))\nax = sns.barplot(x=label_shift.index, y=np.array(num2class)[label_shift.index],\n                 data=label_shift, ci=None, palette=class_colors[label_shift.index], orient='h' )\nax.set_title(f\"{num2class[class_num]} Changed to\", fontsize=15)\nax.set_xlabel (\"Percentage\")\nax.set_ylabel (\"Disease Type\")\nfor rect in ax.patches:\n    ax.text (rect.get_width(), rect.get_y() + rect.get_height() / 2,\"%.2f%%\"% rect.get_width(), weight='bold' )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class_num = 1\nlabel_shift = all_data[all_data.given_label==class_num]['guess_label'].value_counts(normalize=True)\n\nsns.set_style(\"whitegrid\")\nbar,ax = plt.subplots(figsize=(10,6))\nax = sns.barplot(x=label_shift.index, y=np.array(num2class)[label_shift.index],\n                 data=label_shift, ci=None, palette=class_colors[label_shift.index], orient='h' )\nax.set_title(f\"{num2class[class_num]} Changed to\", fontsize=15)\nax.set_xlabel (\"Percentage\")\nax.set_ylabel (\"Disease Type\")\nfor rect in ax.patches:\n    ax.text (rect.get_width(), rect.get_y() + rect.get_height() / 2,\"%.2f%%\"% rect.get_width(), weight='bold' )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class_num = 2\nlabel_shift = all_data[all_data.given_label==class_num]['guess_label'].value_counts(normalize=True)\n\nsns.set_style(\"whitegrid\")\nbar,ax = plt.subplots(figsize=(10,6))\nax = sns.barplot(x=label_shift.index, y=np.array(num2class)[label_shift.index],\n                 data=label_shift, ci=None, palette=class_colors[label_shift.index], orient='h' )\nax.set_title(f\"{num2class[class_num]} Changed to\", fontsize=15)\nax.set_xlabel (\"Percentage\")\nax.set_ylabel (\"Disease Type\")\nfor rect in ax.patches:\n    ax.text (rect.get_width(), rect.get_y() + rect.get_height() / 2,\"%.2f%%\"% rect.get_width(), weight='bold' )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class_num = 3\nlabel_shift = all_data[all_data.given_label==class_num]['guess_label'].value_counts(normalize=True)\n\nsns.set_style(\"whitegrid\")\nbar,ax = plt.subplots(figsize=(10,6))\nax = sns.barplot(x=label_shift.index, y=np.array(num2class)[label_shift.index],\n                 data=label_shift, ci=None, palette=class_colors[label_shift.index], orient='h' )\nax.set_title(f\"{num2class[class_num]} Changed to\", fontsize=15)\nax.set_xlabel (\"Percentage\")\nax.set_ylabel (\"Disease Type\")\nfor rect in ax.patches:\n    ax.text (rect.get_width(), rect.get_y() + rect.get_height() / 2,\"%.2f%%\"% rect.get_width(), weight='bold' )","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"class_num = 4\nlabel_shift = all_data[all_data.given_label==class_num]['guess_label'].value_counts(normalize=True)\n\nsns.set_style(\"whitegrid\")\nbar,ax = plt.subplots(figsize=(10,6))\nax = sns.barplot(x=label_shift.index, y=np.array(num2class)[label_shift.index],\n                 data=label_shift, ci=None, palette=class_colors[label_shift.index], orient='h' )\nax.set_title(f\"{num2class[class_num]} Changed to\", fontsize=15)\nax.set_xlabel (\"Percentage\")\nax.set_ylabel (\"Disease Type\")\nfor rect in ax.patches:\n    ax.text (rect.get_width(), rect.get_y() + rect.get_height() / 2,\"%.2f%%\"% rect.get_width(), weight='bold' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}