{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# About\n* This notebook achieved top11%, with ensembing efn-b4-5folds and cassava-resnext50-32x4d-weights.\n* The resnext50 model weights is public weights, and others are trained on Kaggle by myself.\n    \n# Source Kernels\n* This notebook refered these great kernels below which I upvoted, thanks for your great job and best regards\n    \n    1. [EfficientNet CV](https://www.kaggle.com/ajaykumar7778/efficientnet-cv#Training)\n    2. [Pytorch Efficientnet Baseline ](https://www.kaggle.com/zzy990106/pytorch-efficientnet-baseline/notebook?select=weight.pt)\n    3. [EfficientNet and CutMixUp with TPU Predict Phase](https://www.kaggle.com/itsuki9180/efficientnet-and-cutmixup-with-tpu-predict-phase)\n    4. [Ensemble: Resnext50_32x4d + Efficientnet = 0.903](https://www.kaggle.com/japandata509/ensemble-resnext50-32x4d-efficientnet-0-903)\n    5. [ViT: CUDA as usual [Ensemble Inference]](https://www.kaggle.com/szuzhangzhi/vit-cuda-as-usual-ensemble-inference/data)\n    6. [LEAF CLASSIFICATION RESNEXT 50_32*4D](https://www.kaggle.com/manojprabhaakr/leaf-classification-resnext-50-32-4d/data)"},{"metadata":{},"cell_type":"markdown","source":"# Background\n\n## The Challenge\n\nAs the second-largest provider of carbohydrates in Africa, cassava is a key food security crop grown by smallholder farmers because it can withstand harsh conditions. At least 80% of household farms in Sub-Saharan Africa grow this starchy root, but viral diseases are major sources of poor yields. With the help of data science, it may be possible to identify common diseases so they can be treated.\n\nExisting methods of disease detection require farmers to solicit the help of government-funded agricultural experts to visually inspect and diagnose the plants. This suffers from being labor-intensive, low-supply and costly. As an added challenge, effective solutions for farmers must perform well under significant constraints, since African farmers may only have access to mobile-quality cameras with low-bandwidth.\n\nIn this competition, we introduce a dataset of 21,367 labeled images collected during a regular survey in Uganda. Most images were crowdsourced from farmers taking photos of their gardens, and annotated by experts at the National Crops Resources Research Institute (NaCRRI) in collaboration with the AI lab at Makerere University, Kampala. This is in a format that most realistically represents what farmers would need to diagnose in real life.\n\nYour task is to classify each cassava image into four disease categories or a fifth category indicating a healthy leaf. With your help, farmers may be able to quickly identify diseased plants, potentially saving their crops before they inflict irreparable damage.\n\n## Files\n\n**[train/test]_images** the image files. The full set of test images will only be available to your notebook when it is submitted for scoring. Expect to see roughly 15,000 images in the test set.\n\n**train.csv**\n* image_id the image file name.\n* label the ID code for the disease.\n\n**sample_submission.csv** A properly formatted sample submission, given the disclosed test set content.\n* image_id the image file name.\n* label the predicted ID code for the disease.\n\n**[train/test]_tfrecords** the image files in tfrecord format.\n\n**label_num_to_disease_map.json** The mapping between each disease code and the real disease name."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# load data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\n\ntrain_path = '../input/cassava-leaf-disease-classification/train_images/'\ntest_path = '../input/cassava-leaf-disease-classification/test_images/'\n\ntrain_csv = pd.read_csv('../input/cassava-leaf-disease-classification/train.csv')\ntest_csv = pd.read_csv('../input/cassava-leaf-disease-classification/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import cv2\nfrom torch.utils.data import Dataset\n\nclass MyDataset(Dataset):\n    \n    def __init__(self, dataframe, transform=None, test=False):\n        self.df = dataframe\n        self.transform = transform\n        self.test = test\n    \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, idx):\n        \n        label = self.df.label.values[idx]\n        p = self.df.image_id.values[idx]\n        \n        if self.test == False:\n            p_path = train_path + p\n        else:\n            p_path = test_path + p\n            \n        image = cv2.imread(p_path)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        \n        if self.transform:\n            image = self.transform(image=image)['image']\n        \n        return image, label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# process data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import albumentations\nfrom albumentations.pytorch import ToTensorV2\nfrom torch.utils.data import DataLoader\nfrom sklearn.model_selection import train_test_split, StratifiedKFold\n\nIMG_SIZE = 512\nBATCH_SIZE = 12\n\ntrain_transform = albumentations.Compose([\n    albumentations.RandomResizedCrop(IMG_SIZE, IMG_SIZE),\n    albumentations.Transpose(p=0.5),\n    albumentations.HorizontalFlip(p=0.5),\n    albumentations.VerticalFlip(p=0.5),\n    albumentations.ShiftScaleRotate(p=0.5),\n    albumentations.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit=0.2, val_shift_limit=0.2, p=0.5),\n    albumentations.RandomBrightnessContrast(brightness_limit=(-0.1,0.1), contrast_limit=(-0.1, 0.1), p=0.5),\n    albumentations.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n    albumentations.CoarseDropout(p=0.5),\n    albumentations.Cutout(p=0.5),\n    ToTensorV2(p=1.0),\n], p=1.)\n\nval_transform = albumentations.Compose([\n     albumentations.CenterCrop(IMG_SIZE, IMG_SIZE, p=1.),\n     albumentations.Resize(IMG_SIZE, IMG_SIZE),\n     albumentations.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], max_pixel_value=255.0, p=1.0),\n     ToTensorV2(p=1.0),\n], p=1.)\n\nkFold = StratifiedKFold(n_splits=10, shuffle=True, random_state=2021).split(train_csv.image_id, train_csv.label)\n\ntestset = MyDataset(test_csv, transform=val_transform, test=True)\ntest_loader = DataLoader(testset, batch_size=BATCH_SIZE, shuffle=False, num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(testset[0][0].shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# create model"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn as nn\nimport torch.optim as optim\nimport torch.nn.functional as F\nfrom transformers import get_cosine_schedule_with_warmup\n\n!pip install ../input/timm-install/timm-0.1.30-py3-none-any.whl\nimport timm\n\nimport sys\n\npackage_path = '../input/visiontransformer-pakage/VisionTransformer-Pytorch'\nsys.path.append(package_path)\nfrom vision_transformer_pytorch import VisionTransformer\n\nLR = 1e-3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# train model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_epoch(net, data_loader, device):\n    net.train()\n    train_batch_num = len(data_loader)\n    total_loss = 0\n    correct = 0\n    sample_num = 0\n\n    for batch_idx, (data, target) in enumerate(data_loader):\n        data = data.to(device).float()\n        target = target.to(device).long()\n\n        optimizer.zero_grad()\n        output = net(data)\n        loss = criterion(output, target)\n        loss.backward()\n        optimizer.step()\n\n        total_loss += loss.item()\n        prediction = torch.argmax(output, dim=1)\n        correct += (prediction == target).sum().item()\n        sample_num += len(data)\n    \n    loss = total_loss / train_batch_num\n    acc = correct / sample_num\n    return loss, acc\n\ndef test_epoch(net, data_loader, device):\n    net.eval()\n    test_batch_num = len(data_loader)\n    total_loss = 0\n    correct = 0\n    sample_num = 0\n\n    with torch.no_grad():\n        for batch_idx, (data, target) in enumerate(data_loader):\n            data = data.to(device).float()\n            target = target.to(device).long()\n\n            output = net(data)\n            loss = criterion(output, torch.squeeze(target).long())\n            total_loss += loss.item()\n            prediction = torch.argmax(output, dim=1)\n            correct += (prediction == target).sum().item()\n            sample_num += len(data)\n        \n    loss = total_loss / test_batch_num\n    acc = correct / sample_num\n    return loss, acc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# from tqdm import tqdm\n# fold_train_loss_list = []\n# fold_train_acc_list = []\n# fold_val_loss_list = []\n# fold_val_acc_list = []\n\n# for n_fold,(train_idx, val_idx) in enumerate(kFold):\n#     if n_fold != 9: continue\n#     # get i-fold\n#     train_df, val_df = train_csv.iloc[train_idx], train_csv.iloc[val_idx]\n    \n#     trainset = MyDataset(train_df, transform=train_transform)\n#     train_loader = DataLoader(trainset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)\n    \n#     valset = MyDataset(val_df, transform=val_transform)\n#     val_loader = DataLoader(valset, batch_size=BATCH_SIZE, shuffle=False, num_workers=4)\n    \n#     # define a new net\n#     net = timm.create_model('tf_efficientnet_b4_ns', pretrained=True, num_classes=5).to(device)\n#     #net.load_state_dict(torch.load('../input/efn-b4-ap-fold3-1/efn_b4_ap_fold3_weight_0.8841.pt'))\n#     # net = VisionTransformer.from_name('ViT-B_16', num_classes=5).to(device)\n#     criterion = nn.CrossEntropyLoss()\n#     optimizer = torch.optim.Adam(net.parameters(), lr=LR)\n#     scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n#         optimizer, mode='max', factor=0.5, patience=1, verbose=True, min_lr=1e-9)\n\n#     # train at this fold\n#     train_loss_list = []\n#     train_acc_list = []\n#     val_loss_list = []\n#     val_acc_list = []\n\n#     torch.cuda.empty_cache()\n\n#     EPOCHS = 50\n#     BEST_ACC = 0\n    \n#     print('Fold %d:' % n_fold)\n#     for epoch in range(EPOCHS):\n#         tk = tqdm(train_loader, total=len(train_loader), position=0, leave=True)\n#         train_loss, train_acc = train_epoch(net, tk, device)\n\n#         tk = tqdm(val_loader, total=len(val_loader), position=0, leave=True)\n#         val_loss, val_acc = test_epoch(net, tk, device)\n\n#         train_loss_list.append(train_loss)\n#         train_acc_list.append(train_acc)\n#         val_loss_list.append(val_loss)\n#         val_acc_list.append(val_acc)\n\n#         if val_acc > BEST_ACC:\n#             BEST_ACC = val_acc\n#             dirr = 'tf_efficientnet_b4_ns_fold' + str(n_fold) + '_weight_' + str(round(BEST_ACC, 4)) + '.pt'\n#             torch.save(net.state_dict(), dirr)\n\n#         print('epoch %d, train loss %.4lf, train acc %.4lf, test loss %.4lf, test acc %.4lf, , best acc %.4lf' %\n#         (epoch + 1, train_loss, train_acc, val_loss, val_acc, BEST_ACC))\n\n#         scheduler.step(val_acc)\n    \n#     fold_train_loss_list.append(train_loss_list)\n#     fold_train_acc_list.append(train_acc_list)\n#     fold_val_loss_list.append(val_loss_list)\n#     fold_val_acc_list.append(val_acc_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import matplotlib.pyplot as plt\n\n# for i in range(5):\n#     plt.figure(figsize=[20,9])\n#     plt.plot(fold_train_loss_list[i],label=\"train_loss\",color=\"red\")\n#     plt.plot(fold_train_acc_list[i],label=\"train_acc\",color=\"orange\")\n#     plt.plot(fold_val_loss_list[i],label=\"val_loss\",color=\"blue\")\n#     plt.plot(fold_val_acc_list[i],label=\"val_acc\",color=\"green\")\n#     plt.legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# predict data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# MODEL\n# ====================================================\nclass CustomResNext(nn.Module):\n    def __init__(self, model_name='resnext50_32x4d', pretrained=False):\n        super().__init__()\n        self.model = timm.create_model(model_name, pretrained=pretrained)\n        n_features = self.model.fc.in_features\n        self.model.fc = nn.Linear(n_features, 5)\n\n    def forward(self, x):\n        x = self.model(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Helper functions\n# ====================================================\ndef load_state(model_path):\n    model = CustomResNext('resnext50_32x4d', pretrained=False)\n    try:  # single GPU model_file\n        model.load_state_dict(torch.load(model_path)['model'], strict=True)\n        state_dict = torch.load(model_path)['model']\n    except:  # multi GPU model_file\n        state_dict = torch.load(model_path)['model']\n        state_dict = {k[7:] if k.startswith('module.') else k: state_dict[k] for k in state_dict.keys()}\n\n    return state_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nfrom tqdm import tqdm\n\nmodel_vit = VisionTransformer.from_name('ViT-B_16', num_classes=5).to(device)\nmodel_vit.load_state_dict(torch.load('../input/vit-b-16/ViT-B_16.pt'))\n\nmodel1 = CustomResNext('resnext50_32x4d', pretrained=False).to(device)\nstates1 = [load_state(f'../input/cassava-resnext50-32x4d-weights/resnext50_32x4d_fold{fold}.pth') for fold in range(5)]\n\nmodel2 = timm.create_model('tf_efficientnet_b3_ns', pretrained=False, num_classes=5).to(device)\nstates2 = [torch.load(f'../input/efn-b3-5folds/efn_b3_fold{fold}_weight.pt') for fold in range(5)]\n\nmodel3 = timm.create_model('tf_efficientnet_b4_ns', pretrained=False, num_classes=5).to(device)\nstates3 = [torch.load(f'../input/efn-b4-5folds/efn_b4_fold{fold}_weight.pt') for fold in range(5)]\n\nmodel4 = timm.create_model('tf_efficientnet_b4_ns', pretrained=False, num_classes=5).to(device)\nstates4 = [torch.load(f'../input/cassava-efn-b4-ns-10folds-weights/efn_b4_ns_fold{fold}_weight.pt') for fold in range(10)]\n\nmodel5 = timm.create_model('resnet50', pretrained=False, num_classes=5).to(device)\nstates5 = [torch.load(f'../input/resnet50-5folds/resNet50_fold{fold}_weight.pt') for fold in range(5)]\n\ntest_pred = []\nwith torch.no_grad():\n    for i, data in enumerate(tqdm(test_loader, position=0, leave=True)):\n        images, _ = data\n        images = images.to(device)\n        \n        avg_preds = []\n            \n        for state in states1:\n            model1.load_state_dict(state)\n            model1.eval()\n            pred = model1(images)\n            avg_preds.append(pred.to('cpu').numpy())\n            \n        for state in states2:\n            model2.load_state_dict(state)\n            model2.eval()\n            pred = model2(images)\n            avg_preds.append(pred.to('cpu').numpy())\n        \n        for state in states3:\n            model3.load_state_dict(state)\n            model3.eval()\n            pred = model3(images)\n            avg_preds.append(pred.to('cpu').numpy())\n        \n        for state in states4:\n            model4.load_state_dict(state)\n            model4.eval()\n            pred = model4(images)\n            avg_preds.append(pred.to('cpu').numpy())\n        \n        for state in states5:\n            model5.load_state_dict(state)\n            model5.eval()\n            pred = model5(images)\n            avg_preds.append(pred.to('cpu').numpy())\n\n        avg_preds = np.mean(avg_preds, axis=0)\n        \n        test_pred.append(avg_preds)\n\ntest_pred = np.concatenate(test_pred)\ntest_csv.label = test_pred.argmax(1)\ntest_csv.to_csv('submission.csv',index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}