{"cells":[{"metadata":{},"cell_type":"markdown","source":"note: Added comparison of 2019 data and 2020 data. As a final push to the LB score, it seems worthwhile to add an inference model with reference to the heatmap."},{"metadata":{},"cell_type":"markdown","source":"## I hope this prob-list is helpful for you!"},{"metadata":{},"cell_type":"markdown","source":"It is a derivative of [this notebook](https://www.kaggle.com/marutama/optuna-tuning-resnext50-blending-inference).\n"},{"metadata":{},"cell_type":"markdown","source":"Than kyou for great previous work. please, upvote them!\n\n* [Cassava Leaf Disease - Exploratory Data Analysis](https://www.kaggle.com/ihelon/cassava-leaf-disease-exploratory-data-analysis)\n* [Ensemble: Resnext50_32x4d + Efficientnet = 0.903](https://www.kaggle.com/japandata509/ensemble-resnext50-32x4d-efficientnet-0-903)\n* [[No TTA] Cassava Resnext50_32x4d Inference lb0.903](https://www.kaggle.com/piantic/no-tta-cassava-resnext50-32x4d-inference-lb0-903/output)\n* [Clean_Inference_Kernel_8xTTA_LB902](https://www.kaggle.com/underwearfitting/clean-inference-kernel-8xtta-lb902/data)\n* [Cassava-ensemble-(efnetb3-resnet50)](https://www.kaggle.com/shubham108/cassava-ensemble-efnetb3-resnet50)"},{"metadata":{},"cell_type":"markdown","source":"# EDA of Resnext and Efficientnet inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\n\nimport scipy as sp\nfrom scipy.special import softmax\nimport numpy as np\nimport pandas as pd\nimport json\n\nfrom sklearn.metrics import accuracy_score\n\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\n\nimport cv2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_DIR = \"../input/cassava-leaf-disease-classification/\"\nwith open(os.path.join(BASE_DIR, \"label_num_to_disease_map.json\")) as file:\n    j = json.loads(file.read())\n\nmap_classes = {int(k) : v for k, v in j.items()}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(json.dumps(j, indent=4))\nprint(j[\"1\"]) # index is character string","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"map_classes[0] = 'CBB'\nmap_classes[1] = 'CBSD'\nmap_classes[2] = 'CGM'\nmap_classes[3] = 'CMD'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(json.dumps(map_classes, indent=4))\nprint(map_classes[4]) # index is integer","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# You can also use pprint to display the json object. This is generally useful.\nfrom pprint import pprint\nprint('Index is character string.')\npprint(j)\nprint('Index is integer.')\npprint(map_classes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Read all-rnxt-effn.csv\nThis Dataframe is inference results of Resnext504d and Effcientnet-b4.\n\n* image_id : Image filename of merged.csv including train.csv\n* correct : label of merged.csv including train.csv\n* Rnxt : Resnext inference labels\n* Effn : Efficientnet inference labels\n* 50%+50% : lResnext and Effcientnet ensemble labels\n* r0_0 : Probability of Resnext model 0, category 0 \n* r0_1 : Probability of Resnext model 0, category 1 \n* ...\n* e0_0 : Probability of Efficientnet model 0, category 0 \n* e0_1 : Probability of Efficientnet model 0, category 1 \n* ..."},{"metadata":{"trusted":true},"cell_type":"code","source":"a=pd.read_csv('../input/all-rnxt-effn/all-rnxt-effn.csv')\na","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculate confidence\n\nCalculate confidence of '50%+50%'"},{"metadata":{"trusted":true},"cell_type":"code","source":"a['r_0']=a['r0_0']+a['r1_0']+a['r2_0']+a['r3_0']+a['r4_0']\na['r_1']=a['r0_1']+a['r1_1']+a['r2_1']+a['r3_1']+a['r4_1']\na['r_2']=a['r0_2']+a['r1_2']+a['r2_2']+a['r3_2']+a['r4_2']\na['r_3']=a['r0_3']+a['r1_3']+a['r2_3']+a['r3_3']+a['r4_3']\na['r_4']=a['r0_4']+a['r1_4']+a['r2_4']+a['r3_4']+a['r4_4']\n\na['e_0']=a['e0_0']+a['e1_0']+a['e2_0']+a['e3_0']+a['e4_0']\na['e_1']=a['e0_1']+a['e1_1']+a['e2_1']+a['e3_1']+a['e4_1']\na['e_2']=a['e0_2']+a['e1_2']+a['e2_2']+a['e3_2']+a['e4_2']\na['e_3']=a['e0_3']+a['e1_3']+a['e2_3']+a['e3_3']+a['e4_3']\na['e_4']=a['e0_4']+a['e1_4']+a['e2_4']+a['e3_4']+a['e4_4']\n\na['re_0']=a['r_0']+a['e_0']\na['re_1']=a['r_1']+a['e_1']\na['re_2']=a['r_2']+a['e_2']\na['re_3']=a['r_3']+a['e_3']\na['re_4']=a['r_4']+a['e_4']\n\na['confidence']=a[['re_0', 're_1', 're_2', 're_3', 're_4']].max(1)/10","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b=a[['image_id', 'correct', 'Rnxt', 'Effn', '50%+50%', 'confidence']]\nb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## Boundary between 2019 and 2020 data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b2020 = b[:21396]\nb2020","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b2019 = b[21396:]\nb2019","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot of confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 4))\nplt.plot(b2020['confidence'])\nplt.plot(b2019['confidence'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Confidence seems to be a bit high in the second half of 2019 data"},{"metadata":{"trusted":true},"cell_type":"code","source":"b['confidence'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.displot(b['confidence'], height=5, aspect=2)\n#sns.distplot(b['confidence']) # if your seaborn version is old","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b2020['confidence'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.displot(b2020['confidence'], height=5, aspect=2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"b2019['confidence'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.displot(b2019['confidence'], height=5, aspect=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There seems to be no big difference in confidence distribution between 2020 and 2019."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Merged acc', accuracy_score(b['correct'], b['50%+50%']))\nprint('2020   acc', accuracy_score(b2020['correct'], b2020['50%+50%']))\nprint('2019   acc', accuracy_score(b2019['correct'], b2019['50%+50%']))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a difference in accuracy between 2020 and 2019, but it is about 20%."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 4))\nfig=sns.countplot(y='50%+50%', hue='correct', data=b)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 4))\nfig=sns.countplot(y='50%+50%', hue='correct', data=b)\nfig.set(xlim=(0,350))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 4))\nfig=sns.countplot(y='50%+50%', hue='correct', data=b2020)\nfig.set(xlim=(0,350))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(12, 4))\nfig=sns.countplot(y='50%+50%', hue='correct', data=b2019)\nfig.set(xlim=(0,350))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a pivot table with the number of occurrences\nm=b.pivot_table(index='correct', columns='50%+50%', values='image_id', aggfunc='count')\n# Clip maximums for heatmap\nm=m.clip(upper=int(278*1.2)) # 1.2 times the largest number other than the diagonal\nplt.figure(figsize=(15, 5)) \nsns.heatmap(m, cmap='rainbow', annot=True, fmt='g')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a pivot table with the number of occurrences\nm=b2020.pivot_table(index='correct', columns='50%+50%', values='image_id', aggfunc='count')\n# Clip maximums for heatmap\nm=m.clip(upper=int(220*1.2)) # 1.2 times the largest number other than the diagonal\nplt.figure(figsize=(15, 5)) \nsns.heatmap(m, cmap='rainbow', annot=True, fmt='g')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a pivot table with the number of occurrences\nm=b2019.pivot_table(index='correct', columns='50%+50%', values='image_id', aggfunc='count')\n# Clip maximums for heatmap\nm=m.clip(upper=int(58*1.2)) # 1.2 times the largest number other than the diagonal\nplt.figure(figsize=(15, 5)) \nsns.heatmap(m, cmap='rainbow', annot=True, fmt='g')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ヒートマップを参考に別モデルを導入することは有効そうです。ただ対応パターンは2019と2020で違いそうです。\n\nIt seems effective to introduce another model with reference to the heat map. However, the correspondence pattern seems to be different between 2019 and 2020."},{"metadata":{},"cell_type":"markdown","source":"2020データでは下記のケースが比較的多いようだ:\n* correct:4が間違えて50%+50%:0に予測されるケース\n* correct:4が間違えて50%+50%:3に予測されるケース\n* correct:2が間違えて50%+50%:3に予測されるケース\n* correct:1が間違えて50%+50%:4に予測されるケース\n* correct:0が間違えて50%+50%:4に予測されるケース\n\n\nThe following cases seem to be relatively common in the 2020 data:\n* correct: 4 is mistakenly predicted to be 50%+50%: 0\n* correct: 4 is mistakenly predicted to be 50%+50%: 3\n* correct: 2 is mistakenly predicted to be 50%+50%: 3\n* correct: 1 is mistakenly predicted to be 50%+50%: 4\n* correct: 0 is mistakenly predicted to be 50%+50%: 4"},{"metadata":{},"cell_type":"markdown","source":"# Show images"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_batch(df):\n    \n    image_ids = df[\"image_id\"].values\n    corrects = df['correct'].values\n    labels = df[\"50%+50%\"].values\n    confidences = df['confidence'].values\n    \n    plt.figure(figsize=(16, 12))\n    \n    for ind, (image_id, correct, label, conf) in enumerate(zip(image_ids, corrects, labels, confidences)):\n        plt.subplot(3, 3, ind + 1)\n        image = cv2.imread(os.path.join('../input/cassava-leaf-disease-merged/train', image_id))\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n        plt.imshow(image)\n        plt.title(f\"correct:{correct}({map_classes[correct]}), 50%+50%:{label}({map_classes[label]}),\\n{image_id}, conf={conf:.4f}\", fontsize=12)\n        plt.axis(\"off\")\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Display 9 images with low confidence value"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b.sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Images with different aspect ratios are 2019 data."},{"metadata":{},"cell_type":"markdown","source":"## Display 9 images with high confidence value"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b.sort_values('confidence', ascending=False)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correct label:0 Cassava Bacterial Blight (CBB)\n\nAngular necrotic spotting of the leaves—often with a chlorotic ring encircling the spots.\n\n葉の角のある壊死性の斑点—多くの場合、斑点を取り囲むクロロティックリングを伴う。"},{"metadata":{},"cell_type":"markdown","source":"## High confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==0].sort_values('confidence', ascending=False)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Low confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==0].sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correct label:1 Cassava Brown Streak Disease (CBSD)\n\nSevere chlorosis and necrosis on infected leaves, giving them a yellowish, mottled appearance.\n\n感染した葉の重度の白化と壊死により、黄色がかったまだらの外観になります。"},{"metadata":{},"cell_type":"markdown","source":"## High confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==1].sort_values('confidence', ascending=False)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hmmm, only the cross section of the root ..."},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==1].sort_values('confidence', ascending=False)[100:109]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Low confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==1].sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correct label:2 Cassava Green Mottle (CGM)\n\nYoung leaves are puckered with faint to distinct yellow spots, green patterns (mosaics), and twisted margins.\n\n若い葉は、かすかにはっきりとした黄色の斑点、緑色のパターン（モザイク）、およびねじれた縁でしわが寄っています。"},{"metadata":{},"cell_type":"markdown","source":"## High confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==2].sort_values('confidence', ascending=False)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Low confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==2].sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correct label:3 Cassava Mosaic Disease (CMD)\n\nThese symptoms include chlorotic mosaic of the leaves, leaf distortion, and stunted growth. Leaf stalks have a characteristic S-shape.\n\n植物全体に病徴が発現する。病徴には葉のモザイク状の白化、葉の変形、生育の阻害が含まれる。"},{"metadata":{},"cell_type":"markdown","source":"## High confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==3].sort_values('confidence', ascending=False)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Low confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==3].sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correct label:4 Healthy"},{"metadata":{},"cell_type":"markdown","source":"## High confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==4].sort_values('confidence', ascending=False)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Low confidence"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[b['correct']==4].sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In particular, there are clearly many images of illness on the Healthy correct label of the 2020 data. Label noise may be high in the 2020 data.\n2020 healthy low confidence : about 0.24 - 0.34\n2019 healthy low confidence : about 0.40 - 0.49"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b2019[b2019['correct']==4].sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Displays images with the same label of correct, Rnxt, Effn and 50% + 50%\n\nImages can be displayed in various patterns in this way."},{"metadata":{"trusted":true},"cell_type":"code","source":"t=b[(b['correct']==b['Rnxt'])&(b['correct']==b['Effn'])&(b['correct']==b['50%+50%'])]\n#t=b[(b['correct']==3) & (b['Rnxt']==3) & (b['Effn']==3) & (b['50%+50%']==3)]\ns=t.sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.displot(t['confidence'],height=5, aspect=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Other images"},{"metadata":{"trusted":true},"cell_type":"code","source":"t=b[~((b['correct']==b['Rnxt'])&(b['correct']==b['Effn'])&(b['correct']==b['50%+50%']))]\n#t=b[(b['correct']==3) & (b['Rnxt']==3) & (b['Effn']==3) & (b['50%+50%']==3)]\ns=t.sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.displot(t['confidence'],height=5, aspect=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"confidence が高いのに correctと異なる予測になっているのは label noiseが多いからでしょう。\n\nThe reason why the prediction is different from correct even though the confidence is high is probably because there is a lot of label noise."},{"metadata":{},"cell_type":"markdown","source":"# Let's check our assumptions\n\n* correct: 4 is mistakenly predicted to be 50%+50%: 0\n* correct: 4 is mistakenly predicted to be 50%+50%: 3\n* correct: 2 is mistakenly predicted to be 50%+50%: 3\n* correct: 1 is mistakenly predicted to be 50%+50%: 4\n* correct: 0 is mistakenly predicted to be 50%+50%: 4\n\nLet's look at only the first two here.\n\n\n## correct: 4 is mistakenly predicted to be 50%+50%: 0"},{"metadata":{"trusted":true},"cell_type":"code","source":"b.groupby('50%+50%').mean()['confidence']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[(b['correct']==4)&(b['50%+50%']==0)].sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## correct: 4 is mistakenly predicted to be 50%+50%: 3"},{"metadata":{"trusted":true},"cell_type":"code","source":"s=b[(b['correct']==4)&(b['50%+50%']==3)].sort_values('confidence', ascending=True)[:9]\nvisualize_batch(s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## I hope it helps you ..."},{"metadata":{"trusted":true},"cell_type":"code","source":"# If you want to execute the following code, comment out this Assert and turn it on with GPU Accelerator.\n# It takes 4 hours.\nassert(False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How to make 'all-rnxt-effn.csv'"},{"metadata":{},"cell_type":"markdown","source":"## Import Library"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Library\n# ====================================================\nimport sys\nimport os\nimport math\nimport time\nimport random\nimport shutil\nimport albumentations\nfrom pathlib import Path\nfrom contextlib import contextmanager\nfrom collections import defaultdict, Counter\n\nimport scipy as sp\nfrom scipy.special import softmax\nimport numpy as np\nimport pandas as pd\n\nfrom sklearn import preprocessing\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import StratifiedKFold\n\nfrom tqdm.auto import tqdm\nfrom functools import partial\n\nimport cv2\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Library for Pytorch and GPU\n# ====================================================\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.optim import Adam, SGD\nimport torchvision.models as models\nfrom torch.nn.parameter import Parameter\nfrom torch.utils.data import DataLoader, Dataset\nfrom torch.optim.lr_scheduler import CosineAnnealingWarmRestarts, CosineAnnealingLR, ReduceLROnPlateau\n\nimport albumentations as A\nfrom albumentations.pytorch import ToTensorV2\n\nsys.path.append('../input/pytorch-image-models/pytorch-image-models-master')\nimport timm\n\nimport warnings \nwarnings.filterwarnings('ignore')\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint(device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Directory settings"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Directory settings for Resnext\n# ====================================================\nOUTPUT_DIR = './'\nMODEL_DIR = '../input/cassava-resnext50-32x4d-weights/'\nif not os.path.exists(OUTPUT_DIR):\n    os.makedirs(OUTPUT_DIR)\n    \nTRAIN_PATH = '../input/cassava-leaf-disease-classification/train_images'\n#TEST_PATH = '../input/cassava-leaf-disease-classification/test_images'\nMERGED_PATH = '../input/cassava-leaf-disease-merged/train'\nTEST_PATH = MERGED_PATH\nMERGED_CSV = '../input/cassava-leaf-disease-merged/merged.csv'\n#SUBM_CSV = '../input/cassava-leaf-disease-classification/sample_submission.csv   '","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Configuration setting"},{"metadata":{"trusted":true},"cell_type":"code","source":"#BATCH_SIZE ... ResNext inference time(min) : in case of 26337(MERGED num) 512x512 img, GCP Tesla T4\n#       T4     ,  P100(Kaggle)\n#01 ... 86(min)\n#02 ... 64(min)\n#04 ... 53(min)\n#08 ... 51(min)\n#16 ... 47(min),  35(min)\n#32 ... 46(min),  34(min)\n#64 ... 45(min)\nBATCH_SIZE = 32\ntotal_file_nums = len(os.listdir(TEST_PATH))\nd = total_file_nums // BATCH_SIZE\nmax_file_nums = d * BATCH_SIZE\nprint(f'Adjust the number of input files by batch size: {total_file_nums}->{max_file_nums}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ninp_imgs= []\nfor dirname, _, filenames in os.walk(TEST_PATH):\n    for filename in filenames[:max_file_nums]:\n    #for filename in filenames[:BATCH_SIZE*16]: # デバッグ用入力画像数絞れる\n        #print(os.path.join(dirname, filename))\n        #print(filename)\n        inp_imgs.append(filename)\ninp_imgs.sort()\nprint(len(inp_imgs))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# CFG for ResNext\n# ====================================================\nclass CFG:\n    debug=False\n    num_workers=0  # original is 8\n    model_name='resnext50_32x4d'\n    size=512\n    batch_size=BATCH_SIZE # original is 32\n    seed=2020\n    target_size=5\n    target_col='label'\n    n_fold=5\n    trn_fold=[0, 1, 2, 3, 4]\n    inference=True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#for efficientnet\nEFF_BATCH_SIZE = 1 # Important: Increasing the batch size will decrease the Effn score\n# EFF_BATCH_SIZE=1 ... Effn inference is about 60min@T4(50min@P100) x 5 models\nimage_size = 512\nenet_type = ['tf_efficientnet_b4_ns'] * 5\nmodel_path = ['../input/moa-b4-baseline/baseline_cld_fold0_epoch8_tf_efficientnet_b4_ns_512.pth', \n              '../input/moa-b4-baseline/baseline_cld_fold1_epoch9_tf_efficientnet_b4_ns_512.pth', \n              '../input/moa-b4-baseline/baseline_cld_fold2_epoch9_tf_efficientnet_b4_ns_512.pth',\n              '../input/moa-b4-baseline/baseline_cld_fold3_epoch5_tf_efficientnet_b4_ns_512.pth',\n              '../input/moa-b4-baseline/baseline_cld_fold4_epoch11_tf_efficientnet_b4_ns_512.pth']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Transform for efficientnet\ntransforms_valid = albumentations.Compose([\n    albumentations.Resize(800, 600), # For 2019 data, it is smaller than 512x512\n    albumentations.CenterCrop(image_size, image_size, p=1),\n    albumentations.Resize(image_size, image_size),\n    albumentations.Normalize()\n])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Utils"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Utils for Resnext\n# ====================================================\ndef get_score(y_true, y_pred):\n    return accuracy_score(y_true, y_pred)\n\n\n@contextmanager\ndef timer(name):\n    t0 = time.time()\n    LOGGER.info(f'[{name}] start')\n    yield\n    LOGGER.info(f'[{name}] done in {time.time() - t0:.0f} s.')\n\n\ndef init_logger(log_file=OUTPUT_DIR+'inference.log'):\n    from logging import getLogger, INFO, FileHandler,  Formatter,  StreamHandler\n    logger = getLogger(__name__)\n    logger.setLevel(INFO)\n    handler1 = StreamHandler()\n    handler1.setFormatter(Formatter(\"%(message)s\"))\n    handler2 = FileHandler(filename=log_file)\n    handler2.setFormatter(Formatter(\"%(message)s\"))\n    logger.addHandler(handler1)\n    logger.addHandler(handler2)\n    return logger\n\n#LOGGER = init_logger()\n\ndef seed_torch(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nseed_torch(seed=CFG.seed)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Loading"},{"metadata":{"trusted":true},"cell_type":"code","source":"merged = pd.read_csv(MERGED_CSV)\nmerged = merged.set_index('image_id')\n#merged","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(inp_imgs)\n# 21397 TRAIN num\n# 26337 MERGED num","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp = merged.loc[inp_imgs]\ntest = tmp.rename(columns={'label':'correct'}).drop('source', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = test.reset_index()\n#test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# filepath is for Effcientnet dataloader\n#test = pd.read_csv('../input/cassava-leaf-disease-classification/sample_submission.csv')\ntest['filepath'] = test.image_id.apply(lambda x: os.path.join(MERGED_PATH, f'{x}'))\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Dataset for Resnext\n# ====================================================\nclass TestDataset(Dataset):\n    def __init__(self, df, transform=None):\n        self.df = df\n        self.file_names = df['image_id'].values\n        self.transform = transform\n        \n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        file_name = self.file_names[idx]\n        file_path = f'{TEST_PATH}/{file_name}'\n        image = cv2.imread(file_path)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        if self.transform:\n            augmented = self.transform(image=image)\n            image = augmented['image']\n        return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Dataset for efficientnet\n# ====================================================\nclass CLDDataset(Dataset):\n    def __init__(self, df, mode, transform=None):\n        self.df = df.reset_index(drop=True)\n        self.mode = mode\n        self.transform = transform\n        \n    def __len__(self):\n        return len(self.df)\n    \n    def __getitem__(self, index):\n        row = self.df.loc[index]\n        image = cv2.imread(row.filepath)\n        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n        \n        if self.transform is not None:\n            res = self.transform(image=image)\n            image = res['image']\n        \n        image = image.astype(np.float32)\n        image = image.transpose(2,0,1)\n        if self.mode == 'test':\n            return torch.tensor(image).float()\n        else:\n            return torch.tensor(image).float(), torch.tensor(row.label).float()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#for efficientnet\ntest_dataset_efficient = CLDDataset(test, 'test', transform=transforms_valid)\ntest_loader_efficient = torch.utils.data.DataLoader(test_dataset_efficient, batch_size=EFF_BATCH_SIZE, shuffle=False,  num_workers=0) # original num_workders=4","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Transforms for Resnext\n# ====================================================\ndef get_transforms(*, data):\n    if data == 'valid':\n        return A.Compose([\n            A.Resize(CFG.size, CFG.size),\n            A.Normalize(\n                mean=[0.485, 0.456, 0.406],\n                std=[0.229, 0.224, 0.225],\n            ),\n            ToTensorV2(),\n        ])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## MODELS"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# ResNext Model\n# ====================================================\nclass CustomResNext(nn.Module):\n    def __init__(self, model_name='resnext50_32x4d', pretrained=False):\n        super().__init__()\n        self.model = timm.create_model(model_name, pretrained=pretrained)\n        n_features = self.model.fc.in_features\n        self.model.fc = nn.Linear(n_features, CFG.target_size)\n\n    def forward(self, x):\n        x = self.model(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# EfficientNet Model\n# ====================================================\nclass enet_v2(nn.Module):\n\n    def __init__(self, backbone, out_dim, pretrained=False):\n        super(enet_v2, self).__init__()\n        self.enet = timm.create_model(backbone, pretrained=pretrained)\n        in_ch = self.enet.classifier.in_features\n        self.myfc = nn.Linear(in_ch, out_dim)\n        self.enet.classifier = nn.Identity()\n\n    def forward(self, x):\n        x = self.enet(x)\n        x = self.myfc(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Helper functions for Resnext\n# ====================================================\ndef load_state(model_path):\n    model = CustomResNext(CFG.model_name, pretrained=False)\n    try:  # single GPU model_file\n        model.load_state_dict(torch.load(model_path)['model'], strict=True)\n        state_dict = torch.load(model_path)['model']\n    except:  # multi GPU model_file\n        state_dict = torch.load(model_path)['model']\n        state_dict = {k[7:] if k.startswith('module.') else k: state_dict[k] for k in state_dict.keys()}\n\n    return state_dict\n\ndef inference(model, states, test_loader, device):\n    model.to(device)\n    tk0 = tqdm(enumerate(test_loader), total=len(test_loader), desc='[Rnxt All]')\n    probs = []\n    for i, (images) in tk0:\n        images = images.to(device)\n        avg_preds = []\n        for state in states:\n            #model.load_state_dict(state['model'])\n            model.load_state_dict(state)\n            model.eval()\n            with torch.no_grad():\n                y_preds = model(images)\n            avg_preds.append(y_preds.softmax(1).to('cpu').numpy())\n        #avg_preds = np.mean(avg_preds, axis=0)\n        probs.append(avg_preds)\n    probs = np.concatenate(probs)\n    return probs\n\n#tta_inference for ResNextを作るべきか？","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====================================================\n# Helper functions for efficientnet\n# ====================================================\ndef inference_func(test_loader):\n    model.eval()\n    bar = tqdm(test_loader, desc='[Effn]')\n\n    LOGITS = []\n    PREDS = []\n    \n    with torch.no_grad():\n        for batch_idx, images in enumerate(bar):\n            x = images.to(device)\n            logits = model(x)\n            LOGITS.append(logits.cpu())\n            PREDS += [torch.softmax(logits, 1).detach().cpu()]\n        PREDS = torch.cat(PREDS).cpu().numpy()\n        LOGITS = torch.cat(LOGITS).cpu().numpy()\n    return PREDS\n\ndef tta_inference_func(test_loader):\n    model.eval()\n    bar = tqdm(test_loader, desc='[Effn TTA]')\n    PREDS = []\n    LOGITS = []\n\n    with torch.no_grad():\n        for batch_idx, images in enumerate(bar):\n            x = images.to(device)\n            x = torch.stack([x,x.flip(-1),x.flip(-2),x.flip(-1,-2),\n            x.transpose(-1,-2),x.transpose(-1,-2).flip(-1),\n            x.transpose(-1,-2).flip(-2),x.transpose(-1,-2).flip(-1,-2)],0)\n            x = x.view(-1, 3, image_size, image_size)\n            logits = model(x)\n            logits = logits.view(EFF_BATCH_SIZE, 8, -1).mean(1)\n            PREDS += [torch.softmax(logits, 1).detach().cpu()]\n            LOGITS.append(logits.cpu())\n\n        PREDS = torch.cat(PREDS).cpu().numpy()\n        \n    return PREDS","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference ResNext"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nmodel = CustomResNext(CFG.model_name, pretrained=False)\n#model = enet_v2(enet_type[i], out_dim=5)\nstates = [load_state(MODEL_DIR+f'{CFG.model_name}_fold{fold}.pth') for fold in CFG.trn_fold]\ntest_dataset = TestDataset(test, transform=get_transforms(data='valid')) # NO TTA\ntest_loader = DataLoader(test_dataset, batch_size=CFG.batch_size, shuffle=False, \n                         num_workers=CFG.num_workers, pin_memory=True)\nprobs_rnxt = inference(model, states, test_loader, device)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split by model\nMODELS = 5\nr = [0]*MODELS\nfor i in range(len(probs_rnxt)//MODELS):\n    for m in range(MODELS):\n        if i==0:\n            r[m]=probs_rnxt[m]\n        else:\n            r[m] = np.concatenate([r[m], probs_rnxt[i*5+m]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inference Effcientnet"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ne = []\nfor i in range(len(enet_type)):\n    model = enet_v2(enet_type[i], out_dim=5)\n    model = model.to(device)\n    model.load_state_dict(torch.load(model_path[i]))\n    e += [tta_inference_func(test_loader_efficient)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(np.array(r).shape, np.array(e).shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" # Save Results "},{"metadata":{"trusted":true},"cell_type":"code","source":"test = test.drop('filepath', axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Score : RexNext\npred = np.mean(r, axis=0)\ntest['Rnxt'] = softmax(pred).argmax(1)\n# acc_scoreで検算チェック\nacc_score = accuracy_score(test['correct'], test['Rnxt'])\nprint(acc_score)\n# 0.909753703790251 ... TRAIN score\n# 0.8880282492311197 ... MERGED score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Score : Effcientnet\npred = np.mean(e, axis=0)\ntest['Effn'] = softmax(pred).argmax(1)\n# acc_scoreで検算チェック\nacc_score = accuracy_score(test['correct'], test['Effn'])\nprint(acc_score)\n# 0.909753703790251 ... TRAIN score\n# 0.8880282492311197 ... MERGED score","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pred = 0.50*np.mean(r, axis=0) + 0.50*np.mean(e, axis=0) # lb 0.903\ntest['50%+50%'] = softmax(pred).argmax(1)\n#test[['image_id', '50%+50%']].to_csv(OUTPUT_DIR+'submission.csv', index=False) # must change 50%+50% -> label\n# acc_scoreで検算チェック\nacc_score = accuracy_score(test['correct'], test['50%+50%'])\nprint(acc_score)\n# 0.909753703790251 ... TRAIN score\n# 0.8880282492311197 ... MERGED score\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Merge"},{"metadata":{"trusted":true},"cell_type":"code","source":"col = ['r0_0', 'r0_1', 'r0_2', 'r0_3', 'r0_4']\ndr0 = pd.DataFrame(data=r[0], columns=col, dtype='float32')\ncol = ['r1_0', 'r1_1', 'r1_2', 'r1_3', 'r1_4']\ndr1 = pd.DataFrame(data=r[1], columns=col, dtype='float32')\ncol = ['r2_0', 'r2_1', 'r2_2', 'r2_3', 'r2_4']\ndr2 = pd.DataFrame(data=r[2], columns=col, dtype='float32')\ncol = ['r3_0', 'r3_1', 'r3_2', 'r3_3', 'r3_4']\ndr3 = pd.DataFrame(data=r[3], columns=col, dtype='float32')\ncol = ['r4_0', 'r4_1', 'r4_2', 'r4_3', 'r4_4']\ndr4 = pd.DataFrame(data=r[4], columns=col, dtype='float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"col = ['e0_0', 'e0_1', 'e0_2', 'e0_3', 'e0_4']\nde0 = pd.DataFrame(data=e[0], columns=col, dtype='float32')\ncol = ['e1_0', 'e1_1', 'e1_2', 'e1_3', 'e1_4']\nde1 = pd.DataFrame(data=e[1], columns=col, dtype='float32')\ncol = ['e2_0', 'e2_1', 'e2_2', 'e2_3', 'e2_4']\nde2 = pd.DataFrame(data=e[2], columns=col, dtype='float32')\ncol = ['e3_0', 'e3_1', 'e3_2', 'e3_3', 'e3_4']\nde3 = pd.DataFrame(data=e[3], columns=col, dtype='float32')\ncol = ['e4_0', 'e4_1', 'e4_2', 'e4_3', 'e4_4']\nde4 = pd.DataFrame(data=e[4], columns=col, dtype='float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df = pd.merge(test,   dr0, left_index=True, right_index=True)\nall_df = pd.merge(all_df, dr1, left_index=True, right_index=True)\nall_df = pd.merge(all_df, dr2, left_index=True, right_index=True)\nall_df = pd.merge(all_df, dr3, left_index=True, right_index=True)\nall_df = pd.merge(all_df, dr4, left_index=True, right_index=True)\n\nall_df = pd.merge(all_df, de0, left_index=True, right_index=True)\nall_df = pd.merge(all_df, de1, left_index=True, right_index=True)\nall_df = pd.merge(all_df, de2, left_index=True, right_index=True)\nall_df = pd.merge(all_df, de3, left_index=True, right_index=True)\nall_df = pd.merge(all_df, de4, left_index=True, right_index=True)\n\nall_df.to_csv(\"all-rnxt-effn.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"all_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test[test['Rnxt']!=test['Effn']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}