{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Cassava Leaf Disease Classification - Exploratory Data Analysis / キャッサバの葉の病気分類 - 探索的データ分析\n\n### Japanese translation of [Cassava Leaf Disease Classification - Exploratory Data Analysis](https://www.kaggle.com/ihelon/cassava-leaf-disease-exploratory-data-analysis)\n\nQuick Exploratory Data Analysis for [Cassava Leaf Disease Classification](https://www.kaggle.com/c/cassava-leaf-disease-classification) challenge \n[キャッサバの葉の病気分類](https://www.kaggle.com/c/cassava-leaf-disease-classification)チャレンジのための簡単な探索的データ分析 \n\n\nThis competition will challenge you to distinguish between several diseases that cause material harm to the food supply of many African countries. In some cases the main remedy is to burn the infected plants to prevent further spread, which can make a rapid automated turnaround quite useful to the farmers. \nこのコンペティションでは、多くのアフリカ諸国の食糧供給に重大な害を及ぼすいくつかの病気を区別することが求められます。病気の対策として感染した植物を燃やすことで拡大を防ぐことが行われることもあり、これによる迅速な回復は農家にとって非常に有益です。"},{"metadata":{},"cell_type":"markdown","source":"![](https://storage.googleapis.com/kaggle-competitions/kaggle/13836/logos/header.png)"},{"metadata":{},"cell_type":"markdown","source":"\n\n
\n

Quick Navigation / クイックナビゲーション

\n\n* [1. Overview / 概要](#1)\n* [2. General Visualization / 一般的な可視化](#2)\n* [3. Cassava Bacterial Blight (CBB) / キャッサバ・バクテリア・ブライト (CBB)](#3)\n* [4. Cassava Brown Streak Disease (CBSD) / キャッサバ褐色条斑病 (CBSD)](#4)\n* [5. Cassava Green Mottle (CGM) / キャッサバ・グリーン・モットル (CGM)](#5)\n* [6. Cassava Mosaic Disease (CMD) / キャッサバモザイク病 (CMD)](#6)\n* [7. Healthy / 健康](#7)\n \n \n* [Augmentation Examples / データ拡張の例](#50)\n \n \n* [Submission Example / 提出ファイルの例](#100)"},{"metadata":{},"cell_type":"markdown","source":"\n

Overview / 概要

"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport json\n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sn\nimport matplotlib.pyplot as plt\nimport cv2\nimport albumentations as A\nfrom sklearn import metrics as sk_metrics","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"BASE_DIR = \"../input/cassava-leaf-disease-classification/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this competition we have 5 classes: **4 diseases** and **1 healthy** \nWe can find the mapping between the class number and its name in the file label_num_to_disease_map.json \nこのコンペティションでは5つの分類を行います。: **4つの病気と1つの健康** \nクラス番号とその名前のマッピングは、label_num_to_disease_map.json ファイルにあります。"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open(os.path.join(BASE_DIR, \"label_num_to_disease_map.json\")) as file:\n map_classes = json.loads(file.read())\n \nprint(json.dumps(map_classes, indent=4))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"input_files = os.listdir(os.path.join(BASE_DIR, \"train_images\"))\nprint(f\"Number of train images: {len(input_files)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's take a look at the dimensions of the first 300 images \nAs you can see below, all images are the same size (600, 800, 3) \n最初の300枚の画像の寸法を見てみましょう。 \n以下に示すように全ての画像は同じサイズ（600、800、3）です。"},{"metadata":{"trusted":true},"cell_type":"code","source":"img_shapes = {}\nfor image_name in os.listdir(os.path.join(BASE_DIR, \"train_images\"))[:300]:\n image = cv2.imread(os.path.join(BASE_DIR, \"train_images\", image_name))\n img_shapes[image.shape] = img_shapes.get(image.shape, 0) + 1\n\nprint(img_shapes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's load the training dataframe and add a column with the real class name to it. \nトレーニングデータのデータフレームをロードし、実際のクラス名を含むカラムを追加しましょう。"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(os.path.join(BASE_DIR, \"train.csv\"))\n\ndf_train[\"class_name\"] = df_train[\"label\"].astype(str).map(map_classes)\n\ndf_train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at the number of pictures in each class. \n各クラスの写真の数を見てみましょう。"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(8, 4))\nsn.countplot(x=\"label\", data=df_train);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, the dataset has a fairly large imbalance. \nこのようにデータセットにはかなり大きな不均衡があります。"},{"metadata":{},"cell_type":"markdown","source":"\n

General Visualization / 一般的な可視化

"},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_batch(image_ids, labels):\n plt.figure(figsize=(16, 12))\n \n for ind, (image_id, label) in enumerate(zip(image_ids, labels)):\n plt.subplot(3, 3, ind + 1)\n image = cv2.imread(os.path.join(BASE_DIR, \"train_images\", image_id))\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n plt.imshow(image)\n plt.title(f\"Class: {label}\", fontsize=12)\n plt.axis(\"off\")\n \n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_df = df_train.sample(9)\nimage_ids = tmp_df[\"image_id\"].values\nlabels = tmp_df[\"class_name\"].values\n\nvisualize_batch(image_ids, labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

Cassava Bacterial Blight (CBB) / キャッサバ・バクテリア・ブライト (CBB)

"},{"metadata":{},"cell_type":"markdown","source":"\nThe image from discussion / ディスカッションからの写真: Cassava Leaf Diseases: Overview / キャッサバの葉の病気: 概要"},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_df = df_train[df_train[\"label\"] == 0]\nprint(f\"Total train images for class 0: {tmp_df.shape[0]}\")\n\ntmp_df = tmp_df.sample(9)\nimage_ids = tmp_df[\"image_id\"].values\nlabels = tmp_df[\"label\"].values\n\nvisualize_batch(image_ids, labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

Cassava Brown Streak Disease (CBSD) / キャッサバ褐色条斑病 (CBSD)

"},{"metadata":{},"cell_type":"markdown","source":"\nThe image from discussion / ディスカッションからの写真: Cassava Leaf Diseases: Overview / キャッサバの葉の病気: 概要"},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_df = df_train[df_train[\"label\"] == 1]\nprint(f\"Total train images for class 1: {tmp_df.shape[0]}\")\n\ntmp_df = tmp_df.sample(9)\nimage_ids = tmp_df[\"image_id\"].values\nlabels = tmp_df[\"label\"].values\n\nvisualize_batch(image_ids, labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

Cassava Green Mottle (CGM) / キャッサバ・グリーン・モットル (CGM)

"},{"metadata":{},"cell_type":"markdown","source":"\nThe image from discussion / ディスカッションからの写真: Cassava Leaf Diseases: Overview / キャッサバの葉の病気: 概要"},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_df = df_train[df_train[\"label\"] == 2]\nprint(f\"Total train images for class 2: {tmp_df.shape[0]}\")\n\ntmp_df = tmp_df.sample(9)\nimage_ids = tmp_df[\"image_id\"].values\nlabels = tmp_df[\"label\"].values\n\nvisualize_batch(image_ids, labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

Cassava Mosaic Disease (CMD) / キャッサバモザイク病 (CMD)

"},{"metadata":{},"cell_type":"markdown","source":"\nThe image from discussion / ディスカッションからの写真: Cassava Leaf Diseases: Overview / キャッサバの葉の病気: 概要"},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_df = df_train[df_train[\"label\"] == 3]\nprint(f\"Total train images for class 3: {tmp_df.shape[0]}\")\n\ntmp_df = tmp_df.sample(9)\nimage_ids = tmp_df[\"image_id\"].values\nlabels = tmp_df[\"label\"].values\n\nvisualize_batch(image_ids, labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

Healthy / 健康

"},{"metadata":{"trusted":true},"cell_type":"code","source":"tmp_df = df_train[df_train[\"label\"] == 4]\nprint(f\"Total train images for class 4: {tmp_df.shape[0]}\")\n\ntmp_df = tmp_df.sample(9)\nimage_ids = tmp_df[\"image_id\"].values\nlabels = tmp_df[\"label\"].values\n\nvisualize_batch(image_ids, labels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

Augmentation Examples / データ拡張の例

"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_augmentation(image_id, transform):\n plt.figure(figsize=(16, 4))\n img = cv2.imread(os.path.join(BASE_DIR, \"train_images\", image_id))\n img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n\n plt.subplot(1, 3, 1)\n plt.imshow(img)\n plt.axis(\"off\")\n\n plt.subplot(1, 3, 2)\n x = transform(image=img)[\"image\"]\n plt.imshow(x)\n plt.axis(\"off\")\n\n plt.subplot(1, 3, 3)\n x = transform(image=img)[\"image\"]\n plt.imshow(x)\n plt.axis(\"off\")\n \n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since we have a fairly limited number of some classes, we can use augmentation \nThis section shows examples of augmentation using the [albumentations](https://albumentations.ai/) library \n一部のクラスの数はかなり限られているため、データ拡張を使用します。 \nここでは、 [albumentations](https://albumentations.ai/) ライブラリを使用したデータ拡張の例を示します。"},{"metadata":{},"cell_type":"markdown","source":"The example below uses rotate-shift-scale augmentation with specular edge complementation. For this kind of pictures, this augmentation looks quite natural. \n以下の例では、鏡面エッジ補完を使用した回転・シフト・スケール拡張を使用しています。この種の写真の場合、このデータ拡張は非常に自然に見えます。"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = A.ShiftScaleRotate(\n p=1.0, \n shift_limit=(-0.3, 0.3), \n scale_limit=(-0.1, 0.1), \n rotate_limit=(-180, 180), \n interpolation=0, \n border_mode=4, \n)\n\nplot_augmentation(\"1003442061.jpg\", transform)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Another useful augmentation could be CoarseDropout. Thanks to this augmentation, you can complicate the life of the model so that she does not look too closely at some of the details of the image. \nLet's look at the example below: \nもう1つの便利な拡張機能はCoarseDropoutです。この拡張機能により、モデルを複雑にして、モデルが画像の細部の一部をあまり詳しく見ないようにすることができます。 \n以下の例を見てみましょう。"},{"metadata":{"trusted":true},"cell_type":"code","source":"transform = A.CoarseDropout(\n p=1.0, \n max_holes=100, \n max_height=50, \n max_width=50, \n min_holes=30, \n min_height=20, \n min_width=20,\n)\n\nplot_augmentation(\"1003442061.jpg\", transform)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n

Submission Example / 提出ファイルの例

"},{"metadata":{},"cell_type":"markdown","source":"Load the submission template \n提出ファイルのテンプレートをロードします。"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sub = pd.read_csv(\"../input/cassava-leaf-disease-classification/sample_submission.csv\", index_col=0)\ndf_sub","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see only one file in the submission file \n提出ファイルには1つのファイルしかありません。"},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir(os.path.join(BASE_DIR, \"test_images\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is because [it is a Code Competition](https://www.kaggle.com/c/cassava-leaf-disease-classification/overview/code-requirements), and the test data is hidden \nYour notebook should correct working with unseen test dataset \nこれは、[コード・コンペティション](https://www.kaggle.com/c/cassava-leaf-disease-classification/overview/code-requirements)であるため、テストデータが非表示になっています。 \nノートブックは、目に見えないテストデータセットでの修正作業を行う必要があります。"},{"metadata":{},"cell_type":"markdown","source":"Let's choose the most popular class of training set as the label for all images in test set \nテストデータセット内のすべての画像のラベルの中で最も数の多いクラスのデータを選択しましょう。"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sub[\"label\"] = 3","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For example, let's calculate our accuracy (competition metric) with this forecast for a train data \nたとえば、トレインデータのこの予測を使用して、精度（コンペティション指標）を計算してみましょう。"},{"metadata":{"trusted":true},"cell_type":"code","source":"y_true = df_train[\"label\"].values\ny_pred = np.full_like(y_true, 3)\nprint(f\"accuracy score (predict 3): {sk_metrics.accuracy_score(y_true, y_pred):.3f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"in the same way, you can calculate the estimated accuracy for other predictions: \n同様に、他の予測についても精度を計算できます。"},{"metadata":{"trusted":true},"cell_type":"code","source":"for pred_class in range(0, 5):\n y_true = df_train[\"label\"].values\n y_pred = np.full_like(y_true, pred_class)\n print(f\"accuracy score (predict {pred_class}): {sk_metrics.accuracy_score(y_true, y_pred):.3f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Write result to the submission file \n結果を提出ファイルに書き込む。"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_sub.to_csv(\"submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If you submit the result for evaluation, you will get an accuracy of 0.614 on a public liderboard (on the train it is 0.615). This may indicate that there is also an imbalance of classes on the public test distribution. \n評価のために結果を送信すると、パブリックリーダーボードで0.614の精度が得られます（トレインデータでは0.615です）。 これは、パブリックテスト配布のクラスにも不均衡があることを示している可能性があります。"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}