{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<img src=\"https://i.imgur.com/gb6B4ig.png\" width=\"400\" alt=\"Weights & Biases\" />\n\n\nThis kernel is the walkthrough of the newest feature by [Weights and Biases](https://wandb.ai/site) for Dataset and Prediction Visualization called [Tables](https://docs.wandb.ai/guides/data-vis). Learn how to quickly integrate W&B in your Kaggle workflow [here](https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases).\n\nData is at the core of every ML workflow. Tables let you visualize and query datasets and model evaluations at the example level. We can use this new tool to analyze and understand datasets, and to measure and debug model performance. I hope you will find it useful for this Kaggle competition and other competitions alike. \n\nThis Kernel is designed with Kaggle workflow in perspective. I would love to get your suggestions around Kaggle workflow and how Tables can help with the same. \n\n![img](https://i.imgur.com/P7bRGmR.gif)","metadata":{}},{"cell_type":"markdown","source":"# 🔧 Imports and Setups","metadata":{}},{"cell_type":"code","source":"# Install latest version of W&B\n!pip install -q --upgrade wandb\n\n# Import wandb\nimport wandb\n\n# Login with your autorization key\nwandb.login()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nprint(tf.__version__)\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import models\nimport tensorflow_addons as tfa\n\nimport os\nimport json\nimport numpy as np\nimport pandas as pd\npd.set_option('mode.chained_assignment', None)\n\nfrom tqdm import tqdm\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom sklearn.model_selection import train_test_split","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I will be using images of resolution 256x256 for this kernel. Thanks to [Ankur Singh](https://www.kaggle.com/ankursingh12) for this [dataset](https://www.kaggle.com/ankursingh12/resized-plant2021). Feel free to use higher resolution but that might take more time to upload.","metadata":{}},{"cell_type":"code","source":"# Global variables/hyperparameters\n\nTRAIN_PATH = '../input/resized-plant2021/img_sz_256/'\nTEST_PATH = '../input/plant-pathology-2021-fgvc8/test_images/'\n\nIMG_RES = 256\nWANDB_PROJECT = 'plant-pathology-21'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read csv file\ndf = pd.read_csv('../input/plant-pathology-2021-fgvc8/train.csv')\n\n# Integer encode multi-label labels. \nlabels = list(df['labels'].value_counts().keys())\nlabels_dict = dict(zip(labels, range(12)))\nid_to_labels = {val: key for key, val in labels_dict.items()}\n\ndf = df.replace({\"labels\": labels_dict})\ndf.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1️⃣ Step 1: Upload Raw Dataset \n\nAfter understanding the problem statement of the competition, the next most important step is to have a good understanding of the dataset. A naive approach is to plot batches of data using matplotlib but its not interactive and convenient. Convenince here would be,\n\n* to visualize as many samples as possible without running Jupyter cell(s) multiple times.\n* to revisit this step multiple times without maintaining a notebook or coding again. \n* to be able to group together images by labels\n\n📌 In the cell below the raw `train.csv` file is uploaded as [W&B Artifacts](https://wandb.ai/site/artifacts). We can even upload the entire image dataset but from a Kaggle perspective it's not very relevant since the dataset is going to be constant (more or less). \n\n📌 Beside uploading the csv file, I iterated over the entire dataframe and logged each image along with it's label as [W&B Tables](https://docs.wandb.ai/guides/data-vis/tables). Consider this table to be a pandas dataframe with data, meta-data and rich media all in one view. ","metadata":{}},{"cell_type":"code","source":"RAW_DATA_AT = f'raw_data_{IMG_RES}'\n\n# Initialize a new W&B run\nrun = wandb.init(project=WANDB_PROJECT, job_type=\"upload\")\n\n# create an artifact for all the raw data\nraw_data_at = wandb.Artifact(RAW_DATA_AT, type=\"raw_data\")\n# Upload raw csv file as artifact\nraw_data_at.add_file('../input/plant-pathology-2021-fgvc8/train.csv')\n\n# create a table with columns we want to track/compare\npreview_dt = wandb.Table(columns=[\"id\", \"image\", \"label\"])\n\nfor i in tqdm(range(len(df))):\n    image_name = df.loc[i]['image']\n    label = df.loc[i]['labels']\n    full_path = TRAIN_PATH+image_name\n    \n    # Append each example as a new row in Table.\n    preview_dt.add_data(image_name, wandb.Image(full_path), id_to_labels[label])\n\n# save artifact to W&B\nraw_data_at.add(preview_dt, \"data_split\")\nrun.log_artifact(raw_data_at)\nrun.finish()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Explore the dataset interactively $\\rightarrow$](https://wandb.ai/ayush-thakur/plant-pathology-21/artifacts/raw_data/raw_data_256/9e52a72a554c05224a80/files/data_split.table.json)\n\n![img](https://i.imgur.com/TrvS9VE.png)\n\n## 😍 Group by Labels\n\n* Learn about the number of images per label.\n* Visualize images per label interactively.\n* Revisit the table again to explore more. \n\n![img](https://i.imgur.com/3wG0RGM.gif)\n\n## 😍 Filter by Label\n\n* Visualize images per label.\n\n![img](https://i.imgur.com/iMk7j3v.gif)","metadata":{}},{"cell_type":"markdown","source":"# 2️⃣ Step 2: Split Raw Dataset\n\nThe most important and crucial step is to setup a reliable local validation. Weights and Biases Artifacts can be really helpful here especially with the Tables feature. \n\n📌 In order to build local validation, we need to split the `train.csv` file into training and validation (hold-out) split. In the cell below I have created a stratified train-val split. \n\n📌 This split is uploaded as an Artifact. Note the use of `use_artifact` method. It is used to establish the relationship between different artifacts. In this instance the split is created using the raw csv file that we uploaded as an artifact in the previous section.","metadata":{}},{"cell_type":"code","source":"# Split into train and validation\ntrain_df, valid_df = train_test_split(df, test_size=0.2, stratify=df['labels'].values)\nprint(f'Number of train images: {len(train_df)} and validation images: {len(valid_df)}')\n\n# Add a new column\ntrain_df.loc[:, 'split'] = 'train'\nvalid_df.loc[:, 'split'] = 'validation'\ntmp_df = pd.concat([train_df, valid_df], axis=0, ignore_index=True)\n\n# Save as csv files\ntrain_df.to_csv('stratified_train.csv', index=False)\nvalid_df.to_csv('stratified_valid.csv', index=False)\n\nSPLIT_DATA_AT = f'split_{IMG_RES}'\n\n# Create new W&B run\nrun = wandb.init(project=WANDB_PROJECT, job_type=\"data_split\")\n# Use artifact\nraw_data_at = run.use_artifact('ayush-thakur/plant-pathology-21/raw_data_256:v0', type='raw_data')\n\n# create an artifact for all the raw data\ndata_split_at = wandb.Artifact(SPLIT_DATA_AT, type=\"stratified_split\")\n# Upload split as artifact\ndata_split_at.add_file('../input/plant-pathology-tables-support/stratified_train.csv')\ndata_split_at.add_file('../input/plant-pathology-tables-support/stratified_valid.csv')\n\n# create a table with columns we want to track/compare\npreview_dt = wandb.Table(columns=[\"id\", \"image\", \"label\", \"split\"])\n\nfor i in tqdm(range(len(tmp_df))):\n    image_name = tmp_df.loc[i]['image']\n    label = tmp_df.loc[i]['labels']\n    split = tmp_df.loc[i]['split']\n    full_path = TRAIN_PATH+image_name\n    \n    # Append each example as a new row in Table.\n    preview_dt.add_data(image_name, wandb.Image(full_path), id_to_labels[label], split)\n\n# save artifact to W&B\ndata_split_at.add(preview_dt, \"data_split\")\nrun.log_artifact(data_split_at)\nrun.finish()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Explore Split Dataset Interactively $\\rightarrow$](https://wandb.ai/ayush-thakur/plant-pathology-21/artifacts/stratified_split/split_256/a6b48c097d30d654e5ed/files/data_split.table.json)\n\n## 😍 Group by \"Split\"\n\n* Visualize images in train and validation split interactively along with the image name.\n* Visualize the distribution of labels per split.\n\n![img](https://i.imgur.com/P7bRGmR.gif)\n\n## 😍 Group by \"Labels\"\n\n* Visualize the distribution of images by split per label.\n\n![img](https://i.imgur.com/AUBdDEr.gif)","metadata":{}},{"cell_type":"markdown","source":"# 3️⃣ Step 3: Train Model Using Split\n\nWith the split in place the next task is to train model(s). For sanity sake, this kernel will not go over the training proceduce. You can refer to this [kernel](https://www.kaggle.com/ayuraj/experiment-tracking-with-weights-and-biases) for a training pipeline. However, I have shown how you can use W&B Artifacts in your training pipeline to save your trained model weights. \n\nNote that I have trained two models (using the same split) on a GCP instance and have provided weights as [Kaggle dataset](https://www.kaggle.com/ayuraj/plant-pathology-tables-support) so that we can use it for the mentioned purpose. \n\n* For demonstraction purposes, we will upload only one model weight above. \n* We will use the other model for comparison later in this kernel. ","metadata":{}},{"cell_type":"code","source":"# Initialize a new W&B run\nrun = wandb.init(project=WANDB_PROJECT, job_type=\"train\")\n# Use raw csv file\nsplit_data_at = run.use_artifact('ayush-thakur/plant-pathology-21/split_256:v0', type='stratified_split')\n\n# TRAIN MODEL\n\nmodel_path = '../input/plant-pathology-tables-support/efficientnetb0_3.h5'\n# Create an artifact to save model weights\nmodel_at = wandb.Artifact('efficientnetb0', type=\"trained_model\")\n# Add csv file to be uploaded\nmodel_at.add_file(model_path)\n# Log the csv file\nrun.log_artifact(model_at)\n# End W&B run\nrun.finish()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"![img](https://i.imgur.com/mbzGEl4.png)","metadata":{}},{"cell_type":"markdown","source":"# 4️⃣ Step 4: Prediction Visualization\n\nNow that we have a trained model, the obvious next step is to evaluate on the validation set. Using W&B Tables we can do just so much more than simple `model.evaluate` (Keras). I felt like I got SUPERPOWERs the first time I used this. You are up for a ride my fellow Kagglers. ","metadata":{}},{"cell_type":"code","source":"# Utils\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n\nCONFIG = dict (\n    num_labels = 12,\n    img_width = 224,\n    img_height = 224,\n    batch_size = 64,\n)\n\n@tf.function\ndef decode_image(image):\n    # convert the compressed string to a 3D uint8 tensor\n    image = tf.image.decode_jpeg(image, channels=3)\n    # Normalize image\n    image = tf.image.convert_image_dtype(image, dtype=tf.float32)\n    # resize the image to the desired size\n    return image\n\n@tf.function\ndef load_image(df_dict):\n    # Load image\n    image = tf.io.read_file(TRAIN_PATH+df_dict['image'])\n    image = decode_image(image)\n    \n    # Resize image\n    image = tf.image.resize(image, (CONFIG['img_height'], CONFIG['img_width']))\n        \n    return image","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Initialize new W&B run\nrun = wandb.init(project=WANDB_PROJECT, job_type=\"prediction\")\n\n# Use split csv artifacts and download (for demonstration)\nsplit_csv = run.use_artifact('ayush-thakur/plant-pathology-21/split_256:v0', type='stratified_split')\nsplit_data_dir = split_csv.download()\n\n# Use trained model and download\ntrained_model = artifact = run.use_artifact('ayush-thakur/plant-pathology-21/efficientnetb4:v0', type='trained_model')\nmodel_dir = trained_model.download()\n\n# PREPARE DATALOADER\nvalidation_df = pd.read_csv(split_data_dir+'/stratified_valid.csv')\ntestloader = tf.data.Dataset.from_tensor_slices(dict(validation_df))\n\ntestloader = (\n    testloader\n    .map(load_image, num_parallel_calls=AUTOTUNE)\n    .batch(CONFIG['batch_size'])\n    .prefetch(AUTOTUNE)\n)\n\n# LOAD MODEL\ntf.keras.backend.clear_session()\nmodel = tf.keras.models.load_model(model_dir+'/efficientnetb4_0.h5')\nmodel.summary()\n\n# Prediction\nraw_preds = []\npredictions = []\nfor image_batch in tqdm(testloader):\n    # Get model prediction\n    preds = model.predict(image_batch)\n    # Get label\n    top1 = np.argmax(preds, axis=1)\n    predictions.extend(top1)\n    raw_preds.extend(preds)\n\n# Guess\nvalidation_df['guess'] = predictions\nvalidation_df = validation_df.replace({\"guess\": id_to_labels})\n\n# Logits\nscore_columns = [f'{label}_score' for label in labels_dict.keys()]\nvalidation_df[score_columns] = raw_preds\n\n# Initialize new artifact\nprediction_at = wandb.Artifact('val_predictions_effnetb4', type=\"val_prediction\")\n\ncolumns=[\"id\", \"image\", \"truth\", \"guess\"]\ncolumns.extend(score_columns)\npreview_dt = wandb.Table(columns=columns)\n\nfor i in tqdm(range(len(validation_df))):\n    image_name = validation_df.loc[i]['image']\n    label = validation_df.loc[i]['labels']\n    guess = validation_df.loc[i]['guess']\n    \n    full_path = TRAIN_PATH+image_name\n    \n    row = [image_name, wandb.Image(full_path), id_to_labels[label], guess]\n    for score in validation_df.loc[i][3:]:\n        row.append(score)\n        \n    preview_dt.add_data(*row)\n\n# save artifact to W&B\nprediction_at.add(preview_dt, \"val_prediction\")\nrun.log_artifact(prediction_at)\nrun.finish()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### [Explore Prediction Validation Interactively $\\rightarrow$](https://wandb.ai/ayush-thakur/plant-pathology-21/artifacts/val_prediction/val_predictions/368a98edd260ab844b24/files/val_prediction.table.json)\n\n## 😍 Check the predictions: Group by \"truth\"\n\n* The model is doing well on images with single grouth truth label. \n* `complex` label is being confused the most. Leaves (images) with too many diseases to classify visually are grouped under complex class. Need better strategy to understand what is \"too many\" diseases. \n* Even images with more than one label is hard for the model to clasify. Need to device better strategy to classify. \n\n![img](https://i.imgur.com/4pmINim.gif)\n\n## 😍 Dive into the confusing classes: Group by \"guess\"\n\n![img](https://i.imgur.com/nEJ3Use.png)\n\n## 😍 Find images with complex class and guessed right\n\nLet's investigate `complex` label and find out the images belonging to this label that are guesses correctly. Filter using this query: `row[\"truth\"] = \"complex\" and row[\"guess\"] = \"complex\"`. \n\n![img](https://i.imgur.com/xrFkOoD.gif)\n\n### 😮 Group by truth - the model is not very confident in it's prediction for complex label\n\n![img](https://i.imgur.com/aKQ8sOF.png)","metadata":{}},{"cell_type":"markdown","source":"# 💥 Compare Across Model Versions\n\nTill now we visualized the model performance for one model (EfficientNetB0). With W&B Tables we can also visualize the performance across multiple models. \n\nBelow I am comparing two models - EfficientNetB0 and EfficientNetb4. \n\n| Model | CV Score | LB Score | \n|:---| ----| ---: |\n| EfficientNetB0 | 79.61 | 71.3 |\n| EfficientNetB4 | 82.13 | 73.0 |\n\n## 😍 Find the performance of the models grouped by ground truth\n\n* Visualize how two models are performing on the same validation set. \n* Group by ground truth value to find the most confusing label.\n\n![img](https://i.imgur.com/AjcfpiI.gif)\n\n## 😍 Find score distribution across models.\n\n* The blue color corresponds to EfficientNetB4 while yellow is EfficientNetB0.\n* Clearly see the confusing labels.\n\n![img](https://i.imgur.com/GUlVUW2.gif)","metadata":{}},{"cell_type":"markdown","source":"# ✨ Resources\n\nHere are few other resources that will help you get started with Tables:\n\n* [Visualize Data for Image Classification](https://wandb.ai/stacey/mendeleev/reports/Visualize-Data-for-Image-Classification--VmlldzozNjE3NjA)\n* [Visualize Predictions over Time](https://wandb.ai/stacey/mnist-viz/reports/Visualize-Predictions-over-Time--Vmlldzo1OTQxMTk)\n* [Visualize Audio Data in W&B](https://wandb.ai/stacey/cshanty/reports/Visualize-Audio-Data-in-W-B--Vmlldzo1NDMxMDk)\n\nCheck out the official documentation [here](https://docs.wandb.ai/guides/data-vis).\n\n**Try it out yourself and let me know what more insight you found out.**","metadata":{}}]}