{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Yum or Yuck Butterfly Mimics 2022 – Baseline Model\n\n**Author:** [Keith Pinson](https://github.com/keithpinson)<br>\n**Date created:** 2022/06/11<br>\n**Version:** 1.0.0001<br>\n**Description:** A simple transfer learning model to establish a baseline score for a Kaggle Community Competition where we try to classify the name of the butterfly in each given image.<br>\n**Platform:** Kaggle Packages including Tensorflow 2.6.3 with GPU support<br>\n<br>\n\n![Butterfly Classification Model Diagram](https://github.com/KeithPinson/Yum-or-Yuck-2022-Butterfly-Mimics/raw/main/DocResources/ButterflyClassificationTransferLearning-854.png)\n<br>\n\nWe will use EfficientNet convolutional neural network with the Imagenet dataset and re-train the weights.  The final classification layers will be replaced with our own very simple layer to make the butterfly classifier. We will not do any fine-tuning of the pre-trained model although we will allow the network to retrain by not freezing the weights.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"import datetime\n\nprint(\"executed\",datetime.datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\"),\"local time\")","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:36:59.833959Z","iopub.execute_input":"2022-08-01T11:36:59.834463Z","iopub.status.idle":"2022-08-01T11:36:59.860651Z","shell.execute_reply.started":"2022-08-01T11:36:59.834381Z","shell.execute_reply":"2022-08-01T11:36:59.859634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\nUsing TensorFlow we will a build a butterfly image classifier that will be able to identify 6 remarkable North American butterflies. The Monarch and Pipevine caterpillars sequester toxins: cardiac glycoside and aristolochic acid, from their host plants as they feed. Birds find these butterflies distasteful. They are yucky. The other butterflies mimic the Monarch or Pipevine butterflies and are yummy. Well, except for the Viceroy, since it is both yucky and a mimic, as it sequesters salicylic acid when a caterpillar.\n\nOur goal in this competition is to accurately identify the butterflies by name.\n\n![Images of Black, Monarch, Pipevine, Spicebush, Tiger and Viceroy butterflies from the dataset](https://github.com/KeithPinson/Yum-or-Yuck-2022-Butterfly-Mimics/raw/main/DocResources/the-butterflies.png)\n\n<br>\n\nThe YOYMimics-2022 dataset has labels to support:\n\n 1. Classification (butterfly species)\n 2. Binary classification (yum or yuck)\n 3. Multi-label classification (species and wing side)\n\nThe competition uses only classification of butterfly species in its scoring metric.\n<br>\n\n### <u>This notebook will</u>:\n\n - Load EfficientNet with the pre-trained Imagenet weights\n - Load the Butterfly Mimics dataset\n - Create a new set of feature weights using our own classifier\n - Predict the butterflies from the test\n - Show a sample of the results\n - Quantify the accuracy of the results\n\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"---\n## Set Hyperparameters\n---","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# Hyper-parameters\nLEARNING_RATE = 3e-6\nEPOCHS = 400\nREPEATS = 2\n\n# Other constants\nIMAGE_WIDTH = IMAGE_HEIGHT = 224\nIMAGE_SIZE = (IMAGE_HEIGHT, IMAGE_WIDTH)  # Row,Column order\nIMAGE_DEPTH = 3\nBATCH_SIZE = 32\nBUFFER_SIZE = 1024\nMODEL_NAME = 'yoymimics'\n\nSEED = 43","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:04.865101Z","iopub.execute_input":"2022-08-01T11:37:04.865474Z","iopub.status.idle":"2022-08-01T11:37:04.871470Z","shell.execute_reply.started":"2022-08-01T11:37:04.865443Z","shell.execute_reply":"2022-08-01T11:37:04.870342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Set Environment\n---\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"import os\nimport platform\nimport random\n\nimport tensorflow as tf\n\nfrom tensorflow.keras import mixed_precision\nmixed_precision.set_global_policy('mixed_float16')\n\n# Using TensorFlow's enhanced version of Numpy\nimport tensorflow.experimental.numpy as np\nnp.experimental_enable_numpy_behavior()\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n\nimport pandas as pd\nimport matplotlib as mpl\nfrom matplotlib import pyplot as plt, patches\nfrom matplotlib.offsetbox import OffsetImage, AnnotationBbox\n\nfrom sklearn.metrics import \\\n    fbeta_score, \\\n    confusion_matrix, \\\n    ConfusionMatrixDisplay, \\\n    classification_report\n\nimport IPython.display as display\n\nfrom keras.callbacks import EarlyStopping\nfrom keras.layers import \\\n    Dense, \\\n    RandomRotation, \\\n    RandomZoom\n\n\nos.environ['PYTHONHASHSEED'] = str(SEED)\nrandom.seed = SEED\nnp.random.seed = SEED\ntf.random.set_seed(SEED)\n\nprint(f\"Tensorflow {tf.__version__}\", \"with GPU support\" if len(tf.config.list_physical_devices('GPU')) > 0 else \"for CPU only\")","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:06.574579Z","iopub.execute_input":"2022-08-01T11:37:06.574993Z","iopub.status.idle":"2022-08-01T11:37:12.449179Z","shell.execute_reply.started":"2022-08-01T11:37:06.574960Z","shell.execute_reply":"2022-08-01T11:37:12.448174Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <u>Dataset paths and names</u>\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"os_system = platform.system()  # 'Windows', 'Linux'\n\nhosted_by = 'Kaggle' if os.environ.get('KAGGLE_URL_BASE') else \\\n            ('Windows' if os.environ.get('WINDIR') else \\\n            'Unknown')\n\nif hosted_by == 'Kaggle':\n    dataset_name = \"yum-or-yuck-butterfly-mimics-2022\"\n\n    # Setting the variables assuming a Kaggle platform\n    base_dir = \"/kaggle\"\n    dataset_dir = os.path.join(base_dir, 'input', dataset_name)\n    data_dir = os.path.join(dataset_dir, 'data', 'butterfly_mimics')\n    working_dir = os.path.join(base_dir, 'working')\n    temp_dir = os.path.join(base_dir, 'temp')\n\ntrain_dir = os.path.join(data_dir, 'images')\ntest_dir = os.path.join(data_dir, 'image_holdouts')\ntrain_csv = os.path.join(data_dir, 'images.csv')\ntest_csv = os.path.join(data_dir, 'image_holdouts.csv')\nsubmit_csv = os.path.join(working_dir, 'submission.csv')\n\nclass_names = ['black', 'monarch', 'pipevine', 'spicebush', 'tiger', 'viceroy']\nname_tastes = {'black':'yum', 'monarch':'yuck', 'pipevine':'yuck', 'spicebush':'yum', 'tiger':'yum', 'viceroy':'yuck'}\nclass_count = len(class_names)\nyums = ['black','spicebush','tiger']\nyucks = ['monarch','pipevine','viceroy']\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:12.451363Z","iopub.execute_input":"2022-08-01T11:37:12.452083Z","iopub.status.idle":"2022-08-01T11:37:12.463513Z","shell.execute_reply.started":"2022-08-01T11:37:12.452043Z","shell.execute_reply":"2022-08-01T11:37:12.462548Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <u>Convenient functions</u>","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"def decode_image(image):\n    # image.shape == tf.TensorShape([IMAGE_HEIGHT, IMAGE_WIDTH, IMAGE_DEPTH])\n    return tf.keras.utils.array_to_img(image.numpy())\n\ndef decode_label(label):\n    # label.shape == tf.TensorShape([class_count])\n    return butterfly_classes[tf.argmax(label)].numpy().decode('UTF-8')\n\ndef decode_image_id(image_id):\n    # image_id.shape == tf.TensorShape([1])\n    return image_id.numpy()[0].decode('UTF-8')\n\ndef get_error_type(name:str, predicted_name:str):\n    if name_tastes[name]=='yum' and name_tastes[predicted_name]=='yum':\n        error_string='TP'\n    elif name_tastes[name]=='yum' and name_tastes[predicted_name]=='yuck':\n        error_string='FN'\n    elif name_tastes[name]=='yuck' and name_tastes[predicted_name]=='yum':\n        error_string='FP'\n    elif name_tastes[name]=='yuck' and name_tastes[predicted_name]=='yuck':\n        error_string='TN'\n    else:\n        error_string = ''\n\n    return error_string\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:12.465149Z","iopub.execute_input":"2022-08-01T11:37:12.465846Z","iopub.status.idle":"2022-08-01T11:37:12.484033Z","shell.execute_reply.started":"2022-08-01T11:37:12.465808Z","shell.execute_reply":"2022-08-01T11:37:12.483048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Load the Model\n---\n\nNote: that final, top layer is not included. We will replace these dense layers of the image classifier with our own.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# EfficientNetV2 was not added until TensorFlow 2.8\nfrom keras.applications.efficientnet import EfficientNetB1\n\nheadless_efficientnet = EfficientNetB1(\n    weights='imagenet',\n    include_top=False,\n    input_shape=(IMAGE_HEIGHT, IMAGE_WIDTH, IMAGE_DEPTH)\n)\n\nprint(\"Shape of pre-trained network:\", headless_efficientnet.output_shape[1:])\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:12.487707Z","iopub.execute_input":"2022-08-01T11:37:12.488422Z","iopub.status.idle":"2022-08-01T11:37:17.861918Z","shell.execute_reply.started":"2022-08-01T11:37:12.488362Z","shell.execute_reply":"2022-08-01T11:37:17.860956Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"base_model = headless_efficientnet","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:17.863968Z","iopub.execute_input":"2022-08-01T11:37:17.864750Z","iopub.status.idle":"2022-08-01T11:37:17.870340Z","shell.execute_reply.started":"2022-08-01T11:37:17.864709Z","shell.execute_reply":"2022-08-01T11:37:17.868937Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Build the Butterfly Classifier\n---\n\nThis is the new classifier that replaces the top layer of the pre-trained model. We will *fit* the model with our dateset in a later step, below.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"def butterfly_classifier(x):\n\n    x = tf.keras.layers.GlobalAveragePooling2D()(x)\n\n    # This is the place to improve the classifier.\n    # As an example you could:\n    #\n    # x = Dropout(0.2)(x)\n    # x = Dense(512, activation='relu')(x)\n\n    return x\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:24.452544Z","iopub.execute_input":"2022-08-01T11:37:24.453550Z","iopub.status.idle":"2022-08-01T11:37:24.458872Z","shell.execute_reply.started":"2022-08-01T11:37:24.453501Z","shell.execute_reply":"2022-08-01T11:37:24.457704Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"inputs = tf.keras.Input(shape=(IMAGE_HEIGHT, IMAGE_WIDTH, IMAGE_DEPTH))\n\n# We wind our way from inputs to outputs\nx = inputs\n\nx = RandomRotation(factor=0.2, fill_mode='nearest')(x)\nx = RandomZoom((-0.2, 0.2), fill_mode='nearest')(x)\nx = base_model(x, training=True)  # To run in inference mode set training=False\nx = butterfly_classifier(x)\n\noutputs = Dense(class_count, activation='softmax')(x)\n\n\nbutterfly_model = tf.keras.Model(inputs, outputs, name=MODEL_NAME)\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:26.804040Z","iopub.execute_input":"2022-08-01T11:37:26.804405Z","iopub.status.idle":"2022-08-01T11:37:27.981176Z","shell.execute_reply.started":"2022-08-01T11:37:26.804375Z","shell.execute_reply":"2022-08-01T11:37:27.980036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"butterfly_model.summary()","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:29.211892Z","iopub.execute_input":"2022-08-01T11:37:29.212555Z","iopub.status.idle":"2022-08-01T11:37:29.241003Z","shell.execute_reply.started":"2022-08-01T11:37:29.212519Z","shell.execute_reply":"2022-08-01T11:37:29.239782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"layer = butterfly_model.layers[-1]\nprint(\"weights:\", len(layer.weights))\nprint(\"trainable_weights:\", len(layer.trainable_weights))\nprint(\"non_trainable_weights:\", len(layer.non_trainable_weights))\nprint(\"trainable_variables:\", len(butterfly_model.trainable_variables))\nprint(\"non_trainable_variables:\", len(butterfly_model.non_trainable_variables))\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:32.805776Z","iopub.execute_input":"2022-08-01T11:37:32.806657Z","iopub.status.idle":"2022-08-01T11:37:32.819233Z","shell.execute_reply.started":"2022-08-01T11:37:32.806620Z","shell.execute_reply":"2022-08-01T11:37:32.817976Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Build Dataset Pipeline\n---\n\nThe Butterfly Mimics 2022 Dataset consists of JPG images of a single butterfly. In the training data we are given a label that identifies the class of each butterfly as either *black*, *monarch*, *pipevine*, *spicebush*, *tiger*, or *viceroy*.<br>\n\n![Tiger Swallowtail Butterfly](https://github.com/KeithPinson/Yum-or-Yuck-2022-Butterfly-Mimics/raw/main/DocResources/tiger_female_dark_form_vyaa1ee082.jpg)\n<br>\n\n```python\nX = vyaa1ee082.jpg # Features\ny = \"tiger\"        # Target\n```\n\nThe dataset provides two folders of images. One with class labels for training. And, one without class labels to test your model for the Public Leaderboard.\n\n- Training data is referred to as the `'images'`\n- Test data is referred to as the `'image_holdouts'`\n\nNo data from the `'image_holdouts'` is used to generate our baseline scores.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"### Our ETL Pipeline:\n| <u>Extract</u> | <u>Transform</u> | <u>Load</u> |\n| :----- | :------- | :-- |\n| *Read* dataset stored locally on disk | Step 1: *Split* into training and evaluating sets | *Feed* the data to the model |\n|                                     | Step 2: *Pre-process* |\n|                                     | Step 3: *Map* image names to image features |\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"### <u>Read</u>","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"presplit_csv_data = pd.read_csv(train_csv)\ntest_csv_data = pd.read_csv(test_csv)\n\nbutterfly_count = len(presplit_csv_data)\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:38.971054Z","iopub.execute_input":"2022-08-01T11:37:38.971426Z","iopub.status.idle":"2022-08-01T11:37:39.006354Z","shell.execute_reply.started":"2022-08-01T11:37:38.971395Z","shell.execute_reply":"2022-08-01T11:37:39.005374Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <u>Our own Test Split</u>\nA test split is provided with the dataset but it does not include the class labels. For this notebook we want to know the actual class labels. So, instead of using the provided test split we will create our own. We call it 'eval'. With it, we will compare actual labels to predicted near the end of the notebook.\n\nWe will split the dataset into training and validation in a little bit further below.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# Eval is a very simple split\neval_fraction = 0.25\neval_count = int(butterfly_count//BATCH_SIZE * eval_fraction + 0.5) * BATCH_SIZE\ntrain_count = butterfly_count - eval_count\n\ntrain_csv_data = presplit_csv_data[:train_count]\neval_csv_data = presplit_csv_data[train_count:]","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:42.494699Z","iopub.execute_input":"2022-08-01T11:37:42.495080Z","iopub.status.idle":"2022-08-01T11:37:42.501538Z","shell.execute_reply.started":"2022-08-01T11:37:42.495048Z","shell.execute_reply":"2022-08-01T11:37:42.500580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <u>Map and Preprocess</u>\n\nWe define some mapping functions that we will use in the TensorFlow pipeline. Pay particular attention to `tf.numpy_function()`, without it this workflow of reading the CSV and then loading our images later would not easily work using TensorFlow data.Dataset.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"butterfly_classes = np.array(class_names, dtype='str')\n\ndef get_feature_and_label_function(image_id, class_name):\n    _image_id = image_id[0].decode('UTF-8')\n    _class_name = class_name[0].decode('UTF-8')\n\n    _img = tf.io.read_file(os.path.join(\n        train_dir, _image_id + '.jpg'))\n\n    _img = tf.image.decode_jpeg(_img,\n        channels=IMAGE_DEPTH,\n        dct_method='INTEGER_ACCURATE',\n        name=_image_id)\n\n    _img = tf.image.resize(_img,IMAGE_SIZE)\n\n    _img = tf.cast(_img, tf.float32)/255.0\n\n    name_label = tf.convert_to_tensor(_class_name)\n\n    one_hot = name_label == butterfly_classes\n\n    # encoded_label = tf.argmax(one_hot)\n    encoded_label = one_hot.astype(np.float32)\n\n    return _img, encoded_label, image_id\n\ndef get_feature_and_label(x,y):\n\n    features_labels = tf.numpy_function(\n        get_feature_and_label_function,\n        [x,y],\n        [tf.float32,tf.float32,tf.string]\n    )\n\n    # numpy_function() loses the shapes, we will need to restore them\n\n    features_labels[0].set_shape(\n        tf.TensorShape([IMAGE_HEIGHT, IMAGE_WIDTH, IMAGE_DEPTH])\n    )\n\n    features_labels[1].set_shape(tf.TensorShape([class_count]))\n\n    features_labels[2].set_shape(tf.TensorShape([1]))\n    tf.cast(features_labels[2], tf.string, name='image_id')\n\n    return features_labels\n\ndef get_feature_function(image_id):\n    _image_id = image_id[0].decode('UTF-8')\n\n    _img = tf.io.read_file(os.path.join(\n        test_dir, _image_id + '.jpg'))\n\n    _img = tf.image.decode_jpeg(_img,\n        channels=IMAGE_DEPTH,\n        dct_method='INTEGER_ACCURATE',\n        name=_image_id)\n\n    _img = tf.image.resize(_img,IMAGE_SIZE)\n\n    _img = tf.cast(_img, tf.float32)/255.0\n\n    return _img, image_id\n\ndef get_feature(x):\n\n    features_labels = tf.numpy_function(\n        get_feature_function,\n        [x],\n        [tf.float32,tf.string]\n    )\n\n    # numpy_function() loses the shapes, we will need to restore them\n\n    features_labels[0].set_shape(\n        tf.TensorShape([IMAGE_HEIGHT, IMAGE_WIDTH, IMAGE_DEPTH])\n    )\n\n    features_labels[1].set_shape(tf.TensorShape([1]))\n    tf.cast(features_labels[1], tf.string, name='image_id')\n\n    return features_labels\n\ndef clear_label(image_feature, label, image_id):\n    _label = np.array([0.,0.,0.,0.,0.,0.],dtype=np.float32)\n    return image_feature, _label, image_id\n\ndef drop_image_id(image_feature, label, image_id):\n    return image_feature, label\n\ndef train_preprocess(image_feature, label):\n    _img = tf.image.random_flip_left_right(image_feature)\n\n    _img = tf.image.random_brightness(_img, max_delta=32.0 / 255.0)\n    _img = tf.image.random_saturation(_img, lower=0.5, upper=1.5)\n\n    # Make sure the image is still in [0, 1]\n    _img = tf.clip_by_value(_img, 0.0, 1.0)\n\n    return _img, label\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:44.904626Z","iopub.execute_input":"2022-08-01T11:37:44.905133Z","iopub.status.idle":"2022-08-01T11:37:44.928921Z","shell.execute_reply.started":"2022-08-01T11:37:44.905096Z","shell.execute_reply":"2022-08-01T11:37:44.927898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Map the strings into the actual tensors and vectors.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# Map the entire dataset as well as an eval dataset that\n# is a stand-in in this notebook for the test dataset\n\n# Butterfly Dataset\nbutterfly_images = pd.DataFrame(presplit_csv_data[['image']].values.tolist())\nbutterfly_names = pd.DataFrame(presplit_csv_data[['name']].values.tolist())\n\nbutterfly_ds_prebatch = tf.data.Dataset.from_tensor_slices((butterfly_images,butterfly_names))\nbutterfly_ds_prebatch = butterfly_ds_prebatch.map(get_feature_and_label , num_parallel_calls=AUTOTUNE)\n\n# Eval Dataset\neval_images = pd.DataFrame(eval_csv_data[['image']].values.tolist())\neval_names = pd.DataFrame(eval_csv_data[['name']].values.tolist())\n\neval_ds_prebatch = tf.data.Dataset.from_tensor_slices((eval_images,eval_names))\neval_ds_prebatch = eval_ds_prebatch.map(get_feature_and_label , num_parallel_calls=AUTOTUNE)\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:48.181495Z","iopub.execute_input":"2022-08-01T11:37:48.182483Z","iopub.status.idle":"2022-08-01T11:37:48.410495Z","shell.execute_reply.started":"2022-08-01T11:37:48.182434Z","shell.execute_reply":"2022-08-01T11:37:48.409260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <u>Training/Validation Split</u>","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# Shuffle and the do a simple split\n\nbutterfly_ds_prebatch.shuffle(butterfly_count, reshuffle_each_iteration=False)\n\ntrain_ds_prebatch = butterfly_ds_prebatch.take(train_count)     # For training\nvalidate_ds_prebatch = butterfly_ds_prebatch.skip(train_count)  #   and validating during Fit()\n\nprint(f\"length: {len(train_ds_prebatch)} train / {len(validate_ds_prebatch)} validate\")\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:50.109171Z","iopub.execute_input":"2022-08-01T11:37:50.109547Z","iopub.status.idle":"2022-08-01T11:37:50.130325Z","shell.execute_reply.started":"2022-08-01T11:37:50.109517Z","shell.execute_reply":"2022-08-01T11:37:50.129281Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <u>Augmentation Preprocessing</u>","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# At this point our datasets have an image, label, and image_id component...\n\ntrain_ds_prebatch = train_ds_prebatch.map(drop_image_id, num_parallel_calls=AUTOTUNE)        # Drop image id\nvalidate_ds_prebatch = validate_ds_prebatch.map(drop_image_id, num_parallel_calls=AUTOTUNE)  # Drop image id\n\n# Finish preprocessing Training Dataset\ntrain_ds_prebatch = train_ds_prebatch.repeat(REPEATS)\ntrain_ds_prebatch = train_ds_prebatch.map(train_preprocess, num_parallel_calls=AUTOTUNE)\n\nif hosted_by != 'Kaggle':\n    # Throws a warning in kaggle...\n    train_ds_prebatch = train_ds_prebatch.cache()\n    \ntrain_ds_prebatch = train_ds_prebatch.shuffle(buffer_size=BUFFER_SIZE, reshuffle_each_iteration=True)\n\n# Just cache the validation dataset, hold off on any preprocessing\nif hosted_by != 'Kaggle':\n    # Throws a warning in kaggle...\n    validate_ds_prebatch = validate_ds_prebatch.cache()\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:53.259856Z","iopub.execute_input":"2022-08-01T11:37:53.260511Z","iopub.status.idle":"2022-08-01T11:37:53.398137Z","shell.execute_reply.started":"2022-08-01T11:37:53.260475Z","shell.execute_reply":"2022-08-01T11:37:53.397169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's pause for moment and walk a batch through our model and make sure things are working as we expect.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"x, y = next(iter(train_ds_prebatch.batch(BATCH_SIZE)))\n\nprint(\"                   Shapes\")\nprint(\"                   ------\")\nprint(\"     input batch: \", x.shape)\nx = base_model(x)\nprint(\"      base model: \", x.shape)\nx = butterfly_classifier(x)\nprint(\"classifier model: \", x.shape)\nx = Dense(class_count)(x)\nprint(\"prediction layer: \", x.shape)\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:37:55.088275Z","iopub.execute_input":"2022-08-01T11:37:55.088672Z","iopub.status.idle":"2022-08-01T11:38:08.561263Z","shell.execute_reply.started":"2022-08-01T11:37:55.088639Z","shell.execute_reply":"2022-08-01T11:38:08.559650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check the shapes that the mapping gave us\nprint(\"Shapes:\")\nprint(train_ds_prebatch)\nprint()\nimage, label = next(iter(train_ds_prebatch.skip(2)))\nprint(\"image shape:\", image.shape)\nprint(\"label shape:\", label.shape, \"\\t\\t\\t\", label)\nprint()\nprint(decode_label(label))\ndisplay.display(decode_image(image))\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:38:08.563019Z","iopub.execute_input":"2022-08-01T11:38:08.563998Z","iopub.status.idle":"2022-08-01T11:38:12.503123Z","shell.execute_reply.started":"2022-08-01T11:38:08.563956Z","shell.execute_reply":"2022-08-01T11:38:12.502180Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <u>Load</u>\n\nWe first batch the datasets and then they will be loaded by the TensorFlow model.fit() method.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"# Batch and prefetch the datasets now\ntrain_ds_batch = train_ds_prebatch.batch(BATCH_SIZE)\ntrain_ds = train_ds_batch.prefetch(AUTOTUNE)\n\neval_ds = eval_ds_prebatch.batch(BATCH_SIZE)\n\nvalidate_ds_batch = validate_ds_prebatch.batch(BATCH_SIZE)\nvalidate_ds = validate_ds_batch.prefetch(AUTOTUNE)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:38:16.368626Z","iopub.execute_input":"2022-08-01T11:38:16.369002Z","iopub.status.idle":"2022-08-01T11:38:16.378745Z","shell.execute_reply.started":"2022-08-01T11:38:16.368971Z","shell.execute_reply":"2022-08-01T11:38:16.377530Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can confirm the dataset is ready by taking a peek at the first batch.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"plt.style.use(\"default\")\nplt.figure(figsize=(13,12))\n\n# Get the first batch\nimage_batch, label_batch = next(iter(train_ds))\n\nrows = 5\ncols = 6\nfor i in range(rows*cols):\n    butterfly_image = decode_image(image_batch[i])\n    butterfly_label = decode_label(label_batch[i])\n\n    ax = plt.subplot(rows, cols, i+1)\n    plt.imshow(butterfly_image)\n    plt.title(butterfly_label, color=\"dodgerblue\", fontsize=17)\n    plt.axis(\"off\")\n\nplt.show()\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:38:19.697529Z","iopub.execute_input":"2022-08-01T11:38:19.700831Z","iopub.status.idle":"2022-08-01T11:38:25.700544Z","shell.execute_reply.started":"2022-08-01T11:38:19.700777Z","shell.execute_reply":"2022-08-01T11:38:25.699648Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Train the Model\n---\n\nWe will compile() and fit() to train our model with given dataset.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"stop_early = EarlyStopping(\n    monitor='val_loss',\n    min_delta=0.0001,\n    patience=50,\n    verbose=1,\n    mode='auto',\n    restore_best_weights=True\n)\n\nbase_model.trainable = True  # Freeze or don't freeze weights in all layers\n\nbutterfly_model.compile(\n    optimizer=tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE),\n    loss=tf.keras.losses.categorical_crossentropy,\n    metrics=['accuracy'])\n\nfit_history = butterfly_model.fit(\n    train_ds,\n    shuffle=True,\n    epochs=EPOCHS,\n    callbacks=[stop_early],\n    validation_data=validate_ds,\n    verbose=1\n)\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T11:38:25.702493Z","iopub.execute_input":"2022-08-01T11:38:25.703108Z","iopub.status.idle":"2022-08-01T13:05:32.694806Z","shell.execute_reply.started":"2022-08-01T11:38:25.703069Z","shell.execute_reply":"2022-08-01T13:05:32.693063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Visualize the Training\n---\n\nIf the val_loss is wildly oscillating and/or the accuracy has quickly achieved good accuracy, then cooling the learning rate down may help. The LEARNING_RATE is a hyperparameter set in a cell near the top of the notebook.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"plt.figure(figsize=(15,6))\nplt.style.use(\"default\")\n\nplt.suptitle('Loss and Accuracy', fontsize=18, color=\"forestgreen\")\n\n# Visualize Loss vs Epochs\nplt.subplot(1,2,1)\nplt.plot(fit_history.history['loss'], label='Training Loss')\nplt.plot(fit_history.history['val_loss'], label='Validation Loss')\nplt.legend()\nplt.xlabel('Epochs', fontsize=15, color=\"forestgreen\")\nplt.ylabel('Loss', fontsize=15, color=\"forestgreen\")\n\n# Visualize Accuracy vs Epochs\nplt.subplot(1,2,2)\nplt.plot(fit_history.history['accuracy'], label='Train Accuracy')\nplt.plot(fit_history.history['val_accuracy'], label='Validation Accuracy')\nplt.legend()\nplt.xlabel('Epochs', fontsize=15, color=\"forestgreen\")\nplt.ylabel('Accuracy', fontsize=15, color=\"forestgreen\")\nplt.show()","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T13:35:40.094546Z","iopub.execute_input":"2022-08-01T13:35:40.095207Z","iopub.status.idle":"2022-08-01T13:35:40.568455Z","shell.execute_reply.started":"2022-08-01T13:35:40.095169Z","shell.execute_reply":"2022-08-01T13:35:40.567518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now that the model is trained we can evaluate it\neval_result = butterfly_model.evaluate(validate_ds)\ndict(zip(butterfly_model.metrics_names, eval_result))","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T13:35:44.168144Z","iopub.execute_input":"2022-08-01T13:35:44.168842Z","iopub.status.idle":"2022-08-01T13:35:46.773362Z","shell.execute_reply.started":"2022-08-01T13:35:44.168763Z","shell.execute_reply":"2022-08-01T13:35:46.772338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Make the Predictions\n---\n\nLike `evaluate()` which we just ran, we can run `predict()` but we will have to do some work to visualize the results.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"y_predictions = butterfly_model.predict(eval_ds, verbose=1)","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T13:35:52.610314Z","iopub.execute_input":"2022-08-01T13:35:52.610918Z","iopub.status.idle":"2022-08-01T13:35:56.633304Z","shell.execute_reply.started":"2022-08-01T13:35:52.610877Z","shell.execute_reply":"2022-08-01T13:35:56.632297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def decode_predictions(preds, name_list, top=1):\n    results = []\n    for pred in preds:\n        top_indices = pred.argsort()[-top:][::-1]\n\n        result = [tuple([name_list[i]]) + (pred[i], ) for i in top_indices]\n        result.sort(key=lambda x: x[1], reverse=True)\n        results.append(result)\n\n    return results\n\npredicted_label_confidence = decode_predictions(y_predictions, class_names)\n","metadata":{"pycharm":{"name":"#%%\n"},"execution":{"iopub.status.busy":"2022-08-01T13:35:59.790585Z","iopub.execute_input":"2022-08-01T13:35:59.791356Z","iopub.status.idle":"2022-08-01T13:35:59.803430Z","shell.execute_reply.started":"2022-08-01T13:35:59.791318Z","shell.execute_reply":"2022-08-01T13:35:59.802219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Combine the actual and predicted data together to help visualize our results later\nes = eval_ds_decoded = [(decode_image(m),decode_label(l),decode_image_id(id)) for (m, l, id) in eval_ds.unbatch()]\nps = predicted_label_confidence\n\n# Zip the tuples, manually\npredictions_df = pd.DataFrame(\n    [(es[i][2],             # id\n      es[i][0],             # X\n      es[i][1],             # y\n      es[i][1] in yums,     # yum\n      ps[i][0][0],          # y_pred\n      ps[i][0][0] in yums,  # yum_pred\n      ps[i][0][1],          # y_conf\n      get_error_type(es[i][1], ps[i][0][0])\n    ) for i in range(len(es))],\n    columns=['id','X','y','yum','y_pred','yum_pred','y_conf','yum_err'])\n\npredictions_df.head()\n","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:03.801415Z","iopub.execute_input":"2022-08-01T13:36:03.802116Z","iopub.status.idle":"2022-08-01T13:36:05.000681Z","shell.execute_reply.started":"2022-08-01T13:36:03.802079Z","shell.execute_reply":"2022-08-01T13:36:04.999537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 4\n\nprint(f\"{predictions_df.loc[i,'id']}, {predictions_df.loc[i,'y']}\")\ndisplay.display(predictions_df.loc[i,'X'])","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:19.574023Z","iopub.execute_input":"2022-08-01T13:36:19.574386Z","iopub.status.idle":"2022-08-01T13:36:19.602159Z","shell.execute_reply.started":"2022-08-01T13:36:19.574356Z","shell.execute_reply":"2022-08-01T13:36:19.601106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Visualize Results\n---","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"j = 0\nshow_actual = True\n\nfp_icon = os.path.join(base_dir, dataset_name, 'DocResources', 'face-vomiting_1f92e.png')\n\nyuck_icon = None  # mpl.image.imread(fp_icon)\nicon_size = 1  # yuck_icon.shape[1], yuck_icon.shape[0]\n","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:24.542529Z","iopub.execute_input":"2022-08-01T13:36:24.543589Z","iopub.status.idle":"2022-08-01T13:36:24.549354Z","shell.execute_reply.started":"2022-08-01T13:36:24.543551Z","shell.execute_reply":"2022-08-01T13:36:24.548189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Run this cell over and over to step through the predictions\n\nrows = 5\ncols = 5\n\nplt.style.use(\"default\")\nfig, axs = plt.subplots(nrows=rows, ncols=cols, figsize=(13, 13),\n                        subplot_kw={'xticks': [], 'yticks': []})\n\nplt.suptitle(\"Predictions (blue: correct, red: incorrect)\\n\", color=\"mediumseagreen\", fontsize=20)\n\nmaxloop = len(predictions_df)//(rows*cols)\n\nif j <= maxloop:\n    for i, ax in enumerate(axs.flat):\n        k = (j*rows*cols) + i\n\n        if k < len(predictions_df):\n\n            a_name = predictions_df.loc[k,'y']\n            p_name = predictions_df.loc[k,'y_pred']\n            p_confidence = predictions_df.loc[k,'y_conf']\n            p_percent = \"{:.2%}\".format(p_confidence) \\\n                if p_confidence < 1.0 else \"100%\"\n\n            a_description = f\"{predictions_df.loc[k,'id']}\\n{a_name}\"\n            p_description = f\"{p_name} {p_percent}\"\n\n            # Disable False-positive icon display\n            is_fp = False  # = predictions_df.loc[k,'yum_err'] == 'FP'\n\n            color = \"dodgerblue\" if a_name == p_name else \"crimson\"\n\n            ax.grid(None)\n            ax.axis('on')\n\n            ax.set_title(p_description, color=color, fontsize=17)\n\n            ax.imshow(predictions_df.loc[k,'X'], interpolation_stage='rgb', alpha=1.0)\n\n            if is_fp and yuck_icon:\n                offset_img = OffsetImage(yuck_icon, zoom=.7)\n                ab = AnnotationBbox(offset_img, (2, 22), frameon=False)\n                ax.add_artist(ab)\n\n            if show_actual:\n                rectangle = patches.Rectangle((0, IMAGE_HEIGHT-62), IMAGE_WIDTH, 58, facecolor=\"forestgreen\", linewidth=0, alpha=.5)\n                ax.add_patch(rectangle)\n\n                ax.annotate(a_description, (IMAGE_WIDTH//2, IMAGE_HEIGHT-33), color='ivory', weight='bold', fontsize=17, ha='center', va='center')\n\n            ax.set(xticks=[], yticks=[], xlabel='')\n        else:\n            ax.set_visible(False)\n\n    fig.text(.05, .96, f\"page {j+1}\", color='k', fontsize=17)\n\n    j = j + 1 if j < maxloop else 0\n    plt.tight_layout(h_pad=1.5)\n    plt.show()","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:26.938392Z","iopub.execute_input":"2022-08-01T13:36:26.938770Z","iopub.status.idle":"2022-08-01T13:36:29.427075Z","shell.execute_reply.started":"2022-08-01T13:36:26.938739Z","shell.execute_reply":"2022-08-01T13:36:29.426130Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Code for Visualizations\n---\nThis section contains Python definitions to help visualize our results. Apologies for the length of the code, but it is necessary for making better visualizations. This code does not need to be reviewed to understand the model.\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"def plot_confusion_matrix(cm, cmap=None, title=None,\n                          y_labels=None, x_labels=None,\n                          binary_flip=True,\n                          error_1_text = '',\n                          error_2_text = '',\n                          show_annotations=True,\n                          quad_line=True,  # True/False or width of true-positives\n                          size=(8,6)):\n\n    is_binary = len(cm.ravel()) == 4\n\n    # Binary matrices will unravel in this order:\n    # tn, fp, fn, tp = cm.ravel()\n    # Binary labels should be in False,True or 0,1 order\n\n    tn_c = (0,0)  # True Negative top-left corner\n    fp_c = (0,1)\n    fn_c = (1,0)\n    tp_c = (1,1)\n\n    _cm = cm\n\n    if is_binary and binary_flip:\n        _cm = np.rot90(np.rot90(cm)).numpy()\n        x_labels = list(reversed(x_labels)) if x_labels else None\n        y_labels = list(reversed(y_labels)) if y_labels else None\n        tn_c = (1,1)  # True Negative bottom-right corner\n        fp_c = (1,0)\n        fn_c = (0,1)\n        tp_c = (0,0)\n\n    plt.style.use(\"default\")\n\n    if y_labels and not x_labels:\n        x_labels = y_labels\n\n    if x_labels and not y_labels:\n        y_labels = x_labels\n\n    labels = y_labels # either None or both labels set\n\n\n    # Set the optional ax argument so we can change the figsize()\n    fig, ax = plt.subplots(figsize=size)\n\n    if title:\n        plt.suptitle(title, color=\"darkgreen\", fontsize=20, y=1.1)\n\n    disp = ConfusionMatrixDisplay(\n        confusion_matrix=_cm,\n        display_labels=labels,\n    )\n\n    if not cmap:\n        # Contract the color space\n        cmap = plt.get_cmap('RdGy')\n        min_v = .15\n        max_v = .9\n        _cmap = cmap(np.linspace(min_v, max_v, 100))\n        cmap = mpl.colors.LinearSegmentedColormap.from_list(\n            'trunc({n},{a:.2f},{b:.2f})'.format(n=cmap.name, a=min_v, b=max_v),\n            _cmap)\n    else:\n        cmap = plt.get_cmap(cmap)\n\n    cm_plot = disp.plot(\n        ax=ax,\n        cmap=cmap,              # 'seismic_r'\n        xticks_rotation=37,\n        include_values=False,\n    )\n\n    ax.set_title(\"Confusion Matrix\", fontsize=18)\n\n    ax.set_ylabel('Actual', fontsize=17)\n    ax.set_xlabel('Prediction', fontsize=17)\n\n    if not x_labels:\n        x_labels = ax.xaxis.get_ticklabels()\n\n    ax.xaxis.set_ticklabels(x_labels, ha='right')\n\n    ax.tick_params(axis='both', which='major', labelsize=14, labelcolor='dimgrey')\n\n    #\n    # Display our own content in the grid\n    #\n\n    tm = np.empty_like(_cm, dtype=object).numpy() # Text Matrix\n\n    # print text with appropriate color depending on background\n    cmap_ls = cmap(np.linspace(0, 1.0, (_cm.max()-_cm.min())))\n    lum_white = 0.9277833117792471\n    lum_black = 0.0\n\n    for j in range(tm.shape[0]):\n        for i in range(tm.shape[1]):\n\n            # Determine the highest contrast text color\n            r,g,b,a = tuple(cmap_ls[_cm[j, i]-_cm.min()-1].tolist())\n\n            lum_r = r / 12.92 if r <= 0.03928 else (r + 0.055)/1.055 ** 2.4\n            lum_g = g / 12.92 if g <= 0.03928 else (g + 0.055)/1.055 ** 2.4\n            lum_b = b / 12.92 if b <= 0.03928 else (b + 0.055)/1.055 ** 2.4\n\n            lum = 0.2126 * lum_r + 0.7152 * lum_g + 0.0722 * lum_b\n\n            contrast_ratio_light = ((max(lum,lum_white) + 0.05) / (min(lum,lum_white) + 0.05))\n            contrast_ratio_dark = ((max(lum,lum_black) + 0.05) / (min(lum,lum_black) + 0.05))\n\n            tm_color = 'k' if contrast_ratio_dark > contrast_ratio_light else 'w'\n\n            # Determine the number formatting\n            tm_fmt = format(_cm[j, i], \".2g\")\n            if _cm.dtype.kind != \"f\":\n                text_d = format(_cm[j, i], \"d\")\n                if len(text_d) < len(tm_fmt):\n                    tm_fmt = text_d\n\n            tm[j,i] = {'value':tm_fmt, 'color':tm_color}\n            ax.text(y=j, x=i, s=tm_fmt, ha=\"center\", va=\"center\", color=tm_color, fontsize=17)\n\n\n    if is_binary and show_annotations:\n\n        tm[tn_c]['quadrant'] = '(True Negative)'\n        tm[fp_c]['quadrant'] = '(False Positive)'\n        tm[fn_c]['quadrant'] = '(False Negative)'\n        tm[tp_c]['quadrant'] = '(True Positive)'\n        tm[fn_c]['description'] = 'Type II Error\\n' + error_2_text\n        tm[fp_c]['description'] = 'Type I Error\\n' + error_1_text\n\n        for j in range(tm.shape[0]):\n            for i in range(tm.shape[1]):\n                if tm[j,i].get('quadrant'):\n                    ax.text(s=f\"{tm[j,i]['quadrant']}\",\n                        y=j,x=i,\n                        position=(i,j+0.25),\n                        color=tm[j,i]['color'],\n                        fontsize='16',\n                        verticalalignment='bottom',\n                        horizontalalignment='center',\n                    )\n                if tm[j,i].get('description'):\n                    ax.text(s=f\"{tm[j,i]['description']}\",\n                        y=j,x=i,\n                        position=(i,j+0.3),\n                        color=tm[j,i]['color'],\n                        fontsize='12',\n                        fontstyle='italic',\n                        verticalalignment='top',\n                        horizontalalignment='center',\n                    )\n\n\n    if quad_line:\n        # Check to see we have to calculate line position and we have right shape\n        if isinstance(quad_line,bool):\n            if (len(cm)==len(cm[0]) and len(cm)%2==0):\n                quad_line = len(cm)//2\n            else:\n                quad_line = 0\n\n        if quad_line > 0:\n            ax.axhline(y=quad_line-0.5, xmin=0.0, xmax=1.0, color='k')\n            ax.axvline(x=quad_line-0.5, ymin=0.0, ymax=1.0, color='k')\n\n    plt.show()\n","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:31.465096Z","iopub.execute_input":"2022-08-01T13:36:31.465453Z","iopub.status.idle":"2022-08-01T13:36:31.499998Z","shell.execute_reply.started":"2022-08-01T13:36:31.465423Z","shell.execute_reply":"2022-08-01T13:36:31.498717Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Check our Score\n---\nThe competition is using the F1 score as a scoring metric.\n\nFor a tabular display `sklearn classification_report()` and `confusion_matrix()` provide nicely formatted displays of useful metrics.\n\nA graphical display of the confusion table can be useful too.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"beta = 1\naverage = 'micro'\nprint(f\"F{beta} score:\", fbeta_score(predictions_df[['y']], predictions_df[['y_pred']], beta=beta, average=average))","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:39.050389Z","iopub.execute_input":"2022-08-01T13:36:39.050774Z","iopub.status.idle":"2022-08-01T13:36:39.064304Z","shell.execute_reply.started":"2022-08-01T13:36:39.050743Z","shell.execute_reply":"2022-08-01T13:36:39.063093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"butterfly_cr = classification_report(predictions_df[['y']], predictions_df[['y_pred']], digits=3, output_dict=False)\nprint(\"Butterfly Classification Report\")\nprint()\nprint(butterfly_cr)\n","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:49.546074Z","iopub.execute_input":"2022-08-01T13:36:49.546438Z","iopub.status.idle":"2022-08-01T13:36:49.569110Z","shell.execute_reply.started":"2022-08-01T13:36:49.546407Z","shell.execute_reply":"2022-08-01T13:36:49.568221Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"butterfly_cm = confusion_matrix(predictions_df[['y']], predictions_df[['y_pred']])\nprint(\"Butterfly Confusion Matrix\")\nprint()\nprint(butterfly_cm)\n","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:53.072541Z","iopub.execute_input":"2022-08-01T13:36:53.073522Z","iopub.status.idle":"2022-08-01T13:36:53.084661Z","shell.execute_reply.started":"2022-08-01T13:36:53.073473Z","shell.execute_reply":"2022-08-01T13:36:53.083616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_confusion_matrix(butterfly_cm, title=\"Butterfly Classification\", cmap='Spectral_r', y_labels=class_names, size=(5,4), quad_line=False)","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:36:55.419300Z","iopub.execute_input":"2022-08-01T13:36:55.419987Z","iopub.status.idle":"2022-08-01T13:36:55.790637Z","shell.execute_reply.started":"2022-08-01T13:36:55.419949Z","shell.execute_reply":"2022-08-01T13:36:55.789715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Further Visualizations\n---\nMany datasets have additional labels (sometimes stated and sometimes hidden). These additional labels can be incorporated into the model and if done right can decrease overfitting and increase accuracy.\n\nThe *2022 Dataset of Butterfly Mimics* provides an additional stated label, `side` which indicates the visible side of the wing shown in the image. A model that first classifies *side* and then butterfly *name* might boost accuracy.\n\nSome hidden labels, these are really heuristics and don't require teasing out with another model, are color and taste. Monarch and viceroys are orange, yellow swallowtails are tigers (the lighter form). Black, spicebush, and tiger butterflies taste yummy. Monarch, pipevine, and viceroy butterflies taste yucky (to birds).\n\n<u>Yum and Yuck in the Confusion Matrix</u>\n\nLet's look at the \"yum or yuck\" label and see how incorporating it into the Butterfly Confusion Matrix can reveal more information.\n\n","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"yum_cm = confusion_matrix(predictions_df[['yum']], predictions_df[['yum_pred']])\n\nk_class_names = ['black', 'spicebush', 'tiger', 'monarch', 'pipevine', 'viceroy']\nk_butterfly_cm = confusion_matrix(predictions_df[['y']], predictions_df[['y_pred']], labels=k_class_names)\n\n# Note: with binary classifications it is best unravel the confusion matrix as follows:\ntn, fp, fn, tp = yum_cm.ravel()\n\nprint('TP', tp, \"\\t| \", 'FN', fn)  # \"if Prediction left-right axis then FN is left-right and clockwise to TP\"\nprint('------------------')\nprint('FP', fp, \"\\t| \", 'TN', tn)\nprint()\n\nprint(\"Or, viewed as a visualization:\\n\")\n\nplot_confusion_matrix(yum_cm,\n                      title=\"Yum or Yuck Binary Classification\",\n                      y_labels=['Yuck', 'Yum'],\n                      x_labels=['No Eat', 'Ate'],\n                      error_1_text='\"Ate yucky butterfly\"',\n                      error_2_text='\"Missed yummy butterfly\"'\n)\n","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:37:52.264641Z","iopub.execute_input":"2022-08-01T13:37:52.265618Z","iopub.status.idle":"2022-08-01T13:37:52.596478Z","shell.execute_reply.started":"2022-08-01T13:37:52.265562Z","shell.execute_reply":"2022-08-01T13:37:52.595532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at the corresponding butterfly confusion matrix still in yum/yuck order...","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"plot_confusion_matrix(k_butterfly_cm, title=\"Butterfly Classification\", cmap='Spectral_r', y_labels=k_class_names, size=(5,4))","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:38:06.849462Z","iopub.execute_input":"2022-08-01T13:38:06.850152Z","iopub.status.idle":"2022-08-01T13:38:07.231363Z","shell.execute_reply.started":"2022-08-01T13:38:06.850113Z","shell.execute_reply":"2022-08-01T13:38:07.230419Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"While it may not have been clear before, now we can see that the additional label provides Type I and II Error information that could prove useful in improving the overall accuracy of the model.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"markdown","source":"---\n## Save the CSV\n---\n\nIf our model is good this is the last step.","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"def make_predictions(test_data : pd.core.frame.DataFrame):\n\n    # Test Dataset\n    test_images = pd.DataFrame(test_data[['image']].values.tolist())\n\n    test_ds_prebatch = tf.data.Dataset.from_tensor_slices((test_images))\n    test_ds_prebatch = test_ds_prebatch.map(get_feature, num_parallel_calls=AUTOTUNE)\n\n    test_ds = test_ds_prebatch.batch(BATCH_SIZE)\n\n    y_predictions = butterfly_model.predict(test_ds, verbose=1)\n\n    predicted_label_confidence = decode_predictions(y_predictions, class_names)\n\n    # Combine the actual and predicted data together into a single list\n    es = test_ds_decoded = [(decode_image(m), decode_image_id(id)) for (m, id) in test_ds.unbatch()]\n    ps = predicted_label_confidence\n\n    # Zip the tuples, manually\n    predictions_df = pd.DataFrame(\n        [(es[i][1],             # id\n          es[i][0],             # X\n          ps[i][0][0],          # y_pred\n          ps[i][0][0] in yums,  # yum_pred\n          ps[i][0][1],          # y_conf\n        ) for i in range(len(es))],\n        columns=['id','X','y_pred','yum_pred','y_conf'])\n\n    return predictions_df\n\n\ntest_csv_data = pd.read_csv(test_csv)\n\ntest_predictions_df = make_predictions(test_csv_data)\ntest_predictions_df.head()\n\n","metadata":{"collapsed":false,"pycharm":{"name":"#%%\n"},"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-08-01T13:38:42.679184Z","iopub.execute_input":"2022-08-01T13:38:42.679544Z","iopub.status.idle":"2022-08-01T13:38:51.039556Z","shell.execute_reply.started":"2022-08-01T13:38:42.679514Z","shell.execute_reply":"2022-08-01T13:38:51.038619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---\n## Next Steps\n---\n\nThere is much room for improvement. So, some things to try:\n - Use a different base model or roll-your-own model\n - Freeze a portion of the base model\n - Fine-tune the model\n - Change the image augmenting\n - Improve the classifier\n - Incorporate additional label information into the model","metadata":{"pycharm":{"name":"#%% md\n"}}},{"cell_type":"code","source":"tf.keras.backend.clear_session()","metadata":{"pycharm":{"name":"#%%\n"}},"execution_count":null,"outputs":[]}]}