{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"1. # Deep learning with python project: plant pathology recognition\n- David Coba (student number: 12439665)\n- Enrico Erler (student number: 13287214)\n\nhttps://www.kaggle.com/dcobac/dl-project","metadata":{}},{"cell_type":"markdown","source":"# 1. Setup","metadata":{}},{"cell_type":"code","source":"import os\nimport time\nimport numpy as np\nimport pandas as pd\nfrom IPython.display import Image\nimport tensorflow as tf \nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\nimport matplotlib\nimport pickle\nprint(\"All libs loaded\")","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:09.973022Z","iopub.execute_input":"2021-12-21T13:22:09.97335Z","iopub.status.idle":"2021-12-21T13:22:09.981354Z","shell.execute_reply.started":"2021-12-21T13:22:09.973315Z","shell.execute_reply":"2021-12-21T13:22:09.980093Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Introduction","metadata":{}},{"cell_type":"markdown","source":"This project is based on the Kaggle competition [Plant Pathology 2021 - FGVC8](https://www.kaggle.com/c/plant-pathology-2021-fgvc8).\nThe goal of the competition is to classify leaves of apple trees. Some of them are healthy leaves and other show evidence of having diseases.\nDeveloping a classifier is useful in this situation because it can reduce the number of trees that have to be checked by an inspector, potentially saving a considerable amount of hours. \n\nOriginally images can be labeled as either:\n  - Healthy\n  - Having apple `scab`\n  - Having apple `rust`\n  - Having `frog_eye_leaf_spot`\n  - Having a `complex` disease, which can be\n    - Just the `complex`label\n    - The `complex`label & other disease labels\n    \nAt first we attempted to tackle the problem as it is presented: with a multi-label classifier in which classes can occur simultaneously with sigmoid activation functions mapping activations to probabilities.\nHowever, after unsuccessful attempts we decided to simplify the problem to just the categories present in the [2020 edition of the competition](https://www.kaggle.com/c/plant-pathology-2020-fgvc7), and treat the `complex` label as a single category. It becomes a multi-label problem with 4 mutually exclusive categories. The authors of the competition report that they achieved 97% classification accuracy with \"\"an off the shelf\" `resnet50` classifier [(Thapa, Zhang, Snavely, Belongie & Khan, 2020)](https://bsapubs.onlinelibrary.wiley.com/doi/10.1002/aps3.11390), and this is the benchmark we are trying to match.","metadata":{}},{"cell_type":"markdown","source":"- Example healthy image","metadata":{}},{"cell_type":"code","source":"Image(filename=\"../input/plant-pathology-2021-fgvc8/train_images/b7bcca8ce84f5046.jpg\", width=602, height=476) ","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:09.984187Z","iopub.execute_input":"2021-12-21T13:22:09.985073Z","iopub.status.idle":"2021-12-21T13:22:10.033138Z","shell.execute_reply.started":"2021-12-21T13:22:09.985018Z","shell.execute_reply":"2021-12-21T13:22:10.031874Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Example image with a visible disease (in this case `scab`)","metadata":{}},{"cell_type":"code","source":"Image(filename=\"../input/plant-pathology-2021-fgvc8/train_images/803e3bd17a16e65c.jpg\", width=602, height=476) ","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:10.034728Z","iopub.execute_input":"2021-12-21T13:22:10.035743Z","iopub.status.idle":"2021-12-21T13:22:10.100117Z","shell.execute_reply.started":"2021-12-21T13:22:10.0357Z","shell.execute_reply":"2021-12-21T13:22:10.099366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Data loading","metadata":{}},{"cell_type":"code","source":"data_df = pd.read_csv(\"../input/plant-pathology-2021-fgvc8/train.csv\")\nprint(\"Data shape: \",data_df.shape)\ndata_df.head()","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:10.102057Z","iopub.execute_input":"2021-12-21T13:22:10.103182Z","iopub.status.idle":"2021-12-21T13:22:10.141251Z","shell.execute_reply.started":"2021-12-21T13:22:10.103129Z","shell.execute_reply":"2021-12-21T13:22:10.140565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_df[\"labels\"] = data_df.labels.apply(lambda x: x.split())\ndata_df.head()","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:10.142483Z","iopub.execute_input":"2021-12-21T13:22:10.143634Z","iopub.status.idle":"2021-12-21T13:22:10.173629Z","shell.execute_reply.started":"2021-12-21T13:22:10.14356Z","shell.execute_reply":"2021-12-21T13:22:10.172379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Simplify data from 6 categories (including multi-label `complex` category) to 4 mutually exclusive categories to match 2020 competition outcomes:\n  - `healthy`\n  - `rust`\n  - `scab`\n  - `complex`","metadata":{}},{"cell_type":"code","source":"def simplify_outcome(y):\n    \"\"\"\n    Function to apply to single dataframe cell. \n\n    Converts entries to [\"complex\"], if:\n        1) it\"s a multi-label outcome and,\n        2) it contains one of the original 2020 plant pathology competition outcomes.\n\n    Input: single cell entry\n\n    Output: original outcome (if conditions not met) or [\"complex\"]\n    \"\"\"\n\n    if (len(y) > 1) & any(x in [\"healthy\",\"rust\",\"scab\"] for x in y):\n        return [\"complex\"]\n    else:\n        return y\n    \n# apply function per row in column \"labels\"\ndata_df.labels = data_df.labels.apply(simplify_outcome)\n# only select original 2020 plant pathology competition outcomes\nsimple_labels = [\"healthy\",\"rust\",\"scab\",\"complex\"]\nonly_simple_labels = data_df.labels.apply(lambda x:\n    any(item for item in simple_labels if (item in x) and (len(x) == 1))\n)\n\ndata_df = data_df[only_simple_labels]\n\n# Unlist labels: [a] -> a\ndata_df.loc[:, \"labels\"] = data_df.loc[:, \"labels\"].apply(lambda x: x[0])\n\nprint(\"Outcomes:\\n\",data_df.labels.value_counts())\nprint(\"\\nData shape: \", data_df.shape)\ndata_df.head()","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:10.175299Z","iopub.execute_input":"2021-12-21T13:22:10.175561Z","iopub.status.idle":"2021-12-21T13:22:10.263794Z","shell.execute_reply.started":"2021-12-21T13:22:10.175531Z","shell.execute_reply":"2021-12-21T13:22:10.262991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Split data into testing, validation and testing sets.\n- Data was split so that 10% of the data was reserved to testing, ~20% to validation, and ~70% to training \n- Images were loaded with Tensorflow's `DataGenerator` and `flow_from_dataframe` pipeline.\n- The images were resized to 256 by 256 pixels as bigger images couldn't be stored in the available VRAM capacity (11Gb) without reducing the batch size too much\n","metadata":{}},{"cell_type":"code","source":"train_df, test_df = train_test_split(\n    data_df, #.iloc[1:100, ] # change to only load a subset of the data\n    test_size = 0.1,\n    random_state = 123 # set seed for reproducibility\n)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:10.265689Z","iopub.execute_input":"2021-12-21T13:22:10.266186Z","iopub.status.idle":"2021-12-21T13:22:10.277547Z","shell.execute_reply.started":"2021-12-21T13:22:10.266139Z","shell.execute_reply":"2021-12-21T13:22:10.276457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_size = 256","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:10.280027Z","iopub.execute_input":"2021-12-21T13:22:10.280866Z","iopub.status.idle":"2021-12-21T13:22:10.288234Z","shell.execute_reply.started":"2021-12-21T13:22:10.280818Z","shell.execute_reply":"2021-12-21T13:22:10.28722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_generator = ImageDataGenerator(\n    rescale=1./255,\n    validation_split = 0.22 # results in: ~70% train, ~20% valid & 10% test data\n)\n\ntest_generator = ImageDataGenerator(rescale=1./255)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:10.292136Z","iopub.execute_input":"2021-12-21T13:22:10.292902Z","iopub.status.idle":"2021-12-21T13:22:10.302003Z","shell.execute_reply.started":"2021-12-21T13:22:10.292699Z","shell.execute_reply":"2021-12-21T13:22:10.300865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_loader = data_generator.flow_from_dataframe(\n    seed = 123,\n    dataframe = train_df,\n    directory = \"../input/plant-pathology-2021-fgvc8/train_images/\",\n    subset=\"training\",\n    x_col = \"image\",\n    y_col = \"labels\",\n    target_size = (img_size, img_size), \n    class_mode = \"categorical\",\n    batch_size = 32,\n    shuffle = True\n)\n\nvalid_loader = data_generator.flow_from_dataframe(\n    seed = 123,\n    dataframe = train_df,\n    directory= \"../input/plant-pathology-2021-fgvc8/train_images/\",\n    subset=\"validation\",\n    x_col = \"image\",\n    y_col = \"labels\",\n    target_size = (img_size, img_size), \n    class_mode = \"categorical\",\n    batch_size = 32,\n    shuffle = True\n)\n\ntest_loader = test_generator.flow_from_dataframe(\n    seed = 123,\n    dataframe = test_df,\n    directory= \"../input/plant-pathology-2021-fgvc8/train_images/\",\n    x_col = \"image\",\n    y_col = \"labels\",\n    target_size = (img_size, img_size), \n    class_mode = \"categorical\",\n    batch_size = 32,\n    shuffle = False # Makes easier to assess test predictions\n)\n\nprint(\"Finished loading data.\")","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:10.303507Z","iopub.execute_input":"2021-12-21T13:22:10.303939Z","iopub.status.idle":"2021-12-21T13:22:45.504479Z","shell.execute_reply.started":"2021-12-21T13:22:10.303877Z","shell.execute_reply":"2021-12-21T13:22:45.503625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Resnet & feature extraction","metadata":{}},{"cell_type":"markdown","source":"We have used `resnet50` to extract features from the pictures. The body of the network outputs 2048 8x8 filters, which made training not viable with the hardware we have access to. We considered doing a `MaxPool` pass reducing the filter size to 4x4 before flattening the extracted features, or using global average pooling. We settled on using an extra `MaxPool` pass with a 2x2 kernel, valid padding and a stride of 2, resulting in 2048 feature maps of size 4x4. ","metadata":{}},{"cell_type":"markdown","source":"- Select whether to load the `resnet50` body or not. It can be trainable or not.\n  - Unless we are training a `resnet50` model or extracting its features this is not necessary.","metadata":{}},{"cell_type":"code","source":"LOAD_RESNET = False\nTRAINABLE = False","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.505756Z","iopub.execute_input":"2021-12-21T13:22:45.506017Z","iopub.status.idle":"2021-12-21T13:22:45.510278Z","shell.execute_reply.started":"2021-12-21T13:22:45.505984Z","shell.execute_reply":"2021-12-21T13:22:45.509578Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if LOAD_RESNET:\n    resnet_base = keras.applications.resnet.ResNet50(\n        include_top = False,\n        weights = \"imagenet\",\n        input_shape = (img_size, img_size, 3)\n    )\n    \n    resnet_base.trainable = TRAINABLE\n\n    model_resnet_body = keras.Sequential([\n        resnet_base,\n        keras.layers.MaxPool2D(\n            pool_size=(2, 2), \n            strides=2, \n            padding=\"valid\"\n        ), \n        layers.Flatten()\n        # layers.GlobalAveragePooling2D()\n    ])\n\n    model_resnet_body.compile()\n    print(\"resnet loaded\")\n    ","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.511613Z","iopub.execute_input":"2021-12-21T13:22:45.512051Z","iopub.status.idle":"2021-12-21T13:22:45.52426Z","shell.execute_reply.started":"2021-12-21T13:22:45.512019Z","shell.execute_reply":"2021-12-21T13:22:45.523338Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Training models with this data set takes a long time, mainly because of bottlenecks in the IO streams loading pictures into memory.\nIf we are not interesting in fine-tuning the `resnet50` weights, we can pre-extract all features from the images before training the classifier to avoid having to evaluate the `resnet50` body at every iteration.\nHowever, we ultimately abandoned this idea, since it made augmenting the dataset significantly more difficult and we wanted to fine-tune the model weights.\n\n- It works, but without data augmentation.","metadata":{}},{"cell_type":"code","source":"EXPORT_RESNET_FEATURES = False\nIMPORT_RESNET_FEATURES = False","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.525637Z","iopub.execute_input":"2021-12-21T13:22:45.526346Z","iopub.status.idle":"2021-12-21T13:22:45.537418Z","shell.execute_reply.started":"2021-12-21T13:22:45.526311Z","shell.execute_reply":"2021-12-21T13:22:45.536498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if EXPORT_RESNET_FEATURES:\n    def serialize_features(resnet, loader, filename):\n        df = pd.DataFrame({\"image\": loader.filenames, \"labels\": loader.classes})\n        with open(f\"../input/resnet_features/{filename}_label.h5\", \"wb\") as file:\n            pickle.dump(df, file);\n        \n        time_0 = time.time()\n        features = resnet.predict(loader)\n        features = tf.convert_to_tensor(features)        \n        with open(f\"../input/resnet_features/{filename}\", \"wb\") as file:\n            pickle.dump(features, file);\n        time_1 = time.time()\n\n        ellapsed = round(time_1 - time_0, ndigits = 1)\n        print(f\"{filename} features extracted in {ellapsed}s.\")\n        return df, features\n\n    labels_train, features_train = serialize_features(model_resnet_body, train_loader, \"train.h5\")\n    labels_valid, features_valid = serialize_features(model_resnet_body, valid_loader, \"valid.h5\")\n    labels_test, features_test = serialize_features(model_resnet_body, test_loader, \"test.h5\")\n\nif IMPORT_RESNET_FEATURES: # load pre-saved resnet features\n    print(\"Loading pre-extracted features\")\n\n    with open(\"../input/resnet_features/train.h5\", \"rb\") as file:\n        features_train = pickle.load(file)\n    with open(\"../input/resnet_features/train_label.h5\", \"rb\") as file:\n        labels_train = pickle.load(file)\n    print(\"Training features loaded as a tensor\")\n\n    with open(\"../input/resnet_features/valid.h5\", \"rb\") as file:\n        features_valid = pickle.load(file)\n    with open(\"../input/resnet_features/valid_label.h5\", \"rb\") as file:\n        labels_valid = pickle.load(file)\n    print(\"Validation features loaded as a tensor\")\n\n    with open(\"../input/resnet_features/test.h5\", \"rb\") as file:\n        features_test = pickle.load(file)\n    with open(\"../input/resnet_features/test_label.h5\", \"rb\") as file:\n        labels_test = pickle.load(file)\n    print(\"Testing features loaded as a tensor\")","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.539133Z","iopub.execute_input":"2021-12-21T13:22:45.539634Z","iopub.status.idle":"2021-12-21T13:22:45.55538Z","shell.execute_reply.started":"2021-12-21T13:22:45.53959Z","shell.execute_reply":"2021-12-21T13:22:45.554399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Sanity check","metadata":{}},{"cell_type":"code","source":"if EXPORT_RESNET_FEATURES or IMPORT_RESNET_FEATURES:\n    if train_loader.n != features_train.shape[0] or \\\n        valid_loader.n != features_valid.shape[0] or \\\n        test_loader.n != features_test.shape[0] :\n        print(f\"Training data: {len(train_loader.filenames)} in the data loader vs {features_train.shape[0]} in the data array.\")\n        print(f\"Validating data: {len(valid_loader.filenames)} in the data loader vs {features_valid.shape[0]} in the data array.\")\n        print(f\"Testing data: {len(test_loader.filenames)} in the data loader vs {features_test.shape[0]} in the data array.\")\n        raise Exception(\"The dimensions of the loaded features and the data set does not match.\")\n    else:\n        print(\"All good\")","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.557248Z","iopub.execute_input":"2021-12-21T13:22:45.557502Z","iopub.status.idle":"2021-12-21T13:22:45.573834Z","shell.execute_reply.started":"2021-12-21T13:22:45.557471Z","shell.execute_reply":"2021-12-21T13:22:45.572612Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Encode labels as one-hot tensors.","metadata":{}},{"cell_type":"code","source":"if EXPORT_RESNET_FEATURES or IMPORT_RESNET_FEATURES:\n    target_train = tf.one_hot(labels_train.labels, len(train_loader.class_indices))\n    target_valid = tf.one_hot(labels_valid.labels, len(valid_loader.class_indices))\n    target_test  = tf.one_hot(labels_test.labels, len(test_loader.class_indices))","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.576083Z","iopub.execute_input":"2021-12-21T13:22:45.576426Z","iopub.status.idle":"2021-12-21T13:22:45.589951Z","shell.execute_reply.started":"2021-12-21T13:22:45.576383Z","shell.execute_reply":"2021-12-21T13:22:45.589012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Model training","metadata":{}},{"cell_type":"markdown","source":"- All models we have trained have similar architectures:\n  - A set of data augmentation layers\n  - A CNN body\n    - resnet50 or \n    - resnet50 with trainable weights or\n    - a custom CNN defined below\n  - A regularized classifier with two hidden layers of 64 & 32 neurons","metadata":{}},{"cell_type":"markdown","source":"- Data augmentation for training","metadata":{}},{"cell_type":"code","source":"data_augmentation_layers = keras.Sequential([\n        layers.RandomFlip(\"horizontal_and_vertical\"),\n        layers.RandomContrast(0.5),\n        layers.RandomRotation(\n                factor = 0.3,\n                fill_mode = \"nearest\",\n                interpolation = \"nearest\"\n                ),\n        layers.RandomTranslation(\n                height_factor = 0.2, \n                width_factor = 0.2,\n                fill_mode = \"nearest\",\n                interpolation = \"nearest\"\n                ),\n])","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.591769Z","iopub.execute_input":"2021-12-21T13:22:45.592304Z","iopub.status.idle":"2021-12-21T13:22:45.693134Z","shell.execute_reply.started":"2021-12-21T13:22:45.59226Z","shell.execute_reply":"2021-12-21T13:22:45.692502Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Custom CNN ","metadata":{}},{"cell_type":"code","source":"default_conv = layers.Conv2D(\n    10, \n    kernel_size = 3, \n    strides = 1, \n    padding = \"same\",\n    kernel_initializer = keras.initializers.GlorotNormal()\n)\n\ncustom_cnn = keras.Sequential([\n    # default_conv + input_shape\n    layers.Conv2D(\n        10, \n        kernel_size = 3, \n        strides = 1, \n        padding = \"same\",\n        kernel_initializer = keras.initializers.GlorotNormal(),\n        input_shape = (img_size, img_size, 3)\n    ),\n    default_conv,\n    layers.MaxPool2D(),\n    default_conv,\n    default_conv,\n    layers.MaxPool2D(),\n    layers.Flatten()\n])","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.694597Z","iopub.execute_input":"2021-12-21T13:22:45.695082Z","iopub.status.idle":"2021-12-21T13:22:45.789387Z","shell.execute_reply.started":"2021-12-21T13:22:45.695042Z","shell.execute_reply":"2021-12-21T13:22:45.788249Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_model(cnn, name):\n    \"\"\"\n    Trains a model with a specific CNN body.\n    \"\"\"\n    model = keras.Sequential([\n        data_augmentation_layers,\n        cnn,\n        # Classifier:\n        layers.BatchNormalization(),\n        layers.Dropout(rate = 0.3),\n        layers.Dense(64, activation = \"relu\"),\n        layers.Dropout(rate = 0.3),\n        layers.Dense(32, activation = \"relu\"),\n        layers.Dropout(rate = 0.3),\n        layers.Dense(4, activation = \"softmax\")\n    ])\n\n    model.compile(\n        optimizer = \"adam\",\n        loss = \"categorical_crossentropy\", \n        metrics = [\"accuracy\"]\n    )\n    \n    all_callbacks = [\n    tf.keras.callbacks.EarlyStopping(\n        monitor = \"val_loss\",\n        min_delta = 0.005,\n        patience=15, \n        restore_best_weights = True,\n        ),\n    tf.keras.callbacks.ModelCheckpoint(\n        filepath = f\"../input/models/models/{name}/best_model.h5\", # only saving best model\n        verbose = 0,\n        save_best_only = True,\n        save_weights_only = False,\n        mode = \"auto\",\n        save_freq = \"epoch\"\n        ),                                \n    tf.keras.callbacks.TensorBoard(\n        log_dir = f\"../input/models/models/{name}/logs\",\n        histogram_freq = 1\n        ),\n    ]\n\n    history = model.fit(train_loader,\n        validation_data = valid_loader,\n        epochs = 50,\n        callbacks = all_callbacks\n    )\n\n    print(f\"Saving {name} model ...\")\n    model.save(f\"../input/models/models/{name}/model.h5\"); \n    print(\"Model saved\")\n    print(\"Saving history...\")\n    history = history.history\n    with open(f\"../input/models/models/{name}/history.h5\", \"wb\") as file:\n        pickle.dump(history, file)\n    print(\"History saved\")\n    return model, history","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:25:22.142357Z","iopub.execute_input":"2021-12-21T13:25:22.143321Z","iopub.status.idle":"2021-12-21T13:25:22.155492Z","shell.execute_reply.started":"2021-12-21T13:25:22.143278Z","shell.execute_reply":"2021-12-21T13:25:22.15482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_model(name):\n    \"\"\"\n    Loads a pre-trained model & history.\n    \"\"\"\n    print(f\"Loading {name} model\")\n    model = keras.models.load_model(f\"../input/models/models/{name}/model.h5\");\n    with open(f\"../input/models/models/{name}/history.h5\", \"rb\") as file:\n        history = pickle.load(file)\n    print(\"Last model loaded\")\n    return model, history\n","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:25:24.918171Z","iopub.execute_input":"2021-12-21T13:25:24.918492Z","iopub.status.idle":"2021-12-21T13:25:24.925548Z","shell.execute_reply.started":"2021-12-21T13:25:24.918463Z","shell.execute_reply":"2021-12-21T13:25:24.924557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Select whether to train a model in this session, and which model.","metadata":{}},{"cell_type":"code","source":"TRAIN_MODEL = False","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.823812Z","iopub.execute_input":"2021-12-21T13:22:45.824221Z","iopub.status.idle":"2021-12-21T13:22:45.839832Z","shell.execute_reply.started":"2021-12-21T13:22:45.824191Z","shell.execute_reply":"2021-12-21T13:22:45.839002Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_resnet_name():\n    if TRAINABLE:\n        return \"resnet_trainable\"\n    else: return \"resnet_not_trainable\"","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.841353Z","iopub.execute_input":"2021-12-21T13:22:45.841756Z","iopub.status.idle":"2021-12-21T13:22:45.851447Z","shell.execute_reply.started":"2021-12-21T13:22:45.841723Z","shell.execute_reply":"2021-12-21T13:22:45.850669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Select which model to train.","metadata":{}},{"cell_type":"code","source":"# CNN, CNN_NAME = model_resnet_body, get_resnet_name() # or\n# CNN, CNN_NAME = custom_cnn, \"custom_cnn\"","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:45.85281Z","iopub.execute_input":"2021-12-21T13:22:45.853238Z","iopub.status.idle":"2021-12-21T13:22:45.865093Z","shell.execute_reply.started":"2021-12-21T13:22:45.853202Z","shell.execute_reply":"2021-12-21T13:22:45.86393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Train chosen model and load the last version of the others, or load all previous versions","metadata":{}},{"cell_type":"code","source":"if TRAIN_MODEL:\n    if CNN_NAME == \"custom_cnn\": \n        model_custom_cnn, history_custom_cnn  = train_model(CNN, CNN_NAME)\n        model_resnet_trainable, history_resnet_trainable = load_model(\"resnet_trainable\")\n        model_resnet_not_trainable, history_resnet_not_trainable = load_model(\"resnet_not_trainable\")\n    elif CNN_NAME == \"resnet_trainable\":\n        model_resnet_trainable, history_resnet_trainable = train_model(CNN, CNN_NAME)\n        model_custom_cnn, history_custom_cnn = load_model(\"custom_cnn\")\n        model_resnet_not_trainable, history_resnet_not_trainable = load_model(\"resnet_not_trainable\")\n    elif CNN_NAME == \"resnet_not_trainable\":\n        model_resnet_not_trainable, history_resnet_not_trainable = train_model(CNN, CNN_NAME)\n        model_custom_cnn, history_custom_cnn = load_model(\"custom_cnn\")\n        model_resnet_trainable, history_resnet_trainable = load_model(\"resnet_trainable\")\nelse: # Just load all pre-trained models\n    model_custom_cnn, history_custom_cnn = load_model(\"custom_cnn\")\n    model_resnet_trainable, history_resnet_trainable = load_model(\"resnet_trainable\")\n    model_resnet_not_trainable, history_resnet_not_trainable = load_model(\"resnet_not_trainable\")\n ","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:25:32.125896Z","iopub.execute_input":"2021-12-21T13:25:32.126867Z","iopub.status.idle":"2021-12-21T13:25:53.154775Z","shell.execute_reply.started":"2021-12-21T13:25:32.126816Z","shell.execute_reply":"2021-12-21T13:25:53.153744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Model evaluation & discussion","metadata":{}},{"cell_type":"markdown","source":"## 6.1 Accuracy","metadata":{}},{"cell_type":"code","source":"test_accuracy = pd.Series(\n    {\"custom_cnn\" : model_custom_cnn.evaluate(test_loader)[1],\n    \"resnet_trainable\" : model_resnet_trainable.evaluate(test_loader)[1],\n    \"resnet_not_trainable\" : model_resnet_not_trainable.evaluate(test_loader)[1]\n    }\n)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.269691Z","iopub.status.idle":"2021-12-21T13:22:46.270046Z","shell.execute_reply.started":"2021-12-21T13:22:46.269848Z","shell.execute_reply":"2021-12-21T13:22:46.269865Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"acc_plot = test_accuracy.plot(\n    kind = \"bar\", \n    title = \"Accuracy on the testing set\",\n    ylabel = 'Accuracy in %', \n    ylim = (0, 1)\n)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.271757Z","iopub.status.idle":"2021-12-21T13:22:46.272096Z","shell.execute_reply.started":"2021-12-21T13:22:46.271915Z","shell.execute_reply":"2021-12-21T13:22:46.271945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig = acc_plot.get_figure()\n# fig.savefig(\"../output/local/accuracies.svg\", format = \"svg\", dpi = 1200)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.273258Z","iopub.status.idle":"2021-12-21T13:22:46.273556Z","shell.execute_reply.started":"2021-12-21T13:22:46.273399Z","shell.execute_reply":"2021-12-21T13:22:46.273414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- The custom CNN did not perform well at all under this training conditions.\n- The `resnet50` with its original `imagenet` weights does not reach high accuracy either (around 50%).\n- Fine tuning its weights over training increases accuracy to around 86% accuracy.","metadata":{}},{"cell_type":"markdown","source":"## 6.2 Learning slopes","metadata":{}},{"cell_type":"markdown","source":"However, training these models takes around 20-40 min per epoch, depending on which model and on which hardware they are trained.\nThis means that we cannot compare training times across models.\nBecause of the long training times, we have not been able to run models for longer than ~30 epochs and the performance of the models could potentially improve with more training.\nHowever, we can check how the loss function & accuracy evolve over training to assess if more training would be beneficial to increase out-of-sample prediction accuracy.","metadata":{}},{"cell_type":"code","source":"pd.DataFrame(history_custom_cnn).loc[:, [\"loss\", \"val_loss\"]].plot(title = \"Custom CNN\")\npd.DataFrame(history_custom_cnn).loc[:, [\"Accuracy\", \"val_Accuracy\"]].plot(title = \"Custom CNN\")","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.274566Z","iopub.status.idle":"2021-12-21T13:22:46.274863Z","shell.execute_reply.started":"2021-12-21T13:22:46.274709Z","shell.execute_reply":"2021-12-21T13:22:46.274725Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this case it is hard to tell whether further training would improve the performance of the model. Although the loss function and the accuracy metric have slopes that indicate that they would probably keep improving, it is hard to see if that would be the case for their counterparts on the validation set, since the values make big jumps after each iteration of training. Nevertheless, overfitting is clearly visible after only a few epochs.","metadata":{}},{"cell_type":"code","source":"pd.DataFrame(history_resnet_not_trainable).loc[:, [\"loss\", \"val_loss\"]].plot(title = \"Non-trainable resnet50\")\npd.DataFrame(history_resnet_not_trainable).loc[:, [\"accuracy\", \"val_accuracy\"]].plot(title = \"Non-trainable resnet50\")","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.276628Z","iopub.status.idle":"2021-12-21T13:22:46.276976Z","shell.execute_reply.started":"2021-12-21T13:22:46.276783Z","shell.execute_reply":"2021-12-21T13:22:46.276799Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The validation-set metrics from the model with `resnet50` with non-trainable weights seem to have flattened out, so it looks like the model would not benefit from additional training. However, this is not entirely clear and it might be possible that these are \"local\" flats and that after further training the performance of the model would start improving. Overfitting can also be seen to occur after around epoch 20, however.","metadata":{}},{"cell_type":"code","source":"pd.DataFrame(history_resnet_trainable).loc[:, [\"loss\", \"val_loss\"]].plot()\ntrainable_history = pd.DataFrame(history_resnet_trainable).loc[:, [\"Accuracy\", \"val_Accuracy\"]].plot()","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.278324Z","iopub.status.idle":"2021-12-21T13:22:46.278626Z","shell.execute_reply.started":"2021-12-21T13:22:46.278467Z","shell.execute_reply":"2021-12-21T13:22:46.278483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# fig = trainable_history.get_figure()\n# fig.savefig(\"../output/local/trainable_history.svg\", format = \"svg\", dpi = 1200)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.280164Z","iopub.status.idle":"2021-12-21T13:22:46.280456Z","shell.execute_reply.started":"2021-12-21T13:22:46.280301Z","shell.execute_reply":"2021-12-21T13:22:46.280316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 6.3 Category predictions","metadata":{}},{"cell_type":"markdown","source":"Next, we examine the actual predictions of each model.","metadata":{}},{"cell_type":"code","source":"test_filenames = pd.DataFrame({\"image\": test_loader.filenames})\ntest_labels = pd.DataFrame({\"labels\": test_loader.classes})\n# Rename labels to original strings\ntest_labels = test_labels.replace({\"labels\": {v: k for (k, v) in test_loader.class_indices.items()}})\n\ndef get_model_predictions(model):\n    predicted_probs = pd.DataFrame(model.predict(x = test_loader))\n    # Rename columns to class labels\n    predicted_probs.rename(columns = {v : k for (k, v) in\n                                      test_loader.class_indices.items() },\n                           inplace = True)\n    y_pred = pd.DataFrame(predicted_probs.iloc[:, :].idxmax(axis = 1)).rename(columns = {0 : \"y_pred\"})\n    predictions = pd.concat([\n        test_filenames,\n        test_labels,\n        y_pred,\n        predicted_probs\n    ],\n                            axis = 1)\n    return predictions","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.285387Z","iopub.status.idle":"2021-12-21T13:22:46.286196Z","shell.execute_reply.started":"2021-12-21T13:22:46.28587Z","shell.execute_reply":"2021-12-21T13:22:46.285903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions_custom_cnn = get_model_predictions(model_custom_cnn)\npredictions_resnet_not_trainable = get_model_predictions(model_resnet_not_trainable)\npredictions_resnet_trainable = get_model_predictions(model_resnet_trainable)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.287556Z","iopub.status.idle":"2021-12-21T13:22:46.288204Z","shell.execute_reply.started":"2021-12-21T13:22:46.287906Z","shell.execute_reply":"2021-12-21T13:22:46.287952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions_custom_cnn","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.289511Z","iopub.status.idle":"2021-12-21T13:22:46.290296Z","shell.execute_reply.started":"2021-12-21T13:22:46.28999Z","shell.execute_reply":"2021-12-21T13:22:46.290022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We already showed that the model with a custom CNN did not preform well, and we can see that it is just predicting the `rust` label for every picture.","metadata":{}},{"cell_type":"code","source":"predictions_resnet_not_trainable","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.29162Z","iopub.status.idle":"2021-12-21T13:22:46.292261Z","shell.execute_reply.started":"2021-12-21T13:22:46.292003Z","shell.execute_reply":"2021-12-21T13:22:46.292028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model with `resnet50` with non-trainable weights does not predict the same label for every picture, but it is not performing very well either since it had ~0.5 accuracy.\nHowever, we cannot detect any unusual pattern in the activations of the last layer either.","metadata":{}},{"cell_type":"code","source":"predictions_resnet_trainable","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.293457Z","iopub.status.idle":"2021-12-21T13:22:46.294097Z","shell.execute_reply.started":"2021-12-21T13:22:46.293816Z","shell.execute_reply":"2021-12-21T13:22:46.293842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model with `resnet50` with trainable weights predicts almost all pictures correctly.","metadata":{}},{"cell_type":"markdown","source":"We can see with more details which categories the `resnet50` models classify properly and which ones does not by taking a look at the confusion matrix.","metadata":{}},{"cell_type":"code","source":"def generate_cm(predictions):\n    cm = confusion_matrix(predictions.labels, predictions.y_pred, labels = [\"complex\", \"healthy\", \"rust\", \"scab\"], normalize = None)\n    disp = ConfusionMatrixDisplay(confusion_matrix = cm, display_labels = np.array([\"complex\",\"healthy\",\"rust\",\"scab\"]))\n    disp.plot(cmap = \"Blues\") \n    print(\"Accuracy from confusion matrix: \", round(np.diag(cm).sum()/cm.sum(), 4))\n    return disp","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.295228Z","iopub.status.idle":"2021-12-21T13:22:46.296294Z","shell.execute_reply.started":"2021-12-21T13:22:46.295998Z","shell.execute_reply":"2021-12-21T13:22:46.296034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generate_cm(predictions_resnet_not_trainable)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.297512Z","iopub.status.idle":"2021-12-21T13:22:46.298235Z","shell.execute_reply.started":"2021-12-21T13:22:46.298005Z","shell.execute_reply":"2021-12-21T13:22:46.298031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that the classifier with non-trainable weights never predicts the label `rust`, and that it overestimates the probability of a picture belonging to the `scab` category.","metadata":{}},{"cell_type":"code","source":"cm_plot = generate_cm(predictions_resnet_trainable)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.299455Z","iopub.status.idle":"2021-12-21T13:22:46.300237Z","shell.execute_reply.started":"2021-12-21T13:22:46.300005Z","shell.execute_reply":"2021-12-21T13:22:46.300029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# cm_plot.figure_.savefig(\"../output/local/cm_plot.svg\", format = \"svg\", dpi = 1200)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.301409Z","iopub.status.idle":"2021-12-21T13:22:46.302155Z","shell.execute_reply.started":"2021-12-21T13:22:46.301953Z","shell.execute_reply":"2021-12-21T13:22:46.301975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As is evident from our best-performing model (the `resnet50` with trainable weights), it predicts around 86% of the pictures correctly and most of the values lie on the diagonal of the confusion matrix. Nevertheless, you also see that it misspecifies quite a few of the  `healthy` leaves as `scab`. We think this is likely due to the fact that scab often shows as light dots on the leave in its early stages. Depending on the lighting conditions and sun reflections, healthy leaves can also have light spots (due to the sun shining on them or through them partially). We think this is one of the reasons all models have a higher misclassification rate on `healthy` vs. `scab`.","metadata":{}},{"cell_type":"markdown","source":"Here you can see one example of a `healthy` leaf that was misclassified by the untrained resnet as `scab`:","metadata":{}},{"cell_type":"code","source":"Image(filename=\"../input/plant-pathology-2021-fgvc8/train_images/8bd27e8d6124a5b3.jpg\", width=301, height=238) ","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.303416Z","iopub.status.idle":"2021-12-21T13:22:46.304156Z","shell.execute_reply.started":"2021-12-21T13:22:46.303948Z","shell.execute_reply":"2021-12-21T13:22:46.303975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Due to shadows and lighting, the leaf appears to have bright spots. Using our (human) experience, you can clearly make out that those are shadows and sunrays hitting the leaf's surface. However, we think that the models could not make this generalization as they only have the leaf pictures and do not know about how the sun and shadows interact. All they see are bright spots on the leaf and they think it's `scab`, because a lot of pictures with bright spots are labelled `scab`. We think this is a prime example of a generalization problem of machine learning/ AI.","metadata":{}},{"cell_type":"markdown","source":"## 6.4 Mean F1-scores\n","metadata":{}},{"cell_type":"markdown","source":"The Kaggle competition actually uses mean F1-scores to assess the model accuracies. To make our results comparable to other teams of the competition leaderboard, we transform our accuracies into mean F1-scores.\n\nThe F1-score per category is calculated as follows:\n\n$$\n  F1 = \\frac{TP}{TP+1/2(TP+FN)}\n$$ \n\n\nWhereby $TP$ is the number of true positives (i.e., correctly identified pictures) and $FN$ the number of false negatives (i.e., wrongly identified pictures) per category.\n\nA mean F1-score calculates as the mean of individual F1-scores for each category:\n\n$$\n  F1_{mean}  = \\frac{1}{n}  \\sum_{k=1}^n F1_{k}\n$$ ","metadata":{}},{"cell_type":"code","source":"def mean_f1_score(predictions_table):\n    '''\n    Function to return the mean F1-score.\n    \n    Input: \n    - prediction table with columns named 'labels' for true values and 'y_pred' for predicted values\n\n    Output:\n    - mean F1-score\n    '''\n    \n    cm = confusion_matrix(predictions_table.labels, predictions_table.y_pred, labels = [\"complex\", \"healthy\", \"rust\", \"scab\"], normalize = None)\n\n    TPS = []\n    FPS = []\n    F = []\n\n    for i in range(len(cm)):\n       TPS.append(np.diagonal(cm)[i])\n       FPS.append(cm[:,i].sum() - TPS[i])\n    \n    for r,value in enumerate(TPS):\n      F.append(value / (value + 0.5*(value + FPS[r])))\n      #print(F[r]) # debugging\n    return np.mean(F)\n    ","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.305373Z","iopub.status.idle":"2021-12-21T13:22:46.305689Z","shell.execute_reply.started":"2021-12-21T13:22:46.305524Z","shell.execute_reply":"2021-12-21T13:22:46.305541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mean_f1_score(predictions_resnet_trainable)","metadata":{"collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2021-12-21T13:22:46.307135Z","iopub.status.idle":"2021-12-21T13:22:46.307867Z","shell.execute_reply.started":"2021-12-21T13:22:46.30766Z","shell.execute_reply":"2021-12-21T13:22:46.307682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The mean F1-score is about 0.63 for our best model. We thereby score in the mid-field of the 2021 competition leaderboard. We think this a good result given the short time period we had for this project, the relatively small number of hidden layers and low number of epochs. This can certainly be improved, however, and we offer some suggestions for improvement in the conclusion. ","metadata":{}},{"cell_type":"markdown","source":"## 6.5 Conclusions","metadata":{}},{"cell_type":"markdown","source":"We have trained 3 different models: A custom CNN, an unaltered `resnet50` with a custom dense network, and a re-trained `resnet50`. It was not unsurprising to us that the re-trained `resnet50` model performed best. However, it did not reach the 97% accuracy reported by \n[Thapa et al. (2020)](https://bsapubs.onlinelibrary.wiley.com/doi/10.1002/aps3.11390) and it is unclear whether the performance of our model would improve with further training.\n\nThe biggest issue we have encountered is dealing with the IO bottleneck, which made testing new approaches and models very time consuming.\nFinally, we present a list of possible things we have discussed implementing that could increase the predictive accuracy of our models, but we have not had the time to check.\n\n- Using a subset of (the subset of) the data set. \n    - This will increase how much iterations we can train our models for, but also reduce the data available at each iteration. There is a possibly a sweet spot in this trade-off.\n- Trying different classifiers, since we've very much sticked to a basic one throughout the project.\n- Doing manual data augmentation to pre-extract features with `resnet50`.\n    - This would allow to train the classifier in a fraction of the current time re-loading all images per iteration.\n- Going back to using original labels. \n    - Maybe our simplification has increased the noise to signal ratio in our data.\n  ","metadata":{}}]}