{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":6799,"databundleVersionId":4225553,"sourceType":"competition"},{"sourceId":156719079,"sourceType":"kernelVersion"}],"dockerImageVersionId":30588,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Overview\n\nIn this notebook, we aim to reproduce the VGG convolutional neural networks introduced by [_Simonyan & Zisserman 2015_](https://arxiv.org/pdf/1409.1556v6.pdf). In other words, we will design an analogous model on TensorFlow, train it with the data from the [ImageNet Object Classification Challenge](https://www.kaggle.com/competitions/imagenet-object-localization-challenge) and ultimately benchmark its performance with the original model.","metadata":{}},{"cell_type":"markdown","source":"# Environment Setup\n\nFirst off, we import the necessary dependencies. Importantly, the custom functions to preprocess data and design model components were defined in a separate utility script available at [Utilities for VGG](https://www.kaggle.com/code/giuliobenedetti/utilities-for-vgg?scriptVersionId=157103969).","metadata":{}},{"cell_type":"code","source":"# Import packages\n\n# Data analysis\nimport numpy as np\nimport pandas as pd\n\n# File management\nimport os\nimport shutil\n\n# Image visualisation\nimport matplotlib.pyplot as plt\n\n# Neural network\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom keras import layers\n\n# Custom utilities\nimport utilities_for_vgg as vggutils\n\n# Hide warnings\nimport warnings\nwarnings.filterwarnings(\"ignore\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:40:56.740769Z","iopub.execute_input":"2023-12-31T10:40:56.741133Z","iopub.status.idle":"2023-12-31T10:41:11.921744Z","shell.execute_reply.started":"2023-12-31T10:40:56.741102Z","shell.execute_reply":"2023-12-31T10:41:11.920027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Due to the large size of the data and the great computational demand, a [Tensor Processing Unit](https://www.kaggle.com/docs/tpu) (TPU) provided by Kaggle was used to execute this notebook. Here, we set up the TPU strategy for highly efficient execution.","metadata":{}},{"cell_type":"code","source":"# If TPU is available\ntry:\n    # Detect TPU\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver() \n    print('Running on TPU ', tpu.master())\n\nexcept:\n    tpu = None\n    \n\n# If TPU is defined\nif tpu:\n    # Initialise TPU\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    # Instantiate a TPU distribution strategy\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\n\nelse:\n    # Or instanstiate available strategy\n    strategy = tf.distribute.get_strategy() \n\nprint(\"REPLICAS: \", strategy.num_replicas_in_sync)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:42:30.826901Z","iopub.execute_input":"2023-12-31T10:42:30.827682Z","iopub.status.idle":"2023-12-31T10:42:30.841686Z","shell.execute_reply.started":"2023-12-31T10:42:30.827646Z","shell.execute_reply":"2023-12-31T10:42:30.840565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As a last step to setup our environment, we defined the constants that will be used to preprocess data and design model architectures. A batch size of 256 was fixed to agree with the original training strategy, whereas the number of epochs was changed from the original 74 to 10 due to training issues explained later.","metadata":{}},{"cell_type":"code","source":"# Define constants\n\n# Set seed for reproducibility\nSEED = 123\n# Set batch size for mini-batch gradient descent\nBATCH_SIZE = 32 * strategy.num_replicas_in_sync\n# Set number of epochs to train model\nEPOCH_NUM = 15\n\n# Set kernel size for Conv2D layers\nKERNEL_SIZE = 3\n# Set padding mode for Conv2D layers\nPAD_MODE = \"same\"\n# Set activation function for Conv2D layers\nACTIVATION = \"relu\"\n\n# Set pool size for MaxPool2D layers\nPOOL_SIZE = 2\n# Set strides for MaxPool2D layers\nPOOL_STRIDES = 2","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:42:33.760224Z","iopub.execute_input":"2023-12-31T10:42:33.760791Z","iopub.status.idle":"2023-12-31T10:42:33.769358Z","shell.execute_reply.started":"2023-12-31T10:42:33.76074Z","shell.execute_reply":"2023-12-31T10:42:33.767969Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Preprocessing\n\nDue to computational limitations, we restricted the train set to classes starting with `n01`, which corresponds to about 170000 images or roughly 11% of the total train set.","metadata":{}},{"cell_type":"code","source":"# Store paths to base, train set and subset dirs\nbase_dir = \"/kaggle/input/imagenet-object-localization-challenge/\"\ntrain_dir = base_dir + \"ILSVRC/Data/CLS-LOC/train/\"\ndevel_dir = \"/kaggle/working/devel/\"\n\n# Fetch train set\nraw_train_ds = tf.data.Dataset.list_files(train_dir + \"n01*/*.JPEG\", shuffle=False)\n\n# Find size of train set\ntrain_size = tf.data.experimental.cardinality(raw_train_ds).numpy()\n\n# Shuffle train set\nraw_train_ds = raw_train_ds.shuffle(train_size, reshuffle_each_iteration=False, seed=SEED)\n\nprint(f\"Size of train set: {train_size}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:42:38.282322Z","iopub.execute_input":"2023-12-31T10:42:38.282736Z","iopub.status.idle":"2023-12-31T10:43:06.241062Z","shell.execute_reply.started":"2023-12-31T10:42:38.282706Z","shell.execute_reply":"2023-12-31T10:43:06.238849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The class names were derived from the names of the training image directories. Both the train and the devel subsets included 128 classes as opposed to the original 1000 classes.","metadata":{}},{"cell_type":"code","source":"# Find class names from dir names\nclass_names = np.array(sorted([dir for dir in os.listdir(train_dir) if dir.startswith(\"n01\")]))\n\n# Set number of classes\nCLASS_NUM = len(class_names)\n\nprint(class_names[:10])","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:43:09.050579Z","iopub.execute_input":"2023-12-31T10:43:09.050973Z","iopub.status.idle":"2023-12-31T10:43:09.059344Z","shell.execute_reply.started":"2023-12-31T10:43:09.050943Z","shell.execute_reply":"2023-12-31T10:43:09.057788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The class to which the devel images belong was obtained from the `LOC_val_solution` file. In total, the devel set contained 6350 images.","metadata":{}},{"cell_type":"code","source":"# Import devel labels\ndevel_df = pd.read_csv(base_dir + \"LOC_val_solution.csv\")\n\n# Select devel images belonging to subset of classes\nkeep = devel_df[\"PredictionString\"].str.startswith(\"n01\")\ndevel_df = devel_df[keep]\n\n# Replace prediction string with only class name\ndevel_df[\"PredictionString\"] = devel_df[\"PredictionString\"].str.split(expand=True)[0]\ndevel_df[\"ImageId\"] = base_dir + \"ILSVRC/Data/CLS-LOC/val/\" + devel_df[\"ImageId\"] + \".JPEG\"\n\n# Sort and polish dataframe\ndevel_df.sort_values(by=[\"ImageId\"], inplace=True)\ndevel_df.index = range(len(devel_df))","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:43:11.208385Z","iopub.execute_input":"2023-12-31T10:43:11.208774Z","iopub.status.idle":"2023-12-31T10:43:11.504661Z","shell.execute_reply.started":"2023-12-31T10:43:11.208746Z","shell.execute_reply":"2023-12-31T10:43:11.50339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create devel dir if it doesn't exists\nif not os.path.exists(devel_dir):\n    os.makedirs(devel_dir)\n\n# Create class subdirs if they don't exist\nfor class_name in class_names:\n    if not os.path.exists(devel_dir + class_name):\n        os.makedirs(devel_dir + class_name)\n    \n# For every file in devel set\nfor index, row in devel_df.iterrows():\n    \n    # Define origin path\n    origin = row[\"ImageId\"]\n    # Define class name\n    class_name = row[\"PredictionString\"] + \"/\"\n    # Fetch file name\n    file_name = origin.split(\"/\")[-1]\n    # Define destination path\n    destination = devel_dir + class_name + file_name\n    \n    # Copy devel file to devel dir if it doesn't exist\n    if not os.path.exists(destination):\n        shutil.copy(origin, destination)","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:43:15.061213Z","iopub.execute_input":"2023-12-31T10:43:15.061717Z","iopub.status.idle":"2023-12-31T10:43:15.603692Z","shell.execute_reply.started":"2023-12-31T10:43:15.061677Z","shell.execute_reply":"2023-12-31T10:43:15.602559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Fetch devel set\nraw_devel_ds = tf.data.Dataset.list_files(devel_dir + \"n01*/*.JPEG\", shuffle=False)\n\n# Find size of devel set\ndevel_size = tf.data.experimental.cardinality(raw_devel_ds).numpy()\n\n# Shuffle devel set\nraw_devel_ds = raw_train_ds.shuffle(devel_size, reshuffle_each_iteration=False, seed=SEED)\n\nprint(f\"Size of devel set: {devel_size}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:43:18.802053Z","iopub.execute_input":"2023-12-31T10:43:18.803078Z","iopub.status.idle":"2023-12-31T10:43:18.881611Z","shell.execute_reply.started":"2023-12-31T10:43:18.803025Z","shell.execute_reply":"2023-12-31T10:43:18.880484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Below we visualise how the preprocessing function changes a sample image by comparing the original image with its preprocessed version.","metadata":{}},{"cell_type":"code","source":"# Set path of original image\nfile_path = devel_dir + \"n01531178/ILSVRC2012_val_00025535.JPEG\"\n\n# Preprocess sample image\nimage, label = vggutils.process_path(file_path, class_names)\n\n# Store input shape for model design\ninput_shape = image.shape\n\n# Plot original and preprocessed image\nfig, axes = plt.subplots(1, 2)\n\nfig.set_figheight(3)\nfig.set_figwidth(8)\n\naxes[0].imshow(plt.imread(file_path))\naxes[1].imshow(image)\nprint(f\"Class: {label.numpy()}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-31T10:50:10.116262Z","iopub.execute_input":"2023-12-31T10:50:10.116776Z","iopub.status.idle":"2023-12-31T10:50:10.671061Z","shell.execute_reply.started":"2023-12-31T10:50:10.116736Z","shell.execute_reply":"2023-12-31T10:50:10.669855Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In principle, train and devel images were preprocessed in the same way.  They were first normalised by subtracting the mean RGB value of the entire ImageNet dataset, then rescaled by a factor S equal to 256, and finally cropped randomly to a square shape of (228, 228). In addition, the train images were flipped and enriched in colour contrast to augment the data.","metadata":{}},{"cell_type":"code","source":"# Preprocess train set\ntrain_ds = raw_train_ds.map(\n    lambda path: vggutils.process_path(path, class_names),\n    num_parallel_calls=tf.data.AUTOTUNE\n)\n\n# Preprocess devel set\ndevel_ds = raw_devel_ds.map(\n    lambda path: vggutils.process_path(path, class_names, max_delta=0),\n    num_parallel_calls=tf.data.AUTOTUNE\n)\n\n# Show example\nfor image, label in train_ds.take(1):\n    print(\"Image shape: \", image.numpy().shape)\n    print(\"Label: \", label.numpy())","metadata":{"execution":{"iopub.status.busy":"2023-12-28T21:24:13.767304Z","iopub.execute_input":"2023-12-28T21:24:13.768052Z","iopub.status.idle":"2023-12-28T21:24:14.59443Z","shell.execute_reply.started":"2023-12-28T21:24:13.768019Z","shell.execute_reply":"2023-12-28T21:24:14.593524Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, both the train and test sets are processed for optimal performance by caching, prefetching and splitting them into batches.","metadata":{}},{"cell_type":"code","source":"# Improve train set performance\ntrain_ds = train_ds \\\n    .cache() \\\n    .shuffle(buffer_size=1000, seed=SEED) \\\n    .batch(BATCH_SIZE) \\\n    .prefetch(buffer_size=tf.data.AUTOTUNE)\n\n# Improve devel set performance\ndevel_ds = devel_ds \\\n    .batch(BATCH_SIZE) \\\n    .cache() \\\n    .prefetch(buffer_size=tf.data.AUTOTUNE)","metadata":{"execution":{"iopub.status.busy":"2023-12-28T21:24:26.843664Z","iopub.execute_input":"2023-12-28T21:24:26.843976Z","iopub.status.idle":"2023-12-28T21:24:26.86078Z","shell.execute_reply.started":"2023-12-28T21:24:26.843948Z","shell.execute_reply":"2023-12-28T21:24:26.859963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The train images below were successfully preprocessed and shuffled. Thus, they appear in normalised colour and belong to various classes.","metadata":{}},{"cell_type":"code","source":"# Extract a sample of images from the first batch\nimage_batch, label_batch = next(iter(train_ds))\n\n# Plot a sample of images from the first batch\nplt.figure(figsize=(10, 10))\nfor i in range(9):\n    ax = plt.subplot(3, 3, i + 1)\n    plt.imshow(image_batch[i])\n    label = label_batch[i]\n    plt.title(class_names[label])\n    plt.axis(\"off\")","metadata":{"execution":{"iopub.status.busy":"2023-12-28T21:24:30.93631Z","iopub.execute_input":"2023-12-28T21:24:30.937019Z","iopub.status.idle":"2023-12-28T21:24:35.322124Z","shell.execute_reply.started":"2023-12-28T21:24:30.936981Z","shell.execute_reply":"2023-12-28T21:24:35.321116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Design","metadata":{}},{"cell_type":"markdown","source":"In general, the model architecture resembles the model from the paper pretty well, even though some details may still be missing.","metadata":{}},{"cell_type":"code","source":"# Set up model within scope of tpu strategy\nwith strategy.scope():\n    \n    # Make environment more reproducible\n    vggutils.reproduce_environment(random_state=SEED)\n    \n    # Build model\n    model = vggutils.design_vgg11(\n        class_num=CLASS_NUM,\n        input_shape=input_shape,\n        kernel_size=KERNEL_SIZE,\n        padding=PAD_MODE,\n        activation=ACTIVATION,\n        pool_size=POOL_SIZE,\n        strides=POOL_STRIDES\n    )\n    \n    # Define optimiser\n    sgd_optimiser = vggutils.create_optimiser()\n    \n    # Compile model\n    model.compile(\n        optimizer = sgd_optimiser,\n        loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n        metrics=[\"sparse_categorical_accuracy\"]\n    )","metadata":{"execution":{"iopub.status.busy":"2023-12-28T21:24:43.150228Z","iopub.execute_input":"2023-12-28T21:24:43.150966Z","iopub.status.idle":"2023-12-28T21:24:48.88185Z","shell.execute_reply.started":"2023-12-28T21:24:43.150933Z","shell.execute_reply":"2023-12-28T21:24:48.881005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To test that the model works, we train it for a few epochs. Ideally, we will train the model on the full dataset when once it resembles the model from the paper in terms of parameter number. ","metadata":{}},{"cell_type":"code","source":"# Update learning rate\nlr_decay = keras.callbacks.ReduceLROnPlateau(\n    monitor=\"val_loss\",\n    factor=0.1,\n    patience=2,\n    mode=\"min\",\n    min_delta=1e-4,\n    min_lr=1e-5\n)\n\n# Stop early\nearly_stopping = keras.callbacks.EarlyStopping(\n    monitor=\"val_loss\",\n    patience=3,\n    restore_best_weights=True,\n    start_from_epoch=5\n)\n\n# Train model\nhistory = model.fit(\n    train_ds,\n    validation_data=devel_ds,\n    epochs=EPOCH_NUM,\n    batch_size=BATCH_SIZE,\n    verbose=False,\n    callbacks=[lr_decay, early_stopping]\n)\n\n# Store training history as a dataframe\nhistory_df = pd.DataFrame(history.history)\n\nprint(f\"Train loss: {history_df['loss'].iloc[-1]:.3f}, Devel loss: {history_df['val_loss'].iloc[-1]:.3f}\")\nprint(f\"Train accuracy: {history_df['sparse_categorical_accuracy'].iloc[-1]:.3f}, Devel accuracy: {history_df['val_sparse_categorical_accuracy'].iloc[-1]:.3f}\")","metadata":{"execution":{"iopub.status.busy":"2023-12-27T10:52:03.648996Z","iopub.execute_input":"2023-12-27T10:52:03.649444Z","iopub.status.idle":"2023-12-27T10:52:51.565273Z","shell.execute_reply.started":"2023-12-27T10:52:03.649409Z","shell.execute_reply":"2023-12-27T10:52:51.56316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model contains 128778627 trainable parameters, which aligns pretty well with the 33 million in the paper.\n\nHowever, our model differs from the original in that weights were initialised from a uniform glorot distribution and not from a normal distribution as described in the paper. This variation was chosen because custom initialisation would cause the loss to become a nan value during training. Such issue was likely due to a vanishing gradient and could be solved by defining a custom loss function with a small positive noise (epsilon), so that zero operations cannot occur.","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2)\n\nfig.set_figheight(3)\nfig.set_figwidth(10)\n\n# Visualise performance\nhistory_df.loc[:, [\"loss\", \"val_loss\"]].plot(title=\"Crossentropy\", ax=axes[0])\nhistory_df.loc[:, [\"sparse_categorical_accuracy\", \"val_sparse_categorical_accuracy\"]].plot(title=\"Accuracy\", ax=axes[1])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"VGG11 achieved an accuracy of about 45% on the devel set after 15 epochs. Unexpectedly, the model started overfitting much earlier than the original model, which was trained for 74 epochs according to the paper. Such divergence might be due to the smaller train set used in the current replication.","metadata":{}},{"cell_type":"markdown","source":"# Transfer Learning\n\nThe next step involved building deeper architectures by:\n1. extracting first four convolutional layers and classification head from VGG11,\n2. interlaying them with untrained convolutional layers and\n3. training and validating the resulting model on the data\n\nBy transferring the learnt weights from the base model, we can build more complex and deeper architectures without too much computational burden. In particular, here we aim to reproduce the following three models from _Simonyan & Zisserman 2015_:\n- VGG13 (13-layer ConvNet)\n- VGG16 (16-layer ConvNet)\n- VGG19 (19-layer ConvNet)\n\nIn the following cell, we store the VGG11 in a keras file that will be used to transfer the weights to the deeper architectures.","metadata":{}},{"cell_type":"code","source":"# Define model file\nvgg11_file = f\"/kaggle/working/{CLASS_NUM}class_vgg11.keras\"\n\n# Save model into file for replication purposes\nmodel.save(vgg11_file)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## VGG13\n\nHere, we built, trained and validated a 13-layer architecture based on VGG13. Weights for four convolutional layers and the three dense layers in the classification head were transferred from the pre-trained VGG11 model developed above.","metadata":{}},{"cell_type":"code","source":"# Set up model within scope of tpu strategy\nwith strategy.scope():\n    \n    # Make environment more reproducible\n    vggutils.reproduce_environment(random_state=SEED)\n    \n    # Build model\n    model = vggutils.design_vgg13(\n        model_file=vgg11_file,\n        kernel_size=KERNEL_SIZE,\n        padding=PAD_MODE,\n        activation=ACTIVATION,\n        pool_size=POOL_SIZE,\n        strides=POOL_STRIDES\n    )\n    \n    # Define optimiser\n    sgd_optimiser = vggutils.create_optimiser()\n    \n    # Compile model\n    model.compile(\n        optimizer = sgd_optimiser,\n        loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n        metrics=[\"sparse_categorical_accuracy\"]\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Update learning rate\nlr_decay = keras.callbacks.ReduceLROnPlateau(\n    monitor=\"val_loss\",\n    factor=0.1,\n    patience=2,\n    min_delta=1e-4,\n    min_lr=1e-5\n)\n\n# Stop early\nearly_stopping = keras.callbacks.EarlyStopping(\n    monitor=\"val_loss\",\n    patience=3,\n    restore_best_weights=True,\n    start_from_epoch=5\n)\n\n# Train model\nhistory = model.fit(\n    train_ds,\n    validation_data=devel_ds,\n    epochs=EPOCH_NUM - 5,\n    batch_size=BATCH_SIZE,\n    verbose=False,\n    callbacks=[lr_decay, early_stopping]\n)\n\n# Store training history as a dataframe\nhistory_df = pd.DataFrame(history.history)\n\nprint(f\"Train loss: {history_df['loss'].iloc[-1]:.3f}, Devel loss: {history_df['val_loss'].iloc[-1]:.3f}\")\nprint(f\"Train accuracy: {history_df['sparse_categorical_accuracy'].iloc[-1]:.3f}, Devel accuracy: {history_df['val_sparse_categorical_accuracy'].iloc[-1]:.3f}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2)\n\nfig.set_figheight(3)\nfig.set_figwidth(10)\n\n# Visualise performance\nhistory_df.loc[:, [\"loss\", \"val_loss\"]].plot(title=\"Crossentropy\", ax=axes[0])\nhistory_df.loc[:, [\"sparse_categorical_accuracy\", \"val_sparse_categorical_accuracy\"]].plot(title=\"Accuracy\", ax=axes[1])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define model file\nvgg13_file = f\"/kaggle/working/{CLASS_NUM}class_vgg13.keras\"\n\n# Save model into file for replication purposes\nmodel.save(vgg13_file)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## VGG16\n\nHere, we built, trained and validated a 16-layer architecture based on VGG16. Weights for four convolutional layers and the three dense layers in the classification head were transferred from the pre-trained VGG11 model developed above.","metadata":{}},{"cell_type":"code","source":"# Set up model within scope of tpu strategy\nwith strategy.scope():\n    \n    # Make environment more reproducible\n    vggutils.reproduce_environment(random_state=SEED)\n    \n    # Build model\n    model = vggutils.design_vgg16(\n        model_file=vgg11_file,\n        kernel_size=KERNEL_SIZE,\n        padding=PAD_MODE,\n        activation=ACTIVATION,\n        pool_size=POOL_SIZE,\n        strides=POOL_STRIDES\n    )\n    \n    # Define optimiser\n    sgd_optimiser = vggutils.create_optimiser()\n    \n    # Compile model\n    model.compile(\n        optimizer = sgd_optimiser,\n        loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n        metrics=[\"sparse_categorical_accuracy\"]\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Update learning rate\nlr_decay = keras.callbacks.ReduceLROnPlateau(\n    monitor=\"val_loss\",\n    factor=0.1,\n    patience=2,\n    mode=\"min\",\n    min_delta=1e-4,\n    min_lr=1e-5\n)\n\n# Stop early\nearly_stopping = keras.callbacks.EarlyStopping(\n    monitor=\"val_loss\",\n    patience=3,\n    restore_best_weights=True,\n    start_from_epoch=5\n)\n\n# Train model\nhistory = model.fit(\n    train_ds,\n    validation_data=devel_ds,\n    epochs=EPOCH_NUM - 5,\n    batch_size=BATCH_SIZE,\n    verbose=False,\n    callbacks=[lr_decay, early_stopping]\n)\n\n# Store training history as a dataframe\nhistory_df = pd.DataFrame(history.history)\n\nprint(f\"Train loss: {history_df['loss'].iloc[-1]:.3f}, Devel loss: {history_df['val_loss'].iloc[-1]:.3f}\")\nprint(f\"Train accuracy: {history_df['sparse_categorical_accuracy'].iloc[-1]:.3f}, Devel accuracy: {history_df['val_sparse_categorical_accuracy'].iloc[-1]:.3f}\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2)\n\nfig.set_figheight(3)\nfig.set_figwidth(10)\n\n# Visualise performance\nhistory_df.loc[:, [\"loss\", \"val_loss\"]].plot(title=\"Crossentropy\", ax=axes[0])\nhistory_df.loc[:, [\"sparse_categorical_accuracy\", \"val_sparse_categorical_accuracy\"]].plot(title=\"Accuracy\", ax=axes[1])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define model file\nvgg16_file = f\"/kaggle/working/{CLASS_NUM}class_vgg16.keras\"\n\n# Save model into file for replication purposes\nmodel.save(vgg16_file)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## VGG19\n\nHere, we built, trained and validated a 19-layer architecture based on VGG19. Weights for four convolutional layers and the three dense layers in the classification head were transferred from the pre-trained VGG11 model developed above.","metadata":{}},{"cell_type":"code","source":"# Set up model within scope of tpu strategy\nwith strategy.scope():\n    \n    # Make environment more reproducible\n    vggutils.reproduce_environment(random_state=SEED)\n    \n    # Build model\n    model = vggutils.design_vgg19(\n        model_file=vgg11_file,\n        kernel_size=KERNEL_SIZE,\n        padding=PAD_MODE,\n        activation=ACTIVATION,\n        pool_size=POOL_SIZE,\n        strides=POOL_STRIDES\n    )\n    \n    # Define optimiser\n    sgd_optimiser = vggutils.create_optimiser()\n    \n    # Compile model\n    model.compile(\n        optimizer = sgd_optimiser,\n        loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n        metrics=[\"sparse_categorical_accuracy\"]\n    )","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Update learning rate\nlr_decay = keras.callbacks.ReduceLROnPlateau(\n    monitor=\"val_loss\",\n    factor=0.1,\n    patience=2,\n    mode=\"min\",\n    min_delta=1e-4,\n    min_lr=1e-5\n)\n\n# Stop early\nearly_stopping = keras.callbacks.EarlyStopping(\n    monitor=\"val_loss\",\n    patience=3,\n    restore_best_weights=True,\n    start_from_epoch=5\n)\n\n# Train model\nhistory = model.fit(\n    train_ds,\n    validation_data=devel_ds,\n    epochs=EPOCH_NUM - 5,\n    batch_size=BATCH_SIZE,\n    verbose=False,\n    callbacks=[lr_decay, early_stopping]\n)\n\n# Store training history as a dataframe\nhistory_df = pd.DataFrame(history.history)\n\nprint(f\"Train loss: {history_df['loss'].iloc[-1]:.3f}, Devel loss: {history_df['val_loss'].iloc[-1]:.3f}\")\nprint(f\"Train accuracy: {history_df['sparse_categorical_accuracy'].iloc[-1]:.3f}, Devel accuracy: {history_df['val_sparse_categorical_accuracy'].iloc[-1]:.3f}\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 2)\n\nfig.set_figheight(3)\nfig.set_figwidth(10)\n\n# Visualise performance\nhistory_df.loc[:, [\"loss\", \"val_loss\"]].plot(title=\"Crossentropy\", ax=axes[0])\nhistory_df.loc[:, [\"sparse_categorical_accuracy\", \"val_sparse_categorical_accuracy\"]].plot(title=\"Accuracy\", ax=axes[1])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define model file\nvgg19_file = f\"/kaggle/working/{CLASS_NUM}class_vgg19.keras\"\n\n# Save model into file for replication purposes\nmodel.save(vgg19_file)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusions\n\nIn a nutshell, results were partially successful.\n\nOn the positive side, neural network architectures that are nearly identical to the original VGG models could be designed, trained and validated. Although a smaller dataset was used (roughly 10% of the full set), the data was normalised and processed according to the original pipeline. The great computational demand of the training task was satisfied by executing the current notebook on a TPU, which represents a very relevant skill for deep learning of big data.\n\nOn the negative side, low performance was achieved on the development set. Unlike the original models, VGG11 started overfitting as early as 10 epochs, and the deeper ones as soon as training began. Remarkably, training accuracy easily approached 100%, whereas the development one reached a plateau at about 50%. This large divergence in train and devel accuracies might hint to 1). a too complex architecture for the subset of data used or 2). a significant difference between the train and the devel set. While the former is an inevitable consequence of our computational limitations, the latter would originate from improper and differential data preprocessing, which could be easily fixed after a thorough debugging process.\n\nIn conclusion, this notebook attempted to replicate the deep learning procedure for computer vision described by [_Simonyan & Zisserman 2015_](https://arxiv.org/pdf/1409.1556v6.pdf), also known as VGG11, VGG13, VGG16 and VGG19 architectures. The dataset was taken from a subset of ImageNet and represented 10% of the entire data. Results were successful with respect to designing and training the models as well as preprocessing data. However, state-of-the-art accuracy could be achieved on the training set but not the devel set, due to a large amount of overfitting which is not in line with the original paper. Overall, the current notebook represented an effort and showed the challenges to replicate a highly performant computer vision architecture.","metadata":{}}]}