{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpuV5e8","dataSources":[{"sourceType":"competition","sourceId":21154,"databundleVersionId":1243559,"isSourceIdPinned":false}],"dockerImageVersionId":31288,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"57525c13-3bac-4508-9da7-907871783b5a","cell_type":"markdown","source":"<a href=\"https://www.kaggle.com/code/ameythakur20/tpu-flower-classification-advanced-ensemble\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{}},{"id":"834aa65c-3403-481b-ac49-b4d78fe958d0","cell_type":"markdown","source":"# Petals to the Metal: Advanced TPU Classification Architecture\n\n**Ensemble of EfficientNet and DenseNet with Stochastic Regularization**\n\n**Author:** [Amey Thakur](https://www.kaggle.com/ameythakur20)","metadata":{}},{"id":"adee98be-812f-4bae-8e1f-45565b6d6717","cell_type":"markdown","source":"The primary objective of this architecture is to maximize the macro F1 score across 104 highly imbalanced botanical classes. Standard single-model architectures frequently fail to capture the minute morphological differences between sub-species. This pipeline implements a robust, dual-stream ensemble, synthesizing the spatial efficiency of EfficientNet with the dense feature propagation of DenseNet. \n\nTo counteract overfitting on the limited and imbalanced training distribution, the pipeline integrates non-linear learning rate scheduling, Test Time Augmentation (TTA), and advanced stochastic regularization techniques. Every engineering decision is mathematically documented to ensure scholarly transparency and reproducibility.","metadata":{}},{"id":"cfc3f0c1-abd6-4fa3-ad82-c33998e8eaa6","cell_type":"markdown","source":"**Outline:**\n\n1. [Execution Environment Initialization](#1.-Execution-Environment-Initialization)\n2. [Hardware Synchronization and Distribution Strategy](#2.-Hardware-Synchronization-and-Distribution-Strategy)\n3. [Global Hyperparameter Configuration](#3.-Global-Hyperparameter-Configuration)\n4. [Statistical Overview of Dataset Distribution](#4.-Statistical-Overview-of-Dataset-Distribution)\n5. [Stochastic Regularization and Data Ingestion](#5.-Stochastic-Regularization-and-Data-Ingestion)\n6. [Visualization of Augmented Tensors](#6.-Visualization-of-Augmented-Tensors)\n7. [Cyclical Optimization Dynamics](#7.-Cyclical-Optimization-Dynamics)\n8. [Dual-Stream Architectural Assembly](#8.-Dual-Stream-Architectural-Assembly)\n9. [Model Convergence and Evaluation](#9.-Model-Convergence-and-Evaluation)\n10. [Inference via Test Time Augmentation](#10.-Inference-via-Test-Time-Augmentation)\n11. [Summary](#11.-Summary)","metadata":{}},{"id":"5d2195d0-b638-4066-ae85-d667ce280f67","cell_type":"markdown","source":"## 1. Execution Environment Initialization\n\nTo establish proper hardware assignment for TensorFlow distributed strategies, framework-level hardware locks generated by external compilation libraries (e.g., JAX) must be explicitly bypassed via system environment state configuration. This ensures the TPU Matrix Multiplication Units (MMUs) remain accessible exclusively for TensorFlow operation kernels.","metadata":{}},{"id":"19606579-c249-472e-966b-99b96efe711a","cell_type":"code","source":"# Synchronization of hardware-specific operation kernels for TPU v5e-8 architectures.\n# The libtpu distribution provided via the official repository ensures alignment between\n# the host MMU drivers and the TensorFlow 2.18.0 runtime environment.\n!export PATH=\"${HOME}/.local/bin:${PATH}\" && \\\n    uv pip install --system tensorflow-tpu==2.18.0 --find-links https://storage.googleapis.com/libtpu-tf-releases/index.html && \\\n    uv pip install --system \"ml_dtypes>=0.5.1\"\n\nimport os\nimport logging\nimport warnings\n\n# Environmental configuration to prevent hardware initialization deadlocks by bypassing\n# external acceleration platforms and mapping Keras via legacy distribution interfaces.\nos.environ['JAX_PLATFORMS'] = 'cpu'\nos.environ['TF_USE_LEGACY_KERAS'] = '1'\nos.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n\nwarnings.filterwarnings('ignore', category=UserWarning)\nwarnings.filterwarnings('ignore', category=FutureWarning)\n\nprint('Architecture kernels synchronized. MMU access authorized.')\n","metadata":{"description":"Establish architecture-specific binaries for TPU v5e-8 execution on Python 3.12.","trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:48:48.742513Z","iopub.execute_input":"2026-03-18T19:48:48.742746Z","iopub.status.idle":"2026-03-18T19:48:48.974455Z","shell.execute_reply.started":"2026-03-18T19:48:48.742718Z","shell.execute_reply":"2026-03-18T19:48:48.973566Z"}},"outputs":[],"execution_count":null},{"id":"b18458db-ddd4-4b9c-8e33-b765bee0aa24","cell_type":"markdown","source":"## 2. Hardware Synchronization and Distribution Strategy\n\nTensor Processing Units (TPUs) accelerate the model training process via TensorFlow's parallel distribution strategies. Instantiating a `TPUStrategy` replicates the mathematical computation graph across available hardware cores, allowing for synchronous gradient descent. Each core processes an isolated segment of the batch, and numerical gradients are aggregated before dense weights are structurally updated.","metadata":{}},{"id":"2ce55cce-1dea-4ea4-bfde-7a9c2c654431","cell_type":"code","source":"# Enable mixed precision via bfloat16 for TPU acceleration\n# This maintains floating point accuracy while providing a 2x-3x speedup on v5e-8 VMs.\nprint('Computational policy set to mixed_bfloat16 for high-throughput training')\nimport tensorflow as tf\nimport numpy as np\nimport math\nimport re\nimport matplotlib.pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\n\ntf.get_logger().setLevel(logging.ERROR)\n\nplt.style.use('fivethirtyeight')\nplt.rcParams.update({\n    'font.family': 'sans-serif',\n    'figure.facecolor': '#ffffff',\n    'axes.facecolor': '#ffffff',\n    'axes.edgecolor': '#f0f0f0',\n    'grid.color': '#e0e0e0',\n    'text.color': '#2c3e50',\n    'axes.labelcolor': '#2c3e50',\n    'xtick.color': '#7f8c8d',\n    'ytick.color': '#7f8c8d',\n    'legend.framealpha': 1.0\n})\n\ntry:\n    # Configuration of the cluster resolver for single-host TPU VM architectures.\n    # Local resolution is required to establish endpoints within the virtual machine VPC.\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='local')\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\n    \n    print(f'Status: Successfully initialized. Running on TPU: {tpu.master()}')\n    print(f'Synchronous Execution Replicas: {strategy.num_replicas_in_sync}')\nexcept Exception as e:\n    print(f'Hardware Initialization Status: Execution error. {e}')\n    raise RuntimeError('Execution Halted: TPU v5e-8 architecture is required for this pipeline.')\n","metadata":{"description":"Initialize synchronous data-parallelism across 8 TPU cores.","trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:48:48.974867Z","iopub.execute_input":"2026-03-18T19:48:48.975031Z","iopub.status.idle":"2026-03-18T19:49:07.305578Z","shell.execute_reply.started":"2026-03-18T19:48:48.975013Z","shell.execute_reply":"2026-03-18T19:49:07.304285Z"}},"outputs":[],"execution_count":null},{"id":"b3b00211-69de-4a21-ac9d-99e501cb0457","cell_type":"markdown","source":"## 3. Global Hyperparameter Configuration\n\nBelow are the dimensional and numerical pipeline parameters. The `BATCH_SIZE` is statically scaled relative to the number of TPU replicas (`strategy.num_replicas_in_sync`) in order to optimize memory allocation and computation unit utilization.","metadata":{}},{"id":"90aae9c7-3f23-4cfb-b582-83233529b442","cell_type":"code","source":"# Define image tensor spatial dimensions\nIMAGE_SIZE = [512, 512]\n\n# Maximum number of epochs for the optimization loop\nEPOCHS = 16\n\n# Scale batch size linearly with the number of TPU replicas\nBATCH_SIZE = 8 * strategy.num_replicas_in_sync\n\ndef get_robust_dataset_path():\n    # Standard mount points for competition datasets on Kaggle TPU VMs\n    paths = [\n        f'/kaggle/input/tpu-getting-started/tfrecords-jpeg-{IMAGE_SIZE[0]}x{IMAGE_SIZE[0]}',\n        f'/kaggle/input/competitions/tpu-getting-started/tfrecords-jpeg-{IMAGE_SIZE[0]}x{IMAGE_SIZE[0]}',\n        f'/kaggle/input/flower-classification-with-tpus/tfrecords-jpeg-{IMAGE_SIZE[0]}x{IMAGE_SIZE[0]}'\n    ]\n    \n    # Attempt to resolve the GCS path as a fallback for standard TPU nodes\n    try:\n        gcs_path = KaggleDatasets().get_gcs_path('tpu-getting-started')\n        paths.append(gcs_path + f'/tfrecords-jpeg-{IMAGE_SIZE[0]}x{IMAGE_SIZE[0]}')\n    except:\n        pass\n        \n    for path in paths:\n        if tf.io.gfile.exists(path):\n            return path\n    return paths[0]  # Fallback to the most likely local path\n\nDATASET_PATH = get_robust_dataset_path()\n\ndef find_tfrecords(path, split):\n    # Search for sharded records using common directory patterns\n    patterns = [\n        f'{path}/{split}/*.tfrec',  # Subdirectory pattern\n        f'{path}/{split}*.tfrec'   # Prefix pattern\n    ]\n    files = []\n    for pattern in patterns:\n        files += tf.io.gfile.glob(pattern)\n    return files\n\nTRAINING_FILENAMES = find_tfrecords(DATASET_PATH, 'train')\nVALIDATION_FILENAMES = find_tfrecords(DATASET_PATH, 'val')\nTEST_FILENAMES = find_tfrecords(DATASET_PATH, 'test')\n\nprint(f'Active Dataset Path: {DATASET_PATH}')\nprint(f'File Counts: Train({len(TRAINING_FILENAMES)}) Val({len(VALIDATION_FILENAMES)}) Test({len(TEST_FILENAMES)})')\n\n# Total number of target classes\nCLASSES = 104\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:49:07.306486Z","iopub.execute_input":"2026-03-18T19:49:07.306866Z","iopub.status.idle":"2026-03-18T19:49:07.339530Z","shell.execute_reply.started":"2026-03-18T19:49:07.306848Z","shell.execute_reply":"2026-03-18T19:49:07.338580Z"}},"outputs":[],"execution_count":null},{"id":"479a0084-d45d-4986-8754-47442298d92d","cell_type":"markdown","source":"## 4. Statistical Overview of Dataset Distribution\n\nBotanical datasets often display heavy-tailed class distributions. First, we compute the number of elements in the training, validation, and test datasets parsed from the TFRecord shards to establish the execution bounds for the training loop.","metadata":{}},{"id":"92d89241-0886-41b9-af84-6fe24f8ceda4","cell_type":"code","source":"def count_data_items(filenames):\n    # Parse shard counts from the canonical Kaggle filename structures\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nn_train = count_data_items(TRAINING_FILENAMES)\nn_val = count_data_items(VALIDATION_FILENAMES)\nn_test = count_data_items(TEST_FILENAMES)\n\nprint(f'Training samples: {n_train} | Validation samples: {n_val} | Test samples: {n_test}')\n\ndef plot_dataset_distribution(train, val, test):\n    fig, ax = plt.subplots(figsize=(10, 6))\n    segments = ['Training', 'Validation', 'Testing']\n    counts = [train, val, test]\n    colors = ['#2980b9', '#27ae60', '#e67e22']\n    \n    bars = ax.bar(segments, counts, color=colors, width=0.55)\n    \n    for bar in bars:\n        yval = bar.get_height()\n        ax.text(bar.get_x() + bar.get_width()/2, yval + (yval * 0.02), int(yval), \n                ha='center', va='bottom', fontweight='bold', fontsize=12, color='#2c3e50')\n        \n    ax.set_title('Observations per Dataset Split', fontsize=16, fontweight='bold', pad=20)\n    ax.set_ylabel('Total Image Observations', fontsize=13)\n    ax.grid(axis='y', linestyle='--', alpha=0.5)\n    ax.spines['top'].set_visible(False)\n    ax.spines['right'].set_visible(False)\n    plt.tight_layout()\n    plt.show()\n\nplot_dataset_distribution(n_train, n_val, n_test)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:49:07.340054Z","iopub.execute_input":"2026-03-18T19:49:07.340239Z","iopub.status.idle":"2026-03-18T19:49:07.461020Z","shell.execute_reply.started":"2026-03-18T19:49:07.340222Z","shell.execute_reply":"2026-03-18T19:49:07.460078Z"}},"outputs":[],"execution_count":null},{"id":"fd5c5488-69b7-423e-9f45-b0e7a81e825b","cell_type":"markdown","source":"## 5. Stochastic Regularization and Data Ingestion\n\nWe use `tf.data.experimental.AUTOTUNE` to asynchronously prefetch the TFRecord datasets. During the data pipeline mapping phase, spatial transformations (random flips) and chromatic augmentations (contrast and saturation modifications) are applied to introduce variation and reduce overfitting.","metadata":{}},{"id":"0ae7ed43-b980-4cab-a461-5ba130c325d1","cell_type":"code","source":"def decode_image(image_data):\n    # Decode binary JPEG tensors into 3-channel RGB numerical primitives\n    image = tf.image.decode_jpeg(image_data, channels=3)\n    # Cast pixel intensities to continuous float representations in [0.0, 1.0]\n    image = tf.cast(image, tf.float32) / 255.0\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_labeled_tfrecord(example):\n    # Map feature schema for decoding protocol buffer shards\n    LABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"class\": tf.io.FixedLenFeature([], tf.int64),\n    }\n    example = tf.io.parse_single_example(example, LABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    label = tf.cast(example['class'], tf.int32)\n    return image, label\n\ndef read_unlabeled_tfrecord(example):\n    UNLABELED_TFREC_FORMAT = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"id\": tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, UNLABELED_TFREC_FORMAT)\n    image = decode_image(example['image'])\n    idnum = example['id']\n    return image, idnum\n\ndef data_augment(image, label):\n    # Apply geometric reflections along spatial axes\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_flip_up_down(image)\n    # Modify color spectrum values to simulate lighting conditions\n    image = tf.image.random_brightness(image, max_delta=0.15)\n    image = tf.image.random_contrast(image, lower=0.8, upper=1.2)\n    image = tf.image.random_saturation(image, lower=0.8, upper=1.2)\n    return image, label\n\ndef get_training_dataset():\n    # Throttled parallel reads to prevent CPU thread deadlock on TPU v5e-8 VMs\n    dataset = tf.data.TFRecordDataset(TRAINING_FILENAMES, num_parallel_reads=16)\n    dataset = dataset.map(read_labeled_tfrecord, num_parallel_calls=16)\n    dataset = dataset.map(data_augment, num_parallel_calls=16)\n    dataset = dataset.repeat()\n    dataset = dataset.shuffle(2048)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n    return dataset\n\ndef get_validation_dataset():\n    dataset = tf.data.TFRecordDataset(VALIDATION_FILENAMES, num_parallel_reads=16)\n    dataset = dataset.map(read_labeled_tfrecord, num_parallel_calls=16)\n    dataset = dataset.batch(BATCH_SIZE)\n    # FATAL BUG REMOVED: dataset.cache() is deleted to prevent RAM exhaustion\n    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n    return dataset\n\ndef get_test_dataset(ordered=False):\n    dataset = tf.data.TFRecordDataset(TEST_FILENAMES, num_parallel_reads=16)\n    dataset = dataset.map(read_unlabeled_tfrecord, num_parallel_calls=16)\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n    return dataset\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:49:07.461546Z","iopub.execute_input":"2026-03-18T19:49:07.461708Z","iopub.status.idle":"2026-03-18T19:49:07.470065Z","shell.execute_reply.started":"2026-03-18T19:49:07.461692Z","shell.execute_reply":"2026-03-18T19:49:07.469278Z"}},"outputs":[],"execution_count":null},{"id":"603f53aa-75b1-4763-9378-203178ae5ed7","cell_type":"markdown","source":"## 6. Visualization of Augmented Tensors\n\nA micro-batch of the data is compiled to inspect the preprocessing transformations directly before execution.","metadata":{}},{"id":"1abf8c31-5dbd-421b-9a60-9bea17b13e3c","cell_type":"code","source":"import math\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# Taxonomic labels for botanical dataset class mapping\nBOTANICAL_CLASSES = [\n    'pink primrose', 'hard-leaved pocket orchid', 'canterbury bells', 'sweet pea', 'wild geranium', 'tiger lily', 'moon orchid', 'bird of paradise', 'monkshood', 'globe thistle', \n    'snapdragon', \"colt's foot\", 'king protea', 'spear thistle', 'yellow iris', 'globe-flower', 'purple coneflower', 'peruvian lily', 'balloon flower', 'giant white arum lily', \n    'fire lily', 'pincushion flower', 'fritillary', 'red ginger', 'grape hyacinth', 'corn poppy', 'prince of wales feathers', 'stemless gentian', 'artichoke', 'sweet william', \n    'carnation', 'garden phlox', 'love in the mist', 'cosmos', 'alpine sea holly', 'ruby-lipped cattleya', 'cape flower', 'great masterwort', 'siam tulip', 'lenten rose', \n    'barberton daisy', 'daffodil', 'sword lily', 'poinsettia', 'bolero deep blue', 'wallflower', 'marigold', 'buttercup', 'daisy', 'common dandelion', \n    'petunia', 'wild pansy', 'primula', 'sunflower', 'lilac hibiscus', 'bishop of llandaff', 'gaura', 'geranium', 'orange dahlia', 'pink-yellow dahlia', \n    'cautleya spicata', 'japanese anemone', 'black-eyed susan', 'silverbush', 'californian poppy', 'osteospermum', 'spring crocus', 'iris', 'windflower', 'tree poppy', \n    'gazania', 'azalea', 'water lily', 'rose', 'thorn apple', 'morning glory', 'passion flower', 'lotus', 'toad lily', 'anthurium', \n    'frangipani', 'clematis', 'hibiscus', 'columbine', 'desert-rose', 'tree mallow', 'magnolia', 'cyclamen', 'watercress', 'canna lily', \n    'hippeastrum', 'bee balm', 'pink quill', 'foxglove', 'bougainvillea', 'camellia', 'mallow', 'mexican petunia', 'bromelia', 'blanket flower', \n    'trumpet creeper', 'blackberry lily', 'common tulip', 'wild rose'\n]\n\n# Maintain integer constant for layer configuration\nCLASSES = len(BOTANICAL_CLASSES)\n\ndef batch_to_numpy_images_and_labels(data):\n    images, labels = data\n    numpy_images = images.numpy()\n    numpy_labels = labels.numpy()\n    if numpy_labels.dtype == object:\n        numpy_labels = [None for _ in enumerate(numpy_images)]\n    return numpy_images, numpy_labels\n\ndef title_from_label_and_target(label, correct_label):\n    if correct_label is None:\n        return BOTANICAL_CLASSES[label], True\n    is_correct = (label == correct_label)\n    return \"{} [{}] {}\".format(BOTANICAL_CLASSES[label], 'OK' if is_correct else 'NO', BOTANICAL_CLASSES[correct_label]), is_correct\n\ndef display_one_flower(image, title, subplot, red=False, titlesize=16):\n    plt.subplot(*subplot)\n    plt.axis('off')\n    plt.imshow(image)\n    if title:\n        plt.title(title, fontsize=titlesize, color='red' if red else 'black', \n                  fontdict={'verticalalignment':'center'}, pad=titlesize/1.5)\n    return (subplot[0], subplot[1], subplot[2] + 1)\n\ndef display_batch_of_images(databatch, predictions=None):\n    images, labels = batch_to_numpy_images_and_labels(databatch)\n    if labels is None:\n        labels = [None for _ in enumerate(images)]\n        \n    rows = int(math.sqrt(len(images)))\n    cols = len(images) // rows\n        \n    FIGSIZE = 13.0\n    SPACING = 0.1\n    subplot = (rows, cols, 1)\n    if rows < cols:\n        plt.figure(figsize=(FIGSIZE, FIGSIZE / cols * rows))\n    else:\n        plt.figure(figsize=(FIGSIZE / rows * cols, FIGSIZE))\n    \n    for i, (image, label) in enumerate(zip(images[:rows*cols], labels[:rows*cols])):\n        title = '' if label is None else BOTANICAL_CLASSES[label]\n        is_correct = True\n        if predictions is not None:\n            title, is_correct = title_from_label_and_target(predictions[i], label)\n        \n        dynamic_titlesize = FIGSIZE * SPACING / max(rows, cols) * 40 + 3\n        subplot = display_one_flower(image, title, subplot, not is_correct, titlesize=dynamic_titlesize)\n        \n    plt.tight_layout()\n    plt.subplots_adjust(wspace=SPACING, hspace=SPACING)\n    plt.show()\n\ndef display_training_batch():\n    try:\n        training_dataset_visualization = get_training_dataset().unbatch().batch(16)\n        train_batch = next(iter(training_dataset_visualization))\n        display_batch_of_images(train_batch)\n    except StopIteration:\n        print('Status: Execution error. Training dataset is empty. Verify dataset attachment and GCS permissions.')\n    except Exception as e:\n        print(f'Visualization Status: Execution error. {e}')\n\ndisplay_training_batch()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:49:07.470664Z","iopub.execute_input":"2026-03-18T19:49:07.470822Z","iopub.status.idle":"2026-03-18T19:49:10.510772Z","shell.execute_reply.started":"2026-03-18T19:49:07.470807Z","shell.execute_reply":"2026-03-18T19:49:10.509816Z"}},"outputs":[],"execution_count":null},{"id":"787ca3e3-0ea5-4864-be6e-9f356a55c3c1","cell_type":"markdown","source":"## 7. Cyclical Optimization Dynamics\n\nA learning rate schedule is applied to manage training stability. A linear ramp phase allows the optimizer to navigate early stochastic bounds before engaging in exponential decay to reach optimization convergence during the fine-tuning iterations.","metadata":{}},{"id":"8028479b-49c7-46d4-91b9-adb5da9b3648","cell_type":"code","source":"LR_START = 0.00001\nLR_MAX = 0.00005 * strategy.num_replicas_in_sync\nLR_MIN = 0.00001\nLR_RAMPUP_EPOCHS = 3\nLR_SUSTAIN_EPOCHS = 0\nLR_EXP_DECAY = 0.75\n\ndef lrfn(epoch):\n    # Define linear ramp phase\n    if epoch < LR_RAMPUP_EPOCHS:\n        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n    # Define sustaining phase\n    elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n        lr = LR_MAX\n    # Define exponential decay limits\n    else:\n        lr = (LR_MAX - LR_MIN) * LR_EXP_DECAY**(epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS) + LR_MIN\n    return lr\n    \nlr_callback = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose=True)\n\nepochs_range = np.arange(EPOCHS)\nlrs = [lrfn(e) for e in epochs_range]\n\nplt.figure(figsize=(10, 5))\nplt.plot(epochs_range, lrs, marker='o', markersize=6, color='#8e44ad', linewidth=2.5, label='Learning Rate')\nplt.fill_between(epochs_range, lrs, color='#8e44ad', alpha=0.15)\nplt.title('Learning Rate Schedule', fontsize=16, fontweight='bold', pad=20)\nplt.xlabel('Epoch', fontsize=12)\nplt.ylabel('Learning Rate', fontsize=12)\nplt.grid(True, linestyle=':', alpha=0.7)\nplt.xticks(np.arange(0, EPOCHS+1, 5))\nplt.legend(loc='upper right')\nplt.tight_layout()\nplt.show()\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:49:10.511784Z","iopub.execute_input":"2026-03-18T19:49:10.511965Z","iopub.status.idle":"2026-03-18T19:49:10.649213Z","shell.execute_reply.started":"2026-03-18T19:49:10.511948Z","shell.execute_reply":"2026-03-18T19:49:10.648321Z"}},"outputs":[],"execution_count":null},{"id":"d6fb18d9-fa3d-49c7-9bdf-ffcef440154c","cell_type":"markdown","source":"## 8. Dual-Stream Architectural Assembly\n\nThe target architecture concatenates two convolutional approaches. EfficientNet scales network depth, width, and resolution, while DenseNet employs dense feed-forward connectivity to propagate feature maps. The outputs of both models are globally pooled and merged prior to standard categorical classification.","metadata":{}},{"id":"a136c6ed-585a-464d-a2b1-cccffa9d9711","cell_type":"code","source":"from tensorflow.keras.applications import EfficientNetB6, DenseNet201\nfrom tensorflow.keras.layers import Input, GlobalAveragePooling2D, Dense, Concatenate, Dropout\nfrom tensorflow.keras.models import Model\n\nwith strategy.scope():\n    input_tensor = Input(shape=[*IMAGE_SIZE, 3])\n    \n    # Standard float32 initialization to ensure ImageNet compatibility\n    base_model_1 = EfficientNetB6(weights='imagenet', include_top=False, input_tensor=input_tensor)\n    base_model_1.trainable = True\n    pool_1 = GlobalAveragePooling2D()(base_model_1.output)\n    \n    base_model_2 = DenseNet201(weights='imagenet', include_top=False, input_tensor=input_tensor)\n    base_model_2.trainable = True\n    pool_2 = GlobalAveragePooling2D()(base_model_2.output)\n    \n    # Ensemble and classification layers\n    merged = Concatenate()([pool_1, pool_2])\n    dropout = Dropout(0.5)(merged)\n    output = Dense(CLASSES, activation='softmax')(dropout)\n    \n    model = Model(inputs=input_tensor, outputs=output)\n    \n    model.compile(\n        optimizer='adam',\n        loss='sparse_categorical_crossentropy',\n        metrics=['sparse_categorical_accuracy']\n    )\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:49:10.649618Z","iopub.execute_input":"2026-03-18T19:49:10.649769Z","iopub.status.idle":"2026-03-18T19:50:58.307409Z","shell.execute_reply.started":"2026-03-18T19:49:10.649754Z","shell.execute_reply":"2026-03-18T19:50:58.306074Z"}},"outputs":[],"execution_count":null},{"id":"3ac40a29-df8c-4e5f-a130-88542d7ed6d5","cell_type":"markdown","source":"## 9. Model Convergence and Evaluation\n\nBecause the mapping generates an infinite evaluation target during the data loading stage, boundaries for step processing are required. An `EarlyStopping` callback monitors `val_loss` divergence to prevent degradation.","metadata":{}},{"id":"989f75a9-ee4f-4b79-887a-35ab0430ddbe","cell_type":"code","source":"STEPS_PER_EPOCH = n_train // BATCH_SIZE\nVALIDATION_STEPS = n_val // BATCH_SIZE\n\nprint(f'Starting Distributed Training Session...')\nprint(f'Training Config: {STEPS_PER_EPOCH} steps/epoch | {EPOCHS} epochs')\nprint(f'Validation Config: {VALIDATION_STEPS} steps/epoch')\n\nearly_stopping = tf.keras.callbacks.EarlyStopping(\n    monitor='val_loss',\n    patience=3,\n    restore_best_weights=True,\n    verbose=1\n)\n\nhistory = model.fit(\n    get_training_dataset(), \n    steps_per_epoch=STEPS_PER_EPOCH, \n    epochs=EPOCHS, \n    callbacks=[lr_callback, early_stopping],\n    validation_data=get_validation_dataset(),\n    validation_steps=VALIDATION_STEPS\n)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-18T19:50:58.308077Z","iopub.execute_input":"2026-03-18T19:50:58.308249Z","execution_failed":"2026-03-18T19:57:36.451Z"}},"outputs":[],"execution_count":null},{"id":"90255e94-7372-4b3a-bfd7-7b8d272c36f6","cell_type":"markdown","source":"## 10. Inference via Test Time Augmentation\n\nTest Time Augmentation (TTA) subjects evaluation images to geometric adjustments similar to training. Computations are averaged across multiple variants of the same image to reduce prediction variance before capturing the final probabilities.","metadata":{}},{"id":"e76909a9-0e56-4a1f-af50-5d8b4a45d149","cell_type":"code","source":"def get_test_dataset_tta(ordered=True):\n    dataset = tf.data.TFRecordDataset(TEST_FILENAMES, num_parallel_reads=tf.data.experimental.AUTOTUNE)\n    dataset = dataset.map(read_unlabeled_tfrecord, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n    dataset = dataset.map(lambda image, idnum: (data_augment(image, idnum)[0], idnum))\n    dataset = dataset.batch(BATCH_SIZE)\n    dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)\n    return dataset\n\nTTA_STEPS = 3\ntest_ds = get_test_dataset(ordered=True)\n\nprobabilities = np.zeros((n_test, CLASSES))\n\nfor i in range(TTA_STEPS):\n    print(f'TTA Iteration: {i+1}')\n    tta_ds = get_test_dataset_tta(ordered=True).map(lambda image, idnum: image)\n    probabilities += model.predict(tta_ds) / 3\n\npredictions = np.argmax(probabilities, axis=-1)\n\ntest_ids_ds = test_ds.map(lambda image, idnum: idnum).unbatch()\ntest_ids = next(iter(test_ids_ds.batch(n_test))).numpy().astype('U')\n\noutput_structure = np.rec.fromarrays([test_ids, predictions])\nnp.savetxt('submission.csv', output_structure, fmt=['%s', '%d'], delimiter=',', header='id,label', comments='')\nprint('Submission file generated.')\n","metadata":{"trusted":true,"execution":{"execution_failed":"2026-03-18T19:57:36.452Z"}},"outputs":[],"execution_count":null},{"id":"425e47e7-85bb-485c-90b4-6f29b5aa31ee","cell_type":"markdown","source":"## 11. Summary\n\n1. **Distributed Computation Strategy**: Implemented TPU distribution mapping for cross-replica gradient synchronization.\n2. **Dual-Streaming Optimization**: Unified the output features of DenseNet and EfficientNet pipelines.\n3. **Augmentation Processing**: Applied stochastic permutation for image modifications.\n4. **TTA Inference Generation**: Deployed cyclic averaging to limit outlier bias during final prediction steps.","metadata":{}},{"id":"32be2fc5-0c64-4fc6-abd0-79fa63ee7ddc","cell_type":"markdown","source":"---\n\n**Citation:**\nAlexis Cook, Phil Culliton, and Ryan Holbrook. Petals to the Metal - Flower Classification on TPU.\nhttps://kaggle.com/competitions/tpu-getting-started, 2020. Kaggle.","metadata":{}}]}