{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[],"dockerImageVersionId":31089,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Cassava Leaf Disease Classification using EfficientNetB3\n\n## 📖 Introduction\n\n**Cassava** is a critical staple crop for millions of people in Africa, Asia, and Latin America, providing a vital source of calories. However, its cultivation is severely threatened by various viral diseases, which can lead to significant yield losses and jeopardize food security. Accurate and early detection of these diseases is crucial for effective management and control.\n\nThis notebook tackles the **Cassava Leaf Disease Classification** challenge. The primary goal is to develop a machine learning model capable of accurately identifying the type of disease present on a cassava plant, or classifying it as healthy, based on an image of its leaf.\n\n---\n\n### 🌿 The Diseases\n\nThe dataset focuses on the following five categories:\n1.  **Cassava Bacterial Blight (CBB)**\n2.  **Cassava Brown Streak Disease (CBSD)**\n3.  **Cassava Green Mottle (CGM)**\n4.  **Cassava Mosaic Disease (CMD)**\n5.  **Healthy**\n\n---\n\n### 🎯 Our Approach\n\nWe will employ a powerful deep learning technique called **transfer learning**. Specifically, we will fine-tune a pre-trained **EfficientNetB3** model, a state-of-the-art convolutional neural network (CNN), on the cassava leaf dataset. To accelerate the training process, we'll leverage Google's Tensor Processing Units (TPUs).\n\n**This notebook focuses on the training phase and saving the model. The inference (prediction generation) will be handled in a separate notebook.**  \n[Cassava Leaf inference with TTA](https://www.kaggle.com/code/amirmohamadaskari/cassava-leaf-inference-tta-with-loading-my-model)\n\nThe workflow in *this training notebook* includes:\n\n-   **Setting up the Environment**: Configuring the TPU strategy for distributed training.\n-   **Data Preprocessing**: Creating an efficient data pipeline using TFRecords.\n-   **Data Augmentation**: Applying various image transformations to enhance model robustness and prevent overfitting.\n-   **Model Building**: Constructing a custom classifier on top of the EfficientNetB3 base.\n-   **Two-Phase Training**:\n    1.  Training only the classifier head.\n    2.  Fine-tuning the entire model with a low learning rate.\n-   **Model Saving**: Saving the trained `final_model_cassava.keras` to this notebook's output, making it available for use in an inference notebook (or via [Cassava Leaf Model](https://www.kaggle.com/models/amirmohamadaskari/cassava-leaf-model)).","metadata":{"execution":{"iopub.status.busy":"2025-08-16T18:14:05.588740Z","iopub.execute_input":"2025-08-16T18:14:05.589077Z","iopub.status.idle":"2025-08-16T18:14:05.606909Z","shell.execute_reply.started":"2025-08-16T18:14:05.589051Z","shell.execute_reply":"2025-08-16T18:14:05.601112Z"}}},{"cell_type":"markdown","source":"## 🛠️ 1. Initial Setup and Imports\n\nLet's begin by importing the essential libraries for our project. We'll need `numpy` and `pandas` for data manipulation, `tensorflow` for building and training our deep learning model, `cv2` and `matplotlib` for image processing and visualization, and `os` and `json` for file handling.","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport random\nimport pandas as pd\nimport numpy as np\nimport tensorflow as tf\nimport cv2\nimport matplotlib.pyplot as plt\nimport json\nimport seaborn as sns\nfrom tensorflow.keras import mixed_precision\nfrom tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\nfrom tensorflow.keras.regularizers import l2","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T19:35:42.326753Z","iopub.execute_input":"2025-08-16T19:35:42.327072Z","iopub.status.idle":"2025-08-16T19:35:56.621484Z","shell.execute_reply.started":"2025-08-16T19:35:42.327051Z","shell.execute_reply":"2025-08-16T19:35:56.620717Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🌱 2. Ensuring Reproducibility\n\nTo make our experiments reproducible, it's crucial to set a global seed. This ensures that any process involving randomness—such as weight initialization, data shuffling, and augmentation—produces the same results every time the code is run. The function below sets the seed for Python's `random` module, `NumPy`, and `TensorFlow`.","metadata":{}},{"cell_type":"code","source":"def seed_everthing(SEED=28):\n    # Set the Python's random module seed\n    random.seed(SEED)\n    \n    # Set the NumPy random seed\n    np.random.seed(SEED)\n    \n    # Set the TensorFlow random seed\n    tf.random.set_seed(SEED)\n\n    print(f\"Global seed set to {SEED} 🌱\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T19:35:56.622695Z","iopub.execute_input":"2025-08-16T19:35:56.623192Z","iopub.status.idle":"2025-08-16T19:35:56.627455Z","shell.execute_reply.started":"2025-08-16T19:35:56.623172Z","shell.execute_reply":"2025-08-16T19:35:56.626626Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"seed_everthing()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T19:36:00.081332Z","iopub.execute_input":"2025-08-16T19:36:00.081796Z","iopub.status.idle":"2025-08-16T19:36:00.085928Z","shell.execute_reply.started":"2025-08-16T19:36:00.081767Z","shell.execute_reply":"2025-08-16T19:36:00.085232Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🚀 3. Hardware Accelerator Configuration\n\nTraining deep learning models can be computationally intensive. To speed up this process, we'll use a hardware accelerator like a TPU (Tensor Processing Unit) or GPU (Graphics Processing Unit). The following code detects the available hardware and sets up the appropriate TensorFlow distribution strategy. \n\n- **TPUStrategy**: For training on TPUs.\n- **MirroredStrategy**: For training on multiple GPUs on a single machine.\n- **Default Strategy**: For training on a single GPU or CPU.\n\nThe number of `REPLICAS` indicates how many parallel processing units are available, which is essential for scaling our batch size and distributing the training workload.","metadata":{}},{"cell_type":"code","source":"print(\"Available devices:\")\nfor device in tf.config.list_logical_devices():\n    print(device.name, device.device_type)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T19:36:01.069457Z","iopub.execute_input":"2025-08-16T19:36:01.069720Z","iopub.status.idle":"2025-08-16T19:36:01.831491Z","shell.execute_reply.started":"2025-08-16T19:36:01.069702Z","shell.execute_reply":"2025-08-16T19:36:01.830710Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import tensorflow as tf\n\ndef get_strategy():\n    \"\"\"\n    Detects and returns the best TensorFlow distribution strategy.\n    - TPUStrategy for TPU(s)\n    - MirroredStrategy for GPU(s)\n    - Default strategy for CPU\n    \"\"\"\n    try:\n        tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu= 'local')\n        tf.config.experimental_connect_to_cluster(tpu)\n        tf.tpu.experimental.initialize_tpu_system(tpu)\n        strategy = tf.distribute.TPUStrategy(tpu)\n        print(\"Using TPU strategy:\", type(strategy).__name__)\n    except ValueError:\n        # If TPU is not found, try GPU\n        physical_devices = tf.config.list_physical_devices('GPU')\n        if physical_devices:\n            strategy = tf.distribute.MirroredStrategy()\n            print(\"Using GPU strategy:\", type(strategy).__name__)\n        else:\n            strategy = tf.distribute.get_strategy()  # default CPU\n            print(\"No TPU/GPU found. Using default strategy:\", type(strategy).__name__)\n    except Exception as e:\n        print(\"Failed to initialize strategy:\", e)\n        strategy = tf.distribute.get_strategy()\n        print(\"Using fallback strategy:\", type(strategy).__name__)\n\n    print(\"REPLICAS:\", strategy.num_replicas_in_sync)\n    return strategy\n\n# Call this function once at the beginning of your script\nstrategy = get_strategy()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T19:36:44.579216Z","iopub.execute_input":"2025-08-16T19:36:44.579436Z","iopub.status.idle":"2025-08-16T19:36:44.586621Z","shell.execute_reply.started":"2025-08-16T19:36:44.579420Z","shell.execute_reply":"2025-08-16T19:36:44.586081Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"REPLICAS:\", strategy.num_replicas_in_sync)\nprint(\"TensorFlow version:\", tf.__version__)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T19:36:44.589572Z","iopub.execute_input":"2025-08-16T19:36:44.589739Z","iopub.status.idle":"2025-08-16T19:36:44.599182Z","shell.execute_reply.started":"2025-08-16T19:36:44.589726Z","shell.execute_reply":"2025-08-16T19:36:44.598627Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ⚡ 4. Mixed Precision Training\n\nTo further optimize our training process, we'll enable mixed precision. This technique uses a combination of 16-bit and 32-bit floating-point types during training. It can significantly speed up computations and reduce memory usage, especially on modern GPUs and TPUs, without a substantial loss in model accuracy.","metadata":{}},{"cell_type":"code","source":"# List all physical devices labeled as 'GPU'\ngpus = tf.config.list_physical_devices('GPU')\n\nif gpus:\n    print(f\"✅ GPU detected: {gpus}\")\n    mixed_precision.set_global_policy('mixed_float16')\nelse:\n    print(\"❌ No GPU found. Using CPU instead.\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T16:13:23.875992Z","iopub.execute_input":"2025-08-16T16:13:23.876271Z","iopub.status.idle":"2025-08-16T16:13:23.885848Z","shell.execute_reply.started":"2025-08-16T16:13:23.876247Z","shell.execute_reply":"2025-08-16T16:13:23.881394Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 📂 5. Data Loading and Exploration\n\nNow, let's define the paths to our data files. We need the directory containing the training images, the CSV file with image labels, and the JSON file that maps numerical labels to their corresponding disease names.","metadata":{}},{"cell_type":"code","source":"DATA_DIR = '/kaggle/input/cassava-leaf-disease-classification'\nTRAIN_DIR = os.path.join(DATA_DIR, 'train_images')\nCSV_PATH = os.path.join(DATA_DIR, 'train.csv')\nLABEL_PATH = os.path.join(DATA_DIR, 'label_num_to_disease_map.json')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:08.512011Z","iopub.execute_input":"2025-08-16T17:33:08.512342Z","iopub.status.idle":"2025-08-16T17:33:08.521018Z","shell.execute_reply.started":"2025-08-16T17:33:08.512317Z","shell.execute_reply":"2025-08-16T17:33:08.517828Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Let's have a look at some of the image file names\nprint(os.listdir(TRAIN_DIR)[:5])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T11:24:52.770704Z","iopub.execute_input":"2025-08-16T11:24:52.770970Z","iopub.status.idle":"2025-08-16T11:24:53.022889Z","shell.execute_reply.started":"2025-08-16T11:24:52.770948Z","shell.execute_reply":"2025-08-16T11:24:53.022230Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Read one image to check its dimensions\nimg_files = os.listdir(TRAIN_DIR)\nimg_path = os.path.join(TRAIN_DIR, img_files[0])\nimg = cv2.imread(img_path)\nprint(img.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T11:24:54.783668Z","iopub.execute_input":"2025-08-16T11:24:54.783913Z","iopub.status.idle":"2025-08-16T11:24:54.826143Z","shell.execute_reply.started":"2025-08-16T11:24:54.783896Z","shell.execute_reply":"2025-08-16T11:24:54.825530Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Load the label-to-disease name mapping from the JSON file\nwith open(LABEL_PATH, 'r') as f:\n    label_map = json.load(f)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T11:24:56.508861Z","iopub.execute_input":"2025-08-16T11:24:56.509152Z","iopub.status.idle":"2025-08-16T11:24:56.517502Z","shell.execute_reply.started":"2025-08-16T11:24:56.509131Z","shell.execute_reply":"2025-08-16T11:24:56.516814Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Loading the Training DataFrame\nWe load the `train.csv` file into a pandas DataFrame. This file contains the `image_id` and its corresponding `label`. We'll convert the labels to string type for easier handling.","metadata":{}},{"cell_type":"code","source":"df = pd.read_csv(CSV_PATH)\n# Convert the 'label' column to string type for consistency\ndf['label'] = df['label'].astype(str)\nprint(df.tail())\nprint(df.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T11:25:05.182901Z","iopub.execute_input":"2025-08-16T11:25:05.183140Z","iopub.status.idle":"2025-08-16T11:25:05.224919Z","shell.execute_reply.started":"2025-08-16T11:25:05.183122Z","shell.execute_reply":"2025-08-16T11:25:05.224238Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Analyzing Class Distribution\nUnderstanding the distribution of classes is a vital step in any classification problem. It helps us identify if there is a class imbalance, which can affect the model's performance. A significant imbalance might require special techniques like class weighting or over/under-sampling.","metadata":{}},{"cell_type":"code","source":"# Get the count of images for each class\ndf['label'].value_counts()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T11:25:09.004102Z","iopub.execute_input":"2025-08-16T11:25:09.004387Z","iopub.status.idle":"2025-08-16T11:25:09.016481Z","shell.execute_reply.started":"2025-08-16T11:25:09.004363Z","shell.execute_reply":"2025-08-16T11:25:09.015777Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualize the class distribution using a bar chart\nplt.figure(figsize=(25,8))\nax=sns.countplot(x=df[\"label\"],palette=\"viridis\",order=df['label'].value_counts().index)\n# Add labels on top of the bars for clarity\nfor p in ax.containers:\n    ax.bar_label(p, fontsize=20, color='black', padding=5);","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T11:25:09.567332Z","iopub.execute_input":"2025-08-16T11:25:09.567634Z","iopub.status.idle":"2025-08-16T11:25:09.846869Z","shell.execute_reply.started":"2025-08-16T11:25:09.567612Z","shell.execute_reply":"2025-08-16T11:25:09.846078Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Visualizing Sample Images\nTo get a better feel for the dataset, let's visualize one sample image from each of the five categories. This helps us understand the visual characteristics of healthy leaves and leaves affected by different diseases.","metadata":{}},{"cell_type":"code","source":"def show_images_from_each_category(df, label_map, rows= 1, images_per_class= 1):\n    classes = df['label'].unique()\n    num_classes = len(classes)\n    columns = images_per_class\n    \n    plt.figure(figsize= (10* rows,10* columns))\n    \n    for i, c in enumerate(classes):\n        # Get random samples for the current class\n        class_samples = df[df['label'] == c].sample(images_per_class)\n        for j ,(_, row) in enumerate(class_samples.iterrows()):\n            # Construct the full image path\n            img_path = os.path.join(TRAIN_DIR, row['image_id'])\n            img = cv2.imread(img_path)\n            # Convert image from BGR (OpenCV default) to RGB for correct display with Matplotlib\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n\n            # Get the disease name from the label map\n            label_name = label_map.get(row['label'], row['label'])\n\n            # Create a subplot for each image\n            plt.subplot(num_classes, columns, i* columns + j +1)\n            plt.imshow(img)\n            plt.title(f'{label_name}')\n            plt.axis('off')\n\n    plt.tight_layout()\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T11:25:12.235203Z","iopub.execute_input":"2025-08-16T11:25:12.235770Z","iopub.status.idle":"2025-08-16T11:25:12.241249Z","shell.execute_reply.started":"2025-08-16T11:25:12.235745Z","shell.execute_reply":"2025-08-16T11:25:12.240468Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"show_images_from_each_category(df, label_map, rows= 1, images_per_class= 1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T11:25:12.754981Z","iopub.execute_input":"2025-08-16T11:25:12.755260Z","iopub.status.idle":"2025-08-16T11:25:13.406912Z","shell.execute_reply.started":"2025-08-16T11:25:12.755238Z","shell.execute_reply":"2025-08-16T11:25:13.406237Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ⚙️ 6. Data Pipeline with TFRecords\n\nFor efficient data handling, especially with large datasets and TPUs, we will use the **TFRecord** format. TFRecord is a binary file format that stores a sequence of protocol buffer messages, which is highly optimized for reading data in TensorFlow.\n\nFirst, we'll locate the directory containing our pre-processed TFRecord files.","metadata":{}},{"cell_type":"code","source":"tfrecord_dir = '/kaggle/input/cassava-leaf-disease-classification/train_tfrecords'\nprint(os.listdir(tfrecord_dir))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:12.489886Z","iopub.execute_input":"2025-08-16T17:33:12.490246Z","iopub.status.idle":"2025-08-16T17:33:12.508203Z","shell.execute_reply.started":"2025-08-16T17:33:12.490218Z","shell.execute_reply":"2025-08-16T17:33:12.503416Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Configuration Parameters\nHere, we define several key parameters for our data pipeline and model. \n\n- `AUTO`: An instruction for TensorFlow to automatically tune the level of parallelism for data pipeline operations.\n- `IMAGE_SIZE`: The dimensions to which all images will be resized. EfficientNetB3 performs well with larger image sizes.\n- `NUM_CLASSES`: The number of distinct categories in our classification task.\n- `BATCH_SIZE`: The total number of samples processed in one forward/backward pass. We calculate the **global batch size** by multiplying the per-replica batch size by the number of available replicas (TPU cores or GPUs). This ensures that the model sees the same total number of images per step, regardless of the distribution strategy.","metadata":{}},{"cell_type":"code","source":"AUTO = tf.data.AUTOTUNE\nIMAGE_SIZE = (512, 512)\nBATCH_SIZE_PER_REPLICA = 8\nNUM_CLASSES = 5\nBATCH_SIZE = BATCH_SIZE_PER_REPLICA * strategy.num_replicas_in_sync\nprint(f'Global Batch size: {BATCH_SIZE}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:12.787893Z","iopub.execute_input":"2025-08-16T17:33:12.788171Z","iopub.status.idle":"2025-08-16T17:33:12.798570Z","shell.execute_reply.started":"2025-08-16T17:33:12.788147Z","shell.execute_reply":"2025-08-16T17:33:12.793348Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Decoding TFRecord Examples\nThis function is responsible for parsing a single example from a TFRecord file. It reads the raw byte strings for the image and its target label, decodes the JPEG image, resizes it to our standard `IMAGE_SIZE`, and finally, one-hot encodes the label. One-hot encoding converts the integer label (e.g., 3) into a binary vector (e.g., `[0, 0, 0, 1, 0]`), which is the required format for categorical cross-entropy loss.","metadata":{}},{"cell_type":"code","source":"def decode_example(example):\n    # Define the structure of the features in the TFRecord file\n    feature_description = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'target': tf.io.FixedLenFeature([], tf.int64),\n    }\n    # Parse the input `example` protocol buffer using the feature description\n    example = tf.io.parse_single_example(example, feature_description)\n    \n    # Decode the JPEG-encoded image string to a tensor\n    image = tf.image.decode_jpeg(example['image'], channels=3)\n    # Resize the image to the desired dimensions\n    image = tf.image.resize(image, IMAGE_SIZE)\n    \n    # Get the integer label\n    label_int = example['target']\n    \n    # One-hot encode the label for the model\n    label = tf.one_hot(label_int, NUM_CLASSES)\n    \n    return image, label","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:14.039123Z","iopub.execute_input":"2025-08-16T17:33:14.039381Z","iopub.status.idle":"2025-08-16T17:33:14.051398Z","shell.execute_reply.started":"2025-08-16T17:33:14.039357Z","shell.execute_reply":"2025-08-16T17:33:14.046085Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Preprocessing Input for EfficientNet\nPre-trained models like EfficientNet expect their input data to be preprocessed in a specific way (e.g., pixel values scaled to a certain range). The `preprocess_input` function from `tf.keras.applications.efficientnet` handles this for us, ensuring our images are in the correct format for the model.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications.efficientnet import preprocess_input\n\ndef preprocess(image, label):\n    # Apply the specific preprocessing required by the EfficientNet model\n    image = preprocess_input(image)\n    return image, label","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:14.362486Z","iopub.execute_input":"2025-08-16T17:33:14.362723Z","iopub.status.idle":"2025-08-16T17:33:14.377701Z","shell.execute_reply.started":"2025-08-16T17:33:14.362701Z","shell.execute_reply":"2025-08-16T17:33:14.373535Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🎨 Data Augmentation\nData augmentation is a powerful technique to increase the diversity of the training data without collecting new samples. By applying random transformations like rotations, flips, and zooms to the training images, we can make our model more robust and less prone to overfitting. We define a `Sequential` model that will apply these augmentations on the fly during training.","metadata":{}},{"cell_type":"code","source":"data_augmentation = tf.keras.Sequential([\n    # Geometric Transformations\n    tf.keras.layers.RandomRotation(40/ 360), # Randomly rotate images\n    tf.keras.layers.RandomTranslation(0.2, 0.2), # Randomly shift images horizontally and vertically\n    tf.keras.layers.RandomZoom(0.2, 0.2), # Randomly zoom into images\n    tf.keras.layers.RandomFlip('horizontal'), # Randomly flip images horizontally\n    tf.keras.layers.RandomFlip('vertical') # Randomly flip images vertically\n], name=\"data_augmentation\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:15.420079Z","iopub.execute_input":"2025-08-16T17:33:15.420437Z","iopub.status.idle":"2025-08-16T17:33:15.448730Z","shell.execute_reply.started":"2025-08-16T17:33:15.420408Z","shell.execute_reply":"2025-08-16T17:33:15.443436Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Define the augmentation function that will be mapped to the dataset\ndef apply_augmentation(image, label):\n    # Keras augmentation layers expect a batch of images.\n    # We add a batch dimension, apply the augmentation, and then remove it.\n    image = tf.expand_dims(image, 0) # Add batch dimension\n    image = data_augmentation(image)\n    image = tf.squeeze(image, 0)      # Remove batch dimension\n    return image, label","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:15.733734Z","iopub.execute_input":"2025-08-16T17:33:15.734059Z","iopub.status.idle":"2025-08-16T17:33:15.744800Z","shell.execute_reply.started":"2025-08-16T17:33:15.734032Z","shell.execute_reply":"2025-08-16T17:33:15.740385Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Assembling the Data Pipeline\nThis function brings everything together to create our final `tf.data.Dataset` object. It reads the TFRecord files, decodes and preprocesses the data, and applies augmentation (only to the training set). Key steps include:\n\n- `.with_options(ignore_order)`: Disables deterministic order to improve performance.\n- `.map()`: Applies our decoding, preprocessing, and augmentation functions in parallel.\n- `.shuffle()`: Shuffles the training data to ensure the model doesn't learn from the order of examples.\n- `.batch()`: Groups the data into batches.\n- `.prefetch()`: Prepares subsequent batches while the current one is being processed, which helps to prevent data pipeline bottlenecks.","metadata":{}},{"cell_type":"code","source":"def load_dataset(filenames, is_training=True):\n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = False  # for performance\n    # Create a dataset from the TFRecord files\n    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads= AUTO)\n    # Disable deterministic order for better performance\n    dataset = dataset.with_options(ignore_order)\n    # Decode each example in the dataset\n    dataset = dataset.map(decode_example, num_parallel_calls= AUTO)\n    # Preprocess the images for the model\n    dataset = dataset.map(preprocess, num_parallel_calls= AUTO)\n    # Apply augmentations and shuffling only to the training set\n    if is_training:\n        dataset = dataset.shuffle(8196)\n        dataset = dataset.map(apply_augmentation, num_parallel_calls= AUTO)\n    # Batch the dataset and prefetch for performance\n    dataset = dataset.batch(BATCH_SIZE, drop_remainder= True).prefetch(AUTO)\n    return dataset","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:16.018306Z","iopub.execute_input":"2025-08-16T17:33:16.018706Z","iopub.status.idle":"2025-08-16T17:33:16.032058Z","shell.execute_reply.started":"2025-08-16T17:33:16.018670Z","shell.execute_reply":"2025-08-16T17:33:16.026030Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Creating Training and Validation Datasets\nWe'll split our TFRecord files into a training set and a validation set. A common split is 80% for training and 20% for validation. The validation set is crucial for monitoring the model's performance on unseen data during training and for tuning hyperparameters.","metadata":{}},{"cell_type":"code","source":"# Get a list of all TFRecord files\nall_files = sorted(tf.io.gfile.glob(os.path.join(tfrecord_dir, '*.tfrec')))\n\n# Split the files into training and validation sets (e.g., 80% train, 20% val)\nsplit_index = int(0.8 * len(all_files)) + 1\ntrain_files = all_files[:split_index]\nval_files = all_files[split_index:]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:16.949811Z","iopub.execute_input":"2025-08-16T17:33:16.950172Z","iopub.status.idle":"2025-08-16T17:33:16.965951Z","shell.execute_reply.started":"2025-08-16T17:33:16.950141Z","shell.execute_reply":"2025-08-16T17:33:16.961302Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create the dataset objects using our pipeline function\ntrain_dataset = load_dataset(train_files, is_training=True)\nval_dataset = load_dataset(val_files, is_training=False)\nprint(f'Train and validation set created successfully!')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:17.238478Z","iopub.execute_input":"2025-08-16T17:33:17.238729Z","iopub.status.idle":"2025-08-16T17:33:17.607196Z","shell.execute_reply.started":"2025-08-16T17:33:17.238706Z","shell.execute_reply":"2025-08-16T17:33:17.603045Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Calculating Steps Per Epoch\nTo properly train our model, we need to know how many batches of data constitute one full epoch. We calculate `steps_per_epoch` for the training set and `validation_steps` for the validation set. This is done by counting the total number of images in each set and dividing by the global batch size.","metadata":{}},{"cell_type":"code","source":"# Helper function to count the total number of examples in a set of TFRecord files\ndef count_total_examples(tfrecord_files):\n    count = 0\n    for fname in tfrecord_files:\n        count += sum(1 for _ in tf.data.TFRecordDataset(fname))\n    return count","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:18.253186Z","iopub.execute_input":"2025-08-16T17:33:18.253523Z","iopub.status.idle":"2025-08-16T17:33:18.264794Z","shell.execute_reply.started":"2025-08-16T17:33:18.253496Z","shell.execute_reply":"2025-08-16T17:33:18.258978Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Count the number of images in the training and validation sets\nnum_train_images = count_total_examples(train_files)\nnum_val_images = count_total_examples(val_files)\n\n# Calculate the number of steps (batches) per epoch\nsteps_per_epoch = num_train_images // BATCH_SIZE\nvalidation_steps = num_val_images // BATCH_SIZE\n\nprint(f'Batch steps on training: {steps_per_epoch}\\nSteps on validation: {validation_steps}')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:19.179388Z","iopub.execute_input":"2025-08-16T17:33:19.179730Z","iopub.status.idle":"2025-08-16T17:33:50.882082Z","shell.execute_reply.started":"2025-08-16T17:33:19.179700Z","shell.execute_reply":"2025-08-16T17:33:50.877304Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Verifying the Data Pipeline\nLet's inspect a single batch from our training dataset to ensure that the images and labels have the correct shapes and data types. This is a good sanity check before starting the training process.","metadata":{}},{"cell_type":"code","source":"# Take one batch from the training dataset\nfor images, labels in train_dataset.take(1):\n    # Print the shape of the image and label tensors\n    print(\"Image batch shape:\", images.shape)\n    print(\"Label batch shape:\", labels.shape)\n    # Print an example one-hot encoded label\n    print(\"Label example (one-hot):\", labels[0].numpy())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-15T21:12:12.656230Z","iopub.execute_input":"2025-08-15T21:12:12.656500Z","iopub.status.idle":"2025-08-15T21:12:23.030536Z","shell.execute_reply.started":"2025-08-15T21:12:12.656481Z","shell.execute_reply":"2025-08-15T21:12:23.029960Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🧠 7. Model Building with Transfer Learning\n\nWe will now construct our model using **transfer learning**. We'll use **EfficientNetB3**, a powerful and efficient model pre-trained on the large ImageNet dataset. The idea is to leverage the features learned by this model (like edges, textures, and shapes) and adapt them to our specific task of classifying cassava leaf diseases.\n\nOur model architecture will consist of:\n1.  **The Base Model**: The pre-trained `EfficientNetB3` with its top classification layer removed (`include_top=False`). We'll initially freeze its weights so they don't change during the first phase of training.\n2.  **A Custom Classifier Head**: We will add our own layers on top of the base model:\n    - `GlobalAveragePooling2D`: To flatten the feature maps from the base model.\n    - `Dense` layer with `relu` activation: A hidden layer to learn more complex patterns.\n    - `Dropout`: A regularization technique to prevent overfitting.\n    - `Dense` output layer with `softmax` activation: To produce a probability distribution over the 5 classes.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.regularizers import l2\ndef cassava_model(IMAGE_SIZE):\n    \n    # Load the EfficientNetB3 model, pre-trained on ImageNet\n    base_model = tf.keras.applications.EfficientNetB3(\n        weights= 'imagenet',\n        include_top= False, # Do not include the final ImageNet classifier layer\n        input_shape= IMAGE_SIZE + (3,)\n    )\n    \n    # Freeze the weights of the base model. We will only train the new classifier head initially.\n    base_model.trainable = False\n        \n    # Define the model input\n    inputs = tf.keras.layers.Input(shape= IMAGE_SIZE + (3, ))\n    # Pass the inputs through the base model\n    eff = base_model(inputs)\n    # Add our custom classifier head\n    avg = tf.keras.layers.GlobalAveragePooling2D()(eff)\n    fc = tf.keras.layers.Dense(256, activation= 'relu', kernel_regularizer= l2(0.001))(avg)\n    dropout = tf.keras.layers.Dropout(0.3)(fc)\n    outputs = tf.keras.layers.Dense(NUM_CLASSES, activation= 'softmax')(dropout)\n    \n    # Create the final model\n    model = tf.keras.Model(inputs= inputs, outputs= outputs)\n    # Print a summary of the model architecture\n    model.summary()\n\n    return model","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:33:50.884465Z","iopub.execute_input":"2025-08-16T17:33:50.885573Z","iopub.status.idle":"2025-08-16T17:33:50.903741Z","shell.execute_reply.started":"2025-08-16T17:33:50.885532Z","shell.execute_reply":"2025-08-16T17:33:50.897973Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🏋️ 8. Training Phase 1: Training the Head\n\nIn the first phase of training, we only train the weights of the custom classifier head we added. The weights of the EfficientNetB3 base model remain frozen. This allows the new layers to learn to interpret the features extracted by the base model for our specific dataset without disrupting the valuable pre-trained knowledge.\n\n### Callbacks\nWe will use several callbacks to manage the training process:\n- `ModelCheckpoint`: Saves the model with the best validation loss.\n- `EarlyStopping`: Stops training if the validation loss doesn't improve for a set number of epochs, preventing overfitting.\n- `ReduceLROnPlateau`: Reduces the learning rate if the training plateaus, helping the model to find a better minimum.","metadata":{}},{"cell_type":"code","source":"# Save the best model based on validation loss\ncheckpoint_cb = ModelCheckpoint(\n    '/kaggle/working/initial_model_cassava.keras', # File to save the best model\n    monitor='val_loss',\n    save_best_only=True,\n    mode='min' # We want to minimize loss\n)\n\n# Stop training if validation loss doesn't improve for 3 epochs\nearly_stopping_cb = EarlyStopping(\n    monitor='val_loss',\n    patience=3,\n    restore_best_weights=True # This is great, it restores the weights from the best epoch\n)\n\n# Reduce learning rate when learning plateaus\nreduce_lr_cb = ReduceLROnPlateau(\n    monitor='val_loss',\n    factor=0.2,\n    patience=2,\n    min_lr=1e-6\n)\n\ncallbacks = [checkpoint_cb, early_stopping_cb, reduce_lr_cb]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:34:42.241126Z","iopub.execute_input":"2025-08-16T17:34:42.241346Z","iopub.status.idle":"2025-08-16T17:34:42.257169Z","shell.execute_reply.started":"2025-08-16T17:34:42.241324Z","shell.execute_reply":"2025-08-16T17:34:42.252735Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Compiling and Fitting the Model\nWe compile the model within the `strategy.scope()` to ensure it's distributed across the available TPUs/GPUs. We use the `Adam` optimizer, `CategoricalCrossentropy` loss with **label smoothing** (a regularization technique that prevents the model from becoming overconfident), and track `accuracy` as our performance metric. Then, we start the training process.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    # Build the model\n    model = cassava_model(IMAGE_SIZE)\n    # Compile the model with optimizer, loss, and metrics\n    model.compile(optimizer = tf.keras.optimizers.Adam(3e-4), \n                  loss= tf.keras.losses.CategoricalCrossentropy(label_smoothing= 0.01), \n                  metrics= ['accuracy'],\n                  steps_per_execution= 32 if isinstance(strategy, tf.distribute.TPUStrategy) else 1)\n\n# Set the number of epochs for this training phase\ninitial_epoch = 15\nprint(\"--- Starting Phase 1: Training Head Classifier ---\")\n# Fit the model to the training data\nhistory = model.fit(train_dataset.repeat(), \n                    validation_data= val_dataset.repeat(),\n                    epochs= initial_epoch,\n                    callbacks= callbacks,\n                    steps_per_epoch= steps_per_epoch,\n                    validation_steps= validation_steps)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:34:44.056207Z","iopub.execute_input":"2025-08-16T17:34:44.056501Z","iopub.status.idle":"2025-08-16T17:49:31.930597Z","shell.execute_reply.started":"2025-08-16T17:34:44.056478Z","shell.execute_reply":"2025-08-16T17:49:31.925281Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Check the history object from your training run\nval_loss_history = history.history['val_loss']\nbest_val_loss = min(val_loss_history)\nbest_epoch = val_loss_history.index(best_val_loss) + 1\n\nprint(f\"Lowest Validation Loss: {best_val_loss:.4f} at Epoch {best_epoch}\")\nprint(f\"Final Validation Loss: {val_loss_history[-1]:.4f} at Epoch {len(val_loss_history)}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:50:05.998588Z","iopub.execute_input":"2025-08-16T17:50:05.998964Z","iopub.status.idle":"2025-08-16T17:50:06.010103Z","shell.execute_reply.started":"2025-08-16T17:50:05.998933Z","shell.execute_reply":"2025-08-16T17:50:06.005643Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 🎨 9. Training Phase 2: Fine-Tuning\n\nAfter the classifier head has been trained and has converged, we can move to the fine-tuning phase. Here, we **unfreeze** the entire base model (or some of its top layers) and continue training with a **very low learning rate**. \n\nThis allows the model to make small adjustments to the pre-trained weights, adapting them more closely to the specifics of the cassava leaf dataset. Using a low learning rate is critical to avoid destroying the valuable features learned during pre-training.","metadata":{}},{"cell_type":"code","source":"# Define a new set of callbacks for the fine-tuning phase\nfn_checkpoint_cb = ModelCheckpoint(\n    '/kaggle/working/final_model_cassava.keras', # File to save the best model\n    monitor='val_loss',\n    save_best_only=True,\n    mode='min' # We want to minimize loss\n)\n\n# Stop training if validation loss doesn't improve for 4 epochs\nearly_stopping_cb = EarlyStopping(\n    monitor='val_loss',\n    patience=4,\n    restore_best_weights=True # This is great, it restores the weights from the best epoch\n)\n\n# Reduce learning rate when learning plateaus\nreduce_lr_cb = ReduceLROnPlateau(\n    monitor='val_loss',\n    factor=0.2,\n    patience=2,\n    min_lr=1e-7\n)\n\ncallbacks = [fn_checkpoint_cb, early_stopping_cb, reduce_lr_cb]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-16T17:50:07.690778Z","iopub.execute_input":"2025-08-16T17:50:07.691111Z","iopub.status.idle":"2025-08-16T17:50:07.704781Z","shell.execute_reply.started":"2025-08-16T17:50:07.691085Z","shell.execute_reply":"2025-08-16T17:50:07.699102Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"with strategy.scope():\n\n    # Unfreeze the base model to make it trainable\n    base_model = model.get_layer('efficientnetb3')\n    base_model.trainable = True\n\n    # Re-compile the model with a much lower learning rate for fine-tuning\n    model.compile(\n        optimizer=tf.keras.optimizers.Adam(3e-5),\n        loss=tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.01),\n        metrics=['accuracy'],\n        steps_per_execution= 32 if isinstance(strategy, tf.distribute.TPUStrategy) else 1\n    )\n\nprint(\"--- Starting Phase 2: Fine tune some last layers ---\")\nfinal_epochs = initial_epoch + 50\n# Continue training the model\nhistory_final = model.fit(\n    train_dataset.repeat(),\n    validation_data= val_dataset.repeat(),\n    epochs= final_epochs,\n    initial_epoch= initial_epoch, # Start from the epoch number where the first phase left off\n    callbacks= callbacks,\n    steps_per_epoch= steps_per_epoch,\n    validation_steps= validation_steps\n)","metadata":{"trusted":true,"execution":{"iopub.status.idle":"2025-08-16T18:11:50.057828Z","shell.execute_reply.started":"2025-08-16T17:50:25.707736Z","shell.execute_reply":"2025-08-16T18:11:50.054506Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✨ End of Training Notebook: What's Next?\n\nCongratulations! You've successfully completed the training phase of your model. This notebook's primary purpose was to **train your deep learning model (e.g., EfficientNet)** using the provided dataset and accelerators like TPUs or GPUs.\n\n---\n\n### Your Accomplishments in This Notebook:\n\n* **Model Training:** You've trained `final_model_cassava.keras`, learning patterns and features from the large training dataset.\n* **Model Saving:** Crucially, the trained model has been **saved to this notebook's output directory** (`/kaggle/working/`). When you \"Save Version\" of this notebook (especially with \"Save & Run All`), this saved model becomes a persistent asset.\n\n---\n\n### Moving to Inference: The Next Step\n\nFor Kaggle competitions, particularly those with hidden test sets or long inference times, it's best practice to separate training from prediction. Your trained model is now ready for the **Inference Notebook**.\n\n**Here's the planned workflow for making your final submission:**\n\n1.  **Create a New Inference Notebook:** Start a fresh Kaggle notebook. This notebook will be solely for making predictions.\n2.  **Add This Notebook's Output as Input:** In the new Inference Notebook, go to the \"Add Data\" section. You'll find the output of *this* training notebook (which includes `final_model_cassava.keras`) and add it as an input. This makes your trained model available for use.\n    * Alternatively, you can also use the model directly from Kaggle Models: [Cassava Leaf Model](https://www.kaggle.com/models/amirmohamadaskari/cassava-leaf-model).\n3.  **Load the Model:** In the Inference Notebook, you'll load `final_model_cassava.keras` from its path within `/kaggle/input/` (e.g., `/kaggle/input/your-training-notebook-output-name/final_model_cassava.keras`).\n4.  **Perform Inference & Create Submission File:** The Inference Notebook will then load the competition's test data, use your loaded model to make predictions, and generate the `submission.csv` file in the correct format.\n5.  **Submit!** Once the Inference Notebook successfully runs and creates `submission.csv`, you can submit it to the competition leaderboard.\n\nThis two-notebook approach keeps your training and inference workflows clean, efficient, and compliant with Kaggle's submission system. Good luck with your submission! 🚀\n\nInference and Test time Augmentation will make on this notebook:  \n[Cassava Leaf inference with TTA](https://www.kaggle.com/code/amirmohamadaskari/cassava-leaf-inference-tta-with-loading-my-model)  \n**Note**: For simplicity and also did'nt have access to ouput versions of my notebook as input to inference notebook, I download my model as just uploaded again on models. so in inference notebook, we loaded our model that I already uploaded in models and don't add output of my notebook as input, but this is a standard way to avoid wasting time through training with CPU or GPU in some competitions like Cassavs that using TPU was banned:)","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}