{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":11848,"databundleVersionId":862157,"sourceType":"competition"}],"dockerImageVersionId":30920,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Cancer Detection with Keras\n\nThis notebook explores the use of Convolutional Neural Networks (CNNs) to identify malignant cancers in histopathologic scans of lymph node sections. The data is provide by Kaggle as a slightly modified version of the PatchCamelyon (PCam) benchmark dataset.\n\nThe dataset consists of a pre-split training and testing set, each with a CSV containing file ids and their labels (0: benign, 1: malignant) and a directory of 96 x 96 pixel RGB TIFF images.\n\nThe task is to predict the probability that the center 32x32 pixel region contains at least one pixel of tumor tissue. ","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import data as tf_data\nimport pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nimport keras\nfrom keras import layers\nimport keras_tuner as kt\nimport os\nimport cv2\nfrom tqdm import tqdm\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_curve, auc, roc_auc_score\nfrom scipy.special import expit\nimport glob\nimport tifffile as tiff\n\nsns.set(style='whitegrid')\nprint(\"Tensorflow version \" + tf.__version__)\n\nimport warnings\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:09.029160Z","iopub.execute_input":"2025-04-09T21:58:09.029409Z","iopub.status.idle":"2025-04-09T21:58:23.902494Z","shell.execute_reply.started":"2025-04-09T21:58:09.029386Z","shell.execute_reply":"2025-04-09T21:58:23.901580Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"test_dir = '/kaggle/input/histopathologic-cancer-detection/test/'\ntrain_dir = '/kaggle/input/histopathologic-cancer-detection/train/'\ntrain_labels_filepath = '/kaggle/input/histopathologic-cancer-detection/train_labels.csv'\ntest_labels_filepath = '/kaggle/input/histopathologic-cancer-detection/sample_submission.csv'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:23.903433Z","iopub.execute_input":"2025-04-09T21:58:23.904061Z","iopub.status.idle":"2025-04-09T21:58:23.907634Z","shell.execute_reply.started":"2025-04-09T21:58:23.904029Z","shell.execute_reply":"2025-04-09T21:58:23.906821Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train_labels = pd.read_csv(train_labels_filepath)\ntest_labels = pd.read_csv(test_labels_filepath)\ntrain_labels['path'] = train_dir + \"/\" + train_labels['id'].astype(str) + \".tif\"\ntest_labels['path'] = test_dir + \"/\" + test_labels['id'].astype(str) + \".tif\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:23.908549Z","iopub.execute_input":"2025-04-09T21:58:23.908877Z","iopub.status.idle":"2025-04-09T21:58:24.524394Z","shell.execute_reply.started":"2025-04-09T21:58:23.908844Z","shell.execute_reply":"2025-04-09T21:58:24.523732Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Exploratory Data Analysis\nHere I explore the training and test datasets to understand \n- the data structure\n- the relative number of images for each label class\n- visualize examples from each class","metadata":{}},{"cell_type":"code","source":"train_labels.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:24.525122Z","iopub.execute_input":"2025-04-09T21:58:24.525422Z","iopub.status.idle":"2025-04-09T21:58:24.545784Z","shell.execute_reply.started":"2025-04-09T21:58:24.525395Z","shell.execute_reply":"2025-04-09T21:58:24.545009Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Compare training labels and number of images\nnum_labels_train = train_labels.shape[0]\nnum_images_train = len(os.listdir(train_dir))\nprint(f'Training data contains {num_images_train} with {num_labels_train} labels.')\n\n# Compare test labels and number of images\nnum_labels_test = test_labels.shape[0]\nnum_images_test = len(os.listdir(test_dir))\nprint(f'Training data contains {num_images_test} with {num_labels_test} labels.')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:24.546604Z","iopub.execute_input":"2025-04-09T21:58:24.546935Z","iopub.status.idle":"2025-04-09T21:58:48.387030Z","shell.execute_reply.started":"2025-04-09T21:58:24.546905Z","shell.execute_reply":"2025-04-09T21:58:48.386137Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Count labels in each class\nlabel_counts = train_labels['label'].value_counts(normalize=True).reset_index()\nlabel_counts.columns = ['label', 'proportion']\n\n# Visualize class distribution\nsns.barplot(data=label_counts, x='label', y='proportion')\n\nplt.title('Relative Distribution of Labels')\nplt.xlabel('Label')\nplt.ylabel('Proportion')\nplt.ylim(0, 1)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:48.389213Z","iopub.execute_input":"2025-04-09T21:58:48.389449Z","iopub.status.idle":"2025-04-09T21:58:48.679712Z","shell.execute_reply.started":"2025-04-09T21:58:48.389431Z","shell.execute_reply":"2025-04-09T21:58:48.678691Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Visualize benign and malignant samples\nbenign_samples = train_labels[train_labels[\"label\"] == 0].sample(3)[\"id\"].values\nmalignant_samples = train_labels[train_labels[\"label\"] == 1].sample(3)[\"id\"].values\n\nfig, axes = plt.subplots(2, 3, figsize=(10,6))\n\ndef plot_sample(sample_id, sample_dir, ax, title):\n    img_path = os.path.join(sample_dir, sample_id + \".tif\")\n    img = Image.open(img_path)\n    ax.imshow(img)\n    ax.axis(\"off\")\n    ax.set_title(title)\n\nfor i, img_id in enumerate(benign_samples):\n    ax = axes[0, i]\n    plot_sample(img_id, train_dir, ax, 'Benign')\n\nfor i, img_id in enumerate(malignant_samples):\n    ax = axes[1, i]\n    plot_sample(img_id, train_dir, ax, 'Malignant')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:48.681191Z","iopub.execute_input":"2025-04-09T21:58:48.681434Z","iopub.status.idle":"2025-04-09T21:58:49.893300Z","shell.execute_reply.started":"2025-04-09T21:58:48.681413Z","shell.execute_reply":"2025-04-09T21:58:49.892036Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"benign_samples = train_labels[train_labels[\"label\"] == 0].sample(100)[\"id\"].values\nmalignant_samples = train_labels[train_labels[\"label\"] == 1].sample(100)[\"id\"].values\nnum_bins = 32\n\nhistograms = []\n\ndef extract_color_histograms(samples, num_bins, label_name):\n    # Initialize histograms for each color channel\n    hist_r = np.zeros(num_bins)\n    hist_g = np.zeros(num_bins)\n    hist_b = np.zeros(num_bins)\n    \n    # Extract and combine data\n    for img_id in samples:\n        img = cv2.imread(os.path.join(train_dir, img_id + \".tif\"))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        hist_r += cv2.calcHist([img], [0], None, [num_bins], [0, 256]).flatten()\n        hist_g += cv2.calcHist([img], [1], None, [num_bins], [0, 256]).flatten()\n        hist_b += cv2.calcHist([img], [2], None, [num_bins], [0, 256]).flatten()\n        \n    # Normalize histograms\n    hist_r /= hist_r.sum()\n    hist_g /= hist_g.sum()\n    hist_b /= hist_b.sum()\n\n    histogram_dict = {'red': hist_r, 'green': hist_g, 'blue': hist_b}\n    histograms.append({'label': label_name, 'histograms': histogram_dict})\n\n# Process samples\nextract_color_histograms(benign_samples, num_bins, 'Benign')\nextract_color_histograms(malignant_samples, num_bins, 'Malignant')\n\n# Plot\nfig, axes = plt.subplots(2, 3, figsize=(10,6), sharey=True)\n\ndef plot_color_histogram(histogram_values, label, color, ax):\n    ax.plot(histogram_values, color=color)\n    ax.set_title(f'{label} {color.title()} Channel Histogram')\n    ax.set_xlabel('Bin')\n    ax.set_ylabel('Proportion')\n    ax.grid(True)\n\nfor row, result_dict in enumerate(histograms):\n    label = result_dict['label']\n    histogram_dict = result_dict['histograms']\n    for i, color in enumerate(histogram_dict):\n        ax = axes[row, i]\n        plot_color_histogram(histogram_dict[color], label, color, ax)\n        \nplt.tight_layout()\nplt.show()      ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:49.894257Z","iopub.execute_input":"2025-04-09T21:58:49.894542Z","iopub.status.idle":"2025-04-09T21:58:53.260045Z","shell.execute_reply.started":"2025-04-09T21:58:49.894519Z","shell.execute_reply":"2025-04-09T21:58:53.259229Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"intensities = []\n\n# Function to compute mean grayscale intensity per image\ndef extract_grayscale_intensity(sample_ids, label_name):\n    for img_id in sample_ids:\n        img_path = os.path.join(train_dir, img_id + \".tif\")\n        img = cv2.imread(img_path, cv2.IMREAD_GRAYSCALE)\n        mean_intensity = np.mean(img)\n        intensities.append({'label': label_name, 'intensity': mean_intensity})\n\n# Process samples\nextract_grayscale_intensity(benign_samples, 'Benign')\nextract_grayscale_intensity(malignant_samples, 'Malignant')\n\n# Create DataFrame for Seaborn\ndf_gray = pd.DataFrame(intensities)\n\n# Plot\nplt.figure(figsize=(8, 5))\n\nsns.histplot(data=df_gray, x='intensity', hue='label', kde=True, stat='density', common_norm=False, bins=30)\n\nplt.title('Grayscale Intensity Distribution by Class')\nplt.xlabel('Mean Grayscale Intensity')\nplt.ylabel('Density')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:53.260976Z","iopub.execute_input":"2025-04-09T21:58:53.261312Z","iopub.status.idle":"2025-04-09T21:58:54.264754Z","shell.execute_reply.started":"2025-04-09T21:58:53.261258Z","shell.execute_reply":"2025-04-09T21:58:54.263973Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Architecture (Analysis Plan)\nHaving explored the data and a few simple features including RBG histograms and greyscale intensities, we can see that the malignant and benign slides may be difficult to distinguish with traditional machine learning techniques. Therefore, my plan is to use a CNN to attempt to classify images. CNN may work better as it can more easily identify shape-like features in the datasets which may be useful in classifying the images.\n\nFollowing the Keras tutorial [Image classification from scratch](https://keras.io/examples/vision/image_classification_from_scratch/), I'll train a neural net using the Xception network architecture, which performed well in the tutorial for binary classification of cats and dogs.\n\nI'll then proceed with hyperparameter tuning using [Keras Tuner](https://github.com/keras-team/keras-tuner) for a basic model to compare the Xception architecture with a simpler architecture. Hyperparameters to be tuned will include the number of convolution layers, the sizes of each convolution layer, learning rates, and dropout rates. The intent is to discover a model architecture that is well suited for this problem.\n\nThe CNN architecture for all models will include a few important components for this dataset, including:\n- Normalizing the data to 0-1 scale from 1-256 pixel values\n- Cropping the center of the image to get the middle 32x32 pixels (which is specified as the critical area for the test dataset)\n\nAll models will use the Adam optimizer and Binary cross entropy as loss function, which is appropriate for this binary classificaiton task. In the model pipeline, I will also apply data augmentation by rotating and flipping the images","metadata":{}},{"cell_type":"code","source":"# Balance training data (subsample for debugging)\nn_train = train_labels['label'].value_counts().min()\nn_train = round(n_train * 0.05)  # Limit to 10% of total for debugging\n\nbenign = train_labels[train_labels['label'] == 0].sample(n_train)\nmalignant = train_labels[train_labels['label'] == 1].sample(n_train)\ndf_train_all = pd.concat([benign, malignant], axis=0).reset_index(drop=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:54.265678Z","iopub.execute_input":"2025-04-09T21:58:54.265996Z","iopub.status.idle":"2025-04-09T21:58:54.305242Z","shell.execute_reply.started":"2025-04-09T21:58:54.265966Z","shell.execute_reply":"2025-04-09T21:58:54.304485Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Split training and testing data\ntrain_df, val_df = train_test_split(\n    df_train_all, \n    test_size=0.2, \n    stratify=df_train_all['label'], \n    random_state=1337\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:54.306075Z","iopub.execute_input":"2025-04-09T21:58:54.306309Z","iopub.status.idle":"2025-04-09T21:58:54.319615Z","shell.execute_reply.started":"2025-04-09T21:58:54.306267Z","shell.execute_reply":"2025-04-09T21:58:54.318771Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Set up TensorFlow Data Service\ndef make_dataset(df, image_dir, image_size, batch_size, shuffle=True, repeat=True, include_id=False):\n    file_paths = df['path'].astype(str).tolist()\n    labels = df['label'].tolist()\n    ids = df['id'].astype(str).tolist()\n\n    ds = tf.data.Dataset.from_tensor_slices((file_paths, labels, ids))\n\n    if shuffle:\n        ds = ds.shuffle(buffer_size=len(df), reshuffle_each_iteration=True)\n\n    def map_fn(path, label, id_):\n        def _load_image(p):\n            p = p.numpy().decode(\"utf-8\")\n            img = cv2.imread(p)\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n            img = cv2.resize(img, image_size)\n            return img.astype(np.float32) / 255.0\n\n        image = tf.py_function(_load_image, inp=[path], Tout=tf.float32)\n        image.set_shape((*image_size, 3))\n        return (image, label, id_) if include_id else (image, label)\n\n    ds = ds.map(map_fn, num_parallel_calls=tf.data.AUTOTUNE)\n\n    if repeat:\n        ds = ds.repeat()\n\n    ds = ds.batch(batch_size)\n    ds = ds.prefetch(tf.data.AUTOTUNE)\n\n    return ds","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:54.320453Z","iopub.execute_input":"2025-04-09T21:58:54.320674Z","iopub.status.idle":"2025-04-09T21:58:54.327866Z","shell.execute_reply.started":"2025-04-09T21:58:54.320655Z","shell.execute_reply":"2025-04-09T21:58:54.327034Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create data streams\nimage_size = (96, 96)\nbatch_size = 128\n\ntrain_ds = make_dataset(train_df, train_dir, image_size, batch_size, shuffle=True)\nval_ds = make_dataset(val_df, train_dir, image_size, batch_size, shuffle=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:54.328686Z","iopub.execute_input":"2025-04-09T21:58:54.328921Z","iopub.status.idle":"2025-04-09T21:58:55.456864Z","shell.execute_reply.started":"2025-04-09T21:58:54.328902Z","shell.execute_reply":"2025-04-09T21:58:55.455951Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Augment data with rotation and flip\ndata_augmentation = keras.Sequential([\n    layers.RandomFlip(\"horizontal\"),\n    layers.RandomRotation(0.1)\n])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:55.457742Z","iopub.execute_input":"2025-04-09T21:58:55.457966Z","iopub.status.idle":"2025-04-09T21:58:55.471299Z","shell.execute_reply.started":"2025-04-09T21:58:55.457947Z","shell.execute_reply":"2025-04-09T21:58:55.470673Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Train Xception model","metadata":{}},{"cell_type":"code","source":"def make_model(input_shape, num_classes):\n    inputs = keras.Input(shape=input_shape)\n    x = data_augmentation(inputs)\n    # x = layers.Rescaling(1./255)(x)\n    x = layers.CenterCrop(32, 32)(x)\n\n    # Entry block\n    x = layers.Conv2D(128, 3, strides=2, padding=\"same\")(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(\"relu\")(x)\n\n    previous_block_activation = x  # Set aside residual\n\n    for size in [256, 512, 728]:\n        x = layers.Activation(\"relu\")(x)\n        x = layers.SeparableConv2D(size, 3, padding=\"same\")(x)\n        x = layers.BatchNormalization()(x)\n\n        x = layers.Activation(\"relu\")(x)\n        x = layers.SeparableConv2D(size, 3, padding=\"same\")(x)\n        x = layers.BatchNormalization()(x)\n\n        x = layers.MaxPooling2D(3, strides=2, padding=\"same\")(x)\n\n        # Project residual\n        residual = layers.Conv2D(size, 1, strides=2, padding=\"same\")(\n            previous_block_activation\n        )\n        x = layers.add([x, residual])\n        previous_block_activation = x\n\n    x = layers.SeparableConv2D(1024, 3, padding=\"same\")(x)\n    x = layers.BatchNormalization()(x)\n    x = layers.Activation(\"relu\")(x)\n\n    x = layers.GlobalAveragePooling2D()(x)\n    if num_classes == 2:\n        units = 1\n    else:\n        units = num_classes\n\n    x = layers.Dropout(0.25)(x)\n    # We specify activation=None so as to return logits\n    outputs = layers.Dense(units, activation=None)(x)\n    return keras.Model(inputs, outputs)\n\nxception_model = make_model(input_shape=image_size + (3,), num_classes=2)\nkeras.utils.plot_model(xception_model, show_shapes=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:55.472052Z","iopub.execute_input":"2025-04-09T21:58:55.472317Z","iopub.status.idle":"2025-04-09T21:58:58.807099Z","shell.execute_reply.started":"2025-04-09T21:58:55.472297Z","shell.execute_reply":"2025-04-09T21:58:58.806231Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Because the model returns logit rather than 0-1 probabilities, the validation loss is still high when confidence is low, and thus validation accuracy tends to grow very slowly. With early stopping, the training often stops prematurely before sufficient burn in time for weights. I create the MinimumEpochEarlyStopping to overcome this. ","metadata":{}},{"cell_type":"code","source":"class MinimumEpochEarlyStopping(keras.callbacks.EarlyStopping):\n    def __init__(self, min_epochs=10, **kwargs):\n        super().__init__(**kwargs)\n        self.min_epochs = min_epochs\n\n    def on_epoch_end(self, epoch, logs=None):\n        # Only start checking for stopping after reaching min_epochs\n        if epoch + 1 >= self.min_epochs:\n            super().on_epoch_end(epoch, logs)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:58.807850Z","iopub.execute_input":"2025-04-09T21:58:58.808085Z","iopub.status.idle":"2025-04-09T21:58:58.812461Z","shell.execute_reply.started":"2025-04-09T21:58:58.808065Z","shell.execute_reply":"2025-04-09T21:58:58.811675Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"epochs = 25\nsteps_per_epoch = int(np.ceil(len(train_df) / batch_size))\nvalidation_steps = int(np.ceil(len(val_df) / batch_size))\n    \ncallbacks = [\n    MinimumEpochEarlyStopping(\n        monitor='val_loss',\n        patience=3,\n        restore_best_weights=True\n    ),\n    keras.callbacks.ReduceLROnPlateau(\n        monitor='val_loss',\n        factor=0.5,\n        patience=2,\n        min_lr=1e-6,\n        verbose=1\n    )\n]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:58.813526Z","iopub.execute_input":"2025-04-09T21:58:58.813840Z","iopub.status.idle":"2025-04-09T21:58:58.838827Z","shell.execute_reply.started":"2025-04-09T21:58:58.813802Z","shell.execute_reply":"2025-04-09T21:58:58.838003Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"xception_model.compile(\n    optimizer=keras.optimizers.Adam(3e-4),\n    loss=keras.losses.BinaryCrossentropy(from_logits=True),\n    metrics=[keras.metrics.BinaryAccuracy(name=\"accuracy\")],\n)\n\nhistory_xception = xception_model.fit(\n    train_ds,\n    epochs=epochs,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps,\n    callbacks=callbacks,\n    validation_data=val_ds,\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T21:58:58.839548Z","iopub.execute_input":"2025-04-09T21:58:58.839779Z","iopub.status.idle":"2025-04-09T22:03:59.107928Z","shell.execute_reply.started":"2025-04-09T21:58:58.839760Z","shell.execute_reply":"2025-04-09T22:03:59.107190Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"I was able to achieve fairly high training accuracy using the untuned Xception model architecture. Based on hyperparameter tuning (not presented in this notebook) I feel confident that the architecture of Xception is fairly well tuned for this dataset (although of course some additional performance could be squeezed out; as this is a demonstration notebook I don't invest the training time to explore all hyperparameter combinations).\n\nLet's explore a more basic model where hyperparameter tuning is faster to see how well Xception performs relative to a basic architecture.","metadata":{}},{"cell_type":"markdown","source":"## Train basic CNN with hyperparameter tuning","metadata":{}},{"cell_type":"code","source":"def build_basic_model(hp):\n    model = keras.Sequential()\n    \n    model.add(keras.Input(shape=image_size + (3,)))\n    # model.add(layers.Rescaling(1./255))\n    model.add(layers.CenterCrop(32, 32))\n\n    # Tune number of layers and layer sizes\n    for i in range(hp.Int('num_blocks', 1, 3)):\n        filters = hp.Choice(f'block_filters_{i}', values=[32, 64, 128])\n        model.add(layers.Conv2D(filters, kernel_size=(3, 3), activation=\"relu\"))\n        model.add(layers.MaxPooling2D(pool_size=(2, 2)))\n\n    model.add(layers.Flatten())\n\n    # Tune dropout \n    model.add(layers.Dropout(hp.Choice('dropout', values=[0.2, 0.3, 0.5])))\n    model.add(layers.Dense(1, activation=\"sigmoid\"))\n\n    # Compile model with variable learning rate\n    model.compile(\n        optimizer=keras.optimizers.Adam(\n            hp.Float(\"learning_rate\", 1e-4, 1e-2, sampling=\"log\")\n        ),\n        loss=keras.losses.BinaryCrossentropy(),\n        metrics=[keras.metrics.BinaryAccuracy(name=\"accuracy\")]\n    )\n\n    return model","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T22:03:59.108711Z","iopub.execute_input":"2025-04-09T22:03:59.108948Z","iopub.status.idle":"2025-04-09T22:03:59.115190Z","shell.execute_reply.started":"2025-04-09T22:03:59.108927Z","shell.execute_reply":"2025-04-09T22:03:59.114258Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Delete the tuner directory if it exists\nimport shutil\ntuner_dir = 'my_tuners/basic'\nif os.path.exists(tuner_dir):\n    shutil.rmtree(tuner_dir)\n\n# Initialize tuner\nbasic_tuner = kt.RandomSearch(\n    lambda hp: build_basic_model(hp),\n    objective='val_accuracy',\n    directory='my_tuners',\n    project_name='basic'\n)\n\n# Summarize tuner\nbasic_tuner.search_space_summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T22:03:59.118482Z","iopub.execute_input":"2025-04-09T22:03:59.118714Z","iopub.status.idle":"2025-04-09T22:03:59.184422Z","shell.execute_reply.started":"2025-04-09T22:03:59.118694Z","shell.execute_reply":"2025-04-09T22:03:59.183453Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Run tuner\nbasic_tuner.search(\n    train_ds,\n    epochs=epochs,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps,\n    callbacks=callbacks,\n    validation_data=val_ds,\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T22:03:59.185559Z","iopub.execute_input":"2025-04-09T22:03:59.185772Z","iopub.status.idle":"2025-04-09T22:26:04.509222Z","shell.execute_reply.started":"2025-04-09T22:03:59.185753Z","shell.execute_reply":"2025-04-09T22:26:04.508372Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"basic_tuner.results_summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T22:26:04.530433Z","iopub.execute_input":"2025-04-09T22:26:04.530686Z","iopub.status.idle":"2025-04-09T22:26:04.546608Z","shell.execute_reply.started":"2025-04-09T22:26:04.530662Z","shell.execute_reply":"2025-04-09T22:26:04.541488Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Get the top model\nmodels = basic_tuner.get_best_models(num_models=1)\nbest_model = models[0]\nbest_model.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T22:26:04.547412Z","iopub.execute_input":"2025-04-09T22:26:04.547702Z","iopub.status.idle":"2025-04-09T22:26:05.327995Z","shell.execute_reply.started":"2025-04-09T22:26:04.547671Z","shell.execute_reply":"2025-04-09T22:26:05.327128Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The top model architecture for a basic `Sequential` model in `keras` is shown above. ","metadata":{}},{"cell_type":"markdown","source":"## Results & Analysis\nI'll fit both the basic model and the Xception model to produce submissions for the public leaderboard so we can compare true performance. I'll also plot training versus validation accuracy over epochs to get a sense of model stability and the AUC-ROC curve as that is the accuracy metric used by the competition.","metadata":{}},{"cell_type":"code","source":"# Balance training data (subsample for debugging)\nn_train = train_labels['label'].value_counts().min()\n\nbenign = train_labels[train_labels['label'] == 0].sample(n_train)\nmalignant = train_labels[train_labels['label'] == 1].sample(n_train)\ndf_train_all = pd.concat([benign, malignant], axis=0).reset_index(drop=True)\n\n# Split training and testing data\ntrain_df, val_df = train_test_split(\n    df_train_all, \n    test_size=0.2, \n    stratify=df_train_all['label'], \n    random_state=1337\n)\n\nsteps_per_epoch = int(np.ceil(len(train_df) / batch_size))\nvalidation_steps = int(np.ceil(len(val_df) / batch_size))\n\n# Make dataset\ntrain_ds_full = make_dataset(train_df, train_dir, image_size, batch_size, shuffle=True)\nval_ds_full = make_dataset(val_df, train_dir, image_size, batch_size, shuffle=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T22:26:05.328887Z","iopub.execute_input":"2025-04-09T22:26:05.329157Z","iopub.status.idle":"2025-04-09T22:26:06.553008Z","shell.execute_reply.started":"2025-04-09T22:26:05.329125Z","shell.execute_reply":"2025-04-09T22:26:06.552361Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Fit final basic model\nbest_basic_hps = basic_tuner.get_best_hyperparameters(num_trials=1)\nbest_basic_model = build_basic_model(best_basic_hps[0])\nhistory_basic = best_basic_model.fit(\n    train_ds_full,\n    epochs=epochs,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps,\n    callbacks=callbacks,\n    validation_data=val_ds_full  \n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T22:26:06.553680Z","iopub.execute_input":"2025-04-09T22:26:06.553912Z","iopub.status.idle":"2025-04-09T23:40:17.192749Z","shell.execute_reply.started":"2025-04-09T22:26:06.553892Z","shell.execute_reply":"2025-04-09T23:40:17.192008Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Fit Xception model to all data\nhistory_xception = xception_model.fit(\n    train_ds_full,\n    epochs=epochs,\n    steps_per_epoch=steps_per_epoch,\n    validation_steps=validation_steps,\n    callbacks=callbacks,\n    validation_data=val_ds_full,\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-09T23:40:17.193493Z","iopub.execute_input":"2025-04-09T23:40:17.193712Z","iopub.status.idle":"2025-04-10T00:48:34.003947Z","shell.execute_reply.started":"2025-04-09T23:40:17.193693Z","shell.execute_reply":"2025-04-10T00:48:34.003158Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_accuracy(model, ax, title):\n    ax.plot(model.history['accuracy'], label='Train Accuracy', color='#1f77b4', linestyle='dashed')\n    ax.plot(model.history['val_accuracy'], label='Val Accuracy', color='#1f77b4')\n    ax.set_title(f'CNN ({title}) Accuracy')\n    ax.set_xlabel('Epochs')\n    ax.set_ylabel('Accuracy')\n    ax.legend()\n\nfig, axes = plt.subplots(1, 2, figsize=(10, 6), sharey=True)\n\n# Plot history_basic\nplot_accuracy(history_basic, axes[0], \"Basic\")\nplot_accuracy(history_xception, axes[1], \"Xception\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-10T00:48:34.004791Z","iopub.execute_input":"2025-04-10T00:48:34.005032Z","iopub.status.idle":"2025-04-10T00:48:34.536226Z","shell.execute_reply.started":"2025-04-10T00:48:34.005011Z","shell.execute_reply":"2025-04-10T00:48:34.535331Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Get true and predicted labels\ny_true = np.concatenate([y.numpy() for _, y in val_ds.take(validation_steps)], axis=0)\ny_pred_basic = best_basic_model.predict(val_ds, steps=validation_steps).ravel()\ny_pred_xception = xception_model.predict(val_ds, steps=validation_steps).ravel()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-10T00:48:34.537054Z","iopub.execute_input":"2025-04-10T00:48:34.537314Z","iopub.status.idle":"2025-04-10T00:50:54.724024Z","shell.execute_reply.started":"2025-04-10T00:48:34.537271Z","shell.execute_reply":"2025-04-10T00:50:54.723270Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_roc(y_true, y_pred, ax, title):\n    fpr, tpr, thresholds = roc_curve(y_true, y_pred)\n    roc_auc = auc(fpr, tpr)\n    ax.plot(fpr, tpr, label=f\"ROC curve (AUC = {roc_auc:.2f})\")\n    ax.plot([0, 1], [0, 1], \"k--\", label=\"Random baseline\")\n    ax.set_title(f\"Receiver Operating Characteristic (ROC) | {title}\")\n    ax.set_xlabel(\"False Positive Rate\")\n    ax.set_ylabel(\"True Positive Rate\")\n    ax.legend(loc=\"lower right\")\n\nfig, axes = plt.subplots(1, 2, figsize=(10, 6), sharey=True)\n\n# Plot\nplot_roc(y_true, y_pred_basic, axes[0], \"Basic\")\nplot_roc(y_true, y_pred_xception, axes[1], \"Xception\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-10T00:50:54.724896Z","iopub.execute_input":"2025-04-10T00:50:54.725116Z","iopub.status.idle":"2025-04-10T00:50:55.235887Z","shell.execute_reply.started":"2025-04-10T00:50:54.725097Z","shell.execute_reply":"2025-04-10T00:50:55.234978Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can see that the best basic model performed fairly well and was able to stablize training in the first few epochs for both training and validation sets. Training and validation accuracies are similar which indicates low likelihood of overfitting. In comparison, the Xception model struggled to stabilize around the validation set which may indicate an issue with the learning rate or validation sets. The low validation rate and steep climb after 8 epochs is also of note, but may be due to using logits rather than probabilities as output for that model. ","metadata":{}},{"cell_type":"markdown","source":"## Create Submission\nI'll create a submission for both the Xception model and basic model using the test dataset.","metadata":{}},{"cell_type":"code","source":"# Create test data stream\ntest_ds = make_dataset(test_labels, test_dir, image_size, batch_size, \n                       shuffle=False, repeat=False, include_id=True)\n\n# Strip out the IDs for prediction\npredict_ds = test_ds.map(lambda image, label, id_: image)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-10T00:50:55.236628Z","iopub.execute_input":"2025-04-10T00:50:55.236859Z","iopub.status.idle":"2025-04-10T00:50:55.541063Z","shell.execute_reply.started":"2025-04-10T00:50:55.236822Z","shell.execute_reply":"2025-04-10T00:50:55.540384Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"all_ids = []\nfor batch in test_ds:\n    _, _, ids = batch\n    all_ids.extend([id_.numpy().decode(\"utf-8\") for id_ in ids])\nlen(all_ids)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-10T00:50:55.541749Z","iopub.execute_input":"2025-04-10T00:50:55.541958Z","iopub.status.idle":"2025-04-10T00:53:33.470902Z","shell.execute_reply.started":"2025-04-10T00:50:55.541939Z","shell.execute_reply":"2025-04-10T00:53:33.469959Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Create submission\ny_pred_basic = best_basic_model.predict(predict_ds).ravel()\nsubmission_basic_df = pd.DataFrame(\n    {\n        'id': all_ids,\n        'label': y_pred_basic\n    }\n)\nsubmission_basic_df.to_csv('submission_basic.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-10T01:01:38.563952Z","iopub.execute_input":"2025-04-10T01:01:38.564255Z","iopub.status.idle":"2025-04-10T01:02:53.426972Z","shell.execute_reply.started":"2025-04-10T01:01:38.564232Z","shell.execute_reply":"2025-04-10T01:02:53.426276Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"y_pred_xception = xception_model.predict(predict_ds).ravel()\ny_pred_xception_probs = expit(y_pred_xception)\nsubmission_xception_df = pd.DataFrame(\n    {\n        'id': all_ids,\n        'label': y_pred_xception_probs\n    }\n)\nsubmission_xception_df.to_csv('submission_xception.csv', index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-10T00:59:38.297787Z","iopub.execute_input":"2025-04-10T00:59:38.298067Z","iopub.status.idle":"2025-04-10T01:00:55.337200Z","shell.execute_reply.started":"2025-04-10T00:59:38.298044Z","shell.execute_reply":"2025-04-10T01:00:55.336536Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Conclusion\nThe basic model architecture was surprisingly competitive with the more complex Xception model architecture. The complexity of the Xception model may have resulted in some unstable behavior as the validation accuracy never settled during training. \n\nAs an aside, I invested heavily in developing a data streaming pipeline that would work efficiently for TIFF data in an interest in reusing this pipeline for more data intensive tasks. It certainly would have been feasible to either copy and convert all of the data to a file format supported by Keras for image processing or even read the data into memory, which would have allowed more time for hyperparameter tuning.\n\nFuture directions may be to test other model architectures like Resnet which seemed to work well for others and ensemble multiple models. I would also invest more in debugging the unstable validation error for the Xception model.","metadata":{}},{"cell_type":"markdown","source":"## References\n- https://keras.io/getting_started/intro_to_keras_for_engineers/\n- https://keras.io/examples/vision/image_classification_from_scratch/\n- https://keras.io/examples/vision/mnist_convnet/\n- https://keras.io/keras_tuner/getting_started/\n- https://www.tensorflow.org/tutorials/load_data/images","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}