{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":11848,"databundleVersionId":862157,"sourceType":"competition"}],"dockerImageVersionId":30887,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Histopathologic Cancer Detection: Deep Learning for Metastatic Tissue Classification","metadata":{}},{"cell_type":"markdown","source":"## Introduction","metadata":{}},{"cell_type":"markdown","source":"Cancer detection in histopathological images is a critical task in medical diagnostics, aiding in early detection and treatment planning. Deep learning-based approaches, particularly Convolutional Neural Networks (CNNs), have demonstrated state-of-the-art performance in automated tumor classification. This project leverages a CNN-based model to classify histopathology image patches into tumor and non-tumor categories, providing an efficient and scalable solution for digital pathology","metadata":{}},{"cell_type":"markdown","source":"## Objective","metadata":{}},{"cell_type":"markdown","source":"The primary goal of this project is to develop and evaluate a deep learning model capable of accurately identifying metastatic tissue in histopathologic scans of lymph node sections. This involves:\n\n- Preprocessing high-resolution pathology images for deep learning.\n- Designing and training a CNN architecture tailored for medical image classification.\n- Optimizing the model using advanced training techniques, including learning rate scheduling and regularization.\n- Evaluating model performance using standard classification metrics such as accuracy, precision, recall, F1-score, and AUC-ROC.\n- Generating predictions for unseen test images and analyzing the results for potential clinical applications","metadata":{}},{"cell_type":"markdown","source":"## Model Architecture Overview","metadata":{}},{"cell_type":"markdown","source":"This project employs a custom CNN model with multiple convolutional and pooling layers, followed by fully connected layers for classification. Key design aspects include:\n\n- Feature extraction through stacked convolutional layers with ReLU activation.\n- Max pooling for spatial dimensionality reduction and feature selection.\n- Fully connected layers for high-level representation learning.\n- Dropout regularization to prevent overfitting.\n- Softmax activation for multi-class probability distribution.\n\nThe model is trained using the Negative Log-Likelihood Loss (NLLLoss) function with an Adam optimizer, ensuring efficient convergence and robustness.","metadata":{}},{"cell_type":"markdown","source":"## Significance & Impact","metadata":{}},{"cell_type":"markdown","source":"- Medical Advancement: Automating tumor detection in histopathology images enhances diagnostic efficiency, reducing pathologists' workload.\n- Clinical Application: A reliable deep learning model can assist in early cancer diagnosis, improving patient outcomes.\n- Deep Learning Research: This project contributes to medical AI research, demonstrating CNN capabilities in digital pathology.\n\nBy integrating state-of-the-art deep learning techniques, this project aims to push the boundaries of AI-assisted cancer detection, bridging the gap between medical imaging and artificial intelligence.","metadata":{}},{"cell_type":"markdown","source":"## Dataset","metadata":{}},{"cell_type":"markdown","source":"The Histopathologic Cancer Detection dataset is designed to identify metastatic tissue in histopathologic scans of lymph node sections. The dataset contains small pathology image patches, each labeled based on the presence of tumor tissue in the central 32x32px region.","metadata":{}},{"cell_type":"markdown","source":"### Dataset Structure","metadata":{}},{"cell_type":"markdown","source":"\n1. Image Files\n\n- Each image file is named with a unique image ID.\n- The dataset consists of high-resolution pathology images, used for training and testing a binary classification model\n\n2. Labels and Ground Truth\n\n- The train_labels.csv file contains the ground truth labels for images in the training set.\n- The test set images do not have labels, as they are used for model evaluation and submission.\n- A positive label (1) indicates that the central 32×32px region contains tumor tissue.\n- The outer regions of each image do not affect the label but are included to help convolutional models generalize.","metadata":{}},{"cell_type":"markdown","source":"## Summary","metadata":{}},{"cell_type":"markdown","source":"- Goal: Classify pathology images as tumor (1) or non-tumor (0).\n- Input: Small pathology image patches (high-resolution)\n- Labels: Defined by the central 32×32px region.\n- No Duplicates: Ensuring data integrity for model training.\n- Benchmark-Compatible: Maintains PCam dataset structure for comparison with existing methods.","metadata":{}},{"cell_type":"code","source":"import os\nimport json\nfrom pathlib import Path\nfrom tqdm.notebook import trange, tqdm\nimport copy\nimport random\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom PIL import Image\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score, confusion_matrix, classification_report\n\nimport torch\nfrom torch.utils.data import Dataset, DataLoader, Subset, random_split\nfrom torch.optim import Adam\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torchvision.transforms as transforms\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\nrandom.seed(2025)\ntorch.manual_seed(2025)\n\nsns.set_context('notebook')\nsns.set_style('white')\n\n%matplotlib inline","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:23:57.361151Z","iopub.execute_input":"2025-02-21T20:23:57.361528Z","iopub.status.idle":"2025-02-21T20:23:57.375351Z","shell.execute_reply.started":"2025-02-21T20:23:57.361480Z","shell.execute_reply":"2025-02-21T20:23:57.373914Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"if torch.cuda.is_available():\n    print(f\"Compatible GPU ({torch.cuda.get_device_name()}) found\")\nelse:\n    print(f\"No compatible GPU found.\")\n    \ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:23:57.393055Z","iopub.execute_input":"2025-02-21T20:23:57.393379Z","iopub.status.idle":"2025-02-21T20:23:57.399131Z","shell.execute_reply.started":"2025-02-21T20:23:57.393354Z","shell.execute_reply":"2025-02-21T20:23:57.398145Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data Loading and Preprocessing","metadata":{}},{"cell_type":"markdown","source":"This section defines a custom PyTorch Dataset for the Histopathologic Cancer Detection images. The dataset is structured to load images and their corresponding labels from a given directory, apply specified transformations, and allow for random subsampling. Detailed steps are provided below.","metadata":{}},{"cell_type":"markdown","source":"The dataset is initialized with a root data directory, a transformation function, a specified data type (e.g., \"train\"), and the number of samples to load\\\nIt randomly selects a defined number of images and retrieves their corresponding labels from the CSV file","metadata":{}},{"cell_type":"code","source":"class CancerDataset(Dataset):\n    \"\"\"\n    Custom Dataset for loading histopathologic cancer detection images.\n\n    Args:\n        data_dir (str or Path): Root directory containing the image data and labels.\n        transform (callable, optional): Transformation function to apply to the images.\n        data_type (str): Directory selection - \"train\", \"test\", or \"val\".\n        num_samples (int, optional): Number of samples to randomly select from the dataset.\n    \"\"\"\n    \n    def __init__(self, data_dir, transform=None, data_type=\"train\"):\n        self.data_dir = Path(data_dir)\n        self.data_type = data_type\n        self.transform = transform\n\n        # Define the image directory based on the data_type (e.g., train, test)\n        image_dir = self.data_dir / data_type\n        if not image_dir.exists():\n            raise FileNotFoundError(f\"Directory '{image_dir}' not found.\")        \n        \n        # Load all valid image files (.tif) in the directory\n        all_files = list(image_dir.glob(\"*.tif\"))\n\n        # No sample\n        num_samples = len(all_files)\n        \n        if num_samples > len(all_files):\n            raise ValueError(f\"num_samples ({num_samples}) exceeds available images ({len(all_files)}).\")\n        \n        # Randomly select a subset of image files\n        self.full_filenames = np.random.choice(all_files, num_samples, replace=False).tolist()\n\n        # Load labels from a CSV file (expects columns 'id' and 'label')\n        labels_file = self.data_dir / \"train_labels.csv\"\n        if not labels_file.exists():\n            raise FileNotFoundError(f\"Labels file '{labels_file}' not found.\")\n        \n        labels_df = pd.read_csv(labels_file).set_index(\"id\")\n        self.labels = [labels_df.loc[img.stem].values[0] for img in self.full_filenames]\n\n    def __len__(self):\n        # Return the number of samples in the dataset\n        return len(self.full_filenames)\n\n    def __getitem__(self, idx):\n        # Retrieve the image path and corresponding label using the index\n        img_path = self.full_filenames[idx]\n        image = Image.open(img_path).convert(\"RGB\")  # Ensure image is in RGB format\n    \n        if self.transform:\n            image = self.transform(image)\n    \n        label = self.labels[idx]\n        img_id = img_path.stem  # Extract the image ID (filename without extension)\n        return image, label, img_id","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:23:57.409737Z","iopub.execute_input":"2025-02-21T20:23:57.410058Z","iopub.status.idle":"2025-02-21T20:23:57.419517Z","shell.execute_reply.started":"2025-02-21T20:23:57.410031Z","shell.execute_reply":"2025-02-21T20:23:57.418572Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"For the training set, random horizontal/vertical flips and random rotations are applied to augment the dataset","metadata":{}},{"cell_type":"markdown","source":"For validation (and test), only basic tensor conversion is performed to ensure consistency during evaluation","metadata":{}},{"cell_type":"code","source":"# Define transformations for training and validation datasets\ntrain_transforms = transforms.Compose([\n    transforms.RandomHorizontalFlip(p=0.5),\n    transforms.RandomVerticalFlip(p=0.5),\n    transforms.RandomRotation(45),\n    transforms.ToTensor()\n])\n\nval_transforms = transforms.Compose([\n    transforms.ToTensor()\n])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:23:57.462876Z","iopub.execute_input":"2025-02-21T20:23:57.463316Z","iopub.status.idle":"2025-02-21T20:23:57.469312Z","shell.execute_reply.started":"2025-02-21T20:23:57.463283Z","shell.execute_reply":"2025-02-21T20:23:57.468208Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Set the path to the dataset directory\ndata_dir = '/kaggle/input/histopathologic-cancer-detection/'\n\n# Initialize the CancerDataset for training data\ndataset = CancerDataset(data_dir, transform=train_transforms, data_type=\"train\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:23:57.471015Z","iopub.execute_input":"2025-02-21T20:23:57.471297Z","iopub.status.idle":"2025-02-21T20:24:01.922503Z","shell.execute_reply.started":"2025-02-21T20:23:57.471273Z","shell.execute_reply":"2025-02-21T20:24:01.921554Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The dataset indices are split into training (80%), validation (10%), and test (10%) sets. This is achieved in two steps using train_test_split.","metadata":{}},{"cell_type":"code","source":"# Create an array of indices for the entire dataset\nindices = np.arange(len(dataset))\n\n# First split: Separate 20% (temporary set) from the 80% training set\ntrain_indices, temp_indices = train_test_split(indices, test_size=0.2, random_state=2025)\n\n# Second split: Divide the temporary set into two halves for validation and test (10% each)\nval_indices, test_indices = train_test_split(temp_indices, test_size=0.5, random_state=2025)\n\n# Create subsets for training, validation, and testing using the indices\ntrain_dataset = Subset(dataset, train_indices)\nval_dataset = Subset(dataset, val_indices)\ntest_dataset = Subset(dataset, test_indices)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:01.924563Z","iopub.execute_input":"2025-02-21T20:24:01.925094Z","iopub.status.idle":"2025-02-21T20:24:01.932402Z","shell.execute_reply.started":"2025-02-21T20:24:01.925043Z","shell.execute_reply":"2025-02-21T20:24:01.931245Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Apply appropriate transformations\ntrain_dataset.dataset.transform = train_transforms\nval_dataset.dataset.transform = val_transforms\ntest_dataset.dataset.transform = val_transforms","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:01.934067Z","iopub.execute_input":"2025-02-21T20:24:01.934442Z","iopub.status.idle":"2025-02-21T20:24:01.953992Z","shell.execute_reply.started":"2025-02-21T20:24:01.934410Z","shell.execute_reply":"2025-02-21T20:24:01.952871Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Print dataset sizes for verification\nprint(f\"Training dataset size: {len(train_dataset)}\")\nprint(f\"Validation dataset size: {len(val_dataset)}\")\nprint(f\"Test dataset size: {len(test_dataset)}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:01.955157Z","iopub.execute_input":"2025-02-21T20:24:01.955582Z","iopub.status.idle":"2025-02-21T20:24:01.972244Z","shell.execute_reply.started":"2025-02-21T20:24:01.955543Z","shell.execute_reply":"2025-02-21T20:24:01.970866Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"DataLoaders are defined for the training and validation sets with appropriate batch sizes and shuffling enabled for training","metadata":{}},{"cell_type":"code","source":"# Define DataLoaders for training and validation\ntrain_loader = DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=4)\nval_loader = DataLoader(val_dataset, batch_size=32, shuffle=False, num_workers=4)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:01.973308Z","iopub.execute_input":"2025-02-21T20:24:01.973660Z","iopub.status.idle":"2025-02-21T20:24:01.989194Z","shell.execute_reply.started":"2025-02-21T20:24:01.973627Z","shell.execute_reply":"2025-02-21T20:24:01.988027Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data Visualization","metadata":{}},{"cell_type":"markdown","source":"Before training the model, it is crucial to visualize a subset of images to gain an intuitive understanding of the dataset. The function below randomly selects a given number of images from the dataset and displays them in a grid format","metadata":{}},{"cell_type":"code","source":"def plot_sample_images(dataset, data_dir, data_type=\"train\", num_images=12, title=\"Sample Images\"):\n    \"\"\"\n    Plots a grid of sample images from the dataset to visualize their appearance.\n\n    Args:\n        dataset (Dataset or Subset): The dataset object (train, validation, or test).\n        data_dir (str or Path): Root directory containing image data and labels.\n        data_type (str): Directory selection - \"train\", \"test\", or \"val\".\n        num_images (int): Number of images to display in the grid.\n        title (str): Title of the plot.\n    \n    Displays:\n        A matplotlib figure with sample images and their corresponding labels.\n    \"\"\"\n\n    # Define the grid layout (3 rows, num_images/3 columns)\n    fig, axes = plt.subplots(nrows=3, ncols=num_images // 3, figsize=(15, 7))\n    fig.suptitle(title, fontsize=16, fontweight='bold')\n\n    for ax in axes.flat:\n        # Select a random image index\n        idx = random.randint(0, len(dataset) - 1)\n\n        # Retrieve image and label (handling test set separately)\n        if data_type == \"test\":\n            filename, label = dataset[idx]\n        else:\n            img, label, filename = dataset[idx]        \n\n        # Load and display the image\n        img_path = os.path.join(data_dir, data_type, filename + '.tif')\n        ax.imshow(Image.open(img_path))\n        ax.set_title(f\"{'Cancer' if label == 1 else 'Normal'}\", fontsize=10)\n        ax.axis(\"off\")\n\n    # Adjust layout to prevent overlap\n    plt.tight_layout()\n    plt.subplots_adjust(top=0.9)\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:01.990226Z","iopub.execute_input":"2025-02-21T20:24:01.990595Z","iopub.status.idle":"2025-02-21T20:24:02.007133Z","shell.execute_reply.started":"2025-02-21T20:24:01.990568Z","shell.execute_reply":"2025-02-21T20:24:02.006046Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Plot sample images from training and validation sets\nplot_sample_images(train_dataset, data_dir, data_type=\"train\", title=\"Training Set Samples\")\nplot_sample_images(val_dataset, data_dir, data_type=\"train\", title=\"Validation Set Samples\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:02.010150Z","iopub.execute_input":"2025-02-21T20:24:02.010484Z","iopub.status.idle":"2025-02-21T20:24:05.768202Z","shell.execute_reply.started":"2025-02-21T20:24:02.010440Z","shell.execute_reply":"2025-02-21T20:24:05.766584Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Architecture","metadata":{}},{"cell_type":"markdown","source":"Before defining the architecture, we need to compute how the convolutional layers affect the spatial dimensions of the input. The function below calculates the height (hout) and width (wout) of the output feature maps after applying a convolutional layer","metadata":{}},{"cell_type":"code","source":"def findConv2dOutShape(hin, win, conv, pool=2):\n    \"\"\"\n    Computes the output shape of a convolutional layer.\n\n    Args:\n        hin (int): Input height.\n        win (int): Input width.\n        conv (torch.nn.Conv2d): Convolutional layer instance.\n        pool (int, optional): Pooling factor (default: 2).\n\n    Returns:\n        tuple: Output height and width after convolution and pooling.\n    \"\"\"\n    kernel_size = conv.kernel_size\n    stride = conv.stride\n    padding = conv.padding\n    dilation = conv.dilation\n\n    hout = np.floor((hin + 2 * padding[0] - dilation[0] * (kernel_size[0] - 1) - 1) / stride[0] + 1)\n    wout = np.floor((win + 2 * padding[1] - dilation[1] * (kernel_size[1] - 1) - 1) / stride[1] + 1)\n\n    if pool:\n        hout /= pool\n        wout /= pool\n\n    return int(hout), int(wout)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.770764Z","iopub.execute_input":"2025-02-21T20:24:05.771287Z","iopub.status.idle":"2025-02-21T20:24:05.779949Z","shell.execute_reply.started":"2025-02-21T20:24:05.771231Z","shell.execute_reply":"2025-02-21T20:24:05.778715Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The CNN consists of four convolutional layers followed by fully connected (FC) layers for classification. The architecture is built dynamically based on the given hyperparameters.","metadata":{}},{"cell_type":"code","source":"# Define the Convolutional Neural Network\nclass Network(nn.Module):\n    \"\"\"\n    Convolutional Neural Network for histopathological cancer detection.\n\n    Args:\n        params (dict): Dictionary containing model hyperparameters.\n\n    Model Architecture:\n        - 4 Convolutional layers with ReLU activation and MaxPooling\n        - 2 Fully Connected (FC) layers with ReLU activation and Dropout\n        - LogSoftmax activation in the output layer\n    \"\"\"\n\n    def __init__(self, params):\n        super(Network, self).__init__()\n\n        # Extract parameters\n        Cin, Hin, Win = params[\"shape_in\"]\n        init_f = params[\"initial_filters\"]\n        num_fc1 = params[\"num_fc1\"]\n        num_classes = params[\"num_classes\"]\n        self.dropout_rate = params[\"dropout_rate\"]\n\n        # Convolutional Layers\n        self.conv1 = nn.Conv2d(Cin, init_f, kernel_size=3)\n        h, w = findConv2dOutShape(Hin, Win, self.conv1)\n        self.conv2 = nn.Conv2d(init_f, 2 * init_f, kernel_size=3)\n        h, w = findConv2dOutShape(h, w, self.conv2)\n        self.conv3 = nn.Conv2d(2 * init_f, 4 * init_f, kernel_size=3)\n        h, w = findConv2dOutShape(h, w, self.conv3)\n        self.conv4 = nn.Conv2d(4 * init_f, 8 * init_f, kernel_size=3)\n        h, w = findConv2dOutShape(h, w, self.conv4)\n\n        # Fully Connected Layers\n        self.fc1 = nn.Linear(1024, num_fc1)\n        self.fc2 = nn.Linear(num_fc1, num_classes)\n\n    def forward(self, X):\n        \"\"\"\n        Forward pass of the CNN.\n\n        Args:\n            X (torch.Tensor): Input batch of images.\n\n        Returns:\n            torch.Tensor: Log probabilities for each class.\n        \"\"\"\n        # Convolution + Activation + Pooling\n        X = F.relu(self.conv1(X))\n        X = F.max_pool2d(X, 2, 2)\n        X = F.relu(self.conv2(X))\n        X = F.max_pool2d(X, 2, 2)\n        X = F.relu(self.conv3(X))\n        X = F.max_pool2d(X, 2, 2)\n        X = F.relu(self.conv4(X))\n        X = F.max_pool2d(X, 2, 2)\n\n        # Flatten feature maps\n        X = X.view(X.shape[0], -1)  \n\n        # Fully Connected Layers\n        X = F.relu(self.fc1(X))\n        X = F.dropout(X, self.dropout_rate)\n        X = self.fc2(X)\n\n        return F.log_softmax(X, dim=1)  # Output log-probabilities","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.781189Z","iopub.execute_input":"2025-02-21T20:24:05.781525Z","iopub.status.idle":"2025-02-21T20:24:05.797971Z","shell.execute_reply.started":"2025-02-21T20:24:05.781482Z","shell.execute_reply":"2025-02-21T20:24:05.796888Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Hyperparameters","metadata":{}},{"cell_type":"markdown","source":"The model is initialized using predefined hyperparameters that define:\n\n- Input shape: (3, 46, 46) corresponding to (channels, height, width)\n- Number of initial filters: 8\n- Fully connected layer size: 100\n- Dropout rate: 0.25\n- Number of output classes: 2 (Cancer vs. Normal)","metadata":{}},{"cell_type":"code","source":"# Define model hyperparameters\nparams_model = {\n    \"shape_in\": (3, 46, 46),  # Input shape: (Channels, Height, Width)\n    \"initial_filters\": 8,      # Number of filters in the first Conv layer\n    \"num_fc1\": 100,            # Number of neurons in the first fully connected layer\n    \"dropout_rate\": 0.25,      # Dropout rate\n    \"num_classes\": 2           # Number of output classes (Cancer, Normal)\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.798970Z","iopub.execute_input":"2025-02-21T20:24:05.799297Z","iopub.status.idle":"2025-02-21T20:24:05.821917Z","shell.execute_reply.started":"2025-02-21T20:24:05.799259Z","shell.execute_reply":"2025-02-21T20:24:05.820639Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Instantiate the CNN model\ncnn_model = Network(params_model)\n\n# Move model to the specified device (CPU or GPU)\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nmodel = cnn_model.to(device)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.823093Z","iopub.execute_input":"2025-02-21T20:24:05.823518Z","iopub.status.idle":"2025-02-21T20:24:05.842034Z","shell.execute_reply.started":"2025-02-21T20:24:05.823474Z","shell.execute_reply":"2025-02-21T20:24:05.840829Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Training","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"This section defines the functions and procedures required to train the CNN model, validate its performance, and visualize training progress","metadata":{}},{"cell_type":"markdown","source":"The function below extracts the current learning rate from the optimizer","metadata":{}},{"cell_type":"code","source":"def get_lr(optimizer):\n    \"\"\"\n    Retrieve the current learning rate from the optimizer.\n\n    Args:\n        optimizer (torch.optim.Optimizer): The optimizer used for training.\n\n    Returns:\n        float: The current learning rate.\n    \"\"\"\n    for param_group in optimizer.param_groups:\n        return param_group['lr']\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.843105Z","iopub.execute_input":"2025-02-21T20:24:05.843500Z","iopub.status.idle":"2025-02-21T20:24:05.857743Z","shell.execute_reply.started":"2025-02-21T20:24:05.843428Z","shell.execute_reply":"2025-02-21T20:24:05.856420Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The function below computes the loss and accuracy for a single batch of data. If an optimizer is provided, it also performs a backward pass and updates model weights","metadata":{}},{"cell_type":"code","source":"def loss_batch(loss_func, output, target, optimizer=None):\n    \"\"\"\n    Computes loss and accuracy for a batch of data.\n\n    Args:\n        loss_func (torch.nn.Module): The loss function.\n        output (torch.Tensor): Model predictions.\n        target (torch.Tensor): Ground truth labels.\n        optimizer (torch.optim.Optimizer, optional): The optimizer for updating model weights. Defaults to None.\n\n    Returns:\n        tuple: (loss value, number of correct predictions)\n    \"\"\"\n    loss = loss_func(output, target)\n    pred = output.argmax(dim=1, keepdim=True)\n    batch_correct = pred.eq(target.view_as(pred)).sum().item()\n\n    if optimizer is not None:\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n\n    return loss.item(), batch_correct\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.858815Z","iopub.execute_input":"2025-02-21T20:24:05.859113Z","iopub.status.idle":"2025-02-21T20:24:05.876844Z","shell.execute_reply.started":"2025-02-21T20:24:05.859084Z","shell.execute_reply":"2025-02-21T20:24:05.875749Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This function evaluates the model’s performance across an entire epoch, iterating over all batches in the dataset","metadata":{}},{"cell_type":"code","source":"def loss_epoch(model, loss_func, dataloader, optimizer=None, device='cpu'):\n    \"\"\"\n    Computes the average loss and accuracy for an entire epoch.\n\n    Args:\n        model: The neural network model.\n        loss_func: Loss function.\n        dataloader: DataLoader providing the dataset.\n        optimizer: Optimizer instance for training; if None, evaluation is assumed.\n        device: Device to run computation on.\n\n    Returns:\n        Tuple of (average loss, average accuracy).\n    \"\"\"\n    model = model.to(device)\n    total_loss = 0.0\n    total_correct = 0\n    total_samples = len(dataloader.dataset)\n\n    for xb, yb, _ in dataloader:\n        xb, yb = xb.to(device), yb.to(device)\n        outputs = model(xb)\n        loss_value, batch_correct = loss_batch(loss_func, outputs, yb, optimizer)\n        total_loss += loss_value\n        total_correct += batch_correct\n\n    avg_loss = total_loss / total_samples\n    avg_accuracy = total_correct / total_samples\n    return avg_loss, avg_accuracy","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.877942Z","iopub.execute_input":"2025-02-21T20:24:05.878234Z","iopub.status.idle":"2025-02-21T20:24:05.899353Z","shell.execute_reply.started":"2025-02-21T20:24:05.878207Z","shell.execute_reply":"2025-02-21T20:24:05.898353Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The following function trains and validates the model over multiple epochs, while also implementing:\n\n- Learning rate scheduling\n- Model checkpointing (saving the best weights)\n- Verbose logging of progress","metadata":{}},{"cell_type":"code","source":"def train_validate(model, params, device='cpu', verbose=True):\n    \"\"\"\n    Trains and validates the model over a specified number of epochs.\n\n    Args:\n        model (torch.nn.Module): The neural network model.\n        params (dict): Dictionary of training parameters.\n        device (str): Device to run computation ('cpu' or 'cuda').\n        verbose (bool): Whether to print training progress.\n\n    Returns:\n        tuple: (trained model, loss history, accuracy history)\n    \"\"\"\n    epochs = params[\"epochs\"]\n    loss_func = params[\"f_loss\"]\n    optimizer = params[\"optimiser\"]\n    train_dl = params[\"train\"]\n    val_dl = params[\"val\"]\n    lr_scheduler = params[\"lr_change\"]\n    weight_path = params[\"weight_path\"]\n\n    loss_history = {\"train\": [], \"val\": []}\n    metric_history = {\"train\": [], \"val\": []}\n\n    best_model_wts = copy.deepcopy(model.state_dict())\n    best_loss = float('inf')\n\n    for epoch in tqdm(range(epochs), desc=\"Training Epochs\"):\n        current_lr = get_lr(optimizer)\n        if verbose:\n            print(f\"Epoch {epoch+1}/{epochs}, Current LR: {current_lr:.6f}\")\n\n        # Training Phase\n        model.train()\n        train_loss, train_accuracy = loss_epoch(model, loss_func, train_dl, optimizer, device)\n        loss_history[\"train\"].append(train_loss)\n        metric_history[\"train\"].append(train_accuracy)\n\n        # Validation Phase\n        model.eval()\n        with torch.no_grad():\n            val_loss, val_accuracy = loss_epoch(model, loss_func, val_dl, None, device)\n        loss_history[\"val\"].append(val_loss)\n        metric_history[\"val\"].append(val_accuracy)\n\n        # Save best model weights\n        if val_loss < best_loss:\n            best_loss = val_loss\n            best_model_wts = copy.deepcopy(model.state_dict())\n            torch.save(model.state_dict(), weight_path)\n            if verbose:\n                print(\"Saved best model weights.\")\n\n        # Adjust learning rate\n        lr_scheduler.step(val_loss)\n        new_lr = get_lr(optimizer)\n        if new_lr != current_lr:\n            if verbose:\n                print(\"LR reduced; reloading best model weights.\")\n            model.load_state_dict(best_model_wts)\n\n        if verbose:\n            print(f\"Train Loss: {train_loss:.6f}, Val Loss: {val_loss:.6f}, Val Accuracy: {100*val_accuracy:.2f}%\")\n            print(\"-\" * 30)\n\n    # Load best model weights after training\n    model.load_state_dict(best_model_wts)\n    return model, loss_history, metric_history","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.900570Z","iopub.execute_input":"2025-02-21T20:24:05.900923Z","iopub.status.idle":"2025-02-21T20:24:05.913728Z","shell.execute_reply.started":"2025-02-21T20:24:05.900896Z","shell.execute_reply":"2025-02-21T20:24:05.912577Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Optimizer & Learning Rate Scheduler","metadata":{}},{"cell_type":"code","source":"# Define optimizer and learning rate scheduler\noptimizer = Adam(cnn_model.parameters(), lr=3e-4)\nlr_scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=20, verbose=True)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.914760Z","iopub.execute_input":"2025-02-21T20:24:05.915104Z","iopub.status.idle":"2025-02-21T20:24:05.934506Z","shell.execute_reply.started":"2025-02-21T20:24:05.915074Z","shell.execute_reply":"2025-02-21T20:24:05.933211Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training Parameters","metadata":{}},{"cell_type":"code","source":"params_train = {\n    \"train\": train_loader,\n    \"val\": val_loader,\n    \"epochs\": 50,\n    \"optimiser\": optimizer,\n    \"lr_change\": lr_scheduler,\n    \"f_loss\": nn.NLLLoss(reduction=\"sum\"),\n    \"weight_path\": \"cnn_weights.pt\",\n}","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.935801Z","iopub.execute_input":"2025-02-21T20:24:05.936175Z","iopub.status.idle":"2025-02-21T20:24:05.954444Z","shell.execute_reply.started":"2025-02-21T20:24:05.936138Z","shell.execute_reply":"2025-02-21T20:24:05.953033Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training the Model","metadata":{}},{"cell_type":"code","source":"trained_model, loss_hist, metric_hist = train_validate(cnn_model, params_train, device=device, verbose=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:05.955876Z","iopub.execute_input":"2025-02-21T20:24:05.956420Z","iopub.status.idle":"2025-02-21T20:24:35.324880Z","shell.execute_reply.started":"2025-02-21T20:24:05.956373Z","shell.execute_reply":"2025-02-21T20:24:35.323591Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training Performance Visualization","metadata":{}},{"cell_type":"markdown","source":"To evaluate training, we plot the loss and accuracy trends over epochs","metadata":{}},{"cell_type":"code","source":"epochs_range = range(1, params_train[\"epochs\"] + 1)\nfig, axes = plt.subplots(1, 2, figsize=(12, 5))\n\n# Loss Convergence Plot\nsns.lineplot(x=list(epochs_range), y=loss_hist[\"train\"], ax=axes[0], label=\"Train Loss\")\nsns.lineplot(x=list(epochs_range), y=loss_hist[\"val\"], ax=axes[0], label=\"Validation Loss\")\naxes[0].set_title(\"Loss Convergence History\")\naxes[0].set_xlabel(\"Epoch\")\naxes[0].set_ylabel(\"Loss\")\n\n# Accuracy Convergence Plot\nsns.lineplot(x=list(epochs_range), y=metric_hist[\"train\"], ax=axes[1], label=\"Train Accuracy\")\nsns.lineplot(x=list(epochs_range), y=metric_hist[\"val\"], ax=axes[1], label=\"Validation Accuracy\")\naxes[1].set_title(\"Accuracy Convergence History\")\naxes[1].set_xlabel(\"Epoch\")\naxes[1].set_ylabel(\"Accuracy\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:35.326297Z","iopub.execute_input":"2025-02-21T20:24:35.326648Z","iopub.status.idle":"2025-02-21T20:24:36.174522Z","shell.execute_reply.started":"2025-02-21T20:24:35.326615Z","shell.execute_reply":"2025-02-21T20:24:36.173371Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Evaluation","metadata":{}},{"cell_type":"markdown","source":"This section defines the functions required to evaluate the CNN model on test data, compare predictions with ground truth labels, and generate detailed performance metrics","metadata":{}},{"cell_type":"markdown","source":"### Perform Inference on the Dataset","metadata":{}},{"cell_type":"code","source":"def generate_predictions(model, dataset, device, batch_size=32):\n    \"\"\"\n    Performs inference on the dataset and returns a dictionary of predictions.\n\n    Args:\n        model (torch.nn.Module): The trained CNN model.\n        dataset (torch.utils.data.Dataset): The dataset for inference.\n        device (str): Device for computation ('cpu' or 'cuda').\n        batch_size (int, optional): Batch size for inference. Defaults to 32.\n\n    Returns:\n        dict: A dictionary where keys are filenames and values are predicted classes.\n    \"\"\"\n    model.to(device)\n    model.eval()  # Set model to evaluation mode\n    dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=False)\n    predictions = {}\n\n    with torch.no_grad():\n        for images, _, filenames in tqdm(dataloader, desc=\"Generating Predictions\"):\n            images = images.to(device)\n            outputs = model(images)\n\n            # Get the index of the highest probability class\n            _, preds = torch.max(outputs, 1)\n\n            # Store predictions with corresponding filenames\n            for filename, pred in zip(filenames, preds.cpu().numpy()):\n                predictions[filename] = pred\n\n    return predictions","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:36.175807Z","iopub.execute_input":"2025-02-21T20:24:36.176201Z","iopub.status.idle":"2025-02-21T20:24:36.182843Z","shell.execute_reply.started":"2025-02-21T20:24:36.176163Z","shell.execute_reply":"2025-02-21T20:24:36.181888Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Load Model Weights","metadata":{}},{"cell_type":"code","source":"# load any model weights for the model\ncnn_model.load_state_dict(torch.load('cnn_weights.pt'))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:36.183779Z","iopub.execute_input":"2025-02-21T20:24:36.184093Z","iopub.status.idle":"2025-02-21T20:24:36.213504Z","shell.execute_reply.started":"2025-02-21T20:24:36.184054Z","shell.execute_reply":"2025-02-21T20:24:36.212481Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Compare Predictions with Ground Truth","metadata":{}},{"cell_type":"markdown","source":"This function evaluates the model by comparing predictions with actual labels from a CSV file. It calculates key performance metrics like accuracy, precision, recall, F1-score, and AUC (Area Under Curve).","metadata":{}},{"cell_type":"code","source":"def evaluate_model(model, dataset, labels_csv, device, batch_size=32):\n    \"\"\"\n    Evaluates the CNN model by comparing predictions with actual labels.\n\n    Args:\n        model (torch.nn.Module): The trained CNN model.\n        dataset (torch.utils.data.Dataset): The dataset for inference.\n        labels_csv (str): Path to CSV file containing ground truth labels. \n                          The CSV must have 'id' and 'label' columns.\n        device (str): Device for inference ('cpu' or 'cuda').\n        batch_size (int, optional): Batch size for inference. Defaults to 32.\n\n    Returns:\n        dict: Dictionary containing evaluation metrics.\n    \"\"\"\n    \n    # Generate model predictions\n    predictions = generate_predictions(model, dataset, device, batch_size=batch_size)\n    \n    # Load ground truth labels from CSV\n    labels_df = pd.read_csv(labels_csv)\n    \n    # Create a dictionary mapping file names to their true labels\n    labels_dict = dict(zip(labels_df['id'], labels_df['label']))\n    \n    y_true = []\n    y_pred = [] \n    \n    # Match predictions with true labels\n    for filename, pred in predictions.items():\n        if filename in labels_dict:\n            y_true.append(labels_dict[filename])\n            y_pred.append(pred)\n    \n    # Convert lists to NumPy arrays for metric calculations\n    y_true = np.array(y_true)\n    y_pred = np.array(y_pred)\n    \n    # Compute key evaluation metrics\n    accuracy = accuracy_score(y_true, y_pred)\n    precision = precision_score(y_true, y_pred)\n    recall = recall_score(y_true, y_pred)\n    f1 = f1_score(y_true, y_pred)\n    auc = roc_auc_score(y_true, y_pred)    \n    \n    print(f\"Accuracy: {accuracy:.4f}\")\n    print(f\"Precision: {precision:.4f}\")\n    print(f\"Recall: {recall:.4f}\")\n    print(f\"F1-score: {f1:.4f}\")\n    print(f\"AUC: {auc:.4f}\")\n    \n    # Generate confusion matrix\n    conf_matrix = confusion_matrix(y_true, y_pred)\n    plt.figure(figsize=(6, 5))\n    sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues', \n                xticklabels=[\"No Tumor\", \"Tumor\"], yticklabels=[\"No Tumor\", \"Tumor\"])\n    plt.xlabel(\"Predicted Label\")\n    plt.ylabel(\"True Label\")\n    plt.title(\"Confusion Matrix\")\n    plt.show()\n    \n    # Print classification report\n    print(\"\\nClassification Report:\\n\", classification_report(y_true, y_pred, target_names=[\"No Tumor\", \"Tumor\"]))\n\n    return {\n        \"accuracy\": accuracy,\n        \"precision\": precision,\n        \"recall\": recall,\n        \"f1_score\": f1,\n        \"auc\": auc,\n    }","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:36.218537Z","iopub.execute_input":"2025-02-21T20:24:36.218852Z","iopub.status.idle":"2025-02-21T20:24:36.230584Z","shell.execute_reply.started":"2025-02-21T20:24:36.218825Z","shell.execute_reply":"2025-02-21T20:24:36.229422Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"labels_csv = \"/kaggle/input/histopathologic-cancer-detection/train_labels.csv\"","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:36.232276Z","iopub.execute_input":"2025-02-21T20:24:36.232637Z","iopub.status.idle":"2025-02-21T20:24:36.254826Z","shell.execute_reply.started":"2025-02-21T20:24:36.232607Z","shell.execute_reply":"2025-02-21T20:24:36.253637Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"metrics = evaluate_model(cnn_model, test_dataset, labels_csv, device)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:36.255867Z","iopub.execute_input":"2025-02-21T20:24:36.256151Z","iopub.status.idle":"2025-02-21T20:24:41.657525Z","shell.execute_reply.started":"2025-02-21T20:24:36.256127Z","shell.execute_reply":"2025-02-21T20:24:41.656574Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Save Evaluation Metrics","metadata":{}},{"cell_type":"code","source":"with open(\"metrics.json\", \"w\") as f:\n    json.dump(metrics, f)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:41.658221Z","iopub.execute_input":"2025-02-21T20:24:41.658510Z","iopub.status.idle":"2025-02-21T20:24:41.663707Z","shell.execute_reply.started":"2025-02-21T20:24:41.658471Z","shell.execute_reply":"2025-02-21T20:24:41.662808Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Predictions","metadata":{}},{"cell_type":"markdown","source":"This section describes how to prepare the dataset for inference, generate predictions, and save the results in a structured CSV format.","metadata":{}},{"cell_type":"markdown","source":"This dataset class:\n- Loads test images from the given directory\n- Applies transformations before inference\n- Returns the image filename for reference","metadata":{}},{"cell_type":"code","source":"class PredictionsDataset(Dataset):\n    \"\"\"\n    Custom dataset class for loading test images for inference.\n\n    Args:\n        data_dir (str): Path to the directory containing test images.\n        transform (torchvision.transforms.Compose): Image transformations.\n\n    Returns:\n        image (Tensor): Transformed image.\n        filename (str): Name of the image file.\n    \"\"\"\n    def __init__(self, data_dir, transform):\n        self.path2data = data_dir\n        self.filenames = os.listdir(self.path2data)  # List all image files\n        self.full_filenames = [os.path.join(self.path2data, f) for f in self.filenames]        \n        self.transform = transform       \n\n    def __len__(self):\n        return len(self.full_filenames)\n\n    def __getitem__(self, idx):\n        image = Image.open(self.full_filenames[idx])  # Open image with PIL\n        image = self.transform(image)  # Apply transformations\n        filename = self.filenames[idx]  # Get filename\n        return image, _, filename  # Return image, placeholder for label and filename","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:41.664700Z","iopub.execute_input":"2025-02-21T20:24:41.665080Z","iopub.status.idle":"2025-02-21T20:24:41.682866Z","shell.execute_reply.started":"2025-02-21T20:24:41.665042Z","shell.execute_reply":"2025-02-21T20:24:41.681820Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data_dir = '/kaggle/input/histopathologic-cancer-detection/test'\n\n# Since the model expects input images in tensor format, we use torchvision.transforms to convert images to tensors\ndata_transformer = transforms.Compose([\n    transforms.ToTensor()\n])\n\ndataset_predictions = PredictionsDataset(data_dir=data_dir, transform=data_transformer)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:41.683760Z","iopub.execute_input":"2025-02-21T20:24:41.684049Z","iopub.status.idle":"2025-02-21T20:24:41.786335Z","shell.execute_reply.started":"2025-02-21T20:24:41.684012Z","shell.execute_reply":"2025-02-21T20:24:41.785330Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"predictions = generate_predictions(cnn_model, dataset_predictions, device)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:24:41.787246Z","iopub.execute_input":"2025-02-21T20:24:41.787521Z","iopub.status.idle":"2025-02-21T20:28:54.696133Z","shell.execute_reply.started":"2025-02-21T20:24:41.787497Z","shell.execute_reply":"2025-02-21T20:28:54.694874Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"df = pd.DataFrame(list(predictions.items()), columns=[\"id\", \"label\"])\ndf[\"id\"] = df[\"id\"].apply(lambda x: x.replace(\".tif\", \"\"))\ndf.to_csv(\"prediction.csv\", index=False)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:28:54.697321Z","iopub.execute_input":"2025-02-21T20:28:54.697709Z","iopub.status.idle":"2025-02-21T20:28:54.947605Z","shell.execute_reply.started":"2025-02-21T20:28:54.697679Z","shell.execute_reply":"2025-02-21T20:28:54.946370Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Upload to huggingface","metadata":{}},{"cell_type":"code","source":"from kaggle_secrets import UserSecretsClient\nuser_secrets = UserSecretsClient()\nsecret_value_0 = user_secrets.get_secret(\"huggingface\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:28:54.948685Z","iopub.execute_input":"2025-02-21T20:28:54.949083Z","iopub.status.idle":"2025-02-21T20:28:55.105308Z","shell.execute_reply.started":"2025-02-21T20:28:54.949049Z","shell.execute_reply":"2025-02-21T20:28:55.104053Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from huggingface_hub import login\n\nlogin(token=secret_value_0)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:28:55.106306Z","iopub.execute_input":"2025-02-21T20:28:55.106728Z","iopub.status.idle":"2025-02-21T20:28:55.318493Z","shell.execute_reply.started":"2025-02-21T20:28:55.106685Z","shell.execute_reply":"2025-02-21T20:28:55.317596Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Upload model","metadata":{}},{"cell_type":"code","source":"from huggingface_hub import HfApi\n\nusername = \"maiurilorenzo\"\nrepo_name = \"histoplastic-cancer-CNN-classifier\"\n\napi = HfApi()\napi.create_repo(f\"{username}/{repo_name}\", exist_ok=True)  # Create repo if it doesn't exist\n\n# Upload the model file\napi.upload_file(\n    path_or_fileobj=\"cnn_weights.pt\",\n    path_in_repo=\"cnn_weights.pt\",\n    repo_id=f\"{username}/{repo_name}\"\n)\n\napi.upload_file(\n    path_or_fileobj=\"metrics.json\",\n    path_in_repo=\"metrics.json\",\n    repo_id=f\"{username}/{repo_name}\"\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:28:55.319442Z","iopub.execute_input":"2025-02-21T20:28:55.319787Z","iopub.status.idle":"2025-02-21T20:28:57.091009Z","shell.execute_reply.started":"2025-02-21T20:28:55.319744Z","shell.execute_reply":"2025-02-21T20:28:57.090014Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Test the uploaded model","metadata":{}},{"cell_type":"code","source":"from huggingface_hub import hf_hub_download\nimport tensorflow as tf\nimport cv2\nimport numpy as np\nimport json\nimport matplotlib.pyplot as plt\n\n# Load model\nrepo_id = f\"{username}/{repo_name}\"\n\nmodel_path = hf_hub_download(repo_id=repo_id, filename=\"cnn_weights.pt\")\n\n# Neural Network Predefined Parameters\nparams_model={\n        \"shape_in\": (3,46,46), \n        \"initial_filters\": 8,    \n        \"num_fc1\": 100,\n        \"dropout_rate\": 0.25,\n        \"num_classes\": 2}\n\n# Create instantiation of Network class\ncnn_model = Network(params_model)\n\ncnn_model.load_state_dict(torch.load('cnn_weights.pt'))\n\npredictions = generate_predictions(model, dataset_predictions, device)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:28:57.091984Z","iopub.execute_input":"2025-02-21T20:28:57.092250Z","iopub.status.idle":"2025-02-21T20:32:14.986024Z","shell.execute_reply.started":"2025-02-21T20:28:57.092228Z","shell.execute_reply":"2025-02-21T20:32:14.984735Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data_dir = '/kaggle/input/histopathologic-cancer-detection'","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:32:14.987235Z","iopub.execute_input":"2025-02-21T20:32:14.987594Z","iopub.status.idle":"2025-02-21T20:32:14.991699Z","shell.execute_reply.started":"2025-02-21T20:32:14.987563Z","shell.execute_reply":"2025-02-21T20:32:14.990672Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plot_sample_images(\n    [(filename.replace(\".tif\", \"\"), label) for filename, label in predictions.items()], \n    data_dir, \n    data_type=\"test\", \n    title=\"Test Set Samples\"\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-02-21T20:32:14.992579Z","iopub.execute_input":"2025-02-21T20:32:14.992910Z","iopub.status.idle":"2025-02-21T20:32:16.863105Z","shell.execute_reply.started":"2025-02-21T20:32:14.992884Z","shell.execute_reply":"2025-02-21T20:32:16.861960Z"}},"outputs":[],"execution_count":null}]}