{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":11848,"databundleVersionId":862157,"sourceType":"competition"}],"dockerImageVersionId":31041,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"id":"2145536b","cell_type":"markdown","source":"# Dinov2 PCam Classification\n\nThis notebook demonstrates image classification on the PatchCamelyon (PCam) dataset using a self-supervised Dinov2 model. We will cover cloning the Dinov2 repository, installing dependencies, downloading and preparing the PCam dataset, performing exploratory data analysis, preprocessing the data, loading the Dinov2 model, training the model, evaluating its performance, and saving the trained model.","metadata":{}},{"id":"12fdd5e4","cell_type":"markdown","source":"## 1. Clone Dinov2 Repository and Install Dependencies\n\nFirst, we need to clone the Dinov2 repository from GitHub and install the necessary dependencies. This includes PyTorch, torchvision, timm, scikit-learn, matplotlib, opendatasets, and pandas.","metadata":{}},{"id":"9c76ab9d","cell_type":"code","source":"# Clone Dinov2 repository\n!git clone https://github.com/facebookresearch/dinov2.git\n\n# Install other packages\n!pip install timm scikit-learn matplotlib opendatasets pandas\n\nimport os\nimport sys\nsys.path.append('./dinov2') # Add dinov2 to system path","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:12:44.908693Z","iopub.execute_input":"2025-06-10T05:12:44.908925Z","iopub.status.idle":"2025-06-10T05:12:48.643286Z","shell.execute_reply.started":"2025-06-10T05:12:44.908894Z","shell.execute_reply":"2025-06-10T05:12:48.642517Z"},"collapsed":true,"jupyter":{"outputs_hidden":true}},"outputs":[],"execution_count":null},{"id":"c59ade9b","cell_type":"markdown","source":"## 2. Download and Prepare PCam Dataset\n\nNext, we download the PCam dataset using the `opendatasets` library. After downloading, we will extract the dataset and organize it into training, validation, and test sets.","metadata":{}},{"id":"1c6f2569","cell_type":"code","source":"import opendatasets as od\nimport pandas as pd\nimport shutil\nfrom sklearn.model_selection import train_test_split\nimport os # Ensure os is imported if not already\n\n# The dataset is expected to be available in the Kaggle input directory\n# # Download PCam dataset (commented out as data is expected in Kaggle environment)\n# dataset_url = 'https://www.kaggle.com/competitions/histopathologic-cancer-detection/data'\n# od.download(dataset_url, data_dir='./pcam_dataset') # This line would create ./pcam_dataset\n\n# Define paths assuming data is in /kaggle/input/\n# If running locally after download, adjust this base_dir accordingly.\n# For Kaggle environment, it's typically /kaggle/input/<dataset-folder-name>\nbase_dir = '/kaggle/input/histopathologic-cancer-detection' \n# If you downloaded it locally using the above od.download, base_dir would be './pcam_dataset/histopathologic-cancer-detection'\n\ntrain_labels_path = os.path.join(base_dir, 'train_labels.csv')\ntrain_images_dir = os.path.join(base_dir, 'train')\ntest_images_dir = os.path.join(base_dir, 'test') # This is the unlabeled test set from Kaggle\n\n# Load labels\ntry:\n    train_labels_df = pd.read_csv(train_labels_path)\nexcept FileNotFoundError:\n    print(f\"Error: train_labels.csv not found at {train_labels_path}\")\n    print(\"Please ensure the 'base_dir' variable is set correctly to your dataset location.\")\n    print(\"If running on Kaggle, the path should be /kaggle/input/histopathologic-cancer-detection/train_labels.csv\")\n    # Example for local download:\n    # print(\"If you downloaded data locally to './pcam_dataset', set: base_dir = './pcam_dataset/histopathologic-cancer-detection'\")\n    raise\n\n# Split training data into train and validation sets (using the labels dataframe)\ntrain_df, val_df = train_test_split(train_labels_df, test_size=0.2, stratify=train_labels_df['label'], random_state=42)\n\nprint(f\"Total training samples: {len(train_labels_df)}\")\nprint(f\"Number of training samples after split: {len(train_df)}\")\nprint(f\"Number of validation samples: {len(val_df)}\")\nprint(f\"Training images directory: {train_images_dir}\")\nprint(f\"Test images directory (unlabeled): {test_images_dir}\")\n\n# The pcam_organized directory and file copying are no longer needed.\n# Images will be loaded directly from train_images_dir using a custom Dataset.\n\nprint(\"Dataset preparation complete (labels loaded and split). File copying skipped.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:12:48.647765Z","iopub.execute_input":"2025-06-10T05:12:48.648013Z","iopub.status.idle":"2025-06-10T05:12:50.267160Z","shell.execute_reply.started":"2025-06-10T05:12:48.647984Z","shell.execute_reply":"2025-06-10T05:12:50.266101Z"}},"outputs":[],"execution_count":null},{"id":"ce43741f","cell_type":"markdown","source":"## 3. Perform Exploratory Data Analysis (EDA)\n\nIn this section, we will analyze the PCam dataset. This includes visualizing sample images from each class, checking the class distribution, and calculating basic statistics about the images.","metadata":{}},{"id":"425ddba4-c7b1-4f94-9190-2d87511a5cc9","cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Set plot style\nsns.set(style=\"whitegrid\")\n\n# 1. Bar plot of class distribution (Benign vs Malignant)\nplt.figure(figsize=(6, 4))\nsns.countplot(x='label', data=train_df, palette='pastel')\nplt.title('Label Distribution in Training Set')\nplt.xticks([0, 1], ['Benign (0)', 'Malignant (1)'])\nplt.xlabel('Label')\nplt.ylabel('Number of Samples')\nplt.show()\n\n# 2. Pie chart for class proportion\nlabel_counts = train_df['label'].value_counts()\nplt.figure(figsize=(5, 5))\nplt.pie(label_counts, labels=['Benign (0)', 'Malignant (1)'], autopct='%1.1f%%', colors=['skyblue', 'lightcoral'])\nplt.title('Class Proportion (Benign vs Malignant)')\nplt.show()\n\n# 3. Bar plot showing class distribution in training vs validation set\nsplit_dist = pd.DataFrame({\n    'Training': train_df['label'].value_counts(),\n    'Validation': val_df['label'].value_counts()\n}).T\n\nsplit_dist.plot(kind='bar', stacked=True, figsize=(8, 5), colormap='coolwarm')\nplt.title('Class Distribution in Training vs Validation Set')\nplt.xlabel('Dataset Split')\nplt.ylabel('Number of Samples')\nplt.xticks(rotation=0)\nplt.legend(['Benign (0)', 'Malignant (1)'])\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:12:50.268244Z","iopub.execute_input":"2025-06-10T05:12:50.268649Z","iopub.status.idle":"2025-06-10T05:12:51.424292Z","shell.execute_reply.started":"2025-06-10T05:12:50.268606Z","shell.execute_reply":"2025-06-10T05:12:51.423552Z"}},"outputs":[],"execution_count":null},{"id":"4bd64591","cell_type":"code","source":"import matplotlib.pyplot as plt\nimport random\nfrom PIL import Image\nimport numpy as np\nimport os\nimport pandas as pd\n\n# train_images_dir, train_df, val_df, and train_labels_df should be defined from running cell 5.\n# The line that previously redefined train_images_dir here has been removed.\n\n# Function to display sample images\ndef display_sample_images(df, image_dir, num_samples=2):\n    plt.figure(figsize=(10, 5 * num_samples // 2))\n    # Get samples from both classes if possible\n    samples_class_0 = df[df['label'] == 0].sample(min(num_samples // 2, len(df[df['label'] == 0])))\n    samples_class_1 = df[df['label'] == 1].sample(min(num_samples // 2, len(df[df['label'] == 1])))\n    \n    samples_to_display = pd.concat([samples_class_0, samples_class_1])\n    if samples_to_display.empty:\n        print(\"No samples to display. Check your DataFrame and image directory.\")\n        return\n\n    for i, (idx, row) in enumerate(samples_to_display.iterrows()):\n        img_id = row['id']\n        label = row['label']\n        img_path = os.path.join(image_dir, f'{img_id}.tif')\n        \n        try:\n            img = Image.open(img_path)\n            plt.subplot(num_samples // 2, 2, i + 1)\n            plt.imshow(img)\n            plt.title(f\"ID: {img_id}\\nClass: {label} ({'No Cancer' if label == 0 else 'Cancer'})\")\n            plt.axis('off')\n        except FileNotFoundError:\n            print(f\"Image not found: {img_path}\")\n        except Exception as e:\n            print(f\"Error loading image {img_path}: {e}\")\n    plt.tight_layout()\n    plt.show()\n\nprint(\"Displaying sample images from training set:\")\n# Display 4 samples (2 from each class if available)\n# This will now use train_images_dir from cell 5\ndisplay_sample_images(train_df, train_images_dir, num_samples=4) \n\n# Check class distribution using the DataFrame\nprint(\"\\nClass distribution in the original training labels:\")\nprint(train_labels_df['label'].value_counts())\n\nprint(\"\\nClass distribution in the split training set (train_df):\")\nprint(train_df['label'].value_counts())\n\nprint(\"\\nClass distribution in the split validation set (val_df):\")\nprint(val_df['label'].value_counts())\n\n# Basic image statistics (dimensions of a few samples)\nprint(\"\\nBasic image statistics (dimensions from a few samples in train_df):\")\nif not train_df.empty:\n    for i in range(min(4, len(train_df))): # Check up to 4 images\n        sample_row = train_df.sample(1).iloc[0]\n        img_id = sample_row['id']\n        label = sample_row['label']\n        # This will now use train_images_dir from cell 5\n        img_path = os.path.join(train_images_dir, f'{img_id}.tif')\n        try:\n            img = Image.open(img_path)\n            print(f\"Image ID: {img_id}, Class: {label}, Dimensions: {img.size}, Mode: {img.mode}\")\n        except FileNotFoundError:\n            print(f\"Image not found: {img_path}\")\n        except Exception as e:\n            print(f\"Error loading image {img_path}: {e}\")\nelse:\n    print(\"train_df is empty, cannot display image statistics.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:12:51.425132Z","iopub.execute_input":"2025-06-10T05:12:51.425522Z","iopub.status.idle":"2025-06-10T05:12:54.288780Z","shell.execute_reply.started":"2025-06-10T05:12:51.425504Z","shell.execute_reply":"2025-06-10T05:12:54.287962Z"}},"outputs":[],"execution_count":null},{"id":"d60ba86b-2b9c-44aa-bfaf-1d9a0276dad6","cell_type":"code","source":"### show sample image files\nimport cv2\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom pathlib import Path\n\ntrain_dir = Path(\"/kaggle/input/histopathologic-cancer-detection/train\")\ntrain_labels = pd.read_csv('/kaggle/input/histopathologic-cancer-detection/train_labels.csv')\ntrain_labels['path'] = train_labels['id'].apply(lambda x: str(train_dir / f'{x}.tif'))\n\ndef show_samples(label, df=train_labels, num_images=5):\n    subset = df[df['label'] == label].sample(num_images, random_state=42)\n    plt.figure(figsize=(12, 4))\n    for i, row in enumerate(subset.itertuples()):\n        img = cv2.imread(row.path)\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        center = (32, 32, 64, 64)  \n        cv2.rectangle(img, (center[0], center[1]), (center[2], center[3]), (255, 0, 0), 1)\n        plt.subplot(1, num_images, i+1)\n        plt.imshow(img)\n        plt.title(f'Label: {row.label}')\n        plt.axis('off')\n    plt.show()\n\n\nshow_samples(0)  # Non-cancer\nshow_samples(1)  # Cancer\n\n# check image size and channel\nimg_path = f\"/kaggle/input/histopathologic-cancer-detection/train/{train_labels['id'].iloc[0]}.tif\"\nimg = Image.open(img_path)\nprint(f\"image size: {img.size}\") \nprint(f\"image mode: {img.mode}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:15:59.001587Z","iopub.execute_input":"2025-06-10T05:15:59.001963Z","iopub.status.idle":"2025-06-10T05:16:01.525645Z","shell.execute_reply.started":"2025-06-10T05:15:59.001937Z","shell.execute_reply":"2025-06-10T05:16:01.524690Z"}},"outputs":[],"execution_count":null},{"id":"968c6e36","cell_type":"markdown","source":"## 4. Preprocess Dataset for Dinov2\n\nNow, we preprocess the dataset to be compatible with the Dinov2 model. This involves resizing the images, normalizing pixel values according to Dinov2's requirements, and creating PyTorch DataLoaders for efficient batching during training and evaluation.","metadata":{}},{"id":"798a1b06","cell_type":"code","source":"import torch\nfrom torchvision import transforms # Removed datasets as ImageFolder is not used\nfrom torch.utils.data import DataLoader, Dataset # Added Dataset\nfrom PIL import Image # For loading images in the custom dataset\nimport os # Ensure os is imported\nimport pandas as pd # Ensure pandas is imported for DataFrame operations\nfrom sklearn.model_selection import train_test_split # For stratified splitting if needed\n\n# train_images_dir, train_df, and val_df are defined in cell 5\n# However, we will now re-create train_df and val_df from the original train_labels_df for subsetting\n# IMG_SIZE is defined here, or could be moved to a config section\nIMG_SIZE = 224 \n\n# --- Subsetting the dataset ---\n# Load the original full training labels dataframe again to ensure we start fresh for subsetting\noriginal_train_labels_df = pd.read_csv(os.path.join(base_dir, 'train_labels.csv'))\n\n# Define desired number of samples\nnum_total_samples = 30000\ntrain_ratio = 0.8\nnum_train_samples_target = int(num_total_samples * train_ratio) # 2400\nnum_val_samples_target = num_total_samples - num_train_samples_target # 600\n\nprint(f\"Targeting {num_train_samples_target} training samples and {num_val_samples_target} validation samples.\")\n\n# Create a smaller, stratified subset for training\n# First, ensure we have enough samples in each class for the target split\n# If not, we might need to adjust or take all available for the smaller class\n# For simplicity, we'll try to sample proportionally. If a class has fewer than target/2, it will take all of them.\n\n# Stratified split from the original_train_labels_df to get a pool of 3000 images first\nif len(original_train_labels_df) > num_total_samples:\n    pool_df, _ = train_test_split(\n        original_train_labels_df, \n        train_size=num_total_samples, \n        stratify=original_train_labels_df['label'], \n        random_state=42\n    )\nelse:\n    pool_df = original_train_labels_df # Use all data if it's less than 3000\n\n# Now, split this pool_df into train_df and val_df\ntrain_df, val_df = train_test_split(\n    pool_df, \n    train_size=num_train_samples_target, # Target for train from the pool\n    stratify=pool_df['label'], \n    random_state=42\n)\n\n# Adjust val_df size if the split didn't perfectly match num_val_samples_target due to pool size\nif len(val_df) > num_val_samples_target:\n    val_df = val_df.sample(n=num_val_samples_target, random_state=42) # Further subsample if val is too large\n\ntrain_df = train_df.reset_index(drop=True)\nval_df = val_df.reset_index(drop=True)\n\nprint(f\"Actual subset sizes: {len(train_df)} training images, {len(val_df)} validation images.\")\nprint(\"Training set class distribution:\")\nprint(train_df['label'].value_counts())\nprint(\"Validation set class distribution:\")\nprint(val_df['label'].value_counts())\n# --- End of subsetting ---\n\n\n# Define a custom Dataset\nclass PCamDataset(Dataset):\n    def __init__(self, dataframe, image_dir, transform=None):\n        \"\"\"\n        Args:\n            dataframe (pd.DataFrame): DataFrame with 'id' and 'label' columns.\n            image_dir (str): Directory with all the images.\n            transform (callable, optional): Optional transform to be applied on a sample.\n        \"\"\"\n        self.dataframe = dataframe\n        self.image_dir = image_dir\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.dataframe)\n\n    def __getitem__(self, idx):\n        if torch.is_tensor(idx):\n            idx = idx.tolist()\n\n        img_id = self.dataframe.iloc[idx, self.dataframe.columns.get_loc('id')]\n        label = self.dataframe.iloc[idx, self.dataframe.columns.get_loc('label')]\n        \n        img_name = os.path.join(self.image_dir, f\"{img_id}.tif\")\n        \n        try:\n            image = Image.open(img_name).convert(\"RGB\") # Ensure image is RGB\n        except FileNotFoundError:\n            print(f\"ERROR: Image not found at {img_name} for id {img_id}. Check image_dir and dataframe.\")\n            # Return a placeholder or raise an error. For now, let's raise it to stop execution.\n            raise FileNotFoundError(f\"Image not found: {img_name}\")\n        except Exception as e:\n            print(f\"ERROR: Could not load image {img_name}: {e}\")\n            raise\n\n        if self.transform:\n            image = self.transform(image)\n        \n        return image, int(label)\n\n# Dinov2 preprocessing / Standard ImageNet normalization\ndata_transforms = {\n    'train': transforms.Compose([\n        transforms.Resize((IMG_SIZE, IMG_SIZE)),\n        transforms.RandomHorizontalFlip(),\n        transforms.RandomRotation(10),\n        transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ]),\n    'val': transforms.Compose([\n        transforms.Resize((IMG_SIZE, IMG_SIZE)),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ]),\n    # Transform for the unlabeled test set (if used for prediction later)\n    'test_data': transforms.Compose([\n        transforms.Resize((IMG_SIZE, IMG_SIZE)),\n        transforms.ToTensor(),\n        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n    ]),\n}\n\n# Create custom datasets using the new subsetted train_df and val_df\n# train_images_dir should be /kaggle/input/histopathologic-cancer-detection/train\nimage_datasets = {\n    'train': PCamDataset(dataframe=train_df, image_dir=train_images_dir, transform=data_transforms['train']),\n    'val': PCamDataset(dataframe=val_df, image_dir=train_images_dir, transform=data_transforms['val'])\n}\n\n# Create DataLoaders\nBATCH_SIZE = 32 # Adjust based on your GPU memory. \n                # If using multiple GPUs (e.g., with DataParallel), you might be able to increase this.\n                # With a smaller dataset (3000 images), 32 should be fine, might even increase if memory allows.\ndataloaders = {x: DataLoader(image_datasets[x],\n                                batch_size=BATCH_SIZE,\n                                shuffle=True if x == 'train' else False,\n                                num_workers=2) # Adjust num_workers based on your system\n               for x in ['train', 'val']}\n\ndataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}\n# Class names are implicitly [0, 1] based on the labels in the CSV.\n# If you need explicit class names for plotting or reports later, define them:\nclass_names = ['0', '1'] # Or ['No Cancer', 'Cancer'] if preferred for display\n\nprint(f\"Class names used: {class_names}\")\nprint(f\"Dataset sizes: Train: {dataset_sizes['train']}, Val: {dataset_sizes['val']}\")\nprint(f\"Number of training batches: {len(dataloaders['train'])}\")\nprint(f\"Number of validation batches: {len(dataloaders['val'])}\")\n\n# For now, we will set test_dataloader to None as it's not strictly needed for training/validation loop\ntest_dataloader = None\nprint(\"test_dataloader is set to None. Modify above if predictions on Kaggle test set are needed.\")\n\n# Verify a sample from the dataloader\nif dataloaders['train'] and len(dataloaders['train']) > 0:\n    try:\n        sample_inputs, sample_labels = next(iter(dataloaders['train']))\n        print(f\"Sample batch - Inputs shape: {sample_inputs.shape}, Labels shape: {sample_labels.shape}\")\n    except Exception as e:\n        print(f\"Error when trying to get a sample batch from train dataloader: {e}\")\n        print(\"This might indicate an issue with PCamDataset or the underlying data.\")\nelif not dataloaders['train']:\n    print(\"Train dataloader is not initialized.\")\nelse:\n    print(\"Train dataloader is empty. This likely means the training dataset (train_df) is empty after subsetting.\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:16:30.375197Z","iopub.execute_input":"2025-06-10T05:16:30.375890Z","iopub.status.idle":"2025-06-10T05:16:39.181577Z","shell.execute_reply.started":"2025-06-10T05:16:30.375863Z","shell.execute_reply":"2025-06-10T05:16:39.180720Z"}},"outputs":[],"execution_count":null},{"id":"f24dc030","cell_type":"markdown","source":"## 5. Load Dinov2 Model\n\nWe will load a pre-trained Dinov2 model. Dinov2 models are powerful vision transformers trained using self-supervised learning. We can use them as feature extractors or fine-tune them for specific downstream tasks like PCam classification. Here, we'll load a `dinov2_vits14` model and adapt its classification head for our binary classification problem.","metadata":{}},{"id":"6bd826aa","cell_type":"code","source":"import torch.nn as nn\n\n# Load a pre-trained Dinov2 model\n# Available models: dinov2_vits14, dinov2_vitb14, dinov2_vitl14, dinov2_vitg14\n# We'll use dinov2_vits14 as an example. It's smaller and faster to train.\ndinov2_model_name = 'dinov2_vits14' # Small model\n# dinov2_model_name = 'dinov2_vitb14' # Base model\n\n# Using torch.hub to load the model\n# Ensure you have the dinov2 repository cloned and in your sys.path as done in step 1\ntry:\n    model = torch.hub.load('facebookresearch/dinov2', dinov2_model_name)\nexcept Exception as e:\n    print(f\"Error loading model from torch.hub: {e}\")\n    print(\"Make sure the dinov2 repository is correctly cloned and accessible.\")\n    # Fallback or alternative loading if needed, e.g. using timm if the model is available there\n    # import timm\n    # model = timm.create_model('vit_small_patch14_dinov2.lvd142m', pretrained=True)\n    raise e\n\n# Dinov2 models from torch.hub usually don't have a classification head suitable for direct fine-tuning\n# or the head is for ImageNet (1000 classes). We need to replace it or use the model as a feature extractor.\n\n# Option 1: Use as a feature extractor and add a new classifier head\n# The feature dimension for dinov2_vits14 is 384\n# For dinov2_vitb14, it's 768\nfeature_dim = model.embed_dim # This should give the feature dimension\n\n# Freeze the backbone (Dinov2) parameters\nfor param in model.parameters():\n    param.requires_grad = False\n\n# Define a new classification head\nnum_classes = len(class_names) # Should be 2 for PCam (Cancer/No Cancer)\nclassifier_head = nn.Linear(feature_dim, num_classes)\n\n# Combine the Dinov2 backbone with the new head\n# Some Dinov2 hub models might return features directly, others might have a `head` attribute.\n# If it has a `head` attribute, we replace it.\nif hasattr(model, 'head') and isinstance(model.head, nn.Linear):\n    model.head = classifier_head\n    print(f\"Replaced model.head with a new Linear layer for {num_classes} classes.\")\nelse:\n    # If no head attribute or it's not what we expect, create a sequential model\n    # This assumes the base Dinov2 model outputs features directly from its forward pass\n    # or we need to call a specific method like `model.forward_features()`\n    # For torch.hub.load('facebookresearch/dinov2', ...), the model itself is the backbone.\n    # We need to wrap it if we want to append a head and treat it as a single nn.Module for training.\n    class Dinov2WithHead(nn.Module):\n        def __init__(self, backbone, head):\n            super().__init__()\n            self.backbone = backbone\n            self.head = head\n        \n        def forward(self, x):\n            # The backbone might return a dict or a tensor\n            # For Dinov2, it often returns a dict like {'x_norm_patchtokens': ..., 'x_norm_clstoken': ...}\n            # We are interested in the CLS token for classification\n            features = self.backbone.forward_features(x)\n            # Try to get CLS token, otherwise use patch tokens (might need pooling)\n            if isinstance(features, dict) and 'x_norm_clstoken' in features:\n                cls_token = features['x_norm_clstoken']\n            elif isinstance(features, dict) and 'x_norm_patchtokens' in features: # Fallback: average patch tokens\n                patch_tokens = features['x_norm_patchtokens']\n                cls_token = torch.mean(patch_tokens, dim=1) # Average pool patch tokens\n            elif torch.is_tensor(features):\n                # If it's a tensor, assume it's [batch_size, num_tokens, embed_dim]\n                # and the first token is the CLS token, or we average pool\n                # This depends on the specific Dinov2 variant from torch.hub\n                # For safety, let's assume we need to average if it's not a dict with clstoken\n                if features.ndim == 3:\n                    cls_token = torch.mean(features, dim=1)\n                else:\n                    cls_token = features # Assume it's already [batch_size, embed_dim]\n            else:\n                raise ValueError(f\"Unexpected feature output format from Dinov2 backbone: {type(features)}\")\n            \n            return self.head(cls_token)\n\n    model = Dinov2WithHead(model, classifier_head)\n    print(f\"Wrapped Dinov2 backbone with a new Linear layer for {num_classes} classes.\")\n\n# Move model to GPU if available\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nmodel = model.to(device)\n\n# Check for multiple GPUs and use DataParallel\nif torch.cuda.device_count() > 1:\n    print(f\"Let's use {torch.cuda.device_count()} GPUs!\")\n    model = nn.DataParallel(model)\n    print(\"Model wrapped with nn.DataParallel.\")\n\nprint(f\"Model: {dinov2_model_name} loaded and adapted for {num_classes}-class classification.\")\nprint(f\"Using device: {device}\")\n# print(model) # Uncomment to see model structure","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:16:39.183195Z","iopub.execute_input":"2025-06-10T05:16:39.183451Z","iopub.status.idle":"2025-06-10T05:16:41.701794Z","shell.execute_reply.started":"2025-06-10T05:16:39.183426Z","shell.execute_reply":"2025-06-10T05:16:41.700904Z"}},"outputs":[],"execution_count":null},{"id":"5091dd10","cell_type":"markdown","source":"## 6. Train Dinov2 on PCam Dataset\n\nIn this section, we will train (fine-tune the head of) the Dinov2 model on the PCam dataset. We will define a loss function (CrossEntropyLoss for classification) and an optimizer (AdamW). The training loop will iterate through the data, perform forward and backward passes, and update the model weights. We will also monitor validation accuracy to save the best performing model.","metadata":{}},{"id":"d27d3aa6","cell_type":"code","source":"import torch.optim as optim\nfrom torch.optim import lr_scheduler\nimport time\nimport copy\nimport torch.nn as nn # Ensure nn is imported for DataParallel check\n\n# Define loss function and optimizer\n\n# Access the original model if wrapped by DataParallel\noriginal_model = model.module if isinstance(model, nn.DataParallel) else model\n\n# Only training the classifier head, so pass only its parameters to the optimizer\nif isinstance(original_model, Dinov2WithHead):\n    optimizer = optim.AdamW(original_model.head.parameters(), lr=1e-3, weight_decay=1e-4)\n    print(\"Optimizing only the parameters of the custom classifier head (original_model.head).\")\nelif hasattr(original_model, 'head') and isinstance(original_model.head, nn.Linear):\n    optimizer = optim.AdamW(original_model.head.parameters(), lr=1e-3, weight_decay=1e-4)\n    print(\"Optimizing only the parameters of original_model.head.\")\nelse:\n    # This case should ideally not be hit if model adaptation was correct\n    optimizer = optim.AdamW(filter(lambda p: p.requires_grad, original_model.parameters()), lr=1e-3, weight_decay=1e-4)\n    print(\"Warning: Optimizing all parameters of original_model with requires_grad=True. Ensure this is intended.\")\n\ncriterion = nn.CrossEntropyLoss()\n\n# Learning rate scheduler (optional, but often helpful)\n# Reduce learning rate when a metric has stopped improving\nscheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)\n\n# Training function\ndef train_model(model, criterion, optimizer, scheduler, num_epochs=25):\n    overall_start_time = time.time() # Record start time for overall training\n    best_model_wts = copy.deepcopy(model.state_dict())\n    best_acc = 0.0\n\n    history = {'train_loss': [], 'train_acc': [], 'val_loss': [], 'val_acc': []}\n\n    for epoch in range(num_epochs):\n        epoch_start_time = time.time() # Record start time for the current epoch\n        print(f'Epoch {epoch+1}/{num_epochs}')\n        print('-' * 10)\n\n        # Each epoch has a training and validation phase\n        for phase in ['train', 'val']:\n            if phase == 'train':\n                model.train()  # Set model to training mode\n            else:\n                model.eval()   # Set model to evaluate mode\n\n            running_loss = 0.0\n            running_corrects = 0\n\n            # Iterate over data.\n            for inputs, labels in dataloaders[phase]:\n                inputs = inputs.to(device)\n                labels = labels.to(device)\n\n                # Zero the parameter gradients\n                optimizer.zero_grad()\n\n                # Forward\n                # Track history if only in train\n                with torch.set_grad_enabled(phase == 'train'):\n                    outputs = model(inputs)\n                    _, preds = torch.max(outputs, 1)\n                    loss = criterion(outputs, labels)\n\n                    # Backward + optimize only if in training phase\n                    if phase == 'train':\n                        loss.backward()\n                        optimizer.step()\n\n                # Statistics\n                running_loss += loss.item() * inputs.size(0)\n                running_corrects += torch.sum(preds == labels.data)\n            \n            if phase == 'train':\n                scheduler.step() # Step the learning rate scheduler\n\n            epoch_loss = running_loss / dataset_sizes[phase]\n            epoch_acc = running_corrects.double() / dataset_sizes[phase]\n\n            history[f'{phase}_loss'].append(epoch_loss)\n            history[f'{phase}_acc'].append(epoch_acc.item()) # .item() to get Python number\n\n            print(f'{phase} Loss: {epoch_loss:.4f} Acc: {epoch_acc:.4f}')\n\n            # Deep copy the model if it's the best validation accuracy so far\n            if phase == 'val' and epoch_acc > best_acc:\n                best_acc = epoch_acc\n                best_model_wts = copy.deepcopy(model.state_dict())\n                # Save the best model checkpoint\n                torch.save(model.state_dict(), 'dinov2_pcam_best_model.pth')\n                print(f\"Best validation accuracy improved to {best_acc:.4f}. Model saved.\")\n        \n        epoch_time_elapsed = time.time() - epoch_start_time\n        print(f\"Epoch {epoch+1} completed in {epoch_time_elapsed // 60:.0f}m {epoch_time_elapsed % 60:.0f}s\")\n        print()\n\n    overall_time_elapsed = time.time() - overall_start_time\n    print(f'Overall training complete in {overall_time_elapsed // 60:.0f}m {overall_time_elapsed % 60:.0f}s')\n    print(f'Best val Acc: {best_acc:4f}')\n\n    # Load best model weights\n    model.load_state_dict(best_model_wts)\n    return model, history\n\n# Start training\n# Consider a small number of epochs first to ensure everything runs correctly.\nNUM_EPOCHS = 10 # Adjust as needed. For good results, more epochs might be required.\n\nprint(\"Starting model training...\")\nmodel_ft, training_history = train_model(model, criterion, optimizer, scheduler, num_epochs=NUM_EPOCHS)\n\n# Plot training history\nplt.figure(figsize=(12, 4))\nplt.subplot(1, 2, 1)\nplt.plot(training_history['train_loss'], label='Train Loss')\nplt.plot(training_history['val_loss'], label='Val Loss')\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.legend()\nplt.title('Loss Over Epochs')\n\nplt.subplot(1, 2, 2)\nplt.plot(training_history['train_acc'], label='Train Accuracy')\nplt.plot(training_history['val_acc'], label='Val Accuracy')\nplt.xlabel('Epoch')\nplt.ylabel('Accuracy')\nplt.legend()\nplt.title('Accuracy Over Epochs')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:16:41.702665Z","iopub.execute_input":"2025-06-10T05:16:41.702902Z","iopub.status.idle":"2025-06-10T05:43:47.092193Z","shell.execute_reply.started":"2025-06-10T05:16:41.702882Z","shell.execute_reply":"2025-06-10T05:43:47.091295Z"}},"outputs":[],"execution_count":null},{"id":"f009a10b","cell_type":"markdown","source":"## 7. Evaluate the Model\n\nAfter training, we evaluate the performance of our fine-tuned Dinov2 model on the validation set (or a dedicated test set if available and prepared). We will calculate metrics such as accuracy, precision, recall, F1-score, and visualize a confusion matrix.","metadata":{}},{"id":"c32eb5b7","cell_type":"code","source":"from sklearn.metrics import classification_report, confusion_matrix\nimport seaborn as sns\n\n# Function to evaluate the model\ndef evaluate_model(model, dataloader):\n    model.eval()  # Set model to evaluation mode\n    all_preds = []\n    all_labels = []\n\n    with torch.no_grad(): # No need to track gradients during evaluation\n        for inputs, labels in dataloader:\n            inputs = inputs.to(device)\n            labels = labels.to(device)\n\n            outputs = model(inputs)\n            _, preds = torch.max(outputs, 1)\n\n            all_preds.extend(preds.cpu().numpy())\n            all_labels.extend(labels.cpu().numpy())\n\n    return np.array(all_labels), np.array(all_preds)\n\nprint(\"Evaluating the model on the validation set...\")\nval_labels, val_preds = evaluate_model(model_ft, dataloaders['val'])\n\n# Classification Report\nprint(\"\\nClassification Report (Validation Set):\")\n# Ensure class_names are strings if they are not already for classification_report\nstr_class_names = [str(cn) for cn in class_names]\nprint(classification_report(val_labels, val_preds, target_names=str_class_names))\n\n# Confusion Matrix\nprint(\"\\nConfusion Matrix (Validation Set):\")\ncm = confusion_matrix(val_labels, val_preds)\nplt.figure(figsize=(6, 5))\nsns.heatmap(cm, annot=True, fmt=\"d\", cmap=\"Blues\", xticklabels=str_class_names, yticklabels=str_class_names)\nplt.xlabel(\"Predicted Label\")\nplt.ylabel(\"True Label\")\nplt.title(\"Confusion Matrix\")\nplt.show()\n\n# Visualize some predictions (optional)\ndef visualize_predictions(model, dataloader, num_images=10):\n    model.eval()\n    images_so_far = 0\n    fig = plt.figure(figsize=(15, 10))\n    \n    with torch.no_grad():\n        for i, (inputs, labels) in enumerate(dataloader):\n            inputs = inputs.to(device)\n            labels = labels.to(device)\n            \n            outputs = model(inputs)\n            _, preds = torch.max(outputs, 1)\n            \n            for j in range(inputs.size()[0]):\n                images_so_far += 1\n                ax = plt.subplot(num_images // 5 + (1 if num_images % 5 > 0 else 0), 5, images_so_far)\n                ax.axis('off')\n                ax.set_title(f'Pred: {class_names[preds[j]]} (True: {class_names[labels[j]]})')\n                # Inverse normalize and display image\n                img = inputs.cpu().data[j].numpy().transpose((1, 2, 0))\n                mean = np.array([0.485, 0.456, 0.406])\n                std = np.array([0.229, 0.224, 0.225])\n                img = std * img + mean\n                img = np.clip(img, 0, 1)\n                plt.imshow(img)\n                \n                if images_so_far == num_images:\n                    plt.tight_layout()\n                    plt.show()\n                    return\n    plt.tight_layout()\n    plt.show()\n\nprint(\"\\nVisualizing some predictions from the validation set:\")\nvisualize_predictions(model_ft, dataloaders['val'], num_images=10)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:43:47.093902Z","iopub.execute_input":"2025-06-10T05:43:47.094166Z","iopub.status.idle":"2025-06-10T05:44:10.217969Z","shell.execute_reply.started":"2025-06-10T05:43:47.094145Z","shell.execute_reply":"2025-06-10T05:44:10.216937Z"}},"outputs":[],"execution_count":null},{"id":"962a8c63","cell_type":"markdown","source":"## 8. Save the Trained Model\n\nFinally, we save the state dictionary of our best performing fine-tuned model. This allows us to load it later for inference or further training without needing to repeat the entire training process.","metadata":{}},{"id":"d3a581c8","cell_type":"code","source":"# The best model was already saved during training as 'dinov2_pcam_best_model.pth'\n# Here we can explicitly save the final model if needed, or re-save the best one for clarity.\nfinal_model_path = \"dinov2_pcam_final_trained_model.pth\"\ntorch.save(model_ft.state_dict(), final_model_path)\nprint(f\"Final trained model state dictionary saved to: {final_model_path}\")\n\n# To load the model later:\n# 1. Re-define the model architecture (as done in Section 5)\n#    model_to_load = ... (same architecture as model_ft)\n# 2. Load the state dictionary:\n#    model_to_load.load_state_dict(torch.load(final_model_path, map_location=device))\n#    model_to_load.eval() # Set to evaluation mode","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-06-10T05:44:10.219314Z","iopub.execute_input":"2025-06-10T05:44:10.219689Z","iopub.status.idle":"2025-06-10T05:44:10.359723Z","shell.execute_reply.started":"2025-06-10T05:44:10.219665Z","shell.execute_reply":"2025-06-10T05:44:10.358952Z"}},"outputs":[],"execution_count":null},{"id":"5ae2db30","cell_type":"markdown","source":"## 9. Conclusion and Expected Results\n\nThis notebook demonstrated a pipeline for classifying PCam histopathological images using a pre-trained Dinov2 model. We covered:\n1.  Setting up the environment and cloning Dinov2.\n2.  Downloading and organizing the PCam dataset.\n3.  Performing basic EDA.\n4.  Preprocessing images and creating DataLoaders.\n5.  Loading a Dinov2 model (`dinov2_vits14`) and adapting its head for binary classification.\n6.  Training the new classification head while keeping the Dinov2 backbone frozen.\n7.  Evaluating the model using accuracy, classification report, and confusion matrix.\n8.  Saving the trained model.\n\n**Expected Results:**\n\n*   **Performance:** Using a powerful pre-trained model like Dinov2, even with just fine-tuning the head, can yield strong results on medical image datasets like PCam. Accuracy can be expected to be significantly above random chance and often competitive with models trained from scratch with extensive data, especially when training data is limited. For PCam, accuracies in the range of 90-98% have been reported with various CNN and Transformer architectures, depending on the exact preprocessing, augmentation, model size, and training regime.\n*   **Feature Extraction Power:** Dinov2's self-supervised pre-training allows it to learn robust and generalizable visual features. This is particularly beneficial for specialized domains like medical imaging where large labeled datasets might be scarce.\n*   **Training Efficiency:** By freezing the backbone and only training a small classifier head, training can be relatively fast and require less computational resources compared to full fine-tuning of a large transformer.\n\n**Potential Improvements:**\n\n*   **Full Fine-tuning:** Unfreeze some or all layers of the Dinov2 backbone for further fine-tuning. This might yield better performance but requires more data and computational resources, and careful hyperparameter tuning (e.g., lower learning rates for the backbone).\n*   **Larger Dinov2 Models:** Experiment with larger Dinov2 variants (e.g., `dinov2_vitb14`, `dinov2_vitl14`) if computational resources allow. These models have higher capacity but also require more memory and time to train.\n*   **Advanced Augmentation:** Explore more sophisticated data augmentation techniques specific to histopathological images.\n*   **Hyperparameter Optimization:** Systematically tune hyperparameters like learning rate, batch size, optimizer settings, and scheduler parameters.\n*   **Cross-Validation:** Use k-fold cross-validation for a more robust estimate of model performance.\n*   **Test Set Evaluation:** If a dedicated labeled test set is available (separate from the validation set used for model selection), evaluate the final model on it for an unbiased performance measure. The current notebook uses the Kaggle `test` folder which is unlabeled; predictions would typically be submitted to Kaggle for scoring.\n\nThis notebook provides a solid foundation for using Dinov2 for image classification tasks.","metadata":{}}]}