{"metadata":{"colab":{"provenance":[]},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":11848,"databundleVersionId":862157,"sourceType":"competition"}],"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Problem Description and Data Overview","metadata":{"id":"ZfNz5rX8b3yB"}},{"cell_type":"markdown","source":"This project is part of a Kaggle competition focused on automatically identifying metastatic cancer in small image patches taken from larger digital pathology scans. The objective is to build a binary classifier to predict whether a given image patch contains cancerous tissue.","metadata":{}},{"cell_type":"markdown","source":"# **Data Overview**\nDataset: The dataset consists of image patches of size 96x96 pixels.\nTraining Set: Contains labeled images to train the model.\nTest Set: Contains unlabeled images for which predictions are needed.\nClass Distribution: The class labels indicate the presence (1) or absence (0) of cancer.","metadata":{}},{"cell_type":"code","source":"# Import the necessary libraries for data manipulation, visualization, and model building\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nfrom PIL import Image\nimport tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout\nfrom tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom keras.regularizers import l2\n\n# Define paths to the directories containing the train and test datasets\ntrain_dir = '/kaggle/input/histopathologic-cancer-detection/train'\ntest_dir = '/kaggle/input/histopathologic-cancer-detection/test'\n\n# Load train labels from CSV file\ntrain_labels = pd.read_csv('/kaggle/input/histopathologic-cancer-detection/train_labels.csv')\n\n# Display basic information about the train labels dataframe\nprint(\"Training Labels Info:\")\nprint(train_labels.info())\nprint(\"\\nSample of Training Labels:\")\nprint(train_labels.head())\n","metadata":{"id":"ClFMYW9fcLWb","execution":{"iopub.status.busy":"2024-09-04T04:50:24.618672Z","iopub.execute_input":"2024-09-04T04:50:24.619611Z","iopub.status.idle":"2024-09-04T04:50:24.877066Z","shell.execute_reply.started":"2024-09-04T04:50:24.619556Z","shell.execute_reply":"2024-09-04T04:50:24.875834Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Explanation:**\nIn this section, libraries are imported to handle data operations, and paths for training and test datasets are set. Basic information about training labels is printed to understand the dataset structure.","metadata":{"id":"74Grgph9cTPf"}},{"cell_type":"markdown","source":"# **Exploratory Data Analysis (EDA)**","metadata":{"id":"sRueIPfqcXLm"}},{"cell_type":"markdown","source":"During the EDA phase, we:\n\nVisualized the data using histograms and pie charts to understand class distribution and image characteristics.\nData Cleaning: Ensured the data quality by checking for missing values or corrupted images.\nPlan of Analysis: Based on initial findings, we decided to address class imbalance using techniques like data augmentation.","metadata":{}},{"cell_type":"code","source":"# Function to display images with their respective labels\ndef display_images(img_ids, labels, path, title):\n    \"\"\"\n    Display selected images with their labels.\n    Args:\n    - img_ids: List of image IDs\n    - labels: Corresponding labels for the images\n    - path: Directory path where images are located\n    - title: Title for the plot\n    \"\"\"\n    plt.figure(figsize=(15, 5))\n    for i, (img_id, label) in enumerate(zip(img_ids, labels)):\n        img_path = os.path.join(path, img_id + '.tif')\n        img = Image.open(img_path)\n        plt.subplot(1, len(img_ids), i + 1)\n        plt.imshow(img)\n        plt.title(f\"Label: {label}\")\n        plt.axis('off')\n    plt.suptitle(title)\n    plt.show()\n\n# Show examples with and without cancer\ndisplay_images(train_labels[train_labels['label'] == 0]['id'][:5], [0]*5, train_dir, \"Images Without Cancer\")\ndisplay_images(train_labels[train_labels['label'] == 1]['id'][:5], [1]*5, train_dir, \"Images With Cancer\")\n\n# Analyze distribution of labels in the training dataset\nplt.figure(figsize=(8, 6))\nsns.countplot(x='label', data=train_labels)\nplt.title('Training Labels Distribution')\nplt.xlabel('Label (0 = No Cancer, 1 = Cancer)')\nplt.ylabel('Frequency')\nplt.show()\n","metadata":{"id":"vffiQa7Ocby9","execution":{"iopub.status.busy":"2024-09-04T04:50:30.363317Z","iopub.execute_input":"2024-09-04T04:50:30.363984Z","iopub.status.idle":"2024-09-04T04:50:31.850082Z","shell.execute_reply.started":"2024-09-04T04:50:30.363942Z","shell.execute_reply":"2024-09-04T04:50:31.849087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Explanation:**\nThe function display_images visualizes several examples from each class of the dataset—cancerous and non-cancerous. A count plot shows the distribution of labels to understand class balance.","metadata":{"id":"vFSSuplBceav"}},{"cell_type":"markdown","source":"# **Image Preprocessing**","metadata":{"id":"lQgNuQfrci6N"}},{"cell_type":"code","source":"# Rescale pixel values, split for validation, and prepare ImageDataGenerators\nbatch_size = 64\ntarget_size = (60, 60)\n\ntrain_datagen = ImageDataGenerator(\n    rescale=1./255,  # Normalize image pixel values\n    validation_split=0.25  # Reserve a portion for validation\n)\n\n# Update 'id' column to match file names and convert 'label' to string for generator\ntrain_labels['id'] = train_labels['id'].apply(lambda x: x + '.tif')\ntrain_labels['label'] = train_labels['label'].astype(str)\n\n# Initialize generators for train and validation datasets\ntrain_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_labels,\n    directory=train_dir,\n    x_col='id',\n    y_col='label',\n    target_size=target_size,\n    batch_size=batch_size,\n    class_mode='binary',\n    subset='training',\n    shuffle=True\n)\n\nvalidation_generator = train_datagen.flow_from_dataframe(\n    dataframe=train_labels,\n    directory=train_dir,\n    x_col='id',\n    y_col='label',\n    target_size=target_size,\n    batch_size=batch_size,\n    class_mode='binary',\n    subset='validation',\n    shuffle=True\n)\n","metadata":{"id":"aZVmVclvcnZA","execution":{"iopub.status.busy":"2024-09-04T04:50:37.692275Z","iopub.execute_input":"2024-09-04T04:50:37.693049Z","iopub.status.idle":"2024-09-04T04:56:35.324353Z","shell.execute_reply.started":"2024-09-04T04:50:37.693009Z","shell.execute_reply":"2024-09-04T04:56:35.323543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Explanation:**\nThe ImageDataGenerator resizes and normalizes images, and splits the training data for validation. Labels are adjusted to ensure compatibility with the generator functions.","metadata":{"id":"Io0gm8vjcsLA"}},{"cell_type":"markdown","source":"# **Model Architecture and Training**","metadata":{"id":"rR40XFdHcuPU"}},{"cell_type":"markdown","source":"The chosen model is a Convolutional Neural Network (CNN) due to its effectiveness in image classification tasks:\n\nArchitecture: A sequential CNN with three convolutional layers followed by max-pooling layers, a dropout layer, and fully connected layers.\nReasoning: CNNs automatically learn spatial hierarchies of features, which is critical for medical image analysis.\nHyperparameter Tuning: We experimented with learning rates, batch sizes, and dropout rates to optimize performance.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Input\n\n# Define the CNN architecture using Input object\nmodel = Sequential([\n    Input(shape=(60, 60, 3)),  # Define the input shape here\n    Conv2D(32, (3, 3), activation='relu'),\n    MaxPooling2D(2, 2),\n    Conv2D(64, (3, 3), activation='relu'),\n    MaxPooling2D(2, 2),\n    Conv2D(128, (3, 3), activation='relu'),\n    MaxPooling2D(2, 2),\n    Flatten(),\n    Dense(256, activation='relu'),\n    Dropout(0.5),\n    Dense(1, activation='sigmoid')\n])\n\n\n# Compile the model\nmodel.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\n# Display a summary of the model\nmodel.summary()\n\n# Callbacks for early stopping and reducing learning rate\nearly_stopping = EarlyStopping(monitor='val_loss', patience=3, restore_best_weights=True)\nreduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=2, min_lr=0.001)\n\n# Train the model\nhistory = model.fit(\n    train_generator,\n    steps_per_epoch=train_generator.samples // batch_size,\n    validation_data=validation_generator,\n    validation_steps=validation_generator.samples // batch_size,\n    epochs=10,\n    callbacks=[early_stopping, reduce_lr]\n)\n","metadata":{"id":"FDWHAFU1cxxF","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Explanation:**\nA Sequential model is built with multiple convolution and pooling layers, a flatten layer, and dense layers. Early stopping and learning rate reduction callbacks are used during training to enhance model performance.","metadata":{"id":"BzYB4Aogc0sb"}},{"cell_type":"markdown","source":"# **Results Visualization and Analysis**","metadata":{"id":"uAIC5ihGc6kY"}},{"cell_type":"markdown","source":"Several experimental setups were conducted to improve the model:\n\nHyperparameter Tuning: Cross-validation was used to select optimal hyperparameters.\nComparisons: We compared different model architectures, including deeper networks, to assess performance.\nTraining Optimization: Techniques like early stopping and learning rate reduction improved convergence.\nPerformance Metrics: Used accuracy and AUC-ROC to evaluate the model, with results presented in comparative tables and plots.","metadata":{}},{"cell_type":"code","source":"# Function to plot training history\nimport matplotlib.pyplot as plt\ndef plot_history(history, metric):\n    \"\"\"\n    Plot model training history.\n    Args:\n    - history: Training history returned by model.fit()\n    - metric: Metric to be plotted (e.g., 'accuracy', 'loss')\n    \"\"\"\n    plt.plot(history.history[metric])\n    plt.plot(history.history['val_' + metric])\n    plt.title('Model ' + metric.title())\n    plt.xlabel('Epoch')\n    plt.ylabel(metric.title())\n    plt.legend(['Train', 'Validation'], loc='best')\n    plt.show()\n\n# Displaying results\nplt.figure(figsize=(12, 5))\nplot_history(history, 'accuracy')\nplot_history(history, 'loss')\n","metadata":{"id":"b28M-tBBc5qs","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Explanation:**\nThe plot_history function plots training and validation accuracy and loss over epochs, providing insights into the model’s learning curve.","metadata":{"id":"kf_5tlxddBqL"}},{"cell_type":"code","source":"# Prepare test data generator\ntest_datagen = ImageDataGenerator(rescale=1./255)\n\ntest_generator = test_datagen.flow_from_dataframe(\n    dataframe=pd.DataFrame({'id': os.listdir(test_dir)}),\n    directory=test_dir,\n    x_col='id',\n    y_col=None,\n    target_size=target_size,\n    batch_size=1,\n    class_mode=None,\n    shuffle=False\n)\n\n# Predict on test data\ntest_generator.reset()\npredictions = model.predict(test_generator, steps=test_generator.samples)\n\n# Prepare the submission DataFrame\nfilenames = test_generator.filenames\nids = [filename.split('.')[0] for filename in filenames]\npredicted_labels = (predictions > 0.5).astype(int).reshape(-1)\n\nsubmission_df = pd.DataFrame({\n    'id': ids,\n    'label': predicted_labels\n})\n\n# Save submission.csv\nsubmission_df.to_csv('submission.csv', index=False)\n\nprint(\"submission.csv file has been created successfully!\")\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **Conclusion**","metadata":{}},{"cell_type":"markdown","source":"Findings: Techniques like data augmentation and dropout significantly improved model generalization.\nChallenges: Imbalanced data was a concern, mitigated by oversampling.\nFuture Work: Consider exploring more advanced architectures like ResNet or implementing ensemble methods.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}