{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":71549,"databundleVersionId":8561470,"sourceType":"competition"},{"sourceId":137055,"sourceType":"modelInstanceVersion","isSourceIdPinned":true,"modelInstanceId":116008,"modelId":139240}],"dockerImageVersionId":30761,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Inference with Fine-Tuned EfficientNet Model\n\n## Introduction\n\nIn this notebook, we will perform inference using a fine-tuned EfficientNet model on a dataset related to lumbar spine degenerative classification. The EfficientNet architecture, known for its efficiency and accuracy, has been pre-trained on a large dataset and fine-tuned to adapt to our specific classification task.\n\n### Objectives\n\n1. **Set Up Environment**: Import necessary libraries and load the dataset.\n2. **Load the Fine-Tuned Model**: Load the previously saved model to make predictions on the test dataset.\n3. **Process Test Images**: Prepare the test images using appropriate transformations.\n4. **Make Predictions**: Utilize the model to predict the classes of the test images.\n5. **Prepare Submission**: Format the predictions for submission in a required format.\n\n### Prerequisites\n\n- Basic knowledge of PyTorch and deep learning concepts.\n- Understanding of model fine-tuning and inference processes.\n\nBy the end of this notebook, you will have a clear understanding of how to utilize a fine-tuned model for making predictions on unseen data and preparing those predictions for submission.\n\nLet's get started!\n","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torchvision.models as models\nfrom torch.utils.data import DataLoader\nfrom torchvision import transforms\nfrom sklearn.model_selection import train_test_split\nfrom tqdm import tqdm\nimport pandas as pd ","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:46.058219Z","iopub.execute_input":"2024-10-16T19:36:46.059219Z","iopub.status.idle":"2024-10-16T19:36:52.134162Z","shell.execute_reply.started":"2024-10-16T19:36:46.059172Z","shell.execute_reply":"2024-10-16T19:36:52.133038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Exploring the Test Dataset Descriptions\n\nTo understand the structure and contents of the `test_description` DataFrame, we use the `info()` method provided by pandas. This method summarizes important details about the DataFrame, including the number of entries, column names, non-null counts, and data types.\n","metadata":{}},{"cell_type":"code","source":"# the path to the dataset\n\ntrain_path = '/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/'\n\ntest_description = pd.read_csv(train_path + 'test_series_descriptions.csv')\nsubmission = pd.read_csv(train_path + 'sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.136033Z","iopub.execute_input":"2024-10-16T19:36:52.136573Z","iopub.status.idle":"2024-10-16T19:36:52.160972Z","shell.execute_reply.started":"2024-10-16T19:36:52.136526Z","shell.execute_reply":"2024-10-16T19:36:52.160011Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_description.info()","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.162258Z","iopub.execute_input":"2024-10-16T19:36:52.162627Z","iopub.status.idle":"2024-10-16T19:36:52.194201Z","shell.execute_reply.started":"2024-10-16T19:36:52.162591Z","shell.execute_reply":"2024-10-16T19:36:52.193029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Summary of the DataFrame\n\nThe output reveals that the `test_description` DataFrame consists of 3 entries (or rows), indexed from 0 to 2. It contains a total of 3 columns with the following characteristics:\n\n- **study_id**: \n  - Type: `int64` \n  - Description: This column contains unique identifiers for each study. It has 3 non-null entries, indicating that all entries are populated.\n\n- **series_id**: \n  - Type: `int64`\n  - Description: This column holds unique identifiers for each series within a study, also with 3 non-null entries.\n\n- **series_description**: \n  - Type: `object` \n  - Description: This column contains string descriptions of each series. It similarly has 3 non-null entries, meaning that all descriptions are present.\n\n#### Memory Usage\n\nThe DataFrame utilizes approximately 200 bytes of memory, which is a relatively small amount given the number of entries.\n\n### Conclusion\n\nThis summary of the `test_description` DataFrame provides insight into the data we will be working with for inference. Understanding the structure and contents of the DataFrame is crucial for processing the test images and making accurate predictions.\n","metadata":{}},{"cell_type":"markdown","source":"------","metadata":{}},{"cell_type":"markdown","source":"### Generating Image Paths for Inference\n\nIn this section, we create a function that generates the file paths for the test images based on the structure of the dataset. This step is crucial for loading the images later, enabling us to process them and make predictions.\n\n#### Libraries Used\n\n- **os**: This module provides functionalities for interacting with the operating system, allowing us to create and manipulate file paths efficiently.\n- **cv2**: Part of the OpenCV library, commonly used for image processing tasks. Although not used directly in this snippet, it will be helpful later.\n- **matplotlib.pyplot**: This library is typically used for visualization and plotting, which may be utilized later to display images.\n\n#### Function Overview\n\nA function named `generate_image_paths` is defined to take a DataFrame and a base directory as input. Its purpose is to generate a list of paths for all the images related to each study and series identified in the provided DataFrame.\n\n1. **Parameters**:\n   - **DataFrame (`df`)**: This contains the `study_id` and `series_id` that identify the images.\n   - **Base Directory (`data_dir`)**: This is the path to the directory where the test images are stored.\n\n2. **Process**:\n   - The function initializes an empty list to store the image paths.\n   - It iterates through the `study_id` and `series_id` pairs in the DataFrame.\n   - For each pair, it constructs the paths to the respective study and series directories.\n   - It lists all files in the series directory and generates full paths for each image.\n   - Finally, it returns the list of image paths.\n\n#### Generating Test Image Paths\n\nAfter defining the function, it is invoked to generate the paths for the test images using the `test_description` DataFrame and the specified directory path. The resulting list of paths is stored for later use.\n\n\n\nThis section ensures that all relevant test images can be accessed by dynamically generating their file paths based on the dataset's directory structure. This approach facilitates efficient loading and processing of images in the subsequent steps of our inference process.\n","metadata":{}},{"cell_type":"code","source":"import os\nimport cv2\nimport matplotlib.pyplot as plt\n\n# Function to generate image paths based on directory structure\ndef generate_image_paths(df, data_dir):\n    image_paths = []\n    for study_id, series_id in zip(df['study_id'], df['series_id']):\n        study_dir = os.path.join(data_dir, str(study_id))\n        series_dir = os.path.join(study_dir, str(series_id))\n        images = os.listdir(series_dir)\n        image_paths.extend([os.path.join(series_dir, img) for img in images])\n    return image_paths\n\n\ntest_image_paths = generate_image_paths(test_description, f'{train_path}/test_images')\n","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.197306Z","iopub.execute_input":"2024-10-16T19:36:52.197902Z","iopub.status.idle":"2024-10-16T19:36:52.488669Z","shell.execute_reply.started":"2024-10-16T19:36:52.197850Z","shell.execute_reply":"2024-10-16T19:36:52.487506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Condition Mapping for Medical Imaging\n\nIn this section, we define a mapping dictionary called `condition_mapping`, which is essential for linking specific imaging modalities to their corresponding medical conditions. This mapping is crucial for interpreting the results of medical images effectively.\n\n#### Overview of the Mapping\n\nThe `condition_mapping` dictionary organizes the relationship between different types of imaging conditions (modalities) and associated diagnoses. It allows for quick access to conditions based on the type of imaging performed.\n\n#### Structure of the Dictionary\n\n- **Keys**: The keys represent various imaging conditions:\n  - **`'Sagittal T1'`**: Refers to images taken in the sagittal plane using T1-weighted sequences.\n  - **`'Axial T2'`**: Refers to images taken in the axial plane using T2-weighted sequences.\n  - **`'Sagittal T2/STIR'`**: Refers to images taken in the sagittal plane using T2-weighted or Short Tau Inversion Recovery (STIR) sequences.\n\n- **Values**: The values associated with these keys indicate the specific medical conditions:\n  - For **`'Sagittal T1'`** and **`'Axial T2'`**, the values are nested dictionaries that further distinguish conditions based on left and right side afflictions:\n    - **Left Conditions**: Indicate conditions affecting the left side of the spine, such as `left_neural_foraminal_narrowing` or `left_subarticular_stenosis`.\n    - **Right Conditions**: Indicate conditions affecting the right side, such as `right_neural_foraminal_narrowing` or `right_subarticular_stenosis`.\n  \n  - For **`'Sagittal T2/STIR'`**, the value directly corresponds to a single condition, `spinal_canal_stenosis`, indicating that this imaging modality is associated with this specific diagnosis.\n\n#### Purpose of the Mapping\n\nThe `condition_mapping` dictionary serves several important functions:\n\n- **Organization**: It provides a structured way to relate complex imaging types with specific medical conditions, facilitating data processing.\n- **Clarity**: The mapping offers a clear reference to understand which imaging modalities correlate with particular diagnoses, aiding in clinical decision-making.\n- **Efficiency**: Using a dictionary allows for quick lookups of conditions based on imaging types, which enhances the efficiency of data analysis and prediction tasks.\n\n\nThe `condition_mapping` dictionary is a critical component of our analysis workflow. It enables us to efficiently link medical imaging modalities with their respective conditions, thereby improving our ability to interpret and analyze the results of the imaging data effectively.\n","metadata":{}},{"cell_type":"code","source":"condition_mapping = {\n    'Sagittal T1': {'left': 'left_neural_foraminal_narrowing', 'right': 'right_neural_foraminal_narrowing'},\n    'Axial T2': {'left': 'left_subarticular_stenosis', 'right': 'right_subarticular_stenosis'},\n    'Sagittal T2/STIR': 'spinal_canal_stenosis'\n}\n","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.490186Z","iopub.execute_input":"2024-10-16T19:36:52.490941Z","iopub.status.idle":"2024-10-16T19:36:52.496416Z","shell.execute_reply.started":"2024-10-16T19:36:52.490882Z","shell.execute_reply":"2024-10-16T19:36:52.495358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Expanding Image Paths into a DataFrame\n\nIn this section, we define a process for expanding the image paths from our test dataset and organizing them into a structured DataFrame. This step is crucial for facilitating data analysis and model inference.\n\n#### Overview of the Process\n\n1. **Function Definition**:\n   - A function called `get_image_paths` is defined to retrieve image file paths from a given study and series. \n\n2. **Function Logic**:\n   - The function constructs the path to the series directory using the `study_id` and `series_id` from the provided row.\n   - It checks if the constructed path exists. If it does, it returns a list of file paths for all images in that directory. The paths are constructed using `os.path.join()`, ensuring proper handling of the filesystem structure.\n   - If the series path does not exist, the function returns an empty list.\n\n3. **Expanding Rows**:\n   - An empty list called `expanded_rows` is initialized to store the expanded information for each image.\n   - The code iterates through each row of the `test_description` DataFrame using `iterrows()`. For each row:\n     - The function `get_image_paths` is called to retrieve the list of image paths.\n     - The conditions associated with the `series_description` are obtained from the `condition_mapping` dictionary. If the conditions are a single string, they are converted into a dictionary format for uniformity.\n     - The code then loops through each side (left and right) and condition pair, creating a new entry for every image path found. Each entry includes:\n       - `study_id`\n       - `series_id`\n       - `series_description`\n       - `image_path`: The path to the individual image.\n       - `condition`: The corresponding condition from the mapping.\n       - `row_id`: A unique identifier constructed from the `study_id` and the condition.\n\n4. **Creating the DataFrame**:\n   - After expanding all image paths, a new DataFrame called `test_df` is created from the `expanded_rows` list. This DataFrame contains comprehensive information about each image, including its associated conditions.\n\n\nThis section effectively constructs a structured DataFrame, `test_df`, that holds all relevant details about each image in the test dataset. By expanding the rows based on the conditions and image paths, we facilitate easier access and processing of the data for subsequent analysis and inference tasks.\n\n\n----------","metadata":{}},{"cell_type":"code","source":"base_path = '/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/test_images/'\n","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.497975Z","iopub.execute_input":"2024-10-16T19:36:52.498328Z","iopub.status.idle":"2024-10-16T19:36:52.507893Z","shell.execute_reply.started":"2024-10-16T19:36:52.498294Z","shell.execute_reply":"2024-10-16T19:36:52.506549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_image_paths(row):\n    series_path = os.path.join(base_path, str(row['study_id']), str(row['series_id']))\n    if os.path.exists(series_path):\n        return [os.path.join(series_path, f) for f in os.listdir(series_path) if os.path.isfile(os.path.join(series_path, f))]\n    return []\n","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.509412Z","iopub.execute_input":"2024-10-16T19:36:52.509793Z","iopub.status.idle":"2024-10-16T19:36:52.519387Z","shell.execute_reply.started":"2024-10-16T19:36:52.509757Z","shell.execute_reply":"2024-10-16T19:36:52.518141Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"expanded_rows = []\nfor index, row in test_description.iterrows():\n    image_paths = get_image_paths(row)\n    conditions = condition_mapping.get(row['series_description'], {})\n    if isinstance(conditions, str):  # Single condition\n        conditions = {'left': conditions, 'right': conditions}\n    for side, condition in conditions.items():\n        for image_path in image_paths:\n            expanded_rows.append({\n                'study_id': row['study_id'],\n                'series_id': row['series_id'],\n                'series_description': row['series_description'],\n                'image_path': image_path,\n                'condition': condition,\n                'row_id': f\"{row['study_id']}_{condition}\"\n            })\n","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.521138Z","iopub.execute_input":"2024-10-16T19:36:52.521629Z","iopub.status.idle":"2024-10-16T19:36:52.537990Z","shell.execute_reply.started":"2024-10-16T19:36:52.521581Z","shell.execute_reply":"2024-10-16T19:36:52.536663Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_df = pd.DataFrame(expanded_rows)","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.539579Z","iopub.execute_input":"2024-10-16T19:36:52.540349Z","iopub.status.idle":"2024-10-16T19:36:52.548055Z","shell.execute_reply.started":"2024-10-16T19:36:52.540308Z","shell.execute_reply":"2024-10-16T19:36:52.546943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Updating `row_id` with Levels\n\nIn this section, we enhance the `row_id` in the `test_df` DataFrame by incorporating anatomical levels associated with spinal conditions. This improvement provides additional context to each entry in the dataset.\n\n#### Overview of the Process\n\n1. **Defining Levels**:\n   - A list named `levels` is defined, containing the following anatomical levels commonly associated with spinal imaging:\n     - `'l1_l2'`\n     - `'l2_l3'`\n     - `'l3_l4'`\n     - `'l4_l5'`\n     - `'l5_s1'`\n   - These levels correspond to the intervertebral spaces between lumbar vertebrae.\n\n2. **Function Definition**:\n   - The function `update_row_id` is defined to modify the `row_id` for each row in the DataFrame. It takes two parameters: `row` (the current row of the DataFrame) and `levels` (the list of anatomical levels).\n   - Within the function:\n     - The anatomical level is determined using `row.name % len(levels)`, which calculates the index based on the current row's position. This ensures a cyclic assignment of levels to the `row_id`.\n     - The function returns a new `row_id` formatted as a string that combines the `study_id`, the associated `condition`, and the corresponding anatomical `level`.\n\n3. **Applying the Update**:\n   - The `apply()` method is used to update the `row_id` in `test_df` by applying the `update_row_id` function across all rows.\n   - The `axis=1` parameter indicates that the function should be applied row-wise.\n\n4. **Viewing the Updated DataFrame**:\n   - Finally, the first few rows of the updated DataFrame (`test_df`) are displayed using `test_df.head()`, allowing us to inspect the changes made to the `row_id`.\n\n\nThis section successfully enhances the `row_id` of each entry in the `test_df` DataFrame by incorporating relevant spinal levels. This addition not only improves the identification of conditions but also provides better context for the analysis of imaging data related to lumbar spine conditions.\n\n------------","metadata":{}},{"cell_type":"code","source":"# Levels for row_id\nlevels = ['l1_l2', 'l2_l3', 'l3_l4', 'l4_l5', 'l5_s1']\n\n# update row_id with levels\ndef update_row_id(row, levels):\n    level = levels[row.name % len(levels)]  \n    return f\"{row['study_id']}_{row['condition']}_{level}\"\n\n# Update row_id in expanded_test_desc to include levels\ntest_df['row_id'] = test_df.apply(lambda row: update_row_id(row, levels), axis=1)\n\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.551635Z","iopub.execute_input":"2024-10-16T19:36:52.552005Z","iopub.status.idle":"2024-10-16T19:36:52.577319Z","shell.execute_reply.started":"2024-10-16T19:36:52.551960Z","shell.execute_reply":"2024-10-16T19:36:52.576324Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loading DICOM Images and Custom Dataset Class\n\nIn this section, we define a method for loading DICOM images and create a custom dataset class to facilitate the handling of test data in our machine learning model. This is essential for preparing the images for inference.\n\n#### Loading DICOM Images\n\n1. **Function Definition**:\n   - The function `load_dicom` is designed to load an image from a DICOM file given its path.\n   - It takes a single parameter, `image_path`, which specifies the location of the DICOM file.\n\n2. **Error Handling**:\n   - The function employs a try-except block to manage exceptions that may arise during the loading process.\n   - If an error occurs, a `FileNotFoundError` is raised, detailing the nature of the error.\n\n3. **Reading the DICOM File**:\n   - The `pydicom.dcmread()` function reads the DICOM file, with the `force=True` argument ensuring that it attempts to read the file even if it does not conform to the standard.\n\n4. **Image Processing**:\n   - The pixel data is extracted using `dicom.pixel_array`.\n   - If the image data type is not already `uint8`, it is converted to this format to standardize the pixel values.\n   - If the image is grayscale (i.e., has two dimensions), it is converted to RGB format by stacking the grayscale channel three times.\n\n5. **Return Statement**:\n   - The processed image is returned for further use.\n\n#### Custom Dataset Class\n\n1. **Class Definition**:\n   - A custom dataset class named `CustomTestDataset` is defined, inheriting from `torch.utils.data.Dataset`. This allows us to create a dataset compatible with PyTorch data loaders.\n\n2. **Initialization**:\n   - The `__init__` method takes two parameters: `dataframe`, which contains the metadata and paths for the images, and an optional `transform` parameter to apply transformations to the images.\n   - These parameters are stored as instance variables for use in other methods.\n\n3. **Length Method**:\n   - The `__len__` method returns the length of the dataset, which is the number of entries in the DataFrame.\n\n4. **Item Retrieval**:\n   - The `__getitem__` method retrieves an image and its associated metadata for a specific index:\n     - It uses `iloc` to safely access the image path corresponding to the given index.\n     - The `load_dicom` function is called to load the image from the specified path.\n     - If a transformation is provided, it is applied to the image.\n     - A dictionary called `row_data` is created to hold the metadata associated with the current index (in this case, only `row_id`).\n\n5. **Return Statement**:\n   - The method returns a tuple containing the processed image and its associated metadata (`row_data`).\n\n\nThis section provides the functionality to load DICOM images and define a custom dataset class for our test dataset. This setup is crucial for preparing and managing the data needed for model inference, ensuring that the images are correctly loaded and transformed for further analysis.\n\n\n----------------","metadata":{}},{"cell_type":"code","source":"import pydicom\nimport numpy as np\nfrom torch.utils.data import Dataset\nfrom torchvision import transforms\nimport torch\n\ndef load_dicom(image_path):\n    \"\"\"Load an image from a DICOM file.\"\"\"\n    try:\n        dicom = pydicom.dcmread(image_path, force=True)\n        image = dicom.pixel_array\n        # Convert to uint8 if not already in that format\n        if image.dtype != np.uint8:\n            image = image.astype(np.uint8)\n        # Convert grayscale to RGB if needed\n        if len(image.shape) == 2:  # Grayscale image\n            image = np.stack([image] * 3, axis=-1)  # Repeat the channel\n        return image\n    except Exception as e:\n        raise FileNotFoundError(f\"Error loading DICOM file {image_path}: {str(e)}\")\n\nclass CustomTestDataset(Dataset):\n    def __init__(self, dataframe, transform=None):\n        self.dataframe = dataframe\n        self.transform = transform\n\n    def __len__(self):\n        return len(self.dataframe)\n\n    def __getitem__(self, index):\n        image_path = self.dataframe['image_path'].iloc[index]  # Ensure using iloc for safe access\n        image = load_dicom(image_path)\n        \n        if self.transform:\n            image = self.transform(image)\n        \n         # Create row_data as a dictionary for the current index\n        row_data = {\n            'row_id': self.dataframe['row_id'].iloc[index],\n            # Add other fields as necessary\n        }\n        return image, row_data","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.578550Z","iopub.execute_input":"2024-10-16T19:36:52.578876Z","iopub.status.idle":"2024-10-16T19:36:52.792302Z","shell.execute_reply.started":"2024-10-16T19:36:52.578840Z","shell.execute_reply":"2024-10-16T19:36:52.791036Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Image Transformations for Model Input\n\nIn this section, we define a series of image transformations to prepare the images for input into our deep learning model. These transformations are essential for ensuring that the images are in the correct format and size, allowing the model to make accurate predictions.\n\n#### Overview of Transformations\n\n1. **Importing the Necessary Library**:\n   - The `transforms` module from `torchvision` is imported to leverage various image transformation functions provided by PyTorch.\n\n2. **Transformation Pipeline**:\n   - We create a transformation pipeline using `transforms.Compose()`, which allows us to chain multiple transformations together. The transformations applied in the pipeline are as follows:\n   \n3. **Detailed Explanation of Each Transformation**:\n   - **`transforms.ToPILImage()`**:\n     - This transformation converts a NumPy array (or a tensor) into a PIL image format. This step is necessary if the input images are in a format that is not directly compatible with the subsequent transformations.\n  \n   - **`transforms.Resize((224, 224))`**:\n     - Resizes the image to dimensions of 224x224 pixels, which is a common input size for many deep learning models, including EfficientNet. This ensures that all images fed into the model have a consistent size, preventing shape mismatches.\n\n   - **`transforms.ToTensor()`**:\n     - Converts the PIL image to a PyTorch tensor. This is a crucial step because the model requires input data in tensor format for computation.\n\n   - **`transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])`**:\n     - Normalizes the tensor image by adjusting the pixel values to have a mean and standard deviation that are consistent with those of the ImageNet dataset. This normalization helps in improving the model's convergence during training and inference by ensuring that the input data has a standard scale.\n\n4. **Importance of Transformations**:\n   - These transformations are essential for preparing the images so that they conform to the expected input specifications of the model. Properly transformed images contribute significantly to the performance of the model by improving its ability to generalize from the training data.\n\n\n\nThis section effectively outlines the transformations applied to the images before they are fed into the model. By standardizing the input images through resizing, tensor conversion, and normalization, we set the stage for improved model performance and accurate predictions.\n\n------------","metadata":{}},{"cell_type":"code","source":"import torchvision.transforms as transforms\n\ntransform = transforms.Compose([\n    transforms.ToPILImage(),  # Convert NumPy array to PIL image (if necessary)\n    transforms.Resize((224, 224)),  # Resize the image to match the model input size\n    transforms.ToTensor(),  # Convert image to tensor\n    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])  # Apply ImageNet normalization\n])\n","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.793591Z","iopub.execute_input":"2024-10-16T19:36:52.793915Z","iopub.status.idle":"2024-10-16T19:36:52.800769Z","shell.execute_reply.started":"2024-10-16T19:36:52.793882Z","shell.execute_reply":"2024-10-16T19:36:52.799557Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Creating a Test Dataset and DataLoader\n\nIn this section, we create a custom test dataset and a corresponding DataLoader to facilitate the batch processing of test images for inference.\n\n#### Overview\n\n1. **Custom Test Dataset**:\n   - We initialize a `CustomTestDataset` object named `test_dataset`. This dataset is responsible for loading the test images and their associated metadata.\n   - The parameters passed during initialization include:\n     - `test_df`: The DataFrame containing the paths to the images and other metadata.\n     - `batch_size`: Set to 8, which specifies the number of samples to be loaded in each batch.\n     - `transform`: The previously defined transformation pipeline that processes the images as they are loaded.\n\n### Creating the DataLoader\n\nIn this section, we create a `DataLoader` object named `test_loader` to facilitate the iterative loading of test images from the dataset.\n\n#### Purpose of DataLoader\n\nThe `DataLoader` is crucial for efficiently handling the data during inference, allowing us to process multiple samples at once and manage batch sizes effectively.\n\n#### Parameters of the DataLoader\n\n1. **`test_dataset`**:\n   - This parameter specifies the dataset from which the `DataLoader` will load data. In this case, it is our previously defined `test_dataset`, which contains the test images and their corresponding metadata.\n\n2. **`batch_size`**:\n   - The `batch_size` is set to 8, meaning that each iteration of the `DataLoader` will yield 8 images. This allows for efficient processing of the images in manageable groups.\n\n3. **`shuffle`**:\n   - The `shuffle` parameter is set to `False`. This means that the data will be loaded in the order it appears in the dataset without any randomization. Maintaining the correct order is important during inference to ensure that predictions can be accurately matched to their respective inputs.\n\n\n\nBy creating the `test_loader`, we enable the efficient loading of test data, optimizing memory usage and computational efficiency during the inference phase of model evaluation.\n\n-----------","metadata":{}},{"cell_type":"code","source":"test_dataset = CustomTestDataset(test_df, transform=transform)  \n\ntest_loader = DataLoader(test_dataset,batch_size = 8, shuffle=False)","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.802089Z","iopub.execute_input":"2024-10-16T19:36:52.802794Z","iopub.status.idle":"2024-10-16T19:36:52.813718Z","shell.execute_reply.started":"2024-10-16T19:36:52.802732Z","shell.execute_reply":"2024-10-16T19:36:52.812600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loading the Pre-trained Model\n\nIn this section, we demonstrate how to load the pre-trained model that has been saved to disk. This process is crucial for performing inference on new data using the model we have trained.\n\n#### Model Loading Process\n\n1. **Importing the Necessary Library**:\n   - We import the `torch` library, which provides functionalities for loading and managing models in PyTorch.\n\n2. **Model Path**:\n   - We specify the path to the saved model weights. In this case, the model is located at:\n     ```python\n     model_path = \"/kaggle/input/rsna-model/pytorch/default/1/full_model30epochs.pth\"\n     ```\n\n3. **Loading the Model**:\n   - We use `torch.load()` to load the model from the specified path. The `map_location='cpu'` argument ensures that the model is loaded to the CPU, which is particularly useful if the model was trained on a GPU but is being used on a machine without GPU support. \n   - The loaded model is assigned to the variable `unified_model`, which we can use for inference.\n  \n-------------","metadata":{}},{"cell_type":"markdown","source":"### UnifiedEfficientNetV2 Class\n\nIn this section, we define the `UnifiedEfficientNetV2` class, which builds upon the EfficientNetV2 architecture. This class is tailored for multi-class classification tasks, utilizing transfer learning while allowing for the customization of the final classification layers.\n\n#### Overview\n\n1. **Class Definition**:\n   - The `UnifiedEfficientNetV2` class inherits from `torch.nn.Module` and defines the architecture for our model. \n\n2. **Constructor (`__init__` Method)**:\n   - **Parameters**:\n     - `num_classes`: Specifies the number of output classes for classification (default is 3).\n     - `pretrained`: A boolean flag indicating whether to load pretrained weights (default is `True`).\n     - `weights_path`: A string path to load local weights if available (default is `None`).\n\n   - **Model Initialization**:\n     - Loads the EfficientNetV2-S model from the `torchvision` library, optionally using pretrained weights.\n     - If a local weights path is provided, it loads those weights into the model.\n\n   - **Classifier Replacement**:\n     - Replaces the default classifier of EfficientNetV2 with an identity function. This allows us to define custom classification layers.\n\n   - **Fully Connected Layers**:\n     - Defines three fully connected layers (`fc1`, `fc2`, `fc3`) with Batch Normalization and ReLU activation.\n     - The output layer (`fc3`) maps to the specified number of classes.\n\n   - **Dropout Layers**:\n     - Incorporates dropout layers (`dropout1` and `dropout2`) for regularization to help prevent overfitting during training.\n\n3. **Forward Method**:\n   - Defines the forward pass of the network:\n     - Extracts embeddings from the EfficientNetV2 model.\n     - Passes the embeddings through the fully connected layers and applies dropout as specified.\n\n\nThe `UnifiedEfficientNetV2` class provides a flexible architecture for image classification tasks, leveraging the EfficientNetV2 backbone while allowing customization of the classifier. \n\nFor detailed documentation and explanations of the training process and architecture decisions, please refer to the training notebook, where I have provided a comprehensive overview of each component and its purpose.\n\n-----","metadata":{}},{"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torchvision import models\n\nclass UnifiedEfficientNetV2(nn.Module):\n    def __init__(self, num_classes=3, pretrained=True, weights_path=None):\n        super(UnifiedEfficientNetV2, self).__init__()\n\n        # Load the EfficientNetV2 with optional pretrained weights\n        self.model = models.efficientnet_v2_s(weights=models.EfficientNet_V2_S_Weights.DEFAULT if pretrained else None)\n\n        # Load local weights if a path is provided\n        if weights_path is not None:\n            self.model.load_state_dict(torch.load(weights_path))\n\n        # Capture in_features from the last layer of the original classifier\n        in_features = self.model.classifier[-1].in_features  # Should be 1280 for EfficientNetV2-S\n        \n        # Replace the classifier with an identity function (we'll handle classification manually)\n        self.model.classifier = nn.Identity()\n\n        # Define fully connected layers with BatchNorm and ReLU activation\n        self.fc1 = nn.Sequential(\n            nn.Linear(in_features, 256),  # Update input size to 1280\n            nn.BatchNorm1d(256),\n            nn.ReLU(inplace=True)\n        )\n        self.fc2 = nn.Sequential(\n            nn.Linear(256, 128),\n            nn.BatchNorm1d(128),\n            nn.ReLU(inplace=True)\n        )\n        self.fc3 = nn.Linear(128, num_classes)\n\n        # Dropout layers for regularization\n        self.dropout1 = nn.Dropout(p=0.5)\n        self.dropout2 = nn.Dropout(p=0.3)\n\n    def forward(self, x):\n        # Get embeddings from EfficientNetV2\n        embeddings = self.model(x)  # Should output shape (batch_size, 1280)\n\n        # Fully connected layers with dropout and activations\n        x = self.fc1(embeddings)\n        x = self.dropout1(x)\n        x = self.fc2(x)\n        x = self.dropout2(x)\n        x = self.fc3(x)\n\n        return x","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.815422Z","iopub.execute_input":"2024-10-16T19:36:52.815893Z","iopub.status.idle":"2024-10-16T19:36:52.828354Z","shell.execute_reply.started":"2024-10-16T19:36:52.815855Z","shell.execute_reply":"2024-10-16T19:36:52.826898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import torch\n\n# Load the full model\n\nmodel_path = \"/kaggle/input/fullfinetuned/pytorch/default/1/full_fine_tuned_model.pth\" \nunified_model = torch.load(model_path, map_location='cpu')  # Load directly\n","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:52.830098Z","iopub.execute_input":"2024-10-16T19:36:52.830521Z","iopub.status.idle":"2024-10-16T19:36:54.382713Z","shell.execute_reply.started":"2024-10-16T19:36:52.830455Z","shell.execute_reply":"2024-10-16T19:36:54.381545Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inference on the Test Set\n\nIn this section, we perform inference using the pre-trained model on the test dataset. The goal is to predict the probabilities of different conditions for each test image.\n\n#### Setup\n\n1. **Importing Libraries**:\n   - We import the necessary libraries, including `torch`, `pandas`, and `tqdm`. The `tqdm` library is used to display a progress bar during batch processing, providing visual feedback on the status of inference.\n\n2. **Setting the Device**:\n   - We determine whether to run the model on a GPU or CPU. If a CUDA-capable GPU is available, we use it; otherwise, we fall back to the CPU. This is done using:\n     ```python\n     device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n     ```\n\n3. **Model Preparation**:\n   - We ensure the model is set to evaluation mode by calling `unified_model.eval()`. This is crucial as it changes the behavior of certain layers (like dropout and batch normalization) to behave appropriately during inference.\n\n#### Inference Process\n\n1. **Initializing Results Dictionary**:\n   - We create a dictionary named `results` to store the output probabilities for each condition corresponding to each image. This dictionary includes:\n     - `row_id`: Unique identifier for each test image.\n     - `normal_mild`: Probability of the condition being normal or mild.\n     - `moderate`: Probability of the condition being moderate.\n     - `severe`: Probability of the condition being severe.\n\n2. **Inference Loop**:\n   - Using a `with torch.no_grad()` context manager, we disable gradient calculation to reduce memory usage and improve performance during inference.\n   - We loop through the test DataLoader (`test_loader`), processing images in batches. The `tqdm` function provides a progress bar to indicate the batch processing status.\n\n3. **Processing Batches**:\n   - For each batch of images:\n     - We transfer the images to the specified device (CPU or GPU) using `images.to(device)`.\n     - The model outputs predictions for the images using `unified_model(images)`.\n     - We apply the softmax function to the output logits to convert them into probabilities for each class, storing the results in `probs`.\n\n4. **Storing Results**:\n   - For each image in the batch, we extract the corresponding `row_id` and the predicted probabilities for each condition (normal/mild, moderate, severe). These values are appended to the results dictionary.\n\n#### Normalization and Validation\n\n1. **Creating a DataFrame**:\n   - After processing all batches, we convert the results dictionary into a pandas DataFrame named `results_df`.\n\n2. **Normalizing Probabilities**:\n   - To ensure that the predicted probabilities for each condition sum to 1, we normalize the probabilities by dividing each probability by the sum of probabilities for each image.\n\n3. **Validation**:\n   - We check for any invalid (negative) probabilities in the DataFrame. If any negative probabilities are found, a `ValueError` is raised, indicating a potential issue in the inference process.\n\n\nThis section demonstrates how to perform inference on a test dataset using the pre-trained model. By processing images in batches, we efficiently compute the probabilities for different conditions while ensuring the results are valid and properly normalized for further analysis or submission.\n\n----","metadata":{}},{"cell_type":"code","source":"import torch\nimport pandas as pd\nfrom tqdm import tqdm\nimport torch\n\n# Set device\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\n# Assuming model is already loaded and set to eval mode\nunified_model.eval()\n\n# Initialize a dictionary to store the results\nresults = {\n    'row_id': [],\n    'normal_mild': [],\n    'moderate': [],\n    'severe': []\n}\n\nprint(\"Starting inference on the test set...\")\n\nwith torch.no_grad():\n    for batch_idx, (images, row_data) in enumerate(tqdm(test_loader, desc=\"Processing batches\")):\n        images = images.to(device)\n\n        outputs = unified_model(images)\n        probs = torch.softmax(outputs, dim=1).cpu().numpy()\n\n        for i in range(len(probs)):\n            row_id = row_data['row_id'][i]\n            results['row_id'].append(row_id)\n            results['normal_mild'].append(probs[i][0])  # Probability for 'Normal/Mild'\n            results['moderate'].append(probs[i][1])     # Probability for 'Moderate'\n            results['severe'].append(probs[i][2])       # Probability for 'Severe'\n\n# Convert results to DataFrame\nresults_df = pd.DataFrame(results)\n\n# Normalize probabilities to ensure they sum to 1\nresults_df[['normal_mild', 'moderate', 'severe']] = results_df[['normal_mild', 'moderate', 'severe']].div(\n    results_df[['normal_mild', 'moderate', 'severe']].sum(axis=1), axis=0\n)\n\n# Check for any invalid values\nif (results_df[['normal_mild', 'moderate', 'severe']] < 0).any().any():\n    raise ValueError(\"Found negative probabilities in submission.\")\n\nresults_df = pd.DataFrame(results)","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:36:54.383950Z","iopub.execute_input":"2024-10-16T19:36:54.384284Z","iopub.status.idle":"2024-10-16T19:37:14.341780Z","shell.execute_reply.started":"2024-10-16T19:36:54.384250Z","shell.execute_reply":"2024-10-16T19:37:14.340753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Overview of the Results DataFrame\n\nAfter generating the predictions, we utilize the `info()` method on the `results_df` DataFrame to obtain a summary of its structure and contents. This step is crucial for understanding the data we have produced during the inference process.\n\n#### Key Aspects of `results_df.info()`:\n\n1. **DataFrame Summary:**\n   - The `info()` method provides a concise summary of the DataFrame, including:\n     - The number of entries (rows) and columns.\n     - The index range.\n     - The data types of each column.\n     - The count of non-null entries in each column.\n\n2. **Column Breakdown:**\n   - The DataFrame consists of the following columns:\n     - `row_id`: Unique identifier for each image, linking it to its respective condition and study.\n     - `normal_mild`: Probability score for the image being classified as normal or mild.\n     - `moderate`: Probability score indicating a moderate classification.\n     - `severe`: Probability score suggesting a severe classification.\n\n3. **Data Types:**\n   - The data types of the columns will typically include:\n     - `row_id`: Object (string type).\n     - `normal_mild`, `moderate`, `severe`: Float (probability values).\n\n4. **Non-null Counts:**\n   - The count of non-null entries helps verify that there are no missing values in the predictions, ensuring the integrity of the results.\n\n5. **Importance of the Summary:**\n   - By examining this information, we can confirm that the predictions have been successfully generated and structured correctly. It also helps identify any issues, such as missing values or incorrect data types, before proceeding with further analysis or visualizations.\n\nOverall, running `results_df.info()` is an important step to validate the output of our model and prepare for subsequent tasks like analysis, visualization, or reporting of the results.\n","metadata":{}},{"cell_type":"code","source":"results_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:37:14.343160Z","iopub.execute_input":"2024-10-16T19:37:14.343609Z","iopub.status.idle":"2024-10-16T19:37:14.354626Z","shell.execute_reply.started":"2024-10-16T19:37:14.343563Z","shell.execute_reply":"2024-10-16T19:37:14.353535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Averaging Results per `row_id`\n\nIn this section, we aggregate the prediction results for each unique `row_id` by calculating the average probabilities across the corresponding images. This process is essential for consolidating the predictions to reflect the overall assessment for each condition associated with a specific study.\n\n#### Key Steps:\n\n1. **Grouping by `row_id`:**\n   - We use the `groupby()` method on the `results_df` DataFrame to group the data based on the `row_id`. This ensures that we consolidate predictions for each unique image identifier.\n\n   - The `as_index=False` parameter is set to maintain `row_id` as a regular column in the resulting DataFrame instead of using it as an index.\n\n2. **Calculating Mean Probabilities:**\n   - For each `row_id`, we compute the mean of the probability scores for the classes: `normal_mild`, `moderate`, and `severe`. This results in a new DataFrame called `averaged_results_df`, which contains the average probability scores for each `row_id`.\n\n3. **Normalizing Probabilities:**\n   - To ensure that the probabilities across the three classes sum to 1 for each `row_id`, we perform normalization:\n     - We calculate the sum of the probabilities for each `row_id` using `sum(axis=1)`.\n     - Each probability column (`normal_mild`, `moderate`, `severe`) is divided by the corresponding sum of probabilities. This step ensures that the predicted probabilities are properly scaled.\n\n4. **Checking for Invalid Values:**\n   - We perform a validation check to ensure that there are no negative probability values in the normalized probabilities. This is critical, as probabilities must always be non-negative and fall within the range [0, 1].\n   - If any negative probabilities are found, a `ValueError` is raised, indicating that the submission contains invalid values.\n\n5. **Importance of Averaging and Normalization:**\n   - Averaging results across multiple predictions for the same `row_id` provides a more robust estimate of the true condition.\n   - Normalizing the probabilities is essential for interpretation, as it allows for direct comparisons among the different conditions.\n\nThis step is crucial for preparing the final prediction results, ensuring they are valid and ready for analysis, visualization, or submission.\n","metadata":{}},{"cell_type":"code","source":"# Average results per row_id\naveraged_results_df = results_df.groupby('row_id', as_index=False).mean()\n\n# Normalize probabilities to ensure they sum to 1\nsum_probs = averaged_results_df[['normal_mild', 'moderate', 'severe']].sum(axis=1)\naveraged_results_df['normal_mild'] = averaged_results_df['normal_mild'] / sum_probs\naveraged_results_df['moderate'] = averaged_results_df['moderate'] / sum_probs\naveraged_results_df['severe'] = averaged_results_df['severe'] / sum_probs\n\n# Check for any invalid values\nif (averaged_results_df[['normal_mild', 'moderate', 'severe']] < 0).any().any():\n    raise ValueError(\"Found negative probabilities in submission.\")","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:37:14.356173Z","iopub.execute_input":"2024-10-16T19:37:14.356615Z","iopub.status.idle":"2024-10-16T19:37:14.374617Z","shell.execute_reply.started":"2024-10-16T19:37:14.356578Z","shell.execute_reply":"2024-10-16T19:37:14.373559Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Verification of Probability Normalization\n\nIn this step, we verify that the normalized probabilities for each condition sum to 1 for every `row_id` in the `averaged_results_df` DataFrame. This check ensures that the normalization process was successful and that the probability values are valid.\n\n#### Key Steps:\n\n1. **Calculating the Sum of Probabilities:**\n   - We create a new column called `sum_check` in the `averaged_results_df` DataFrame. This column will hold the sum of the probabilities for the three conditions: `normal_mild`, `moderate`, and `severe`.\n   - The sum is calculated using the `sum(axis=1)` method, which sums the values across the specified columns for each row.\n\n2. **Rounding the Sum Values:**\n   - To enhance readability and ensure consistency, we apply the `round(x, 2)` function to round the sum to two decimal places. This rounding helps in clearly observing the results without excessive precision that may not be meaningful in the context of probabilities.\n\n3. **Displaying the Normalization Check:**\n   - We print a confirmation message \"Normalization Check:\" to indicate that we are about to display the results of the verification process.\n   - The `head()` method is used to show the first few entries of the `row_id` along with their corresponding `sum_check` values. This display allows us to quickly assess whether the normalization was successful across several samples.\n\n4. **Importance of the Normalization Check:**\n   - Ensuring that the sum of probabilities equals 1 is crucial for validating the model's outputs. Probabilities must adhere to the properties of a probability distribution, where the sum of all possible outcomes should equal 1.\n   - This check serves as a final validation step before proceeding to utilize or submit the results, providing confidence in the integrity of the data.\n\nBy performing this verification, we confirm that our predictions are reliable and ready for further analysis or reporting.\n","metadata":{}},{"cell_type":"code","source":"# Verify that the sum of probabilities is 1 for each row\naveraged_results_df['sum_check'] = averaged_results_df[['normal_mild', 'moderate', 'severe']].sum(axis=1).apply(lambda x: round(x, 2))\nprint(\"Normalization Check:\")\nprint(averaged_results_df[['row_id', 'sum_check']])","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:37:14.376117Z","iopub.execute_input":"2024-10-16T19:37:14.376536Z","iopub.status.idle":"2024-10-16T19:37:14.403927Z","shell.execute_reply.started":"2024-10-16T19:37:14.376489Z","shell.execute_reply":"2024-10-16T19:37:14.402691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Interpretation of Results\n\n- **Consistent Results**: Each `row_id` shows a `sum_check` value of **1.0**. This indicates that the probabilities for `normal/mild`, `moderate`, and `severe` conditions have been successfully normalized.\n  \n- **Validity of Probabilities**: The normalization ensures that the output probabilities are valid and reflect the model's confidence in its predictions.\n\n\nThe normalization check confirms that the predictions for each `row_id` are valid, reinforcing the reliability of the model's output. This step is critical before proceeding to the submission or further analysis of the results\n\n-----","metadata":{}},{"cell_type":"markdown","source":"### Preparing the Submission DataFrame\n\nIn this section, we create a new DataFrame specifically designed for submission purposes. This DataFrame will contain the final average probability results for each unique `row_id`, allowing for straightforward analysis or submission.\n\n#### Key Steps:\n\n1. **Creating the Submission DataFrame:**\n   - We define `submission_df` by selecting specific columns from the `averaged_results_df`. The columns included are:\n     - `row_id`: The unique identifier for each image, linking it to its respective condition and study.\n     - `normal_mild`: The averaged probability score indicating the likelihood that the condition is classified as normal or mild.\n     - `moderate`: The averaged probability score representing a moderate classification.\n     - `severe`: The averaged probability score suggesting a severe classification.\n\n   This selection ensures that only the relevant data for submission is retained in the new DataFrame.\n\n2. **Structure of the Submission DataFrame:**\n   - The resulting `submission_df` will have the following columns:\n     - **row_id**: A string that uniquely identifies each row.\n     - **normal_mild**: A float representing the predicted probability of the normal/mild condition.\n     - **moderate**: A float representing the predicted probability of the moderate condition.\n     - **severe**: A float representing the predicted probability of the severe condition.\n\n3. **Viewing the Submission DataFrame:**\n   - By calling `submission_df`, we can preview the structure and contents of the DataFrame. This step is essential for verifying that the data has been correctly organized and is ready for further processing, such as exporting to a CSV file for submission or reporting.\n\n4. **Importance of the Submission DataFrame:**\n   - The `submission_df` serves as the final output of our inference process. It consolidates all the predictions into a format that can be easily interpreted, shared, or submitted to relevant stakeholders or platforms.\n   - Ensuring that the data is structured correctly is crucial for effective communication of the model’s findings and performance on the test dataset.\n\nThis preparation step is vital for the completion of the project, marking the transition from model inference to result dissemination.\n","metadata":{}},{"cell_type":"code","source":"submission_df = averaged_results_df[['row_id', 'normal_mild', 'moderate', 'severe']]\nsubmission_df","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:37:14.405357Z","iopub.execute_input":"2024-10-16T19:37:14.405855Z","iopub.status.idle":"2024-10-16T19:37:14.427377Z","shell.execute_reply.started":"2024-10-16T19:37:14.405806Z","shell.execute_reply":"2024-10-16T19:37:14.426168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Saving the Submission File\n\nIn this final step, we save the prepared submission DataFrame (`submission_df`) to a CSV file. This file can be used for further analysis, sharing, or submission to relevant platforms.\n\n#### Key Steps:\n\n1. **Exporting the DataFrame to CSV:**\n   - We utilize the `to_csv()` method from the pandas library to export `submission_df` to a CSV file.\n   - The argument `index=False` is specified to prevent pandas from writing row indices to the CSV file. This is important because we want to keep the file clean and ensure it only contains the relevant data columns.\n\n2. **Naming the Submission File:**\n   - The submission file is named `submission.csv`, and this name is indicated in the print statement for clarity. The CSV file will contain the `row_id` and the corresponding averaged probability scores for the conditions: normal/mild, moderate, and severe.\n\n3. **Outputting a Confirmation Message:**\n   - A confirmation message is printed to the console to inform the user that the submission file has been successfully saved. This provides assurance that the data has been correctly exported.\n\n4. **Saving to Kaggle Working Directory:**\n   - The command `submission_df.to_csv('/kaggle/working/submission.csv', index=False)` ensures that the file is saved to the Kaggle working directory, making it accessible for download or further use within the Kaggle environment.\n\n5. **Importance of Saving the Submission File:**\n   - Saving the predictions in a structured format (CSV) is essential for effective communication of results.\n   - The submission file can be submitted to competitions or assessments where model performance needs to be evaluated.\n   - This step marks the conclusion of the data processing and inference pipeline, transitioning from analysis to actionable outcomes.\n\nBy completing this step, we ensure that our results are preserved and ready for future reference or evaluation.\n\n------------","metadata":{}},{"cell_type":"code","source":"submission_df.to_csv('submission.csv', index=False)\n# Save the submission file\nsubmission_df.to_csv('/kaggle/working/submission.csv', index=False)\nprint(\"Submission file saved as 'submission.csv'.\")","metadata":{"execution":{"iopub.status.busy":"2024-10-16T19:37:14.428914Z","iopub.execute_input":"2024-10-16T19:37:14.429310Z","iopub.status.idle":"2024-10-16T19:37:14.444904Z","shell.execute_reply.started":"2024-10-16T19:37:14.429271Z","shell.execute_reply":"2024-10-16T19:37:14.443773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion\n\nIn this notebook, we successfully implemented the inference process for the RSNA Lumbar Spine Degenerative Classification task using a pre-trained Convolutional Neural Network (CNN). The key steps undertaken include:\n\n1. **Data Preparation:** \n   - We loaded and processed the test dataset, ensuring that all images were correctly accessed and organized.\n\n2. **Model Loading and Predictions:** \n   - We utilized a pre-trained CNN model to make predictions on the test dataset. The model output was carefully handled to ensure accurate results.\n\n3. **Results Aggregation and Normalization:** \n   - We computed the average probabilities for each condition across multiple images corresponding to each `row_id`, ensuring that the probabilities were normalized to sum to 1.\n\n4. **Verification:** \n   - A final check confirmed that the normalized probabilities adhered to the fundamental properties of probability distributions.\n\n5. **Submission Preparation:**\n   - The processed results were saved in a structured CSV format, ready for submission or further analysis.\n\nThrough these steps, we have built a robust inference pipeline that can be adapted for similar classification tasks in medical imaging. The results generated can be valuable for clinical evaluations and decision-making regarding lumbar spine conditions. Future work may focus on improving model accuracy through fine-tuning and exploring additional data augmentation techniques.\n\n## Thank you for reviewing this notebook, and I look forward to any questions or feedback!\n\n-----------","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}