{"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":71549,"databundleVersionId":8561470,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Dataset creation","metadata":{}},{"cell_type":"markdown","source":"#### Here we will: \n#### - 🔍 Analyze the data and get some important **insights**.\n#### - 📊 Create dataset **dataframe** with all the necessary info about MRI scans and paths to them. \n#### - 📝 Write a consise **PyTorch dataset**, so you could looks at the samples.","metadata":{}},{"cell_type":"code","source":"import os\nimport cv2\nimport pydicom\nimport torch\nimport numpy as np\nimport pandas as pd\nimport plotly.express as px\nimport plotly.graph_objects as go\n\nfrom typing import Tuple, Union\nfrom tqdm import tqdm\nfrom pathlib import Path\nfrom torchvision import transforms\nfrom torch.utils.data import Dataset","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:10.174602Z","iopub.execute_input":"2024-09-22T12:56:10.175004Z","iopub.status.idle":"2024-09-22T12:56:16.306213Z","shell.execute_reply.started":"2024-09-22T12:56:10.174950Z","shell.execute_reply":"2024-09-22T12:56:16.305082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_folder = \"/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification\"\n\ntrain_dataframe = pd.read_csv(os.path.join(data_folder, \"train.csv\"))\ntrain_dataframe.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:16.308518Z","iopub.execute_input":"2024-09-22T12:56:16.309254Z","iopub.status.idle":"2024-09-22T12:56:16.374461Z","shell.execute_reply.started":"2024-09-22T12:56:16.309201Z","shell.execute_reply":"2024-09-22T12:56:16.373199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### So, we have ***1975*** patients each with a unique `study_id`. \n#### There are **three** diseases: *Spinal Canal Stenosis, Subarticular Stenosis and Neural Foraminal Narrowing* that can be either *Normal/Mild*, **Moderate** or ***Severe***.\n#### The disease may occur in nervs between different vertebrae of *Lumbar* denoted with **L1/L2,L2/L3, L3/L4, L4/L5, L5/S1**.\n\n
","metadata":{}},{"cell_type":"code","source":"train_descriptions_dataframe = pd.read_csv(os.path.join(data_folder, \"train_series_descriptions.csv\"))\ntrain_descriptions_dataframe.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:16.375999Z","iopub.execute_input":"2024-09-22T12:56:16.376374Z","iopub.status.idle":"2024-09-22T12:56:16.399541Z","shell.execute_reply.started":"2024-09-22T12:56:16.376335Z","shell.execute_reply":"2024-09-22T12:56:16.398279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_descriptions_dataframe['series_description'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:16.401878Z","iopub.execute_input":"2024-09-22T12:56:16.402259Z","iopub.status.idle":"2024-09-22T12:56:16.417157Z","shell.execute_reply.started":"2024-09-22T12:56:16.402220Z","shell.execute_reply":"2024-09-22T12:56:16.415974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### For each patient we have at least **three** different MRI scans: **Axial T2**, **Sagittal T1** and **Sagittal T2/STIR**. \n#### (Note that if we have more than 3 scans, it is mostly additional Axial T2 MRI).\n\n
\n\n\n#### In case you wonder what's the difference between T1 and T2:\n#### - T1 scans are good for anatomical detail, showing structures like fat and post-contrast enhancement.\n#### - T2 scans emphasize areas of high water content, making them better for identifying pathologies related to fluid, inflammation, or edema.","metadata":{}},{"cell_type":"code","source":"train_label_coordinates_dataframe = pd.read_csv(os.path.join(data_folder, \"train_label_coordinates.csv\"))\ntrain_label_coordinates_dataframe.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:16.418804Z","iopub.execute_input":"2024-09-22T12:56:16.419332Z","iopub.status.idle":"2024-09-22T12:56:16.542164Z","shell.execute_reply.started":"2024-09-22T12:56:16.419278Z","shell.execute_reply":"2024-09-22T12:56:16.541106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### We also have a dataframe with the coordinates of the center of the area that defined the label. **Could they be used to train a better model?**","metadata":{}},{"cell_type":"code","source":"def plot_conditions_coordinates_distribution():\n fig = px.scatter(\n train_label_coordinates_dataframe,\n x='x',\n y='y',\n color='condition',\n opacity=0.33,\n title='Conditions coordinates',\n )\n \n fig.update_layout(\n title={\n 'text': 'Conditions Coordinates Distribution',\n 'y': 0.99,\n 'x': 0.5,\n 'xanchor': 'center',\n 'yanchor': 'top',\n 'font': {'size': 24, 'color': '#4f4f4f'}\n },\n xaxis_title='X Coordinate',\n yaxis_title='Y Coordinate',\n legend_title_text='Condition',\n xaxis_range=[0, train_label_coordinates_dataframe['x'].max()],\n yaxis_range=[0, train_label_coordinates_dataframe['y'].max()],\n width=1200,\n height=600,\n margin=dict(l=40, r=40, b=40, t=80),\n font=dict(size=18),\n )\n \n fig.update_xaxes(\n showgrid=True, gridwidth=0.7, gridcolor='gray', \n title_font=dict(size=20), tickfont=dict(size=16)\n )\n fig.update_yaxes(\n showgrid=True, gridwidth=0.7, gridcolor='gray', \n title_font=dict(size=20), tickfont=dict(size=16)\n )\n fig.show()\n\nplot_conditions_coordinates_distribution()","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:16.543455Z","iopub.execute_input":"2024-09-22T12:56:16.543856Z","iopub.status.idle":"2024-09-22T12:56:18.712578Z","shell.execute_reply.started":"2024-09-22T12:56:16.543814Z","shell.execute_reply":"2024-09-22T12:56:18.710353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Maybe subarticalar stenosis may be more relative to vertebrea in the image center like L3/L4? We will come back to coordinates and see how to use them later","metadata":{}},{"cell_type":"code","source":"def parse_descriptions(image_folders_path: str, descriptions_path: str) -> dict:\n \"\"\"\n Parse and structure metadata for medical image scans and their descriptions.\n\n This function creates a metadata dictionary for each scan folder. Each entry contains the folder path, \n the SeriesInstanceUIDs of the scans within that folder, and the corresponding series descriptions.\n\n The structure of the returned metadata is as follows:\n\n .. code-block:: python\n\n metadata = {\n 'StudyInstanceUID': {\n 'folder_path': str, # Path to the folder containing the scan series.\n 'SeriesInstanceUIDs': list, # List of SeriesInstanceUIDs for the study.\n 'SeriesDescriptions': list # List of Series descriptions for each series.\n },\n ...\n }\n\n Parameters:\n ----------\n image_folders_path : str\n Path to the folder containing subdirectories for each study. Each subdirectory represents a unique scan study.\n descriptions_path : str\n Path to the CSV file containing scan descriptions, including `study_id` and `series_id` columns.\n\n Returns:\n ----------\n dict\n A dictionary where each key is a `StudyInstanceUID` and the value is another dictionary containing:\n - `folder_path`: str, the path to the study folder.\n - `SeriesInstanceUIDs`: list of str, the UIDs of each series in the study folder.\n - `SeriesDescriptions`: list of str, the descriptions of the corresponding series.\n \n Notes:\n ----------\n - This function skips folders or files containing '.DS' in their name.\n - It attempts to match the `study_id` and `series_id` from the descriptions CSV with the actual folder and series names.\n - If a series description is not found, an error is printed for the specific series and study.\n\n Raises:\n ----------\n IndexError\n If a matching series description for a study and series cannot be found in the descriptions CSV.\n \"\"\"\n image_folders = list(filter(lambda x: x.find('.DS') == -1, os.listdir(image_folders_path)))\n descriptions_df = pd.read_csv(descriptions_path)\n \n metadata = {\n folder_name: {\n 'folder_path': os.path.join(image_folders_path, folder_name), \n 'SeriesInstanceUIDs': list(\n filter(lambda x: x.find('.DS') == -1, os.listdir(os.path.join(image_folders_path, folder_name)))\n )\n } for folder_name in image_folders\n }\n\n # grabs the corresponding series descriptions\n for k in tqdm(metadata, desc=\"Parsing metadata\"):\n for s in metadata[k]['SeriesInstanceUIDs']:\n if 'SeriesDescriptions' not in metadata[k]:\n metadata[k]['SeriesDescriptions'] = []\n try:\n metadata[k]['SeriesDescriptions'].append(\n descriptions_df[(descriptions_df['study_id'] == int(k)) & \n (descriptions_df['series_id'] == int(s))]['series_description'].iloc[0])\n except IndexError:\n print(\"Failed on\", s, k)\n \n return metadata","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:18.714306Z","iopub.execute_input":"2024-09-22T12:56:18.714713Z","iopub.status.idle":"2024-09-22T12:56:18.725169Z","shell.execute_reply.started":"2024-09-22T12:56:18.714671Z","shell.execute_reply":"2024-09-22T12:56:18.724069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def create_samples_dataframe(image_folders_path: str, labels_path: str, descriptions_path: str, label_coordinates_path: str) -> pd.DataFrame:\n \"\"\"\n Create a DataFrame containing metadata and labels for DICOM image samples.\n\n This function combines metadata from the image folders, label coordinates, descriptions, and labels\n to generate a comprehensive DataFrame for each image sample, including its corresponding labels,\n coordinates (if applicable), and file paths.\n\n The generated DataFrame contains the following columns:\n \n - `study_id`: int, the ID of the study.\n - `series_id`: int, the ID of the series within the study.\n - `instance_number`: int, the DICOM instance number for the image.\n - `image_path`: str, the file path to the DICOM image.\n - `scan_type`: str, the type of scan (derived from SeriesDescriptions).\n - `condition`: str, the medical condition (if applicable).\n - `level`: str, the spine level (if applicable).\n - `x`: float, the x-coordinate of the condition point (if applicable).\n - `y`: float, the y-coordinate of the condition point (if applicable).\n - `label`: int, the label for the condition (0 for normal cases, and 1, 2, 3 for increasing severity).\n\n Parameters:\n ----------\n image_folders_path : str\n Path to the folder containing the image study subfolders.\n labels_path : str\n Path to the CSV file containing the labels for each study and series.\n descriptions_path : str\n Path to the CSV file containing the descriptions of each series.\n label_coordinates_path : str\n Path to the CSV file containing the label coordinates (x, y) for specific series in the dataset.\n\n Returns:\n ----------\n pd.DataFrame\n A pandas DataFrame where each row corresponds to a DICOM image sample, including metadata, coordinates, \n and the associated labels.\n\n Notes:\n ----------\n - The `labels_mapping` dictionary maps label descriptions (`Normal/Mild`, `Moderate`, `Severe`) to integer values.\n - Images without coordinates (normal cases) are labeled with 0.\n - Series descriptions and labels are matched using `study_id` and `series_id` from the input CSV files.\n\n Raises:\n ----------\n FileNotFoundError\n If any of the provided paths (e.g., image folders, labels CSV, descriptions CSV) do not exist or cannot be read.\n \"\"\"\n metadata = parse_descriptions(image_folders_path, descriptions_path)\n label_coordinates_df = pd.read_csv(label_coordinates_path)\n labels_df = pd.read_csv(labels_path)\n \n labels_mapping = {\n \"Normal/Mild\": 1,\n \"Moderate\": 2,\n \"Severe\": 3,\n }\n \n data = []\n for study_id, patient in tqdm(metadata.items(), desc=\"Creating a dataframe\", total=len(metadata)):\n labels_per_study = labels_df[labels_df[\"study_id\"] == int(study_id)]\n for series_folder, scan_type in zip(patient[\"SeriesInstanceUIDs\"], patient[\"SeriesDescriptions\"]):\n # Pre-filter the label_coordinates_df for this specific study and series\n relevant_coordinates = label_coordinates_df[\n (label_coordinates_df[\"study_id\"] == int(study_id)) &\n (label_coordinates_df[\"series_id\"] == int(series_folder))\n ]\n # Dictionary of instance_number -> relevant_coordinates\n dicom_files = list(Path(os.path.join(patient[\"folder_path\"], series_folder)).rglob(\"*.dcm\"))\n instance_map = {int(Path(dicom_file).stem): dicom_file for dicom_file in dicom_files}\n \n # Cases with coordinates\n for _, entry in relevant_coordinates.iterrows():\n instance_number = entry[\"instance_number\"]\n if instance_number not in instance_map:\n continue\n \n dicom_file = instance_map[instance_number]\n condition_column = entry[\"condition\"].lower().replace(\" \", \"_\") + \"_\" + entry[\"level\"].lower().replace(\"/\", \"_\")\n label_row = labels_per_study[condition_column].iloc[0]\n \n if not isinstance(label_row, str):\n continue\n \n data.append({\n \"study_id\": int(study_id),\n \"series_id\": int(series_folder),\n \"instance_number\": instance_number,\n \"image_path\": str(dicom_file),\n \"scan_type\": scan_type,\n \"condition\": entry[\"condition\"],\n \"level\": entry[\"level\"],\n \"x\": entry[\"x\"],\n \"y\": entry[\"y\"],\n \"label\": labels_mapping[label_row],\n })\n \n # Rows without coordinates (Normal cases)\n untagged_instances = set(instance_map.keys()) - set(relevant_coordinates[\"instance_number\"].values)\n for instance_number in untagged_instances:\n dicom_file = instance_map[instance_number]\n data.append({\n \"study_id\": int(study_id),\n \"series_id\": int(series_folder),\n \"instance_number\": instance_number,\n \"image_path\": str(dicom_file),\n \"scan_type\": scan_type,\n \"condition\": None,\n \"level\": None,\n \"x\": None,\n \"y\": None,\n \"label\": 0,\n })\n\n return pd.DataFrame(data)","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:18.726390Z","iopub.execute_input":"2024-09-22T12:56:18.726771Z","iopub.status.idle":"2024-09-22T12:56:18.777518Z","shell.execute_reply.started":"2024-09-22T12:56:18.726733Z","shell.execute_reply":"2024-09-22T12:56:18.776353Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def save_samples_dataframe(dataframe: pd.DataFrame, save_path: str) -> None:\n dataframe.to_csv(save_path, index=False)","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:18.779103Z","iopub.execute_input":"2024-09-22T12:56:18.779641Z","iopub.status.idle":"2024-09-22T12:56:18.797682Z","shell.execute_reply.started":"2024-09-22T12:56:18.779589Z","shell.execute_reply":"2024-09-22T12:56:18.796177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_samples = create_samples_dataframe(\n os.path.join(data_folder, \"train_images\"),\n os.path.join(data_folder, \"train.csv\"),\n os.path.join(data_folder, \"train_series_descriptions.csv\"),\n os.path.join(data_folder, \"train_label_coordinates.csv\")\n)\nsave_samples_dataframe(train_samples, \"all_train_samples.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:56:18.801441Z","iopub.execute_input":"2024-09-22T12:56:18.801848Z","iopub.status.idle":"2024-09-22T12:57:41.581783Z","shell.execute_reply.started":"2024-09-22T12:56:18.801810Z","shell.execute_reply":"2024-09-22T12:57:41.580520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_samples","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:41.583449Z","iopub.execute_input":"2024-09-22T12:57:41.583810Z","iopub.status.idle":"2024-09-22T12:57:41.602036Z","shell.execute_reply.started":"2024-09-22T12:57:41.583773Z","shell.execute_reply":"2024-09-22T12:57:41.600335Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_column_distribution(dataframe: pd.DataFrame, column_name: str):\n value_counts = dataframe[column_name].value_counts().reset_index()\n value_counts.columns = ['Category', 'Count']\n fig = px.pie(value_counts, names='Category', values='Count', title='Distribution of Categories', hole=0.3)\n fig.show()\n\nplot_column_distribution(train_samples, \"scan_type\")","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:41.603563Z","iopub.execute_input":"2024-09-22T12:57:41.603942Z","iopub.status.idle":"2024-09-22T12:57:41.696973Z","shell.execute_reply.started":"2024-09-22T12:57:41.603904Z","shell.execute_reply":"2024-09-22T12:57:41.695861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Let's split the dataframes based on the MRI scan type:","metadata":{}},{"cell_type":"code","source":"for scan_type in train_samples['scan_type'].unique():\n df = train_samples[train_samples['scan_type'] == scan_type]\n df.to_csv(f\"{scan_type.lower().replace(' ', '_').replace('/stir', '')}_train_samples.csv\", index=False)","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:41.698246Z","iopub.execute_input":"2024-09-22T12:57:41.698665Z","iopub.status.idle":"2024-09-22T12:57:43.134802Z","shell.execute_reply.started":"2024-09-22T12:57:41.698625Z","shell.execute_reply":"2024-09-22T12:57:43.133562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let look at the labels distribution now:","metadata":{}},{"cell_type":"code","source":"def plot_labels_distribution(dataframe: pd.DataFrame, label_column: str = \"label\"):\n labels_mapping = {\n 0: \"Normal\",\n 1: \"Mild\",\n 2: \"Moderate\",\n 3: \"Severe\",\n }\n\n dataframe['label_name'] = dataframe[label_column].map(labels_mapping)\n fig = px.pie(dataframe, names='label_name', title='Label Distribution', hole=0.3)\n fig.show()\n \nplot_labels_distribution(train_samples)","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:43.136115Z","iopub.execute_input":"2024-09-22T12:57:43.137306Z","iopub.status.idle":"2024-09-22T12:57:43.553362Z","shell.execute_reply.started":"2024-09-22T12:57:43.137237Z","shell.execute_reply":"2024-09-22T12:57:43.552138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### As expected, *most* patients in the provided dataset **dot not** have any of the conditions. However, this means that we will have to deal with **class imbalance** in the future.","metadata":{}},{"cell_type":"markdown","source":"### Now let's finally build a PyTorch dataset to use in our machine learning processing later!","metadata":{}},{"cell_type":"code","source":"class LumbarSpineDataset(Dataset):\n \"\"\"\n A PyTorch Dataset for loading and processing lumbar spine images and labels,\n including creating Gaussian masks for certain image points.\n\n Attributes:\n ----------\n _image_size : Tuple[int, int]\n The target size for the image (height, width).\n _mask_sigma_coefficient : int\n Coefficient used to calculate the sigma for the Gaussian mask.\n _mask_sigma : Tuple[int, int]\n Sigma values (x, y) for generating the Gaussian mask.\n _samples : pd.DataFrame\n DataFrame containing paths to images and their associated metadata.\n _encoded_labels : pd.DataFrame\n One-hot encoded labels for the dataset samples.\n _transform : torchvision.transforms.Compose\n A set of transformations applied to the images and masks.\n \"\"\"\n\n def __init__(self, samples_df_path: str, image_size: Tuple[int, int] = (512, 512)):\n \"\"\"\n Initialize the LumbarSpineDataset.\n\n Parameters:\n ----------\n samples_df_path : str\n The path to the CSV file containing dataset samples, including image paths and metadata.\n image_size : Tuple[int, int], optional\n The desired image size for all samples. Default is (512, 512).\n \"\"\"\n self._image_size = image_size\n self._mask_sigma_coefficient = 64 # Manually chosen\n self._mask_sigma = (\n max(image_size[0] // self._mask_sigma_coefficient, 1),\n max(image_size[1] // self._mask_sigma_coefficient, 1)\n )\n self._samples = pd.read_csv(samples_df_path)\n self._encoded_labels = pd.get_dummies(self._samples[\"label\"], prefix=\"label\", dtype=\"float32\")\n self._transform = transforms.Compose([\n transforms.ToPILImage(),\n transforms.Resize(image_size),\n transforms.ToTensor(),\n ])\n \n def __len__(self) -> int:\n \"\"\"\n Return the number of samples in the dataset.\n\n Returns:\n ----------\n int\n The total number of samples in the dataset.\n \"\"\"\n return len(self._samples)\n \n def __getitem__(self, idx: int) -> Tuple[torch.Tensor, torch.Tensor, np.ndarray]:\n \"\"\"\n Retrieve a sample from the dataset.\n\n Parameters:\n ----------\n idx : int\n Index of the sample to retrieve.\n\n Returns:\n ----------\n Tuple[torch.Tensor, torch.Tensor, np.ndarray]\n A tuple containing the image tensor, the Gaussian mask tensor, and the encoded labels.\n \"\"\"\n sample = self._samples.iloc[idx]\n image = self.read_dicom_data(sample[\"image_path\"])\n coordinates = (sample[\"x\"], sample[\"y\"]) if sample[\"label\"] > 0 else (-1, -1)\n labels = self._encoded_labels.iloc[idx].values\n \n gaussian_mask = self.create_gaussian_mask(\n image.shape,\n coordinates,\n self._mask_sigma\n ) \n \n if self._transform:\n image = self._transform(image)\n gaussian_mask = self._transform(gaussian_mask)\n \n return image, gaussian_mask, labels\n \n @staticmethod\n def read_dicom_data(dicom_file_path: str) -> np.ndarray:\n \"\"\"\n Read and convert a DICOM file to a NumPy array.\n\n Parameters:\n ----------\n dicom_file_path : str\n The path to the DICOM file.\n\n Returns:\n ----------\n np.ndarray\n The image data from the DICOM file as a NumPy array.\n \"\"\"\n dicom_data = pydicom.dcmread(dicom_file_path)\n return dicom_data.pixel_array.astype(\"uint8\")\n \n @staticmethod\n def create_gaussian_mask(shape: Tuple[int, int], center: Tuple[float, float], sigma: Union[float, Tuple[float, float]]) -> np.ndarray:\n \"\"\"\n Create a 2D Gaussian mask array centered at a specified point.\n\n Parameters:\n ----------\n shape : Tuple[int, int]\n The shape of the image/mask array.\n center : Tuple[float, float]\n The coordinates of the Gaussian center. If negative, returns a zero mask.\n sigma : Union[float, Tuple[float, float]]\n The standard deviation(s) of the Gaussian. Can be a scalar or a tuple for x and y directions.\n\n Returns:\n ----------\n np.ndarray\n The generated Gaussian mask as a 2D array.\n \"\"\"\n if center[0] < 0 or center[1] < 0:\n return np.zeros(shape)\n \n if np.isscalar(sigma):\n sigma_x = sigma_y = sigma\n else:\n sigma_x, sigma_y = sigma\n \n y = np.arange(shape[0]).reshape(-1, 1)\n x = np.arange(shape[1]).reshape(1, -1)\n gauss = np.exp(-(((x - center[0]) ** 2) / (2 * sigma_x ** 2) + ((y - center[1]) ** 2) / (2 * sigma_y ** 2)))\n \n return (gauss * 255).astype(\"uint8\")\n \n def show_sample(self, index: int, alpha: float = 0.75, steps: int = 5) -> None:\n \"\"\"\n Display a dataset sample with a Gaussian mask overlay using Plotly.\n\n Parameters:\n ----------\n index : int\n The index of the sample to display.\n alpha : float, optional\n The transparency of the mask overlay. Default is 0.75.\n steps : int, optional\n The number of steps in the transparency slider. Default is 5.\n \"\"\"\n sample, mask, label = self.__getitem__(index)\n rgb_sample = cv2.cvtColor(sample.squeeze().numpy(), cv2.COLOR_GRAY2RGB)\n mask = mask.squeeze().numpy()\n \n rgb_mask = cv2.merge((mask, np.zeros_like(mask), np.zeros_like(mask)))\n \n def create_overlay(alpha_value):\n return ((rgb_sample + alpha_value * rgb_mask).clip(0, 1) * 255).astype(np.uint8)\n \n result = create_overlay(alpha)\n fig = go.Figure(go.Image(z=result))\n \n step_values = np.linspace(0, 1, steps)\n overlays = [create_overlay(step_alpha) for step_alpha in step_values]\n \n sliders = [dict(\n active=int(alpha * steps),\n currentvalue={\"prefix\": \"Alpha: \"},\n pad={\"t\": 50},\n steps=[\n dict(\n method=\"restyle\",\n args=[{\"z\": [overlay]}],\n label=f\"{step_alpha:.2f}\"\n ) for overlay, step_alpha in zip(overlays, step_values)\n ]\n )]\n fig.update_layout(\n sliders=sliders,\n title=f\"Sample {index} with Mask\",\n coloraxis_showscale=False\n )\n fig.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:43.554969Z","iopub.execute_input":"2024-09-22T12:57:43.555343Z","iopub.status.idle":"2024-09-22T12:57:43.581235Z","shell.execute_reply.started":"2024-09-22T12:57:43.555303Z","shell.execute_reply":"2024-09-22T12:57:43.580038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"axial_t2_train_dataset = LumbarSpineDataset(\"/kaggle/working/axial_t2_train_samples.csv\")\nsagittal_t1_train_dataset = LumbarSpineDataset(\"/kaggle/working/sagittal_t1_train_samples.csv\")\nsagittal_t2_train_dataset = LumbarSpineDataset(\"/kaggle/working/sagittal_t2_train_samples.csv\")","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:43.582725Z","iopub.execute_input":"2024-09-22T12:57:43.583103Z","iopub.status.idle":"2024-09-22T12:57:44.116357Z","shell.execute_reply.started":"2024-09-22T12:57:43.583061Z","shell.execute_reply":"2024-09-22T12:57:44.115048Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### This dataset allows us also to retrieve a **gaussian mask** centered at the label coordinates for samples that do have the condition (the mask has only zeroes for others). \n#### This may help us to train more **robust** models that can generalize better to new data (hopefully). ","metadata":{}},{"cell_type":"code","source":"axial_t2_train_dataset.show_sample(0)","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:44.117822Z","iopub.execute_input":"2024-09-22T12:57:44.118279Z","iopub.status.idle":"2024-09-22T12:57:44.888928Z","shell.execute_reply.started":"2024-09-22T12:57:44.118225Z","shell.execute_reply":"2024-09-22T12:57:44.887655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sagittal_t1_train_dataset.show_sample(0)","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:44.890509Z","iopub.execute_input":"2024-09-22T12:57:44.890909Z","iopub.status.idle":"2024-09-22T12:57:45.522738Z","shell.execute_reply.started":"2024-09-22T12:57:44.890866Z","shell.execute_reply":"2024-09-22T12:57:45.521018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sagittal_t2_train_dataset.show_sample(0)","metadata":{"execution":{"iopub.status.busy":"2024-09-22T12:57:45.525008Z","iopub.execute_input":"2024-09-22T12:57:45.525715Z","iopub.status.idle":"2024-09-22T12:57:46.125287Z","shell.execute_reply.started":"2024-09-22T12:57:45.525636Z","shell.execute_reply":"2024-09-22T12:57:46.123270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### This dataset is ready to be used with your PyTorch **DataLoader**.","metadata":{}},{"cell_type":"markdown","source":"# Thank you for reading\n#### If you find this helpful, don't forget to upvote","metadata":{}}]}