{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":71549,"databundleVersionId":8561470,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"colab":{"name":"RSNA2024 EDA + Vis","provenance":[]}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# RSNA 2024 Lumbar Spine Degenerative Classification\n\n## Overview\nThe goal of this competition is to detect and cassify degeneratiove spine conditions using lumbar spine MR images. There are three core conditions: **spinal canal stenosis**, **neural foraminal narrowing**, and **subarticular stenosis**\n\n## Anatomical overview\n\nThe spine is a complex structure divided into four regions:\n\n- **Cervical region**: Consists of 7 vertebral bodies, located in the neck.\n- **Thoracic region**: Contains 12 vertebral bodies, located in the upper and mid-back.\n- **Lumbar region**: Comprises 5 vertebral bodies, found in the lower back.\n- **Sacral region**: Made up of 3-5 fused vertebral bodies, situated at the base of the spine.\n\nFor each study in this dataset we have following disc levels: L1/L2, L2/L3, L3/L4, L4/L5, and L5/S1.\n\nThe disc levels refer to the specific regions of the lumbar spine where the intervertebral discs are located.\n\n- **L1/L2**: The disc between the first (L1) and second (L2) lumbar vertebrae\n- **L2/L3**: The disc between the second (L2) and third (L3) lumbar vertebrae\n- **L3/L4**: The disc between the third (L3) and fourth (L4) lumbar vertebrae\n- **L4/L5**: The disc between the fourth (L4) and fifth (L5) lumbar vertebrae\n- **L5/S1**: The disc between the fifth lumbar vertebra (L5) and the first sacral vertebra (S1)\n\n### Canal Stenosis\n\n**Canal stenosis** involves impingement of the spinal canal, the passageway through which the spinal cord travels. Causes include bulging discs, trauma, bony outgrowths (osteophytes), and thickened ligaments. The degree of compression is typically assessed in the axial plane.\n\n
\n
\n \"Canal\n
\n
\n \"Grading\n
\n
\nFrom Radiopedia

\n\n### Forminal narrowing stenosis\n\nForamina, the openings through which spinal nerves exit, can become narrowed, a condition best visualized in the sagittal plane (a vertical slice parallel to the spine).\n- Foraminal narrowing results in nerve compression, leading to pain along the affected nerve's distribution, which is the path the nerve travels in the body.\n\n
\n
\n \"Foraminal\n
\n
\n \"Grading\n
\n


\n\n### Subarticular stenosis\n\n**Subarticular stenosis** occurs when there is compression of the spinal cord in the subarticular zone, a region just below the articular surface of the vertebrae. This condition is best visualized in the axial plane (a horizontal slice perpendicular to the spine). The compression in this zone is often due to similar factors that cause foraminal narrowing.\n\n
\n
\n \"Subarticular\n
\n
\n \"Grading\n
\n
\n\nLeft image from Miami Neuroscience Center

\n\n\n## MRI overview\n\n\n**MRI imaging** (Magnetic Resonance Imaging) of the spine can be performed in three planes: axial, sagittal, and coronal. For this challenge, the axial and sagittal planes are most relevant:\n\n- **Axial plane**: Horizontal slices perpendicular to the spine.\n- **Sagittal plane**: Vertical slices parallel to the spine.\n\nMRI images come in two primary types:\n\n- **T1-weighted images**:\n - These images highlight fat, making the inner parts of bones appear brighter.\n- **T2-weighted images**:\n - These images highlight water, making the spinal canal appear brighter.\n\n\nCase Western Reserve University

\n\nUnlike CT images (Computed Tomography), MRI images lack standardized pixel values, requiring careful consideration for standardization during analysis.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","id":"HilUnX6FTl-N"}},{"cell_type":"markdown","source":"## Imports and Setup","metadata":{"id":"kNnaAu5STl-P"}},{"cell_type":"code","source":"import os\nimport pydicom\nimport numpy as np\nimport pandas as pd\nimport math, random\nfrom glob import glob\nfrom tqdm import tqdm\nimport seaborn as sns\nimport pydicom as dicom\nfrom pathlib import Path\nimport matplotlib.pyplot as plt\nfrom matplotlib import animation, rc; rc('animation', html='jshtml')\n\nimport torch\nfrom torch import nn\nfrom torch.optim import AdamW\nimport torch.nn.functional as F\nfrom torch.utils.data import DataLoader, Dataset\n\nimport timm\nfrom transformers import get_cosine_schedule_with_warmup\n\nimport albumentations as A\n","metadata":{"id":"NoJR_svoTl-R","execution":{"iopub.status.busy":"2024-09-03T09:52:06.032039Z","iopub.execute_input":"2024-09-03T09:52:06.032641Z","iopub.status.idle":"2024-09-03T09:52:06.043954Z","shell.execute_reply.started":"2024-09-03T09:52:06.032582Z","shell.execute_reply":"2024-09-03T09:52:06.042122Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_rows', 100)\npd.set_option('display.max_columns', 100)\npd.set_option('display.width', 1000)\n\ndata_dir = Path('/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification')","metadata":{"id":"ADodj6wFTl-T","execution":{"iopub.status.busy":"2024-09-03T09:52:06.046330Z","iopub.execute_input":"2024-09-03T09:52:06.046843Z","iopub.status.idle":"2024-09-03T09:52:06.060194Z","shell.execute_reply.started":"2024-09-03T09:52:06.046798Z","shell.execute_reply":"2024-09-03T09:52:06.058154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helper functions","metadata":{"id":"cthehl6ITl-V"}},{"cell_type":"code","source":"def visualize_condition_counts(df, title, path=None):\n\n \"\"\"\n Visualize condition counts on training set\n\n Parameters\n ----------\n df: pandas.DataFrame\n Counts and percentages of conditions\n\n title: str\n Title of the plot\n\n path: str, pathlib.Path or None\n Path of the output file (if path is None, plot is displayed with selected backend)\n \"\"\"\n\n fig, ax = plt.subplots(figsize=(24, 16))\n\n ax.barh(\n y=np.arange(df.shape[0] // 3) - 0.2,\n width=df['count'].values[0::3],\n height=0.2,\n align='center',\n label='Normal/Mild'\n )\n ax.barh(\n y=np.arange(df.shape[0] // 3),\n width=df['count'].values[1::3],\n height=0.2,\n align='center',\n label='Moderate'\n )\n ax.barh(\n y=np.arange(df.shape[0] // 3) + 0.2,\n width=df['count'].values[2::3],\n height=0.2,\n align='center',\n label='Severe'\n )\n\n ax.set_yticks(np.arange(df.shape[0] // 3))\n ax.set_yticklabels([\n f'{level}\\nNormal Count: {normal_count} ({normal_percentage:.2f}%)\\nModerate Count: {moderate_count} ({moderate_percentage:.2f}%)\\nSevere Count: {severe_count} ({severe_percentage:.2f}%)' for level, normal_count, normal_percentage, moderate_count, moderate_percentage, severe_count, severe_percentage, in zip(\n df['level'].values[0::3],\n df['count'].values[0::3],\n df['percentage'].values[0::3],\n df['count'].values[1::3],\n df['percentage'].values[1::3],\n df['count'].values[2::3],\n df['percentage'].values[2::3],\n )\n ])\n ax.set_xlabel('')\n ax.tick_params(axis='x', labelsize=17.5, pad=10)\n ax.tick_params(axis='y', labelsize=17.5, pad=10)\n ax.set_title(title, size=20, pad=15)\n ax.legend(loc='best', prop={'size': 18})\n plt.gca().invert_yaxis()\n\n plt.show()\n\n if path is None:\n plt.show()\n else:\n plt.savefig(path)\n plt.close(fig)\n\ndef visualize_counts(df, value, title, path=None):\n\n \"\"\"\n Visualize counts on training set\n\n Parameters\n ----------\n df: pandas.DataFrame\n Counts and percentages of conditions\n\n title: str\n Title of the plot\n\n path: str, pathlib.Path or None\n Path of the output file (if path is None, plot is displayed with selected backend)\n \"\"\"\n\n fig, ax = plt.subplots(figsize=(24, 16))\n\n ax.barh(\n y=np.arange(df.shape[0]),\n width=df['count'].values,\n align='center'\n )\n\n ax.set_yticks(np.arange(df.shape[0]))\n ax.set_yticklabels([\n f'{val} - Count: {count} ({percentage:.2f}%)' for val, count, percentage in zip(\n df[value].values,\n df['count'].values,\n df['percentage'].values,\n )\n ])\n ax.set_xlabel('')\n ax.tick_params(axis='x', labelsize=17.5, pad=10)\n ax.tick_params(axis='y', labelsize=17.5, pad=10)\n ax.set_title(title, size=20, pad=15)\n ax.legend(loc='best', prop={'size': 18})\n plt.gca().invert_yaxis()\n\n plt.show()\n\n if path is None:\n plt.show()\n else:\n plt.savefig(path)\n plt.close(fig)\n\ndef visualize_continuous_metadata_column_histogram(df, title, path=None):\n\n \"\"\"\n Visualize histogram of the given continuous column within modalities\n\n Parameters\n ----------\n df: pandas.DataFrame\n Dataframe with given continuous column and series description as index\n\n title: str\n Title of the plot\n\n path: str, pathlib.Path or None\n Path of the output file or None (if path is None, plot is displayed with selected backend)\n \"\"\"\n\n fig, ax = plt.subplots(figsize=(32, 8), dpi=100)\n _, bins, _ = ax.hist(df.loc['Axial T2'], bins=24, label='Axial T2', alpha=0.5)\n ax.hist(df.loc['Sagittal T1'], bins=bins, label='Sagittal T1', alpha=0.5)\n ax.hist(df.loc['Sagittal T2/STIR'], bins=bins, label='Sagittal T2/STIR', alpha=0.5)\n ax.tick_params(axis='x', labelsize=15)\n ax.tick_params(axis='y', labelsize=15)\n ax.set_xlabel('')\n ax.set_ylabel('')\n ax.legend(prop={'size': 18})\n ax.set_title(\n title + f'''\n Axial T2 Mean: {np.mean(df.loc['Axial T2']):.2f} Std: {np.std(df.loc['Axial T2']):.2f} Min: {np.min(df.loc['Axial T2']):.2f} Max: {np.max(df.loc['Axial T2']):.2f}\n Sagittal T1 Mean: {np.mean(df.loc['Sagittal T1']):.2f} Std: {np.std(df.loc['Sagittal T1']):.2f} Min: {np.min(df.loc['Sagittal T1']):.2f} Max: {np.max(df.loc['Sagittal T1']):.2f}\n Sagittal T2/STIR Mean: {np.mean(df.loc['Sagittal T2/STIR']):.2f} Std: {np.std(df.loc['Sagittal T2/STIR']):.2f} Min: {np.min(df.loc['Sagittal T2/STIR']):.2f} Max: {np.max(df.loc['Sagittal T2/STIR']):.2f}\n ''',\n size=15,\n pad=12.5,\n loc='center',\n wrap=True\n )\n\n if path is None:\n plt.show()\n else:\n plt.savefig(path, bbox_inches='tight')\n plt.close(fig)\n\ndef create_animation(\n img_stack: np.ndarray,\n study_id: str = \"\",\n series_id: str = \"\",\n imaging_plane: str = \"\",\n figsize: tuple[int, int] = (7, 7),\n interval: int = 83, # 1000//12=83[.333]\n cmap: str = 'bone',\n do_save: bool = False,\n save_dir: str = \"/kaggle/working/animations\"\n) -> animation.FuncAnimation:\n \"\"\"Create an animation from a stack of DICOM images.\n\n Args:\n img_stack (np.ndarray):\n A 3D numpy array representing a stack of DICOM images (shape: [num_frames, height, width, channels]).\n plot_title (str, optional)\n figsize (tuple[int, int], optional):\n A tuple representing the size of the figure (width, height) in inches.\n interval (int, optional):\n Delay between frames in milliseconds. Default is approximately 24 frames per second.\n cmap (str, optional):\n Colormap to use for displaying the images.\n\n Returns:\n animation.FuncAnimation:\n An animation object that can be displayed in a Jupyter notebook or saved to a file.\n \"\"\"\n\n animation_title = f\"STUDY_{study_id}__SERIES_{series_id}__{imaging_plane}_PLANE_VIEW\"\n\n # Create a new figure with the specified size\n fig = plt.figure(figsize=figsize)\n\n # Turn off the axis\n plt.axis('off')\n\n # Display the first image from the stack\n img = plt.imshow(img_stack[0], cmap=cmap)\n plt.title(animation_title.replace(\"__\", \" – \").replace(\"_\", \" \").title(), fontweight=\"bold\")\n def animate_func(i: int) -> list[Any]:\n \"\"\"Update the image for the animation.\n\n Args:\n i (int): The frame index.\n\n Returns:\n list: A list containing the updated image.\n \"\"\"\n img.set_array(img_stack[i])\n return [img]\n plt.close()\n\n # Create and return the animation object\n anim = animation.FuncAnimation(fig, animate_func, frames=len(img_stack), interval=interval)\n\n if do_save:\n if not os.path.isdir(save_dir):\n os.makedirs(save_dir, exist_ok=True)\n anim.save(os.path.join(save_dir, animation_title+\".gif\"), fps=12, writer='imagemagick')\n\n return anim\n\ndef convert_to_8bit(image: np.ndarray) -> np.ndarray:\n \"\"\"Convert a 16-bit image to 8-bit.\n\n Args:\n image:\n A 16-bit image array.\n\n Returns:\n An 8-bit image array.\n \"\"\"\n lower, upper = np.percentile(image, (1, 99))\n image = np.clip(image, lower, upper)\n image = image - np.min(image)\n image = image / np.max(image)\n return (image * 255).astype(\"uint8\")\n\n\ndef load_dicom_stack(dicom_folder: str, plane: str, reverse_sort: bool = False) -> dict[str, np.ndarray]:\n \"\"\"Load a stack of DICOM images and convert them to 8-bit.\n\n Args:\n dicom_folder (str):\n Path to the folder containing DICOM files.\n plane (str):\n The plane of the images ('sagittal', 'coronal', 'axial').\n reverse_sort (bool, optional):\n Whether to reverse sort the images.\n\n Returns:\n A dictionary containing the image array, positions, and pixel spacing.\n \"\"\"\n dicom_files = glob(os.path.join(dicom_folder, \"*.dcm\"))\n dicoms = [pydicom.dcmread(f) for f in dicom_files]\n plane_index = {\"sagittal\": 0, \"coronal\": 1, \"axial\": 2}[plane.lower()]\n positions = np.asarray([float(d.ImagePositionPatient[plane_index]) for d in dicoms])\n idx = np.argsort(-positions if reverse_sort else positions)\n ipp = np.asarray([d.ImagePositionPatient for d in dicoms]).astype(\"float\")[idx]\n array = np.stack([d.pixel_array.astype(\"float32\") for d in dicoms])\n array = array[idx]\n return {\n \"array\": convert_to_8bit(array),\n \"positions\": ipp,\n \"pixel_spacing\": np.asarray(dicoms[0].PixelSpacing).astype(\"float\")\n }\n\n\ndef plot_dicom_images(study: pd.DataFrame, image_dir: str) -> None:\n \"\"\"Plot DICOM images for a given study.\n\n Args:\n study (pd.DataFrame):\n DataFrame containing study information.\n image_dir (str):\n Directory containing DICOM images.\n \"\"\"\n for row in study.itertuples():\n if row.series_description == \"Sagittal T2/STIR\":\n sag_t2 = load_dicom_stack(os.path.join(image_dir, str(row.study_id), str(row.series_id)), plane=\"sagittal\")\n elif row.series_description == \"Sagittal T1\":\n sag_t1 = load_dicom_stack(os.path.join(image_dir, str(row.study_id), str(row.series_id)), plane=\"sagittal\")\n elif row.series_description == \"Axial T2\":\n ax_t2 = load_dicom_stack(os.path.join(image_dir, str(row.study_id), str(row.series_id)), plane=\"axial\", reverse_sort=True)\n\n plt.figure(figsize=(14, 5))\n plt.subplot(1, 3, 1)\n plt.imshow(sag_t2[\"array\"][len(sag_t2[\"array\"]) // 2], cmap=\"gray\")\n plt.subplot(1, 3, 2)\n plt.imshow(sag_t1[\"array\"][len(sag_t1[\"array\"]) // 2], cmap=\"gray\")\n plt.subplot(1, 3, 3)\n plt.imshow(ax_t2[\"array\"][len(ax_t2[\"array\"]) // 2], cmap=\"gray\")\n plt.show()\n\n\ndef map_axial_to_sagittal(sag_t2: dict[str, np.ndarray], ax_t2: dict[str, np.ndarray], plot_all_slices: bool = True) -> None:\n \"\"\"Map axial slices to sagittal slices and plot the results.\n\n Args:\n sag_t2 (dict[str, np.ndarray]):\n Dictionary containing sagittal T2 image data.\n ax_t2 (dict[str, np.ndarray]):\n Dictionary containing axial T2 image data.\n plot_all_slices (bool, optional):\n Whether to show each individual side-by-side plot.\n \"\"\"\n top_left_hand_corner_sag_t2 = sag_t2[\"positions\"][len(sag_t2[\"array\"]) // 2]\n sag_y_axis_to_pixel_space = [top_left_hand_corner_sag_t2[2]]\n while len(sag_y_axis_to_pixel_space) < sag_t2[\"array\"].shape[1]:\n sag_y_axis_to_pixel_space.append(sag_y_axis_to_pixel_space[-1] - sag_t2[\"pixel_spacing\"][1])\n\n sag_y_coord_to_axial_slice = {}\n for ax_t2_slice, ax_t2_pos in zip(ax_t2[\"array\"], ax_t2[\"positions\"]):\n diffs = np.abs(np.asarray(sag_y_axis_to_pixel_space) - ax_t2_pos[2])\n sag_y_coord = np.argmin(diffs)\n sag_y_coord_to_axial_slice[sag_y_coord] = ax_t2_slice\n\n sag_midline_slice = sag_t2[\"array\"][len(sag_t2[\"array\"]) // 2]\n plt.imshow(sag_midline_slice, cmap=\"gray\")\n for k in [*sag_y_coord_to_axial_slice]:\n plt.axhline(y=k, color=\"red\", linestyle=\"--\")\n plt.show()\n\n if plot_all_slices:\n for k, v in sag_y_coord_to_axial_slice.items():\n plt.figure(figsize=(10, 4))\n plt.subplot(1, 2, 1)\n plt.imshow(sag_midline_slice, cmap=\"gray\")\n plt.axhline(y=k, color=\"red\", linestyle=\"--\")\n plt.subplot(1, 2, 2)\n plt.imshow(v, cmap=\"gray\")\n plt.show()","metadata":{"id":"bv1zuOCoTl-W","execution":{"iopub.status.busy":"2024-09-03T09:52:06.167558Z","iopub.execute_input":"2024-09-03T09:52:06.168018Z","iopub.status.idle":"2024-09-03T09:52:06.237771Z","shell.execute_reply.started":"2024-09-03T09:52:06.167968Z","shell.execute_reply":"2024-09-03T09:52:06.236477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Load and Inspection","metadata":{"id":"H1gUSRUCTl-e"}},{"cell_type":"code","source":"train_images = glob(str(data_dir / 'train_images' / '*' / '*' / '*.dcm'))\ntrain_images_dir = glob(str(data_dir / 'train_images'))\n\ndf_train = pd.read_csv(data_dir / 'train.csv')\ndf_train_series_descriptions = pd.read_csv(data_dir / 'train_series_descriptions.csv')\ndf_train_label_coordinates = pd.read_csv(data_dir / 'train_label_coordinates.csv')","metadata":{"id":"htllbqCuTl-g","execution":{"iopub.status.busy":"2024-09-03T09:52:06.240492Z","iopub.execute_input":"2024-09-03T09:52:06.240952Z","iopub.status.idle":"2024-09-03T09:52:12.315064Z","shell.execute_reply.started":"2024-09-03T09:52:06.240905Z","shell.execute_reply":"2024-09-03T09:52:12.313795Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Training Images Count {len(train_images)}')\nprint(f'Training Set Shape: {df_train.shape}')\nprint(f'Train Series Descriptions Shape: {df_train_series_descriptions.shape}')\nprint(f'Train Label Coordinates Shape: {df_train_label_coordinates.shape}')","metadata":{"id":"lIIl2EisTl-h","execution":{"iopub.status.busy":"2024-09-03T09:52:12.316498Z","iopub.execute_input":"2024-09-03T09:52:12.316897Z","iopub.status.idle":"2024-09-03T09:52:12.324343Z","shell.execute_reply.started":"2024-09-03T09:52:12.316855Z","shell.execute_reply":"2024-09-03T09:52:12.323193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Trian data","metadata":{"execution":{"iopub.status.busy":"2024-08-29T18:58:43.192154Z","iopub.execute_input":"2024-08-29T18:58:43.192636Z","iopub.status.idle":"2024-08-29T18:58:43.198796Z","shell.execute_reply.started":"2024-08-29T18:58:43.192589Z","shell.execute_reply":"2024-08-29T18:58:43.197226Z"},"id":"FgHfXImxTl-j"}},{"cell_type":"code","source":"display(df_train)\ndisplay(df_train.info())","metadata":{"_kg_hide-input":true,"id":"NQhWtE78Tl-j","execution":{"iopub.status.busy":"2024-09-03T09:52:12.325868Z","iopub.execute_input":"2024-09-03T09:52:12.326369Z","iopub.status.idle":"2024-09-03T09:52:12.384268Z","shell.execute_reply.started":"2024-09-03T09:52:12.326326Z","shell.execute_reply":"2024-09-03T09:52:12.382839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train Series Descriptions\n\n","metadata":{"id":"_6UXQ0XbTl-k"}},{"cell_type":"code","source":"display(df_train_series_descriptions)\ndisplay(df_train_series_descriptions.info())","metadata":{"_kg_hide-input":true,"id":"zZh8HfXPTl-l","execution":{"iopub.status.busy":"2024-09-03T09:52:12.388443Z","iopub.execute_input":"2024-09-03T09:52:12.389003Z","iopub.status.idle":"2024-09-03T09:52:12.420379Z","shell.execute_reply.started":"2024-09-03T09:52:12.388924Z","shell.execute_reply":"2024-09-03T09:52:12.418952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train Label Coordinates","metadata":{"id":"MOE3_VGdTl-l"}},{"cell_type":"code","source":"display(df_train_label_coordinates)\ndisplay(df_train_label_coordinates.info())\ndisplay(df_train_label_coordinates.describe())","metadata":{"_kg_hide-input":true,"id":"_gWkxNlGTl-m","execution":{"iopub.status.busy":"2024-09-03T09:52:12.422105Z","iopub.execute_input":"2024-09-03T09:52:12.422560Z","iopub.status.idle":"2024-09-03T09:52:12.493794Z","shell.execute_reply.started":"2024-09-03T09:52:12.422515Z","shell.execute_reply":"2024-09-03T09:52:12.492477Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Merge data","metadata":{"id":"LcglXVdbTl-m"}},{"cell_type":"code","source":"merged_data = pd.merge(df_train, df_train_label_coordinates, on='study_id', how='left')\n\nmerged_data = pd.merge(merged_data, df_train_series_descriptions, on=['study_id','series_id'], how='left')\n\ndisplay(merged_data.head(10))\nprint(merged_data.info())","metadata":{"id":"ls1kWWzvTl-n","execution":{"iopub.status.busy":"2024-09-03T09:52:12.495471Z","iopub.execute_input":"2024-09-03T09:52:12.495885Z","iopub.status.idle":"2024-09-03T09:52:12.780465Z","shell.execute_reply.started":"2024-09-03T09:52:12.495844Z","shell.execute_reply":"2024-09-03T09:52:12.779231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"merged_data.isna().sum()","metadata":{"id":"J2rLrIqETl-n","execution":{"iopub.status.busy":"2024-09-03T09:52:12.782087Z","iopub.execute_input":"2024-09-03T09:52:12.782492Z","iopub.status.idle":"2024-09-03T09:52:12.939191Z","shell.execute_reply.started":"2024-09-03T09:52:12.782438Z","shell.execute_reply":"2024-09-03T09:52:12.937886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conditions","metadata":{"id":"IDSivl-iTl-o"}},{"cell_type":"markdown","source":"### Spinal Canal Stenosis","metadata":{"id":"I1i5-gUkTl-o"}},{"cell_type":"code","source":"spinal_canal_stenosis_columns = [column for column in df_train.columns if column.startswith('spinal_canal_stenosis')]\ndf_train_spinal_canal_stenosis = []\n\nfor column in spinal_canal_stenosis_columns:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'spinal_canal_stenosis'})\n df_train_spinal_canal_stenosis.append(df)\n\ndf_train_spinal_canal_stenosis = pd.concat(df_train_spinal_canal_stenosis, axis=0).reset_index(drop=True)\n\ndf_train_spinal_canal_stenosis_counts = df_train_spinal_canal_stenosis.groupby('level').value_counts().reset_index()\ndf_train_spinal_canal_stenosis_counts['severity'] = df_train_spinal_canal_stenosis_counts['spinal_canal_stenosis'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_spinal_canal_stenosis_counts = df_train_spinal_canal_stenosis_counts.sort_values(by=['level', 'severity'], ascending=True)\ndf_train_spinal_canal_stenosis_counts['percentage'] = df_train_spinal_canal_stenosis_counts['count'] / df_train_spinal_canal_stenosis_counts.groupby('level')['count'].transform('sum') * 100","metadata":{"id":"XWr36lWnTl-p","execution":{"iopub.status.busy":"2024-09-03T09:52:12.941380Z","iopub.execute_input":"2024-09-03T09:52:12.941854Z","iopub.status.idle":"2024-09-03T09:52:12.980667Z","shell.execute_reply.started":"2024-09-03T09:52:12.941810Z","shell.execute_reply":"2024-09-03T09:52:12.979214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_condition_counts(\n df=df_train_spinal_canal_stenosis_counts,\n title='Spinal Canal Stenosis Counts On Levels'\n)","metadata":{"_kg_hide-input":true,"id":"aSOVWyxRTl-p","execution":{"iopub.status.busy":"2024-09-03T09:52:12.982444Z","iopub.execute_input":"2024-09-03T09:52:12.982895Z","iopub.status.idle":"2024-09-03T09:52:13.582404Z","shell.execute_reply.started":"2024-09-03T09:52:12.982845Z","shell.execute_reply":"2024-09-03T09:52:13.581168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Neural Foraminal Narrowing","metadata":{"id":"ZAig1QGxTl-p"}},{"cell_type":"code","source":"neural_foraminal_narrowing_columns = [column for column in df_train.columns if 'neural_foraminal_narrowing' in column]\ndf_train_neural_foraminal_narrowing = []\n\nfor column in neural_foraminal_narrowing_columns:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'neural_foraminal_narrowing'})\n df_train_neural_foraminal_narrowing.append(df)\n\ndf_train_neural_foraminal_narrowing = pd.concat(df_train_neural_foraminal_narrowing, axis=0).reset_index(drop=True)\n\ndf_train_neural_foraminal_narrowing_counts = df_train_neural_foraminal_narrowing.groupby('level').value_counts().reset_index()\ndf_train_neural_foraminal_narrowing_counts['severity'] = df_train_neural_foraminal_narrowing_counts['neural_foraminal_narrowing'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_neural_foraminal_narrowing_counts = df_train_neural_foraminal_narrowing_counts.sort_values(by=['level', 'severity'], ascending=True)\ndf_train_neural_foraminal_narrowing_counts['percentage'] = df_train_neural_foraminal_narrowing_counts['count'] / df_train_neural_foraminal_narrowing_counts.groupby('level')['count'].transform('sum') * 100\n\nleft_neural_foraminal_narrowing_columns = [column for column in df_train.columns if column.startswith('left_neural_foraminal_narrowing')]\ndf_train_left_neural_foraminal_narrowing = []\n\nfor column in left_neural_foraminal_narrowing_columns:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'left_neural_foraminal_narrowing'})\n df_train_left_neural_foraminal_narrowing.append(df)\n\ndf_train_left_neural_foraminal_narrowing = pd.concat(df_train_left_neural_foraminal_narrowing, axis=0).reset_index(drop=True)\n\ndf_train_left_neural_foraminal_narrowing_counts = df_train_left_neural_foraminal_narrowing.groupby('level').value_counts().reset_index()\ndf_train_left_neural_foraminal_narrowing_counts['severity'] = df_train_left_neural_foraminal_narrowing_counts['left_neural_foraminal_narrowing'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_left_neural_foraminal_narrowing_counts = df_train_left_neural_foraminal_narrowing_counts.sort_values(by=['level', 'severity'], ascending=True)\ndf_train_left_neural_foraminal_narrowing_counts['percentage'] = df_train_left_neural_foraminal_narrowing_counts['count'] / df_train_left_neural_foraminal_narrowing_counts.groupby('level')['count'].transform('sum') * 100\n\nright_neural_foraminal_narrowing_columns = [column for column in df_train.columns if column.startswith('right_neural_foraminal_narrowing')]\ndf_train_right_neural_foraminal_narrowing = []\n\nfor column in right_neural_foraminal_narrowing_columns:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'right_neural_foraminal_narrowing'})\n df_train_right_neural_foraminal_narrowing.append(df)\n\ndf_train_right_neural_foraminal_narrowing = pd.concat(df_train_right_neural_foraminal_narrowing, axis=0).reset_index(drop=True)\n\ndf_train_right_neural_foraminal_narrowing_counts = df_train_right_neural_foraminal_narrowing.groupby('level').value_counts().reset_index()\ndf_train_right_neural_foraminal_narrowing_counts['severity'] = df_train_right_neural_foraminal_narrowing_counts['right_neural_foraminal_narrowing'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_right_neural_foraminal_narrowing_counts = df_train_right_neural_foraminal_narrowing_counts.sort_values(by=['level', 'severity'], ascending=True)\ndf_train_right_neural_foraminal_narrowing_counts['percentage'] = df_train_right_neural_foraminal_narrowing_counts['count'] / df_train_right_neural_foraminal_narrowing_counts.groupby('level')['count'].transform('sum') * 100","metadata":{"id":"mbve-ugqTl-q","execution":{"iopub.status.busy":"2024-09-03T09:52:13.583874Z","iopub.execute_input":"2024-09-03T09:52:13.584260Z","iopub.status.idle":"2024-09-03T09:52:13.698375Z","shell.execute_reply.started":"2024-09-03T09:52:13.584219Z","shell.execute_reply":"2024-09-03T09:52:13.697363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_condition_counts(\n df=df_train_neural_foraminal_narrowing_counts,\n title='Neural Foraminal Narrowing Counts On Levels'\n)\n\nvisualize_condition_counts(\n df=df_train_left_neural_foraminal_narrowing_counts,\n title='Left Neural Foraminal Narrowing Counts On Levels'\n)\n\nvisualize_condition_counts(\n df=df_train_right_neural_foraminal_narrowing_counts,\n title='Right Neural Foraminal Narrowing Counts On Levels'\n)","metadata":{"_kg_hide-input":true,"id":"XiUxhTs0Tl-r","execution":{"iopub.status.busy":"2024-09-03T09:52:13.700007Z","iopub.execute_input":"2024-09-03T09:52:13.700529Z","iopub.status.idle":"2024-09-03T09:52:15.769032Z","shell.execute_reply.started":"2024-09-03T09:52:13.700473Z","shell.execute_reply":"2024-09-03T09:52:15.767806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"neural_foraminal_narrowing_columns = [column for column in df_train.columns if 'neural_foraminal_narrowing' in column]\ndf_train_left_right_neural_foraminal_narrowing = []\n\nfor column in neural_foraminal_narrowing_columns[:5]:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'left_neural_foraminal_narrowing'})\n df_train_left_right_neural_foraminal_narrowing.append(df)\n\nfor column in neural_foraminal_narrowing_columns[5:]:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'right_neural_foraminal_narrowing'})\n df_train_left_right_neural_foraminal_narrowing.append(df)\n\ndf_train_left_right_neural_foraminal_narrowing = pd.concat((\n pd.concat(df_train_left_right_neural_foraminal_narrowing[:5], axis=0).reset_index(drop=True).iloc[:, :1],\n pd.concat(df_train_left_right_neural_foraminal_narrowing[5:], axis=0).reset_index(drop=True)\n), axis=1, ignore_index=False)\n\ndf_train_left_right_neural_foraminal_narrowing['left_severity'] = df_train_left_right_neural_foraminal_narrowing['left_neural_foraminal_narrowing'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_left_right_neural_foraminal_narrowing['right_severity'] = df_train_left_right_neural_foraminal_narrowing['right_neural_foraminal_narrowing'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_left_right_neural_foraminal_narrowing['level_'] = df_train_left_right_neural_foraminal_narrowing['level'].map({\n 'l1_l2': 0,\n 'l2_l3': 1,\n 'l3_l4': 2,\n 'l4_l5': 3,\n 'l5_s1': 4\n})","metadata":{"id":"wRrb7FtTTl-r","execution":{"iopub.status.busy":"2024-09-03T09:52:15.770748Z","iopub.execute_input":"2024-09-03T09:52:15.771206Z","iopub.status.idle":"2024-09-03T09:52:15.818910Z","shell.execute_reply.started":"2024-09-03T09:52:15.771153Z","shell.execute_reply":"2024-09-03T09:52:15.817650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_neural_foraming_narrowing_correlations = df_train_left_right_neural_foraminal_narrowing[['level_', 'left_severity', 'right_severity']].corr()\ndisplay(df_neural_foraming_narrowing_correlations)","metadata":{"id":"Q0N9JjNtTl-r","execution":{"iopub.status.busy":"2024-09-03T09:52:15.825481Z","iopub.execute_input":"2024-09-03T09:52:15.826360Z","iopub.status.idle":"2024-09-03T09:52:15.843627Z","shell.execute_reply.started":"2024-09-03T09:52:15.826312Z","shell.execute_reply":"2024-09-03T09:52:15.842169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Subarticular Stenosis","metadata":{"id":"6wf479BOTl-s"}},{"cell_type":"code","source":"subarticular_stenosis_columns = [column for column in df_train.columns if 'subarticular_stenosis' in column]\ndf_train_subarticular_stenosis = []\n\nfor column in subarticular_stenosis_columns:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'subarticular_stenosis'})\n df_train_subarticular_stenosis.append(df)\n\ndf_train_subarticular_stenosis = pd.concat(df_train_subarticular_stenosis, axis=0).reset_index(drop=True)\n\ndf_train_subarticular_stenosis_counts = df_train_subarticular_stenosis.groupby('level').value_counts().reset_index()\ndf_train_subarticular_stenosis_counts['severity'] = df_train_subarticular_stenosis_counts['subarticular_stenosis'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_subarticular_stenosis_counts = df_train_subarticular_stenosis_counts.sort_values(by=['level', 'severity'], ascending=True)\ndf_train_subarticular_stenosis_counts['percentage'] = df_train_subarticular_stenosis_counts['count'] / df_train_subarticular_stenosis_counts.groupby('level')['count'].transform('sum') * 100\n\nleft_subarticular_stenosis_columns = [column for column in df_train.columns if column.startswith('left_subarticular_stenosis')]\ndf_train_left_subarticular_stenosis = []\n\nfor column in left_subarticular_stenosis_columns:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'left_subarticular_stenosis'})\n df_train_left_subarticular_stenosis.append(df)\n\ndf_train_left_subarticular_stenosis = pd.concat(df_train_left_subarticular_stenosis, axis=0).reset_index(drop=True)\n\ndf_train_left_subarticular_stenosis_counts = df_train_left_subarticular_stenosis.groupby('level').value_counts().reset_index()\ndf_train_left_subarticular_stenosis_counts['severity'] = df_train_left_subarticular_stenosis_counts['left_subarticular_stenosis'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_left_subarticular_stenosis_counts = df_train_left_subarticular_stenosis_counts.sort_values(by=['level', 'severity'], ascending=True)\ndf_train_left_subarticular_stenosis_counts['percentage'] = df_train_left_subarticular_stenosis_counts['count'] / df_train_left_subarticular_stenosis_counts.groupby('level')['count'].transform('sum') * 100\n\nright_subarticular_stenosis_columns = [column for column in df_train.columns if column.startswith('right_subarticular_stenosis')]\ndf_train_right_subarticular_stenosis = []\n\nfor column in right_subarticular_stenosis_columns:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'right_subarticular_stenosis'})\n df_train_right_subarticular_stenosis.append(df)\n\ndf_train_right_subarticular_stenosis = pd.concat(df_train_right_subarticular_stenosis, axis=0).reset_index(drop=True)\n\ndf_train_right_subarticular_stenosis_counts = df_train_right_subarticular_stenosis.groupby('level').value_counts().reset_index()\ndf_train_right_subarticular_stenosis_counts['severity'] = df_train_right_subarticular_stenosis_counts['right_subarticular_stenosis'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_right_subarticular_stenosis_counts = df_train_right_subarticular_stenosis_counts.sort_values(by=['level', 'severity'], ascending=True)\ndf_train_right_subarticular_stenosis_counts['percentage'] = df_train_right_subarticular_stenosis_counts['count'] / df_train_right_subarticular_stenosis_counts.groupby('level')['count'].transform('sum') * 100","metadata":{"id":"5EaIV7m0Tl-s","execution":{"iopub.status.busy":"2024-09-03T09:52:15.845914Z","iopub.execute_input":"2024-09-03T09:52:15.846324Z","iopub.status.idle":"2024-09-03T09:52:15.968687Z","shell.execute_reply.started":"2024-09-03T09:52:15.846280Z","shell.execute_reply":"2024-09-03T09:52:15.967478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_condition_counts(\n df=df_train_subarticular_stenosis_counts,\n title='Subarticular Stenosis Counts On Levels'\n)\n\nvisualize_condition_counts(\n df=df_train_left_subarticular_stenosis_counts,\n title='Left Subarticular Stenosis Counts On Levels'\n)\n\nvisualize_condition_counts(\n df=df_train_right_subarticular_stenosis_counts,\n title='Right Subarticular Stenosis Counts On Levels'\n)","metadata":{"id":"q7DVaHKbTl-s","execution":{"iopub.status.busy":"2024-09-03T09:52:15.970177Z","iopub.execute_input":"2024-09-03T09:52:15.970699Z","iopub.status.idle":"2024-09-03T09:52:17.754847Z","shell.execute_reply.started":"2024-09-03T09:52:15.970644Z","shell.execute_reply":"2024-09-03T09:52:17.753498Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"subarticular_stenosis_columns = [column for column in df_train.columns if 'subarticular_stenosis' in column]\ndf_train_left_right_subarticular_stenosis = []\n\nfor column in subarticular_stenosis_columns[:5]:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'left_subarticular_stenosis'})\n df_train_left_right_subarticular_stenosis.append(df)\n\nfor column in subarticular_stenosis_columns[5:]:\n df = df_train[[column]].copy(deep=True)\n df['level'] = '_'.join(column.split('_')[-2:])\n df = df.rename(columns={column: 'right_subarticular_stenosis'})\n df_train_left_right_subarticular_stenosis.append(df)\n\ndf_train_left_right_subarticular_stenosis = pd.concat((\n pd.concat(df_train_left_right_subarticular_stenosis[:5], axis=0).reset_index(drop=True).iloc[:, :1],\n pd.concat(df_train_left_right_subarticular_stenosis[5:], axis=0).reset_index(drop=True)\n), axis=1, ignore_index=False)\n\ndf_train_left_right_subarticular_stenosis['left_severity'] = df_train_left_right_subarticular_stenosis['left_subarticular_stenosis'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_left_right_subarticular_stenosis['right_severity'] = df_train_left_right_subarticular_stenosis['right_subarticular_stenosis'].map({\n 'Normal/Mild': 0,\n 'Moderate': 1,\n 'Severe': 2\n})\ndf_train_left_right_subarticular_stenosis['level_'] = df_train_left_right_subarticular_stenosis['level'].map({\n 'l1_l2': 0,\n 'l2_l3': 1,\n 'l3_l4': 2,\n 'l4_l5': 3,\n 'l5_s1': 4\n})","metadata":{"id":"9WxD6bfcTl-t","execution":{"iopub.status.busy":"2024-09-03T09:52:17.756575Z","iopub.execute_input":"2024-09-03T09:52:17.756995Z","iopub.status.idle":"2024-09-03T09:52:17.799052Z","shell.execute_reply.started":"2024-09-03T09:52:17.756953Z","shell.execute_reply":"2024-09-03T09:52:17.797620Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_subarticular_stenosis_correlations = df_train_left_right_subarticular_stenosis[['level_', 'left_severity', 'right_severity']].corr()\ndisplay(df_subarticular_stenosis_correlations)","metadata":{"_kg_hide-input":true,"id":"Wc47HilyTl-u","execution":{"iopub.status.busy":"2024-09-03T09:52:17.800622Z","iopub.execute_input":"2024-09-03T09:52:17.801050Z","iopub.status.idle":"2024-09-03T09:52:17.817326Z","shell.execute_reply.started":"2024-09-03T09:52:17.801000Z","shell.execute_reply":"2024-09-03T09:52:17.815902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modalities and Planes","metadata":{"id":"8bOQB2QSTl-u"}},{"cell_type":"code","source":"df_train_series_counts = df_train_series_descriptions.groupby('study_id')['series_id'].count().value_counts().reset_index().rename(columns={'series_id': 'series_count'})\ndf_train_series_counts['percentage'] = df_train_series_counts['count'] / df_train_series_counts['count'].sum(axis=0) * 100","metadata":{"id":"JOdP7laHTl-u","execution":{"iopub.status.busy":"2024-09-03T09:52:17.818874Z","iopub.execute_input":"2024-09-03T09:52:17.819358Z","iopub.status.idle":"2024-09-03T09:52:17.832779Z","shell.execute_reply.started":"2024-09-03T09:52:17.819313Z","shell.execute_reply":"2024-09-03T09:52:17.831549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_counts(df=df_train_series_counts, value='series_count', title='Series Counts in Studies')","metadata":{"_kg_hide-input":true,"id":"wk4ejlgeTl-v","execution":{"iopub.status.busy":"2024-09-03T09:52:17.834341Z","iopub.execute_input":"2024-09-03T09:52:17.835178Z","iopub.status.idle":"2024-09-03T09:52:18.303248Z","shell.execute_reply.started":"2024-09-03T09:52:17.835117Z","shell.execute_reply":"2024-09-03T09:52:18.301635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_train_series_descriptions['plane'] = df_train_series_descriptions['series_description'].apply(lambda x: str(x).split()[0].strip())\ndf_train_series_descriptions['modality'] = df_train_series_descriptions['series_description'].apply(lambda x: str(x).split()[-1].strip())\n\ndf_train_series_description_counts = df_train_series_descriptions['series_description'].value_counts().reset_index()\ndf_train_series_description_counts['percentage'] = df_train_series_description_counts['count'] / df_train_series_description_counts['count'].sum(axis=0) * 100\n\ndf_train_series_plane_counts = df_train_series_descriptions['plane'].value_counts().reset_index()\ndf_train_series_plane_counts['percentage'] = df_train_series_plane_counts['count'] / df_train_series_plane_counts['count'].sum(axis=0) * 100\n\ndf_train_series_modality_counts = df_train_series_descriptions['modality'].value_counts().reset_index()\ndf_train_series_modality_counts['percentage'] = df_train_series_modality_counts['count'] / df_train_series_modality_counts['count'].sum(axis=0) * 100","metadata":{"id":"FUEmCr3dTl-v","execution":{"iopub.status.busy":"2024-09-03T09:52:18.304869Z","iopub.execute_input":"2024-09-03T09:52:18.305278Z","iopub.status.idle":"2024-09-03T09:52:18.340161Z","shell.execute_reply.started":"2024-09-03T09:52:18.305235Z","shell.execute_reply":"2024-09-03T09:52:18.338764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_counts(df=df_train_series_modality_counts, value='modality', title='Modality Counts in Series')","metadata":{"id":"EQ-FveTWTl-v","execution":{"iopub.status.busy":"2024-09-03T09:52:18.341961Z","iopub.execute_input":"2024-09-03T09:52:18.342509Z","iopub.status.idle":"2024-09-03T09:52:18.763772Z","shell.execute_reply.started":"2024-09-03T09:52:18.342453Z","shell.execute_reply":"2024-09-03T09:52:18.762509Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_counts(df=df_train_series_plane_counts, value='plane', title='Plane Counts in Series')","metadata":{"id":"v5peR46rTl-v","execution":{"iopub.status.busy":"2024-09-03T09:52:18.765231Z","iopub.execute_input":"2024-09-03T09:52:18.765626Z","iopub.status.idle":"2024-09-03T09:52:19.190643Z","shell.execute_reply.started":"2024-09-03T09:52:18.765583Z","shell.execute_reply":"2024-09-03T09:52:19.189370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_counts(df=df_train_series_description_counts, value='series_description', title='Series Description Counts in Series')","metadata":{"id":"i9MxMfewTl-4","execution":{"iopub.status.busy":"2024-09-03T09:52:19.192575Z","iopub.execute_input":"2024-09-03T09:52:19.193002Z","iopub.status.idle":"2024-09-03T09:52:19.623924Z","shell.execute_reply.started":"2024-09-03T09:52:19.192950Z","shell.execute_reply":"2024-09-03T09:52:19.622412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## DICOM Metadata","metadata":{"id":"d44fUR5tTl-4"}},{"cell_type":"code","source":"metadata = []\ndicom_attributes = [\n 'StudyInstanceUID', 'SeriesInstanceUID', 'PatientID',\n 'ContentDate', 'ContentTime', 'SeriesDescription',\n 'SliceThickness', 'SpacingBetweenSlices', 'PatientPosition',\n 'InstanceNumber', 'ImagePositionPatient', 'ImageOrientationPatient',\n 'SliceLocation', 'SamplesPerPixel', 'PhotometricInterpretation',\n 'Rows', 'Columns', 'PixelSpacing', 'BitsAllocated', 'BitsStored',\n 'HighBit', 'PixelRepresentation', 'WindowCenter', 'WindowWidth',\n 'RescaleIntercept', 'RescaleSlope'\n]\n\ntrain_images_root_directory = data_dir / 'train_images'\nstudy_ids = os.listdir(str(train_images_root_directory))\n\nfor study_id in tqdm(study_ids):\n\n study_directory = train_images_root_directory / study_id\n series_ids = os.listdir(str(study_directory))\n\n for series_id in series_ids:\n\n series_directory = study_directory / series_id\n dicom_file_names = os.listdir(series_directory)\n\n for dicom_file_name in dicom_file_names:\n\n dicom_path = series_directory / dicom_file_name\n dicom = pydicom.dcmread(dicom_path)\n\n dicom_dict = {}\n dicom_dict['slice_id'] = int(dicom_file_name.split('.')[0])\n for tag in dicom_attributes:\n try:\n dicom_tag_value = dicom[tag].value\n except KeyError:\n dicom_tag_value = np.nan\n dicom_dict[tag] = dicom_tag_value\n\n metadata.append(dicom_dict)\n\n\ndf_metadata = pd.DataFrame(metadata)\ndf_metadata = df_metadata.sort_values(by=['StudyInstanceUID', 'SeriesInstanceUID', 'slice_id'], ascending=True).reset_index(drop=True)\ndf_metadata['StudyInstanceUID'] = df_metadata['StudyInstanceUID'].astype(int)\ndf_metadata['SeriesInstanceUID'] = df_metadata['SeriesInstanceUID'].apply(lambda x: int(str(x).split('.')[-1])).astype(int)\ndf_metadata['PatientID'] = df_metadata['PatientID'].astype(int)\ndf_metadata['ContentDate'] = pd.to_datetime(df_metadata['ContentDate'] + ' ' + df_metadata['ContentTime'].apply(lambda x: f'{x[0:2]}:{x[2:4]}'))\ndf_metadata = df_metadata.drop(columns=['PatientID', 'ContentTime'])\n\ndf_metadata = df_metadata.merge(\n df_train_series_descriptions.rename(columns={\n 'study_id': 'StudyInstanceUID',\n 'series_id': 'SeriesInstanceUID'\n }),\n on=['StudyInstanceUID', 'SeriesInstanceUID'],\n how='left'\n)","metadata":{"id":"_Cv5uvKpTl-5","execution":{"iopub.status.busy":"2024-09-03T09:52:19.626222Z","iopub.execute_input":"2024-09-03T09:52:19.626718Z","iopub.status.idle":"2024-09-03T10:15:30.341881Z","shell.execute_reply.started":"2024-09-03T09:52:19.626660Z","shell.execute_reply":"2024-09-03T10:15:30.340480Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_metadata['PixelSpacingX'] = df_metadata['PixelSpacing'].apply(lambda x: float(x[0]))\ndf_metadata['PixelSpacingY'] = df_metadata['PixelSpacing'].apply(lambda x: float(x[1]))\n\ndf_slice_counts = df_metadata.groupby(['StudyInstanceUID', 'series_description'])['series_description'].count().reset_index(level=[0], drop=True).sort_index()\n\nvisualize_continuous_metadata_column_histogram(\n df=df_slice_counts,\n title='Series Slice Counts Distribution'\n)\n\ncontinuous_columns = [\n 'SliceThickness', 'SpacingBetweenSlices',\n 'SamplesPerPixel', 'Rows', 'Columns', 'PixelSpacingX', 'PixelSpacingY',\n 'BitsAllocated', 'BitsStored', 'HighBit',\n 'PixelRepresentation',\n 'WindowCenter', 'WindowWidth', 'RescaleIntercept', 'RescaleSlope'\n]\ndf_metadata_continuous_columns = df_metadata.groupby(['StudyInstanceUID', 'series_description'])[continuous_columns].first().reset_index(level=[0], drop=True).sort_index()\nfor column in continuous_columns:\n visualize_continuous_metadata_column_histogram(\n df=df_metadata_continuous_columns[column],\n title=f'Series {column} Distribution'\n )","metadata":{"id":"XHHSqh8XTl-5","execution":{"iopub.status.busy":"2024-09-03T10:15:30.343629Z","iopub.execute_input":"2024-09-03T10:15:30.344048Z","iopub.status.idle":"2024-09-03T10:15:47.191806Z","shell.execute_reply.started":"2024-09-03T10:15:30.344005Z","shell.execute_reply":"2024-09-03T10:15:47.190519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualize data","metadata":{"id":"4On2GCfvTl-5"}},{"cell_type":"code","source":"study_sample = df_train_series_descriptions.loc[df_train_series_descriptions.study_id == df_train_series_descriptions.study_id.iloc[10]]\n\nplot_dicom_images(study_sample, train_images_dir[0])\n\nax_t2 = sag_t1 = sag_t2 = None\nfor row in study_sample.itertuples():\n if row.series_description == \"Sagittal T2/STIR\":\n sag_t2 = load_dicom_stack(os.path.join(train_images_dir[0], str(row.study_id), str(row.series_id)), plane=\"sagittal\")\n elif row.series_description == \"Sagittal T1\":\n sag_t1 = load_dicom_stack(os.path.join(train_images_dir[0], str(row.study_id), str(row.series_id)), plane=\"sagittal\")\n elif row.series_description == \"Axial T2\":\n ax_t2 = load_dicom_stack(os.path.join(train_images_dir[0], str(row.study_id), str(row.series_id)), plane=\"axial\", reverse_sort=True)\n\n if sag_t2 and ax_t2:\n break\n\n# Assuming the variables sag_t2 and ax_t2 are defined in the study\nmap_axial_to_sagittal(sag_t2, ax_t2)","metadata":{"id":"BEcJ2fjATl-6","execution":{"iopub.status.busy":"2024-09-03T10:15:47.193998Z","iopub.execute_input":"2024-09-03T10:15:47.194483Z","iopub.status.idle":"2024-09-03T10:16:00.733645Z","shell.execute_reply.started":"2024-09-03T10:15:47.194406Z","shell.execute_reply":"2024-09-03T10:16:00.732211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dicom_smpl_path = '/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/train_images/1004726367/1709080005/12.dcm'\ndicom_dataset = pydicom.dcmread(dicom_smpl_path)\n\n# Extract pixel data\npixel_array = dicom_dataset.pixel_array\n\nscaled = (pixel_array - pixel_array.min()) / (pixel_array.max() - pixel_array.min())\n\n# Print some basic information\nprint(f\"Shape of pixel array: {pixel_array.shape}\")\nprint(f\"Data type of pixel array: {pixel_array.dtype}\")\n\n# Display the pixel values\nprint(\"Pixel values:\")\nprint(pixel_array)\nprint(f\"Min and max of pixel values: {np.min(pixel_array), np.max(pixel_array)}\")\n\n\n# Display scaled pixel values\nprint(\"Scaled pixel values:\")\nprint(scaled)\n\nplt.imshow(pixel_array, cmap='gray')\nplt.colorbar()\nplt.title('DICOM Image')\nplt.show()\n\nplt.imshow(scaled, cmap='gray')\nplt.colorbar()\nplt.title('Scaled DICOM Image')\nplt.show()","metadata":{"id":"9X63i1A0Tl-6","execution":{"iopub.status.busy":"2024-09-03T10:16:00.735686Z","iopub.execute_input":"2024-09-03T10:16:00.736181Z","iopub.status.idle":"2024-09-03T10:16:01.676870Z","shell.execute_reply.started":"2024-09-03T10:16:00.736130Z","shell.execute_reply":"2024-09-03T10:16:01.675627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"id":"5MY7EXgNTl-7"},"execution_count":null,"outputs":[]}]}