{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":71549,"databundleVersionId":8561470,"sourceType":"competition"}],"dockerImageVersionId":30746,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# RSNA 2024 Lumbar Spine Degenerative Classification\n\n### Why this competition matters? \n**Low back pain** is the leading cause of disability worldwide, according to the World Health Organization, affecting 619 million people in 2020.\n\n**Magnetic resonance imaging (MRI)** provides a detailed view of the lumbar spine vertebra, discs and nerves, enabling radiologists to assess the presence and severity of these conditions.\n\nThe challenge will focus on the classification of **five lumbar spine degenerative conditions**: Left Neural Foraminal Narrowing, Right Neural Foraminal Narrowing, Left Subarticular Stenosis, Right Subarticular Stenosis, and Spinal Canal Stenosis. For each imaging study in the dataset, we’ve provided severity scores (Normal/Mild, Moderate, or Severe) for each of the five conditions across the intervertebral disc levels L1/L2, L2/L3, L3/L4, L4/L5, and L5/S1.","metadata":{}},{"cell_type":"markdown","source":"# Terminology \n\n### What is the [Lumbar Spine](https://www.theskeletalsystem.net/spine-vertebral-column/lumbar-vertebrae.html)?\nThe **lumbar spine** is the third and lowermost part of the spinal column, consisting of 5 lumbar vertebrae, **L1-L5**.\n\n**Lumbar Vertebrae**\n![image.png](https://www.theskeletalsystem.net/wp-content/uploads/2022/05/Lumbar-Vertebrae.jpg)","metadata":{}},{"cell_type":"markdown","source":"### What are the target symptoms?\n\n[**Neural Foraminal Narrowing**](https://www.usaspinecare.com/back_problems/foraminal_narrowing/)\nForaminal narrowing, or foraminal stenosis, is a condition of the spine that can cause pain and other symptoms resulting from spinal nerve root compression. \n\n![image.png](https://www.usaspinecare.com/wp-content/uploads/2020/10/foraminal-narrowing-3d-render.jpg)\n\n[**Stenosis**](https://mayfieldclinic.com/pe-sten.htm)\nIn a normal vertebra, the spinal canal and the nerve root canals have ample space for passage of the spinal cord and nerves. In spinal stenosis, bone spurs, enlarged facet joints and a bulging disc constrict the nerve root canals causing compression and entrapment of the spinal nerves; also called lateral or foraminal stenosis.\n\n![image.png](https://dbmmu5j6lxew.cloudfront.net/pe-stenosis-fig1b.jpg)","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport os.path as osp\nimport numpy as np\nimport pydicom\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:04.783501Z","iopub.execute_input":"2024-08-06T01:17:04.783883Z","iopub.status.idle":"2024-08-06T01:17:05.409176Z","shell.execute_reply.started":"2024-08-06T01:17:04.783854Z","shell.execute_reply":"2024-08-06T01:17:05.407781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Files\n### train.csv\n- `study_id`: The study ID. Each study may include multiple series of images.\n- `[condition]_[level]`: The target labels, such as **spinal_canal_stenosis_l1_l2**, with the severity levels of Normal/Mild, Moderate, or Severe. Some entries have incomplete labels.","metadata":{}},{"cell_type":"code","source":"root_dir = '/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification'\ntrain_df = pd.read_csv(osp.join(root_dir, 'train.csv'))\ntrain_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:05.411321Z","iopub.execute_input":"2024-08-06T01:17:05.412455Z","iopub.status.idle":"2024-08-06T01:17:05.481568Z","shell.execute_reply.started":"2024-08-06T01:17:05.412410Z","shell.execute_reply":"2024-08-06T01:17:05.480349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### train_label_coordinates.csv\n- `study_id`\n- `series_id`: The imagery series ID.\n- `instance_number`: The image's order number within the 3D stack.\n- `condition`: There are three core conditions: spinal canal stenosis, neural_foraminal_narrowing, and subarticular_stenosis. The latter two are considered for each side of the spine.\n- `level`:  The relevant vertebrae, such as l3_l4\n- `[x/y]`: The x/y coordinates for the center of the area that defined the label.","metadata":{}},{"cell_type":"code","source":"train_label_df = pd.read_csv(osp.join(root_dir, 'train_label_coordinates.csv'))\ntrain_label_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:05.482879Z","iopub.execute_input":"2024-08-06T01:17:05.483222Z","iopub.status.idle":"2024-08-06T01:17:05.604229Z","shell.execute_reply.started":"2024-08-06T01:17:05.483189Z","shell.execute_reply":"2024-08-06T01:17:05.602988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### train_series_descriptions.csv\n\n- `study_id`\n- `series_id`\n- `series_description`: The scan's orientation.\n\nThe [sagittal plane](https://anatomytool.org/content/lecturio-drawing-sagittal-coronal-and-transverse-plane-english-labels) is an anatomical plane that divides the body into right and left sections.\n\n![image.png](https://anatomytool.org/sites/default/files/CT-Image-Planes-768x768.jpg)","metadata":{}},{"cell_type":"code","source":"train_series_desc_df = pd.read_csv(osp.join(root_dir, 'train_series_descriptions.csv'))\ntrain_series_desc_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:05.606674Z","iopub.execute_input":"2024-08-06T01:17:05.607072Z","iopub.status.idle":"2024-08-06T01:17:05.630436Z","shell.execute_reply.started":"2024-08-06T01:17:05.607040Z","shell.execute_reply":"2024-08-06T01:17:05.629018Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### sample_submission.csv\n\n- `row_id`: A slug of the study ID, condition, and level such as **12345_spinal_canal_stenosis_l3_l4**.\n- `[normal_mild/moderate/severe]`: The three prediction columns.","metadata":{}},{"cell_type":"code","source":"sample_submission_df = pd.read_csv(osp.join(root_dir, 'sample_submission.csv'))\nsample_submission_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:05.631819Z","iopub.execute_input":"2024-08-06T01:17:05.632188Z","iopub.status.idle":"2024-08-06T01:17:05.649569Z","shell.execute_reply.started":"2024-08-06T01:17:05.632158Z","shell.execute_reply":"2024-08-06T01:17:05.648060Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### DCM images\n\n- `[train/test]_images/[study_id]/[series_id]/[instance_number].dcm`","metadata":{}},{"cell_type":"code","source":"filtered_df = train_series_desc_df[train_series_desc_df['series_description'].str.contains('Sagittal')]\n\nselected_study_ids = filtered_df['study_id'].sample(n=4, random_state=1).values\nselected_series_ids = filtered_df['series_id'].sample(n=4, random_state=1).values\n\nplt.figure(figsize=(10, 10))\n\nfor i, (study_id, series_id) in enumerate(zip(selected_study_ids, selected_series_ids)):\n    dicom_path = osp.join(root_dir, 'train_images', str(study_id), str(series_id), '10.dcm') \n    dicom_data = pydicom.dcmread(dicom_path)\n    image = dicom_data.pixel_array\n    plt.subplot(2, 2, i + 1)\n    plt.imshow(image, cmap='gray')\n    plt.title(f'Series ID: {series_id}')\n    plt.axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:05.651369Z","iopub.execute_input":"2024-08-06T01:17:05.652210Z","iopub.status.idle":"2024-08-06T01:17:06.633781Z","shell.execute_reply.started":"2024-08-06T01:17:05.652170Z","shell.execute_reply":"2024-08-06T01:17:06.632308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filtered_df = train_series_desc_df[train_series_desc_df['series_description'].str.contains('Axial')]\n\nselected_study_ids = filtered_df['study_id'].sample(n=4, random_state=1).values\nselected_series_ids = filtered_df['series_id'].sample(n=4, random_state=1).values\n\nplt.figure(figsize=(10, 10))\n\nfor i, (study_id, series_id) in enumerate(zip(selected_study_ids, selected_series_ids)):\n    dicom_path = osp.join(root_dir, 'train_images', str(study_id), str(series_id), '10.dcm') \n    dicom_data = pydicom.dcmread(dicom_path)\n    image = dicom_data.pixel_array\n    plt.subplot(2, 2, i + 1)\n    plt.imshow(image, cmap='gray')\n    plt.title(f'Series ID: {series_id}')\n    plt.axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:06.635358Z","iopub.execute_input":"2024-08-06T01:17:06.635749Z","iopub.status.idle":"2024-08-06T01:17:07.713460Z","shell.execute_reply.started":"2024-08-06T01:17:06.635716Z","shell.execute_reply":"2024-08-06T01:17:07.712191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def visualize_dicom_with_coordinates(study_id, series_id, instance_number, coordinates_df):\n    dicom_path = f'/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/train_images/{study_id}/{series_id}/{instance_number}.dcm'\n    if not osp.exists(dicom_path):\n        print(f\"File not found: {dicom_path}\")\n        return\n\n    ds = pydicom.dcmread(dicom_path)\n    \n    plt.figure(figsize=(5, 5))\n    plt.imshow(ds.pixel_array, cmap='gray')\n    \n    # Plot the coordinates\n    relevant_coords = coordinates_df[(coordinates_df['study_id'] == study_id) & \n                                     (coordinates_df['series_id'] == series_id) & \n                                     (coordinates_df['instance_number'] == instance_number)]\n    \n    for _, row in relevant_coords.iterrows():\n        plt.scatter(row['x'], row['y'], c='red', s=50, label=row['condition'] if 'condition' in row else '')\n    \n    plt.title(f'Study: {study_id}, Series: {series_id}, Instance: {instance_number}')\n    plt.axis('off')\n    plt.legend(loc='upper right')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:07.715394Z","iopub.execute_input":"2024-08-06T01:17:07.715788Z","iopub.status.idle":"2024-08-06T01:17:07.726503Z","shell.execute_reply.started":"2024-08-06T01:17:07.715756Z","shell.execute_reply":"2024-08-06T01:17:07.725268Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"conditions = ['Spinal Canal Stenosis', 'Right Neural Foraminal Narrowing',\n 'Left Neural Foraminal Narrowing', 'Left Subarticular Stenosis',\n 'Right Subarticular Stenosis']\n\nrandom_rows = {}\nfor condition in conditions:\n    condition_rows = train_label_df[train_label_df['condition'] == condition]\n    if not condition_rows.empty:\n        random_row = condition_rows.sample(n=1)\n        random_rows[condition] = random_row\n\n# Print the selected rows\nfor condition, row in random_rows.items():\n    print(f\"Random row for condition '{condition}':\")\n    print(row)\n    print()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:07.727987Z","iopub.execute_input":"2024-08-06T01:17:07.728384Z","iopub.status.idle":"2024-08-06T01:17:07.784378Z","shell.execute_reply.started":"2024-08-06T01:17:07.728352Z","shell.execute_reply":"2024-08-06T01:17:07.783219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"random_rows_vals = list(random_rows.values())","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:07.788806Z","iopub.execute_input":"2024-08-06T01:17:07.789311Z","iopub.status.idle":"2024-08-06T01:17:07.794681Z","shell.execute_reply.started":"2024-08-06T01:17:07.789277Z","shell.execute_reply":"2024-08-06T01:17:07.793441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 0\n\nstudy_id = random_rows_vals[idx]['study_id'].values[0]\nseries_id = random_rows_vals[idx]['series_id'].values[0]\ninstance_number = random_rows_vals[idx]['instance_number'].values[0]\n\nvisualize_dicom_with_coordinates(study_id, series_id, instance_number, train_label_df)","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:07.796088Z","iopub.execute_input":"2024-08-06T01:17:07.796425Z","iopub.status.idle":"2024-08-06T01:17:08.183509Z","shell.execute_reply.started":"2024-08-06T01:17:07.796398Z","shell.execute_reply":"2024-08-06T01:17:08.182172Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 1\n\nstudy_id = random_rows_vals[idx]['study_id'].values[0]\nseries_id = random_rows_vals[idx]['series_id'].values[0]\ninstance_number = random_rows_vals[idx]['instance_number'].values[0]\n\nvisualize_dicom_with_coordinates(study_id, series_id, instance_number, train_label_df)","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:08.185010Z","iopub.execute_input":"2024-08-06T01:17:08.185433Z","iopub.status.idle":"2024-08-06T01:17:08.513523Z","shell.execute_reply.started":"2024-08-06T01:17:08.185399Z","shell.execute_reply":"2024-08-06T01:17:08.512291Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 2\n\nstudy_id = random_rows_vals[idx]['study_id'].values[0]\nseries_id = random_rows_vals[idx]['series_id'].values[0]\ninstance_number = random_rows_vals[idx]['instance_number'].values[0]\n\nvisualize_dicom_with_coordinates(study_id, series_id, instance_number, train_label_df)","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:08.515223Z","iopub.execute_input":"2024-08-06T01:17:08.515586Z","iopub.status.idle":"2024-08-06T01:17:08.735238Z","shell.execute_reply.started":"2024-08-06T01:17:08.515554Z","shell.execute_reply":"2024-08-06T01:17:08.733851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"idx = 3\n\nstudy_id = random_rows_vals[idx]['study_id'].values[0]\nseries_id = random_rows_vals[idx]['series_id'].values[0]\ninstance_number = random_rows_vals[idx]['instance_number'].values[0]\n\nvisualize_dicom_with_coordinates(study_id, series_id, instance_number, train_label_df)","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:08.737049Z","iopub.execute_input":"2024-08-06T01:17:08.737467Z","iopub.status.idle":"2024-08-06T01:17:09.052472Z","shell.execute_reply.started":"2024-08-06T01:17:08.737432Z","shell.execute_reply":"2024-08-06T01:17:09.051129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Missing or Broken Values \n\nWe don't have missing values in `train_label_coordinates.csv` and `train_series_descriptions.csv`. However, we need to implement missing value handling in our dataloader for the training process.","metadata":{}},{"cell_type":"code","source":"total_cells = train_df.size\nmissing_cells = train_df.isnull().sum().sum()\nmissing_percentage = (missing_cells / total_cells) * 100\n\nprint(f\"Total number of data points: {total_cells}\")\nprint(f\"Number of missing data points: {missing_cells}\")\nprint(f\"Percentage of missing data: {missing_percentage:.2f}%\")\n\nprint('-----------------------------------')\nprint(\"Missing values in train.csv:\")\nprint(train_df.isnull().sum())\n\nprint('-----------------------------------')\nprint(\"Missing values in train_label_coordinates.csv:\")\nprint(train_label_df.isnull().sum())\n\nprint('-----------------------------------')\nprint(\"Missing values in train_series_descriptions.csv:\")\nprint(train_series_desc_df.isnull().sum())","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:09.054272Z","iopub.execute_input":"2024-08-06T01:17:09.054742Z","iopub.status.idle":"2024-08-06T01:17:09.083520Z","shell.execute_reply.started":"2024-08-06T01:17:09.054699Z","shell.execute_reply":"2024-08-06T01:17:09.082173Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"rows_with_missing = train_df.isnull().any(axis=1)\nmissing_indices = rows_with_missing[rows_with_missing].index\nexample_index = np.random.choice(missing_indices)\n\nexample_df = train_df.iloc[example_index].copy()\nexample_df","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:09.085513Z","iopub.execute_input":"2024-08-06T01:17:09.086140Z","iopub.status.idle":"2024-08-06T01:17:09.107716Z","shell.execute_reply.started":"2024-08-06T01:17:09.086091Z","shell.execute_reply":"2024-08-06T01:17:09.106462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\n# Create a boolean mask for missing values\nmissing_mask = train_df.isnull()\n\n# Create the plot\nfig, ax = plt.subplots(figsize=(8, 15))\n\n# Plot the missingness map\nim = ax.imshow(missing_mask, cmap='binary', aspect='auto', interpolation='nearest')\n\n# Customize the plot\nax.set_title('Missingness Map of Training Data', fontsize=16)\nax.set_xlabel('Columns', fontsize=14)\nax.set_ylabel('Row Index', fontsize=14)\n\n# Set x-axis ticks to column names\nax.set_xticks(range(len(train_df.columns)))\nax.set_xticklabels(train_df.columns, rotation=90, ha='right')\n\nplt.text(1.02, 0.5, 'Missing (Black) vs Present (White)', \n         rotation=270, transform=ax.transAxes, \n         verticalalignment='center', fontsize=12)\n\n# Adjust layout and display\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:09.109418Z","iopub.execute_input":"2024-08-06T01:17:09.109796Z","iopub.status.idle":"2024-08-06T01:17:09.783385Z","shell.execute_reply.started":"2024-08-06T01:17:09.109765Z","shell.execute_reply":"2024-08-06T01:17:09.782088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Broken Values in CSVs\n\n`train.csv`","metadata":{}},{"cell_type":"code","source":"unique_values = {column: train_df[column].unique() for column in train_df.columns}\n\nfor column, values in unique_values.items():\n    print(f\"Unique values in column '{column}':\")\n    print(values)\n    print()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:09.784936Z","iopub.execute_input":"2024-08-06T01:17:09.785366Z","iopub.status.idle":"2024-08-06T01:17:09.804518Z","shell.execute_reply.started":"2024-08-06T01:17:09.785330Z","shell.execute_reply":"2024-08-06T01:17:09.802947Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`train_label_coordinates.csv`","metadata":{}},{"cell_type":"code","source":"unique_values = {column: train_label_df[column].unique() for column in train_label_df.columns}\n\nfor column, values in unique_values.items():\n    print(f\"Unique values in column '{column}':\")\n    print(values)\n    print()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:09.806184Z","iopub.execute_input":"2024-08-06T01:17:09.806661Z","iopub.status.idle":"2024-08-06T01:17:09.836321Z","shell.execute_reply.started":"2024-08-06T01:17:09.806615Z","shell.execute_reply":"2024-08-06T01:17:09.835207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`train_series_descriptions.csv`","metadata":{}},{"cell_type":"code","source":"unique_values = {column: train_series_desc_df[column].unique() for column in train_series_desc_df.columns}\n\nfor column, values in unique_values.items():\n    print(f\"Unique values in column '{column}':\")\n    print(values)\n    print()","metadata":{"execution":{"iopub.status.busy":"2024-08-06T01:17:09.837989Z","iopub.execute_input":"2024-08-06T01:17:09.838338Z","iopub.status.idle":"2024-08-06T01:17:09.848352Z","shell.execute_reply.started":"2024-08-06T01:17:09.838307Z","shell.execute_reply":"2024-08-06T01:17:09.847096Z"},"trusted":true},"execution_count":null,"outputs":[]}]}