{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":71549,"databundleVersionId":8561470,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction","metadata":{}},{"cell_type":"markdown","source":"## Topic\nLow back pain, a leading cause of disability, is often due to degenerative spine conditions like spondylosis, which involves disc deterioration and nerve compression. MRI is crucial for diagnosis and treatment planning. The RSNA and ASNR have launched a competition to explore AI's ability to identify and classify five key lumbar spine conditions using MRI, supported by a comprehensive, globally sourced dataset.\n\n## Objective of the competition\nThe picture below shows the different areas of the spine. This study focuses on the lumbar spine, which consists of five vertrebrals, each separated by an vertebral disc.\n\nThe objective of the competetion is to utilize neural networks to assess the degree of degenerative diseases in the lumbar spine based on MRI scans taken from three perspectives, also called planes (Sagittal T1, Sagittal T2/STIR and Axial T2). The planes are displayed in the second picture below. \n\nSpecifically, the focus is on identifying and classifying three conditions: foraminal narrowing (on left and right side), subarticular stenosis (on left and right side) and canal stenosis. These conditions can appear between the 5 vertrebrals L1/L2, L2/L3, L3/L4, L4/L5, L5/S1.\n\nThe severity of these conditions at each vertebral transition has to be categorized into three levels: normal/mild, moderate, and severe. For each level, the probability that the condition is present at that severity has to be estimated.\n\n## Objective of the study\nThis study specifically focuses on the classification of left subarticular stenosis between the L4/L5 vertebral levels. The selection of this focus area is driven by the observation that the data distribution for this particular condition is more balanced compared to others, making it more suitable for a proof-of-concept study. By establishing a reliable model for left subarticular stenosis at the L4/L5 transition, the effectiveness of the proposed neural network approach can be validated and provide a solid foundation for extending the classification framework to other conditions and vertebral levels.\n\n
\n
\n

Lumbar spine

\n \"Lumbar\n

Source: Leading Medicine Guide

\n
\n
\n

Planes

\n \"Planes\"\n

Source: Lecturio

\n
\n
\n\n\n## Project Methodology\n\nBefore commencing with the actual implementation, an overview of existing research on the use of CNNs for the classification of spinal disorders using MRI scans will be provided. The insights gained from these studies will be used to draw conclusions for the approach of the present work. Subsequently, the implementation will begin, following these steps:\n\n1. **Data Exploration**: \n Initially, a thorough exploration of the dataset will be undertaken to gain insights into the distribution and characteristics of the data. This step will involve analyzing how the different degenerative conditions of the lumbar spine present across varying levels of severity. The purpose of this exploration is to understand the nuances of the data, which will inform subsequent steps in the project.\n\n2. **Data Preparation**: \n Following exploration, the data will be prepared for analysis. This process includes merging relevant information from multiple tables, performing necessary preprocessing steps and cleaning the data to remove any inconsistencies or noise. Additionally, the scans will be preprocessed and standardized to ensure all are in the appropriate format and quality required for input into the neural networks. The objective of this stage is to transform the raw data into a structured and standardized format suitable for training the neural network.\n\n3. **Model Implementation**: \n The core of the project involves developing a Convolutional Neural Network (CNN) specifically tailored to classify the severity of left subarticular stenosis between the L4/L5 vertebral levels. This model will predict the probability of the disease severity levels: normal/mild, moderate, and severe. This approach allows for a focused and condition-specific analysis, aiming to achieve high accuracy in severity classification in upcoming work. Upon completion of the model development, the results will be summarized. This summary will include an evaluation of the model accuracy and any observed limitations. The findings will be used for model improvements.\n \n4. **Summary of Results and Future Work**:\n At least the study will be summarized and recommendations for future work will be provided, focusing on strategies to optimize the results. This will include exploring advanced preprocessing techniques and alternative neural network architectures.","metadata":{}},{"cell_type":"markdown","source":"# Recent research\nThis chapter brings together three studies that have focused on the classification of spinal disorders using convolutional neural networks (CNNs).\n\n## Chosen papers\nOne notable study by [Liawrungrueang et al. (2023)](https://www.mdpi.com/2075-4418/13/4/663) explored the use of a CNN model to automatically detect and classify intervertebral disc degeneration (IDD) based on sagittal T2-weighted MRI scans. The study utilized the YOLOv5 model as CNN architecture to identify various levels of disc degeneration. They used the Pfirrmann grading system to label different levels of disc degeneration and demonstrated that their CNN model achieved over 95% accuracy in classification, proving the effectiveness of deep learning in diagnosing spinal conditions.\n\nAnother contribution comes from [Al-Kubaisi and Khamiss (2022)](https://www.mdpi.com/2079-9292/11/1/85). They applied transfer learning to enhance the classification of lumbar disc conditions. Their approach utilized pretrained models such as ResNet50 and VGG19. They introduced a region of interest (ROI) technique to focus on specific areas of the spine. This technique improved the performance of the classifiers significantly, particularly when dealing with limited data. The study demonstrated that transfer learning from domain-relevant datasets outperformed models trained from scratch.\n\nAdditionally, [Tumko et al. (2023)](https://link.springer.com/article/10.1007/s00586-023-08089-2) developed a multi-stage neural network that integrated segmentation and classification models to detect and grade stenosis in the lumbar spine. This architecture employed a U-Net model for anatomical segmentation, which processed grayscale MRI images and produced pixel-wise masks delineating 17 anatomical structures within the lumbar region. Following this, RegNetY800MF-based classification models were utilized to determine the presence of central, foraminal, and lateral recess stenosis. For each type of stenosis, a severity classification was performed using separate models based on the RegNetY32GF architecture.\n\n## Design decisions for study\nThis study will follow the ROI-approach and preprocess the scans to include only the relevant patches of the spine. But: This study won’t use transfer learning, it will develop a CNN from scratch. This decision is mainly influenced by the author's first-time experience working with neural networks. Pretrained models, while efficient, abstract much of the critical design process, limiting the opportunity to fully engage with the foundational principles of network construction. By building the model from scratch, the author aims to gain deeper insights into how architectural choices affect performance.","metadata":{}},{"cell_type":"markdown","source":"# Import libraries","metadata":{}},{"cell_type":"code","source":"# Python\nimport os\nimport glob\nfrom typing import TypedDict, List, Dict\nfrom collections import Counter\n\n# Data\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n\n# Scans\nimport cv2\nimport pydicom\nfrom matplotlib.patches import Circle\n\n# Helpers\nfrom tqdm import tqdm\nimport warnings","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.008349Z","iopub.execute_input":"2024-09-15T15:46:03.008724Z","iopub.status.idle":"2024-09-15T15:46:03.014989Z","shell.execute_reply.started":"2024-09-15T15:46:03.008688Z","shell.execute_reply":"2024-09-15T15:46:03.014049Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Data Exploration\nIn this chapter the existing data is read, described and explored.\n\nA short note about the existing scans: In the directory /kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/train_images, each *study_id* has its own folder, named accordingly. Within each study folder, there are additional subfolders for each series, named after the corresponding *series_id*. Inside these series folders, multiple DICOM (dcm) files are stored, each containing an image. The filenames of these images include the instance number, which corresponds to the *instance_number* column in the train_label_coordinates.csv file.","metadata":{}},{"cell_type":"markdown","source":"## Read data","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/train.csv')\nseries_description_df = pd.read_csv('/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/train_series_descriptions.csv')\nlabel_coordinates_df = pd.read_csv('/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/train_label_coordinates.csv')\n\npath_train_images = '/kaggle/input/rsna-2024-lumbar-spine-degenerative-classification/train_images'","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.016743Z","iopub.execute_input":"2024-09-15T15:46:03.017075Z","iopub.status.idle":"2024-09-15T15:46:03.116934Z","shell.execute_reply.started":"2024-09-15T15:46:03.017011Z","shell.execute_reply":"2024-09-15T15:46:03.116190Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Explore data\nThree CSV files are provided, which are connected by *study_id* and *series_id*. The *study_id* identifies the study corresponding to a specific patient. Each study consists of multiple series, and each series contains several scans (*instance_number*) captured from a specific perspective. The images are not standardized, with varying image sizes and zoom levels.","metadata":{}},{"cell_type":"markdown","source":"### Train data\nThis dataframe includes a row for each patient, categorizing the severity for each condition for each vertebral disc. As the .info() displays, the dataframe includes data for 1975 patients.","metadata":{}},{"cell_type":"code","source":"train_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.118448Z","iopub.execute_input":"2024-09-15T15:46:03.118771Z","iopub.status.idle":"2024-09-15T15:46:03.140176Z","shell.execute_reply.started":"2024-09-15T15:46:03.118736Z","shell.execute_reply":"2024-09-15T15:46:03.139251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.141554Z","iopub.execute_input":"2024-09-15T15:46:03.141849Z","iopub.status.idle":"2024-09-15T15:46:03.160974Z","shell.execute_reply.started":"2024-09-15T15:46:03.141806Z","shell.execute_reply":"2024-09-15T15:46:03.160014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Series data\nThis dataframe identifies the perspective for each series. There are three perspectives: Sagittal T1, Sagittal T2/STIR and Axial T2. They're visualized in the introduction Each perspective emphasizes a different area of the anatomy. Therefore, depending on the condition, a different perspective is preferred. This can be seen later in *grouped_conditions*. \n\nThere are 6293 series for 1975 studies, which results in ~3,1 series per study. It's important to mention that the some studies have more series than others.","metadata":{}},{"cell_type":"code","source":"series_description_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.163497Z","iopub.execute_input":"2024-09-15T15:46:03.163775Z","iopub.status.idle":"2024-09-15T15:46:03.172412Z","shell.execute_reply.started":"2024-09-15T15:46:03.163745Z","shell.execute_reply":"2024-09-15T15:46:03.171538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series_description_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.173683Z","iopub.execute_input":"2024-09-15T15:46:03.173986Z","iopub.status.idle":"2024-09-15T15:46:03.184853Z","shell.execute_reply.started":"2024-09-15T15:46:03.173943Z","shell.execute_reply":"2024-09-15T15:46:03.183906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"series_description_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.185946Z","iopub.execute_input":"2024-09-15T15:46:03.186255Z","iopub.status.idle":"2024-09-15T15:46:03.199530Z","shell.execute_reply.started":"2024-09-15T15:46:03.186224Z","shell.execute_reply":"2024-09-15T15:46:03.198516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Coordinates data\nThis dataframe identifies the center of the area that defined the label in train_df. This information will be used to cut off the relevant part (including a padding) of the scans to minimize the information the CNN has to handle. This is done in *Preprocess images*.\n\nThere are 48632 scans with related coordinates for 6293 series, which means that every series includes about 7,7 scans.","metadata":{}},{"cell_type":"code","source":"label_coordinates_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.200871Z","iopub.execute_input":"2024-09-15T15:46:03.201444Z","iopub.status.idle":"2024-09-15T15:46:03.215071Z","shell.execute_reply.started":"2024-09-15T15:46:03.201402Z","shell.execute_reply":"2024-09-15T15:46:03.214105Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"label_coordinates_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.216200Z","iopub.execute_input":"2024-09-15T15:46:03.216487Z","iopub.status.idle":"2024-09-15T15:46:03.236555Z","shell.execute_reply.started":"2024-09-15T15:46:03.216456Z","shell.execute_reply":"2024-09-15T15:46:03.235609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data distribution\nThe distribution shows an important issue: The severities aren't evenly distributed for each condition between the vertebrals. For foraminal narrowing and subarticular stenosis there are much more normal/mild than moderate and severe cases. For the canal distribution there are much more normal/mild cases than moderate and severe cases. This has to be considered when training the model by weighting the categories or generating additional training data by data augmentation (in case of scans: varying existing scans, for example by rotating them).","metadata":{}},{"cell_type":"code","source":"# Remove study_id\nconditions = train_df.columns[1:] \n\n# Aggregate for each condition\nseverity_data = []\n\nfor condition in conditions:\n severity_counts = train_df[condition].value_counts()\n severity_data.append(severity_counts)\n \nseverity_df = pd.DataFrame(severity_data, index=conditions).fillna(0)\n\n# Display\nax = severity_df.plot(kind='bar', stacked=True, figsize=(10, 7))\nax.set_title('Severity Distribution per Condition')\nax.set_xlabel('Condition')\nax.set_ylabel('Frequency')\nax.legend(title='Severity')\nplt.xticks(rotation=45, ha=\"right\")\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:03.237575Z","iopub.execute_input":"2024-09-15T15:46:03.237910Z","iopub.status.idle":"2024-09-15T15:46:04.087874Z","shell.execute_reply.started":"2024-09-15T15:46:03.237878Z","shell.execute_reply":"2024-09-15T15:46:04.086957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Frequency tables","metadata":{}},{"cell_type":"code","source":"train_df.melt(value_name='value')['value'].value_counts().loc[['Normal/Mild', 'Moderate', 'Severe']].reset_index()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:04.091521Z","iopub.execute_input":"2024-09-15T15:46:04.091856Z","iopub.status.idle":"2024-09-15T15:46:04.119514Z","shell.execute_reply.started":"2024-09-15T15:46:04.091820Z","shell.execute_reply":"2024-09-15T15:46:04.118677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since the distribution for left subarticular stenosis between L4/L5 is most evenly, this class will be used for a proof of concept of the model. ","metadata":{}},{"cell_type":"code","source":"train_df['left_subarticular_stenosis_l4_l5'].value_counts().reset_index()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:04.120559Z","iopub.execute_input":"2024-09-15T15:46:04.120849Z","iopub.status.idle":"2024-09-15T15:46:04.131181Z","shell.execute_reply.started":"2024-09-15T15:46:04.120817Z","shell.execute_reply":"2024-09-15T15:46:04.130123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Data Preparation\nIn this chapter the three separated dataframes are merged in one dataframe. The resulting dataframe enables the fast selection of the relevant cases (X) and their related scan-paths (y). This will be utilized to display scans with several attribute-combinations (plane, condition, severity) to get a better understanding of the appearance of the conditions and their severities and the format of the scans. At least the scans will be preprocessed to ensure a propper format for the CNN training.","metadata":{}},{"cell_type":"markdown","source":"## Preprocess data\nThe result is a dataframe which includes all existing information for the categorization of a severity in one row: the study_id, series_id, instance_number, condition, level, x and y coordinates of the center of the area defined in condition, series_description and transformed severity (Normal/Mind -> 0, Moderate -> 1, Severe -> 2).","metadata":{}},{"cell_type":"markdown","source":"### Helper functions","metadata":{}},{"cell_type":"code","source":"def __transform_label(label):\n \"\"\"Transform categorized severity labels\"\"\"\n if label == 'Normal/Mild':\n return 0\n elif label == 'Moderate':\n return 1\n elif label == 'Severe':\n return 2\n else:\n return np.nan\n \ndef get_transformed_severity(row):\n \"\"\"Get severity for a row of the train_df\"\"\"\n condition = row['condition'].lower().replace(\" \", \"_\")\n level = level = row['level'].lower().replace(\"/\", \"_\")\n severity_column_name = f\"{condition}_{level}\"\n \n return __transform_label(row[severity_column_name])","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:04.132570Z","iopub.execute_input":"2024-09-15T15:46:04.132910Z","iopub.status.idle":"2024-09-15T15:46:04.140612Z","shell.execute_reply.started":"2024-09-15T15:46:04.132874Z","shell.execute_reply":"2024-09-15T15:46:04.139461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Apply helpers","metadata":{}},{"cell_type":"code","source":"# Merge coordinates with series description to get perspective information\ntrain_merged_df = label_coordinates_df.merge(series_description_df, on=['study_id', 'series_id'])\n\n# Merge the merged data with the train labels to add severity information\ntrain_final_df = train_merged_df.merge(train_df, on='study_id')\n\n# Add column for relevant severity per row & transform to numeric value\ntrain_final_df['severity'] = train_final_df.apply(get_transformed_severity, axis=1)\n\n# Get all _ column names (will be used later)\nlabel_columns = list(train_final_df.columns)[8:]\nlabel_columns.pop() # remove severity-column\n\n# Remove all unused column names\ntrain_final_df = train_final_df.drop(columns=label_columns)\n\n# Remove na values\ntrain_final_df = train_final_df.dropna()\n\ntrain_final_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:04.141625Z","iopub.execute_input":"2024-09-15T15:46:04.141891Z","iopub.status.idle":"2024-09-15T15:46:05.256750Z","shell.execute_reply.started":"2024-09-15T15:46:04.141861Z","shell.execute_reply":"2024-09-15T15:46:05.255724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Scan distribution\nThe competetion description doesn't provide detailed information which condition is scanned with which plane, it only says that some conditions can be observed better in the axial plane (subarticular stenosis, canal stenosis) and others in the sagittal plane (foraminal narrowing). The following grouped df provides deeper information: Each plane is specified for one condition. There are five exceptions in the sagittal T1 plane for spinal canal stenosis. These are removed to have a clean split between the planes.\n\nIn general, the 48692 scans are even distributed over the several conditions.","metadata":{}},{"cell_type":"code","source":"grouped_conditions = train_final_df.groupby(['series_description', 'condition']).size().unstack(fill_value=0)\n\n# Remove five spinal canal stenosis entries in Sagittal T1\ntrain_final_df = train_final_df[~((train_final_df['condition'] == 'Spinal Canal Stenosis') & (train_final_df['series_description'] == 'Sagittal T1'))]\n\ngrouped_conditions","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:05.257886Z","iopub.execute_input":"2024-09-15T15:46:05.258223Z","iopub.status.idle":"2024-09-15T15:46:05.299089Z","shell.execute_reply.started":"2024-09-15T15:46:05.258187Z","shell.execute_reply":"2024-09-15T15:46:05.298168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_final_df.info()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:05.300479Z","iopub.execute_input":"2024-09-15T15:46:05.300769Z","iopub.status.idle":"2024-09-15T15:46:05.324668Z","shell.execute_reply.started":"2024-09-15T15:46:05.300737Z","shell.execute_reply":"2024-09-15T15:46:05.323445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Display scans\nWith help of the merged dataframe the scans can now easily displayed.","metadata":{}},{"cell_type":"code","source":"def create_file_path(study_id, series_id, instance_number):\n \"\"\"Get path to the scans which are persisted by the competition-creators\"\"\"\n return os.path.join(path_train_images, str(study_id), str(series_id), f\"{instance_number}.dcm\")\n\ndef load_dicom_image(filepath):\n \"\"\"Load image with help of lib\"\"\"\n dicom = pydicom.dcmread(filepath)\n image = dicom.pixel_array\n return image","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:05.325804Z","iopub.execute_input":"2024-09-15T15:46:05.326097Z","iopub.status.idle":"2024-09-15T15:46:05.331585Z","shell.execute_reply.started":"2024-09-15T15:46:05.326064Z","shell.execute_reply":"2024-09-15T15:46:05.330538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Comparison of planes\nFirst let's compare the three different planes to get an idea how these differ. The introduction gave a first overview of the three planes. By displaying them now, they can be descibred in more detail: The axial plane provides horizontal cross-sectional images, cutting across the body from top to bottom and perpendicular to the spine. On the other hand, the sagittal plane offers vertical cross-sectional views, running parallel to the spine from left to right.\n\nThe scans are either T1 or T2 weighted. T1-weighted scans highlight fat, making the interior of bones appear brighter. Conversely, T2-weighted images emphasize water, which causes the spinal canal to appear brighter.","metadata":{}},{"cell_type":"code","source":"def get_scan_for_plane(df, study_id, series_description):\n \"\"\"Load image for study_id and series_description\"\"\"\n filtered_row = df[(df['series_description'] == series_description) & (df['study_id'] == study_id)].iloc[0]\n file_path = create_file_path(filtered_row['study_id'], filtered_row['series_id'], filtered_row['instance_number'])\n dicom_image = load_dicom_image(file_path)\n return dicom_image\n\nseries_descriptions = ['Axial T2', 'Sagittal T2/STIR', 'Sagittal T1']\n\n# Display in one row\nfig, axes = plt.subplots(1, len(series_descriptions), figsize=(15, 5))\n\nfor i, series in enumerate(series_descriptions):\n image = get_scan_for_plane(train_final_df, 4003253, series)\n axes[i].imshow(image, cmap='gray')\n axes[i].set_title(f'Plane: {series}')\n axes[i].axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:05.332572Z","iopub.execute_input":"2024-09-15T15:46:05.332855Z","iopub.status.idle":"2024-09-15T15:46:06.020226Z","shell.execute_reply.started":"2024-09-15T15:46:05.332823Z","shell.execute_reply":"2024-09-15T15:46:06.019343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Comparison of severities for Left Subarticular Stenosis for Level L4/L5 (Plane: Axial T2)\nSince the proof of concept focuses on the Left Subarticular Stenosis for Level L4/L5, the three severities for this area are now compared to get a better understanding of their differences. In general subarticular stenosis is due to compression of the spinal cord in the subarticular zone. This can be seen in the picture below.\n\n\"Subarticular\n\n*Source: [Miami Neuroscience Center](https://files.miamineurosciencecenter.com/media/filer_public_thumbnails/filer_public/d5/08/d508ae6a-a4f2-4796-be9f-455f8df45fe1/herniation_zones.jpg__1700.0x1308.0_q85_subject_location-850%2C656_subsampling-2.jpg)*\n\nBelow one scan for each severity is shown. The x,y coordinates are used to draw a circle around the condition.","metadata":{}},{"cell_type":"code","source":"lss_l4_l5_df = train_final_df[(train_final_df['condition'] == 'Left Subarticular Stenosis') & (train_final_df['level'] == 'L4/L5')]\nlss_l4_l5_df.head()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:06.021431Z","iopub.execute_input":"2024-09-15T15:46:06.021791Z","iopub.status.idle":"2024-09-15T15:46:06.059117Z","shell.execute_reply.started":"2024-09-15T15:46:06.021747Z","shell.execute_reply":"2024-09-15T15:46:06.058239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_scan_for_severity(df, severity):\n \"\"\"Filter df by severity\"\"\"\n filtered_row = lss_l4_l5_df[lss_l4_l5_df['severity'] == severity].iloc[0]\n \n file_path = create_file_path(filtered_row['study_id'], filtered_row['series_id'], filtered_row['instance_number'])\n dicom_image = load_dicom_image(file_path)\n \n return dicom_image, filtered_row['x'], filtered_row['y']\n\nseverities = [0, 1, 2]\n\n# Display in one row\nfig, axes = plt.subplots(1, len(severities), figsize=(15, 5))\n\nfor i, severity in enumerate(severities):\n image, x, y = get_scan_for_severity(train_final_df, severity)\n \n circle = Circle((x, y), radius=20, color='red', fill=False, linewidth=2)\n axes[i].add_patch(circle)\n \n axes[i].imshow(image, cmap='gray')\n axes[i].set_title(f'Severity: {severity} (Shape: {image.shape})')\n axes[i].axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:06.060086Z","iopub.execute_input":"2024-09-15T15:46:06.060347Z","iopub.status.idle":"2024-09-15T15:46:06.731983Z","shell.execute_reply.started":"2024-09-15T15:46:06.060317Z","shell.execute_reply":"2024-09-15T15:46:06.731095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The three scans are located in different series and point up an important challenge: They aren't standardized. They have different shapes, zoom-factors, sharpness and brightness. The distribution of the shapes can also be seen in the following bar-diagram. This has to be taken into account before the model is trained. This is done in the next chapter.","metadata":{}},{"cell_type":"code","source":"def get_scan_shapes(df):\n shapes = []\n \n for index, row in df.iterrows():\n file_path = create_file_path(row['study_id'], row['series_id'], row['instance_number'])\n dicom_image = load_dicom_image(file_path)\n shapes.append(dicom_image.shape)\n \n return shapes\n\n# Add scan shapes to the original DataFrame\nlss_l4_l5_df['Shape'] = get_scan_shapes(lss_l4_l5_df)\n\n# Count height-width-combinations per severity\nshape_counts_df = (\n lss_l4_l5_df.groupby(['Shape', 'severity'])\n .size()\n .unstack(fill_value=0)\n)\n\n# Create a stacked bar plot\nshape_counts_df.plot(kind='bar', stacked=True, figsize=(14, 8))\n\n# Set plot title and labels\nplt.title('Distribution of Image Shapes for Left Subarticular Stenosis at L4/L5 by Severity')\nplt.xlabel('Image Shape (Height, Width)')\nplt.ylabel('Count')\nplt.xticks(rotation=45, ha=\"right\")\nplt.legend(title='Severity')\nplt.tight_layout()\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:06.733190Z","iopub.execute_input":"2024-09-15T15:46:06.733541Z","iopub.status.idle":"2024-09-15T15:46:42.965986Z","shell.execute_reply.started":"2024-09-15T15:46:06.733499Z","shell.execute_reply":"2024-09-15T15:46:42.965095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preprocess scans\nAs mentioned in the chapter before, the scans for the same condition and vertebral differ between the series. Before they can be used to train the model, they'll be preprocessed with the following steps. The final output is a dataset of preprocessed image patches paired with their corresponding labels which can than be used for the training.\n\n","metadata":{}},{"cell_type":"markdown","source":"### Step 1: Resize\nAll scans are tansformed to have the same shape. Since (1024, 1024) is the highest solution, all scans with lower solutions are filled with a padding to have the same solution.","metadata":{}},{"cell_type":"code","source":"def resize_with_padding(image, resize_size):\n \"\"\"Resizes an image while adding padding around it to achieve the desired dimensions. This ensures that the image remains centered without any distortion.\"\"\"\n original_height, original_width = image.shape\n target_height, target_width = resize_size\n\n # Calc Padding\n pad_left = (target_width - original_width) // 2\n pad_right = target_width - original_width - pad_left\n pad_top = (target_height - original_height) // 2\n pad_bottom = target_height - original_height - pad_top\n\n # Add padding around scan\n padded_image = cv2.copyMakeBorder(image, pad_top, pad_bottom, pad_left, pad_right, cv2.BORDER_CONSTANT, value=0)\n\n return padded_image, pad_left, pad_top","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:42.967099Z","iopub.execute_input":"2024-09-15T15:46:42.967383Z","iopub.status.idle":"2024-09-15T15:46:42.973549Z","shell.execute_reply.started":"2024-09-15T15:46:42.967351Z","shell.execute_reply":"2024-09-15T15:46:42.972577Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Step 2: Extract Patch\nFor each scan exist x and y coordinates which identify the center of the area of the condition. These coordinates are used to extract a patch of the scan. The goal is to reduce the information of the scan to the relevant parts.","metadata":{}},{"cell_type":"code","source":"def extract_patch(image, x, y, patch_size):\n \"\"\"Extracts relevant, quadratic patch of image around (x, y) coordinates\"\"\"\n half_patch = patch_size // 2\n x = int(x)\n y = int(y)\n \n # Ensure that patch isn't outside the image\n start_x = max(x - half_patch, 0)\n end_x = min(x + half_patch, image.shape[1])\n \n start_y = max(y - half_patch, 0)\n end_y = min(y + half_patch, image.shape[0])\n \n # Adjust the patch size to make sure it is square\n if (end_x - start_x) != patch_size:\n if start_x == 0:\n end_x = min(start_x + patch_size, image.shape[1])\n else:\n start_x = max(end_x - patch_size, 0)\n \n if (end_y - start_y) != patch_size:\n if start_y == 0:\n end_y = min(start_y + patch_size, image.shape[0])\n else:\n start_y = max(end_y - patch_size, 0)\n\n patch = image[start_y:end_y, start_x:end_x]\n return patch","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:42.974582Z","iopub.execute_input":"2024-09-15T15:46:42.974872Z","iopub.status.idle":"2024-09-15T15:46:42.987844Z","shell.execute_reply.started":"2024-09-15T15:46:42.974841Z","shell.execute_reply":"2024-09-15T15:46:42.986940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Step 3: Normalize Patch\nThe patches are normalized. Normalization of image data scales pixel values to a consistent range between 0-1. This improves the model performance and stability.","metadata":{}},{"cell_type":"code","source":"def normalize_patch(patch): \n \"\"\"Normalizes the pixel values of an image patch to a [0, 1] range and adds an additional dimension, preparing it for further processing\"\"\"\n normalized_patch = patch / 255.0\n normalized_patch = np.expand_dims(normalized_patch, axis=-1)\n return normalized_patch","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:42.988785Z","iopub.execute_input":"2024-09-15T15:46:42.989074Z","iopub.status.idle":"2024-09-15T15:46:42.996385Z","shell.execute_reply.started":"2024-09-15T15:46:42.989016Z","shell.execute_reply":"2024-09-15T15:46:42.995611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n### Step 4: Add second dimension with circle\nAn additional data channel is created that encodes a filled circular region corresponding to the area of interest. This channel is combined with the normalized image data, enhancing the representation by explicitly marking the focal point within the image. The CNN had better results with help of this circle.\n\n","metadata":{}},{"cell_type":"code","source":"def add_circle_channel(image, x, y, radius):\n \"\"\"Adds a new channel to the scan, representing a filled circle at a specific location. This helps to visually encode the position of interest within the scan.\"\"\"\n circle_channel = np.zeros_like(image, dtype=np.float32)\n cv2.circle(circle_channel, (x, y), radius, 1, thickness=-1)\n combined_image = np.stack([image, circle_channel], axis=-1)\n \n return combined_image","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:42.997349Z","iopub.execute_input":"2024-09-15T15:46:42.997613Z","iopub.status.idle":"2024-09-15T15:46:43.006056Z","shell.execute_reply.started":"2024-09-15T15:46:42.997583Z","shell.execute_reply":"2024-09-15T15:46:43.005239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Combine all steps","metadata":{}},{"cell_type":"code","source":"def display_scans_in_row(scans_with_titles_and_coords):\n \"\"\"Display all modification stages of a scan in a row. The coordinates are always visualized with a circle.\"\"\"\n fig, axes = plt.subplots(1, len(scans_with_titles_and_coords), figsize=(20, 5))\n \n for i, (image, title, x, y) in enumerate(scans_with_titles_and_coords):\n x = int(x)\n y = int(y)\n circle = Circle((x, y), radius=image.shape[0]/100*(1000/image.shape[0]), color='red', fill=False, linewidth=2)\n axes[i].add_patch(circle)\n \n axes[i].imshow(image, cmap='gray')\n axes[i].set_title(title)\n axes[i].axis('off')\n \n plt.tight_layout()\n plt.show()\n \ndef preprocess_scans(df, n, resize_size, patch_size, display_scans):\n \"\"\"Combines all functions from before\"\"\"\n patches = []\n labels = []\n displayed_first = False # Ensure to display only the first scan\n \n for index, row in tqdm(df[:n].iterrows()): \n filepath = create_file_path(row['study_id'], row['series_id'], row['instance_number'])\n \n if os.path.exists(filepath):\n image = load_dicom_image(filepath)\n \n # Step 1: Resize the image with padding \n resized_image, pad_left, pad_top = resize_with_padding(image, resize_size=resize_size)\n \n # Update x and y coordinates based on resized image and padding\n x_resized = int(row['x'] + pad_left)\n y_resized = int(row['y'] + pad_top)\n \n # Step 2: Extract relevant patch\n patch = extract_patch(resized_image, x_resized, y_resized, patch_size)\n \n # Step 3: Normalize \n normalized_patch = normalize_patch(patch)\n normalized_patch_with_circle = add_circle_channel(normalized_patch, patch_size//2, patch_size//2, radius=int(normalized_patch.shape[0]/100*(1000/normalized_patch.shape[0])))\n \n # Displaying too many scans wil crash the notebook and make restart impossible\n if display_scans and not displayed_first:\n images_with_titles_and_coords = [\n (image, f\"Original {image.shape}\", row['x'], row['y']),\n (resized_image, f\"Resized {resized_image.shape}\", x_resized, y_resized),\n (patch, f\"Patch {patch.shape}\", patch_size/2, patch_size/2),\n (normalized_patch, f\"Normalized Patch {normalized_patch.shape}\", patch_size/2, patch_size/2)\n ]\n display_scans_in_row(images_with_titles_and_coords)\n displayed_first = True\n\n patches.append(normalized_patch_with_circle)\n labels.append(row['severity'])\n \n return np.array(patches).reshape(-1, patch_size, patch_size, 2), np.array(labels)","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:43.007213Z","iopub.execute_input":"2024-09-15T15:46:43.007583Z","iopub.status.idle":"2024-09-15T15:46:43.024624Z","shell.execute_reply.started":"2024-09-15T15:46:43.007543Z","shell.execute_reply":"2024-09-15T15:46:43.023723Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Apply defined functions\nBelow all scans for Left Subarticular Stenosis for level L4/L5 are preprocessed. To get a better understanding of the preprocession, all four processing-stages of the first scan are displayed. \nThe patch_size of 256 is an important choice: Lower patch_sizes lead to a big zom-in and by this to lower sharpness. The result was a strong overfitting while fitting the model. Higher patch_sizes include too much complexity for the CNN and lead to a black padding for smaller scans. \n\nSince most of the data has a shape >= (256, 256) (see 'Distribution of Scan Shapes for Left Subarticular Stenosis at L4/L5') and fitting the model doesn't overfit too fast, this is used as patch_size. ","metadata":{}},{"cell_type":"code","source":"resize_size = (1024, 1024) # Size of biggest scan\npatch_size = 256\ninput_shape = (patch_size, patch_size, 2) # CNN\n\nlss_l4_l5_df = train_final_df[(train_final_df['condition'] == 'Left Subarticular Stenosis') & (train_final_df['level'] == 'L4/L5')]\npreprocessed_scans, labels = preprocess_scans(df=lss_l4_l5_df, n=2500, resize_size=resize_size, patch_size = patch_size, display_scans=True)","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:46:43.025642Z","iopub.execute_input":"2024-09-15T15:46:43.025923Z","iopub.status.idle":"2024-09-15T15:47:11.667518Z","shell.execute_reply.started":"2024-09-15T15:46:43.025893Z","shell.execute_reply":"2024-09-15T15:47:11.666506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Model Implementation","metadata":{}},{"cell_type":"markdown","source":"After preparing the training data is done, the model implementation can start now. In this chapter the model is build, trained and evaluated. After inspecting the first evaluation results, the model is finetuned using Hyperparameter-Tuning. The first buildup of the model already includes some improvements which had been made while developing: a Spatial Attention Mechanism and weights for the several severities.","metadata":{}},{"cell_type":"markdown","source":"## Build model","metadata":{}},{"cell_type":"markdown","source":"### Libraries","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, Layer, Multiply\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\nfrom tensorflow.keras import Input\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.utils.class_weight import compute_class_weight\nimport keras_tuner\nfrom tensorflow.keras.callbacks import EarlyStopping","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:47:11.668661Z","iopub.execute_input":"2024-09-15T15:47:11.668970Z","iopub.status.idle":"2024-09-15T15:47:24.903372Z","shell.execute_reply.started":"2024-09-15T15:47:11.668936Z","shell.execute_reply":"2024-09-15T15:47:24.902546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Architectur\nThe following model architecture is designed to classify the severity of left subarticular stenosis in the L4/L5 vertebral. The architecture employs a Convolutional Neural Network (CNN) integrated with a spatial attention mechanism, tailored to address the challenges of localizing and analyzing the degenerative condition which is centered in all scans.\n\n### Input Layer\nThe input layer is designed to accommodate the scans with defined dimensions (input_shape).\n\n### Convolutional Layers\nTo extract relevant features from the scans, the model integrates multiple 2D convolutional layers with progressively increasing filter sizes (32, 64, 128). These layers identify and capture features as edges, textures and patterns, which are indicative of the severity of left subarticular stenosis in the L4/L5 region. The use of ReLU activation functions enhances the model's capability to learn complex non-linear relationships within the data.\n\n### MaxPooling Layers\nMaxPooling layers are employed after each convolutional layer to reduce the spatial dimensions of the feature maps, thereby focusing on the most salient features and reducing computational complexity.\n\n### Spatial Attention Mechanism\nA key feature of this model is the incorporation of a spatial attention mechanism. This component is designed to enhance the model's focus on the most relevant regions of the scans, particularly the center of each scan (where the conditions are placed). The spatial attention layer generates an attention map that modulates the input feature maps, highlighting the critical regions and allowing the model to concentrate on the relevant areas with greater precision.\n\n### Dense and Dropout Layers\nAfter the convolutional and attention layers, the model employs a Flatten layer to convert the 2D feature maps into a 1D vector. This vector is then passed through a Dense layer with 128 units, where complex patterns are further learned and refined. A Dropout layer with a rate of 0.5 is included to prevent overfitting. This is required since the scans have a wide variability and the number of samples are limited due to focusing on left subarticular stenosis.\n\n### Output Layer\nThe final Dense layer uses the softmax activation function to output a probability distribution across the three severity levels (normal_mild, moderate, severe). This allows the model to provide a probabilistic assessment of the severity of left subarticular stenosis in the L4/L5 vertebrae for each input image as asked by the competition creators.\n\n### Compilation\nThe model is compiled using the Adam optimizer. The loss function sparse_categorical_crossentropy is used because it's appropriate for multi-class classification tasks, where the target labels are represented as integers. Since the classification has to be done for the integer labeled severity class, this is a good choice. The accuracy metric is used to monitor the model's performance during training and evaluation phases.","metadata":{}},{"cell_type":"code","source":"class SpatialAttention(Layer):\n def __init__(self, kernel_size):\n super(SpatialAttention, self).__init__()\n self.conv = Conv2D(filters=1, kernel_size=kernel_size, strides=1, padding='same', activation='sigmoid')\n\n def call(self, inputs):\n attention_map = self.conv(inputs)\n return Multiply()([inputs, attention_map])\n\nmodel = tf.keras.Sequential([\n Input(shape=input_shape),\n Conv2D(32, (3, 3), activation='relu'),\n MaxPooling2D((2, 2)),\n Conv2D(64, (3, 3), activation='relu'),\n SpatialAttention(kernel_size=1), # Small size for specialization on middle\n MaxPooling2D((2, 2)),\n Conv2D(128, (3, 3), activation='relu'),\n MaxPooling2D((2, 2)),\n Flatten(),\n Dense(128, activation='relu'),\n Dropout(0.5),\n Dense(3, activation='softmax') # 0, 1, 2\n])\n\nmodel.compile(optimizer='adam', \n loss='sparse_categorical_crossentropy', \n metrics=['accuracy'])\n\nmodel.summary()","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:47:24.908011Z","iopub.execute_input":"2024-09-15T15:47:24.908620Z","iopub.status.idle":"2024-09-15T15:47:25.841727Z","shell.execute_reply.started":"2024-09-15T15:47:24.908584Z","shell.execute_reply":"2024-09-15T15:47:25.840825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train model\nAs known from the first observation of the distribution of severities, they aren't even distributed. A small reminder: Focussing on Left Subarticular Stenosis in L4/L5 brings the advantage of the most even distribution in the dataset. But there are still a lot more cases of severity 0 than of the other severities (as shown below). This challenge can be solved in two ways: using weights or augmenting data by transforming existing data. This study uses the first way.","metadata":{}},{"cell_type":"code","source":"Counter(labels)","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:47:25.842807Z","iopub.execute_input":"2024-09-15T15:47:25.843128Z","iopub.status.idle":"2024-09-15T15:47:25.849972Z","shell.execute_reply.started":"2024-09-15T15:47:25.843092Z","shell.execute_reply":"2024-09-15T15:47:25.849006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, X_val, y_train, y_val = train_test_split(preprocessed_scans, labels, test_size=0.2, random_state=42)\n\nclass_weights = compute_class_weight(class_weight='balanced', classes=np.unique(y_train), y=y_train)\n\nclass_weight_dict = {0: class_weights[0], 1: class_weights[1], 2: class_weights[2]}\nclass_weight_dict","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:47:25.851215Z","iopub.execute_input":"2024-09-15T15:47:25.851609Z","iopub.status.idle":"2024-09-15T15:47:26.446277Z","shell.execute_reply.started":"2024-09-15T15:47:25.851571Z","shell.execute_reply":"2024-09-15T15:47:26.445280Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The model demonstrates good learning behavior with increasing training and validation accuracy. But after a few epochs there emerge signs of overfitting, particularly as seen in the increasing validation loss. That's why early_stopping stopped the model fitting. One way to solve this issue is to adjust the regularization and learning rate by Hyperparameter Tuning. This will be done in the next chapter. But first let's evaluate the model results with help of the validation data.","metadata":{}},{"cell_type":"code","source":"early_stopping = EarlyStopping(monitor='val_loss', \n patience=4, \n min_delta=0.001, \n restore_best_weights=True)\n\nhistory = model.fit(X_train, y_train, \n epochs=20, \n validation_data=(X_val, y_val), \n batch_size=32,\n class_weight=class_weight_dict,\n callbacks=[early_stopping])","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:47:26.447439Z","iopub.execute_input":"2024-09-15T15:47:26.447756Z","iopub.status.idle":"2024-09-15T15:48:14.101730Z","shell.execute_reply.started":"2024-09-15T15:47:26.447721Z","shell.execute_reply":"2024-09-15T15:48:14.100767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluate model","metadata":{}},{"cell_type":"markdown","source":"The classification results indicate an overall accuracy of 65%, with varying performance across different classes. The model performs well in identifying Normal/Mild and Severe cases, as reflected by the higher F1-scores of 0.76 and 0.70, respectively. However, it struggles with the moderate cases, which have a lower F1-score of 0.43, suggesting difficulties in distinguishing this class from others. The macro and weighted averages reflect this imbalance, indicating that while the model is relatively effective in identifying extreme cases, its ability to accurately classify moderate cases remains limited.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import classification_report\n\ndef evaluate_model(model, X_val, y_val, target_names=['Normal/Mild', 'Moderate', 'Severe']):\n y_pred = model.predict(X_val)\n y_pred_classes = np.argmax(y_pred, axis=1)\n report = classification_report(y_val, y_pred_classes, target_names=target_names)\n print(report)\n\nevaluate_model(model, X_val, y_val)","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:48:14.103222Z","iopub.execute_input":"2024-09-15T15:48:14.103897Z","iopub.status.idle":"2024-09-15T15:48:15.440929Z","shell.execute_reply.started":"2024-09-15T15:48:14.103852Z","shell.execute_reply":"2024-09-15T15:48:15.439952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Hyperparameter-Tuning\nThis chapter tries to solve the overfitting issue by adjusting the parameter of the model. These parameters are:\n- number of filters and kernel sizes for the convolutional layers based on values sampled by the hyperparameter tuner\n- kernel size for the SpatialAttention layer\n- The number of units in the dense layer and the dropout rate will be adjusted, allowing the model to manage its complexity and incorporate regularization to prevent overfitting\n- The learning rate will be sampled on a logarithmic scale (sampling='LOG'), enabling the tuner to find the optimal learning rate\n\nRandomSearch is used as tuner because it's a good compromiss between performance, flexibility and scalability.","metadata":{}},{"cell_type":"code","source":"def build_model(hp):\n model = tf.keras.Sequential()\n\n # Input Layer\n model.add(Input(shape=input_shape))\n\n # Convolutional Layers\n model.add(Conv2D(filters=hp.Int('conv_1_filters', min_value=32, max_value=128, step=16),\n kernel_size=hp.Choice('conv_1_kernel', values=[3, 5]),\n activation='relu'))\n model.add(MaxPooling2D(pool_size=(2, 2)))\n model.add(Conv2D(filters=hp.Int('conv_2_filters', min_value=64, max_value=256, step=32),\n kernel_size=hp.Choice('conv_2_kernel', values=[3, 5]),\n activation='relu'))\n model.add(SpatialAttention(kernel_size=hp.Choice('spatial_kernel', values=[1, 3, 5])))\n model.add(MaxPooling2D(pool_size=(2, 2)))\n\n model.add(Flatten())\n\n # Fully Connected Layers\n model.add(Dense(units=hp.Int('dense_units', min_value=64, max_value=256, step=32),\n activation='relu'))\n model.add(Dropout(rate=hp.Float('dropout', min_value=0.2, max_value=0.5, step=0.1)))\n\n # Output Layer\n model.add(Dense(3, activation='softmax'))\n\n # Compile Model\n model.compile(optimizer=tf.keras.optimizers.Adam(\n hp.Float('learning_rate', min_value=1e-4, max_value=1e-2, sampling='LOG')),\n loss='sparse_categorical_crossentropy',\n metrics=['accuracy'])\n\n return model","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:48:15.445654Z","iopub.execute_input":"2024-09-15T15:48:15.446047Z","iopub.status.idle":"2024-09-15T15:48:15.456530Z","shell.execute_reply.started":"2024-09-15T15:48:15.445999Z","shell.execute_reply":"2024-09-15T15:48:15.455546Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tuner = keras_tuner.RandomSearch(\n build_model,\n objective='val_accuracy',\n max_trials=10, # Amount of tested models\n executions_per_trial=1, # Amount of runs per tested model\n directory='hyperparam_tuning',\n project_name='1'\n)","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:48:15.458079Z","iopub.execute_input":"2024-09-15T15:48:15.458443Z","iopub.status.idle":"2024-09-15T15:48:15.627732Z","shell.execute_reply.started":"2024-09-15T15:48:15.458400Z","shell.execute_reply":"2024-09-15T15:48:15.626750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tuner.search(X_train, y_train, \n epochs=10, \n validation_data=(X_val, y_val),\n class_weight=class_weight_dict,\n batch_size=32)","metadata":{"execution":{"iopub.status.busy":"2024-09-15T15:48:15.628782Z","iopub.execute_input":"2024-09-15T15:48:15.629169Z","iopub.status.idle":"2024-09-15T16:05:54.166358Z","shell.execute_reply.started":"2024-09-15T15:48:15.629133Z","shell.execute_reply":"2024-09-15T16:05:54.165439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_hps = tuner.get_best_hyperparameters(num_trials=1)[0]\ntuned_model = tuner.hypermodel.build(best_hps)\ntuned_history = tuned_model.fit(X_train, y_train,\n epochs=10, \n validation_data=(X_val, y_val),\n batch_size=32,\n class_weight=class_weight_dict,\n callbacks=[early_stopping])\nevaluate_model(tuned_model, X_val, y_val)","metadata":{"execution":{"iopub.status.busy":"2024-09-15T16:05:54.167633Z","iopub.execute_input":"2024-09-15T16:05:54.168412Z","iopub.status.idle":"2024-09-15T16:06:10.677733Z","shell.execute_reply.started":"2024-09-15T16:05:54.168363Z","shell.execute_reply":"2024-09-15T16:06:10.675962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Unfortunately, the evaluation could not be carried out due to OOM. It could be completed previously, but did not lead to better results than the initial model.","metadata":{}},{"cell_type":"markdown","source":"# 4. Summary of Results and Future Work\n\n## Summary of Results\nIn conclusion, this study successfully demonstrated the feasibility of using a convolutional neural network (CNN) to classify the severity of left subarticular stenosis between the L4/L5 vertebral levels using MRI scans. By focusing on this specific condition with a balanced dataset, the model achieved promising results in distinguishing between normal/mild, moderate, and severe cases. However, some limitations were observed, particularly for the classification of moderate cases.\n\nIt is important to note that this study serves as a proof of concept and represents an initial exploration of neural networks. Therefore, the focused analysis of the specific subset, which will be critically examined in the following sections, is fundamentally justified.\n\n## Learnings\nDuring the course of this study, several important learnings emerged. One key insight was the challenge posed by the preprocessing step, where MRI scans were centered around the region of interest, specifically the affected vertebral area. While this approach helped in aligning the data, it introduced significant limitations for future classifications. Namely, this method relies on the availability of precise x and y coordinates for the vertebrae, which may not be present in new, unprocessed scans. Moreover, it necessitates an additional preprocessing step for new scans, potentially limiting the model's practicality in real-world applications where raw, unprocessed scans need to be classified immediately.\n\nAnother learning involved the limitations of the CNN architecture employed in this study. The narrow focus on a single condition within a specific spinal region significantly reduced the dataset, which, combined with the highly standardized image preprocessing, may have contributed to the model's tendency to overfit. The lack of sufficient data variability likely hindered the model's ability to generalize beyond the training set. As a result, overfitting became a notable issue. In response, an alternative neural network architecture will be discussed in the following section, aimed at addressing these concerns by increasing data variability and reducing overfitting tendencies.\n \n ## Future Work\nBefore the notebook can be submitted for participation in the competition, several modifications need to be implemented.\n\nFirst, addressing class imbalance through data augmentation is necessary, especially to compensate for the scarcity of severe cases. However, for certain regions, fewer than 10 cases classified as \"severe\" are available. How to handle this extreme imbalance remains a challenge.\n\nA potential solution is to adopt a holistic approach, which would require a different CNN architecture. Instead of training a separate CNN for each disease in a specific region, a unified CNN could be designed to take a patient's complete set of scans and labels as input. This model would feature a shared backbone responsible for feature extraction across all scans. Transfer learning could be employed here by using a pretrained CNN, such as ResNet or DenseNet, as the backbone for feature extraction. The final layers would then be adapted for the individual classification of each condition across the vertebrae. This architecture would culminate in a multi-output layer, with one output for each classification task.\n\nLastly, a revised strategy for hyperparameter tuning is recommended, such as a more targeted optimization approach using Grid Search, which could improve model performance and reduce overfitting.","metadata":{}}]}