\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1820636%2F7d69bbb4c4a112042c288b1780157350%2Fpatches.png?generation=1728463071124063&alt=media\")
Similar to other competitors, we implemented a 2-stage system composed of three core models (plus an additional one for pretraining):
\nFor detailed training specifics, please refer to the accompanying code.
\nWe initially experimented with bounding box detectors and segmentation models, but keypoints provided the best results and flexibility. The model outputs 6 pairs of XY coordinates (12 total), where 5 pairs are used for sagittal images, and 1 pair is designated for axial images. We incorporated the corrected dataset labels shared by @brendanartley. This allowed us to use tilted crops, although the performance improvement was minimal.
\nThe loss function was based on Euclidean distance, with masking applied to avoid propagating axial keypoint errors in sagittal images and vice versa. We also introduced position encoding to the backbone features before pooling to enhance the model’s performance.
\nFor axial images, we developed a straightforward image model that predicts the corresponding level. Some metadata was concatenated with the backbone features to enhance the model's performance.
\nWe observed occasional prediction inconsistencies (e.g., L5-S1 predicted next to L1-L2 within the same series). To address this, we redefined the task as a regression problem, predicting values between 0.0 and 1.0 at 0.25 intervals (e.g., 0.0 for L1-L2, 0.25 for L2-L3, etc.), using Mean Squared Error (MSE) as the loss function. This approach penalized predictions further from the true value more heavily.
\nLeveraging anatomical symmetries, we trained one model with study-level-side inputs.
\nDetails are explained further below.
\nWe used the keypoints and levels models to create patches, with each axial patch corresponding to one or more spinal levels. Since the levels model is a regressor, we applied a threshold with a tolerance range to capture additional context. For example, patches corresponding to L2-L3 (0.25 output) could span from 0.10 to 0.40.
\nTo ensure consistency, we maintained uniform pixel spacing across each plane. We used 96x96 patches. Sagittal patches were offset at the top and bottom to prevent level leakage.
\nSome examples:
\n
Before training the sequence model, we pre-trained a patch model using individual patches with the same loss function (defined later). This step significantly improved the convergence of the sequence model.
\n2D model + encoder-decoder transformer.
\nThe architecture combined a 2D image model with an encoder-decoder transformer. The best backbones we found were regnetz_b16.ra3_in1k and hgnet_tiny.ssld_in1k.
Metadata (modified XYZ world coordinates) was fed into the encoder, while image features were passed to the decoder. We added three learnable vectors to the decoder, each representing one disease classification (analogous to CLS tokens). Various pooling strategies were tested, including max/mean pooling and attention pooling.
\nThe model was designed to be flexible, allowing it to process either an entire study or individual level-sides. This adaptability enabled us to train the model at the level-side while evaluating it on the whole study.
\nWhat to do with spinal?
\nSince spinal stenosis affects the center rather than specific sides, we duplicated the label for both sides during training. At inference, we aggregated the learnable vectors from both sides (those added in the decoder) before passing them through the classification head. We tested multiple aggregation methods and ultimately chose max pooling.
\nWe used the competition-specific loss function implemented in PyTorch. Although we explored several variations, such as a focal version of the competition loss, none yielded significantly better results.
\n| description | \nbackbone | \nlocal avg cv | \npublic | \nprivate | \n
|---|---|---|---|---|
| best public: ensemble 4 (dropped one) TTA 20 | \nregnet | \n0.4027 | \n0.3452 | \n0.4121 | \n
| best local: ensemble 5 TTA 40 | \nhgnet | \n0.3883 | \n0.3612 | \n0.4215 | \n
| best private: ensemble 5 TTA 16 | \nregnet | \n0.4076 | \n0.3472 | \n0.4118 | \n
The two selected submissions were the best local and the best in the public leaderboard. The best in the public leaderboard was the 3rd best in the private leaderboard.
\nhttps://github.com/claverru/RSNA-lumbar
", "messages": [ { "id": "3012689", "postDate": "10/09/2024 09:05:00", "content": "Similar to other competitors, we implemented a 2-stage system composed of three core models (plus an additional one for pretraining):
\nFor detailed training specifics, please refer to the accompanying code.
\nWe initially experimented with bounding box detectors and segmentation models, but keypoints provided the best results and flexibility. The model outputs 6 pairs of XY coordinates (12 total), where 5 pairs are used for sagittal images, and 1 pair is designated for axial images. We incorporated the corrected dataset labels shared by @brendanartley. This allowed us to use tilted crops, although the performance improvement was minimal.
\nThe loss function was based on Euclidean distance, with masking applied to avoid propagating axial keypoint errors in sagittal images and vice versa. We also introduced position encoding to the backbone features before pooling to enhance the model’s performance.
\nFor axial images, we developed a straightforward image model that predicts the corresponding level. Some metadata was concatenated with the backbone features to enhance the model's performance.
\nWe observed occasional prediction inconsistencies (e.g., L5-S1 predicted next to L1-L2 within the same series). To address this, we redefined the task as a regression problem, predicting values between 0.0 and 1.0 at 0.25 intervals (e.g., 0.0 for L1-L2, 0.25 for L2-L3, etc.), using Mean Squared Error (MSE) as the loss function. This approach penalized predictions further from the true value more heavily.
\nLeveraging anatomical symmetries, we trained one model with study-level-side inputs.
\nDetails are explained further below.
\nWe used the keypoints and levels models to create patches, with each axial patch corresponding to one or more spinal levels. Since the levels model is a regressor, we applied a threshold with a tolerance range to capture additional context. For example, patches corresponding to L2-L3 (0.25 output) could span from 0.10 to 0.40.
\nTo ensure consistency, we maintained uniform pixel spacing across each plane. We used 96x96 patches. Sagittal patches were offset at the top and bottom to prevent level leakage.
\nSome examples:
\n
Before training the sequence model, we pre-trained a patch model using individual patches with the same loss function (defined later). This step significantly improved the convergence of the sequence model.
\n2D model + encoder-decoder transformer.
\nThe architecture combined a 2D image model with an encoder-decoder transformer. The best backbones we found were regnetz_b16.ra3_in1k and hgnet_tiny.ssld_in1k.
Metadata (modified XYZ world coordinates) was fed into the encoder, while image features were passed to the decoder. We added three learnable vectors to the decoder, each representing one disease classification (analogous to CLS tokens). Various pooling strategies were tested, including max/mean pooling and attention pooling.
\nThe model was designed to be flexible, allowing it to process either an entire study or individual level-sides. This adaptability enabled us to train the model at the level-side while evaluating it on the whole study.
\nWhat to do with spinal?
\nSince spinal stenosis affects the center rather than specific sides, we duplicated the label for both sides during training. At inference, we aggregated the learnable vectors from both sides (those added in the decoder) before passing them through the classification head. We tested multiple aggregation methods and ultimately chose max pooling.
\nWe used the competition-specific loss function implemented in PyTorch. Although we explored several variations, such as a focal version of the competition loss, none yielded significantly better results.
\n| description | \nbackbone | \nlocal avg cv | \npublic | \nprivate | \n
|---|---|---|---|---|
| best public: ensemble 4 (dropped one) TTA 20 | \nregnet | \n0.4027 | \n0.3452 | \n0.4121 | \n
| best local: ensemble 5 TTA 40 | \nhgnet | \n0.3883 | \n0.3612 | \n0.4215 | \n
| best private: ensemble 5 TTA 16 | \nregnet | \n0.4076 | \n0.3472 | \n0.4118 | \n
The two selected submissions were the best local and the best in the public leaderboard. The best in the public leaderboard was the 3rd best in the private leaderboard.
\nhttps://github.com/claverru/RSNA-lumbar
", "rawMarkdown": "# 2 stage system\n\n## Summary\n\nSimilar to other competitors, we implemented a 2-stage system composed of three core models (plus an additional one for pretraining):\n\n- Keypoints: Localizing intervertebral spots.\n- Levels: Classifying axial images into spinal levels.\n- Pretraining (Patch): Classifying individual patches.\n- Sequence: A 2D model combined with a Transformer for final predictions.\n\nFor detailed training specifics, please refer to the accompanying code.\n\n## Models\n\n### 1. Keypoints\n\nWe initially experimented with bounding box detectors and segmentation models, but keypoints provided the best results and flexibility. The model outputs 6 pairs of XY coordinates (12 total), where 5 pairs are used for sagittal images, and 1 pair is designated for axial images. We incorporated the [corrected dataset labels](https://www.kaggle.com/code/brendanartley/lumbar-coordinate-dataset-code) shared by @brendanartley. This allowed us to use tilted crops, although the performance improvement was minimal.\n\nThe loss function was based on Euclidean distance, with masking applied to avoid propagating axial keypoint errors in sagittal images and vice versa. We also introduced position encoding to the backbone features before pooling to enhance the model’s performance.\n\n### 2. Levels\n\nFor axial images, we developed a straightforward image model that predicts the corresponding level. Some metadata was concatenated with the backbone features to enhance the model's performance.\n\nWe observed occasional prediction inconsistencies (e.g., L5-S1 predicted next to L1-L2 within the same series). To address this, we redefined the task as a regression problem, predicting values between 0.0 and 1.0 at 0.25 intervals (e.g., 0.0 for L1-L2, 0.25 for L2-L3, etc.), using Mean Squared Error (MSE) as the loss function. This approach penalized predictions further from the true value more heavily.\n\n### 3. Sequence\n\nLeveraging anatomical symmetries, we trained one model with study-level-side inputs. \n- Input: For example, one row would be all patches in the L1-L2 right side, and another row all patches in the L4-L5 left side.\n- Output: neural_foraminal_narrowing, subarticular_stenosis, spinal_canal_stenosis.\n\nDetails are explained further below.\n\n#### Input\n\nWe used the keypoints and levels models to create patches, with each axial patch corresponding to one or more spinal levels. Since the levels model is a regressor, we applied a threshold with a tolerance range to capture additional context. For example, patches corresponding to L2-L3 (0.25 output) could span from 0.10 to 0.40.\n\nTo ensure consistency, we maintained uniform pixel spacing across each plane. We used 96x96 patches. Sagittal patches were offset at the top and bottom to prevent level leakage.\n\nSome examples:\n\n\n#### Pretraining: Patch model\n\nBefore training the sequence model, we pre-trained a patch model using individual patches with the same loss function (defined later). This step significantly improved the convergence of the sequence model.\n\n#### Architecture\n\n2D model + encoder-decoder transformer. \n\nThe architecture combined a 2D image model with an encoder-decoder transformer. The best backbones we found were `regnetz_b16.ra3_in1k` and `hgnet_tiny.ssld_in1k`.\n\nMetadata (modified XYZ world coordinates) was fed into the encoder, while image features were passed to the decoder. We added three learnable vectors to the decoder, each representing one disease classification (analogous to CLS tokens). Various pooling strategies were tested, including max/mean pooling and attention pooling.\n\n#### Reshapable model\n\nThe model was designed to be flexible, allowing it to process either an entire study or individual level-sides. This adaptability enabled us to train the model at the level-side while evaluating it on the whole study.\n\nWhat to do with spinal?\n\nSince spinal stenosis affects the center rather than specific sides, we duplicated the label for both sides during training. At inference, we aggregated the learnable vectors from both sides (those added in the decoder) before passing them through the classification head. We tested multiple aggregation methods and ultimately chose max pooling.\n\n#### Loss\n\nWe used the competition-specific loss function implemented in PyTorch. Although we explored several variations, such as a focal version of the competition loss, none yielded significantly better results.\n\n## Further information\n\n| description | backbone | local avg cv | public | private |\n| --- | --- | --- | --- | --- |\n| best public: ensemble 4 (dropped one) TTA 20 | regnet | 0.4027 | 0.3452 | 0.4121 | \n| best local: ensemble 5 TTA 40 | hgnet | 0.3883 | 0.3612 | 0.4215 | \n| best private: ensemble 5 TTA 16 | regnet | 0.4076 | 0.3472 | 0.4118 | \n\n\nThe two selected submissions were the best local and the best in the public leaderboard. The best in the public leaderboard was the 3rd best in the private leaderboard.\n\n\n## Code\n\nhttps://github.com/claverru/RSNA-lumbar", "votes": null }, { "id": "3019312", "postDate": "10/16/2024 12:36:56", "content": "Congrats on the gold and becoming Competitions Master!!
", "rawMarkdown": "Congrats on the gold and becoming Competitions Master!!", "votes": null }, { "id": "3019916", "postDate": "10/17/2024 02:46:06", "content": "Congratulations on this impressive development! great job
", "rawMarkdown": "Congratulations on this impressive development! great job", "votes": null } ], "comments": [ { "id": 3019312, "author_name": "bacterio", "author_url": "", "post_date": "10/16/2024 12:36:56", "content": "Congrats on the gold and becoming Competitions Master!!
", "votes": null, "replies": [] }, { "id": 3019916, "author_name": "darwinberrio", "author_url": "", "post_date": "10/17/2024 02:46:06", "content": "Congratulations on this impressive development! great job
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3012689": "# 2 stage system\n\n## Summary\n\nSimilar to other competitors, we implemented a 2-stage system composed of three core models (plus an additional one for pretraining):\n\n- Keypoints: Localizing intervertebral spots.\n- Levels: Classifying axial images into spinal levels.\n- Pretraining (Patch): Classifying individual patches.\n- Sequence: A 2D model combined with a Transformer for final predictions.\n\nFor detailed training specifics, please refer to the accompanying code.\n\n## Models\n\n### 1. Keypoints\n\nWe initially experimented with bounding box detectors and segmentation models, but keypoints provided the best results and flexibility. The model outputs 6 pairs of XY coordinates (12 total), where 5 pairs are used for sagittal images, and 1 pair is designated for axial images. We incorporated the [corrected dataset labels](https://www.kaggle.com/code/brendanartley/lumbar-coordinate-dataset-code) shared by @brendanartley. This allowed us to use tilted crops, although the performance improvement was minimal.\n\nThe loss function was based on Euclidean distance, with masking applied to avoid propagating axial keypoint errors in sagittal images and vice versa. We also introduced position encoding to the backbone features before pooling to enhance the model’s performance.\n\n### 2. Levels\n\nFor axial images, we developed a straightforward image model that predicts the corresponding level. Some metadata was concatenated with the backbone features to enhance the model's performance.\n\nWe observed occasional prediction inconsistencies (e.g., L5-S1 predicted next to L1-L2 within the same series). To address this, we redefined the task as a regression problem, predicting values between 0.0 and 1.0 at 0.25 intervals (e.g., 0.0 for L1-L2, 0.25 for L2-L3, etc.), using Mean Squared Error (MSE) as the loss function. This approach penalized predictions further from the true value more heavily.\n\n### 3. Sequence\n\nLeveraging anatomical symmetries, we trained one model with study-level-side inputs. \n- Input: For example, one row would be all patches in the L1-L2 right side, and another row all patches in the L4-L5 left side.\n- Output: neural_foraminal_narrowing, subarticular_stenosis, spinal_canal_stenosis.\n\nDetails are explained further below.\n\n#### Input\n\nWe used the keypoints and levels models to create patches, with each axial patch corresponding to one or more spinal levels. Since the levels model is a regressor, we applied a threshold with a tolerance range to capture additional context. For example, patches corresponding to L2-L3 (0.25 output) could span from 0.10 to 0.40.\n\nTo ensure consistency, we maintained uniform pixel spacing across each plane. We used 96x96 patches. Sagittal patches were offset at the top and bottom to prevent level leakage.\n\nSome examples:\n\n\n#### Pretraining: Patch model\n\nBefore training the sequence model, we pre-trained a patch model using individual patches with the same loss function (defined later). This step significantly improved the convergence of the sequence model.\n\n#### Architecture\n\n2D model + encoder-decoder transformer. \n\nThe architecture combined a 2D image model with an encoder-decoder transformer. The best backbones we found were `regnetz_b16.ra3_in1k` and `hgnet_tiny.ssld_in1k`.\n\nMetadata (modified XYZ world coordinates) was fed into the encoder, while image features were passed to the decoder. We added three learnable vectors to the decoder, each representing one disease classification (analogous to CLS tokens). Various pooling strategies were tested, including max/mean pooling and attention pooling.\n\n#### Reshapable model\n\nThe model was designed to be flexible, allowing it to process either an entire study or individual level-sides. This adaptability enabled us to train the model at the level-side while evaluating it on the whole study.\n\nWhat to do with spinal?\n\nSince spinal stenosis affects the center rather than specific sides, we duplicated the label for both sides during training. At inference, we aggregated the learnable vectors from both sides (those added in the decoder) before passing them through the classification head. We tested multiple aggregation methods and ultimately chose max pooling.\n\n#### Loss\n\nWe used the competition-specific loss function implemented in PyTorch. Although we explored several variations, such as a focal version of the competition loss, none yielded significantly better results.\n\n## Further information\n\n| description | backbone | local avg cv | public | private |\n| --- | --- | --- | --- | --- |\n| best public: ensemble 4 (dropped one) TTA 20 | regnet | 0.4027 | 0.3452 | 0.4121 | \n| best local: ensemble 5 TTA 40 | hgnet | 0.3883 | 0.3612 | 0.4215 | \n| best private: ensemble 5 TTA 16 | regnet | 0.4076 | 0.3472 | 0.4118 | \n\n\nThe two selected submissions were the best local and the best in the public leaderboard. The best in the public leaderboard was the 3rd best in the private leaderboard.\n\n\n## Code\n\nhttps://github.com/claverru/RSNA-lumbar", "3019312": "Congrats on the gold and becoming Competitions Master!!", "3019916": "Congratulations on this impressive development! great job" }, "source": "meta" }