{ "id": 539439, "title": "[7th solution, my part] Single Stage Model Wins!", "url": "/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/539439", "author_name": "", "post_date": "2024-10-09T00:03:36.555180100Z", "votes": 35, "comment_count": 6, "views": 0, "content": "

Acknowledgement

\n

\"We extend our thanks to HP for providing the Z8 Fury-G5 Data Science Workstation, which empowered our deep learning experiments. The high computational power and large GPU memory enabled us to design our models swiftly.\"

\n
\n

Section1. Objective

\n\n

Since my teammate @lihaoweicvch already has very good two-stage model, my task is to design one-stage model to improve his results.

\n

Note: We later discovered that top teams solved the same issue by using a \"multi-crop, multi-level\" prediction model with transformers and LSTMs, \ninstead of a \"single-crop, single-level\" prediction model.

\n

Section2. Modeling

\n\n

[1] 'Numerical Coordinate Regression with Convolutional Neural Networks' - Aiden Nibali, Arvix 2018

\n\n

Section3. Local cross validation

\n

\"\"

\n

\"\"

\n

Section4. Submission

\n\n

\"\"

\n

Section5. Training code

\n

https://github.com/hengck23/solution-rsna-2024-lumbar-spine

\n

demo:
\nhttps://www.kaggle.com/code/hengck23/clean-final-submit02-scs-nfn-ensemble

", "messages": [ { "id": "3012337", "postDate": "10/09/2024 00:03:36", "content": "

Acknowledgement

\n

\"We extend our thanks to HP for providing the Z8 Fury-G5 Data Science Workstation, which empowered our deep learning experiments. The high computational power and large GPU memory enabled us to design our models swiftly.\"

\n
\n

Section1. Objective

\n\n

Since my teammate @lihaoweicvch already has very good two-stage model, my task is to design one-stage model to improve his results.

\n

Note: We later discovered that top teams solved the same issue by using a \"multi-crop, multi-level\" prediction model with transformers and LSTMs, \ninstead of a \"single-crop, single-level\" prediction model.

\n

Section2. Modeling

\n\n

[1] 'Numerical Coordinate Regression with Convolutional Neural Networks' - Aiden Nibali, Arvix 2018

\n\n

Section3. Local cross validation

\n

\"\"

\n

\"\"

\n

Section4. Submission

\n\n

\"\"

\n

Section5. Training code

\n

https://github.com/hengck23/solution-rsna-2024-lumbar-spine

\n

demo:
\nhttps://www.kaggle.com/code/hengck23/clean-final-submit02-scs-nfn-ensemble

", "rawMarkdown": "### Acknowledgement\n### \"We extend our thanks to HP for providing the Z8 Fury-G5 Data Science Workstation, which empowered our deep learning experiments. The high computational power and large GPU memory enabled us to design our models swiftly.\"\n\n----\n\n\n# Section1. Objective\n- Most teams will focus on the straightforward two-stage approach: crop and classify.\n- However, we believe that a one-stage model can complement the results of the two-stage approach when used in an ensemble. \n- The two-stage classification method only considers the context within the crop, but grade scores across different levels are correlated, \nand leveraging the entire image context in one-stage nethod may lead to better predictions.\n\nSince my teammate @lihaoweicvch already has very good two-stage model, my task is to design one-stage model to improve his results.\n\n_Note: We later discovered that top teams solved the same issue by using a \"multi-crop, multi-level\" prediction model with transformers and LSTMs, \ninstead of a \"single-crop, single-level\" prediction model._\n\n# Section2. Modeling\n- Two-stage model uses \"crop and classify\". One-stage models uses \"point-masking and pool\".\n- Below shows the model architecture of NFN (neural foraminal narrowing).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F950a45f55ada1fad21422edaef930ad7%2FSelection_512.png?generation=1728959974347938&alt=media)\n- Below shows the model architecture of SCS (spinal canal stenosis).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F6e7901f2f205ad9d868c4b0cc2408e02%2FSelection_511.png?generation=1728960029417304&alt=media)\n\n- Main process block:\n - Predict the pixelwise level point heatmap. Use Jensen-Shannon divergence loss for learn the net parameters for this.\n - Convert heatmap target xyz coordinates using the differentiable spatial-to-numerical transform (DSNT) as described in paper [1].\n - Predict the pixelwise grade.\n - Use the heatmap to pool pixelwise grades and compute the levelwise grade.\n \n[1] 'Numerical Coordinate Regression with Convolutional Neural Networks' - Aiden Nibali, Arvix 2018 \n\n\n- Training:\n - We observe that we can temporarily modify the network to use the ground truth level point heatmap and focus solely on learning grade prediction. This serves as an upper bound for grade prediction performance.\n - One-stage networks are challenging to train because the optimal parameters for coordinate prediction are not necessarily the best for grade prediction.\n - This can be confirmed by observing the loss for both level and grade during training iterations, as they seem to compete with each other.\n - We believe one reason for this is labeling errors, particularly the significant confusion between L5 and S1 level labels. Additionally, the point labels themselves are not very precise.\n \n\n\n# Section3. Local cross validation\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Ffd7bcea022cbe18fba966ae183d0ee1b%2FSelection_513.png?generation=1728970086417304&alt=media)\n \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fef75c96ae7091ca4ec81350b134b4d8b%2FSelection_514.png?generation=1728970132039656&alt=media)\n\n# Section4. Submission\n- blending for submission is: 0.75\\*2stage + 0.25\\*1stage\n- even with bugged implementation (and training), one-stage model improves two-stage model.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7d8d21bc1fac3ccb80bf2474f51fd97a%2FSelection_502.png?generation=1728780736026952&alt=media)\n\n\n#Section5. Training code\nhttps://github.com/hengck23/solution-rsna-2024-lumbar-spine\n\ndemo:\nhttps://www.kaggle.com/code/hengck23/clean-final-submit02-scs-nfn-ensemble", "votes": null }, { "id": "3012348", "postDate": "10/09/2024 00:23:24", "content": "

Congrats!

", "rawMarkdown": "Congrats!", "votes": null }, { "id": "3012370", "postDate": "10/09/2024 01:19:48", "content": "

That was very informative. Congratulations!

", "rawMarkdown": "That was very informative. Congratulations!", "votes": null }, { "id": "3012818", "postDate": "10/09/2024 12:02:51", "content": "

Congrats :)

", "rawMarkdown": "Congrats :)", "votes": null }, { "id": "3013047", "postDate": "10/09/2024 15:58:31", "content": "

Congratulations on winning the 7th prize in this competition. Commendable solution approach with single model.

", "rawMarkdown": "Congratulations on winning the 7th prize in this competition. Commendable solution approach with single model.", "votes": null }, { "id": "3016170", "postDate": "10/13/2024 12:58:26", "content": "

Congratulations!

\n

I wrote a training script for your single stage method that you published in this notebook, and I managed to successfully train a 5-fold model for Sagittal T2/STIR series with a local CV score of 0.29. I used a few custom implemented data augmentation methods for the volumes, namely, Random Brightness, Random Additive Noise, Random Horizontal Flip, Random Rotation along centered z-axis and Random Translation along x- and y-axis that modifies coordinates accordingly.
\nHowever, I failed to train the same model with Sagittal T1 and Axial T2 series. No matter how much I tried different tricks, I couldn't get any local CV scores better than 0.55 (Sagittal T1 ) and 0.63 (Axial T2). I am eager to look at your training code to see how you managed to train Sagittal T1 and Axial T2 series. Could you share your experiment with these two series?

", "rawMarkdown": "Congratulations!\n\nI wrote a training script for your single stage method that you published in this [notebook](https://www.kaggle.com/code/hengck23/ver-1-magic-single-stage-model), and I managed to successfully train a 5-fold model for Sagittal T2/STIR series with a local CV score of 0.29. I used a few custom implemented data augmentation methods for the volumes, namely, Random Brightness, Random Additive Noise, Random Horizontal Flip, Random Rotation along centered z-axis and Random Translation along x- and y-axis that modifies coordinates accordingly.\nHowever, I failed to train the same model with Sagittal T1 and Axial T2 series. No matter how much I tried different tricks, I couldn't get any local CV scores better than 0.55 (Sagittal T1 ) and 0.63 (Axial T2). I am eager to look at your training code to see how you managed to train Sagittal T1 and Axial T2 series. Could you share your experiment with these two series?", "votes": null }, { "id": "3017227", "postDate": "10/14/2024 16:49:58", "content": "

please wait till the end of this week for the final update of this discussion post.
\nhere is a preliminary version of the training code:
\nhttps://github.com/hengck23/solution-rsna-2024-lumbar-spine/

\n

… note that it is not the final version yet as we are finalising post-submission experiments…

", "rawMarkdown": "please wait till the end of this week for the final update of this discussion post.\nhere is a preliminary version of the training code:\nhttps://github.com/hengck23/solution-rsna-2024-lumbar-spine/\n\n... note that it is not the final version yet as we are finalising post-submission experiments...", "votes": null } ], "comments": [ { "id": 3012348, "author_name": "vsahin", "author_url": "", "post_date": "10/09/2024 00:23:24", "content": "

Congrats!

", "votes": null, "replies": [] }, { "id": 3012370, "author_name": "shunkuraishi", "author_url": "", "post_date": "10/09/2024 01:19:48", "content": "

That was very informative. Congratulations!

", "votes": null, "replies": [] }, { "id": 3012818, "author_name": "evanhislupus", "author_url": "", "post_date": "10/09/2024 12:02:51", "content": "

Congrats :)

", "votes": null, "replies": [] }, { "id": 3013047, "author_name": "crsuthikshnkumar", "author_url": "", "post_date": "10/09/2024 15:58:31", "content": "

Congratulations on winning the 7th prize in this competition. Commendable solution approach with single model.

", "votes": null, "replies": [] }, { "id": 3016170, "author_name": "rasoulmojtahedzadeh", "author_url": "", "post_date": "10/13/2024 12:58:26", "content": "

Congratulations!

\n

I wrote a training script for your single stage method that you published in this notebook, and I managed to successfully train a 5-fold model for Sagittal T2/STIR series with a local CV score of 0.29. I used a few custom implemented data augmentation methods for the volumes, namely, Random Brightness, Random Additive Noise, Random Horizontal Flip, Random Rotation along centered z-axis and Random Translation along x- and y-axis that modifies coordinates accordingly.
\nHowever, I failed to train the same model with Sagittal T1 and Axial T2 series. No matter how much I tried different tricks, I couldn't get any local CV scores better than 0.55 (Sagittal T1 ) and 0.63 (Axial T2). I am eager to look at your training code to see how you managed to train Sagittal T1 and Axial T2 series. Could you share your experiment with these two series?

", "votes": null, "replies": [ { "id": 3017227, "author_name": "hengck23", "author_url": "", "post_date": "10/14/2024 16:49:58", "content": "

please wait till the end of this week for the final update of this discussion post.
\nhere is a preliminary version of the training code:
\nhttps://github.com/hengck23/solution-rsna-2024-lumbar-spine/

\n

… note that it is not the final version yet as we are finalising post-submission experiments…

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3012337": "### Acknowledgement\n### \"We extend our thanks to HP for providing the Z8 Fury-G5 Data Science Workstation, which empowered our deep learning experiments. The high computational power and large GPU memory enabled us to design our models swiftly.\"\n\n----\n\n\n# Section1. Objective\n- Most teams will focus on the straightforward two-stage approach: crop and classify.\n- However, we believe that a one-stage model can complement the results of the two-stage approach when used in an ensemble. \n- The two-stage classification method only considers the context within the crop, but grade scores across different levels are correlated, \nand leveraging the entire image context in one-stage nethod may lead to better predictions.\n\nSince my teammate @lihaoweicvch already has very good two-stage model, my task is to design one-stage model to improve his results.\n\n_Note: We later discovered that top teams solved the same issue by using a \"multi-crop, multi-level\" prediction model with transformers and LSTMs, \ninstead of a \"single-crop, single-level\" prediction model._\n\n# Section2. Modeling\n- Two-stage model uses \"crop and classify\". One-stage models uses \"point-masking and pool\".\n- Below shows the model architecture of NFN (neural foraminal narrowing).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F950a45f55ada1fad21422edaef930ad7%2FSelection_512.png?generation=1728959974347938&alt=media)\n- Below shows the model architecture of SCS (spinal canal stenosis).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F6e7901f2f205ad9d868c4b0cc2408e02%2FSelection_511.png?generation=1728960029417304&alt=media)\n\n- Main process block:\n - Predict the pixelwise level point heatmap. Use Jensen-Shannon divergence loss for learn the net parameters for this.\n - Convert heatmap target xyz coordinates using the differentiable spatial-to-numerical transform (DSNT) as described in paper [1].\n - Predict the pixelwise grade.\n - Use the heatmap to pool pixelwise grades and compute the levelwise grade.\n \n[1] 'Numerical Coordinate Regression with Convolutional Neural Networks' - Aiden Nibali, Arvix 2018 \n\n\n- Training:\n - We observe that we can temporarily modify the network to use the ground truth level point heatmap and focus solely on learning grade prediction. This serves as an upper bound for grade prediction performance.\n - One-stage networks are challenging to train because the optimal parameters for coordinate prediction are not necessarily the best for grade prediction.\n - This can be confirmed by observing the loss for both level and grade during training iterations, as they seem to compete with each other.\n - We believe one reason for this is labeling errors, particularly the significant confusion between L5 and S1 level labels. Additionally, the point labels themselves are not very precise.\n \n\n\n# Section3. Local cross validation\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Ffd7bcea022cbe18fba966ae183d0ee1b%2FSelection_513.png?generation=1728970086417304&alt=media)\n \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fef75c96ae7091ca4ec81350b134b4d8b%2FSelection_514.png?generation=1728970132039656&alt=media)\n\n# Section4. Submission\n- blending for submission is: 0.75\\*2stage + 0.25\\*1stage\n- even with bugged implementation (and training), one-stage model improves two-stage model.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7d8d21bc1fac3ccb80bf2474f51fd97a%2FSelection_502.png?generation=1728780736026952&alt=media)\n\n\n#Section5. Training code\nhttps://github.com/hengck23/solution-rsna-2024-lumbar-spine\n\ndemo:\nhttps://www.kaggle.com/code/hengck23/clean-final-submit02-scs-nfn-ensemble", "3012348": "Congrats!", "3012370": "That was very informative. Congratulations!", "3012818": "Congrats :)", "3013047": "Congratulations on winning the 7th prize in this competition. Commendable solution approach with single model.", "3016170": "Congratulations!\n\nI wrote a training script for your single stage method that you published in this [notebook](https://www.kaggle.com/code/hengck23/ver-1-magic-single-stage-model), and I managed to successfully train a 5-fold model for Sagittal T2/STIR series with a local CV score of 0.29. I used a few custom implemented data augmentation methods for the volumes, namely, Random Brightness, Random Additive Noise, Random Horizontal Flip, Random Rotation along centered z-axis and Random Translation along x- and y-axis that modifies coordinates accordingly.\nHowever, I failed to train the same model with Sagittal T1 and Axial T2 series. No matter how much I tried different tricks, I couldn't get any local CV scores better than 0.55 (Sagittal T1 ) and 0.63 (Axial T2). I am eager to look at your training code to see how you managed to train Sagittal T1 and Axial T2 series. Could you share your experiment with these two series?", "3017227": "please wait till the end of this week for the final update of this discussion post.\nhere is a preliminary version of the training code:\nhttps://github.com/hengck23/solution-rsna-2024-lumbar-spine/\n\n... note that it is not the final version yet as we are finalising post-submission experiments..." }, "source": "meta" }