{ "id": 274243, "title": "Choice of Image depth along the network", "url": "/competitions/rsna-miccai-brain-tumor-radiogenomic-classification/discussion/274243", "author_name": "", "post_date": "2021-09-24T19:23:41.883313500Z", "votes": null, "comment_count": 3, "views": 0, "content": "

Hello,
\nThis is my first time dealing with 3D data or medical image classification and I'm having some problems, I can't seem to decide what is the best depth to choose for the model.

\n

I thought of two things and I would like to explore both if possible:

\n

Using each slice individually and running a 2D conv network as a starter in order not to over complicate stuff

\n

This is my first thought as that would be an easy baseline model, which I previously built for different tasks. However, I face other problems doing this.

\n

I don't know which formula should be used as a prediction.

\n

Imagine that there are 200 slices in an MRI, and there is a tumor that is only visible in 20 of them.

\n

So, If I run this on a classifier, It should predict 1 (tumor exists) for 20 images out of the 200 assuming it works perfectly.

\n

The same logic can be applied to different type of imaging ( T1w, Tw1ce, FLAIR,T2) what if the model predicts that only one of them has a tumor and fails to see it in the others.

\n
\n

Make a 3D CNN after making all images have the same depth

\n

I was thinking that maybe I could check

\n

if the image had less than 40 slices, I would resize using scipy.ndimage.zoom to make it to 40

\n

If it has more than than, I would divide the array of slices by 40. (i.e. if the array had 120 slices, I would group each 3 slices together and average them). and pass that to the 3D CNN.

\n

and I'm wondering if that would be a good idea or would the quality of the images be affected by the zoom and averaging.

\n

This method, I still have the question of what should be done if the model predicted that a patient had a tumor according to T1w scan and the patient not having a tumor on the rest of the scans.

", "messages": [ { "id": "1522968", "postDate": "09/24/2021 19:23:41", "content": "

Hello,
\nThis is my first time dealing with 3D data or medical image classification and I'm having some problems, I can't seem to decide what is the best depth to choose for the model.

\n

I thought of two things and I would like to explore both if possible:

\n

Using each slice individually and running a 2D conv network as a starter in order not to over complicate stuff

\n

This is my first thought as that would be an easy baseline model, which I previously built for different tasks. However, I face other problems doing this.

\n

I don't know which formula should be used as a prediction.

\n

Imagine that there are 200 slices in an MRI, and there is a tumor that is only visible in 20 of them.

\n

So, If I run this on a classifier, It should predict 1 (tumor exists) for 20 images out of the 200 assuming it works perfectly.

\n

The same logic can be applied to different type of imaging ( T1w, Tw1ce, FLAIR,T2) what if the model predicts that only one of them has a tumor and fails to see it in the others.

\n
\n

Make a 3D CNN after making all images have the same depth

\n

I was thinking that maybe I could check

\n

if the image had less than 40 slices, I would resize using scipy.ndimage.zoom to make it to 40

\n

If it has more than than, I would divide the array of slices by 40. (i.e. if the array had 120 slices, I would group each 3 slices together and average them). and pass that to the 3D CNN.

\n

and I'm wondering if that would be a good idea or would the quality of the images be affected by the zoom and averaging.

\n

This method, I still have the question of what should be done if the model predicted that a patient had a tumor according to T1w scan and the patient not having a tumor on the rest of the scans.

", "rawMarkdown": "Hello,\nThis is my first time dealing with 3D data or medical image classification and I'm having some problems, I can't seem to decide what is the best depth to choose for the model.\n\nI thought of two things and I would like to explore both if possible:\n\n### Using each slice individually and running a 2D conv network as a starter in order not to over complicate stuff\n\nThis is my first thought as that would be an easy baseline model, which I previously built for different tasks. However, I face other problems doing this.\n\nI don't know which formula should be used as a prediction.\n\nImagine that there are 200 slices in an MRI, and there is a tumor that is only visible in 20 of them.\n\nSo, If I run this on a classifier, It should predict 1 (tumor exists) for 20 images out of the 200 assuming it works perfectly.\n\nThe same logic can be applied to different type of imaging ( T1w, Tw1ce, FLAIR,T2) what if the model predicts that only one of them has a tumor and fails to see it in the others.\n\n****\n\n### Make a 3D CNN after making all images have the same depth\n\nI was thinking that maybe I could check\n \nif the image had less than 40 slices, I would resize using `scipy.ndimage.zoom` to make it to 40\n\nIf it has more than than, I would divide the array of slices by 40. (i.e. if the array had 120 slices, I would group each 3 slices together and average them). and pass that to the 3D CNN.\n\nand I'm wondering if that would be a good idea or would the quality of the images be affected by the `zoom` and averaging.\n\nThis method, I still have the question of what should be done if the model predicted that a patient had a tumor according to T1w scan and the patient not having a tumor on the rest of the scans.", "votes": null }, { "id": "1523076", "postDate": "09/25/2021 00:15:29", "content": "

Due to the differences in each series, I wouldn't expect a single classifier to work well on all series. For instance, fluid is bright on a T2w series, but it's dark on T1. Bone and hard tumor tissue is bright on both. A classifier trained on T1 will easily misclassify the ventricles or sulcus as tumor on a T2w image.

\n

One of the challenges with either approach is that the series aren't all in the same plane across studies. That is, some are axial, some are coronal or sagittal. Most are a mixture of all three planes.

\n

Getting the entire test dataset in co-planar series orientations is the first challenge. I made a notebook that does just that -> https://www.kaggle.com/davidbroberts/export-dicom-series-by-plane

\n

Averaging slices together (slabbing) might make some interesting images. Typically averaging helps dissimilar tissues stand out from one another. Consider using MIPing for brain tumors. You will need to incorporate things like slice thickness, position and distance between slices to do it correctly though.

", "rawMarkdown": "Due to the differences in each series, I wouldn't expect a single classifier to work well on all series. For instance, fluid is bright on a T2w series, but it's dark on T1. Bone and hard tumor tissue is bright on both. A classifier trained on T1 will easily misclassify the ventricles or sulcus as tumor on a T2w image.\n\nOne of the challenges with either approach is that the series aren't all in the same plane across studies. That is, some are axial, some are coronal or sagittal. Most are a mixture of all three planes. \n\nGetting the entire test dataset in co-planar series orientations is the first challenge. I made a notebook that does just that -> https://www.kaggle.com/davidbroberts/export-dicom-series-by-plane\n\nAveraging slices together (slabbing) might make some interesting images. Typically averaging helps dissimilar tissues stand out from one another. Consider using MIPing for brain tumors. You will need to incorporate things like slice thickness, position and distance between slices to do it correctly though.", "votes": null }, { "id": "1523750", "postDate": "09/25/2021 17:09:58", "content": "
\n

A classifier trained on T1 will easily misclassify the ventricles or sulcus as tumor on a T2w image.

\n
\n

I thought the same thing but I wasn't sure as I'm unfamiliar with how tumor identification happens. thanks for help.

\n
\n

One of the challenges with either approach is that the series aren't all in the same plane across studies. That is, some are axial, some are coronal or sagittal. Most are a mixture of all three planes.

\n
\n

Shouldn't this be okay when working with 3D CNNs, I am thinking that it's similar to adding augmentations for a 2D classifier, so just as a 2D CNN should be able to recognize a rotated image, the 3D classifier should be able to classify a 3D object taken from a different direction.

\n

I might be wrong about this but it makes sense for me when I compare it to a rotated 2D picture.

\n
\n

You will need to incorporate things like slice thickness, position and distance between slices to do it correctly though.

\n
\n

I looked up MIPing and thanks for the recommendation, my current process for slabbing is to order them using position then stack them. Are there any resources on how to utilize the distance and slice thickness.

\n
\n

You were tremendously helpful, Thanks alot

", "rawMarkdown": "> A classifier trained on T1 will easily misclassify the ventricles or sulcus as tumor on a T2w image.\n\nI thought the same thing but I wasn't sure as I'm unfamiliar with how tumor identification happens. thanks for help.\n\n> One of the challenges with either approach is that the series aren't all in the same plane across studies. That is, some are axial, some are coronal or sagittal. Most are a mixture of all three planes.\n\nShouldn't this be okay when working with 3D CNNs, I am thinking that it's similar to adding augmentations for a 2D classifier, so just as a 2D CNN should be able to recognize a rotated image, the 3D classifier should be able to classify a 3D object taken from a different direction.\n\nI might be wrong about this but it makes sense for me when I compare it to a rotated 2D picture. \n\n> You will need to incorporate things like slice thickness, position and distance between slices to do it correctly though.\n\nI looked up MIPing and thanks for the recommendation, my current process for slabbing is to order them using position then stack them. Are there any resources on how to utilize the distance and slice thickness.\n\n****\nYou were tremendously helpful, Thanks alot", "votes": null }, { "id": "1523843", "postDate": "09/25/2021 18:37:51", "content": "
\n

Shouldn't this be okay when working with 3D CNNs, I am thinking that it's similar to adding augmentations for a 2D classifier, so just as a 2D CNN should be able to recognize a rotated image, the 3D classifier should be able to classify a 3D object taken from a different direction.

\n
\n

If you were able to accurately create a 3D object from a series (using DICOM position, orientation, thickness and spacing tags), and there were enough samples in each plane to create a balanced dataset in each plane .. It might be possible.

\n

Stacking JPG images into a 3D array is not the proper way to create a 3D object with DICOM images. Especially on series that are obliqued (not orthogonal), which many of these are .. it results in distorted and misshapen images. These images have already been exported from the original voxel array. So we're not working with the raw data per-se. Reformatting them in any way will almost always be a lossy operation.

", "rawMarkdown": "> Shouldn't this be okay when working with 3D CNNs, I am thinking that it's similar to adding augmentations for a 2D classifier, so just as a 2D CNN should be able to recognize a rotated image, the 3D classifier should be able to classify a 3D object taken from a different direction.\n\nIf you were able to accurately create a 3D object from a series (using DICOM position, orientation, thickness and spacing tags), and there were enough samples in each plane to create a balanced dataset in each plane .. It might be possible.\n\nStacking JPG images into a 3D array is not the proper way to create a 3D object with DICOM images. Especially on series that are obliqued (not orthogonal), which many of these are .. it results in distorted and misshapen images. These images have already been exported from the original voxel array. So we're not working with the raw data per-se. Reformatting them in any way will almost always be a lossy operation.", "votes": null } ], "comments": [ { "id": 1523076, "author_name": "davidbroberts", "author_url": "", "post_date": "09/25/2021 00:15:29", "content": "

Due to the differences in each series, I wouldn't expect a single classifier to work well on all series. For instance, fluid is bright on a T2w series, but it's dark on T1. Bone and hard tumor tissue is bright on both. A classifier trained on T1 will easily misclassify the ventricles or sulcus as tumor on a T2w image.

\n

One of the challenges with either approach is that the series aren't all in the same plane across studies. That is, some are axial, some are coronal or sagittal. Most are a mixture of all three planes.

\n

Getting the entire test dataset in co-planar series orientations is the first challenge. I made a notebook that does just that -> https://www.kaggle.com/davidbroberts/export-dicom-series-by-plane

\n

Averaging slices together (slabbing) might make some interesting images. Typically averaging helps dissimilar tissues stand out from one another. Consider using MIPing for brain tumors. You will need to incorporate things like slice thickness, position and distance between slices to do it correctly though.

", "votes": null, "replies": [ { "id": 1523750, "author_name": "abdelrhmanhosny", "author_url": "", "post_date": "09/25/2021 17:09:58", "content": "
\n

A classifier trained on T1 will easily misclassify the ventricles or sulcus as tumor on a T2w image.

\n
\n

I thought the same thing but I wasn't sure as I'm unfamiliar with how tumor identification happens. thanks for help.

\n
\n

One of the challenges with either approach is that the series aren't all in the same plane across studies. That is, some are axial, some are coronal or sagittal. Most are a mixture of all three planes.

\n
\n

Shouldn't this be okay when working with 3D CNNs, I am thinking that it's similar to adding augmentations for a 2D classifier, so just as a 2D CNN should be able to recognize a rotated image, the 3D classifier should be able to classify a 3D object taken from a different direction.

\n

I might be wrong about this but it makes sense for me when I compare it to a rotated 2D picture.

\n
\n

You will need to incorporate things like slice thickness, position and distance between slices to do it correctly though.

\n
\n

I looked up MIPing and thanks for the recommendation, my current process for slabbing is to order them using position then stack them. Are there any resources on how to utilize the distance and slice thickness.

\n
\n

You were tremendously helpful, Thanks alot

", "votes": null, "replies": [] }, { "id": 1523843, "author_name": "davidbroberts", "author_url": "", "post_date": "09/25/2021 18:37:51", "content": "
\n

Shouldn't this be okay when working with 3D CNNs, I am thinking that it's similar to adding augmentations for a 2D classifier, so just as a 2D CNN should be able to recognize a rotated image, the 3D classifier should be able to classify a 3D object taken from a different direction.

\n
\n

If you were able to accurately create a 3D object from a series (using DICOM position, orientation, thickness and spacing tags), and there were enough samples in each plane to create a balanced dataset in each plane .. It might be possible.

\n

Stacking JPG images into a 3D array is not the proper way to create a 3D object with DICOM images. Especially on series that are obliqued (not orthogonal), which many of these are .. it results in distorted and misshapen images. These images have already been exported from the original voxel array. So we're not working with the raw data per-se. Reformatting them in any way will almost always be a lossy operation.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1522968": "Hello,\nThis is my first time dealing with 3D data or medical image classification and I'm having some problems, I can't seem to decide what is the best depth to choose for the model.\n\nI thought of two things and I would like to explore both if possible:\n\n### Using each slice individually and running a 2D conv network as a starter in order not to over complicate stuff\n\nThis is my first thought as that would be an easy baseline model, which I previously built for different tasks. However, I face other problems doing this.\n\nI don't know which formula should be used as a prediction.\n\nImagine that there are 200 slices in an MRI, and there is a tumor that is only visible in 20 of them.\n\nSo, If I run this on a classifier, It should predict 1 (tumor exists) for 20 images out of the 200 assuming it works perfectly.\n\nThe same logic can be applied to different type of imaging ( T1w, Tw1ce, FLAIR,T2) what if the model predicts that only one of them has a tumor and fails to see it in the others.\n\n****\n\n### Make a 3D CNN after making all images have the same depth\n\nI was thinking that maybe I could check\n \nif the image had less than 40 slices, I would resize using `scipy.ndimage.zoom` to make it to 40\n\nIf it has more than than, I would divide the array of slices by 40. (i.e. if the array had 120 slices, I would group each 3 slices together and average them). and pass that to the 3D CNN.\n\nand I'm wondering if that would be a good idea or would the quality of the images be affected by the `zoom` and averaging.\n\nThis method, I still have the question of what should be done if the model predicted that a patient had a tumor according to T1w scan and the patient not having a tumor on the rest of the scans.", "1523076": "Due to the differences in each series, I wouldn't expect a single classifier to work well on all series. For instance, fluid is bright on a T2w series, but it's dark on T1. Bone and hard tumor tissue is bright on both. A classifier trained on T1 will easily misclassify the ventricles or sulcus as tumor on a T2w image.\n\nOne of the challenges with either approach is that the series aren't all in the same plane across studies. That is, some are axial, some are coronal or sagittal. Most are a mixture of all three planes. \n\nGetting the entire test dataset in co-planar series orientations is the first challenge. I made a notebook that does just that -> https://www.kaggle.com/davidbroberts/export-dicom-series-by-plane\n\nAveraging slices together (slabbing) might make some interesting images. Typically averaging helps dissimilar tissues stand out from one another. Consider using MIPing for brain tumors. You will need to incorporate things like slice thickness, position and distance between slices to do it correctly though.", "1523750": "> A classifier trained on T1 will easily misclassify the ventricles or sulcus as tumor on a T2w image.\n\nI thought the same thing but I wasn't sure as I'm unfamiliar with how tumor identification happens. thanks for help.\n\n> One of the challenges with either approach is that the series aren't all in the same plane across studies. That is, some are axial, some are coronal or sagittal. Most are a mixture of all three planes.\n\nShouldn't this be okay when working with 3D CNNs, I am thinking that it's similar to adding augmentations for a 2D classifier, so just as a 2D CNN should be able to recognize a rotated image, the 3D classifier should be able to classify a 3D object taken from a different direction.\n\nI might be wrong about this but it makes sense for me when I compare it to a rotated 2D picture. \n\n> You will need to incorporate things like slice thickness, position and distance between slices to do it correctly though.\n\nI looked up MIPing and thanks for the recommendation, my current process for slabbing is to order them using position then stack them. Are there any resources on how to utilize the distance and slice thickness.\n\n****\nYou were tremendously helpful, Thanks alot", "1523843": "> Shouldn't this be okay when working with 3D CNNs, I am thinking that it's similar to adding augmentations for a 2D classifier, so just as a 2D CNN should be able to recognize a rotated image, the 3D classifier should be able to classify a 3D object taken from a different direction.\n\nIf you were able to accurately create a 3D object from a series (using DICOM position, orientation, thickness and spacing tags), and there were enough samples in each plane to create a balanced dataset in each plane .. It might be possible.\n\nStacking JPG images into a 3D array is not the proper way to create a 3D object with DICOM images. Especially on series that are obliqued (not orthogonal), which many of these are .. it results in distorted and misshapen images. These images have already been exported from the original voxel array. So we're not working with the raw data per-se. Reformatting them in any way will almost always be a lossy operation." }, "source": "meta" }