![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2Fe6bac333f58d7702a1a016a0549f511e%2FScreenshot%20from%202024-07-07%2011-33-53.png?generation=1720366446450490&alt=media\")
The image metadata has a bunch of helpful features like
\n \n \n \n \n \n \n \n \n \n \n \n [, , ]\n [, , , , , ]\n \n \nSo the last one is interesting and differs from the file indices
\n = pydicom.dcmread()\n.SliceLocation # .\nWhereas
\n = pydicom.dcmread()\n.SliceLocation # -.\nAs such
\n = glob.glob()\n = [pydicom.dcmread(fname) for fname in files]\n = sorted(slices, key=lambda s: s.SliceLocation)\nvs
\n = .()\n = sorted(, key=lambda : (.()[-].()[]))\nslices = [pydicom.dcmread(fname) fname in ]\nReference: https://pydicom.github.io/pydicom/stable/auto_examples/image_processing/reslice.html
", "messages": [ { "id": "2909352", "postDate": "07/06/2024 21:26:34", "content": "The image metadata has a bunch of helpful features like
\n \n \n \n \n \n \n \n \n \n \n \n [, , ]\n [, , , , , ]\n \n \nSo the last one is interesting and differs from the file indices
\n = pydicom.dcmread()\n.SliceLocation # .\nWhereas
\n = pydicom.dcmread()\n.SliceLocation # -.\nAs such
\n = glob.glob()\n = [pydicom.dcmread(fname) for fname in files]\n = sorted(slices, key=lambda s: s.SliceLocation)\nvs
\n = .()\n = sorted(, key=lambda : (.()[-].()[]))\nslices = [pydicom.dcmread(fname) fname in ]\nReference: https://pydicom.github.io/pydicom/stable/auto_examples/image_processing/reslice.html
", "rawMarkdown": "The image metadata has a bunch of helpful features like\n```\n (0008, 0018) SOP Instance UID UI: 4646740.1.4\n (0008, 0023) Content Date DA: '20240503'\n (0008, 0033) Content Time TM: '223735.025639'\n (0008, 103e) Series Description LO: ''\n (0010, 0020) Patient ID LO: '4646740'\n (0018, 0050) Slice Thickness DS: '3.0'\n (0018, 0088) Spacing Between Slices DS: '3.3'\n (0018, 5100) Patient Position CS: 'HFS'\n (0020, 000d) Study Instance UID UI: 4646740\n (0020, 000e) Series Instance UID UI: 4646740.3486248476\n (0020, 0013) Instance Number IS: '4'\n (0020, 0032) Image Position (Patient) DS: [26.0466, -54.1397, 155.921]\n (0020, 0037) Image Orientation (Patient) DS: [0, 1, 4.897e-12, 0, 4.897e-12, -1]\n (0020, 0052) Frame of Reference UID UI: 1.2.826.0.1.3680043.8.498.10607309798034473096588236962421719480\n (0020, 1041) Slice Location DS: '26.0466'\n\n```\n\nSo the last one is interesting and differs from the file indices\n```\nds = pydicom.dcmread(\"../data/rsna-2024-lumbar-spine-degenerative-classification/train_images/4646740/3486248476/9.dcm\")\nds.SliceLocation # 9.54656\n```\nWhereas\n```\nds = pydicom.dcmread(\"../data/rsna-2024-lumbar-spine-degenerative-classification/train_images/4646740/3486248476/12.dcm\")\nds.SliceLocation # -0.353442\n```\n\nAs such\n```\nfiles = glob.glob(\"../data/rsna-2024-lumbar-spine-degenerative-classification/train_images/4646740/3486248476/*\")\nslices = [pydicom.dcmread(fname) for fname in files]\nslices = sorted(slices, key=lambda s: s.SliceLocation)\n```\n\nvs\n\n```\nfiles = glob.glob(\"../data/rsna-2024-lumbar-spine-degenerative-classification/train_images/4646740/3486248476/*\")\nfiles = sorted(files, key=lambda x: int(x.split('/')[-1].split('.')[0]))\nslices = [pydicom.dcmread(fname) for fname in files]\n```\n\nReference: https://pydicom.github.io/pydicom/stable/auto_examples/image_processing/reslice.html", "votes": null }, { "id": "2909992", "postDate": "07/07/2024 12:42:42", "content": "Thank you! That was informative.
", "rawMarkdown": "Thank you! That was informative.", "votes": null }, { "id": "2910229", "postDate": "07/07/2024 15:34:16", "content": "@vsahin Hi Victor, it is a nice catch. Though I would be very careful with SliceLocation order. Sometimes there is a reversed order, sometimes there is an equality and sometimes there is not.
\n\ndicom_files = (dicom_files, key= x: (x.split()[-].split()[]))\n
\ndicom_files = (dicom_files, key= s: s.SliceLocation)\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2Fcb6a3ed16fab6975e70d6e5d4e61689e%2FScreenshot%20from%202024-07-07%2011-33-11.png?generation=1720366410274831&alt=media\")
glob glob\n pandas pd\n pydicom\n tqdm.auto tqdm\n joblib Parallel, delayed\n\n\n ():\n (x.split()[-].split()[])\n\n ():\n dicom_paths = glob()\n dicom_paths_instance_sorted = (dicom_paths, key=get_instance_number_from_path)\n\n slices = [(pydicom.dcmread(fp), fp) fp dicom_paths]\n slices = (slices, key= s: s[].SliceLocation)\n dicom_paths_slice_location_sorted = [fp s, fp slices]\n\n dicom_paths_instance_sorted == dicom_paths_slice_location_sorted:\n (, (st, sr))\n dicom_paths_instance_sorted == dicom_paths_slice_location_sorted[::-]:\n (, (st, sr))\n :\n (, (st, sr))\n\ndesc = pd.read_csv()\nresults = Parallel(n_jobs=)(delayed(process_series)(st, sr) st, sr tqdm(desc[[, ]].values))\n\nequals = [item[] item results item[] == ]\nreversed_equals = [item[] item results item[] == ]\nnot_equals = [item[] item results item[] == ]\n\n()\n()\n()\n\nEquals: \nReversed Equals: \nNot Equals: \n(example - where is no sorting order equality preserved either reversed or direct for study: 11340341, series: 1224932122)
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2Fb049765bdfb6514e117da9cf954cf1ba%2Finsta.png?generation=1720370203779646&alt=media\")
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F6259210%2F390c6c98545613a9960005d1b2e31058%2Fslice.png?generation=1720370225097509&alt=media\")
It looks 36.dcm does not simply belong to the place between 41.dcm and 40.dcm if we apply SliceLocation order.
\nIt will be interesting to see other people reporting the score difference because of it.
\n\n@vsahin Hi Victor, it is a nice catch. Though I would be very careful with SliceLocation order. Sometimes there is a reversed order, sometimes there is an equality and sometimes there is not.
\n
Now that's really interesting. It is supposed to be the ground truth per the DICOM API specs
\nhttps://dicom.innolitics.com/ciods/ct-image/image-plane/00201041
If the difference was only from the reversed order, then I would not expect much difference beyond the reconstructed 3D images being mirrored, which I can see on the coronal axis (2nd image) with my example:
\nSorted by SliceLocation
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1734709%2Faab86500365d07f125344c6dfb4fb9b4%2Fcoronal1.png?generation=1720390964491460&alt=media\")
Sorted by file index
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1734709%2F891a847b3850156ab7e64dab31aff08f%2Fcoronal2.png?generation=1720390989900121&alt=media\")
But…
\nWhen looking at your example, it seems the file order is the correct one:
\nSorted by SliceLocation. Note the vertical band artifacts on the sagittal axis which is sideways on the 2nd image now.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1734709%2Fff5b69c3baf92aad99436ab43b2c0721%2Fsagittal1.png?generation=1720391314128058&alt=media\")
Sorted by file index, note the artifacts not being visible.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1734709%2F9af9cccf48423ccab4b1e1c969d94927%2Fsagittal2.png?generation=1720391565156470&alt=media\")
So it seems you are correct. Although I find it really weird for the metadata coming directly from the imaging equipment to be the incorrect one.
", "rawMarkdown": "> @vsahin Hi Victor, it is a nice catch. Though I would be very careful with SliceLocation order. Sometimes there is a reversed order, sometimes there is an equality and sometimes there is not.\n\nNow that's really interesting. It is supposed to be the ground truth per the DICOM API specs\nhttps://dicom.innolitics.com/ciods/ct-image/image-plane/00201041\n\nIf the difference was only from the reversed order, then I would not expect much difference beyond the reconstructed 3D images being mirrored, which I can see on the coronal axis (2nd image) with my example:\n\nSorted by SliceLocation\n\n\nSorted by file index\n\n\nBut...\n\nWhen looking at your example, it seems the file order is the correct one:\nSorted by SliceLocation. Note the vertical band artifacts on the sagittal axis which is sideways on the 2nd image now.\n\n\nSorted by file index, note the artifacts not being visible.\n\n\nSo it seems you are correct. Although I find it really weird for the metadata coming directly from the imaging equipment to be the incorrect one.", "votes": null }, { "id": "2910803", "postDate": "07/07/2024 23:38:05", "content": "There is a workaround to use ImagePositionPatient meta for sorting. Though there are still 83 axial series to be decided on after that. BTW I have tried all 3 sorting methods and my local CV have not changed significantly. I don't know if my model that bad and not really sensitive to that. Though I expected that logloss for foraminal part of predictions would go down, and it did by 0.03.
The reason for this discrepancy is because this axial series (and most of the axial series in general) has multiple groups of slices with different orientations (ImageOrientationPatient attribute). In this case (study: 11340341, series: 1224932122) there are 5 groups:
So, in my option, these 45 slices can't be grouped into a single 3D volume. To show this you can see artifacts in the image in the \"sorted by file index\" volume above at the intersection of \"10 to 18\"/\"19 to 27\" groups and \"19 to 27\"/\"28 to 36\" groups:
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1303569%2Fb4c1a95d06c57012a6e49d2155fdcbf8%2FScreenshot%202024-07-08%20061209(2).png?generation=1720399722741275&alt=media\")
I think those are just imaging noise from the subject moving around etc. You can see the structures being contiguous despite the brightness shifts, and the brightness shifts not being consistent across the entire slice.
\nThat is unlike the artifacts in the first example which is definitely due to the slices being out of order.
", "rawMarkdown": "I think those are just imaging noise from the subject moving around etc. You can see the structures being contiguous despite the brightness shifts, and the brightness shifts not being consistent across the entire slice. \n\nThat is unlike the artifacts in the first example which is definitely due to the slices being out of order.", "votes": null }, { "id": "2911400", "postDate": "07/08/2024 09:11:33", "content": "@vsahin Yes, you are right, the regions I marked were just imaging noise. Maybe the angle between the \"19 to 27\"/\"28 to 36\" groups are too small to be noticed in the image, but by the SliceLocation attribute, the volumes generated by these 2 groups should have a common area. This is why sorting by SliceLocation creates vertical band artifacts on the sagittal axis.
To give a more easily visualizable example: study: 391103067, series: 1771893480 sagittal view:
\n![\"Sagittal](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1303569%2F83b9e5b00acc9fc8e19c715d928762f9%2FScreenshot%202024-07-08%20143745.png?generation=1720429691756553&alt=media\")
Here instances 1 to 5 have ascending SliceLocation ordering while 6 to 25 have descending SliceLocation ordering.
Oh you make a really good point. I found this one example reconstructed volume from axial slices, you can clearly see the orientation change around the middle on the sagittal aspect.![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1734709%2F9c83ad165fb69a4ca4489e6bf7f90ea9%2Forientation_change.png?generation=1720590494106580&alt=media\")
Thank you! That was informative.
", "votes": null, "replies": [] }, { "id": 2910229, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "07/07/2024 15:34:16", "content": "@vsahin Hi Victor, it is a nice catch. Though I would be very careful with SliceLocation order. Sometimes there is a reversed order, sometimes there is an equality and sometimes there is not.
\n\ndicom_files = (dicom_files, key= x: (x.split()[-].split()[]))\n
\ndicom_files = (dicom_files, key= s: s.SliceLocation)\n
glob glob\n pandas pd\n pydicom\n tqdm.auto tqdm\n joblib Parallel, delayed\n\n\n ():\n (x.split()[-].split()[])\n\n ():\n dicom_paths = glob()\n dicom_paths_instance_sorted = (dicom_paths, key=get_instance_number_from_path)\n\n slices = [(pydicom.dcmread(fp), fp) fp dicom_paths]\n slices = (slices, key= s: s[].SliceLocation)\n dicom_paths_slice_location_sorted = [fp s, fp slices]\n\n dicom_paths_instance_sorted == dicom_paths_slice_location_sorted:\n (, (st, sr))\n dicom_paths_instance_sorted == dicom_paths_slice_location_sorted[::-]:\n (, (st, sr))\n :\n (, (st, sr))\n\ndesc = pd.read_csv()\nresults = Parallel(n_jobs=)(delayed(process_series)(st, sr) st, sr tqdm(desc[[, ]].values))\n\nequals = [item[] item results item[] == ]\nreversed_equals = [item[] item results item[] == ]\nnot_equals = [item[] item results item[] == ]\n\n()\n()\n()\n\nEquals: \nReversed Equals: \nNot Equals: \n(example - where is no sorting order equality preserved either reversed or direct for study: 11340341, series: 1224932122)
\n
It looks 36.dcm does not simply belong to the place between 41.dcm and 40.dcm if we apply SliceLocation order.
\nIt will be interesting to see other people reporting the score difference because of it.
\n\n@vsahin Hi Victor, it is a nice catch. Though I would be very careful with SliceLocation order. Sometimes there is a reversed order, sometimes there is an equality and sometimes there is not.
\n
Now that's really interesting. It is supposed to be the ground truth per the DICOM API specs
\nhttps://dicom.innolitics.com/ciods/ct-image/image-plane/00201041
If the difference was only from the reversed order, then I would not expect much difference beyond the reconstructed 3D images being mirrored, which I can see on the coronal axis (2nd image) with my example:
\nSorted by SliceLocation
\n
Sorted by file index
\n
But…
\nWhen looking at your example, it seems the file order is the correct one:
\nSorted by SliceLocation. Note the vertical band artifacts on the sagittal axis which is sideways on the 2nd image now.
\n
Sorted by file index, note the artifacts not being visible.
\n
So it seems you are correct. Although I find it really weird for the metadata coming directly from the imaging equipment to be the incorrect one.
", "votes": null, "replies": [ { "id": 2910803, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "07/07/2024 23:38:05", "content": "There is a workaround to use ImagePositionPatient meta for sorting. Though there are still 83 axial series to be decided on after that. BTW I have tried all 3 sorting methods and my local CV have not changed significantly. I don't know if my model that bad and not really sensitive to that. Though I expected that logloss for foraminal part of predictions would go down, and it did by 0.03.
The reason for this discrepancy is because this axial series (and most of the axial series in general) has multiple groups of slices with different orientations (ImageOrientationPatient attribute). In this case (study: 11340341, series: 1224932122) there are 5 groups:
So, in my option, these 45 slices can't be grouped into a single 3D volume. To show this you can see artifacts in the image in the \"sorted by file index\" volume above at the intersection of \"10 to 18\"/\"19 to 27\" groups and \"19 to 27\"/\"28 to 36\" groups:
\n
I think those are just imaging noise from the subject moving around etc. You can see the structures being contiguous despite the brightness shifts, and the brightness shifts not being consistent across the entire slice.
\nThat is unlike the artifacts in the first example which is definitely due to the slices being out of order.
", "votes": null, "replies": [ { "id": 2911400, "author_name": "coderrkj", "author_url": "", "post_date": "07/08/2024 09:11:33", "content": "@vsahin Yes, you are right, the regions I marked were just imaging noise. Maybe the angle between the \"19 to 27\"/\"28 to 36\" groups are too small to be noticed in the image, but by the SliceLocation attribute, the volumes generated by these 2 groups should have a common area. This is why sorting by SliceLocation creates vertical band artifacts on the sagittal axis.
To give a more easily visualizable example: study: 391103067, series: 1771893480 sagittal view:
\n
Here instances 1 to 5 have ascending SliceLocation ordering while 6 to 25 have descending SliceLocation ordering.
Oh you make a really good point. I found this one example reconstructed volume from axial slices, you can clearly see the orientation change around the middle on the sagittal aspect.