My data reader passed all public kaggle train data but is stuck with \"notebook exception error\" for a few days.
\nI am running out of submissions and it is now impossible for me to \"submit and debug\".
I wonder if anyone can help it identify which hidden dicom test tag is giving problem?
\nupdate:
\nthese are tested ok for test and train:
for the following, I never submitted without try-catch, so I do not know if they are issues
\nalso, verified problems:
\nIf there are other issues, please let me know. thanks.
\nreference:
\n[1] https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/528284
My data reader passed all public kaggle train data but is stuck with \"notebook exception error\" for a few days.
\nI am running out of submissions and it is now impossible for me to \"submit and debug\".
I wonder if anyone can help it identify which hidden dicom test tag is giving problem?
\nupdate:
\nthese are tested ok for test and train:
for the following, I never submitted without try-catch, so I do not know if they are issues
\nalso, verified problems:
\nIf there are other issues, please let me know. thanks.
\nreference:
\n[1] https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/528284
I think is a sneaky bug in my case. But I've already lost 3 submissions since I've included Sagittal T2 3D projections. If that's the case I'm only using IPP,IOP and PixelSpacing from your shared 3D projection code. I've also wasted 5 GPU hours running the entire public train… a couple of times :__P
\nhttps://www.kaggle.com/code/hengck23/2d-to-3d-projection-for-dicom
\n` xx,yy,zz = [],[],[]
\n for i in range(len(dicom)):
\n sx,sy,sz = [float(v) for v in dicom[i].ImagePositionPatient]
\n o0, o1, o2, o3, o4, o5 = [float(v) for v in dicom[i].ImageOrientationPatient]
\n delx,dely = dicom[i].PixelSpacing
x,y all_centers[i]:\n xx.append(o0lx*x + o3ly*y + sx)\n yy.append(o1lx*x + o4ly*y + sy)\n zz.append(o2lx*x + o5ly*y + sz)`\nFYI, I directly used these DICOM tags: PixelSpacing, ImageOrientationPatient, ImagePositionPatient, Rows, Columns.
\nThese tags are used by dcm.get(), which allows missing values: SliceThinkness, RescaleSlope, RescaleIntercept.
One more submission error. I'm now pretty sure that there is something wrong with IPP, IOP or PixelSpacing with some Sagittal T2 file/folder.
", "rawMarkdown": "One more submission error. I'm now pretty sure that there is something wrong with IPP, IOP or PixelSpacing with some Sagittal T2 file/folder.", "votes": null }, { "id": "3005707", "postDate": "10/03/2024 08:10:37", "content": "\nyou are saying that all test axial-t2 have IOP,PixelSpacing,SliceThickness and the problem only occurs for sagittal-t2?
", "rawMarkdown": "sacuscreed \n\nyou are saying that all test axial-t2 have IOP,PixelSpacing,SliceThickness and the problem only occurs for sagittal-t2?", "votes": null }, { "id": "3005711", "postDate": "10/03/2024 08:30:27", "content": "My code started to give errors when I added segmented level coordinates to Sagittal T2 to 3D projection processing,. No errors projecting all other MRI segmented coordinates. So may be we should focus on Sagittal T2 to find the problem (no idea how many of them). But right now I'm not sure of nothing, I did other apparently trivial changes to the code at same time. Few days to finish and few submissions available. Tryng to define the correct way to solve it.
", "rawMarkdown": "My code started to give errors when I added segmented level coordinates to Sagittal T2 to 3D projection processing,. No errors projecting all other MRI segmented coordinates. So may be we should focus on Sagittal T2 to find the problem (no idea how many of them). But right now I'm not sure of nothing, I did other apparently trivial changes to the code at same time. Few days to finish and few submissions available. Tryng to define the correct way to solve it.", "votes": null }, { "id": "3005895", "postDate": "10/03/2024 13:32:24", "content": "I think this is the way. If the tag does not exist, it will return None instead of error.
", "rawMarkdown": "I think this is the way. If the tag does not exist, it will return None instead of error.", "votes": null }, { "id": "3005919", "postDate": "10/03/2024 14:00:14", "content": "I didn't understood at first read. So if the tag doen't exist and you try acces it directly it raises error but with dcm.get() is like a try/except.
\nIf that's the case thank you very much and I'll use if I get another submission error.
Oh, I only picked 1 of Sagittal T2/STIR, 1 of Sagittal T1, 1 of Axial T2. Probably there are bug series I skipped.
In general, I believe the IPP, IOP, and PixelSpacing must all be present. If any of these are missing, it’s impossible to accurately map the image to physical coordinates. It’s fine to handle this with a try-except block to skip such problematic cases.
\n@hengck23 , The SliceThickness attribute is not always guaranteed to be present. Using dcm.SliceThickness may cause errors. I used dcm.get('SliceThickness', 1).
That makes sense, now I remember have changed also process all MRI available, not only first one. Then may be would be safe taking the first one available only.
", "rawMarkdown": "That makes sense, now I remember have changed also process all MRI available, not only first one. Then may be would be safe taking the first one available only.", "votes": null }, { "id": "3005975", "postDate": "10/03/2024 15:23:55", "content": "Hi, @hengck23,
\nI have had the similar issue and my workaround was to wrap the getitem in try except:
\n:\n arr, study_id, series_id, ipp, iop\n Exception e:\n , , , , \n\n\n torch.no_grad():\n i, batch (tqdm(test_dl, desc=)):\n :\n image, study_id, series_id, ipp, iop = batch\n study_id:\n \nIt is not the most elegant solution and might not work for you since you guys fighting for every inch, but hope it serves well for others.
", "rawMarkdown": "Hi, @hengck23, \n\nI have had the similar issue and my workaround was to wrap the __getitem__ in try except:\n```python\ntry:\n return arr, study_id, series_id, ipp, iop\nexcept Exception as e:\n return 0, 0, 0, 0, 0\n\n\nwith torch.no_grad():\n for i, batch in enumerate(tqdm(test_dl, desc='Predicting...')):\n try:\n image, study_id, series_id, ipp, iop = batch\n if not study_id:\n continue\n```\n\nIt is not the most elegant solution and might not work for you since you guys fighting for every inch, but hope it serves well for others.", "votes": null }, { "id": "3006039", "postDate": "10/03/2024 16:50:03", "content": "Must be last minut panic but I don't undestand anything. Well, at least I can confirm that IPP, IOP and PixelSpacing of evrything in hidden test is safe. If there is something wrong must be with another field.
\nEDIT: I've also optimized memmory usage. May be loading evreything, 3D coordinates included caused not notified OOM in my case.
", "rawMarkdown": "Must be last minut panic but I don't undestand anything. Well, at least I can confirm that IPP, IOP and PixelSpacing of evrything in hidden test is safe. If there is something wrong must be with another field.\n\nEDIT: I've also optimized memmory usage. May be loading evreything, 3D coordinates included caused not notified OOM in my case.", "votes": null }, { "id": "3006421", "postDate": "10/04/2024 06:07:29", "content": "I have the same problem.
\nI use try-except block.
", "rawMarkdown": "I have the same problem.\n\nI use try-except block.", "votes": null }, { "id": "3006512", "postDate": "10/04/2024 08:56:52", "content": "\n\nI didn't understood at first read. So if the tag doen't exist and you try acces it directly it raises error but with dcm.get() is like a try/except.
\n
Yes, exactly. So if it returns None instead of some value, I handled it by try different approach instead for just that paritcular one. basically it is just a bit easier to handle try/except.
", "rawMarkdown": ">I didn't understood at first read. So if the tag doen't exist and you try acces it directly it raises error but with dcm.get() is like a try/except.\n\nYes, exactly. So if it returns None instead of some value, I handled it by try different approach instead for just that paritcular one. basically it is just a bit easier to handle try/except.", "votes": null }, { "id": "3008250", "postDate": "10/06/2024 11:54:53", "content": "\"Your notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset See more debugging tips\" An error that doen't shows when processing the entire training set… is very frustating. I've checked everything, no idea what can explode in test…
", "rawMarkdown": "\"Your notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset See more debugging tips\" An error that doen't shows when processing the entire training set... is very frustating. I've checked everything, no idea what can explode in test...", "votes": null } ], "comments": [ { "id": 3005390, "author_name": "sacuscreed", "author_url": "", "post_date": "10/02/2024 20:46:15", "content": "I think is a sneaky bug in my case. But I've already lost 3 submissions since I've included Sagittal T2 3D projections. If that's the case I'm only using IPP,IOP and PixelSpacing from your shared 3D projection code. I've also wasted 5 GPU hours running the entire public train… a couple of times :__P
\nhttps://www.kaggle.com/code/hengck23/2d-to-3d-projection-for-dicom
\n` xx,yy,zz = [],[],[]
\n for i in range(len(dicom)):
\n sx,sy,sz = [float(v) for v in dicom[i].ImagePositionPatient]
\n o0, o1, o2, o3, o4, o5 = [float(v) for v in dicom[i].ImageOrientationPatient]
\n delx,dely = dicom[i].PixelSpacing
x,y all_centers[i]:\n xx.append(o0lx*x + o3ly*y + sx)\n yy.append(o1lx*x + o4ly*y + sy)\n zz.append(o2lx*x + o5ly*y + sz)`\nOne more submission error. I'm now pretty sure that there is something wrong with IPP, IOP or PixelSpacing with some Sagittal T2 file/folder.
", "votes": null, "replies": [ { "id": 3005707, "author_name": "hengck23", "author_url": "", "post_date": "10/03/2024 08:10:37", "content": "\nyou are saying that all test axial-t2 have IOP,PixelSpacing,SliceThickness and the problem only occurs for sagittal-t2?
", "votes": null, "replies": [ { "id": 3005711, "author_name": "sacuscreed", "author_url": "", "post_date": "10/03/2024 08:30:27", "content": "My code started to give errors when I added segmented level coordinates to Sagittal T2 to 3D projection processing,. No errors projecting all other MRI segmented coordinates. So may be we should focus on Sagittal T2 to find the problem (no idea how many of them). But right now I'm not sure of nothing, I did other apparently trivial changes to the code at same time. Few days to finish and few submissions available. Tryng to define the correct way to solve it.
", "votes": null, "replies": [] } ] }, { "id": 3005957, "author_name": "together", "author_url": "", "post_date": "10/03/2024 14:53:48", "content": "Oh, I only picked 1 of Sagittal T2/STIR, 1 of Sagittal T1, 1 of Axial T2. Probably there are bug series I skipped.
In general, I believe the IPP, IOP, and PixelSpacing must all be present. If any of these are missing, it’s impossible to accurately map the image to physical coordinates. It’s fine to handle this with a try-except block to skip such problematic cases.
\n@hengck23 , The SliceThickness attribute is not always guaranteed to be present. Using dcm.SliceThickness may cause errors. I used dcm.get('SliceThickness', 1).
That makes sense, now I remember have changed also process all MRI available, not only first one. Then may be would be safe taking the first one available only.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3005430, "author_name": "together", "author_url": "", "post_date": "10/02/2024 22:04:35", "content": "FYI, I directly used these DICOM tags: PixelSpacing, ImageOrientationPatient, ImagePositionPatient, Rows, Columns.
\nThese tags are used by dcm.get(), which allows missing values: SliceThinkness, RescaleSlope, RescaleIntercept.
I think this is the way. If the tag does not exist, it will return None instead of error.
", "votes": null, "replies": [] }, { "id": 3005919, "author_name": "sacuscreed", "author_url": "", "post_date": "10/03/2024 14:00:14", "content": "I didn't understood at first read. So if the tag doen't exist and you try acces it directly it raises error but with dcm.get() is like a try/except.
\nIf that's the case thank you very much and I'll use if I get another submission error.
\n\nI didn't understood at first read. So if the tag doen't exist and you try acces it directly it raises error but with dcm.get() is like a try/except.
\n
Yes, exactly. So if it returns None instead of some value, I handled it by try different approach instead for just that paritcular one. basically it is just a bit easier to handle try/except.
", "votes": null, "replies": [] } ] } ] }, { "id": 3005975, "author_name": "sergiosaharovskiy", "author_url": "", "post_date": "10/03/2024 15:23:55", "content": "Hi, @hengck23,
\nI have had the similar issue and my workaround was to wrap the getitem in try except:
\n:\n arr, study_id, series_id, ipp, iop\n Exception e:\n , , , , \n\n\n torch.no_grad():\n i, batch (tqdm(test_dl, desc=)):\n :\n image, study_id, series_id, ipp, iop = batch\n study_id:\n \nIt is not the most elegant solution and might not work for you since you guys fighting for every inch, but hope it serves well for others.
", "votes": null, "replies": [] }, { "id": 3006039, "author_name": "sacuscreed", "author_url": "", "post_date": "10/03/2024 16:50:03", "content": "Must be last minut panic but I don't undestand anything. Well, at least I can confirm that IPP, IOP and PixelSpacing of evrything in hidden test is safe. If there is something wrong must be with another field.
\nEDIT: I've also optimized memmory usage. May be loading evreything, 3D coordinates included caused not notified OOM in my case.
", "votes": null, "replies": [] }, { "id": 3006421, "author_name": "chihantsai", "author_url": "", "post_date": "10/04/2024 06:07:29", "content": "I have the same problem.
\nI use try-except block.
", "votes": null, "replies": [] }, { "id": 3008250, "author_name": "sacuscreed", "author_url": "", "post_date": "10/06/2024 11:54:53", "content": "\"Your notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset See more debugging tips\" An error that doen't shows when processing the entire training set… is very frustating. I've checked everything, no idea what can explode in test…
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3005279": "My data reader passed all public kaggle train data but is stuck with \"notebook exception error\" for a few days.\nI am running out of submissions and it is now impossible for me to \"submit and debug\".\n\nI wonder if anyone can help it identify which hidden dicom test tag is giving problem?\n\n- ImagePositionPatient : ok for train and test\n- SpacingBetweenSlices: ok for train, problem for test[1]\n- pixel_array.shape : ok for train and test\n\nupdate:\nthese are tested ok for test and train:\n- ImageOrientationPatient: ok for train, ok for test\n- PixelSpacing: ok for train, ok for test\n\n\nfor the following, I never submitted without try-catch, so I do not know if they are issues \n- SliceThickness: ok for train, ??? for test\n- SliceLocation: ok for train, ??? for test\n\n\nalso, verified problems: \n- are some empty dicom folder in hidden test. \n- images can be of different shapes within a dicom folder.\n\nIf there are other issues, please let me know. thanks.\n\nreference:\n[1] https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/528284", "3005390": "I think is a sneaky bug in my case. But I've already lost 3 submissions since I've included Sagittal T2 3D projections. If that's the case I'm only using IPP,IOP and PixelSpacing from your shared 3D projection code. I've also wasted 5 GPU hours running the entire public train... a couple of times :__P\n\n https://www.kaggle.com/code/hengck23/2d-to-3d-projection-for-dicom\n\n` xx,yy,zz = [],[],[]\n for i in range(len(dicom)):\n sx,sy,sz = [float(v) for v in dicom[i].ImagePositionPatient]\n o0, o1, o2, o3, o4, o5 = [float(v) for v in dicom[i].ImageOrientationPatient]\n delx,dely = dicom[i].PixelSpacing\n\n for x,y in all_centers[i]:\n xx.append(o0*delx*x + o3*dely*y + sx)\n yy.append(o1*delx*x + o4*dely*y + sy)\n zz.append(o2*delx*x + o5*dely*y + sz)`", "3005430": "FYI, I directly used these DICOM tags: PixelSpacing, ImageOrientationPatient, ImagePositionPatient, Rows, Columns.\nThese tags are used by `dcm.get()`, which allows missing values: SliceThinkness, RescaleSlope, RescaleIntercept.", "3005434": "One more submission error. I'm now pretty sure that there is something wrong with IPP, IOP or PixelSpacing with some Sagittal T2 file/folder.", "3005707": "sacuscreed \n\nyou are saying that all test axial-t2 have IOP,PixelSpacing,SliceThickness and the problem only occurs for sagittal-t2?", "3005711": "My code started to give errors when I added segmented level coordinates to Sagittal T2 to 3D projection processing,. No errors projecting all other MRI segmented coordinates. So may be we should focus on Sagittal T2 to find the problem (no idea how many of them). But right now I'm not sure of nothing, I did other apparently trivial changes to the code at same time. Few days to finish and few submissions available. Tryng to define the correct way to solve it.", "3005895": "I think this is the way. If the tag does not exist, it will return None instead of error.", "3005919": "I didn't understood at first read. So if the tag doen't exist and you try acces it directly it raises error but with dcm.get() is like a try/except.\nIf that's the case thank you very much and I'll use if I get another submission error.", "3005957": "Oh, I only picked 1 of `Sagittal T2/STIR`, 1 of `Sagittal T1`, 1 of `Axial T2`. Probably there are bug series I skipped. \n\nIn general, I believe the IPP, IOP, and PixelSpacing must all be present. If any of these are missing, it’s impossible to accurately map the image to physical coordinates. It’s fine to handle this with a try-except block to skip such problematic cases.\n\n@hengck23 , The SliceThickness attribute is not always guaranteed to be present. Using `dcm.SliceThickness` may cause errors. I used `dcm.get('SliceThickness', 1)`.", "3005963": "That makes sense, now I remember have changed also process all MRI available, not only first one. Then may be would be safe taking the first one available only.", "3005975": "Hi, @hengck23, \n\nI have had the similar issue and my workaround was to wrap the __getitem__ in try except:\n```python\ntry:\n return arr, study_id, series_id, ipp, iop\nexcept Exception as e:\n return 0, 0, 0, 0, 0\n\n\nwith torch.no_grad():\n for i, batch in enumerate(tqdm(test_dl, desc='Predicting...')):\n try:\n image, study_id, series_id, ipp, iop = batch\n if not study_id:\n continue\n```\n\nIt is not the most elegant solution and might not work for you since you guys fighting for every inch, but hope it serves well for others.", "3006039": "Must be last minut panic but I don't undestand anything. Well, at least I can confirm that IPP, IOP and PixelSpacing of evrything in hidden test is safe. If there is something wrong must be with another field.\n\nEDIT: I've also optimized memmory usage. May be loading evreything, 3D coordinates included caused not notified OOM in my case.", "3006421": "I have the same problem.\n\nI use try-except block.", "3006512": ">I didn't understood at first read. So if the tag doen't exist and you try acces it directly it raises error but with dcm.get() is like a try/except.\n\nYes, exactly. So if it returns None instead of some value, I handled it by try different approach instead for just that paritcular one. basically it is just a bit easier to handle try/except.", "3008250": "\"Your notebook hit an unhandled error while rerunning your code. Note that the hidden dataset can be larger/smaller/different than the public dataset See more debugging tips\" An error that doen't shows when processing the entire training set... is very frustating. I've checked everything, no idea what can explode in test..." }, "source": "meta" }