\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2841366%2F6d456b529856448fbca7c4480f4a0bb1%2FID00012637202177665765362.gif?generation=1595919768770348&alt=media\")
I went through a few notebooks but didn't find out how they are using image data for training purposes. can anyone please explain what is going on?
", "messages": [ { "id": 947775, "postDate": "2020-07-27T13:30:48.540Z", "content": "I went through a few notebooks but didn't find out how they are using image data for training purposes. can anyone please explain what is going on?
", "rawMarkdown": "I went through a few notebooks but didn't find out how they are using image data for training purposes. can anyone please explain what is going on?", "votes": 10 }, { "id": 948672, "postDate": "2020-07-28T06:28:38.930Z", "content": "The images are DICOM images of one single CT Scan. CT scans are not images rather videos, so the dataset contains frames of a single CT scan video.
", "rawMarkdown": "The images are DICOM images of one single CT Scan. CT scans are not images rather videos, so the dataset contains frames of a single CT scan video.", "votes": 1, "replies": [ { "id": 948695, "postDate": "2020-07-28T06:43:07.073Z", "content": "That's not correct, CT scans (as long as they are not sequential) are three-dimensional images when stacked.
", "rawMarkdown": "That's not correct, CT scans (as long as they are not sequential) are three-dimensional images when stacked.", "votes": 2, "replies": [ { "id": 948723, "postDate": "2020-07-28T07:03:01.917Z", "content": "Thanks for the reply @hojijoji , i should correct my self. The images in the dataset are images of a single CT scan. CT scans are, as @hojijoji said are 3D images when stacked, but in the dataset all the stacked images are individually present. \nFor Example, this GIF shows the CT scan with Patient Id ID00012637202177665765362 -
\n
Thank you for replying @hojijoji , I should correct myself, The dataset contains images of a single CT scan, and CT scans, as @hojijoji said are 3 Dimensional images when stacked. In the given dataset we have individual images of a single CT scan of a Patient.
\n\nFor example, This is a GIF of CT scan of Patient ID00012637202177665765362,
\n\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2841366%2Fda1ab4df2955500ba5c1f33541e4c739%2FID00012637202177665765362.gif?generation=1595931619408140&alt=media\")
@vijayyadav22 , \nas @hojijoji mentioned in the comment section of this discusion, the image data of this competition has many flaws... heterogenous, totally different scan protocolls, some scans have even sequential images (meaning: not aquired within a single breathold maneuver).
\n\nI am completely on the same opinion (especially the second one):
\n\n\n", "rawMarkdown": "@vijayyadav22 , \nas @hojijoji mentioned in the comment section of [this discusion](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/165044), the image data of this competition has many flaws... heterogenous, totally different scan protocolls, some scans have even sequential images (meaning: not aquired within a single breathold maneuver).\n\nI am completely on the same opinion (especially the second one):\n\n> (1) The best performing algorithm will perform worse compared to what would be possible if clean unmodified data would have been used (data from the scanner).\n(2) The best performing algorithm wins not because it is the best in predicting FVC but because it is good in spotting flawed scans. E.g. there is a corrupted scan in the hidden test set and one algorithm wins because it detects the corruption and predicts the population average. An algorithm detects the wrong orientation etc. A lot of development\nmay go into anticipating data corruptions and artifacts rather than the real thing.\n\n", "votes": 1, "replies": [ { "id": 949301, "postDate": "2020-07-28T14:40:29.037Z", "content": "(1) The best performing algorithm will perform worse compared to what would be possible if clean unmodified data would have been used (data from the scanner).\n (2) The best performing algorithm wins not because it is the best in predicting FVC but because it is good in spotting flawed scans. E.g. there is a corrupted scan in the hidden test set and one algorithm wins because it detects the corruption and predicts the population average. An algorithm detects the wrong orientation etc. A lot of development\n may go into anticipating data corruptions and artifacts rather than the real thing.
\n
Sir, can you please explain to me how to use these image data in training the model?
", "rawMarkdown": "Sir, can you please explain to me how to use these image data in training the model?" }, { "id": 952196, "postDate": "2020-07-30T17:46:09.987Z", "content": "if you know how to work with 3D Conv nets you can easily use these images for training.\nbut first, you have to pre-process these images as already mentioned above ~100x512x512 is a huge number and 3D convnet cannot handle that much data so first you have to reduce this number say 30x150x150 although it's still big.\nOnce you have these preprocessed data you can a model with two parallel branches\nBranch 1 -- will process and learn on tabular data (Dense layers)\nBranch 2 -- will process image data (3D Conv net)\nthen you can concatenate these branches together to form a single model use Keras functional API\nfollowing is my notebook which goes through preprocessing the Dicom data and saves the processed training images in npy format.\npreprocessing dicom data\n it has reduced each scan to a size of 30x150x150 which is still huge but you can use this approach to reduce the size as much as you want
", "rawMarkdown": "if you know how to work with 3D Conv nets you can easily use these images for training.\nbut first, you have to pre-process these images as already mentioned above ~100x512x512 is a huge number and 3D convnet cannot handle that much data so first you have to reduce this number say 30x150x150 although it's still big.\nOnce you have these preprocessed data you can a model with two parallel branches\nBranch 1 -- will process and learn on tabular data (Dense layers)\nBranch 2 -- will process image data (3D Conv net)\nthen you can concatenate these branches together to form a single model use Keras functional API\nfollowing is my notebook which goes through preprocessing the Dicom data and saves the processed training images in npy format.\n[preprocessing dicom data](https://www.kaggle.com/zainahmad/preprocessing-the-dicom-data)\n it has reduced each scan to a size of 30x150x150 which is still huge but you can use this approach to reduce the size as much as you want", "votes": 3 } ] }, { "id": 949297, "postDate": "2020-07-28T14:39:02.287Z", "content": "Thank you everyone, this discussion was very helpful in understanding the image but still, I do not understand whether image data was used in model training or not?
", "rawMarkdown": "Thank you everyone, this discussion was very helpful in understanding the image but still, I do not understand whether image data was used in model training or not? ", "replies": [ { "id": 949302, "postDate": "2020-07-28T14:42:19.833Z", "content": "The only public one with lots of upvotes is @carlossouza: https://www.kaggle.com/carlossouza/end-to-end-model-ct-scans-tabular
\n\nThough I suspect lots of us (myself including) are in the background trying to build models that use image data and improve on the score.
", "rawMarkdown": "The only public one with lots of upvotes is @carlossouza: https://www.kaggle.com/carlossouza/end-to-end-model-ct-scans-tabular\n\nThough I suspect lots of us (myself including) are in the background trying to build models that use image data and improve on the score.", "votes": 3 }, { "id": 949315, "postDate": "2020-07-28T14:54:22.743Z", "content": "The submission you made does include image data for training?
", "rawMarkdown": "The submission you made does include image data for training?" }, { "id": 949365, "postDate": "2020-07-28T15:33:40.513Z", "content": "not got one that improves on the tabular data. The problem is that the image data has such high dimensionality (~100x512x512=25,000,000 features!) that using the image data increases the likelihood of simply overfitting the training data. That's why approaches like the one above try to reduce the dimensionality first.
", "rawMarkdown": "not got one that improves on the tabular data. The problem is that the image data has such high dimensionality (~100x512x512=25,000,000 features!) that using the image data increases the likelihood of simply overfitting the training data. That's why approaches like the one above try to reduce the dimensionality first.", "votes": 1 }, { "id": 949376, "postDate": "2020-07-28T15:43:06.997Z", "content": "Thank you James for helping me out.
", "rawMarkdown": "Thank you James for helping me out." } ] }, { "id": 986456, "postDate": "2020-08-26T14:12:29.413Z", "rawMarkdown": "", "isDeleted": true }, { "id": 980132, "postDate": "2020-08-21T10:52:20.680Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 980200, "postDate": "2020-08-21T11:48:33.137Z", "content": "Firstly, The image data is just a single 3D scans.
\nSecondly, The CT Scans were taken on Week 0.
\nSince some training samples do not have FVC value for Week 0 then in that case you have to apply some measures - either you use the data and use the nearest week FVC as the target value or any other method. Also you can opt to not train over those samples which does not contain Week0 FVC value.
Yes. Help needed!
\nI can not understand the same point. How can you use a single 3D scan from the very first day with all the records of a particular patient in train.csv?
", "rawMarkdown": "Yes. Help needed! \n\nI can not understand the same point. How can you use a single 3D scan from the very first day with all the records of a particular patient in **train.csv**?" }, { "id": 995298, "postDate": "2020-09-02T10:32:46.550Z", "content": "Here the task is to predict the FVC values for the final 3 measurements of the patient. The challenge is to find a pattern in the first CT Scan in order to predict the FVC values. for example, you get your first CT scan of your lungs, the doctor uses it to predict your final three FVC measurements to be prepared in advance for best possible results and to provide the best possible treatment.
", "rawMarkdown": "Here the task is to predict the FVC values for the final 3 measurements of the patient. The challenge is to find a pattern in the first CT Scan in order to predict the FVC values. for example, you get your first CT scan of your lungs, the doctor uses it to predict your final three FVC measurements to be prepared in advance for best possible results and to provide the best possible treatment.\n\n" } ] }, { "id": 953613, "postDate": "2020-07-31T23:15:09.923Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 948672, "author_name": "Sambhav Garg", "author_url": "", "post_date": "2020-07-28T06:28:38.930000", "content": "The images are DICOM images of one single CT Scan. CT scans are not images rather videos, so the dataset contains frames of a single CT scan video.
", "votes": 1, "replies": [ { "id": 948695, "author_name": "Johannes Hofmanninger", "author_url": "", "post_date": "2020-07-28T06:43:07.073000", "content": "That's not correct, CT scans (as long as they are not sequential) are three-dimensional images when stacked.
", "votes": 2, "replies": [ { "id": 948723, "author_name": "Sambhav Garg", "author_url": "", "post_date": "2020-07-28T07:03:01.917000", "content": "Thanks for the reply @hojijoji , i should correct my self. The images in the dataset are images of a single CT scan. CT scans are, as @hojijoji said are 3D images when stacked, but in the dataset all the stacked images are individually present. \nFor Example, this GIF shows the CT scan with Patient Id ID00012637202177665765362 -
\n
Thank you for replying @hojijoji , I should correct myself, The dataset contains images of a single CT scan, and CT scans, as @hojijoji said are 3 Dimensional images when stacked. In the given dataset we have individual images of a single CT scan of a Patient.
\n\nFor example, This is a GIF of CT scan of Patient ID00012637202177665765362,
\n\n
@vijayyadav22 , \nas @hojijoji mentioned in the comment section of this discusion, the image data of this competition has many flaws... heterogenous, totally different scan protocolls, some scans have even sequential images (meaning: not aquired within a single breathold maneuver).
\n\nI am completely on the same opinion (especially the second one):
\n\n\n", "votes": 1, "replies": [ { "id": 949301, "author_name": "Vijay Yadav", "author_url": "", "post_date": "2020-07-28T14:40:29.037000", "content": "(1) The best performing algorithm will perform worse compared to what would be possible if clean unmodified data would have been used (data from the scanner).\n (2) The best performing algorithm wins not because it is the best in predicting FVC but because it is good in spotting flawed scans. E.g. there is a corrupted scan in the hidden test set and one algorithm wins because it detects the corruption and predicts the population average. An algorithm detects the wrong orientation etc. A lot of development\n may go into anticipating data corruptions and artifacts rather than the real thing.
\n
Sir, can you please explain to me how to use these image data in training the model?
", "votes": 0, "replies": [] }, { "id": 952196, "author_name": "Zain Ahmad", "author_url": "", "post_date": "2020-07-30T17:46:09.987000", "content": "if you know how to work with 3D Conv nets you can easily use these images for training.\nbut first, you have to pre-process these images as already mentioned above ~100x512x512 is a huge number and 3D convnet cannot handle that much data so first you have to reduce this number say 30x150x150 although it's still big.\nOnce you have these preprocessed data you can a model with two parallel branches\nBranch 1 -- will process and learn on tabular data (Dense layers)\nBranch 2 -- will process image data (3D Conv net)\nthen you can concatenate these branches together to form a single model use Keras functional API\nfollowing is my notebook which goes through preprocessing the Dicom data and saves the processed training images in npy format.\npreprocessing dicom data\n it has reduced each scan to a size of 30x150x150 which is still huge but you can use this approach to reduce the size as much as you want
", "votes": 3, "replies": [] } ] }, { "id": 949297, "author_name": "Vijay Yadav", "author_url": "", "post_date": "2020-07-28T14:39:02.287000", "content": "Thank you everyone, this discussion was very helpful in understanding the image but still, I do not understand whether image data was used in model training or not?
", "votes": 0, "replies": [ { "id": 949302, "author_name": "jameschapman19", "author_url": "", "post_date": "2020-07-28T14:42:19.833000", "content": "The only public one with lots of upvotes is @carlossouza: https://www.kaggle.com/carlossouza/end-to-end-model-ct-scans-tabular
\n\nThough I suspect lots of us (myself including) are in the background trying to build models that use image data and improve on the score.
", "votes": 3, "replies": [] }, { "id": 949315, "author_name": "Vijay Yadav", "author_url": "", "post_date": "2020-07-28T14:54:22.743000", "content": "The submission you made does include image data for training?
", "votes": 0, "replies": [] }, { "id": 949365, "author_name": "jameschapman19", "author_url": "", "post_date": "2020-07-28T15:33:40.513000", "content": "not got one that improves on the tabular data. The problem is that the image data has such high dimensionality (~100x512x512=25,000,000 features!) that using the image data increases the likelihood of simply overfitting the training data. That's why approaches like the one above try to reduce the dimensionality first.
", "votes": 1, "replies": [] }, { "id": 949376, "author_name": "Vijay Yadav", "author_url": "", "post_date": "2020-07-28T15:43:06.997000", "content": "Thank you James for helping me out.
", "votes": 0, "replies": [] } ] }, { "id": 986456, "author_name": "", "author_url": "", "post_date": "2020-08-26T14:12:29.413000", "content": "", "votes": 0, "replies": [] }, { "id": 980132, "author_name": "", "author_url": "", "post_date": "2020-08-21T10:52:20.680000", "content": "", "votes": 1, "replies": [ { "id": 980200, "author_name": "Sambhav Garg", "author_url": "", "post_date": "2020-08-21T11:48:33.137000", "content": "Firstly, The image data is just a single 3D scans.
\nSecondly, The CT Scans were taken on Week 0.
\nSince some training samples do not have FVC value for Week 0 then in that case you have to apply some measures - either you use the data and use the nearest week FVC as the target value or any other method. Also you can opt to not train over those samples which does not contain Week0 FVC value.
Yes. Help needed!
\nI can not understand the same point. How can you use a single 3D scan from the very first day with all the records of a particular patient in train.csv?
", "votes": 0, "replies": [] }, { "id": 995298, "author_name": "Sambhav Garg", "author_url": "", "post_date": "2020-09-02T10:32:46.550000", "content": "Here the task is to predict the FVC values for the final 3 measurements of the patient. The challenge is to find a pattern in the first CT Scan in order to predict the FVC values. for example, you get your first CT scan of your lungs, the doctor uses it to predict your final three FVC measurements to be prepared in advance for best possible results and to provide the best possible treatment.
", "votes": 0, "replies": [] } ] }, { "id": 953613, "author_name": "", "author_url": "", "post_date": "2020-07-31T23:15:09.923000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "947775": "I went through a few notebooks but didn't find out how they are using image data for training purposes. can anyone please explain what is going on?", "948672": "The images are DICOM images of one single CT Scan. CT scans are not images rather videos, so the dataset contains frames of a single CT scan video.", "948224": "@vijayyadav22 , \nas @hojijoji mentioned in the comment section of [this discusion](https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/discussion/165044), the image data of this competition has many flaws... heterogenous, totally different scan protocolls, some scans have even sequential images (meaning: not aquired within a single breathold maneuver).\n\nI am completely on the same opinion (especially the second one):\n\n> (1) The best performing algorithm will perform worse compared to what would be possible if clean unmodified data would have been used (data from the scanner).\n(2) The best performing algorithm wins not because it is the best in predicting FVC but because it is good in spotting flawed scans. E.g. there is a corrupted scan in the hidden test set and one algorithm wins because it detects the corruption and predicts the population average. An algorithm detects the wrong orientation etc. A lot of development\nmay go into anticipating data corruptions and artifacts rather than the real thing.\n\n", "949297": "Thank you everyone, this discussion was very helpful in understanding the image but still, I do not understand whether image data was used in model training or not? ", "986456": "", "980132": "", "953613": "" } }