As you may know, TFRecord stores the data in a binary file format and can have a significant impact on the performance of training time of your model. Binary data takes up less space on disk, takes less time to copy and can be read much more efficiently from disk.[1]
\nAnother advantage of using TFRecord is that you can leverage TPU for training your model much faster, otherwise reading inputs will be a bottleneck for TPU, since TPUs are very fast.
\nIn order to convert DICOMs to TFRecord I have created a notebook which you can find it here. But don't worry, I have created a dataset from output files, which you can find it here, and import it to your kernel and use it for faster training.
\nThere is also a notebook, which I have provided an example of how use it, here.
\nNote that, the dataset has 2 files: 1 for training with 465 samples and 1 for validation with 117 samples. All samples stored in the (128,128,32,4) shape. Last dimension (channel) is for modalities. So it includes FLAIR, T1W and… .
Also note that you can merge all samples (train and validation samples) in 1 files by something like dataset=train_set.concatenate(valid_set).
As you may know, TFRecord stores the data in a binary file format and can have a significant impact on the performance of training time of your model. Binary data takes up less space on disk, takes less time to copy and can be read much more efficiently from disk.[1]
\nAnother advantage of using TFRecord is that you can leverage TPU for training your model much faster, otherwise reading inputs will be a bottleneck for TPU, since TPUs are very fast.
\nIn order to convert DICOMs to TFRecord I have created a notebook which you can find it here. But don't worry, I have created a dataset from output files, which you can find it here, and import it to your kernel and use it for faster training.
\nThere is also a notebook, which I have provided an example of how use it, here.
\nNote that, the dataset has 2 files: 1 for training with 465 samples and 1 for validation with 117 samples. All samples stored in the (128,128,32,4) shape. Last dimension (channel) is for modalities. So it includes FLAIR, T1W and… .
Also note that you can merge all samples (train and validation samples) in 1 files by something like dataset=train_set.concatenate(valid_set).
No. Just wanted to get everything ready.
", "rawMarkdown": "No. Just wanted to get everything ready. " }, { "id": 1495411, "postDate": "2021-08-29T14:41:13.927Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 1495189, "author_name": "", "author_url": "", "post_date": "2021-08-29T11:31:16.077000", "content": "", "votes": 1, "replies": [ { "id": 1495295, "author_name": "Kaveh Shahhosseini", "author_url": "", "post_date": "2021-08-29T12:54:24.157000", "content": "No. Just wanted to get everything ready.
", "votes": 0, "replies": [] }, { "id": 1495411, "author_name": "", "author_url": "", "post_date": "2021-08-29T14:41:13.927000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1493370": "As you may know, TFRecord stores the data in a binary file format and can have a significant impact on the performance of training time of your model. **Binary data takes up less space on disk, takes less time to copy and can be read much more efficiently from disk**.[[1](https://medium.com/mostly-ai/tensorflow-records-what-they-are-and-how-to-use-them-c46bc4bbb564)]\n\nAnother **advantage of using TFRecord is that you can leverage TPU** for training your model much faster, otherwise reading inputs will be a bottleneck for TPU, since TPUs are very fast. \n\nIn order to convert DICOMs to TFRecord I have created a notebook which you can find it [here](https://www.kaggle.com/kavehshahhosseini/rsna-brain-tumor-convert-dicom-to-tfrecord). But don't worry, I have created a dataset from output files, which you can find it [here](https://www.kaggle.com/kavehshahhosseini/rsna-brain-tumor-classification-tfrecords), and import it to your kernel and use it for faster training. \n\nThere is also a notebook, which I have provided an example of how use it, [here](https://www.kaggle.com/kavehshahhosseini/how-to-load-and-use-data). \n\nNote that, the dataset has 2 files: 1 for training with 465 samples and 1 for validation with 117 samples. All samples stored in the `(128,128,32,4)` shape. Last dimension (channel) is for modalities. So it includes FLAIR, T1W and... . \n\nAlso note that you can merge all samples (train and validation samples) in 1 files by something like `dataset=train_set.concatenate(valid_set)`.", "1495189": "" } }