NIH Chest X-rays, one of the largest X-ray images dataset available, seems to be in the spotlight now. As it's been recently reconfirmed by the host of the competition here, the entire train (at least) and test (at most) data is 100% relabeled data from the NIH CXR. Therefore one should be careful to avoid overfitting thus making more harm than good from using this external data either for pretraining models or as an additional data source.
\nAnother point of concern is time and resources, as training a single EfficientNetB4 on TPU with TFRecordDataset of 112,120 samples (and images downscaled to 600x600) for 20 epochs must take over 2.5 hours. Skipping serialization and going with from_tensor_slices must be at least 3-4 more time-consuming.
So I've decided to do a bit of preprocessing and serialized this dataset to TFRecords, which are now available in 600x600 image quality in this dataset. To make your custom TFRecords by filtering out the duplicates, tuning image quality, etc., please consider using this starter notebook.
I'd like to ask you to share your personal experience on using NIH CXR (or other external datasets) either for pretraining models or enlarging the training dataset and concerns about overfitting.
\nHappy coding!
", "messages": [ { "id": 1216973, "postDate": "2021-02-24T17:00:45.357Z", "content": "NIH Chest X-rays, one of the largest X-ray images dataset available, seems to be in the spotlight now. As it's been recently reconfirmed by the host of the competition here, the entire train (at least) and test (at most) data is 100% relabeled data from the NIH CXR. Therefore one should be careful to avoid overfitting thus making more harm than good from using this external data either for pretraining models or as an additional data source.
\nAnother point of concern is time and resources, as training a single EfficientNetB4 on TPU with TFRecordDataset of 112,120 samples (and images downscaled to 600x600) for 20 epochs must take over 2.5 hours. Skipping serialization and going with from_tensor_slices must be at least 3-4 more time-consuming.
So I've decided to do a bit of preprocessing and serialized this dataset to TFRecords, which are now available in 600x600 image quality in this dataset. To make your custom TFRecords by filtering out the duplicates, tuning image quality, etc., please consider using this starter notebook.
I'd like to ask you to share your personal experience on using NIH CXR (or other external datasets) either for pretraining models or enlarging the training dataset and concerns about overfitting.
\nHappy coding!
", "rawMarkdown": "### Hello!\n**[NIH Chest X-rays](https://www.kaggle.com/nih-chest-xrays/data)**, one of the largest X-ray images dataset available, seems to be in the spotlight now. As it's been recently reconfirmed by the host of the competition **[here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/221808)**, the entire train (at least) and test (at most) data is 100% relabeled data from the NIH CXR. Therefore one should be careful to avoid overfitting thus making more harm than good from using this external data either for pretraining models or as an additional data source.\n\nAnother point of concern is time and resources, as training a single `EfficientNetB4` on TPU with TFRecordDataset of `112,120` samples (and images downscaled to `600x600`) for 20 epochs must take over 2.5 hours. Skipping serialization and going with `from_tensor_slices` must be at least 3-4 more time-consuming. \n\nSo I've decided to do a bit of preprocessing and serialized this dataset to TFRecords, which are now available in `600x600` image quality in this **[dataset](https://www.kaggle.com/nickuzmenkov/nih-chest-xrays-tfrecords)**. To make your custom TFRecords by filtering out the duplicates, tuning image quality, etc., please consider using this **[starter notebook](https://www.kaggle.com/nickuzmenkov/ranzcr-clip-add-nih-chest-x-rays-tfrecords)**.\n\nI'd like to ask you to share your personal experience on using NIH CXR (or other external datasets) either for pretraining models or enlarging the training dataset and concerns about overfitting.\n\nHappy coding!\n\n", "votes": 7 } ], "comments": [], "raw_markdown_by_id": { "1216973": "### Hello!\n**[NIH Chest X-rays](https://www.kaggle.com/nih-chest-xrays/data)**, one of the largest X-ray images dataset available, seems to be in the spotlight now. As it's been recently reconfirmed by the host of the competition **[here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/221808)**, the entire train (at least) and test (at most) data is 100% relabeled data from the NIH CXR. Therefore one should be careful to avoid overfitting thus making more harm than good from using this external data either for pretraining models or as an additional data source.\n\nAnother point of concern is time and resources, as training a single `EfficientNetB4` on TPU with TFRecordDataset of `112,120` samples (and images downscaled to `600x600`) for 20 epochs must take over 2.5 hours. Skipping serialization and going with `from_tensor_slices` must be at least 3-4 more time-consuming. \n\nSo I've decided to do a bit of preprocessing and serialized this dataset to TFRecords, which are now available in `600x600` image quality in this **[dataset](https://www.kaggle.com/nickuzmenkov/nih-chest-xrays-tfrecords)**. To make your custom TFRecords by filtering out the duplicates, tuning image quality, etc., please consider using this **[starter notebook](https://www.kaggle.com/nickuzmenkov/ranzcr-clip-add-nih-chest-x-rays-tfrecords)**.\n\nI'd like to ask you to share your personal experience on using NIH CXR (or other external datasets) either for pretraining models or enlarging the training dataset and concerns about overfitting.\n\nHappy coding!\n\n" } }