{ "id": 208689, "title": "Dataset with Stratified-GroupKFold TFRecords", "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/208689", "author_name": "Prateek Mishra", "post_date": "2021-01-04T15:32:40.467000", "votes": 4, "comment_count": 0, "views": 0, "content": "

Hello all, While going through all the TPU notebooks created so far in this competition I found that they don't have a proper cross-validation setup. Since the dataset is highly imbalanced and Some patients have more than one record.
\nTo make things easy, I have created Datasets that contain image data for Kaggle's RANZCR-CLiP Competition. They are MulitLabel stratified Group KFold. The original jpegs have been resized using cv2.resize().

\n

All the Dataset can be found at : [ (128x128) , (256x256) , (384x384) , (512x512) ]

\n

A sample notebook that presents Stratified GroupKFold cross-validation and Efficient Net architecture getting trained using TPUs can be found here.

\n

Want to modify or check how this Dataset was created? See this Notebook.

\n

What do TFRecords contain?

\n
feature = {\n      'image'  :   _bytes_feature\n      'ETT - Abnormal'  :  _int64_feature\n      'ETT - Borderline'  :  _int64_feature\n      'ETT - Normal'  :  _int64_feature\n      'NGT - Abnormal'  :  _int64_feature\n      'NGT - Borderline'  :  _int64_feature\n      'NGT - Incompletely Imaged'  :  _int64_feature\n      'NGT - Normal'  :  _int64_feature\n      'CVC - Abnormal'  :  _int64_feature\n      'CVC - Borderline'  :  _int64_feature\n      'CVC - Normal'  :_int64_feature\n      'Swan Ganz Catheter Present'  :  _int64_feature\n      'StudyInstanceUID':   _bytes_feature\n      'PatientID'  : _bytes_feature\n    }\n
\n

Visualizing distribution of labels in each TFRecords

\n

\"\"

\n

Visualizing distribution of Sample across the TFRecords

\n

\"\"

\n

Thanks.!

", "messages": [ { "id": 1138323, "postDate": "2021-01-04T15:32:40.467Z", "content": "

Hello all, While going through all the TPU notebooks created so far in this competition I found that they don't have a proper cross-validation setup. Since the dataset is highly imbalanced and Some patients have more than one record.
\nTo make things easy, I have created Datasets that contain image data for Kaggle's RANZCR-CLiP Competition. They are MulitLabel stratified Group KFold. The original jpegs have been resized using cv2.resize().

\n

All the Dataset can be found at : [ (128x128) , (256x256) , (384x384) , (512x512) ]

\n

A sample notebook that presents Stratified GroupKFold cross-validation and Efficient Net architecture getting trained using TPUs can be found here.

\n

Want to modify or check how this Dataset was created? See this Notebook.

\n

What do TFRecords contain?

\n
feature = {\n      'image'  :   _bytes_feature\n      'ETT - Abnormal'  :  _int64_feature\n      'ETT - Borderline'  :  _int64_feature\n      'ETT - Normal'  :  _int64_feature\n      'NGT - Abnormal'  :  _int64_feature\n      'NGT - Borderline'  :  _int64_feature\n      'NGT - Incompletely Imaged'  :  _int64_feature\n      'NGT - Normal'  :  _int64_feature\n      'CVC - Abnormal'  :  _int64_feature\n      'CVC - Borderline'  :  _int64_feature\n      'CVC - Normal'  :_int64_feature\n      'Swan Ganz Catheter Present'  :  _int64_feature\n      'StudyInstanceUID':   _bytes_feature\n      'PatientID'  : _bytes_feature\n    }\n
\n

Visualizing distribution of labels in each TFRecords

\n

\"\"

\n

Visualizing distribution of Sample across the TFRecords

\n

\"\"

\n

Thanks.!

", "rawMarkdown": "Hello all, While going through all the TPU notebooks created so far in this competition I found that they don't have a proper cross-validation setup. Since the dataset is highly imbalanced and Some patients have more than one record. \nTo make things easy, I have created Datasets that contain image data for Kaggle's RANZCR-CLiP Competition. They are MulitLabel stratified Group KFold. The original jpegs have been resized using cv2.resize(). \n\nAll the Dataset can be found at : [ [(128x128)](https://www.kaggle.com/prateek0x/ranzcr-128x128) , ([256x256](https://www.kaggle.com/prateek0x/ranzcr-256x256)) , ([384x384](https://www.kaggle.com/prateek0x/ranzcr-384x384/)) , ([512x512](https://www.kaggle.com/prateek0x/ranzcr-512x512)) ]\n\n**A sample notebook** that presents Stratified GroupKFold cross-validation and Efficient Net architecture getting trained using TPUs can be found [here.](https://www.kaggle.com/prateek0x/stratified-groupkfold-with-efn-tfrecords)\n\nWant to modify or check how this Dataset was **created**? See this [Notebook](https://www.kaggle.com/prateek0x/creating-stratified-groupkfold-tfrecords-256x256).\n\n### What do TFRecords contain?\n\n```\nfeature = {\n 'image' : _bytes_feature\n 'ETT - Abnormal' : _int64_feature\n 'ETT - Borderline' : _int64_feature\n 'ETT - Normal' : _int64_feature\n 'NGT - Abnormal' : _int64_feature\n 'NGT - Borderline' : _int64_feature\n 'NGT - Incompletely Imaged' : _int64_feature\n 'NGT - Normal' : _int64_feature\n 'CVC - Abnormal' : _int64_feature\n 'CVC - Borderline' : _int64_feature\n 'CVC - Normal' :_int64_feature\n 'Swan Ganz Catheter Present' : _int64_feature\n 'StudyInstanceUID': _bytes_feature\n 'PatientID' : _bytes_feature\n }\n\n```\n## Visualizing distribution of labels in each TFRecords\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3530433%2Fb93ecb90b62af8377e91d7b693cfb9bd%2FScreenshot_20210104_201758.png?generation=1609771893934120&alt=media)\n\n\n## Visualizing distribution of Sample across the TFRecords\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3530433%2Fecc8cecedeb2355e01389f33a7d04342%2FScreenshot_20210104_201733.png?generation=1609771789672605&alt=media)\n\nThanks.!", "votes": 4 } ], "comments": [], "raw_markdown_by_id": { "1138323": "Hello all, While going through all the TPU notebooks created so far in this competition I found that they don't have a proper cross-validation setup. Since the dataset is highly imbalanced and Some patients have more than one record. \nTo make things easy, I have created Datasets that contain image data for Kaggle's RANZCR-CLiP Competition. They are MulitLabel stratified Group KFold. The original jpegs have been resized using cv2.resize(). \n\nAll the Dataset can be found at : [ [(128x128)](https://www.kaggle.com/prateek0x/ranzcr-128x128) , ([256x256](https://www.kaggle.com/prateek0x/ranzcr-256x256)) , ([384x384](https://www.kaggle.com/prateek0x/ranzcr-384x384/)) , ([512x512](https://www.kaggle.com/prateek0x/ranzcr-512x512)) ]\n\n**A sample notebook** that presents Stratified GroupKFold cross-validation and Efficient Net architecture getting trained using TPUs can be found [here.](https://www.kaggle.com/prateek0x/stratified-groupkfold-with-efn-tfrecords)\n\nWant to modify or check how this Dataset was **created**? See this [Notebook](https://www.kaggle.com/prateek0x/creating-stratified-groupkfold-tfrecords-256x256).\n\n### What do TFRecords contain?\n\n```\nfeature = {\n 'image' : _bytes_feature\n 'ETT - Abnormal' : _int64_feature\n 'ETT - Borderline' : _int64_feature\n 'ETT - Normal' : _int64_feature\n 'NGT - Abnormal' : _int64_feature\n 'NGT - Borderline' : _int64_feature\n 'NGT - Incompletely Imaged' : _int64_feature\n 'NGT - Normal' : _int64_feature\n 'CVC - Abnormal' : _int64_feature\n 'CVC - Borderline' : _int64_feature\n 'CVC - Normal' :_int64_feature\n 'Swan Ganz Catheter Present' : _int64_feature\n 'StudyInstanceUID': _bytes_feature\n 'PatientID' : _bytes_feature\n }\n\n```\n## Visualizing distribution of labels in each TFRecords\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3530433%2Fb93ecb90b62af8377e91d7b693cfb9bd%2FScreenshot_20210104_201758.png?generation=1609771893934120&alt=media)\n\n\n## Visualizing distribution of Sample across the TFRecords\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F3530433%2Fecc8cecedeb2355e01389f33a7d04342%2FScreenshot_20210104_201733.png?generation=1609771789672605&alt=media)\n\nThanks.!" } }