I would like to continue the topic started in this kernel by reigHns.
\n\nOne of the challenges that we are facing in this competition is finding the best way to implement a K-fold cross-validation. On one hand, we have a very imbalanced dataset, so some sort of stratification seems to be in order here. On the other hand, we know that some of the patients present in the dataset are represented by multiple images and it is probably a good idea not to mix images corresponding to the same patient_id in our training and validation sets. The latter can be achieved by using sklearn Group K-Fold split. Unfortunately, sklearn does not have an implementation of Stratified Group K-Folds, so we have to make one from scratch. Fortunately, one such implementation of Stratified Group K-Folds was already developed in the PetFinder competition, so we can just take it and use it here. And it seems to be working pretty well as can be seen from my public kernel:
SIIM Stratified GroupKFold 5-folds
\n\n-- the 5-fold split has almost the same percentages of 0's and 1's in each fold and the unique patient_id values do not overlap between different folds. I can think of two possible ways how these folds can be utilized: we can use them to do cross-validation on jpeg files or we can make a new set of tfrecord files based on these folds.
I would like to continue the topic started in this kernel by reigHns.
\n\nOne of the challenges that we are facing in this competition is finding the best way to implement a K-fold cross-validation. On one hand, we have a very imbalanced dataset, so some sort of stratification seems to be in order here. On the other hand, we know that some of the patients present in the dataset are represented by multiple images and it is probably a good idea not to mix images corresponding to the same patient_id in our training and validation sets. The latter can be achieved by using sklearn Group K-Fold split. Unfortunately, sklearn does not have an implementation of Stratified Group K-Folds, so we have to make one from scratch. Fortunately, one such implementation of Stratified Group K-Folds was already developed in the PetFinder competition, so we can just take it and use it here. And it seems to be working pretty well as can be seen from my public kernel:
SIIM Stratified GroupKFold 5-folds
\n\n-- the 5-fold split has almost the same percentages of 0's and 1's in each fold and the unique patient_id values do not overlap between different folds. I can think of two possible ways how these folds can be utilized: we can use them to do cross-validation on jpeg files or we can make a new set of tfrecord files based on these folds.
Thanks @graf10a for your CV strategy. Now I'm using it as my baseline.\nMine CV AUC each folder is: 0.913/0.937/0.884/0.883/0.906\nOverall OOF AUC is 0.892; LB 0.929 (with external, 256x256, Eff-b0)\nDo you also observe the AUC difference in each folder?
", "rawMarkdown": "Thanks @graf10a for your CV strategy. Now I'm using it as my baseline.\nMine CV AUC each folder is: 0.913/0.937/0.884/0.883/0.906\nOverall OOF AUC is 0.892; LB 0.929 (with external, 256x256, Eff-b0)\nDo you also observe the AUC difference in each folder?\n", "votes": 1, "replies": [ { "id": 926854, "postDate": "2020-07-13T03:42:57.530Z", "content": "@waylongo Yes, I do see a great deal of variability across the folds. Makes me very nervous about the private LB 😲 .
", "rawMarkdown": "@waylongo Yes, I do see a great deal of variability across the folds. Makes me very nervous about the private LB 😲 .", "votes": 1 } ] }, { "id": 892301, "postDate": "2020-06-18T19:29:50.857Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 892433, "postDate": "2020-06-18T21:43:28.580Z", "content": "Thank you! It seems to be working reasonably well.
", "rawMarkdown": "Thank you! It seems to be working reasonably well.", "votes": 1 } ] } ], "comments": [ { "id": 926847, "author_name": "Waylon Wu", "author_url": "", "post_date": "2020-07-13T03:32:39.577000", "content": "Thanks @graf10a for your CV strategy. Now I'm using it as my baseline.\nMine CV AUC each folder is: 0.913/0.937/0.884/0.883/0.906\nOverall OOF AUC is 0.892; LB 0.929 (with external, 256x256, Eff-b0)\nDo you also observe the AUC difference in each folder?
", "votes": 1, "replies": [ { "id": 926854, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-07-13T03:42:57.530000", "content": "@waylongo Yes, I do see a great deal of variability across the folds. Makes me very nervous about the private LB 😲 .
", "votes": 1, "replies": [] } ] }, { "id": 892301, "author_name": "", "author_url": "", "post_date": "2020-06-18T19:29:50.857000", "content": "", "votes": 2, "replies": [ { "id": 892433, "author_name": "Alexey Pronin", "author_url": "", "post_date": "2020-06-18T21:43:28.580000", "content": "Thank you! It seems to be working reasonably well.
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "873306": "I would like to continue the topic started in [this kernel](https://www.kaggle.com/reighns/groupkfold-efficientbnet-trial) by [reigHns](https://www.kaggle.com/reighns). \n\nOne of the challenges that we are facing in this competition is finding the best way to implement a K-fold cross-validation. On one hand, we have a very imbalanced dataset, so some sort of stratification seems to be in order here. On the other hand, we know that some of the patients present in the dataset are represented by multiple images and it is probably a good idea not to mix images corresponding to the same `patient_id` in our training and validation sets. The latter can be achieved by using sklearn Group K-Fold split. Unfortunately, sklearn does not have an implementation of Stratified Group K-Folds, so we have to make one from scratch. Fortunately, one such [implementation of Stratified Group K-Folds](https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation) was already developed in the PetFinder competition, so we can just take it and use it here. And it seems to be working pretty well as can be seen from my public kernel:\n\n[SIIM Stratified GroupKFold 5-folds](https://www.kaggle.com/graf10a/siim-stratified-groupkfold) \n\n-- the 5-fold split has almost the same percentages of 0's and 1's in each fold and the unique `patient_id` values do not overlap between different folds. I can think of two possible ways how these folds can be utilized: we can use them to do cross-validation on jpeg files or we can make a new set of tfrecord files based on these folds. ", "926847": "Thanks @graf10a for your CV strategy. Now I'm using it as my baseline.\nMine CV AUC each folder is: 0.913/0.937/0.884/0.883/0.906\nOverall OOF AUC is 0.892; LB 0.929 (with external, 256x256, Eff-b0)\nDo you also observe the AUC difference in each folder?\n", "892301": "" } }