In the current public notebooks, most of people use GroupKFold to split the train/valid data.
\nIt assures no leak happens in validation dataset. But you also have to consider sufficient #COTSs appears in both train/validation sets for each fold. Naively applying GroupKFold may cause unbalanced folds.
So I came up with the idea splitting each sequence to subsequence, to get more balanced fold.
\nThe algorithms is simple, but it mitigates the unbalance -- decreasing standard deviation of #COTSs and #COTSs/frame between folds.
The algorithm is naive, but the idea should be applied to other more sophisticated fold splitting algorithms.
\nThe code is here:
\nhttps://www.kaggle.com/tatamikenn/a-balanced-fold-splitting-algorithm?scriptVersionId=81854118
In the current public notebooks, most of people use GroupKFold to split the train/valid data.
\nIt assures no leak happens in validation dataset. But you also have to consider sufficient #COTSs appears in both train/validation sets for each fold. Naively applying GroupKFold may cause unbalanced folds.
So I came up with the idea splitting each sequence to subsequence, to get more balanced fold.
\nThe algorithms is simple, but it mitigates the unbalance -- decreasing standard deviation of #COTSs and #COTSs/frame between folds.
The algorithm is naive, but the idea should be applied to other more sophisticated fold splitting algorithms.
\nThe code is here:
\nhttps://www.kaggle.com/tatamikenn/a-balanced-fold-splitting-algorithm?scriptVersionId=81854118
Hi @tatamikenn, Do you have a good result with new split algorithm than groupkfold ?
\nHas the gap between CV and LB been reduced with new algorithm ?
Personally I didn't test. But someone reported CV/LB was correlated well using a similar split [1] (the one which @julian3833 proposed[2]. It is almost identical to the approach except for the fold allocation algorithm).
\nThank you!
", "rawMarkdown": "Thank you!", "votes": null } ], "comments": [ { "id": 1617262, "author_name": "seongwook93", "author_url": "", "post_date": "12/14/2021 01:58:15", "content": "Hi @tatamikenn, Do you have a good result with new split algorithm than groupkfold ?
\nHas the gap between CV and LB been reduced with new algorithm ?
Personally I didn't test. But someone reported CV/LB was correlated well using a similar split [1] (the one which @julian3833 proposed[2]. It is almost identical to the approach except for the fold allocation algorithm).
\nThank you!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1611880": "In the current public notebooks, most of people use GroupKFold to split the train/valid data.\nIt assures no leak happens in validation dataset. But you also have to consider sufficient #COTSs appears in both train/validation sets for each fold. Naively applying GroupKFold may cause unbalanced folds. \n\nSo I came up with the idea splitting each sequence to subsequence, to get more balanced fold.\nThe algorithms is simple, but it mitigates the unbalance -- decreasing standard deviation of #COTSs and #COTSs/frame between folds.\n\nThe algorithm is naive, but the idea should be applied to other more sophisticated fold splitting algorithms.\n\nThe code is here:\nhttps://www.kaggle.com/tatamikenn/a-balanced-fold-splitting-algorithm?scriptVersionId=81854118", "1617262": "Hi @tatamikenn, Do you have a good result with new split algorithm than groupkfold ? \nHas the gap between CV and LB been reduced with new algorithm ?", "1617360": "seongwook93 \n\nPersonally I didn't test. But someone reported CV/LB was correlated well using a similar split [1] (the one which @julian3833 proposed[2]. It is almost identical to the approach except for the fold allocation algorithm).\n\n* [1] https://www.kaggle.com/c/tensorflow-great-barrier-reef/discussion/290757#1612788\n* [2] https://www.kaggle.com/julian3833/reef-a-cv-strategy-subsequences", "1617427": "Thank you!" }, "source": "meta" }