Maybe it is too early to ask this question, but as everybody seems to be focused on the metrick hack and may have lost interest on it, I would like to know which CV strategies have performed better on the LB before everybody jumps into the next competition.
\nSo far, all the public notebooks I have seen are using AUCROC with StratifiedGroupKFold using a weeks as group variable, and from my point of view that doesn’t really fit in with the problem. I have tried using the stability metric with a kind of time series 5 fold split: first fold train with first 20% of the data and validate the stability metric with the rest 80%; 2nd fold train with first 40% and validate with 60%; and so on… but this approach didn’t give me good results on LB. I have ended up using AUCROC with a mix StratifiedKFold with the label and weeks, also tried with AUCPR
\nCurious to know what have worked for you
", "messages": [ { "id": "2827437", "postDate": "05/21/2024 13:21:44", "content": "Maybe it is too early to ask this question, but as everybody seems to be focused on the metrick hack and may have lost interest on it, I would like to know which CV strategies have performed better on the LB before everybody jumps into the next competition.
\nSo far, all the public notebooks I have seen are using AUCROC with StratifiedGroupKFold using a weeks as group variable, and from my point of view that doesn’t really fit in with the problem. I have tried using the stability metric with a kind of time series 5 fold split: first fold train with first 20% of the data and validate the stability metric with the rest 80%; 2nd fold train with first 40% and validate with 60%; and so on… but this approach didn’t give me good results on LB. I have ended up using AUCROC with a mix StratifiedKFold with the label and weeks, also tried with AUCPR
\nCurious to know what have worked for you
", "rawMarkdown": "Maybe it is too early to ask this question, but as everybody seems to be focused on the metrick hack and may have lost interest on it, I would like to know which CV strategies have performed better on the LB before everybody jumps into the next competition.\n\nSo far, all the public notebooks I have seen are using AUCROC with StratifiedGroupKFold using a weeks as group variable, and from my point of view that doesn’t really fit in with the problem. I have tried using the stability metric with a kind of time series 5 fold split: first fold train with first 20% of the data and validate the stability metric with the rest 80%; 2nd fold train with first 40% and validate with 60%; and so on… but this approach didn’t give me good results on LB. I have ended up using AUCROC with a mix StratifiedKFold with the label and weeks, also tried with AUCPR\n\nCurious to know what have worked for you", "votes": null }, { "id": "2827486", "postDate": "05/21/2024 14:00:10", "content": "I have used both StratifiedGroupKFold (5 and 10 fold training) and Holdout validation. For the latter, I trained my model on the first 82 weeks (90% of total weeks on the train set - week 0 to week 81) and evaluated on the last 10 weeks (10% of total weeks on the train set - week 82 to week 91).
\ninitially, Holdout validation seemed more correct to predict future, unseen data (the validation set) than StratifiedGroupKFold. Training 2 LGB and 2 CTB models and ensembling them based on the mean of the predictions, I got a public LB score of 0.578, while the best AUC scores for LGB and CTB classifiers respectively, were 0.88 and 0.87.
\nWith StratifiedGroupKFold however, the results are slightly different: the mean AUC CV scores for 5-fold and 10-fold training are 0.8589 and 0.8594 and the public LB score is 0.569 and 0.568 respectively (only for LGB models).
\nConsidering the top10 solutions of the AMEX competition - with the same objective - most solutions used StratifiedKFold and not time-series methods (TimeSeriesSplit or Holdout validation). Thus, for my two selected solutions, I will select solutions with StratifiedGroupKFold training.
", "rawMarkdown": "I have used both StratifiedGroupKFold (5 and 10 fold training) and Holdout validation. For the latter, I trained my model on the first 82 weeks (90% of total weeks on the train set - week 0 to week 81) and evaluated on the last 10 weeks (10% of total weeks on the train set - week 82 to week 91). \n\ninitially, Holdout validation seemed more correct to predict future, unseen data (the validation set) than StratifiedGroupKFold. Training 2 LGB and 2 CTB models and ensembling them based on the mean of the predictions, I got a public LB score of 0.578, while the best AUC scores for LGB and CTB classifiers respectively, were 0.88 and 0.87. \n\nWith StratifiedGroupKFold however, the results are slightly different: the mean AUC CV scores for 5-fold and 10-fold training are 0.8589 and 0.8594 and the public LB score is 0.569 and 0.568 respectively (only for LGB models). \n\nConsidering the top10 solutions of the [AMEX competition](https://www.kaggle.com/competitions/amex-default-prediction) - with the same objective - most solutions used StratifiedKFold and not time-series methods (TimeSeriesSplit or Holdout validation). Thus, for my two selected solutions, I will select solutions with StratifiedGroupKFold training.", "votes": null }, { "id": "2827920", "postDate": "05/21/2024 18:16:57", "content": "Interesting, I didn't know about that AMEX competition. Thanks for sharing and good luck for the private leaderboard
", "rawMarkdown": "Interesting, I didn't know about that AMEX competition. Thanks for sharing and good luck for the private leaderboard", "votes": null } ], "comments": [ { "id": 2827486, "author_name": "andreasbis", "author_url": "", "post_date": "05/21/2024 14:00:10", "content": "I have used both StratifiedGroupKFold (5 and 10 fold training) and Holdout validation. For the latter, I trained my model on the first 82 weeks (90% of total weeks on the train set - week 0 to week 81) and evaluated on the last 10 weeks (10% of total weeks on the train set - week 82 to week 91).
\ninitially, Holdout validation seemed more correct to predict future, unseen data (the validation set) than StratifiedGroupKFold. Training 2 LGB and 2 CTB models and ensembling them based on the mean of the predictions, I got a public LB score of 0.578, while the best AUC scores for LGB and CTB classifiers respectively, were 0.88 and 0.87.
\nWith StratifiedGroupKFold however, the results are slightly different: the mean AUC CV scores for 5-fold and 10-fold training are 0.8589 and 0.8594 and the public LB score is 0.569 and 0.568 respectively (only for LGB models).
\nConsidering the top10 solutions of the AMEX competition - with the same objective - most solutions used StratifiedKFold and not time-series methods (TimeSeriesSplit or Holdout validation). Thus, for my two selected solutions, I will select solutions with StratifiedGroupKFold training.
", "votes": null, "replies": [ { "id": 2827920, "author_name": "diegoiglesias", "author_url": "", "post_date": "05/21/2024 18:16:57", "content": "Interesting, I didn't know about that AMEX competition. Thanks for sharing and good luck for the private leaderboard
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2827437": "Maybe it is too early to ask this question, but as everybody seems to be focused on the metrick hack and may have lost interest on it, I would like to know which CV strategies have performed better on the LB before everybody jumps into the next competition.\n\nSo far, all the public notebooks I have seen are using AUCROC with StratifiedGroupKFold using a weeks as group variable, and from my point of view that doesn’t really fit in with the problem. I have tried using the stability metric with a kind of time series 5 fold split: first fold train with first 20% of the data and validate the stability metric with the rest 80%; 2nd fold train with first 40% and validate with 60%; and so on… but this approach didn’t give me good results on LB. I have ended up using AUCROC with a mix StratifiedKFold with the label and weeks, also tried with AUCPR\n\nCurious to know what have worked for you", "2827486": "I have used both StratifiedGroupKFold (5 and 10 fold training) and Holdout validation. For the latter, I trained my model on the first 82 weeks (90% of total weeks on the train set - week 0 to week 81) and evaluated on the last 10 weeks (10% of total weeks on the train set - week 82 to week 91). \n\ninitially, Holdout validation seemed more correct to predict future, unseen data (the validation set) than StratifiedGroupKFold. Training 2 LGB and 2 CTB models and ensembling them based on the mean of the predictions, I got a public LB score of 0.578, while the best AUC scores for LGB and CTB classifiers respectively, were 0.88 and 0.87. \n\nWith StratifiedGroupKFold however, the results are slightly different: the mean AUC CV scores for 5-fold and 10-fold training are 0.8589 and 0.8594 and the public LB score is 0.569 and 0.568 respectively (only for LGB models). \n\nConsidering the top10 solutions of the [AMEX competition](https://www.kaggle.com/competitions/amex-default-prediction) - with the same objective - most solutions used StratifiedKFold and not time-series methods (TimeSeriesSplit or Holdout validation). Thus, for my two selected solutions, I will select solutions with StratifiedGroupKFold training.", "2827920": "Interesting, I didn't know about that AMEX competition. Thanks for sharing and good luck for the private leaderboard" }, "source": "meta" }