![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4367831%2F271f42e5ebce40fa6d6e4028369d4e32%2Fhold_out.jpg?generation=1609935747356886&alt=media\")
Hi! I try to summarize my knowledge about cross-validation (CV). It would be nice if a more experienced kaggler can correct me out if I am wrong, or add some information if I miss it. I hope it can help others. I assume that the data is shuffled already. Cross-validation is more representative of the private leader board, then the public leader board score.
\nThe basic approach is the holdout, where we subdivide the data into a train set and test set. We train the neural network on the train set and, measure the accuracy on the test set. This is when in k = 1. Code for it [1]
Cross-validation means, that we create k type of subdivision between on the data. We train on each fold one model, and in the end, we calculate the average. Code for this [2]
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4367831%2Fa7eaffa417f9905be8f0e22af7326ac0%2Fk-fold.jpg?generation=1609935772242624&alt=media\")
Additional materials:
\n[1] https://stackoverflow.com/questions/60883696/k-fold-cross-validation-using-dataloaders-in-pytorch
\ntrain_size = int(0.8 * len(full_dataset))\nvalidation_size = len(full_dataset) - train_size\ntrain_dataset, validation_dataset = random_split(full_dataset, [train_size, validation_size])\n\nfull_loader = DataLoader(full_dataset, batch_size=4,sampler = sampler_(full_dataset), pin_memory=True) \ntrain_loader = DataLoader(train_dataset, batch_size=4, sampler = sampler_(train_dataset))\nval_loader = DataLoader(validation_dataset, batch_size=1, sampler = sampler_(validation_dataset))\n[2] Taken from: https://www.kaggle.com/piantic/train-cassava-starter-using-various-loss-funcs
\nfolds = train.copy()\nFold = StratifiedKFold(n_splits=CFG.n_fold, shuffle=True, random_state=CFG.seed)\nfor n, (train_index, val_index) in enumerate(Fold.split(folds, folds[CFG.target_col])):\n folds.loc[val_index, 'fold'] = int(n)\nfolds['fold'] = folds['fold'].astype(int)\nHi! I try to summarize my knowledge about cross-validation (CV). It would be nice if a more experienced kaggler can correct me out if I am wrong, or add some information if I miss it. I hope it can help others. I assume that the data is shuffled already. Cross-validation is more representative of the private leader board, then the public leader board score.
\nThe basic approach is the holdout, where we subdivide the data into a train set and test set. We train the neural network on the train set and, measure the accuracy on the test set. This is when in k = 1. Code for it [1]
Cross-validation means, that we create k type of subdivision between on the data. We train on each fold one model, and in the end, we calculate the average. Code for this [2]
\n
Additional materials:
\n[1] https://stackoverflow.com/questions/60883696/k-fold-cross-validation-using-dataloaders-in-pytorch
\ntrain_size = int(0.8 * len(full_dataset))\nvalidation_size = len(full_dataset) - train_size\ntrain_dataset, validation_dataset = random_split(full_dataset, [train_size, validation_size])\n\nfull_loader = DataLoader(full_dataset, batch_size=4,sampler = sampler_(full_dataset), pin_memory=True) \ntrain_loader = DataLoader(train_dataset, batch_size=4, sampler = sampler_(train_dataset))\nval_loader = DataLoader(validation_dataset, batch_size=1, sampler = sampler_(validation_dataset))\n[2] Taken from: https://www.kaggle.com/piantic/train-cassava-starter-using-various-loss-funcs
\nfolds = train.copy()\nFold = StratifiedKFold(n_splits=CFG.n_fold, shuffle=True, random_state=CFG.seed)\nfor n, (train_index, val_index) in enumerate(Fold.split(folds, folds[CFG.target_col])):\n folds.loc[val_index, 'fold'] = int(n)\nfolds['fold'] = folds['fold'].astype(int)\n@bessenyeiszilrd There are 2 more I can add to this list:
\nStratified k-fold cross-validation
\nAdversarial validation
https://medium.com/datadriveninvestor/k-fold-and-other-cross-validation-techniques-6c03a2563f1e
\nHope this helps you. You can find some other folding techniques discussed here.
", "rawMarkdown": "https://medium.com/datadriveninvestor/k-fold-and-other-cross-validation-techniques-6c03a2563f1e\n\nHope this helps you. You can find some other folding techniques discussed here.", "votes": null }, { "id": "1157063", "postDate": "01/17/2021 16:11:25", "content": "Thanks a lot for posting this! One thing that has been a bit unclear to me is when to use the \"hold out\" approach and when to use the \"K-fold cross validation\" or if you should actually do both? I've seen that you sometimes make the split: train/test and then \"K-fold cross validation\" on this training set but leave the test set to the very last (as how it's done in Kaggle competitions). In a way I think this should be the safest approach, to always have a test set that never comes near any of your calculations, to avoid overfitting.
", "rawMarkdown": "Thanks a lot for posting this! One thing that has been a bit unclear to me is when to use the \"hold out\" approach and when to use the \"K-fold cross validation\" or if you should actually do both? I've seen that you sometimes make the split: train/test and then \"K-fold cross validation\" on this training set but leave the test set to the very last (as how it's done in Kaggle competitions). In a way I think this should be the safest approach, to always have a test set that never comes near any of your calculations, to avoid overfitting.", "votes": null }, { "id": "1157071", "postDate": "01/17/2021 16:20:14", "content": "In the hold out approach you split the dataset into a test and train set. However, you might have seen that the result (train/validation error) is not consistent if you re-run the same setup. As a result, to show consistency, we take an average after repeating the test train set for k folds. And there are many kinds of cross validation techniques. You can go through the link below to know more.
\nhttps://medium.com/datadriveninvestor/k-fold-and-other-cross-validation-techniques-6c03a2563f1e
", "rawMarkdown": "In the hold out approach you split the dataset into a test and train set. However, you might have seen that the result (train/validation error) is not consistent if you re-run the same setup. As a result, to show consistency, we take an average after repeating the test train set for k folds. And there are many kinds of cross validation techniques. You can go through the link below to know more.\n\nhttps://medium.com/datadriveninvestor/k-fold-and-other-cross-validation-techniques-6c03a2563f1e", "votes": null }, { "id": "1268132", "postDate": "04/09/2021 06:46:22", "content": "But in a Kaggle competition there will always be a test data set which the model never get to see. So I guess that for this case the setup is rather train/validation/test? (And you can do cross validation on the train/validation part as you please, but you can never get the test data into the cross validation).
", "rawMarkdown": "But in a Kaggle competition there will always be a test data set which the model never get to see. So I guess that for this case the setup is rather train/validation/test? (And you can do cross validation on the train/validation part as you please, but you can never get the test data into the cross validation).", "votes": null }, { "id": "1270173", "postDate": "04/11/2021 11:33:49", "content": "In my image, the test set is the validation set, so during the model training only train / validation sets
Great contextual overview @bessenyeiszilrd !!
\nk=10 is recommended for ML application. k is selected such that each train_test of sample is large enough to statistically depict the wider dataset.
\nThe test set score is a better estimate of model performance on unseen data. So, cross validation score is insignificant!
\nA better validation score than the test score indicates the model is overfitting, a recipe to doubt the test score.
\nSolution: Utilizing more training data against live test data with little or no isolated patterns, one can avoid statistical bias and overfitting (model easily memorizing known patterns). This pushes models to reach predictions that are accommodative to more parameters.
@bessenyeiszilrd There are 2 more I can add to this list:
\nStratified k-fold cross-validation
\nAdversarial validation
https://medium.com/datadriveninvestor/k-fold-and-other-cross-validation-techniques-6c03a2563f1e
\nHope this helps you. You can find some other folding techniques discussed here.
", "votes": null, "replies": [] }, { "id": 1157063, "author_name": "martinlarsalbert", "author_url": "", "post_date": "01/17/2021 16:11:25", "content": "Thanks a lot for posting this! One thing that has been a bit unclear to me is when to use the \"hold out\" approach and when to use the \"K-fold cross validation\" or if you should actually do both? I've seen that you sometimes make the split: train/test and then \"K-fold cross validation\" on this training set but leave the test set to the very last (as how it's done in Kaggle competitions). In a way I think this should be the safest approach, to always have a test set that never comes near any of your calculations, to avoid overfitting.
", "votes": null, "replies": [ { "id": 1157071, "author_name": "sadmanaraf", "author_url": "", "post_date": "01/17/2021 16:20:14", "content": "In the hold out approach you split the dataset into a test and train set. However, you might have seen that the result (train/validation error) is not consistent if you re-run the same setup. As a result, to show consistency, we take an average after repeating the test train set for k folds. And there are many kinds of cross validation techniques. You can go through the link below to know more.
\nhttps://medium.com/datadriveninvestor/k-fold-and-other-cross-validation-techniques-6c03a2563f1e
", "votes": null, "replies": [] }, { "id": 1268132, "author_name": "martinlarsalbert", "author_url": "", "post_date": "04/09/2021 06:46:22", "content": "But in a Kaggle competition there will always be a test data set which the model never get to see. So I guess that for this case the setup is rather train/validation/test? (And you can do cross validation on the train/validation part as you please, but you can never get the test data into the cross validation).
", "votes": null, "replies": [] }, { "id": 1270173, "author_name": "bessenyeiszilrd", "author_url": "", "post_date": "04/11/2021 11:33:49", "content": "In my image, the test set is the validation set, so during the model training only train / validation sets
Great contextual overview @bessenyeiszilrd !!
\nk=10 is recommended for ML application. k is selected such that each train_test of sample is large enough to statistically depict the wider dataset.
\nThe test set score is a better estimate of model performance on unseen data. So, cross validation score is insignificant!
\nA better validation score than the test score indicates the model is overfitting, a recipe to doubt the test score.
\nSolution: Utilizing more training data against live test data with little or no isolated patterns, one can avoid statistical bias and overfitting (model easily memorizing known patterns). This pushes models to reach predictions that are accommodative to more parameters.