![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2F79668c5b80f69cbe78b5ee91deb6798b%2Fkfold.png?generation=1597776677559528&alt=media\")
One thing I learned at Kaggle is how to ensemble models using OOF files. This was very helpful in Melanoma Competition. I would like to share a simple procedure below called hill climbing and I posted a starter notebook here.
The acronym CV refers to cross validation. We start with the full training dataset picture below to the far left. Next, we divide it into 5 subsets, called Fold 1, Fold 2, Fold 3, Fold 4, Fold 5. Then we train 5 models. We train our first model using data from Folds 2-5 and predict Fold 1. Next we train model 2 using Folds 1, 3, 4, 5 and predict 2. Next 1, 2, 4, 5 and predict 3, etc etc.
Afterward we have predictions for every training image. This compete set of predictions is called OOF, \"out of fold\" predictions. It is a good practice to save these predictions for every model you build during a competition as oof.csv.
The CV score (or OOF AUC) is then calculated with OOF_AUC = roc_auc_score( train.target, oof.prediction). And this is the best indicator of how your model performs. It is a better indicator than public LB.
During the competition, every model should use the same folds. This is accomplished by using the same seed with sklearn.model_selection.KFold(n_splits = 5, shuffle = True, random_seed = 42)
For each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image. We take the average of these 5 sets of predictions and this is our submission.csv file that we submit to Kaggle. When you submit this to Kaggle, your LB score should be similar to your CV score.
Now say that you build 2 models (that means that you did 5 KFold twice). You now have oof_1.csv, oof_2.csv, sub_1.csv, and sub_2.csv. How do we blend the two models?
We find the weight w such that w * oof_1.predictions + (1-w) * oof_2.predictions has the largest AUC.
all = []\n for w in [0.00, 0.01, 0.02, ..., 0.98, 0.99, 1.00]:\n ensemble_pred = w * oof_1.predictions + (1-w) * oof_2.predictions\n ensemble_auc = roc_auc_score( oof.target , ensemble_pred )\n all.append( ensemble_auc )\n best_weight = np.argmax( all ) / 100.\nThen our submission to kaggle will be
\n kaggle_sub = best_weight * sub_1.target + (1-best_weight) * sub_2.target\nAfter weeks working on a competition, we will have more than 2 models. So we need a more sophisticated approach than a single for-loop.
\nThe simplest approach is hill climbing (or forward selection). Start with the one model that has highest CV score. Next iterate through all your additional models and find the one model that combines with the first model to generate the highest two model ensemble CV score. Then search for the best third model. Repeat until ensemble CV does not increase anymore.
I posted a starter notebook here which includes all my oof.csv and sub.csv for this Melanoma comp. There are 39 models. Forward selection chooses 8 of them and the resultant ensemble has OOF CV 0.950, Public LB 0.958, and Private LB 0.942
One thing I learned at Kaggle is how to ensemble models using OOF files. This was very helpful in Melanoma Competition. I would like to share a simple procedure below called hill climbing and I posted a starter notebook here.
The acronym CV refers to cross validation. We start with the full training dataset picture below to the far left. Next, we divide it into 5 subsets, called Fold 1, Fold 2, Fold 3, Fold 4, Fold 5. Then we train 5 models. We train our first model using data from Folds 2-5 and predict Fold 1. Next we train model 2 using Folds 1, 3, 4, 5 and predict 2. Next 1, 2, 4, 5 and predict 3, etc etc.
Afterward we have predictions for every training image. This compete set of predictions is called OOF, \"out of fold\" predictions. It is a good practice to save these predictions for every model you build during a competition as oof.csv.
The CV score (or OOF AUC) is then calculated with OOF_AUC = roc_auc_score( train.target, oof.prediction). And this is the best indicator of how your model performs. It is a better indicator than public LB.
During the competition, every model should use the same folds. This is accomplished by using the same seed with sklearn.model_selection.KFold(n_splits = 5, shuffle = True, random_seed = 42)
For each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image. We take the average of these 5 sets of predictions and this is our submission.csv file that we submit to Kaggle. When you submit this to Kaggle, your LB score should be similar to your CV score.
Now say that you build 2 models (that means that you did 5 KFold twice). You now have oof_1.csv, oof_2.csv, sub_1.csv, and sub_2.csv. How do we blend the two models?
We find the weight w such that w * oof_1.predictions + (1-w) * oof_2.predictions has the largest AUC.
all = []\n for w in [0.00, 0.01, 0.02, ..., 0.98, 0.99, 1.00]:\n ensemble_pred = w * oof_1.predictions + (1-w) * oof_2.predictions\n ensemble_auc = roc_auc_score( oof.target , ensemble_pred )\n all.append( ensemble_auc )\n best_weight = np.argmax( all ) / 100.\nThen our submission to kaggle will be
\n kaggle_sub = best_weight * sub_1.target + (1-best_weight) * sub_2.target\nAfter weeks working on a competition, we will have more than 2 models. So we need a more sophisticated approach than a single for-loop.
\nThe simplest approach is hill climbing (or forward selection). Start with the one model that has highest CV score. Next iterate through all your additional models and find the one model that combines with the first model to generate the highest two model ensemble CV score. Then search for the best third model. Repeat until ensemble CV does not increase anymore.
I posted a starter notebook here which includes all my oof.csv and sub.csv for this Melanoma comp. There are 39 models. Forward selection chooses 8 of them and the resultant ensemble has OOF CV 0.950, Public LB 0.958, and Private LB 0.942
Kagglers refer to this as hill climbing just as a side note if others see this phrase somewhere else.
thanks. I updated my post.
", "rawMarkdown": "thanks. I updated my post.", "votes": 3 }, { "id": 984401, "postDate": "2020-08-25T04:46:25.477Z", "content": "@philippsinger When you and others use hill climbing, do you continue adding models as long as CV increases, or do you only add a new model if it increases CV by some tolerance?
For my submission, i used TOL = 0.0003 thinking I would prevent noise. But if you compare hill climbing to XGB stacking, then XGB uses TOL = 0, right? XGB just keeps adding as long as CV increases.
@cdeotte I usually apply it for a certain number of iterations. To avoid overfitting you can also add certain restrictions, like a single model can only be added once or twice.
", "rawMarkdown": "@cdeotte I usually apply it for a certain number of iterations. To avoid overfitting you can also add certain restrictions, like a single model can only be added once or twice.", "votes": 1 } ] }, { "id": 979176, "postDate": "2020-08-20T17:12:15.270Z", "content": "Thanks for sharing this! Good explanation for CV!
\nAfter reading some discussions in the past few days after the competition end, I gradually realized that what more people need is not the top solutions, but the basic ML knowledge, just like this posts.
\nI see many people still don't understand why they should not trust public LB score (especially in this comp). But I don’t know how to explain to them in a simple and understandable way.
\nI think your post can be a good explanation for it as well. Thanks again!
", "rawMarkdown": "Thanks for sharing this! Good explanation for CV!\n\nAfter reading some discussions in the past few days after the competition end, I gradually realized that what more people need is not the top solutions, but the basic ML knowledge, just like this posts.\n\nI see many people still don't understand why they should not trust public LB score (especially in this comp). But I don’t know how to explain to them in a simple and understandable way.\n\nI think your post can be a good explanation for it as well. Thanks again!", "votes": 14, "replies": [ { "id": 979203, "postDate": "2020-08-20T17:31:26.207Z", "content": "Thanks Qishen. I updated the discussion title to `\"How To CV and How To Ensemble OOF\". I thought i was just explaining ensemble, but you are right, this is a nice explanation of CV too.
\nCongrats again on you and your team's awesome model with amazing 0.960 CV!
", "rawMarkdown": "Thanks Qishen. I updated the discussion title to `\"How To CV and How To Ensemble OOF\". I thought i was just explaining ensemble, but you are right, this is a nice explanation of CV too.\n\nCongrats again on you and your team's awesome model with amazing 0.960 CV!", "votes": 7 } ] }, { "id": 1654315, "postDate": "2022-01-18T11:35:17.327Z", "content": "Thanks @cdeotte, your Hill Climbing approach is a great trick. I'll for sure try it out in my next competition.
\nI have small doubt, in the post you've linked you say
\n\nDuring the competition, every model should use the same folds.
\n
Does that really matter when your oof predictions have all the train samples anyway? I specifically ask in a context of ensembling models as a team, where different members worked with different splits.
", "rawMarkdown": "Thanks @cdeotte, your Hill Climbing approach is a great trick. I'll for sure try it out in my next competition.\nI have small doubt, in the post you've linked you say \n>During the competition, every model should use the same folds. \n\nDoes that really matter when your oof predictions have all the train samples anyway? I specifically ask in a context of ensembling models as a team, where different members worked with different splits.", "votes": 2 }, { "id": 988805, "postDate": "2020-08-28T09:42:28.270Z", "content": "Hi @cdeotte, have you seen this paper \"On Cross-Validation and Stacking: Building seemingly predictive models on random data\"?
\nhttps://www.kdd.org/exploration_files/v12-02-4-UR-Perlich.pdf
I'd be interested to know what you and other Kagglers think about it
", "rawMarkdown": "Hi @cdeotte, have you seen this paper \"On Cross-Validation and Stacking: Building seemingly predictive models on random data\"?\nhttps://www.kdd.org/exploration_files/v12-02-4-UR-Perlich.pdf \n\nI'd be interested to know what you and other Kagglers think about it", "votes": 3, "replies": [ { "id": 989300, "postDate": "2020-08-28T17:42:23.213Z", "content": "Thanks for the paper, i'll check it out and post my reaction here.
", "rawMarkdown": "Thanks for the paper, i'll check it out and post my reaction here.", "votes": 4 } ] }, { "id": 979325, "postDate": "2020-08-20T19:03:15.597Z", "content": "Hi Chris, the explanation is very clear but I have a question and after reading the post a couple of times I can't figure out the answer on my own. Talking about the submission file, you say
\n\n\nFor each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image. We take the average of these 5 sets of predictions and this is our submission.csv file that we submit to Kaggle. When you submit this to Kaggle, your LB score should be similar to your CV score.
\n
and this makes perfect sense, but why do you never train a model using all train data once you have a high CV score for that model?
\nIs it because you would not know how it performs when trained with an additional fold, making all the work useless and destroying the ensembling process?
\nI am asking because so far, when I perform cross-validation, I always fit the model on all train data with the best parameters to make the test predictions and so your way of doing it by averaging is completely new to me.
\nThank you!
Doing it as I describe associates the CV to LB score. Because the identical models that produced the CV score are the same models producing the LB score. (When you run twice as your describe, you don't have this guarantee). That being said, I sometimes do what you suggest. Both ways have pros and cons.
\nNote that the way I describe in this post is a form of bagging similar to a random forest. Each of the 5 fold models are training on a random 80% of the train data. Therefore even if each of the five fold models overfits their 80% train data, the ensemble of the 5 folds will not overfit the entire 100% train data. So predicting test using CV is also a way to prevent overfitting the train data (i.e. generalizing better to test data) in the same way that a random forest uses bagging to prevent overfitting train data.
", "rawMarkdown": "Doing it as I describe associates the CV to LB score. Because the identical models that produced the CV score are the same models producing the LB score. (When you run twice as your describe, you don't have this guarantee). That being said, I sometimes do what you suggest. Both ways have pros and cons.\n\nNote that the way I describe in this post is a form of bagging similar to a random forest. Each of the 5 fold models are training on a random 80% of the train data. Therefore even if each of the five fold models overfits their 80% train data, the ensemble of the 5 folds will not overfit the entire 100% train data. So predicting test using CV is also a way to prevent overfitting the train data (i.e. generalizing better to test data) in the same way that a random forest uses bagging to prevent overfitting train data.", "votes": 5 }, { "id": 979418, "postDate": "2020-08-20T20:31:06.320Z", "content": "Ok, got it, thank you so much.
\nBy doing this, you are training on less data but as a tradeoff you have a more reliable estimate of the performance of your model and also a free ensemble of 5 classifiers which is never a bad thing.
\nI guess I'll try to apply this method in House Prices competition when I have my final model ready!
The strategy I usually go for is finding a set of models that give the best CV when averaged together.
\nDo you have any idea how your ensemble performs (especially on the leaderboard) when giving equal weights to every model ?
\nWhat about using the 8 (or any number of) models that have the best CV when averging the predictions together ?
This would be nice to see how much hill climbing actually helps.
", "rawMarkdown": "The strategy I usually go for is finding a set of models that give the best CV when averaged together.\n\nDo you have any idea how your ensemble performs (especially on the leaderboard) when giving equal weights to every model ? \nWhat about using the 8 (or any number of) models that have the best CV when averging the predictions together ? \n\nThis would be nice to see how much hill climbing actually helps.", "votes": 4, "replies": [ { "id": 976425, "postDate": "2020-08-18T20:53:24.477Z", "content": "Great suggestions Theo. I will investigate this.
\nI should probably change the language in my post. I should be more clear that high climbing is only one way and there are many others like the ones you suggest.
", "rawMarkdown": "Great suggestions Theo. I will investigate this.\n\nI should probably change the language in my post. I should be more clear that high climbing is only one way and there are many others like the ones you suggest.", "votes": 1 }, { "id": 976447, "postDate": "2020-08-18T21:22:20.207Z", "content": "\n\nI should probably change the language in my post. I should be more clear that high climbing is only one way and there are many others like the ones you suggest.
\n
No worries, this was clear to me :)
", "rawMarkdown": "> I should probably change the language in my post. I should be more clear that high climbing is only one way and there are many others like the ones you suggest.\n\nNo worries, this was clear to me :)", "votes": 1 }, { "id": 976452, "postDate": "2020-08-18T21:29:34.490Z", "content": "Interesting! thanks for sharing.
\n| no_of_models | \nOOF_avg_auc | \nOOF_rank_auc | \nOOF_pow_auc | \nOOF_bo_auc | \nOverall_mean | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|---|---|---|
| 8 | \n0.93898 | \n0.93993 | \n0.937304 | \n0.93963 | \n0.93963 | \n0.9522 | \n0.9409 | \n
| 8 | \n0.93787 | \n0.93806 | \n0.93652 | \n0.93918 | \n0.93791 | \n0.9518 | \n0.9411 | \n
| 6 | \n0.938307 | \n0.938737 | \n0.93630 | \n0.93900 | \n0.93808 | \n0.9512 | \n0.9413 | \n
\n\nWhat about using the 8 (or any number of) models that have the best CV when averging the predictions together ?
\n
@theoviel I just averaged all 39 models and it performed identical to hill climbing. Also I randomly chose 8 repeatedly 10,000 times and just averaged them. Then used the 8 with best CV. That also performed the same. So maybe everything just does the same haha.
I know if you only have 2 or 3 models, then finding weights is better than just averaging them but perhaps when you have dozens of models just averaging works as well as finding weights.
\n(I've never had so many models before so I don't have much experience what is best when you have dozens of models)
", "rawMarkdown": "> What about using the 8 (or any number of) models that have the best CV when averging the predictions together ?\n\n@theoviel I just averaged all 39 models and it performed identical to `hill climbing`. Also I randomly chose 8 repeatedly 10,000 times and just averaged them. Then used the 8 with best CV. That also performed the same. So maybe everything just does the same haha.\n\nI know if you only have 2 or 3 models, then finding weights is better than just averaging them but perhaps when you have dozens of models just averaging works as well as finding weights.\n\n(I've never had so many models before so I don't have much experience what is best when you have dozens of models)", "votes": 6 }, { "id": 976462, "postDate": "2020-08-18T21:54:49.323Z", "content": "But i guess there is an advantage to hill climbing or your select 8 method. In the end, you will have fewer models in your ensemble which is usually a good thing.
", "rawMarkdown": "But i guess there is an advantage to hill climbing or your select 8 method. In the end, you will have fewer models in your ensemble which is usually a good thing.", "votes": 2 }, { "id": 977039, "postDate": "2020-08-19T08:52:27.083Z", "content": "Interesting, thanks a lot for checking
", "rawMarkdown": "Interesting, thanks a lot for checking ", "votes": 1 }, { "id": 983075, "postDate": "2020-08-24T03:09:30.487Z", "content": "I discovered that if i set TOL = 0 to allow hill climbing to add any new model that increases CV AUC greater than 0, then the CV and private LB will become higher than averaging all or using a random 8. So perhaps to get the most out of high climbing, you need to set TOL = 0. I'm still unsure what is the optimal value because if an additional model only increase CV AUC 0.0001 it may just be random noise. (That's why I initially choose TOL = 0.0003 in my hill climbing notebook. Perhaps I should analyze this mathematically).
Suppose we are using a model and I have to choose the best hyperparameters for that model. Now, will it be right that for each hyperparameter I do a new cv and note down the best parameters? By doing this am I introducing any leakage?
", "rawMarkdown": "Suppose we are using a model and I have to choose the best hyperparameters for that model. Now, will it be right that for each hyperparameter I do a new cv and note down the best parameters? By doing this am I introducing any leakage?", "votes": 1, "replies": [ { "id": 998899, "postDate": "2020-09-05T07:15:17.133Z", "content": "That is not right in my opinion since the hyperparameters like this themselves might not give the best result in tandem. Choose the cv set with the least loss value and work on that but remember changing the hyperparameters too much would make it biased to your cv set and then might fail at generalizing. There is always another way to make sure, try it yourself once. I hope this helps : ).
", "rawMarkdown": "That is not right in my opinion since the hyperparameters like this themselves might not give the best result in tandem. Choose the cv set with the least loss value and work on that but remember changing the hyperparameters too much would make it biased to your cv set and then might fail at generalizing. There is always another way to make sure, try it yourself once. I hope this helps : ).", "votes": 1 } ] }, { "id": 997407, "postDate": "2020-09-04T03:09:53.400Z", "content": "Thanks for sharing. I do follow these approaches on Stacking while working on the projects.
\nHave you tried Voting Classifiers as well?
A good explaination, was very helpful.
", "rawMarkdown": "A good explaination, was very helpful.", "votes": 1 }, { "id": 987504, "postDate": "2020-08-27T10:04:31.347Z", "content": "Good one
", "rawMarkdown": "Good one", "votes": 1 }, { "id": 983330, "postDate": "2020-08-24T07:37:11.813Z", "content": "Very crisp, clear & to the point explanation. I am a novice at kaggle and I am learning amazing stuff reading your notebook & discussion. Thanks!
", "rawMarkdown": "Very crisp, clear & to the point explanation. I am a novice at kaggle and I am learning amazing stuff reading your notebook & discussion. Thanks!", "votes": 1 }, { "id": 983068, "postDate": "2020-08-24T03:03:32.770Z", "content": "Thanks for sharing great knowledge about ensembling!
\nI didn't know hill climbing approach to ensemble the model and helped me a lot!
", "rawMarkdown": "Thanks for sharing great knowledge about ensembling! \n\nI didn't know hill climbing approach to ensemble the model and helped me a lot! ", "votes": 1 }, { "id": 979255, "postDate": "2020-08-20T18:06:47.880Z", "content": "Because of hardware limitations I have been trying this….
\nfor each epoch in 1:30
\ntrain for 100 steps; in each step use a batch of n images that is a random sample of the train set with augmentations (resampled at each step)
\nvalidate for 30 steps; in each step use a batch of n images that is a random sample of the train set without augmentations (resampled at each step)
n is 16 32 64 depending on image size
\nDifferent runs for the same model got consistent scores between \"this cv score\" and private set… not with public set thas was a bit lower…
\nchanging the set of images at each step may result in underfitting? it certainly did not overfit
", "rawMarkdown": "Because of hardware limitations I have been trying this....\n\nfor each epoch in 1:30\ntrain for 100 steps; in each step use a batch of n images that is a random sample of the train set with augmentations (resampled at each step)\nvalidate for 30 steps; in each step use a batch of n images that is a random sample of the train set without augmentations (resampled at each step)\n\nn is 16 32 64 depending on image size\n\nDifferent runs for the same model got consistent scores between \"this cv score\" and private set... not with public set thas was a bit lower...\n\nchanging the set of images at each step may result in underfitting? it certainly did not overfit", "votes": 1, "replies": [ { "id": 979275, "postDate": "2020-08-20T18:23:29.133Z", "content": "Nice trick Marcelo.
\nNote that if a full epoch has 1000 steps and you are only training on 100 steps, then after 10 of your epochs, you will do the same thing as 1 of the original epochs. The difference would be that you are changing the learning rate more often than it would occur in the original epoch.
\nAnother trick to speed up training is to train on random crops. So if you wish to train a model on 384x384. Then you randomly crop each train image as 192x192 (as data augmentation). Then after training for a while your model will see the entire 384x384 image, but you train fast because your are only processing 192x192
", "rawMarkdown": "Nice trick Marcelo. \n\nNote that if a full epoch has 1000 steps and you are only training on 100 steps, then after 10 of your epochs, you will do the same thing as 1 of the original epochs. The difference would be that you are changing the learning rate more often than it would occur in the original epoch.\n\nAnother trick to speed up training is to train on random crops. So if you wish to train a model on 384x384. Then you randomly crop each train image as 192x192 (as data augmentation). Then after training for a while your model will see the entire 384x384 image, but you train fast because your are only processing 192x192 ", "votes": 1 }, { "id": 979677, "postDate": "2020-08-21T03:55:37.117Z", "content": "That seems very interesting! Can an EffNet model trained with one image size tranfer weights to an EffNet model to predict a larger image size? or it only applies to simpler models?
", "rawMarkdown": "That seems very interesting! Can an EffNet model trained with one image size tranfer weights to an EffNet model to predict a larger image size? or it only applies to simpler models?", "votes": 1 }, { "id": 979693, "postDate": "2020-08-21T04:14:32.493Z", "content": "Yes. You can train an EffNet with 192x192 and then predict with 384x384. (This is what I did in Steel Comp image segmetation, explained here). You can even train where each batch has a different size. One batch can be 256x256 and another batch can be 512x512. An EffNet model doesn't care about input size. An EffNet is just a bunch of convolution filters that get trained. Convolution filters are little 3x3 \"windows\" that move over an image.
\nHowever, I'm not suggesting mixing training sizes and inference sizes in Melanoma Comp (even though it is possible but maybe it would be good to investigate). I'm suggesting using 384x384 but during training always show it a different random 192x192 crop. It will then learn to classify images based on just using a 192x192 piece of a full image. Then during inference, you use TTA=21, so you will have your CNN predict on 21 different 192x192 random pieces (of the same image) and then you average those 21 predictions as your final prediction for that single image. (So the model is always training and inferring on 192x192 pieces).
", "rawMarkdown": "Yes. You can train an EffNet with 192x192 and then predict with 384x384. (This is what I did in Steel Comp image segmetation, explained [here][1]). You can even train where each batch has a different size. One batch can be 256x256 and another batch can be 512x512. An EffNet model doesn't care about input size. An EffNet is just a bunch of convolution filters that get trained. Convolution filters are little 3x3 \"windows\" that move over an image.\n\nHowever, I'm not suggesting mixing training sizes and inference sizes in Melanoma Comp (even though it is possible but maybe it would be good to investigate). I'm suggesting using 384x384 but during training always show it a different random 192x192 crop. It will then learn to classify images based on just using a 192x192 piece of a full image. Then during inference, you use TTA=21, so you will have your CNN predict on 21 different 192x192 random pieces (of the same image) and then you average those 21 predictions as your final prediction for that single image. (So the model is always training and inferring on 192x192 pieces).\n\n[1]: https://www.kaggle.com/c/severstal-steel-defect-detection/discussion/114321", "votes": 2 }, { "id": 980236, "postDate": "2020-08-21T12:33:33.543Z", "content": "Is it possible to train EffNet on 192x192 images just to learn the weights of the kernels to speed up learning at first and then only retrain the FC layers at the tip of the architecture to predict for 384x384 images at the end of training? Are there any drawbacks to this approach?
", "rawMarkdown": "Is it possible to train EffNet on 192x192 images just to learn the weights of the kernels to speed up learning at first and then only retrain the FC layers at the tip of the architecture to predict for 384x384 images at the end of training? Are there any drawbacks to this approach?" }, { "id": 980295, "postDate": "2020-08-21T13:24:39.770Z", "content": "Thank you very much Sir…
\nI found this possibility is very suitable for fine-grained images that are more or less homogeneous. With this approach I could train models with small cropos much more quickly. I will give it a try with data from human protein classification to see how it works …
\n\nIs it possible to train EffNet on 192x192 images just to learn the weights of the kernels to speed up learning at first and then only retrain the FC layers at the tip of the architecture to predict for 384x384 images at the end of training? Are there any drawbacks to this approach?
\n
Yes, you can do this. The best way to see how well it works is try it and evaluate CV LB.
", "rawMarkdown": "> Is it possible to train EffNet on 192x192 images just to learn the weights of the kernels to speed up learning at first and then only retrain the FC layers at the tip of the architecture to predict for 384x384 images at the end of training? Are there any drawbacks to this approach?\n\nYes, you can do this. The best way to see how well it works is try it and evaluate CV LB." }, { "id": 980720, "postDate": "2020-08-21T19:52:49.990Z", "rawMarkdown": "", "isDeleted": true }, { "id": 980721, "postDate": "2020-08-21T19:52:49.997Z", "rawMarkdown": "", "isDeleted": true }, { "id": 980722, "postDate": "2020-08-21T19:52:50.007Z", "content": "And one more thing: The LB scores are very close to one another (.0005 isn't much especially when you take into consideration that test accuracy isn't true real world accuracy) so there must be some randomness involved to who might be 1st beyond just skill. So how do hosts deal with this kind of randomness? Do they deem it insignificant? Is it insignificant?
", "rawMarkdown": "And one more thing: The LB scores are very close to one another (.0005 isn't much especially when you take into consideration that test accuracy isn't true real world accuracy) so there must be some randomness involved to who might be 1st beyond just skill. So how do hosts deal with this kind of randomness? Do they deem it insignificant? Is it insignificant?", "votes": 1 }, { "id": 980885, "postDate": "2020-08-22T01:02:56.293Z", "content": "Data science is both skill and luck. For example, I built a model from meta data: age, sex, site, image size. (By itself it had CV 0.77 LB 0.77). When I added it to my ensemble, It increased my CV by 0.0005. The question is do I include it in my final submission with such a small CV increase? How much can I gain, and how much do I risk?
\nI simulated 1000 private leaderboards. You can see below that in some simulated private leaderboards, including the meta model (as 90% image 10% meta) will decrease my private LB score as much as AUC 0.002! There is also the chance it could increase my private LB 0.003!
\nThe expected increase in private LB score is 0.0005 with standard deviation 0.0008. That means that it will increase my score in 75% of private leaderboards. Should I take the risk? The best we can do is calculate expected values, variances, and risk and decide whether to include things in our models.
\nI did include it and it lowered my private LB by 0.0002 but there was the chance that it could have increased me to Gold !!
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1723677%2Feb01f9d371f4a49be62093d5d2b638bd%2Fmeta.png?generation=1598057575239798&alt=media\")
Thank you so much for investing your time into replying to such a newbie I really appreciate it. Due to the MASSIVE information spread about this field, it's really hard to get yourself around. If it wasn't for people like you, new comers won't have any chance keeping up. Especially if they were self-taught like me. Thank you so much
", "rawMarkdown": "Thank you so much for investing your time into replying to such a newbie I really appreciate it. Due to the MASSIVE information spread about this field, it's really hard to get yourself around. If it wasn't for people like you, new comers won't have any chance keeping up. Especially if they were self-taught like me. Thank you so much", "votes": 2 }, { "id": 1253672, "postDate": "2021-03-26T23:35:26.560Z", "content": "\n\nI simulated 1000 private leaderboards.
\n
@cdeotte may I ask you how one could simulate private leaderboards?
", "rawMarkdown": "> I simulated 1000 private leaderboards.\n\n@cdeotte may I ask you how one could simulate private leaderboards?" } ] }, { "id": 977172, "postDate": "2020-08-19T10:35:08.297Z", "content": "Thanks a lot for sharing and it is really good explanation for oof ensemble!
\nMay I ask something:
\nSome Kagglers save their folds to folds.csv and just load folds.csv for more experiments.
If I use same seed(e.g. random_seed = 42), is it same folds result?
new_folds = sklearn.model_selection.KFold(n_splits = 5, shuffle = True, random_seed = 42)
\nfold.csv == new_folds
Thanks again @cdeotte.
", "rawMarkdown": "Thanks a lot for sharing and it is really good explanation for oof ensemble!\n\nMay I ask something:\nSome Kagglers save their folds to `folds.csv` and just load `folds.csv` for more experiments.\n\nIf I use same seed(`e.g. random_seed = 42`), is it same folds result?\n\n`new_folds = sklearn.model_selection.KFold(n_splits = 5, shuffle = True, random_seed = 42)`\n`fold.csv == new_folds`\n\nThanks again @cdeotte.", "votes": 1, "replies": [ { "id": 977791, "postDate": "2020-08-19T17:53:07.827Z", "content": "Yes. If two different computers use the same random_seed = 42 then they will use the same folds. This is how working on a Kaggle team does it. Every team member uses the same seed on their own computer. Then you can compare everyone's CV score and use everyone's OOF to ensemble your final submission.
Thanks a lot for reply @cdeotte !
\nI want to use what I have learned from you in the next competition. :)
What clarity!
\nI knew that intuitively, but I wasn't confident enough in my methods to implement it. I want to take advantage of this in the future.
Simple yet very efficient, thanks @cdeotte , for me it seems obvious now, but before this competition I never saved my model's OOF predictions, after seeing your public notebook this is a practice that I will keep from now on!
", "rawMarkdown": "Simple yet very efficient, thanks @cdeotte , for me it seems obvious now, but before this competition I never saved my model's OOF predictions, after seeing your public notebook this is a practice that I will keep from now on!", "votes": 1, "replies": [ { "id": 976604, "postDate": "2020-08-19T01:19:06.400Z", "content": "Yes. For the past 2 months, I save every model's (1) OOF (2) Submission CSV (3) Model weights. These things always come in handy later.
\nFor example, maybe we want to build a model that reuses 3 models' weights then build a new model using those three backbones, concatenate the outputs and train a new MLP on top of that.
\nOr maybe we want to extract embeddings from old model weight backbones, then use XGB to build a head, etc etc.
", "rawMarkdown": "Yes. For the past 2 months, I save every model's (1) OOF (2) Submission CSV (3) Model weights. These things always come in handy later.\n\nFor example, maybe we want to build a model that reuses 3 models' weights then build a new model using those three backbones, concatenate the outputs and train a new MLP on top of that.\n\nOr maybe we want to extract embeddings from old model weight backbones, then use XGB to build a head, etc etc.", "votes": 5 } ] }, { "id": 976583, "postDate": "2020-08-19T00:52:34.967Z", "content": "Could you clarify why you do a for loop instead of fitting a logistic regression or other optimization approaches?
", "rawMarkdown": "Could you clarify why you do a for loop instead of fitting a logistic regression or other optimization approaches?", "votes": 1, "replies": [ { "id": 976598, "postDate": "2020-08-19T01:15:16.573Z", "content": "I've tried logistic regression, linear regression and XGB. For some reason hill climbing achieves a higher ensemble CV.
(Not sure why. Maybe I'm doing the other methods wrong).
", "rawMarkdown": "I've tried logistic regression, linear regression and XGB. For some reason `hill climbing` achieves a higher ensemble CV. \n\n(Not sure why. Maybe I'm doing the other methods wrong).", "votes": 1 }, { "id": 976688, "postDate": "2020-08-19T03:30:10.147Z", "content": "Very interesting. I see many top kagglers use the same approach as yours, so there might be some underlying reasons.
\nBy the way thank you for all of your hard work!
", "rawMarkdown": "Very interesting. I see many top kagglers use the same approach as yours, so there might be some underlying reasons.\n\nBy the way thank you for all of your hard work! ", "votes": 1 }, { "id": 983070, "postDate": "2020-08-24T03:04:21.437Z", "content": "Note that my forward selection notebook here uses TOL = 0.0003. I'm still unsure what is the best selection for this variable. (This is CV AUC threshold increase for adding new models and I didn't want to clutter the ensemble with unnecessary additional models). Also I use DUPLICATES = False. That means that after it adds a model, it won't add that model again.
But now that I think about it, it would be better to allow adding the same model again because that is equivalent to adding more weight to that model's coefficient. (Kind of how logistic regression or linear regression adjusts weights to find optimal).
\nSo, FYI, if you use TOL = 0 and DUPLICATES = True in my public notebook, the CV gets up to 0.9507 and the private LB gets up to 0.9433! Therefore if you try to compare high climbing to logistic regression or linear regression, those are the CV and LB, you need to beat.
\n\nAlso every model should use the same folds.
\n
Is it necessary to have the same folds? I guess we can have different folds OOF csv files. Will it cause leak?
", "rawMarkdown": "> Also every model should use the same folds. \n\nIs it necessary to have the same folds? I guess we can have different folds OOF csv files. Will it cause leak?", "votes": 1, "replies": [ { "id": 976454, "postDate": "2020-08-18T21:38:55.690Z", "content": "It won't cause a leak but you will get artificial CV gains. If one OOF uses folds A, B, C, D, E. Then every train image in fold A is predicted using 80% of train (B, C, D, E). Let's call one image in fold A, img_123. If a second OOF uses folds F, G, H, I, J. Then img_123 is in fold G and it gets predicted with the 80% of train in F, H, I, J.
The 80% that predicted img_123 in OOF_1 is different than the 80% that predicted img_123 in OOF_2. Therefore you will most certainly get an increase when you average OOF_1 and OOF_2. So using different folds gives increases even when the two models are similar and not diverse.
However when you blend SUB_1 and SUB_2 you will not get an increase because both are already similar. In conclusion, using different folds gives increases in CV which do not give increases in SUB blending.
", "rawMarkdown": "It won't cause a leak but you will get artificial CV gains. If one OOF uses folds A, B, C, D, E. Then every train image in fold A is predicted using 80% of train (B, C, D, E). Let's call one image in fold A, `img_123`. If a second OOF uses folds F, G, H, I, J. Then `img_123` is in fold G and it gets predicted with the 80% of train in F, H, I, J. \n\nThe 80% that predicted `img_123` in OOF_1 is different than the 80% that predicted `img_123` in OOF_2. Therefore you will most certainly get an increase when you average OOF_1 and OOF_2. So using different folds gives increases even when the two models are similar and not diverse.\n\nHowever when you blend SUB_1 and SUB_2 you will not get an increase because both are already similar. In conclusion, using different folds gives increases in CV which do not give increases in SUB blending.", "votes": 7 }, { "id": 976461, "postDate": "2020-08-18T21:53:53.423Z", "content": "Thanks @cdeotte I can see the reason why CV score increases. But I think SUB_1 and SUB_2 will be slightly different because models are trained on different folds of data. So my feeling is that if we run different folds OOF ensemble, CV will increase and LB will increase small or no increase.
", "rawMarkdown": "Thanks @cdeotte I can see the reason why CV score increases. But I think SUB_1 and SUB_2 will be slightly different because models are trained on different folds of data. So my feeling is that if we run different folds OOF ensemble, CV will increase and LB will increase small or no increase." }, { "id": 976638, "postDate": "2020-08-19T02:12:07.977Z", "content": "SUB_1 and SUB2_1 may be slightly different. But if their combination increases public LB, it is a different reason than combining OOF_1 and OOF_2. Therefore the weights that optimize the increase of OOF_1 plus OOF_2 isn't necessarily the same weights that would optimize SUB_1 plus SUB_2.
", "rawMarkdown": "SUB_1 and SUB2_1 may be slightly different. But if their combination increases public LB, it is a different reason than combining OOF_1 and OOF_2. Therefore the weights that optimize the increase of OOF_1 plus OOF_2 isn't necessarily the same weights that would optimize SUB_1 plus SUB_2.", "votes": 1 } ] }, { "id": 976391, "postDate": "2020-08-18T20:18:19.303Z", "content": "This is great. As a beginner, ensembling has always puzzled me, What are more creative ways to do ensembling?
", "rawMarkdown": "This is great. As a beginner, ensembling has always puzzled me, What are more creative ways to do ensembling?", "votes": 1, "replies": [ { "id": 976412, "postDate": "2020-08-18T20:34:40.413Z", "content": "What we're doing here is similar to linear regression or logistic regression. We are using the OOF as \"train data\" where each OOF is one column feature and we are predicting target. Since our model is linear, we don't worry above overfitting (which seldom occurs for linear models).
\nInstead of linear regression, we can use any model as our \"ensemble model\". For example we could train an XGB using the OOF to predict target. Also we can add additional features. For example, some columns can be the OOF and other columns can be meta data like age, sex, site, etc.
\nEssentially this is stacking.
\nX_train[:,0] = oof_1.predictions\nX_train[:,1] = oof_2.predictions\nX_train[:,2] = oof_3.predictions\nX_train[:,3] = oof.age\nX_train[:,4] = oof.sex\ny_train = oof.target\n\nmodel.fit(X_train,y_train)\n\nX_test[:,0] = sub_1.target\nX_test[:,1] = sub_2.target\nX_test[:,2] = sub_3.target\nX_test[:,3] = sub.age\nX_test[:,4] = sub.sex\n\nkaggle_sub = model.predict(X_test)\nAfterward we have predictions for every training image. This complete set of predictions is called OOF, i think it should be test images
No. Fold 1, Fold 2, Fold 3, Fold 4, Fold 5 are each 20% of the train images. So when we build a model with Fold 2, 3, 4, 5 and predict Fold 1. We now have predictions for the 20% of train images in Fold 1.
\nAfter we do this five times, we have predictions for 100% of train images. These are called OOF
\nAdditionally, each fold model also makes predictions on the test images. After averaging these 5 sets of predictions, these are called SUB
", "rawMarkdown": "No. Fold 1, Fold 2, Fold 3, Fold 4, Fold 5 are each 20% of the train images. So when we build a model with Fold 2, 3, 4, 5 and predict Fold 1. We now have predictions for the 20% of train images in Fold 1.\n\nAfter we do this five times, we have predictions for 100% of train images. These are called OOF\n\nAdditionally, each fold model also makes predictions on the test images. After averaging these 5 sets of predictions, these are called SUB", "votes": 3 } ] }, { "id": 976370, "postDate": "2020-08-18T20:13:25.350Z", "content": "Gonna bookmark this! thanks for the detailed explanation.
", "rawMarkdown": "Gonna bookmark this! thanks for the detailed explanation.", "votes": 1 }, { "id": 976311, "postDate": "2020-08-18T19:25:47.447Z", "content": "thanks @cdeotte !
\nI used your starter notebook and had 12 models.
\ni did ensembling using another starter kernel.
\nthis is my final ensemble with 6 models (Private LB = 0.9413).
\nhttps://www.kaggle.com/mpsampat/final-simple-oof-ensembling-methods-6-models
I had a hard time to find the optimal subset of models. Your starter notebook will be very handy in learning this task!
\nthanks again for sharing this and other notebooks!
\nCheers!
\n\nFor each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image.
\n
Amazing article. I keep revisiting this again & again. I was wondering how there are 5 predictions for the test image since the 5 different folds will be a part of single model? So essentially, one model will predict one output for a single test image? I understand for different models the prediction of test image will vary but how is it varying for single model (and different folds)? Do we take the average for test image output for different models or for different folds of the same model?
", "rawMarkdown": "> For each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image.\n\n\nAmazing article. I keep revisiting this again & again. I was wondering how there are 5 predictions for the test image since the 5 different folds will be a part of single model? So essentially, one model will predict one output for a single test image? I understand for different models the prediction of test image will vary but how is it varying for single model (and different folds)? Do we take the average for test image output for different models or for different folds of the same model?", "votes": 2, "replies": [ { "id": 986903, "postDate": "2020-08-26T21:34:31.537Z", "content": "In 5 KFold, you actually build 5 models. All models use the same architecture and hyperparameters so they will perform similarily, but each is trained on a different 80% of data. (Their 4 folds).
\nSo after performing 5 KFold, we have 5 models and each model predicts all the test images. Therefore each test image has 5 predictions (where each pred is between 0 and 1). We take the simple average of these 5 numbers as the prediction for test.
", "rawMarkdown": "In 5 KFold, you actually build 5 models. All models use the same architecture and hyperparameters so they will perform similarily, but each is trained on a different 80% of data. (Their 4 folds).\n\nSo after performing 5 KFold, we have 5 models and each model predicts all the test images. Therefore each test image has 5 predictions (where each pred is between 0 and 1). We take the simple average of these 5 numbers as the prediction for test.", "votes": 1 }, { "id": 987068, "postDate": "2020-08-27T00:48:28.893Z", "content": "@ujjwal29jain You can think about CV KFold like a random forest. After building a random forest, you call it \"one model\" but really, the forest is many models each trained with a different 80% of the data and then ensembled together.
\nSimilarily, KFold is K models each trained with a different 80% of the data and then ensembled together. (And both help prevent overfitting and generalize better to unseen data).
", "rawMarkdown": "@ujjwal29jain You can think about CV KFold like a random forest. After building a random forest, you call it \"one model\" but really, the forest is many models each trained with a different 80% of the data and then ensembled together.\n\nSimilarily, KFold is K models each trained with a different 80% of the data and then ensembled together. (And both help prevent overfitting and generalize better to unseen data).", "votes": 3 }, { "id": 1110842, "postDate": "2020-12-13T06:07:41.173Z", "content": "@cdeotte Dear Chris, was revisiting your old posts, and as usual, I have noticed that for such classification problems (and even the recent Cassava Comp), I generally build 5 folds and average out the predictions towards the end when I inference. Usually, this produces better results - but is there any statistical justification for this. Appreciate it if you can link me to one article that explains so…
", "rawMarkdown": "@cdeotte Dear Chris, was revisiting your old posts, and as usual, I have noticed that for such classification problems (and even the recent Cassava Comp), I generally build 5 folds and average out the predictions towards the end when I inference. Usually, this produces better results - but is there any statistical justification for this. Appreciate it if you can link me to one article that explains so..." } ] }, { "id": 976706, "postDate": "2020-08-19T03:49:00.227Z", "content": "Thanks for sharing Chris! Probably the clearest explanation of this I’ve seen.
\nIs it necessary to use the same folds you used to train your models or can this CV scheme be independent of the initial scheme?
", "rawMarkdown": "Thanks for sharing Chris! Probably the clearest explanation of this I’ve seen.\n\nIs it necessary to use the same folds you used to train your models or can this CV scheme be independent of the initial scheme?", "votes": 2, "replies": [ { "id": 976758, "postDate": "2020-08-19T04:34:36.367Z", "content": "You should always use the same folds. Otherwise if you ensemble models using different folds, you will always see a CV increase but the LB may not increase.
", "rawMarkdown": "You should always use the same folds. Otherwise if you ensemble models using different folds, you will always see a CV increase but the LB may not increase.", "votes": 3 }, { "id": 976947, "postDate": "2020-08-19T07:30:46.203Z", "content": "Chris, can you please elaborate this a bit further?
\nBecause, in this competition I blended different models using oof (each having a different seed value, merged oof files using image id) and it seems to have increased both CV and LB(public and private).
", "rawMarkdown": "Chris, can you please elaborate this a bit further? \n\nBecause, in this competition I blended different models using oof (each having a different seed value, merged oof files using image id) and it seems to have increased both CV and LB(public and private).", "votes": 2 }, { "id": 981999, "postDate": "2020-08-23T00:02:14.210Z", "content": "@manjeshg03 Using different seeds does add variety to your models. However it is hard to compare the models. For example if you have one model with CV 0.920 using seed=42 and another with CV 0.922 with seed=123, which is better? We don't know.
\nAlso if you ensemble two identical models where one uses seed=42 and the another uses seed=123, the ensemble CV score will always increase. This is because a 5 fold model only uses 80% of train data to predict the OOF. When you ensemble two different seeds, the result are predictions that uses more than 80% (most likely around 95%) of train data for each OOF prediction.
\nHowever when you ensemble the submission.csv files of two identical models where one uses seed=42 and the another uses seed=123, you won't automatically have an increase. Because each model already uses the full 100% train to make the submission file (as result of blending the 5 folds together). So, the CV will always increase but the LB will not always increase. This makes using different seeds a confusing situation. (Not necessarily a bad situation, but certainly a confusing situation to evaluate CV LB).
", "rawMarkdown": "@manjeshg03 Using different seeds does add variety to your models. However it is hard to compare the models. For example if you have one model with CV 0.920 using seed=42 and another with CV 0.922 with seed=123, which is better? We don't know.\n\nAlso if you ensemble two identical models where one uses seed=42 and the another uses seed=123, the ensemble CV score will always increase. This is because a 5 fold model only uses 80% of train data to predict the OOF. When you ensemble two different seeds, the result are predictions that uses more than 80% (most likely around 95%) of train data for each OOF prediction.\n\nHowever when you ensemble the submission.csv files of two identical models where one uses seed=42 and the another uses seed=123, you won't automatically have an increase. Because each model already uses the full 100% train to make the submission file (as result of blending the 5 folds together). So, the CV will always increase but the LB will **not** always increase. This makes using different seeds a confusing situation. (Not necessarily a bad situation, but certainly a confusing situation to evaluate CV LB).", "votes": 11 }, { "id": 984837, "postDate": "2020-08-25T10:20:51.067Z", "content": "@cdeotte Got the point. Thanks a lot!
", "rawMarkdown": "@cdeotte Got the point. Thanks a lot!", "votes": 1 } ] }, { "id": 976329, "postDate": "2020-08-18T19:42:00.883Z", "content": "Nice brute force algorithm. I will definitely give it a try to improve my optimization techniques. How would you do this for OOF predictions for models of different seeds?
", "rawMarkdown": "Nice brute force algorithm. I will definitely give it a try to improve my optimization techniques. How would you do this for OOF predictions for models of different seeds?", "votes": 2, "replies": [ { "id": 976639, "postDate": "2020-08-19T02:12:34.507Z", "content": "I would suggest training a NN on the OOF predictions, adding a small amount of noise as data augmentation.
\nAs Chris has pointed out below - if you train exactly the same architecture twice, with different seeds each time, then the OOF predictions of the two trained models will be different, but their submissions will be similar.
\nI think this can be overcome using noise because I found, with my own submissions, that I could almost perfectly distinguish the combined OOF predictions ([oof_1,oof_2,…,oof_n]) from the combined submissions ([sub_1,sub_2,…,sub_n]). Meaning that the distributions were slightly different. This isn't surprising because each OOF prediction is made by a single model, while each submission prediction comes from a combination of models. However, I found that I could partially avoid this issue by using a small amount of gaussian noise as augmentation while training a NN. With a small amount of noise the two sets could not be distinguished very well. Performing TTA with this noise and using stratified cross validation I trained a model that combined my OOF predictions slightly better than a straightforward linear blend in terms of CV score.
\nWhat I am saying is that if the right amount of noise is added (something with less variance than that of the model that is having noise added to it) then the combined OOF predictions are difficult to distinguish from the combined submissions. As this was the case when using the same seed every time, I imagine it will also be the case when you do not have the same seed. And, in your case, it would also ensure that any of the OOF predictions that are similar become difficult to distinguish, removing their ability to provide artificial gains in the combined OOF predictions.
\nI hope that makes sense, and it would be interesting to know if it works for you!
", "rawMarkdown": "I would suggest training a NN on the OOF predictions, adding a small amount of noise as data augmentation. \n\nAs Chris has pointed out below - if you train exactly the same architecture twice, with different seeds each time, then the OOF predictions of the two trained models will be different, but their submissions will be similar. \n\nI think this can be overcome using noise because I found, with my own submissions, that I could almost perfectly distinguish the combined OOF predictions ([oof_1,oof_2,...,oof_n]) from the combined submissions ([sub_1,sub_2,...,sub_n]). Meaning that the distributions were slightly different. This isn't surprising because each OOF prediction is made by a single model, while each submission prediction comes from a combination of models. However, I found that I could partially avoid this issue by using a small amount of gaussian noise as augmentation while training a NN. With a small amount of noise the two sets could not be distinguished very well. Performing TTA with this noise and using stratified cross validation I trained a model that combined my OOF predictions slightly better than a straightforward linear blend in terms of CV score.\n\nWhat I am saying is that if the right amount of noise is added (something with less variance than that of the model that is having noise added to it) then the combined OOF predictions are difficult to distinguish from the combined submissions. As this was the case when using the same seed every time, I imagine it will also be the case when you do not have the same seed. And, in your case, it would also ensure that any of the OOF predictions that are similar become difficult to distinguish, removing their ability to provide artificial gains in the combined OOF predictions.\n\nI hope that makes sense, and it would be interesting to know if it works for you!", "votes": 1 } ] }, { "id": 2919597, "postDate": "2024-07-13T03:10:49.583Z", "content": "@cdeotte, this was a nice discussion. But I was wondering maybe if you could give some advice in context of code competitions, like the current ISIC competition. Here, we do not have the entire test set, and hence need to do the full inference within the limit of 9 hours.
\nSay, I have trained using 5-fold cross validation. So essentially I have 5 models to use for inference. Which one should I choose? Infering using all of them and then averaging just takes too much time.
", "rawMarkdown": "@cdeotte, this was a nice discussion. But I was wondering maybe if you could give some advice in context of code competitions, like the [current ISIC competition](https://www.kaggle.com/competitions/isic-2024-challenge). Here, we do not have the entire test set, and hence need to do the full inference within the limit of 9 hours. \n\nSay, I have trained using 5-fold cross validation. So essentially I have 5 models to use for inference. Which one should I choose? Infering using all of them and then averaging just takes too much time." }, { "id": 2017546, "postDate": "2022-11-04T23:54:03.187Z", "content": "The best concept I've ever seen. And I always learn new things from you.
", "rawMarkdown": "The best concept I've ever seen. And I always learn new things from you." }, { "id": 1871067, "postDate": "2022-07-26T04:03:15.060Z", "content": "Thank you for sharing your Hill Climbing approach.
", "rawMarkdown": "Thank you for sharing your Hill Climbing approach." }, { "id": 1822540, "postDate": "2022-06-16T13:40:08.877Z", "content": "thanks for sharing. hill climbing,good job.
", "rawMarkdown": "thanks for sharing. hill climbing,good job." }, { "id": 1729854, "postDate": "2022-03-20T15:53:22.940Z", "content": "Thankyou helped a lot
", "rawMarkdown": "Thankyou helped a lot" }, { "id": 1654624, "postDate": "2022-01-18T17:11:02.800Z", "content": "Thanks for sharing this!
\nIts been really helpful.
Great post! Crisp and Clear! Thanks for sharing
", "rawMarkdown": "Great post! Crisp and Clear! Thanks for sharing" }, { "id": 976390, "postDate": "2020-08-18T20:18:06.273Z", "content": "Are you by finding these optimal weights not overfitting the validation set?
", "rawMarkdown": "Are you by finding these optimal weights not overfitting the validation set?", "replies": [ { "id": 976409, "postDate": "2020-08-18T20:26:26.717Z", "content": "These weights create a linear model. A line rarely overfits anything.
\nIf you trained XGB using the OOF files, then you could overfit validation and must be careful.
", "rawMarkdown": "These weights create a linear model. A line rarely overfits anything.\n\nIf you trained XGB using the OOF files, then you could overfit validation and must be careful. \n\n", "votes": 7 }, { "id": 977235, "postDate": "2020-08-19T11:13:17.920Z", "content": "Buts let's assume we are ensembling multiple models:
\nsubmission = 0.45 * model1 + 0.03 * model2 + … + 0.11 modelx
\nThis way you are just selecting the highest score on the validation set, which may be overfitting it and not generalizing well at all.
\nI guess what you do is not add all the models at once?
", "rawMarkdown": "Buts let's assume we are ensembling multiple models:\n\nsubmission = 0.45 * model1 + 0.03 * model2 + ... + 0.11 modelx\n\nThis way you are just selecting the highest score on the validation set, which may be overfitting it and not generalizing well at all.\n\nI guess what you do is not add all the models at once?" } ] }, { "id": 1946594, "postDate": "2022-09-19T23:39:58.900Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1947541, "postDate": "2022-09-20T14:14:43.423Z", "content": "No, because the model will already have been trained on the validation data. Doing this will introduce leakage
", "rawMarkdown": "No, because the model will already have been trained on the validation data. Doing this will introduce leakage", "votes": 1 } ] }, { "id": 987263, "postDate": "2020-08-27T05:58:34.753Z", "content": "Great stuff. Thanks for explaining
", "rawMarkdown": "Great stuff. Thanks for explaining ", "votes": 1 }, { "id": 986044, "postDate": "2020-08-26T06:57:01.327Z", "content": "nice explanation
\nThanks for sharing
thanks for sharing this!
", "rawMarkdown": "thanks for sharing this!", "votes": 1 }, { "id": 976322, "postDate": "2020-08-18T19:35:36.740Z", "content": "Thanks sir
", "rawMarkdown": "Thanks sir", "votes": 1 }, { "id": 976303, "postDate": "2020-08-18T19:22:13.703Z", "content": "This the light!! Thanks!
", "rawMarkdown": "This the light!! Thanks!", "votes": 1 }, { "id": 1797454, "postDate": "2022-05-22T01:26:52.757Z", "content": "Thanks for sharing, extremely helpful!!
", "rawMarkdown": "Thanks for sharing, extremely helpful!!" } ], "comments": [ { "id": 976302, "author_name": "Psi", "author_url": "", "post_date": "2020-08-18T19:21:45.140000", "content": "Kagglers refer to this as hill climbing just as a side note if others see this phrase somewhere else.
thanks. I updated my post.
", "votes": 3, "replies": [] }, { "id": 984401, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-25T04:46:25.477000", "content": "@philippsinger When you and others use hill climbing, do you continue adding models as long as CV increases, or do you only add a new model if it increases CV by some tolerance?
For my submission, i used TOL = 0.0003 thinking I would prevent noise. But if you compare hill climbing to XGB stacking, then XGB uses TOL = 0, right? XGB just keeps adding as long as CV increases.
@cdeotte I usually apply it for a certain number of iterations. To avoid overfitting you can also add certain restrictions, like a single model can only be added once or twice.
", "votes": 1, "replies": [] } ] }, { "id": 979176, "author_name": "Qishen Ha", "author_url": "", "post_date": "2020-08-20T17:12:15.270000", "content": "Thanks for sharing this! Good explanation for CV!
\nAfter reading some discussions in the past few days after the competition end, I gradually realized that what more people need is not the top solutions, but the basic ML knowledge, just like this posts.
\nI see many people still don't understand why they should not trust public LB score (especially in this comp). But I don’t know how to explain to them in a simple and understandable way.
\nI think your post can be a good explanation for it as well. Thanks again!
", "votes": 14, "replies": [ { "id": 979203, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-20T17:31:26.207000", "content": "Thanks Qishen. I updated the discussion title to `\"How To CV and How To Ensemble OOF\". I thought i was just explaining ensemble, but you are right, this is a nice explanation of CV too.
\nCongrats again on you and your team's awesome model with amazing 0.960 CV!
", "votes": 7, "replies": [] } ] }, { "id": 1654315, "author_name": "Slawek Biel", "author_url": "", "post_date": "2022-01-18T11:35:17.327000", "content": "Thanks @cdeotte, your Hill Climbing approach is a great trick. I'll for sure try it out in my next competition.
\nI have small doubt, in the post you've linked you say
\n\nDuring the competition, every model should use the same folds.
\n
Does that really matter when your oof predictions have all the train samples anyway? I specifically ask in a context of ensembling models as a team, where different members worked with different splits.
", "votes": 2, "replies": [] }, { "id": 988805, "author_name": "datasaurus", "author_url": "", "post_date": "2020-08-28T09:42:28.270000", "content": "Hi @cdeotte, have you seen this paper \"On Cross-Validation and Stacking: Building seemingly predictive models on random data\"?
\nhttps://www.kdd.org/exploration_files/v12-02-4-UR-Perlich.pdf
I'd be interested to know what you and other Kagglers think about it
", "votes": 3, "replies": [ { "id": 989300, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-28T17:42:23.213000", "content": "Thanks for the paper, i'll check it out and post my reaction here.
", "votes": 4, "replies": [] } ] }, { "id": 979325, "author_name": "Massimiliano Viola", "author_url": "", "post_date": "2020-08-20T19:03:15.597000", "content": "Hi Chris, the explanation is very clear but I have a question and after reading the post a couple of times I can't figure out the answer on my own. Talking about the submission file, you say
\n\n\nFor each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image. We take the average of these 5 sets of predictions and this is our submission.csv file that we submit to Kaggle. When you submit this to Kaggle, your LB score should be similar to your CV score.
\n
and this makes perfect sense, but why do you never train a model using all train data once you have a high CV score for that model?
\nIs it because you would not know how it performs when trained with an additional fold, making all the work useless and destroying the ensembling process?
\nI am asking because so far, when I perform cross-validation, I always fit the model on all train data with the best parameters to make the test predictions and so your way of doing it by averaging is completely new to me.
\nThank you!
Doing it as I describe associates the CV to LB score. Because the identical models that produced the CV score are the same models producing the LB score. (When you run twice as your describe, you don't have this guarantee). That being said, I sometimes do what you suggest. Both ways have pros and cons.
\nNote that the way I describe in this post is a form of bagging similar to a random forest. Each of the 5 fold models are training on a random 80% of the train data. Therefore even if each of the five fold models overfits their 80% train data, the ensemble of the 5 folds will not overfit the entire 100% train data. So predicting test using CV is also a way to prevent overfitting the train data (i.e. generalizing better to test data) in the same way that a random forest uses bagging to prevent overfitting train data.
", "votes": 5, "replies": [] }, { "id": 979418, "author_name": "Massimiliano Viola", "author_url": "", "post_date": "2020-08-20T20:31:06.320000", "content": "Ok, got it, thank you so much.
\nBy doing this, you are training on less data but as a tradeoff you have a more reliable estimate of the performance of your model and also a free ensemble of 5 classifiers which is never a bad thing.
\nI guess I'll try to apply this method in House Prices competition when I have my final model ready!
The strategy I usually go for is finding a set of models that give the best CV when averaged together.
\nDo you have any idea how your ensemble performs (especially on the leaderboard) when giving equal weights to every model ?
\nWhat about using the 8 (or any number of) models that have the best CV when averging the predictions together ?
This would be nice to see how much hill climbing actually helps.
", "votes": 4, "replies": [ { "id": 976425, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T20:53:24.477000", "content": "Great suggestions Theo. I will investigate this.
\nI should probably change the language in my post. I should be more clear that high climbing is only one way and there are many others like the ones you suggest.
", "votes": 1, "replies": [] }, { "id": 976447, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-18T21:22:20.207000", "content": "\n\nI should probably change the language in my post. I should be more clear that high climbing is only one way and there are many others like the ones you suggest.
\n
No worries, this was clear to me :)
", "votes": 1, "replies": [] }, { "id": 976452, "author_name": "Mehul Sampat", "author_url": "", "post_date": "2020-08-18T21:29:34.490000", "content": "Interesting! thanks for sharing.
\n| no_of_models | \nOOF_avg_auc | \nOOF_rank_auc | \nOOF_pow_auc | \nOOF_bo_auc | \nOverall_mean | \nPublic LB | \nPrivate LB | \n
|---|---|---|---|---|---|---|---|
| 8 | \n0.93898 | \n0.93993 | \n0.937304 | \n0.93963 | \n0.93963 | \n0.9522 | \n0.9409 | \n
| 8 | \n0.93787 | \n0.93806 | \n0.93652 | \n0.93918 | \n0.93791 | \n0.9518 | \n0.9411 | \n
| 6 | \n0.938307 | \n0.938737 | \n0.93630 | \n0.93900 | \n0.93808 | \n0.9512 | \n0.9413 | \n
\n\nWhat about using the 8 (or any number of) models that have the best CV when averging the predictions together ?
\n
@theoviel I just averaged all 39 models and it performed identical to hill climbing. Also I randomly chose 8 repeatedly 10,000 times and just averaged them. Then used the 8 with best CV. That also performed the same. So maybe everything just does the same haha.
I know if you only have 2 or 3 models, then finding weights is better than just averaging them but perhaps when you have dozens of models just averaging works as well as finding weights.
\n(I've never had so many models before so I don't have much experience what is best when you have dozens of models)
", "votes": 6, "replies": [] }, { "id": 976462, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T21:54:49.323000", "content": "But i guess there is an advantage to hill climbing or your select 8 method. In the end, you will have fewer models in your ensemble which is usually a good thing.
", "votes": 2, "replies": [] }, { "id": 977039, "author_name": "Theo Viel", "author_url": "", "post_date": "2020-08-19T08:52:27.083000", "content": "Interesting, thanks a lot for checking
", "votes": 1, "replies": [] }, { "id": 983075, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-24T03:09:30.487000", "content": "I discovered that if i set TOL = 0 to allow hill climbing to add any new model that increases CV AUC greater than 0, then the CV and private LB will become higher than averaging all or using a random 8. So perhaps to get the most out of high climbing, you need to set TOL = 0. I'm still unsure what is the optimal value because if an additional model only increase CV AUC 0.0001 it may just be random noise. (That's why I initially choose TOL = 0.0003 in my hill climbing notebook. Perhaps I should analyze this mathematically).
Suppose we are using a model and I have to choose the best hyperparameters for that model. Now, will it be right that for each hyperparameter I do a new cv and note down the best parameters? By doing this am I introducing any leakage?
", "votes": 1, "replies": [ { "id": 998899, "author_name": "NoMalady", "author_url": "", "post_date": "2020-09-05T07:15:17.133000", "content": "That is not right in my opinion since the hyperparameters like this themselves might not give the best result in tandem. Choose the cv set with the least loss value and work on that but remember changing the hyperparameters too much would make it biased to your cv set and then might fail at generalizing. There is always another way to make sure, try it yourself once. I hope this helps : ).
", "votes": 1, "replies": [] } ] }, { "id": 997407, "author_name": "Blesson Densil", "author_url": "", "post_date": "2020-09-04T03:09:53.400000", "content": "Thanks for sharing. I do follow these approaches on Stacking while working on the projects.
\nHave you tried Voting Classifiers as well?
A good explaination, was very helpful.
", "votes": 1, "replies": [] }, { "id": 987504, "author_name": "Ravi Singh", "author_url": "", "post_date": "2020-08-27T10:04:31.347000", "content": "Good one
", "votes": 1, "replies": [] }, { "id": 983330, "author_name": "UjjwalJain", "author_url": "", "post_date": "2020-08-24T07:37:11.813000", "content": "Very crisp, clear & to the point explanation. I am a novice at kaggle and I am learning amazing stuff reading your notebook & discussion. Thanks!
", "votes": 1, "replies": [] }, { "id": 983068, "author_name": "lr", "author_url": "", "post_date": "2020-08-24T03:03:32.770000", "content": "Thanks for sharing great knowledge about ensembling!
\nI didn't know hill climbing approach to ensemble the model and helped me a lot!
", "votes": 1, "replies": [] }, { "id": 979255, "author_name": "Marcelo Kittlein", "author_url": "", "post_date": "2020-08-20T18:06:47.880000", "content": "Because of hardware limitations I have been trying this….
\nfor each epoch in 1:30
\ntrain for 100 steps; in each step use a batch of n images that is a random sample of the train set with augmentations (resampled at each step)
\nvalidate for 30 steps; in each step use a batch of n images that is a random sample of the train set without augmentations (resampled at each step)
n is 16 32 64 depending on image size
\nDifferent runs for the same model got consistent scores between \"this cv score\" and private set… not with public set thas was a bit lower…
\nchanging the set of images at each step may result in underfitting? it certainly did not overfit
", "votes": 1, "replies": [ { "id": 979275, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-20T18:23:29.133000", "content": "Nice trick Marcelo.
\nNote that if a full epoch has 1000 steps and you are only training on 100 steps, then after 10 of your epochs, you will do the same thing as 1 of the original epochs. The difference would be that you are changing the learning rate more often than it would occur in the original epoch.
\nAnother trick to speed up training is to train on random crops. So if you wish to train a model on 384x384. Then you randomly crop each train image as 192x192 (as data augmentation). Then after training for a while your model will see the entire 384x384 image, but you train fast because your are only processing 192x192
", "votes": 1, "replies": [] }, { "id": 979677, "author_name": "Marcelo Kittlein", "author_url": "", "post_date": "2020-08-21T03:55:37.117000", "content": "That seems very interesting! Can an EffNet model trained with one image size tranfer weights to an EffNet model to predict a larger image size? or it only applies to simpler models?
", "votes": 1, "replies": [] }, { "id": 979693, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-21T04:14:32.493000", "content": "Yes. You can train an EffNet with 192x192 and then predict with 384x384. (This is what I did in Steel Comp image segmetation, explained here). You can even train where each batch has a different size. One batch can be 256x256 and another batch can be 512x512. An EffNet model doesn't care about input size. An EffNet is just a bunch of convolution filters that get trained. Convolution filters are little 3x3 \"windows\" that move over an image.
\nHowever, I'm not suggesting mixing training sizes and inference sizes in Melanoma Comp (even though it is possible but maybe it would be good to investigate). I'm suggesting using 384x384 but during training always show it a different random 192x192 crop. It will then learn to classify images based on just using a 192x192 piece of a full image. Then during inference, you use TTA=21, so you will have your CNN predict on 21 different 192x192 random pieces (of the same image) and then you average those 21 predictions as your final prediction for that single image. (So the model is always training and inferring on 192x192 pieces).
", "votes": 2, "replies": [] }, { "id": 980236, "author_name": "DarkCube", "author_url": "", "post_date": "2020-08-21T12:33:33.543000", "content": "Is it possible to train EffNet on 192x192 images just to learn the weights of the kernels to speed up learning at first and then only retrain the FC layers at the tip of the architecture to predict for 384x384 images at the end of training? Are there any drawbacks to this approach?
", "votes": 0, "replies": [] }, { "id": 980295, "author_name": "Marcelo Kittlein", "author_url": "", "post_date": "2020-08-21T13:24:39.770000", "content": "Thank you very much Sir…
\nI found this possibility is very suitable for fine-grained images that are more or less homogeneous. With this approach I could train models with small cropos much more quickly. I will give it a try with data from human protein classification to see how it works …
\n\nIs it possible to train EffNet on 192x192 images just to learn the weights of the kernels to speed up learning at first and then only retrain the FC layers at the tip of the architecture to predict for 384x384 images at the end of training? Are there any drawbacks to this approach?
\n
Yes, you can do this. The best way to see how well it works is try it and evaluate CV LB.
", "votes": 0, "replies": [] }, { "id": 980720, "author_name": "", "author_url": "", "post_date": "2020-08-21T19:52:49.990000", "content": "", "votes": 0, "replies": [] }, { "id": 980721, "author_name": "", "author_url": "", "post_date": "2020-08-21T19:52:49.997000", "content": "", "votes": 0, "replies": [] }, { "id": 980722, "author_name": "DarkCube", "author_url": "", "post_date": "2020-08-21T19:52:50.007000", "content": "And one more thing: The LB scores are very close to one another (.0005 isn't much especially when you take into consideration that test accuracy isn't true real world accuracy) so there must be some randomness involved to who might be 1st beyond just skill. So how do hosts deal with this kind of randomness? Do they deem it insignificant? Is it insignificant?
", "votes": 1, "replies": [] }, { "id": 980885, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-22T01:02:56.293000", "content": "Data science is both skill and luck. For example, I built a model from meta data: age, sex, site, image size. (By itself it had CV 0.77 LB 0.77). When I added it to my ensemble, It increased my CV by 0.0005. The question is do I include it in my final submission with such a small CV increase? How much can I gain, and how much do I risk?
\nI simulated 1000 private leaderboards. You can see below that in some simulated private leaderboards, including the meta model (as 90% image 10% meta) will decrease my private LB score as much as AUC 0.002! There is also the chance it could increase my private LB 0.003!
\nThe expected increase in private LB score is 0.0005 with standard deviation 0.0008. That means that it will increase my score in 75% of private leaderboards. Should I take the risk? The best we can do is calculate expected values, variances, and risk and decide whether to include things in our models.
\nI did include it and it lowered my private LB by 0.0002 but there was the chance that it could have increased me to Gold !!
\n
Thank you so much for investing your time into replying to such a newbie I really appreciate it. Due to the MASSIVE information spread about this field, it's really hard to get yourself around. If it wasn't for people like you, new comers won't have any chance keeping up. Especially if they were self-taught like me. Thank you so much
", "votes": 2, "replies": [] }, { "id": 1253672, "author_name": "Jacopo Repossi", "author_url": "", "post_date": "2021-03-26T23:35:26.560000", "content": "\n\nI simulated 1000 private leaderboards.
\n
@cdeotte may I ask you how one could simulate private leaderboards?
", "votes": 0, "replies": [] } ] }, { "id": 977172, "author_name": "Heroseo", "author_url": "", "post_date": "2020-08-19T10:35:08.297000", "content": "Thanks a lot for sharing and it is really good explanation for oof ensemble!
\nMay I ask something:
\nSome Kagglers save their folds to folds.csv and just load folds.csv for more experiments.
If I use same seed(e.g. random_seed = 42), is it same folds result?
new_folds = sklearn.model_selection.KFold(n_splits = 5, shuffle = True, random_seed = 42)
\nfold.csv == new_folds
Thanks again @cdeotte.
", "votes": 1, "replies": [ { "id": 977791, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T17:53:07.827000", "content": "Yes. If two different computers use the same random_seed = 42 then they will use the same folds. This is how working on a Kaggle team does it. Every team member uses the same seed on their own computer. Then you can compare everyone's CV score and use everyone's OOF to ensemble your final submission.
Thanks a lot for reply @cdeotte !
\nI want to use what I have learned from you in the next competition. :)
What clarity!
\nI knew that intuitively, but I wasn't confident enough in my methods to implement it. I want to take advantage of this in the future.
Simple yet very efficient, thanks @cdeotte , for me it seems obvious now, but before this competition I never saved my model's OOF predictions, after seeing your public notebook this is a practice that I will keep from now on!
", "votes": 1, "replies": [ { "id": 976604, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T01:19:06.400000", "content": "Yes. For the past 2 months, I save every model's (1) OOF (2) Submission CSV (3) Model weights. These things always come in handy later.
\nFor example, maybe we want to build a model that reuses 3 models' weights then build a new model using those three backbones, concatenate the outputs and train a new MLP on top of that.
\nOr maybe we want to extract embeddings from old model weight backbones, then use XGB to build a head, etc etc.
", "votes": 5, "replies": [] } ] }, { "id": 976583, "author_name": "M&M", "author_url": "", "post_date": "2020-08-19T00:52:34.967000", "content": "Could you clarify why you do a for loop instead of fitting a logistic regression or other optimization approaches?
", "votes": 1, "replies": [ { "id": 976598, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T01:15:16.573000", "content": "I've tried logistic regression, linear regression and XGB. For some reason hill climbing achieves a higher ensemble CV.
(Not sure why. Maybe I'm doing the other methods wrong).
", "votes": 1, "replies": [] }, { "id": 976688, "author_name": "M&M", "author_url": "", "post_date": "2020-08-19T03:30:10.147000", "content": "Very interesting. I see many top kagglers use the same approach as yours, so there might be some underlying reasons.
\nBy the way thank you for all of your hard work!
", "votes": 1, "replies": [] }, { "id": 983070, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-24T03:04:21.437000", "content": "Note that my forward selection notebook here uses TOL = 0.0003. I'm still unsure what is the best selection for this variable. (This is CV AUC threshold increase for adding new models and I didn't want to clutter the ensemble with unnecessary additional models). Also I use DUPLICATES = False. That means that after it adds a model, it won't add that model again.
But now that I think about it, it would be better to allow adding the same model again because that is equivalent to adding more weight to that model's coefficient. (Kind of how logistic regression or linear regression adjusts weights to find optimal).
\nSo, FYI, if you use TOL = 0 and DUPLICATES = True in my public notebook, the CV gets up to 0.9507 and the private LB gets up to 0.9433! Therefore if you try to compare high climbing to logistic regression or linear regression, those are the CV and LB, you need to beat.
\n\nAlso every model should use the same folds.
\n
Is it necessary to have the same folds? I guess we can have different folds OOF csv files. Will it cause leak?
", "votes": 1, "replies": [ { "id": 976454, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T21:38:55.690000", "content": "It won't cause a leak but you will get artificial CV gains. If one OOF uses folds A, B, C, D, E. Then every train image in fold A is predicted using 80% of train (B, C, D, E). Let's call one image in fold A, img_123. If a second OOF uses folds F, G, H, I, J. Then img_123 is in fold G and it gets predicted with the 80% of train in F, H, I, J.
The 80% that predicted img_123 in OOF_1 is different than the 80% that predicted img_123 in OOF_2. Therefore you will most certainly get an increase when you average OOF_1 and OOF_2. So using different folds gives increases even when the two models are similar and not diverse.
However when you blend SUB_1 and SUB_2 you will not get an increase because both are already similar. In conclusion, using different folds gives increases in CV which do not give increases in SUB blending.
", "votes": 7, "replies": [] }, { "id": 976461, "author_name": "Waylon Wu", "author_url": "", "post_date": "2020-08-18T21:53:53.423000", "content": "Thanks @cdeotte I can see the reason why CV score increases. But I think SUB_1 and SUB_2 will be slightly different because models are trained on different folds of data. So my feeling is that if we run different folds OOF ensemble, CV will increase and LB will increase small or no increase.
", "votes": 0, "replies": [] }, { "id": 976638, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T02:12:07.977000", "content": "SUB_1 and SUB2_1 may be slightly different. But if their combination increases public LB, it is a different reason than combining OOF_1 and OOF_2. Therefore the weights that optimize the increase of OOF_1 plus OOF_2 isn't necessarily the same weights that would optimize SUB_1 plus SUB_2.
", "votes": 1, "replies": [] } ] }, { "id": 976391, "author_name": "DeepUnderstanding", "author_url": "", "post_date": "2020-08-18T20:18:19.303000", "content": "This is great. As a beginner, ensembling has always puzzled me, What are more creative ways to do ensembling?
", "votes": 1, "replies": [ { "id": 976412, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-18T20:34:40.413000", "content": "What we're doing here is similar to linear regression or logistic regression. We are using the OOF as \"train data\" where each OOF is one column feature and we are predicting target. Since our model is linear, we don't worry above overfitting (which seldom occurs for linear models).
\nInstead of linear regression, we can use any model as our \"ensemble model\". For example we could train an XGB using the OOF to predict target. Also we can add additional features. For example, some columns can be the OOF and other columns can be meta data like age, sex, site, etc.
\nEssentially this is stacking.
\nX_train[:,0] = oof_1.predictions\nX_train[:,1] = oof_2.predictions\nX_train[:,2] = oof_3.predictions\nX_train[:,3] = oof.age\nX_train[:,4] = oof.sex\ny_train = oof.target\n\nmodel.fit(X_train,y_train)\n\nX_test[:,0] = sub_1.target\nX_test[:,1] = sub_2.target\nX_test[:,2] = sub_3.target\nX_test[:,3] = sub.age\nX_test[:,4] = sub.sex\n\nkaggle_sub = model.predict(X_test)\nAfterward we have predictions for every training image. This complete set of predictions is called OOF, i think it should be test images
No. Fold 1, Fold 2, Fold 3, Fold 4, Fold 5 are each 20% of the train images. So when we build a model with Fold 2, 3, 4, 5 and predict Fold 1. We now have predictions for the 20% of train images in Fold 1.
\nAfter we do this five times, we have predictions for 100% of train images. These are called OOF
\nAdditionally, each fold model also makes predictions on the test images. After averaging these 5 sets of predictions, these are called SUB
", "votes": 3, "replies": [] } ] }, { "id": 976370, "author_name": "Santiago Viquez", "author_url": "", "post_date": "2020-08-18T20:13:25.350000", "content": "Gonna bookmark this! thanks for the detailed explanation.
", "votes": 1, "replies": [] }, { "id": 976311, "author_name": "Mehul Sampat", "author_url": "", "post_date": "2020-08-18T19:25:47.447000", "content": "thanks @cdeotte !
\nI used your starter notebook and had 12 models.
\ni did ensembling using another starter kernel.
\nthis is my final ensemble with 6 models (Private LB = 0.9413).
\nhttps://www.kaggle.com/mpsampat/final-simple-oof-ensembling-methods-6-models
I had a hard time to find the optimal subset of models. Your starter notebook will be very handy in learning this task!
\nthanks again for sharing this and other notebooks!
\nCheers!
\n\nFor each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image.
\n
Amazing article. I keep revisiting this again & again. I was wondering how there are 5 predictions for the test image since the 5 different folds will be a part of single model? So essentially, one model will predict one output for a single test image? I understand for different models the prediction of test image will vary but how is it varying for single model (and different folds)? Do we take the average for test image output for different models or for different folds of the same model?
", "votes": 2, "replies": [ { "id": 986903, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-26T21:34:31.537000", "content": "In 5 KFold, you actually build 5 models. All models use the same architecture and hyperparameters so they will perform similarily, but each is trained on a different 80% of data. (Their 4 folds).
\nSo after performing 5 KFold, we have 5 models and each model predicts all the test images. Therefore each test image has 5 predictions (where each pred is between 0 and 1). We take the simple average of these 5 numbers as the prediction for test.
", "votes": 1, "replies": [] }, { "id": 987068, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-27T00:48:28.893000", "content": "@ujjwal29jain You can think about CV KFold like a random forest. After building a random forest, you call it \"one model\" but really, the forest is many models each trained with a different 80% of the data and then ensembled together.
\nSimilarily, KFold is K models each trained with a different 80% of the data and then ensembled together. (And both help prevent overfitting and generalize better to unseen data).
", "votes": 3, "replies": [] }, { "id": 1110842, "author_name": "gao-hongnan", "author_url": "", "post_date": "2020-12-13T06:07:41.173000", "content": "@cdeotte Dear Chris, was revisiting your old posts, and as usual, I have noticed that for such classification problems (and even the recent Cassava Comp), I generally build 5 folds and average out the predictions towards the end when I inference. Usually, this produces better results - but is there any statistical justification for this. Appreciate it if you can link me to one article that explains so…
", "votes": 0, "replies": [] } ] }, { "id": 976706, "author_name": "datasaurus", "author_url": "", "post_date": "2020-08-19T03:49:00.227000", "content": "Thanks for sharing Chris! Probably the clearest explanation of this I’ve seen.
\nIs it necessary to use the same folds you used to train your models or can this CV scheme be independent of the initial scheme?
", "votes": 2, "replies": [ { "id": 976758, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-19T04:34:36.367000", "content": "You should always use the same folds. Otherwise if you ensemble models using different folds, you will always see a CV increase but the LB may not increase.
", "votes": 3, "replies": [] }, { "id": 976947, "author_name": "Manjesh Gupta", "author_url": "", "post_date": "2020-08-19T07:30:46.203000", "content": "Chris, can you please elaborate this a bit further?
\nBecause, in this competition I blended different models using oof (each having a different seed value, merged oof files using image id) and it seems to have increased both CV and LB(public and private).
", "votes": 2, "replies": [] }, { "id": 981999, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-23T00:02:14.210000", "content": "@manjeshg03 Using different seeds does add variety to your models. However it is hard to compare the models. For example if you have one model with CV 0.920 using seed=42 and another with CV 0.922 with seed=123, which is better? We don't know.
\nAlso if you ensemble two identical models where one uses seed=42 and the another uses seed=123, the ensemble CV score will always increase. This is because a 5 fold model only uses 80% of train data to predict the OOF. When you ensemble two different seeds, the result are predictions that uses more than 80% (most likely around 95%) of train data for each OOF prediction.
\nHowever when you ensemble the submission.csv files of two identical models where one uses seed=42 and the another uses seed=123, you won't automatically have an increase. Because each model already uses the full 100% train to make the submission file (as result of blending the 5 folds together). So, the CV will always increase but the LB will not always increase. This makes using different seeds a confusing situation. (Not necessarily a bad situation, but certainly a confusing situation to evaluate CV LB).
", "votes": 11, "replies": [] }, { "id": 984837, "author_name": "Manjesh Gupta", "author_url": "", "post_date": "2020-08-25T10:20:51.067000", "content": "@cdeotte Got the point. Thanks a lot!
", "votes": 1, "replies": [] } ] }, { "id": 976329, "author_name": "Tim Yee", "author_url": "", "post_date": "2020-08-18T19:42:00.883000", "content": "Nice brute force algorithm. I will definitely give it a try to improve my optimization techniques. How would you do this for OOF predictions for models of different seeds?
", "votes": 2, "replies": [ { "id": 976639, "author_name": "Sam Klein", "author_url": "", "post_date": "2020-08-19T02:12:34.507000", "content": "I would suggest training a NN on the OOF predictions, adding a small amount of noise as data augmentation.
\nAs Chris has pointed out below - if you train exactly the same architecture twice, with different seeds each time, then the OOF predictions of the two trained models will be different, but their submissions will be similar.
\nI think this can be overcome using noise because I found, with my own submissions, that I could almost perfectly distinguish the combined OOF predictions ([oof_1,oof_2,…,oof_n]) from the combined submissions ([sub_1,sub_2,…,sub_n]). Meaning that the distributions were slightly different. This isn't surprising because each OOF prediction is made by a single model, while each submission prediction comes from a combination of models. However, I found that I could partially avoid this issue by using a small amount of gaussian noise as augmentation while training a NN. With a small amount of noise the two sets could not be distinguished very well. Performing TTA with this noise and using stratified cross validation I trained a model that combined my OOF predictions slightly better than a straightforward linear blend in terms of CV score.
\nWhat I am saying is that if the right amount of noise is added (something with less variance than that of the model that is having noise added to it) then the combined OOF predictions are difficult to distinguish from the combined submissions. As this was the case when using the same seed every time, I imagine it will also be the case when you do not have the same seed. And, in your case, it would also ensure that any of the OOF predictions that are similar become difficult to distinguish, removing their ability to provide artificial gains in the combined OOF predictions.
\nI hope that makes sense, and it would be interesting to know if it works for you!
", "votes": 1, "replies": [] } ] }, { "id": 2919597, "author_name": "Mohammad Sadat Hossain", "author_url": "", "post_date": "2024-07-13T03:10:49.583000", "content": "@cdeotte, this was a nice discussion. But I was wondering maybe if you could give some advice in context of code competitions, like the current ISIC competition. Here, we do not have the entire test set, and hence need to do the full inference within the limit of 9 hours.
\nSay, I have trained using 5-fold cross validation. So essentially I have 5 models to use for inference. Which one should I choose? Infering using all of them and then averaging just takes too much time.
", "votes": 0, "replies": [] }, { "id": 2017546, "author_name": "Myo Min Htet(wnp)🇲🇲", "author_url": "", "post_date": "2022-11-04T23:54:03.187000", "content": "The best concept I've ever seen. And I always learn new things from you.
", "votes": 0, "replies": [] }, { "id": 1871067, "author_name": "Matoo", "author_url": "", "post_date": "2022-07-26T04:03:15.060000", "content": "Thank you for sharing your Hill Climbing approach.
", "votes": 0, "replies": [] }, { "id": 1822540, "author_name": "highscoreman", "author_url": "", "post_date": "2022-06-16T13:40:08.877000", "content": "thanks for sharing. hill climbing,good job.
", "votes": 0, "replies": [] }, { "id": 1729854, "author_name": "CHIRAGhj", "author_url": "", "post_date": "2022-03-20T15:53:22.940000", "content": "Thankyou helped a lot
", "votes": 0, "replies": [] }, { "id": 1654624, "author_name": "", "author_url": "", "post_date": "2022-01-18T17:11:02.800000", "content": "", "votes": 0, "replies": [] }, { "id": 1118700, "author_name": "", "author_url": "", "post_date": "2020-12-19T10:11:24.077000", "content": "", "votes": 0, "replies": [] }, { "id": 976390, "author_name": "", "author_url": "", "post_date": "2020-08-18T20:18:06.273000", "content": "", "votes": 0, "replies": [ { "id": 976409, "author_name": "", "author_url": "", "post_date": "2020-08-18T20:26:26.717000", "content": "", "votes": 7, "replies": [] }, { "id": 977235, "author_name": "", "author_url": "", "post_date": "2020-08-19T11:13:17.920000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1946594, "author_name": "", "author_url": "", "post_date": "2022-09-19T23:39:58.900000", "content": "", "votes": 0, "replies": [ { "id": 1947541, "author_name": "", "author_url": "", "post_date": "2022-09-20T14:14:43.423000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 987263, "author_name": "", "author_url": "", "post_date": "2020-08-27T05:58:34.753000", "content": "", "votes": 1, "replies": [] }, { "id": 986044, "author_name": "", "author_url": "", "post_date": "2020-08-26T06:57:01.327000", "content": "", "votes": 1, "replies": [] }, { "id": 983744, "author_name": "", "author_url": "", "post_date": "2020-08-24T14:54:53.317000", "content": "", "votes": 1, "replies": [] }, { "id": 976322, "author_name": "", "author_url": "", "post_date": "2020-08-18T19:35:36.740000", "content": "", "votes": 1, "replies": [] }, { "id": 976303, "author_name": "", "author_url": "", "post_date": "2020-08-18T19:22:13.703000", "content": "", "votes": 1, "replies": [] }, { "id": 1797454, "author_name": "", "author_url": "", "post_date": "2022-05-22T01:26:52.757000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "976295": "One thing I learned at Kaggle is how to ensemble models using OOF files. This was very helpful in Melanoma Competition. I would like to share a simple procedure below called `hill climbing` and I posted a starter notebook [here][1]. \n\n# Cross Validation\nThe acronym CV refers to cross validation. We start with the full training dataset picture below to the far left. Next, we divide it into 5 subsets, called `Fold 1, Fold 2, Fold 3, Fold 4, Fold 5`. Then we train 5 models. We train our first model using data from Folds 2-5 and predict Fold 1. Next we train model 2 using Folds 1, 3, 4, 5 and predict 2. Next 1, 2, 4, 5 and predict 3, etc etc.\n\nAfterward we have predictions for every training image. This compete set of predictions is called OOF, \"out of fold\" predictions. It is a good practice to save these predictions for every model you build during a competition as `oof.csv`.\n\nThe CV score (or OOF AUC) is then calculated with `OOF_AUC = roc_auc_score( train.target, oof.prediction)`. And this is the best indicator of how your model performs. It is a better indicator than public LB.\n\nDuring the competition, every model should use the same folds. This is accomplished by using the same seed with `sklearn.model_selection.KFold(n_splits = 5, shuffle = True, random_seed = 42)`\n\n\n\n# Submission Files\nFor each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image. We take the average of these 5 sets of predictions and this is our `submission.csv` file that we submit to Kaggle. When you submit this to Kaggle, your LB score should be similar to your CV score.\n\n# Model Ensemble\nNow say that you build 2 models (that means that you did 5 KFold twice). You now have `oof_1.csv`, `oof_2.csv`, `sub_1.csv`, and `sub_2.csv`. How do we blend the two models?\n\nWe find the weight `w` such that `w * oof_1.predictions + (1-w) * oof_2.predictions` has the largest AUC.\n\n all = []\n for w in [0.00, 0.01, 0.02, ..., 0.98, 0.99, 1.00]:\n ensemble_pred = w * oof_1.predictions + (1-w) * oof_2.predictions\n ensemble_auc = roc_auc_score( oof.target , ensemble_pred )\n all.append( ensemble_auc )\n best_weight = np.argmax( all ) / 100.\n\nThen our submission to kaggle will be\n\n kaggle_sub = best_weight * sub_1.target + (1-best_weight) * sub_2.target\n\n# Starter Notebook\nAfter weeks working on a competition, we will have more than 2 models. So we need a more sophisticated approach than a single for-loop. \n\nThe simplest approach is `hill climbing` (or `forward selection`). Start with the one model that has highest CV score. Next iterate through all your additional models and find the one model that combines with the first model to generate the highest two model ensemble CV score. Then search for the best third model. Repeat until ensemble CV does not increase anymore.\n\nI posted a starter notebook [here][1] which includes all my `oof.csv` and `sub.csv` for this Melanoma comp. There are 39 models. Forward selection chooses 8 of them and the resultant ensemble has OOF CV 0.950, Public LB 0.958, and Private LB 0.942\n\n[1]: https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private", "976302": "Kagglers refer to this as `hill climbing` just as a side note if others see this phrase somewhere else.", "979176": "Thanks for sharing this! Good explanation for CV!\n\nAfter reading some discussions in the past few days after the competition end, I gradually realized that what more people need is not the top solutions, but the basic ML knowledge, just like this posts.\n\nI see many people still don't understand why they should not trust public LB score (especially in this comp). But I don’t know how to explain to them in a simple and understandable way.\n\nI think your post can be a good explanation for it as well. Thanks again!", "1654315": "Thanks @cdeotte, your Hill Climbing approach is a great trick. I'll for sure try it out in my next competition.\nI have small doubt, in the post you've linked you say \n>During the competition, every model should use the same folds. \n\nDoes that really matter when your oof predictions have all the train samples anyway? I specifically ask in a context of ensembling models as a team, where different members worked with different splits.", "988805": "Hi @cdeotte, have you seen this paper \"On Cross-Validation and Stacking: Building seemingly predictive models on random data\"?\nhttps://www.kdd.org/exploration_files/v12-02-4-UR-Perlich.pdf \n\nI'd be interested to know what you and other Kagglers think about it", "979325": "Hi Chris, the explanation is very clear but I have a question and after reading the post a couple of times I can't figure out the answer on my own. Talking about the submission file, you say\n> For each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image. We take the average of these 5 sets of predictions and this is our submission.csv file that we submit to Kaggle. When you submit this to Kaggle, your LB score should be similar to your CV score.\n\nand this makes perfect sense, but why do you never train a model using all train data once you have a high CV score for that model? \nIs it because you would not know how it performs when trained with an additional fold, making all the work useless and destroying the ensembling process? \nI am asking because so far, when I perform cross-validation, I always fit the model on all train data with the best parameters to make the test predictions and so your way of doing it by averaging is completely new to me.\nThank you!", "976417": "The strategy I usually go for is finding a set of models that give the best CV when averaged together.\n\nDo you have any idea how your ensemble performs (especially on the leaderboard) when giving equal weights to every model ? \nWhat about using the 8 (or any number of) models that have the best CV when averging the predictions together ? \n\nThis would be nice to see how much hill climbing actually helps.", "998817": "Suppose we are using a model and I have to choose the best hyperparameters for that model. Now, will it be right that for each hyperparameter I do a new cv and note down the best parameters? By doing this am I introducing any leakage?", "997407": "Thanks for sharing. I do follow these approaches on Stacking while working on the projects.\nHave you tried Voting Classifiers as well?", "988133": "A good explaination, was very helpful.", "987504": "Good one", "983330": "Very crisp, clear & to the point explanation. I am a novice at kaggle and I am learning amazing stuff reading your notebook & discussion. Thanks!", "983068": "Thanks for sharing great knowledge about ensembling! \n\nI didn't know hill climbing approach to ensemble the model and helped me a lot! ", "979255": "Because of hardware limitations I have been trying this....\n\nfor each epoch in 1:30\ntrain for 100 steps; in each step use a batch of n images that is a random sample of the train set with augmentations (resampled at each step)\nvalidate for 30 steps; in each step use a batch of n images that is a random sample of the train set without augmentations (resampled at each step)\n\nn is 16 32 64 depending on image size\n\nDifferent runs for the same model got consistent scores between \"this cv score\" and private set... not with public set thas was a bit lower...\n\nchanging the set of images at each step may result in underfitting? it certainly did not overfit", "977172": "Thanks a lot for sharing and it is really good explanation for oof ensemble!\n\nMay I ask something:\nSome Kagglers save their folds to `folds.csv` and just load `folds.csv` for more experiments.\n\nIf I use same seed(`e.g. random_seed = 42`), is it same folds result?\n\n`new_folds = sklearn.model_selection.KFold(n_splits = 5, shuffle = True, random_seed = 42)`\n`fold.csv == new_folds`\n\nThanks again @cdeotte.", "976954": "What clarity!\nI knew that intuitively, but I wasn't confident enough in my methods to implement it. I want to take advantage of this in the future.", "976589": "Simple yet very efficient, thanks @cdeotte , for me it seems obvious now, but before this competition I never saved my model's OOF predictions, after seeing your public notebook this is a practice that I will keep from now on!", "976583": "Could you clarify why you do a for loop instead of fitting a logistic regression or other optimization approaches?", "976448": "> Also every model should use the same folds. \n\nIs it necessary to have the same folds? I guess we can have different folds OOF csv files. Will it cause leak?", "976391": "This is great. As a beginner, ensembling has always puzzled me, What are more creative ways to do ensembling?", "976370": "Gonna bookmark this! thanks for the detailed explanation.", "976311": "thanks @cdeotte ! \nI used your starter notebook and had 12 models. \ni did ensembling using another starter kernel. \nthis is my final ensemble with 6 models (Private LB = 0.9413). \nhttps://www.kaggle.com/mpsampat/final-simple-oof-ensembling-methods-6-models\n\nI had a hard time to find the optimal subset of models. Your starter notebook will be very handy in learning this task! \nthanks again for sharing this and other notebooks! \nCheers! ", "985887": "> For each of the 5 fold models above, we predict the test images. Therefore we have 5 predictions for each test image.\n\n\nAmazing article. I keep revisiting this again & again. I was wondering how there are 5 predictions for the test image since the 5 different folds will be a part of single model? So essentially, one model will predict one output for a single test image? I understand for different models the prediction of test image will vary but how is it varying for single model (and different folds)? Do we take the average for test image output for different models or for different folds of the same model?", "976706": "Thanks for sharing Chris! Probably the clearest explanation of this I’ve seen.\n\nIs it necessary to use the same folds you used to train your models or can this CV scheme be independent of the initial scheme?", "976329": "Nice brute force algorithm. I will definitely give it a try to improve my optimization techniques. How would you do this for OOF predictions for models of different seeds?", "2919597": "@cdeotte, this was a nice discussion. But I was wondering maybe if you could give some advice in context of code competitions, like the [current ISIC competition](https://www.kaggle.com/competitions/isic-2024-challenge). Here, we do not have the entire test set, and hence need to do the full inference within the limit of 9 hours. \n\nSay, I have trained using 5-fold cross validation. So essentially I have 5 models to use for inference. Which one should I choose? Infering using all of them and then averaging just takes too much time.", "2017546": "The best concept I've ever seen. And I always learn new things from you.", "1871067": "Thank you for sharing your Hill Climbing approach.", "1822540": "thanks for sharing. hill climbing,good job.", "1729854": "Thankyou helped a lot", "1654624": "Thanks for sharing this!\nIts been really helpful.", "1118700": "Great post! Crisp and Clear! Thanks for sharing", "976390": "Are you by finding these optimal weights not overfitting the validation set?", "1946594": "", "987263": "Great stuff. Thanks for explaining ", "986044": "nice explanation \nThanks for sharing", "983744": "thanks for sharing this!", "976322": "Thanks sir", "976303": "This the light!! Thanks!", "1797454": "Thanks for sharing, extremely helpful!!" } }