![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7285296%2F52a4af62650961305bedb748ce65e007%2FCapture.PNG?generation=1682495391935741&alt=media\")
When I cross-validate (CV) a model and use the validation set for early_stopping, each fold of CV often have a different number of early-stopped n_estimators due to different validation sets being used. If you are in the same case, what model do you use for inference? What is the best practice in your opinion? I'm thinking of the following options
n_estimators=X and all train data, then using it for inference. I feel like this is the best approach since we're utilizing all train data without long prediction time (like option 2). However I am not sure what X should be - average of all folds, a random fold, etc.?n_estimators (suggested in this Stack Exchange thread). Instead, use CV for only for tuning other hyper-params. After other params have been fixed, re-train the model on the majority of train data and keep out a small subset as validation for early stopping, and use this final model for inference.When I cross-validate (CV) a model and use the validation set for early_stopping, each fold of CV often have a different number of early-stopped n_estimators due to different validation sets being used. If you are in the same case, what model do you use for inference? What is the best practice in your opinion? I'm thinking of the following options
n_estimators=X and all train data, then using it for inference. I feel like this is the best approach since we're utilizing all train data without long prediction time (like option 2). However I am not sure what X should be - average of all folds, a random fold, etc.?n_estimators (suggested in this Stack Exchange thread). Instead, use CV for only for tuning other hyper-params. After other params have been fixed, re-train the model on the majority of train data and keep out a small subset as validation for early stopping, and use this final model for inference.@hoangnguyen719
\nI am not sure about point 4 but I can confirm your point 1 and 2. I've also had the same experience while doing CV.
Point 3 is an interesting approach and I think that you should take a naive approach to determine n_esrimators.
\nBut thanks for bringing this up.
", "rawMarkdown": "hoangnguyen719 \nI am not sure about point 4 but I can confirm your point 1 and 2. I've also had the same experience while doing CV.\n\nPoint 3 is an interesting approach and I think that you should take a naive approach to determine n_esrimators.\n\nBut thanks for bringing this up.", "votes": null }, { "id": "2161831", "postDate": "02/27/2023 19:17:01", "content": "I think saving n_estimators for each fold_question and then retraining on all data using n_estimtors = np.mean(estimators[q]) or the median is a good strategy
Thanks Reacher! That's also the first thing I thought of to calculate X in option (3) above, but I have not found a good logic to back it. For example, some of my folds' n_estimators are twice as many as other folds', in which case I don't feel that mean is the best option.
\nDo you do additional testing after calculating such means? Do you see any differences when the inter-fold differences are large?
I haven't checked the variance between folds tbh , since I'm focusing more on FE but you're right if variance is too high choosing mean or median isn't appropriate!
", "rawMarkdown": "I haven't checked the variance between folds tbh , since I'm focusing more on FE but you're right if variance is too high choosing mean or median isn't appropriate!", "votes": null }, { "id": "2235600", "postDate": "04/26/2023 07:34:04", "content": "Hi @hoangnguyen719 , I believe you should use the largest n_estimators.
Let me elaborate my reason: if your model early stopped at 100 rounds, this means that your model stopped to learn at 100 rounds, and if you force your model to learn 200 rounds, the accuracy of your model should not reduced (on the condition that the learning rate is not too big so that the global optimal point was not jumped during the learning process)
", "rawMarkdown": "Hi @hoangnguyen719 , I believe you should use the largest `n_estimators`. \n\nLet me elaborate my reason: if your model early stopped at 100 rounds, this means that your model stopped to learn at 100 rounds, and if you force your model to learn 200 rounds, the accuracy of your model should not reduced (on the condition that the learning rate is not too big so that the global optimal point was not jumped during the learning process)", "votes": null }, { "id": "2235603", "postDate": "04/26/2023 07:39:37", "content": "I hoped I made myself clear
", "rawMarkdown": "I hoped I made myself clear", "votes": null }, { "id": "2235611", "postDate": "04/26/2023 07:51:32", "content": "you can take a look at the example below. The model stopped to learn at No.182 round, but ask the model to learn more rounds do not harm the accuracy.
\n
@hoangnguyen719
\nI am not sure about point 4 but I can confirm your point 1 and 2. I've also had the same experience while doing CV.
Point 3 is an interesting approach and I think that you should take a naive approach to determine n_esrimators.
\nBut thanks for bringing this up.
", "votes": null, "replies": [] }, { "id": 2161831, "author_name": "ihebch", "author_url": "", "post_date": "02/27/2023 19:17:01", "content": "I think saving n_estimators for each fold_question and then retraining on all data using n_estimtors = np.mean(estimators[q]) or the median is a good strategy
Thanks Reacher! That's also the first thing I thought of to calculate X in option (3) above, but I have not found a good logic to back it. For example, some of my folds' n_estimators are twice as many as other folds', in which case I don't feel that mean is the best option.
\nDo you do additional testing after calculating such means? Do you see any differences when the inter-fold differences are large?
I haven't checked the variance between folds tbh , since I'm focusing more on FE but you're right if variance is too high choosing mean or median isn't appropriate!
", "votes": null, "replies": [] } ] } ] }, { "id": 2235600, "author_name": "zhangyue325", "author_url": "", "post_date": "04/26/2023 07:34:04", "content": "Hi @hoangnguyen719 , I believe you should use the largest n_estimators.
Let me elaborate my reason: if your model early stopped at 100 rounds, this means that your model stopped to learn at 100 rounds, and if you force your model to learn 200 rounds, the accuracy of your model should not reduced (on the condition that the learning rate is not too big so that the global optimal point was not jumped during the learning process)
", "votes": null, "replies": [ { "id": 2235603, "author_name": "zhangyue325", "author_url": "", "post_date": "04/26/2023 07:39:37", "content": "I hoped I made myself clear
", "votes": null, "replies": [] }, { "id": 2235611, "author_name": "zhangyue325", "author_url": "", "post_date": "04/26/2023 07:51:32", "content": "you can take a look at the example below. The model stopped to learn at No.182 round, but ask the model to learn more rounds do not harm the accuracy.
\n