Hi all,
\nI have built a ensemble model that is acheiving cv scores of 0.48 in training, however when I use the model on the leaderboard - I am finding that is overfitting 0.45. I have added various features into my model to help reduce the overfitting including cv, regularization, feature selection and removed outliers. Does anyone know any other techniques I can use to reduce the overfitting?
", "messages": [ { "id": "3014042", "postDate": "10/10/2024 19:56:38", "content": "Hi all,
\nI have built a ensemble model that is acheiving cv scores of 0.48 in training, however when I use the model on the leaderboard - I am finding that is overfitting 0.45. I have added various features into my model to help reduce the overfitting including cv, regularization, feature selection and removed outliers. Does anyone know any other techniques I can use to reduce the overfitting?
", "rawMarkdown": "Hi all, \n\nI have built a ensemble model that is acheiving cv scores of 0.48 in training, however when I use the model on the leaderboard - I am finding that is overfitting 0.45. I have added various features into my model to help reduce the overfitting including cv, regularization, feature selection and removed outliers. Does anyone know any other techniques I can use to reduce the overfitting?", "votes": null }, { "id": "3014063", "postDate": "10/10/2024 20:41:23", "content": "Hi @peterhopkinson,
\nYour techniques to reduce overfitting are the good ones (cross-validation, regularization, feature selection …)
\nWhat makes you think you are overfitting ?
\nYou will start to overfit when you will make decisions based on your score on the public leaderboard ; as long as you don't do it, and as long as your CV score is .48 or higher even when changing your random seed, you will not overfit.
", "rawMarkdown": "Hi @peterhopkinson,\n\nYour techniques to reduce overfitting are the good ones (cross-validation, regularization, feature selection ...)\n\nWhat makes you think you are overfitting ? \n\nYou will start to overfit when you will make decisions based on your score on the public leaderboard ; as long as you don't do it, and as long as your CV score is .48 or higher even when changing your random seed, you will not overfit.", "votes": null }, { "id": "3014082", "postDate": "10/10/2024 21:53:57", "content": "Just the difference between the cross validation score and the leaderboard score suggests the model is overfitting - I am evaluating the hyperparameter ranges to adjust them to reduce the chance of overfitting.
", "rawMarkdown": "Just the difference between the cross validation score and the leaderboard score suggests the model is overfitting - I am evaluating the hyperparameter ranges to adjust them to reduce the chance of overfitting.", "votes": null }, { "id": "3014242", "postDate": "10/11/2024 05:02:58", "content": "So you are not overfitting the public LB.
\nAnother thing to verify, is the difference between your score during traing and your validation score : as long as difference is less than .1, you are not overfitting training samples too much (my opinion, in this comp). Best public notebooks could have a .3 difference : their training score generalize poprly.
\nWe can compare those two scores because it should not be any difference between train and validation samples (it depends on your CV strategy and regularization)
\nBut we must not compare CV score and score on public LB, because we don't known anything about samples on public LB.
\nThere is an interesting discussion about scores differences to bookmark here
", "rawMarkdown": "So you are not overfitting the public LB.\n\nAnother thing to verify, is the difference between your score during traing and your validation score : as long as difference is less than .1, you are not overfitting training samples too much (my opinion, in this comp). Best public notebooks could have a .3 difference : their training score generalize poprly.\n\nWe can compare those two scores because it should not be any difference between train and validation samples (it depends on your CV strategy and regularization)\n\nBut we must not compare CV score and score on public LB, because we don't known anything about samples on public LB.\n\nThere is an interesting discussion about scores differences to bookmark [here](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/discussion/535523)", "votes": null }, { "id": "3014401", "postDate": "10/11/2024 07:57:33", "content": "hey @peterhopkinson , it's not overfitting . It's completely natural to have a small score difference between cv and Public test data.
", "rawMarkdown": "hey @peterhopkinson , it's not overfitting . It's completely natural to have a small score difference between cv and Public test data.", "votes": null }, { "id": "3016013", "postDate": "10/13/2024 08:57:07", "content": "You are not overfitting. The data available to us is not enough to overfit unless you do something too far.
", "rawMarkdown": "You are not overfitting. The data available to us is not enough to overfit unless you do something too far.", "votes": null } ], "comments": [ { "id": 3014063, "author_name": "adaubas", "author_url": "", "post_date": "10/10/2024 20:41:23", "content": "Hi @peterhopkinson,
\nYour techniques to reduce overfitting are the good ones (cross-validation, regularization, feature selection …)
\nWhat makes you think you are overfitting ?
\nYou will start to overfit when you will make decisions based on your score on the public leaderboard ; as long as you don't do it, and as long as your CV score is .48 or higher even when changing your random seed, you will not overfit.
", "votes": null, "replies": [ { "id": 3014082, "author_name": "peterhopkinson", "author_url": "", "post_date": "10/10/2024 21:53:57", "content": "Just the difference between the cross validation score and the leaderboard score suggests the model is overfitting - I am evaluating the hyperparameter ranges to adjust them to reduce the chance of overfitting.
", "votes": null, "replies": [ { "id": 3014242, "author_name": "adaubas", "author_url": "", "post_date": "10/11/2024 05:02:58", "content": "So you are not overfitting the public LB.
\nAnother thing to verify, is the difference between your score during traing and your validation score : as long as difference is less than .1, you are not overfitting training samples too much (my opinion, in this comp). Best public notebooks could have a .3 difference : their training score generalize poprly.
\nWe can compare those two scores because it should not be any difference between train and validation samples (it depends on your CV strategy and regularization)
\nBut we must not compare CV score and score on public LB, because we don't known anything about samples on public LB.
\nThere is an interesting discussion about scores differences to bookmark here
", "votes": null, "replies": [] } ] } ] }, { "id": 3014401, "author_name": "mohammedahmedxx12", "author_url": "", "post_date": "10/11/2024 07:57:33", "content": "hey @peterhopkinson , it's not overfitting . It's completely natural to have a small score difference between cv and Public test data.
", "votes": null, "replies": [] }, { "id": 3016013, "author_name": "chanpreetsingh07", "author_url": "", "post_date": "10/13/2024 08:57:07", "content": "You are not overfitting. The data available to us is not enough to overfit unless you do something too far.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3014042": "Hi all, \n\nI have built a ensemble model that is acheiving cv scores of 0.48 in training, however when I use the model on the leaderboard - I am finding that is overfitting 0.45. I have added various features into my model to help reduce the overfitting including cv, regularization, feature selection and removed outliers. Does anyone know any other techniques I can use to reduce the overfitting?", "3014063": "Hi @peterhopkinson,\n\nYour techniques to reduce overfitting are the good ones (cross-validation, regularization, feature selection ...)\n\nWhat makes you think you are overfitting ? \n\nYou will start to overfit when you will make decisions based on your score on the public leaderboard ; as long as you don't do it, and as long as your CV score is .48 or higher even when changing your random seed, you will not overfit.", "3014082": "Just the difference between the cross validation score and the leaderboard score suggests the model is overfitting - I am evaluating the hyperparameter ranges to adjust them to reduce the chance of overfitting.", "3014242": "So you are not overfitting the public LB.\n\nAnother thing to verify, is the difference between your score during traing and your validation score : as long as difference is less than .1, you are not overfitting training samples too much (my opinion, in this comp). Best public notebooks could have a .3 difference : their training score generalize poprly.\n\nWe can compare those two scores because it should not be any difference between train and validation samples (it depends on your CV strategy and regularization)\n\nBut we must not compare CV score and score on public LB, because we don't known anything about samples on public LB.\n\nThere is an interesting discussion about scores differences to bookmark [here](https://www.kaggle.com/competitions/child-mind-institute-problematic-internet-use/discussion/535523)", "3014401": "hey @peterhopkinson , it's not overfitting . It's completely natural to have a small score difference between cv and Public test data.", "3016013": "You are not overfitting. The data available to us is not enough to overfit unless you do something too far." }, "source": "meta" }