Hi,
\n\nHas anyone been able to get local cv scores above 0.69 and get public lb scores above 0.7?
\n\nI'm asking this because there are teams currently in gold and silver positions with LB scores above 0.7 even though people have reported that cv score decreases as LB score increases beyond 0.68. Is it more likely that they are overfitting to the public lb or are they doing something more advanced and will still be at the top of the leaderboard at the end of the competition?
", "messages": [ { "id": "457530", "postDate": "01/17/2019 16:27:41", "content": "Hi,
\n\nHas anyone been able to get local cv scores above 0.69 and get public lb scores above 0.7?
\n\nI'm asking this because there are teams currently in gold and silver positions with LB scores above 0.7 even though people have reported that cv score decreases as LB score increases beyond 0.68. Is it more likely that they are overfitting to the public lb or are they doing something more advanced and will still be at the top of the leaderboard at the end of the competition?
", "rawMarkdown": "Hi,\n\nHas anyone been able to get local cv scores above 0.69 and get public lb scores above 0.7?\n\nI'm asking this because there are teams currently in gold and silver positions with LB scores above 0.7 even though people have reported that cv score decreases as LB score increases beyond 0.68. Is it more likely that they are overfitting to the public lb or are they doing something more advanced and will still be at the top of the leaderboard at the end of the competition?", "votes": null }, { "id": "457702", "postDate": "01/18/2019 00:42:32", "content": "cv 693 -> lb above 700
", "rawMarkdown": "cv 693 -> lb above 700", "votes": null }, { "id": "458633", "postDate": "01/20/2019 05:15:32", "content": "There have been a lot of misunderstanding in evaluating the model performance (the common misunderstood one is simply compare CV and LB). In short your CV prediction is based on single-classifiers (OOF) performance, and LB prediction is based on an ensemble performance, so they are not directly comparable ... you should take into account the diversity of each base classifier when they combine their power in predicting LB.
\n\nFor full details, please take a look at this kernel to demythify the subject : https://www.kaggle.com/bminixhofer/a-validation-framework-impact-of-the-random-seed
", "rawMarkdown": "There have been a lot of misunderstanding in evaluating the model performance (the common misunderstood one is simply compare CV and LB). In short your CV prediction is based on single-classifiers (OOF) performance, and LB prediction is based on an ensemble performance, so **they are not directly comparable** ... you should take into account the diversity of each base classifier when they combine their power in predicting LB.\n\nFor full details, please take a look at this kernel to demythify the subject : https://www.kaggle.com/bminixhofer/a-validation-framework-impact-of-the-random-seed", "votes": null }, { "id": "458656", "postDate": "01/20/2019 07:29:53", "content": "It's possible that they are creating at least two models - one that is focused on high phase-1 public LB scores, while the other is meant as a more generalized model.
", "rawMarkdown": "It's possible that they are creating at least two models - one that is focused on high phase-1 public LB scores, while the other is meant as a more generalized model.", "votes": null } ], "comments": [ { "id": 457702, "author_name": "baomengjiao", "author_url": "", "post_date": "01/18/2019 00:42:32", "content": "cv 693 -> lb above 700
", "votes": null, "replies": [] }, { "id": 458633, "author_name": "ratthachat", "author_url": "", "post_date": "01/20/2019 05:15:32", "content": "There have been a lot of misunderstanding in evaluating the model performance (the common misunderstood one is simply compare CV and LB). In short your CV prediction is based on single-classifiers (OOF) performance, and LB prediction is based on an ensemble performance, so they are not directly comparable ... you should take into account the diversity of each base classifier when they combine their power in predicting LB.
\n\nFor full details, please take a look at this kernel to demythify the subject : https://www.kaggle.com/bminixhofer/a-validation-framework-impact-of-the-random-seed
", "votes": null, "replies": [] }, { "id": 458656, "author_name": "julius6", "author_url": "", "post_date": "01/20/2019 07:29:53", "content": "It's possible that they are creating at least two models - one that is focused on high phase-1 public LB scores, while the other is meant as a more generalized model.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "457530": "Hi,\n\nHas anyone been able to get local cv scores above 0.69 and get public lb scores above 0.7?\n\nI'm asking this because there are teams currently in gold and silver positions with LB scores above 0.7 even though people have reported that cv score decreases as LB score increases beyond 0.68. Is it more likely that they are overfitting to the public lb or are they doing something more advanced and will still be at the top of the leaderboard at the end of the competition?", "457702": "cv 693 -> lb above 700", "458633": "There have been a lot of misunderstanding in evaluating the model performance (the common misunderstood one is simply compare CV and LB). In short your CV prediction is based on single-classifiers (OOF) performance, and LB prediction is based on an ensemble performance, so **they are not directly comparable** ... you should take into account the diversity of each base classifier when they combine their power in predicting LB.\n\nFor full details, please take a look at this kernel to demythify the subject : https://www.kaggle.com/bminixhofer/a-validation-framework-impact-of-the-random-seed", "458656": "It's possible that they are creating at least two models - one that is focused on high phase-1 public LB scores, while the other is meant as a more generalized model." }, "source": "meta" }