Out of curiosity, I tried modifying a few params and introducing a few features in the top-scoring public notebook, using both GroupKFold(5) and StratifiedGroupKFold(5) - I noticed that the CV values for both do not provide a reliable indicator of performance in the public LB, in any way. In other words, improvement in CV MAP does not indicate improvement in LB.
\nI've seen this topic be raised up a few times already, but was wondering what thoughts the community had on this.
", "messages": [ { "id": "2261444", "postDate": "05/16/2023 10:24:42", "content": "Out of curiosity, I tried modifying a few params and introducing a few features in the top-scoring public notebook, using both GroupKFold(5) and StratifiedGroupKFold(5) - I noticed that the CV values for both do not provide a reliable indicator of performance in the public LB, in any way. In other words, improvement in CV MAP does not indicate improvement in LB.
\nI've seen this topic be raised up a few times already, but was wondering what thoughts the community had on this.
", "rawMarkdown": "Out of curiosity, I tried modifying a few params and introducing a few features in the top-scoring public notebook, using both GroupKFold(5) and StratifiedGroupKFold(5) - I noticed that the CV values for both do not provide a reliable indicator of performance in the public LB, in any way. In other words, improvement in CV MAP does not indicate improvement in LB.\n\nI've seen this topic be raised up a few times already, but was wondering what thoughts the community had on this.", "votes": null }, { "id": "2262059", "postDate": "05/16/2023 17:53:08", "content": "Public LB could be heavy on 1 class, private one may be very different and could match CV better
", "rawMarkdown": "Public LB could be heavy on 1 class, private one may be very different and could match CV better", "votes": null }, { "id": "2263726", "postDate": "05/17/2023 20:40:26", "content": "This has been show to be false in some of the other CV threads, some folks have figured it out.
", "rawMarkdown": "This has been show to be false in some of the other CV threads, some folks have figured it out.", "votes": null }, { "id": "2265470", "postDate": "05/19/2023 09:06:54", "content": "You need to deal with class imbalance one way or another in order to have more reliable CV (see https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/406139)
", "rawMarkdown": "You need to deal with class imbalance one way or another in order to have more reliable CV (see https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/406139)", "votes": null }, { "id": "2265628", "postDate": "05/19/2023 11:46:19", "content": "I also recommend plotting a confusion matrix in order to see how your model performs on different classes.
", "rawMarkdown": "I also recommend plotting a confusion matrix in order to see how your model performs on different classes.", "votes": null }, { "id": "2288319", "postDate": "06/05/2023 10:13:42", "content": "@exjustice
\nThank you for the information!
\nDo you remember which discussion thread the conversation was on?
Public LB could be heavy on 1 class, private one may be very different and could match CV better
", "votes": null, "replies": [ { "id": 2263726, "author_name": "exjustice", "author_url": "", "post_date": "05/17/2023 20:40:26", "content": "This has been show to be false in some of the other CV threads, some folks have figured it out.
", "votes": null, "replies": [ { "id": 2288319, "author_name": "daikimori", "author_url": "", "post_date": "06/05/2023 10:13:42", "content": "@exjustice
\nThank you for the information!
\nDo you remember which discussion thread the conversation was on?
You need to deal with class imbalance one way or another in order to have more reliable CV (see https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/406139)
", "votes": null, "replies": [] }, { "id": 2265628, "author_name": "atamazian", "author_url": "", "post_date": "05/19/2023 11:46:19", "content": "I also recommend plotting a confusion matrix in order to see how your model performs on different classes.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2261444": "Out of curiosity, I tried modifying a few params and introducing a few features in the top-scoring public notebook, using both GroupKFold(5) and StratifiedGroupKFold(5) - I noticed that the CV values for both do not provide a reliable indicator of performance in the public LB, in any way. In other words, improvement in CV MAP does not indicate improvement in LB.\n\nI've seen this topic be raised up a few times already, but was wondering what thoughts the community had on this.", "2262059": "Public LB could be heavy on 1 class, private one may be very different and could match CV better", "2263726": "This has been show to be false in some of the other CV threads, some folks have figured it out.", "2265470": "You need to deal with class imbalance one way or another in order to have more reliable CV (see https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/406139)", "2265628": "I also recommend plotting a confusion matrix in order to see how your model performs on different classes.", "2288319": "exjustice \nThank you for the information!\nDo you remember which discussion thread the conversation was on?" }, "source": "meta" }