{ "id": 485697, "title": "CV vs LB gap shortens?", "url": "/competitions/hms-harmful-brain-activity-classification/discussion/485697", "author_name": "Cody_Null", "post_date": "2024-03-21T20:11:13.391000", "votes": 10, "comment_count": 23, "views": 0, "content": "

As most have found by now, there are two groups in our data set. Those who had many votes and those who did not. Calculating CV for the subsets separately we can see that the CV LB gap is actually quite small if we consider that the public LB is mostly high vote data. Of course, there are many assumptions people are making about what the private dataset may hold for us. I thought I would share some of my numbers in hope other may share as well.

\n

High votes group CV: 0.33
\nLow votes group CV: 0.6
\nLB: 0.35

\n

This should be a fun competition at the end!

", "messages": [ { "id": 2709761, "postDate": "2024-03-21T20:11:13.390Z", "content": "

As most have found by now, there are two groups in our data set. Those who had many votes and those who did not. Calculating CV for the subsets separately we can see that the CV LB gap is actually quite small if we consider that the public LB is mostly high vote data. Of course, there are many assumptions people are making about what the private dataset may hold for us. I thought I would share some of my numbers in hope other may share as well.

\n

High votes group CV: 0.33
\nLow votes group CV: 0.6
\nLB: 0.35

\n

This should be a fun competition at the end!

", "rawMarkdown": "As most have found by now, there are two groups in our data set. Those who had many votes and those who did not. Calculating CV for the subsets separately we can see that the CV LB gap is actually quite small if we consider that the public LB is mostly high vote data. Of course, there are many assumptions people are making about what the private dataset may hold for us. I thought I would share some of my numbers in hope other may share as well.\n\nHigh votes group CV: 0.33 \nLow votes group CV: 0.6\nLB: 0.35\n\nThis should be a fun competition at the end!", "votes": 10 }, { "id": 2710168, "postDate": "2024-03-22T04:31:59.207Z", "content": "

Our vote count >= 10 OOF score is 0.2174. Top 5 teams probably reached beyond 0.2.

", "rawMarkdown": "Our vote count >= 10 OOF score is 0.2174. Top 5 teams probably reached beyond 0.2.", "votes": 8, "replies": [ { "id": 2710462, "postDate": "2024-03-22T09:21:18.293Z", "content": "

is this cv on Chris's created training data filtered out with votes >= 10

", "rawMarkdown": "is this cv on Chris's created training data filtered out with votes >= 10", "replies": [ { "id": 2710463, "postDate": "2024-03-22T09:22:15.347Z", "content": "

Nope, we are using a custom cv scheme.

", "rawMarkdown": "Nope, we are using a custom cv scheme.", "votes": 1 } ] } ] }, { "id": 2710549, "postDate": "2024-03-22T10:54:24.913Z", "content": "

Here is the prediction distribution by total_evaluators in oof_df.
\nY: kl divergence from label
\nX: total_evaluators

\n

This prediction is made by baseline efficient net model trained with all data, cv score with all data is 0.5X.
\nThe baseline model is almost same as moth's published notebook.

\n

As you can see, kl divergence is lager in data with total_evaluators <= 3.
\nIt seems that data with total_evaluators <= 3 have difficult sample or noizy sample.
\nIn my experiments, this distribution often happens when training baseline efficient net model with all data.

\n

\"viz_of_kl_div_by_total_evaluators\"

\n

I will publish code soon.

", "rawMarkdown": "Here is the prediction distribution by `total_evaluators` in oof_df.\nY: kl divergence from label\nX: total_evaluators\n\nThis prediction is made by baseline efficient net model trained with all data, cv score with all data is 0.5X.\nThe baseline model is almost same as [moth's published notebook](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train).\n\nAs you can see, kl divergence is lager in data with total_evaluators <= 3.\nIt seems that data with total_evaluators <= 3 have difficult sample or noizy sample.\nIn my experiments, this distribution often happens when training baseline efficient net model with all data. \n\n![viz_of_kl_div_by_total_evaluators](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4250230%2F9f2d180d88de96d03a41fa119a0703c2%2Fviz_.png?generation=1711104172826486&alt=media)\n\nI will publish code soon.", "votes": 5, "replies": [ { "id": 2710824, "postDate": "2024-03-22T15:23:37.120Z", "content": "

Nice looking plot, but I think thetotal_evaluators axis might be incorrect. I don't see any columns with 8 or 9 votes.

\n
 pandas  pd\n\ndf= pd.read_csv()\ntotal_votes= df[[, , , , , ]].(axis=)\n(((total_votes.unique())))\n
", "rawMarkdown": "Nice looking plot, but I think the`total_evaluators` axis might be incorrect. I don't see any columns with 8 or 9 votes.\n\n```\nimport pandas as pd\n\ndf= pd.read_csv(\"/kaggle/input/hms-harmful-brain-activity-classification/train.csv\")\ntotal_votes= df[['gpd_vote', 'grda_vote', 'lpd_vote', 'lrda_vote', 'other_vote', 'seizure_vote']].sum(axis=1)\nprint(str(sorted(total_votes.unique())))\n```", "replies": [ { "id": 2710834, "postDate": "2024-03-22T15:37:17.317Z", "content": "

This plot is based on eeg-unique-df, not train.csv
\nThe eeg-unique-df is created in the same way as in chris or moth published notebook
\nThe eeg-unique-df is made by aggregation, so \"total_evaluators \" column contains float value.
\nI use df.astype(int) to plot this, so this is rough plot of result.

", "rawMarkdown": "This plot is based on *`eeg-unique-df`*, not train.csv\nThe eeg-unique-df is created in the same way as in [chris or moth published notebook](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train?scriptVersionId=161067167&cellId=11)\nThe eeg-unique-df is made by aggregation, so \"total_evaluators \" column contains float value.\nI use `df.astype(int)` to plot this, so this is rough plot of result.", "votes": 2 } ] }, { "id": 2710871, "postDate": "2024-03-22T16:17:41.940Z", "content": "

2stage learning looks like this.
\nThe model is trained in this way, EffNetB0 model trained twice, once for each of two training populations - [LB 0.39]

\n\n

As you can see, in data with total_evaluators > 9, score get better.
\nThis 2stage model get around 0.35 in public LB.
\n\"viz_compare_2stage\"

", "rawMarkdown": "2stage learning looks like this.\nThe model is trained in this way, [EffNetB0 model trained twice, once for each of two training populations - [LB 0.39]](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135)\n\n- Blue: stage1, CV(total_evaluators > 9)=0.40\n- Orange: srage2, CV(total_evaluators > 9)=0.33\n\nAs you can see, in data with total_evaluators > 9, score get better. \nThis 2stage model get around 0.35 in public LB.\n![viz_compare_2stage](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4250230%2F950351391c5b1c9226c18a501a690dce%2Foutput_2.png?generation=1711123884044601&alt=media)\n" } ] }, { "id": 2710373, "postDate": "2024-03-22T07:52:23.347Z", "content": "

4 folds votes >= 10 cv 2366 with online maybe 2499 LB.
\nOne thing I've noticed is if you change your folds seed as well as other seeds, you might see different cv, sometimes as large as from 256 to 249.
\nNow I got 4 folds model LB 0.24 on LB with cv 0.244.

", "rawMarkdown": "4 folds votes >= 10 cv 2366 with online maybe 2499 LB.\nOne thing I've noticed is if you change your folds seed as well as other seeds, you might see different cv, sometimes as large as from 256 to 249.\nNow I got 4 folds model LB 0.24 on LB with cv 0.244.", "votes": 4, "replies": [ { "id": 2710563, "postDate": "2024-03-22T11:05:13.500Z", "content": "

Super score! What do you mean \"with online\"?

", "rawMarkdown": "Super score! What do you mean \"with online\"?", "replies": [ { "id": 2710622, "postDate": "2024-03-22T12:07:13.153Z", "content": "

Online just mean submission LB result.

", "rawMarkdown": "Online just mean submission LB result." } ] }, { "id": 2710922, "postDate": "2024-03-22T16:47:05.083Z", "content": "

Thanks for sharing.
\nHow about cv of vote < 10 ? Not calculated ?\nDo you calculate cv in custom way or just filtered oof with vote >= 10 ?

", "rawMarkdown": "Thanks for sharing. \nHow about cv of vote < 10 ? Not calculated ?\nDo you calculate cv in custom way or just filtered oof with vote >= 10 ?", "replies": [ { "id": 2713023, "postDate": "2024-03-23T23:04:31.333Z", "content": "

I removed vote < 10 for eval part to speedup. I used custom cv.

", "rawMarkdown": "I removed vote < 10 for eval part to speedup. I used custom cv.", "votes": 2 }, { "id": 2713472, "postDate": "2024-03-24T08:18:45.323Z", "content": "
\n

I used custom cv

\n
\n

thanks. If just filtered vote >= 10, I got 0.27 in CV with vote >= 10, but it's not good in public LB.

", "rawMarkdown": "> I used custom cv\n\nthanks. If just filtered vote >= 10, I got 0.27 in CV with vote >= 10, but it's not good in public LB." }, { "id": 2713543, "postDate": "2024-03-24T09:02:51.520Z", "content": "

Have been looking at the breakdown of number of votes e.g., < 3 or >=10 etc. and at the high level expert consensus given by unique eeg id (17089) in train.

\n

Other has the majority of entries 7196 and roughly 40% of these have >=10 votes 37.5% < 3\nso fits in well with the <3 or >=10 idea. Possibly due to the majority class it has too much influence with >=10 filters (or < 3). GPD and LPD are somewhat similar but have less entries by comparison

\n

Seizure has 2785 entries but only around 6.6% of these have >= 10 votes and around 8% <3. Most ~85% are in the 3-9 votes. GRDA and LRDA are less extreme but over 50% in the 3-9 votes range.

\n

It is possible public LB has greater representation of Other and Seizure since these are the majority in train.
\nNotably Seizure as the only pattern with votes make up nearly 80% of entries. So losing these in splits by number of votes probably is bad for LB even if CV is good due to the influence of Other, etc.

", "rawMarkdown": "Have been looking at the breakdown of number of votes e.g., < 3 or >=10 etc. and at the high level expert consensus given by unique eeg id (17089) in train. \n\nOther has the majority of entries 7196 and roughly 40% of these have >=10 votes 37.5% < 3\nso fits in well with the <3 or >=10 idea. Possibly due to the majority class it has too much influence with >=10 filters (or < 3). GPD and LPD are somewhat similar but have less entries by comparison\n\nSeizure has 2785 entries but only around 6.6% of these have >= 10 votes and around 8% <3. Most ~85% are in the 3-9 votes. GRDA and LRDA are less extreme but over 50% in the 3-9 votes range.\n\nIt is possible public LB has greater representation of Other and Seizure since these are the majority in train. \nNotably Seizure as the only pattern with votes make up nearly 80% of entries. So losing these in splits by number of votes probably is bad for LB even if CV is good due to the influence of Other, etc. ", "votes": 9 }, { "id": 2713854, "postDate": "2024-03-24T13:13:33.420Z", "content": "

Thanks for insight !!
\nAggregating or filtering train.csv influences the \"expert_consensus\" distribution like following plot.

\n

\"label_distribution\"

", "rawMarkdown": "Thanks for insight !!\nAggregating or filtering train.csv influences the \"expert_consensus\" distribution like following plot.\n\n![label_distribution](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4250230%2F1dc4366bcc96ca123d710e392df41bdd%2Flabel_dist.png?generation=1711285918774479&alt=media)", "votes": 8 }, { "id": 2713944, "postDate": "2024-03-24T14:42:28.440Z", "content": "

Thanks for the plot!

", "rawMarkdown": "Thanks for the plot! " }, { "id": 2714037, "postDate": "2024-03-24T16:08:16.687Z", "content": "

Thanks for your plot, it is clear to see now the main distribution diff is on Seizure.

", "rawMarkdown": "Thanks for your plot, it is clear to see now the main distribution diff is on Seizure.", "votes": 1 } ] }, { "id": 2711260, "postDate": "2024-03-22T20:09:10.123Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2725489, "postDate": "2024-03-31T15:26:12.253Z", "content": "

do you apply \"label+0.1666\"？to get cv0.244

", "rawMarkdown": "do you apply \"label+0.1666\"？to get cv0.244", "replies": [ { "id": 2725508, "postDate": "2024-03-31T15:41:22.410Z", "content": "

No I do not use that.

", "rawMarkdown": "No I do not use that." } ] } ] }, { "id": 2716441, "postDate": "2024-03-26T03:41:13.670Z", "content": "

actually ignore the kl divergence score and only consider e.g. top1, top2 classification accuracy, they are quite high.
\nanother wrong design of evaluation metric?

", "rawMarkdown": "actually ignore the kl divergence score and only consider e.g. top1, top2 classification accuracy, they are quite high.\nanother wrong design of evaluation metric?" }, { "id": 2710298, "postDate": "2024-03-22T06:43:00.497Z", "content": "

In two-stage model, high quality(vote >= 10) CV always same with LB

", "rawMarkdown": "In two-stage model, high quality(vote >= 10) CV always same with LB " }, { "id": 2710521, "postDate": "2024-03-22T10:27:10.007Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2710168, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-03-22T04:31:59.207000", "content": "

Our vote count >= 10 OOF score is 0.2174. Top 5 teams probably reached beyond 0.2.

", "votes": 8, "replies": [ { "id": 2710462, "author_name": "NikhilMishra", "author_url": "", "post_date": "2024-03-22T09:21:18.293000", "content": "

is this cv on Chris's created training data filtered out with votes >= 10

", "votes": 0, "replies": [ { "id": 2710463, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-03-22T09:22:15.347000", "content": "

Nope, we are using a custom cv scheme.

", "votes": 1, "replies": [] } ] } ] }, { "id": 2710549, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-03-22T10:54:24.913000", "content": "

Here is the prediction distribution by total_evaluators in oof_df.
\nY: kl divergence from label
\nX: total_evaluators

\n

This prediction is made by baseline efficient net model trained with all data, cv score with all data is 0.5X.
\nThe baseline model is almost same as moth's published notebook.

\n

As you can see, kl divergence is lager in data with total_evaluators <= 3.
\nIt seems that data with total_evaluators <= 3 have difficult sample or noizy sample.
\nIn my experiments, this distribution often happens when training baseline efficient net model with all data.

\n

\"viz_of_kl_div_by_total_evaluators\"

\n

I will publish code soon.

", "votes": 5, "replies": [ { "id": 2710824, "author_name": "Bartley", "author_url": "", "post_date": "2024-03-22T15:23:37.120000", "content": "

Nice looking plot, but I think thetotal_evaluators axis might be incorrect. I don't see any columns with 8 or 9 votes.

\n
 pandas  pd\n\ndf= pd.read_csv()\ntotal_votes= df[[, , , , , ]].(axis=)\n(((total_votes.unique())))\n
", "votes": 0, "replies": [ { "id": 2710834, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-03-22T15:37:17.317000", "content": "

This plot is based on eeg-unique-df, not train.csv
\nThe eeg-unique-df is created in the same way as in chris or moth published notebook
\nThe eeg-unique-df is made by aggregation, so \"total_evaluators \" column contains float value.
\nI use df.astype(int) to plot this, so this is rough plot of result.

", "votes": 2, "replies": [] } ] }, { "id": 2710871, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-03-22T16:17:41.940000", "content": "

2stage learning looks like this.
\nThe model is trained in this way, EffNetB0 model trained twice, once for each of two training populations - [LB 0.39]

\n\n

As you can see, in data with total_evaluators > 9, score get better.
\nThis 2stage model get around 0.35 in public LB.
\n\"viz_compare_2stage\"

", "votes": 0, "replies": [] } ] }, { "id": 2710373, "author_name": "gezi", "author_url": "", "post_date": "2024-03-22T07:52:23.347000", "content": "

4 folds votes >= 10 cv 2366 with online maybe 2499 LB.
\nOne thing I've noticed is if you change your folds seed as well as other seeds, you might see different cv, sometimes as large as from 256 to 249.
\nNow I got 4 folds model LB 0.24 on LB with cv 0.244.

", "votes": 4, "replies": [ { "id": 2710563, "author_name": "Phaedrus", "author_url": "", "post_date": "2024-03-22T11:05:13.500000", "content": "

Super score! What do you mean \"with online\"?

", "votes": 0, "replies": [ { "id": 2710622, "author_name": "gezi", "author_url": "", "post_date": "2024-03-22T12:07:13.153000", "content": "

Online just mean submission LB result.

", "votes": 0, "replies": [] } ] }, { "id": 2710922, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-03-22T16:47:05.083000", "content": "

Thanks for sharing.
\nHow about cv of vote < 10 ? Not calculated ?\nDo you calculate cv in custom way or just filtered oof with vote >= 10 ?

", "votes": 0, "replies": [ { "id": 2713023, "author_name": "gezi", "author_url": "", "post_date": "2024-03-23T23:04:31.333000", "content": "

I removed vote < 10 for eval part to speedup. I used custom cv.

", "votes": 2, "replies": [] }, { "id": 2713472, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-03-24T08:18:45.323000", "content": "
\n

I used custom cv

\n
\n

thanks. If just filtered vote >= 10, I got 0.27 in CV with vote >= 10, but it's not good in public LB.

", "votes": 0, "replies": [] }, { "id": 2713543, "author_name": "something4kag", "author_url": "", "post_date": "2024-03-24T09:02:51.520000", "content": "

Have been looking at the breakdown of number of votes e.g., < 3 or >=10 etc. and at the high level expert consensus given by unique eeg id (17089) in train.

\n

Other has the majority of entries 7196 and roughly 40% of these have >=10 votes 37.5% < 3\nso fits in well with the <3 or >=10 idea. Possibly due to the majority class it has too much influence with >=10 filters (or < 3). GPD and LPD are somewhat similar but have less entries by comparison

\n

Seizure has 2785 entries but only around 6.6% of these have >= 10 votes and around 8% <3. Most ~85% are in the 3-9 votes. GRDA and LRDA are less extreme but over 50% in the 3-9 votes range.

\n

It is possible public LB has greater representation of Other and Seizure since these are the majority in train.
\nNotably Seizure as the only pattern with votes make up nearly 80% of entries. So losing these in splits by number of votes probably is bad for LB even if CV is good due to the influence of Other, etc.

", "votes": 9, "replies": [] }, { "id": 2713854, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-03-24T13:13:33.420000", "content": "

Thanks for insight !!
\nAggregating or filtering train.csv influences the \"expert_consensus\" distribution like following plot.

\n

\"label_distribution\"

", "votes": 8, "replies": [] }, { "id": 2713944, "author_name": "something4kag", "author_url": "", "post_date": "2024-03-24T14:42:28.440000", "content": "

Thanks for the plot!

", "votes": 0, "replies": [] }, { "id": 2714037, "author_name": "gezi", "author_url": "", "post_date": "2024-03-24T16:08:16.687000", "content": "

Thanks for your plot, it is clear to see now the main distribution diff is on Seizure.

", "votes": 1, "replies": [] } ] }, { "id": 2711260, "author_name": "", "author_url": "", "post_date": "2024-03-22T20:09:10.123000", "content": "", "votes": 0, "replies": [] }, { "id": 2725489, "author_name": "pky", "author_url": "", "post_date": "2024-03-31T15:26:12.253000", "content": "

do you apply \"label+0.1666\"？to get cv0.244

", "votes": 0, "replies": [ { "id": 2725508, "author_name": "gezi", "author_url": "", "post_date": "2024-03-31T15:41:22.410000", "content": "

No I do not use that.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2716441, "author_name": "hengck23", "author_url": "", "post_date": "2024-03-26T03:41:13.670000", "content": "

actually ignore the kl divergence score and only consider e.g. top1, top2 classification accuracy, they are quite high.
\nanother wrong design of evaluation metric?

", "votes": 0, "replies": [] }, { "id": 2710298, "author_name": "Peter", "author_url": "", "post_date": "2024-03-22T06:43:00.497000", "content": "

In two-stage model, high quality(vote >= 10) CV always same with LB

", "votes": 0, "replies": [] }, { "id": 2710521, "author_name": "", "author_url": "", "post_date": "2024-03-22T10:27:10.007000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2709761": "As most have found by now, there are two groups in our data set. Those who had many votes and those who did not. Calculating CV for the subsets separately we can see that the CV LB gap is actually quite small if we consider that the public LB is mostly high vote data. Of course, there are many assumptions people are making about what the private dataset may hold for us. I thought I would share some of my numbers in hope other may share as well.\n\nHigh votes group CV: 0.33 \nLow votes group CV: 0.6\nLB: 0.35\n\nThis should be a fun competition at the end!", "2710168": "Our vote count >= 10 OOF score is 0.2174. Top 5 teams probably reached beyond 0.2.", "2710549": "Here is the prediction distribution by `total_evaluators` in oof_df.\nY: kl divergence from label\nX: total_evaluators\n\nThis prediction is made by baseline efficient net model trained with all data, cv score with all data is 0.5X.\nThe baseline model is almost same as [moth's published notebook](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train).\n\nAs you can see, kl divergence is lager in data with total_evaluators <= 3.\nIt seems that data with total_evaluators <= 3 have difficult sample or noizy sample.\nIn my experiments, this distribution often happens when training baseline efficient net model with all data. \n\n![viz_of_kl_div_by_total_evaluators](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4250230%2F9f2d180d88de96d03a41fa119a0703c2%2Fviz_.png?generation=1711104172826486&alt=media)\n\nI will publish code soon.", "2710373": "4 folds votes >= 10 cv 2366 with online maybe 2499 LB.\nOne thing I've noticed is if you change your folds seed as well as other seeds, you might see different cv, sometimes as large as from 256 to 249.\nNow I got 4 folds model LB 0.24 on LB with cv 0.244.", "2716441": "actually ignore the kl divergence score and only consider e.g. top1, top2 classification accuracy, they are quite high.\nanother wrong design of evaluation metric?", "2710298": "In two-stage model, high quality(vote >= 10) CV always same with LB ", "2710521": "" } }