![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4250230%2F5e2f00904e50cfc6e303eea31fddb7a1%2Foutput.png?generation=1710000306691110&alt=media\")
Currently, the two-stage method is popular in public methods. I think the main reason why this method is effective is because it is found that LB data tends to a certain distribution, but this distribution is far from the distribution of the training set. Private data sets may not and the characteristics presented by LB data, so I think we still need to be wary of this method to avoid overfitting LB data and causing shake.
", "messages": [ { "id": 2687170, "postDate": "2024-03-08T10:41:00.367Z", "content": "From the competition data page:
\n\n\nNote that the test samples had between 3 and 20 annotators.
\n
If we train our models on subsets which are more focused on votes > 10 (skipping almost half of the test data).
\nHow can we not expect a shakeup?
", "rawMarkdown": "From the competition data page:\n>Note that the test samples had between 3 and 20 annotators.\n\nIf we train our models on subsets which are more focused on votes > 10 (skipping almost half of the test data). \n\nHow can we not expect a shakeup?", "votes": 7, "replies": [ { "id": 2687271, "postDate": "2024-03-08T12:14:37.843Z", "content": "I can't think of any logical reason why public and private would be different, and 35% is a good representative ratio.
", "rawMarkdown": "I can't think of any logical reason why public and private would be different, and 35% is a good representative ratio.", "replies": [ { "id": 2687659, "postDate": "2024-03-08T17:14:26.370Z", "content": "@gunesevitan different distributions for public/private lbs is very common in Kaggle competitions
", "rawMarkdown": "@gunesevitan different distributions for public/private lbs is very common in Kaggle competitions ", "votes": 5 }, { "id": 2687738, "postDate": "2024-03-08T18:32:03.823Z", "content": "Yes, but most of the time it's part of the problem i.e. using one type of data and predicting another type. Organizers either want you to do domain adaptation (computer vision) or build a stable model (time series). I can't imagine either of those scenarios to be expected here.
", "rawMarkdown": "Yes, but most of the time it's part of the problem i.e. using one type of data and predicting another type. Organizers either want you to do domain adaptation (computer vision) or build a stable model (time series). I can't imagine either of those scenarios to be expected here." }, { "id": 2687945, "postDate": "2024-03-08T21:59:55.723Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2688188, "postDate": "2024-03-09T04:48:09.860Z", "content": "So what this there in public LB 35 % data distribution same as the stage 2 training data so public LB good. If the private 65% contains more 1 - 10 annotators data. Then there is chance of shake up.. Is my understanding is correct?
", "rawMarkdown": "So what this there in public LB 35 % data distribution same as the stage 2 training data so public LB good. If the private 65% contains more 1 - 10 annotators data. Then there is chance of shake up.. Is my understanding is correct? ", "votes": 1 } ] }, { "id": 2688016, "postDate": "2024-03-08T23:42:29.330Z", "content": "This is interesting, in train about 50% samples are with exactly 3 annotators.
", "rawMarkdown": "This is interesting, in train about 50% samples are with exactly 3 annotators.", "votes": 3 } ] }, { "id": 2686945, "postDate": "2024-03-08T07:02:25.320Z", "content": "Currently, the two-stage method is popular in public methods. I think the main reason why this method is effective is because it is found that LB data tends to a certain distribution, but this distribution is far from the distribution of the training set. Private data sets may not and the characteristics presented by LB data, so I think we still need to be wary of this method to avoid overfitting LB data and causing shake.
", "rawMarkdown": "Currently, the two-stage method is popular in public methods. I think the main reason why this method is effective is because it is found that LB data tends to a certain distribution, but this distribution is far from the distribution of the training set. Private data sets may not and the characteristics presented by LB data, so I think we still need to be wary of this method to avoid overfitting LB data and causing shake.", "votes": 5 }, { "id": 2689044, "postDate": "2024-03-09T16:05:53.830Z", "content": "Here is a Cumulative Distribution of Total Evaluators in eeg_id-unique-train_df for reference.
\nsorted_counts = df_train[].value_counts().sort_index()\ncumulative_dist = sorted_counts.cumsum() / sorted_counts.()\nplt.plot(cumulative_dist.index, cumulative_dist, marker=, linestyle=)\n
The public to private ratio is ~1:2.
\nThe ratio of small and large number of experts on train data is ~1:2.
\nThink about it… ))
If that's true, we need two submissions, one for the public set (10~20 annotators), and one for the private set (3~10 annotators).
", "rawMarkdown": "If that's true, we need two submissions, one for the public set (10~20 annotators), and one for the private set (3~10 annotators).\n", "votes": 2, "replies": [ { "id": 2687420, "postDate": "2024-03-08T15:17:49.700Z", "content": "I think you can make a suitable integration for these two distributions
", "rawMarkdown": "I think you can make a suitable integration for these two distributions", "votes": 1 }, { "id": 2687555, "postDate": "2024-03-08T16:05:15.140Z", "content": "so if we ensample stage 1 + stage 2 models combined will avoid the 10~20 annotators and 3~10 annotators different distributions issue.
\nBut i am currently submitting only stage 2. so chances of shake is lot form my understanding.
If the organizers made such a split, it would be extremely unfair towards the competitors (us).
\nAnd also make the competition a bit meaningless.
\nAs Andrew Ng said in one of his courses, validation should be approximately from the same distribution of the test/final target (not exact words, but close enough, I think)
\nOtherwise, what are we even doing other than mindlessly guessing? Lol.
Totally agree. It doesn't make sense to do such thing and very uncharacteristic of Kaggle. When public and private test sets are different in competitions, they always tell us and want us to build robust models in competitions like hubmap or ubc ocean. Doing the same thing here doesn't make any sense and fails the objective of this competition.
", "rawMarkdown": "Totally agree. It doesn't make sense to do such thing and very uncharacteristic of Kaggle. When public and private test sets are different in competitions, they always tell us and want us to build robust models in competitions like hubmap or ubc ocean. Doing the same thing here doesn't make any sense and fails the objective of this competition.", "votes": 3 }, { "id": 2687532, "postDate": "2024-03-08T15:58:20.473Z", "content": "Eactly, also in the Ribonanza competition there was a case that the private test had a different distribution from public/train, but it was OK since they tolds as it is the case and how the distribution differs, so we could plan our validation scheme accordingly.
\nBut if they just put random distribution in the private test, that is different both from the train distribution and the public test distribution, and does not inform us on the matter, it is just…well…idk, data sciencve done wrong?
and throw away 50k for no value whatsoever
", "rawMarkdown": "and throw away 50k for no value whatsoever", "votes": 2 }, { "id": 2687948, "postDate": "2024-03-08T22:03:02.230Z", "content": "I totally agree with this but in reality I see most comps have different distributions and different shakeup. CV normally wins
", "rawMarkdown": "I totally agree with this but in reality I see most comps have different distributions and different shakeup. CV normally wins", "votes": 1 } ] } ] }, { "id": 2687976, "postDate": "2024-03-08T22:37:54.163Z", "content": "The fact that cv and lb scores are so different make me think there could be some shakeup.
", "rawMarkdown": "The fact that cv and lb scores are so different make me think there could be some shakeup. ", "votes": 2 }, { "id": 2686953, "postDate": "2024-03-08T07:11:55.267Z", "content": "I'm not expecting any shake up.
", "rawMarkdown": "I'm not expecting any shake up.", "votes": 1, "replies": [ { "id": 2686988, "postDate": "2024-03-08T07:30:29.943Z", "content": "Hope nothing shakes your position😀
", "rawMarkdown": "Hope nothing shakes your position😀", "votes": 1, "replies": [ { "id": 2687038, "postDate": "2024-03-08T08:08:56.907Z", "content": "We are hoping to reach 1st place 🙏
", "rawMarkdown": "We are hoping to reach 1st place 🙏", "votes": 2 } ] } ] }, { "id": 2688512, "postDate": "2024-03-09T09:55:01.607Z", "content": "In fact, the reason why I feel this way is because in the two months before the game, most teams were stuck at 0.34. Even though the offline cv was optimized, the LB did not change until the second stage method emerged. The situation has improved, but for the front-row team, this does not affect them:)
", "rawMarkdown": "In fact, the reason why I feel this way is because in the two months before the game, most teams were stuck at 0.34. Even though the offline cv was optimized, the LB did not change until the second stage method emerged. The situation has improved, but for the front-row team, this does not affect them:)", "replies": [ { "id": 2688517, "postDate": "2024-03-09T09:58:01.830Z", "content": "https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479776#2671166 That's why I think what he said makes sense
", "rawMarkdown": "https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479776#2671166 That's why I think what he said makes sense" }, { "id": 2688526, "postDate": "2024-03-09T10:06:25.217Z", "content": "You mean people in top 20 LB .. Not used stage wise training.. So the shake up in private LB will be less for the front row people… Is my understanding is correct
", "rawMarkdown": "You mean people in top 20 LB .. Not used stage wise training.. So the shake up in private LB will be less for the front row people... Is my understanding is correct", "replies": [ { "id": 2688551, "postDate": "2024-03-09T10:11:45.287Z", "content": "They should probably realize that the occurrence of this situation will make the model more robust. As for whether to use the second stage, I can’t judge.
", "rawMarkdown": "They should probably realize that the occurrence of this situation will make the model more robust. As for whether to use the second stage, I can’t judge.", "votes": 1 } ] } ] }, { "id": 2688484, "postDate": "2024-03-09T09:14:12.643Z", "content": "considering SenNet + HOA - Hacking the Human Vasculature in 3D anything is possible.
\nIt is not just number of annotators, but quality of spectrograms - differences between competition provided and generated from EEG raw data have been noted. For the latter no information has been provided on preprocessing methodology which could be different based on source of original data.
\nIt is not specified if source of patients are the same facilities, equipment the same, etc. And distribution of patterns represented could vary, patients should be different for public test from private test.
Usually hosts/sponsors are looking for robust solutions or to verify what is already known to work best, so time will tell…
\nGood luck to everyone!
From the competition data page:
\n\n\nNote that the test samples had between 3 and 20 annotators.
\n
If we train our models on subsets which are more focused on votes > 10 (skipping almost half of the test data).
\nHow can we not expect a shakeup?
", "votes": 7, "replies": [ { "id": 2687271, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-03-08T12:14:37.843000", "content": "I can't think of any logical reason why public and private would be different, and 35% is a good representative ratio.
", "votes": 0, "replies": [ { "id": 2687659, "author_name": "Muhammad Ahmed", "author_url": "", "post_date": "2024-03-08T17:14:26.370000", "content": "@gunesevitan different distributions for public/private lbs is very common in Kaggle competitions
", "votes": 5, "replies": [] }, { "id": 2687738, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-03-08T18:32:03.823000", "content": "Yes, but most of the time it's part of the problem i.e. using one type of data and predicting another type. Organizers either want you to do domain adaptation (computer vision) or build a stable model (time series). I can't imagine either of those scenarios to be expected here.
", "votes": 0, "replies": [] }, { "id": 2687945, "author_name": "", "author_url": "", "post_date": "2024-03-08T21:59:55.723000", "content": "", "votes": 0, "replies": [] }, { "id": 2688188, "author_name": "xavier fernando", "author_url": "", "post_date": "2024-03-09T04:48:09.860000", "content": "So what this there in public LB 35 % data distribution same as the stage 2 training data so public LB good. If the private 65% contains more 1 - 10 annotators data. Then there is chance of shake up.. Is my understanding is correct?
", "votes": 1, "replies": [] } ] }, { "id": 2688016, "author_name": "gezi", "author_url": "", "post_date": "2024-03-08T23:42:29.330000", "content": "This is interesting, in train about 50% samples are with exactly 3 annotators.
", "votes": 3, "replies": [] } ] }, { "id": 2689044, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-03-09T16:05:53.830000", "content": "Here is a Cumulative Distribution of Total Evaluators in eeg_id-unique-train_df for reference.
\nsorted_counts = df_train[].value_counts().sort_index()\ncumulative_dist = sorted_counts.cumsum() / sorted_counts.()\nplt.plot(cumulative_dist.index, cumulative_dist, marker=, linestyle=)\n
The public to private ratio is ~1:2.
\nThe ratio of small and large number of experts on train data is ~1:2.
\nThink about it… ))
If that's true, we need two submissions, one for the public set (10~20 annotators), and one for the private set (3~10 annotators).
", "votes": 2, "replies": [ { "id": 2687420, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2024-03-08T15:17:49.700000", "content": "I think you can make a suitable integration for these two distributions
", "votes": 1, "replies": [] }, { "id": 2687555, "author_name": "xavier fernando", "author_url": "", "post_date": "2024-03-08T16:05:15.140000", "content": "so if we ensample stage 1 + stage 2 models combined will avoid the 10~20 annotators and 3~10 annotators different distributions issue.
\nBut i am currently submitting only stage 2. so chances of shake is lot form my understanding.
If the organizers made such a split, it would be extremely unfair towards the competitors (us).
\nAnd also make the competition a bit meaningless.
\nAs Andrew Ng said in one of his courses, validation should be approximately from the same distribution of the test/final target (not exact words, but close enough, I think)
\nOtherwise, what are we even doing other than mindlessly guessing? Lol.
Totally agree. It doesn't make sense to do such thing and very uncharacteristic of Kaggle. When public and private test sets are different in competitions, they always tell us and want us to build robust models in competitions like hubmap or ubc ocean. Doing the same thing here doesn't make any sense and fails the objective of this competition.
", "votes": 3, "replies": [] }, { "id": 2687532, "author_name": "greySnow", "author_url": "", "post_date": "2024-03-08T15:58:20.473000", "content": "Eactly, also in the Ribonanza competition there was a case that the private test had a different distribution from public/train, but it was OK since they tolds as it is the case and how the distribution differs, so we could plan our validation scheme accordingly.
\nBut if they just put random distribution in the private test, that is different both from the train distribution and the public test distribution, and does not inform us on the matter, it is just…well…idk, data sciencve done wrong?
and throw away 50k for no value whatsoever
", "votes": 2, "replies": [] }, { "id": 2687948, "author_name": "Cody_Null", "author_url": "", "post_date": "2024-03-08T22:03:02.230000", "content": "I totally agree with this but in reality I see most comps have different distributions and different shakeup. CV normally wins
", "votes": 1, "replies": [] } ] } ] }, { "id": 2687976, "author_name": "chemdatafarmer", "author_url": "", "post_date": "2024-03-08T22:37:54.163000", "content": "The fact that cv and lb scores are so different make me think there could be some shakeup.
", "votes": 2, "replies": [] }, { "id": 2686953, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-03-08T07:11:55.267000", "content": "I'm not expecting any shake up.
", "votes": 1, "replies": [ { "id": 2686988, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2024-03-08T07:30:29.943000", "content": "Hope nothing shakes your position😀
", "votes": 1, "replies": [ { "id": 2687038, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-03-08T08:08:56.907000", "content": "We are hoping to reach 1st place 🙏
", "votes": 2, "replies": [] } ] } ] }, { "id": 2688512, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2024-03-09T09:55:01.607000", "content": "In fact, the reason why I feel this way is because in the two months before the game, most teams were stuck at 0.34. Even though the offline cv was optimized, the LB did not change until the second stage method emerged. The situation has improved, but for the front-row team, this does not affect them:)
", "votes": 0, "replies": [ { "id": 2688517, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2024-03-09T09:58:01.830000", "content": "https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/479776#2671166 That's why I think what he said makes sense
", "votes": 0, "replies": [] }, { "id": 2688526, "author_name": "xavier fernando", "author_url": "", "post_date": "2024-03-09T10:06:25.217000", "content": "You mean people in top 20 LB .. Not used stage wise training.. So the shake up in private LB will be less for the front row people… Is my understanding is correct
", "votes": 0, "replies": [ { "id": 2688551, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2024-03-09T10:11:45.287000", "content": "They should probably realize that the occurrence of this situation will make the model more robust. As for whether to use the second stage, I can’t judge.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2688484, "author_name": "something4kag", "author_url": "", "post_date": "2024-03-09T09:14:12.643000", "content": "considering SenNet + HOA - Hacking the Human Vasculature in 3D anything is possible.
\nIt is not just number of annotators, but quality of spectrograms - differences between competition provided and generated from EEG raw data have been noted. For the latter no information has been provided on preprocessing methodology which could be different based on source of original data.
\nIt is not specified if source of patients are the same facilities, equipment the same, etc. And distribution of patterns represented could vary, patients should be different for public test from private test.
Usually hosts/sponsors are looking for robust solutions or to verify what is already known to work best, so time will tell…
\nGood luck to everyone!