![\"scatter](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F15433644%2F19065e32cf0f6cabdec631924707a537%2FUntitled.png?generation=1708067270830919&alt=media\")
@Sean R.B. Bearden provides a great idea in his insightful notebook. He proposes a two-stage training method to address the KL-divergence issue caused by samples with 1-7 votes.
\nHowever, there is no clear correlation between the number of votes and the distribution of votes. As evidence, the scatter diagram of total votes - KL Loss (KL Loss between votes and uniform distribution) is as follows.
\n
The calculation of KL Loss is as follows:
\ny_data = data[label_cols].values\ny_data = y_data / y_data.(axis=, keepdims=)\ndata[label_cols] = y_data\nlabels = data[[, , , , , ]].values + \n\n\ndata[] = torch.nn.functional.kl_div(\n torch.log(torch.tensor(labels)),\n torch.tensor([ / ] * ),\n reduction=\n).(dim=).numpy()\n(, data[].(), , data[].())\nThere is an important observation: samples with peaked distributions (KL Loss > 7) present in both groups
\nTherefore, according to the motivation and the observation, I suggest dividing train data according to KL Loss. My experimental configs are as follows:
\n| Method | \nCV (first stage) | \nCV (second stage) | \nLB | \n
|---|---|---|---|
| single-stage | \n0.60 | \n- | \n0.41 | \n
| two-stage (KL Loss) | \n0.60 | \n0.33 | \n0.37 | \n
| two-stage (total votes) | \n0.60 | \n0.38 | \n0.41 | \n
Please note that CVs of different stages cannot be compared. Because the training and validation of the 2nd stage are based on filtered data. This is a fast implementation. For a more detailed discussion, please refer to my comment. I will fix this issue and update the results.
\nI fix the issue and the training framework is as follows:
\n ():\n ....\n total_df = read_train_metadata(data_root) \n split_train_validation(total_df, **self.data_conf[]) \n\n \n train_df = total_df[total_df[] != self.data_conf[]].reset_index(drop=)\n val_df = total_df[total_df[] == self.data_conf[]].reset_index(drop=)\n\n self.optim_conf[][] == :\n .... \n ....\nAs long as the tagging of training and validation is the same (this can be achieved by fixing random seeds), these two stages use the same validation data. The result are as follows:
\n| Method | \nCV (first stage) | \nCV (second stage) | \n
|---|---|---|
| two-stage (KL Loss) | \n0.59836 | \n0.58767 | \n
CV has been improved.
\nMore detailed result:
\n| stage | \nfold-1 | \nfold-2 | \nfold-3 | \nfold-4 | \nfold-5 | \n
|---|---|---|---|---|---|
| 1st | \n0.63056 | \n0.55998 | \n0.53577 | \n0.58305 | \n0.68245 | \n
| 2nd | \n0.60282 | \n0.55192 | \n0.53050 | \n0.60580 | \n0.64730 | \n
This result is surprising. Although there is an improvement in CV (0.60->0.59), the magnitude of the improvement is much smaller than that of LB (0.41->0.37). I'm not sure if there's any special reason.
", "messages": [ { "id": 2654532, "postDate": "2024-02-16T07:38:41.517Z", "content": "@Sean R.B. Bearden provides a great idea in his insightful notebook. He proposes a two-stage training method to address the KL-divergence issue caused by samples with 1-7 votes.
\nHowever, there is no clear correlation between the number of votes and the distribution of votes. As evidence, the scatter diagram of total votes - KL Loss (KL Loss between votes and uniform distribution) is as follows.
\n
The calculation of KL Loss is as follows:
\ny_data = data[label_cols].values\ny_data = y_data / y_data.(axis=, keepdims=)\ndata[label_cols] = y_data\nlabels = data[[, , , , , ]].values + \n\n\ndata[] = torch.nn.functional.kl_div(\n torch.log(torch.tensor(labels)),\n torch.tensor([ / ] * ),\n reduction=\n).(dim=).numpy()\n(, data[].(), , data[].())\nThere is an important observation: samples with peaked distributions (KL Loss > 7) present in both groups
\nTherefore, according to the motivation and the observation, I suggest dividing train data according to KL Loss. My experimental configs are as follows:
\n| Method | \nCV (first stage) | \nCV (second stage) | \nLB | \n
|---|---|---|---|
| single-stage | \n0.60 | \n- | \n0.41 | \n
| two-stage (KL Loss) | \n0.60 | \n0.33 | \n0.37 | \n
| two-stage (total votes) | \n0.60 | \n0.38 | \n0.41 | \n
Please note that CVs of different stages cannot be compared. Because the training and validation of the 2nd stage are based on filtered data. This is a fast implementation. For a more detailed discussion, please refer to my comment. I will fix this issue and update the results.
\nI fix the issue and the training framework is as follows:
\n ():\n ....\n total_df = read_train_metadata(data_root) \n split_train_validation(total_df, **self.data_conf[]) \n\n \n train_df = total_df[total_df[] != self.data_conf[]].reset_index(drop=)\n val_df = total_df[total_df[] == self.data_conf[]].reset_index(drop=)\n\n self.optim_conf[][] == :\n .... \n ....\nAs long as the tagging of training and validation is the same (this can be achieved by fixing random seeds), these two stages use the same validation data. The result are as follows:
\n| Method | \nCV (first stage) | \nCV (second stage) | \n
|---|---|---|
| two-stage (KL Loss) | \n0.59836 | \n0.58767 | \n
CV has been improved.
\nMore detailed result:
\n| stage | \nfold-1 | \nfold-2 | \nfold-3 | \nfold-4 | \nfold-5 | \n
|---|---|---|---|---|---|
| 1st | \n0.63056 | \n0.55998 | \n0.53577 | \n0.58305 | \n0.68245 | \n
| 2nd | \n0.60282 | \n0.55192 | \n0.53050 | \n0.60580 | \n0.64730 | \n
This result is surprising. Although there is an improvement in CV (0.60->0.59), the magnitude of the improvement is much smaller than that of LB (0.41->0.37). I'm not sure if there's any special reason.
", "rawMarkdown": "[@Sean R.B. Bearden](https://www.kaggle.com/seanbearden) provides a great idea in his insightful [notebook](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135). He proposes a two-stage training method to address the KL-divergence issue caused by samples with 1-7 votes. \n\n- **Motivation**: samples with peaked distributions lead models to predict with exaggerated confidence, resulting in worse performance.\n- **Method**: two-stage training method (the first stage only uses samples with 1-7 votes, then samples with 10-28 votes are used to fine-tune models).\n\nHowever, there is no clear correlation between the number of votes and the distribution of votes. As evidence, the scatter diagram of total votes - KL Loss (KL Loss between votes and uniform distribution) is as follows.\n\n\n\nThe calculation of KL Loss is as follows:\n```python\ny_data = data[label_cols].values\ny_data = y_data / y_data.sum(axis=1, keepdims=True)\ndata[label_cols] = y_data\nlabels = data[['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']].values + 1e-5\n\n# compute kl-loss with uniform distribution by pytorch\ndata['kl'] = torch.nn.functional.kl_div(\n torch.log(torch.tensor(labels)),\n torch.tensor([1 / 6] * 6),\n reduction='none'\n).sum(dim=1).numpy()\nprint('min:', data['kl'].min(), 'max:', data['kl'].max())\n```\n\nThere is an important observation: samples with peaked distributions (KL Loss > 7) present in both groups\n\nTherefore, according to the motivation and the observation, I suggest dividing train data according to KL Loss. My experimental configs are as follows:\n- EffNetB2 (based on [@alejopaullier](https://www.kaggle.com/alejopaullier)'s [notebook](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train/notebook))\n- First stage: training with all data\n- Second stage: training with samples (KL Loss < 5.5).\n\n| Method | CV (first stage) | CV (second stage) | LB |\n| --- | --- | --- | --- |\n| single-stage | 0.60 | - | 0.41 |\n|two-stage (KL Loss)|0.60|0.33|**0.37**|\n|two-stage (total votes)|0.60|0.38|0.41|\n\n---\nPlease note that CVs of different stages cannot be compared. Because the training and validation of the 2nd stage are based on filtered data. This is a fast implementation. For a more detailed discussion, please refer to [my comment](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461#2655472). I will fix this issue and update the results.\n\n---\nI fix the issue and the training framework is as follows:\n```python\ndef training():\n ....\n total_df = read_train_metadata(data_root) # load metadata\n split_train_validation(total_df, **self.data_conf['fold']) # tag training and validation\n\n # create train_df & val_df\n train_df = total_df[total_df['fold'] != self.data_conf['val_fold_id']].reset_index(drop=True)\n val_df = total_df[total_df['fold'] == self.data_conf['val_fold_id']].reset_index(drop=True)\n\n if self.optim_conf['train']['stage'] == 2:\n .... # filter only on train_df\n ....\n```\n\nAs long as the tagging of training and validation is the same (this can be achieved by fixing random seeds), these two stages use the same validation data. The result are as follows:\n\n| Method | CV (first stage) | CV (second stage) |\n| --- | --- | --- |\n|two-stage (KL Loss)|0.59836|0.58767|\n\nCV has been improved.\n\nMore detailed result:\n\n| stage | fold-1 | fold-2 | fold-3 | fold-4 | fold-5 |\n| --- | --- | --- | --- | --- | --- |\n|1st|0.63056|0.55998|0.53577|**0.58305**|0.68245|\n|2nd|**0.60282**|**0.55192**|**0.53050**|0.60580|**0.64730**|\n\n\n**This result is surprising**. Although there is an improvement in CV (0.60->0.59), the magnitude of the improvement is much smaller than that of LB (0.41->0.37). I'm not sure if there's any special reason.", "votes": 112 }, { "id": 2654899, "postDate": "2024-02-16T14:41:04.903Z", "content": "did you just train and validated with filtered record in stage 2 ? I would have thought that you would filter out the <5.5 KLLoss records for train set and continue with the same validation set as stage1 ?
", "rawMarkdown": "did you just train and validated with filtered record in stage 2 ? I would have thought that you would filter out the <5.5 KLLoss records for train set and continue with the same validation set as stage1 ? ", "votes": 5, "replies": [ { "id": 2655477, "postDate": "2024-02-16T22:51:04.567Z", "content": "Training and validation are based on filtered data in the 2nd stage. I discuss my training framework in this comment. And you are right, using the same validation set is a better method, I will fix it and update CVs.
", "rawMarkdown": "Training and validation are based on filtered data in the 2nd stage. I discuss my training framework in this [comment](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461#2655472). And you are right, using the same validation set is a better method, I will fix it and update CVs.", "votes": 1 } ] }, { "id": 2654920, "postDate": "2024-02-16T15:04:58.210Z", "content": "Do you make sure you are using same split in stage 1 and stage 2? Looks to me that choosing samples with KL<5.5 is causing mild leakage to your strategy, as CV is too optimal.
", "rawMarkdown": "Do you make sure you are using same split in stage 1 and stage 2? Looks to me that choosing samples with KL<5.5 is causing mild leakage to your strategy, as CV is too optimal.", "votes": 6, "replies": [ { "id": 2654949, "postDate": "2024-02-16T15:37:23.423Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2654953, "postDate": "2024-02-16T15:37:57.683Z", "content": "stage 1: all data
\nstage 2: all data[kl<5.5]
\nWouldn't this strategy create leaks?
But how do you create splits?
\nlet's say you create splits in stage-1 and then in stage-2, you need to make sure both have the same sample that is (all samples of fold0 stage-1 must remain in all samples of fold 0 in stage-2, same for all other folds).
You are right, the validation set with the same distribution should be used in the second stage of training
", "rawMarkdown": "You are right, the validation set with the same distribution should be used in the second stage of training" } ] }, { "id": 2655480, "postDate": "2024-02-16T22:56:32.917Z", "content": "I discuss my training framework in this comment. Overall, there is no leakage. However, training and validation are based on filtered data in stage 2, it is better to use the same validation data in both stages. I will fix it.
", "rawMarkdown": "I discuss my training framework in this [comment](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461#2655472). Overall, there is no leakage. However, training and validation are based on filtered data in stage 2, it is better to use the same validation data in both stages. I will fix it." } ] }, { "id": 2655613, "postDate": "2024-02-17T04:53:28.110Z", "content": "My experience is that the Public LB score for one-stage training is 0.43, while the Public LB score for two-stage training is 0.36, which is a strange result.
", "rawMarkdown": "My experience is that the Public LB score for one-stage training is 0.43, while the Public LB score for two-stage training is 0.36, which is a strange result.", "votes": 5, "replies": [ { "id": 2655625, "postDate": "2024-02-17T05:10:10.710Z", "content": "Is it possible that it is caused by the evaluation metric (KL divergence), which is sensitive to extreme predictions? Experiments seem to imply that smooth prediction results have more obvious advantages.
", "rawMarkdown": "Is it possible that it is caused by the evaluation metric (KL divergence), which is sensitive to extreme predictions? Experiments seem to imply that smooth prediction results have more obvious advantages.", "votes": 3, "replies": [ { "id": 2655633, "postDate": "2024-02-17T05:21:59.050Z", "content": "Maybe hard samples like your title describes are more important in this competition metrics. However, the distribution of Private and Public may lead to large differences in the final results.
", "rawMarkdown": "Maybe hard samples like your title describes are more important in this competition metrics. However, the distribution of Private and Public may lead to large differences in the final results.", "votes": 3 }, { "id": 2655654, "postDate": "2024-02-17T05:45:47.047Z", "content": "This is indeed a crucial issue.
\nHave you tried using the same validation data in two stages, or is it similar to my implementation method (validation of the 2nd stage is filtered data)? I think using the same validation data can better evaluate the importance of hard samples.
I'm now validating with validation data from the respective stages.
", "rawMarkdown": "I'm now validating with validation data from the respective stages." }, { "id": 2655851, "postDate": "2024-02-17T08:44:12.513Z", "content": "I update the result with the same validation data. The result is indeed quite surprising. The improvement of CV is much smaller than LB.
", "rawMarkdown": "I update the result with the same validation data. The result is indeed quite surprising. The improvement of CV is much smaller than LB.", "votes": 1 }, { "id": 2655895, "postDate": "2024-02-17T09:08:22.637Z", "content": "I tried the same configuration as you, stage1: train on all data, stage2: train and verify on data with kl<5.5, this did not improve my single-mode lb
", "rawMarkdown": "I tried the same configuration as you, stage1: train on all data, stage2: train and verify on data with kl<5.5, this did not improve my single-mode lb" }, { "id": 2656689, "postDate": "2024-02-18T00:14:31.390Z", "content": "@gentlezdh have you tried to lower the learning rate?
\nI tried the Two stages on - kaggle specs only - model, but I've reset the learning rate for the stages, CV and LB got worst.
\nMaybe the learning rate for the second stage should be much lower.
I conducted more experiments based on different backbones (effnetB0/1/2, resnet18/34) and training configs. Two-stage training seems to have limitations. The efficiency is limited for baselines with LB<=0.39 (same or worse). But for baselines with LB>0.41, there is an improvement in most cases.
", "rawMarkdown": "I conducted more experiments based on different backbones (effnetB0/1/2, resnet18/34) and training configs. Two-stage training seems to have limitations. The efficiency is limited for baselines with LB<=0.39 (same or worse). But for baselines with LB>0.41, there is an improvement in most cases.", "votes": 3 } ] } ] }, { "id": 2670651, "postDate": "2024-02-27T03:39:06.027Z", "content": "Nice work!!
\nsingle-stage CV (first stage) [0.485] LB [0.34]
\ntwo-stage CV (first stage) [0.485] CV (second stage) [0.325] LB [0.32]
Its result is Single Models? or ensemble?
", "rawMarkdown": "Its result is Single Models? or ensemble?", "replies": [ { "id": 2670727, "postDate": "2024-02-27T05:01:43.800Z", "content": "ensemble spe + raweeg
", "rawMarkdown": "ensemble spe + raweeg", "votes": 3 } ] } ] }, { "id": 2657511, "postDate": "2024-02-18T14:52:01.500Z", "content": "I have to say that I am impressed, it worked!!!
\nI managed to get the - kaggle only - model score from [CV 0.62 – LB 0.47] to [CV 0.6365 – LB 0.43], CV got worst, but LB got a lot better. I am planning on retraining the other model types then rensemble.
\nI released the new version of the code with LB 0.36 here.
@nartaa Good job! This really helps me a lot.
", "rawMarkdown": "@nartaa Good job! This really helps me a lot.", "votes": 1, "replies": [ { "id": 2657893, "postDate": "2024-02-18T20:16:01.680Z", "content": "@gentlezdh Thank you! You can check out the code from here
", "rawMarkdown": "@gentlezdh Thank you! You can check out the code from [here](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469666)", "votes": 1 }, { "id": 2658135, "postDate": "2024-02-19T01:59:14.270Z", "content": "I'm glad this method works for you! The code shared by you is insightful. I think this method is worth exploring further.
", "rawMarkdown": "I'm glad this method works for you! The code shared by you is insightful. I think this method is worth exploring further.", "votes": 1 } ] } ] }, { "id": 2655472, "postDate": "2024-02-16T22:42:58.260Z", "content": "Hi. The training loop is as follows (I use Hydra to manage hyper-parameters):
\n fold_id (n_fold):\n current_cfg = cfg.copy()\n current_cfg[][] = fold_id\n\n \n current_cfg[][][] = \n best_model_score_1, best_model_path_1 = training(current_cfg)\n\n \n current_cfg[][][] = \n best_model_score_2, _ = training(current_cfg)\nAll seeds have also been fixed. Filtering is performed after tag training and validation.
\n ():\n ....\n total_df = read_train_metadata(data_root) \n split_train_validation(total_df, **self.data_conf[]) \n self.optim_conf[][] == :\n .... \n ....\nSo I am certain that there is no leakage.
\nThe reason for the small CV in the second stage is that it was only trained and tested on the filtered data (as with motivation, samples with peaked distributions are more sensitive in evaluation). Therefore, there are some minor flaws: the data used for validation in the second stage is a subset (KL Loss < 5.5) of the data used in the first stage.
\nFor my training framework, this is a fast implementation because it only requires filtering after tagging (a few lines of code). CVs of different stages don't have comparability (at the same stage, it is possible to).
\nThank you for pointing out this issue. I will fix it and update my experimental results.
", "rawMarkdown": "Hi. The training loop is as follows (I use Hydra to manage hyper-parameters): \n```python\nfor fold_id in range(n_fold):\n current_cfg = cfg.copy()\n current_cfg['data_conf']['val_fold_id'] = fold_id\n\n # stage I\n current_cfg['optim_conf']['train']['stage'] = 1\n best_model_score_1, best_model_path_1 = training(current_cfg)\n\n # stage II\n current_cfg['optim_conf']['train']['stage'] = 2\n best_model_score_2, _ = training(current_cfg)\n```\nAll seeds have also been fixed. Filtering is performed after tag training and validation. \n```python\ndef training():\n ....\n total_df = read_train_metadata(data_root) # load metadata\n split_train_validation(total_df, **self.data_conf['fold']) # tag training and validation\n if self.optim_conf['train']['stage'] == 2:\n .... # filter\n ....\n```\nSo I am certain that there is no leakage.\n\nThe reason for the small CV in the second stage is that it was only trained and tested on the filtered data (as with motivation, samples with peaked distributions are more sensitive in evaluation). Therefore, there are some minor flaws: the data used for validation in the second stage is a **subset** (KL Loss < 5.5) of the data used in the first stage.\n\nFor my training framework, this is a fast implementation because it only requires filtering after tagging (a few lines of code). CVs of different stages don't have comparability (at the same stage, it is possible to).\n\nThank you for pointing out this issue. I will fix it and update my experimental results.", "votes": 1 }, { "id": 2703636, "postDate": "2024-03-18T10:14:18.380Z", "content": "Hello @zijiangyang1116
\nShouldn't the title be Soft samples are more important , since we are training the second stage on soft samples, I m assuming by a hard sample you mean - the ones closer to a degenerate distribution.
\nAnd the soft samples are the ones closer to a uniform distribution.
I think he meant hard sample in the sense that the experts find it hard to classify. I.e. they are unsure which option it is and the distribution is closer to uniform
", "rawMarkdown": "I think he meant hard sample in the sense that the experts find it hard to classify. I.e. they are unsure which option it is and the distribution is closer to uniform" } ] }, { "id": 2665331, "postDate": "2024-02-23T15:42:47.610Z", "content": "Thanks for sharing. I have two questions.
\nAt first, it was just intuition. Because threshold=5.5 can retain enough samples for training while excluding extreme samples. I also conducted many experiments later, and threshold=5.5 is an optimal choice in most cases.
I train with all data in the 1st stage and then with filtered data (the 2nd stage is the same as Sean R.B. Bearden's method).
Thanks. Filtering train_df by kl_div worked for me
", "rawMarkdown": "Thanks. Filtering train_df by kl_div worked for me", "votes": 1 } ] } ] }, { "id": 2655261, "postDate": "2024-02-16T19:05:05.063Z", "content": "did you just train and validated with filtered record in stage 2!! plz sure once again
", "rawMarkdown": "did you just train and validated with filtered record in stage 2!! plz sure once again", "replies": [ { "id": 2655473, "postDate": "2024-02-16T22:47:43.877Z", "content": "Yes, training and validation are based on filtered data. I discuss my training framework in this comment. You can refer to it.
", "rawMarkdown": "Yes, training and validation are based on filtered data. I discuss my training framework in this [comment](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461#2655472). You can refer to it." } ] }, { "id": 2663269, "postDate": "2024-02-22T12:01:37.273Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2654899, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2024-02-16T14:41:04.903000", "content": "did you just train and validated with filtered record in stage 2 ? I would have thought that you would filter out the <5.5 KLLoss records for train set and continue with the same validation set as stage1 ?
", "votes": 5, "replies": [ { "id": 2655477, "author_name": "Koolo", "author_url": "", "post_date": "2024-02-16T22:51:04.567000", "content": "Training and validation are based on filtered data in the 2nd stage. I discuss my training framework in this comment. And you are right, using the same validation set is a better method, I will fix it and update CVs.
", "votes": 1, "replies": [] } ] }, { "id": 2654920, "author_name": "Priyanshu Chaudhary", "author_url": "", "post_date": "2024-02-16T15:04:58.210000", "content": "Do you make sure you are using same split in stage 1 and stage 2? Looks to me that choosing samples with KL<5.5 is causing mild leakage to your strategy, as CV is too optimal.
", "votes": 6, "replies": [ { "id": 2654949, "author_name": "", "author_url": "", "post_date": "2024-02-16T15:37:23.423000", "content": "", "votes": 0, "replies": [] }, { "id": 2654953, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2024-02-16T15:37:57.683000", "content": "stage 1: all data
\nstage 2: all data[kl<5.5]
\nWouldn't this strategy create leaks?
But how do you create splits?
\nlet's say you create splits in stage-1 and then in stage-2, you need to make sure both have the same sample that is (all samples of fold0 stage-1 must remain in all samples of fold 0 in stage-2, same for all other folds).
You are right, the validation set with the same distribution should be used in the second stage of training
", "votes": 0, "replies": [] } ] }, { "id": 2655480, "author_name": "Koolo", "author_url": "", "post_date": "2024-02-16T22:56:32.917000", "content": "I discuss my training framework in this comment. Overall, there is no leakage. However, training and validation are based on filtered data in stage 2, it is better to use the same validation data in both stages. I will fix it.
", "votes": 0, "replies": [] } ] }, { "id": 2655613, "author_name": "ynhuhu", "author_url": "", "post_date": "2024-02-17T04:53:28.110000", "content": "My experience is that the Public LB score for one-stage training is 0.43, while the Public LB score for two-stage training is 0.36, which is a strange result.
", "votes": 5, "replies": [ { "id": 2655625, "author_name": "Koolo", "author_url": "", "post_date": "2024-02-17T05:10:10.710000", "content": "Is it possible that it is caused by the evaluation metric (KL divergence), which is sensitive to extreme predictions? Experiments seem to imply that smooth prediction results have more obvious advantages.
", "votes": 3, "replies": [ { "id": 2655633, "author_name": "ynhuhu", "author_url": "", "post_date": "2024-02-17T05:21:59.050000", "content": "Maybe hard samples like your title describes are more important in this competition metrics. However, the distribution of Private and Public may lead to large differences in the final results.
", "votes": 3, "replies": [] }, { "id": 2655654, "author_name": "Koolo", "author_url": "", "post_date": "2024-02-17T05:45:47.047000", "content": "This is indeed a crucial issue.
\nHave you tried using the same validation data in two stages, or is it similar to my implementation method (validation of the 2nd stage is filtered data)? I think using the same validation data can better evaluate the importance of hard samples.
I'm now validating with validation data from the respective stages.
", "votes": 0, "replies": [] }, { "id": 2655851, "author_name": "Koolo", "author_url": "", "post_date": "2024-02-17T08:44:12.513000", "content": "I update the result with the same validation data. The result is indeed quite surprising. The improvement of CV is much smaller than LB.
", "votes": 1, "replies": [] }, { "id": 2655895, "author_name": "Donghui Zhang", "author_url": "", "post_date": "2024-02-17T09:08:22.637000", "content": "I tried the same configuration as you, stage1: train on all data, stage2: train and verify on data with kl<5.5, this did not improve my single-mode lb
", "votes": 0, "replies": [] }, { "id": 2656689, "author_name": "Danial Zakaria", "author_url": "", "post_date": "2024-02-18T00:14:31.390000", "content": "@gentlezdh have you tried to lower the learning rate?
\nI tried the Two stages on - kaggle specs only - model, but I've reset the learning rate for the stages, CV and LB got worst.
\nMaybe the learning rate for the second stage should be much lower.
I conducted more experiments based on different backbones (effnetB0/1/2, resnet18/34) and training configs. Two-stage training seems to have limitations. The efficiency is limited for baselines with LB<=0.39 (same or worse). But for baselines with LB>0.41, there is an improvement in most cases.
", "votes": 3, "replies": [] } ] } ] }, { "id": 2670651, "author_name": "Yuri Sun", "author_url": "", "post_date": "2024-02-27T03:39:06.027000", "content": "Nice work!!
\nsingle-stage CV (first stage) [0.485] LB [0.34]
\ntwo-stage CV (first stage) [0.485] CV (second stage) [0.325] LB [0.32]
Its result is Single Models? or ensemble?
", "votes": 0, "replies": [ { "id": 2670727, "author_name": "Yuri Sun", "author_url": "", "post_date": "2024-02-27T05:01:43.800000", "content": "ensemble spe + raweeg
", "votes": 3, "replies": [] } ] } ] }, { "id": 2657511, "author_name": "Danial Zakaria", "author_url": "", "post_date": "2024-02-18T14:52:01.500000", "content": "I have to say that I am impressed, it worked!!!
\nI managed to get the - kaggle only - model score from [CV 0.62 – LB 0.47] to [CV 0.6365 – LB 0.43], CV got worst, but LB got a lot better. I am planning on retraining the other model types then rensemble.
\nI released the new version of the code with LB 0.36 here.
@nartaa Good job! This really helps me a lot.
", "votes": 1, "replies": [ { "id": 2657893, "author_name": "Danial Zakaria", "author_url": "", "post_date": "2024-02-18T20:16:01.680000", "content": "@gentlezdh Thank you! You can check out the code from here
", "votes": 1, "replies": [] }, { "id": 2658135, "author_name": "Koolo", "author_url": "", "post_date": "2024-02-19T01:59:14.270000", "content": "I'm glad this method works for you! The code shared by you is insightful. I think this method is worth exploring further.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2655472, "author_name": "Koolo", "author_url": "", "post_date": "2024-02-16T22:42:58.260000", "content": "Hi. The training loop is as follows (I use Hydra to manage hyper-parameters):
\n fold_id (n_fold):\n current_cfg = cfg.copy()\n current_cfg[][] = fold_id\n\n \n current_cfg[][][] = \n best_model_score_1, best_model_path_1 = training(current_cfg)\n\n \n current_cfg[][][] = \n best_model_score_2, _ = training(current_cfg)\nAll seeds have also been fixed. Filtering is performed after tag training and validation.
\n ():\n ....\n total_df = read_train_metadata(data_root) \n split_train_validation(total_df, **self.data_conf[]) \n self.optim_conf[][] == :\n .... \n ....\nSo I am certain that there is no leakage.
\nThe reason for the small CV in the second stage is that it was only trained and tested on the filtered data (as with motivation, samples with peaked distributions are more sensitive in evaluation). Therefore, there are some minor flaws: the data used for validation in the second stage is a subset (KL Loss < 5.5) of the data used in the first stage.
\nFor my training framework, this is a fast implementation because it only requires filtering after tagging (a few lines of code). CVs of different stages don't have comparability (at the same stage, it is possible to).
\nThank you for pointing out this issue. I will fix it and update my experimental results.
", "votes": 1, "replies": [] }, { "id": 2703636, "author_name": "Danial Zakaria", "author_url": "", "post_date": "2024-03-18T10:14:18.380000", "content": "Hello @zijiangyang1116
\nShouldn't the title be Soft samples are more important , since we are training the second stage on soft samples, I m assuming by a hard sample you mean - the ones closer to a degenerate distribution.
\nAnd the soft samples are the ones closer to a uniform distribution.
I think he meant hard sample in the sense that the experts find it hard to classify. I.e. they are unsure which option it is and the distribution is closer to uniform
", "votes": 0, "replies": [] } ] }, { "id": 2665331, "author_name": "Aurora_blue", "author_url": "", "post_date": "2024-02-23T15:42:47.610000", "content": "Thanks for sharing. I have two questions.
\nAt first, it was just intuition. Because threshold=5.5 can retain enough samples for training while excluding extreme samples. I also conducted many experiments later, and threshold=5.5 is an optimal choice in most cases.
I train with all data in the 1st stage and then with filtered data (the 2nd stage is the same as Sean R.B. Bearden's method).
Thanks. Filtering train_df by kl_div worked for me
", "votes": 1, "replies": [] } ] } ] }, { "id": 2655261, "author_name": "Tanishq dublish", "author_url": "", "post_date": "2024-02-16T19:05:05.063000", "content": "did you just train and validated with filtered record in stage 2!! plz sure once again
", "votes": 0, "replies": [ { "id": 2655473, "author_name": "Koolo", "author_url": "", "post_date": "2024-02-16T22:47:43.877000", "content": "Yes, training and validation are based on filtered data. I discuss my training framework in this comment. You can refer to it.
", "votes": 0, "replies": [] } ] }, { "id": 2663269, "author_name": "", "author_url": "", "post_date": "2024-02-22T12:01:37.273000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2654532": "[@Sean R.B. Bearden](https://www.kaggle.com/seanbearden) provides a great idea in his insightful [notebook](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135). He proposes a two-stage training method to address the KL-divergence issue caused by samples with 1-7 votes. \n\n- **Motivation**: samples with peaked distributions lead models to predict with exaggerated confidence, resulting in worse performance.\n- **Method**: two-stage training method (the first stage only uses samples with 1-7 votes, then samples with 10-28 votes are used to fine-tune models).\n\nHowever, there is no clear correlation between the number of votes and the distribution of votes. As evidence, the scatter diagram of total votes - KL Loss (KL Loss between votes and uniform distribution) is as follows.\n\n\n\nThe calculation of KL Loss is as follows:\n```python\ny_data = data[label_cols].values\ny_data = y_data / y_data.sum(axis=1, keepdims=True)\ndata[label_cols] = y_data\nlabels = data[['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']].values + 1e-5\n\n# compute kl-loss with uniform distribution by pytorch\ndata['kl'] = torch.nn.functional.kl_div(\n torch.log(torch.tensor(labels)),\n torch.tensor([1 / 6] * 6),\n reduction='none'\n).sum(dim=1).numpy()\nprint('min:', data['kl'].min(), 'max:', data['kl'].max())\n```\n\nThere is an important observation: samples with peaked distributions (KL Loss > 7) present in both groups\n\nTherefore, according to the motivation and the observation, I suggest dividing train data according to KL Loss. My experimental configs are as follows:\n- EffNetB2 (based on [@alejopaullier](https://www.kaggle.com/alejopaullier)'s [notebook](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train/notebook))\n- First stage: training with all data\n- Second stage: training with samples (KL Loss < 5.5).\n\n| Method | CV (first stage) | CV (second stage) | LB |\n| --- | --- | --- | --- |\n| single-stage | 0.60 | - | 0.41 |\n|two-stage (KL Loss)|0.60|0.33|**0.37**|\n|two-stage (total votes)|0.60|0.38|0.41|\n\n---\nPlease note that CVs of different stages cannot be compared. Because the training and validation of the 2nd stage are based on filtered data. This is a fast implementation. For a more detailed discussion, please refer to [my comment](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461#2655472). I will fix this issue and update the results.\n\n---\nI fix the issue and the training framework is as follows:\n```python\ndef training():\n ....\n total_df = read_train_metadata(data_root) # load metadata\n split_train_validation(total_df, **self.data_conf['fold']) # tag training and validation\n\n # create train_df & val_df\n train_df = total_df[total_df['fold'] != self.data_conf['val_fold_id']].reset_index(drop=True)\n val_df = total_df[total_df['fold'] == self.data_conf['val_fold_id']].reset_index(drop=True)\n\n if self.optim_conf['train']['stage'] == 2:\n .... # filter only on train_df\n ....\n```\n\nAs long as the tagging of training and validation is the same (this can be achieved by fixing random seeds), these two stages use the same validation data. The result are as follows:\n\n| Method | CV (first stage) | CV (second stage) |\n| --- | --- | --- |\n|two-stage (KL Loss)|0.59836|0.58767|\n\nCV has been improved.\n\nMore detailed result:\n\n| stage | fold-1 | fold-2 | fold-3 | fold-4 | fold-5 |\n| --- | --- | --- | --- | --- | --- |\n|1st|0.63056|0.55998|0.53577|**0.58305**|0.68245|\n|2nd|**0.60282**|**0.55192**|**0.53050**|0.60580|**0.64730**|\n\n\n**This result is surprising**. Although there is an improvement in CV (0.60->0.59), the magnitude of the improvement is much smaller than that of LB (0.41->0.37). I'm not sure if there's any special reason.", "2654899": "did you just train and validated with filtered record in stage 2 ? I would have thought that you would filter out the <5.5 KLLoss records for train set and continue with the same validation set as stage1 ? ", "2654920": "Do you make sure you are using same split in stage 1 and stage 2? Looks to me that choosing samples with KL<5.5 is causing mild leakage to your strategy, as CV is too optimal.", "2655613": "My experience is that the Public LB score for one-stage training is 0.43, while the Public LB score for two-stage training is 0.36, which is a strange result.", "2670651": "Nice work!!\nsingle-stage \tCV (first stage) [0.485] LB [0.34]\ntwo-stage CV (first stage) [0.485] CV (second stage) [0.325]\tLB [0.32]\n", "2657511": "I have to say that I am impressed, it worked!!!\nI managed to get the - kaggle only - model score from [CV 0.62 – LB 0.47] to [CV 0.6365 – LB 0.43], CV got worst, but LB got a lot better. I am planning on retraining the other model types then rensemble.\nI released the new version of the code with LB 0.36 [here](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469666).", "2655472": "Hi. The training loop is as follows (I use Hydra to manage hyper-parameters): \n```python\nfor fold_id in range(n_fold):\n current_cfg = cfg.copy()\n current_cfg['data_conf']['val_fold_id'] = fold_id\n\n # stage I\n current_cfg['optim_conf']['train']['stage'] = 1\n best_model_score_1, best_model_path_1 = training(current_cfg)\n\n # stage II\n current_cfg['optim_conf']['train']['stage'] = 2\n best_model_score_2, _ = training(current_cfg)\n```\nAll seeds have also been fixed. Filtering is performed after tag training and validation. \n```python\ndef training():\n ....\n total_df = read_train_metadata(data_root) # load metadata\n split_train_validation(total_df, **self.data_conf['fold']) # tag training and validation\n if self.optim_conf['train']['stage'] == 2:\n .... # filter\n ....\n```\nSo I am certain that there is no leakage.\n\nThe reason for the small CV in the second stage is that it was only trained and tested on the filtered data (as with motivation, samples with peaked distributions are more sensitive in evaluation). Therefore, there are some minor flaws: the data used for validation in the second stage is a **subset** (KL Loss < 5.5) of the data used in the first stage.\n\nFor my training framework, this is a fast implementation because it only requires filtering after tagging (a few lines of code). CVs of different stages don't have comparability (at the same stage, it is possible to).\n\nThank you for pointing out this issue. I will fix it and update my experimental results.", "2703636": "Hello @zijiangyang1116\nShouldn't the title be **Soft samples are more important** , since we are training the second stage on soft samples, I m assuming by a hard sample you mean - the ones closer to a degenerate distribution.\nAnd the soft samples are the ones closer to a uniform distribution.", "2665331": "Thanks for sharing. I have two questions.\n1. How did you find threshold=5.5 ? By a lot of experiment?\n2. About two-stage (total votes) in result table, how did you train model ? Did you train in the same way as Sean R.B. Bearden ? or Did you train with all data and then filter like total_evaluators > threshold ?", "2655261": "did you just train and validated with filtered record in stage 2!! plz sure once again", "2663269": "" } }