\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4495635%2F6b9d655b354f3550f0123e512822a262%2Fhms.png?generation=1708084757579739&alt=media\")
Greetings, Everyone,
\ntrain[TARGETS] = K*train[TARGETS] + 0.166666667 # K=1
Introducing a single line of code has notably reduced the Cross-Validation (CV) score, aligning it more closely with the Leaderboard (LB) scores. It's worth noting that the LB score itself remains unchanged. This observation suggests that during model initialization, a uniform probability distribution were predicted by the model at the begining, because it predicts classes poorly but equally, then eventually the model reduces the loss and learns the probability distributions of the classes.
\nBy incorporating an offset of 0.16667 (1/6) across all classes at the target probability distributions, the CV score decreases, approaching the learned distribution. The lack of change in the LB score potentially supports the hypothesis that this adjustment brings the learned probability closer to the target probability distribution. Consequently, the gap diminishes between the training loss and validation loss, while the underlying learned probability remains consistent. (further should be investigated)
\nHere, the second row is the probability distribution of the target labels with the offset
\n
Side Note: Adjusting the Coefficient K < 1 for Voted Weighting: Possibly optimizing Model Confidence for Improved Leaderboard Scores
\nLet me know your thoughts.
Greetings, Everyone,
\ntrain[TARGETS] = K*train[TARGETS] + 0.166666667 # K=1
Introducing a single line of code has notably reduced the Cross-Validation (CV) score, aligning it more closely with the Leaderboard (LB) scores. It's worth noting that the LB score itself remains unchanged. This observation suggests that during model initialization, a uniform probability distribution were predicted by the model at the begining, because it predicts classes poorly but equally, then eventually the model reduces the loss and learns the probability distributions of the classes.
\nBy incorporating an offset of 0.16667 (1/6) across all classes at the target probability distributions, the CV score decreases, approaching the learned distribution. The lack of change in the LB score potentially supports the hypothesis that this adjustment brings the learned probability closer to the target probability distribution. Consequently, the gap diminishes between the training loss and validation loss, while the underlying learned probability remains consistent. (further should be investigated)
\nHere, the second row is the probability distribution of the target labels with the offset
\n
Side Note: Adjusting the Coefficient K < 1 for Voted Weighting: Possibly optimizing Model Confidence for Improved Leaderboard Scores
\nLet me know your thoughts.
When you change this , do all the labels summation for each row still remain 1 ?
", "rawMarkdown": "When you change this , do all the labels summation for each row still remain 1 ?", "votes": null }, { "id": "2654951", "postDate": "02/16/2024 15:37:38", "content": "Yes, The summation of each row still remains 1.
\ntrain[TARGETS] = 1*train[TARGETS] + 0.166666667
\ntrain[TARGETS] = train[TARGETS]/train[TARGETS].values.sum(axis=1,keepdims=True)
just to confirm, the models are trained with the new targets right ? And not that just cv is recalculated with new targets. Although it might be worth looking at the second case too
", "rawMarkdown": "just to confirm, the models are trained with the new targets right ? And not that just cv is recalculated with new targets. Although it might be worth looking at the second case too", "votes": null }, { "id": "2656010", "postDate": "02/17/2024 11:44:53", "content": "Yes, the models are trained with the new targets. But I think the second case should give consistent results).
\nBasically, the targets are started off with a uniform distribution i.e. ([0.16, 0.16, 0.16, 0.16, 0.16, 0.16]), then this distribution is reshaped with the provided votes for all classes, we can control how much the votes effect the distribution with the coefficient K.
", "rawMarkdown": "Yes, the models are trained with the new targets. But I think the second case should give consistent results).\n\nBasically, the targets are started off with a uniform distribution i.e. ([0.16, 0.16, 0.16, 0.16, 0.16, 0.16]), then this distribution is reshaped with the provided votes for all classes, we can control how much the votes effect the distribution with the coefficient K.", "votes": null }, { "id": "2656020", "postDate": "02/17/2024 11:50:39", "content": "What would you arbitrarily change the labels of validation set? The change in score should be reflected in the original validation, otherwise you're going to end up with unreliable results.
", "rawMarkdown": "What would you arbitrarily change the labels of validation set? The change in score should be reflected in the original validation, otherwise you're going to end up with unreliable results.", "votes": null }, { "id": "2656043", "postDate": "02/17/2024 12:05:14", "content": "The target labels are processed the same for all the training set once grouped by eeg_id (Chris's approach), that also includes validation sets. Let me know if I didn't answer your question
", "rawMarkdown": "The target labels are processed the same for all the training set once grouped by eeg_id (Chris's approach), that also includes validation sets. Let me know if I didn't answer your question", "votes": null }, { "id": "2656046", "postDate": "02/17/2024 12:15:22", "content": "just keep in mind that in order to have reliable CV results you need to do this after the train/val split only in train targets and keep valid untouched. So CV score is calculated on the original targets
\nThink that in test time (submission) you won't have the test[TARGETS] to apply the same transformation.
\nI'm guessing this is what also other guys above are trying to say.
", "rawMarkdown": "just keep in mind that in order to have reliable CV results you need to do this after the train/val split only in train targets and keep valid untouched. So CV score is calculated on the original targets \n\nThink that in test time (submission) you won't have the test[TARGETS] to apply the same transformation.\n\nI'm guessing this is what also other guys above are trying to say.", "votes": null }, { "id": "2656140", "postDate": "02/17/2024 13:43:30", "content": "I think this can go either way, I am seeing metric correlation either way.
\nIf we operate on the original targets for CV, the scores will be more familliar to us, but that will add additional complexity to the code (we would have to account for that in the data generator).
I agree with slime. I also think that more noise tends to less total votes for each patient, because it may be hard to diagose/predict and we should keep the original targets when calculating cv. Noise is also important features for these hard samples. keep hard samples/edge cases hard/edge, and don't change them.
", "rawMarkdown": "I agree with slime. I also think that more noise tends to less total votes for each patient, because it may be hard to diagose/predict and we should keep the original targets when calculating cv. Noise is also important features for these hard samples. keep hard samples/edge cases hard/edge, and don't change them.", "votes": null } ], "comments": [ { "id": 2654911, "author_name": "phoenix9032", "author_url": "", "post_date": "02/16/2024 14:50:33", "content": "When you change this , do all the labels summation for each row still remain 1 ?
", "votes": null, "replies": [ { "id": 2654951, "author_name": "nartaa", "author_url": "", "post_date": "02/16/2024 15:37:38", "content": "Yes, The summation of each row still remains 1.
\ntrain[TARGETS] = 1*train[TARGETS] + 0.166666667
\ntrain[TARGETS] = train[TARGETS]/train[TARGETS].values.sum(axis=1,keepdims=True)
just keep in mind that in order to have reliable CV results you need to do this after the train/val split only in train targets and keep valid untouched. So CV score is calculated on the original targets
\nThink that in test time (submission) you won't have the test[TARGETS] to apply the same transformation.
\nI'm guessing this is what also other guys above are trying to say.
", "votes": null, "replies": [ { "id": 2656140, "author_name": "nartaa", "author_url": "", "post_date": "02/17/2024 13:43:30", "content": "I think this can go either way, I am seeing metric correlation either way.
\nIf we operate on the original targets for CV, the scores will be more familliar to us, but that will add additional complexity to the code (we would have to account for that in the data generator).
just to confirm, the models are trained with the new targets right ? And not that just cv is recalculated with new targets. Although it might be worth looking at the second case too
", "votes": null, "replies": [ { "id": 2656010, "author_name": "nartaa", "author_url": "", "post_date": "02/17/2024 11:44:53", "content": "Yes, the models are trained with the new targets. But I think the second case should give consistent results).
\nBasically, the targets are started off with a uniform distribution i.e. ([0.16, 0.16, 0.16, 0.16, 0.16, 0.16]), then this distribution is reshaped with the provided votes for all classes, we can control how much the votes effect the distribution with the coefficient K.
", "votes": null, "replies": [] } ] }, { "id": 2656020, "author_name": "martynoveduard", "author_url": "", "post_date": "02/17/2024 11:50:39", "content": "What would you arbitrarily change the labels of validation set? The change in score should be reflected in the original validation, otherwise you're going to end up with unreliable results.
", "votes": null, "replies": [ { "id": 2656043, "author_name": "nartaa", "author_url": "", "post_date": "02/17/2024 12:05:14", "content": "The target labels are processed the same for all the training set once grouped by eeg_id (Chris's approach), that also includes validation sets. Let me know if I didn't answer your question
", "votes": null, "replies": [] }, { "id": 2656775, "author_name": "sweetyheehee", "author_url": "", "post_date": "02/18/2024 02:27:04", "content": "I agree with slime. I also think that more noise tends to less total votes for each patient, because it may be hard to diagose/predict and we should keep the original targets when calculating cv. Noise is also important features for these hard samples. keep hard samples/edge cases hard/edge, and don't change them.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2654758": "Greetings, Everyone,\n\n`train[TARGETS] = K*train[TARGETS] + 0.166666667 # K=1`\n\nIntroducing a single line of code has notably reduced the Cross-Validation (CV) score, aligning it more closely with the Leaderboard (LB) scores. It's worth noting that the LB score itself remains unchanged. This observation suggests that during model initialization, a uniform probability distribution were predicted by the model at the begining, because it predicts classes poorly but equally, then eventually the model reduces the loss and learns the probability distributions of the classes.\n\nBy incorporating an offset of 0.16667 (1/6) across all classes at the target probability distributions, the CV score decreases, approaching the learned distribution. The lack of change in the LB score potentially supports the hypothesis that this adjustment brings the learned probability closer to the target probability distribution. Consequently, the gap diminishes between the training loss and validation loss, while the underlying learned probability remains consistent. (further should be investigated)\n\nHere, the second row is the probability distribution of the target labels with the offset\n\n\n*Side Note: Adjusting the Coefficient K < 1 for Voted Weighting: Possibly optimizing Model Confidence for Improved Leaderboard Scores*\nLet me know your thoughts.", "2654911": "When you change this , do all the labels summation for each row still remain 1 ?", "2654951": "Yes, The summation of each row still remains 1.\n\ntrain[TARGETS] = 1*train[TARGETS] + 0.166666667\ntrain[TARGETS] = train[TARGETS]/train[TARGETS].values.sum(axis=1,keepdims=True)", "2655490": "just to confirm, the models are trained with the new targets right ? And not that just cv is recalculated with new targets. Although it might be worth looking at the second case too", "2656010": "Yes, the models are trained with the new targets. But I think the second case should give consistent results).\n\nBasically, the targets are started off with a uniform distribution i.e. ([0.16, 0.16, 0.16, 0.16, 0.16, 0.16]), then this distribution is reshaped with the provided votes for all classes, we can control how much the votes effect the distribution with the coefficient K.", "2656020": "What would you arbitrarily change the labels of validation set? The change in score should be reflected in the original validation, otherwise you're going to end up with unreliable results.", "2656043": "The target labels are processed the same for all the training set once grouped by eeg_id (Chris's approach), that also includes validation sets. Let me know if I didn't answer your question", "2656046": "just keep in mind that in order to have reliable CV results you need to do this after the train/val split only in train targets and keep valid untouched. So CV score is calculated on the original targets \n\nThink that in test time (submission) you won't have the test[TARGETS] to apply the same transformation.\n\nI'm guessing this is what also other guys above are trying to say.", "2656140": "I think this can go either way, I am seeing metric correlation either way.\nIf we operate on the original targets for CV, the scores will be more familliar to us, but that will add additional complexity to the code (we would have to account for that in the data generator).", "2656775": "I agree with slime. I also think that more noise tends to less total votes for each patient, because it may be hard to diagose/predict and we should keep the original targets when calculating cv. Noise is also important features for these hard samples. keep hard samples/edge cases hard/edge, and don't change them." }, "source": "meta" }