![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10604629%2Fe806b9b43a4c8d4ce22673032f132e7e%2Fpicture_pc_ad440002b1e991af34e35e4785200043.webp?generation=1734348983215112&alt=media\")
We know from public notebook results that small differences in seed values(=42,2024,…) can yield very noisy scores.
\nOne of the reasons for this may be insufficient training data, but in my opinion, target (=sii) distribution of the test data is different from the training data is also an important factor.
Through several experiments, I found that the variance of QWK scores changes just by changing the distribution of sii (=0~3). This is due to the way QWK is calculated. In QWK calculations, the larger the difference between the predicted and correct labels, the more squared the penalty is given. (The figure below shows the penalties for each class.)
\n
This experiments suggests that even in a completely random environment, the standard deviation of QWK increases when there are many labels such as 0 and 2. For example, the higher the ratio of sii=2, the higher the random QWK variation.
\n| sii0/all=0.5, sii1/all=0.5-j, sii2/all=j | \nj | \nQWK's std | \n
|---|---|---|
| \n | 0.0 | \n0.02226 | \n
| \n | 0.1 | \n0.02727 | \n
| \n | 0.2 | \n0.02945 | \n
| \n | 0.3 | \n0.03056 | \n
| \n | 0.4 | \n0.03099 | \n
| \n | 0.5 | \n0.03120 | \n
\nIn other words, at least in the public test data, the dataset has many such labels(=0,2). For proof of this, just look at the 0.495 and 0.497 notebooks. The only difference between these two notebooks is that x0 (=initial value for optimization) in optim.minimize is changed from [0.5,1.5,2.5] to [0.5,1.49,2.5]. This means that only the threshold for allocation to each class has changed.
j = , thr = [ ]\n----> || Optimized QWK SCORE :: \nj = , thr = [ ]\n----> || Optimized QWK SCORE :: \nThe two values of thresholds are important. (You can ignore the last value, because the maximum predicted value in CV is about 2.00 at most, and there is no row that predicts sii=3.)
\nThe difference between these two thresholds shows that there are more rows that predict sii=0 than before.(because threshold[0] change from 0.588 to 0.600.) This means that the public test data contains more sii=0 than the predicted distribution in notebooks of 0.495.
\nIn other words, this means that the label distribution in test data is more biased toward 0 or 3 than in the training data, and when the above assumptions are combined, QWK with a larger variance can be obtained.
\nNote: However, even if we can predict the rough distribution of public test data, we can't predict the distribution of private test data. It can also be seen that simply changing threshold is simply overfitting public test data. But we are not sure it would give bad results on private test data.
\nIf it is close to the training data, it is advantageous to have a high CV, if it is close to the public test data, it is advantageous to have a high LB, and if it is neither, it is chaotic.
We know from public notebook results that small differences in seed values(=42,2024,…) can yield very noisy scores.
\nOne of the reasons for this may be insufficient training data, but in my opinion, target (=sii) distribution of the test data is different from the training data is also an important factor.
Through several experiments, I found that the variance of QWK scores changes just by changing the distribution of sii (=0~3). This is due to the way QWK is calculated. In QWK calculations, the larger the difference between the predicted and correct labels, the more squared the penalty is given. (The figure below shows the penalties for each class.)
\n
This experiments suggests that even in a completely random environment, the standard deviation of QWK increases when there are many labels such as 0 and 2. For example, the higher the ratio of sii=2, the higher the random QWK variation.
\n| sii0/all=0.5, sii1/all=0.5-j, sii2/all=j | \nj | \nQWK's std | \n
|---|---|---|
| \n | 0.0 | \n0.02226 | \n
| \n | 0.1 | \n0.02727 | \n
| \n | 0.2 | \n0.02945 | \n
| \n | 0.3 | \n0.03056 | \n
| \n | 0.4 | \n0.03099 | \n
| \n | 0.5 | \n0.03120 | \n
\nIn other words, at least in the public test data, the dataset has many such labels(=0,2). For proof of this, just look at the 0.495 and 0.497 notebooks. The only difference between these two notebooks is that x0 (=initial value for optimization) in optim.minimize is changed from [0.5,1.5,2.5] to [0.5,1.49,2.5]. This means that only the threshold for allocation to each class has changed.
j = , thr = [ ]\n----> || Optimized QWK SCORE :: \nj = , thr = [ ]\n----> || Optimized QWK SCORE :: \nThe two values of thresholds are important. (You can ignore the last value, because the maximum predicted value in CV is about 2.00 at most, and there is no row that predicts sii=3.)
\nThe difference between these two thresholds shows that there are more rows that predict sii=0 than before.(because threshold[0] change from 0.588 to 0.600.) This means that the public test data contains more sii=0 than the predicted distribution in notebooks of 0.495.
\nIn other words, this means that the label distribution in test data is more biased toward 0 or 3 than in the training data, and when the above assumptions are combined, QWK with a larger variance can be obtained.
\nNote: However, even if we can predict the rough distribution of public test data, we can't predict the distribution of private test data. It can also be seen that simply changing threshold is simply overfitting public test data. But we are not sure it would give bad results on private test data.
\nIf it is close to the training data, it is advantageous to have a high CV, if it is close to the public test data, it is advantageous to have a high LB, and if it is neither, it is chaotic.
Agreed, I did something similar in my initial experiments and have no major indicator of the distribution and if the private test data is close to the train/ public test set @minar44
\nI think the results will be chaotic given the nature of the data and the overall CV-LB score
In conclusion, get ready for the big lottery? 😍🤣
", "rawMarkdown": "In conclusion, get ready for the big lottery? 😍🤣", "votes": null }, { "id": "3073425", "postDate": "12/16/2024 14:03:08", "content": "Yes, this is a lottery @shlomoron
\nPeople winning golds here are one of the luckiest ones out there!
\"In other words, this means that the label distribution in test data is more biased toward 0 or 3 than in the training data\"
\nI don't think it necessarily has to mean that the distribution of sii=0 is higher. The thresholds depends on the data (not only the sii value). There could simply be a feature with a higher distribution than in the training df, which leads your model to a higher bias in general, so the thresholds move up aswell.
\nI haven't tested this, but I would assume that if you do the following:
\nSplit df into 3 parts with equal distribution of sii values.
\nTrain on one. Predict the other two. The optimal qwk for the predicted dfs will be different from each other, eventhough they have the same distribution of sii.
\nAnyway, we can't predict how the private LB looks anyway. We need alot of luck :)
Well, we can still maximize our chances of winning gold. If all are random, it's 0.0046, so with some robustness and avoiding pitfalls that exist in public NB, we can easily exceed the 0.01 chance for gold—and maybe much higher! For a lottery, it is really great odds.
", "rawMarkdown": "Well, we can still maximize our chances of winning gold. If all are random, it's 0.0046, so with some robustness and avoiding pitfalls that exist in public NB, we can easily exceed the 0.01 chance for gold—and maybe much higher! For a lottery, it is really great odds.", "votes": null }, { "id": "3073453", "postDate": "12/16/2024 14:18:10", "content": "I think there isn't much to do other than minimizing another metric on soft predictions and then make your bets.
", "rawMarkdown": "I think there isn't much to do other than minimizing another metric on soft predictions and then make your bets.", "votes": null }, { "id": "3073535", "postDate": "12/16/2024 15:36:39", "content": "\n\nThe two values of thresholds are important. (You can ignore the last value,
\n
The last value produces a local maximum, isn't it? The global maximum is obtained from a value around 1.5 if we look in the 0.495 notebook. What if we updated 0.497 notebook with these changes? In my experiments with NN, global maximums lead to a 0.01 improvement in CV and 0.005 in LB. But sometimes this improvement is even more if the first step optimization wasn't successful.
", "rawMarkdown": ">The two values of thresholds are important. (You can ignore the last value,\n\nThe last value produces a **local** maximum, isn't it? The **global** maximum is obtained from a value around 1.5 if we look in the 0.495 notebook. What if we updated 0.497 notebook with these changes? In my experiments with NN, global maximums lead to a 0.01 improvement in CV and 0.005 in LB. But sometimes this improvement is even more if the first step optimization wasn't successful.", "votes": null }, { "id": "3074128", "postDate": "12/17/2024 10:54:50", "content": "The two values and last value here are thresholds. Thresholds are used to divide each class into 0, 1, 2, and 3.
\nFor example, the last value of the 0.497 notebook threshold is 2.71…, but the maximum predicted value of each model CV is as follows, and there is no value that is classified as 3.
| model | \nglobal maximum | \n
|---|---|
| model_1 | \n1.730926 | \n
| model_2 | \n2.003144 | \n
| model_3 | \n1.926503 | \n
\nAlso, I checked your notebook, it seems that the way the threshold is determined is different from that of the 0.497 notebook. Following that approach, the threshold would be [0.46, 0.91, 1.499]. I don't know if this will work as I haven't tried it.
Although it is higher than the notebook predicted sii=0 of 0.495, it cannot be said that bias is higher than training data. Thank you for pointing it out.
", "rawMarkdown": "Although it is higher than the notebook predicted sii=0 of 0.495, it cannot be said that bias is higher than training data. Thank you for pointing it out.", "votes": null }, { "id": "3074294", "postDate": "12/17/2024 14:31:44", "content": "I understand what you are pointing out about thresholds; it's okay. My question is why the author of the 0.495 notebook presents such great insight about global maximums and doesn't incorporate the calculation of the thresholds in that way in the further pipeline. The author of the 0.497 notebook is missing this discovery as well. So both best public notebooks can be even better, I think.
", "rawMarkdown": "I understand what you are pointing out about thresholds; it's okay. My question is why the author of the 0.495 notebook presents such great insight about global maximums and doesn't incorporate the calculation of the thresholds in that way in the further pipeline. The author of the 0.497 notebook is missing this discovery as well. So both best public notebooks can be even better, I think.", "votes": null }, { "id": "3074303", "postDate": "12/17/2024 14:45:12", "content": "I haven't tried it yet, so I can't say anything about it, but I think that using global maximum method may be overfits training data.
\nSure, that method allows us to find a combination of thresholds that optimizes CV score, but it doesn't mean it's the optimal threshold for test data too.
\n\nusing global maximum method may be overfits training data.
\n
Is true also for local minimum.
\nYou can avoid CV overfit by further splitting: train, train_treshold and validation.
\nBut it's probably not worth it. Just submit and see if global minimum perform better than local minimum. Since it's only a small and very focused experiment you are not in danger of LB overfitting.
The noisy Quadratic Weighted Kappa (QWK) scores can be attributed to a fundamental mismatch between training and test data distributions, particularly in how the target labels (sii values 0-3) are distributed. The test data contains a higher proportion of sii=0 cases than what was predicted from training data patterns, as evidenced by the threshold shift from 0.588 to 0.600 between optimization attempts.
\nThe nature of QWK calculation itself contributes to this volatility, as it employs a squared penalty system for prediction errors. This is clearly demonstrated in the penalty matrix (ω_i,j), where predictions that deviate further from the true class incur substantially larger penalties. For instance, confusing class 3 with class 1 results in a severe penalty of 4.
\nThe sensitivity of QWK scores becomes particularly apparent in how the standard deviation increases with parameter j, especially when there are numerous labels of 0 and 2. This sensitivity is so pronounced that even minor variations in seed values (like choosing between 42 and 2024) can lead to markedly different scores. This high variance in performance metrics stems from the combined effect of distributional differences between training and test sets, QWK's inherent quadratic penalty structure, and potentially insufficient training data, making the model's performance highly dependent on both initialization conditions and the underlying data distribution.
", "rawMarkdown": "The noisy Quadratic Weighted Kappa (QWK) scores can be attributed to a fundamental mismatch between training and test data distributions, particularly in how the target labels (sii values 0-3) are distributed. The test data contains a higher proportion of sii=0 cases than what was predicted from training data patterns, as evidenced by the threshold shift from 0.588 to 0.600 between optimization attempts.\n\nThe nature of QWK calculation itself contributes to this volatility, as it employs a squared penalty system for prediction errors. This is clearly demonstrated in the penalty matrix (ω_i,j), where predictions that deviate further from the true class incur substantially larger penalties. For instance, confusing class 3 with class 1 results in a severe penalty of 4.\n\nThe sensitivity of QWK scores becomes particularly apparent in how the standard deviation increases with parameter j, especially when there are numerous labels of 0 and 2. This sensitivity is so pronounced that even minor variations in seed values (like choosing between 42 and 2024) can lead to markedly different scores. This high variance in performance metrics stems from the combined effect of distributional differences between training and test sets, QWK's inherent quadratic penalty structure, and potentially insufficient training data, making the model's performance highly dependent on both initialization conditions and the underlying data distribution.", "votes": null } ], "comments": [ { "id": 3073394, "author_name": "ravi20076", "author_url": "", "post_date": "12/16/2024 13:18:59", "content": "Agreed, I did something similar in my initial experiments and have no major indicator of the distribution and if the private test data is close to the train/ public test set @minar44
\nI think the results will be chaotic given the nature of the data and the overall CV-LB score
In conclusion, get ready for the big lottery? 😍🤣
", "votes": null, "replies": [ { "id": 3073425, "author_name": "ravi20076", "author_url": "", "post_date": "12/16/2024 14:03:08", "content": "Yes, this is a lottery @shlomoron
\nPeople winning golds here are one of the luckiest ones out there!
Well, we can still maximize our chances of winning gold. If all are random, it's 0.0046, so with some robustness and avoiding pitfalls that exist in public NB, we can easily exceed the 0.01 chance for gold—and maybe much higher! For a lottery, it is really great odds.
", "votes": null, "replies": [ { "id": 3073453, "author_name": "gunesevitan", "author_url": "", "post_date": "12/16/2024 14:18:10", "content": "I think there isn't much to do other than minimizing another metric on soft predictions and then make your bets.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3073431, "author_name": "mariusheuser", "author_url": "", "post_date": "12/16/2024 14:04:38", "content": "\"In other words, this means that the label distribution in test data is more biased toward 0 or 3 than in the training data\"
\nI don't think it necessarily has to mean that the distribution of sii=0 is higher. The thresholds depends on the data (not only the sii value). There could simply be a feature with a higher distribution than in the training df, which leads your model to a higher bias in general, so the thresholds move up aswell.
\nI haven't tested this, but I would assume that if you do the following:
\nSplit df into 3 parts with equal distribution of sii values.
\nTrain on one. Predict the other two. The optimal qwk for the predicted dfs will be different from each other, eventhough they have the same distribution of sii.
\nAnyway, we can't predict how the private LB looks anyway. We need alot of luck :)
Although it is higher than the notebook predicted sii=0 of 0.495, it cannot be said that bias is higher than training data. Thank you for pointing it out.
", "votes": null, "replies": [] } ] }, { "id": 3073535, "author_name": "yekenot", "author_url": "", "post_date": "12/16/2024 15:36:39", "content": "\n\nThe two values of thresholds are important. (You can ignore the last value,
\n
The last value produces a local maximum, isn't it? The global maximum is obtained from a value around 1.5 if we look in the 0.495 notebook. What if we updated 0.497 notebook with these changes? In my experiments with NN, global maximums lead to a 0.01 improvement in CV and 0.005 in LB. But sometimes this improvement is even more if the first step optimization wasn't successful.
", "votes": null, "replies": [ { "id": 3074128, "author_name": "minar44", "author_url": "", "post_date": "12/17/2024 10:54:50", "content": "The two values and last value here are thresholds. Thresholds are used to divide each class into 0, 1, 2, and 3.
\nFor example, the last value of the 0.497 notebook threshold is 2.71…, but the maximum predicted value of each model CV is as follows, and there is no value that is classified as 3.
| model | \nglobal maximum | \n
|---|---|
| model_1 | \n1.730926 | \n
| model_2 | \n2.003144 | \n
| model_3 | \n1.926503 | \n
\nAlso, I checked your notebook, it seems that the way the threshold is determined is different from that of the 0.497 notebook. Following that approach, the threshold would be [0.46, 0.91, 1.499]. I don't know if this will work as I haven't tried it.
I understand what you are pointing out about thresholds; it's okay. My question is why the author of the 0.495 notebook presents such great insight about global maximums and doesn't incorporate the calculation of the thresholds in that way in the further pipeline. The author of the 0.497 notebook is missing this discovery as well. So both best public notebooks can be even better, I think.
", "votes": null, "replies": [] }, { "id": 3074303, "author_name": "minar44", "author_url": "", "post_date": "12/17/2024 14:45:12", "content": "I haven't tried it yet, so I can't say anything about it, but I think that using global maximum method may be overfits training data.
\nSure, that method allows us to find a combination of thresholds that optimizes CV score, but it doesn't mean it's the optimal threshold for test data too.
\n\nusing global maximum method may be overfits training data.
\n
Is true also for local minimum.
\nYou can avoid CV overfit by further splitting: train, train_treshold and validation.
\nBut it's probably not worth it. Just submit and see if global minimum perform better than local minimum. Since it's only a small and very focused experiment you are not in danger of LB overfitting.
The noisy Quadratic Weighted Kappa (QWK) scores can be attributed to a fundamental mismatch between training and test data distributions, particularly in how the target labels (sii values 0-3) are distributed. The test data contains a higher proportion of sii=0 cases than what was predicted from training data patterns, as evidenced by the threshold shift from 0.588 to 0.600 between optimization attempts.
\nThe nature of QWK calculation itself contributes to this volatility, as it employs a squared penalty system for prediction errors. This is clearly demonstrated in the penalty matrix (ω_i,j), where predictions that deviate further from the true class incur substantially larger penalties. For instance, confusing class 3 with class 1 results in a severe penalty of 4.
\nThe sensitivity of QWK scores becomes particularly apparent in how the standard deviation increases with parameter j, especially when there are numerous labels of 0 and 2. This sensitivity is so pronounced that even minor variations in seed values (like choosing between 42 and 2024) can lead to markedly different scores. This high variance in performance metrics stems from the combined effect of distributional differences between training and test sets, QWK's inherent quadratic penalty structure, and potentially insufficient training data, making the model's performance highly dependent on both initialization conditions and the underlying data distribution.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3073374": "We know from public notebook results that small differences in seed values(=42,2024,...) can yield very noisy scores.\nOne of the reasons for this may be insufficient training data, but in my opinion, **target (=sii) distribution** of the test data is different from the training data is also an important factor.\n\nThrough several experiments, I found that the variance of QWK scores changes just by changing the distribution of sii (=0~3). This is due to the way QWK is calculated. In QWK calculations, the larger the difference between the predicted and correct labels, the more squared the penalty is given. (The figure below shows the penalties for each class.)\n\n\n\nThis experiments suggests that even in a completely random environment, the standard deviation of QWK increases when there are many labels such as 0 and 2. For example, the higher the ratio of sii=2, the higher the random QWK variation.\n| sii0/all=0.5, sii1/all=0.5-j, sii2/all=j | j | QWK's std |\n| --- | --- | --- |\n| | 0.0 | 0.02226 |\n| | 0.1 | 0.02727 |\n| | 0.2 | 0.02945 |\n| | 0.3 | 0.03056 |\n| | 0.4 | 0.03099 |\n| | 0.5 | 0.03120 |\n\n