\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12870466%2F5176185e2ed37b2170c29398b031bfcd%2FScreenshot_7.jpg?generation=1677782997268274&alt=media\")
\n(I didn't use the best_threshold after seeing this notebook)
\nBut my global F1 dropped significantly, any idea why is that ? souldn't the model be more accurate globally ?
\n/
\nNote: I am using a XGB model.
I have tried to optimise each question separately to improve overall F1 score:
\n
\n(I didn't use the best_threshold after seeing this notebook)
\nBut my global F1 dropped significantly, any idea why is that ? souldn't the model be more accurate globally ?
\n/
\nNote: I am using a XGB model.
I have tried to optimise each question separately to improve overall F1 score:
\n
\n(I didn't use the best_threshold after seeing this notebook)
\nBut my global F1 dropped significantly, any idea why is that ? souldn't the model be more accurate globally ?
\n/
\nNote: I am using a XGB model.
I think you may optimize each model, but you have to do it at the same time: not one model at a time.
", "rawMarkdown": "I think you may optimize each model, but you have to do it at the same time: not one model at a time.", "votes": null }, { "id": "2166615", "postDate": "03/02/2023 21:48:59", "content": "Thanks. Can you pleae guide us with a way to do it?
", "rawMarkdown": "Thanks. Can you pleae guide us with a way to do it?", "votes": null }, { "id": "2168941", "postDate": "03/04/2023 16:46:02", "content": "How do you \"optimize\" lr? The lower you set it - the better. It's all about the computational budget. 0.3 and 0.1 do look too high. Set lr at something like 0.02 and use the validation set to do the early stopping.
", "rawMarkdown": "How do you \"optimize\" lr? The lower you set it - the better. It's all about the computational budget. 0.3 and 0.1 do look too high. Set lr at something like 0.02 and use the validation set to do the early stopping.", "votes": null }, { "id": "2170853", "postDate": "03/06/2023 10:44:48", "content": "actually, in theory that it how it should work
\nbut as you can see, lr is not the same for each question, and even the global f1 isn't obtained when the lower lr is set because of overfitting.
Not sure what you are doing, but the optimization of 'best_threshold' might be partially to blame. If you are doing something like this:
\nThis will fail because the F1 score is macro averaged. That is, the true values of 1 F1 score is calculated, then the true values of 0 F1 score is calculated, then those two F1 scores are averaged.
\nSo for some questions, like #2, which most everyone gets right, it is much more important to forecast the people who get it wrong when you are fitting the threshold of the individual question. However, in the overall scheme when the number of wrong answers for #2 is lower, this will not matter as much.
\nTry looking at the output of classification_report in the sklearn.metrics module.
If you want to individually optimize thresholds, code like this might be better
\nFirst get a dataframe predictions with your predictions and the true values with columns like:
\n[index, session_id, correct, question#, prediction_prob]
Then something like this will work (Excuse the poor formatting):
\nimport scipy.optimize as spo
\ndef fun(x):
\n\\t for q in range(1,19):
\n\\t\\t thresh = x[q-1]
\n\\t\\t mask = predictions['question'] == q
\n\\t\\t predictions.loc[mask, 'temp'] = predictions.loc[mask, 'pred'].map(lambda x: 1 if x > thresh else 0)
\n\\t m = f1_score(predictions['correct'].values, predictions['temp'].values, average='macro')
\n\\t return -m
x0 = np.ones(18)*0.5 + np.random.normal(scale=0.01, size=18)
soln = spo.minimize(fun=fun, x0=x0, method='Nelder-Mead', options={'disp': True, 'fatol': 1.e-5})
I've explored this idea, but I optimized for accuracy rather than F1 score for each model, in many cases I can get a better CV. I haven't checked on the math, but my intuition tells me that accuracy correlates more with macro F1 score in different subsets (questions) with varying positive-negative ratios.
", "rawMarkdown": "I've explored this idea, but I optimized for accuracy rather than F1 score for each model, in many cases I can get a better CV. I haven't checked on the math, but my intuition tells me that accuracy correlates more with macro F1 score in different subsets (questions) with varying positive-negative ratios.", "votes": null } ], "comments": [ { "id": 2166455, "author_name": "steubk", "author_url": "", "post_date": "03/02/2023 20:06:48", "content": "I think you may optimize each model, but you have to do it at the same time: not one model at a time.
", "votes": null, "replies": [ { "id": 2166615, "author_name": "shashwatraman", "author_url": "", "post_date": "03/02/2023 21:48:59", "content": "Thanks. Can you pleae guide us with a way to do it?
", "votes": null, "replies": [] } ] }, { "id": 2168941, "author_name": "sakvaua", "author_url": "", "post_date": "03/04/2023 16:46:02", "content": "How do you \"optimize\" lr? The lower you set it - the better. It's all about the computational budget. 0.3 and 0.1 do look too high. Set lr at something like 0.02 and use the validation set to do the early stopping.
", "votes": null, "replies": [ { "id": 2170853, "author_name": "janmpia", "author_url": "", "post_date": "03/06/2023 10:44:48", "content": "actually, in theory that it how it should work
\nbut as you can see, lr is not the same for each question, and even the global f1 isn't obtained when the lower lr is set because of overfitting.
Not sure what you are doing, but the optimization of 'best_threshold' might be partially to blame. If you are doing something like this:
\nThis will fail because the F1 score is macro averaged. That is, the true values of 1 F1 score is calculated, then the true values of 0 F1 score is calculated, then those two F1 scores are averaged.
\nSo for some questions, like #2, which most everyone gets right, it is much more important to forecast the people who get it wrong when you are fitting the threshold of the individual question. However, in the overall scheme when the number of wrong answers for #2 is lower, this will not matter as much.
\nTry looking at the output of classification_report in the sklearn.metrics module.
If you want to individually optimize thresholds, code like this might be better
\nFirst get a dataframe predictions with your predictions and the true values with columns like:
\n[index, session_id, correct, question#, prediction_prob]
Then something like this will work (Excuse the poor formatting):
\nimport scipy.optimize as spo
\ndef fun(x):
\n\\t for q in range(1,19):
\n\\t\\t thresh = x[q-1]
\n\\t\\t mask = predictions['question'] == q
\n\\t\\t predictions.loc[mask, 'temp'] = predictions.loc[mask, 'pred'].map(lambda x: 1 if x > thresh else 0)
\n\\t m = f1_score(predictions['correct'].values, predictions['temp'].values, average='macro')
\n\\t return -m
x0 = np.ones(18)*0.5 + np.random.normal(scale=0.01, size=18)
soln = spo.minimize(fun=fun, x0=x0, method='Nelder-Mead', options={'disp': True, 'fatol': 1.e-5})
I've explored this idea, but I optimized for accuracy rather than F1 score for each model, in many cases I can get a better CV. I haven't checked on the math, but my intuition tells me that accuracy correlates more with macro F1 score in different subsets (questions) with varying positive-negative ratios.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2166381": "I have tried to **optimise each question separately** to improve overall F1 score:\n\n*(I didn't use the best_threshold after seeing [this](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/389217) notebook)*\nBut my global F1 dropped significantly, any idea why is that ? souldn't the model be more accurate globally ?\n/\n**Note: I am using a XGB model.**", "2166455": "I think you may optimize each model, but you have to do it at the same time: not one model at a time.", "2166615": "Thanks. Can you pleae guide us with a way to do it?", "2168941": "How do you \"optimize\" lr? The lower you set it - the better. It's all about the computational budget. 0.3 and 0.1 do look too high. Set lr at something like 0.02 and use the validation set to do the early stopping.", "2170853": "actually, in theory that it how it should work\nbut as you can see, lr is not the same for each question, and even the global f1 isn't obtained when the lower lr is set because of overfitting.", "2202166": "Not sure what you are doing, but the optimization of 'best_threshold' might be partially to blame. If you are doing something like this:\n\n1. Create my features\n2. FOREACH question:\n - Fit a model forecasting each question\n - Find the best threshold for marking as 0 or 1 to optimize the F1 score on this question\n3. Calculate the total F1 score with best parameters\n\nThis will fail because the F1 score is macro averaged. That is, the true values of 1 F1 score is calculated, then the true values of 0 F1 score is calculated, then those two F1 scores are averaged.\n\nSo for some questions, like #2, which most everyone gets right, it is much more important to forecast the people who get it wrong when you are fitting the threshold of the individual question. However, in the overall scheme when the number of wrong answers for #2 is lower, this will not matter as much.\n\nTry looking at the output of `classification_report` in the `sklearn.metrics` module.\n\nIf you want to individually optimize thresholds, code like this might be better\n\nFirst get a dataframe `predictions` with your predictions and the true values with columns like: \n`[index, session_id, correct, question#, prediction_prob]`\n\nThen something like this will work (Excuse the poor formatting):\n`import scipy.optimize as spo`\n`def fun(x):`\n`\\t for q in range(1,19):`\n`\\t\\t thresh = x[q-1]`\n`\\t\\t mask = predictions['question'] == q`\n`\\t\\t predictions.loc[mask, 'temp'] = predictions.loc[mask, 'pred'].map(lambda x: 1 if x > thresh else 0)`\n`\\t m = f1_score(predictions['correct'].values, predictions['temp'].values, average='macro')`\n`\\t return -m`\n\n`x0 = np.ones(18)*0.5 + np.random.normal(scale=0.01, size=18)`\n\n`soln = spo.minimize(fun=fun, x0=x0, method='Nelder-Mead', options={'disp': True, 'fatol': 1.e-5})`", "2202775": "I've explored this idea, but I optimized for accuracy rather than F1 score for each model, in many cases I can get a better CV. I haven't checked on the math, but my intuition tells me that accuracy correlates more with macro F1 score in different subsets (questions) with varying positive-negative ratios." }, "source": "meta" }