{ "id": 389169, "title": "Per question log loss [with code snippet]", "url": "/competitions/predict-student-performance-from-game-play/discussion/389169", "author_name": "", "post_date": "2023-02-21T03:21:26.785414100Z", "votes": 5, "comment_count": 1, "views": 0, "content": "

F1 macro score is the evaluation metric for this competition, but perhaps isn't the best way to estimate a model's skill on a single question. Especially compared to other questions or compared to predicting a constant.

\n

I like using a type of adjusted log loss. Taking the percentage improvement in the loss from the model's predictions compared with blindly predicting the class balance as the constant probability. It's probably more clear in code:

\n
 k  ():\n\n    \n    m = log_loss(true[k].values, oof[k].values)\n    blind = log_loss(true[k].values, np.full(true[k].values.shape, true[k].()/true[k].count()))\n    (, (blind - m) / blind, , m)\n\n\nall_true = true.values.reshape((-))\nm = log_loss(all_true, oof.values.reshape((-)))\nblind = log_loss(all_true, np.full(all_true.shape, all_true.() / all_true.shape[]))\n(, (blind - m) / blind, , m)\n
\n

with output like this for the XGBoost Baseline:

\n
Q1: Loss improvement =  Log loss = \nQ2: Loss improvement =  Log loss = \nQ3: Loss improvement =  Log loss = \nQ4: Loss improvement =  Log loss = \nQ5: Loss improvement =  Log loss = \nQ6: Loss improvement =  Log loss = \nQ7: Loss improvement =  Log loss = \nQ8: Loss improvement =  Log loss = \nQ9: Loss improvement =  Log loss = \nQ10: Loss improvement =  Log loss = \nQ11: Loss improvement =  Log loss = \nQ12: Loss improvement =  Log loss = \nQ13: Loss improvement =  Log loss = \nQ14: Loss improvement =  Log loss = \nQ15: Loss improvement =  Log loss = \nQ16: Loss improvement =  Log loss = \nQ17: Loss improvement =  Log loss = \nQ18: Loss improvement =  Log loss = \n==> Overall loss improvement =  log loss = \n
\n

But output like this for the Random Forest baseline:

\n
: Loss improvement =  Log loss = \nQ2: Loss improvement = - Log loss = \nQ3: Loss improvement = - Log loss = \nQ4: Loss improvement =  Log loss = \nQ5: Loss improvement =  Log loss = \nQ6: Loss improvement =  Log loss = \nQ7: Loss improvement =  Log loss = \nQ8: Loss improvement = - Log loss = \nQ9: Loss improvement =  Log loss = \nQ10: Loss improvement =  Log loss = \nQ11: Loss improvement =  Log loss = \nQ12: Loss improvement = - Log loss = \nQ13: Loss improvement =  Log loss = \nQ14: Loss improvement =  Log loss = \nQ15: Loss improvement =  Log loss = \nQ16: Loss improvement = - Log loss = \nQ17: Loss improvement = - Log loss = \nQ18: Loss improvement = - Log loss = \n==> Overall loss improvement =  log loss = \n
\n

This output is a bit more human readable in my view, and highlights where a model is doing particularly well vs poorly.

", "messages": [ { "id": "2152807", "postDate": "02/21/2023 03:21:26", "content": "

F1 macro score is the evaluation metric for this competition, but perhaps isn't the best way to estimate a model's skill on a single question. Especially compared to other questions or compared to predicting a constant.

\n

I like using a type of adjusted log loss. Taking the percentage improvement in the loss from the model's predictions compared with blindly predicting the class balance as the constant probability. It's probably more clear in code:

\n
 k  ():\n\n    \n    m = log_loss(true[k].values, oof[k].values)\n    blind = log_loss(true[k].values, np.full(true[k].values.shape, true[k].()/true[k].count()))\n    (, (blind - m) / blind, , m)\n\n\nall_true = true.values.reshape((-))\nm = log_loss(all_true, oof.values.reshape((-)))\nblind = log_loss(all_true, np.full(all_true.shape, all_true.() / all_true.shape[]))\n(, (blind - m) / blind, , m)\n
\n

with output like this for the XGBoost Baseline:

\n
Q1: Loss improvement =  Log loss = \nQ2: Loss improvement =  Log loss = \nQ3: Loss improvement =  Log loss = \nQ4: Loss improvement =  Log loss = \nQ5: Loss improvement =  Log loss = \nQ6: Loss improvement =  Log loss = \nQ7: Loss improvement =  Log loss = \nQ8: Loss improvement =  Log loss = \nQ9: Loss improvement =  Log loss = \nQ10: Loss improvement =  Log loss = \nQ11: Loss improvement =  Log loss = \nQ12: Loss improvement =  Log loss = \nQ13: Loss improvement =  Log loss = \nQ14: Loss improvement =  Log loss = \nQ15: Loss improvement =  Log loss = \nQ16: Loss improvement =  Log loss = \nQ17: Loss improvement =  Log loss = \nQ18: Loss improvement =  Log loss = \n==> Overall loss improvement =  log loss = \n
\n

But output like this for the Random Forest baseline:

\n
: Loss improvement =  Log loss = \nQ2: Loss improvement = - Log loss = \nQ3: Loss improvement = - Log loss = \nQ4: Loss improvement =  Log loss = \nQ5: Loss improvement =  Log loss = \nQ6: Loss improvement =  Log loss = \nQ7: Loss improvement =  Log loss = \nQ8: Loss improvement = - Log loss = \nQ9: Loss improvement =  Log loss = \nQ10: Loss improvement =  Log loss = \nQ11: Loss improvement =  Log loss = \nQ12: Loss improvement = - Log loss = \nQ13: Loss improvement =  Log loss = \nQ14: Loss improvement =  Log loss = \nQ15: Loss improvement =  Log loss = \nQ16: Loss improvement = - Log loss = \nQ17: Loss improvement = - Log loss = \nQ18: Loss improvement = - Log loss = \n==> Overall loss improvement =  log loss = \n
\n

This output is a bit more human readable in my view, and highlights where a model is doing particularly well vs poorly.

", "rawMarkdown": "F1 macro score is the evaluation metric for this competition, but perhaps isn't the best way to estimate a model's skill on a single question. Especially compared to other questions or compared to predicting a constant.\n\nI like using a type of adjusted log loss. Taking the percentage improvement in the loss from the model's predictions compared with blindly predicting the class balance as the constant probability. It's probably more clear in code:\n\n```python\nfor k in range(18):\n\n # COMPUTE LOG LOSS SCORE PER QUESTION\n m = log_loss(true[k].values, oof[k].values)\n blind = log_loss(true[k].values, np.full(true[k].values.shape, true[k].sum()/true[k].count()))\n print(f'Q{k+1}: Loss improvement =', (blind - m) / blind, \"Log loss =\", m)\n \n# COMPUTE F1 SCORE OVERALL\nall_true = true.values.reshape((-1))\nm = log_loss(all_true, oof.values.reshape((-1)))\nblind = log_loss(all_true, np.full(all_true.shape, all_true.sum() / all_true.shape[0]))\nprint('==> Overall loss improvement =', (blind - m) / blind, \"log loss =\", m)\n```\n\nwith output like this for the XGBoost Baseline:\n```python\nQ1: Loss improvement = 0.055942362273853886 Log loss = 0.5561777215935708\nQ2: Loss improvement = 0.056882065526941625 Log loss = 0.09692049366278292\nQ3: Loss improvement = 0.05578567513432904 Log loss = 0.2341614541712648\nQ4: Loss improvement = 0.09890758549293428 Log loss = 0.4517770859635777\nQ5: Loss improvement = 0.058881852129344645 Log loss = 0.6482758529069098\nQ6: Loss improvement = 0.07732157678681215 Log loss = 0.49527996201122343\nQ7: Loss improvement = 0.05715527124334885 Log loss = 0.5506114822068406\nQ8: Loss improvement = 0.016250543440364192 Log loss = 0.655943869450482\nQ9: Loss improvement = 0.06133176813028352 Log loss = 0.5423163255179014\nQ10: Loss improvement = 0.05625131587505579 Log loss = 0.6541564640755049\nQ11: Loss improvement = 0.038137372862145864 Log loss = 0.6261388290805355\nQ12: Loss improvement = 0.03285490199674833 Log loss = 0.396095159272896\nQ13: Loss improvement = 0.06415919402592156 Log loss = 0.5462849318654703\nQ14: Loss improvement = 0.06314108737324568 Log loss = 0.5640663508170124\nQ15: Loss improvement = 0.0639807517906829 Log loss = 0.6482565892765335\nQ16: Loss improvement = 0.005470441964189067 Log loss = 0.57215845512635\nQ17: Loss improvement = 0.015394477913998042 Log loss = 0.6132315734264148\nQ18: Loss improvement = 0.07509626130488153 Log loss = 0.18199723811562224\n==> Overall loss improvement = 0.17383722765274012 log loss = 0.5018805465856052\n```\n\n\nBut output like this for the Random Forest baseline:\n```python\n1: Loss improvement = 0.009968893371547544 Log loss = 0.5832623170314342\nQ2: Loss improvement = -0.9671763473460785 Log loss = 0.20215891961916513\nQ3: Loss improvement = -0.17970166412649088 Log loss = 0.29256139192700725\nQ4: Loss improvement = 0.052017156807821964 Log loss = 0.475286351927748\nQ5: Loss improvement = 0.03730382318801655 Log loss = 0.6631395712908762\nQ6: Loss improvement = 0.04084273431542177 Log loss = 0.5148612584400759\nQ7: Loss improvement = 0.033877459474270996 Log loss = 0.5642054813562084\nQ8: Loss improvement = -0.0044815686794867126 Log loss = 0.669767614668431\nQ9: Loss improvement = 0.03679239079161823 Log loss = 0.5564939705014693\nQ10: Loss improvement = 0.03322440226903215 Log loss = 0.6701174975968976\nQ11: Loss improvement = 0.014963487698008112 Log loss = 0.6412242154055002\nQ12: Loss improvement = -0.009924967223552515 Log loss = 0.4136156938312294\nQ13: Loss improvement = 0.02911362443671399 Log loss = 0.5667423285434232\nQ14: Loss improvement = 0.03367676461436953 Log loss = 0.5818063037532506\nQ15: Loss improvement = 0.039101513278229805 Log loss = 0.6654871433839781\nQ16: Loss improvement = -0.012803987746548416 Log loss = 0.5826718374457864\nQ17: Loss improvement = -0.0024215171666258656 Log loss = 0.6243277235600253\nQ18: Loss improvement = -0.2245399432084328 Log loss = 0.24095792708181418\n==> Overall loss improvement = 0.1304124148710452 log loss = 0.5282604192980178\n```\n\nThis output is a bit more human readable in my view, and highlights where a model is doing particularly well vs poorly.", "votes": null }, { "id": "2162165", "postDate": "02/28/2023 03:18:49", "content": "

Thanks for sharing Robert! Would you mind sharing what adjustments in your metric you made compared to scikit-learn's version?

\n

Also, I wonder if class imbalance may cause any bias when we use log loss (to explore features, for example). However, I think it shouldn't be the case if we observe the score improvement instead of the absolute score value, should it?

", "rawMarkdown": "Thanks for sharing Robert! Would you mind sharing what adjustments in your metric you made compared to [scikit-learn's version](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html)?\n\nAlso, I wonder if class imbalance may cause any bias when we use log loss (to explore features, for example). However, I think it shouldn't be the case if we observe the **score improvement** instead of the absolute **score value**, should it?", "votes": null } ], "comments": [ { "id": 2162165, "author_name": "hoangnguyen719", "author_url": "", "post_date": "02/28/2023 03:18:49", "content": "

Thanks for sharing Robert! Would you mind sharing what adjustments in your metric you made compared to scikit-learn's version?

\n

Also, I wonder if class imbalance may cause any bias when we use log loss (to explore features, for example). However, I think it shouldn't be the case if we observe the score improvement instead of the absolute score value, should it?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2152807": "F1 macro score is the evaluation metric for this competition, but perhaps isn't the best way to estimate a model's skill on a single question. Especially compared to other questions or compared to predicting a constant.\n\nI like using a type of adjusted log loss. Taking the percentage improvement in the loss from the model's predictions compared with blindly predicting the class balance as the constant probability. It's probably more clear in code:\n\n```python\nfor k in range(18):\n\n # COMPUTE LOG LOSS SCORE PER QUESTION\n m = log_loss(true[k].values, oof[k].values)\n blind = log_loss(true[k].values, np.full(true[k].values.shape, true[k].sum()/true[k].count()))\n print(f'Q{k+1}: Loss improvement =', (blind - m) / blind, \"Log loss =\", m)\n \n# COMPUTE F1 SCORE OVERALL\nall_true = true.values.reshape((-1))\nm = log_loss(all_true, oof.values.reshape((-1)))\nblind = log_loss(all_true, np.full(all_true.shape, all_true.sum() / all_true.shape[0]))\nprint('==> Overall loss improvement =', (blind - m) / blind, \"log loss =\", m)\n```\n\nwith output like this for the XGBoost Baseline:\n```python\nQ1: Loss improvement = 0.055942362273853886 Log loss = 0.5561777215935708\nQ2: Loss improvement = 0.056882065526941625 Log loss = 0.09692049366278292\nQ3: Loss improvement = 0.05578567513432904 Log loss = 0.2341614541712648\nQ4: Loss improvement = 0.09890758549293428 Log loss = 0.4517770859635777\nQ5: Loss improvement = 0.058881852129344645 Log loss = 0.6482758529069098\nQ6: Loss improvement = 0.07732157678681215 Log loss = 0.49527996201122343\nQ7: Loss improvement = 0.05715527124334885 Log loss = 0.5506114822068406\nQ8: Loss improvement = 0.016250543440364192 Log loss = 0.655943869450482\nQ9: Loss improvement = 0.06133176813028352 Log loss = 0.5423163255179014\nQ10: Loss improvement = 0.05625131587505579 Log loss = 0.6541564640755049\nQ11: Loss improvement = 0.038137372862145864 Log loss = 0.6261388290805355\nQ12: Loss improvement = 0.03285490199674833 Log loss = 0.396095159272896\nQ13: Loss improvement = 0.06415919402592156 Log loss = 0.5462849318654703\nQ14: Loss improvement = 0.06314108737324568 Log loss = 0.5640663508170124\nQ15: Loss improvement = 0.0639807517906829 Log loss = 0.6482565892765335\nQ16: Loss improvement = 0.005470441964189067 Log loss = 0.57215845512635\nQ17: Loss improvement = 0.015394477913998042 Log loss = 0.6132315734264148\nQ18: Loss improvement = 0.07509626130488153 Log loss = 0.18199723811562224\n==> Overall loss improvement = 0.17383722765274012 log loss = 0.5018805465856052\n```\n\n\nBut output like this for the Random Forest baseline:\n```python\n1: Loss improvement = 0.009968893371547544 Log loss = 0.5832623170314342\nQ2: Loss improvement = -0.9671763473460785 Log loss = 0.20215891961916513\nQ3: Loss improvement = -0.17970166412649088 Log loss = 0.29256139192700725\nQ4: Loss improvement = 0.052017156807821964 Log loss = 0.475286351927748\nQ5: Loss improvement = 0.03730382318801655 Log loss = 0.6631395712908762\nQ6: Loss improvement = 0.04084273431542177 Log loss = 0.5148612584400759\nQ7: Loss improvement = 0.033877459474270996 Log loss = 0.5642054813562084\nQ8: Loss improvement = -0.0044815686794867126 Log loss = 0.669767614668431\nQ9: Loss improvement = 0.03679239079161823 Log loss = 0.5564939705014693\nQ10: Loss improvement = 0.03322440226903215 Log loss = 0.6701174975968976\nQ11: Loss improvement = 0.014963487698008112 Log loss = 0.6412242154055002\nQ12: Loss improvement = -0.009924967223552515 Log loss = 0.4136156938312294\nQ13: Loss improvement = 0.02911362443671399 Log loss = 0.5667423285434232\nQ14: Loss improvement = 0.03367676461436953 Log loss = 0.5818063037532506\nQ15: Loss improvement = 0.039101513278229805 Log loss = 0.6654871433839781\nQ16: Loss improvement = -0.012803987746548416 Log loss = 0.5826718374457864\nQ17: Loss improvement = -0.0024215171666258656 Log loss = 0.6243277235600253\nQ18: Loss improvement = -0.2245399432084328 Log loss = 0.24095792708181418\n==> Overall loss improvement = 0.1304124148710452 log loss = 0.5282604192980178\n```\n\nThis output is a bit more human readable in my view, and highlights where a model is doing particularly well vs poorly.", "2162165": "Thanks for sharing Robert! Would you mind sharing what adjustments in your metric you made compared to [scikit-learn's version](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html)?\n\nAlso, I wonder if class imbalance may cause any bias when we use log loss (to explore features, for example). However, I think it shouldn't be the case if we observe the **score improvement** instead of the absolute **score value**, should it?" }, "source": "meta" }