Citation: Fed Zee (https://stats.stackexchange.com/users/239985/fed-zee), What's considered a good log loss?, URL (version: 2019-03-14): https://stats.stackexchange.com/q/395774
\n\"Like any metric, a good metric is the one better that the \"dumb\", by-chance guess, if you would have to guess with no information on the observations. This is called the intercept-only model in statistics.\"
\nThis \"dumb\"-guess depends on 2 factors :
\nthe number of classes
\nthe balance of classes : their prevalence in the observed dataset
\n\"In the case of the LogLoss metric, one usual \"well-known\" metric is to say that 0.693 is the non-informative value. This figure is obtained by predicting p = 0.5 for any class of a binary problem. This is valid only for balanced binary problems. Because when prevalence of one class is of 10%, then you will predict p =0.1 for that class, always. This will be your baseline of dumb, by-chance prediction, because predicting 0.5 will be dumber.\"
I. Impact of the number of classes N on the dumb-logloss:
\n\"In the balanced case (every class has the same prevalence), when you predict p = prevalence = 1 / N for every observation, the equation becomes simply :\"
\nLogloss = -log(1 / N)
\nlog being Ln, neperian logarithm for those who use that convention.
\nIn the binary case, N = 2 : Logloss = - log(1/2) = 0.693
\nII.Impact of the prevalence of classes on the dumb-Logloss:
\na. Binary classification case
\n \"When classes are very unbalanced (prevalence <2%), a logloss of 0.1 can actually be very bad ! Such as an accuracy of 98% would be bad in that case. So maybe Logloss would not be the best metric to use.
b. Three-class case
\n\"Dumb\"-logloss depending on prevalence.
CONCLUSION
\n\"A logloss of 0.69 may be good in a multiclass problem, and very bad in a binary biased case.\"
\n\"Depending of your case, you would better compute yourself the baseline of the problem, to check the meaning of your prediction.\"
\n\"In the biased cases, I understand that logloss has the same problem as the accuracy and other loss functions : it provides only a global measurement of your performance. So you would better complement your understanding with metrics focused on the minority classes (recall and precision), or maybe not use logloss at all.\"
\nhttps://stats.stackexchange.com/questions/276067/whats-considered-a-good-log-loss
\ncitation: http://www.exegetic.biz/blog/2015/12/making-sense-logarithmic-loss/
\n\"Log Loss is a measure of how well a classifier is working Given a classifier that assigns probabilities to each of the classes it is predicting, (predict_proba in sklearn), Log Loss is the sum of the log of the probabilities assigned to the correct class.\"
\nAssigning a probability of 1.0 to every correct class results in a Log Loss of 0.0.
\nhttps://www.kaggle.com/code/grfiv4/log-loss-depicted-1
KAGGLE NOTEBOOKS
\nWhat is Log Loss? By Dan B.
\nhttps://www.kaggle.com/code/dansbecker/what-is-log-loss/notebook
Log loss example By Robin East
\nhttps://www.kaggle.com/code/robineast/log-loss-example
Log-loss with New York City Building Sales By Aleksey Bilogur
\nhttps://www.kaggle.com/code/residentmario/log-loss-with-new-york-city-building-sales
0.335 log-loss in a dozen lines By Marco Spinaci
\nhttps://www.kaggle.com/code/marcospinaci/0-335-log-loss-in-a-dozen-lines
Log Loss depicted R script By George Fisher
\nhttps://www.kaggle.com/code/grfiv4/log-loss-depicted-1
Weighted Multi Class Log Loss Explained By Moth
\nhttps://www.kaggle.com/code/alejopaullier/weighted-multi-class-log-loss-explained
KAGGLE DISCUSSION TOPICS
\nConfidence intervals for Log Loss metric? By Tijl Kindt
\nhttps://www.kaggle.com/discussions/general/10885
Comment By Jason Sumpter
\n\"The log loss for all your predictions is just the mean of the losses of all the predictions. You should be able to do a test for the difference of the means. Throw \"difference of means confidence intervals\" into Google and you should be able to find what you are looking for. If you are using R then t.test(results1, results2), where results1 is the vector of individual log losses for model 1 and results 2 for model 2, should tell you what you want to know.\"
\nhttps://www.kaggle.com/discussions/general/10885
Log Loss Confusion By Ahsoka
\nhttps://www.kaggle.com/discussions/general/26365
sklearn.metrics.log_loss
\nhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html
I have so much to learn in this challenging competition.
", "messages": [ { "id": "1966273", "postDate": "10/01/2022 20:16:48", "content": "Citation: Fed Zee (https://stats.stackexchange.com/users/239985/fed-zee), What's considered a good log loss?, URL (version: 2019-03-14): https://stats.stackexchange.com/q/395774
\n\"Like any metric, a good metric is the one better that the \"dumb\", by-chance guess, if you would have to guess with no information on the observations. This is called the intercept-only model in statistics.\"
\nThis \"dumb\"-guess depends on 2 factors :
\nthe number of classes
\nthe balance of classes : their prevalence in the observed dataset
\n\"In the case of the LogLoss metric, one usual \"well-known\" metric is to say that 0.693 is the non-informative value. This figure is obtained by predicting p = 0.5 for any class of a binary problem. This is valid only for balanced binary problems. Because when prevalence of one class is of 10%, then you will predict p =0.1 for that class, always. This will be your baseline of dumb, by-chance prediction, because predicting 0.5 will be dumber.\"
I. Impact of the number of classes N on the dumb-logloss:
\n\"In the balanced case (every class has the same prevalence), when you predict p = prevalence = 1 / N for every observation, the equation becomes simply :\"
\nLogloss = -log(1 / N)
\nlog being Ln, neperian logarithm for those who use that convention.
\nIn the binary case, N = 2 : Logloss = - log(1/2) = 0.693
\nII.Impact of the prevalence of classes on the dumb-Logloss:
\na. Binary classification case
\n \"When classes are very unbalanced (prevalence <2%), a logloss of 0.1 can actually be very bad ! Such as an accuracy of 98% would be bad in that case. So maybe Logloss would not be the best metric to use.
b. Three-class case
\n\"Dumb\"-logloss depending on prevalence.
CONCLUSION
\n\"A logloss of 0.69 may be good in a multiclass problem, and very bad in a binary biased case.\"
\n\"Depending of your case, you would better compute yourself the baseline of the problem, to check the meaning of your prediction.\"
\n\"In the biased cases, I understand that logloss has the same problem as the accuracy and other loss functions : it provides only a global measurement of your performance. So you would better complement your understanding with metrics focused on the minority classes (recall and precision), or maybe not use logloss at all.\"
\nhttps://stats.stackexchange.com/questions/276067/whats-considered-a-good-log-loss
\ncitation: http://www.exegetic.biz/blog/2015/12/making-sense-logarithmic-loss/
\n\"Log Loss is a measure of how well a classifier is working Given a classifier that assigns probabilities to each of the classes it is predicting, (predict_proba in sklearn), Log Loss is the sum of the log of the probabilities assigned to the correct class.\"
\nAssigning a probability of 1.0 to every correct class results in a Log Loss of 0.0.
\nhttps://www.kaggle.com/code/grfiv4/log-loss-depicted-1
KAGGLE NOTEBOOKS
\nWhat is Log Loss? By Dan B.
\nhttps://www.kaggle.com/code/dansbecker/what-is-log-loss/notebook
Log loss example By Robin East
\nhttps://www.kaggle.com/code/robineast/log-loss-example
Log-loss with New York City Building Sales By Aleksey Bilogur
\nhttps://www.kaggle.com/code/residentmario/log-loss-with-new-york-city-building-sales
0.335 log-loss in a dozen lines By Marco Spinaci
\nhttps://www.kaggle.com/code/marcospinaci/0-335-log-loss-in-a-dozen-lines
Log Loss depicted R script By George Fisher
\nhttps://www.kaggle.com/code/grfiv4/log-loss-depicted-1
Weighted Multi Class Log Loss Explained By Moth
\nhttps://www.kaggle.com/code/alejopaullier/weighted-multi-class-log-loss-explained
KAGGLE DISCUSSION TOPICS
\nConfidence intervals for Log Loss metric? By Tijl Kindt
\nhttps://www.kaggle.com/discussions/general/10885
Comment By Jason Sumpter
\n\"The log loss for all your predictions is just the mean of the losses of all the predictions. You should be able to do a test for the difference of the means. Throw \"difference of means confidence intervals\" into Google and you should be able to find what you are looking for. If you are using R then t.test(results1, results2), where results1 is the vector of individual log losses for model 1 and results 2 for model 2, should tell you what you want to know.\"
\nhttps://www.kaggle.com/discussions/general/10885
Log Loss Confusion By Ahsoka
\nhttps://www.kaggle.com/discussions/general/26365
sklearn.metrics.log_loss
\nhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html
I have so much to learn in this challenging competition.
", "rawMarkdown": "#What's considered a good log loss?\n\nCitation: Fed Zee (https://stats.stackexchange.com/users/239985/fed-zee), What's considered a good log loss?, URL (version: 2019-03-14): https://stats.stackexchange.com/q/395774\n\n\"Like any metric, a good metric is the one better that the \"dumb\", by-chance guess, if you would have to guess with no information on the observations. This is called the intercept-only model in statistics.\"\n\nThis \"dumb\"-guess depends on 2 factors :\n\nthe number of classes\nthe balance of classes : their prevalence in the observed dataset\n\"In the case of the LogLoss metric, one usual \"well-known\" metric is to say that 0.693 is the non-informative value. This figure is obtained by predicting p = 0.5 for any class of a binary problem. This is valid only for balanced binary problems. Because when prevalence of one class is of 10%, then you will predict p =0.1 for that class, always. This will be your baseline of dumb, by-chance prediction, because predicting 0.5 will be dumber.\"\n\nI. Impact of the number of classes N on the dumb-logloss:\n\n\"In the balanced case (every class has the same prevalence), when you predict p = prevalence = 1 / N for every observation, the equation becomes simply :\"\n\nLogloss = -log(1 / N)\n\nlog being Ln, neperian logarithm for those who use that convention.\n\nIn the binary case, N = 2 : Logloss = - log(1/2) = 0.693\n\nII.Impact of the prevalence of classes on the dumb-Logloss:\n\na. Binary classification case\n \"When classes are very unbalanced (prevalence <2%), a logloss of 0.1 can actually be very bad ! Such as an accuracy of 98% would be bad in that case. So maybe Logloss would not be the best metric to use.\n\nb. Three-class case\n\"Dumb\"-logloss depending on prevalence.\n\nCONCLUSION\n\n\"A logloss of 0.69 may be good in a multiclass problem, and very bad in a binary biased case.\"\n\n\"Depending of your case, you would better compute yourself the baseline of the problem, to check the meaning of your prediction.\"\n\n\"In the biased cases, I understand that logloss has the same problem as the accuracy and other loss functions : it provides only a global measurement of your performance. So you would better complement your understanding with metrics focused on the minority classes (recall and precision), or maybe not use logloss at all.\"\n\nhttps://stats.stackexchange.com/questions/276067/whats-considered-a-good-log-loss\n\n#Log Loss on Kaggle\n\ncitation: http://www.exegetic.biz/blog/2015/12/making-sense-logarithmic-loss/\n\n\"Log Loss is a measure of how well a classifier is working Given a classifier that assigns probabilities to each of the classes it is predicting, (predict_proba in sklearn), Log Loss is the sum of the log of the probabilities assigned to the correct class.\"\n\nAssigning a probability of 1.0 to every correct class results in a Log Loss of 0.0.\nhttps://www.kaggle.com/code/grfiv4/log-loss-depicted-1\n\nKAGGLE NOTEBOOKS\n\nWhat is Log Loss? By Dan B.\nhttps://www.kaggle.com/code/dansbecker/what-is-log-loss/notebook\n\nLog loss example By Robin East\nhttps://www.kaggle.com/code/robineast/log-loss-example\n\nLog-loss with New York City Building Sales By Aleksey Bilogur\nhttps://www.kaggle.com/code/residentmario/log-loss-with-new-york-city-building-sales\n\n0.335 log-loss in a dozen lines By Marco Spinaci\nhttps://www.kaggle.com/code/marcospinaci/0-335-log-loss-in-a-dozen-lines\n\nLog Loss depicted R script By George Fisher\nhttps://www.kaggle.com/code/grfiv4/log-loss-depicted-1\n\nWeighted Multi Class Log Loss Explained By Moth\nhttps://www.kaggle.com/code/alejopaullier/weighted-multi-class-log-loss-explained\n\nKAGGLE DISCUSSION TOPICS\n\nConfidence intervals for Log Loss metric? By Tijl Kindt\nhttps://www.kaggle.com/discussions/general/10885\n\nComment By Jason Sumpter \n\"The log loss for all your predictions is just the mean of the losses of all the predictions. You should be able to do a test for the difference of the means. Throw \"difference of means confidence intervals\" into Google and you should be able to find what you are looking for. If you are using R then t.test(results1, results2), where results1 is the vector of individual log losses for model 1 and results 2 for model 2, should tell you what you want to know.\" \nhttps://www.kaggle.com/discussions/general/10885\n\nLog Loss Confusion By Ahsoka\nhttps://www.kaggle.com/discussions/general/26365\n\nsklearn.metrics.log_loss\nhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html\n\n#Are you still lost with Log Loss?\n\nI have so much to learn in this challenging competition.", "votes": null }, { "id": "1966364", "postDate": "10/01/2022 22:53:21", "content": "It may be worth pointing out that the scoring metric for this competition is not the standard log-loss for a 3-class classification problem (team A wins, team B wins, neither wins). The metric for this competition is
\n\\[
\n-\\frac{1}{N}\\sum_{i=1}^N\\sum_{m=1}^2\\frac{y_{i,m}\\log(p_{i,m})+(1-y_{i,m})\\log(1-p_{i,m})}{2}
\n\\]
whereas the standard log-loss for 3-class classification is
\n\\[
\n-\\frac{1}{N}\\sum_{i=1}^N\\sum_{m=1}^3y_{i,m}\\log(p_{i,m})
\n\\]
where \\(y_{i,3}=1-y_{i,1}-y_{i,2}\\) and \\(p_{i,3}=1-p_{i,1}-p_{i,2}\\). For example, for a sample \\(i\\) with team A winning, the contributing term is \\(\\frac{1}{2}(\\log(p_{i,1})+\\log(1-p_{i,2}))\\) for the competition metric, whereas for the standard log-loss it is simply \\(\\log(p_{i,1})\\). If you are formulating a model with standard softmax and log-loss, you are not exactly minimizing the metric of this competition.
", "rawMarkdown": "It may be worth pointing out that the scoring metric for this competition is not the standard log-loss for a 3-class classification problem (team A wins, team B wins, neither wins). The metric for this competition is \n\n\\\\[\n-\\frac{1}{N}\\sum_{i=1}^N\\sum_{m=1}^2\\frac{y_{i,m}\\log(p_{i,m})+(1-y_{i,m})\\log(1-p_{i,m})}{2}\n\\\\]\n\nwhereas the standard log-loss for 3-class classification is\n\n\\\\[\n-\\frac{1}{N}\\sum_{i=1}^N\\sum_{m=1}^3y_{i,m}\\log(p_{i,m})\n\\\\]\n\nwhere \\\\(y_{i,3}=1-y_{i,1}-y_{i,2}\\\\) and \\\\(p_{i,3}=1-p_{i,1}-p_{i,2}\\\\). For example, for a sample \\\\(i\\\\) with team A winning, the contributing term is \\\\(\\frac{1}{2}(\\log(p_{i,1})+\\log(1-p_{i,2}))\\\\) for the competition metric, whereas for the standard log-loss it is simply \\\\(\\log(p_{i,1})\\\\). If you are formulating a model with standard softmax and log-loss, you are not exactly minimizing the metric of this competition.", "votes": null }, { "id": "1966370", "postDate": "10/01/2022 23:16:36", "content": "Valuable comment contribution Siukeitin. Thank you.
", "rawMarkdown": "Valuable comment contribution Siukeitin. Thank you.", "votes": null } ], "comments": [ { "id": 1966364, "author_name": "siukeitin", "author_url": "", "post_date": "10/01/2022 22:53:21", "content": "It may be worth pointing out that the scoring metric for this competition is not the standard log-loss for a 3-class classification problem (team A wins, team B wins, neither wins). The metric for this competition is
\n\\[
\n-\\frac{1}{N}\\sum_{i=1}^N\\sum_{m=1}^2\\frac{y_{i,m}\\log(p_{i,m})+(1-y_{i,m})\\log(1-p_{i,m})}{2}
\n\\]
whereas the standard log-loss for 3-class classification is
\n\\[
\n-\\frac{1}{N}\\sum_{i=1}^N\\sum_{m=1}^3y_{i,m}\\log(p_{i,m})
\n\\]
where \\(y_{i,3}=1-y_{i,1}-y_{i,2}\\) and \\(p_{i,3}=1-p_{i,1}-p_{i,2}\\). For example, for a sample \\(i\\) with team A winning, the contributing term is \\(\\frac{1}{2}(\\log(p_{i,1})+\\log(1-p_{i,2}))\\) for the competition metric, whereas for the standard log-loss it is simply \\(\\log(p_{i,1})\\). If you are formulating a model with standard softmax and log-loss, you are not exactly minimizing the metric of this competition.
", "votes": null, "replies": [ { "id": 1966370, "author_name": "mpwolke", "author_url": "", "post_date": "10/01/2022 23:16:36", "content": "Valuable comment contribution Siukeitin. Thank you.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1966273": "#What's considered a good log loss?\n\nCitation: Fed Zee (https://stats.stackexchange.com/users/239985/fed-zee), What's considered a good log loss?, URL (version: 2019-03-14): https://stats.stackexchange.com/q/395774\n\n\"Like any metric, a good metric is the one better that the \"dumb\", by-chance guess, if you would have to guess with no information on the observations. This is called the intercept-only model in statistics.\"\n\nThis \"dumb\"-guess depends on 2 factors :\n\nthe number of classes\nthe balance of classes : their prevalence in the observed dataset\n\"In the case of the LogLoss metric, one usual \"well-known\" metric is to say that 0.693 is the non-informative value. This figure is obtained by predicting p = 0.5 for any class of a binary problem. This is valid only for balanced binary problems. Because when prevalence of one class is of 10%, then you will predict p =0.1 for that class, always. This will be your baseline of dumb, by-chance prediction, because predicting 0.5 will be dumber.\"\n\nI. Impact of the number of classes N on the dumb-logloss:\n\n\"In the balanced case (every class has the same prevalence), when you predict p = prevalence = 1 / N for every observation, the equation becomes simply :\"\n\nLogloss = -log(1 / N)\n\nlog being Ln, neperian logarithm for those who use that convention.\n\nIn the binary case, N = 2 : Logloss = - log(1/2) = 0.693\n\nII.Impact of the prevalence of classes on the dumb-Logloss:\n\na. Binary classification case\n \"When classes are very unbalanced (prevalence <2%), a logloss of 0.1 can actually be very bad ! Such as an accuracy of 98% would be bad in that case. So maybe Logloss would not be the best metric to use.\n\nb. Three-class case\n\"Dumb\"-logloss depending on prevalence.\n\nCONCLUSION\n\n\"A logloss of 0.69 may be good in a multiclass problem, and very bad in a binary biased case.\"\n\n\"Depending of your case, you would better compute yourself the baseline of the problem, to check the meaning of your prediction.\"\n\n\"In the biased cases, I understand that logloss has the same problem as the accuracy and other loss functions : it provides only a global measurement of your performance. So you would better complement your understanding with metrics focused on the minority classes (recall and precision), or maybe not use logloss at all.\"\n\nhttps://stats.stackexchange.com/questions/276067/whats-considered-a-good-log-loss\n\n#Log Loss on Kaggle\n\ncitation: http://www.exegetic.biz/blog/2015/12/making-sense-logarithmic-loss/\n\n\"Log Loss is a measure of how well a classifier is working Given a classifier that assigns probabilities to each of the classes it is predicting, (predict_proba in sklearn), Log Loss is the sum of the log of the probabilities assigned to the correct class.\"\n\nAssigning a probability of 1.0 to every correct class results in a Log Loss of 0.0.\nhttps://www.kaggle.com/code/grfiv4/log-loss-depicted-1\n\nKAGGLE NOTEBOOKS\n\nWhat is Log Loss? By Dan B.\nhttps://www.kaggle.com/code/dansbecker/what-is-log-loss/notebook\n\nLog loss example By Robin East\nhttps://www.kaggle.com/code/robineast/log-loss-example\n\nLog-loss with New York City Building Sales By Aleksey Bilogur\nhttps://www.kaggle.com/code/residentmario/log-loss-with-new-york-city-building-sales\n\n0.335 log-loss in a dozen lines By Marco Spinaci\nhttps://www.kaggle.com/code/marcospinaci/0-335-log-loss-in-a-dozen-lines\n\nLog Loss depicted R script By George Fisher\nhttps://www.kaggle.com/code/grfiv4/log-loss-depicted-1\n\nWeighted Multi Class Log Loss Explained By Moth\nhttps://www.kaggle.com/code/alejopaullier/weighted-multi-class-log-loss-explained\n\nKAGGLE DISCUSSION TOPICS\n\nConfidence intervals for Log Loss metric? By Tijl Kindt\nhttps://www.kaggle.com/discussions/general/10885\n\nComment By Jason Sumpter \n\"The log loss for all your predictions is just the mean of the losses of all the predictions. You should be able to do a test for the difference of the means. Throw \"difference of means confidence intervals\" into Google and you should be able to find what you are looking for. If you are using R then t.test(results1, results2), where results1 is the vector of individual log losses for model 1 and results 2 for model 2, should tell you what you want to know.\" \nhttps://www.kaggle.com/discussions/general/10885\n\nLog Loss Confusion By Ahsoka\nhttps://www.kaggle.com/discussions/general/26365\n\nsklearn.metrics.log_loss\nhttps://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html\n\n#Are you still lost with Log Loss?\n\nI have so much to learn in this challenging competition.", "1966364": "It may be worth pointing out that the scoring metric for this competition is not the standard log-loss for a 3-class classification problem (team A wins, team B wins, neither wins). The metric for this competition is \n\n\\\\[\n-\\frac{1}{N}\\sum_{i=1}^N\\sum_{m=1}^2\\frac{y_{i,m}\\log(p_{i,m})+(1-y_{i,m})\\log(1-p_{i,m})}{2}\n\\\\]\n\nwhereas the standard log-loss for 3-class classification is\n\n\\\\[\n-\\frac{1}{N}\\sum_{i=1}^N\\sum_{m=1}^3y_{i,m}\\log(p_{i,m})\n\\\\]\n\nwhere \\\\(y_{i,3}=1-y_{i,1}-y_{i,2}\\\\) and \\\\(p_{i,3}=1-p_{i,1}-p_{i,2}\\\\). For example, for a sample \\\\(i\\\\) with team A winning, the contributing term is \\\\(\\frac{1}{2}(\\log(p_{i,1})+\\log(1-p_{i,2}))\\\\) for the competition metric, whereas for the standard log-loss it is simply \\\\(\\log(p_{i,1})\\\\). If you are formulating a model with standard softmax and log-loss, you are not exactly minimizing the metric of this competition.", "1966370": "Valuable comment contribution Siukeitin. Thank you." }, "source": "meta" }