{
  "id": 2421,
  "title": "multiclass logloss in R",
  "url": "/competitions/predict-closed-questions-on-stack-overflow/discussion/2421",
  "author_name": "",
  "post_date": "2012-08-23T19:53:12.653Z",
  "votes": null,
  "comment_count": 4,
  "views": 7680,
  "content": "<p>Here's the code I'm using, but the result doesn't quiete match the <a href=\"http://www.kaggle.com/wiki/MultiClassLogLoss\">\r\nwiki page</a>:</p>\r\n<blockquote>\r\n<pre>LogLoss &lt;- function(actual, predicted, eps=1e-15) {<br>  predicted &lt;- pmin(pmax(predicted, eps), 1-eps)<br>  -1/length(actual)*(sum(actual*log(predicted)&#43;(1-actual)*log(1-predicted)))<br>}<br>a &lt;- matrix(c(1,1,1,0,0,0,0,0,0,1,1,1), ncol=2)<br>p &lt;- matrix(c(0.5, 0.1, 0.01, 0.9, 0.75, 0.001, 0.5, 0.9, 0.99, 0.2, 0.25, 0.999), ncol=2)<br>LogLoss(a,p)<br>stopifnot(1.881797068998267==LogLoss(a,p))</pre>\r\n</blockquote>\r\n<p>Can anyone spot my error?</p>",
  "messages": [
    {
      "id": "13372",
      "postDate": "08/23/2012 19:53:12",
      "content": "<p>Here's the code I'm using, but the result doesn't quiete match the <a href=\"http://www.kaggle.com/wiki/MultiClassLogLoss\">\r\nwiki page</a>:</p>\r\n<blockquote>\r\n<pre>LogLoss &lt;- function(actual, predicted, eps=1e-15) {<br>  predicted &lt;- pmin(pmax(predicted, eps), 1-eps)<br>  -1/length(actual)*(sum(actual*log(predicted)&#43;(1-actual)*log(1-predicted)))<br>}<br>a &lt;- matrix(c(1,1,1,0,0,0,0,0,0,1,1,1), ncol=2)<br>p &lt;- matrix(c(0.5, 0.1, 0.01, 0.9, 0.75, 0.001, 0.5, 0.9, 0.99, 0.2, 0.25, 0.999), ncol=2)<br>LogLoss(a,p)<br>stopifnot(1.881797068998267==LogLoss(a,p))</pre>\r\n</blockquote>\r\n<p>Can anyone spot my error?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13373",
      "postDate": "08/23/2012 19:55:06",
      "content": "<p>Here's where I got the original code:<br>\r\nhttp://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "13374",
      "postDate": "08/23/2012 20:21:54",
      "content": "<p>That code is calculating two-class log-loss, with the values given as the probability of class 1 (the probability of class 0 is the inverse). This problem uses multiclass log-loss, which is slightly different. You want a function more like this:</p>\r\n<pre>LogLoss &lt;- function(actual, predicted, eps=1e-15) {<br>  predicted[predicted &lt; eps] &lt;- eps;<br>  predicted[predicted &gt; 1 - eps] &lt;- 1 - eps;<br>  -1/nrow(actual)*(sum(actual*log(predicted)))<br>}</pre>\r\n<p>You've also mistranscribed the example values: p[4,2] should be 0.1, not 0.2</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "72773",
      "postDate": "04/20/2015 22:50:15",
      "content": "<p>[quote=Martin O'Leary;13374]</p>\n<p>LogLoss &lt;- function(actual, predicted, eps=1e-15) {</p>\n<pre>  predicted[predicted &lt; eps] &lt;- eps;<br>  predicted[predicted &gt; 1 - eps] &lt;- 1 - eps;<br>  -1/nrow(actual)*(sum(actual*log(predicted)))<br>}</pre>\n<p>[/quote]</p>\n<p>What is actual in this multiclass case?&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "72939",
      "postDate": "04/21/2015 14:57:39",
      "content": "<p>We calculate logloss for a set of observations for which we already know the actual class. &nbsp;Actual is a probability value of 1 corresponding to a given class. &nbsp;Effectively we are summing all the predicted probabilities for the actual class.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 13373,
      "author_name": "zachmayer",
      "author_url": "",
      "post_date": "08/23/2012 19:55:06",
      "content": "<p>Here's where I got the original code:<br>\r\nhttp://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 13374,
      "author_name": "martinoleary",
      "author_url": "",
      "post_date": "08/23/2012 20:21:54",
      "content": "<p>That code is calculating two-class log-loss, with the values given as the probability of class 1 (the probability of class 0 is the inverse). This problem uses multiclass log-loss, which is slightly different. You want a function more like this:</p>\r\n<pre>LogLoss &lt;- function(actual, predicted, eps=1e-15) {<br>  predicted[predicted &lt; eps] &lt;- eps;<br>  predicted[predicted &gt; 1 - eps] &lt;- 1 - eps;<br>  -1/nrow(actual)*(sum(actual*log(predicted)))<br>}</pre>\r\n<p>You've also mistranscribed the example values: p[4,2] should be 0.1, not 0.2</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 72773,
      "author_name": "anshul",
      "author_url": "",
      "post_date": "04/20/2015 22:50:15",
      "content": "<p>[quote=Martin O'Leary;13374]</p>\n<p>LogLoss &lt;- function(actual, predicted, eps=1e-15) {</p>\n<pre>  predicted[predicted &lt; eps] &lt;- eps;<br>  predicted[predicted &gt; 1 - eps] &lt;- 1 - eps;<br>  -1/nrow(actual)*(sum(actual*log(predicted)))<br>}</pre>\n<p>[/quote]</p>\n<p>What is actual in this multiclass case?&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 72939,
      "author_name": "outliar",
      "author_url": "",
      "post_date": "04/21/2015 14:57:39",
      "content": "<p>We calculate logloss for a set of observations for which we already know the actual class. &nbsp;Actual is a probability value of 1 corresponding to a given class. &nbsp;Effectively we are summing all the predicted probabilities for the actual class.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "13372": "",
    "13373": "",
    "13374": "",
    "72773": "",
    "72939": ""
  },
  "source": "meta"
}