{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "68e78c2f-f3c3-fc33-d4f3-6a4fc5200a00"
      },
      "source": [
        "# Log Loss\n",
        "\n",
        "citation: http://www.exegetic.biz/blog/2015/12/making-sense-logarithmic-loss/\n",
        "\n",
        "### Log Loss is a measure of how well a classifier is working\n",
        "\n",
        "\n",
        "\n",
        "Given a classifier that assigns probabilities to each of the classes it is predicting, (`predict_proba` in sklearn), Log Loss is the sum of the log of the probabilities assigned to the correct\n",
        "class.\n",
        "\n",
        "Assigning a probability of 1.0 to every correct class results in a Log Loss of 0.0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "16126696-7641-d010-7405-75d03dfa7b50"
      },
      "outputs": [],
      "source": [
        "LogLossBinary = function(actual, predicted, eps = 1e-15) {\n",
        "    # return the Log Loss\n",
        "    #\n",
        "    # Arguments\n",
        "    # ---------\n",
        "    #   actual      the correct prediction\n",
        "    #   predicted   the probability assigned\n",
        "    #   eps         earnings per share\n",
        "    #\n",
        "    # Returns\n",
        "    # -------\n",
        "    #   logloss\n",
        "    \n",
        "   predicted = pmin(pmax(predicted, eps), 1-eps)\n",
        "   - (sum(actual * log(predicted) + (1 - actual) * log(1 - predicted))) / length(actual)\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "6f777e9e-56e1-0ea1-2de4-ec04f394325a"
      },
      "source": [
        "# Examples"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "a04e7113-4557-5f5c-9071-ad67508b360c"
      },
      "source": [
        "### correct class given a 60% probabiity"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "114ff97b-48a5-4121-d3c0-be30a2db7e7b"
      },
      "outputs": [],
      "source": [
        "LogLossBinary(1, c(0.60))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "1c632748-e425-6e35-d1b6-3a84686e4cf7"
      },
      "source": [
        "### correct class given 90% probability"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "742ccdab-d52e-a5d7-c2a9-84d9fec78e44"
      },
      "outputs": [],
      "source": [
        "LogLossBinary(1, c(0.9))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "f00629b3-e3a7-f2d2-0ed1-80c1e9972fc3"
      },
      "source": [
        "### correct class given only 10% probability"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3aa0e431-1735-7f7e-ec5c-9bd2a4d40d2e"
      },
      "outputs": [],
      "source": [
        "LogLossBinary(1, c(0.1))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "7a28fbda-42d4-bd20-8278-5779d73364ae"
      },
      "outputs": [],
      "source": [
        "lgloss = c()\n",
        "x      = seq(from = 0.01, to = 1.0, by = 0.01)\n",
        "\n",
        "for (prob in x) {\n",
        "    lgloss = c(lgloss, LogLossBinary(1, c(prob)))\n",
        "}\n",
        "plot(x=x, y=lgloss, type='l',\n",
        "     xlab=\"Probability Assigned to Correct Class\",\n",
        "     ylab=\"Log Loss\",\n",
        "     main=\"Log Loss as Predictions Improve\")\n",
        "abline(h=0,lty=2,col='red')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "_cell_guid": "687d5039-9d14-08bc-dd42-5380f7625ca9"
      },
      "source": [
        "# Log Loss for a given predicted probability"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "3109e018-f9a9-290d-d322-7030544d5b41"
      },
      "outputs": [],
      "source": [
        "data.frame(probability=x,logloss=round(lgloss, 5))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "_cell_guid": "ad3e13b5-32a0-71e2-d5a4-d5368e81a8de"
      },
      "outputs": [],
      "source": [
        ""
      ]
    }
  ],
  "metadata": {
    "_change_revision": 0,
    "_is_fork": false,
    "kernelspec": {
      "display_name": "R",
      "language": "R",
      "name": "ir"
    },
    "language_info": {
      "codemirror_mode": "r",
      "file_extension": ".r",
      "mimetype": "text/x-r-source",
      "name": "R",
      "pygments_lexer": "r",
      "version": "3.3.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}