Is the following Python code an accurate representation of how submissions are evaluated? I've played with this to help me evaluate my modelling, but wanted to make sure I understood how the evaluator worked. I believe I'll need to add a PostId to the prediction\r\n data when I submit, but have not included that for simplicity's sake in this example code.
\r\nfrom __future__ import division\r\n\r\nimport csv\r\nimport os\r\nimport scipy as sp\r\n\r\ndef llfun(act, pred):\r\n epsilon = 1e-15\r\n pred = sp.maximum(epsilon, pred)\r\n pred = sp.minimum(1-epsilon, pred)\r\n ll = sum(act*sp.log(pred) + sp.subtract(1,act)*sp.log(sp.subtract(1,pred)))\r\n ll = ll * -1.0/len(act)\r\n return ll\r\n\r\ndef main():\r\n pred = [\r\n [0.05,0.05,0.05,0.8,0.05],\r\n [0.73,0.05,0.01,0.20,0.02],\r\n [0.02,0.03,0.01,0.75,0.19],\r\n [0.01,0.02,0.83,0.12,0.02]\r\n ]\r\n act = [\r\n [0,0,0,1,0],\r\n [1,0,0,0,0],\r\n [0,0,0,1,0],\r\n [0,0,1,0,0]\r\n ]\r\n\r\n scores = []\r\n for index in range(0, len(pred)):\r\n result = llfun(act[index], pred[index])\r\n scores.append(result)\r\n\r\n print(sum(scores) / len(scores)) # 0.0985725708595\r\n\r\nif __name__ == '__main__':\r\n main()\r\n", "messages": [ { "id": "14247", "postDate": "09/12/2012 01:31:59", "content": "
Is the following Python code an accurate representation of how submissions are evaluated? I've played with this to help me evaluate my modelling, but wanted to make sure I understood how the evaluator worked. I believe I'll need to add a PostId to the prediction\r\n data when I submit, but have not included that for simplicity's sake in this example code.
\r\nfrom __future__ import division\r\n\r\nimport csv\r\nimport os\r\nimport scipy as sp\r\n\r\ndef llfun(act, pred):\r\n epsilon = 1e-15\r\n pred = sp.maximum(epsilon, pred)\r\n pred = sp.minimum(1-epsilon, pred)\r\n ll = sum(act*sp.log(pred) + sp.subtract(1,act)*sp.log(sp.subtract(1,pred)))\r\n ll = ll * -1.0/len(act)\r\n return ll\r\n\r\ndef main():\r\n pred = [\r\n [0.05,0.05,0.05,0.8,0.05],\r\n [0.73,0.05,0.01,0.20,0.02],\r\n [0.02,0.03,0.01,0.75,0.19],\r\n [0.01,0.02,0.83,0.12,0.02]\r\n ]\r\n act = [\r\n [0,0,0,1,0],\r\n [1,0,0,0,0],\r\n [0,0,0,1,0],\r\n [0,0,1,0,0]\r\n ]\r\n\r\n scores = []\r\n for index in range(0, len(pred)):\r\n result = llfun(act[index], pred[index])\r\n scores.append(result)\r\n\r\n print(sum(scores) / len(scores)) # 0.0985725708595\r\n\r\nif __name__ == '__main__':\r\n main()\r\n", "rawMarkdown": "", "votes": null }, { "id": "14248", "postDate": "09/12/2012 02:36:25", "content": "
ll = sum(act*sp.log(pred) + sp.subtract(1,act)*sp.log(sp.subtract(1,pred)))\nThat's not right. Since the prediction is a normalized multinomial distribution, you just take log(pred[label]), and ignore the other predictions not covered by the label (their impact on the score is via the normalization). If your prediction is not actually\r\n normalized, you need to normalize it (after clamping to 1e-15).
", "rawMarkdown": "", "votes": null }, { "id": "14250", "postDate": "09/12/2012 07:34:58", "content": "Here's the function I use:
\r\nimport numpy as np\r\n\r\ndef multiclass_log_loss(y_true, y_pred, eps=1e-15):\r\n """Multi class version of Logarithmic Loss metric.\r\n https://www.kaggle.com/wiki/MultiClassLogLoss\r\n\r\n idea from this post:\r\n http://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error/12209#post12209\r\n\r\n Parameters\r\n ----------\r\n y_true : array, shape = [n_samples]\r\n y_pred : array, shape = [n_samples, n_classes]\r\n\r\n Returns\r\n -------\r\n loss : float\r\n """\r\n predictions = np.clip(y_pred, eps, 1 - eps)\r\n\r\n # normalize row sums to 1\r\n predictions /= predictions.sum(axis=1)[:, np.newaxis]\r\n\r\n actual = np.zeros(y_pred.shape)\r\n rows = actual.shape[0]\r\n actual[np.arange(rows), y_true.astype(int)] = 1\r\n vsota = np.sum(actual * np.log(predictions))\r\n return -1.0 / rows * vsota\r\n", "rawMarkdown": "", "votes": null }, { "id": "14446", "postDate": "09/18/2012 00:24:58", "content": "
[quote=ephes;14250]
\r\nHere's the function I use:
\r\nimport numpy as np\r\n\r\ndef multiclass_log_loss(y_true, y_pred, eps=1e-15):\r\n """Multi class version of Logarithmic Loss metric.\r\n https://www.kaggle.com/wiki/MultiClassLogLoss\r\n\r\n idea from this post:\r\n http://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error/12209#post12209\r\n\r\n Parameters\r\n ----------\r\n y_true : array, shape = [n_samples]\r\n y_pred : array, shape = [n_samples, n_classes]\r\n\r\n Returns\r\n -------\r\n loss : float\r\n """\r\n predictions = np.clip(y_pred, eps, 1 - eps)\r\n\r\n # normalize row sums to 1\r\n predictions /= predictions.sum(axis=1)[:, np.newaxis]\r\n\r\n actual = np.zeros(y_pred.shape)\r\n rows = actual.shape[0]\r\n actual[np.arange(rows), y_true.astype(int)] = 1\r\n vsota = np.sum(actual * np.log(predictions))\r\n return -1.0 / rows * vsota\r\n\r\n
[/quote]
\r\nWhat type of objects are the inputs?
\r\ny_true : array, shape = [n_samples]
\r\ny_pred : array, shape = [n_samples, n_classes]
\r\n\r\n
I'm using a simple list of list and isn't working properly =(
\r\n
\r\nthanks for any help
The function assumes that two numpy ndarrays are supplied.
\r\nThe first is a 1-d array, where each element is the goldstandard class ID of the instance.
\r\nThe second is a 2-d array, where each element is the predicted distribution over the classes.
\r\nHere are some example uses:
\r\n>>> import numpy as np", "rawMarkdown": "", "votes": null }, { "id": "145265", "postDate": "11/17/2016 15:54:07", "content": "
>>> multiclass_log_loss(np.array([0,1,2]),np.array([[1,0,0],[0,1,0],[0,0,1]]))
2.1094237467877998e-15
>>> multiclass_log_loss(np.array([0,1,2]),np.array([[1,1,1],[0,1,0],[0,0,1]]))
0.36620409622270467
The other option would be to use the built in log_loss function in scikit learn. I've checked the results of that out of the box function against the ones posted above and it matches exactly. just use
\n\n>>> from sklearn.metrics import log_loss\n>>> log_loss(np.array([0,1,2]),np.array([[1,0,0],[0,1,0],[0,0,1]]))\n2.1094237467877998e-15\n>>> log_loss(np.array([0,1,2]),np.array([[1,1,1],[0,1,0],[0,0,1]]))\n0.36620409622270467\nOne of the advantages of the sklearn package is that you can leave your actual values coded as characters or strings. It assumes that the probabilities in the prediction array are ordered alphabetically. For example...
\n\n>>> from sklearn.metrics import log_loss\n>>> log_loss(np.array(['a','b','c']),np.array([[1,0,0],[0,1,0],[0,0,1]]))\n2.1094237467877998e-15\n>>> log_loss(np.array(['a','b','c']),np.array([[1,1,1],[0,1,0],[0,0,1]]))\n0.36620409622270467\n>>> log_loss(np.array(['cat','ant','boy']),np.array([[1,1,1],[0,1,0],[0,0,1]]))\n23.39205502616316\nll = sum(act*sp.log(pred) + sp.subtract(1,act)*sp.log(sp.subtract(1,pred)))\nThat's not right. Since the prediction is a normalized multinomial distribution, you just take log(pred[label]), and ignore the other predictions not covered by the label (their impact on the score is via the normalization). If your prediction is not actually\r\n normalized, you need to normalize it (after clamping to 1e-15).
", "votes": null, "replies": [] }, { "id": 14250, "author_name": "ephesus", "author_url": "", "post_date": "09/12/2012 07:34:58", "content": "Here's the function I use:
\r\nimport numpy as np\r\n\r\ndef multiclass_log_loss(y_true, y_pred, eps=1e-15):\r\n """Multi class version of Logarithmic Loss metric.\r\n https://www.kaggle.com/wiki/MultiClassLogLoss\r\n\r\n idea from this post:\r\n http://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error/12209#post12209\r\n\r\n Parameters\r\n ----------\r\n y_true : array, shape = [n_samples]\r\n y_pred : array, shape = [n_samples, n_classes]\r\n\r\n Returns\r\n -------\r\n loss : float\r\n """\r\n predictions = np.clip(y_pred, eps, 1 - eps)\r\n\r\n # normalize row sums to 1\r\n predictions /= predictions.sum(axis=1)[:, np.newaxis]\r\n\r\n actual = np.zeros(y_pred.shape)\r\n rows = actual.shape[0]\r\n actual[np.arange(rows), y_true.astype(int)] = 1\r\n vsota = np.sum(actual * np.log(predictions))\r\n return -1.0 / rows * vsota\r\n", "votes": null, "replies": [] }, { "id": 14446, "author_name": "alessandrosena", "author_url": "", "post_date": "09/18/2012 00:24:58", "content": "
[quote=ephes;14250]
\r\nHere's the function I use:
\r\nimport numpy as np\r\n\r\ndef multiclass_log_loss(y_true, y_pred, eps=1e-15):\r\n """Multi class version of Logarithmic Loss metric.\r\n https://www.kaggle.com/wiki/MultiClassLogLoss\r\n\r\n idea from this post:\r\n http://www.kaggle.com/c/emc-data-science/forums/t/2149/is-anyone-noticing-difference-betwen-validation-and-leaderboard-error/12209#post12209\r\n\r\n Parameters\r\n ----------\r\n y_true : array, shape = [n_samples]\r\n y_pred : array, shape = [n_samples, n_classes]\r\n\r\n Returns\r\n -------\r\n loss : float\r\n """\r\n predictions = np.clip(y_pred, eps, 1 - eps)\r\n\r\n # normalize row sums to 1\r\n predictions /= predictions.sum(axis=1)[:, np.newaxis]\r\n\r\n actual = np.zeros(y_pred.shape)\r\n rows = actual.shape[0]\r\n actual[np.arange(rows), y_true.astype(int)] = 1\r\n vsota = np.sum(actual * np.log(predictions))\r\n return -1.0 / rows * vsota\r\n\r\n
[/quote]
\r\nWhat type of objects are the inputs?
\r\ny_true : array, shape = [n_samples]
\r\ny_pred : array, shape = [n_samples, n_classes]
\r\n\r\n
I'm using a simple list of list and isn't working properly =(
\r\n
\r\nthanks for any help
The function assumes that two numpy ndarrays are supplied.
\r\nThe first is a 1-d array, where each element is the goldstandard class ID of the instance.
\r\nThe second is a 2-d array, where each element is the predicted distribution over the classes.
\r\nHere are some example uses:
\r\n>>> import numpy as np", "votes": null, "replies": [] }, { "id": 145265, "author_name": "jedisom", "author_url": "", "post_date": "11/17/2016 15:54:07", "content": "
>>> multiclass_log_loss(np.array([0,1,2]),np.array([[1,0,0],[0,1,0],[0,0,1]]))
2.1094237467877998e-15
>>> multiclass_log_loss(np.array([0,1,2]),np.array([[1,1,1],[0,1,0],[0,0,1]]))
0.36620409622270467
The other option would be to use the built in log_loss function in scikit learn. I've checked the results of that out of the box function against the ones posted above and it matches exactly. just use
\n\n>>> from sklearn.metrics import log_loss\n>>> log_loss(np.array([0,1,2]),np.array([[1,0,0],[0,1,0],[0,0,1]]))\n2.1094237467877998e-15\n>>> log_loss(np.array([0,1,2]),np.array([[1,1,1],[0,1,0],[0,0,1]]))\n0.36620409622270467\nOne of the advantages of the sklearn package is that you can leave your actual values coded as characters or strings. It assumes that the probabilities in the prediction array are ordered alphabetically. For example...
\n\n>>> from sklearn.metrics import log_loss\n>>> log_loss(np.array(['a','b','c']),np.array([[1,0,0],[0,1,0],[0,0,1]]))\n2.1094237467877998e-15\n>>> log_loss(np.array(['a','b','c']),np.array([[1,1,1],[0,1,0],[0,0,1]]))\n0.36620409622270467\n>>> log_loss(np.array(['cat','ant','boy']),np.array([[1,1,1],[0,1,0],[0,0,1]]))\n23.39205502616316\n