{
  "id": 189030,
  "title": "l5kit evaluation metric",
  "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/189030",
  "author_name": "",
  "post_date": "2020-10-06T12:01:51.345747500Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I get very large error values for what look like quite good test inputs, mean absolute error for ten steps - 250 samples</p>\n<ol>\n<li>0.20865453 0.34789981 0.45752303<br>\n0.53371379 0.59657276 0.66059172 0.74480307 0.8253196  0.89017859</li>\n</ol>\n<p>But error values per sample from the metrics.neg_multi_log_likelihood function</p>\n<p>5495995.0119675435<br>\n5572643.463186087<br>\n5290248.74200898</p>",
  "messages": [
    {
      "id": "1039179",
      "postDate": "10/06/2020 12:01:51",
      "content": "<p>I get very large error values for what look like quite good test inputs, mean absolute error for ten steps - 250 samples</p>\n<ol>\n<li>0.20865453 0.34789981 0.45752303<br>\n0.53371379 0.59657276 0.66059172 0.74480307 0.8253196  0.89017859</li>\n</ol>\n<p>But error values per sample from the metrics.neg_multi_log_likelihood function</p>\n<p>5495995.0119675435<br>\n5572643.463186087<br>\n5290248.74200898</p>",
      "rawMarkdown": "I get very large error values for what look like quite good test inputs, mean absolute error for ten steps - 250 samples\n\n0.         0.20865453 0.34789981 0.45752303\n 0.53371379 0.59657276 0.66059172 0.74480307 0.8253196  0.89017859\n\nBut error values per sample from the metrics.neg_multi_log_likelihood function\n\n5495995.0119675435\n5572643.463186087\n5290248.74200898",
      "votes": null
    },
    {
      "id": "1039197",
      "postDate": "10/06/2020 12:16:57",
      "content": "<p>Your mean absolute errors are quite large, especially taking into account a large number of stationary cars. A small number samples with large errors when squared can lead to large average neg_multi_log_likelihood value.</p>",
      "rawMarkdown": "Your mean absolute errors are quite large, especially taking into account a large number of stationary cars. A small number samples with large errors when squared can lead to large average neg_multi_log_likelihood value.",
      "votes": null
    },
    {
      "id": "1039223",
      "postDate": "10/06/2020 12:40:52",
      "content": "<p>I'm not quite convinced, but I can share the notebook. There's a fair chance i've done something incorrectly, but it seems ok</p>\n<p><a href=\"https://www.kaggle.com/n3n77i/recon-err\" target=\"_blank\">https://www.kaggle.com/n3n77i/recon-err</a></p>\n<p>The testing is against reconstructed inputs rather than predictions</p>",
      "rawMarkdown": "I'm not quite convinced, but I can share the notebook. There's a fair chance i've done something incorrectly, but it seems ok\n\nhttps://www.kaggle.com/n3n77i/recon-err\n\nThe testing is against reconstructed inputs rather than predictions",
      "votes": null
    },
    {
      "id": "1039917",
      "postDate": "10/06/2020 21:56:18",
      "content": "<p>Resolved, I was using the wrong inputs </p>\n<p>add: for reference, the underlying metric is sum mean squared coordinate error, ((x_err^2 + y_err^2) / 2 ) * no_steps </p>",
      "rawMarkdown": "Resolved, I was using the wrong inputs \n\nadd: for reference, the underlying metric is sum mean squared coordinate error, ((x_err^2 + y_err^2) / 2 ) * no_steps",
      "votes": null
    },
    {
      "id": "1039958",
      "postDate": "10/07/2020 00:02:20",
      "content": "<p>Confidence values are also a bit funny, the penalty for reasonably good predictions with very low confidence seems quite low? +100% score for C = 0.0001</p>\n<p>Roughly linear + 1.0-c error down to around 0.3 confidence, then up to +10 for c &lt; 0.001<br>\nupdate: confidence_err = log(c)</p>",
      "rawMarkdown": "Confidence values are also a bit funny, the penalty for reasonably good predictions with very low confidence seems quite low? +100% score for C = 0.0001\n\nRoughly linear + 1.0-c error down to around 0.3 confidence, then up to +10 for c < 0.001\nupdate: confidence_err = log(c)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1039197,
      "author_name": "dmytropoplavskiy",
      "author_url": "",
      "post_date": "10/06/2020 12:16:57",
      "content": "<p>Your mean absolute errors are quite large, especially taking into account a large number of stationary cars. A small number samples with large errors when squared can lead to large average neg_multi_log_likelihood value.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1039223,
          "author_name": "n3n77i",
          "author_url": "",
          "post_date": "10/06/2020 12:40:52",
          "content": "<p>I'm not quite convinced, but I can share the notebook. There's a fair chance i've done something incorrectly, but it seems ok</p>\n<p><a href=\"https://www.kaggle.com/n3n77i/recon-err\" target=\"_blank\">https://www.kaggle.com/n3n77i/recon-err</a></p>\n<p>The testing is against reconstructed inputs rather than predictions</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1039958,
          "author_name": "n3n77i",
          "author_url": "",
          "post_date": "10/07/2020 00:02:20",
          "content": "<p>Confidence values are also a bit funny, the penalty for reasonably good predictions with very low confidence seems quite low? +100% score for C = 0.0001</p>\n<p>Roughly linear + 1.0-c error down to around 0.3 confidence, then up to +10 for c &lt; 0.001<br>\nupdate: confidence_err = log(c)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1039917,
      "author_name": "n3n77i",
      "author_url": "",
      "post_date": "10/06/2020 21:56:18",
      "content": "<p>Resolved, I was using the wrong inputs </p>\n<p>add: for reference, the underlying metric is sum mean squared coordinate error, ((x_err^2 + y_err^2) / 2 ) * no_steps </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1039179": "I get very large error values for what look like quite good test inputs, mean absolute error for ten steps - 250 samples\n\n0.         0.20865453 0.34789981 0.45752303\n 0.53371379 0.59657276 0.66059172 0.74480307 0.8253196  0.89017859\n\nBut error values per sample from the metrics.neg_multi_log_likelihood function\n\n5495995.0119675435\n5572643.463186087\n5290248.74200898",
    "1039197": "Your mean absolute errors are quite large, especially taking into account a large number of stationary cars. A small number samples with large errors when squared can lead to large average neg_multi_log_likelihood value.",
    "1039223": "I'm not quite convinced, but I can share the notebook. There's a fair chance i've done something incorrectly, but it seems ok\n\nhttps://www.kaggle.com/n3n77i/recon-err\n\nThe testing is against reconstructed inputs rather than predictions",
    "1039917": "Resolved, I was using the wrong inputs \n\nadd: for reference, the underlying metric is sum mean squared coordinate error, ((x_err^2 + y_err^2) / 2 ) * no_steps",
    "1039958": "Confidence values are also a bit funny, the penalty for reasonably good predictions with very low confidence seems quite low? +100% score for C = 0.0001\n\nRoughly linear + 1.0-c error down to around 0.3 confidence, then up to +10 for c < 0.001\nupdate: confidence_err = log(c)"
  },
  "source": "meta"
}