{
  "id": 323402,
  "title": "Can somebody explain the evaluation technique",
  "url": "/competitions/smartphone-decimeter-2022/discussion/323402",
  "author_name": "",
  "post_date": "2022-05-06T09:57:56.951367700Z",
  "votes": 4,
  "comment_count": 2,
  "views": 0,
  "content": "<p>What does \"Submissions are scored on the mean of the 50th and 95th percentile distance errors\" mean?</p>",
  "messages": [
    {
      "id": "1779329",
      "postDate": "05/06/2022 09:57:56",
      "content": "<p>What does \"Submissions are scored on the mean of the 50th and 95th percentile distance errors\" mean?</p>",
      "rawMarkdown": "What does \"Submissions are scored on the mean of the 50th and 95th percentile distance errors\" mean?",
      "votes": null
    },
    {
      "id": "1779837",
      "postDate": "05/06/2022 21:43:51",
      "content": "<p>Hi Jimat,</p>\n<p>Your solution will predict the latitude and longitude of the vehicle at all the requested timestamps, typically every 1s.  The journeys are typically an hours long, so you'll be making roughly 3,600 predictions per journey.  For evaluation, the distance error will be computed for each of the data points along the journey.  This will give you roughly 3,600 distance errors (i.e. the distance in meters between your prediction and the true value).  If you were to line all these errors up from smallest to largest, the 50th percentile would be the middle value (i.e. the 1,800th in the sorted list) and the 95th percentile would be fairly near the biggest error (3,420th value in the sorted list).  The evaluation takes the average of these two values as the score for the journey.</p>\n<p>Then it does all of the above for each journey and takes the average of the individual journey scores.</p>\n<p>The upshot of all of that is that it's important both to make good predictions on average, but also to reduce the number of outliers (and the size of those outliers).  (Extreme outliers - in the worst 5% of your predictions per route - don't make any difference to your score, but the rest do.)</p>\n<p>I hope that helps.  Let me know if anything isn't clear.</p>\n<p>Andrew</p>",
      "rawMarkdown": "Hi Jimat,\n\nYour solution will predict the latitude and longitude of the vehicle at all the requested timestamps, typically every 1s.  The journeys are typically an hours long, so you'll be making roughly 3,600 predictions per journey.  For evaluation, the distance error will be computed for each of the data points along the journey.  This will give you roughly 3,600 distance errors (i.e. the distance in meters between your prediction and the true value).  If you were to line all these errors up from smallest to largest, the 50th percentile would be the middle value (i.e. the 1,800th in the sorted list) and the 95th percentile would be fairly near the biggest error (3,420th value in the sorted list).  The evaluation takes the average of these two values as the score for the journey.\n\nThen it does all of the above for each journey and takes the average of the individual journey scores.\n\nThe upshot of all of that is that it's important both to make good predictions on average, but also to reduce the number of outliers (and the size of those outliers).  (Extreme outliers - in the worst 5% of your predictions per route - don't make any difference to your score, but the rest do.)\n\nI hope that helps.  Let me know if anything isn't clear.\n\nAndrew",
      "votes": null
    },
    {
      "id": "1780674",
      "postDate": "05/07/2022 17:22:15",
      "content": "<p>Wow, it was a very nice explanation! thanks <a href=\"https://www.kaggle.com/andrewrrose\" target=\"_blank\">@andrewrrose</a> </p>",
      "rawMarkdown": "Wow, it was a very nice explanation! thanks @andrewrrose",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1779837,
      "author_name": "andrewrrose",
      "author_url": "",
      "post_date": "05/06/2022 21:43:51",
      "content": "<p>Hi Jimat,</p>\n<p>Your solution will predict the latitude and longitude of the vehicle at all the requested timestamps, typically every 1s.  The journeys are typically an hours long, so you'll be making roughly 3,600 predictions per journey.  For evaluation, the distance error will be computed for each of the data points along the journey.  This will give you roughly 3,600 distance errors (i.e. the distance in meters between your prediction and the true value).  If you were to line all these errors up from smallest to largest, the 50th percentile would be the middle value (i.e. the 1,800th in the sorted list) and the 95th percentile would be fairly near the biggest error (3,420th value in the sorted list).  The evaluation takes the average of these two values as the score for the journey.</p>\n<p>Then it does all of the above for each journey and takes the average of the individual journey scores.</p>\n<p>The upshot of all of that is that it's important both to make good predictions on average, but also to reduce the number of outliers (and the size of those outliers).  (Extreme outliers - in the worst 5% of your predictions per route - don't make any difference to your score, but the rest do.)</p>\n<p>I hope that helps.  Let me know if anything isn't clear.</p>\n<p>Andrew</p>",
      "votes": null,
      "replies": [
        {
          "id": 1780674,
          "author_name": "jimitshah777",
          "author_url": "",
          "post_date": "05/07/2022 17:22:15",
          "content": "<p>Wow, it was a very nice explanation! thanks <a href=\"https://www.kaggle.com/andrewrrose\" target=\"_blank\">@andrewrrose</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1779329": "What does \"Submissions are scored on the mean of the 50th and 95th percentile distance errors\" mean?",
    "1779837": "Hi Jimat,\n\nYour solution will predict the latitude and longitude of the vehicle at all the requested timestamps, typically every 1s.  The journeys are typically an hours long, so you'll be making roughly 3,600 predictions per journey.  For evaluation, the distance error will be computed for each of the data points along the journey.  This will give you roughly 3,600 distance errors (i.e. the distance in meters between your prediction and the true value).  If you were to line all these errors up from smallest to largest, the 50th percentile would be the middle value (i.e. the 1,800th in the sorted list) and the 95th percentile would be fairly near the biggest error (3,420th value in the sorted list).  The evaluation takes the average of these two values as the score for the journey.\n\nThen it does all of the above for each journey and takes the average of the individual journey scores.\n\nThe upshot of all of that is that it's important both to make good predictions on average, but also to reduce the number of outliers (and the size of those outliers).  (Extreme outliers - in the worst 5% of your predictions per route - don't make any difference to your score, but the rest do.)\n\nI hope that helps.  Let me know if anything isn't clear.\n\nAndrew",
    "1780674": "Wow, it was a very nice explanation! thanks @andrewrrose"
  },
  "source": "meta"
}