{
  "id": 110167,
  "title": "What Metric should be followed?",
  "url": "/competitions/3d-object-detection-for-autonomous-vehicles/discussion/110167",
  "author_name": "",
  "post_date": "2019-09-25T13:51:15.598831800Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi everyone.\nI wanted to ask you Kagglers which metric you follow during training?</p>\n\n<p>Until now have followed the categorical cross entropy  loss of Guido's Kernel\n(<a href=\"https://www.kaggle.com/gzuidhof/reference-model\">https://www.kaggle.com/gzuidhof/reference-model</a>). However, early stopping on the best validation error yielded a significantly lower LB score.\nDo I actually need to follow a categorical loss? Or does it make more sense to follow a IoU?</p>\n\n<p>I am open for any suggestion and to learn new stuff\n:)</p>",
  "messages": [
    {
      "id": "633853",
      "postDate": "09/25/2019 13:51:15",
      "content": "<p>Hi everyone.\nI wanted to ask you Kagglers which metric you follow during training?</p>\n\n<p>Until now have followed the categorical cross entropy  loss of Guido's Kernel\n(<a href=\"https://www.kaggle.com/gzuidhof/reference-model\">https://www.kaggle.com/gzuidhof/reference-model</a>). However, early stopping on the best validation error yielded a significantly lower LB score.\nDo I actually need to follow a categorical loss? Or does it make more sense to follow a IoU?</p>\n\n<p>I am open for any suggestion and to learn new stuff\n:)</p>",
      "rawMarkdown": "Hi everyone.\nI wanted to ask you Kagglers which metric you follow during training?\n\nUntil now have followed the categorical cross entropy  loss of Guido's Kernel\n(https://www.kaggle.com/gzuidhof/reference-model). However, early stopping on the best validation error yielded a significantly lower LB score.\nDo I actually need to follow a categorical loss? Or does it make more sense to follow a IoU?\n\nI am open for any suggestion and to learn new stuff\n:)",
      "votes": null
    },
    {
      "id": "633863",
      "postDate": "09/25/2019 14:07:38",
      "content": "<p>I think default approach is to implement LB metric, monitor it during training and choose the best model based on it. In some cases though calculation of the full validation metric might be slow so you can use various surrogates (calculate it in a different/faster way which still correlates with target metric, calculate it on a smaller subset, etc.).\nOverall it's very typical that validation loss stops improving and is even rising, while validation metrics still improve significantly.</p>",
      "rawMarkdown": "I think default approach is to implement LB metric, monitor it during training and choose the best model based on it. In some cases though calculation of the full validation metric might be slow so you can use various surrogates (calculate it in a different/faster way which still correlates with target metric, calculate it on a smaller subset, etc.).\nOverall it's very typical that validation loss stops improving and is even rising, while validation metrics still improve significantly.",
      "votes": null
    },
    {
      "id": "633878",
      "postDate": "09/25/2019 14:31:13",
      "content": "<p>Thank you <a href=\"/lopuhin\">@lopuhin</a> for the insight.</p>",
      "rawMarkdown": "Thank you @lopuhin for the insight.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 633863,
      "author_name": "lopuhin",
      "author_url": "",
      "post_date": "09/25/2019 14:07:38",
      "content": "<p>I think default approach is to implement LB metric, monitor it during training and choose the best model based on it. In some cases though calculation of the full validation metric might be slow so you can use various surrogates (calculate it in a different/faster way which still correlates with target metric, calculate it on a smaller subset, etc.).\nOverall it's very typical that validation loss stops improving and is even rising, while validation metrics still improve significantly.</p>",
      "votes": null,
      "replies": [
        {
          "id": 633878,
          "author_name": "fanconic",
          "author_url": "",
          "post_date": "09/25/2019 14:31:13",
          "content": "<p>Thank you <a href=\"/lopuhin\">@lopuhin</a> for the insight.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "633853": "Hi everyone.\nI wanted to ask you Kagglers which metric you follow during training?\n\nUntil now have followed the categorical cross entropy  loss of Guido's Kernel\n(https://www.kaggle.com/gzuidhof/reference-model). However, early stopping on the best validation error yielded a significantly lower LB score.\nDo I actually need to follow a categorical loss? Or does it make more sense to follow a IoU?\n\nI am open for any suggestion and to learn new stuff\n:)",
    "633863": "I think default approach is to implement LB metric, monitor it during training and choose the best model based on it. In some cases though calculation of the full validation metric might be slow so you can use various surrogates (calculate it in a different/faster way which still correlates with target metric, calculate it on a smaller subset, etc.).\nOverall it's very typical that validation loss stops improving and is even rising, while validation metrics still improve significantly.",
    "633878": "Thank you @lopuhin for the insight."
  },
  "source": "meta"
}