{
  "id": 208990,
  "title": "Which metric to use for saving the models?",
  "url": "/competitions/rfcx-species-audio-detection/discussion/208990",
  "author_name": "",
  "post_date": "2021-01-05T20:44:19.314579800Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>I wonder that which metric do you use to save models during training? Like validation loss, correct prediction count or the competition metric LWLRAP. In my case, validation loss is somewhat misleading. Correct counts and the LWLRAP seem to be more robust.</p>",
  "messages": [
    {
      "id": "1140175",
      "postDate": "01/05/2021 20:44:19",
      "content": "<p>Hi everyone,</p>\n<p>I wonder that which metric do you use to save models during training? Like validation loss, correct prediction count or the competition metric LWLRAP. In my case, validation loss is somewhat misleading. Correct counts and the LWLRAP seem to be more robust.</p>",
      "rawMarkdown": "Hi everyone,\n\nI wonder that which metric do you use to save models during training? Like validation loss, correct prediction count or the competition metric LWLRAP. In my case, validation loss is somewhat misleading. Correct counts and the LWLRAP seem to be more robust.",
      "votes": null
    },
    {
      "id": "1140480",
      "postDate": "01/06/2021 03:21:08",
      "content": "<p>In my opinion, LWLRAP is the way to go. On changing the metrics the model is fed different signals for the back prop step. I find LWLRAP more inclined towards the competition and hence more robust in this use case. 😄</p>",
      "rawMarkdown": "In my opinion, LWLRAP is the way to go. On changing the metrics the model is fed different signals for the back prop step. I find LWLRAP more inclined towards the competition and hence more robust in this use case. :smile:",
      "votes": null
    },
    {
      "id": "1150043",
      "postDate": "01/12/2021 10:18:07",
      "content": "<p>Do you mean using LWLRAP as training loss function and do back prop on this?  Or you just use LWLRAP as validation loss and determine best epoch/model on this? Thanks</p>",
      "rawMarkdown": "Do you mean using LWLRAP as training loss function and do back prop on this?  Or you just use LWLRAP as validation loss and determine best epoch/model on this? Thanks",
      "votes": null
    },
    {
      "id": "1150062",
      "postDate": "01/12/2021 10:31:46",
      "content": "<p>you can't backprop LWLRAP</p>",
      "rawMarkdown": "you can't backprop LWLRAP",
      "votes": null
    },
    {
      "id": "1151012",
      "postDate": "01/13/2021 04:23:40",
      "content": "<p>Thanks <a href=\"https://www.kaggle.com/ilu000\" target=\"_blank\">@ilu000</a> , I tried use LWLRAP as my validation metric and looks now my training is better aligned with competition metric. But for training loss, which loss function do you think is best? BCEWithLogitsLoss, AUC?</p>",
      "rawMarkdown": "Thanks @ilu000 , I tried use LWLRAP as my validation metric and looks now my training is better aligned with competition metric. But for training loss, which loss function do you think is best? BCEWithLogitsLoss, AUC?",
      "votes": null
    },
    {
      "id": "1191184",
      "postDate": "02/08/2021 09:49:37",
      "content": "<p>A <strong>loss</strong> function should be <strong>differentiable</strong> in order to be able to calculate the gradients (i.e. do backprop) - AUC, LWLRAP are not, so it can be used only as <strong>metrics</strong></p>\n<blockquote>\n  <p>which loss function do you think is best?</p>\n</blockquote>\n<p>you can start with BCE, FocalLoss and variants amongst other</p>",
      "rawMarkdown": "A **loss** function should be **differentiable** in order to be able to calculate the gradients (i.e. do backprop) - AUC, LWLRAP are not, so it can be used only as **metrics**\n\n>which loss function do you think is best?\n\nyou can start with BCE, FocalLoss and variants amongst other",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1140480,
      "author_name": "aritrag",
      "author_url": "",
      "post_date": "01/06/2021 03:21:08",
      "content": "<p>In my opinion, LWLRAP is the way to go. On changing the metrics the model is fed different signals for the back prop step. I find LWLRAP more inclined towards the competition and hence more robust in this use case. 😄</p>",
      "votes": null,
      "replies": [
        {
          "id": 1150043,
          "author_name": "superchenhao",
          "author_url": "",
          "post_date": "01/12/2021 10:18:07",
          "content": "<p>Do you mean using LWLRAP as training loss function and do back prop on this?  Or you just use LWLRAP as validation loss and determine best epoch/model on this? Thanks</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1150062,
          "author_name": "ilu000",
          "author_url": "",
          "post_date": "01/12/2021 10:31:46",
          "content": "<p>you can't backprop LWLRAP</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1151012,
          "author_name": "superchenhao",
          "author_url": "",
          "post_date": "01/13/2021 04:23:40",
          "content": "<p>Thanks <a href=\"https://www.kaggle.com/ilu000\" target=\"_blank\">@ilu000</a> , I tried use LWLRAP as my validation metric and looks now my training is better aligned with competition metric. But for training loss, which loss function do you think is best? BCEWithLogitsLoss, AUC?</p>",
          "votes": null,
          "replies": [
            {
              "id": 1191184,
              "author_name": "imeintanis",
              "author_url": "",
              "post_date": "02/08/2021 09:49:37",
              "content": "<p>A <strong>loss</strong> function should be <strong>differentiable</strong> in order to be able to calculate the gradients (i.e. do backprop) - AUC, LWLRAP are not, so it can be used only as <strong>metrics</strong></p>\n<blockquote>\n  <p>which loss function do you think is best?</p>\n</blockquote>\n<p>you can start with BCE, FocalLoss and variants amongst other</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1140175": "Hi everyone,\n\nI wonder that which metric do you use to save models during training? Like validation loss, correct prediction count or the competition metric LWLRAP. In my case, validation loss is somewhat misleading. Correct counts and the LWLRAP seem to be more robust.",
    "1140480": "In my opinion, LWLRAP is the way to go. On changing the metrics the model is fed different signals for the back prop step. I find LWLRAP more inclined towards the competition and hence more robust in this use case. :smile:",
    "1150043": "Do you mean using LWLRAP as training loss function and do back prop on this?  Or you just use LWLRAP as validation loss and determine best epoch/model on this? Thanks",
    "1150062": "you can't backprop LWLRAP",
    "1151012": "Thanks @ilu000 , I tried use LWLRAP as my validation metric and looks now my training is better aligned with competition metric. But for training loss, which loss function do you think is best? BCEWithLogitsLoss, AUC?",
    "1191184": "A **loss** function should be **differentiable** in order to be able to calculate the gradients (i.e. do backprop) - AUC, LWLRAP are not, so it can be used only as **metrics**\n\n>which loss function do you think is best?\n\nyou can start with BCE, FocalLoss and variants amongst other"
  },
  "source": "meta"
}