{
  "id": 475748,
  "title": "Why these Submissions are evaluated on the Kullback Liebler divergence",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/475748",
  "author_name": "",
  "post_date": "2024-02-09T16:34:26.151656500Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello All, there are other loss functions, why specifically Kullback Liebler Divergence Loss Func'n in this competition ? </p>",
  "messages": [
    {
      "id": "2644738",
      "postDate": "02/09/2024 16:34:26",
      "content": "<p>Hello All, there are other loss functions, why specifically Kullback Liebler Divergence Loss Func'n in this competition ? </p>",
      "rawMarkdown": "Hello All, there are other loss functions, why specifically Kullback Liebler Divergence Loss Func'n in this competition ?",
      "votes": null
    },
    {
      "id": "2645061",
      "postDate": "02/09/2024 22:36:47",
      "content": "<p>Are you wondering why the competition hosts chose KL Divergence as the metric for this competition, or why folks are using KL divergence for their training? I'll provide my thoughts on both.</p>\n<p>If it's the former, I believe it's because experts don't always agree on which kind of event they are observing in the eeg and the competition hosts likely want the model to reflect uncertainty in the model. Essentially, they want the model to agree with the experts, which I think is reasonable.</p>\n<p>If it's the latter you're interested in, I believe it's because the metric being used for this competition also happens to be differentiable and a totally valid loss function, which is convenient. Why not train the model with the objective of doing as well with the competition metric as possible?</p>\n<p>I'm just getting started in this competition today, but those are my thoughts.</p>",
      "rawMarkdown": "Are you wondering why the competition hosts chose KL Divergence as the metric for this competition, or why folks are using KL divergence for their training? I'll provide my thoughts on both.\n\nIf it's the former, I believe it's because experts don't always agree on which kind of event they are observing in the eeg and the competition hosts likely want the model to reflect uncertainty in the model. Essentially, they want the model to agree with the experts, which I think is reasonable.\n\nIf it's the latter you're interested in, I believe it's because the metric being used for this competition also happens to be differentiable and a totally valid loss function, which is convenient. Why not train the model with the objective of doing as well with the competition metric as possible?\n\nI'm just getting started in this competition today, but those are my thoughts.",
      "votes": null
    },
    {
      "id": "2645064",
      "postDate": "02/09/2024 22:54:11",
      "content": "<p>If your question is why KL divergence and not something else such as multiclass logloss (to cite only this one), it's because the votes do not always reflect a consensus. Whenever the probability of seizure is 0.5, the probability of LPD is 0.35, and the probability of other is 0.15, we can't just resume the results to a single class, which is what multiclass logloss evaluates. In this competition, we are asked to replicate the distribution of experts' votes in all its complexity. Therefore, we need to use a metric which compares distributions:  the distribution of experts' votes, and our model's predicted probability distribution. KL divergence is one metric enabling to do so, but there are others (such as Jensen-Shannon divergence).</p>",
      "rawMarkdown": "If your question is why KL divergence and not something else such as multiclass logloss (to cite only this one), it's because the votes do not always reflect a consensus. Whenever the probability of seizure is 0.5, the probability of LPD is 0.35, and the probability of other is 0.15, we can't just resume the results to a single class, which is what multiclass logloss evaluates. In this competition, we are asked to replicate the distribution of experts' votes in all its complexity. Therefore, we need to use a metric which compares distributions:  the distribution of experts' votes, and our model's predicted probability distribution. KL divergence is one metric enabling to do so, but there are others (such as Jensen-Shannon divergence).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2645061,
      "author_name": "chemdatafarmer",
      "author_url": "",
      "post_date": "02/09/2024 22:36:47",
      "content": "<p>Are you wondering why the competition hosts chose KL Divergence as the metric for this competition, or why folks are using KL divergence for their training? I'll provide my thoughts on both.</p>\n<p>If it's the former, I believe it's because experts don't always agree on which kind of event they are observing in the eeg and the competition hosts likely want the model to reflect uncertainty in the model. Essentially, they want the model to agree with the experts, which I think is reasonable.</p>\n<p>If it's the latter you're interested in, I believe it's because the metric being used for this competition also happens to be differentiable and a totally valid loss function, which is convenient. Why not train the model with the objective of doing as well with the competition metric as possible?</p>\n<p>I'm just getting started in this competition today, but those are my thoughts.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2645064,
      "author_name": "fabienpv",
      "author_url": "",
      "post_date": "02/09/2024 22:54:11",
      "content": "<p>If your question is why KL divergence and not something else such as multiclass logloss (to cite only this one), it's because the votes do not always reflect a consensus. Whenever the probability of seizure is 0.5, the probability of LPD is 0.35, and the probability of other is 0.15, we can't just resume the results to a single class, which is what multiclass logloss evaluates. In this competition, we are asked to replicate the distribution of experts' votes in all its complexity. Therefore, we need to use a metric which compares distributions:  the distribution of experts' votes, and our model's predicted probability distribution. KL divergence is one metric enabling to do so, but there are others (such as Jensen-Shannon divergence).</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2644738": "Hello All, there are other loss functions, why specifically Kullback Liebler Divergence Loss Func'n in this competition ?",
    "2645061": "Are you wondering why the competition hosts chose KL Divergence as the metric for this competition, or why folks are using KL divergence for their training? I'll provide my thoughts on both.\n\nIf it's the former, I believe it's because experts don't always agree on which kind of event they are observing in the eeg and the competition hosts likely want the model to reflect uncertainty in the model. Essentially, they want the model to agree with the experts, which I think is reasonable.\n\nIf it's the latter you're interested in, I believe it's because the metric being used for this competition also happens to be differentiable and a totally valid loss function, which is convenient. Why not train the model with the objective of doing as well with the competition metric as possible?\n\nI'm just getting started in this competition today, but those are my thoughts.",
    "2645064": "If your question is why KL divergence and not something else such as multiclass logloss (to cite only this one), it's because the votes do not always reflect a consensus. Whenever the probability of seizure is 0.5, the probability of LPD is 0.35, and the probability of other is 0.15, we can't just resume the results to a single class, which is what multiclass logloss evaluates. In this competition, we are asked to replicate the distribution of experts' votes in all its complexity. Therefore, we need to use a metric which compares distributions:  the distribution of experts' votes, and our model's predicted probability distribution. KL divergence is one metric enabling to do so, but there are others (such as Jensen-Shannon divergence)."
  },
  "source": "meta"
}