{
  "id": 481584,
  "title": "Is there a way to know ultimate performance?",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/481584",
  "author_name": "",
  "post_date": "2024-03-04T08:43:37.153566400Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>For example if the classifier always found the same top results as the experts, this would be a pretty good result.<br>\nCan I take the cv and understand what percentage of the time that the correct choice was made? Is there a way to say that there is a certain cv that represents the classifier performance limit?<br>\nAt some level, experts and classifier performance is noisy.</p>",
  "messages": [
    {
      "id": "2680504",
      "postDate": "03/04/2024 08:43:37",
      "content": "<p>For example if the classifier always found the same top results as the experts, this would be a pretty good result.<br>\nCan I take the cv and understand what percentage of the time that the correct choice was made? Is there a way to say that there is a certain cv that represents the classifier performance limit?<br>\nAt some level, experts and classifier performance is noisy.</p>",
      "rawMarkdown": "For example if the classifier always found the same top results as the experts, this would be a pretty good result.\nCan I take the cv and understand what percentage of the time that the correct choice was made? Is there a way to say that there is a certain cv that represents the classifier performance limit?\nAt some level, experts and classifier performance is noisy.",
      "votes": null
    },
    {
      "id": "2681030",
      "postDate": "03/04/2024 12:56:05",
      "content": "<p>I guess it depends on what you mean by \"correct choice\". We're not predicting the class, we're predicting the distribution of classes. If you mean predicting the distribution perfectly, the KLDivergence loss would simply be 0 and that's how you'd know you have a perfect model. </p>\n<p>If you're wondering \"what's the best I could feasibly do given that experts don't agree\" perhaps you could use the current distributions as an assumed probability and you could generate 30 or so \"random\" guesses for each sample using a probability mass function, then check the average KLDivergence loss across this simulated data…I might try this to see how it looks and publish a notebook on it it it's interesting.</p>",
      "rawMarkdown": "I guess it depends on what you mean by \"correct choice\". We're not predicting the class, we're predicting the distribution of classes. If you mean predicting the distribution perfectly, the KLDivergence loss would simply be 0 and that's how you'd know you have a perfect model. \n\nIf you're wondering \"what's the best I could feasibly do given that experts don't agree\" perhaps you could use the current distributions as an assumed probability and you could generate 30 or so \"random\" guesses for each sample using a probability mass function, then check the average KLDivergence loss across this simulated data...I might try this to see how it looks and publish a notebook on it it it's interesting.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2681030,
      "author_name": "chemdatafarmer",
      "author_url": "",
      "post_date": "03/04/2024 12:56:05",
      "content": "<p>I guess it depends on what you mean by \"correct choice\". We're not predicting the class, we're predicting the distribution of classes. If you mean predicting the distribution perfectly, the KLDivergence loss would simply be 0 and that's how you'd know you have a perfect model. </p>\n<p>If you're wondering \"what's the best I could feasibly do given that experts don't agree\" perhaps you could use the current distributions as an assumed probability and you could generate 30 or so \"random\" guesses for each sample using a probability mass function, then check the average KLDivergence loss across this simulated data…I might try this to see how it looks and publish a notebook on it it it's interesting.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2680504": "For example if the classifier always found the same top results as the experts, this would be a pretty good result.\nCan I take the cv and understand what percentage of the time that the correct choice was made? Is there a way to say that there is a certain cv that represents the classifier performance limit?\nAt some level, experts and classifier performance is noisy.",
    "2681030": "I guess it depends on what you mean by \"correct choice\". We're not predicting the class, we're predicting the distribution of classes. If you mean predicting the distribution perfectly, the KLDivergence loss would simply be 0 and that's how you'd know you have a perfect model. \n\nIf you're wondering \"what's the best I could feasibly do given that experts don't agree\" perhaps you could use the current distributions as an assumed probability and you could generate 30 or so \"random\" guesses for each sample using a probability mass function, then check the average KLDivergence loss across this simulated data...I might try this to see how it looks and publish a notebook on it it it's interesting."
  },
  "source": "meta"
}