{
  "id": 476587,
  "title": "Human factor",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/476587",
  "author_name": "",
  "post_date": "2024-02-12T23:06:27.056501600Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Ok. Someone has to ask that question, I guess. Maybe I understand it's all wrong, but I will be happy if that's the case.<br>\nConsider the event when there were 4 doctors and two of them <br>\nvoted for seizure and two for GPD</p>\n<p>votes:   2, 0, 2, 0, 0, 0<br>\nExpected result: 0.3935, 0.0533, 0.3935, 0.0533, 0.0533, 0.0533<br>\nNow. Out of nowhere comes the fifth doctor and he votes for GPD <br>\nnow <br>\nvotes: 2, 0, 3, 0, 0, 0<br>\nexpected result:  0.2348, 0.0318, 0.6382, 0.0318, 0.0318, 0.0318</p>\n<p>So the results are ~44% different, </p>\n<p>Yet the source data contains literally no information about the number of doctors.<br>\nLet's say the EEG and Spectrograms are inconclusive and suggest that there's around a 50% chance that it's a seizure and 50% that it's GPD, hence a 50/50 split in the votes. If we had 10000 votes, statistically, we would have half there and half there,  But 5 doctors are making it 60/40, and that difference doubles after the softmax is applied. And the fact that there will be 5th doctor is absolutely and utterly unpredictable, which is making maximum possible accuracy by my estimation less than 50%, which is laughable, and it sounds like a joke.</p>\n<p>Please tell me I am wrong.</p>",
  "messages": [
    {
      "id": "2649530",
      "postDate": "02/12/2024 23:06:27",
      "content": "<p>Ok. Someone has to ask that question, I guess. Maybe I understand it's all wrong, but I will be happy if that's the case.<br>\nConsider the event when there were 4 doctors and two of them <br>\nvoted for seizure and two for GPD</p>\n<p>votes:   2, 0, 2, 0, 0, 0<br>\nExpected result: 0.3935, 0.0533, 0.3935, 0.0533, 0.0533, 0.0533<br>\nNow. Out of nowhere comes the fifth doctor and he votes for GPD <br>\nnow <br>\nvotes: 2, 0, 3, 0, 0, 0<br>\nexpected result:  0.2348, 0.0318, 0.6382, 0.0318, 0.0318, 0.0318</p>\n<p>So the results are ~44% different, </p>\n<p>Yet the source data contains literally no information about the number of doctors.<br>\nLet's say the EEG and Spectrograms are inconclusive and suggest that there's around a 50% chance that it's a seizure and 50% that it's GPD, hence a 50/50 split in the votes. If we had 10000 votes, statistically, we would have half there and half there,  But 5 doctors are making it 60/40, and that difference doubles after the softmax is applied. And the fact that there will be 5th doctor is absolutely and utterly unpredictable, which is making maximum possible accuracy by my estimation less than 50%, which is laughable, and it sounds like a joke.</p>\n<p>Please tell me I am wrong.</p>",
      "rawMarkdown": "Ok. Someone has to ask that question, I guess. Maybe I understand it's all wrong, but I will be happy if that's the case.\nConsider the event when there were 4 doctors and two of them \nvoted for seizure and two for GPD\n\nvotes:   2, 0, 2, 0, 0, 0\nExpected result: 0.3935, 0.0533, 0.3935, 0.0533, 0.0533, 0.0533\nNow. Out of nowhere comes the fifth doctor and he votes for GPD \nnow \nvotes: 2, 0, 3, 0, 0, 0\nexpected result:  0.2348, 0.0318, 0.6382, 0.0318, 0.0318, 0.0318\n\nSo the results are ~44% different, \n\nYet the source data contains literally no information about the number of doctors.\nLet's say the EEG and Spectrograms are inconclusive and suggest that there's around a 50% chance that it's a seizure and 50% that it's GPD, hence a 50/50 split in the votes. If we had 10000 votes, statistically, we would have half there and half there,  But 5 doctors are making it 60/40, and that difference doubles after the softmax is applied. And the fact that there will be 5th doctor is absolutely and utterly unpredictable, which is making maximum possible accuracy by my estimation less than 50%, which is laughable, and it sounds like a joke.\n\nPlease tell me I am wrong.",
      "votes": null
    },
    {
      "id": "2649541",
      "postDate": "02/12/2024 23:26:13",
      "content": "<p>The ground truth is <strong>not</strong> created using softmax. Kaggle said <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468705#2606605\" target=\"_blank\">here</a> that the ground truth is created by dividing by the total number of counts. So <code>[2, 0, 2, 0, 0, 0]</code> has a ground truth of <code>[2/4, 0, 2/4, 0, 0, 0]</code>.</p>",
      "rawMarkdown": "The ground truth is **not** created using softmax. Kaggle said [here][1] that the ground truth is created by dividing by the total number of counts. So `[2, 0, 2, 0, 0, 0]` has a ground truth of `[2/4, 0, 2/4, 0, 0, 0]`.\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468705#2606605",
      "votes": null
    },
    {
      "id": "2649545",
      "postDate": "02/12/2024 23:57:25",
      "content": "<p>So it will be <br>\n0.5, 0, 0.5, 0, 0, 0   vs   0.4, 0, 0.6, 0, 0. 0   <br>\nIt doesn't help, there is still a 35% difference after Softmax. It is still less than 50% maximum possible accuracy, and the Founder effect is ridiculously high.    </p>",
      "rawMarkdown": "So it will be \n0.5, 0, 0.5, 0, 0, 0   vs   0.4, 0, 0.6, 0, 0. 0   \nIt doesn't help, there is still a 35% difference after Softmax. It is still less than 50% maximum possible accuracy, and the Founder effect is ridiculously high.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2649541,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "02/12/2024 23:26:13",
      "content": "<p>The ground truth is <strong>not</strong> created using softmax. Kaggle said <a href=\"https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468705#2606605\" target=\"_blank\">here</a> that the ground truth is created by dividing by the total number of counts. So <code>[2, 0, 2, 0, 0, 0]</code> has a ground truth of <code>[2/4, 0, 2/4, 0, 0, 0]</code>.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2649545,
          "author_name": "vladimirgolendukhin",
          "author_url": "",
          "post_date": "02/12/2024 23:57:25",
          "content": "<p>So it will be <br>\n0.5, 0, 0.5, 0, 0, 0   vs   0.4, 0, 0.6, 0, 0. 0   <br>\nIt doesn't help, there is still a 35% difference after Softmax. It is still less than 50% maximum possible accuracy, and the Founder effect is ridiculously high.    </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2649530": "Ok. Someone has to ask that question, I guess. Maybe I understand it's all wrong, but I will be happy if that's the case.\nConsider the event when there were 4 doctors and two of them \nvoted for seizure and two for GPD\n\nvotes:   2, 0, 2, 0, 0, 0\nExpected result: 0.3935, 0.0533, 0.3935, 0.0533, 0.0533, 0.0533\nNow. Out of nowhere comes the fifth doctor and he votes for GPD \nnow \nvotes: 2, 0, 3, 0, 0, 0\nexpected result:  0.2348, 0.0318, 0.6382, 0.0318, 0.0318, 0.0318\n\nSo the results are ~44% different, \n\nYet the source data contains literally no information about the number of doctors.\nLet's say the EEG and Spectrograms are inconclusive and suggest that there's around a 50% chance that it's a seizure and 50% that it's GPD, hence a 50/50 split in the votes. If we had 10000 votes, statistically, we would have half there and half there,  But 5 doctors are making it 60/40, and that difference doubles after the softmax is applied. And the fact that there will be 5th doctor is absolutely and utterly unpredictable, which is making maximum possible accuracy by my estimation less than 50%, which is laughable, and it sounds like a joke.\n\nPlease tell me I am wrong.",
    "2649541": "The ground truth is **not** created using softmax. Kaggle said [here][1] that the ground truth is created by dividing by the total number of counts. So `[2, 0, 2, 0, 0, 0]` has a ground truth of `[2/4, 0, 2/4, 0, 0, 0]`.\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468705#2606605",
    "2649545": "So it will be \n0.5, 0, 0.5, 0, 0, 0   vs   0.4, 0, 0.6, 0, 0. 0   \nIt doesn't help, there is still a 35% difference after Softmax. It is still less than 50% maximum possible accuracy, and the Founder effect is ridiculously high."
  },
  "source": "meta"
}