So to test this, I am using @cdeotte notebook: https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43
\nSo basically the intuition is the more votes, the more weight it should have on training. There is no way that 1 voter is as accurate as the some 10000 voters that some of this data has. The problem is to determine how much more weight these greater voting sample size data should have. I don't think I have found anything on this other than here: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469666
\nI encourage people who have found anything on this to comment on their findings here, because I think this may help models, and its kinda fun.
\nSo anyways things that I have found:
\nSo the cv score for the original (no weight) is: 0.59015 (std: 0.0296)
\nWith log_128(voteCount + 1) weighting it drops to: 0.57349 (std: 0.0191) (quite stable)
\nWith ln(log_128(voteCount))/3+1: 0.57690 (std: 0.0301)
\n@caelhasse idea of:
\nalpha = 1 / (smoothing+np.sqrt(voteCount))
\n(1-alpha) + alpha/6
\nworked quite well with a cv score: 0.57803 (std: 0.0234) (smoothing = 1 for this one with going to 2 and 0.5 getting worse)
Um whatever I did here 1/(1+e^(-log_10(weights/((1.960.01)/0.1)^2)))1/0.9 (intuition sigmoid curve maybe clean it up): 0.58436 (std: 0.0286)
", "messages": [ { "id": "2668897", "postDate": "02/26/2024 01:07:30", "content": "So to test this, I am using @cdeotte notebook: https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43
\nSo basically the intuition is the more votes, the more weight it should have on training. There is no way that 1 voter is as accurate as the some 10000 voters that some of this data has. The problem is to determine how much more weight these greater voting sample size data should have. I don't think I have found anything on this other than here: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469666
\nI encourage people who have found anything on this to comment on their findings here, because I think this may help models, and its kinda fun.
\nSo anyways things that I have found:
\nSo the cv score for the original (no weight) is: 0.59015 (std: 0.0296)
\nWith log_128(voteCount + 1) weighting it drops to: 0.57349 (std: 0.0191) (quite stable)
\nWith ln(log_128(voteCount))/3+1: 0.57690 (std: 0.0301)
\n@caelhasse idea of:
\nalpha = 1 / (smoothing+np.sqrt(voteCount))
\n(1-alpha) + alpha/6
\nworked quite well with a cv score: 0.57803 (std: 0.0234) (smoothing = 1 for this one with going to 2 and 0.5 getting worse)
Um whatever I did here 1/(1+e^(-log_10(weights/((1.960.01)/0.1)^2)))1/0.9 (intuition sigmoid curve maybe clean it up): 0.58436 (std: 0.0286)
", "rawMarkdown": "So to test this, I am using @cdeotte notebook: https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43\n\nSo basically the intuition is the more votes, the more weight it should have on training. There is no way that 1 voter is as accurate as the some 10000 voters that some of this data has. The problem is to determine how much more weight these greater voting sample size data should have. I don't think I have found anything on this other than here: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469666\n\nI encourage people who have found anything on this to comment on their findings here, because I think this may help models, and its kinda fun.\n\nSo anyways things that I have found:\n\nSo the cv score for the original (no weight) is: **0.59015** (std: 0.0296)\n\nWith **log_128(voteCount + 1)** weighting it drops to: **0.57349** (std: 0.0191) (quite stable)\n\nWith **ln(log_128(voteCount))/3+1**: **0.57690** (std: 0.0301)\n\n@caelhasse idea of:\n**alpha = 1 / (smoothing+np.sqrt(voteCount))**\n**(1-alpha) + alpha/6**\nworked quite well with a cv score: **0.57803** (std: 0.0234) (smoothing = 1 for this one with going to 2 and 0.5 getting worse)\n\nUm whatever I did here 1/(1+e^(-log_10(weights/((1.96*0.01)/0.1)^2)))*1/0.9 (intuition sigmoid curve maybe clean it up): **0.58436** (std: 0.0286)", "votes": null }, { "id": "2668923", "postDate": "02/26/2024 02:25:45", "content": "The risk about your intuition is \"what did kaggle and the host do to build the test set\"
\nMy intuition is that the single vote was Dr. Lawrence Hirsch. (or another expert of similiar skills). You intuition would fit if a intern was assigned the job of rating some nice eeg's that were available to fill up the data set.
\nSo - the big question - does the test set have the same distribution of voters as the train?
", "rawMarkdown": "The risk about your intuition is \"what did kaggle and the host do to build the test set\"\n\nMy intuition is that the single vote was Dr. Lawrence Hirsch. (or another expert of similiar skills). You intuition would fit if a intern was assigned the job of rating some nice eeg's that were available to fill up the data set.\n\nSo - the big question - does the test set have the same distribution of voters as the train?", "votes": null }, { "id": "2668961", "postDate": "02/26/2024 03:15:39", "content": "Hmmm interesting, maybe that's why dropping cv does not really help the leaderboard. I still do think that more accurate voting should be weighted more, but I'm not sure. I'll probably just pick my best model and submit one with weighted training and another with unweighted (Since 2 can be selected). Still, as they say, always trust that cv score.
", "rawMarkdown": "Hmmm interesting, maybe that's why dropping cv does not really help the leaderboard. I still do think that more accurate voting should be weighted more, but I'm not sure. I'll probably just pick my best model and submit one with weighted training and another with unweighted (Since 2 can be selected). Still, as they say, always trust that cv score.", "votes": null }, { "id": "2669560", "postDate": "02/26/2024 10:35:21", "content": "You're right but that would be a real problem if the model should predict the true label of the patient. What has to predict is the probability distribution of many expert votations. Not?
", "rawMarkdown": "You're right but that would be a real problem if the model should predict the true label of the patient. What has to predict is the probability distribution of many expert votations. Not?", "votes": null }, { "id": "2670524", "postDate": "02/26/2024 23:29:30", "content": "I had an idea related to this that I posted elsewhere:
\nConsider the normalised votes as estimates of an underlying probability for a physician to vote for that class. The less votes the more uncertainty in that estimate; even though the estimate may be the most probable value it may not be the median value. When trying to predict out of sample normalised votes, it may be better to make a more conservative prediction.
\nI used the formula:
\nalpha = 1/(smoothing + np.sqrt(y_sum)) #y_sum being the number of votes
\ntrain[SMOOTH_TARGETS] = (1-alpha)*train[TARGETS] + alpha/6
where smoothing is some parameter (higher == less smoothing). In my tests I applied the label smoothing to the training labels and not the validation labels. So far I haven't found any benefits but maybe in the right context or with improvements, someone could get something out of it.
", "rawMarkdown": "I had an idea related to this that I posted elsewhere:\n\nConsider the normalised votes as estimates of an underlying probability for a physician to vote for that class. The less votes the more uncertainty in that estimate; even though the estimate may be the most probable value it may not be the median value. When trying to predict out of sample normalised votes, it may be better to make a more conservative prediction.\n\nI used the formula:\n\nalpha = 1/(smoothing + np.sqrt(y_sum)) #y_sum being the number of votes\ntrain[SMOOTH_TARGETS] = (1-alpha)*train[TARGETS] + alpha/6\n\nwhere smoothing is some parameter (higher == less smoothing). In my tests I applied the label smoothing to the training labels and not the validation labels. So far I haven't found any benefits but maybe in the right context or with improvements, someone could get something out of it.", "votes": null }, { "id": "2671601", "postDate": "02/27/2024 15:52:20", "content": "Thanks for sharing, keep up the good work, in my opinion, the more votes you have, the higher the quality of the data, that is, the more experts were able to give a verdict on that data
", "rawMarkdown": "Thanks for sharing, keep up the good work, in my opinion, the more votes you have, the higher the quality of the data, that is, the more experts were able to give a verdict on that data", "votes": null }, { "id": "2671958", "postDate": "02/27/2024 19:40:43", "content": "Very interesting, will try it now. I just graphed it in desmos and I definitely like how higher values stop scaling as much in weight.
\nEdit: I’m using a smoothing of 2 and just graphed the histogram and it has a very nice distribution (similar to the log one but in my opinion better). Going to try smoothing of 1 and 0 as well.
Your equation worked quite well with a smoothing value of 1, ending with a cv score: 0.57803 (std: 0.0234)
", "rawMarkdown": "Your equation worked quite well with a smoothing value of 1, ending with a cv score: 0.57803 (std: 0.0234)", "votes": null } ], "comments": [ { "id": 2668923, "author_name": "pcjimmmy", "author_url": "", "post_date": "02/26/2024 02:25:45", "content": "The risk about your intuition is \"what did kaggle and the host do to build the test set\"
\nMy intuition is that the single vote was Dr. Lawrence Hirsch. (or another expert of similiar skills). You intuition would fit if a intern was assigned the job of rating some nice eeg's that were available to fill up the data set.
\nSo - the big question - does the test set have the same distribution of voters as the train?
", "votes": null, "replies": [ { "id": 2668961, "author_name": "kawaiicoderuwu", "author_url": "", "post_date": "02/26/2024 03:15:39", "content": "Hmmm interesting, maybe that's why dropping cv does not really help the leaderboard. I still do think that more accurate voting should be weighted more, but I'm not sure. I'll probably just pick my best model and submit one with weighted training and another with unweighted (Since 2 can be selected). Still, as they say, always trust that cv score.
", "votes": null, "replies": [] }, { "id": 2669560, "author_name": "sacuscreed", "author_url": "", "post_date": "02/26/2024 10:35:21", "content": "You're right but that would be a real problem if the model should predict the true label of the patient. What has to predict is the probability distribution of many expert votations. Not?
", "votes": null, "replies": [] } ] }, { "id": 2670524, "author_name": "caelhasse", "author_url": "", "post_date": "02/26/2024 23:29:30", "content": "I had an idea related to this that I posted elsewhere:
\nConsider the normalised votes as estimates of an underlying probability for a physician to vote for that class. The less votes the more uncertainty in that estimate; even though the estimate may be the most probable value it may not be the median value. When trying to predict out of sample normalised votes, it may be better to make a more conservative prediction.
\nI used the formula:
\nalpha = 1/(smoothing + np.sqrt(y_sum)) #y_sum being the number of votes
\ntrain[SMOOTH_TARGETS] = (1-alpha)*train[TARGETS] + alpha/6
where smoothing is some parameter (higher == less smoothing). In my tests I applied the label smoothing to the training labels and not the validation labels. So far I haven't found any benefits but maybe in the right context or with improvements, someone could get something out of it.
", "votes": null, "replies": [ { "id": 2671958, "author_name": "kawaiicoderuwu", "author_url": "", "post_date": "02/27/2024 19:40:43", "content": "Very interesting, will try it now. I just graphed it in desmos and I definitely like how higher values stop scaling as much in weight.
\nEdit: I’m using a smoothing of 2 and just graphed the histogram and it has a very nice distribution (similar to the log one but in my opinion better). Going to try smoothing of 1 and 0 as well.
Your equation worked quite well with a smoothing value of 1, ending with a cv score: 0.57803 (std: 0.0234)
", "votes": null, "replies": [] } ] } ] }, { "id": 2671601, "author_name": "rafaelzimmermann1", "author_url": "", "post_date": "02/27/2024 15:52:20", "content": "Thanks for sharing, keep up the good work, in my opinion, the more votes you have, the higher the quality of the data, that is, the more experts were able to give a verdict on that data
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2668897": "So to test this, I am using @cdeotte notebook: https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43\n\nSo basically the intuition is the more votes, the more weight it should have on training. There is no way that 1 voter is as accurate as the some 10000 voters that some of this data has. The problem is to determine how much more weight these greater voting sample size data should have. I don't think I have found anything on this other than here: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469666\n\nI encourage people who have found anything on this to comment on their findings here, because I think this may help models, and its kinda fun.\n\nSo anyways things that I have found:\n\nSo the cv score for the original (no weight) is: **0.59015** (std: 0.0296)\n\nWith **log_128(voteCount + 1)** weighting it drops to: **0.57349** (std: 0.0191) (quite stable)\n\nWith **ln(log_128(voteCount))/3+1**: **0.57690** (std: 0.0301)\n\n@caelhasse idea of:\n**alpha = 1 / (smoothing+np.sqrt(voteCount))**\n**(1-alpha) + alpha/6**\nworked quite well with a cv score: **0.57803** (std: 0.0234) (smoothing = 1 for this one with going to 2 and 0.5 getting worse)\n\nUm whatever I did here 1/(1+e^(-log_10(weights/((1.96*0.01)/0.1)^2)))*1/0.9 (intuition sigmoid curve maybe clean it up): **0.58436** (std: 0.0286)", "2668923": "The risk about your intuition is \"what did kaggle and the host do to build the test set\"\n\nMy intuition is that the single vote was Dr. Lawrence Hirsch. (or another expert of similiar skills). You intuition would fit if a intern was assigned the job of rating some nice eeg's that were available to fill up the data set.\n\nSo - the big question - does the test set have the same distribution of voters as the train?", "2668961": "Hmmm interesting, maybe that's why dropping cv does not really help the leaderboard. I still do think that more accurate voting should be weighted more, but I'm not sure. I'll probably just pick my best model and submit one with weighted training and another with unweighted (Since 2 can be selected). Still, as they say, always trust that cv score.", "2669560": "You're right but that would be a real problem if the model should predict the true label of the patient. What has to predict is the probability distribution of many expert votations. Not?", "2670524": "I had an idea related to this that I posted elsewhere:\n\nConsider the normalised votes as estimates of an underlying probability for a physician to vote for that class. The less votes the more uncertainty in that estimate; even though the estimate may be the most probable value it may not be the median value. When trying to predict out of sample normalised votes, it may be better to make a more conservative prediction.\n\nI used the formula:\n\nalpha = 1/(smoothing + np.sqrt(y_sum)) #y_sum being the number of votes\ntrain[SMOOTH_TARGETS] = (1-alpha)*train[TARGETS] + alpha/6\n\nwhere smoothing is some parameter (higher == less smoothing). In my tests I applied the label smoothing to the training labels and not the validation labels. So far I haven't found any benefits but maybe in the right context or with improvements, someone could get something out of it.", "2671601": "Thanks for sharing, keep up the good work, in my opinion, the more votes you have, the higher the quality of the data, that is, the more experts were able to give a verdict on that data", "2671958": "Very interesting, will try it now. I just graphed it in desmos and I definitely like how higher values stop scaling as much in weight.\nEdit: I’m using a smoothing of 2 and just graphed the histogram and it has a very nice distribution (similar to the log one but in my opinion better). Going to try smoothing of 1 and 0 as well.", "2672145": "Your equation worked quite well with a smoothing value of 1, ending with a cv score: 0.57803 (std: 0.0234)" }, "source": "meta" }