There are some discussion about label smoothing:
\nBy the way, I think label smoothing and two-stage training have similar motivations: samples with extreme votes lead models to predict with exaggerated confidence, resulting in worse performance. Label smoothing attempts to weaken these extreme samples. On the other hand, two-stage training tries to eliminate them in the 2nd stage.
\nPlease share your mind or experiments about label smoothing.
", "messages": [ { "id": "2655675", "postDate": "02/17/2024 06:10:27", "content": "There are some discussion about label smoothing:
\nBy the way, I think label smoothing and two-stage training have similar motivations: samples with extreme votes lead models to predict with exaggerated confidence, resulting in worse performance. Label smoothing attempts to weaken these extreme samples. On the other hand, two-stage training tries to eliminate them in the 2nd stage.
\nPlease share your mind or experiments about label smoothing.
", "rawMarkdown": "There are some discussion about label smoothing:\n1. [@djhuth](https://www.kaggle.com/djhuth) discusses it in [this discussion](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477554). And there seem to be some positive outcomes (better CV).\n2. [@nartaa](https://www.kaggle.com/nartaa) discusses it in [this comment](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135#2653865). It also seems to be able to improve CV with label smoothing.\n\nBy the way, I think label smoothing and [two-stage training](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461#2655654) have similar motivations: samples with extreme votes lead models to predict with exaggerated confidence, resulting in worse performance. Label smoothing attempts to weaken these extreme samples. On the other hand, two-stage training tries to eliminate them in the 2nd stage.\n\nPlease share your mind or experiments about label smoothing.", "votes": null }, { "id": "2655833", "postDate": "02/17/2024 08:28:10", "content": "In some of my experiments label smoothing helped with the CV and LB and in others it didn't - kinda odd but with every other change I keep on testing both variations. So far no system emerged. Happy to hear from others!
\nBest,
\nJan
@zijiangyang1116 I have conducted some experiments, more about this in this post
", "rawMarkdown": "zijiangyang1116 I have conducted some experiments, more about this in [this post](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477498)", "votes": null }, { "id": "2656370", "postDate": "02/17/2024 16:39:22", "content": "It kinda didn't work for me. The results weren't significant enough but it might work for you, so it's worth trying.
", "rawMarkdown": "It kinda didn't work for me. The results weren't significant enough but it might work for you, so it's worth trying.", "votes": null } ], "comments": [ { "id": 2655833, "author_name": "janbrederecke", "author_url": "", "post_date": "02/17/2024 08:28:10", "content": "In some of my experiments label smoothing helped with the CV and LB and in others it didn't - kinda odd but with every other change I keep on testing both variations. So far no system emerged. Happy to hear from others!
\nBest,
\nJan
@zijiangyang1116 I have conducted some experiments, more about this in this post
", "votes": null, "replies": [] }, { "id": 2656370, "author_name": "gunesevitan", "author_url": "", "post_date": "02/17/2024 16:39:22", "content": "It kinda didn't work for me. The results weren't significant enough but it might work for you, so it's worth trying.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2655675": "There are some discussion about label smoothing:\n1. [@djhuth](https://www.kaggle.com/djhuth) discusses it in [this discussion](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477554). And there seem to be some positive outcomes (better CV).\n2. [@nartaa](https://www.kaggle.com/nartaa) discusses it in [this comment](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477135#2653865). It also seems to be able to improve CV with label smoothing.\n\nBy the way, I think label smoothing and [two-stage training](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461#2655654) have similar motivations: samples with extreme votes lead models to predict with exaggerated confidence, resulting in worse performance. Label smoothing attempts to weaken these extreme samples. On the other hand, two-stage training tries to eliminate them in the 2nd stage.\n\nPlease share your mind or experiments about label smoothing.", "2655833": "In some of my experiments label smoothing helped with the CV and LB and in others it didn't - kinda odd but with every other change I keep on testing both variations. So far no system emerged. Happy to hear from others!\n\nBest,\nJan", "2656070": "zijiangyang1116 I have conducted some experiments, more about this in [this post](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477498)", "2656370": "It kinda didn't work for me. The results weren't significant enough but it might work for you, so it's worth trying." }, "source": "meta" }