Hi, I am new to Kaggle and I found the use of label smoothing in the binary cross-entropy loss function which smoothens the labels towards 0.5.
\n\nOn the internet, I found people claiming that it increases the accuracy somehow, can somebody please explain why does it happen and is it useful in an AUC score competition?
", "messages": [ { "id": "956104", "postDate": "08/03/2020 08:37:54", "content": "Hi, I am new to Kaggle and I found the use of label smoothing in the binary cross-entropy loss function which smoothens the labels towards 0.5.
\n\nOn the internet, I found people claiming that it increases the accuracy somehow, can somebody please explain why does it happen and is it useful in an AUC score competition?
", "rawMarkdown": "Hi, I am new to Kaggle and I found the use of label smoothing in the binary cross-entropy loss function which smoothens the labels towards 0.5. \n\nOn the internet, I found people claiming that it increases the accuracy somehow, can somebody please explain why does it happen and is it useful in an AUC score competition?", "votes": null }, { "id": "956547", "postDate": "08/03/2020 15:39:45", "content": "Label smoothing and weight decay is just regularization. it's very easy to overfit the loss function to the points in train/validation so adding a small regularization term keeps the model from overfitting. It's also a hyper parameter.
", "rawMarkdown": "Label smoothing and weight decay is just regularization. it's very easy to overfit the loss function to the points in train/validation so adding a small regularization term keeps the model from overfitting. It's also a hyper parameter.", "votes": null }, { "id": "956595", "postDate": "08/03/2020 16:06:44", "content": "Exactly think of label smoothing as a form of regularization. I've found this blog post Label Smoothing Explained using Microsoft Excel by @aroraaman really useful to get a intuition of what is happening behind the scenes.
", "rawMarkdown": "Exactly think of label smoothing as a form of regularization. I've found this blog post [Label Smoothing Explained using Microsoft Excel](https://amaarora.github.io/2020/07/18/label-smoothing.html) by @aroraaman really useful to get a intuition of what is happening behind the scenes.", "votes": null }, { "id": "956652", "postDate": "08/03/2020 17:06:19", "content": "I think you may find useful this discussion and blog by @aroraaman
\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168124\nhttps://amaarora.github.io/2020/07/18/label-smoothing.html#comparing-microsoft-excel-results-with-pytorch
\n\nPS Sorry, didn't see that Santiago has posted it already. But it really helped me to get the idea behind smoothing.
", "rawMarkdown": "I think you may find useful this discussion and blog by @aroraaman \n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168124\nhttps://amaarora.github.io/2020/07/18/label-smoothing.html#comparing-microsoft-excel-results-with-pytorch\n\nPS Sorry, didn't see that Santiago has posted it already. But it really helped me to get the idea behind smoothing.", "votes": null } ], "comments": [ { "id": 956547, "author_name": "teeyee314", "author_url": "", "post_date": "08/03/2020 15:39:45", "content": "Label smoothing and weight decay is just regularization. it's very easy to overfit the loss function to the points in train/validation so adding a small regularization term keeps the model from overfitting. It's also a hyper parameter.
", "votes": null, "replies": [ { "id": 956595, "author_name": "santiviquez", "author_url": "", "post_date": "08/03/2020 16:06:44", "content": "Exactly think of label smoothing as a form of regularization. I've found this blog post Label Smoothing Explained using Microsoft Excel by @aroraaman really useful to get a intuition of what is happening behind the scenes.
", "votes": null, "replies": [] } ] }, { "id": 956652, "author_name": "dunklerwald", "author_url": "", "post_date": "08/03/2020 17:06:19", "content": "I think you may find useful this discussion and blog by @aroraaman
\n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168124\nhttps://amaarora.github.io/2020/07/18/label-smoothing.html#comparing-microsoft-excel-results-with-pytorch
\n\nPS Sorry, didn't see that Santiago has posted it already. But it really helped me to get the idea behind smoothing.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "956104": "Hi, I am new to Kaggle and I found the use of label smoothing in the binary cross-entropy loss function which smoothens the labels towards 0.5. \n\nOn the internet, I found people claiming that it increases the accuracy somehow, can somebody please explain why does it happen and is it useful in an AUC score competition?", "956547": "Label smoothing and weight decay is just regularization. it's very easy to overfit the loss function to the points in train/validation so adding a small regularization term keeps the model from overfitting. It's also a hyper parameter.", "956595": "Exactly think of label smoothing as a form of regularization. I've found this blog post [Label Smoothing Explained using Microsoft Excel](https://amaarora.github.io/2020/07/18/label-smoothing.html) by @aroraaman really useful to get a intuition of what is happening behind the scenes.", "956652": "I think you may find useful this discussion and blog by @aroraaman \n\nhttps://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/168124\nhttps://amaarora.github.io/2020/07/18/label-smoothing.html#comparing-microsoft-excel-results-with-pytorch\n\nPS Sorry, didn't see that Santiago has posted it already. But it really helped me to get the idea behind smoothing." }, "source": "meta" }