![\"\"](\"https://raw.githubusercontent.com/iridiumblue/about/master/newplot.png\")
\nPaper: https://www.aaai.org/Papers/ICML/2003/ICML03-110.pdf
\nGithub: https://github.com/iridiumblue/roc-star
\nPaper: https://www.aaai.org/Papers/ICML/2003/ICML03-110.pdf
\nGithub: https://github.com/iridiumblue/roc-star
ROC-Star uses an approximation of the Wilcoxon-Mann-Whitney Statistic in order to train on AUC more directly than BCE. The paper achieved better AUC when comparing to BCE on average, that seems to be also the case from my experiment:
\nI've adapted the github repo above to my notebook below to run on my training loop on Pytorch. It also performed better than BCE as the paper suggested with the comparison below:
\nSame Params:
\nBCE: LB 0.829 (https://www.kaggle.com/capiru/g2net-starter-easy-training-model-gpu-tpu?scriptVersionId=67471237)
\nROC-Star: LB 0.836 (https://www.kaggle.com/capiru/g2net-roc-star-maximize-auc)
\nPaper: https://www.aaai.org/Papers/ICML/2003/ICML03-110.pdf
\nGithub: https://github.com/iridiumblue/roc-star
ROC-Star uses an approximation of the Wilcoxon-Mann-Whitney Statistic in order to train on AUC more directly than BCE. The paper achieved better AUC when comparing to BCE on average, that seems to be also the case from my experiment:
\nI've adapted the github repo above to my notebook below to run on my training loop on Pytorch. It also performed better than BCE as the paper suggested with the comparison below:
\nSame Params:
\nBCE: LB 0.829 (https://www.kaggle.com/capiru/g2net-starter-easy-training-model-gpu-tpu?scriptVersionId=67471237)
\nROC-Star: LB 0.836 (https://www.kaggle.com/capiru/g2net-roc-star-maximize-auc)
Does it take about twice longer to run, as it seems from your notebooks?
", "rawMarkdown": "Does it take about twice longer to run, as it seems from your notebooks?", "votes": null }, { "id": "1379532", "postDate": "07/07/2021 12:47:18", "content": "The main difference i found was that it could train for more epochs, after the first minimum with BCE it wouldn't improve anymore, but with ROC-Star, it manages to keep finding new minima.
\nEpoch by Epoch comparison is:
\nBCE 14 min per epoch
\nROC-Star 16 min per epoch (14% slower)
Got it, thanks!
", "rawMarkdown": "Got it, thanks!", "votes": null }, { "id": "1382207", "postDate": "07/09/2021 15:31:15", "content": "Thanks for sharing @capiru , is the ensemble between BCE and ROC Star working better that either or worse?
", "rawMarkdown": "Thanks for sharing @capiru , is the ensemble between BCE and ROC Star working better that either or worse?", "votes": null }, { "id": "1384761", "postDate": "07/12/2021 07:37:48", "content": "good job!
\nimage_size=640
\nROC-Star loss
\nepochs=4
\n1fold(5fold)
\ncv0.8685
\nlb0.871
thanks! seems that I can train 1-2 epochs more…
", "rawMarkdown": "thanks! seems that I can train 1-2 epochs more...", "votes": null }, { "id": "1384981", "postDate": "07/12/2021 11:29:27", "content": "Good compared to what? How much boost do you get form roc star loss?
", "rawMarkdown": "Good compared to what? How much boost do you get form roc star loss?", "votes": null }, { "id": "1384993", "postDate": "07/12/2021 11:40:52", "content": "@cpmpml want to compare with my image (640) of the same size. They are all 5fold. One uses ROC star loss, and the other is not used. Unfortunately, there is a small problem in the code. After running 1fold, CUDA out of memory. The result needs to wait until tomorrow.
", "rawMarkdown": "cpmpml want to compare with my image (640) of the same size. They are all 5fold. One uses ROC star loss, and the other is not used. Unfortunately, there is a small problem in the code. After running 1fold, CUDA out of memory. The result needs to wait until tomorrow.", "votes": null }, { "id": "1385900", "postDate": "07/13/2021 05:12:31", "content": "After modifying the code, the result is 0.872. It seems that this loss function is not very obvious
\nimage_size=640
\nROC-Star loss
\nepochs=4
\n5fold(5fold)
\ncv0.86828
\nlb0.872
When this loss function is used, the larger the image is, the smaller the improvement of CV is. Until 640, the improvement of CV is too small to change lb
", "rawMarkdown": "After modifying the code, the result is 0.872. It seems that this loss function is not very obvious\nimage_size=640\nROC-Star loss\nepochs=4\n5fold(5fold)\ncv0.86828\nlb0.872\n\nWhen this loss function is used, the larger the image is, the smaller the improvement of CV is. Until 640, the improvement of CV is too small to change lb", "votes": null }, { "id": "1391783", "postDate": "07/18/2021 04:45:53", "content": "Thanks a lot for sharing! How much longer do you need to train with this loss function?
", "rawMarkdown": "Thanks a lot for sharing! How much longer do you need to train with this loss function?", "votes": null }, { "id": "1400475", "postDate": "07/26/2021 10:16:09", "content": "Thanks for sharing! It worked for me! I didn't need longer time per epoch for learning.
", "rawMarkdown": "Thanks for sharing! It worked for me! I didn't need longer time per epoch for learning.", "votes": null }, { "id": "1561281", "postDate": "10/27/2021 13:12:06", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
Does it take about twice longer to run, as it seems from your notebooks?
", "votes": null, "replies": [ { "id": 1379532, "author_name": "capiru", "author_url": "", "post_date": "07/07/2021 12:47:18", "content": "The main difference i found was that it could train for more epochs, after the first minimum with BCE it wouldn't improve anymore, but with ROC-Star, it manages to keep finding new minima.
\nEpoch by Epoch comparison is:
\nBCE 14 min per epoch
\nROC-Star 16 min per epoch (14% slower)
Got it, thanks!
", "votes": null, "replies": [] }, { "id": 1384763, "author_name": "zhangeng", "author_url": "", "post_date": "07/12/2021 07:39:50", "content": "thanks! seems that I can train 1-2 epochs more…
", "votes": null, "replies": [] } ] }, { "id": 1382207, "author_name": "saurabhbagchi", "author_url": "", "post_date": "07/09/2021 15:31:15", "content": "Thanks for sharing @capiru , is the ensemble between BCE and ROC Star working better that either or worse?
", "votes": null, "replies": [] }, { "id": 1384761, "author_name": "zhangeng", "author_url": "", "post_date": "07/12/2021 07:37:48", "content": "good job!
\nimage_size=640
\nROC-Star loss
\nepochs=4
\n1fold(5fold)
\ncv0.8685
\nlb0.871
Good compared to what? How much boost do you get form roc star loss?
", "votes": null, "replies": [] }, { "id": 1384993, "author_name": "zhangeng", "author_url": "", "post_date": "07/12/2021 11:40:52", "content": "@cpmpml want to compare with my image (640) of the same size. They are all 5fold. One uses ROC star loss, and the other is not used. Unfortunately, there is a small problem in the code. After running 1fold, CUDA out of memory. The result needs to wait until tomorrow.
", "votes": null, "replies": [] }, { "id": 1385900, "author_name": "zhangeng", "author_url": "", "post_date": "07/13/2021 05:12:31", "content": "After modifying the code, the result is 0.872. It seems that this loss function is not very obvious
\nimage_size=640
\nROC-Star loss
\nepochs=4
\n5fold(5fold)
\ncv0.86828
\nlb0.872
When this loss function is used, the larger the image is, the smaller the improvement of CV is. Until 640, the improvement of CV is too small to change lb
", "votes": null, "replies": [] } ] }, { "id": 1391783, "author_name": "richx86", "author_url": "", "post_date": "07/18/2021 04:45:53", "content": "Thanks a lot for sharing! How much longer do you need to train with this loss function?
", "votes": null, "replies": [] }, { "id": 1400475, "author_name": "yoshimasaizaki", "author_url": "", "post_date": "07/26/2021 10:16:09", "content": "Thanks for sharing! It worked for me! I didn't need longer time per epoch for learning.
", "votes": null, "replies": [] }, { "id": 1561281, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 13:12:06", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris