Have some of you experienced this? When training my validation loss is increasing which is an indicator of overfitting however my AUC score is increasing and LB is also better.. What would that mean?
", "messages": [ { "id": 911784, "postDate": "2020-07-02T02:52:12.380Z", "content": "The problem is that loss is cross entropy which is different than AUC. So one could go down while the other goes down. We need to discover a loss that correlates better with AUC.
", "rawMarkdown": "The problem is that loss is cross entropy which is different than AUC. So one could go down while the other goes down. We need to discover a loss that correlates better with AUC.", "votes": 3, "replies": [ { "id": 911820, "postDate": "2020-07-02T03:28:02.583Z", "content": "Yes! Thanks for your answer :)
", "rawMarkdown": "Yes! Thanks for your answer :)" } ] }, { "id": 911026, "postDate": "2020-07-01T14:11:37.853Z", "content": "Have some of you experienced this? When training my validation loss is increasing which is an indicator of overfitting however my AUC score is increasing and LB is also better.. What would that mean?
", "rawMarkdown": "Have some of you experienced this? When training my validation loss is increasing which is an indicator of overfitting however my AUC score is increasing and LB is also better.. What would that mean?" }, { "id": 911051, "postDate": "2020-07-01T14:32:33.317Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 911064, "postDate": "2020-07-01T14:47:51.263Z", "content": "Thank you for your answer, this confirms what I thought! I think my CV is reliable, i removed duplicates and used groukfold, and my CV and LB scores dont have much gap.
\n\nWhat would be the best way to know if im overfitting if i cant really rely on validation loss?
", "rawMarkdown": "Thank you for your answer, this confirms what I thought! I think my CV is reliable, i removed duplicates and used groukfold, and my CV and LB scores dont have much gap. \n\nWhat would be the best way to know if im overfitting if i cant really rely on validation loss?" }, { "id": 911068, "postDate": "2020-07-01T14:50:02.800Z", "rawMarkdown": "", "isDeleted": true }, { "id": 911082, "postDate": "2020-07-01T14:58:33.813Z", "content": "You are right im not comparing training AUC vs validation AUC, thanks for the tips!
", "rawMarkdown": "You are right im not comparing training AUC vs validation AUC, thanks for the tips!" }, { "id": 911104, "postDate": "2020-07-01T15:10:04.733Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 911388, "postDate": "2020-07-01T17:39:05.727Z", "content": "@yannmajewski how much is your cv and lb score correlating?
", "rawMarkdown": "@yannmajewski how much is your cv and lb score correlating?\n" }, { "id": 911398, "postDate": "2020-07-01T17:47:39.637Z", "content": "Single model, one fold, only images i got CV 0.9329 and lb 0.928, its not perfect but with my current set up when i get a better CV I increase my lb. Note that i'm only testing with one fold because I don't have the compute and time to run 5 folds so there can be variations between folds.
\n\nHowever i just trained a similar model with meta data with CV 0.9371 and lLB 0.911.. Maybe overfitting on meta data? i don't really know anymore haha
", "rawMarkdown": "Single model, one fold, only images i got CV 0.9329 and lb 0.928, its not perfect but with my current set up when i get a better CV I increase my lb. Note that i'm only testing with one fold because I don't have the compute and time to run 5 folds so there can be variations between folds.\n\nHowever i just trained a similar model with meta data with CV 0.9371 and lLB 0.911.. Maybe overfitting on meta data? i don't really know anymore haha" }, { "id": 911791, "postDate": "2020-07-02T02:56:11.683Z", "content": "did you tried external data? from my experiments using external data improve cv but severely hurt the lb score 0.92-0.93 down to 0.86-0.87
", "rawMarkdown": "did you tried external data? from my experiments using external data improve cv but severely hurt the lb score 0.92-0.93 down to 0.86-0.87" }, { "id": 911821, "postDate": "2020-07-02T03:31:13.497Z", "content": "Yes with the isic 2019 dataset added! Not sure why adding the meta data in my model changed my lb that much but improved my CV
", "rawMarkdown": "Yes with the isic 2019 dataset added! Not sure why adding the meta data in my model changed my lb that much but improved my CV" } ] } ], "comments": [ { "id": 911784, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-07-02T02:52:12.380000", "content": "The problem is that loss is cross entropy which is different than AUC. So one could go down while the other goes down. We need to discover a loss that correlates better with AUC.
", "votes": 3, "replies": [ { "id": 911820, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-07-02T03:28:02.583000", "content": "Yes! Thanks for your answer :)
", "votes": 0, "replies": [] } ] }, { "id": 911051, "author_name": "", "author_url": "", "post_date": "2020-07-01T14:32:33.317000", "content": "", "votes": 0, "replies": [ { "id": 911064, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-07-01T14:47:51.263000", "content": "Thank you for your answer, this confirms what I thought! I think my CV is reliable, i removed duplicates and used groukfold, and my CV and LB scores dont have much gap.
\n\nWhat would be the best way to know if im overfitting if i cant really rely on validation loss?
", "votes": 0, "replies": [] }, { "id": 911068, "author_name": "", "author_url": "", "post_date": "2020-07-01T14:50:02.800000", "content": "", "votes": 0, "replies": [] }, { "id": 911082, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-07-01T14:58:33.813000", "content": "You are right im not comparing training AUC vs validation AUC, thanks for the tips!
", "votes": 0, "replies": [] }, { "id": 911104, "author_name": "", "author_url": "", "post_date": "2020-07-01T15:10:04.733000", "content": "", "votes": 1, "replies": [] }, { "id": 911388, "author_name": "MhdSharuk", "author_url": "", "post_date": "2020-07-01T17:39:05.727000", "content": "@yannmajewski how much is your cv and lb score correlating?
", "votes": 0, "replies": [] }, { "id": 911398, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-07-01T17:47:39.637000", "content": "Single model, one fold, only images i got CV 0.9329 and lb 0.928, its not perfect but with my current set up when i get a better CV I increase my lb. Note that i'm only testing with one fold because I don't have the compute and time to run 5 folds so there can be variations between folds.
\n\nHowever i just trained a similar model with meta data with CV 0.9371 and lLB 0.911.. Maybe overfitting on meta data? i don't really know anymore haha
", "votes": 0, "replies": [] }, { "id": 911791, "author_name": "DatNT", "author_url": "", "post_date": "2020-07-02T02:56:11.683000", "content": "did you tried external data? from my experiments using external data improve cv but severely hurt the lb score 0.92-0.93 down to 0.86-0.87
", "votes": 0, "replies": [] }, { "id": 911821, "author_name": "Yann Majewski", "author_url": "", "post_date": "2020-07-02T03:31:13.497000", "content": "Yes with the isic 2019 dataset added! Not sure why adding the meta data in my model changed my lb that much but improved my CV
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "911784": "The problem is that loss is cross entropy which is different than AUC. So one could go down while the other goes down. We need to discover a loss that correlates better with AUC.", "911026": "Have some of you experienced this? When training my validation loss is increasing which is an indicator of overfitting however my AUC score is increasing and LB is also better.. What would that mean?", "911051": "" } }