Let's share CV and LB scores here.
\nMy CV is 0.85. Will update with LB score soon :)
\nUPDATE: LB 0.90 . My CV calculation is erroneous ;)
", "messages": [ { "id": "1127461", "postDate": "12/26/2020 15:01:21", "content": "Let's share CV and LB scores here.
\nMy CV is 0.85. Will update with LB score soon :)
\nUPDATE: LB 0.90 . My CV calculation is erroneous ;)
", "rawMarkdown": "Let's share CV and LB scores here. \n\nMy CV is 0.85. Will update with LB score soon :) \n\nUPDATE: LB 0.90 . My CV calculation is erroneous ;)", "votes": null }, { "id": "1187734", "postDate": "02/05/2021 17:05:15", "content": "My worst discrepancy is 0.965 CV and 0.942 LB.
", "rawMarkdown": "My worst discrepancy is 0.965 CV and 0.942 LB.", "votes": null }, { "id": "1218011", "postDate": "02/25/2021 13:38:39", "content": "How do you deal with it after that ??? My single fold 0 could reach 0.967 cv, but it has a 0.958 lb, and a 0.947 cv has 0.960 lb which is much higher than the 0.967 guy… Such amazing.
", "rawMarkdown": "How do you deal with it after that ??? My single fold 0 could reach 0.967 cv, but it has a 0.958 lb, and a 0.947 cv has 0.960 lb which is much higher than the 0.967 guy... Such amazing.", "votes": null }, { "id": "1218178", "postDate": "02/25/2021 16:12:49", "content": "Depends on how good is your CV strategy. If your CV strategy is solid and you know there is no data leakage (because you have solid CV strategy), then trust CV. Trusting CV is the only thing that make senses, LB is just part of the entire testing data.
\nYou wouldn't have LB in real-life application.
", "rawMarkdown": "Depends on how good is your CV strategy. If your CV strategy is solid and you know there is no data leakage (because you have solid CV strategy), then trust CV. Trusting CV is the only thing that make senses, LB is just part of the entire testing data.\n\nYou wouldn't have LB in real-life application.", "votes": null }, { "id": "1218201", "postDate": "02/25/2021 16:35:31", "content": "Tks for your kindly reply!.
\nFor me I trust my cv most. I am wondering why my model cv 5 fold 0.955 could get a 0.965 score, but a model 5 fold cv is 0.962 can only get a same lb score. The cv strategy is totally same. I just feel confused because they are just using same folds and parameter. I just change the learning rate strategy….. Maybe I am stuck in my local minimum. Maybe I should try to deal with CVC problems first. May scores boost after I get up~~~
CV is 95.2 and LB is 95.6.
", "rawMarkdown": "CV is 95.2 and LB is 95.6.", "votes": null }, { "id": "1227207", "postDate": "03/05/2021 09:55:09", "content": "@fanwenping This is my first time training a large model and I am experiencing the same thing. Choosing a small learning rate makes my single-fold-Resnet200d's CV higher (0.955 to 0.960), but its LB gets lower (0.963 to 0.958). Have you found the reason? I know that the generalization of models becomes worse near some local minimums, but why is the CV still rising?
", "rawMarkdown": "fanwenping This is my first time training a large model and I am experiencing the same thing. Choosing a small learning rate makes my single-fold-Resnet200d's CV higher (0.955 to 0.960), but its LB gets lower (0.963 to 0.958). Have you found the reason? I know that the generalization of models becomes worse near some local minimums, but why is the CV still rising?", "votes": null }, { "id": "1227219", "postDate": "03/05/2021 10:07:36", "content": "I think overfitting in this game is useful, maybe…..
", "rawMarkdown": "I think overfitting in this game is useful, maybe.....", "votes": null } ], "comments": [ { "id": 1187734, "author_name": "jy2tong", "author_url": "", "post_date": "02/05/2021 17:05:15", "content": "My worst discrepancy is 0.965 CV and 0.942 LB.
", "votes": null, "replies": [ { "id": 1218011, "author_name": "fanwenping", "author_url": "", "post_date": "02/25/2021 13:38:39", "content": "How do you deal with it after that ??? My single fold 0 could reach 0.967 cv, but it has a 0.958 lb, and a 0.947 cv has 0.960 lb which is much higher than the 0.967 guy… Such amazing.
", "votes": null, "replies": [] }, { "id": 1218178, "author_name": "jy2tong", "author_url": "", "post_date": "02/25/2021 16:12:49", "content": "Depends on how good is your CV strategy. If your CV strategy is solid and you know there is no data leakage (because you have solid CV strategy), then trust CV. Trusting CV is the only thing that make senses, LB is just part of the entire testing data.
\nYou wouldn't have LB in real-life application.
", "votes": null, "replies": [] }, { "id": 1218201, "author_name": "fanwenping", "author_url": "", "post_date": "02/25/2021 16:35:31", "content": "Tks for your kindly reply!.
\nFor me I trust my cv most. I am wondering why my model cv 5 fold 0.955 could get a 0.965 score, but a model 5 fold cv is 0.962 can only get a same lb score. The cv strategy is totally same. I just feel confused because they are just using same folds and parameter. I just change the learning rate strategy….. Maybe I am stuck in my local minimum. Maybe I should try to deal with CVC problems first. May scores boost after I get up~~~
@fanwenping This is my first time training a large model and I am experiencing the same thing. Choosing a small learning rate makes my single-fold-Resnet200d's CV higher (0.955 to 0.960), but its LB gets lower (0.963 to 0.958). Have you found the reason? I know that the generalization of models becomes worse near some local minimums, but why is the CV still rising?
", "votes": null, "replies": [] }, { "id": 1227219, "author_name": "fanwenping", "author_url": "", "post_date": "03/05/2021 10:07:36", "content": "I think overfitting in this game is useful, maybe…..
", "votes": null, "replies": [] } ] }, { "id": 1218274, "author_name": "vickygoyal", "author_url": "", "post_date": "02/25/2021 17:19:27", "content": "CV is 95.2 and LB is 95.6.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1127461": "Let's share CV and LB scores here. \n\nMy CV is 0.85. Will update with LB score soon :) \n\nUPDATE: LB 0.90 . My CV calculation is erroneous ;)", "1187734": "My worst discrepancy is 0.965 CV and 0.942 LB.", "1218011": "How do you deal with it after that ??? My single fold 0 could reach 0.967 cv, but it has a 0.958 lb, and a 0.947 cv has 0.960 lb which is much higher than the 0.967 guy... Such amazing.", "1218178": "Depends on how good is your CV strategy. If your CV strategy is solid and you know there is no data leakage (because you have solid CV strategy), then trust CV. Trusting CV is the only thing that make senses, LB is just part of the entire testing data.\n\nYou wouldn't have LB in real-life application.", "1218201": "Tks for your kindly reply!.\nFor me I trust my cv most. I am wondering why my model cv 5 fold 0.955 could get a 0.965 score, but a model 5 fold cv is 0.962 can only get a same lb score. The cv strategy is totally same. I just feel confused because they are just using same folds and parameter. I just change the learning rate strategy..... Maybe I am stuck in my local minimum. Maybe I should try to deal with CVC problems first. May scores boost after I get up~~~", "1218274": "CV is 95.2 and LB is 95.6.", "1227207": "fanwenping This is my first time training a large model and I am experiencing the same thing. Choosing a small learning rate makes my single-fold-Resnet200d's CV higher (0.955 to 0.960), but its LB gets lower (0.963 to 0.958). Have you found the reason? I know that the generalization of models becomes worse near some local minimums, but why is the CV still rising?", "1227219": "I think overfitting in this game is useful, maybe....." }, "source": "meta" }