The discussion about using leave-one-quake-out cross validation and early stopping started in this thread when @cpmpml attached his oof plot. I was intrigued that he is using early stopping in quake-wise CV and getting good results. At least when I tried it, my LB score was > 1.6 due to pre-mature stopping resulting from leakage. Others like @philippsinger shared my frustration. Even @cpmpml started doubting himself and started this thread about early stopping where many had very good arguments about leakage, but @cpmpml almost could swear his model is not leaking.
\n\nSo I investigated the issue and figured out the reason for this discrepancy. Those who are suffering from leak when using this strategy are probably using catboost (at least me). Others are using LGB. \nIt turned out that catboost initialize the boosting with zeros. In small quakes, the error in the validation set reaches a minimum very fast before convergence because the values start increasing from zero. However, LGB has a parameter boost_from_average, which is true by default. Therefore, LGB doesn't have the same problem.
To test, set boost_from_average to false, and the CV score will drop ~0.2 due to leakage. I couldn't find a parameter in catboost that controls this behavior.
So to conclude, if you want to use quake-wise CV, switch to LGB. That doesn't exclude the possibility of leakage, but at least it won't be as severe as we have experienced. \nI don't know yet how that will affect my LB score, but fingers crossed.
\n\nUpdate: after trying quake-wise CV with early stopping in LGB (boostfromaverage = true), the leak is still there. Not as it was with catboost, but it is still severely under/over-fitting. \nSo I am sticking with shuffling, at least my models stop optimally in this setting.
", "messages": [ { "id": "527521", "postDate": "05/05/2019 18:10:56", "content": "The discussion about using leave-one-quake-out cross validation and early stopping started in this thread when @cpmpml attached his oof plot. I was intrigued that he is using early stopping in quake-wise CV and getting good results. At least when I tried it, my LB score was > 1.6 due to pre-mature stopping resulting from leakage. Others like @philippsinger shared my frustration. Even @cpmpml started doubting himself and started this thread about early stopping where many had very good arguments about leakage, but @cpmpml almost could swear his model is not leaking.
\n\nSo I investigated the issue and figured out the reason for this discrepancy. Those who are suffering from leak when using this strategy are probably using catboost (at least me). Others are using LGB. \nIt turned out that catboost initialize the boosting with zeros. In small quakes, the error in the validation set reaches a minimum very fast before convergence because the values start increasing from zero. However, LGB has a parameter boost_from_average, which is true by default. Therefore, LGB doesn't have the same problem.
To test, set boost_from_average to false, and the CV score will drop ~0.2 due to leakage. I couldn't find a parameter in catboost that controls this behavior.
So to conclude, if you want to use quake-wise CV, switch to LGB. That doesn't exclude the possibility of leakage, but at least it won't be as severe as we have experienced. \nI don't know yet how that will affect my LB score, but fingers crossed.
\n\nUpdate: after trying quake-wise CV with early stopping in LGB (boostfromaverage = true), the leak is still there. Not as it was with catboost, but it is still severely under/over-fitting. \nSo I am sticking with shuffling, at least my models stop optimally in this setting.
", "rawMarkdown": "The discussion about using leave-one-quake-out cross validation and early stopping started in [this thread](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/91125#latest-527210) when @cpmpml attached his oof plot. I was intrigued that he is using early stopping in quake-wise CV and getting good results. At least when I tried it, my LB score was > 1.6 due to pre-mature stopping resulting from leakage. Others like @philippsinger shared my frustration. Even @cpmpml started doubting himself and started [this thread about early stopping](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/91328#latest-527457) where many had very good arguments about leakage, but @cpmpml almost could swear his model is not leaking.\n\nSo I investigated the issue and figured out the reason for this discrepancy. Those who are suffering from leak when using this strategy are probably using catboost (at least me). Others are using LGB. \nIt turned out that catboost initialize the boosting with zeros. In small quakes, the error in the validation set reaches a minimum very fast before convergence because the values start increasing from zero. However, LGB has a parameter `boost_from_average`, which is true by default. Therefore, LGB doesn't have the same problem.\n\nTo test, set `boost_from_average` to false, and the CV score will drop ~0.2 due to leakage. I couldn't find a parameter in catboost that controls this behavior.\n\nSo to conclude, if you want to use quake-wise CV, switch to LGB. That doesn't exclude the possibility of leakage, but at least it won't be as severe as we have experienced. \nI don't know yet how that will affect my LB score, but fingers crossed. \n\nUpdate: after trying quake-wise CV with early stopping in LGB (boostfromaverage = true), the leak is still there. Not as it was with catboost, but it is still severely under/over-fitting. \nSo I am sticking with shuffling, at least my models stop optimally in this setting.", "votes": null }, { "id": "527522", "postDate": "05/05/2019 18:16:31", "content": "\n\n\nhe is using early stopping in quake-wise CV
\n
I am not doing that. I even said it explicitly in the discussion you cite ;)
", "rawMarkdown": "> he is using early stopping in quake-wise CV \n\nI am not doing that. I even said it explicitly in the discussion you cite ;)", "votes": null }, { "id": "527525", "postDate": "05/05/2019 18:21:17", "content": "Sorry for misunderstanding then. I guess this was only my assumption because I could only reproduce a similar pattern to your oof with early stopping.
", "rawMarkdown": "Sorry for misunderstanding then. I guess this was only my assumption because I could only reproduce a similar pattern to your oof with early stopping.", "votes": null }, { "id": "527531", "postDate": "05/05/2019 18:28:48", "content": "What is right in your post is that lgb has many parameters, and some can have a significant effect like the one you single out.
", "rawMarkdown": "What is right in your post is that lgb has many parameters, and some can have a significant effect like the one you single out.", "votes": null }, { "id": "527533", "postDate": "05/05/2019 18:37:54", "content": "Everything is right about my post, except my assumption about your secret fancy strategy ;)
", "rawMarkdown": "Everything is right about my post, except my assumption about your secret fancy strategy ;)", "votes": null }, { "id": "527568", "postDate": "05/05/2019 20:40:30", "content": "Very interesting. Thanks for posting this insight. Is it safe to say that XGBoost would exhibit the same behavior as catboost? Quick google search found xgboost doesn't have boost_from_average implemented yet but is being discussed on their github: https://github.com/dmlc/xgboost/issues/4321
I haven't tried XGBoost in this competition yet. I'll post an update when I try it.
", "rawMarkdown": "I haven't tried XGBoost in this competition yet. I'll post an update when I try it.", "votes": null }, { "id": "527583", "postDate": "05/05/2019 21:46:30", "content": "You can use base_score parameter to starts from whatever value you want with XGBoost. Setting it to target average gives you the behavior you discuss.
", "rawMarkdown": "You can use base_score parameter to starts from whatever value you want with XGBoost. Setting it to target average gives you the behavior you discuss.", "votes": null }, { "id": "527584", "postDate": "05/05/2019 21:49:19", "content": "It's all about the mean... What is funny BTW is that boost_from_average behaves differently between l1 and gamma loss.
Ehm... help? Setting 'boost_from_average': 'false', in the parameters causes the best iteration for each fold to be iteration 1, and overall CV jumps to 5.679
That's odd. It will stop prematurely but not like that.
", "rawMarkdown": "That's odd. It will stop prematurely but not like that.", "votes": null }, { "id": "528249", "postDate": "05/07/2019 10:50:34", "content": "Update: after trying quake-wise CV with early stopping in LGB (boost_from_average = true), the leak is still there. Not as it was with catboost, but it is still severely under/over-fitting. \n So I am sticking with shuffling, at least my models stop optimally in this setting.
", "rawMarkdown": "Update: after trying quake-wise CV with early stopping in LGB (boost_from_average = true), the leak is still there. Not as it was with catboost, but it is still severely under/over-fitting. \n So I am sticking with shuffling, at least my models stop optimally in this setting.", "votes": null }, { "id": "528252", "postDate": "05/07/2019 11:08:19", "content": "I have the same with objective mae.
", "rawMarkdown": "I have the same with objective mae.", "votes": null }, { "id": "528282", "postDate": "05/07/2019 12:20:15", "content": "Well, how do we avoid this error? I am also using objective=mean_absolute_error. Are you using gamma loss instead of mae ? Or maybe, the parameters need to be retuned.
> Well, how do we avoid this error?
\n\nDon't use mae with boost_from_average set to False ;)
I'll say more after competition end.
", "rawMarkdown": "> Well, how do we avoid this error? \n\nDon't use mae with `boost_from_average` set to False ;)\n\nI'll say more after competition end.", "votes": null }, { "id": "530197", "postDate": "05/12/2019 04:26:31", "content": "I just tried xgboost with similar results as catboost (Much better CV score than LGB but an abysmal LB score). I need to look more into this base_score parameter that @cpmpml speaks of....
I have started with xgboost and so far my CV is higher than with lgb. I haven't submitted yet.
", "rawMarkdown": "I have started with xgboost and so far my CV is higher than with lgb. I haven't submitted yet.", "votes": null } ], "comments": [ { "id": 527522, "author_name": "cpmpml", "author_url": "", "post_date": "05/05/2019 18:16:31", "content": "\n\n\nhe is using early stopping in quake-wise CV
\n
I am not doing that. I even said it explicitly in the discussion you cite ;)
", "votes": null, "replies": [ { "id": 527525, "author_name": "amjad85", "author_url": "", "post_date": "05/05/2019 18:21:17", "content": "Sorry for misunderstanding then. I guess this was only my assumption because I could only reproduce a similar pattern to your oof with early stopping.
", "votes": null, "replies": [] }, { "id": 527531, "author_name": "cpmpml", "author_url": "", "post_date": "05/05/2019 18:28:48", "content": "What is right in your post is that lgb has many parameters, and some can have a significant effect like the one you single out.
", "votes": null, "replies": [] }, { "id": 527533, "author_name": "amjad85", "author_url": "", "post_date": "05/05/2019 18:37:54", "content": "Everything is right about my post, except my assumption about your secret fancy strategy ;)
", "votes": null, "replies": [] } ] }, { "id": 527568, "author_name": "robikscube", "author_url": "", "post_date": "05/05/2019 20:40:30", "content": "Very interesting. Thanks for posting this insight. Is it safe to say that XGBoost would exhibit the same behavior as catboost? Quick google search found xgboost doesn't have boost_from_average implemented yet but is being discussed on their github: https://github.com/dmlc/xgboost/issues/4321
I haven't tried XGBoost in this competition yet. I'll post an update when I try it.
", "votes": null, "replies": [] }, { "id": 527583, "author_name": "cpmpml", "author_url": "", "post_date": "05/05/2019 21:46:30", "content": "You can use base_score parameter to starts from whatever value you want with XGBoost. Setting it to target average gives you the behavior you discuss.
", "votes": null, "replies": [] }, { "id": 530197, "author_name": "robikscube", "author_url": "", "post_date": "05/12/2019 04:26:31", "content": "I just tried xgboost with similar results as catboost (Much better CV score than LGB but an abysmal LB score). I need to look more into this base_score parameter that @cpmpml speaks of....
I have started with xgboost and so far my CV is higher than with lgb. I haven't submitted yet.
", "votes": null, "replies": [] } ] }, { "id": 527584, "author_name": "philippsinger", "author_url": "", "post_date": "05/05/2019 21:49:19", "content": "It's all about the mean... What is funny BTW is that boost_from_average behaves differently between l1 and gamma loss.
Ehm... help? Setting 'boost_from_average': 'false', in the parameters causes the best iteration for each fold to be iteration 1, and overall CV jumps to 5.679
That's odd. It will stop prematurely but not like that.
", "votes": null, "replies": [] }, { "id": 528252, "author_name": "cpmpml", "author_url": "", "post_date": "05/07/2019 11:08:19", "content": "I have the same with objective mae.
", "votes": null, "replies": [] }, { "id": 528282, "author_name": "returnofsputnik", "author_url": "", "post_date": "05/07/2019 12:20:15", "content": "Well, how do we avoid this error? I am also using objective=mean_absolute_error. Are you using gamma loss instead of mae ? Or maybe, the parameters need to be retuned.
> Well, how do we avoid this error?
\n\nDon't use mae with boost_from_average set to False ;)
I'll say more after competition end.
", "votes": null, "replies": [] } ] }, { "id": 528249, "author_name": "amjad85", "author_url": "", "post_date": "05/07/2019 10:50:34", "content": "Update: after trying quake-wise CV with early stopping in LGB (boost_from_average = true), the leak is still there. Not as it was with catboost, but it is still severely under/over-fitting. \n So I am sticking with shuffling, at least my models stop optimally in this setting.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "527521": "The discussion about using leave-one-quake-out cross validation and early stopping started in [this thread](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/91125#latest-527210) when @cpmpml attached his oof plot. I was intrigued that he is using early stopping in quake-wise CV and getting good results. At least when I tried it, my LB score was > 1.6 due to pre-mature stopping resulting from leakage. Others like @philippsinger shared my frustration. Even @cpmpml started doubting himself and started [this thread about early stopping](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/91328#latest-527457) where many had very good arguments about leakage, but @cpmpml almost could swear his model is not leaking.\n\nSo I investigated the issue and figured out the reason for this discrepancy. Those who are suffering from leak when using this strategy are probably using catboost (at least me). Others are using LGB. \nIt turned out that catboost initialize the boosting with zeros. In small quakes, the error in the validation set reaches a minimum very fast before convergence because the values start increasing from zero. However, LGB has a parameter `boost_from_average`, which is true by default. Therefore, LGB doesn't have the same problem.\n\nTo test, set `boost_from_average` to false, and the CV score will drop ~0.2 due to leakage. I couldn't find a parameter in catboost that controls this behavior.\n\nSo to conclude, if you want to use quake-wise CV, switch to LGB. That doesn't exclude the possibility of leakage, but at least it won't be as severe as we have experienced. \nI don't know yet how that will affect my LB score, but fingers crossed. \n\nUpdate: after trying quake-wise CV with early stopping in LGB (boostfromaverage = true), the leak is still there. Not as it was with catboost, but it is still severely under/over-fitting. \nSo I am sticking with shuffling, at least my models stop optimally in this setting.", "527522": "> he is using early stopping in quake-wise CV \n\nI am not doing that. I even said it explicitly in the discussion you cite ;)", "527525": "Sorry for misunderstanding then. I guess this was only my assumption because I could only reproduce a similar pattern to your oof with early stopping.", "527531": "What is right in your post is that lgb has many parameters, and some can have a significant effect like the one you single out.", "527533": "Everything is right about my post, except my assumption about your secret fancy strategy ;)", "527568": "Very interesting. Thanks for posting this insight. Is it safe to say that XGBoost would exhibit the same behavior as catboost? Quick google search found xgboost doesn't have `boost_from_average` implemented yet but is being discussed on their github: https://github.com/dmlc/xgboost/issues/4321", "527572": "I haven't tried XGBoost in this competition yet. I'll post an update when I try it.", "527583": "You can use base_score parameter to starts from whatever value you want with XGBoost. Setting it to target average gives you the behavior you discuss.", "527584": "It's all about the mean... What is funny BTW is that `boost_from_average` behaves differently between l1 and gamma loss.", "528240": "Ehm... help? Setting `'boost_from_average': 'false',` in the parameters causes the best iteration for each fold to be iteration 1, and overall CV jumps to 5.679", "528247": "That's odd. It will stop prematurely but not like that.", "528249": "Update: after trying quake-wise CV with early stopping in LGB (boost_from_average = true), the leak is still there. Not as it was with catboost, but it is still severely under/over-fitting. \n So I am sticking with shuffling, at least my models stop optimally in this setting.", "528252": "I have the same with objective mae.", "528282": "Well, how do we avoid this error? I am also using `objective=mean_absolute_error`. Are you using `gamma` loss instead of `mae` ? Or maybe, the parameters need to be retuned.", "528287": "> Well, how do we avoid this error? \n\nDon't use mae with `boost_from_average` set to False ;)\n\nI'll say more after competition end.", "530197": "I just tried xgboost with similar results as catboost (Much better CV score than LGB but an abysmal LB score). I need to look more into this `base_score` parameter that @cpmpml speaks of....", "530271": "I have started with xgboost and so far my CV is higher than with lgb. I haven't submitted yet." }, "source": "meta" }