![\"catboost](\"https://i.imgur.com/f0KphfH.png\")
Hello, everyone.
\nI use GPU to train catboost, I have tried many parameters, and the best cv now is 0.7973, LB is 0.797?.
\nBecause some parameters such as 'rsm' cannot use on GPU and CPU is too slow, so I haven't tune these parameters.
\nI'm in trouble now. I have tried nearly all parameters. I can't improve my score again unless I change dataset
\n(I use dataset which is same with public lgbm 0.7977 dart).
I want to know if anyone has achieved a score of 0.798 or above using catboost? Or is catboost worse than lgbm?
\nIf someone has reached it, please reply to me. I want to know if I still have room to improve.
\nThank you!
", "messages": [ { "id": 1866487, "postDate": "2022-07-22T14:08:09.207Z", "content": "Hello, everyone.
\nI use GPU to train catboost, I have tried many parameters, and the best cv now is 0.7973, LB is 0.797?.
\nBecause some parameters such as 'rsm' cannot use on GPU and CPU is too slow, so I haven't tune these parameters.
\nI'm in trouble now. I have tried nearly all parameters. I can't improve my score again unless I change dataset
\n(I use dataset which is same with public lgbm 0.7977 dart).
I want to know if anyone has achieved a score of 0.798 or above using catboost? Or is catboost worse than lgbm?
\nIf someone has reached it, please reply to me. I want to know if I still have room to improve.
\nThank you!
", "rawMarkdown": "Hello, everyone.\n\nI use GPU to train catboost, I have tried many parameters, and the best cv now is 0.7973, LB is 0.797?.\n\nBecause some parameters such as 'rsm' cannot use on GPU and CPU is too slow, so I haven't tune these parameters.\n\nI'm in trouble now. I have tried nearly all parameters. I can't improve my score again unless I change dataset\n(I use dataset which is same with public lgbm 0.7977 dart).\n\nI want to know if anyone has achieved a score of 0.798 or above using catboost? Or is catboost worse than lgbm?\n\nIf someone has reached it, please reply to me. I want to know if I still have room to improve.\n\nThank you!\n", "votes": 13 }, { "id": 1866621, "postDate": "2022-07-22T15:59:30.840Z", "content": "CV 0.79664 and lb 0.798 is possible with catboost and 675 features:
\n
Thanks, I will try to get better parameters
", "rawMarkdown": "Thanks, I will try to get better parameters" }, { "id": 1866938, "postDate": "2022-07-22T21:34:50.083Z", "content": "That's great Feature Selection Kudos 🙌
", "rawMarkdown": "That's great Feature Selection Kudos 🙌" }, { "id": 1867156, "postDate": "2022-07-23T03:30:50.083Z", "content": "Is this catboost gpu version or cpu version?
", "rawMarkdown": "Is this catboost gpu version or cpu version?" }, { "id": 1867347, "postDate": "2022-07-23T06:40:44.563Z", "content": "It is the CPU version.
", "rawMarkdown": "It is the CPU version.", "votes": 2 }, { "id": 1908805, "postDate": "2022-08-22T03:41:54.780Z", "content": "Finally reached 0.798 : |
", "rawMarkdown": "Finally reached 0.798 : | " }, { "id": 1908812, "postDate": "2022-08-22T03:54:25.180Z", "content": "@gauravbrills great work😬
", "rawMarkdown": "@gauravbrills great work😬", "votes": 1 }, { "id": 1909233, "postDate": "2022-08-22T12:56:34.570Z", "content": "Haha all credits to you @tonymarkchris
", "rawMarkdown": "Haha all credits to you @tonymarkchris " } ] }, { "id": 1866497, "postDate": "2022-07-22T14:22:02.190Z", "content": "My catboost prediction was always better the lgb and xgb on cv and LB. To save time, I've been using only xgb recently because the gpu version of xgb is the fastest amoung them. I think use catboost to get a of 0.798 or 0.799+ result is possible, it depends on your feature engineering.
", "rawMarkdown": "My catboost prediction was always better the lgb and xgb on cv and LB. To save time, I've been using only xgb recently because the gpu version of xgb is the fastest amoung them. I think use catboost to get a of 0.798 or 0.799+ result is possible, it depends on your feature engineering.", "votes": 7, "replies": [ { "id": 1866522, "postDate": "2022-07-22T14:46:41.947Z", "content": "Ok, I will pay more attention on feature engineering
", "rawMarkdown": "Ok, I will pay more attention on feature engineering", "votes": 1 }, { "id": 1866535, "postDate": "2022-07-22T14:58:24.677Z", "content": "good luck, I just did minor change in the feature, 3 more lines of code. only use the lgb dart (no param optimization), 5 folds output already reached 0.798.
", "rawMarkdown": "good luck, I just did minor change in the feature, 3 more lines of code. only use the lgb dart (no param optimization), 5 folds output already reached 0.798.", "votes": 4 }, { "id": 1866800, "postDate": "2022-07-22T18:29:17.753Z", "content": "Just for those who voted against this discussion: I seriously want to express that the role of feature engineering is much greater than model parameter tuning. Generally, it is more efficient to adjust the hyperparameters of the model when everything else was tried out. Besides, if you made any improvement in features, you still need to optimize the hyperparameters again. It is just my own opionion, wellcome to leave your comments below and discuss with me.
", "rawMarkdown": "Just for those who voted against this discussion: I seriously want to express that the role of feature engineering is much greater than model parameter tuning. Generally, it is more efficient to adjust the hyperparameters of the model when everything else was tried out. Besides, if you made any improvement in features, you still need to optimize the hyperparameters again. It is just my own opionion, wellcome to leave your comments below and discuss with me.", "votes": 15, "replies": [ { "id": 1901185, "postDate": "2022-08-16T13:55:39.293Z", "content": "Thanks for sharing your insight @meli19 . Could you elaborate a little bit more how is your approach to FE and HPO? From what you mentioned in the comment I understand:
\nStarting features and HPs -> Include new features -> If new features improve the CV -> Tune HPs -> Include new features -> If new features improve the CV -> Tune HPs, etc.
\nIs this correct? Doesn't this lead you to overfitting?
\nBtw when you tune the HPs after each improvement in features, do you try to find the optimal or do you increase for example the number of leaves to a bigger number than the optimal? My concern is that the HP choice sometimes impede that new useful features show up.
", "rawMarkdown": "Thanks for sharing your insight @meli19 . Could you elaborate a little bit more how is your approach to FE and HPO? From what you mentioned in the comment I understand:\n\nStarting features and HPs -> Include new features -> If new features improve the CV -> Tune HPs -> Include new features -> If new features improve the CV -> Tune HPs, etc.\n\nIs this correct? Doesn't this lead you to overfitting?\n\nBtw when you tune the HPs after each improvement in features, do you try to find the optimal or do you increase for example the number of leaves to a bigger number than the optimal? My concern is that the HP choice sometimes impede that new useful features show up." } ] }, { "id": 1906882, "postDate": "2022-08-20T09:55:12.393Z", "content": "you are right. I didn't change any of the default parameters (copied from the public notebook) at all actually.
", "rawMarkdown": "you are right. I didn't change any of the default parameters (copied from the public notebook) at all actually." } ] }, { "id": 1866493, "postDate": "2022-07-22T14:14:41.197Z", "content": "You have a very very good catboost score. my CV score is similar to yours. It is really hard to improve at that point I think.
", "rawMarkdown": "You have a very very good catboost score. my CV score is similar to yours. It is really hard to improve at that point I think.", "votes": 3, "replies": [ { "id": 1866521, "postDate": "2022-07-22T14:46:11.503Z", "content": "Thank you for your reply
", "rawMarkdown": "Thank you for your reply" } ] }, { "id": 1866549, "postDate": "2022-07-22T15:10:11.403Z", "content": "Our catboost effort CV 0.7965 LB 0.797, will try to see it can reach 0.798 and match LGBM
", "rawMarkdown": "Our catboost effort CV 0.7965 LB 0.797, will try to see it can reach 0.798 and match LGBM ", "votes": 1, "replies": [ { "id": 1866742, "postDate": "2022-07-22T17:38:10.810Z", "content": "Thanks you, I will try to get better parameters
", "rawMarkdown": "Thanks you, I will try to get better parameters" } ] }, { "id": 1866587, "postDate": "2022-07-22T15:39:58.977Z", "content": "I think you have reached a glass ceiling with the catboost algorithm, this is a great score with this method!
", "rawMarkdown": "I think you have reached a glass ceiling with the catboost algorithm, this is a great score with this method!", "votes": -1 }, { "id": 1908742, "postDate": "2022-08-22T01:20:16.900Z", "content": "The same quesiton to you. The result of my all experiments, catboost model never broke through to 0.797.
", "rawMarkdown": "The same quesiton to you. The result of my all experiments, catboost model never broke through to 0.797." }, { "id": 1871332, "postDate": "2022-07-26T08:06:14.747Z", "content": "Time to engineer features..
", "rawMarkdown": "Time to engineer features..\n" }, { "id": 1868263, "postDate": "2022-07-23T20:53:09.803Z", "content": "I have tuned some parameters, now cv is 0.7975-0.7977，LB unknown，I think the ceiling maybe is 0.798, which is worse than lgbm
", "rawMarkdown": "I have tuned some parameters, now cv is 0.7975-0.7977，LB unknown，I think the ceiling maybe is 0.798, which is worse than lgbm" }, { "id": 1867133, "postDate": "2022-07-23T02:49:55.947Z", "content": "Hi, have you tried using other bootstraping options for catboost such as Bernoulli, MVS, and Poisson, aside from Bayesian?
", "rawMarkdown": "Hi, have you tried using other bootstraping options for catboost such as Bernoulli, MVS, and Poisson, aside from Bayesian? " }, { "id": 1866580, "postDate": "2022-07-22T15:38:16.453Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1866621, "author_name": "AmbrosM", "author_url": "", "post_date": "2022-07-22T15:59:30.840000", "content": "CV 0.79664 and lb 0.798 is possible with catboost and 675 features:
\n
Thanks, I will try to get better parameters
", "votes": 0, "replies": [] }, { "id": 1866938, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-07-22T21:34:50.083000", "content": "That's great Feature Selection Kudos 🙌
", "votes": 0, "replies": [] }, { "id": 1867156, "author_name": "AKR", "author_url": "", "post_date": "2022-07-23T03:30:50.083000", "content": "Is this catboost gpu version or cpu version?
", "votes": 0, "replies": [] }, { "id": 1867347, "author_name": "AmbrosM", "author_url": "", "post_date": "2022-07-23T06:40:44.563000", "content": "It is the CPU version.
", "votes": 2, "replies": [] }, { "id": 1908805, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-08-22T03:41:54.780000", "content": "Finally reached 0.798 : |
", "votes": 0, "replies": [] }, { "id": 1908812, "author_name": "DJ_Xia", "author_url": "", "post_date": "2022-08-22T03:54:25.180000", "content": "@gauravbrills great work😬
", "votes": 1, "replies": [] }, { "id": 1909233, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-08-22T12:56:34.570000", "content": "Haha all credits to you @tonymarkchris
", "votes": 0, "replies": [] } ] }, { "id": 1866497, "author_name": "Mengfei Li", "author_url": "", "post_date": "2022-07-22T14:22:02.190000", "content": "My catboost prediction was always better the lgb and xgb on cv and LB. To save time, I've been using only xgb recently because the gpu version of xgb is the fastest amoung them. I think use catboost to get a of 0.798 or 0.799+ result is possible, it depends on your feature engineering.
", "votes": 7, "replies": [ { "id": 1866522, "author_name": "Windrunner", "author_url": "", "post_date": "2022-07-22T14:46:41.947000", "content": "Ok, I will pay more attention on feature engineering
", "votes": 1, "replies": [] }, { "id": 1866535, "author_name": "Mengfei Li", "author_url": "", "post_date": "2022-07-22T14:58:24.677000", "content": "good luck, I just did minor change in the feature, 3 more lines of code. only use the lgb dart (no param optimization), 5 folds output already reached 0.798.
", "votes": 4, "replies": [] }, { "id": 1866800, "author_name": "Mengfei Li", "author_url": "", "post_date": "2022-07-22T18:29:17.753000", "content": "Just for those who voted against this discussion: I seriously want to express that the role of feature engineering is much greater than model parameter tuning. Generally, it is more efficient to adjust the hyperparameters of the model when everything else was tried out. Besides, if you made any improvement in features, you still need to optimize the hyperparameters again. It is just my own opionion, wellcome to leave your comments below and discuss with me.
", "votes": 15, "replies": [ { "id": 1901185, "author_name": "delai50", "author_url": "", "post_date": "2022-08-16T13:55:39.293000", "content": "Thanks for sharing your insight @meli19 . Could you elaborate a little bit more how is your approach to FE and HPO? From what you mentioned in the comment I understand:
\nStarting features and HPs -> Include new features -> If new features improve the CV -> Tune HPs -> Include new features -> If new features improve the CV -> Tune HPs, etc.
\nIs this correct? Doesn't this lead you to overfitting?
\nBtw when you tune the HPs after each improvement in features, do you try to find the optimal or do you increase for example the number of leaves to a bigger number than the optimal? My concern is that the HP choice sometimes impede that new useful features show up.
", "votes": 0, "replies": [] } ] }, { "id": 1906882, "author_name": "Mengfei Li", "author_url": "", "post_date": "2022-08-20T09:55:12.393000", "content": "you are right. I didn't change any of the default parameters (copied from the public notebook) at all actually.
", "votes": 0, "replies": [] } ] }, { "id": 1866493, "author_name": "raddar", "author_url": "", "post_date": "2022-07-22T14:14:41.197000", "content": "You have a very very good catboost score. my CV score is similar to yours. It is really hard to improve at that point I think.
", "votes": 3, "replies": [ { "id": 1866521, "author_name": "Windrunner", "author_url": "", "post_date": "2022-07-22T14:46:11.503000", "content": "Thank you for your reply
", "votes": 0, "replies": [] } ] }, { "id": 1866549, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-07-22T15:10:11.403000", "content": "Our catboost effort CV 0.7965 LB 0.797, will try to see it can reach 0.798 and match LGBM
", "votes": 1, "replies": [ { "id": 1866742, "author_name": "Windrunner", "author_url": "", "post_date": "2022-07-22T17:38:10.810000", "content": "Thanks you, I will try to get better parameters
", "votes": 0, "replies": [] } ] }, { "id": 1866587, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2022-07-22T15:39:58.977000", "content": "I think you have reached a glass ceiling with the catboost algorithm, this is a great score with this method!
", "votes": -1, "replies": [] }, { "id": 1908742, "author_name": "Jackson You", "author_url": "", "post_date": "2022-08-22T01:20:16.900000", "content": "The same quesiton to you. The result of my all experiments, catboost model never broke through to 0.797.
", "votes": 0, "replies": [] }, { "id": 1871332, "author_name": "The Devastator", "author_url": "", "post_date": "2022-07-26T08:06:14.747000", "content": "Time to engineer features..
", "votes": 0, "replies": [] }, { "id": 1868263, "author_name": "Windrunner", "author_url": "", "post_date": "2022-07-23T20:53:09.803000", "content": "I have tuned some parameters, now cv is 0.7975-0.7977，LB unknown，I think the ceiling maybe is 0.798, which is worse than lgbm
", "votes": 0, "replies": [] }, { "id": 1867133, "author_name": "Tarrasque9", "author_url": "", "post_date": "2022-07-23T02:49:55.947000", "content": "Hi, have you tried using other bootstraping options for catboost such as Bernoulli, MVS, and Poisson, aside from Bayesian?
", "votes": 0, "replies": [] }, { "id": 1866580, "author_name": "", "author_url": "", "post_date": "2022-07-22T15:38:16.453000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1866487": "Hello, everyone.\n\nI use GPU to train catboost, I have tried many parameters, and the best cv now is 0.7973, LB is 0.797?.\n\nBecause some parameters such as 'rsm' cannot use on GPU and CPU is too slow, so I haven't tune these parameters.\n\nI'm in trouble now. I have tried nearly all parameters. I can't improve my score again unless I change dataset\n(I use dataset which is same with public lgbm 0.7977 dart).\n\nI want to know if anyone has achieved a score of 0.798 or above using catboost? Or is catboost worse than lgbm?\n\nIf someone has reached it, please reply to me. I want to know if I still have room to improve.\n\nThank you!\n", "1866621": "CV 0.79664 and lb 0.798 is possible with catboost and 675 features:\n\n", "1866497": "My catboost prediction was always better the lgb and xgb on cv and LB. To save time, I've been using only xgb recently because the gpu version of xgb is the fastest amoung them. I think use catboost to get a of 0.798 or 0.799+ result is possible, it depends on your feature engineering.", "1866493": "You have a very very good catboost score. my CV score is similar to yours. It is really hard to improve at that point I think.", "1866549": "Our catboost effort CV 0.7965 LB 0.797, will try to see it can reach 0.798 and match LGBM ", "1866587": "I think you have reached a glass ceiling with the catboost algorithm, this is a great score with this method!", "1908742": "The same quesiton to you. The result of my all experiments, catboost model never broke through to 0.797.", "1871332": "Time to engineer features..\n", "1868263": "I have tuned some parameters, now cv is 0.7975-0.7977，LB unknown，I think the ceiling maybe is 0.798, which is worse than lgbm", "1867133": "Hi, have you tried using other bootstraping options for catboost such as Bernoulli, MVS, and Poisson, aside from Bayesian? ", "1866580": "" } }