This has been discussed frequently, but I’m curious how others handle this issue. When I run gradient boosting on a GPU, I notice the score fluctuates. In my case, it varies between 0.467 and 0.479 on my Notebook. Is this just randomness or luck, or is there a way to stabilize the score?
", "messages": [ { "id": "3015246", "postDate": "10/12/2024 05:59:07", "content": "This has been discussed frequently, but I’m curious how others handle this issue. When I run gradient boosting on a GPU, I notice the score fluctuates. In my case, it varies between 0.467 and 0.479 on my Notebook. Is this just randomness or luck, or is there a way to stabilize the score?
", "rawMarkdown": "This has been discussed frequently, but I’m curious how others handle this issue. When I run gradient boosting on a GPU, I notice the score fluctuates. In my case, it varies between 0.467 and 0.479 on my Notebook. Is this just randomness or luck, or is there a way to stabilize the score?", "votes": null }, { "id": "3015268", "postDate": "10/12/2024 06:27:42", "content": "Try and save the model object and reuse it to prevent it from providing you different scores across runs. I am also experiencing the same problem. I think the CV score here is extremely volatile and the LB is not highly trustworthy @itsukiito
", "rawMarkdown": "Try and save the model object and reuse it to prevent it from providing you different scores across runs. I am also experiencing the same problem. I think the CV score here is extremely volatile and the LB is not highly trustworthy @itsukiito", "votes": null }, { "id": "3015281", "postDate": "10/12/2024 07:01:26", "content": "With LightGBM, the 'deterministic' parameter is only available for cpu. How much fluctuation do you see? Is it large compared to the variation in CV score?
", "rawMarkdown": "With LightGBM, the 'deterministic' parameter is only available for cpu. How much fluctuation do you see? Is it large compared to the variation in CV score?", "votes": null }, { "id": "3015284", "postDate": "10/12/2024 07:08:14", "content": "@ravi20076
\nThank you for the advice! I was feeling concerned about the fluctuations in the CV score, so this gives me some relief. It’s important to focus on the correlation between CV and LB without getting caught up in the immediate LB score, and to trust the CV.
@passengerc07
\nThanks for the comments! In my case, the Optimized QWK (CV) also fluctuates slightly, ranging from 0.446 to 0.450, but the LB score fluctuates even more, between 0.467 and 0.479.
CV and LB are not related for me till date at least @itsukiito
", "rawMarkdown": "CV and LB are not related for me till date at least @itsukiito", "votes": null }, { "id": "3016795", "postDate": "10/14/2024 07:06:00", "content": "I tried a few times.Using a CPU cannot solve the problem, and the score also fluctuates. It is possible that the score fluctuation is smaller when using a CPU.I didn't submit any more times today
", "rawMarkdown": "I tried a few times.Using a CPU cannot solve the problem, and the score also fluctuates. It is possible that the score fluctuation is smaller when using a CPU.I didn't submit any more times today", "votes": null }, { "id": "3038093", "postDate": "11/06/2024 15:18:32", "content": "seed everything to make prediction stable , for torch
\n torch.backends.cudnn cudnn\ncudnn.deterministic = \ncudnn.benchmark = \nfor xxboost
\nparams = {\n : ,\n : ,\n : ,\n : ,\n : \n}\netc ….
", "rawMarkdown": "seed everything to make prediction stable , for torch \n``` \nimport torch.backends.cudnn as cudnn\ncudnn.deterministic = True\ncudnn.benchmark = False \n```\nfor xxboost \n```\n\nparams = {\n 'objective': 'reg:squarederror',\n 'max_depth': 10,\n 'learning_rate': 0.1,\n 'n_estimators': 100,\n 'seed': 42 # Set seed for XGBoost\n}\n```\netc ....", "votes": null } ], "comments": [ { "id": 3015268, "author_name": "ravi20076", "author_url": "", "post_date": "10/12/2024 06:27:42", "content": "Try and save the model object and reuse it to prevent it from providing you different scores across runs. I am also experiencing the same problem. I think the CV score here is extremely volatile and the LB is not highly trustworthy @itsukiito
", "votes": null, "replies": [ { "id": 3015284, "author_name": "itsukiito", "author_url": "", "post_date": "10/12/2024 07:08:14", "content": "@ravi20076
\nThank you for the advice! I was feeling concerned about the fluctuations in the CV score, so this gives me some relief. It’s important to focus on the correlation between CV and LB without getting caught up in the immediate LB score, and to trust the CV.
CV and LB are not related for me till date at least @itsukiito
", "votes": null, "replies": [] } ] } ] }, { "id": 3015281, "author_name": "passengerc07", "author_url": "", "post_date": "10/12/2024 07:01:26", "content": "With LightGBM, the 'deterministic' parameter is only available for cpu. How much fluctuation do you see? Is it large compared to the variation in CV score?
", "votes": null, "replies": [ { "id": 3015300, "author_name": "itsukiito", "author_url": "", "post_date": "10/12/2024 07:33:21", "content": "@passengerc07
\nThanks for the comments! In my case, the Optimized QWK (CV) also fluctuates slightly, ranging from 0.446 to 0.450, but the LB score fluctuates even more, between 0.467 and 0.479.
I tried a few times.Using a CPU cannot solve the problem, and the score also fluctuates. It is possible that the score fluctuation is smaller when using a CPU.I didn't submit any more times today
", "votes": null, "replies": [] }, { "id": 3038093, "author_name": "saidkoussi", "author_url": "", "post_date": "11/06/2024 15:18:32", "content": "seed everything to make prediction stable , for torch
\n torch.backends.cudnn cudnn\ncudnn.deterministic = \ncudnn.benchmark = \nfor xxboost
\nparams = {\n : ,\n : ,\n : ,\n : ,\n : \n}\netc ….
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3015246": "This has been discussed frequently, but I’m curious how others handle this issue. When I run gradient boosting on a GPU, I notice the score fluctuates. In my case, it varies between 0.467 and 0.479 on my Notebook. Is this just randomness or luck, or is there a way to stabilize the score?", "3015268": "Try and save the model object and reuse it to prevent it from providing you different scores across runs. I am also experiencing the same problem. I think the CV score here is extremely volatile and the LB is not highly trustworthy @itsukiito", "3015281": "With LightGBM, the 'deterministic' parameter is only available for cpu. How much fluctuation do you see? Is it large compared to the variation in CV score?", "3015284": "ravi20076 \nThank you for the advice! I was feeling concerned about the fluctuations in the CV score, so this gives me some relief. It’s important to focus on the correlation between CV and LB without getting caught up in the immediate LB score, and to trust the CV.", "3015300": "passengerc07 \nThanks for the comments! In my case, the Optimized QWK (CV) also fluctuates slightly, ranging from 0.446 to 0.450, but the LB score fluctuates even more, between 0.467 and 0.479.", "3015342": "CV and LB are not related for me till date at least @itsukiito", "3016795": "I tried a few times.Using a CPU cannot solve the problem, and the score also fluctuates. It is possible that the score fluctuation is smaller when using a CPU.I didn't submit any more times today", "3038093": "seed everything to make prediction stable , for torch \n``` \nimport torch.backends.cudnn as cudnn\ncudnn.deterministic = True\ncudnn.benchmark = False \n```\nfor xxboost \n```\n\nparams = {\n 'objective': 'reg:squarederror',\n 'max_depth': 10,\n 'learning_rate': 0.1,\n 'n_estimators': 100,\n 'seed': 42 # Set seed for XGBoost\n}\n```\netc ...." }, "source": "meta" }