Hi fellow kagglers:
\nI have trained a xgb model locally and when I want to move it to kaggle environment, I fail to do that and meet many weird things, I'm wondering if anyone meet the same situations with me?
\nthe following is the detail:
I learned a lot from Simple EDA on Time for targets and my first version of code is quiet similar to that(main difference is I use xgb except lgbm).
\nI notice writer sample 10% of data to train (otherwise code will exceed memory limits).So my first thoughts is use external memory to avoid that, here is relevant code
skf = GroupKFold(n_kfolds)\n fold, (train_idx, valid_idx) tqdm((skf.split(.....)):\n\n train_data_parts = train_data.iloc[train_idx, :]\n valid_data_parts = train_data.iloc[valid_idx, :]\n \n dump_svmlight_file(X=train_data_parts[feas_all]\n , y=train_data_parts[y_label]\n , f=)\n train_data_parts\n \n \n dtrain = xgb.DMatrix(join(path_save, )\n , feature_names=feas_all)\n\n dvalid = xgb.DMatrix(valid_data_parts[feas_all].values\n , label=valid_data_parts[y_label]\n , feature_names=feas_all)\nAnd what I get when I use \"save version\" func(after I click the button, kaggle will automatically run the code) is the code stop when it finish dump_svmlight_file func at first fold and don't run the following code, and nothing else happen, it just stop running and generate a new version of code.
Later, I try to train my xgbmodel locally and upload it to kaggle envirnment (dump_svmlight_file cost nearly one hour if I use all data from defog+tdcafog, it's not a time saving solution so I didn't spend more time on it). But I get \"Notebook Threw Exception\" instead.
\nI swear I read most of revelent discussions like: [solved] Notebook threw Exception error- I've tried fixing the submission loop!, Gait Prediction - Notebook Threw Exception \"Please Help\", Need Help - Submission Scoring Error??, I need help with the error: Notebook Threw Exception, How I (usually) solve my \"submission score error\", is there any problem on my submission file?, Notebook runs successfully, but there is no score etc. I avoid all the problem mentioned in discussions but still code can't run successfully until I change xgb to lgbm and finally I can get score!
\nThe same thing happens when I train an online xgbmodel (after sampling 5% data to avoid memory explosion), the code doesn't work until I use lgbm model, and I don't change ohter part of code
I really spend several day on this problem and I understand in What is: \"Notebook Threw Exception\", kaggle team said: \"Your notebook is rerun with the private dataset. You don't get to see the log from that run because it could be used to exfiltrate the private host data. This error means your notebook threw an exception while processing the private data, even if it doesn't encounter such an error in the pubic data.\"
\nbut I really spend a lot of time to guess what's wrong with my code.😅 Or is it a common thing everyone know besides me? maybe it's why I don't see code using xgboost to train model in \"Code\"
\nDoes anyone has the same confusion?👀
Hi fellow kagglers:
\nI have trained a xgb model locally and when I want to move it to kaggle environment, I fail to do that and meet many weird things, I'm wondering if anyone meet the same situations with me?
\nthe following is the detail:
I learned a lot from Simple EDA on Time for targets and my first version of code is quiet similar to that(main difference is I use xgb except lgbm).
\nI notice writer sample 10% of data to train (otherwise code will exceed memory limits).So my first thoughts is use external memory to avoid that, here is relevant code
skf = GroupKFold(n_kfolds)\n fold, (train_idx, valid_idx) tqdm((skf.split(.....)):\n\n train_data_parts = train_data.iloc[train_idx, :]\n valid_data_parts = train_data.iloc[valid_idx, :]\n \n dump_svmlight_file(X=train_data_parts[feas_all]\n , y=train_data_parts[y_label]\n , f=)\n train_data_parts\n \n \n dtrain = xgb.DMatrix(join(path_save, )\n , feature_names=feas_all)\n\n dvalid = xgb.DMatrix(valid_data_parts[feas_all].values\n , label=valid_data_parts[y_label]\n , feature_names=feas_all)\nAnd what I get when I use \"save version\" func(after I click the button, kaggle will automatically run the code) is the code stop when it finish dump_svmlight_file func at first fold and don't run the following code, and nothing else happen, it just stop running and generate a new version of code.
Later, I try to train my xgbmodel locally and upload it to kaggle envirnment (dump_svmlight_file cost nearly one hour if I use all data from defog+tdcafog, it's not a time saving solution so I didn't spend more time on it). But I get \"Notebook Threw Exception\" instead.
\nI swear I read most of revelent discussions like: [solved] Notebook threw Exception error- I've tried fixing the submission loop!, Gait Prediction - Notebook Threw Exception \"Please Help\", Need Help - Submission Scoring Error??, I need help with the error: Notebook Threw Exception, How I (usually) solve my \"submission score error\", is there any problem on my submission file?, Notebook runs successfully, but there is no score etc. I avoid all the problem mentioned in discussions but still code can't run successfully until I change xgb to lgbm and finally I can get score!
\nThe same thing happens when I train an online xgbmodel (after sampling 5% data to avoid memory explosion), the code doesn't work until I use lgbm model, and I don't change ohter part of code
I really spend several day on this problem and I understand in What is: \"Notebook Threw Exception\", kaggle team said: \"Your notebook is rerun with the private dataset. You don't get to see the log from that run because it could be used to exfiltrate the private host data. This error means your notebook threw an exception while processing the private data, even if it doesn't encounter such an error in the pubic data.\"
\nbut I really spend a lot of time to guess what's wrong with my code.😅 Or is it a common thing everyone know besides me? maybe it's why I don't see code using xgboost to train model in \"Code\"
\nDoes anyone has the same confusion?👀