Don't use DF to create lightgbm dataset, rather use np array:
\nSample code:
\nX_train_np = train.values.astype(np.float32)\nX_valid_np = valid.values.astype(np.float32)\nfeatures = train.columns\nlgb_train = lgb.Dataset(X_train_np, label=y_tr, feature_name=list(features))\nlgb_valid = lgb.Dataset(X_valid_np, label=y_va, feature_name=list(features))\ndel train, y_tr\n_=gc.collect()\nReference - https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/149754
\ncredit: @alankabisov
Don't use DF to create lightgbm dataset, rather use np array:
\nSample code:
\nX_train_np = train.values.astype(np.float32)\nX_valid_np = valid.values.astype(np.float32)\nfeatures = train.columns\nlgb_train = lgb.Dataset(X_train_np, label=y_tr, feature_name=list(features))\nlgb_valid = lgb.Dataset(X_valid_np, label=y_va, feature_name=list(features))\ndel train, y_tr\n_=gc.collect()\nReference - https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/149754
\ncredit: @alankabisov
Good tip!
\nYou can however get a memory spike when creating the numpy array from the dataframe. My solution was to convert the columns one by one.
X_train = np.ndarray(shape=(N_ROWS, N_COLS), dtype=np.float32)\nfor idx, feature in enumerate(features):\n X_train[:,idx] = train[feature].values.astype(np.float32)\nthanks @markwijkhuizen , I will try your method for further improvement
", "rawMarkdown": "thanks @markwijkhuizen , I will try your method for further improvement" }, { "id": 1090454, "postDate": "2020-11-25T10:40:27.570Z", "content": "This is the solution that worked for me. Many thanks @markwijkhuizen! However instead of casting it column wise and assigning them to an empty array (this would still exhaust memory if the array you are initializing is large enough). A safer solution would be to take chunks of the DF row wise, cast them to float and do a stack operation. Putting a time.sleep() in between helped me as well.
Also, if you are using features such as timestamp it is probably safer to cast them to np.float64.
Thanks, it works to avoid memory spike but converting to np.float32 looks to modify AUC (from 0.7721 to 0.7742 on one for my test) even if not I'm not using any np.int64 (for timestamp). Pandas dataframe only contains np.int8, bool, np.int32, np.int16, np.float32. That's weird as lgb.Dataset converts everything to np.float32:
\nhttps://lightgbm.readthedocs.io/en/latest/_modules/lightgbm/basic.html#Dataset
\nAnyone got similar issue?
What about categorical features, like user_id? If I convert it to float32 type, does the model recognize it as a categorical features?
Yes, It would. We can specify this using the categorical_feature parameter during model training when we do a lgb.train. We also need to specify the feature_name parameter when we create the lgb.Dataset.
I see, but why do we need to specify feature_name? If this is because of categorical_feature, we can specify feature_names by their index!
The model does not recognize categorical features automatically when using numpy arrays, you need to specify the indices of the columns containing categorical features. The code below is a method I use to obtain these indices and specify them for training.
\ncategorical_feature_idxs = []\nfor v in ['part', 'bundle_id', 'etc']: # your cat features\n # TRAIN_FEATURES is a list with your training features\n categorical_feature_idxs.append(TRAIN_FEATURES.index(v))\n\n# Create dataset\ntrain_data = lgb.Dataset(data=X_train, label=y_train, categorical_feature=None)\n\n# train \nlgb.train(\n train_set = train_data,\n # specify the column indices of the categorical features\n categorical_feature = categorical_feature_idxs,\n)\nIt's like you said, feature_name is needed only when we have specify categorical_feature by their column names. This is not required if we specify the columns using their index. It's just that I find it more convenient to use the column names so as to not keep track of the indices :D
I tried but this converting to np.array makes my training slower :(
", "rawMarkdown": "I tried but this converting to np.array makes my training slower :(" } ] }, { "id": 1085643, "postDate": "2020-11-21T03:48:39.750Z", "content": "Does this hurt auc score? Although I had same condition (imputation of missing value, and keeping float32), auc score in validation of pandas improves (auc over 0.7), but not in numpy (auc 0.5). I guess part of reason is I use xgboost, not lightgbm
", "rawMarkdown": "Does this hurt auc score? Although I had same condition (imputation of missing value, and keeping float32), auc score in validation of pandas improves (auc over 0.7), but not in numpy (auc 0.5). I guess part of reason is I use xgboost, not lightgbm", "replies": [ { "id": 1087456, "postDate": "2020-11-22T18:08:08.843Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1090439, "postDate": "2020-11-25T10:32:34.240Z", "content": "The problem may arise if when you are using features such as timestamp. Which has a max value much greater than what np.float32 could contain. These values would then be rounded off leading to drop in AUC. Check discussion here.
@doctorkael
\nThanks. So is it about trade-off, getting rid of features like timestamp or setting all features as float64?
You could still use np.float64 without running into memory issues. @markwijkhuizen solution would help you do just that. If column wise casting still causes you memory issues, trying casting rows in chunks, say 10,000 rows at a time, stacking them to a numpy array going all the way to end of your dataset. The problem is caused not since Kaggle RAM can't contain all the data in float64 format (16 GB can actually hold quite a lot more). Rather it is caused due to a sudden memory spike. This can be mitigated by doing the operation ourselves in a slower fashion (time.sleep comes to mind).
>>> lgb.Dataset(dataframe)
would internally be translated to this ->
\n>>> lgb.Dataset(dataframe.values.astype('float'))
Well, not exactly but a rough equivalent. Hence the abrupt memory spike. Hope this helps :)
", "rawMarkdown": "You could still use `np.float64` without running into memory issues. @markwijkhuizen solution would help you do just that. If column wise casting still causes you memory issues, trying casting rows in chunks, say 10,000 rows at a time, stacking them to a numpy array going all the way to end of your dataset. The problem is caused *not* since Kaggle RAM can't contain all the data in float64 format (16 GB can actually hold quite a lot more). Rather it is caused due to a _**sudden** memory spike_. This can be mitigated by doing the operation ourselves in a *slower fashion* (time.sleep comes to mind).\n\n```>>> lgb.Dataset(dataframe)```\n\nwould internally be translated to this -> \n\n```>>> lgb.Dataset(dataframe.values.astype('float'))```\n\nWell, not exactly but a rough equivalent. Hence the abrupt memory spike. Hope this helps :)", "votes": 1 }, { "id": 1091726, "postDate": "2020-11-26T08:30:40.587Z", "content": "@doctorkael
\nThis \"sudden\" memory spike makes sense to me because success/error occurs arbitrary despite same cell trial…
\nI'm using sklearn api and not sure your method could work as same as python-api. But I'll try it anyway, thank you!
Good tip!
\nYou can however get a memory spike when creating the numpy array from the dataframe. My solution was to convert the columns one by one.
X_train = np.ndarray(shape=(N_ROWS, N_COLS), dtype=np.float32)\nfor idx, feature in enumerate(features):\n X_train[:,idx] = train[feature].values.astype(np.float32)\nthanks @markwijkhuizen , I will try your method for further improvement
", "votes": 0, "replies": [] }, { "id": 1090454, "author_name": "Naresh Jagadeesan", "author_url": "", "post_date": "2020-11-25T10:40:27.570000", "content": "This is the solution that worked for me. Many thanks @markwijkhuizen! However instead of casting it column wise and assigning them to an empty array (this would still exhaust memory if the array you are initializing is large enough). A safer solution would be to take chunks of the DF row wise, cast them to float and do a stack operation. Putting a time.sleep() in between helped me as well.
Also, if you are using features such as timestamp it is probably safer to cast them to np.float64.
Thanks, it works to avoid memory spike but converting to np.float32 looks to modify AUC (from 0.7721 to 0.7742 on one for my test) even if not I'm not using any np.int64 (for timestamp). Pandas dataframe only contains np.int8, bool, np.int32, np.int16, np.float32. That's weird as lgb.Dataset converts everything to np.float32:
\nhttps://lightgbm.readthedocs.io/en/latest/_modules/lightgbm/basic.html#Dataset
\nAnyone got similar issue?
What about categorical features, like user_id? If I convert it to float32 type, does the model recognize it as a categorical features?
Yes, It would. We can specify this using the categorical_feature parameter during model training when we do a lgb.train. We also need to specify the feature_name parameter when we create the lgb.Dataset.
I see, but why do we need to specify feature_name? If this is because of categorical_feature, we can specify feature_names by their index!
The model does not recognize categorical features automatically when using numpy arrays, you need to specify the indices of the columns containing categorical features. The code below is a method I use to obtain these indices and specify them for training.
\ncategorical_feature_idxs = []\nfor v in ['part', 'bundle_id', 'etc']: # your cat features\n # TRAIN_FEATURES is a list with your training features\n categorical_feature_idxs.append(TRAIN_FEATURES.index(v))\n\n# Create dataset\ntrain_data = lgb.Dataset(data=X_train, label=y_train, categorical_feature=None)\n\n# train \nlgb.train(\n train_set = train_data,\n # specify the column indices of the categorical features\n categorical_feature = categorical_feature_idxs,\n)\nIt's like you said, feature_name is needed only when we have specify categorical_feature by their column names. This is not required if we specify the columns using their index. It's just that I find it more convenient to use the column names so as to not keep track of the indices :D
I tried but this converting to np.array makes my training slower :(
", "votes": 0, "replies": [] } ] }, { "id": 1085643, "author_name": "tand", "author_url": "", "post_date": "2020-11-21T03:48:39.750000", "content": "Does this hurt auc score? Although I had same condition (imputation of missing value, and keeping float32), auc score in validation of pandas improves (auc over 0.7), but not in numpy (auc 0.5). I guess part of reason is I use xgboost, not lightgbm
", "votes": 0, "replies": [ { "id": 1087456, "author_name": "", "author_url": "", "post_date": "2020-11-22T18:08:08.843000", "content": "", "votes": 0, "replies": [] }, { "id": 1090439, "author_name": "Naresh Jagadeesan", "author_url": "", "post_date": "2020-11-25T10:32:34.240000", "content": "The problem may arise if when you are using features such as timestamp. Which has a max value much greater than what np.float32 could contain. These values would then be rounded off leading to drop in AUC. Check discussion here.
@doctorkael
\nThanks. So is it about trade-off, getting rid of features like timestamp or setting all features as float64?
You could still use np.float64 without running into memory issues. @markwijkhuizen solution would help you do just that. If column wise casting still causes you memory issues, trying casting rows in chunks, say 10,000 rows at a time, stacking them to a numpy array going all the way to end of your dataset. The problem is caused not since Kaggle RAM can't contain all the data in float64 format (16 GB can actually hold quite a lot more). Rather it is caused due to a sudden memory spike. This can be mitigated by doing the operation ourselves in a slower fashion (time.sleep comes to mind).
>>> lgb.Dataset(dataframe)
would internally be translated to this ->
\n>>> lgb.Dataset(dataframe.values.astype('float'))
Well, not exactly but a rough equivalent. Hence the abrupt memory spike. Hope this helps :)
", "votes": 1, "replies": [] }, { "id": 1091726, "author_name": "tand", "author_url": "", "post_date": "2020-11-26T08:30:40.587000", "content": "@doctorkael
\nThis \"sudden\" memory spike makes sense to me because success/error occurs arbitrary despite same cell trial…
\nI'm using sklearn api and not sure your method could work as same as python-api. But I'll try it anyway, thank you!