I have found it quite challenging to run LGBM successfully on a free Colab, Paperspace or Kaggle instance without running out of memory
\nI modified @ragnar123's notebook to run successfully within a Kaggle notebook, however in the process I sacrificed CV (just trusting their hyperparameters for now) and also CV ensembling as a result. This was done using incremental training to load the train data in 5 parts, and continuously train the model while dumping previous data through the use of parameters keep_training_booster=True and init_model = model.
However, I noticed that my model performs significantly worse. at 0.785 with a single split and no lag features, and 0.793 by ensembling multiple seeds (for the incremental training splits) and adding lag features.
\nI don't fully understand why, but I could guess that there would be a difference between training on the full train set for n rounds, versus training on one fifth of the train set each time, for n rounds, totaling 5n rounds.
\nI am aware that there is a method to iteratively pass data to XGBoost using DeviceQuantileDMatrix, which seems to function identically to passing the full dataset initially.
Would the corresponding method with LGBM be using the Sequence interface in LGBM instead of using init_model and training multiple times?
I have found it quite challenging to run LGBM successfully on a free Colab, Paperspace or Kaggle instance without running out of memory
\nI modified @ragnar123's notebook to run successfully within a Kaggle notebook, however in the process I sacrificed CV (just trusting their hyperparameters for now) and also CV ensembling as a result. This was done using incremental training to load the train data in 5 parts, and continuously train the model while dumping previous data through the use of parameters keep_training_booster=True and init_model = model.
However, I noticed that my model performs significantly worse. at 0.785 with a single split and no lag features, and 0.793 by ensembling multiple seeds (for the incremental training splits) and adding lag features.
\nI don't fully understand why, but I could guess that there would be a difference between training on the full train set for n rounds, versus training on one fifth of the train set each time, for n rounds, totaling 5n rounds.
\nI am aware that there is a method to iteratively pass data to XGBoost using DeviceQuantileDMatrix, which seems to function identically to passing the full dataset initially.
Would the corresponding method with LGBM be using the Sequence interface in LGBM instead of using init_model and training multiple times?
To my understanding neither XGBoost nor LGBM support incremental training in the sense that you could train the model on data and than continue training on never seen data. The continue training feature is thought to continue training on the same data (more epochs). For clarification: The same data means data that the model has already seen, not data from the same Kaggle competition. :-)
\nThe main reasons seem to be:
\nThe above points are from a discussion on the XGBoost github about continued learing.
\nOne of the participants adds:
\nHowever, there are applications when training continuation in new data makes good practical sense. E.g., in a situation when you get some new data that is related but has some sort of \"concept drift\", there are sometimes good chances that by taking an old model learned in old data as \"prior knowledge\", and adapting it to the new data by training continuation in that new data, you would get a better performing model for future use in data that would be like the new data than when training from scratch either with only this new data or with a combined sample of old + new data.
So if you like you could try to split the data per month, train and predict for that month and then continue training the model on the next month and use the final model for inference. But I'm not sure if that helps with the current data.
\nAs you already mentioned you could pass the whole data incrementally to the model with DeviceQuantileDMatrix or a similar dataloader.
\nThe thing is that if you load the data from files, then random access to that data is hard to implement with csv or parquet, so
\n you probably don't want to use shuffling or splits over the whole dataset.
\nOne thing you could do is to split and shuffle the data into files beforehand and then yield them from the dataloader, meaning the index going into the dataloader to get the data is the number of the file and not the row of the dataset, thus the length returned by the dataloader is the number of files.
Here are some suggestions for \"train a model on a larger-than-memory dataset on a single node\" by an LGBM Maintainer:
\nWhat? Use Training Continuation? How?
\nUnder the above link you'll find code.
\nThe trick seems to be: \"reference=first_dataset\" in the loop over files.
dataset = lgb.Dataset(\n next_file,\n params={\"label_column\": \"y\"},\n free_raw_data=True,\n reference=first_dataset\n )\nThis seems to align the bin mapper for the dataset.
\nI'm not sure about how this influences the training continuation:
\n\nIf the distribution of data is very different, the bin mapper could be very different.
\n
I don't know it they handle that completely in their dataloader or if one has to check that the distribution of values in the samples is similar beforehand.
\nIf you try this approch a followup would be nice. I'd like to know if that works for our data.
Another, maybe easier idea is to take an ensemble approach.
\nSplit the data into multiple batches and train a model on each.
\nThen at inference time simply make a prediction with each model using all the new data and take the average to get the final prediction.
To my understanding neither XGBoost nor LGBM support incremental training in the sense that you could train the model on data and than continue training on never seen data. The continue training feature is thought to continue training on the same data (more epochs). For clarification: The same data means data that the model has already seen, not data from the same Kaggle competition. :-)
\nThe main reasons seem to be:
\nThe above points are from a discussion on the XGBoost github about continued learing.
\nOne of the participants adds:
\nHowever, there are applications when training continuation in new data makes good practical sense. E.g., in a situation when you get some new data that is related but has some sort of \"concept drift\", there are sometimes good chances that by taking an old model learned in old data as \"prior knowledge\", and adapting it to the new data by training continuation in that new data, you would get a better performing model for future use in data that would be like the new data than when training from scratch either with only this new data or with a combined sample of old + new data.
So if you like you could try to split the data per month, train and predict for that month and then continue training the model on the next month and use the final model for inference. But I'm not sure if that helps with the current data.
\nAs you already mentioned you could pass the whole data incrementally to the model with DeviceQuantileDMatrix or a similar dataloader.
\nThe thing is that if you load the data from files, then random access to that data is hard to implement with csv or parquet, so
\n you probably don't want to use shuffling or splits over the whole dataset.
\nOne thing you could do is to split and shuffle the data into files beforehand and then yield them from the dataloader, meaning the index going into the dataloader to get the data is the number of the file and not the row of the dataset, thus the length returned by the dataloader is the number of files.
Here are some suggestions for \"train a model on a larger-than-memory dataset on a single node\" by an LGBM Maintainer:
\nWhat? Use Training Continuation? How?
\nUnder the above link you'll find code.
\nThe trick seems to be: \"reference=first_dataset\" in the loop over files.
dataset = lgb.Dataset(\n next_file,\n params={\"label_column\": \"y\"},\n free_raw_data=True,\n reference=first_dataset\n )\nThis seems to align the bin mapper for the dataset.
\nI'm not sure about how this influences the training continuation:
\n\nIf the distribution of data is very different, the bin mapper could be very different.
\n
I don't know it they handle that completely in their dataloader or if one has to check that the distribution of values in the samples is similar beforehand.
\nIf you try this approch a followup would be nice. I'd like to know if that works for our data.
Another, maybe easier idea is to take an ensemble approach.
\nSplit the data into multiple batches and train a model on each.
\nThen at inference time simply make a prediction with each model using all the new data and take the average to get the final prediction.