I'm stuck in timeout problem recently. I test nearly 9k features in training and select 1k-2k features for 3 level-group models respectively. However, in inference part, I still need to generate all the 9k features and then select the useful features by loading the importance dict from the training result. Seems this process will cost too much time.
\nIs there any way too avoid these redundant features generate again during inference?
I'm stuck in timeout problem recently. I test nearly 9k features in training and select 1k-2k features for 3 level-group models respectively. However, in inference part, I still need to generate all the 9k features and then select the useful features by loading the importance dict from the training result. Seems this process will cost too much time.
\nIs there any way too avoid these redundant features generate again during inference?
hey,
\nIf we don't know how you create the features we can't tell you what you should change,
\nhowever, from my understanding of your problem, you have features that are created looping over rooms, fqid and other stuff and you don't want all of them but only specific ones correct ? if so, consider creating a new list that is kind of a subset of the original fqid or room list too loop over less items.
\nIf your problem is that all of your 9k features are useful but not for every model, a simple if statement would do the job where you would only create them for specific level_groups, which can easily be done since for inference they give us data level groups at a time.
Off topic, but you said you only used 3 models, you seem to be doing fine but check out my discussion on the topic here, maybe you'll try it out and get an even better score. I gave arguments to why you shoundn't consider your approach.
", "rawMarkdown": "hey, \nIf we don't know how you create the features we can't tell you what you should change, \nhowever, from my understanding of your problem, you have features that are created looping over rooms, fqid and other stuff and you **don't want all of them but only specific ones** correct ? if so, consider creating a new list that is kind of a subset of the original fqid or room list too loop over less items.\nIf your problem is that all of your 9k features are useful but not for every model, a simple `if` statement would do the job where you would only create them for specific level_groups, which can easily be done since for **inference they give us data level groups at a time.**\n\n*Off topic*, but you said you only used 3 models, you seem to be doing fine but check out my discussion on the topic [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/393614), maybe you'll try it out and get an even better score. I gave arguments to why you shoundn't consider your approach.", "votes": null }, { "id": "2182254", "postDate": "03/15/2023 02:54:43", "content": "One last thing, have you considered polars? it's a pandas's like library that has way faster computing speed. If not, give it a try.
", "rawMarkdown": "One last thing, have you considered **polars**? it's a pandas's like library that has way faster computing speed. If not, give it a try.", "votes": null }, { "id": "2182721", "postDate": "03/15/2023 09:11:42", "content": "I reuse the @cdeotte xgb baseline to make a proof of concept on how to estimate inference time and how to debug bottlenecks in inference pipeline. you can find notebook here
", "rawMarkdown": "I reuse the @cdeotte xgb baseline to make a proof of concept on how to estimate inference time and how to debug bottlenecks in inference pipeline. you can find notebook [here](https://www.kaggle.com/code/steubk/xgboost-baseline-and-inference-time-estimation)", "votes": null }, { "id": "2182892", "postDate": "03/15/2023 11:20:04", "content": "Completely agree with @janmpia on this one - using subsets of the column values you want to calculate, if statements to only include specific aggregates, and using polars are all great steps.
\nIt could also be worth making a submission (or running the excellent inference time estimation notebook that @steubk shared) with just your feature engineering to see how long that takes and compare it to the model predictions.
\nOut of interest, how are you selecting your features? Is it just using the top feature importances from the models with a different number of features for each level group, or are you doing something different like permutation importance or recursively removing features to find the optimal subset for each model? I think feature selection is probably the next key step for these models, and I'm still playing around with some different methods so would be really interested in hearing how you're selecting your features!
", "rawMarkdown": "Completely agree with @janmpia on this one - using subsets of the column values you want to calculate, if statements to only include specific aggregates, and using polars are all great steps. \n\nIt could also be worth making a submission (or running the excellent inference time estimation notebook that @steubk shared) with just your feature engineering to see how long that takes and compare it to the model predictions.\n\nOut of interest, how are you selecting your features? Is it just using the top feature importances from the models with a different number of features for each level group, or are you doing something different like permutation importance or recursively removing features to find the optimal subset for each model? I think feature selection is probably the next key step for these models, and I'm still playing around with some different methods so would be really interested in hearing how you're selecting your features!", "votes": null }, { "id": "2182897", "postDate": "03/15/2023 11:22:10", "content": "And speaking of polars, @carnozhao made these great notebooks showing how we can use polars at inference time. I think they've got another notebook somewhere for training too :) https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-inference
", "rawMarkdown": "And speaking of polars, @carnozhao made these great notebooks showing how we can use polars at inference time. I think they've got another notebook somewhere for training too :) https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-inference", "votes": null }, { "id": "2183452", "postDate": "03/15/2023 17:09:11", "content": "Based on my experiments, if you can run your inference code within 8-9s with the sample test dataset, it will not be timeout when submitting
", "rawMarkdown": "Based on my experiments, if you can run your inference code within 8-9s with the sample test dataset, it will not be timeout when submitting", "votes": null }, { "id": "2183850", "postDate": "03/16/2023 00:59:57", "content": "Thanks a lot. It's really helpful for me!
", "rawMarkdown": "Thanks a lot. It's really helpful for me!", "votes": null }, { "id": "2183852", "postDate": "03/16/2023 01:01:17", "content": "Thanks for your reply. I have used polars. Maybe I should loop only specific items as you said.
", "rawMarkdown": "Thanks for your reply. I have used polars. Maybe I should loop only specific items as you said.", "votes": null }, { "id": "2183858", "postDate": "03/16/2023 01:07:44", "content": "Thanks for your reply, I will try as u said. I simply add the remove collinear variables method to the baseline. Here is my feature selection log. By the way, your \"Saving predictions from previous predictions\" helps me a lot :)
\n:)
\ndf1 shape: (11779, 9344)
\ndf2 shape: (11779, 9356)
\ndf3 shape: (11779, 9352)
we need to drop number of columns with over 90% nan values in df1: 6788
\nwe need to drop number of columns with over 90% nan values in df2: 5263
\nwe need to drop number of columns with over 90% nan values in df3: 4705
then we need to drop number of columns with all the same value in df1: 211
\nthen we need to drop number of columns with all the same value in df2: 153
\nthen we need to drop number of columns with all the same value in df3: 112
then we need to drop number of Collinear Variables in df1. 1360
\nthen we need to drop number of Collinear Variables in df2. 2163
\nthen we need to drop number of Collinear Variables in df3. 2374
we need to drop total number of columns in df1: 8359
\nwe need to drop total number of columns in df2: 7579
\nwe need to drop total number of columns in df3: 7191
One thing that I found really helpful especially for polars is that you can select features even at the feature generation process.
\nFor example,
pruned_cols = []\nfor col in cols:\n if select_col(col):\n pruned_cols.append(col)\ndf = df.groupby(\"session_id\", \"level_group).agg(pruned_cols)\nThe select_col function might check whether the column's name is in the list of features that you want. With this code snippet, you only generate the features that you need.
hey,
\nIf we don't know how you create the features we can't tell you what you should change,
\nhowever, from my understanding of your problem, you have features that are created looping over rooms, fqid and other stuff and you don't want all of them but only specific ones correct ? if so, consider creating a new list that is kind of a subset of the original fqid or room list too loop over less items.
\nIf your problem is that all of your 9k features are useful but not for every model, a simple if statement would do the job where you would only create them for specific level_groups, which can easily be done since for inference they give us data level groups at a time.
Off topic, but you said you only used 3 models, you seem to be doing fine but check out my discussion on the topic here, maybe you'll try it out and get an even better score. I gave arguments to why you shoundn't consider your approach.
", "votes": null, "replies": [ { "id": 2182254, "author_name": "janmpia", "author_url": "", "post_date": "03/15/2023 02:54:43", "content": "One last thing, have you considered polars? it's a pandas's like library that has way faster computing speed. If not, give it a try.
", "votes": null, "replies": [ { "id": 2183852, "author_name": "chris2042", "author_url": "", "post_date": "03/16/2023 01:01:17", "content": "Thanks for your reply. I have used polars. Maybe I should loop only specific items as you said.
", "votes": null, "replies": [] } ] } ] }, { "id": 2182721, "author_name": "steubk", "author_url": "", "post_date": "03/15/2023 09:11:42", "content": "I reuse the @cdeotte xgb baseline to make a proof of concept on how to estimate inference time and how to debug bottlenecks in inference pipeline. you can find notebook here
", "votes": null, "replies": [ { "id": 2183850, "author_name": "chris2042", "author_url": "", "post_date": "03/16/2023 00:59:57", "content": "Thanks a lot. It's really helpful for me!
", "votes": null, "replies": [] } ] }, { "id": 2182892, "author_name": "judehunt23", "author_url": "", "post_date": "03/15/2023 11:20:04", "content": "Completely agree with @janmpia on this one - using subsets of the column values you want to calculate, if statements to only include specific aggregates, and using polars are all great steps.
\nIt could also be worth making a submission (or running the excellent inference time estimation notebook that @steubk shared) with just your feature engineering to see how long that takes and compare it to the model predictions.
\nOut of interest, how are you selecting your features? Is it just using the top feature importances from the models with a different number of features for each level group, or are you doing something different like permutation importance or recursively removing features to find the optimal subset for each model? I think feature selection is probably the next key step for these models, and I'm still playing around with some different methods so would be really interested in hearing how you're selecting your features!
", "votes": null, "replies": [ { "id": 2182897, "author_name": "judehunt23", "author_url": "", "post_date": "03/15/2023 11:22:10", "content": "And speaking of polars, @carnozhao made these great notebooks showing how we can use polars at inference time. I think they've got another notebook somewhere for training too :) https://www.kaggle.com/code/carnozhao/cpu-catboost-baseline-using-polars-inference
", "votes": null, "replies": [ { "id": 2183858, "author_name": "chris2042", "author_url": "", "post_date": "03/16/2023 01:07:44", "content": "Thanks for your reply, I will try as u said. I simply add the remove collinear variables method to the baseline. Here is my feature selection log. By the way, your \"Saving predictions from previous predictions\" helps me a lot :)
\n:)
\ndf1 shape: (11779, 9344)
\ndf2 shape: (11779, 9356)
\ndf3 shape: (11779, 9352)
we need to drop number of columns with over 90% nan values in df1: 6788
\nwe need to drop number of columns with over 90% nan values in df2: 5263
\nwe need to drop number of columns with over 90% nan values in df3: 4705
then we need to drop number of columns with all the same value in df1: 211
\nthen we need to drop number of columns with all the same value in df2: 153
\nthen we need to drop number of columns with all the same value in df3: 112
then we need to drop number of Collinear Variables in df1. 1360
\nthen we need to drop number of Collinear Variables in df2. 2163
\nthen we need to drop number of Collinear Variables in df3. 2374
we need to drop total number of columns in df1: 8359
\nwe need to drop total number of columns in df2: 7579
\nwe need to drop total number of columns in df3: 7191
Based on my experiments, if you can run your inference code within 8-9s with the sample test dataset, it will not be timeout when submitting
", "votes": null, "replies": [] }, { "id": 2188800, "author_name": "mrhantato", "author_url": "", "post_date": "03/20/2023 02:08:48", "content": "One thing that I found really helpful especially for polars is that you can select features even at the feature generation process.
\nFor example,
pruned_cols = []\nfor col in cols:\n if select_col(col):\n pruned_cols.append(col)\ndf = df.groupby(\"session_id\", \"level_group).agg(pruned_cols)\nThe select_col function might check whether the column's name is in the list of features that you want. With this code snippet, you only generate the features that you need.