I wanted to share a simple pipeline to start a tabular competition for beginners. It's been a long time since Kaggle had a true tabular competition and I wish it was easier for newbies to get started.
\nThe following references will be helpful:
\nLink discussion: Integer columns in the data - here you go!- Raddar have found it: all float type columns have random uniform noise of [0,0.01] added to each column
\nLink discussion: How To Reduce Data Size- Thanks Chris
\nRead the following helpful discussion by Chris if you want a deeper understanding of reduce data size.
Link discussion: The distribution of missing values over time
\nNote: The distribution of the missing B_29 differs between the train and test datasets. Whereas in the training and public leaderboard data >90 % are missing, during the last five months of private leaderboard, we have B_29 data for almost every customer. If we use this feature in our models, we should be prepared for surprises in the private leaderboard
Engineering features is key to improving your LB score. We have seen some basic features from public notebook:
\nLook on
\nBelow are few note on how to engineer new features Feature Engineering Techniques by Chris Deotte of Chris Deotte in IEEE-CIS Fraud Detection competition
\nYou can read my discussions about feature selection here with basic feature selection techniques that everyone should know:
\nThere are many notebooks starting with tree-based model: LightGBM, XGBoost, CatBoost on both CPU and GPU (RAPID) you can refer.
\nSome note about tree-based from AMEX LightGBM Quickstart: Thanks AmbrosM
Preprocessing for LightGBM is much simpler than for neural networks:
\nImportance: Validation score with the competition's scoring function
\nNotebook example for using XGBoost with RAPID & GPU faster: RAPIDS cudf Feature Engineering + XGB
\nXGBoost Tip & Trick: Few note about XGBoost & Feature Engineering/Feature Selection
\nThere are many method for Hyper-parameters optimization like:
\n[UPDATE…]
", "messages": [ { "id": 1825176, "postDate": "2022-06-19T04:06:17.423Z", "content": "I wanted to share a simple pipeline to start a tabular competition for beginners. It's been a long time since Kaggle had a true tabular competition and I wish it was easier for newbies to get started.
\nThe following references will be helpful:
\nLink discussion: Integer columns in the data - here you go!- Raddar have found it: all float type columns have random uniform noise of [0,0.01] added to each column
\nLink discussion: How To Reduce Data Size- Thanks Chris
\nRead the following helpful discussion by Chris if you want a deeper understanding of reduce data size.
Link discussion: The distribution of missing values over time
\nNote: The distribution of the missing B_29 differs between the train and test datasets. Whereas in the training and public leaderboard data >90 % are missing, during the last five months of private leaderboard, we have B_29 data for almost every customer. If we use this feature in our models, we should be prepared for surprises in the private leaderboard
Engineering features is key to improving your LB score. We have seen some basic features from public notebook:
\nLook on
\nBelow are few note on how to engineer new features Feature Engineering Techniques by Chris Deotte of Chris Deotte in IEEE-CIS Fraud Detection competition
\nYou can read my discussions about feature selection here with basic feature selection techniques that everyone should know:
\nThere are many notebooks starting with tree-based model: LightGBM, XGBoost, CatBoost on both CPU and GPU (RAPID) you can refer.
\nSome note about tree-based from AMEX LightGBM Quickstart: Thanks AmbrosM
Preprocessing for LightGBM is much simpler than for neural networks:
\nImportance: Validation score with the competition's scoring function
\nNotebook example for using XGBoost with RAPID & GPU faster: RAPIDS cudf Feature Engineering + XGB
\nXGBoost Tip & Trick: Few note about XGBoost & Feature Engineering/Feature Selection
\nThere are many method for Hyper-parameters optimization like:
\n[UPDATE…]
", "rawMarkdown": "I wanted to share a simple pipeline to start a tabular competition for beginners. It's been a long time since Kaggle had a true tabular competition and I wish it was easier for newbies to get started.\n# Step 1: Look at data + Preprocessing data: EDA, Data cleaning, Reduce data size without loss of information,..\nThe following references will be helpful:\n## Magic data format: random uniform noise added to each column\nLink discussion: [Integer columns in the data - here you go!](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514)- Raddar have found it: **all float type columns have random uniform noise of [0,0.01] added to each column**\n- originally we had 188 float/categorical type features. These were transformed into\n - 95 np.int8/np.int16 types\n - 93 np.float32 types\n- Most float columns with [0, 0.01] and [1, 1.01] have these values rounded up at 0 and 1 respectively. This was done to ensure no data loss, as not all features could be rounded up safely.\n- saved in parquet format (only 1.7GB training data!)\nThis is really helpful when you look at the original data set size (50.31GB), now using Kaggle Notebook is possible and convenient for people with limited resources.\n## Step by Step: How to reduce data size\nLink discussion: [How To Reduce Data Size](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328054)- Thanks Chris\nRead the following helpful discussion by Chris if you want a deeper understanding of reduce data size. \n- Step 1 - Reduce Data Types!: \n - convert int32 or int64 which only use 4 bytes or 8 bytes.\n - Reduce 10 bytes to 3 bytes: date with time\n - Reduce 88 bytes to 11 bytes: categorical columns: \n - 177 Numeric Columns - Reduce 1416 bytes to 353 bytes\n- Step 2 - Choose Your File Format\n- Step 3 - Choose Multiple Files or Not\n- Step 4 - Read Raddar's Discussion\n## Look at data: Missing value\nLink discussion: [The distribution of missing values over time](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328756)\n**Note**: The distribution of the missing B_29 differs between the train and test datasets. Whereas in the training and public leaderboard data >90 % are missing, during the last five months of private leaderboard, we have B_29 data for almost every customer. If we use this feature in our models, we should be prepared for surprises in the private leaderboard\n# Step 2: Feature Engineering: \nEngineering features is key to improving your LB score. We have seen some basic features from public notebook:\n- Aggregated features: [Amex Agg Data How It Created](https://www.kaggle.com/code/huseyincot/amex-agg-data-how-it-created)\n- Aggregated features + Rapid GPU for faster: [XGBoost Starter - [0.793]](https://www.kaggle.com/code/cdeotte/xgboost-starter-0-793)\n- \"After-pay\" features. It makes intuitive semse that subtracting the payments from balance/spend etc provides new information about the users' behavior: [RAPIDS cudf Feature Engineering + XGB](https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb)\n- Statistical features (mean, std, min, max,..): [AMEX LightGBM Quickstart](https://www.kaggle.com/code/ambrosm/amex-lightgbm-quickstart)\n - Selected features averaged over all statements of a customer\n - The minimum or maximum of selected features over all statements of a customer\n - Selected features taken from the last statement of a customer\n\n**Look on**\n1. Features with missing values\n2. Features with low variance\n3. Highly correlated features\n4. Univariate features\n5. Train/Test distributions\n\nBelow are few note on how to engineer new features [Feature Engineering Techniques](https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/108575) by Chris Deotte of Chris Deotte in IEEE-CIS Fraud Detection competition\n\n# Step 3: Feature Selection\nYou can read my discussions about feature selection here with basic feature selection techniques that everyone should know:\n- [Some Feature Selection Technique](https://www.kaggle.com/competitions/optiver-realized-volatility-prediction/discussion/269283)\n- [Few note about XGBoost & Feature Engineering/Feature Selection](https://www.kaggle.com/competitions/foursquare-location-matching/discussion/321992)\n- AmbrosM's following discussion is also really helpful: [Which is the right feature importance?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/331131)- In this post he compare the three methods for feature selection and show that the last one is the right one.\n - Split feature importance?\n - Gain feature importance?\n - Permutation feature importance?\n**Conclusion**: Although permutation feature importance is slower than the gradient boosters' built-in importance measures, it is much more useful because it measures generalization and distinguishes positive from negative importances. Features with negative importance should be dropped from the model.\n\n# Step 4: Baseline Model & Hyper-parameters optimization\nThere are many notebooks starting with tree-based model: LightGBM, XGBoost, CatBoost on both CPU and GPU (RAPID) you can refer.\nSome note about tree-based from [AMEX LightGBM Quickstart](https://www.kaggle.com/code/ambrosm/amex-lightgbm-quickstart): Thanks AmbrosM\n\nPreprocessing for LightGBM is much simpler than for neural networks:\n\n1. Neural networks can't process missing values; LightGBM handles them automatically.\n2. Categorical features need to be one-hot encoded for neural networks; LightGBM handles them automatically.\n3. With neural networks, you need to think about outliers; tree-based algorithms deal with outliers easily.\n4. Neural networks need scaled inputs; tree-based algorithms don't depend on scaling.\n\n**Importance**: Validation score with the competition's scoring function\n\nNotebook example for using XGBoost with RAPID & GPU faster: [RAPIDS cudf Feature Engineering + XGB](https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb)\n\n**XGBoost Tip & Trick**: [Few note about XGBoost & Feature Engineering/Feature Selection](https://www.kaggle.com/competitions/foursquare-location-matching/discussion/321992)\n\n## Hyper-parameters optimization\n\nThere are many method for Hyper-parameters optimization like:\n- RandomizedSearchCV [example](https://www.kaggle.com/code/funxexcel/p3-random-forest-tuning-randomizedsearchcv), GridSearchCV [example](https://www.kaggle.com/code/ihelon/titanic-hyperparameter-tuning-with-gridsearchcv)\n- Bayesian Optimization: [example](https://www.kaggle.com/code/vincentlugat/ieee-lgb-bayesian-opt)\n- Optuna: [example](https://www.kaggle.com/code/hamzaghanmi/lgbm-hyperparameter-tuning-using-optuna)\n\n# Step 5: Ensemble Models\n\n- Weighted/Average Ensemble\n- Stacking Model\n\n**[UPDATE...]**", "votes": 57 }, { "id": 1827200, "postDate": "2022-06-20T23:19:09.310Z", "content": "Thanks for sharing this .. this will help a lot!
", "rawMarkdown": "Thanks for sharing this .. this will help a lot!" }, { "id": 1826257, "postDate": "2022-06-20T07:49:51.530Z", "content": "Nice summary. The sentence \"Categorical features need to be one-hot encoded for neural networks\" should be softened, NNs can work with any mapping of categorical features e.g. LabelEncoder but one-hot encoding is one of the best performing (because it also works well with ordinal and non ordinal features)
", "rawMarkdown": "Nice summary. The sentence \"Categorical features need to be one-hot encoded for neural networks\" should be softened, NNs can work with any mapping of categorical features e.g. LabelEncoder but one-hot encoding is one of the best performing (because it also works well with ordinal and non ordinal features)" }, { "id": 1827249, "postDate": "2022-06-21T00:31:14.637Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1828172, "postDate": "2022-06-21T14:18:48.413Z", "content": "Okay, i will update soon!
", "rawMarkdown": "Okay, i will update soon!" } ] }, { "id": 1912435, "postDate": "2022-08-24T18:09:17.327Z", "content": "Thanks for sharing this
", "rawMarkdown": "Thanks for sharing this" }, { "id": 1858593, "postDate": "2022-07-17T03:54:07.247Z", "content": "Very helpful. Thank you
", "rawMarkdown": "Very helpful. Thank you" } ], "comments": [ { "id": 1827200, "author_name": "Chirag Desai", "author_url": "", "post_date": "2022-06-20T23:19:09.310000", "content": "Thanks for sharing this .. this will help a lot!
", "votes": 0, "replies": [] }, { "id": 1826257, "author_name": "Ali Abdin", "author_url": "", "post_date": "2022-06-20T07:49:51.530000", "content": "Nice summary. The sentence \"Categorical features need to be one-hot encoded for neural networks\" should be softened, NNs can work with any mapping of categorical features e.g. LabelEncoder but one-hot encoding is one of the best performing (because it also works well with ordinal and non ordinal features)
", "votes": 0, "replies": [] }, { "id": 1827249, "author_name": "", "author_url": "", "post_date": "2022-06-21T00:31:14.637000", "content": "", "votes": 0, "replies": [ { "id": 1828172, "author_name": "KhanhVD", "author_url": "", "post_date": "2022-06-21T14:18:48.413000", "content": "Okay, i will update soon!
", "votes": 0, "replies": [] } ] }, { "id": 1912435, "author_name": "Gaju Ahmed", "author_url": "", "post_date": "2022-08-24T18:09:17.327000", "content": "Thanks for sharing this
", "votes": 0, "replies": [] }, { "id": 1858593, "author_name": "Little bird", "author_url": "", "post_date": "2022-07-17T03:54:07.247000", "content": "Very helpful. Thank you
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1825176": "I wanted to share a simple pipeline to start a tabular competition for beginners. It's been a long time since Kaggle had a true tabular competition and I wish it was easier for newbies to get started.\n# Step 1: Look at data + Preprocessing data: EDA, Data cleaning, Reduce data size without loss of information,..\nThe following references will be helpful:\n## Magic data format: random uniform noise added to each column\nLink discussion: [Integer columns in the data - here you go!](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514)- Raddar have found it: **all float type columns have random uniform noise of [0,0.01] added to each column**\n- originally we had 188 float/categorical type features. These were transformed into\n - 95 np.int8/np.int16 types\n - 93 np.float32 types\n- Most float columns with [0, 0.01] and [1, 1.01] have these values rounded up at 0 and 1 respectively. This was done to ensure no data loss, as not all features could be rounded up safely.\n- saved in parquet format (only 1.7GB training data!)\nThis is really helpful when you look at the original data set size (50.31GB), now using Kaggle Notebook is possible and convenient for people with limited resources.\n## Step by Step: How to reduce data size\nLink discussion: [How To Reduce Data Size](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328054)- Thanks Chris\nRead the following helpful discussion by Chris if you want a deeper understanding of reduce data size. \n- Step 1 - Reduce Data Types!: \n - convert int32 or int64 which only use 4 bytes or 8 bytes.\n - Reduce 10 bytes to 3 bytes: date with time\n - Reduce 88 bytes to 11 bytes: categorical columns: \n - 177 Numeric Columns - Reduce 1416 bytes to 353 bytes\n- Step 2 - Choose Your File Format\n- Step 3 - Choose Multiple Files or Not\n- Step 4 - Read Raddar's Discussion\n## Look at data: Missing value\nLink discussion: [The distribution of missing values over time](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328756)\n**Note**: The distribution of the missing B_29 differs between the train and test datasets. Whereas in the training and public leaderboard data >90 % are missing, during the last five months of private leaderboard, we have B_29 data for almost every customer. If we use this feature in our models, we should be prepared for surprises in the private leaderboard\n# Step 2: Feature Engineering: \nEngineering features is key to improving your LB score. We have seen some basic features from public notebook:\n- Aggregated features: [Amex Agg Data How It Created](https://www.kaggle.com/code/huseyincot/amex-agg-data-how-it-created)\n- Aggregated features + Rapid GPU for faster: [XGBoost Starter - [0.793]](https://www.kaggle.com/code/cdeotte/xgboost-starter-0-793)\n- \"After-pay\" features. It makes intuitive semse that subtracting the payments from balance/spend etc provides new information about the users' behavior: [RAPIDS cudf Feature Engineering + XGB](https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb)\n- Statistical features (mean, std, min, max,..): [AMEX LightGBM Quickstart](https://www.kaggle.com/code/ambrosm/amex-lightgbm-quickstart)\n - Selected features averaged over all statements of a customer\n - The minimum or maximum of selected features over all statements of a customer\n - Selected features taken from the last statement of a customer\n\n**Look on**\n1. Features with missing values\n2. Features with low variance\n3. Highly correlated features\n4. Univariate features\n5. Train/Test distributions\n\nBelow are few note on how to engineer new features [Feature Engineering Techniques](https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/108575) by Chris Deotte of Chris Deotte in IEEE-CIS Fraud Detection competition\n\n# Step 3: Feature Selection\nYou can read my discussions about feature selection here with basic feature selection techniques that everyone should know:\n- [Some Feature Selection Technique](https://www.kaggle.com/competitions/optiver-realized-volatility-prediction/discussion/269283)\n- [Few note about XGBoost & Feature Engineering/Feature Selection](https://www.kaggle.com/competitions/foursquare-location-matching/discussion/321992)\n- AmbrosM's following discussion is also really helpful: [Which is the right feature importance?](https://www.kaggle.com/competitions/amex-default-prediction/discussion/331131)- In this post he compare the three methods for feature selection and show that the last one is the right one.\n - Split feature importance?\n - Gain feature importance?\n - Permutation feature importance?\n**Conclusion**: Although permutation feature importance is slower than the gradient boosters' built-in importance measures, it is much more useful because it measures generalization and distinguishes positive from negative importances. Features with negative importance should be dropped from the model.\n\n# Step 4: Baseline Model & Hyper-parameters optimization\nThere are many notebooks starting with tree-based model: LightGBM, XGBoost, CatBoost on both CPU and GPU (RAPID) you can refer.\nSome note about tree-based from [AMEX LightGBM Quickstart](https://www.kaggle.com/code/ambrosm/amex-lightgbm-quickstart): Thanks AmbrosM\n\nPreprocessing for LightGBM is much simpler than for neural networks:\n\n1. Neural networks can't process missing values; LightGBM handles them automatically.\n2. Categorical features need to be one-hot encoded for neural networks; LightGBM handles them automatically.\n3. With neural networks, you need to think about outliers; tree-based algorithms deal with outliers easily.\n4. Neural networks need scaled inputs; tree-based algorithms don't depend on scaling.\n\n**Importance**: Validation score with the competition's scoring function\n\nNotebook example for using XGBoost with RAPID & GPU faster: [RAPIDS cudf Feature Engineering + XGB](https://www.kaggle.com/code/jiweiliu/rapids-cudf-feature-engineering-xgb)\n\n**XGBoost Tip & Trick**: [Few note about XGBoost & Feature Engineering/Feature Selection](https://www.kaggle.com/competitions/foursquare-location-matching/discussion/321992)\n\n## Hyper-parameters optimization\n\nThere are many method for Hyper-parameters optimization like:\n- RandomizedSearchCV [example](https://www.kaggle.com/code/funxexcel/p3-random-forest-tuning-randomizedsearchcv), GridSearchCV [example](https://www.kaggle.com/code/ihelon/titanic-hyperparameter-tuning-with-gridsearchcv)\n- Bayesian Optimization: [example](https://www.kaggle.com/code/vincentlugat/ieee-lgb-bayesian-opt)\n- Optuna: [example](https://www.kaggle.com/code/hamzaghanmi/lgbm-hyperparameter-tuning-using-optuna)\n\n# Step 5: Ensemble Models\n\n- Weighted/Average Ensemble\n- Stacking Model\n\n**[UPDATE...]**", "1827200": "Thanks for sharing this .. this will help a lot!", "1826257": "Nice summary. The sentence \"Categorical features need to be one-hot encoded for neural networks\" should be softened, NNs can work with any mapping of categorical features e.g. LabelEncoder but one-hot encoding is one of the best performing (because it also works well with ordinal and non ordinal features)", "1827249": "", "1912435": "Thanks for sharing this", "1858593": "Very helpful. Thank you" } }