![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2405813%2F7db49977cfebf34a15d87e5b4c95ec3f%2F2022-08-28%20%2012.41.40.png?generation=1661686959865932&alt=media\")
Note: Full code to retrain single model will be shared here in a 2 weeks.
\nI would like to say thank you to competition hosts and Kaggle - it was a great pleasure to participate in tabular data competition after many months and years without one.
\nThank you to all participants (and of course winners - @daishu huge jump during last 3 days and fantastic solo result) - our success is your success - you forced us to try harder - without all of you It would be impossible to learn so many new things and achieve such result.
\nAnd special \"thank you\" goes to my fantastic teammates:
\nNo words would be enough to say how much each of you contributed to the end result.
\n\"Do Data Scientists have hobby? Yes -> DS competition\".
We were very hyped by the new tabular data competition release (sorry for the external link: link) and immediately decided to participate. Slack notification -> rules and perspective advertisement (100% chance too lose summer holidays and all free time) -> and here we are - four members. Only one member of the team had previous experience in DS competitions participation.
\nFew rules were established from the beginning:
\nEach of us had own machines / resources (GCP/AWS/local). We used Kaggle platform just for a few times. So the first thing we wanted to solve - unified machine to save all artefacts / experiments. We decided to go with AWS. I would say that it is possible to achieve the same result that we have just with Kaggle resources, but it would be bit more stressful for team management. We didn't want to spend a lot of money on AWS but sometimes (during very hot hours) RAM spikes were 500GB+ to permit simultaneous work.
\nWe tried to use neptune.ai for ML tracking but from July it was not very effective as we entered in brute force zone.
\nAdvise: Resources management is very critical - find bottleneck and remove it to make your team most effective. At the same time don't burn money recklessly - limit your budged. If any optimization possible - do it as soon as possible to save time and resources.
\nEach run was internally versioned (ex. v1.1.1 - (major version).(fe version).(model version))
\nOverall project structure:
All these permitted us to go back and forward and check what worked well and what did not and restore experiments in each particular step.
\nWe wanted to achieve several things with this step:
\nWe didn't use public notebooks for cleaning. Radar's Dataset is fantastic and it is 99% similar to our own transformations.
\nWe used \"isle\" identification without any pre-build coefficients.
\ndummy code is something like this:
for col in process_columns: \n\n df = temp_df[[col]].sort_values(by=[col]) \n df = df[df[col].notna()].drop_duplicates(subset=[col]).reset_index(drop=True)\n\n df['temp'] = np.floor(df[col] * 100000)\n df['group'] = ((df['temp'] - df['temp'].shift()).abs() >= 100).cumsum()\n\n i = 0\n while True:\n min_val = df[df['group']==i]['temp'].min()\n if min_val>0:\n break\n i += 1\n\n df['temp2'] = np.where(df['temp']>=0, \n np.floor(df['temp']/min_val).astype(np.int32),\n np.round(df['temp']/min_val).astype(np.int32))\n\n mapping = dict(zip(df[col],df['temp2']))\n temp_df[col] = temp_df[col].map(mapping)\n\n print(col, df['group'].nunique(), df[col].nunique())\n print(df.groupby(['group'])['temp','temp2'].agg(['min','max','count','nunique']).head(40))\nWe used last statement month to create 0/1/2 feature and store in in \"index\" df
\nFixed random seed (of course 42) to make spits and then took 20% of customers to holdout group (to test stacking / blending / etc)
\nAfter cleaning we had several columns \"groups\".
\nall_files = [\n'p_columns', -> just p columns as we thought that they are very different (and P_2 is internal amex \"scoring\" model)\n'objects_radar_columns', -> order encoding (we were checking where out cleaning differs from public approaches and here was the unique place) \n'objects_columns', -> onehot encoding\n'categorical_cleaned__D__columns', -> no noise categoricals\n'categorical_binary__S__columns', -> cleaned binary\n'categorical_binary__R__columns', -> cleaned binary\n'categorical_binary__D__columns', -> cleaned binary\n'categorical_binary__B__columns', -> cleaned binary\n'categorical__D__columns', -> removed noise categoricals\n'categorical__B__columns', -> removed noise categoricals\n'cleaned__B__columns', -> removed noise continuous \n'cleaned__D__columns', -> removed noise continuous \n'cleaned__R__columns', -> removed noise continuous \n'cleaned__S__columns', -> removed noise continuous \n'rest__B__columns', -> have no idea what to do with it -> floor \n'rest__D__columns', -> have no idea what to do with it -> floor \n'rest__R__columns', -> have no idea what to do with it -> floor \n'rest__S__columns', -> have no idea what to do with it -> floor \n]\nThanks again to @raddar we always used your preprocess as a baseline.
\nWe were able to load just portion of data -> do fe -> concat to \"index\" as all dfs were aligned by index. Also such split permitted us to do fe by feature type to accelerate process and see more statistically valuable metric change.
\nWe started with careful fe column by column or small subset and it worked well until 1xx features and then any metric improvement or degradation was not statistically significant and many features \"overlapped\" on importance and significance.
\nNote: I believe that it is possible to build silver zone robust model with only 3xx features
\nSo we started from scratch with brute force))) Of course there was no need to apply \"nunique\" (for example to binary features) and our previous step helped us to limit fe.
\nall_aggregations = {\n 'agg_func': ['last','mean','std','median','min','max','nunique'],\n 'diff_func': ['first','mean','std','median','min','max'],\n 'ratio_func': ['first','mean','std','median','min','max'],\n 'lags': [1,2,3,6,11],\n 'special': ['ewm','count_month_enc','monotonic_increase','diff_mean','major_class',\n 'normalization','top_outlier','bottom_outlier','normalization_mean','top_outlier_mean','top_outlier_mean']\n} \nagg_func -> normal aggregations by uid
\ndiff_func -> diff last - xxx -> std diff worked better than any other
\nratio_func -> ratio_func last/xxx
\nlags -> diff last - Nx
\nspecial -> some special transformations -> count_month_enc worked well for categorical / emw for continous
We ended up with about 7k features (stored file by group and by agg type for faster loading).
\nNext thing was to figure out what works and what not -> this topic was the most challenging for us.
It's better to call it Standardization (x - m) / s -> as we had also normalization test the name became constant \"normalization\")))
\ndf.groupby(['dt_month','subset'])[col].agg(['mean','std'])\ndt_month -> month of the statement
\nsubset -> train / public / private
\nand mean and std from clients that had full statement history.
We have to have temporal shift to make it work. So we did a \"trick\" removed last statement for each client and applied exactly same transformation for each client and merged appropriate labels. So we had 2 lines in training set for almost each client BUT validated results only on last statement during CV runs and Holdout checks. It more or less same as adding noised data but we had temporal drift and model was able to work better on unknown future data with \"possible\" data drift.
\nOoohh that was really fun.
\nWe used gbdt boosting type during experiments as it was very aligned with dart mode but was significantly faster.
\nAlso, we used ROC AUC score during our experiments as we believed that due to amex instability we can't use it for decision making (of course we tracked log loss and amex).
In previous step we brute forced many features and now is time to clean them out.
\nAll feature selection was done with 5 CV folds training + independent check on 20% holdout data.
Zero importance -> Right from the start we were able to through away 1.5k features that had exactly 0 importance (lgbm importance). That means that with 250 bins and 2**10 data in leaf those features are not participating in any split.
Stepped hierarchical permutation importance -> we defined 300 initial features and looped over all other features subsets (600+) - was very time consuming but very stable.
\nNote: we shuffled order of the subset to force model try different combinations.
\nAdd features subset -> train model -> permutate -> drop negative features (negative mean over 5 seeds) -> add new subset -> …
\nDuring this part that took almost 3 days we limited features to 3k -> 0.800 lb
Stepped permutation importance.
\nTake all features -> train model -> permutate -> drop 20% of worst performed features (only negative) -> repeat. Final subset was 25xx features (and different from previous step) -> 0.800 lb
Forward feature selection.
\nWe defined 300 initial features and simply added subset by subset and compared ROC AUC if metric change was > 0.0003 we kept the subset. -> 0.800 lb
Time series CV.
\nFor very doubtful features as PCA and Normilized values we used to different validation stratagies:
Many other options we tried but result was not stable.
\nFinal subset came from \"Forward feature selection\" plus overlapped features from other technics minus overlapped negative combination. -> lb 0.801 single model.
\nWe tried to blend many models with different subset as we believed that it should give huge LB boost (based on holdout blending tests) but it didn't work well for lb.
\nIn my own experience, DART never worked better and here we have proof that in DS \"all depends.\" We did experiments with DART in the beginning and it did not show any metric improvement with our params and baseline model features subset. Later we found @ragnar123 notebook and gave it one more try and it worked marvelously.
\nFrom the beginning, we tried to build a more complex model with 2**7+ leaves and 0.7+ features but failed. It still puzzles me why a simple model with a very low number of features works here.
\nI saw such behaviour mostly on synthetic data and stacking - so we tried to find out if data is syntetic (at least partly) and deanonimize internal scoring values - but didn't make it.
\nOur best single lgbm model was trained on 29xx features. 5 folds CV - no stratification by any option. Training data - 2 last staements for each client (transformed independently). Params:
\nlgb_params = {\n 'boosting_type': 'dart',\n 'objective': 'cross_entropy', \n 'metric': ['AUC'],\n 'subsample': 0.8, \n 'subsample_freq': 1,\n 'learning_rate': 0.01, \n 'num_leaves': 2 ** 6, \n 'min_data_in_leaf': 2 ** 11, \n 'feature_fraction': 0.2, \n 'feature_fraction_bynode':0.3,\n 'first_metric_only': True,\n 'n_estimators': 17001, # -> 5000 for gbdt \n 'boost_from_average': False,\n 'early_stopping_rounds': 300,\n 'verbose': -1,\n 'num_threads': -1,\n 'seed': SEED,\n}\nBlend -> Power (2) rank blend of Dart lgbm (0.801 public) / GBDT lgbm (0.799 public) / Catboost models (0.799 public)
\nSingle lgbm with 3 last statements showed even better CV by we didn't have enough time to retrain it (full DART run for 5 folds took 12+ hours there).
\nIt was obvious that clients with a little number of statements will not get benefit from all 2k features. So we created a special model that was trained only on 300 features with custom params (also dart). Predictions for clients with <=2 statements came exclusively from such model and were not blended with other models.
\nHow did we combine the result from 2 independent models to not destroy the final ranking?
\n| Client id | \nNumber of statements | \nBasic ranking | \n<=2 prediction | \nFinal ranking | \n
|---|---|---|---|---|
| 1 | \n13 | \n5 | \n… | \n5 | \n
| 1 | \n2 | \n4 | \n0.1 | \n3 | \n
| 1 | \n13 | \n8 | \n… | \n8 | \n
| 1 | \n1 | \n3 | \n0.5 | \n4 | \n
| 1 | \n13 | \n7 | \n… | \n7 | \n
| 1 | \n13 | \n2 | \n… | \n2 | \n
we kept ranking for >2 statements and for rest resorted within initial ranking group (hope it's clear enough))). Was is perfect - no, but it was very stable with really tiny improvement (because of number of such clients in public and private test parts).
\nWhat also worked well:
\nWhat didn't work:
\nAgain, nothing really fancy here. The main thing that helped us align Train / CV with LB was very high 'min_data_in_leaf'.
\nNo optuna used -> just manual old school tuning based on data feeling.
\nWe did many experiments with weights and loss functions but none of them worked.
Due to AMEX metric specification it was obvious that focal loss should work but it didn't. We tried several times to switch loss function during the competition period and the result was the same.
\nError analysis showed that model makes errors without any \"pattern\" -> stacking didn't work for holdout set (25% of data) and we had doubts that it will work on private/public test parts. We kept only LR/Lasso(0.02) for blending options to choose submissions.
\nCross validation -> standard 5folds CV split by client ID. The unique thing that we did here is \"prespliting\" to align all CV between team members to be able to compare results directly.
\nWhat left without mentions:
\nWhat we always wanted but didn't found time to do:
\n########################### Data preprocessing and Data evaluation\n#################################################################################\n\n## Added noise removal -> GOOD2DO\n# There is no doubt that some Noise was injected in data\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327649\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327651\n# We need to find a way to remove it \n# the best option to not follow public approach\n# At least with columns where columns have overlaped population\n\n## Data minification for FE -> GOOD2DO\n# Datatype downcasting\n# Pickle/Parquet/Feather \n# Be careful with floats16 as it may lead to bad agg results\n# Also float16 may lead to some signal degradation due to precision and values changes\n\n## Evaluate values distributions and NaNs -> GOOD2DO\n# Full 13 months history\n# Train against Test Public and Test Private\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327926\n#\n# Need to try:\n# Kolmogorov–Smirnov test -> GOOD2DO\n#\n# Adversarial validation -> GOOD2DO\n# https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation/notebook (just as simple example)\n#\n# Entropy / Distances / etc....\n#\n# Visual checks)))\n#\n# We need to find if ANY feature has very different distribution in PRIVATE test set\n# If that feature works for Public part it doesn't mean that it will work for Private\n\n########################### Targets\n#################################################################################\n\n# We need to find a way to get more targets -> GOOD2DO\n# as we currently training on a single point by client we could greatly improve results\n# by extending our training set with new targets\n#\n# Find default periods in current client history and make appropriate labeling -> GOOD2DO\n# Make 2 level model -> predict p_2 values as normal time-series model and feed it to 2nd level GBT\n\n########################### Separate Models\n#################################################################################\n\n# Probably it's a good idea to make separate models for each subdatasets\n# Full history (13 months)\n# Less than 13 months\n\n########################### External Data\n#################################################################################\n# We can try to add \"Consumer index\" or any other independent temporal feature\n# Will not work if we will not be able to expand targets and add temporal feature\n\n########################### FE\n#################################################################################\n\n# We didn't make anything special here -> July\n# AGGS (Stats by client)\n# Rollings\n# History length feature (not sure if it will help with Private Test)\n# Should we correct statements dates and add NaNs?\n# ReRanking categorical features by P_2 or Target\n# Clusterization (4+ groups feature by feature)\n# Count and Mean encodings for categorical features\n# Features combinations (sum/prod/power) -> bruteforce\n# PCA or any other dimension reduction by features groups\n\n# We need to find if there is \"connection\" between clients in Train -> Public Test -> Private Test\n# we have 458913 + 924621 -> 1383534 If I were AMEX I would export 1M clients (or other round number)\n# so may be 384 534 Clients are overlaps\n\n# Clip by 5 - 95 percentile\n\n########################### Features Selection\n#################################################################################\n# Permutation importance (use all fold only!!!) -> recursive elimination (because of quantity of features -> 3-4 rounds with 0 and 50% negative drop) \n# SHAP\n# Highly correlated features (.98+?)\n# Forward selection (may take ages and due aggs may be not effective - probably by feature block) \n# Backward elimination (may take ages and due aggs may be not effective - probably by feature block) \n\n########################### CV\n#################################################################################\n# Mean Target differs my \"history length\" -> could be wise to do GroupedStratifeidFolds by history length\n# For sure Splits should be done by client\n# Target stratification to balance folds\n\n########################### Loss function / Metric\n#################################################################################\n# Clean and fast np/torch metric\n# Now it's in helper (need to cleanup that)\n# https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations\n#\n# I don't believe that we will have better results with different loss function\n# But it worth to try at least focal loss\n# https://maxhalford.github.io/blog/lightgbm-focal-loss/\n#\n# Weights -> we should try change weights there\n# weights by class\n# weights by some history length\n# weights by internal fe group\n#\n# We need custom metric for catboost\n# example https://catboost.ai/en/docs/concepts/python-usages-examples#logloss1\n\n########################### Models\n#################################################################################\n\n## First level choice\n# LGB/XGB/CTB -> our main models here for sure\n# After stabilizing the baseline model and base feature we need to make 1st round tuning\n\n## Catboos specials\n# Categorical features\n# Embeding features\n\n## NN (GPU/TPU) -> RNN / LSTM / Transformer\n# TPU -> tensorflow (as it works better there)\n\n## NN -> AE / VAE / DAE -> as a denoising model hidden layer as input for GBT models\n# No need complex approach - just fast check the idea and in case of success move to big model\n\n########################### Blending\n#################################################################################\n# Weighted Average\n# Power Average\n# Weighted Rank Average\n# Linear/SVM\n# Postprocessing?\nNote: Full code to retrain single model will be shared here in a 2 weeks.
\nI would like to say thank you to competition hosts and Kaggle - it was a great pleasure to participate in tabular data competition after many months and years without one.
\nThank you to all participants (and of course winners - @daishu huge jump during last 3 days and fantastic solo result) - our success is your success - you forced us to try harder - without all of you It would be impossible to learn so many new things and achieve such result.
\nAnd special \"thank you\" goes to my fantastic teammates:
\nNo words would be enough to say how much each of you contributed to the end result.
\n\"Do Data Scientists have hobby? Yes -> DS competition\".
We were very hyped by the new tabular data competition release (sorry for the external link: link) and immediately decided to participate. Slack notification -> rules and perspective advertisement (100% chance too lose summer holidays and all free time) -> and here we are - four members. Only one member of the team had previous experience in DS competitions participation.
\nFew rules were established from the beginning:
\nEach of us had own machines / resources (GCP/AWS/local). We used Kaggle platform just for a few times. So the first thing we wanted to solve - unified machine to save all artefacts / experiments. We decided to go with AWS. I would say that it is possible to achieve the same result that we have just with Kaggle resources, but it would be bit more stressful for team management. We didn't want to spend a lot of money on AWS but sometimes (during very hot hours) RAM spikes were 500GB+ to permit simultaneous work.
\nWe tried to use neptune.ai for ML tracking but from July it was not very effective as we entered in brute force zone.
\nAdvise: Resources management is very critical - find bottleneck and remove it to make your team most effective. At the same time don't burn money recklessly - limit your budged. If any optimization possible - do it as soon as possible to save time and resources.
\nEach run was internally versioned (ex. v1.1.1 - (major version).(fe version).(model version))
\nOverall project structure:
All these permitted us to go back and forward and check what worked well and what did not and restore experiments in each particular step.
\nWe wanted to achieve several things with this step:
\nWe didn't use public notebooks for cleaning. Radar's Dataset is fantastic and it is 99% similar to our own transformations.
\nWe used \"isle\" identification without any pre-build coefficients.
\ndummy code is something like this:
for col in process_columns: \n\n df = temp_df[[col]].sort_values(by=[col]) \n df = df[df[col].notna()].drop_duplicates(subset=[col]).reset_index(drop=True)\n\n df['temp'] = np.floor(df[col] * 100000)\n df['group'] = ((df['temp'] - df['temp'].shift()).abs() >= 100).cumsum()\n\n i = 0\n while True:\n min_val = df[df['group']==i]['temp'].min()\n if min_val>0:\n break\n i += 1\n\n df['temp2'] = np.where(df['temp']>=0, \n np.floor(df['temp']/min_val).astype(np.int32),\n np.round(df['temp']/min_val).astype(np.int32))\n\n mapping = dict(zip(df[col],df['temp2']))\n temp_df[col] = temp_df[col].map(mapping)\n\n print(col, df['group'].nunique(), df[col].nunique())\n print(df.groupby(['group'])['temp','temp2'].agg(['min','max','count','nunique']).head(40))\nWe used last statement month to create 0/1/2 feature and store in in \"index\" df
\nFixed random seed (of course 42) to make spits and then took 20% of customers to holdout group (to test stacking / blending / etc)
\nAfter cleaning we had several columns \"groups\".
\nall_files = [\n'p_columns', -> just p columns as we thought that they are very different (and P_2 is internal amex \"scoring\" model)\n'objects_radar_columns', -> order encoding (we were checking where out cleaning differs from public approaches and here was the unique place) \n'objects_columns', -> onehot encoding\n'categorical_cleaned__D__columns', -> no noise categoricals\n'categorical_binary__S__columns', -> cleaned binary\n'categorical_binary__R__columns', -> cleaned binary\n'categorical_binary__D__columns', -> cleaned binary\n'categorical_binary__B__columns', -> cleaned binary\n'categorical__D__columns', -> removed noise categoricals\n'categorical__B__columns', -> removed noise categoricals\n'cleaned__B__columns', -> removed noise continuous \n'cleaned__D__columns', -> removed noise continuous \n'cleaned__R__columns', -> removed noise continuous \n'cleaned__S__columns', -> removed noise continuous \n'rest__B__columns', -> have no idea what to do with it -> floor \n'rest__D__columns', -> have no idea what to do with it -> floor \n'rest__R__columns', -> have no idea what to do with it -> floor \n'rest__S__columns', -> have no idea what to do with it -> floor \n]\nThanks again to @raddar we always used your preprocess as a baseline.
\nWe were able to load just portion of data -> do fe -> concat to \"index\" as all dfs were aligned by index. Also such split permitted us to do fe by feature type to accelerate process and see more statistically valuable metric change.
\nWe started with careful fe column by column or small subset and it worked well until 1xx features and then any metric improvement or degradation was not statistically significant and many features \"overlapped\" on importance and significance.
\nNote: I believe that it is possible to build silver zone robust model with only 3xx features
\nSo we started from scratch with brute force))) Of course there was no need to apply \"nunique\" (for example to binary features) and our previous step helped us to limit fe.
\nall_aggregations = {\n 'agg_func': ['last','mean','std','median','min','max','nunique'],\n 'diff_func': ['first','mean','std','median','min','max'],\n 'ratio_func': ['first','mean','std','median','min','max'],\n 'lags': [1,2,3,6,11],\n 'special': ['ewm','count_month_enc','monotonic_increase','diff_mean','major_class',\n 'normalization','top_outlier','bottom_outlier','normalization_mean','top_outlier_mean','top_outlier_mean']\n} \nagg_func -> normal aggregations by uid
\ndiff_func -> diff last - xxx -> std diff worked better than any other
\nratio_func -> ratio_func last/xxx
\nlags -> diff last - Nx
\nspecial -> some special transformations -> count_month_enc worked well for categorical / emw for continous
We ended up with about 7k features (stored file by group and by agg type for faster loading).
\nNext thing was to figure out what works and what not -> this topic was the most challenging for us.
It's better to call it Standardization (x - m) / s -> as we had also normalization test the name became constant \"normalization\")))
\ndf.groupby(['dt_month','subset'])[col].agg(['mean','std'])\ndt_month -> month of the statement
\nsubset -> train / public / private
\nand mean and std from clients that had full statement history.
We have to have temporal shift to make it work. So we did a \"trick\" removed last statement for each client and applied exactly same transformation for each client and merged appropriate labels. So we had 2 lines in training set for almost each client BUT validated results only on last statement during CV runs and Holdout checks. It more or less same as adding noised data but we had temporal drift and model was able to work better on unknown future data with \"possible\" data drift.
\nOoohh that was really fun.
\nWe used gbdt boosting type during experiments as it was very aligned with dart mode but was significantly faster.
\nAlso, we used ROC AUC score during our experiments as we believed that due to amex instability we can't use it for decision making (of course we tracked log loss and amex).
In previous step we brute forced many features and now is time to clean them out.
\nAll feature selection was done with 5 CV folds training + independent check on 20% holdout data.
Zero importance -> Right from the start we were able to through away 1.5k features that had exactly 0 importance (lgbm importance). That means that with 250 bins and 2**10 data in leaf those features are not participating in any split.
Stepped hierarchical permutation importance -> we defined 300 initial features and looped over all other features subsets (600+) - was very time consuming but very stable.
\nNote: we shuffled order of the subset to force model try different combinations.
\nAdd features subset -> train model -> permutate -> drop negative features (negative mean over 5 seeds) -> add new subset -> …
\nDuring this part that took almost 3 days we limited features to 3k -> 0.800 lb
Stepped permutation importance.
\nTake all features -> train model -> permutate -> drop 20% of worst performed features (only negative) -> repeat. Final subset was 25xx features (and different from previous step) -> 0.800 lb
Forward feature selection.
\nWe defined 300 initial features and simply added subset by subset and compared ROC AUC if metric change was > 0.0003 we kept the subset. -> 0.800 lb
Time series CV.
\nFor very doubtful features as PCA and Normilized values we used to different validation stratagies:
Many other options we tried but result was not stable.
\nFinal subset came from \"Forward feature selection\" plus overlapped features from other technics minus overlapped negative combination. -> lb 0.801 single model.
\nWe tried to blend many models with different subset as we believed that it should give huge LB boost (based on holdout blending tests) but it didn't work well for lb.
\nIn my own experience, DART never worked better and here we have proof that in DS \"all depends.\" We did experiments with DART in the beginning and it did not show any metric improvement with our params and baseline model features subset. Later we found @ragnar123 notebook and gave it one more try and it worked marvelously.
\nFrom the beginning, we tried to build a more complex model with 2**7+ leaves and 0.7+ features but failed. It still puzzles me why a simple model with a very low number of features works here.
\nI saw such behaviour mostly on synthetic data and stacking - so we tried to find out if data is syntetic (at least partly) and deanonimize internal scoring values - but didn't make it.
\nOur best single lgbm model was trained on 29xx features. 5 folds CV - no stratification by any option. Training data - 2 last staements for each client (transformed independently). Params:
\nlgb_params = {\n 'boosting_type': 'dart',\n 'objective': 'cross_entropy', \n 'metric': ['AUC'],\n 'subsample': 0.8, \n 'subsample_freq': 1,\n 'learning_rate': 0.01, \n 'num_leaves': 2 ** 6, \n 'min_data_in_leaf': 2 ** 11, \n 'feature_fraction': 0.2, \n 'feature_fraction_bynode':0.3,\n 'first_metric_only': True,\n 'n_estimators': 17001, # -> 5000 for gbdt \n 'boost_from_average': False,\n 'early_stopping_rounds': 300,\n 'verbose': -1,\n 'num_threads': -1,\n 'seed': SEED,\n}\nBlend -> Power (2) rank blend of Dart lgbm (0.801 public) / GBDT lgbm (0.799 public) / Catboost models (0.799 public)
\nSingle lgbm with 3 last statements showed even better CV by we didn't have enough time to retrain it (full DART run for 5 folds took 12+ hours there).
\nIt was obvious that clients with a little number of statements will not get benefit from all 2k features. So we created a special model that was trained only on 300 features with custom params (also dart). Predictions for clients with <=2 statements came exclusively from such model and were not blended with other models.
\nHow did we combine the result from 2 independent models to not destroy the final ranking?
\n| Client id | \nNumber of statements | \nBasic ranking | \n<=2 prediction | \nFinal ranking | \n
|---|---|---|---|---|
| 1 | \n13 | \n5 | \n… | \n5 | \n
| 1 | \n2 | \n4 | \n0.1 | \n3 | \n
| 1 | \n13 | \n8 | \n… | \n8 | \n
| 1 | \n1 | \n3 | \n0.5 | \n4 | \n
| 1 | \n13 | \n7 | \n… | \n7 | \n
| 1 | \n13 | \n2 | \n… | \n2 | \n
we kept ranking for >2 statements and for rest resorted within initial ranking group (hope it's clear enough))). Was is perfect - no, but it was very stable with really tiny improvement (because of number of such clients in public and private test parts).
\nWhat also worked well:
\nWhat didn't work:
\nAgain, nothing really fancy here. The main thing that helped us align Train / CV with LB was very high 'min_data_in_leaf'.
\nNo optuna used -> just manual old school tuning based on data feeling.
\nWe did many experiments with weights and loss functions but none of them worked.
Due to AMEX metric specification it was obvious that focal loss should work but it didn't. We tried several times to switch loss function during the competition period and the result was the same.
\nError analysis showed that model makes errors without any \"pattern\" -> stacking didn't work for holdout set (25% of data) and we had doubts that it will work on private/public test parts. We kept only LR/Lasso(0.02) for blending options to choose submissions.
\nCross validation -> standard 5folds CV split by client ID. The unique thing that we did here is \"prespliting\" to align all CV between team members to be able to compare results directly.
\nWhat left without mentions:
\nWhat we always wanted but didn't found time to do:
\n########################### Data preprocessing and Data evaluation\n#################################################################################\n\n## Added noise removal -> GOOD2DO\n# There is no doubt that some Noise was injected in data\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327649\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327651\n# We need to find a way to remove it \n# the best option to not follow public approach\n# At least with columns where columns have overlaped population\n\n## Data minification for FE -> GOOD2DO\n# Datatype downcasting\n# Pickle/Parquet/Feather \n# Be careful with floats16 as it may lead to bad agg results\n# Also float16 may lead to some signal degradation due to precision and values changes\n\n## Evaluate values distributions and NaNs -> GOOD2DO\n# Full 13 months history\n# Train against Test Public and Test Private\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327926\n#\n# Need to try:\n# Kolmogorov–Smirnov test -> GOOD2DO\n#\n# Adversarial validation -> GOOD2DO\n# https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation/notebook (just as simple example)\n#\n# Entropy / Distances / etc....\n#\n# Visual checks)))\n#\n# We need to find if ANY feature has very different distribution in PRIVATE test set\n# If that feature works for Public part it doesn't mean that it will work for Private\n\n########################### Targets\n#################################################################################\n\n# We need to find a way to get more targets -> GOOD2DO\n# as we currently training on a single point by client we could greatly improve results\n# by extending our training set with new targets\n#\n# Find default periods in current client history and make appropriate labeling -> GOOD2DO\n# Make 2 level model -> predict p_2 values as normal time-series model and feed it to 2nd level GBT\n\n########################### Separate Models\n#################################################################################\n\n# Probably it's a good idea to make separate models for each subdatasets\n# Full history (13 months)\n# Less than 13 months\n\n########################### External Data\n#################################################################################\n# We can try to add \"Consumer index\" or any other independent temporal feature\n# Will not work if we will not be able to expand targets and add temporal feature\n\n########################### FE\n#################################################################################\n\n# We didn't make anything special here -> July\n# AGGS (Stats by client)\n# Rollings\n# History length feature (not sure if it will help with Private Test)\n# Should we correct statements dates and add NaNs?\n# ReRanking categorical features by P_2 or Target\n# Clusterization (4+ groups feature by feature)\n# Count and Mean encodings for categorical features\n# Features combinations (sum/prod/power) -> bruteforce\n# PCA or any other dimension reduction by features groups\n\n# We need to find if there is \"connection\" between clients in Train -> Public Test -> Private Test\n# we have 458913 + 924621 -> 1383534 If I were AMEX I would export 1M clients (or other round number)\n# so may be 384 534 Clients are overlaps\n\n# Clip by 5 - 95 percentile\n\n########################### Features Selection\n#################################################################################\n# Permutation importance (use all fold only!!!) -> recursive elimination (because of quantity of features -> 3-4 rounds with 0 and 50% negative drop) \n# SHAP\n# Highly correlated features (.98+?)\n# Forward selection (may take ages and due aggs may be not effective - probably by feature block) \n# Backward elimination (may take ages and due aggs may be not effective - probably by feature block) \n\n########################### CV\n#################################################################################\n# Mean Target differs my \"history length\" -> could be wise to do GroupedStratifeidFolds by history length\n# For sure Splits should be done by client\n# Target stratification to balance folds\n\n########################### Loss function / Metric\n#################################################################################\n# Clean and fast np/torch metric\n# Now it's in helper (need to cleanup that)\n# https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations\n#\n# I don't believe that we will have better results with different loss function\n# But it worth to try at least focal loss\n# https://maxhalford.github.io/blog/lightgbm-focal-loss/\n#\n# Weights -> we should try change weights there\n# weights by class\n# weights by some history length\n# weights by internal fe group\n#\n# We need custom metric for catboost\n# example https://catboost.ai/en/docs/concepts/python-usages-examples#logloss1\n\n########################### Models\n#################################################################################\n\n## First level choice\n# LGB/XGB/CTB -> our main models here for sure\n# After stabilizing the baseline model and base feature we need to make 1st round tuning\n\n## Catboos specials\n# Categorical features\n# Embeding features\n\n## NN (GPU/TPU) -> RNN / LSTM / Transformer\n# TPU -> tensorflow (as it works better there)\n\n## NN -> AE / VAE / DAE -> as a denoising model hidden layer as input for GBT models\n# No need complex approach - just fast check the idea and in case of success move to big model\n\n########################### Blending\n#################################################################################\n# Weighted Average\n# Power Average\n# Weighted Rank Average\n# Linear/SVM\n# Postprocessing?\n@kyakovlev Congratulations Konstantin and team. I am so happy for you. You are the best tabular data feature engineer that I know. I'm glad to see your skills achieved great success! I'm looking forward to learning about your solution!
", "rawMarkdown": "@kyakovlev Congratulations Konstantin and team. I am so happy for you. You are the best tabular data feature engineer that I know. I'm glad to see your skills achieved great success! I'm looking forward to learning about your solution!", "votes": 13, "replies": [ { "id": 1912768, "postDate": "2022-08-25T00:52:04.023Z", "content": "I think you groom him well from IEEE competition 3 years back :) Congratulations to you both !
", "rawMarkdown": "I think you groom him well from IEEE competition 3 years back :) Congratulations to you both ! ", "votes": 1 }, { "id": 1912775, "postDate": "2022-08-25T00:57:00.003Z", "content": "haha, he groomed me well. I was new to Kaggle then and he invited me to his team when I was like 30th place and he was 1st place! He taught me so much about tabular data and feature engineering. Many groupby aggregate tricks that I use today I learned from Konstantin. Thanks Konstantin!
", "rawMarkdown": "haha, he groomed me well. I was new to Kaggle then and he invited me to his team when I was like 30th place and he was 1st place! He taught me so much about tabular data and feature engineering. Many groupby aggregate tricks that I use today I learned from Konstantin. Thanks Konstantin!", "votes": 11 }, { "id": 1912845, "postDate": "2022-08-25T02:04:13.323Z", "content": "Thank you @cdeotte it's a huge honor to hear such words from you.
\nRight from the start we decided to make \"internal\" team for this competition but I was so many times close to break that rule and ask you to join us - I was sure that you had fantastic and a very unique NN approach (and you had it) - congrats with solo gold with transformers.
", "rawMarkdown": "Thank you @cdeotte it's a huge honor to hear such words from you.\n\nRight from the start we decided to make \"internal\" team for this competition but I was so many times close to break that rule and ask you to join us - I was sure that you had fantastic and a very unique NN approach (and you had it) - congrats with solo gold with transformers. ", "votes": 6 }, { "id": 1912964, "postDate": "2022-08-25T04:28:29.393Z", "content": "Well @cdeotte I think a small notebook about groupby tricks would be your best gift for us after this competition :)
", "rawMarkdown": "Well @cdeotte I think a small notebook about groupby tricks would be your best gift for us after this competition :)", "votes": 3 }, { "id": 1913157, "postDate": "2022-08-25T07:27:17.260Z", "content": "@mohammad2012191 he already made an elaborate post about feature engineering:
\nhttps://www.kaggle.com/competitions/ieee-fraud-detection/discussion/108575
\nIt's probably the highest voted discussion post of all times!
\nGroupby is under \"aggregation\"
Thanks, gonna check it.
", "rawMarkdown": "Thanks, gonna check it.", "votes": 1 }, { "id": 1913550, "postDate": "2022-08-25T11:06:44.673Z", "content": "Yes, the link Yukiya provides has a list of good engineer techniques.
\nAnd here is a link which explains why groupby works here. When we create new columns using groupby from other columns, we give our GBT new ways to classify rows that belong to different groups.
", "rawMarkdown": "Yes, the link Yukiya provides has a list of good engineer techniques. \n\nAnd here is a link which explains why groupby works [here][1]. When we create new columns using groupby from other columns, we give our GBT new ways to classify rows that belong to different groups.\n\n[1]: https://www.kaggle.com/competitions/ieee-fraud-detection/discussion/111453", "votes": 2 }, { "id": 1914175, "postDate": "2022-08-25T19:53:43.670Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1920789, "postDate": "2022-08-31T12:02:27.490Z", "content": "A huge congratulations and thanks for such a detail explanation. As Chris said, the best tabular feature engineer!
\nSome questions from my side:
\n1. Based on your experience, doing brute force feature engineering (i.e. create a ton of potential new features and then perform feature selection) gives better results than going column by column?
\n2. How did you choose the initial LGBM hyperparameters before starting the feature selection phase?
\n3. Did you retune the hyperparameters at some point of the feature selection phase (e.g. when you added/removed a significant number of features) or kept them fixed all the time?
\n4. Could you give some examples of what type of feature subsets you include at each step of the feature selection phase (e.g. one subset could be the mean of cleaned__B__columns)?
\n5. Your final selected features came from \"Forward feature selection\". I tried it a couple of times at work and saw that tends to overfit, do you have the same experience?
\n6. How did you realize that increasing the value of min_data_in_leaf favour CV/LB alignment?
Also waiting for the code 👀
", "rawMarkdown": "A huge congratulations and thanks for such a detail explanation. As Chris said, the best tabular feature engineer! \n\nSome questions from my side:\n\n**1.** Based on your experience, doing brute force feature engineering (i.e. create a ton of potential new features and then perform feature selection) gives better results than going column by column?\n**2.** How did you choose the initial LGBM hyperparameters before starting the feature selection phase? \n**3.** Did you retune the hyperparameters at some point of the feature selection phase (e.g. when you added/removed a significant number of features) or kept them fixed all the time?\n**4.** Could you give some examples of what type of feature subsets you include at each step of the feature selection phase (e.g. one subset could be the mean of `cleaned__B__columns`)?\n**5.** Your final selected features came from \"Forward feature selection\". I tried it a couple of times at work and saw that tends to overfit, do you have the same experience?\n**6.** How did you realize that increasing the value of `min_data_in_leaf` favour CV/LB alignment?\n\nAlso waiting for the code 👀", "votes": 6, "replies": [ { "id": 1920865, "postDate": "2022-08-31T13:01:08.950Z", "content": "Thank you for your kind words and very nice questions.
\n1 | Brute force is a dead end by all meanings - it will not work on production, it will not provide any interpretability or explainability, it is not reproducible result. Column by column and distilled carefully created features are always better and preferable. Even in this competition, manual feature engineering would work better if we could have access to raw data and columns description (meaning).
\n2 | Initial params always depend on data, but I start most of times with 0.7/0.7 sampling/features sampling, 2^7 leaves and 2^8 min data in leaf (all other params as default).
\nFirst baseline run normally gives you a lot of information how to tune it further:
3 | Params tuning normally takes several iterations during model creation - for competitions it means a slight tuning every 2 weeks (after significant feature sets changes or data structure changes) and final deep tuning round 2 weeks before competition ends.
\n4 | Not much to add here - you are right with your assumption:
\nmean + cleaned__B__columns produces:
\nagg_func__mean__cleaned__B__columns|__B_4__mean
\nagg_func__mean__cleaned__B__columns|__B_16__mean
\nagg_func__mean__cleaned__B__columns|__B_20__mean
\nagg_func__mean__cleaned__B__columns|__B_22__mean
\nagg_func__mean__cleaned__B__columns|__B_41__mean
\n…
5 | It works really nice for the start but with more features it becomes less informative and slower. It didn't show any overfitting but we didn't trust it fully as we were training and validating on the same time period (and didn't have any aligned validation set in the “future”).
\n6 | We saw that ROC AUC score on train data went up to 0.9999 and CV/Holdout had cap of 0.96xxx -> many teams decided to use regularization (l1 - punish features with low importance but we would like to keep even small signals, l2 - adds regularization to features with high importance -> that is good move here but we didn't see any statistically significant CV score boost). We decided to make less complex model with 2^6 leaves and increase min data in leaf to force our model generalize better and found that increasing this param to 2^9 - 2^11 works very well on CV/holdout and also worked well on LB.
", "rawMarkdown": "Thank you for your kind words and very nice questions.\n\n\n1 | Brute force is a dead end by all meanings - it will not work on production, it will not provide any interpretability or explainability, it is not reproducible result. Column by column and distilled carefully created features are always better and preferable. Even in this competition, manual feature engineering would work better if we could have access to raw data and columns description (meaning).\n\n---\n\n2 | Initial params always depend on data, but I start most of times with 0.7/0.7 sampling/features sampling, 2^7 leaves and 2^8 min data in leaf (all other params as default). \nFirst baseline run normally gives you a lot of information how to tune it further:\n- if train score is much higher than CV and holdout -> try regularizations / lower subsampling / less complex model\n- if naive score is better than lgbm or difference is minimal -> bad features or/and need more complex model (2**7 is complex enough for majority of tasks)\n- if train score is low but very aligned with CV/holdout -> underfitting -> more complex model with less data in leaf\nThen try to \"tweak\" loss function that is more appropriate for your task.\nThen try to fit goss/dart/extra_trees -> extra_trees works very nice sometimes -> DART never worked better before)))\n\n---\n\n3 | Params tuning normally takes several iterations during model creation - for competitions it means a slight tuning every 2 weeks (after significant feature sets changes or data structure changes) and final deep tuning round 2 weeks before competition ends.\n\n---\n\n4 | Not much to add here - you are right with your assumption:\nmean + cleaned__B__columns produces:\nagg_func__mean__cleaned__B__columns|__B_4__mean\nagg_func__mean__cleaned__B__columns|__B_16__mean\nagg_func__mean__cleaned__B__columns|__B_20__mean\nagg_func__mean__cleaned__B__columns|__B_22__mean\nagg_func__mean__cleaned__B__columns|__B_41__mean\n...\n\n---\n\n5 | It works really nice for the start but with more features it becomes less informative and slower. It didn't show any overfitting but we didn't trust it fully as we were training and validating on the same time period (and didn't have any aligned validation set in the “future”).\n\n---\n\n6 | We saw that ROC AUC score on train data went up to 0.9999 and CV/Holdout had cap of 0.96xxx -> many teams decided to use regularization (l1 - punish features with low importance but we would like to keep even small signals, l2 - adds regularization to features with high importance -> that is good move here but we didn't see any statistically significant CV score boost). We decided to make less complex model with 2^6 leaves and increase min data in leaf to force our model generalize better and found that increasing this param to 2^9 - 2^11 works very well on CV/holdout and also worked well on LB. \n \n ", "votes": 8 }, { "id": 1923496, "postDate": "2022-09-02T09:16:25.547Z", "content": "Thanks for such a complete answer!
", "rawMarkdown": "Thanks for such a complete answer!", "votes": 1 } ] }, { "id": 1913704, "postDate": "2022-08-25T13:06:04.490Z", "content": "It's becoming a long writeup))) will need bit more time
", "rawMarkdown": "It's becoming a long writeup))) will need bit more time", "votes": 5, "replies": [ { "id": 1914039, "postDate": "2022-08-25T17:15:31.977Z", "content": "Take your time! Great details already
", "rawMarkdown": "Take your time! Great details already", "votes": 1 } ] }, { "id": 1915111, "postDate": "2022-08-26T17:11:51.047Z", "content": "Huge congrats Konstantin @kyakovlev, this is incredible commitment your team had! I subscribed 2 months of Colab Pro+ for this comp and that was it…
\nCurious if you know besides diff_mean which of your 'special' agg ideas worked best for you?
btw diff_mean is:
\ndf[f'{col}__diff_mean] = df[col] - df.groupby(['uid'])[col].shift()\ndf[f'{col}__diff_mean] = df.groupby(['uid'])[f'{col}__diff_mean]].transform('mean')\n\ndf = df.drop_duplicates(subset=['uid'], keep='last').reset_index(drop=True)\nstd features worked better than others - diff with std for example
\nnormal diff with client mean was in
\n'diff_func': ['first','mean','std','median','min','max']
and my bad - which of specials.
\nmonotonic functions worked really well
Thanks for the reply!
\nmonotonic functions - Did you create a T/F binary feature or did you calculate for example spearman correlation or similar to quantify monotonicity?
Binary if current more or equal to previous and then took mean.
\nWe wanted to pass to the model information if last statements increase or decrease is normal for the client or not. If due payments are constantly increasing for us it was a sign that risk is growing.
\nAnd such information is not based on absolute values - we constantly wanted to avoid absolute values to be able to generalize well on future data and data with drifts.
", "rawMarkdown": "Binary if current more or equal to previous and then took mean.\n\nWe wanted to pass to the model information if last statements increase or decrease is normal for the client or not. If due payments are constantly increasing for us it was a sign that risk is growing.\n\nAnd such information is not based on absolute values - we constantly wanted to avoid absolute values to be able to generalize well on future data and data with drifts.", "votes": 1 } ] }, { "id": 1912725, "postDate": "2022-08-25T00:21:14.897Z", "content": "Congrats guys .. cant wait for your solution to reach 0.809 ..
", "rawMarkdown": "Congrats guys .. cant wait for your solution to reach 0.809 .. ", "votes": 3 }, { "id": 1918704, "postDate": "2022-08-29T18:58:54.670Z", "content": "very insightful tank youu
", "rawMarkdown": "very insightful tank youu", "votes": 1 }, { "id": 1917682, "postDate": "2022-08-29T01:25:08.577Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!", "votes": 1 }, { "id": 1917052, "postDate": "2022-08-28T11:47:50.033Z", "content": "Here is example how we augmented training Data:
\n
We were removing last statement per client as it would never exists and then applied exactly the same transformations as we did for full history.
\nWe wanted to completely avoid absolute values and with data augmentation the CV and LB were almost on the same level as with absolute values - of course it would not work with clients with 1-2 statements but we had a separate model for them. I believe that that would give much more robust results but on kaggle even slight metric increase is important and we kept \"lasts\".
\nAlso, we were planning to go in -2 max depth and used only 1,2,3,6,11 (most common time-series lags) diff lags to have fe aligned.
\nCombining original data with -1 statement or with -2 statement gave significant CV boost on ROC AUC score (our main metric) and tiny boost on AMEX metric. Combination of 3 sets gave just a tiny boost on ROC AUC but was significantly more memory consuming and slower to train model.
\nAlso, CV splits per original client uid to exclude leaks and validation was performed only on original data to exclude metric degradation tracking for shorten history predictions.
", "rawMarkdown": "Here is example how we augmented training Data:\n\n\n\nWe were removing last statement per client as it would never exists and then applied exactly the same transformations as we did for full history.\n\n- Last values are different but within distributions \n- Agg values are different but within distributions\n- Model has temporal component \n\nWe wanted to completely avoid absolute values and with data augmentation the CV and LB were almost on the same level as with absolute values - of course it would not work with clients with 1-2 statements but we had a separate model for them. I believe that that would give much more robust results but on kaggle even slight metric increase is important and we kept \"lasts\".\n\nAlso, we were planning to go in -2 max depth and used only 1,2,3,6,11 (most common time-series lags) diff lags to have fe aligned. \n\nCombining original data with -1 statement or with -2 statement gave significant CV boost on ROC AUC score (our main metric) and tiny boost on AMEX metric. Combination of 3 sets gave just a tiny boost on ROC AUC but was significantly more memory consuming and slower to train model.\n\nAlso, CV splits per original client uid to exclude leaks and validation was performed only on original data to exclude metric degradation tracking for shorten history predictions.", "votes": 1 }, { "id": 1917023, "postDate": "2022-08-28T11:13:19.733Z", "content": "Few examples how noise cleaning works:
\nFor the most features nunique groups within 998 -> 1001 is a sign of noised cleaning possibility.
Simple feature cleaning - in this case group is our new value:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2405813%2Fea9cd72dbd9666335a1194331479fbc8%2F2022-08-28%20%2012.05.28.png?generation=1661684893433642&alt=media\")
More complex feature with negative \"categories\" we keep negative values (temp2 is a new value):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2405813%2F652007f94ba3b47bc300dfe32da85ce1%2F2022-08-28%20%2012.06.17.png?generation=1661684841167050&alt=media\")
Very \"noised\" feature (we find initial coefficient and then apply it to groups with low population) - temp2 is a new value:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2405813%2F2d811e27de9544d6eaeca47cf268b650%2F2022-08-28%20%2012.11.55.png?generation=1661685137975362&alt=media\")
Thanks for sharing your solution! Very insightful.
", "rawMarkdown": "Thanks for sharing your solution! Very insightful.", "votes": 1 }, { "id": 1913980, "postDate": "2022-08-25T16:05:05.120Z", "content": "Normalizations. Sorry, I'm having trouble understanding. For all EXCEPT last month you took mean and std for all customers fort that exact col, month and subset (train/pub/pri)? What about last? I'm actually super confused what the trick was, and why? Maybe try just restating/rephrasing it and maybe I'll figure it or at least have more precise questions…
", "rawMarkdown": "Normalizations. Sorry, I'm having trouble understanding. For all EXCEPT last month you took mean and std for all customers fort that exact col, month and subset (train/pub/pri)? What about last? I'm actually super confused what the trick was, and why? Maybe try just restating/rephrasing it and maybe I'll figure it or at least have more precise questions...", "votes": 1, "replies": [ { "id": 1920719, "postDate": "2022-08-31T11:00:00Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1913512, "postDate": "2022-08-25T10:49:07.533Z", "content": "Congratulation! Looking forward to dive into your solution!
", "rawMarkdown": "Congratulation! Looking forward to dive into your solution!", "votes": 1 }, { "id": 1913260, "postDate": "2022-08-25T09:03:27.457Z", "content": "Good Job! well deserved
", "rawMarkdown": "Good Job! well deserved", "votes": 1 }, { "id": 1913210, "postDate": "2022-08-25T08:18:53.740Z", "content": "Congrats to the team. One more waiting for the extended explanation
", "rawMarkdown": "Congrats to the team. One more waiting for the extended explanation", "votes": 1 }, { "id": 1913202, "postDate": "2022-08-25T08:11:16.213Z", "content": "Congratulations ! :)
", "rawMarkdown": "Congratulations ! :)", "votes": 1 }, { "id": 1912995, "postDate": "2022-08-25T04:56:41.570Z", "content": "Congratuation!
", "rawMarkdown": "Congratuation!", "votes": 1 }, { "id": 1912963, "postDate": "2022-08-25T04:28:18.733Z", "content": "Glad to see you back on Kaggle competitions with a bang. Congratulations, amazing finish.
", "rawMarkdown": "Glad to see you back on Kaggle competitions with a bang. Congratulations, amazing finish. ", "votes": 1 }, { "id": 1912935, "postDate": "2022-08-25T03:57:12.200Z", "content": "Congratulations! Thank you for sharing model params!! Very useful for next competitions!
", "rawMarkdown": "Congratulations! Thank you for sharing model params!! Very useful for next competitions!", "votes": 1 }, { "id": 1912867, "postDate": "2022-08-25T02:34:02.830Z", "content": "Congratulations! 🎉 Not a lot of experience on Kaggle so not sure if this is normal but was surprised to see that 3/4 of your team is novices and contributors. Gives me hope 😅
", "rawMarkdown": "Congratulations! 🎉 Not a lot of experience on Kaggle so not sure if this is normal but was surprised to see that 3/4 of your team is novices and contributors. Gives me hope 😅", "votes": 1, "replies": [ { "id": 1915182, "postDate": "2022-08-26T18:14:18.977Z", "content": "@shahilap96 our team was formed at the very early stage of the competition and exclusively from my company DS department - many other DS colleagues wanted to participate but were not able to made such 3month commitment and decided not to join. Kaggle experience is nice to have but it’s not an obligatory condition - the most important is a “fire in the eyes” and passion to learn and try something new.
\nYes, my teammates are novice on kaggle but not novice in analytics and DS.
\nOne thing that I can say - all my teammates work with the same passion with regular work tasks and trying really hard to achieve goals. I feel very proud of them.
", "rawMarkdown": "@shahilap96 our team was formed at the very early stage of the competition and exclusively from my company DS department - many other DS colleagues wanted to participate but were not able to made such 3month commitment and decided not to join. Kaggle experience is nice to have but it’s not an obligatory condition - the most important is a “fire in the eyes” and passion to learn and try something new.\n\nYes, my teammates are novice on kaggle but not novice in analytics and DS. \n\nOne thing that I can say - all my teammates work with the same passion with regular work tasks and trying really hard to achieve goals. I feel very proud of them.", "votes": 4 }, { "id": 1916523, "postDate": "2022-08-28T00:39:14.290Z", "content": "Aha, that explains the novice part 😅 People with passion and fire in their eyes are always great to work with. The energy and drive always pushes me to do better. Congrats once again.
", "rawMarkdown": "Aha, that explains the novice part 😅 People with passion and fire in their eyes are always great to work with. The energy and drive always pushes me to do better. Congrats once again." } ] }, { "id": 1912798, "postDate": "2022-08-25T01:14:05.613Z", "content": "Congrats!
\nI'm waiting for your notebook to knwo what had been updated from IEEE competition :)
Congratulations！
", "rawMarkdown": "Congratulations！", "votes": 1 }, { "id": 1912741, "postDate": "2022-08-25T00:29:36.860Z", "content": "Congratulations, guys, you deserved it! Really well done reaching this score
", "rawMarkdown": "Congratulations, guys, you deserved it! Really well done reaching this score", "votes": 1 }, { "id": 1912740, "postDate": "2022-08-25T00:28:09.160Z", "content": "Congratulations 🎉🎉💥
", "rawMarkdown": "Congratulations 🎉🎉💥", "votes": 1 }, { "id": 1912732, "postDate": "2022-08-25T00:23:39.650Z", "content": "Congratulations！
", "rawMarkdown": "Congratulations！", "votes": 1 }, { "id": 1913268, "postDate": "2022-08-25T09:05:23.537Z", "content": "Congratulations 🎉🎉 and Thanks for the solution
", "rawMarkdown": "Congratulations 🎉🎉 and Thanks for the solution", "votes": 2, "replies": [ { "id": 1913318, "postDate": "2022-08-25T09:21:55.303Z", "content": "Solution is still in process. Was posted just initial plan and some information about model. Sorry for the delay, but will need 3-4h more to finish it. Hope you'll find it interesting.
", "rawMarkdown": "Solution is still in process. Was posted just initial plan and some information about model. Sorry for the delay, but will need 3-4h more to finish it. Hope you'll find it interesting.", "votes": 3 } ] }, { "id": 1912814, "postDate": "2022-08-25T01:25:26.207Z", "content": "Congratulations！🎉🎉
", "rawMarkdown": "Congratulations！🎉🎉", "votes": 2 }, { "id": 1914802, "postDate": "2022-08-26T12:41:23.430Z", "content": "Here is a dataset with 2 submissions:
\nsingle_lgbm_model_v12.0.1.csv -> best single lgbm (mean over folds) -> 0.80891 private / 0.80101 public
\n2nd_place_submission_ranked.csv -> ranked 2nd place -> 0.80938 private / 0.80134 public
interesting that single lgbm could give 6th place on lb
\nfeel free to blend it and check results on lb
\nhttps://www.kaggle.com/datasets/kyakovlev/amex-submissions
Is there a way to see the code ?
", "rawMarkdown": "Is there a way to see the code ? " }, { "id": 2010286, "postDate": "2022-10-30T16:47:07.660Z", "content": "thanks for sharing such insights, as a newbie, i loved the part on how you collaborated on cloud, recorded all the trained models to go fwd and back and the way you divided the data to apply ensemble techniques. lot to learn.. thanks for sharing!
", "rawMarkdown": "thanks for sharing such insights, as a newbie, i loved the part on how you collaborated on cloud, recorded all the trained models to go fwd and back and the way you divided the data to apply ensemble techniques. lot to learn.. thanks for sharing!" }, { "id": 1946819, "postDate": "2022-09-20T05:51:59.280Z", "content": "Congratulations！
", "rawMarkdown": "Congratulations！" }, { "id": 1936425, "postDate": "2022-09-12T18:34:58.703Z", "content": "can you share code, as was done for 1st place?
", "rawMarkdown": "can you share code, as was done for 1st place?" }, { "id": 1936351, "postDate": "2022-09-12T17:18:45.667Z", "content": "Congratulations @kyakovlev and team-mates. Thank you for posting such detailed approach for us to learn from.
", "rawMarkdown": "Congratulations @kyakovlev and team-mates. Thank you for posting such detailed approach for us to learn from." }, { "id": 1925646, "postDate": "2022-09-04T07:04:35.363Z", "content": "A huge congratulations! Thanks for sharing the techniques and the best practice, really learn a lot from you! In particular, I am interested in how you organize different versions of your experiments, it is very important and useful for complex research like you mentioned.
\nCan you elaborate more on the details? much appreciated!
", "rawMarkdown": "A huge congratulations! Thanks for sharing the techniques and the best practice, really learn a lot from you! In particular, I am interested in how you organize different versions of your experiments, it is very important and useful for complex research like you mentioned. \n\nCan you elaborate more on the details? much appreciated!" }, { "id": 1921781, "postDate": "2022-09-01T04:07:45.547Z", "content": "Really great！
", "rawMarkdown": "Really great！" }, { "id": 1918844, "postDate": "2022-08-29T22:47:28.043Z", "content": "can you share code pls
", "rawMarkdown": "can you share code pls", "replies": [ { "id": 1918856, "postDate": "2022-08-29T23:02:36.033Z", "content": "Yes, we will share it for sure. Our plan is to make cleanup during the weekend (3-4 September), few days for review and share everything. Last weeks of the competition were very intense and we are taking small “break”. Sorry for the delay but this is the fastest possible timeline for us.
", "rawMarkdown": "Yes, we will share it for sure. Our plan is to make cleanup during the weekend (3-4 September), few days for review and share everything. Last weeks of the competition were very intense and we are taking small “break”. Sorry for the delay but this is the fastest possible timeline for us.", "votes": 7 } ] }, { "id": 1914257, "postDate": "2022-08-25T22:35:16.320Z", "content": "Congratulations！Thanks for sharing!
", "rawMarkdown": "Congratulations！Thanks for sharing!", "votes": 1 }, { "id": 1913065, "postDate": "2022-08-25T05:46:17.893Z", "content": "Thank You!
\nWaiting for more ))
very useful, thanks for sharing!
", "rawMarkdown": "very useful, thanks for sharing!" } ], "comments": [ { "id": 1912746, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-08-25T00:32:24.170000", "content": "@kyakovlev Congratulations Konstantin and team. I am so happy for you. You are the best tabular data feature engineer that I know. I'm glad to see your skills achieved great success! I'm looking forward to learning about your solution!
", "votes": 13, "replies": [ { "id": 1912768, "author_name": "yukiya", "author_url": "", "post_date": "2022-08-25T00:52:04.023000", "content": "I think you groom him well from IEEE competition 3 years back :) Congratulations to you both !
", "votes": 1, "replies": [] }, { "id": 1912775, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-08-25T00:57:00.003000", "content": "haha, he groomed me well. I was new to Kaggle then and he invited me to his team when I was like 30th place and he was 1st place! He taught me so much about tabular data and feature engineering. Many groupby aggregate tricks that I use today I learned from Konstantin. Thanks Konstantin!
", "votes": 11, "replies": [] }, { "id": 1912845, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-25T02:04:13.323000", "content": "Thank you @cdeotte it's a huge honor to hear such words from you.
\nRight from the start we decided to make \"internal\" team for this competition but I was so many times close to break that rule and ask you to join us - I was sure that you had fantastic and a very unique NN approach (and you had it) - congrats with solo gold with transformers.
", "votes": 6, "replies": [] }, { "id": 1912964, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2022-08-25T04:28:29.393000", "content": "Well @cdeotte I think a small notebook about groupby tricks would be your best gift for us after this competition :)
", "votes": 3, "replies": [] }, { "id": 1913157, "author_name": "yukiya", "author_url": "", "post_date": "2022-08-25T07:27:17.260000", "content": "@mohammad2012191 he already made an elaborate post about feature engineering:
\nhttps://www.kaggle.com/competitions/ieee-fraud-detection/discussion/108575
\nIt's probably the highest voted discussion post of all times!
\nGroupby is under \"aggregation\"
Thanks, gonna check it.
", "votes": 1, "replies": [] }, { "id": 1913550, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-08-25T11:06:44.673000", "content": "Yes, the link Yukiya provides has a list of good engineer techniques.
\nAnd here is a link which explains why groupby works here. When we create new columns using groupby from other columns, we give our GBT new ways to classify rows that belong to different groups.
", "votes": 2, "replies": [] }, { "id": 1914175, "author_name": "", "author_url": "", "post_date": "2022-08-25T19:53:43.670000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1920789, "author_name": "delai50", "author_url": "", "post_date": "2022-08-31T12:02:27.490000", "content": "A huge congratulations and thanks for such a detail explanation. As Chris said, the best tabular feature engineer!
\nSome questions from my side:
\n1. Based on your experience, doing brute force feature engineering (i.e. create a ton of potential new features and then perform feature selection) gives better results than going column by column?
\n2. How did you choose the initial LGBM hyperparameters before starting the feature selection phase?
\n3. Did you retune the hyperparameters at some point of the feature selection phase (e.g. when you added/removed a significant number of features) or kept them fixed all the time?
\n4. Could you give some examples of what type of feature subsets you include at each step of the feature selection phase (e.g. one subset could be the mean of cleaned__B__columns)?
\n5. Your final selected features came from \"Forward feature selection\". I tried it a couple of times at work and saw that tends to overfit, do you have the same experience?
\n6. How did you realize that increasing the value of min_data_in_leaf favour CV/LB alignment?
Also waiting for the code 👀
", "votes": 6, "replies": [ { "id": 1920865, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-31T13:01:08.950000", "content": "Thank you for your kind words and very nice questions.
\n1 | Brute force is a dead end by all meanings - it will not work on production, it will not provide any interpretability or explainability, it is not reproducible result. Column by column and distilled carefully created features are always better and preferable. Even in this competition, manual feature engineering would work better if we could have access to raw data and columns description (meaning).
\n2 | Initial params always depend on data, but I start most of times with 0.7/0.7 sampling/features sampling, 2^7 leaves and 2^8 min data in leaf (all other params as default).
\nFirst baseline run normally gives you a lot of information how to tune it further:
3 | Params tuning normally takes several iterations during model creation - for competitions it means a slight tuning every 2 weeks (after significant feature sets changes or data structure changes) and final deep tuning round 2 weeks before competition ends.
\n4 | Not much to add here - you are right with your assumption:
\nmean + cleaned__B__columns produces:
\nagg_func__mean__cleaned__B__columns|__B_4__mean
\nagg_func__mean__cleaned__B__columns|__B_16__mean
\nagg_func__mean__cleaned__B__columns|__B_20__mean
\nagg_func__mean__cleaned__B__columns|__B_22__mean
\nagg_func__mean__cleaned__B__columns|__B_41__mean
\n…
5 | It works really nice for the start but with more features it becomes less informative and slower. It didn't show any overfitting but we didn't trust it fully as we were training and validating on the same time period (and didn't have any aligned validation set in the “future”).
\n6 | We saw that ROC AUC score on train data went up to 0.9999 and CV/Holdout had cap of 0.96xxx -> many teams decided to use regularization (l1 - punish features with low importance but we would like to keep even small signals, l2 - adds regularization to features with high importance -> that is good move here but we didn't see any statistically significant CV score boost). We decided to make less complex model with 2^6 leaves and increase min data in leaf to force our model generalize better and found that increasing this param to 2^9 - 2^11 works very well on CV/holdout and also worked well on LB.
", "votes": 8, "replies": [] }, { "id": 1923496, "author_name": "delai50", "author_url": "", "post_date": "2022-09-02T09:16:25.547000", "content": "Thanks for such a complete answer!
", "votes": 1, "replies": [] } ] }, { "id": 1913704, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-25T13:06:04.490000", "content": "It's becoming a long writeup))) will need bit more time
", "votes": 5, "replies": [ { "id": 1914039, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-08-25T17:15:31.977000", "content": "Take your time! Great details already
", "votes": 1, "replies": [] } ] }, { "id": 1915111, "author_name": "Tonghui Li", "author_url": "", "post_date": "2022-08-26T17:11:51.047000", "content": "Huge congrats Konstantin @kyakovlev, this is incredible commitment your team had! I subscribed 2 months of Colab Pro+ for this comp and that was it…
\nCurious if you know besides diff_mean which of your 'special' agg ideas worked best for you?
btw diff_mean is:
\ndf[f'{col}__diff_mean] = df[col] - df.groupby(['uid'])[col].shift()\ndf[f'{col}__diff_mean] = df.groupby(['uid'])[f'{col}__diff_mean]].transform('mean')\n\ndf = df.drop_duplicates(subset=['uid'], keep='last').reset_index(drop=True)\nstd features worked better than others - diff with std for example
\nnormal diff with client mean was in
\n'diff_func': ['first','mean','std','median','min','max']
and my bad - which of specials.
\nmonotonic functions worked really well
Thanks for the reply!
\nmonotonic functions - Did you create a T/F binary feature or did you calculate for example spearman correlation or similar to quantify monotonicity?
Binary if current more or equal to previous and then took mean.
\nWe wanted to pass to the model information if last statements increase or decrease is normal for the client or not. If due payments are constantly increasing for us it was a sign that risk is growing.
\nAnd such information is not based on absolute values - we constantly wanted to avoid absolute values to be able to generalize well on future data and data with drifts.
", "votes": 1, "replies": [] } ] }, { "id": 1912725, "author_name": "Gaurav Rawat", "author_url": "", "post_date": "2022-08-25T00:21:14.897000", "content": "Congrats guys .. cant wait for your solution to reach 0.809 ..
", "votes": 3, "replies": [] }, { "id": 1918704, "author_name": "Ahmed Taha", "author_url": "", "post_date": "2022-08-29T18:58:54.670000", "content": "very insightful tank youu
", "votes": 1, "replies": [] }, { "id": 1917682, "author_name": "Runzhong Xu", "author_url": "", "post_date": "2022-08-29T01:25:08.577000", "content": "Congratulations!
", "votes": 1, "replies": [] }, { "id": 1917052, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-28T11:47:50.033000", "content": "Here is example how we augmented training Data:
\n
We were removing last statement per client as it would never exists and then applied exactly the same transformations as we did for full history.
\nWe wanted to completely avoid absolute values and with data augmentation the CV and LB were almost on the same level as with absolute values - of course it would not work with clients with 1-2 statements but we had a separate model for them. I believe that that would give much more robust results but on kaggle even slight metric increase is important and we kept \"lasts\".
\nAlso, we were planning to go in -2 max depth and used only 1,2,3,6,11 (most common time-series lags) diff lags to have fe aligned.
\nCombining original data with -1 statement or with -2 statement gave significant CV boost on ROC AUC score (our main metric) and tiny boost on AMEX metric. Combination of 3 sets gave just a tiny boost on ROC AUC but was significantly more memory consuming and slower to train model.
\nAlso, CV splits per original client uid to exclude leaks and validation was performed only on original data to exclude metric degradation tracking for shorten history predictions.
", "votes": 1, "replies": [] }, { "id": 1917023, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-28T11:13:19.733000", "content": "Few examples how noise cleaning works:
\nFor the most features nunique groups within 998 -> 1001 is a sign of noised cleaning possibility.
Simple feature cleaning - in this case group is our new value:
\n
More complex feature with negative \"categories\" we keep negative values (temp2 is a new value):
\n
Very \"noised\" feature (we find initial coefficient and then apply it to groups with low population) - temp2 is a new value:
\n
Thanks for sharing your solution! Very insightful.
", "votes": 1, "replies": [] }, { "id": 1913980, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-08-25T16:05:05.120000", "content": "Normalizations. Sorry, I'm having trouble understanding. For all EXCEPT last month you took mean and std for all customers fort that exact col, month and subset (train/pub/pri)? What about last? I'm actually super confused what the trick was, and why? Maybe try just restating/rephrasing it and maybe I'll figure it or at least have more precise questions…
", "votes": 1, "replies": [ { "id": 1920719, "author_name": "", "author_url": "", "post_date": "2022-08-31T11:00:00", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1913512, "author_name": "Alexander Guldbrand", "author_url": "", "post_date": "2022-08-25T10:49:07.533000", "content": "Congratulation! Looking forward to dive into your solution!
", "votes": 1, "replies": [] }, { "id": 1913260, "author_name": "Mert Burabak", "author_url": "", "post_date": "2022-08-25T09:03:27.457000", "content": "Good Job! well deserved
", "votes": 1, "replies": [] }, { "id": 1913210, "author_name": "Santiago Mota", "author_url": "", "post_date": "2022-08-25T08:18:53.740000", "content": "Congrats to the team. One more waiting for the extended explanation
", "votes": 1, "replies": [] }, { "id": 1913202, "author_name": "Vadim Imaev", "author_url": "", "post_date": "2022-08-25T08:11:16.213000", "content": "Congratulations ! :)
", "votes": 1, "replies": [] }, { "id": 1912995, "author_name": "iliiiiiili", "author_url": "", "post_date": "2022-08-25T04:56:41.570000", "content": "Congratuation!
", "votes": 1, "replies": [] }, { "id": 1912963, "author_name": "Nischay Dhankhar", "author_url": "", "post_date": "2022-08-25T04:28:18.733000", "content": "Glad to see you back on Kaggle competitions with a bang. Congratulations, amazing finish.
", "votes": 1, "replies": [] }, { "id": 1912935, "author_name": "Daisy", "author_url": "", "post_date": "2022-08-25T03:57:12.200000", "content": "Congratulations! Thank you for sharing model params!! Very useful for next competitions!
", "votes": 1, "replies": [] }, { "id": 1912867, "author_name": "shahilpravind", "author_url": "", "post_date": "2022-08-25T02:34:02.830000", "content": "Congratulations! 🎉 Not a lot of experience on Kaggle so not sure if this is normal but was surprised to see that 3/4 of your team is novices and contributors. Gives me hope 😅
", "votes": 1, "replies": [ { "id": 1915182, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-26T18:14:18.977000", "content": "@shahilap96 our team was formed at the very early stage of the competition and exclusively from my company DS department - many other DS colleagues wanted to participate but were not able to made such 3month commitment and decided not to join. Kaggle experience is nice to have but it’s not an obligatory condition - the most important is a “fire in the eyes” and passion to learn and try something new.
\nYes, my teammates are novice on kaggle but not novice in analytics and DS.
\nOne thing that I can say - all my teammates work with the same passion with regular work tasks and trying really hard to achieve goals. I feel very proud of them.
", "votes": 4, "replies": [] }, { "id": 1916523, "author_name": "shahilpravind", "author_url": "", "post_date": "2022-08-28T00:39:14.290000", "content": "Aha, that explains the novice part 😅 People with passion and fire in their eyes are always great to work with. The energy and drive always pushes me to do better. Congrats once again.
", "votes": 0, "replies": [] } ] }, { "id": 1912798, "author_name": "Kim Keonho", "author_url": "", "post_date": "2022-08-25T01:14:05.613000", "content": "Congrats!
\nI'm waiting for your notebook to knwo what had been updated from IEEE competition :)
Congratulations！
", "votes": 1, "replies": [] }, { "id": 1912741, "author_name": "Leandro Destefani", "author_url": "", "post_date": "2022-08-25T00:29:36.860000", "content": "Congratulations, guys, you deserved it! Really well done reaching this score
", "votes": 1, "replies": [] }, { "id": 1912740, "author_name": "Neha", "author_url": "", "post_date": "2022-08-25T00:28:09.160000", "content": "Congratulations 🎉🎉💥
", "votes": 1, "replies": [] }, { "id": 1912732, "author_name": "cong wang", "author_url": "", "post_date": "2022-08-25T00:23:39.650000", "content": "Congratulations！
", "votes": 1, "replies": [] }, { "id": 1913268, "author_name": "Gaju Ahmed", "author_url": "", "post_date": "2022-08-25T09:05:23.537000", "content": "Congratulations 🎉🎉 and Thanks for the solution
", "votes": 2, "replies": [ { "id": 1913318, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-25T09:21:55.303000", "content": "Solution is still in process. Was posted just initial plan and some information about model. Sorry for the delay, but will need 3-4h more to finish it. Hope you'll find it interesting.
", "votes": 3, "replies": [] } ] }, { "id": 1912814, "author_name": "", "author_url": "", "post_date": "2022-08-25T01:25:26.207000", "content": "Congratulations！🎉🎉
", "votes": 2, "replies": [] }, { "id": 1914802, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2022-08-26T12:41:23.430000", "content": "Here is a dataset with 2 submissions:
\nsingle_lgbm_model_v12.0.1.csv -> best single lgbm (mean over folds) -> 0.80891 private / 0.80101 public
\n2nd_place_submission_ranked.csv -> ranked 2nd place -> 0.80938 private / 0.80134 public
interesting that single lgbm could give 6th place on lb
\nfeel free to blend it and check results on lb
\nhttps://www.kaggle.com/datasets/kyakovlev/amex-submissions
Is there a way to see the code ?
", "votes": 0, "replies": [] }, { "id": 2010286, "author_name": "Aman", "author_url": "", "post_date": "2022-10-30T16:47:07.660000", "content": "thanks for sharing such insights, as a newbie, i loved the part on how you collaborated on cloud, recorded all the trained models to go fwd and back and the way you divided the data to apply ensemble techniques. lot to learn.. thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 1946819, "author_name": "", "author_url": "", "post_date": "2022-09-20T05:51:59.280000", "content": "", "votes": 0, "replies": [] }, { "id": 1936425, "author_name": "", "author_url": "", "post_date": "2022-09-12T18:34:58.703000", "content": "", "votes": 0, "replies": [] }, { "id": 1936351, "author_name": "", "author_url": "", "post_date": "2022-09-12T17:18:45.667000", "content": "", "votes": 0, "replies": [] }, { "id": 1925646, "author_name": "", "author_url": "", "post_date": "2022-09-04T07:04:35.363000", "content": "", "votes": 0, "replies": [] }, { "id": 1921781, "author_name": "", "author_url": "", "post_date": "2022-09-01T04:07:45.547000", "content": "", "votes": 0, "replies": [] }, { "id": 1918844, "author_name": "", "author_url": "", "post_date": "2022-08-29T22:47:28.043000", "content": "", "votes": 0, "replies": [ { "id": 1918856, "author_name": "", "author_url": "", "post_date": "2022-08-29T23:02:36.033000", "content": "", "votes": 7, "replies": [] } ] }, { "id": 1914257, "author_name": "", "author_url": "", "post_date": "2022-08-25T22:35:16.320000", "content": "", "votes": 1, "replies": [] }, { "id": 1913065, "author_name": "", "author_url": "", "post_date": "2022-08-25T05:46:17.893000", "content": "", "votes": 0, "replies": [] }, { "id": 1920011, "author_name": "", "author_url": "", "post_date": "2022-08-30T19:42:37.180000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1912721": "*Note: Full code to retrain single model will be shared here in a 2 weeks.*\n\nI would like to say thank you to competition hosts and Kaggle - it was a great pleasure to participate in tabular data competition after many months and years without one. \n\nThank you to all participants (and of course winners - @daishu huge jump during last 3 days and fantastic solo result) - our success is your success - you forced us to try harder - without all of you It would be impossible to learn so many new things and achieve such result.\n\nAnd special \"thank you\" goes to my fantastic teammates:\n- Danila\n- Alexey\n- Igor\n\nNo words would be enough to say how much each of you contributed to the end result.\n\"Do Data Scientists have hobby? Yes -> DS competition\". \n\n---\n\n### Competition announcement\n\nWe were very hyped by the new tabular data competition release (sorry for the external link: [link](https://www.linkedin.com/posts/konstantin-yakovlev-010b46125_american-express-default-prediction-kaggle-activity-6935250015619592194-hqnM)) and immediately decided to participate. Slack notification -> rules and perspective advertisement (100% chance too lose summer holidays and all free time) -> and here we are - four members. Only one member of the team had previous experience in DS competitions participation.\n\nFew rules were established from the beginning:\n- Only free time for competition\n- No \"second\" accounts on Kaggle (even wife/friends to exclude any cheating suspicion)\n- No competition discussion outside of the team\n- We are here to learn and try our best\n\n---\n\n### Infrastructure and pipelines:\n\nEach of us had own machines / resources (GCP/AWS/local). We used Kaggle platform just for a few times. So the first thing we wanted to solve - unified machine to save all artefacts / experiments. We decided to go with AWS. I would say that it is possible to achieve the same result that we have just with Kaggle resources, but it would be bit more stressful for team management. We didn't want to spend a lot of money on AWS but sometimes (during very hot hours) RAM spikes were 500GB+ to permit simultaneous work.\n\nWe tried to use neptune.ai for ML tracking but from July it was not very effective as we entered in brute force zone. \n\n*Advise: Resources management is very critical - find bottleneck and remove it to make your team most effective. At the same time don't burn money recklessly - limit your budged. If any optimization possible - do it as soon as possible to save time and resources.*\n\n---\n\n### Project Structure\nEach run was internally versioned (ex. v1.1.1 - (major version).(fe version).(model version))\nOverall project structure:\n- Initial preprocess -> artifact cleaned and joined df\n- FE -> Many aligned (by uid) dfs with separated features \n- Features selection -> dictionary we selection metadata\n- Holdout Model (fe check and tuning) -> Local validation oof preds / holdout preds/ model / model metadata\n- Full model run -> Model / Model metadata\n- Prediction -> each fold oof predictions / cv split metadata / test predictions\n\nAll these permitted us to go back and forward and check what worked well and what did not and restore experiments in each particular step.\n\n---\n\n### Initial preprocess \nWe wanted to achieve several things with this step:\n- Join Train and Test -> due to many people involved I was afraid that some missed transformation on private test part will be unnoticed. So we sacrifice memory and speed optimization for overall stability and security.\n- Remove detected noise -> (we had options here but ended with unified single one)\n- Transform Customer ID to unified uid \n- Create internal subset feature -> Train / Public / Private\n- Create unified kfold and holdout split -> To align all experiments\n- Separate columns by type and store them separately to minify memory use and load time\n\n#### Remove detected noise\n\nWe didn't use public notebooks for cleaning. Radar's Dataset is fantastic and it is 99% similar to our own transformations.\nWe used \"isle\" identification without any pre-build coefficients.\ndummy code is something like this:\n\n```\n for col in process_columns: \n\n df = temp_df[[col]].sort_values(by=[col]) \n df = df[df[col].notna()].drop_duplicates(subset=[col]).reset_index(drop=True)\n\n df['temp'] = np.floor(df[col] * 100000)\n df['group'] = ((df['temp'] - df['temp'].shift()).abs() >= 100).cumsum()\n\n i = 0\n while True:\n min_val = df[df['group']==i]['temp'].min()\n if min_val>0:\n break\n i += 1\n\n df['temp2'] = np.where(df['temp']>=0, \n np.floor(df['temp']/min_val).astype(np.int32),\n np.round(df['temp']/min_val).astype(np.int32))\n \n mapping = dict(zip(df[col],df['temp2']))\n temp_df[col] = temp_df[col].map(mapping)\n\n print(col, df['group'].nunique(), df[col].nunique())\n print(df.groupby(['group'])['temp','temp2'].agg(['min','max','count','nunique']).head(40))\n```\n\n#### Create internal subset feature\nWe used last statement month to create 0/1/2 feature and store in in \"index\" df\n\n#### Create unified kfold and holdout split\nFixed random seed (of course 42) to make spits and then took 20% of customers to holdout group (to test stacking / blending / etc)\n\n#### Separate columns by type\nAfter cleaning we had several columns \"groups\".\n```\nall_files = [\n'p_columns', -> just p columns as we thought that they are very different (and P_2 is internal amex \"scoring\" model)\n'objects_radar_columns', -> order encoding (we were checking where out cleaning differs from public approaches and here was the unique place) \n'objects_columns', -> onehot encoding\n'categorical_cleaned__D__columns', -> no noise categoricals\n'categorical_binary__S__columns', -> cleaned binary\n'categorical_binary__R__columns', -> cleaned binary\n'categorical_binary__D__columns', -> cleaned binary\n'categorical_binary__B__columns', -> cleaned binary\n'categorical__D__columns', -> removed noise categoricals\n'categorical__B__columns', -> removed noise categoricals\n'cleaned__B__columns', -> removed noise continuous \n'cleaned__D__columns', -> removed noise continuous \n'cleaned__R__columns', -> removed noise continuous \n'cleaned__S__columns', -> removed noise continuous \n'rest__B__columns', -> have no idea what to do with it -> floor \n'rest__D__columns', -> have no idea what to do with it -> floor \n'rest__R__columns', -> have no idea what to do with it -> floor \n'rest__S__columns', -> have no idea what to do with it -> floor \n]\n```\nThanks again to @raddar we always used your preprocess as a baseline.\n\nWe were able to load just portion of data -> do fe -> concat to \"index\" as all dfs were aligned by index. Also such split permitted us to do fe by feature type to accelerate process and see more statistically valuable metric change. \n\n---\n\n### FE\n\nWe started with careful fe column by column or small subset and it worked well until 1xx features and then any metric improvement or degradation was not statistically significant and many features \"overlapped\" on importance and significance. \n\n*Note: I believe that it is possible to build silver zone robust model with only 3xx features*\n\nSo we started from scratch with brute force))) Of course there was no need to apply \"nunique\" (for example to binary features) and our previous step helped us to limit fe.\n\n```\nall_aggregations = {\n 'agg_func': ['last','mean','std','median','min','max','nunique'],\n 'diff_func': ['first','mean','std','median','min','max'],\n 'ratio_func': ['first','mean','std','median','min','max'],\n 'lags': [1,2,3,6,11],\n 'special': ['ewm','count_month_enc','monotonic_increase','diff_mean','major_class',\n 'normalization','top_outlier','bottom_outlier','normalization_mean','top_outlier_mean','top_outlier_mean']\n} \n```\n+ pca (horizonal and vertical) + horizontal combinations + horizontal aggregations.\n\nagg_func -> normal aggregations by uid\ndiff_func -> diff last - xxx -> std diff worked better than any other\nratio_func -> ratio_func last/xxx\nlags -> diff last - Nx\nspecial -> some special transformations -> count_month_enc worked well for categorical / emw for continous \n\nWe ended up with about 7k features (stored file by group and by agg type for faster loading). \nNext thing was to figure out what works and what not -> this topic was the most challenging for us. \n\n#### Normalizations\n\nIt's better to call it Standardization (x - m) / s -> as we had also normalization test the name became constant \"normalization\")))\n```\ndf.groupby(['dt_month','subset'])[col].agg(['mean','std'])\n```\ndt_month -> month of the statement\nsubset -> train / public / private\nand mean and std from clients that had full statement history.\n\nWe have to have temporal shift to make it work. So we did a \"trick\" removed last statement for each client and applied exactly same transformation for each client and merged appropriate labels. So we had 2 lines in training set for almost each client BUT validated results only on last statement during CV runs and Holdout checks. It more or less same as adding noised data but we had temporal drift and model was able to work better on unknown future data with \"possible\" data drift.\n\n---\n\n### Features selection\n\nOoohh that was really fun. \n\nWe used gbdt boosting type during experiments as it was very aligned with dart mode but was significantly faster.\nAlso, we used ROC AUC score during our experiments as we believed that due to amex instability we can't use it for decision making (of course we tracked log loss and amex).\n\nIn previous step we brute forced many features and now is time to clean them out.\nAll feature selection was done with 5 CV folds training + independent check on 20% holdout data.\n\n1. Zero importance -> Right from the start we were able to through away 1.5k features that had exactly 0 importance (lgbm importance). That means that with 250 bins and 2**10 data in leaf those features are not participating in any split.\n\n2. Stepped hierarchical permutation importance -> we defined 300 initial features and looped over all other features subsets (600+) - was very time consuming but very stable.\nNote: we shuffled order of the subset to force model try different combinations.\nAdd features subset -> train model -> permutate -> drop negative features (negative mean over 5 seeds) -> add new subset -> ...\nDuring this part that took almost 3 days we limited features to 3k -> 0.800 lb\n\n3. Stepped permutation importance.\nTake all features -> train model -> permutate -> drop 20% of worst performed features (only negative) -> repeat. Final subset was 25xx features (and different from previous step) -> 0.800 lb\n\n4. Forward feature selection.\nWe defined 300 initial features and simply added subset by subset and compared ROC AUC if metric change was > 0.0003 we kept the subset. -> 0.800 lb\n\n5. Time series CV.\nFor very doubtful features as PCA and Normilized values we used to different validation stratagies:\n- Train on first 6 month values (last statement of the first 6 months went to train set) and validate on last 6 (also just last statement of the last 6 months). We trained model without temporal feature and then with if result was better on CV and on holdout we added to final features subset.\n- We used P_2, B_1, B_2 as a proxy target and MSE loss with combined Train and Test to see if we did right transformation and result did not degrade.\n\nMany other options we tried but result was not stable.\n\nFinal subset came from \"Forward feature selection\" plus overlapped features from other technics minus overlapped negative combination. -> lb 0.801 single model.\n\nWe tried to blend many models with different subset as we believed that it should give huge LB boost (based on holdout blending tests) but it didn't work well for lb. \n\n--- \n\n### Model \n\nIn my own experience, DART never worked better and here we have proof that in DS \"all depends.\" We did experiments with DART in the beginning and it did not show any metric improvement with our params and baseline model features subset. Later we found @ragnar123 notebook and gave it one more try and it worked marvelously.\n\nFrom the beginning, we tried to build a more complex model with 2**7+ leaves and 0.7+ features but failed. It still puzzles me why a simple model with a very low number of features works here. \n\nI saw such behaviour mostly on synthetic data and stacking - so we tried to find out if data is syntetic (at least partly) and deanonimize internal scoring values - but didn't make it.\n\nOur best single lgbm model was trained on 29xx features. 5 folds CV - no stratification by any option. Training data - 2 last staements for each client (transformed independently). Params:\n\n```\nlgb_params = {\n 'boosting_type': 'dart',\n 'objective': 'cross_entropy', \n 'metric': ['AUC'],\n 'subsample': 0.8, \n 'subsample_freq': 1,\n 'learning_rate': 0.01, \n 'num_leaves': 2 ** 6, \n 'min_data_in_leaf': 2 ** 11, \n 'feature_fraction': 0.2, \n 'feature_fraction_bynode':0.3,\n 'first_metric_only': True,\n 'n_estimators': 17001, # -> 5000 for gbdt \n 'boost_from_average': False,\n 'early_stopping_rounds': 300,\n 'verbose': -1,\n 'num_threads': -1,\n 'seed': SEED,\n}\n```\n\nBlend -> Power (2) rank blend of Dart lgbm (0.801 public) / GBDT lgbm (0.799 public) / Catboost models (0.799 public)\n\nSingle lgbm with 3 last statements showed even better CV by we didn't have enough time to retrain it (full DART run for 5 folds took 12+ hours there).\n\nIt was obvious that clients with a little number of statements will not get benefit from all 2k features. So we created a special model that was trained only on 300 features with custom params (also dart). Predictions for clients with <=2 statements came exclusively from such model and were not blended with other models. \n\nHow did we combine the result from 2 independent models to not destroy the final ranking?\n\n| Client id | Number of statements | Basic ranking | <=2 prediction | Final ranking |\n| --- | --- | --- | --- | --- |\n| 1 | 13 | 5 | ... | 5 |\n| 1 | 2 | 4 | 0.1 | 3 |\n| 1 | 13 | 8 | ... | 8 |\n| 1 | 1 | 3 | 0.5 | 4 |\n| 1 | 13 | 7 | ... | 7 |\n| 1 | 13 | 2 | ... | 2 |\n\nwe kept ranking for >2 statements and for rest resorted within initial ranking group (hope it's clear enough))). Was is perfect - no, but it was very stable with really tiny improvement (because of number of such clients in public and private test parts).\n\nWhat also worked well:\n- Train on all data without folds splitting and stop just 1000 rounds further then CV showed. \n\nWhat didn't work:\n- Stacking by any mean\n- Many models with different seed and fe order\n- Massive blend of different models with different types and features (blend worked well till 0.799 and then any low performed model 0.795- made public score worse - we use public CV as additional holdout set and used approaches that worked well on local CV and LB - anything that worked partially was not used in the end).\n\nAgain, nothing really fancy here. The main thing that helped us align Train / CV with LB was very high 'min_data_in_leaf'.\nNo optuna used -> just manual old school tuning based on data feeling.\nWe did many experiments with weights and loss functions but none of them worked.\n\nDue to AMEX metric specification it was obvious that focal loss should work but it didn't. We tried several times to switch loss function during the competition period and the result was the same. \n\nError analysis showed that model makes errors without any \"pattern\" -> stacking didn't work for holdout set (25% of data) and we had doubts that it will work on private/public test parts. We kept only LR/Lasso(0.02) for blending options to choose submissions.\n\nCross validation -> standard 5folds CV split by client ID. The unique thing that we did here is \"prespliting\" to align all CV between team members to be able to compare results directly.\n \n---\n\nWhat left without mentions:\n- EDA on data\n- Denoising experiments\n- Data deanonymization -> didn't manage to make it\n- Features pairs and triples combinations -> that didn't work well\n- NaN filling -> didn't work\n- Clusterization -> didn't work\n- Hundreds of experiments with features selection process and internal discussions about it.\n- Adding noised data (noise / swap noise) that leaded to interesting but doubtful results\n- Model tuning \n- Removing absolute values and keep only diff or ratios -> should be more stable for future data but we saw some lb degradation and didn't proceed\n- pseudo labeling\n\nWhat we always wanted but didn't found time to do:\n- NN - we have no NN in our final blend\n- P_2 or any other column prediction (1/2/3/4 months ahead) with combined data and use it as meta information for lgbm main model\n- 11 / 12 / 13 statements joined training on different subsets (df was too large and training was slow)\n\n--- \n\n### Internal initial plan\n\n```\n########################### Data preprocessing and Data evaluation\n#################################################################################\n\n## Added noise removal -> GOOD2DO\n# There is no doubt that some Noise was injected in data\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327649\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327651\n# We need to find a way to remove it \n# the best option to not follow public approach\n# At least with columns where columns have overlaped population\n\n## Data minification for FE -> GOOD2DO\n# Datatype downcasting\n# Pickle/Parquet/Feather \n# Be careful with floats16 as it may lead to bad agg results\n# Also float16 may lead to some signal degradation due to precision and values changes\n\n## Evaluate values distributions and NaNs -> GOOD2DO\n# Full 13 months history\n# Train against Test Public and Test Private\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327926\n#\n# Need to try:\n# Kolmogorov–Smirnov test -> GOOD2DO\n#\n# Adversarial validation -> GOOD2DO\n# https://www.kaggle.com/code/carlmcbrideellis/what-is-adversarial-validation/notebook (just as simple example)\n#\n# Entropy / Distances / etc....\n#\n# Visual checks)))\n#\n# We need to find if ANY feature has very different distribution in PRIVATE test set\n# If that feature works for Public part it doesn't mean that it will work for Private\n\n########################### Targets\n#################################################################################\n\n# We need to find a way to get more targets -> GOOD2DO\n# as we currently training on a single point by client we could greatly improve results\n# by extending our training set with new targets\n#\n# Find default periods in current client history and make appropriate labeling -> GOOD2DO\n# Make 2 level model -> predict p_2 values as normal time-series model and feed it to 2nd level GBT\n\n########################### Separate Models\n#################################################################################\n\n# Probably it's a good idea to make separate models for each subdatasets\n# Full history (13 months)\n# Less than 13 months\n\n########################### External Data\n#################################################################################\n# We can try to add \"Consumer index\" or any other independent temporal feature\n# Will not work if we will not be able to expand targets and add temporal feature\n\n########################### FE\n#################################################################################\n\n# We didn't make anything special here -> July\n# AGGS (Stats by client)\n# Rollings\n# History length feature (not sure if it will help with Private Test)\n# Should we correct statements dates and add NaNs?\n# ReRanking categorical features by P_2 or Target\n# Clusterization (4+ groups feature by feature)\n# Count and Mean encodings for categorical features\n# Features combinations (sum/prod/power) -> bruteforce\n# PCA or any other dimension reduction by features groups\n\n# We need to find if there is \"connection\" between clients in Train -> Public Test -> Private Test\n# we have 458913 + 924621 -> 1383534 If I were AMEX I would export 1M clients (or other round number)\n# so may be 384 534 Clients are overlaps\n\n# Clip by 5 - 95 percentile\n\n########################### Features Selection\n#################################################################################\n# Permutation importance (use all fold only!!!) -> recursive elimination (because of quantity of features -> 3-4 rounds with 0 and 50% negative drop) \n# SHAP\n# Highly correlated features (.98+?)\n# Forward selection (may take ages and due aggs may be not effective - probably by feature block) \n# Backward elimination (may take ages and due aggs may be not effective - probably by feature block) \n\n########################### CV\n#################################################################################\n# Mean Target differs my \"history length\" -> could be wise to do GroupedStratifeidFolds by history length\n# For sure Splits should be done by client\n# Target stratification to balance folds\n\n########################### Loss function / Metric\n#################################################################################\n# Clean and fast np/torch metric\n# Now it's in helper (need to cleanup that)\n# https://www.kaggle.com/code/rohanrao/amex-competition-metric-implementations\n#\n# I don't believe that we will have better results with different loss function\n# But it worth to try at least focal loss\n# https://maxhalford.github.io/blog/lightgbm-focal-loss/\n#\n# Weights -> we should try change weights there\n# weights by class\n# weights by some history length\n# weights by internal fe group\n#\n# We need custom metric for catboost\n# example https://catboost.ai/en/docs/concepts/python-usages-examples#logloss1\n\n########################### Models\n#################################################################################\n\n## First level choice\n# LGB/XGB/CTB -> our main models here for sure\n# After stabilizing the baseline model and base feature we need to make 1st round tuning\n\n## Catboos specials\n# Categorical features\n# Embeding features\n\n## NN (GPU/TPU) -> RNN / LSTM / Transformer\n# TPU -> tensorflow (as it works better there)\n\n## NN -> AE / VAE / DAE -> as a denoising model hidden layer as input for GBT models\n# No need complex approach - just fast check the idea and in case of success move to big model\n\n########################### Blending\n#################################################################################\n# Weighted Average\n# Power Average\n# Weighted Rank Average\n# Linear/SVM\n# Postprocessing?\n```", "1912746": "@kyakovlev Congratulations Konstantin and team. I am so happy for you. You are the best tabular data feature engineer that I know. I'm glad to see your skills achieved great success! I'm looking forward to learning about your solution!", "1920789": "A huge congratulations and thanks for such a detail explanation. As Chris said, the best tabular feature engineer! \n\nSome questions from my side:\n\n**1.** Based on your experience, doing brute force feature engineering (i.e. create a ton of potential new features and then perform feature selection) gives better results than going column by column?\n**2.** How did you choose the initial LGBM hyperparameters before starting the feature selection phase? \n**3.** Did you retune the hyperparameters at some point of the feature selection phase (e.g. when you added/removed a significant number of features) or kept them fixed all the time?\n**4.** Could you give some examples of what type of feature subsets you include at each step of the feature selection phase (e.g. one subset could be the mean of `cleaned__B__columns`)?\n**5.** Your final selected features came from \"Forward feature selection\". I tried it a couple of times at work and saw that tends to overfit, do you have the same experience?\n**6.** How did you realize that increasing the value of `min_data_in_leaf` favour CV/LB alignment?\n\nAlso waiting for the code 👀", "1913704": "It's becoming a long writeup))) will need bit more time", "1915111": "Huge congrats Konstantin @kyakovlev, this is incredible commitment your team had! I subscribed 2 months of Colab Pro+ for this comp and that was it...\nCurious if you know besides `diff_mean` which of your `'special'` agg ideas worked best for you?", "1912725": "Congrats guys .. cant wait for your solution to reach 0.809 .. ", "1918704": "very insightful tank youu", "1917682": "Congratulations!", "1917052": "Here is example how we augmented training Data:\n\n\n\nWe were removing last statement per client as it would never exists and then applied exactly the same transformations as we did for full history.\n\n- Last values are different but within distributions \n- Agg values are different but within distributions\n- Model has temporal component \n\nWe wanted to completely avoid absolute values and with data augmentation the CV and LB were almost on the same level as with absolute values - of course it would not work with clients with 1-2 statements but we had a separate model for them. I believe that that would give much more robust results but on kaggle even slight metric increase is important and we kept \"lasts\".\n\nAlso, we were planning to go in -2 max depth and used only 1,2,3,6,11 (most common time-series lags) diff lags to have fe aligned. \n\nCombining original data with -1 statement or with -2 statement gave significant CV boost on ROC AUC score (our main metric) and tiny boost on AMEX metric. Combination of 3 sets gave just a tiny boost on ROC AUC but was significantly more memory consuming and slower to train model.\n\nAlso, CV splits per original client uid to exclude leaks and validation was performed only on original data to exclude metric degradation tracking for shorten history predictions.", "1917023": "Few examples how noise cleaning works:\nFor the most features nunique groups within 998 -> 1001 is a sign of noised cleaning possibility.\n\nSimple feature cleaning - in this case group is our new value:\n\n\nMore complex feature with negative \"categories\" we keep negative values (temp2 is a new value):\n\n\nVery \"noised\" feature (we find initial coefficient and then apply it to groups with low population) - temp2 is a new value:\n", "1915380": "Thanks for sharing your solution! Very insightful.", "1913980": "Normalizations. Sorry, I'm having trouble understanding. For all EXCEPT last month you took mean and std for all customers fort that exact col, month and subset (train/pub/pri)? What about last? I'm actually super confused what the trick was, and why? Maybe try just restating/rephrasing it and maybe I'll figure it or at least have more precise questions...", "1913512": "Congratulation! Looking forward to dive into your solution!", "1913260": "Good Job! well deserved", "1913210": "Congrats to the team. One more waiting for the extended explanation", "1913202": "Congratulations ! :)", "1912995": "Congratuation!", "1912963": "Glad to see you back on Kaggle competitions with a bang. Congratulations, amazing finish. ", "1912935": "Congratulations! Thank you for sharing model params!! Very useful for next competitions!", "1912867": "Congratulations! 🎉 Not a lot of experience on Kaggle so not sure if this is normal but was surprised to see that 3/4 of your team is novices and contributors. Gives me hope 😅", "1912798": "Congrats!\nI'm waiting for your notebook to knwo what had been updated from IEEE competition :)", "1912764": "Congratulations！", "1912741": "Congratulations, guys, you deserved it! Really well done reaching this score", "1912740": "Congratulations 🎉🎉💥", "1912732": "Congratulations！", "1913268": "Congratulations 🎉🎉 and Thanks for the solution", "1912814": "Congratulations！🎉🎉", "1914802": "Here is a dataset with 2 submissions:\n\nsingle_lgbm_model_v12.0.1.csv -> best single lgbm (mean over folds) -> 0.80891 private / 0.80101 public\n2nd_place_submission_ranked.csv -> ranked 2nd place -> 0.80938 private / 0.80134 public\n\ninteresting that single lgbm could give 6th place on lb\n\nfeel free to blend it and check results on lb\nhttps://www.kaggle.com/datasets/kyakovlev/amex-submissions", "2245760": "Is there a way to see the code ? ", "2010286": "thanks for sharing such insights, as a newbie, i loved the part on how you collaborated on cloud, recorded all the trained models to go fwd and back and the way you divided the data to apply ensemble techniques. lot to learn.. thanks for sharing!", "1946819": "Congratulations！", "1936425": "can you share code, as was done for 1st place?", "1936351": "Congratulations @kyakovlev and team-mates. Thank you for posting such detailed approach for us to learn from.", "1925646": "A huge congratulations! Thanks for sharing the techniques and the best practice, really learn a lot from you! In particular, I am interested in how you organize different versions of your experiments, it is very important and useful for complex research like you mentioned. \n\nCan you elaborate more on the details? much appreciated!", "1921781": "Really great！", "1918844": "can you share code pls", "1914257": "Congratulations！Thanks for sharing!", "1913065": "Thank You!\nWaiting for more ))", "1920011": "very useful, thanks for sharing!" } }