{ "id": 347668, "title": "10th Place Solution: XGB with Autoregressive RNN features", "url": "/competitions/amex-default-prediction/discussion/347668", "author_name": "Jiwei Liu", "post_date": "2022-08-25T04:12:09.095000", "votes": 93, "comment_count": 29, "views": 0, "content": "

What a competition! I really enjoyed it and only hope I could have found more time. First of all, I would like to thank Raddar @raddar, Martin @ragnar123, and many others who generously shared codes and datasets! The public solutions are of amazing quality. And it also determines my game plan: create something original and blend it with the best public solution. My solution is based on RAPIDS cudf for dataframe processing, XGB for training, and pytorch lightning for feature extraction.

\n

There are two vital observations of this dataset:

\n\n

Let's start with the latter observation:the sequence length of each customer profiles plays a critical role in the model performance:

\n
import cudf\npath = '/raid/amex'\ntrain = cudf.read_parquet(f'{path}/train.parquet',columns=['customer_ID'])\ntrainl = cudf.read_csv(f'{path}/train_labels.csv')\ntrain = train.merge(trainl,on='customer_ID',how='left')\ntrain['seq_len'] = train.groupby('customer_ID')['target'].transform('count')\ntrain = train.drop_duplicates('customer_ID',keep='last')\ntrain.groupby('seq_len').agg({'target':['mean','count']}).sort_index(ascending=False)\n
\n

output:

\n
           target\n       mean    count\nseq_len        \n13    0.231788    386034\n12    0.389344    10623\n11    0.446737    5961\n10    0.462282    6721\n9     0.450164    6411\n8     0.447300    6110\n7     0.418430    5198\n6     0.387670    5515\n5     0.392635    4671\n4     0.416221    4673\n3     0.358602    5778\n2     0.318465    6098\n1     0.335742    5120\n
\n

It is obvious that sequence length 13 is the most common but also with a significantly lower mean default rate. At first glance, I thought it meant shorter sequences are easier to predict since they have more positive samples. But I'm quickly proven wrong when checking my cross-validation results:

\n
Fold 0 amex 0.7990 logloss 0.2144\nFold 0 L13 amex 0.8214 logloss 0.1928\nFold 0 Other amex 0.6724  logloss 0.3289\n
\n

The 1st line is the overall score. The 2nd line is the score of sequences of length 13 and the 3rd line is the score of all the rest sequences. Apparently, shorter sequences have a much worse score than the full sequences of length 13. This is also an implication of how the short sequences are truncated: the more recent profiles are deleted, which could explain the big degradation of the score because more recent profiles have more predicting power in general. For example, let's say for 13 consecutive months (M1~M13) and sequence A is of length 13 and sequence B is of length 8:

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1 \nB  1  1  1  1  1  1  1  1  1  0  0  0  0  0 \n
\n


\nwhere 1 means features exist and 0 means features missing. Of course, there is another possibility:

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1 \nB  0  0  0  0  0  1  1  1  1  1  1  1  1  1 \n
\n


\nWe can actually find out which one is more plausible by unstacking dataframes in the above two ways and run xgboost with them, respectively. As expected, the former has a better CV score which indicates it is likely how the truncation of short sequences is done.

\n

If we could somehow predict the missing profiles of sequence B, the life of the downstream XGB models would be made much easier. An intuitive choice is to generate the missing profiles using a one-dimension auto-regressive RNN. Bascially we want to predict the features of the next month based on the feature values of the current month and all previous months. And when we have the prediction for the next month, we can use it as part of the input and predict again and so on so forth. This is also where the availability of the big test data really shines. Since we are predicting features, not target, we can train our models using both train and test data. The RNN structure is very simple: just one GRU layer and some FC layers. The RNN performance is pretty decent. In terms of RMSE of all 178 numerical features, the GRU achieves validation RMSE 0.019. For simplicity, all features are log-transformed and fillna(0). You might wonder how good is RMSE 0.019. We can simply compare it with the naive baseline: just repeat the last available month. For example, if I'm asked to predict features of M2, the naive baseline is just output features of M1. The RMSE of this naive baseline is 0.03 so our RNN actually learns something and could be useful.

\n

The rest would be straightforward, after predicting missing months, now every sequence is of length 13 so I just unstack the dataframe to increase the number of features 13x. For example, instead of having one feature P_2 of the last month, now we have 13 features P_2_M_1 to P_2_M_13. These are the most useful features I created. For downstream classifiers I only use XGB so that it is not similar to the great LGB DART notebook. By varying RNN hyperparameters, XGB hyperparameters and different combination of features, I end up with 7 XGB models, whose ensemble is 0.7993 CV and 0.799 public LB. Averaging it with the best public solution and with extraordinary luck, my final submission ended up in the gold zone.

\n

\"\"

\n

The final thought is my best auto-regressive features are generated 1 hour before the deadline. I'm very happy it worked!

", "messages": [ { "id": 1912952, "postDate": "2022-08-25T04:12:09.097Z", "content": "

What a competition! I really enjoyed it and only hope I could have found more time. First of all, I would like to thank Raddar @raddar, Martin @ragnar123, and many others who generously shared codes and datasets! The public solutions are of amazing quality. And it also determines my game plan: create something original and blend it with the best public solution. My solution is based on RAPIDS cudf for dataframe processing, XGB for training, and pytorch lightning for feature extraction.

\n

There are two vital observations of this dataset:

\n\n

Let's start with the latter observation:the sequence length of each customer profiles plays a critical role in the model performance:

\n
import cudf\npath = '/raid/amex'\ntrain = cudf.read_parquet(f'{path}/train.parquet',columns=['customer_ID'])\ntrainl = cudf.read_csv(f'{path}/train_labels.csv')\ntrain = train.merge(trainl,on='customer_ID',how='left')\ntrain['seq_len'] = train.groupby('customer_ID')['target'].transform('count')\ntrain = train.drop_duplicates('customer_ID',keep='last')\ntrain.groupby('seq_len').agg({'target':['mean','count']}).sort_index(ascending=False)\n
\n

output:

\n
           target\n       mean    count\nseq_len        \n13    0.231788    386034\n12    0.389344    10623\n11    0.446737    5961\n10    0.462282    6721\n9     0.450164    6411\n8     0.447300    6110\n7     0.418430    5198\n6     0.387670    5515\n5     0.392635    4671\n4     0.416221    4673\n3     0.358602    5778\n2     0.318465    6098\n1     0.335742    5120\n
\n

It is obvious that sequence length 13 is the most common but also with a significantly lower mean default rate. At first glance, I thought it meant shorter sequences are easier to predict since they have more positive samples. But I'm quickly proven wrong when checking my cross-validation results:

\n
Fold 0 amex 0.7990 logloss 0.2144\nFold 0 L13 amex 0.8214 logloss 0.1928\nFold 0 Other amex 0.6724  logloss 0.3289\n
\n

The 1st line is the overall score. The 2nd line is the score of sequences of length 13 and the 3rd line is the score of all the rest sequences. Apparently, shorter sequences have a much worse score than the full sequences of length 13. This is also an implication of how the short sequences are truncated: the more recent profiles are deleted, which could explain the big degradation of the score because more recent profiles have more predicting power in general. For example, let's say for 13 consecutive months (M1~M13) and sequence A is of length 13 and sequence B is of length 8:

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1 \nB  1  1  1  1  1  1  1  1  1  0  0  0  0  0 \n
\n


\nwhere 1 means features exist and 0 means features missing. Of course, there is another possibility:

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1 \nB  0  0  0  0  0  1  1  1  1  1  1  1  1  1 \n
\n


\nWe can actually find out which one is more plausible by unstacking dataframes in the above two ways and run xgboost with them, respectively. As expected, the former has a better CV score which indicates it is likely how the truncation of short sequences is done.

\n

If we could somehow predict the missing profiles of sequence B, the life of the downstream XGB models would be made much easier. An intuitive choice is to generate the missing profiles using a one-dimension auto-regressive RNN. Bascially we want to predict the features of the next month based on the feature values of the current month and all previous months. And when we have the prediction for the next month, we can use it as part of the input and predict again and so on so forth. This is also where the availability of the big test data really shines. Since we are predicting features, not target, we can train our models using both train and test data. The RNN structure is very simple: just one GRU layer and some FC layers. The RNN performance is pretty decent. In terms of RMSE of all 178 numerical features, the GRU achieves validation RMSE 0.019. For simplicity, all features are log-transformed and fillna(0). You might wonder how good is RMSE 0.019. We can simply compare it with the naive baseline: just repeat the last available month. For example, if I'm asked to predict features of M2, the naive baseline is just output features of M1. The RMSE of this naive baseline is 0.03 so our RNN actually learns something and could be useful.

\n

The rest would be straightforward, after predicting missing months, now every sequence is of length 13 so I just unstack the dataframe to increase the number of features 13x. For example, instead of having one feature P_2 of the last month, now we have 13 features P_2_M_1 to P_2_M_13. These are the most useful features I created. For downstream classifiers I only use XGB so that it is not similar to the great LGB DART notebook. By varying RNN hyperparameters, XGB hyperparameters and different combination of features, I end up with 7 XGB models, whose ensemble is 0.7993 CV and 0.799 public LB. Averaging it with the best public solution and with extraordinary luck, my final submission ended up in the gold zone.

\n

\"\"

\n

The final thought is my best auto-regressive features are generated 1 hour before the deadline. I'm very happy it worked!

", "rawMarkdown": "What a competition! I really enjoyed it and only hope I could have found more time. First of all, I would like to thank Raddar @raddar, Martin @ragnar123, and many others who generously shared codes and datasets! The public solutions are of amazing quality. And it also determines my game plan: **create something original and blend it with the best public solution.** My solution is based on RAPIDS cudf for dataframe processing, XGB for training, and pytorch lightning for feature extraction.\n\nThere are two vital observations of this dataset:\n- the big test data which is from the future is available.\n- short sequences (<13) are culprits of bad performance.\n\nLet's start with the latter observation:**the sequence length of each customer profiles** plays a critical role in the model performance:\n```\nimport cudf\npath = '/raid/amex'\ntrain = cudf.read_parquet(f'{path}/train.parquet',columns=['customer_ID'])\ntrainl = cudf.read_csv(f'{path}/train_labels.csv')\ntrain = train.merge(trainl,on='customer_ID',how='left')\ntrain['seq_len'] = train.groupby('customer_ID')['target'].transform('count')\ntrain = train.drop_duplicates('customer_ID',keep='last')\ntrain.groupby('seq_len').agg({'target':['mean','count']}).sort_index(ascending=False)\n```\noutput:\n```\n target\n mean\tcount\nseq_len\t\t\n13\t0.231788\t386034\n12\t0.389344\t10623\n11\t0.446737\t5961\n10\t0.462282\t6721\n9\t 0.450164\t 6411\n8\t 0.447300\t 6110\n7\t 0.418430\t 5198\n6\t 0.387670\t 5515\n5\t 0.392635\t 4671\n4\t 0.416221\t 4673\n3\t 0.358602\t 5778\n2\t 0.318465\t 6098\n1\t 0.335742\t 5120\n```\nIt is obvious that **sequence length 13** is the most common but also with a significantly lower mean default rate. At first glance, I thought it meant shorter sequences are easier to predict since they have more positive samples. But I'm quickly proven wrong when checking my cross-validation results:\n```\nFold 0 amex 0.7990 logloss 0.2144\nFold 0 L13 amex 0.8214 logloss 0.1928\nFold 0 Other amex 0.6724 logloss 0.3289\n```\nThe 1st line is the overall score. The 2nd line is the score of sequences of length 13 and the 3rd line is the score of all the rest sequences. Apparently, shorter sequences have a much worse score than the full sequences of length 13. This is also an implication of how the short sequences are truncated: the more recent profiles are deleted, which could explain the big degradation of the score because more recent profiles have more predicting power in general. For example, let's say for 13 consecutive months (M1~M13) and sequence A is of length 13 and sequence B is of length 8:\n```\n M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA 1 1 1 1 1 1 1 1 1 1 1 1 1 1 \nB 1 1 1 1 1 1 1 1 1 0 0 0 0 0 \n``` \nwhere `1` means features exist and `0` means features missing. Of course, there is another possibility:\n ```\n M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA 1 1 1 1 1 1 1 1 1 1 1 1 1 1 \nB 0 0 0 0 0 1 1 1 1 1 1 1 1 1 \n``` \nWe can actually find out which one is more plausible by `unstacking` dataframes in the above two ways and run xgboost with them, respectively. As expected, the former has a better CV score which indicates it is likely how the truncation of short sequences is done.\n\nIf we could somehow predict the missing profiles of sequence `B`, the life of the downstream XGB models would be made much easier. An intuitive choice is to generate the missing profiles using a one-dimension auto-regressive RNN. Bascially we want to predict the features of the next month based on the feature values of the current month and all previous months. And when we have the prediction for the next month, we can use it as part of the input and predict again and so on so forth. This is also where the availability of the big test data really shines. Since we are predicting features, not `target`, we can train our models using both `train` and `test` data. The RNN structure is very simple: just one GRU layer and some FC layers. The RNN performance is pretty decent. In terms of RMSE of all 178 numerical features, the GRU achieves validation `RMSE 0.019`. For simplicity, all features are log-transformed and `fillna(0)`. You might wonder how good is `RMSE 0.019`. We can simply compare it with the naive baseline: just repeat the last available month. For example, if I'm asked to predict features of M2, the naive baseline is just output features of M1. The RMSE of this naive baseline is `0.03` so our RNN actually learns something and could be useful.\n\nThe rest would be straightforward, after predicting missing months, now every sequence is of length 13 so I just unstack the dataframe to increase the number of features 13x. For example, instead of having one feature `P_2` of the last month, now we have 13 features `P_2_M_1` to `P_2_M_13`. These are the most useful features I created. For downstream classifiers I only use XGB so that it is *not similar* to [the great LGB DART notebook](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977). By varying RNN hyperparameters, XGB hyperparameters and different combination of features, I end up with 7 XGB models, whose ensemble is 0.7993 CV and 0.799 public LB. Averaging it with the best public solution and with extraordinary luck, my final submission ended up in the gold zone.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F100236%2Facf9b55d595c8c030770291e3dd69844%2FWechatIMG5474.png?generation=1661401124767084&alt=media)\n\nThe final thought is my best auto-regressive features are generated 1 hour before the deadline. I'm very happy it worked!", "votes": 93 }, { "id": 1915127, "postDate": "2022-08-26T17:24:46.920Z", "content": "

I forgot to thank @pyagoubi Art Vandelay for his great kernel: https://www.kaggle.com/code/pyagoubi/amex-eda-evolvement-of-numeric-features-over-time
\nhere is a screenshot of his drawing:
\n\"\"
\nThe pattern shows a clear and predictable trend, which motivates me to go down this path.

", "rawMarkdown": "I forgot to thank @pyagoubi Art Vandelay for his great kernel: https://www.kaggle.com/code/pyagoubi/amex-eda-evolvement-of-numeric-features-over-time\nhere is a screenshot of his drawing:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F100236%2F3364ddfbcdfea257e2c047971aaaf0c2%2FScreen%20Shot%202022-08-26%20at%201.24.11%20PM.png?generation=1661534676162068&alt=media)\nThe pattern shows a clear and predictable trend, which motivates me to go down this path. ", "votes": 3, "replies": [ { "id": 1916368, "postDate": "2022-08-27T19:54:50.567Z", "content": "

wow thanks!

", "rawMarkdown": "wow thanks!" } ] }, { "id": 1914643, "postDate": "2022-08-26T09:04:53.403Z", "content": "

Really great idea to generate features for the missing periods!

\n

Therefore just to understand, every cid has 13 records in the end, right?
\nAfter generating the features, did you also predict the target for these new records, or what value for target did you use?

\n

Can you also give more info on the one-dimension auto-regressive RNN? Does the RNN predicts all the features at once or loop / predict for every feature?

\n

Thanks

", "rawMarkdown": "Really great idea to generate features for the missing periods!\n\nTherefore just to understand, every cid has 13 records in the end, right?\nAfter generating the features, did you also predict the target for these new records, or what value for target did you use?\n\nCan you also give more info on the one-dimension auto-regressive RNN? Does the RNN predicts all the features at once or loop / predict for every feature?\n\nThanks", "votes": 1, "replies": [ { "id": 1914875, "postDate": "2022-08-26T13:49:48.363Z", "content": "

Thank you for the question!

\n
\n

After generating the features, did you also predict the target for these new records, or what value for target did you use?

\n
\n

I just used their original targets. The purpose is to generate more recent features with respect to the original targets.

\n
\n

Does the RNN predicts all the features at once?

\n
\n

Yes. please also refer to my answer here

", "rawMarkdown": "Thank you for the question!\n\n> After generating the features, did you also predict the target for these new records, or what value for target did you use?\n\nI just used their original targets. The purpose is to generate more recent features with respect to the original targets.\n\n> Does the RNN predicts all the features at once?\n\nYes. please also refer to my answer [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347668#1914873)\n", "votes": 1 } ] }, { "id": 1914122, "postDate": "2022-08-25T18:47:15.057Z", "content": "

that is nice keep it up!

", "rawMarkdown": "that is nice keep it up!", "votes": 1 }, { "id": 1913131, "postDate": "2022-08-25T06:56:24.130Z", "content": "

Thanks for sharing, I've learned a lot! Congratulations!

", "rawMarkdown": "Thanks for sharing, I've learned a lot! Congratulations!", "votes": 1 }, { "id": 1913087, "postDate": "2022-08-25T06:13:14.513Z", "content": "

Congratulations @jiweiliu, that's a great insight.

", "rawMarkdown": "Congratulations @jiweiliu, that's a great insight.", "votes": 1 }, { "id": 1912988, "postDate": "2022-08-25T04:50:49.160Z", "content": "

Big Congratulations @jiweiliu .
\nI was trying something similar with KNN regressors, couldn't get a good CV LB to make it work. Feels good that this approach worked like magic.

", "rawMarkdown": "Big Congratulations @jiweiliu . \nI was trying something similar with KNN regressors, couldn't get a good CV LB to make it work. Feels good that this approach worked like magic. ", "votes": 1 }, { "id": 1913582, "postDate": "2022-08-25T11:37:32.033Z", "content": "

Congratulations Jiwei! Great solution. I love your use of RNN and test data.

\n

Did you try giving the customers with 13 statements more statements? Like predict them a 14th, 15th, 16th etc etc statement? I wonder if that would boost CV LB

", "rawMarkdown": "Congratulations Jiwei! Great solution. I love your use of RNN and test data. \n\nDid you try giving the customers with 13 statements more statements? Like predict them a 14th, 15th, 16th etc etc statement? I wonder if that would boost CV LB", "votes": 2, "replies": [ { "id": 1913613, "postDate": "2022-08-25T12:01:39.623Z", "content": "

Yes, I did. I only tried to predict the 14th statement of customers with 13 statements and it does help. Maybe predicting more is better. I stopped because the prediction of customers with 13 statements is already very accurate, above 0.82 AMEX metric. So I focused more on shorter sequences.

", "rawMarkdown": "Yes, I did. I only tried to predict the 14th statement of customers with 13 statements and it does help. Maybe predicting more is better. I stopped because the prediction of customers with 13 statements is already very accurate, above 0.82 AMEX metric. So I focused more on shorter sequences.", "votes": 4 }, { "id": 1913621, "postDate": "2022-08-25T12:07:14.073Z", "content": "

Another reason I stopped at 14th is I found that to improve overall AMEX metric, it is more important to make the poor better instead of making the rich richer.

\n
Fold 0 amex 0.7990 logloss 0.2144\nFold 0 L13 amex 0.8214 logloss 0.1928\nFold 0 Other amex 0.6724  logloss 0.3289\n
\n

Many times I found L13 amex is improved with the new features but Other amex with short sequences are worse or not improved, and the overall amex is not improved. If the metric is log loss, I would spend more time optimizing L13.

", "rawMarkdown": "Another reason I stopped at 14th is I found that to improve overall AMEX metric, it is more important to make the poor better instead of making the rich richer.\n```\nFold 0 amex 0.7990 logloss 0.2144\nFold 0 L13 amex 0.8214 logloss 0.1928\nFold 0 Other amex 0.6724 logloss 0.3289\n```\nMany times I found `L13 amex` is improved with the new features but `Other amex` with short sequences are worse or not improved, and the overall `amex` is not improved. If the metric is `log loss`, I would spend more time optimizing `L13`.", "votes": 6 }, { "id": 1913920, "postDate": "2022-08-25T15:22:06.790Z", "content": "

Did you try combining, for example predicted 14th - actual 13th (aka last)? Or last minus predicted last?

\n

It was on my list of things to try I didn't get to

", "rawMarkdown": "Did you try combining, for example predicted 14th - actual 13th (aka last)? Or last minus predicted last?\n\nIt was on my list of things to try I didn't get to", "votes": 1 }, { "id": 1914844, "postDate": "2022-08-26T13:24:31.057Z", "content": "

yes, I tried both. Keeping both predicted 14th and actual 13th (aka last) works best with xgb. Last minus predicted last doesn't add value.

", "rawMarkdown": "yes, I tried both. Keeping both predicted 14th and actual 13th (aka last) works best with xgb. Last minus predicted last doesn't add value.", "votes": 1 } ] }, { "id": 1913400, "postDate": "2022-08-25T10:05:19.020Z", "content": "

@jiweiliu Thanks for sharing and congrats on the solo gold!
\nOne thing I wanted to learn more is your construction of short sequences. According to S_2, the sequence alignment should be the second possibility. Am I thinking about something different here?

", "rawMarkdown": "@jiweiliu Thanks for sharing and congrats on the solo gold! \nOne thing I wanted to learn more is your construction of short sequences. According to `S_2`, the sequence alignment should be the second possibility. Am I thinking about something different here? ", "votes": 2, "replies": [ { "id": 1913603, "postDate": "2022-08-25T11:52:26.943Z", "content": "

Thank you for the comment! Yes, I think you are right and I wasn't clear in my post. If we align two sequences with the date S_2 like Feb/2018, it is the 2nd scenario as you suggested. But if we consider aligning two sequences relative to when the defaults actually happened, it might become the 1st scenario.

\n

For example, if M1 to M13 represent Feb/2018 to Mar/2019, it is aligned as the 2nd case.

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1 \nB  0  0  0  0  0  1  1  1  1  1  1  1  1  1 \n
\n


\nso when do the defaults happen? My assumption is default of B happened sometime after default of A, something like this:

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1         x \nB  0  0  0  0  0  1  1  1  1  1  1  1  1  1                 x \n
\n


\nwhere x indicates default. The point is sequence B is harder to predict which indicates its more recent profiles are missing. So if we align the default dates x, it looks like scenario 1 in my original post:

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1           x\nB  1  1  1  1  1  1  1  1  1  0  0  0  0  0           x \n
\n

Of course, this is all my hypothesis and by no means a precise alignment since the true default dates are unknown. However, the idea might be roughly correct and useful. Eventually, it all comes down to what the new feature should mean. P_2_M_13 could mean the value of P_2 in Mar/2019 or it could mean the value of P_2 in the most recent month relative to when the default happens. I think the downstream XGB learns better with the latter.

", "rawMarkdown": "Thank you for the comment! Yes, I think you are right and I wasn't clear in my post. If we align two sequences with the date `S_2` like Feb/2018, it is the 2nd scenario as you suggested. But if we consider aligning two sequences relative to when the defaults actually happened, it might become the 1st scenario. \n\nFor example, if `M1` to `M13` represent `Feb/2018` to `Mar/2019`, it is aligned as the 2nd case. \n ```\n M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA 1 1 1 1 1 1 1 1 1 1 1 1 1 1 \nB 0 0 0 0 0 1 1 1 1 1 1 1 1 1 \n``` \nso when do the defaults happen? My assumption is default of `B` happened sometime after default of `A`, something like this:\n ```\n M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18\nA 1 1 1 1 1 1 1 1 1 1 1 1 1 1 x \nB 0 0 0 0 0 1 1 1 1 1 1 1 1 1 x \n``` \nwhere `x` indicates default. The point is sequence B is harder to predict which indicates its more recent profiles are missing. So if we align the default dates `x`, it looks like scenario 1 in my original post:\n```\n M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18\nA 1 1 1 1 1 1 1 1 1 1 1 1 1 1 x\nB 1 1 1 1 1 1 1 1 1 0 0 0 0 0 x \n``` \n\nOf course, this is all my hypothesis and by no means a precise alignment since the true default dates are unknown. However, the idea might be roughly correct and useful. Eventually, it all comes down to what the new feature should mean. `P_2_M_13` could mean the value of `P_2` in `Mar/2019` or it could mean the value of `P_2` in the most recent month relative to when the default happens. I think the downstream XGB learns better with the latter.", "votes": 4 } ] }, { "id": 1928754, "postDate": "2022-09-06T15:40:38.713Z", "content": "

Congrats and thanks for sharing such a great solution!

\n

I read that you trained only one RNN with length 8 and only predicted 1 step ahead each time, but I didn't get which statements covers that length 8 sequence (you mentioned truncating/padding and right alignment). Following your example:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
customer_IDM1M2M3M4M5M6M7M8M9M10M11M12M13
A1111111111111
B1111111100000
C1111000000000
\n

The data used for training would be the following? (X's are features, Y the target):

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
customer_IDM1M2M3M4M5M6M7M8M9M10M11M12M13
A----X1X2X3X4X5X6X7X8Y
BX2X3X4X5X6X7X8Y-----
CX6X7X8Y---------
\n

And in the end:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
customer_IDS1S2S3S4S5S6S7S8Target
AX1X2X3X4X5X6X7X8Y
BPadX2X3X4X5X6X7X8Y
CPadPadPadPadPadX6X7X8Y
\n

Is that correct?

", "rawMarkdown": "Congrats and thanks for sharing such a great solution! \n\nI read that you trained only one RNN with length 8 and only predicted 1 step ahead each time, but I didn't get which statements covers that length 8 sequence (you mentioned truncating/padding and right alignment). Following your example:\n\n\n| customer_ID | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 |\n| --- | --- |\n| A | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |\n| B | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |\n| C | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |\n\nThe data used for training would be the following? (X's are features, Y the target):\n\n| customer_ID | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | \n| --- | --- |\n| A | - | - | - | - | X1 | X2 | X3 | X4 | X5 | X6 | X7 | X8 | Y |\n| B | X2 | X3 | X4 | X5 | X6 | X7 | X8 | Y | - | - | - | - | - |\n| C | X6 | X7 | X8 | Y | - | - | - | - | - | - | - | - | - |\n\nAnd in the end:\n\n| customer_ID | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | Target\n| --- | --- |\n| A | X1 | X2 | X3 | X4 | X5 | X6 | X7 | X8 | Y |\n| B | Pad | X2 | X3 | X4 | X5 | X6 | X7 | X8 | Y | \n| C | Pad | Pad | Pad | Pad | Pad | X6 | X7 | X8 | Y |\n\nIs that correct?\n" }, { "id": 1913032, "postDate": "2022-08-25T05:23:24.300Z", "content": "

Thank you very much for the well written, detailed explanation. I have some questions, but if you don't have time or desire to answer, I will not take offense. This is already very enlightening so feel free to ignore.

\n

RNN:

\n

1) Is the below impression nearly correct?
\nMy impression from we have the prediction for the next month, we can use it as part of the input and predict again and so on so forth is that the model predicts only 1 month ahead. Do you train different models for different number of previous months? If not, it would seem to me that the model would need features for up to 12 previous months, and instances with fewer than 12 would have NaN for those features. Also, you could have a large amount of training data, since an instance with no missing months provides 12 points of training data for this RNN model; the second month (with 1 previous month of know data), the third (with 2 previous months …) and so on.

\n

2) Did you provide a flag to the XGB models that the feature sequence of a row had filled data?

\n

CV

\n

3) Do you do a simple k-fold cv (early stop on the validation set and count the score on the same set for your CV score), or did you use nested cv, early stopping on the inner valid set. Or maybe not early stopping, but retraining with an average of best number of iterations. Is there ever a time in which you don't use any early stopping with XGBoost? Sorry for the detail seeking question about CV. I am just really curious as to best practices around cv are. I always figured nested cv was the only \"true\" CV, but I don't find people differentiating which scheme they are using for some reason.

", "rawMarkdown": "Thank you very much for the well written, detailed explanation. I have some questions, but if you don't have time or desire to answer, I will not take offense. This is already very enlightening so feel free to ignore.\n### RNN: \n1) Is the below impression nearly correct? \nMy impression from `we have the prediction for the next month, we can use it as part of the input and predict again and so on so forth` is that the model predicts only 1 month ahead. Do you train different models for different number of previous months? If not, it would seem to me that the model would need features for up to 12 previous months, and instances with fewer than 12 would have NaN for those features. Also, you could have a large amount of training data, since an instance with no missing months provides 12 points of training data for this RNN model; the second month (with 1 previous month of know data), the third (with 2 previous months ...) and so on. \n\n2) Did you provide a flag to the XGB models that the feature sequence of a row had filled data?\n\n### CV\n3) Do you do a simple k-fold cv (early stop on the validation set and count the score on the same set for your CV score), or did you use nested cv, early stopping on the inner valid set. Or maybe not early stopping, but retraining with an average of best number of iterations. Is there ever a time in which you don't use any early stopping with XGBoost? Sorry for the detail seeking question about CV. I am just really curious as to best practices around cv are. I always figured nested cv was the only \"true\" CV, but I don't find people differentiating which scheme they are using for some reason.", "replies": [ { "id": 1913688, "postDate": "2022-08-25T12:54:30.117Z", "content": "

Hi Chris, these are great questions! I'll come back to this and give more details. The short answers are:

\n", "rawMarkdown": "Hi Chris, these are great questions! I'll come back to this and give more details. The short answers are:\n- I only trained one RNN model of length `8` to generate future profiles. The input data are truncated or padded to make length 8 and they are right aligned.\n- No, I don't give a flag to xgb to indicate the features are generated or real. But this is an interesting idea. might work.\n- Yes, I used simple k-fold. but I found very late in the competition that a stratified k-fold on sequence-length might be better.\n- Yes, I used xgb early stopping but very large `early_stopping_rounds`.", "votes": 4 }, { "id": 1913967, "postDate": "2022-08-25T15:57:32.277Z", "content": "

Thank you very much. The answers are very informative.

\n

I forgot to ask, but does the RNN predict all features at once? So the input is 8x188 output 1x188

", "rawMarkdown": "Thank you very much. The answers are very informative. \n\nI forgot to ask, but does the RNN predict all features at once? So the input is 8x188 output 1x188" }, { "id": 1914850, "postDate": "2022-08-26T13:28:06.733Z", "content": "

Yes the RNN predicts all features at once. Both input and output are 8*188 but the output is one time step ahead.

", "rawMarkdown": "Yes the RNN predicts all features at once. Both input and output are 8*188 but the output is one time step ahead.", "votes": 2 }, { "id": 1914873, "postDate": "2022-08-26T13:49:19.650Z", "content": "

To be precise, RNN maps an input [B,S,N] tensor to an output [B,S,N] tensor, where B is the batch size, S is the sequence length and N is the number of all numerical features. The output tensor is one step ahead of the input tensor (output is for the future). In inference, we only take the last step of the output tensor and attach it to the input tensor and predict again.

", "rawMarkdown": "To be precise, RNN maps an input `[B,S,N]` tensor to an output `[B,S,N]` tensor, where `B` is the batch size, `S` is the sequence length and `N` is the number of all numerical features. The output tensor is one step ahead of the input tensor (output is for the future). In inference, we only take the last step of the output tensor and attach it to the input tensor and predict again. ", "votes": 2 } ] }, { "id": 1913022, "postDate": "2022-08-25T05:18:21.877Z", "content": "

Beatiful and original idea. Well done!

", "rawMarkdown": "Beatiful and original idea. Well done!" }, { "id": 1913000, "postDate": "2022-08-25T04:59:18.513Z", "content": "

Nice Job! Congrats!

", "rawMarkdown": "Nice Job! Congrats!" }, { "id": 1912974, "postDate": "2022-08-25T04:43:28.423Z", "content": "

Amazing way to analyze data and generate features!
\nThanks a lot!

", "rawMarkdown": "Amazing way to analyze data and generate features!\nThanks a lot!" }, { "id": 1913082, "postDate": "2022-08-25T06:06:13.753Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 1913611, "postDate": "2022-08-25T11:56:56.170Z", "content": "

Thank you for the question. The short answer is I ignored the gaps in the sequence. I think what's important is if the more recent profiles of a customer are available, not what's missing in the past. Please also refer to my response to Tonghui Li here.

", "rawMarkdown": "Thank you for the question. The short answer is I ignored the gaps in the sequence. I think what's important is if the more recent profiles of a customer are available, not what's missing in the past. Please also refer to my response to Tonghui Li [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/347668#1913603).", "votes": 1 } ] }, { "id": 1913211, "postDate": "2022-08-25T08:19:43.210Z", "content": "

Thanks for the explanation and congrats

", "rawMarkdown": "Thanks for the explanation and congrats", "votes": 1 }, { "id": 1913166, "postDate": "2022-08-25T07:35:32.273Z", "content": "

Thanks for sharing

", "rawMarkdown": "Thanks for sharing~~~~~", "votes": 1 }, { "id": 1912957, "postDate": "2022-08-25T04:19:43.447Z", "content": "

Great job! Thanks for sharing

", "rawMarkdown": "Great job! Thanks for sharing", "votes": 1 } ], "comments": [ { "id": 1915127, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-26T17:24:46.920000", "content": "

I forgot to thank @pyagoubi Art Vandelay for his great kernel: https://www.kaggle.com/code/pyagoubi/amex-eda-evolvement-of-numeric-features-over-time
\nhere is a screenshot of his drawing:
\n\"\"
\nThe pattern shows a clear and predictable trend, which motivates me to go down this path.

", "votes": 3, "replies": [ { "id": 1916368, "author_name": "Art Vandelay", "author_url": "", "post_date": "2022-08-27T19:54:50.567000", "content": "

wow thanks!

", "votes": 0, "replies": [] } ] }, { "id": 1914643, "author_name": "Jonathan Mallia", "author_url": "", "post_date": "2022-08-26T09:04:53.403000", "content": "

Really great idea to generate features for the missing periods!

\n

Therefore just to understand, every cid has 13 records in the end, right?
\nAfter generating the features, did you also predict the target for these new records, or what value for target did you use?

\n

Can you also give more info on the one-dimension auto-regressive RNN? Does the RNN predicts all the features at once or loop / predict for every feature?

\n

Thanks

", "votes": 1, "replies": [ { "id": 1914875, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-26T13:49:48.363000", "content": "

Thank you for the question!

\n
\n

After generating the features, did you also predict the target for these new records, or what value for target did you use?

\n
\n

I just used their original targets. The purpose is to generate more recent features with respect to the original targets.

\n
\n

Does the RNN predicts all the features at once?

\n
\n

Yes. please also refer to my answer here

", "votes": 1, "replies": [] } ] }, { "id": 1914122, "author_name": "rajashekar vt", "author_url": "", "post_date": "2022-08-25T18:47:15.057000", "content": "

that is nice keep it up!

", "votes": 1, "replies": [] }, { "id": 1913131, "author_name": "XiaFire", "author_url": "", "post_date": "2022-08-25T06:56:24.130000", "content": "

Thanks for sharing, I've learned a lot! Congratulations!

", "votes": 1, "replies": [] }, { "id": 1913087, "author_name": "Priyanshu Chaudhary", "author_url": "", "post_date": "2022-08-25T06:13:14.513000", "content": "

Congratulations @jiweiliu, that's a great insight.

", "votes": 1, "replies": [] }, { "id": 1912988, "author_name": "tarick.morty", "author_url": "", "post_date": "2022-08-25T04:50:49.160000", "content": "

Big Congratulations @jiweiliu .
\nI was trying something similar with KNN regressors, couldn't get a good CV LB to make it work. Feels good that this approach worked like magic.

", "votes": 1, "replies": [] }, { "id": 1913582, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-08-25T11:37:32.033000", "content": "

Congratulations Jiwei! Great solution. I love your use of RNN and test data.

\n

Did you try giving the customers with 13 statements more statements? Like predict them a 14th, 15th, 16th etc etc statement? I wonder if that would boost CV LB

", "votes": 2, "replies": [ { "id": 1913613, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-25T12:01:39.623000", "content": "

Yes, I did. I only tried to predict the 14th statement of customers with 13 statements and it does help. Maybe predicting more is better. I stopped because the prediction of customers with 13 statements is already very accurate, above 0.82 AMEX metric. So I focused more on shorter sequences.

", "votes": 4, "replies": [] }, { "id": 1913621, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-25T12:07:14.073000", "content": "

Another reason I stopped at 14th is I found that to improve overall AMEX metric, it is more important to make the poor better instead of making the rich richer.

\n
Fold 0 amex 0.7990 logloss 0.2144\nFold 0 L13 amex 0.8214 logloss 0.1928\nFold 0 Other amex 0.6724  logloss 0.3289\n
\n

Many times I found L13 amex is improved with the new features but Other amex with short sequences are worse or not improved, and the overall amex is not improved. If the metric is log loss, I would spend more time optimizing L13.

", "votes": 6, "replies": [] }, { "id": 1913920, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-08-25T15:22:06.790000", "content": "

Did you try combining, for example predicted 14th - actual 13th (aka last)? Or last minus predicted last?

\n

It was on my list of things to try I didn't get to

", "votes": 1, "replies": [] }, { "id": 1914844, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-26T13:24:31.057000", "content": "

yes, I tried both. Keeping both predicted 14th and actual 13th (aka last) works best with xgb. Last minus predicted last doesn't add value.

", "votes": 1, "replies": [] } ] }, { "id": 1913400, "author_name": "Tonghui Li", "author_url": "", "post_date": "2022-08-25T10:05:19.020000", "content": "

@jiweiliu Thanks for sharing and congrats on the solo gold!
\nOne thing I wanted to learn more is your construction of short sequences. According to S_2, the sequence alignment should be the second possibility. Am I thinking about something different here?

", "votes": 2, "replies": [ { "id": 1913603, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-25T11:52:26.943000", "content": "

Thank you for the comment! Yes, I think you are right and I wasn't clear in my post. If we align two sequences with the date S_2 like Feb/2018, it is the 2nd scenario as you suggested. But if we consider aligning two sequences relative to when the defaults actually happened, it might become the 1st scenario.

\n

For example, if M1 to M13 represent Feb/2018 to Mar/2019, it is aligned as the 2nd case.

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1 \nB  0  0  0  0  0  1  1  1  1  1  1  1  1  1 \n
\n


\nso when do the defaults happen? My assumption is default of B happened sometime after default of A, something like this:

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1         x \nB  0  0  0  0  0  1  1  1  1  1  1  1  1  1                 x \n
\n


\nwhere x indicates default. The point is sequence B is harder to predict which indicates its more recent profiles are missing. So if we align the default dates x, it looks like scenario 1 in my original post:

\n
   M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13 M14 M15 M16 M17 M18\nA  1  1  1  1  1  1  1  1  1  1  1  1  1  1           x\nB  1  1  1  1  1  1  1  1  1  0  0  0  0  0           x \n
\n

Of course, this is all my hypothesis and by no means a precise alignment since the true default dates are unknown. However, the idea might be roughly correct and useful. Eventually, it all comes down to what the new feature should mean. P_2_M_13 could mean the value of P_2 in Mar/2019 or it could mean the value of P_2 in the most recent month relative to when the default happens. I think the downstream XGB learns better with the latter.

", "votes": 4, "replies": [] } ] }, { "id": 1928754, "author_name": "delai50", "author_url": "", "post_date": "2022-09-06T15:40:38.713000", "content": "

Congrats and thanks for sharing such a great solution!

\n

I read that you trained only one RNN with length 8 and only predicted 1 step ahead each time, but I didn't get which statements covers that length 8 sequence (you mentioned truncating/padding and right alignment). Following your example:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
customer_IDM1M2M3M4M5M6M7M8M9M10M11M12M13
A1111111111111
B1111111100000
C1111000000000
\n

The data used for training would be the following? (X's are features, Y the target):

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
customer_IDM1M2M3M4M5M6M7M8M9M10M11M12M13
A----X1X2X3X4X5X6X7X8Y
BX2X3X4X5X6X7X8Y-----
CX6X7X8Y---------
\n

And in the end:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
customer_IDS1S2S3S4S5S6S7S8Target
AX1X2X3X4X5X6X7X8Y
BPadX2X3X4X5X6X7X8Y
CPadPadPadPadPadX6X7X8Y
\n

Is that correct?

", "votes": 0, "replies": [] }, { "id": 1913032, "author_name": "Chris Miles", "author_url": "", "post_date": "2022-08-25T05:23:24.300000", "content": "

Thank you very much for the well written, detailed explanation. I have some questions, but if you don't have time or desire to answer, I will not take offense. This is already very enlightening so feel free to ignore.

\n

RNN:

\n

1) Is the below impression nearly correct?
\nMy impression from we have the prediction for the next month, we can use it as part of the input and predict again and so on so forth is that the model predicts only 1 month ahead. Do you train different models for different number of previous months? If not, it would seem to me that the model would need features for up to 12 previous months, and instances with fewer than 12 would have NaN for those features. Also, you could have a large amount of training data, since an instance with no missing months provides 12 points of training data for this RNN model; the second month (with 1 previous month of know data), the third (with 2 previous months …) and so on.

\n

2) Did you provide a flag to the XGB models that the feature sequence of a row had filled data?

\n

CV

\n

3) Do you do a simple k-fold cv (early stop on the validation set and count the score on the same set for your CV score), or did you use nested cv, early stopping on the inner valid set. Or maybe not early stopping, but retraining with an average of best number of iterations. Is there ever a time in which you don't use any early stopping with XGBoost? Sorry for the detail seeking question about CV. I am just really curious as to best practices around cv are. I always figured nested cv was the only \"true\" CV, but I don't find people differentiating which scheme they are using for some reason.

", "votes": 0, "replies": [ { "id": 1913688, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-25T12:54:30.117000", "content": "

Hi Chris, these are great questions! I'll come back to this and give more details. The short answers are:

\n", "votes": 4, "replies": [] }, { "id": 1913967, "author_name": "Chris Miles", "author_url": "", "post_date": "2022-08-25T15:57:32.277000", "content": "

Thank you very much. The answers are very informative.

\n

I forgot to ask, but does the RNN predict all features at once? So the input is 8x188 output 1x188

", "votes": 0, "replies": [] }, { "id": 1914850, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-26T13:28:06.733000", "content": "

Yes the RNN predicts all features at once. Both input and output are 8*188 but the output is one time step ahead.

", "votes": 2, "replies": [] }, { "id": 1914873, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-26T13:49:19.650000", "content": "

To be precise, RNN maps an input [B,S,N] tensor to an output [B,S,N] tensor, where B is the batch size, S is the sequence length and N is the number of all numerical features. The output tensor is one step ahead of the input tensor (output is for the future). In inference, we only take the last step of the output tensor and attach it to the input tensor and predict again.

", "votes": 2, "replies": [] } ] }, { "id": 1913022, "author_name": "Elias", "author_url": "", "post_date": "2022-08-25T05:18:21.877000", "content": "

Beatiful and original idea. Well done!

", "votes": 0, "replies": [] }, { "id": 1913000, "author_name": "iliiiiiili", "author_url": "", "post_date": "2022-08-25T04:59:18.513000", "content": "

Nice Job! Congrats!

", "votes": 0, "replies": [] }, { "id": 1912974, "author_name": "Oleg Khudyakov", "author_url": "", "post_date": "2022-08-25T04:43:28.423000", "content": "

Amazing way to analyze data and generate features!
\nThanks a lot!

", "votes": 0, "replies": [] }, { "id": 1913082, "author_name": "", "author_url": "", "post_date": "2022-08-25T06:06:13.753000", "content": "", "votes": 2, "replies": [ { "id": 1913611, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2022-08-25T11:56:56.170000", "content": "

Thank you for the question. The short answer is I ignored the gaps in the sequence. I think what's important is if the more recent profiles of a customer are available, not what's missing in the past. Please also refer to my response to Tonghui Li here.

", "votes": 1, "replies": [] } ] }, { "id": 1913211, "author_name": "Santiago Mota", "author_url": "", "post_date": "2022-08-25T08:19:43.210000", "content": "

Thanks for the explanation and congrats

", "votes": 1, "replies": [] }, { "id": 1913166, "author_name": "SgangX", "author_url": "", "post_date": "2022-08-25T07:35:32.273000", "content": "

Thanks for sharing

", "votes": 1, "replies": [] }, { "id": 1912957, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-08-25T04:19:43.447000", "content": "

Great job! Thanks for sharing

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1912952": "What a competition! I really enjoyed it and only hope I could have found more time. First of all, I would like to thank Raddar @raddar, Martin @ragnar123, and many others who generously shared codes and datasets! The public solutions are of amazing quality. And it also determines my game plan: **create something original and blend it with the best public solution.** My solution is based on RAPIDS cudf for dataframe processing, XGB for training, and pytorch lightning for feature extraction.\n\nThere are two vital observations of this dataset:\n- the big test data which is from the future is available.\n- short sequences (<13) are culprits of bad performance.\n\nLet's start with the latter observation:**the sequence length of each customer profiles** plays a critical role in the model performance:\n```\nimport cudf\npath = '/raid/amex'\ntrain = cudf.read_parquet(f'{path}/train.parquet',columns=['customer_ID'])\ntrainl = cudf.read_csv(f'{path}/train_labels.csv')\ntrain = train.merge(trainl,on='customer_ID',how='left')\ntrain['seq_len'] = train.groupby('customer_ID')['target'].transform('count')\ntrain = train.drop_duplicates('customer_ID',keep='last')\ntrain.groupby('seq_len').agg({'target':['mean','count']}).sort_index(ascending=False)\n```\noutput:\n```\n target\n mean\tcount\nseq_len\t\t\n13\t0.231788\t386034\n12\t0.389344\t10623\n11\t0.446737\t5961\n10\t0.462282\t6721\n9\t 0.450164\t 6411\n8\t 0.447300\t 6110\n7\t 0.418430\t 5198\n6\t 0.387670\t 5515\n5\t 0.392635\t 4671\n4\t 0.416221\t 4673\n3\t 0.358602\t 5778\n2\t 0.318465\t 6098\n1\t 0.335742\t 5120\n```\nIt is obvious that **sequence length 13** is the most common but also with a significantly lower mean default rate. At first glance, I thought it meant shorter sequences are easier to predict since they have more positive samples. But I'm quickly proven wrong when checking my cross-validation results:\n```\nFold 0 amex 0.7990 logloss 0.2144\nFold 0 L13 amex 0.8214 logloss 0.1928\nFold 0 Other amex 0.6724 logloss 0.3289\n```\nThe 1st line is the overall score. The 2nd line is the score of sequences of length 13 and the 3rd line is the score of all the rest sequences. Apparently, shorter sequences have a much worse score than the full sequences of length 13. This is also an implication of how the short sequences are truncated: the more recent profiles are deleted, which could explain the big degradation of the score because more recent profiles have more predicting power in general. For example, let's say for 13 consecutive months (M1~M13) and sequence A is of length 13 and sequence B is of length 8:\n```\n M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA 1 1 1 1 1 1 1 1 1 1 1 1 1 1 \nB 1 1 1 1 1 1 1 1 1 0 0 0 0 0 \n``` \nwhere `1` means features exist and `0` means features missing. Of course, there is another possibility:\n ```\n M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12 M13\nA 1 1 1 1 1 1 1 1 1 1 1 1 1 1 \nB 0 0 0 0 0 1 1 1 1 1 1 1 1 1 \n``` \nWe can actually find out which one is more plausible by `unstacking` dataframes in the above two ways and run xgboost with them, respectively. As expected, the former has a better CV score which indicates it is likely how the truncation of short sequences is done.\n\nIf we could somehow predict the missing profiles of sequence `B`, the life of the downstream XGB models would be made much easier. An intuitive choice is to generate the missing profiles using a one-dimension auto-regressive RNN. Bascially we want to predict the features of the next month based on the feature values of the current month and all previous months. And when we have the prediction for the next month, we can use it as part of the input and predict again and so on so forth. This is also where the availability of the big test data really shines. Since we are predicting features, not `target`, we can train our models using both `train` and `test` data. The RNN structure is very simple: just one GRU layer and some FC layers. The RNN performance is pretty decent. In terms of RMSE of all 178 numerical features, the GRU achieves validation `RMSE 0.019`. For simplicity, all features are log-transformed and `fillna(0)`. You might wonder how good is `RMSE 0.019`. We can simply compare it with the naive baseline: just repeat the last available month. For example, if I'm asked to predict features of M2, the naive baseline is just output features of M1. The RMSE of this naive baseline is `0.03` so our RNN actually learns something and could be useful.\n\nThe rest would be straightforward, after predicting missing months, now every sequence is of length 13 so I just unstack the dataframe to increase the number of features 13x. For example, instead of having one feature `P_2` of the last month, now we have 13 features `P_2_M_1` to `P_2_M_13`. These are the most useful features I created. For downstream classifiers I only use XGB so that it is *not similar* to [the great LGB DART notebook](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977). By varying RNN hyperparameters, XGB hyperparameters and different combination of features, I end up with 7 XGB models, whose ensemble is 0.7993 CV and 0.799 public LB. Averaging it with the best public solution and with extraordinary luck, my final submission ended up in the gold zone.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F100236%2Facf9b55d595c8c030770291e3dd69844%2FWechatIMG5474.png?generation=1661401124767084&alt=media)\n\nThe final thought is my best auto-regressive features are generated 1 hour before the deadline. I'm very happy it worked!", "1915127": "I forgot to thank @pyagoubi Art Vandelay for his great kernel: https://www.kaggle.com/code/pyagoubi/amex-eda-evolvement-of-numeric-features-over-time\nhere is a screenshot of his drawing:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F100236%2F3364ddfbcdfea257e2c047971aaaf0c2%2FScreen%20Shot%202022-08-26%20at%201.24.11%20PM.png?generation=1661534676162068&alt=media)\nThe pattern shows a clear and predictable trend, which motivates me to go down this path. ", "1914643": "Really great idea to generate features for the missing periods!\n\nTherefore just to understand, every cid has 13 records in the end, right?\nAfter generating the features, did you also predict the target for these new records, or what value for target did you use?\n\nCan you also give more info on the one-dimension auto-regressive RNN? Does the RNN predicts all the features at once or loop / predict for every feature?\n\nThanks", "1914122": "that is nice keep it up!", "1913131": "Thanks for sharing, I've learned a lot! Congratulations!", "1913087": "Congratulations @jiweiliu, that's a great insight.", "1912988": "Big Congratulations @jiweiliu . \nI was trying something similar with KNN regressors, couldn't get a good CV LB to make it work. Feels good that this approach worked like magic. ", "1913582": "Congratulations Jiwei! Great solution. I love your use of RNN and test data. \n\nDid you try giving the customers with 13 statements more statements? Like predict them a 14th, 15th, 16th etc etc statement? I wonder if that would boost CV LB", "1913400": "@jiweiliu Thanks for sharing and congrats on the solo gold! \nOne thing I wanted to learn more is your construction of short sequences. According to `S_2`, the sequence alignment should be the second possibility. Am I thinking about something different here? ", "1928754": "Congrats and thanks for sharing such a great solution! \n\nI read that you trained only one RNN with length 8 and only predicted 1 step ahead each time, but I didn't get which statements covers that length 8 sequence (you mentioned truncating/padding and right alignment). Following your example:\n\n\n| customer_ID | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 |\n| --- | --- |\n| A | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |\n| B | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |\n| C | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |\n\nThe data used for training would be the following? (X's are features, Y the target):\n\n| customer_ID | M1 | M2 | M3 | M4 | M5 | M6 | M7 | M8 | M9 | M10 | M11 | M12 | M13 | \n| --- | --- |\n| A | - | - | - | - | X1 | X2 | X3 | X4 | X5 | X6 | X7 | X8 | Y |\n| B | X2 | X3 | X4 | X5 | X6 | X7 | X8 | Y | - | - | - | - | - |\n| C | X6 | X7 | X8 | Y | - | - | - | - | - | - | - | - | - |\n\nAnd in the end:\n\n| customer_ID | S1 | S2 | S3 | S4 | S5 | S6 | S7 | S8 | Target\n| --- | --- |\n| A | X1 | X2 | X3 | X4 | X5 | X6 | X7 | X8 | Y |\n| B | Pad | X2 | X3 | X4 | X5 | X6 | X7 | X8 | Y | \n| C | Pad | Pad | Pad | Pad | Pad | X6 | X7 | X8 | Y |\n\nIs that correct?\n", "1913032": "Thank you very much for the well written, detailed explanation. I have some questions, but if you don't have time or desire to answer, I will not take offense. This is already very enlightening so feel free to ignore.\n### RNN: \n1) Is the below impression nearly correct? \nMy impression from `we have the prediction for the next month, we can use it as part of the input and predict again and so on so forth` is that the model predicts only 1 month ahead. Do you train different models for different number of previous months? If not, it would seem to me that the model would need features for up to 12 previous months, and instances with fewer than 12 would have NaN for those features. Also, you could have a large amount of training data, since an instance with no missing months provides 12 points of training data for this RNN model; the second month (with 1 previous month of know data), the third (with 2 previous months ...) and so on. \n\n2) Did you provide a flag to the XGB models that the feature sequence of a row had filled data?\n\n### CV\n3) Do you do a simple k-fold cv (early stop on the validation set and count the score on the same set for your CV score), or did you use nested cv, early stopping on the inner valid set. Or maybe not early stopping, but retraining with an average of best number of iterations. Is there ever a time in which you don't use any early stopping with XGBoost? Sorry for the detail seeking question about CV. I am just really curious as to best practices around cv are. I always figured nested cv was the only \"true\" CV, but I don't find people differentiating which scheme they are using for some reason.", "1913022": "Beatiful and original idea. Well done!", "1913000": "Nice Job! Congrats!", "1912974": "Amazing way to analyze data and generate features!\nThanks a lot!", "1913082": "", "1913211": "Thanks for the explanation and congrats", "1913166": "Thanks for sharing~~~~~", "1912957": "Great job! Thanks for sharing" } }