{ "id": 327828, "title": "Kaggle Dataset for Transformers and RNNs", "url": "/competitions/amex-default-prediction/discussion/327828", "author_name": "Chris Deotte", "post_date": "2022-05-29T13:29:49.977000", "votes": 159, "comment_count": 28, "views": 0, "content": "

Competition Data CSV

\n

This competition provides data about credit card customers in the form of CSV files. We cannot use these files without modification to train Transformers and RNNs because each customer has a different number of credit card statements.

\n

When inputting time series data into a Transformer and RNN, each customer needs the same sequence length (of 13 in this comp).

\n

\"\"

\n

Competition Data NumPy Array

\n

I have converted the competition data into NumPy array files where each customer has sequence length 13 and posted a Kaggle dataset here (This data is created in my notebook here)

\n

For each customer with less than 13 credit card statements, I have padded their data with pad = -1. See example below where the customer has been padded with four -1's:

\n

\"\"

\n

Kaggle Dataset Details

\n

My Kaggle dataset is here. The train data has been split into 10 NumPy arrays named data_1.npy thru data_10.npy. Each array has dimension (45891, 13, 188) which is customer x statement x feature.

\n

The associated targets are contained in the files targets_1.pqt thru targets_10.pqt. These are parquet files with columns customer_ID and target and have dimension (45891, 2). An example training with these files is here. Note that the customer_ID is a int64 instead of the provided string512. It was compressed using the code shown here.

\n

The test data is split in 20 NumPy arrays. Each array has 46231 customers. Important Note: If you concatenate all these files, the row order of the 924621 test customers is not the same as the file sample_submission.csv. The order is contained in my submission.csv here. So infer the 20 NumPy arrays in their order and then overwrite my prediction column with submission['prediction'] = preds.

\n

Transformers and RNNs

\n

It will be exciting to see whether GBT (gradient boosted trees) or NN will achieve the best accuracy in this competition. I posted a RNN notebook starter here and discussion here. If you want to build a Transformer, you can use my Transformer notebook here from Ventilator competition and input Amex competition data. Additionally there are many great RNN notebooks in Ventilator competition for inspiration.

\n

UPDATE: Transformer Starter - LB 0.790

\n

I created and published a Transformer starter notebook here today that uses my Kaggle dataset. It achieves 5-Fold CV 0.787 and LB 0.789. Enjoy!

\n

Have Fun!

", "messages": [ { "id": 1804818, "postDate": "2022-05-29T13:29:49.977Z", "content": "

Competition Data CSV

\n

This competition provides data about credit card customers in the form of CSV files. We cannot use these files without modification to train Transformers and RNNs because each customer has a different number of credit card statements.

\n

When inputting time series data into a Transformer and RNN, each customer needs the same sequence length (of 13 in this comp).

\n

\"\"

\n

Competition Data NumPy Array

\n

I have converted the competition data into NumPy array files where each customer has sequence length 13 and posted a Kaggle dataset here (This data is created in my notebook here)

\n

For each customer with less than 13 credit card statements, I have padded their data with pad = -1. See example below where the customer has been padded with four -1's:

\n

\"\"

\n

Kaggle Dataset Details

\n

My Kaggle dataset is here. The train data has been split into 10 NumPy arrays named data_1.npy thru data_10.npy. Each array has dimension (45891, 13, 188) which is customer x statement x feature.

\n

The associated targets are contained in the files targets_1.pqt thru targets_10.pqt. These are parquet files with columns customer_ID and target and have dimension (45891, 2). An example training with these files is here. Note that the customer_ID is a int64 instead of the provided string512. It was compressed using the code shown here.

\n

The test data is split in 20 NumPy arrays. Each array has 46231 customers. Important Note: If you concatenate all these files, the row order of the 924621 test customers is not the same as the file sample_submission.csv. The order is contained in my submission.csv here. So infer the 20 NumPy arrays in their order and then overwrite my prediction column with submission['prediction'] = preds.

\n

Transformers and RNNs

\n

It will be exciting to see whether GBT (gradient boosted trees) or NN will achieve the best accuracy in this competition. I posted a RNN notebook starter here and discussion here. If you want to build a Transformer, you can use my Transformer notebook here from Ventilator competition and input Amex competition data. Additionally there are many great RNN notebooks in Ventilator competition for inspiration.

\n

UPDATE: Transformer Starter - LB 0.790

\n

I created and published a Transformer starter notebook here today that uses my Kaggle dataset. It achieves 5-Fold CV 0.787 and LB 0.789. Enjoy!

\n

Have Fun!

", "rawMarkdown": "# Competition Data CSV \nThis competition provides data about credit card customers in the form of `CSV` files. We cannot use these files without modification to train Transformers and RNNs because each customer has a different number of credit card statements. \n\nWhen inputting time series data into a Transformer and RNN, each customer needs the same sequence length (of 13 in this comp).\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/May-2022/trans.png)\n\n# Competition Data NumPy Array\nI have converted the competition data into `NumPy array` files where each customer has sequence length 13 and posted a Kaggle dataset [here][2] (This data is created in my notebook [here][1])\n\nFor each customer with less than 13 credit card statements, I have padded their data with `pad = -1`. See example below where the customer has been padded with four -1's:\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/May-2022/data_block.png)\n\n# Kaggle Dataset Details\nMy Kaggle dataset is [here][2]. The train data has been split into 10 NumPy arrays named `data_1.npy` thru `data_10.npy`. Each array has dimension `(45891, 13, 188)` which is `customer x statement x feature`. \n\nThe associated targets are contained in the files `targets_1.pqt` thru `targets_10.pqt`. These are parquet files with columns `customer_ID` and `target` and have dimension `(45891, 2)`. An example training with these files is [here][1]. Note that the `customer_ID` is a `int64` instead of the provided `string512`. It was compressed using the code shown [here][3].\n\nThe test data is split in 20 NumPy arrays. Each array has `46231` customers. **Important Note**: If you concatenate all these files, the row order of the `924621` test customers is **not the same** as the file `sample_submission.csv`. The order is contained in my `submission.csv` [here][4]. So infer the 20 NumPy arrays in their order and then overwrite my prediction column with `submission['prediction'] = preds`.\n\n# Transformers and RNNs\nIt will be exciting to see whether GBT (gradient boosted trees) or NN will achieve the best accuracy in this competition. I posted a RNN notebook starter [here][1] and discussion [here][6]. If you want to build a Transformer, you can use my Transformer notebook [here][5] from Ventilator competition and input Amex competition data. Additionally there are many great RNN notebooks in Ventilator competition for inspiration.\n\n# UPDATE: Transformer Starter - LB 0.790\nI created and published a Transformer starter notebook [here][7] today that uses my Kaggle dataset. It achieves 5-Fold CV 0.787 and LB 0.789. Enjoy!\n\n# Have Fun!\n\n[1]: https://tinyurl.com/4sjwms6v\n[2]: https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns\n[3]: https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/308635\n[4]: https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns?select=submission.csv\n[5]: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-112\n[6]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/327761\n[7]: https://tinyurl.com/45uu256r", "votes": 159 }, { "id": 1852080, "postDate": "2022-07-11T18:32:46.627Z", "content": "

Really nice work. Thanks!

", "rawMarkdown": "Really nice work. Thanks!", "votes": 1, "replies": [ { "id": 1873783, "postDate": "2022-07-27T22:42:49.697Z", "content": "

Thanks Jerry

", "rawMarkdown": "Thanks Jerry" } ] }, { "id": 1846843, "postDate": "2022-07-07T11:49:03.280Z", "content": "

Its awsome

", "rawMarkdown": "Its awsome", "votes": 1 }, { "id": 1844857, "postDate": "2022-07-05T21:23:33.127Z", "content": "

Hi @cdeotte , thanks so much. Your work is so clear.

", "rawMarkdown": "Hi @cdeotte , thanks so much. Your work is so clear. ", "votes": 1, "replies": [ { "id": 1844873, "postDate": "2022-07-05T21:52:33.670Z", "content": "

Thanks Kevin

", "rawMarkdown": "Thanks Kevin", "votes": 1 } ] }, { "id": 1813795, "postDate": "2022-06-07T08:46:09.917Z", "content": "

Hi @cdeotte , thanks for the great kernel.
\nUnfortunately, I didn't see the full code so far but I'm missing something for sure: how can you create the padded sequence for the customers in test data?
\nThey don't have any previous credit statement, so their sequence would be something like [-1, -1, -1…..-1, -1, statement] (basically, no history provided in the sequence).

\n

What am I missing?
\nMany thanks

", "rawMarkdown": "Hi @cdeotte , thanks for the great kernel.\nUnfortunately, I didn't see the full code so far but I'm missing something for sure: how can you create the padded sequence for the customers in test data?\nThey don't have any previous credit statement, so their sequence would be something like [-1, -1, -1.....-1, -1, statement] (basically, no history provided in the sequence).\n\nWhat am I missing?\nMany thanks", "votes": 1, "replies": [ { "id": 1813823, "postDate": "2022-06-07T09:11:33.753Z", "content": "

Most customers in test data have 13 statements. Perhaps you are using a Kaggle dataset that does not contain the full test data. For example, here is one customer in test data
\n\"\"

", "rawMarkdown": "Most customers in test data have 13 statements. Perhaps you are using a Kaggle dataset that does not contain the full test data. For example, here is one customer in test data\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Jun-2022/june.png)", "votes": 4 } ] }, { "id": 1809249, "postDate": "2022-06-02T14:44:15.893Z", "content": "

Lots of Learning!
\nThanks, @cdeotte for the explanation!

", "rawMarkdown": "Lots of Learning!\nThanks, @cdeotte for the explanation!", "votes": 2 }, { "id": 1807205, "postDate": "2022-05-31T20:12:23.393Z", "content": "

UPDATE: I posted TensorFlow Transformer starter code using this dataset here

", "rawMarkdown": "UPDATE: I posted TensorFlow Transformer starter code using this dataset [here][1]\n\n[1]: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790", "votes": 2 }, { "id": 1871637, "postDate": "2022-07-26T11:48:03.243Z", "content": "

Hi is it Time series classification problem ?
\nCan we submit csv with 1 and 0 only not by prediction in decimal like i'm thinking as classification problem or they are asking about **\"probability of target\" ** , that's why our submissions should be in probability of 0 or 1. ??? kindly support

", "rawMarkdown": "Hi is it Time series classification problem ?\nCan we submit csv with 1 and 0 only not by prediction in decimal like i'm thinking as classification problem or they are asking about **\"probability of target\" ** , that's why our submissions should be in probability of 0 or 1. ??? kindly support ", "replies": [ { "id": 1872018, "postDate": "2022-07-26T15:44:33.953Z", "content": "

For each test customer we submit the probability of default which is a number between 0 and 1. For each test customer we have up to 13 observations in time (i.e. credit card statements), so we have a time series (of length 13) for each test customer and we must make 1 classification probability predicton per customer.

", "rawMarkdown": "For each test customer we submit the probability of default which is a number between 0 and 1. For each test customer we have up to 13 observations in time (i.e. credit card statements), so we have a time series (of length 13) for each test customer and we must make 1 classification probability predicton per customer.", "votes": 4 }, { "id": 1872613, "postDate": "2022-07-27T06:02:01.550Z", "content": "

Noted. great support thanks

", "rawMarkdown": "Noted. great support thanks", "votes": 1 } ] }, { "id": 1806068, "postDate": "2022-05-30T18:50:48.927Z", "content": "

Hello, @cdeotte thanks for the explanation. I was looking to learn from the notebook that creates this data but the link was not provided. for example, the [here] doesn't have a link to the Notebook

\n

Below the Section of your post that reference the Link

\n

Competition Data NumPy Array
\nI have converted the competition data into NumPy array files where each customer has a sequence length 13. https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns (This data is created in my notebook [here][1])

", "rawMarkdown": "Hello, @cdeotte thanks for the explanation. I was looking to learn from the notebook that creates this data but the link was not provided. for example, the [here] doesn't have a link to the Notebook\n\nBelow the Section of your post that reference the Link\n\n**Competition Data NumPy Array**\nI have converted the competition data into NumPy array files where each customer has a sequence length 13. https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns (This data is created in my notebook [here][1])", "replies": [ { "id": 1806071, "postDate": "2022-05-30T18:53:53.093Z", "content": "

Thanks. Kaggle has a new spam filter that prevents me from posting the URL because i already posted it in another thread. The link is www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790

", "rawMarkdown": "Thanks. Kaggle has a new spam filter that prevents me from posting the URL because i already posted it in another thread. The link is `www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790`", "votes": 1 }, { "id": 1806073, "postDate": "2022-05-30T18:55:54.123Z", "content": "

@addisonhoward This new Kaggle spam filter is very annoying. I cannot post links to cross reference my posts. Also i cannot edit any of my old posts because when i try to edit and save, i get the error message You've been sharing this link too often. We prevent redundant posts to reduce spam.. This occurs even if i do not change any URLs in the old posts.

", "rawMarkdown": "@addisonhoward This new Kaggle spam filter is very annoying. I cannot post links to cross reference my posts. Also i cannot edit any of my old posts because when i try to edit and save, i get the error message `You've been sharing this link too often. We prevent redundant posts to reduce spam.`. This occurs even if i do not change any URLs in the old posts.", "votes": 5 }, { "id": 1806099, "postDate": "2022-05-30T19:33:50.790Z", "content": "

Thanks for the link, @cdeotte; the new Kaggle spam filter sounds like a headache, especially if you want to create more connected posts.

\n

@cdeotte I was trying to understand how to configure the dataset into a 3D version and created the *.npy files myself; I haven't been able to find this on any of your Notebooks. Did you make them locally?

", "rawMarkdown": "Thanks for the link, @cdeotte; the new Kaggle spam filter sounds like a headache, especially if you want to create more connected posts. \n\n@cdeotte I was trying to understand how to configure the dataset into a 3D version and created the *.npy files myself; I haven't been able to find this on any of your Notebooks. Did you make them locally? ", "votes": 1 }, { "id": 1806102, "postDate": "2022-05-30T19:36:29.410Z", "content": "

My Bad, @cdeotte I should have read the code deeper is all in the link provided, thanks

", "rawMarkdown": "My Bad, @cdeotte I should have read the code deeper is all in the link provided, thanks ", "votes": 1 }, { "id": 1806138, "postDate": "2022-05-30T20:19:28.413Z", "content": "

No worries. The code cell that saves the NumPy files is hidden. It is code cell 6 (in the provided link above). And the dataframe processing occurs in code cell 5.

\n

Note that my NumPy arrays are front padded. Whenever a customer has less than 13 statements, the code pads in the beginning. In actuality, every train customer has a last statement in March 2018 and the missing statements can be anywhere in the previous 12 months. So code for inplace padding is posted below. So far the GRU model with front padding achieves the same CV as inplace padding. (And test data code for inplace padding needs a few changes from below)

\n
# PAD SO EACH CUSTOMER HAS 13 ROWS    \ntrain = train.sort_values(['customer_ID','year','month','day'],\n                         ascending=[True, False, False, False])\ntrain['i'] = train.groupby('customer_ID').year.agg('cumcount')\nfinal_statement = train.loc[train.i == 0].drop(['i'],axis=1)\ntrain = train.loc[train.i != 0].drop(['i'],axis=1)\n# PAD WITHIN FIRST 12 STATEMENTS\ncust = final_statement.customer_ID.unique()\nmonths = cupy.tile( cupy.arange(12)+1,len(cust) )\ncust = cupy.repeat( cust,12 )\nnew_train = cudf.DataFrame({'customer_ID':cust,'month':months})    \nnew_train = new_train.merge(train, on=['customer_ID','month'], how='left')\nnew_train = new_train.fillna(-1)\nnew_train[CATS] = cupy.clip( new_train[CATS].values, a_min=0)\ntime_map = {3:17,4:17,5:17,6:17,7:17,8:17,9:17,10:17,11:17,12:17,1:18,2:18}\nnew_train['year'] = new_train.month.map(time_map)\n# COMBINE FIRST 12 STATEMENTS WITH FINAL STATEMENT FOR 13\ntrain = cudf.concat([new_train,final_statement],axis=0,ignore_index=True)\ndel new_train, cust, months, final_statement\n
", "rawMarkdown": "No worries. The code cell that saves the NumPy files is hidden. It is code cell 6 (in the provided link above). And the dataframe processing occurs in code cell 5.\n\nNote that my NumPy arrays are front padded. Whenever a customer has less than 13 statements, the code pads in the beginning. In actuality, every train customer has a last statement in March 2018 and the missing statements can be anywhere in the previous 12 months. So code for inplace padding is posted below. So far the GRU model with front padding achieves the same CV as inplace padding. (And test data code for inplace padding needs a few changes from below)\n\n # PAD SO EACH CUSTOMER HAS 13 ROWS \n train = train.sort_values(['customer_ID','year','month','day'],\n ascending=[True, False, False, False])\n train['i'] = train.groupby('customer_ID').year.agg('cumcount')\n final_statement = train.loc[train.i == 0].drop(['i'],axis=1)\n train = train.loc[train.i != 0].drop(['i'],axis=1)\n # PAD WITHIN FIRST 12 STATEMENTS\n cust = final_statement.customer_ID.unique()\n months = cupy.tile( cupy.arange(12)+1,len(cust) )\n cust = cupy.repeat( cust,12 )\n new_train = cudf.DataFrame({'customer_ID':cust,'month':months}) \n new_train = new_train.merge(train, on=['customer_ID','month'], how='left')\n new_train = new_train.fillna(-1)\n new_train[CATS] = cupy.clip( new_train[CATS].values, a_min=0)\n time_map = {3:17,4:17,5:17,6:17,7:17,8:17,9:17,10:17,11:17,12:17,1:18,2:18}\n new_train['year'] = new_train.month.map(time_map)\n # COMBINE FIRST 12 STATEMENTS WITH FINAL STATEMENT FOR 13\n train = cudf.concat([new_train,final_statement],axis=0,ignore_index=True)\n del new_train, cust, months, final_statement", "votes": 3 }, { "id": 1806199, "postDate": "2022-05-30T21:47:41.840Z", "content": "

Thanks for the detailed explanation, @cdeotte, and the code snippet; I will experiment with this approach. I believe time base feature engineering and model will have some advantages.

", "rawMarkdown": "Thanks for the detailed explanation, @cdeotte, and the code snippet; I will experiment with this approach. I believe time base feature engineering and model will have some advantages.", "votes": 1 }, { "id": 1807033, "postDate": "2022-05-31T16:49:51.497Z", "content": "

Hey @cdeotte - I've notified our engineering team who developed this filter to provide some more information.

", "rawMarkdown": "Hey @cdeotte - I've notified our engineering team who developed this filter to provide some more information. ", "votes": 2 }, { "id": 1808313, "postDate": "2022-06-01T17:50:01.847Z", "content": "

Hi Chris and others, the spam filter has been helpful in preventing rampant link spam by some users to unfairly gain medals. That said we hear you and recognize there are legitimate reasons to share resources that are good for others. We are working on tweaking the filter now. Thank you for your feedback!

", "rawMarkdown": "Hi Chris and others, the spam filter has been helpful in preventing rampant link spam by some users to unfairly gain medals. That said we hear you and recognize there are legitimate reasons to share resources that are good for others. We are working on tweaking the filter now. Thank you for your feedback!", "votes": 4 } ] }, { "id": 1881182, "postDate": "2022-08-02T10:25:01.143Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1816043, "postDate": "2022-06-09T18:37:08.890Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1847179, "postDate": "2022-07-07T17:30:25.177Z", "content": "

Thanks @cdeotte

", "rawMarkdown": "Thanks @cdeotte ", "votes": 1 }, { "id": 1816246, "postDate": "2022-06-10T02:49:52.640Z", "content": "

Nice work ! Thanks @cdeotte

", "rawMarkdown": "Nice work ! Thanks @cdeotte ", "votes": 1 }, { "id": 1809386, "postDate": "2022-06-02T16:44:44.063Z", "content": "

Wow!
\nThank you so much!

", "rawMarkdown": "Wow!\nThank you so much!", "votes": 1 }, { "id": 1809158, "postDate": "2022-06-02T13:03:01.390Z", "content": "

Thank you!

", "rawMarkdown": "Thank you!", "votes": 1 }, { "id": 2663852, "postDate": "2024-02-22T17:46:12.093Z", "content": "

thanks's man

", "rawMarkdown": "thanks's man" } ], "comments": [ { "id": 1852080, "author_name": "Jerry", "author_url": "", "post_date": "2022-07-11T18:32:46.627000", "content": "

Really nice work. Thanks!

", "votes": 1, "replies": [ { "id": 1873783, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-07-27T22:42:49.697000", "content": "

Thanks Jerry

", "votes": 0, "replies": [] } ] }, { "id": 1846843, "author_name": "Ammara Jabbar", "author_url": "", "post_date": "2022-07-07T11:49:03.280000", "content": "

Its awsome

", "votes": 1, "replies": [] }, { "id": 1844857, "author_name": "Kevin Meates", "author_url": "", "post_date": "2022-07-05T21:23:33.127000", "content": "

Hi @cdeotte , thanks so much. Your work is so clear.

", "votes": 1, "replies": [ { "id": 1844873, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-07-05T21:52:33.670000", "content": "

Thanks Kevin

", "votes": 1, "replies": [] } ] }, { "id": 1813795, "author_name": "Luca Sala", "author_url": "", "post_date": "2022-06-07T08:46:09.917000", "content": "

Hi @cdeotte , thanks for the great kernel.
\nUnfortunately, I didn't see the full code so far but I'm missing something for sure: how can you create the padded sequence for the customers in test data?
\nThey don't have any previous credit statement, so their sequence would be something like [-1, -1, -1…..-1, -1, statement] (basically, no history provided in the sequence).

\n

What am I missing?
\nMany thanks

", "votes": 1, "replies": [ { "id": 1813823, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-06-07T09:11:33.753000", "content": "

Most customers in test data have 13 statements. Perhaps you are using a Kaggle dataset that does not contain the full test data. For example, here is one customer in test data
\n\"\"

", "votes": 4, "replies": [] } ] }, { "id": 1809249, "author_name": "Pawan Sharma", "author_url": "", "post_date": "2022-06-02T14:44:15.893000", "content": "

Lots of Learning!
\nThanks, @cdeotte for the explanation!

", "votes": 2, "replies": [] }, { "id": 1807205, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-05-31T20:12:23.393000", "content": "

UPDATE: I posted TensorFlow Transformer starter code using this dataset here

", "votes": 2, "replies": [] }, { "id": 1871637, "author_name": "Chirag Valani", "author_url": "", "post_date": "2022-07-26T11:48:03.243000", "content": "

Hi is it Time series classification problem ?
\nCan we submit csv with 1 and 0 only not by prediction in decimal like i'm thinking as classification problem or they are asking about **\"probability of target\" ** , that's why our submissions should be in probability of 0 or 1. ??? kindly support

", "votes": 0, "replies": [ { "id": 1872018, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-07-26T15:44:33.953000", "content": "

For each test customer we submit the probability of default which is a number between 0 and 1. For each test customer we have up to 13 observations in time (i.e. credit card statements), so we have a time series (of length 13) for each test customer and we must make 1 classification probability predicton per customer.

", "votes": 4, "replies": [] }, { "id": 1872613, "author_name": "Chirag Valani", "author_url": "", "post_date": "2022-07-27T06:02:01.550000", "content": "

Noted. great support thanks

", "votes": 1, "replies": [] } ] }, { "id": 1806068, "author_name": "C4rl05/V", "author_url": "", "post_date": "2022-05-30T18:50:48.927000", "content": "

Hello, @cdeotte thanks for the explanation. I was looking to learn from the notebook that creates this data but the link was not provided. for example, the [here] doesn't have a link to the Notebook

\n

Below the Section of your post that reference the Link

\n

Competition Data NumPy Array
\nI have converted the competition data into NumPy array files where each customer has a sequence length 13. https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns (This data is created in my notebook [here][1])

", "votes": 0, "replies": [ { "id": 1806071, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-05-30T18:53:53.093000", "content": "

Thanks. Kaggle has a new spam filter that prevents me from posting the URL because i already posted it in another thread. The link is www.kaggle.com/code/cdeotte/tensorflow-gru-starter-0-790

", "votes": 1, "replies": [] }, { "id": 1806073, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-05-30T18:55:54.123000", "content": "

@addisonhoward This new Kaggle spam filter is very annoying. I cannot post links to cross reference my posts. Also i cannot edit any of my old posts because when i try to edit and save, i get the error message You've been sharing this link too often. We prevent redundant posts to reduce spam.. This occurs even if i do not change any URLs in the old posts.

", "votes": 5, "replies": [] }, { "id": 1806099, "author_name": "C4rl05/V", "author_url": "", "post_date": "2022-05-30T19:33:50.790000", "content": "

Thanks for the link, @cdeotte; the new Kaggle spam filter sounds like a headache, especially if you want to create more connected posts.

\n

@cdeotte I was trying to understand how to configure the dataset into a 3D version and created the *.npy files myself; I haven't been able to find this on any of your Notebooks. Did you make them locally?

", "votes": 1, "replies": [] }, { "id": 1806102, "author_name": "C4rl05/V", "author_url": "", "post_date": "2022-05-30T19:36:29.410000", "content": "

My Bad, @cdeotte I should have read the code deeper is all in the link provided, thanks

", "votes": 1, "replies": [] }, { "id": 1806138, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-05-30T20:19:28.413000", "content": "

No worries. The code cell that saves the NumPy files is hidden. It is code cell 6 (in the provided link above). And the dataframe processing occurs in code cell 5.

\n

Note that my NumPy arrays are front padded. Whenever a customer has less than 13 statements, the code pads in the beginning. In actuality, every train customer has a last statement in March 2018 and the missing statements can be anywhere in the previous 12 months. So code for inplace padding is posted below. So far the GRU model with front padding achieves the same CV as inplace padding. (And test data code for inplace padding needs a few changes from below)

\n
# PAD SO EACH CUSTOMER HAS 13 ROWS    \ntrain = train.sort_values(['customer_ID','year','month','day'],\n                         ascending=[True, False, False, False])\ntrain['i'] = train.groupby('customer_ID').year.agg('cumcount')\nfinal_statement = train.loc[train.i == 0].drop(['i'],axis=1)\ntrain = train.loc[train.i != 0].drop(['i'],axis=1)\n# PAD WITHIN FIRST 12 STATEMENTS\ncust = final_statement.customer_ID.unique()\nmonths = cupy.tile( cupy.arange(12)+1,len(cust) )\ncust = cupy.repeat( cust,12 )\nnew_train = cudf.DataFrame({'customer_ID':cust,'month':months})    \nnew_train = new_train.merge(train, on=['customer_ID','month'], how='left')\nnew_train = new_train.fillna(-1)\nnew_train[CATS] = cupy.clip( new_train[CATS].values, a_min=0)\ntime_map = {3:17,4:17,5:17,6:17,7:17,8:17,9:17,10:17,11:17,12:17,1:18,2:18}\nnew_train['year'] = new_train.month.map(time_map)\n# COMBINE FIRST 12 STATEMENTS WITH FINAL STATEMENT FOR 13\ntrain = cudf.concat([new_train,final_statement],axis=0,ignore_index=True)\ndel new_train, cust, months, final_statement\n
", "votes": 3, "replies": [] }, { "id": 1806199, "author_name": "C4rl05/V", "author_url": "", "post_date": "2022-05-30T21:47:41.840000", "content": "

Thanks for the detailed explanation, @cdeotte, and the code snippet; I will experiment with this approach. I believe time base feature engineering and model will have some advantages.

", "votes": 1, "replies": [] }, { "id": 1807033, "author_name": "Addison Howard", "author_url": "", "post_date": "2022-05-31T16:49:51.497000", "content": "

Hey @cdeotte - I've notified our engineering team who developed this filter to provide some more information.

", "votes": 2, "replies": [] }, { "id": 1808313, "author_name": "Jessica Li", "author_url": "", "post_date": "2022-06-01T17:50:01.847000", "content": "

Hi Chris and others, the spam filter has been helpful in preventing rampant link spam by some users to unfairly gain medals. That said we hear you and recognize there are legitimate reasons to share resources that are good for others. We are working on tweaking the filter now. Thank you for your feedback!

", "votes": 4, "replies": [] } ] }, { "id": 1881182, "author_name": "", "author_url": "", "post_date": "2022-08-02T10:25:01.143000", "content": "", "votes": 0, "replies": [] }, { "id": 1816043, "author_name": "", "author_url": "", "post_date": "2022-06-09T18:37:08.890000", "content": "", "votes": 0, "replies": [] }, { "id": 1847179, "author_name": "Pankaj Kumar", "author_url": "", "post_date": "2022-07-07T17:30:25.177000", "content": "

Thanks @cdeotte

", "votes": 1, "replies": [] }, { "id": 1816246, "author_name": "KhanhVD", "author_url": "", "post_date": "2022-06-10T02:49:52.640000", "content": "

Nice work ! Thanks @cdeotte

", "votes": 1, "replies": [] }, { "id": 1809386, "author_name": "Agung Aldevando", "author_url": "", "post_date": "2022-06-02T16:44:44.063000", "content": "

Wow!
\nThank you so much!

", "votes": 1, "replies": [] }, { "id": 1809158, "author_name": "Selbinyyaz Sultanova", "author_url": "", "post_date": "2022-06-02T13:03:01.390000", "content": "

Thank you!

", "votes": 1, "replies": [] }, { "id": 2663852, "author_name": "Sadik Sikder", "author_url": "", "post_date": "2024-02-22T17:46:12.093000", "content": "

thanks's man

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1804818": "# Competition Data CSV \nThis competition provides data about credit card customers in the form of `CSV` files. We cannot use these files without modification to train Transformers and RNNs because each customer has a different number of credit card statements. \n\nWhen inputting time series data into a Transformer and RNN, each customer needs the same sequence length (of 13 in this comp).\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/May-2022/trans.png)\n\n# Competition Data NumPy Array\nI have converted the competition data into `NumPy array` files where each customer has sequence length 13 and posted a Kaggle dataset [here][2] (This data is created in my notebook [here][1])\n\nFor each customer with less than 13 credit card statements, I have padded their data with `pad = -1`. See example below where the customer has been padded with four -1's:\n\n![](https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/May-2022/data_block.png)\n\n# Kaggle Dataset Details\nMy Kaggle dataset is [here][2]. The train data has been split into 10 NumPy arrays named `data_1.npy` thru `data_10.npy`. Each array has dimension `(45891, 13, 188)` which is `customer x statement x feature`. \n\nThe associated targets are contained in the files `targets_1.pqt` thru `targets_10.pqt`. These are parquet files with columns `customer_ID` and `target` and have dimension `(45891, 2)`. An example training with these files is [here][1]. Note that the `customer_ID` is a `int64` instead of the provided `string512`. It was compressed using the code shown [here][3].\n\nThe test data is split in 20 NumPy arrays. Each array has `46231` customers. **Important Note**: If you concatenate all these files, the row order of the `924621` test customers is **not the same** as the file `sample_submission.csv`. The order is contained in my `submission.csv` [here][4]. So infer the 20 NumPy arrays in their order and then overwrite my prediction column with `submission['prediction'] = preds`.\n\n# Transformers and RNNs\nIt will be exciting to see whether GBT (gradient boosted trees) or NN will achieve the best accuracy in this competition. I posted a RNN notebook starter [here][1] and discussion [here][6]. If you want to build a Transformer, you can use my Transformer notebook [here][5] from Ventilator competition and input Amex competition data. Additionally there are many great RNN notebooks in Ventilator competition for inspiration.\n\n# UPDATE: Transformer Starter - LB 0.790\nI created and published a Transformer starter notebook [here][7] today that uses my Kaggle dataset. It achieves 5-Fold CV 0.787 and LB 0.789. Enjoy!\n\n# Have Fun!\n\n[1]: https://tinyurl.com/4sjwms6v\n[2]: https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns\n[3]: https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/discussion/308635\n[4]: https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns?select=submission.csv\n[5]: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-112\n[6]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/327761\n[7]: https://tinyurl.com/45uu256r", "1852080": "Really nice work. Thanks!", "1846843": "Its awsome", "1844857": "Hi @cdeotte , thanks so much. Your work is so clear. ", "1813795": "Hi @cdeotte , thanks for the great kernel.\nUnfortunately, I didn't see the full code so far but I'm missing something for sure: how can you create the padded sequence for the customers in test data?\nThey don't have any previous credit statement, so their sequence would be something like [-1, -1, -1.....-1, -1, statement] (basically, no history provided in the sequence).\n\nWhat am I missing?\nMany thanks", "1809249": "Lots of Learning!\nThanks, @cdeotte for the explanation!", "1807205": "UPDATE: I posted TensorFlow Transformer starter code using this dataset [here][1]\n\n[1]: https://www.kaggle.com/code/cdeotte/tensorflow-transformer-0-790", "1871637": "Hi is it Time series classification problem ?\nCan we submit csv with 1 and 0 only not by prediction in decimal like i'm thinking as classification problem or they are asking about **\"probability of target\" ** , that's why our submissions should be in probability of 0 or 1. ??? kindly support ", "1806068": "Hello, @cdeotte thanks for the explanation. I was looking to learn from the notebook that creates this data but the link was not provided. for example, the [here] doesn't have a link to the Notebook\n\nBelow the Section of your post that reference the Link\n\n**Competition Data NumPy Array**\nI have converted the competition data into NumPy array files where each customer has a sequence length 13. https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns (This data is created in my notebook [here][1])", "1881182": "", "1816043": "", "1847179": "Thanks @cdeotte ", "1816246": "Nice work ! Thanks @cdeotte ", "1809386": "Wow!\nThank you so much!", "1809158": "Thank you!", "2663852": "thanks's man" } }