{ "id": 327361, "title": "Another way to keep only the last month without groupby", "url": "/competitions/amex-default-prediction/discussion/327361", "author_name": "João Pedro Peinado", "post_date": "2022-05-26T20:26:17.319000", "votes": 34, "comment_count": 5, "views": 0, "content": "

Hi guys, I was working with the dataset and the way provided by @inversion to keep only the last statement was causing a big memory usage on because of groupby in the code. So I came up with another way to keep the last statement without groupby.

\n

Inversion way:
\ntrain.groupby('customer_ID').tail(1).drop(['S_2'], axis='columns')
\nInversion post

\n

My way:
\ntrain.drop_duplicates(subset=['customer_ID'], keep='last').drop(['S_2'], axis='columns')

\n

With this way I'm able to create more features with less memory available.

\n

Hope it helps!

", "messages": [ { "id": 1802526, "postDate": "2022-05-26T20:26:17.320Z", "content": "

Hi guys, I was working with the dataset and the way provided by @inversion to keep only the last statement was causing a big memory usage on because of groupby in the code. So I came up with another way to keep the last statement without groupby.

\n

Inversion way:
\ntrain.groupby('customer_ID').tail(1).drop(['S_2'], axis='columns')
\nInversion post

\n

My way:
\ntrain.drop_duplicates(subset=['customer_ID'], keep='last').drop(['S_2'], axis='columns')

\n

With this way I'm able to create more features with less memory available.

\n

Hope it helps!

", "rawMarkdown": "Hi guys, I was working with the dataset and the way provided by @inversion to keep only the last statement was causing a big memory usage on because of groupby in the code. So I came up with another way to keep the last statement without groupby.\n\nInversion way:\n`train.groupby('customer_ID').tail(1).drop(['S_2'], axis='columns')`\n[Inversion post](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327094)\n\nMy way:\n`train.drop_duplicates(subset=['customer_ID'], keep='last').drop(['S_2'], axis='columns')`\n\nWith this way I'm able to create more features with less memory available.\n\nHope it helps!", "votes": 33 }, { "id": 1807908, "postDate": "2022-06-01T12:36:54.353Z", "content": "

@joaopmpeinado we are assuming that dates are sorted (luckily this is the case here) with ascending date.

", "rawMarkdown": "@joaopmpeinado we are assuming that dates are sorted (luckily this is the case here) with ascending date. ", "votes": 1, "replies": [ { "id": 1808154, "postDate": "2022-06-01T15:11:50.543Z", "content": "

@kmmohsin yes, but this is also true if you are using the groupby version that were provided by the organizers

", "rawMarkdown": "@kmmohsin yes, but this is also true if you are using the groupby version that were provided by the organizers", "votes": 2 } ] }, { "id": 1838955, "postDate": "2022-07-01T02:54:39.367Z", "content": "

nice approach!

", "rawMarkdown": "nice approach! " }, { "id": 1833762, "postDate": "2022-06-26T10:12:21.320Z", "content": "

You might want to play with things like average balance over the period. (New to this one so I'm exploring different options.)

", "rawMarkdown": "You might want to play with things like average balance over the period. (New to this one so I'm exploring different options.)" }, { "id": 1804322, "postDate": "2022-05-28T19:03:27.943Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1807908, "author_name": "Mohsin", "author_url": "", "post_date": "2022-06-01T12:36:54.353000", "content": "

@joaopmpeinado we are assuming that dates are sorted (luckily this is the case here) with ascending date.

", "votes": 1, "replies": [ { "id": 1808154, "author_name": "João Pedro Peinado", "author_url": "", "post_date": "2022-06-01T15:11:50.543000", "content": "

@kmmohsin yes, but this is also true if you are using the groupby version that were provided by the organizers

", "votes": 2, "replies": [] } ] }, { "id": 1838955, "author_name": "1110Ra", "author_url": "", "post_date": "2022-07-01T02:54:39.367000", "content": "

nice approach!

", "votes": 0, "replies": [] }, { "id": 1833762, "author_name": "NotGaussian", "author_url": "", "post_date": "2022-06-26T10:12:21.320000", "content": "

You might want to play with things like average balance over the period. (New to this one so I'm exploring different options.)

", "votes": 0, "replies": [] }, { "id": 1804322, "author_name": "", "author_url": "", "post_date": "2022-05-28T19:03:27.943000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1802526": "Hi guys, I was working with the dataset and the way provided by @inversion to keep only the last statement was causing a big memory usage on because of groupby in the code. So I came up with another way to keep the last statement without groupby.\n\nInversion way:\n`train.groupby('customer_ID').tail(1).drop(['S_2'], axis='columns')`\n[Inversion post](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327094)\n\nMy way:\n`train.drop_duplicates(subset=['customer_ID'], keep='last').drop(['S_2'], axis='columns')`\n\nWith this way I'm able to create more features with less memory available.\n\nHope it helps!", "1807908": "@joaopmpeinado we are assuming that dates are sorted (luckily this is the case here) with ascending date. ", "1838955": "nice approach! ", "1833762": "You might want to play with things like average balance over the period. (New to this one so I'm exploring different options.)", "1804322": "" } }