![\"\"](\"https://www.kaggleusercontent.com/kf/97216210/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..Pdd_2RGccTivL0zbTRfUyQ.PO0gPYa6X75S9t6sTY73VoulfpTiZvqDo7lybqD6GV4QFyLPABFdMF7L04BjgJwxaY1mjOZHTLAWG1cb3jtF53myqqWkBKUfpOSw0Diaaqe47wtm7Y0BAvr2KFJrS9lcQuuXh1pDnkcmBVG7UMGqlLLfbU1J8Bn5rVxL9Q_t3D9cvTlQBQLSPA2LyeyT23JXggDi5NgmVdRjSwrzRtatfSCATgwWHqxsR9cIbSxHnRMEC1jIJhQSpMQ8MQf9xDVQ-hDigARg2j826haCGCJPlgI1Yi6802TzaR0C1qs1N6yrdlrEzaIKOiFJT9jGBCkovn9nksg4nRwBAn7SWF1koljUMZSuO0s1i4p2l8663dV4T9RYeyd7kBZaG8_6LjlqGySlAuqwxpmKHgnsxe5iUsfQs9ay5OkGGt4o4X1iQijiMSbBgMpOnXF17zZV0zgMShmcjPho4hC9mknqfymWgUx1lAQDi0tXTscGkD3PFu5CiE-wwUPVyZ-fz_7PlvFcMb-UxclCu9C8SXeY1fm1zNKxaR-EUFY03RHBdtKORN8u9wX26exrgrwPh4AsZWvSAat7XICgSccsh8ntwXbF3BTLNHm_-13CKPyLWbvFCQAlYzD9Vb3i__mYxWy2b2g56i_bzMUNFnKZ3Wlntkynq-4gabg0dw1E8ut45IFThBk.8kwQIfaIdGMD3zlrHCQY-A/__results___files/__results___31_0.png\")
I think I have been very confused about default. This is what I currently believe - please comment where you think I have gone off the path.
\nWe have monthly data for many customers - 13 months is most for a majority of the customers but many have less than a 13 month history. (only a single month is present for some). This history was captured during 2017-2018.
\nAFTER the period of time for which we have data, the customer was tracked for 18 months. If they failed to pay the amount due within 120 days anytime during that 18 months they are default and target value of 1.
\nSo we need to train a model using 13 months (or less) of history - and up to 18 months after this history the customer could have defaulted. But the default did NOT occur during the train data time period. Movie fans may recall a Tom Cruise movie, \"Minority Report\" where 3 precogs could predict future crime. Our model needs to predict future default.
\nAs a future complication the test data is later in time (I believe some EDA's indicate no overlap in time). I am assuming that for the test data the default also DID NOT occur during the 2018-2019 time frame of the data, but rather default occurred up to 18 months after the test data - so 2020-2021. I need to look at EDA's but I believe no over lap in customers occurs in test/train.
\nSo much like precogs in Minority Report, we need to create a model based on 2017 history and predict default for a different group of customers in 2020-2021 time frame.
", "messages": [ { "id": "1803648", "postDate": "05/28/2022 03:46:22", "content": "I think I have been very confused about default. This is what I currently believe - please comment where you think I have gone off the path.
\nWe have monthly data for many customers - 13 months is most for a majority of the customers but many have less than a 13 month history. (only a single month is present for some). This history was captured during 2017-2018.
\nAFTER the period of time for which we have data, the customer was tracked for 18 months. If they failed to pay the amount due within 120 days anytime during that 18 months they are default and target value of 1.
\nSo we need to train a model using 13 months (or less) of history - and up to 18 months after this history the customer could have defaulted. But the default did NOT occur during the train data time period. Movie fans may recall a Tom Cruise movie, \"Minority Report\" where 3 precogs could predict future crime. Our model needs to predict future default.
\nAs a future complication the test data is later in time (I believe some EDA's indicate no overlap in time). I am assuming that for the test data the default also DID NOT occur during the 2018-2019 time frame of the data, but rather default occurred up to 18 months after the test data - so 2020-2021. I need to look at EDA's but I believe no over lap in customers occurs in test/train.
\nSo much like precogs in Minority Report, we need to create a model based on 2017 history and predict default for a different group of customers in 2020-2021 time frame.
", "rawMarkdown": "I think I have been very confused about default. This is what I currently believe - please comment where you think I have gone off the path.\n\nWe have monthly data for many customers - 13 months is most for a majority of the customers but many have less than a 13 month history. (only a single month is present for some). This history was captured during 2017-2018.\n\nAFTER the period of time for which we have data, the customer was tracked for 18 months. If they failed to pay the amount due within 120 days anytime during that 18 months they are default and target value of 1. \n\nSo we need to train a model using 13 months (or less) of history - and up to 18 months after this history the customer could have defaulted. But the default did NOT occur during the train data time period. Movie fans may recall a Tom Cruise movie, \"Minority Report\" where 3 precogs could predict future crime. Our model needs to predict future default.\n\nAs a future complication the test data is later in time (I believe some EDA's indicate no overlap in time). I am assuming that for the test data the default also DID NOT occur during the 2018-2019 time frame of the data, but rather default occurred up to 18 months after the test data - so 2020-2021. I need to look at EDA's but I believe no over lap in customers occurs in test/train.\n\nSo much like precogs in Minority Report, we need to create a model based on 2017 history and predict default for a different group of customers in 2020-2021 time frame.", "votes": null }, { "id": "1803670", "postDate": "05/28/2022 04:22:57", "content": "If you check the train dataset, Starting month is March 2017 and ending month is March 2018, As per my understanding March 2018 is the final statement on which Default is labelled.
\nSo customers who were on the book on March 2017 were evaluated on their 13th statement (which is the last for 120+ Dq for 18 month window). Check notebook mentioned in this thread
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/327595
for details on this assumption.
\nHope this helps
", "rawMarkdown": "If you check the train dataset, Starting month is March 2017 and ending month is March 2018, As per my understanding March 2018 is the final statement on which Default is labelled.\n\nSo customers who were on the book on March 2017 were evaluated on their 13th statement (which is the last for 120+ Dq for 18 month window). Check notebook mentioned in this thread\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/327595\n\n for details on this assumption.\n\nHope this helps", "votes": null }, { "id": "1804146", "postDate": "05/28/2022 15:10:27", "content": "\nSorry - No help to me - what your saying is what I use to believe. I believe that 18 months clock starts AT the last statement (lots of customers who do not have 13 rows).
\nThe objective of this competition is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile
", "rawMarkdown": "[Seeker](https://www.kaggle.com/taran8727)\n\nSorry - No help to me - what your saying is what I use to believe. I believe that 18 months clock starts AT the last statement (lots of customers who do not have 13 rows).\n\n*The objective of this competition is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile*", "votes": null }, { "id": "1804319", "postDate": "05/28/2022 18:59:38", "content": "Only 15% do not have 13 statements,
\nI am understanding your point, so it might be that given 1 years history of customers on the book on March 2018, we need to predict their probability of default in next 18 months
", "rawMarkdown": "Only 15% do not have 13 statements,\n\nI am understanding your point, so it might be that given 1 years history of customers on the book on March 2018, we need to predict their probability of default in next 18 months", "votes": null }, { "id": "1804374", "postDate": "05/28/2022 20:31:18", "content": "Yes - I think we do need to predict for the next 18 months which would seem to make it a much tougher competition. If the default occurs during the 1 year history period we should be able to find the month that payments stopped and 120 days later the default occurs. If I am correct than in the 1 year history we need to find a pattern that predicts future default.
\nSome defaults would seem very hard to predict - I spend a lot of money over the year and my payment is always early or on time for the full amount. I lose my job the next year and wam_bam I go in default. Pretty sure we cannot model that path.
\nI spend a lot of money and my payments are seldom on time and most often not for the full amount. That pattern should be predictable of future default.
\nIt's been many years since I had an American Express card but as I recall at the time AE expected full amount payment each month and you could not slowly generate a large balance due. Do not know if that is the still the case or if users can slowly accumulate a large balance due.
", "rawMarkdown": "Yes - I think we do need to predict for the next 18 months which would seem to make it a much tougher competition. If the default occurs during the 1 year history period we should be able to find the month that payments stopped and 120 days later the default occurs. If I am correct than in the 1 year history we need to find a pattern that predicts future default.\n\nSome defaults would seem very hard to predict - I spend a lot of money over the year and my payment is always early or on time for the full amount. I lose my job the next year and wam_bam I go in default. Pretty sure we cannot model that path.\n\nI spend a lot of money and my payments are seldom on time and most often not for the full amount. That pattern should be predictable of future default. \n\nIt's been many years since I had an American Express card but as I recall at the time AE expected full amount payment each month and you could not slowly generate a large balance due. Do not know if that is the still the case or if users can slowly accumulate a large balance due.", "votes": null }, { "id": "1804878", "postDate": "05/29/2022 14:14:11", "content": "enen,, for every customers, like you said, most of them have 13 months records but here mentioned 18 month windows; I prefer to think, bank observed the next 18 months transactions after credit card statement; but customers only used this credit card on 13 months; so on training dataset, it is a record of customer actions (18 months, but only have 13 records), and binary value of default event (have a overdue pay in this 18 months observation time)
", "rawMarkdown": "enen,, for every customers, like you said, most of them have 13 months records but here mentioned 18 month windows; I prefer to think, bank observed the next 18 months transactions after credit card statement; but customers only used this credit card on 13 months; so on training dataset, it is a record of customer actions (18 months, but only have 13 records), and binary value of default event (have a overdue pay in this 18 months observation time)", "votes": null }, { "id": "1806086", "postDate": "05/30/2022 19:17:39", "content": "This also seems like a decent understanding - is it your belief that all defaults occurred in the 5 months after the end of train records or were some defaults in the 13 month period?
", "rawMarkdown": "This also seems like a decent understanding - is it your belief that all defaults occurred in the 5 months after the end of train records or were some defaults in the 13 month period?", "votes": null }, { "id": "1808978", "postDate": "06/02/2022 09:35:55", "content": "I've checked in my EDA notebook that indeed most of customers have 13 records and only few less but the target distribution for these two groups is quite different. See attached pictures.
\nIt's worth checking who are customers with presence in a database with less than 13 records. Are these customers who quickly defaulted or simply who entered late into a database. This is my next step and I'll keep you updated.
![\"\"](\"https://www.kaggleusercontent.com/kf/97216210/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..Pdd_2RGccTivL0zbTRfUyQ.PO0gPYa6X75S9t6sTY73VoulfpTiZvqDo7lybqD6GV4QFyLPABFdMF7L04BjgJwxaY1mjOZHTLAWG1cb3jtF53myqqWkBKUfpOSw0Diaaqe47wtm7Y0BAvr2KFJrS9lcQuuXh1pDnkcmBVG7UMGqlLLfbU1J8Bn5rVxL9Q_t3D9cvTlQBQLSPA2LyeyT23JXggDi5NgmVdRjSwrzRtatfSCATgwWHqxsR9cIbSxHnRMEC1jIJhQSpMQ8MQf9xDVQ-hDigARg2j826haCGCJPlgI1Yi6802TzaR0C1qs1N6yrdlrEzaIKOiFJT9jGBCkovn9nksg4nRwBAn7SWF1koljUMZSuO0s1i4p2l8663dV4T9RYeyd7kBZaG8_6LjlqGySlAuqwxpmKHgnsxe5iUsfQs9ay5OkGGt4o4X1iQijiMSbBgMpOnXF17zZV0zgMShmcjPho4hC9mknqfymWgUx1lAQDi0tXTscGkD3PFu5CiE-wwUPVyZ-fz_7PlvFcMb-UxclCu9C8SXeY1fm1zNKxaR-EUFY03RHBdtKORN8u9wX26exrgrwPh4AsZWvSAat7XICgSccsh8ntwXbF3BTLNHm_-13CKPyLWbvFCQAlYzD9Vb3i__mYxWy2b2g56i_bzMUNFnKZ3Wlntkynq-4gabg0dw1E8ut45IFThBk.8kwQIfaIdGMD3zlrHCQY-A/__results___files/__results___33_0.png\")
My belief would be these are folks who just started late.
\nI believe no defaults occurred during the train time period, but all were in an 18 month time span after the last statement.
\nChecking dates of statements for the less than 13 crowd could shed some light on the subject.
", "rawMarkdown": "[Robert](https://www.kaggle.com/datark1)\n\nMy belief would be these are folks who just started late.\n\nI believe no defaults occurred during the train time period, but all were in an 18 month time span after the last statement.\n\nChecking dates of statements for the less than 13 crowd could shed some light on the subject.", "votes": null }, { "id": "1809185", "postDate": "06/02/2022 13:36:42", "content": "Good point. Definitely, I will investigate it.
", "rawMarkdown": "Good point. Definitely, I will investigate it.", "votes": null }, { "id": "1809351", "postDate": "06/02/2022 16:24:01", "content": "After checking who are those customers it seems that there are both: who entered late and those who dropped from the observation or made no transactions anymore. Details are in the notebook and below screenshots showing distribution of customers with only 2 observations across time and exemplary one of them.
\n![\"\"](\"https://i.postimg.cc/3JCnjDFL/1.png\")
![\"\"](\"https://i.postimg.cc/K80fwSVB/2.png\")
Robert
\nNice - so those that dropped early - were they also defaults? Think it a decent bet if a customer drops out on their own terms they are probably not going to default in the later 18 month :)
\nYour two selected customers are not defaulters - going to fork your EDA but will not get to work it for a while - have my first real date in 55 years in 2 hours that needs more of my efforts. If she dumps me early I will be back to run your EDA and check out the default question.
\nIf some of these who drop out are defaulters than my theory gets all smashed to dust since I believe that all defaults are in the 18 month future after the last statement in the train set.
the latter one; defaults in the 13 month period; this is training dataset; it would provide all variables for this 18 months records; so there should be nothing which needs to be predicted.
", "rawMarkdown": "the latter one; defaults in the 13 month period; this is training dataset; it would provide all variables for this 18 months records; so there should be nothing which needs to be predicted.", "votes": null } ], "comments": [ { "id": 1803670, "author_name": "taran8727", "author_url": "", "post_date": "05/28/2022 04:22:57", "content": "If you check the train dataset, Starting month is March 2017 and ending month is March 2018, As per my understanding March 2018 is the final statement on which Default is labelled.
\nSo customers who were on the book on March 2017 were evaluated on their 13th statement (which is the last for 120+ Dq for 18 month window). Check notebook mentioned in this thread
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/327595
for details on this assumption.
\nHope this helps
", "votes": null, "replies": [ { "id": 1804146, "author_name": "pcjimmmy", "author_url": "", "post_date": "05/28/2022 15:10:27", "content": "\nSorry - No help to me - what your saying is what I use to believe. I believe that 18 months clock starts AT the last statement (lots of customers who do not have 13 rows).
\nThe objective of this competition is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile
", "votes": null, "replies": [] }, { "id": 1804319, "author_name": "taran8727", "author_url": "", "post_date": "05/28/2022 18:59:38", "content": "Only 15% do not have 13 statements,
\nI am understanding your point, so it might be that given 1 years history of customers on the book on March 2018, we need to predict their probability of default in next 18 months
", "votes": null, "replies": [] }, { "id": 1804374, "author_name": "pcjimmmy", "author_url": "", "post_date": "05/28/2022 20:31:18", "content": "Yes - I think we do need to predict for the next 18 months which would seem to make it a much tougher competition. If the default occurs during the 1 year history period we should be able to find the month that payments stopped and 120 days later the default occurs. If I am correct than in the 1 year history we need to find a pattern that predicts future default.
\nSome defaults would seem very hard to predict - I spend a lot of money over the year and my payment is always early or on time for the full amount. I lose my job the next year and wam_bam I go in default. Pretty sure we cannot model that path.
\nI spend a lot of money and my payments are seldom on time and most often not for the full amount. That pattern should be predictable of future default.
\nIt's been many years since I had an American Express card but as I recall at the time AE expected full amount payment each month and you could not slowly generate a large balance due. Do not know if that is the still the case or if users can slowly accumulate a large balance due.
", "votes": null, "replies": [] } ] }, { "id": 1804878, "author_name": "linlinzeng", "author_url": "", "post_date": "05/29/2022 14:14:11", "content": "enen,, for every customers, like you said, most of them have 13 months records but here mentioned 18 month windows; I prefer to think, bank observed the next 18 months transactions after credit card statement; but customers only used this credit card on 13 months; so on training dataset, it is a record of customer actions (18 months, but only have 13 records), and binary value of default event (have a overdue pay in this 18 months observation time)
", "votes": null, "replies": [ { "id": 1806086, "author_name": "pcjimmmy", "author_url": "", "post_date": "05/30/2022 19:17:39", "content": "This also seems like a decent understanding - is it your belief that all defaults occurred in the 5 months after the end of train records or were some defaults in the 13 month period?
", "votes": null, "replies": [] }, { "id": 1810406, "author_name": "linlinzeng", "author_url": "", "post_date": "06/03/2022 14:32:18", "content": "the latter one; defaults in the 13 month period; this is training dataset; it would provide all variables for this 18 months records; so there should be nothing which needs to be predicted.
", "votes": null, "replies": [] } ] }, { "id": 1808978, "author_name": "datark1", "author_url": "", "post_date": "06/02/2022 09:35:55", "content": "I've checked in my EDA notebook that indeed most of customers have 13 records and only few less but the target distribution for these two groups is quite different. See attached pictures.
\nIt's worth checking who are customers with presence in a database with less than 13 records. Are these customers who quickly defaulted or simply who entered late into a database. This is my next step and I'll keep you updated.
My belief would be these are folks who just started late.
\nI believe no defaults occurred during the train time period, but all were in an 18 month time span after the last statement.
\nChecking dates of statements for the less than 13 crowd could shed some light on the subject.
", "votes": null, "replies": [] }, { "id": 1809185, "author_name": "datark1", "author_url": "", "post_date": "06/02/2022 13:36:42", "content": "Good point. Definitely, I will investigate it.
", "votes": null, "replies": [] }, { "id": 1809351, "author_name": "datark1", "author_url": "", "post_date": "06/02/2022 16:24:01", "content": "After checking who are those customers it seems that there are both: who entered late and those who dropped from the observation or made no transactions anymore. Details are in the notebook and below screenshots showing distribution of customers with only 2 observations across time and exemplary one of them.
\n
Robert
\nNice - so those that dropped early - were they also defaults? Think it a decent bet if a customer drops out on their own terms they are probably not going to default in the later 18 month :)
\nYour two selected customers are not defaulters - going to fork your EDA but will not get to work it for a while - have my first real date in 55 years in 2 hours that needs more of my efforts. If she dumps me early I will be back to run your EDA and check out the default question.
\nIf some of these who drop out are defaulters than my theory gets all smashed to dust since I believe that all defaults are in the 18 month future after the last statement in the train set.