We have seen the top of the leaderboard be fairly stable for a while now. What do you think the final score might get to?
", "messages": [ { "id": 1871370, "postDate": "2022-07-26T08:35:00.387Z", "content": "We have seen the top of the leaderboard be fairly stable for a while now. What do you think the final score might get to?
", "rawMarkdown": "We have seen the top of the leaderboard be fairly stable for a while now. What do you think the final score might get to?", "votes": 11 }, { "id": 1871742, "postDate": "2022-07-26T13:08:41.133Z", "content": "My guess public LB will remain at 0.801.
\nPrivate LB 0.799.
", "rawMarkdown": "My guess public LB will remain at 0.801. \n\nPrivate LB 0.799.", "votes": 9, "replies": [ { "id": 1872232, "postDate": "2022-07-26T18:20:00.523Z", "content": "That will be an interesting finale, I suspect most people will have a similar score on the lb.
", "rawMarkdown": "That will be an interesting finale, I suspect most people will have a similar score on the lb. ", "votes": 2 }, { "id": 1904725, "postDate": "2022-08-18T12:43:41.777Z", "content": "Well someone just hit .802 I didn't expect that!
", "rawMarkdown": "Well someone just hit .802 I didn't expect that!" }, { "id": 1904778, "postDate": "2022-08-18T13:26:27.360Z", "content": "Saw it coming: https://www.kaggle.com/competitions/amex-default-prediction/discussion/342992#1896393
", "rawMarkdown": "Saw it coming: https://www.kaggle.com/competitions/amex-default-prediction/discussion/342992#1896393" }, { "id": 1914714, "postDate": "2022-08-26T10:38:44.147Z", "content": "On reflection, are you surprised at the final results?
", "rawMarkdown": "On reflection, are you surprised at the final results?" }, { "id": 1914723, "postDate": "2022-08-26T10:50:26.443Z", "content": "No I am not. I was confident everyone gonna get a better score on private LB.
\nTo counter the distribution difference between public and private, our team went with more complex ensemble which resulted in a shake down for us. But nonetheless, enjoyed and learned a lot from this competition.
Hi, I found a paper by the competition host and placed it one of the discussions. AMEX GBDT model scored 95.56 (GINI) and 81.60 (RECALL), for defaults occuring within April to October 2018. I don't know if this helps though.
", "rawMarkdown": "Hi, I found a paper by the competition host and placed it one of the discussions. AMEX GBDT model scored 95.56 (GINI) and 81.60 (RECALL), for defaults occuring within April to October 2018. I don't know if this helps though.", "votes": 3 }, { "id": 1872916, "postDate": "2022-07-27T10:13:47.193Z", "content": "The private is affected by covid. I’d guess a 2-3 % loss in AUC. For the D metric it could be as high as 10-15% loss. I’ll start with 5% loss for both components and guess a plateau of score around 0.75.
", "rawMarkdown": "The private is affected by covid. I’d guess a 2-3 % loss in AUC. For the D metric it could be as high as 10-15% loss. I’ll start with 5% loss for both components and guess a plateau of score around 0.75.", "votes": 3, "replies": [ { "id": 1872934, "postDate": "2022-07-27T10:33:10.360Z", "content": "that's doomsday predictions :D
", "rawMarkdown": "that's doomsday predictions :D", "votes": 2 }, { "id": 1873778, "postDate": "2022-07-27T22:38:55.553Z", "content": "Good point. However, the private dataset last statement is Oct 2019. So the customer defaulting window is Nov 2019 thru Feb 2020. This is just when Covid was beginning. So Covid may or may not affect payments.
", "rawMarkdown": "Good point. However, the private dataset last statement is Oct 2019. So the customer defaulting window is Nov 2019 thru Feb 2020. This is just when Covid was beginning. So Covid may or may not affect payments.", "votes": 6 }, { "id": 1874120, "postDate": "2022-07-28T05:07:43.650Z", "content": "18 months of observation… starting Nov 2019 so ending (and including) April 2021.
", "rawMarkdown": "18 months of observation... starting Nov 2019 so ending (and including) April 2021.", "votes": 2 }, { "id": 1875378, "postDate": "2022-07-29T01:20:04.210Z", "content": "No, i think 18 months refers to the rows of the train and test dataset. I think the default period is 120 days.
", "rawMarkdown": "No, i think 18 months refers to the rows of the train and test dataset. I think the default period is 120 days.", "votes": 2 }, { "id": 1875515, "postDate": "2022-07-29T04:44:23.517Z", "content": "Default of 120 days anytime in a 18 month period. If last private is Oct 2019 than default may be recorded anytime in the next 18 months (IMHO).
\nKind of amazing that AE is offering 40K for a solution and we don't even agree on the basics of the problem.
\nProblem solving 101 - define the problem.
", "rawMarkdown": "Default of 120 days anytime in a 18 month period. If last private is Oct 2019 than default may be recorded anytime in the next 18 months (IMHO). \n\nKind of amazing that AE is offering 40K for a solution and we don't even agree on the basics of the problem. \n\nProblem solving 101 - define the problem.\n", "votes": 2 }, { "id": 1875734, "postDate": "2022-07-29T08:48:01.193Z", "content": "From the data tab: \"The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\"
\nThe first part refers to the period of observation that span 18 months after the lastest credit card statement available, that is the horizon of observation of the target. The second part refers to some sort of materiality… to avoid too much noise (technical problems) the default is only counted after 3 months of non payment on their latest credit card statement at that time. The credit card statement refered here is in the future and is not ne essarily available in the training data set.
", "rawMarkdown": "From the data tab: \"The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\"\n \nThe first part refers to the period of observation that span 18 months after the lastest credit card statement available, that is the horizon of observation of the target. The second part refers to some sort of materiality... to avoid too much noise (technical problems) the default is only counted after 3 months of non payment on their latest credit card statement at that time. The credit card statement refered here is in the future and is not ne essarily available in the training data set.", "votes": 1 }, { "id": 1876019, "postDate": "2022-07-29T14:11:42.377Z", "content": "\nDo you think that each customer has 31 credit card statements. And the data only shows us the first 13. And the target refers to defaulting on the 18 we cannot see?
\n(Otherwise i don't understand how it takes 18 months to determine if a customer doesn't pay 120 days after the last of the 13 showing in the data).
", "rawMarkdown": "~~Do you think that each customer has 36 credit card statements. And the data only shows us the first 18. And the target refers to defaulting on the 18 we cannot see?~~\n\nDo you think that each customer has 31 credit card statements. And the data only shows us the first 13. And the target refers to defaulting on the 18 we cannot see?\n\n(Otherwise i don't understand how it takes 18 months to determine if a customer doesn't pay 120 days after the last of the ~~18~~ 13 showing in the data).", "votes": 2 }, { "id": 1876030, "postDate": "2022-07-29T14:27:36.053Z", "content": "I could be wrong … but yes, this is a pretty standard approach (and as you mention other interpretations doesn't make much sense).
", "rawMarkdown": "I could be wrong ... but yes, this is a pretty standard approach (and as you mention other interpretations doesn't make much sense).", "votes": 1 }, { "id": 1876744, "postDate": "2022-07-30T04:10:00.477Z", "content": "No - each customer does not have 36. In the training data we have a decent percentage of customers who have less than 13 statements. Some would appear to have just joined AE - some would appear to be leaving AE.
\nThe train group of customers with a statement at the end of the train time period should have an additional 18 months worth of statements that we never see (unless of course they default and are dropped). My assumption is that AE would not have included customers who left of their own choice in the 18 months following the last train statement.
\nThere is no customer_ID overlap between train and test.
\nThere is no customer_ID overlap between public test and private test.
\n13+18 = the number months that any non default customer should have in train/public test/private test.
", "rawMarkdown": "No - each customer does not have 36. In the training data we have a decent percentage of customers who have less than 13 [statements](https://www.kaggle.com/code/datark1/american-express-eda). Some would appear to have just joined AE - some would appear to be leaving AE. \n\nThe train group of customers with a statement at the end of the train time period should have an additional 18 months worth of statements that we never see (unless of course they default and are dropped). My assumption is that AE would not have included customers who left of their own choice in the 18 months following the last train statement.\n\nThere is no customer_ID overlap between train and test. \n\nThere is no customer_ID overlap between public test and private test. \n\n13+18 = the number months that any non default customer should have in train/public test/private test." }, { "id": 1882461, "postDate": "2022-08-03T09:30:27.330Z", "content": "Correct me if I am wrong, but Covid shouldn't have too big of an impact on the Private Board. if during Covid the default probability increased by 20% uniformly (as an example), then the metrics would be unchanged. Try submitting a submission file with a factor 1.2 on the prediction column and you will see that the score is unchanged. This is due to the way the metric works. we need to sort the customers in the right order, not get the predictions accurately. It is only if Covid messed up with the customer ranking for some second order reasons that we can see an impact in the Private Board ranking.
", "rawMarkdown": "Correct me if I am wrong, but Covid shouldn't have too big of an impact on the Private Board. if during Covid the default probability increased by 20% uniformly (as an example), then the metrics would be unchanged. Try submitting a submission file with a factor 1.2 on the prediction column and you will see that the score is unchanged. This is due to the way the metric works. we need to sort the customers in the right order, not get the predictions accurately. It is only if Covid messed up with the customer ranking for some second order reasons that we can see an impact in the Private Board ranking.", "votes": 1 }, { "id": 1882466, "postDate": "2022-08-03T09:32:41.627Z", "content": "Yeah if we assume uniform impact there is no problem…
", "rawMarkdown": "Yeah if we assume uniform impact there is no problem...", "votes": 1 } ] }, { "id": 1871880, "postDate": "2022-07-26T14:31:53.203Z", "content": "A follow on question is do you think Amex will have got what they want out of this competition given the LB? I get the sense they’re hoping for a) potential hires or b) interesting approach that approves the score but can be also be put into production.
", "rawMarkdown": "A follow on question is do you think Amex will have got what they want out of this competition given the LB? I get the sense they’re hoping for a) potential hires or b) interesting approach that approves the score but can be also be put into production. ", "votes": 3, "replies": [ { "id": 1872229, "postDate": "2022-07-26T18:19:05.857Z", "content": "That's a good question, I wonder how much they would have spent developing their current ML for this task, and if it is cost effective to just have kaggle comps and implement the best model.
", "rawMarkdown": "That's a good question, I wonder how much they would have spent developing their current ML for this task, and if it is cost effective to just have kaggle comps and implement the best model.", "votes": 1 }, { "id": 1877720, "postDate": "2022-07-31T00:36:36.257Z", "content": "This site https://upgradedpoints.com/credit-cards/us-credit-card-market-share-by-network-issuer/ says AmEx has purchase volume is around $600 billion (10^9) per year. That's $50B per month. A 1% default rate costs them $500M each month, and a change of 1 basis point (.01%) is worth $5M per month. So I would think they would have spent a fair amount trying to reduce their default rate. One problem they have that we don't have in this competition is that AmEx has to pass various non-discrimination tests, which the data might not \"realize.\"
", "rawMarkdown": "This site https://upgradedpoints.com/credit-cards/us-credit-card-market-share-by-network-issuer/ says AmEx has purchase volume is around $600 billion (10^9) per year. That's $50B per month. A 1% default rate costs them $500M each month, and a change of 1 basis point (.01%) is worth $5M per month. So I would think they would have spent a fair amount trying to reduce their default rate. One problem they have that we don't have in this competition is that AmEx has to pass various non-discrimination tests, which the data might not \"realize.\"\n", "votes": 6 } ] }, { "id": 1922117, "postDate": "2022-09-01T09:24:20.163Z", "content": "Hi @jakelj, May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants
", "rawMarkdown": "Hi @jakelj, May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants" }, { "id": 1872791, "postDate": "2022-07-27T08:32:15.910Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1871742, "author_name": "raddar", "author_url": "", "post_date": "2022-07-26T13:08:41.133000", "content": "My guess public LB will remain at 0.801.
\nPrivate LB 0.799.
", "votes": 9, "replies": [ { "id": 1872232, "author_name": "Jake", "author_url": "", "post_date": "2022-07-26T18:20:00.523000", "content": "That will be an interesting finale, I suspect most people will have a similar score on the lb.
", "votes": 2, "replies": [] }, { "id": 1904725, "author_name": "Jake", "author_url": "", "post_date": "2022-08-18T12:43:41.777000", "content": "Well someone just hit .802 I didn't expect that!
", "votes": 0, "replies": [] }, { "id": 1904778, "author_name": "tarick.morty", "author_url": "", "post_date": "2022-08-18T13:26:27.360000", "content": "Saw it coming: https://www.kaggle.com/competitions/amex-default-prediction/discussion/342992#1896393
", "votes": 0, "replies": [] }, { "id": 1914714, "author_name": "Jake", "author_url": "", "post_date": "2022-08-26T10:38:44.147000", "content": "On reflection, are you surprised at the final results?
", "votes": 0, "replies": [] }, { "id": 1914723, "author_name": "tarick.morty", "author_url": "", "post_date": "2022-08-26T10:50:26.443000", "content": "No I am not. I was confident everyone gonna get a better score on private LB.
\nTo counter the distribution difference between public and private, our team went with more complex ensemble which resulted in a shake down for us. But nonetheless, enjoyed and learned a lot from this competition.
Hi, I found a paper by the competition host and placed it one of the discussions. AMEX GBDT model scored 95.56 (GINI) and 81.60 (RECALL), for defaults occuring within April to October 2018. I don't know if this helps though.
", "votes": 3, "replies": [] }, { "id": 1872916, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-07-27T10:13:47.193000", "content": "The private is affected by covid. I’d guess a 2-3 % loss in AUC. For the D metric it could be as high as 10-15% loss. I’ll start with 5% loss for both components and guess a plateau of score around 0.75.
", "votes": 3, "replies": [ { "id": 1872934, "author_name": "raddar", "author_url": "", "post_date": "2022-07-27T10:33:10.360000", "content": "that's doomsday predictions :D
", "votes": 2, "replies": [] }, { "id": 1873778, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-07-27T22:38:55.553000", "content": "Good point. However, the private dataset last statement is Oct 2019. So the customer defaulting window is Nov 2019 thru Feb 2020. This is just when Covid was beginning. So Covid may or may not affect payments.
", "votes": 6, "replies": [] }, { "id": 1874120, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-07-28T05:07:43.650000", "content": "18 months of observation… starting Nov 2019 so ending (and including) April 2021.
", "votes": 2, "replies": [] }, { "id": 1875378, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-07-29T01:20:04.210000", "content": "No, i think 18 months refers to the rows of the train and test dataset. I think the default period is 120 days.
", "votes": 2, "replies": [] }, { "id": 1875515, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2022-07-29T04:44:23.517000", "content": "Default of 120 days anytime in a 18 month period. If last private is Oct 2019 than default may be recorded anytime in the next 18 months (IMHO).
\nKind of amazing that AE is offering 40K for a solution and we don't even agree on the basics of the problem.
\nProblem solving 101 - define the problem.
", "votes": 2, "replies": [] }, { "id": 1875734, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-07-29T08:48:01.193000", "content": "From the data tab: \"The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\"
\nThe first part refers to the period of observation that span 18 months after the lastest credit card statement available, that is the horizon of observation of the target. The second part refers to some sort of materiality… to avoid too much noise (technical problems) the default is only counted after 3 months of non payment on their latest credit card statement at that time. The credit card statement refered here is in the future and is not ne essarily available in the training data set.
", "votes": 1, "replies": [] }, { "id": 1876019, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-07-29T14:11:42.377000", "content": "\nDo you think that each customer has 31 credit card statements. And the data only shows us the first 13. And the target refers to defaulting on the 18 we cannot see?
\n(Otherwise i don't understand how it takes 18 months to determine if a customer doesn't pay 120 days after the last of the 13 showing in the data).
", "votes": 2, "replies": [] }, { "id": 1876030, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-07-29T14:27:36.053000", "content": "I could be wrong … but yes, this is a pretty standard approach (and as you mention other interpretations doesn't make much sense).
", "votes": 1, "replies": [] }, { "id": 1876744, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2022-07-30T04:10:00.477000", "content": "No - each customer does not have 36. In the training data we have a decent percentage of customers who have less than 13 statements. Some would appear to have just joined AE - some would appear to be leaving AE.
\nThe train group of customers with a statement at the end of the train time period should have an additional 18 months worth of statements that we never see (unless of course they default and are dropped). My assumption is that AE would not have included customers who left of their own choice in the 18 months following the last train statement.
\nThere is no customer_ID overlap between train and test.
\nThere is no customer_ID overlap between public test and private test.
\n13+18 = the number months that any non default customer should have in train/public test/private test.
", "votes": 0, "replies": [] }, { "id": 1882461, "author_name": "Elias", "author_url": "", "post_date": "2022-08-03T09:30:27.330000", "content": "Correct me if I am wrong, but Covid shouldn't have too big of an impact on the Private Board. if during Covid the default probability increased by 20% uniformly (as an example), then the metrics would be unchanged. Try submitting a submission file with a factor 1.2 on the prediction column and you will see that the score is unchanged. This is due to the way the metric works. we need to sort the customers in the right order, not get the predictions accurately. It is only if Covid messed up with the customer ranking for some second order reasons that we can see an impact in the Private Board ranking.
", "votes": 1, "replies": [] }, { "id": 1882466, "author_name": "Lucas Morin", "author_url": "", "post_date": "2022-08-03T09:32:41.627000", "content": "Yeah if we assume uniform impact there is no problem…
", "votes": 1, "replies": [] } ] }, { "id": 1871880, "author_name": "Burrito Dan", "author_url": "", "post_date": "2022-07-26T14:31:53.203000", "content": "A follow on question is do you think Amex will have got what they want out of this competition given the LB? I get the sense they’re hoping for a) potential hires or b) interesting approach that approves the score but can be also be put into production.
", "votes": 3, "replies": [ { "id": 1872229, "author_name": "Jake", "author_url": "", "post_date": "2022-07-26T18:19:05.857000", "content": "That's a good question, I wonder how much they would have spent developing their current ML for this task, and if it is cost effective to just have kaggle comps and implement the best model.
", "votes": 1, "replies": [] }, { "id": 1877720, "author_name": "SolverWorld", "author_url": "", "post_date": "2022-07-31T00:36:36.257000", "content": "This site https://upgradedpoints.com/credit-cards/us-credit-card-market-share-by-network-issuer/ says AmEx has purchase volume is around $600 billion (10^9) per year. That's $50B per month. A 1% default rate costs them $500M each month, and a change of 1 basis point (.01%) is worth $5M per month. So I would think they would have spent a fair amount trying to reduce their default rate. One problem they have that we don't have in this competition is that AmEx has to pass various non-discrimination tests, which the data might not \"realize.\"
", "votes": 6, "replies": [] } ] }, { "id": 1922117, "author_name": "Yang Liu", "author_url": "", "post_date": "2022-09-01T09:24:20.163000", "content": "Hi @jakelj, May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants
", "votes": 0, "replies": [] }, { "id": 1872791, "author_name": "", "author_url": "", "post_date": "2022-07-27T08:32:15.910000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1871370": "We have seen the top of the leaderboard be fairly stable for a while now. What do you think the final score might get to?", "1871742": "My guess public LB will remain at 0.801. \n\nPrivate LB 0.799.", "1876104": "Hi, I found a paper by the competition host and placed it one of the discussions. AMEX GBDT model scored 95.56 (GINI) and 81.60 (RECALL), for defaults occuring within April to October 2018. I don't know if this helps though.", "1872916": "The private is affected by covid. I’d guess a 2-3 % loss in AUC. For the D metric it could be as high as 10-15% loss. I’ll start with 5% loss for both components and guess a plateau of score around 0.75.", "1871880": "A follow on question is do you think Amex will have got what they want out of this competition given the LB? I get the sense they’re hoping for a) potential hires or b) interesting approach that approves the score but can be also be put into production. ", "1922117": "Hi @jakelj, May I invite you to participate in this survey regarding your experience on Kaggle (10 min)? This is not a scam. We are a group of researchers at the City University of Hong Kong. The survey link is: https://cityuhk.questionpro.com/survey-of-kaggle-contestants", "1872791": "" } }