Does the .8 score mean they had an 80% accuracy or does it mean they had a 100% accuracy, and the scoring metric equation made the top possible score .8?
", "messages": [ { "id": 1852553, "postDate": "2022-07-12T06:31:34.453Z", "content": "Does the .8 score mean they had an 80% accuracy or does it mean they had a 100% accuracy, and the scoring metric equation made the top possible score .8?
", "rawMarkdown": "Does the .8 score mean they had an 80% accuracy or does it mean they had a 100% accuracy, and the scoring metric equation made the top possible score .8?", "votes": 4 }, { "id": 1861065, "postDate": "2022-07-18T18:45:02.557Z", "content": "The metric doesn't look at the accuracy of the predictions. It only looks at the accuracy of the customers ranking. As an experiment you can multiply your submissions by 5 and see that the metric score is unchanged, even with predictions higher than 1.
", "rawMarkdown": "The metric doesn't look at the accuracy of the predictions. It only looks at the accuracy of the customers ranking. As an experiment you can multiply your submissions by 5 and see that the metric score is unchanged, even with predictions higher than 1." }, { "id": 1853592, "postDate": "2022-07-13T01:47:52.610Z", "content": "One of the first things I did was take an existing template, and have it print the 4% and the gini separately. I'm not that high yet, but 0.8 is roughly gini of 0.927, which would be equivalent to auc of 0.9635, plus a 4% accuracy of about 67.3%.
\nNot sure if that helps, but point is that 0.8 is a combination of two very different scores, one higher, one lower
", "rawMarkdown": "One of the first things I did was take an existing template, and have it print the 4% and the gini separately. I'm not that high yet, but 0.8 is roughly gini of 0.927, which would be equivalent to auc of 0.9635, plus a 4% accuracy of about 67.3%. \n\nNot sure if that helps, but point is that 0.8 is a combination of two very different scores, one higher, one lower", "replies": [ { "id": 1853773, "postDate": "2022-07-13T06:01:59.297Z", "content": "So in layman’s terms the models of a 0.8 have an accuracy in the upper 60s, low 70s?
", "rawMarkdown": "So in layman’s terms the models of a 0.8 have an accuracy in the upper 60s, low 70s?" }, { "id": 1853784, "postDate": "2022-07-13T06:19:27.267Z", "content": "Well, if you added 20 times as many non-default customers (to match the real-world distribution), and zoomed in ONLY on the 4% the model predicted the most likely to default, the prediction accuracy WITHIN that small slice is around ~67%
", "rawMarkdown": "Well, if you added 20 times as many non-default customers (to match the real-world distribution), and zoomed in ONLY on the 4% the model predicted the most likely to default, the prediction accuracy WITHIN that small slice is around ~67%" } ] }, { "id": 1853482, "postDate": "2022-07-12T23:17:22.637Z", "content": "These are good resources to understand the AMEX metric:
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/327116
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/327464
The metric doesn't look at the accuracy of the predictions. It only looks at the accuracy of the customers ranking. As an experiment you can multiply your submissions by 5 and see that the metric score is unchanged, even with predictions higher than 1.
", "votes": 0, "replies": [] }, { "id": 1853592, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-07-13T01:47:52.610000", "content": "One of the first things I did was take an existing template, and have it print the 4% and the gini separately. I'm not that high yet, but 0.8 is roughly gini of 0.927, which would be equivalent to auc of 0.9635, plus a 4% accuracy of about 67.3%.
\nNot sure if that helps, but point is that 0.8 is a combination of two very different scores, one higher, one lower
", "votes": 0, "replies": [ { "id": 1853773, "author_name": "Nathan Perkins", "author_url": "", "post_date": "2022-07-13T06:01:59.297000", "content": "So in layman’s terms the models of a 0.8 have an accuracy in the upper 60s, low 70s?
", "votes": 0, "replies": [] }, { "id": 1853784, "author_name": "Robert Hatch", "author_url": "", "post_date": "2022-07-13T06:19:27.267000", "content": "Well, if you added 20 times as many non-default customers (to match the real-world distribution), and zoomed in ONLY on the 4% the model predicted the most likely to default, the prediction accuracy WITHIN that small slice is around ~67%
", "votes": 0, "replies": [] } ] }, { "id": 1853482, "author_name": "1110Ra", "author_url": "", "post_date": "2022-07-12T23:17:22.637000", "content": "These are good resources to understand the AMEX metric:
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/327116
\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/327464