![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9718188%2F979b3f044b3a7278393822fcdfdefd88%2FGandDLearningCurves.png?generation=1656602775384215&alt=media\")
As explained by the organizers here, the competition's metric M is the average of two sub-metrics: G and D.
\n In this discussion, we will look at the learning curve for each sub-metric, while training an XGB model. During the cross validation process, a validation subset of the training data is kept apart (not shown to the model). The curves below show the evolution of G and D on this validation subset, during the XGB training.
The model converges at an M score of 0.793 but the sub-metrics have very different behaviors. G reaches 0.924 while D merely reaches 0.662. This seems to indicate that there is more room for improvement on the D sub-metric. Unfortunately, we only see M on the LeaderBoard.
\nThis Notebook looks into D in more details.
", "messages": [ { "id": "1837967", "postDate": "06/30/2022 04:42:56", "content": "As explained by the organizers here, the competition's metric M is the average of two sub-metrics: G and D.
\n In this discussion, we will look at the learning curve for each sub-metric, while training an XGB model. During the cross validation process, a validation subset of the training data is kept apart (not shown to the model). The curves below show the evolution of G and D on this validation subset, during the XGB training.
The model converges at an M score of 0.793 but the sub-metrics have very different behaviors. G reaches 0.924 while D merely reaches 0.662. This seems to indicate that there is more room for improvement on the D sub-metric. Unfortunately, we only see M on the LeaderBoard.
\nThis Notebook looks into D in more details.
", "rawMarkdown": "As explained by the organizers [here](https://www.kaggle.com/competitions/amex-default-prediction/overview/evaluation), the competition's metric **M is the average of two sub-metrics: G and D**.\n In this discussion, we will look at the learning curve for each sub-metric, while training an XGB model. During the cross validation process, a validation subset of the training data is kept apart (not shown to the model). The curves below show the evolution of G and D on this validation subset, during the XGB training. \n\n\n\nThe model converges at an M score of 0.793 but the sub-metrics have very different behaviors. G reaches 0.924 while D merely reaches 0.662. This seems to indicate that there is more room for improvement on the D sub-metric. Unfortunately, we only see M on the LeaderBoard.\n\n[This Notebook](https://www.kaggle.com/code/gehallak/the-dark-side-of-the-moon) looks into D in more details.", "votes": null }, { "id": "1838764", "postDate": "06/30/2022 19:27:29", "content": "So, there’s a strong bias toward high precision at the given recall. I wondered how the objective function could be modified to take advantage of this. Cue diversion into focal loss and weighted loss. I come back empty handed, but somewhat wiser. My next idea. Look at what errors my model is making in the competition metric. Is it for a certain type of customer?
\nPs you could also plot logloss, the actual objective function.
", "rawMarkdown": "So, there’s a strong bias toward high precision at the given recall. I wondered how the objective function could be modified to take advantage of this. Cue diversion into focal loss and weighted loss. I come back empty handed, but somewhat wiser. My next idea. Look at what errors my model is making in the competition metric. Is it for a certain type of customer?\n\nPs you could also plot logloss, the actual objective function.", "votes": null }, { "id": "1839931", "postDate": "07/01/2022 20:03:05", "content": "I like the idea of the possibility to squeeze some more of D by giving away some of G.
\nThere is this paper and the related github repo that could be helpful. They essentially implement a modified (asymmetric) focal cross-entropy loss by applying different focal strengths for the hard false positives and hard false negatives, as opposed to the standard focal cross-entropy where hard to classify samples are weighted equally for positive and negative labels.
\nIn my case it did not improve the performance though. It seems that G and D are too tightly coupled. Maybe some else has better luck with it.
", "rawMarkdown": "I like the idea of the possibility to squeeze some more of D by giving away some of G.\n\nThere is [this paper](https://arxiv.org/pdf/2009.14119.pdf) and the related [github repo](https://github.com/Alibaba-MIIL/ASL) that could be helpful. They essentially implement a modified (asymmetric) focal cross-entropy loss by applying different focal strengths for the hard false positives and hard false negatives, as opposed to the standard focal cross-entropy where hard to classify samples are weighted equally for positive and negative labels.\n\nIn my case it did not improve the performance though. It seems that G and D are too tightly coupled. Maybe some else has better luck with it.", "votes": null }, { "id": "1840244", "postDate": "07/02/2022 04:38:39", "content": "it's an interesting idea to try different loss functions. This is actually why I wanted to see these curves. It appeared clearly when drawing these curves, that during the XGB learning process, while the logloss goes down regularly and smoothly, G and D go up regularly and smoothly too. The 3 of them converge to their asymptotes together without erratic moves. Empirically, it looks like even if the model knows nothing about the metrics G and D, it is doing a very good job at maximizing them. This is probably because we have a large number of customers. Trying to accurately predict the probability of default for each of them translates correctly in ranking them in order of probability of default (which is the only thing G and D care about).
", "rawMarkdown": "it's an interesting idea to try different loss functions. This is actually why I wanted to see these curves. It appeared clearly when drawing these curves, that during the XGB learning process, while the logloss goes down regularly and smoothly, G and D go up regularly and smoothly too. The 3 of them converge to their asymptotes together without erratic moves. Empirically, it looks like even if the model knows nothing about the metrics G and D, it is doing a very good job at maximizing them. This is probably because we have a large number of customers. Trying to accurately predict the probability of default for each of them translates correctly in ranking them in order of probability of default (which is the only thing G and D care about).", "votes": null }, { "id": "1844191", "postDate": "07/05/2022 12:04:22", "content": "It takes real courage to land on the dark side of the moon, thanks for sharing your knowledge with us!
", "rawMarkdown": "It takes real courage to land on the dark side of the moon, thanks for sharing your knowledge with us!", "votes": null }, { "id": "1844341", "postDate": "07/05/2022 13:34:27", "content": "Thanks for reading and for your sense of humor 😂
", "rawMarkdown": "Thanks for reading and for your sense of humor 😂", "votes": null }, { "id": "1844350", "postDate": "07/05/2022 13:46:54", "content": "Thanks for your thoughts @burritodan. I have tried to focus on ranking (which is all the metrics care about) instead of prediction (loss function) during the blending but without any noticeable increase (or decrease) in the LB. As if with a large number of customers, both problems converge.
", "rawMarkdown": "Thanks for your thoughts @burritodan. I have tried to focus on ranking (which is all the metrics care about) instead of prediction (loss function) during the blending but without any noticeable increase (or decrease) in the LB. As if with a large number of customers, both problems converge.", "votes": null } ], "comments": [ { "id": 1838764, "author_name": "burritodan", "author_url": "", "post_date": "06/30/2022 19:27:29", "content": "So, there’s a strong bias toward high precision at the given recall. I wondered how the objective function could be modified to take advantage of this. Cue diversion into focal loss and weighted loss. I come back empty handed, but somewhat wiser. My next idea. Look at what errors my model is making in the competition metric. Is it for a certain type of customer?
\nPs you could also plot logloss, the actual objective function.
", "votes": null, "replies": [ { "id": 1844350, "author_name": "gehallak", "author_url": "", "post_date": "07/05/2022 13:46:54", "content": "Thanks for your thoughts @burritodan. I have tried to focus on ranking (which is all the metrics care about) instead of prediction (loss function) during the blending but without any noticeable increase (or decrease) in the LB. As if with a large number of customers, both problems converge.
", "votes": null, "replies": [] } ] }, { "id": 1839931, "author_name": "kaerim", "author_url": "", "post_date": "07/01/2022 20:03:05", "content": "I like the idea of the possibility to squeeze some more of D by giving away some of G.
\nThere is this paper and the related github repo that could be helpful. They essentially implement a modified (asymmetric) focal cross-entropy loss by applying different focal strengths for the hard false positives and hard false negatives, as opposed to the standard focal cross-entropy where hard to classify samples are weighted equally for positive and negative labels.
\nIn my case it did not improve the performance though. It seems that G and D are too tightly coupled. Maybe some else has better luck with it.
", "votes": null, "replies": [ { "id": 1840244, "author_name": "gehallak", "author_url": "", "post_date": "07/02/2022 04:38:39", "content": "it's an interesting idea to try different loss functions. This is actually why I wanted to see these curves. It appeared clearly when drawing these curves, that during the XGB learning process, while the logloss goes down regularly and smoothly, G and D go up regularly and smoothly too. The 3 of them converge to their asymptotes together without erratic moves. Empirically, it looks like even if the model knows nothing about the metrics G and D, it is doing a very good job at maximizing them. This is probably because we have a large number of customers. Trying to accurately predict the probability of default for each of them translates correctly in ranking them in order of probability of default (which is the only thing G and D care about).
", "votes": null, "replies": [] } ] }, { "id": 1844191, "author_name": "saurabhbagchi", "author_url": "", "post_date": "07/05/2022 12:04:22", "content": "It takes real courage to land on the dark side of the moon, thanks for sharing your knowledge with us!
", "votes": null, "replies": [ { "id": 1844341, "author_name": "gehallak", "author_url": "", "post_date": "07/05/2022 13:34:27", "content": "Thanks for reading and for your sense of humor 😂
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1837967": "As explained by the organizers [here](https://www.kaggle.com/competitions/amex-default-prediction/overview/evaluation), the competition's metric **M is the average of two sub-metrics: G and D**.\n In this discussion, we will look at the learning curve for each sub-metric, while training an XGB model. During the cross validation process, a validation subset of the training data is kept apart (not shown to the model). The curves below show the evolution of G and D on this validation subset, during the XGB training. \n\n\n\nThe model converges at an M score of 0.793 but the sub-metrics have very different behaviors. G reaches 0.924 while D merely reaches 0.662. This seems to indicate that there is more room for improvement on the D sub-metric. Unfortunately, we only see M on the LeaderBoard.\n\n[This Notebook](https://www.kaggle.com/code/gehallak/the-dark-side-of-the-moon) looks into D in more details.", "1838764": "So, there’s a strong bias toward high precision at the given recall. I wondered how the objective function could be modified to take advantage of this. Cue diversion into focal loss and weighted loss. I come back empty handed, but somewhat wiser. My next idea. Look at what errors my model is making in the competition metric. Is it for a certain type of customer?\n\nPs you could also plot logloss, the actual objective function.", "1839931": "I like the idea of the possibility to squeeze some more of D by giving away some of G.\n\nThere is [this paper](https://arxiv.org/pdf/2009.14119.pdf) and the related [github repo](https://github.com/Alibaba-MIIL/ASL) that could be helpful. They essentially implement a modified (asymmetric) focal cross-entropy loss by applying different focal strengths for the hard false positives and hard false negatives, as opposed to the standard focal cross-entropy where hard to classify samples are weighted equally for positive and negative labels.\n\nIn my case it did not improve the performance though. It seems that G and D are too tightly coupled. Maybe some else has better luck with it.", "1840244": "it's an interesting idea to try different loss functions. This is actually why I wanted to see these curves. It appeared clearly when drawing these curves, that during the XGB learning process, while the logloss goes down regularly and smoothly, G and D go up regularly and smoothly too. The 3 of them converge to their asymptotes together without erratic moves. Empirically, it looks like even if the model knows nothing about the metrics G and D, it is doing a very good job at maximizing them. This is probably because we have a large number of customers. Trying to accurately predict the probability of default for each of them translates correctly in ranking them in order of probability of default (which is the only thing G and D care about).", "1844191": "It takes real courage to land on the dark side of the moon, thanks for sharing your knowledge with us!", "1844341": "Thanks for reading and for your sense of humor 😂", "1844350": "Thanks for your thoughts @burritodan. I have tried to focus on ranking (which is all the metrics care about) instead of prediction (loss function) during the blending but without any noticeable increase (or decrease) in the LB. As if with a large number of customers, both problems converge." }, "source": "meta" }