\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1342094%2Fb6fe6814782903d2d40cdfd0ce8c1931%2Fbad%20model.png?generation=1734261949016183&alt=media\")
Does anyone have predicted vs. actuals plots they can share? Even for an average or basic model? I'm trying to figure out if my current approach is just poor or has a bug. My plot is basically a flatline.
", "messages": [ { "id": "3072445", "postDate": "12/15/2024 07:56:49", "content": "Does anyone have predicted vs. actuals plots they can share? Even for an average or basic model? I'm trying to figure out if my current approach is just poor or has a bug. My plot is basically a flatline.
", "rawMarkdown": "Does anyone have predicted vs. actuals plots they can share? Even for an average or basic model? I'm trying to figure out if my current approach is just poor or has a bug. My plot is basically a flatline.", "votes": null }, { "id": "3072448", "postDate": "12/15/2024 08:05:45", "content": "You may take this from any public kernel and make the plot @jimbeno
", "rawMarkdown": "You may take this from any public kernel and make the plot @jimbeno", "votes": null }, { "id": "3072451", "postDate": "12/15/2024 08:14:28", "content": "try to plot your predictions on a different axis :)
", "rawMarkdown": "try to plot your predictions on a different axis :)", "votes": null }, { "id": "3072548", "postDate": "12/15/2024 11:35:39", "content": "I made visual evaluations in my LGBM models.
\nIt fails miserably to predict the range -5 to +5.
\n
The following chart should be a good model
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1342094%2F3dd52540975a6cf9c7e0cec55f601575%2Fgood%20model.png?generation=1734261986692256&alt=media\")
I can see why it fails to predict the entire range. We can see that most values are in a narrow range between +1 and -1.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1342094%2F669b2ed66ca4e7bc25d3c3e656007f6d%2Fdensity.png?generation=1734262132420656&alt=media\")
I think that even the top leaderboards solutions aren't doing a really good job predicting the responder. The maximum score is 1 and top leaderboards are 0.01x. I think their predicted x actual plot should be closer to my bad model than to the good one.
", "rawMarkdown": "I made visual evaluations in my LGBM models. \n\nIt fails miserably to predict the range -5 to +5.\n\n\nThe following chart should be a good model\n\n\nI can see why it fails to predict the entire range. We can see that most values are in a narrow range between +1 and -1.\n\n\nI think that even the top leaderboards solutions aren't doing a really good job predicting the responder. The maximum score is 1 and top leaderboards are 0.01x. I think their predicted x actual plot should be closer to my bad model than to the good one.", "votes": null }, { "id": "3072668", "postDate": "12/15/2024 14:18:08", "content": "The model doesn’t “dare” to predict large values. This is due to the extreme long tailed target distribution. You can design a weighted loss to encourage the model to predict larger values, similar to what is done in a classification task using the Focal Loss. But this implies that you have put some prior knowledge or assumptions on the hidden test data distributions.
\nNice plot and visuals!
This is from a super simple mlp evaluated on symbol_id 1
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11063283%2Fbee7e11b073c7e442b32c4fd0416c48a%2Fscatter.png?generation=1734294957546393&alt=media\")
Every time I have tried to update my loss function to stop it being \"safe\" and keeping close to 0 I ended up getting worse results.
", "rawMarkdown": "Every time I have tried to update my loss function to stop it being \"safe\" and keeping close to 0 I ended up getting worse results.", "votes": null }, { "id": "3073025", "postDate": "12/16/2024 00:10:01", "content": "Okay, that's basically what I'm seeing. I've never had a predicted vs. actuals plot look so flat 😆 So my takeaway is it may not be a bug in my code, but just a poor model that's not able to learn anything? Looks like it's always predicting between -1 and 1. Almost seems like it could be a bug with the target being normalized and then not de-normalized. But I'm getting this kind of plot without modifying the target responder_6 at all.
", "rawMarkdown": "Okay, that's basically what I'm seeing. I've never had a predicted vs. actuals plot look so flat 😆 So my takeaway is it may not be a bug in my code, but just a poor model that's not able to learn anything? Looks like it's always predicting between -1 and 1. Almost seems like it could be a bug with the target being normalized and then not de-normalized. But I'm getting this kind of plot without modifying the target responder_6 at all.", "votes": null }, { "id": "3073026", "postDate": "12/16/2024 00:13:25", "content": "Thank you, this is fascinating. I'm used to dealing with classification task imbalances. I've never really worked with a regression target imbalance. I wonder, do similar techniques like over-sampling, under-sampling, different weights, apply for regression models?
", "rawMarkdown": "Thank you, this is fascinating. I'm used to dealing with classification task imbalances. I've never really worked with a regression target imbalance. I wonder, do similar techniques like over-sampling, under-sampling, different weights, apply for regression models?", "votes": null }, { "id": "3073275", "postDate": "12/16/2024 08:54:43", "content": "I was asking me exaclty the same question. I did some research powered by AI in the weekend and found that sampling, despite unusual, could be applied. Different weights can be applied also. Lastly, it suggested me this article. I am experimenting with it. For now, it has the same result as training without it.
", "rawMarkdown": "I was asking me exaclty the same question. I did some research powered by AI in the weekend and found that sampling, despite unusual, could be applied. Different weights can be applied also. Lastly, it suggested me this [article](https://towardsdatascience.com/strategies-and-tactics-for-regression-on-imbalanced-data-61eeb0921fca). I am experimenting with it. For now, it has the same result as training without it.", "votes": null }, { "id": "3073285", "postDate": "12/16/2024 09:26:57", "content": "Yes, same techniques dealing with imbalance in classifications can also be applied to regression here. For example, you can bin the targets into ordered classes and resample from each class to get a more balanced dataset. But the trap here is that you have changed the inherent distribution of the data when doing so. It is worth to think if we really want that. My experiment in the beginning of the competition did not give me good results :(
\nBTW Thanks @serjhenrique for the article. It is a great source. I found it too in my early explorations but totally forgot it xD.
", "rawMarkdown": "Yes, same techniques dealing with imbalance in classifications can also be applied to regression here. For example, you can bin the targets into ordered classes and resample from each class to get a more balanced dataset. But the trap here is that you have changed the inherent distribution of the data when doing so. It is worth to think if we really want that. My experiment in the beginning of the competition did not give me good results :(\n\nBTW Thanks @serjhenrique for the article. It is a great source. I found it too in my early explorations but totally forgot it xD.", "votes": null }, { "id": "3073653", "postDate": "12/16/2024 17:41:04", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F16529018%2F9e7b88bd0027b993c52f291177aa8aa4%2F2024-12-17%20013633.png?generation=1734370732783421&alt=media\")
A xgb model trained with 800 date, cv 0.0080 (valid in the last 120 dates)
\nAs we can see, the prediction values are mostly in -1 ~ 1, and it is because most target data are between -1 and 1.
You may take this from any public kernel and make the plot @jimbeno
", "votes": null, "replies": [] }, { "id": 3072451, "author_name": "shiyili", "author_url": "", "post_date": "12/15/2024 08:14:28", "content": "try to plot your predictions on a different axis :)
", "votes": null, "replies": [] }, { "id": 3072548, "author_name": "serjhenrique", "author_url": "", "post_date": "12/15/2024 11:35:39", "content": "I made visual evaluations in my LGBM models.
\nIt fails miserably to predict the range -5 to +5.
\n
The following chart should be a good model
\n
I can see why it fails to predict the entire range. We can see that most values are in a narrow range between +1 and -1.
\n
I think that even the top leaderboards solutions aren't doing a really good job predicting the responder. The maximum score is 1 and top leaderboards are 0.01x. I think their predicted x actual plot should be closer to my bad model than to the good one.
", "votes": null, "replies": [ { "id": 3072668, "author_name": "shiyili", "author_url": "", "post_date": "12/15/2024 14:18:08", "content": "The model doesn’t “dare” to predict large values. This is due to the extreme long tailed target distribution. You can design a weighted loss to encourage the model to predict larger values, similar to what is done in a classification task using the Focal Loss. But this implies that you have put some prior knowledge or assumptions on the hidden test data distributions.
\nNice plot and visuals!
Thank you, this is fascinating. I'm used to dealing with classification task imbalances. I've never really worked with a regression target imbalance. I wonder, do similar techniques like over-sampling, under-sampling, different weights, apply for regression models?
", "votes": null, "replies": [ { "id": 3073275, "author_name": "serjhenrique", "author_url": "", "post_date": "12/16/2024 08:54:43", "content": "I was asking me exaclty the same question. I did some research powered by AI in the weekend and found that sampling, despite unusual, could be applied. Different weights can be applied also. Lastly, it suggested me this article. I am experimenting with it. For now, it has the same result as training without it.
", "votes": null, "replies": [] }, { "id": 3073285, "author_name": "shiyili", "author_url": "", "post_date": "12/16/2024 09:26:57", "content": "Yes, same techniques dealing with imbalance in classifications can also be applied to regression here. For example, you can bin the targets into ordered classes and resample from each class to get a more balanced dataset. But the trap here is that you have changed the inherent distribution of the data when doing so. It is worth to think if we really want that. My experiment in the beginning of the competition did not give me good results :(
\nBTW Thanks @serjhenrique for the article. It is a great source. I found it too in my early explorations but totally forgot it xD.
", "votes": null, "replies": [] } ] } ] }, { "id": 3072954, "author_name": "bradywynn", "author_url": "", "post_date": "12/15/2024 20:37:38", "content": "This is from a super simple mlp evaluated on symbol_id 1
\n
Okay, that's basically what I'm seeing. I've never had a predicted vs. actuals plot look so flat 😆 So my takeaway is it may not be a bug in my code, but just a poor model that's not able to learn anything? Looks like it's always predicting between -1 and 1. Almost seems like it could be a bug with the target being normalized and then not de-normalized. But I'm getting this kind of plot without modifying the target responder_6 at all.
", "votes": null, "replies": [] } ] }, { "id": 3073014, "author_name": "michaeltimbs", "author_url": "", "post_date": "12/15/2024 23:16:56", "content": "Every time I have tried to update my loss function to stop it being \"safe\" and keeping close to 0 I ended up getting worse results.
", "votes": null, "replies": [] }, { "id": 3073653, "author_name": "i2nfinit3y", "author_url": "", "post_date": "12/16/2024 17:41:04", "content": "
A xgb model trained with 800 date, cv 0.0080 (valid in the last 120 dates)
\nAs we can see, the prediction values are mostly in -1 ~ 1, and it is because most target data are between -1 and 1.