I was conducting some experiments and noticed a peculiar behavior when using MSE as my evaluation metric. Specifically, when I shuffled the features, the MSE only increased by around 15% at most. In contrast, R² dropped significantly, as expected. This small change in MSE didn’t make sense to me, so I thoroughly checked my code for bugs but found none.
\nCould someone explain why MSE behaves this way? Why doesn't it reflect the significant drop in model performance as clearly as R²?
", "messages": [ { "id": 3045244, "postDate": "2024-11-14T10:10:05.377Z", "content": "Let's assume that all your predictions are very close to zero, i.e., the mean of the target. Assume then, sum(w*y^2)=1 and sum(w*(y-y')^2)=0.99, Then R^2 = 1-0.99/1 = 0.01.
\nBut sum(w*(y-y')^2)=0.99 is proportional to weighted MSE. So assume your MSE_new = 1.15*MSE_old (changed by 15%). So now new(sum(w*(y-y')^2)) = 1.15*0.99 = 1.1385. Leading to:
\nnew(R^2) = 1-1.1385/1 = -0.1385.
\nYour MSE changed only by 15%, but R^2 changed from 0.01 (1st place on public LB) to -0.1385. There.
The metric here is modified R2score, so I am unsure if trying to optimize anything else is the right approach @aymanallawi
", "rawMarkdown": "The metric here is modified R2score, so I am unsure if trying to optimize anything else is the right approach @aymanallawi ", "votes": 1, "replies": [ { "id": 3045105, "postDate": "2024-11-14T07:08:02.700Z", "content": "You're totally right it isn't the right approach but I'm just curious why MSE is unrepresentative here. So, I just want to know for the sake of knowledge
", "rawMarkdown": "You're totally right it isn't the right approach but I'm just curious why MSE is unrepresentative here. So, I just want to know for the sake of knowledge ", "votes": 1, "replies": [ { "id": 3045119, "postDate": "2024-11-14T07:25:39.960Z", "content": "MSE is not in scope for this competition, so discussing it is kind-of futile @aymanallawi
", "rawMarkdown": "MSE is not in scope for this competition, so discussing it is kind-of futile @aymanallawi " } ] }, { "id": 3045234, "postDate": "2024-11-14T09:49:01.500Z", "content": "Trying to optimize 'anything else' is always a right approach.
\nLEAP was with r-squared metric, and I won it thanks to optimizing MAE.
\nAlways try to tinker with the loss. You would be surprised how much it can be effective.
I was conducting some experiments and noticed a peculiar behavior when using MSE as my evaluation metric. Specifically, when I shuffled the features, the MSE only increased by around 15% at most. In contrast, R² dropped significantly, as expected. This small change in MSE didn’t make sense to me, so I thoroughly checked my code for bugs but found none.
\nCould someone explain why MSE behaves this way? Why doesn't it reflect the significant drop in model performance as clearly as R²?
", "rawMarkdown": "I was conducting some experiments and noticed a peculiar behavior when using MSE as my evaluation metric. Specifically, when I shuffled the features, the MSE only increased by around 15% at most. In contrast, R² dropped significantly, as expected. This small change in MSE didn’t make sense to me, so I thoroughly checked my code for bugs but found none.\n\nCould someone explain why MSE behaves this way? Why doesn't it reflect the significant drop in model performance as clearly as R²?", "votes": 2 }, { "id": 3044921, "postDate": "2024-11-14T01:16:39.147Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3045244, "author_name": "greySnow", "author_url": "", "post_date": "2024-11-14T10:10:05.377000", "content": "Let's assume that all your predictions are very close to zero, i.e., the mean of the target. Assume then, sum(w*y^2)=1 and sum(w*(y-y')^2)=0.99, Then R^2 = 1-0.99/1 = 0.01.
\nBut sum(w*(y-y')^2)=0.99 is proportional to weighted MSE. So assume your MSE_new = 1.15*MSE_old (changed by 15%). So now new(sum(w*(y-y')^2)) = 1.15*0.99 = 1.1385. Leading to:
\nnew(R^2) = 1-1.1385/1 = -0.1385.
\nYour MSE changed only by 15%, but R^2 changed from 0.01 (1st place on public LB) to -0.1385. There.
The metric here is modified R2score, so I am unsure if trying to optimize anything else is the right approach @aymanallawi
", "votes": 1, "replies": [ { "id": 3045105, "author_name": "Ayman Allawi", "author_url": "", "post_date": "2024-11-14T07:08:02.700000", "content": "You're totally right it isn't the right approach but I'm just curious why MSE is unrepresentative here. So, I just want to know for the sake of knowledge
", "votes": 1, "replies": [ { "id": 3045119, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-11-14T07:25:39.960000", "content": "MSE is not in scope for this competition, so discussing it is kind-of futile @aymanallawi
", "votes": 0, "replies": [] } ] }, { "id": 3045234, "author_name": "greySnow", "author_url": "", "post_date": "2024-11-14T09:49:01.500000", "content": "Trying to optimize 'anything else' is always a right approach.
\nLEAP was with r-squared metric, and I won it thanks to optimizing MAE.
\nAlways try to tinker with the loss. You would be surprised how much it can be effective.