{ "id": 520308, "title": "Reflexion about evaluation metric", "url": "/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/520308", "author_name": "", "post_date": "2024-07-15T15:31:05.829787400Z", "votes": null, "comment_count": 4, "views": 0, "content": "

The evaluation metric for this challenge is the log-loss :
\n

\n

For instance, if the ground truth is (1, 0, 0) then the loss is the same if the predicted probability vector is (0, 1, 0) or (0, 0, 1).

\n

In our case, we want to predict severity of a pathology.
\nClinically, it's better to predict \"Moderate\" instead of \"Normal\" if the patient has a \"Severe\" case of a given pathology.

\n

In my opinion, the evaluation metric is not so good, isn't it ?
\nLet me know if you have any idea why we would like to chose this metric instead of a rank metric for instance.

", "messages": [ { "id": "2922921", "postDate": "07/15/2024 15:31:05", "content": "

The evaluation metric for this challenge is the log-loss :
\n

\n

For instance, if the ground truth is (1, 0, 0) then the loss is the same if the predicted probability vector is (0, 1, 0) or (0, 0, 1).

\n

In our case, we want to predict severity of a pathology.
\nClinically, it's better to predict \"Moderate\" instead of \"Normal\" if the patient has a \"Severe\" case of a given pathology.

\n

In my opinion, the evaluation metric is not so good, isn't it ?
\nLet me know if you have any idea why we would like to chose this metric instead of a rank metric for instance.

", "rawMarkdown": "The evaluation metric for this challenge is the log-loss :\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10848256%2F9556d2a27ae49b7d8b05eb2e21f0d8e0%2Flogloss.png?generation=1721057116028683&alt=media)\n\nFor instance, if the ground truth is (1, 0, 0) then the loss is the same if the predicted probability vector is (0, 1, 0) or (0, 0, 1). \n\nIn our case, we want to predict severity of a pathology. \nClinically, it's better to predict \"Moderate\" instead of \"Normal\" if the patient has a \"Severe\" case of a given pathology. \n\nIn my opinion, the evaluation metric is not so good, isn't it ? \nLet me know if you have any idea why we would like to chose this metric instead of a rank metric for instance.", "votes": null }, { "id": "2923323", "postDate": "07/15/2024 19:30:03", "content": "

There is weighting in the metric, the loss is not the same for (0, 1, 0) and (0, 0, 1) in your example.

", "rawMarkdown": "There is weighting in the metric, the loss is not the same for (0, 1, 0) and (0, 0, 1) in your example.", "votes": null }, { "id": "2923730", "postDate": "07/16/2024 05:51:31", "content": "

I think evaluation matric do best scoring compare to because weighted log loss predict better for severe case compare to normal rank matric.

", "rawMarkdown": "I think evaluation matric do best scoring compare to because weighted log loss predict better for severe case compare to normal rank matric.", "votes": null }, { "id": "2926110", "postDate": "07/17/2024 14:45:33", "content": "

Thank you, that's clearer. Do you know where can I find the metric weights ? (I didn't find them)

", "rawMarkdown": "Thank you, that's clearer. Do you know where can I find the metric weights ? (I didn't find them)", "votes": null }, { "id": "2926272", "postDate": "07/17/2024 16:17:54", "content": "

You will find everything in the Overview tab, Evaluation section.

", "rawMarkdown": "You will find everything in the Overview tab, Evaluation section.", "votes": null } ], "comments": [ { "id": 2923323, "author_name": "adamnarai", "author_url": "", "post_date": "07/15/2024 19:30:03", "content": "

There is weighting in the metric, the loss is not the same for (0, 1, 0) and (0, 0, 1) in your example.

", "votes": null, "replies": [ { "id": 2926110, "author_name": "simonqueric", "author_url": "", "post_date": "07/17/2024 14:45:33", "content": "

Thank you, that's clearer. Do you know where can I find the metric weights ? (I didn't find them)

", "votes": null, "replies": [ { "id": 2926272, "author_name": "adamnarai", "author_url": "", "post_date": "07/17/2024 16:17:54", "content": "

You will find everything in the Overview tab, Evaluation section.

", "votes": null, "replies": [] } ] } ] }, { "id": 2923730, "author_name": "sahilverma96", "author_url": "", "post_date": "07/16/2024 05:51:31", "content": "

I think evaluation matric do best scoring compare to because weighted log loss predict better for severe case compare to normal rank matric.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2922921": "The evaluation metric for this challenge is the log-loss :\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10848256%2F9556d2a27ae49b7d8b05eb2e21f0d8e0%2Flogloss.png?generation=1721057116028683&alt=media)\n\nFor instance, if the ground truth is (1, 0, 0) then the loss is the same if the predicted probability vector is (0, 1, 0) or (0, 0, 1). \n\nIn our case, we want to predict severity of a pathology. \nClinically, it's better to predict \"Moderate\" instead of \"Normal\" if the patient has a \"Severe\" case of a given pathology. \n\nIn my opinion, the evaluation metric is not so good, isn't it ? \nLet me know if you have any idea why we would like to chose this metric instead of a rank metric for instance.", "2923323": "There is weighting in the metric, the loss is not the same for (0, 1, 0) and (0, 0, 1) in your example.", "2923730": "I think evaluation matric do best scoring compare to because weighted log loss predict better for severe case compare to normal rank matric.", "2926110": "Thank you, that's clearer. Do you know where can I find the metric weights ? (I didn't find them)", "2926272": "You will find everything in the Overview tab, Evaluation section." }, "source": "meta" }