I was making a baseline with target means, playing with the predictions and metric but some parts didn't make sense.
\nPredictions are stacked on Normal/Mild, Moderate and Severe columns and log losses are calculated on each condition group, but probabilities are not normalized in a way that their sum would be 1. It was like that in last years competition but not here. Is it intentional?
\nI have doubts about any_severe_spinal implementation too. Labels, predictions and weights are max aggregated in study id_groups for spinal canal stenosis rows. Is it really supposed to be for only spinal canal stenosis rows or was that a mistake?
\nIf a study_id has one Moderate and four Normal/Mild labels for spinal canal stenosis, then the max weight becomes 2. Shouldn't the weights for any_severe_spinal be either 1 or 4 since the label name any_severe_spinal implies a binary prediction derived from other predictions. I don't think including Moderate labels on any_severe_spinal makes sense.
\nWhat is any_severe_scalar? I couldn't find its value.
", "messages": [ { "id": "2923910", "postDate": "07/16/2024 07:53:51", "content": "I was making a baseline with target means, playing with the predictions and metric but some parts didn't make sense.
\nPredictions are stacked on Normal/Mild, Moderate and Severe columns and log losses are calculated on each condition group, but probabilities are not normalized in a way that their sum would be 1. It was like that in last years competition but not here. Is it intentional?
\nI have doubts about any_severe_spinal implementation too. Labels, predictions and weights are max aggregated in study id_groups for spinal canal stenosis rows. Is it really supposed to be for only spinal canal stenosis rows or was that a mistake?
\nIf a study_id has one Moderate and four Normal/Mild labels for spinal canal stenosis, then the max weight becomes 2. Shouldn't the weights for any_severe_spinal be either 1 or 4 since the label name any_severe_spinal implies a binary prediction derived from other predictions. I don't think including Moderate labels on any_severe_spinal makes sense.
\nWhat is any_severe_scalar? I couldn't find its value.
", "rawMarkdown": "I was making a baseline with target means, playing with the predictions and metric but some parts didn't make sense.\n\nPredictions are stacked on Normal/Mild, Moderate and Severe columns and log losses are calculated on each condition group, but probabilities are not normalized in a way that their sum would be 1. It was like that in last years competition but not here. Is it intentional?\n\nI have doubts about any_severe_spinal implementation too. Labels, predictions and weights are max aggregated in study id_groups for spinal canal stenosis rows. Is it really supposed to be for only spinal canal stenosis rows or was that a mistake?\n\nIf a study_id has one Moderate and four Normal/Mild labels for spinal canal stenosis, then the max weight becomes 2. Shouldn't the weights for any_severe_spinal be either 1 or 4 since the label name any_severe_spinal implies a binary prediction derived from other predictions. I don't think including Moderate labels on any_severe_spinal makes sense.\n\nWhat is any_severe_scalar? I couldn't find its value.", "votes": null }, { "id": "2924012", "postDate": "07/16/2024 08:52:35", "content": "\n\nWhat is any_severe_scalar? I couldn't find its value.
\n
According to the Evaluation tab:
\n\n\nFor this competition, the any_severe_scalar has been set to 1.0.
\n
Edit:
\nMaybe this (formerly pinned) discussions would help you:
Thanks for the reply. Do you know if the probabilities are normalized or not?
", "rawMarkdown": "Thanks for the reply. Do you know if the probabilities are normalized or not?", "votes": null }, { "id": "2924964", "postDate": "07/16/2024 20:27:21", "content": "You can take a look at the metric code: https://www.kaggle.com/code/metric/rsna-lumbar-metric-71549
\nThe labels and predictions are directly inputted into sklearn.metrics.log_loss. Not sure if normalization occurs inside this function. If not, then no.
I believe sklearn.metrics.log_loss does normalize the probabilities to sum to 1 before calculating the loss. You can try this code snippet to confirm:
import numpy as np\nimport torch\n\n sklearn import log_loss\n\n\ndef torch_log_loss(, t):\n loss = -torch.(t.(), p.()).()\n return loss.()\n\n\ndef (x, y, eps=e-):\n return np.(x - y) < eps\n\n\n# No normalization\ny = torch.((, ))\ny[...] = \ny = torch.(y)\np = torch.((, ))\n\nsk_loss = (y.(), p.())\npt_loss = (p, y).()\n\n((sk_loss, pt_loss))\n\n# After normalization\np = p / p.().()\n\nsk_loss = (y.(), p.())\npt_loss = (p, y).()\n\n((sk_loss, pt_loss))\nThe values are only equal after you normalize the probabilities to sum to 1 (since torch_log_loss does not apply any normalization within the function).
Our metric is in a notebook that still uses v1.2.2, so it normalizes inputs. However, sklearn changed the normalization behavior in v1.3 so if you run a test off Kaggle you'll probably see the newer behavior: https://scikit-learn.org/stable/whats_new/v1.3.html.
Thanks, I was looking for this answer. @vaillant @sohier
", "rawMarkdown": "Thanks, I was looking for this answer. @vaillant @sohier", "votes": null }, { "id": "2930546", "postDate": "07/21/2024 06:02:50", "content": "I have a thought about any_severe_spinal weights implementation. Maybe for your example with moderate label, we have higher weight because it's harder to distinguish between moderate and severe case?
", "rawMarkdown": "I have a thought about any_severe_spinal weights implementation. Maybe for your example with moderate label, we have higher weight because it's harder to distinguish between moderate and severe case?", "votes": null }, { "id": "2931848", "postDate": "07/22/2024 11:51:54", "content": "I was using v1.3.2 when testing the code above, so it appears normalization still occurs.
", "rawMarkdown": "I was using v1.3.2 when testing the code above, so it appears normalization still occurs.", "votes": null } ], "comments": [ { "id": 2924012, "author_name": "coderrkj", "author_url": "", "post_date": "07/16/2024 08:52:35", "content": "\n\nWhat is any_severe_scalar? I couldn't find its value.
\n
According to the Evaluation tab:
\n\n\nFor this competition, the any_severe_scalar has been set to 1.0.
\n
Edit:
\nMaybe this (formerly pinned) discussions would help you:
Thanks for the reply. Do you know if the probabilities are normalized or not?
", "votes": null, "replies": [ { "id": 2924964, "author_name": "coderrkj", "author_url": "", "post_date": "07/16/2024 20:27:21", "content": "You can take a look at the metric code: https://www.kaggle.com/code/metric/rsna-lumbar-metric-71549
\nThe labels and predictions are directly inputted into sklearn.metrics.log_loss. Not sure if normalization occurs inside this function. If not, then no.
I believe sklearn.metrics.log_loss does normalize the probabilities to sum to 1 before calculating the loss. You can try this code snippet to confirm:
import numpy as np\nimport torch\n\n sklearn import log_loss\n\n\ndef torch_log_loss(, t):\n loss = -torch.(t.(), p.()).()\n return loss.()\n\n\ndef (x, y, eps=e-):\n return np.(x - y) < eps\n\n\n# No normalization\ny = torch.((, ))\ny[...] = \ny = torch.(y)\np = torch.((, ))\n\nsk_loss = (y.(), p.())\npt_loss = (p, y).()\n\n((sk_loss, pt_loss))\n\n# After normalization\np = p / p.().()\n\nsk_loss = (y.(), p.())\npt_loss = (p, y).()\n\n((sk_loss, pt_loss))\nThe values are only equal after you normalize the probabilities to sum to 1 (since torch_log_loss does not apply any normalization within the function).
Our metric is in a notebook that still uses v1.2.2, so it normalizes inputs. However, sklearn changed the normalization behavior in v1.3 so if you run a test off Kaggle you'll probably see the newer behavior: https://scikit-learn.org/stable/whats_new/v1.3.html.
Thanks, I was looking for this answer. @vaillant @sohier
", "votes": null, "replies": [ { "id": 2931848, "author_name": "vaillant", "author_url": "", "post_date": "07/22/2024 11:51:54", "content": "I was using v1.3.2 when testing the code above, so it appears normalization still occurs.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] }, { "id": 2930546, "author_name": "glipko", "author_url": "", "post_date": "07/21/2024 06:02:50", "content": "I have a thought about any_severe_spinal weights implementation. Maybe for your example with moderate label, we have higher weight because it's harder to distinguish between moderate and severe case?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2923910": "I was making a baseline with target means, playing with the predictions and metric but some parts didn't make sense.\n\nPredictions are stacked on Normal/Mild, Moderate and Severe columns and log losses are calculated on each condition group, but probabilities are not normalized in a way that their sum would be 1. It was like that in last years competition but not here. Is it intentional?\n\nI have doubts about any_severe_spinal implementation too. Labels, predictions and weights are max aggregated in study id_groups for spinal canal stenosis rows. Is it really supposed to be for only spinal canal stenosis rows or was that a mistake?\n\nIf a study_id has one Moderate and four Normal/Mild labels for spinal canal stenosis, then the max weight becomes 2. Shouldn't the weights for any_severe_spinal be either 1 or 4 since the label name any_severe_spinal implies a binary prediction derived from other predictions. I don't think including Moderate labels on any_severe_spinal makes sense.\n\nWhat is any_severe_scalar? I couldn't find its value.", "2924012": "> What is any_severe_scalar? I couldn't find its value.\n\nAccording to the [Evaluation](https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/overview/evaluation) tab:\n> For this competition, the any_severe_scalar has been set to 1.0.\n\nEdit:\nMaybe this (formerly pinned) discussions would help you:\n- https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/508522\n- https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/510363", "2924278": "Thanks for the reply. Do you know if the probabilities are normalized or not?", "2924964": "You can take a look at the metric code: https://www.kaggle.com/code/metric/rsna-lumbar-metric-71549\n\nThe labels and predictions are directly inputted into `sklearn.metrics.log_loss`. Not sure if normalization occurs inside this function. If not, then no.", "2929141": "I believe `sklearn.metrics.log_loss` does normalize the probabilities to sum to 1 before calculating the loss. You can try this code snippet to confirm:\n\n```\nimport numpy as np\nimport torch\n\nfrom sklearn.metrics import log_loss\n\n\ndef torch_log_loss(p, t):\n\tloss = -torch.xlogy(t.float(), p.float()).sum(1)\n\treturn loss.mean()\n\n\ndef check_equal(x, y, eps=1e-6):\n\treturn np.abs(x - y) < eps\n\n\n# No normalization\ny = torch.empty((1000, 3))\ny[...] = 0.5\ny = torch.bernoulli(y)\np = torch.rand((1000, 3))\n\nsk_loss = log_loss(y.numpy(), p.numpy())\npt_loss = torch_log_loss(p, y).item()\n\nprint(check_equal(sk_loss, pt_loss))\n\n# After normalization\np = p / p.sum(1).unsqueeze(1)\n\nsk_loss = log_loss(y.numpy(), p.numpy())\npt_loss = torch_log_loss(p, y).item()\n\nprint(check_equal(sk_loss, pt_loss))\n```\n\nThe values are only equal after you normalize the probabilities to sum to 1 (since `torch_log_loss` does not apply any normalization within the function).", "2929188": "Our metric is in a notebook that still uses v1.2.2, so it normalizes inputs. However, `sklearn` changed the normalization behavior in v1.3 so if you run a test off Kaggle you'll probably see the newer behavior: https://scikit-learn.org/stable/whats_new/v1.3.html.", "2929458": "Thanks, I was looking for this answer. @vaillant @sohier", "2930546": "I have a thought about any_severe_spinal weights implementation. Maybe for your example with moderate label, we have higher weight because it's harder to distinguish between moderate and severe case?", "2931848": "I was using v1.3.2 when testing the code above, so it appears normalization still occurs." }, "source": "meta" }