Thank you to the hosts and Kaggle for this interesting competition. I have a question about the condition weights for the metric.
\ncondition_losses = \ncondition_weights = \n\n condition :\n condition_indices = solution == condition]\n condition_loss = sklearn(\n y_true=solution,\n y_pred=submission,\n sample_weight=solution\n )\n condition_losses(condition_loss)\n condition_weights( / solution())\nWhen I run this code block, I get condition_weights = [0.2, 0.1, 0.1], then appending any_severe_scalar, it becomes [0.2, 0.1, 0.1, 1.0].
This implies that spinal stenosis grading is worth twice as much as foraminal or subarticular stenosis grading, and that any_severe_spinal is worth 5x as much as spinal and 10x as much as foraminal or subarticular. I am not sure if this is the intended weighting, so I wanted to ask for clarification.
Thank you to the hosts and Kaggle for this interesting competition. I have a question about the condition weights for the metric.
\ncondition_losses = \ncondition_weights = \n\n condition :\n condition_indices = solution == condition]\n condition_loss = sklearn(\n y_true=solution,\n y_pred=submission,\n sample_weight=solution\n )\n condition_losses(condition_loss)\n condition_weights( / solution())\nWhen I run this code block, I get condition_weights = [0.2, 0.1, 0.1], then appending any_severe_scalar, it becomes [0.2, 0.1, 0.1, 1.0].
This implies that spinal stenosis grading is worth twice as much as foraminal or subarticular stenosis grading, and that any_severe_spinal is worth 5x as much as spinal and 10x as much as foraminal or subarticular. I am not sure if this is the intended weighting, so I wanted to ask for clarification.
This is easier to think about if you flip to a wide format, with one column per condition & location pair. If you do that, you'll end up with five spinal columns, ten each for foraminal and subarticular, and one for any_severe_spinal. There are twice as many foraminal and subarticular columns as those conditions consider separate diagnoses for the left and right sides of the spine. The overall effect is that each of those major categories has the same impact in aggregate.
", "rawMarkdown": "This is easier to think about if you flip to a wide format, with one column per condition & location pair. If you do that, you'll end up with five spinal columns, ten each for foraminal and subarticular, and one for any_severe_spinal. There are twice as many foraminal and subarticular columns as those conditions consider separate diagnoses for the left and right sides of the spine. The overall effect is that each of those major categories has the same impact in aggregate.", "votes": null }, { "id": "2848250", "postDate": "05/31/2024 23:40:24", "content": "If condition_loss is the weighted mean log loss across all of the samples, doesn't that remove the effect of having right and left sides for foraminal and subarticular stenoses? Since for those categories, we are averaging across both right and left, if I understand the code correctly.
Yes, but removing the effect of having left and right sides is the intent.
", "rawMarkdown": "Yes, but removing the effect of having left and right sides is the intent.", "votes": null }, { "id": "2849168", "postDate": "06/01/2024 12:53:10", "content": "What I meant was by averaging across the both right and left sides, we have already removed that effect, so there is no need for different weightings.
\nThe log loss as calculated reflects the average loss across all samples in that condition. Let's say there were 1000 studies with 5 levels, so 5000 for spinal, 10000 for foraminal, 10000 for subarticular, and 1000 for spinal.
\nWe are then taking the average log loss across each of the 5000 spinal, 10000 foraminal, 10000 subarticular, and 1000 spinal values, resulting in 1 loss value for each. This weighted loss value does not take into consideration, for example, that there are twice as many foraminal values than spinal values - it just reflects the average loss value across all of them.
\nSo there is no need to weight them differently, if the goal of the weighting is to account for the differences
\nin the number of levels and sides.
If the code was written as:
\nfor condition in ['spinal', 'right_foraminal', 'right_subarticular', 'left_foraminal', 'left_subarticular']:
Then I would understand why foraminal and subarticular would be weighted half as much. Similarly if this were further expanded by level, I would see why any severe spinal would be weighted 5x more than spinal.
", "rawMarkdown": "What I meant was by averaging across the both right and left sides, we have already removed that effect, so there is no need for different weightings.\n\nThe log loss as calculated reflects the average loss across all samples in that condition. Let's say there were 1000 studies with 5 levels, so 5000 for spinal, 10000 for foraminal, 10000 for subarticular, and 1000 for spinal.\n\nWe are then taking the average log loss across each of the 5000 spinal, 10000 foraminal, 10000 subarticular, and 1000 spinal values, resulting in 1 loss value for each. This weighted loss value does not take into consideration, for example, that there are twice as many foraminal values than spinal values - it just reflects the average loss value across all of them. \n\nSo there is no need to weight them differently, if the goal of the weighting is to account for the differences \nin the number of levels and sides.\n\nIf the code was written as:\n\n`for condition in ['spinal', 'right_foraminal', 'right_subarticular', 'left_foraminal', 'left_subarticular']:`\n\nThen I would understand why foraminal and subarticular would be weighted half as much. Similarly if this were further expanded by level, I would see why any severe spinal would be weighted 5x more than spinal.", "votes": null }, { "id": "2851041", "postDate": "06/02/2024 14:15:05", "content": "Within the condition_loss for loop:
Therefore, since the weights for both sides have already been averaged, if we consider the weight from 1 / solution.loc[condition_indices, 'location'].nunique(), does this mean that the conditions for foraminal and subarticular, which are twice as many as spinal, will be further halved.
Correct?
", "rawMarkdown": "Within the `condition_loss` for loop:\n\n1. The average log loss for spinal is calculated.\n2. The average log loss for foraminal is calculated (**this is already averaged across both sides at this point**).\n3. The average log loss for subarticular is calculated (**this too is already averaged across both sides**).\n\nTherefore, since the weights for both sides have already been averaged, if we consider the weight from `1 / solution.loc[condition_indices, 'location'].nunique()`, does this mean that the conditions for foraminal and subarticular, which are twice as many as spinal, will be further halved. \n\nCorrect?", "votes": null }, { "id": "2852663", "postDate": "06/03/2024 12:01:41", "content": "That is my understanding with the current metric.
", "rawMarkdown": "That is my understanding with the current metric.", "votes": null }, { "id": "2853805", "postDate": "06/04/2024 01:32:47", "content": "I also agree.
\nWhat are your thoughts on this? @sohier
Thank you for flagging this, Ian. I see what you mean now and have posted a more detailed response here: https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/510363
", "rawMarkdown": "Thank you for flagging this, Ian. I see what you mean now and have posted a more detailed response here: https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/510363", "votes": null }, { "id": "2857475", "postDate": "06/05/2024 22:28:05", "content": "Thanks for the response and quick turnaround!
", "rawMarkdown": "Thanks for the response and quick turnaround!", "votes": null }, { "id": "2857505", "postDate": "06/05/2024 23:40:08", "content": "Thank you for raising the issue.
\nIn short, am I correct in saying that we have gone from a total of 5 averages of spinal cord, 2 foraminal, and 2 subarticular to a total of 3 averages of spinal cord, foraminal, and subarticular?
Thanks for pointing out this issue
", "rawMarkdown": "Thanks for pointing out this issue", "votes": null }, { "id": "2860542", "postDate": "06/07/2024 16:43:14", "content": "@kosukekita
\nIn my understanding,
any_severe_scalar is set to 1, the overall final score is simply the average of these four losses.@YYama
\nThank you for your instruction.
\nI'm gonna update the loss function in my local environment.
@vaillant can you update on what are new condition weights after 2nd patch of metric ?
", "rawMarkdown": "vaillant can you update on what are new condition weights after 2nd patch of metric ?", "votes": null } ], "comments": [ { "id": 2848243, "author_name": "sohier", "author_url": "", "post_date": "05/31/2024 23:33:36", "content": "This is easier to think about if you flip to a wide format, with one column per condition & location pair. If you do that, you'll end up with five spinal columns, ten each for foraminal and subarticular, and one for any_severe_spinal. There are twice as many foraminal and subarticular columns as those conditions consider separate diagnoses for the left and right sides of the spine. The overall effect is that each of those major categories has the same impact in aggregate.
", "votes": null, "replies": [ { "id": 2848250, "author_name": "vaillant", "author_url": "", "post_date": "05/31/2024 23:40:24", "content": "If condition_loss is the weighted mean log loss across all of the samples, doesn't that remove the effect of having right and left sides for foraminal and subarticular stenoses? Since for those categories, we are averaging across both right and left, if I understand the code correctly.
Yes, but removing the effect of having left and right sides is the intent.
", "votes": null, "replies": [ { "id": 2849168, "author_name": "vaillant", "author_url": "", "post_date": "06/01/2024 12:53:10", "content": "What I meant was by averaging across the both right and left sides, we have already removed that effect, so there is no need for different weightings.
\nThe log loss as calculated reflects the average loss across all samples in that condition. Let's say there were 1000 studies with 5 levels, so 5000 for spinal, 10000 for foraminal, 10000 for subarticular, and 1000 for spinal.
\nWe are then taking the average log loss across each of the 5000 spinal, 10000 foraminal, 10000 subarticular, and 1000 spinal values, resulting in 1 loss value for each. This weighted loss value does not take into consideration, for example, that there are twice as many foraminal values than spinal values - it just reflects the average loss value across all of them.
\nSo there is no need to weight them differently, if the goal of the weighting is to account for the differences
\nin the number of levels and sides.
If the code was written as:
\nfor condition in ['spinal', 'right_foraminal', 'right_subarticular', 'left_foraminal', 'left_subarticular']:
Then I would understand why foraminal and subarticular would be weighted half as much. Similarly if this were further expanded by level, I would see why any severe spinal would be weighted 5x more than spinal.
", "votes": null, "replies": [ { "id": 2851041, "author_name": "yosukeyama", "author_url": "", "post_date": "06/02/2024 14:15:05", "content": "Within the condition_loss for loop:
Therefore, since the weights for both sides have already been averaged, if we consider the weight from 1 / solution.loc[condition_indices, 'location'].nunique(), does this mean that the conditions for foraminal and subarticular, which are twice as many as spinal, will be further halved.
Correct?
", "votes": null, "replies": [ { "id": 2852663, "author_name": "vaillant", "author_url": "", "post_date": "06/03/2024 12:01:41", "content": "That is my understanding with the current metric.
", "votes": null, "replies": [ { "id": 2853805, "author_name": "yosukeyama", "author_url": "", "post_date": "06/04/2024 01:32:47", "content": "I also agree.
\nWhat are your thoughts on this? @sohier
Thank you for flagging this, Ian. I see what you mean now and have posted a more detailed response here: https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/510363
", "votes": null, "replies": [ { "id": 2857475, "author_name": "vaillant", "author_url": "", "post_date": "06/05/2024 22:28:05", "content": "Thanks for the response and quick turnaround!
", "votes": null, "replies": [ { "id": 2858859, "author_name": "manaidu", "author_url": "", "post_date": "06/06/2024 17:18:32", "content": "Thanks for pointing out this issue
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2857505, "author_name": "kosukekita", "author_url": "", "post_date": "06/05/2024 23:40:08", "content": "Thank you for raising the issue.
\nIn short, am I correct in saying that we have gone from a total of 5 averages of spinal cord, 2 foraminal, and 2 subarticular to a total of 3 averages of spinal cord, foraminal, and subarticular?
@kosukekita
\nIn my understanding,
any_severe_scalar is set to 1, the overall final score is simply the average of these four losses.@YYama
\nThank you for your instruction.
\nI'm gonna update the loss function in my local environment.
@vaillant can you update on what are new condition weights after 2nd patch of metric ?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2848237": "Thank you to the hosts and Kaggle for this interesting competition. I have a question about the condition weights for the metric.\n\n```\ncondition_losses = []\ncondition_weights = []\n\nfor condition in ['spinal', 'foraminal', 'subarticular']:\n condition_indices = solution.loc[solution['condition'] == condition].index.values\n condition_loss = sklearn.metrics.log_loss(\n y_true=solution.loc[condition_indices, target_levels].values,\n y_pred=submission.loc[condition_indices, target_levels].values,\n sample_weight=solution.loc[condition_indices, 'sample_weight'].values\n )\n condition_losses.append(condition_loss)\n condition_weights.append(1 / solution.loc[condition_indices, 'location'].nunique())\n```\n\nWhen I run this code block, I get `condition_weights = [0.2, 0.1, 0.1]`, then appending `any_severe_scalar`, it becomes `[0.2, 0.1, 0.1, 1.0]`.\n\nThis implies that spinal stenosis grading is worth twice as much as foraminal or subarticular stenosis grading, and that `any_severe_spinal` is worth 5x as much as spinal and 10x as much as foraminal or subarticular. I am not sure if this is the intended weighting, so I wanted to ask for clarification.", "2848243": "This is easier to think about if you flip to a wide format, with one column per condition & location pair. If you do that, you'll end up with five spinal columns, ten each for foraminal and subarticular, and one for any_severe_spinal. There are twice as many foraminal and subarticular columns as those conditions consider separate diagnoses for the left and right sides of the spine. The overall effect is that each of those major categories has the same impact in aggregate.", "2848250": "If `condition_loss` is the weighted mean log loss across all of the samples, doesn't that remove the effect of having right and left sides for foraminal and subarticular stenoses? Since for those categories, we are averaging across both right and left, if I understand the code correctly.", "2848450": "Yes, but removing the effect of having left and right sides is the intent.", "2849168": "What I meant was by averaging across the both right and left sides, we have already removed that effect, so there is no need for different weightings.\n\nThe log loss as calculated reflects the average loss across all samples in that condition. Let's say there were 1000 studies with 5 levels, so 5000 for spinal, 10000 for foraminal, 10000 for subarticular, and 1000 for spinal.\n\nWe are then taking the average log loss across each of the 5000 spinal, 10000 foraminal, 10000 subarticular, and 1000 spinal values, resulting in 1 loss value for each. This weighted loss value does not take into consideration, for example, that there are twice as many foraminal values than spinal values - it just reflects the average loss value across all of them. \n\nSo there is no need to weight them differently, if the goal of the weighting is to account for the differences \nin the number of levels and sides.\n\nIf the code was written as:\n\n`for condition in ['spinal', 'right_foraminal', 'right_subarticular', 'left_foraminal', 'left_subarticular']:`\n\nThen I would understand why foraminal and subarticular would be weighted half as much. Similarly if this were further expanded by level, I would see why any severe spinal would be weighted 5x more than spinal.", "2851041": "Within the `condition_loss` for loop:\n\n1. The average log loss for spinal is calculated.\n2. The average log loss for foraminal is calculated (**this is already averaged across both sides at this point**).\n3. The average log loss for subarticular is calculated (**this too is already averaged across both sides**).\n\nTherefore, since the weights for both sides have already been averaged, if we consider the weight from `1 / solution.loc[condition_indices, 'location'].nunique()`, does this mean that the conditions for foraminal and subarticular, which are twice as many as spinal, will be further halved. \n\nCorrect?", "2852663": "That is my understanding with the current metric.", "2853805": "I also agree.\nWhat are your thoughts on this? @sohier", "2857471": "Thank you for flagging this, Ian. I see what you mean now and have posted a more detailed response here: https://www.kaggle.com/competitions/rsna-2024-lumbar-spine-degenerative-classification/discussion/510363", "2857475": "Thanks for the response and quick turnaround!", "2857505": "Thank you for raising the issue. \nIn short, am I correct in saying that we have gone from a total of 5 averages of spinal cord, 2 foraminal, and 2 subarticular to a total of 3 averages of spinal cord, foraminal, and subarticular?", "2858859": "Thanks for pointing out this issue", "2860542": "kosukekita \nIn my understanding,\n\n1. The calculation starts by calculating the weighted averages for 'spinal', 'foraminal', and 'subarticular', where each category is weighted according to its severity. (we get \"weighted mean loss of spinal (n=5 per case)\", \"weighted mean loss of foraminal (n=10 per case)\", \"weighted mean loss of subarticular (n=10 per case)\")\n2. Additionally, the weighted average is calculated for any severe spinal conditions. (\"weighted loss of any severe spinal\")\n3. As the `any_severe_scalar` is set to 1, the overall final score is simply the average of these four losses.", "2860952": "YYama\nThank you for your instruction.\nI'm gonna update the loss function in my local environment.", "2892841": "vaillant can you update on what are new condition weights after 2nd patch of metric ?" }, "source": "meta" }