The Metric in Kaggle's RSNA 2024 Lumbar Spine Degenerative Classification. The code came from Kaggle
\nThe Metric in Kaggle's RSNA 2024 Lumbar Spine Degenerative Classification. The code came from Kaggle
\n\nhttps://www.kaggle.com/datasets/seoyunje/kaggle-rsna-2024-metric\n\nThe Metric in Kaggle's RSNA 2024 Lumbar Spine Degenerative Classification. The code came from Kaggle\n\n import sys\n sys.path.append('/kaggle/input/kaggle-rsna-2024-metric')\n from metric import score\n\n score = ('solution', 'submission','row_id_column_name', 'any_severe_scalar')\n score = (solution = true, submission = oof, row_id_column_name=id, any_severe_scalar: 1)",
"votes": null
},
{
"id": "2930712",
"postDate": "07/21/2024 10:39:30",
"content": "If your question is how to use the metric to see the CV score for a test run on train images, then use the following (first part is copied from https://www.kaggle.com/datasets/seoyunje/kaggle-rsna-2024-metric):
\n numpy np\n pandas pd\n pandas.api.types\n sklearn.metrics\n\n\n ():\n \n\n\n () -> :\n \n injury_condition [, , ]:\n injury_condition full_location:\n injury_condition\n ValueError()\n\n\n () -> :\n \n\n target_levels = [, , ]\n\n \n pandas.api.types.is_numeric_dtype(submission[target_levels].values):\n ParticipantVisibleError()\n\n np.isfinite(submission[target_levels].values).():\n ParticipantVisibleError()\n\n solution[target_levels].().() < :\n ParticipantVisibleError()\n submission[target_levels].().() < :\n ParticipantVisibleError()\n\n solution[] = solution[].apply( x: x.split()[])\n solution[] = solution[].apply( x: .join(x.split()[:]))\n solution[] = solution[].apply(get_condition)\n\n solution[row_id_column_name]\n submission[row_id_column_name]\n (submission.columns) == (target_levels)\n\n submission[] = solution[]\n submission[] = solution[]\n submission[] = solution[]\n\n condition_losses = []\n condition_weights = []\n condition [, , ]:\n condition_indices = solution.loc[solution[] == condition].index.values\n condition_loss = sklearn.metrics.log_loss(\n y_true=solution.loc[condition_indices, target_levels].values,\n y_pred=submission.loc[condition_indices, target_levels].values,\n sample_weight=solution.loc[condition_indices, ].values\n )\n condition_losses.append(condition_loss)\n condition_weights.append()\n\n any_severe_spinal_labels = pd.Series(solution.loc[solution[] == ].groupby()[].())\n any_severe_spinal_weights = pd.Series(solution.loc[solution[] == ].groupby()[].())\n any_severe_spinal_predictions = pd.Series(submission.loc[submission[] == ].groupby()[].())\n any_severe_spinal_loss = sklearn.metrics.log_loss(\n y_true=any_severe_spinal_labels,\n y_pred=any_severe_spinal_predictions,\n sample_weight=any_severe_spinal_weights\n )\n condition_losses.append(any_severe_spinal_loss)\n condition_weights.append(any_severe_scalar)\n np.average(condition_losses, weights=condition_weights)\n\n\n ():\n \n target_cols = (sample_weights.keys())\n pred = submission.copy() \n \n pred[target_cols] = pred[target_cols].div(pred[target_cols].(axis=), axis=)\n\n \n indexed_train_df = train_df.set_index(, verify_integrity=)\n\n row_ids = pred[row_id_column_name]\n study_ids = row_ids.apply( x: x.split()[])\n locations = row_ids.apply( x: .join(x.split()[:]))\n\n solution_data = np.zeros_like(pred[target_cols].values)\n sample_weight_list = []\n nan_row_ids = ()\n idx, (row, study_id, location) ((row_ids, study_ids, locations)):\n severity = (indexed_train_df.at[(study_id), location]).replace(, ).lower()\n severity sample_weights:\n solution_data[idx, target_cols.index(severity)] = \n sample_weight_list.append(sample_weights[severity])\n :\n solution_data[idx] = np.nan\n nan_row_ids.add(row)\n sample_weight_list.append(np.nan)\n\n solution = pd.DataFrame({\n row_id_column_name: pred[row_id_column_name],\n : sample_weight_list\n })\n solution[target_cols] = solution_data\n\n \n pred.loc[pred[row_id_column_name].isin(nan_row_ids), target_cols] = np.nan\n \n \n score(solution.dropna().copy(), pred.dropna().copy(), row_id_column_name, any_severe_scalar)\n
\nCreate a flag for testing on train images: FAKE_TEST = len(sample_sub) <= 25 this will ensure that during submission of inference notebook the actual test data is used.
\nIn your notebook swap path to images and other test data dataframes to train path and data. Example:
\n FAKE_TEST:\n df = pd.read_csv(DATA_PATH / )\n study_ids, base_path = (df[].unique()), DATA_PATH / \n (, (study_ids))\n
\nFinally, do the test after submission.csv is created:
\n FAKE_TEST:\n \n (score_from_train(pd.read_csv(), pd.read_csv(DATA_PATH / )))\n
\nHere are my public notebooks, where I have done this:
\n",
"rawMarkdown": "If your question is how to use the metric to see the CV score for a test run on train images, then use the following (first part is copied from https://www.kaggle.com/datasets/seoyunje/kaggle-rsna-2024-metric):\n\n```python\nimport numpy as np\nimport pandas as pd\nimport pandas.api.types\nimport sklearn.metrics\n\n\nclass ParticipantVisibleError(Exception):\n pass\n\n\ndef get_condition(full_location: str) -> str:\n # Given an input like spinal_canal_stenosis_l1_l2 extracts 'spinal'\n for injury_condition in ['spinal', 'foraminal', 'subarticular']:\n if injury_condition in full_location:\n return injury_condition\n raise ValueError(f'condition not found in {full_location}')\n\n\ndef score(\n solution: pd.DataFrame,\n submission: pd.DataFrame,\n row_id_column_name: str,\n any_severe_scalar: float\n ) -> float:\n '''\n Pseudocode:\n 1. Calculate the sample weighted log loss for each medical condition:\n 2. Derive a new any_severe label.\n 3. Calculate the sample weighted log loss for the new any_severe label.\n 4. Return the average of all of the label group log losses as the final score, normalized for the number of columns in each group.\n This mitigates the impact of spinal stenosis having only half as many columns as the other two conditions.\n '''\n\n target_levels = ['normal_mild', 'moderate', 'severe']\n\n # Run basic QC checks on the inputs\n if not pandas.api.types.is_numeric_dtype(submission[target_levels].values):\n raise ParticipantVisibleError('All submission values must be numeric')\n\n if not np.isfinite(submission[target_levels].values).all():\n raise ParticipantVisibleError('All submission values must be finite')\n\n if solution[target_levels].min().min() < 0:\n raise ParticipantVisibleError('All labels must be at least zero')\n if submission[target_levels].min().min() < 0:\n raise ParticipantVisibleError('All predictions must be at least zero')\n\n solution['study_id'] = solution['row_id'].apply(lambda x: x.split('_')[0])\n solution['location'] = solution['row_id'].apply(lambda x: '_'.join(x.split('_')[1:]))\n solution['condition'] = solution['row_id'].apply(get_condition)\n\n del solution[row_id_column_name]\n del submission[row_id_column_name]\n assert sorted(submission.columns) == sorted(target_levels)\n\n submission['study_id'] = solution['study_id']\n submission['location'] = solution['location']\n submission['condition'] = solution['condition']\n\n condition_losses = []\n condition_weights = []\n for condition in ['spinal', 'foraminal', 'subarticular']:\n condition_indices = solution.loc[solution['condition'] == condition].index.values\n condition_loss = sklearn.metrics.log_loss(\n y_true=solution.loc[condition_indices, target_levels].values,\n y_pred=submission.loc[condition_indices, target_levels].values,\n sample_weight=solution.loc[condition_indices, 'sample_weight'].values\n )\n condition_losses.append(condition_loss)\n condition_weights.append(1)\n\n any_severe_spinal_labels = pd.Series(solution.loc[solution['condition'] == 'spinal'].groupby('study_id')['severe'].max())\n any_severe_spinal_weights = pd.Series(solution.loc[solution['condition'] == 'spinal'].groupby('study_id')['sample_weight'].max())\n any_severe_spinal_predictions = pd.Series(submission.loc[submission['condition'] == 'spinal'].groupby('study_id')['severe'].max())\n any_severe_spinal_loss = sklearn.metrics.log_loss(\n y_true=any_severe_spinal_labels,\n y_pred=any_severe_spinal_predictions,\n sample_weight=any_severe_spinal_weights\n )\n condition_losses.append(any_severe_spinal_loss)\n condition_weights.append(any_severe_scalar)\n return np.average(condition_losses, weights=condition_weights)\n\n\ndef score_from_train(\n submission: pd.DataFrame, # Pass submission.csv as a DataFrame\n train_df: pd.DataFrame, # Pass train.csv as a DataFrame\n row_id_column_name=\"row_id\",\n any_severe_scalar=1.0,\n sample_weights: dict[str, int]={\"normal_mild\": 1, \"moderate\": 2, \"severe\": 4},\n):\n \"\"\"Convert train.csv to sample submission format with Nan values removed\"\"\"\n target_cols = list(sample_weights.keys())\n pred = submission.copy() # Copy to prevent changes in original\n # Normalize values to have a sum of 1.0\n pred[target_cols] = pred[target_cols].div(pred[target_cols].sum(axis=1), axis=0)\n\n # Index the study_id in train_df\n indexed_train_df = train_df.set_index(\"study_id\", verify_integrity=True)\n\n row_ids = pred[row_id_column_name]\n study_ids = row_ids.apply(lambda x: x.split('_')[0])\n locations = row_ids.apply(lambda x: '_'.join(x.split('_')[1:]))\n\n solution_data = np.zeros_like(pred[target_cols].values)\n sample_weight_list = []\n nan_row_ids = set()\n for idx, (row, study_id, location) in enumerate(zip(row_ids, study_ids, locations)):\n severity = str(indexed_train_df.at[int(study_id), location]).replace(\"/\", \"_\").lower()\n if severity in sample_weights:\n solution_data[idx, target_cols.index(severity)] = 1.0\n sample_weight_list.append(sample_weights[severity])\n else:\n solution_data[idx] = np.nan\n nan_row_ids.add(row)\n sample_weight_list.append(np.nan)\n\n solution = pd.DataFrame({\n row_id_column_name: pred[row_id_column_name],\n \"sample_weight\": sample_weight_list\n })\n solution[target_cols] = solution_data\n\n # Change row_ids in nan_row_ids to np.nan\n pred.loc[pred[row_id_column_name].isin(nan_row_ids), target_cols] = np.nan\n # Remove nan rows and pass copy to score function\n # score from https://www.kaggle.com/code/metric/rsna-lumbar-metric-71549?scriptVersionId=181722791 (Version 10)\n return score(solution.dropna().copy(), pred.dropna().copy(), row_id_column_name, any_severe_scalar)\n```\nCreate a flag for testing on train images: `FAKE_TEST = len(sample_sub) <= 25` this will ensure that during submission of inference notebook the actual test data is used.\n\nIn your notebook swap path to images and other test data dataframes to train path and data. Example:\n```python\nif FAKE_TEST:\n df = pd.read_csv(DATA_PATH / 'train_series_descriptions.csv')\n study_ids, base_path = list(df['study_id'].unique()), DATA_PATH / \"train_images\"\n print(f\"Num of study_ids:\", len(study_ids))\n```\n\nFinally, do the test after `submission.csv` is created:\n```python\nif FAKE_TEST:\n # Pass submission.csv dataframe and train.csv dataframe\n print(score_from_train(pd.read_csv(\"submission.csv\"), pd.read_csv(DATA_PATH / \"train.csv\")))\n```\n\nHere are my public notebooks, where I have done this:\n- https://www.kaggle.com/code/coderrkj/lumbar-competition-debugging-notebook\n- https://www.kaggle.com/coderrkj/rsna-pytorch-test-infer\n- https://www.kaggle.com/coderrkj/rsna2024-lsdc-submission-baseline",
"votes": null
}
],
"comments": [
{
"id": 2930712,
"author_name": "coderrkj",
"author_url": "",
"post_date": "07/21/2024 10:39:30",
"content": "If your question is how to use the metric to see the CV score for a test run on train images, then use the following (first part is copied from https://www.kaggle.com/datasets/seoyunje/kaggle-rsna-2024-metric):
\n numpy np\n pandas pd\n pandas.api.types\n sklearn.metrics\n\n\n ():\n \n\n\n () -> :\n \n injury_condition [, , ]:\n injury_condition full_location:\n injury_condition\n ValueError()\n\n\n () -> :\n \n\n target_levels = [, , ]\n\n \n pandas.api.types.is_numeric_dtype(submission[target_levels].values):\n ParticipantVisibleError()\n\n np.isfinite(submission[target_levels].values).():\n ParticipantVisibleError()\n\n solution[target_levels].().() < :\n ParticipantVisibleError()\n submission[target_levels].().() < :\n ParticipantVisibleError()\n\n solution[] = solution[].apply( x: x.split()[])\n solution[] = solution[].apply( x: .join(x.split()[:]))\n solution[] = solution[].apply(get_condition)\n\n solution[row_id_column_name]\n submission[row_id_column_name]\n (submission.columns) == (target_levels)\n\n submission[] = solution[]\n submission[] = solution[]\n submission[] = solution[]\n\n condition_losses = []\n condition_weights = []\n condition [, , ]:\n condition_indices = solution.loc[solution[] == condition].index.values\n condition_loss = sklearn.metrics.log_loss(\n y_true=solution.loc[condition_indices, target_levels].values,\n y_pred=submission.loc[condition_indices, target_levels].values,\n sample_weight=solution.loc[condition_indices, ].values\n )\n condition_losses.append(condition_loss)\n condition_weights.append()\n\n any_severe_spinal_labels = pd.Series(solution.loc[solution[] == ].groupby()[].())\n any_severe_spinal_weights = pd.Series(solution.loc[solution[] == ].groupby()[].())\n any_severe_spinal_predictions = pd.Series(submission.loc[submission[] == ].groupby()[].())\n any_severe_spinal_loss = sklearn.metrics.log_loss(\n y_true=any_severe_spinal_labels,\n y_pred=any_severe_spinal_predictions,\n sample_weight=any_severe_spinal_weights\n )\n condition_losses.append(any_severe_spinal_loss)\n condition_weights.append(any_severe_scalar)\n np.average(condition_losses, weights=condition_weights)\n\n\n ():\n \n target_cols = (sample_weights.keys())\n pred = submission.copy() \n \n pred[target_cols] = pred[target_cols].div(pred[target_cols].(axis=), axis=)\n\n \n indexed_train_df = train_df.set_index(, verify_integrity=)\n\n row_ids = pred[row_id_column_name]\n study_ids = row_ids.apply( x: x.split()[])\n locations = row_ids.apply( x: .join(x.split()[:]))\n\n solution_data = np.zeros_like(pred[target_cols].values)\n sample_weight_list = []\n nan_row_ids = ()\n idx, (row, study_id, location) ((row_ids, study_ids, locations)):\n severity = (indexed_train_df.at[(study_id), location]).replace(, ).lower()\n severity sample_weights:\n solution_data[idx, target_cols.index(severity)] = \n sample_weight_list.append(sample_weights[severity])\n :\n solution_data[idx] = np.nan\n nan_row_ids.add(row)\n sample_weight_list.append(np.nan)\n\n solution = pd.DataFrame({\n row_id_column_name: pred[row_id_column_name],\n : sample_weight_list\n })\n solution[target_cols] = solution_data\n\n \n pred.loc[pred[row_id_column_name].isin(nan_row_ids), target_cols] = np.nan\n \n \n score(solution.dropna().copy(), pred.dropna().copy(), row_id_column_name, any_severe_scalar)\n
\nCreate a flag for testing on train images: FAKE_TEST = len(sample_sub) <= 25 this will ensure that during submission of inference notebook the actual test data is used.
\nIn your notebook swap path to images and other test data dataframes to train path and data. Example:
\n FAKE_TEST:\n df = pd.read_csv(DATA_PATH / )\n study_ids, base_path = (df[].unique()), DATA_PATH / \n (, (study_ids))\n
\nFinally, do the test after submission.csv is created:
\n FAKE_TEST:\n \n (score_from_train(pd.read_csv(), pd.read_csv(DATA_PATH / )))\n
\nHere are my public notebooks, where I have done this:
\n",
"votes": null,
"replies": []
}
],
"raw_markdown_by_id": {
"2930403": "\nhttps://www.kaggle.com/datasets/seoyunje/kaggle-rsna-2024-metric\n\nThe Metric in Kaggle's RSNA 2024 Lumbar Spine Degenerative Classification. The code came from Kaggle\n\n import sys\n sys.path.append('/kaggle/input/kaggle-rsna-2024-metric')\n from metric import score\n\n score = ('solution', 'submission','row_id_column_name', 'any_severe_scalar')\n score = (solution = true, submission = oof, row_id_column_name=id, any_severe_scalar: 1)",
"2930712": "If your question is how to use the metric to see the CV score for a test run on train images, then use the following (first part is copied from https://www.kaggle.com/datasets/seoyunje/kaggle-rsna-2024-metric):\n\n```python\nimport numpy as np\nimport pandas as pd\nimport pandas.api.types\nimport sklearn.metrics\n\n\nclass ParticipantVisibleError(Exception):\n pass\n\n\ndef get_condition(full_location: str) -> str:\n # Given an input like spinal_canal_stenosis_l1_l2 extracts 'spinal'\n for injury_condition in ['spinal', 'foraminal', 'subarticular']:\n if injury_condition in full_location:\n return injury_condition\n raise ValueError(f'condition not found in {full_location}')\n\n\ndef score(\n solution: pd.DataFrame,\n submission: pd.DataFrame,\n row_id_column_name: str,\n any_severe_scalar: float\n ) -> float:\n '''\n Pseudocode:\n 1. Calculate the sample weighted log loss for each medical condition:\n 2. Derive a new any_severe label.\n 3. Calculate the sample weighted log loss for the new any_severe label.\n 4. Return the average of all of the label group log losses as the final score, normalized for the number of columns in each group.\n This mitigates the impact of spinal stenosis having only half as many columns as the other two conditions.\n '''\n\n target_levels = ['normal_mild', 'moderate', 'severe']\n\n # Run basic QC checks on the inputs\n if not pandas.api.types.is_numeric_dtype(submission[target_levels].values):\n raise ParticipantVisibleError('All submission values must be numeric')\n\n if not np.isfinite(submission[target_levels].values).all():\n raise ParticipantVisibleError('All submission values must be finite')\n\n if solution[target_levels].min().min() < 0:\n raise ParticipantVisibleError('All labels must be at least zero')\n if submission[target_levels].min().min() < 0:\n raise ParticipantVisibleError('All predictions must be at least zero')\n\n solution['study_id'] = solution['row_id'].apply(lambda x: x.split('_')[0])\n solution['location'] = solution['row_id'].apply(lambda x: '_'.join(x.split('_')[1:]))\n solution['condition'] = solution['row_id'].apply(get_condition)\n\n del solution[row_id_column_name]\n del submission[row_id_column_name]\n assert sorted(submission.columns) == sorted(target_levels)\n\n submission['study_id'] = solution['study_id']\n submission['location'] = solution['location']\n submission['condition'] = solution['condition']\n\n condition_losses = []\n condition_weights = []\n for condition in ['spinal', 'foraminal', 'subarticular']:\n condition_indices = solution.loc[solution['condition'] == condition].index.values\n condition_loss = sklearn.metrics.log_loss(\n y_true=solution.loc[condition_indices, target_levels].values,\n y_pred=submission.loc[condition_indices, target_levels].values,\n sample_weight=solution.loc[condition_indices, 'sample_weight'].values\n )\n condition_losses.append(condition_loss)\n condition_weights.append(1)\n\n any_severe_spinal_labels = pd.Series(solution.loc[solution['condition'] == 'spinal'].groupby('study_id')['severe'].max())\n any_severe_spinal_weights = pd.Series(solution.loc[solution['condition'] == 'spinal'].groupby('study_id')['sample_weight'].max())\n any_severe_spinal_predictions = pd.Series(submission.loc[submission['condition'] == 'spinal'].groupby('study_id')['severe'].max())\n any_severe_spinal_loss = sklearn.metrics.log_loss(\n y_true=any_severe_spinal_labels,\n y_pred=any_severe_spinal_predictions,\n sample_weight=any_severe_spinal_weights\n )\n condition_losses.append(any_severe_spinal_loss)\n condition_weights.append(any_severe_scalar)\n return np.average(condition_losses, weights=condition_weights)\n\n\ndef score_from_train(\n submission: pd.DataFrame, # Pass submission.csv as a DataFrame\n train_df: pd.DataFrame, # Pass train.csv as a DataFrame\n row_id_column_name=\"row_id\",\n any_severe_scalar=1.0,\n sample_weights: dict[str, int]={\"normal_mild\": 1, \"moderate\": 2, \"severe\": 4},\n):\n \"\"\"Convert train.csv to sample submission format with Nan values removed\"\"\"\n target_cols = list(sample_weights.keys())\n pred = submission.copy() # Copy to prevent changes in original\n # Normalize values to have a sum of 1.0\n pred[target_cols] = pred[target_cols].div(pred[target_cols].sum(axis=1), axis=0)\n\n # Index the study_id in train_df\n indexed_train_df = train_df.set_index(\"study_id\", verify_integrity=True)\n\n row_ids = pred[row_id_column_name]\n study_ids = row_ids.apply(lambda x: x.split('_')[0])\n locations = row_ids.apply(lambda x: '_'.join(x.split('_')[1:]))\n\n solution_data = np.zeros_like(pred[target_cols].values)\n sample_weight_list = []\n nan_row_ids = set()\n for idx, (row, study_id, location) in enumerate(zip(row_ids, study_ids, locations)):\n severity = str(indexed_train_df.at[int(study_id), location]).replace(\"/\", \"_\").lower()\n if severity in sample_weights:\n solution_data[idx, target_cols.index(severity)] = 1.0\n sample_weight_list.append(sample_weights[severity])\n else:\n solution_data[idx] = np.nan\n nan_row_ids.add(row)\n sample_weight_list.append(np.nan)\n\n solution = pd.DataFrame({\n row_id_column_name: pred[row_id_column_name],\n \"sample_weight\": sample_weight_list\n })\n solution[target_cols] = solution_data\n\n # Change row_ids in nan_row_ids to np.nan\n pred.loc[pred[row_id_column_name].isin(nan_row_ids), target_cols] = np.nan\n # Remove nan rows and pass copy to score function\n # score from https://www.kaggle.com/code/metric/rsna-lumbar-metric-71549?scriptVersionId=181722791 (Version 10)\n return score(solution.dropna().copy(), pred.dropna().copy(), row_id_column_name, any_severe_scalar)\n```\nCreate a flag for testing on train images: `FAKE_TEST = len(sample_sub) <= 25` this will ensure that during submission of inference notebook the actual test data is used.\n\nIn your notebook swap path to images and other test data dataframes to train path and data. Example:\n```python\nif FAKE_TEST:\n df = pd.read_csv(DATA_PATH / 'train_series_descriptions.csv')\n study_ids, base_path = list(df['study_id'].unique()), DATA_PATH / \"train_images\"\n print(f\"Num of study_ids:\", len(study_ids))\n```\n\nFinally, do the test after `submission.csv` is created:\n```python\nif FAKE_TEST:\n # Pass submission.csv dataframe and train.csv dataframe\n print(score_from_train(pd.read_csv(\"submission.csv\"), pd.read_csv(DATA_PATH / \"train.csv\")))\n```\n\nHere are my public notebooks, where I have done this:\n- https://www.kaggle.com/code/coderrkj/lumbar-competition-debugging-notebook\n- https://www.kaggle.com/coderrkj/rsna-pytorch-test-infer\n- https://www.kaggle.com/coderrkj/rsna2024-lsdc-submission-baseline"
},
"source": "meta"
}