I have experienced a leakage when training ranking model: the order of dataset.
\nSuppose if your dataset has pattern in your dataset (e.g. positive set appears before negative set for each session), the model get extremely high NDCG while the model outputs the constant value for any input features.
\nE.g. assume that your evaluating algorithm selects the first 2 items for each sessions, and calculate Recall@2 in the below dataset (Table 1). You will get maximum recall@2 on the dataset even if your model always outputs a constant score (1.0).
\nTable 1: example of leaked dataset (the positive set appears before negative set).
\n| session | \nfeat1 | \nfeat2 | \nlabel | \npred score | \n
|---|---|---|---|---|
| 0 | \n0 | \nx | \n1 | \n1.0 | \n
| 0 | \n1 | \nx | \n1 | \n1.0 | \n
| 0 | \n0 | \nx | \n0 | \n1.0 | \n
| 1 | \n1 | \nx | \n1 | \n1.0 | \n
| 1 | \n0 | \nx | \n0 | \n1.0 | \n
| 1 | \n0 | \nx | \n0 | \n1.0 | \n
To avoid these kind of leaks,
\nExample code of shuffling:
\n ():\n df[] = np.random.randn((df))\n df = df.sort_values([, , ])\n df = df.drop(, axis=)\n df\nI have experienced a leakage when training ranking model: the order of dataset.
\nSuppose if your dataset has pattern in your dataset (e.g. positive set appears before negative set for each session), the model get extremely high NDCG while the model outputs the constant value for any input features.
\nE.g. assume that your evaluating algorithm selects the first 2 items for each sessions, and calculate Recall@2 in the below dataset (Table 1). You will get maximum recall@2 on the dataset even if your model always outputs a constant score (1.0).
\nTable 1: example of leaked dataset (the positive set appears before negative set).
\n| session | \nfeat1 | \nfeat2 | \nlabel | \npred score | \n
|---|---|---|---|---|
| 0 | \n0 | \nx | \n1 | \n1.0 | \n
| 0 | \n1 | \nx | \n1 | \n1.0 | \n
| 0 | \n0 | \nx | \n0 | \n1.0 | \n
| 1 | \n1 | \nx | \n1 | \n1.0 | \n
| 1 | \n0 | \nx | \n0 | \n1.0 | \n
| 1 | \n0 | \nx | \n0 | \n1.0 | \n
To avoid these kind of leaks,
\nExample code of shuffling:
\n ():\n df[] = np.random.randn((df))\n df = df.sort_values([, , ])\n df = df.drop(, axis=)\n df\nThank you mate!!!But will it be working for all the models?
", "rawMarkdown": "Thank you mate!!!But will it be working for all the models?", "replies": [ { "id": 2094265, "postDate": "2023-01-10T17:20:57.510Z", "content": "\n\nBut will it be working for all the models?
\n
Sorry, I don't understand what you mean.
\nHowever, this thread may be more informative:
\nhttps://www.kaggle.com/competitions/otto-recommender-system/discussion/377094#2093481
Very informative and insightful. Thanks !
\nWe should also print the AUC score to help detect this kind of problem earlier !
Sorry, I don't understand. How AUC score help detect this kind of leakage?
\nI think AUC score also have problem with tie score.
Anyway, I noticed this leakage with extremely hight NDCG (~1.00) with very simple model (always predicts a constant score).
", "rawMarkdown": "Anyway, I noticed this leakage with extremely hight NDCG (~1.00) with very simple model (always predicts a constant score)." } ] } ] }, { "id": 2093474, "postDate": "2023-01-10T03:19:15.580Z", "content": "Does it also affect pairwise-rank model?
", "rawMarkdown": "Does it also affect pairwise-rank model? ", "replies": [ { "id": 2093481, "postDate": "2023-01-10T03:24:03.543Z", "content": "It could be. Actually it does not depend on the model itself, but how we evaluate the metrics (more precisely, how we select the tie-scored items).
\nE.g. assume that your evaluating algorithm selects the first 2 items for each sessions, and calculate Recall@2 in the above example. You will get maximum recall@2 on the dataset.
\nIf you randomly choose N items instead of the first N items, you can also avoid the leakage.
", "rawMarkdown": "It could be. Actually it does not depend on the model itself, but how we evaluate the metrics (more precisely, how we select the tie-scored items).\n\nE.g. assume that your evaluating algorithm selects the first 2 items for each sessions, and calculate Recall@2 in the above example. You will get maximum recall@2 on the dataset.\n\nIf you randomly choose N items instead of the first N items, you can also avoid the leakage.", "replies": [ { "id": 2093485, "postDate": "2023-01-10T03:33:42.527Z", "rawMarkdown": "", "isDeleted": true } ] } ] } ], "comments": [ { "id": 2094263, "author_name": "Senapati Rajesh", "author_url": "", "post_date": "2023-01-10T17:19:40.583000", "content": "Thank you mate!!!But will it be working for all the models?
", "votes": 0, "replies": [ { "id": 2094265, "author_name": "Bilzard", "author_url": "", "post_date": "2023-01-10T17:20:57.510000", "content": "\n\nBut will it be working for all the models?
\n
Sorry, I don't understand what you mean.
\nHowever, this thread may be more informative:
\nhttps://www.kaggle.com/competitions/otto-recommender-system/discussion/377094#2093481
Very informative and insightful. Thanks !
\nWe should also print the AUC score to help detect this kind of problem earlier !
Sorry, I don't understand. How AUC score help detect this kind of leakage?
\nI think AUC score also have problem with tie score.
Anyway, I noticed this leakage with extremely hight NDCG (~1.00) with very simple model (always predicts a constant score).
", "votes": 0, "replies": [] } ] } ] }, { "id": 2093474, "author_name": "EeyoreLee", "author_url": "", "post_date": "2023-01-10T03:19:15.580000", "content": "Does it also affect pairwise-rank model?
", "votes": 0, "replies": [ { "id": 2093481, "author_name": "Bilzard", "author_url": "", "post_date": "2023-01-10T03:24:03.543000", "content": "It could be. Actually it does not depend on the model itself, but how we evaluate the metrics (more precisely, how we select the tie-scored items).
\nE.g. assume that your evaluating algorithm selects the first 2 items for each sessions, and calculate Recall@2 in the above example. You will get maximum recall@2 on the dataset.
\nIf you randomly choose N items instead of the first N items, you can also avoid the leakage.
", "votes": 0, "replies": [ { "id": 2093485, "author_name": "", "author_url": "", "post_date": "2023-01-10T03:33:42.527000", "content": "", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2093149": "I have experienced a leakage when training ranking model: the order of dataset.\n\n## How the leakage occurs?\n\nSuppose if your dataset has pattern in your dataset (e.g. positive set appears before negative set for each session), the model get extremely high NDCG while the model outputs the constant value for any input features.\n\nE.g. assume that your evaluating algorithm selects the first 2 items for each sessions, and calculate Recall@2 in the below dataset (Table 1). You will get maximum recall@2 on the dataset even if your model always outputs a constant score (1.0).\n\nTable 1: example of leaked dataset (the positive set appears before negative set).\n\n| session | feat1 | feat2 | label | pred score |\n|:-------:|:-----:|:-----:|:-----:|------|\n| 0 | 0 | x | 1 | 1.0 |\n| 0 | 1 | x | 1 | 1.0 |\n| 0 | 0 | x | 0 | 1.0 |\n| 1 | 1 | x | 1 | 1.0 |\n| 1 | 0 | x | 0 | 1.0 |\n| 1 | 0 | x | 0 | 1.0 |\n\n## How to avoid the leak\n\nTo avoid these kind of leaks, \n\n1. Shuffle the dataset before training/evaluating\n2. add tiny amount of noise to the model's predicted score\n\nExample code of shuffling:\n\n```python\ndef shuffle_keep_session(df):\n df[\"_noise\"] = np.random.randn(len(df))\n df = df.sort_values([\"type\", \"session\", \"_noise\"])\n df = df.drop(\"_noise\", axis=1)\n return df\n```\n", "2094263": "Thank you mate!!!But will it be working for all the models?", "2094240": "Very informative and insightful. Thanks !\nWe should also print the AUC score to help detect this kind of problem earlier ! ", "2093474": "Does it also affect pairwise-rank model? " } }