![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2666506%2F1489668b50466a28a904dbc3ef1dc0af%2FCapture%20decran%202023-01-09%20a%201.36.28%20PM.png?generation=1673268188825032&alt=media\")
Hello community, I'm training a lightgbmRanker I've got those results:
As you can see, my NDCG@20 increases while the binary loss is increasing. I took care of:
\nI also noticed that I have a feature that has an excessive amount of gain, which is the \"already_see\" feature.
\nThis feature equals 1 if the user has already interacts with the candidate or not.
\nam I the only one who got this ?
", "messages": [ { "id": "2092598", "postDate": "01/09/2023 12:47:40", "content": "Hello community, I'm training a lightgbmRanker I've got those results:
As you can see, my NDCG@20 increases while the binary loss is increasing. I took care of:
\nI also noticed that I have a feature that has an excessive amount of gain, which is the \"already_see\" feature.
\nThis feature equals 1 if the user has already interacts with the candidate or not.
\nam I the only one who got this ?
", "rawMarkdown": "Hello community, I'm training a lightgbmRanker I've got those results:\n\nAs you can see, my NDCG@20 increases while the binary loss is increasing. I took care of:\n\n- Using DART booster.\n- Using lambda_l1 / lambda_l2.\n- Reduce the learning rate.\n- Reduce the max_depth.\n- Increase the min_sum_hessian_in_leaf.\n\n\nI also noticed that I have a feature that has an excessive amount of gain, which is the \"**already_see**\" feature. \n\nThis feature equals 1 if the user has already interacts with the candidate or not.\n\nam I the only one who got this ?", "votes": null }, { "id": "2093113", "postDate": "01/09/2023 20:09:41", "content": "What is the objective function (lambdarank etc)? Since binary log loss is calculated point-wise while NDCG is calculated list-wise, it can be happen where one increase while others decrease.
\nAlso, did you shuffle the order of train/validation dataset before training? If you didn’t, and the trainin/validation set has pattern in order (e.g. positive set appears before negative set) the model got may get extremely high score when it always outputs constant score. That is because evaluation metrics is sensitive to the order of the dataset when the models output is constant.
Thank you @tatamikenn for all those useful informations.
\nYes I use lambdarank as objective function.
Yes I shuffled the data, and I use a subsample of the negative labels. However, I noticed that > 40% of my sessions has no candidates ( all its targets == 0 ). Can this cause a problem ?
\n\nYes I use lambdarank as objective function.
\n
Since lambdarank objective is calculated pair-wise, NDCG should be more correlated with it comparing to the binary logloss. So, I think the shared situation (binary logloss increase while NDCG increase) could be happened.
\nConsider the following two cases (case 1 and 2). While binary logloss is lower in case 1. However, the NDCG is also loser in case 1 because the rank of the prediction is more accurate in case 2. To summarize, binary logloss only evaluates the sample point(event)-wise, while NDCG evaluates the sample list(session)-wise.
\nCase1: binary logloss is lower (0.6596) while NDCG is lower (0.888)
\n| session | \nlabel | \npred | \n
|---|---|---|
| 0 | \n1 | \n0.60 | \n
| 0 | \n1 | \n0.48 | \n
| 0 | \n0 | \n0.52 | \n
Case2: binary logloss is higher (0.9263) while NDCG is higher (1.0)
\n| session | \nlabel | \npred | \n
|---|---|---|
| 0 | \n1 | \n0.30 | \n
| 0 | \n1 | \n0.23 | \n
| 0 | \n0 | \n0.1 | \n
I think binary logloss is not appropriate evaluation metrics for ranking models. That is because when evaluating ranking models, the rank of the predicted score is more important than the predicted scores itself.
", "rawMarkdown": "I think binary logloss is not appropriate evaluation metrics for ranking models. That is because when evaluating ranking models, the **rank** of the predicted score is more important than the predicted scores itself.", "votes": null }, { "id": "2093197", "postDate": "01/09/2023 21:19:27", "content": "\n\nHowever, I noticed that > 40% of my sessions has no candidates ( all its targets == 0 ). Can this cause a problem ?
\n
I think this is irrelevant. Although NDCG is not properly defined when all candidates are zero (since the maximum DCG is 0), it should be constant if it is defined.
", "rawMarkdown": "> However, I noticed that > 40% of my sessions has no candidates ( all its targets == 0 ). Can this cause a problem ?\n\nI think this is irrelevant. Although NDCG is not properly defined when all candidates are zero (since the maximum DCG is 0), it should be constant if it is defined.", "votes": null } ], "comments": [ { "id": 2093113, "author_name": "tatamikenn", "author_url": "", "post_date": "01/09/2023 20:09:41", "content": "What is the objective function (lambdarank etc)? Since binary log loss is calculated point-wise while NDCG is calculated list-wise, it can be happen where one increase while others decrease.
\nAlso, did you shuffle the order of train/validation dataset before training? If you didn’t, and the trainin/validation set has pattern in order (e.g. positive set appears before negative set) the model got may get extremely high score when it always outputs constant score. That is because evaluation metrics is sensitive to the order of the dataset when the models output is constant.
Thank you @tatamikenn for all those useful informations.
\nYes I use lambdarank as objective function.
Yes I shuffled the data, and I use a subsample of the negative labels. However, I noticed that > 40% of my sessions has no candidates ( all its targets == 0 ). Can this cause a problem ?
\n\nYes I use lambdarank as objective function.
\n
Since lambdarank objective is calculated pair-wise, NDCG should be more correlated with it comparing to the binary logloss. So, I think the shared situation (binary logloss increase while NDCG increase) could be happened.
\nConsider the following two cases (case 1 and 2). While binary logloss is lower in case 1. However, the NDCG is also loser in case 1 because the rank of the prediction is more accurate in case 2. To summarize, binary logloss only evaluates the sample point(event)-wise, while NDCG evaluates the sample list(session)-wise.
\nCase1: binary logloss is lower (0.6596) while NDCG is lower (0.888)
\n| session | \nlabel | \npred | \n
|---|---|---|
| 0 | \n1 | \n0.60 | \n
| 0 | \n1 | \n0.48 | \n
| 0 | \n0 | \n0.52 | \n
Case2: binary logloss is higher (0.9263) while NDCG is higher (1.0)
\n| session | \nlabel | \npred | \n
|---|---|---|
| 0 | \n1 | \n0.30 | \n
| 0 | \n1 | \n0.23 | \n
| 0 | \n0 | \n0.1 | \n
I think binary logloss is not appropriate evaluation metrics for ranking models. That is because when evaluating ranking models, the rank of the predicted score is more important than the predicted scores itself.
", "votes": null, "replies": [ { "id": 2093197, "author_name": "tatamikenn", "author_url": "", "post_date": "01/09/2023 21:19:27", "content": "\n\nHowever, I noticed that > 40% of my sessions has no candidates ( all its targets == 0 ). Can this cause a problem ?
\n
I think this is irrelevant. Although NDCG is not properly defined when all candidates are zero (since the maximum DCG is 0), it should be constant if it is defined.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2092598": "Hello community, I'm training a lightgbmRanker I've got those results:\n\nAs you can see, my NDCG@20 increases while the binary loss is increasing. I took care of:\n\n- Using DART booster.\n- Using lambda_l1 / lambda_l2.\n- Reduce the learning rate.\n- Reduce the max_depth.\n- Increase the min_sum_hessian_in_leaf.\n\n\nI also noticed that I have a feature that has an excessive amount of gain, which is the \"**already_see**\" feature. \n\nThis feature equals 1 if the user has already interacts with the candidate or not.\n\nam I the only one who got this ?", "2093113": "What is the objective function (lambdarank etc)? Since binary log loss is calculated point-wise while NDCG is calculated list-wise, it can be happen where one increase while others decrease.\nAlso, did you shuffle the order of train/validation dataset before training? If you didn’t, and the trainin/validation set has pattern in order (e.g. positive set appears before negative set) the model got may get extremely high score when it always outputs constant score. That is because evaluation metrics is sensitive to the order of the dataset when the models output is constant.", "2093145": "Thank you @tatamikenn for all those useful informations.\n1. Yes I use lambdarank as objective function.\n\n2. Yes I shuffled the data, and I use a subsample of the negative labels. However, I noticed that > 40% of my sessions has no candidates ( all its targets == 0 ). Can this cause a problem ?", "2093160": "> Yes I use lambdarank as objective function.\n\nSince lambdarank objective is calculated pair-wise, NDCG should be more correlated with it comparing to the binary logloss. So, I think the shared situation (binary logloss increase while NDCG increase) could be happened.\n\nConsider the following two cases (case 1 and 2). While binary logloss is lower in case 1. However, the NDCG is also loser in case 1 because the rank of the prediction is more accurate in case 2. To summarize, binary logloss only evaluates the sample point(event)-wise, while NDCG evaluates the sample list(session)-wise.\n\nCase1: binary logloss is lower (0.6596) while NDCG is lower (0.888)\n\n| session | label | pred |\n|--------:|-------|------|\n| 0 | 1 | 0.60 |\n| 0 | 1 | 0.48 |\n| 0 | 0 | 0.52 |\n\nCase2: binary logloss is higher (0.9263) while NDCG is higher (1.0)\n\n| session | label | pred |\n|--------:|-------|------|\n| 0 | 1 | 0.30 |\n| 0 | 1 | 0.23 |\n| 0 | 0 | 0.1 |", "2093165": "I think binary logloss is not appropriate evaluation metrics for ranking models. That is because when evaluating ranking models, the **rank** of the predicted score is more important than the predicted scores itself.", "2093197": "> However, I noticed that > 40% of my sessions has no candidates ( all its targets == 0 ). Can this cause a problem ?\n\nI think this is irrelevant. Although NDCG is not properly defined when all candidates are zero (since the maximum DCG is 0), it should be constant if it is defined." }, "source": "meta" }