Hello community, I got 0.99 on ndcg@20 for orders re-ranking, which I find very suspicious. Is it because the number of orders are relatively lower than the number of clicks ?
\nAlso, my click reranker performed well ( 0.54 vs 0.5255 benchmark ). However, when using the same gbdt with the same variables for the orders re-ranker, the performance got worse ( 0.61 vs 0.64 benchmark ).
\nDo you guys use the same parameters and variables for clicks / orders / carts ?
\nDoes the model overfit more quickly when trying to predict orders regarding the low proportion of real candidates vs false candidates ?
Hello community, I got 0.99 on ndcg@20 for orders re-ranking, which I find very suspicious. Is it because the number of orders are relatively lower than the number of clicks ?
\nAlso, my click reranker performed well ( 0.54 vs 0.5255 benchmark ). However, when using the same gbdt with the same variables for the orders re-ranker, the performance got worse ( 0.61 vs 0.64 benchmark ).
\nDo you guys use the same parameters and variables for clicks / orders / carts ?
\nDoes the model overfit more quickly when trying to predict orders regarding the low proportion of real candidates vs false candidates ?
My ndcg@20 for clicks and carts are near 0.86 using 20 trees, for orders it's 0.91. I haven't gotten chance to make a submission with reranking yet. Probably submit one today. Quite busy with the work this year…
", "rawMarkdown": "My ndcg@20 for clicks and carts are near 0.86 using 20 trees, for orders it's 0.91. I haven't gotten chance to make a submission with reranking yet. Probably submit one today. Quite busy with the work this year...", "votes": null }, { "id": "2099887", "postDate": "01/14/2023 18:37:32", "content": "The ultra high ndcg comes from the negtive sample downsampling. Assuming your sampling ratio positive:negtive = 1:20, then recall at 20 means collect 20 samples from 21 samples for each session, therefore even a random guess can reach 90%+ hit rate. For the sake of a fair evalution, put the 'noise' (all negtive samples in your valid fold) back and inference. You will find your model cannot performs that good anymore under such large amount of nosie. You will get the true performance of your ranker model.
", "rawMarkdown": "The ultra high ndcg comes from the negtive sample downsampling. Assuming your sampling ratio positive:negtive = 1:20, then recall at 20 means collect 20 samples from 21 samples for each session, therefore even a random guess can reach 90%+ hit rate. For the sake of a fair evalution, put the 'noise' (all negtive samples in your valid fold) back and inference. You will find your model cannot performs that good anymore under such large amount of nosie. You will get the true performance of your ranker model.", "votes": null }, { "id": "2099923", "postDate": "01/14/2023 19:12:52", "content": "Insightful ! If I well understood you suggest to avoid downsampling negative sample in validation, in order to get a fair evaluation.
", "rawMarkdown": "Insightful ! If I well understood you suggest to avoid downsampling negative sample in validation, in order to get a fair evaluation.", "votes": null }, { "id": "2099963", "postDate": "01/14/2023 19:47:25", "content": "Yes, especially if you use early stopping, ndcg@20 in sessions with only average 21 samples is not a good choice.
", "rawMarkdown": "Yes, especially if you use early stopping, ndcg@20 in sessions with only average 21 samples is not a good choice.", "votes": null }, { "id": "2100513", "postDate": "01/15/2023 08:35:01", "content": "I also face the same problem (ncdg around .95 for carts and orders) even though I didn't downsample. Any ideas from where this comes from?
", "rawMarkdown": "I also face the same problem (ncdg around .95 for carts and orders) even though I didn't downsample. Any ideas from where this comes from?", "votes": null }, { "id": "2100522", "postDate": "01/15/2023 08:39:31", "content": "How does your model generalize performance to test set, namely LB? If it is much lower than your validation, then one of some most posible reasons might be there is some target leakage during your feature generation.
", "rawMarkdown": "How does your model generalize performance to test set, namely LB? If it is much lower than your validation, then one of some most posible reasons might be there is some target leakage during your feature generation.", "votes": null }, { "id": "2100525", "postDate": "01/15/2023 08:41:26", "content": "My CV is 0.552. My LB is 0.568
", "rawMarkdown": "My CV is 0.552. My LB is 0.568", "votes": null }, { "id": "2100528", "postDate": "01/15/2023 08:44:41", "content": "seems correct. Then how about your average session length? If your retrival items per session is very low, ndcg@20 would be high.
", "rawMarkdown": "seems correct. Then how about your average session length? If your retrival items per session is very low, ndcg@20 would be high.", "votes": null }, { "id": "2100530", "postDate": "01/15/2023 08:44:59", "content": "The CV scores for clicks are comparable to my Co visitation matrix approach which scores 0.577. But the carts and orders recall much lower..
\nIf you could explain that to me I would be grateful..
Currently I use 20 candidates per session. I will try more now. But in my Intuition that doesn't explain the extremly high scores for carts and orders.
", "rawMarkdown": "Currently I use 20 candidates per session. I will try more now. But in my Intuition that doesn't explain the extremly high scores for carts and orders.", "votes": null }, { "id": "2100541", "postDate": "01/15/2023 08:50:24", "content": "So that's the reason… You did not downsample, but your session length is so short which equals to a downsampling effect from whatever total number to only 20.
", "rawMarkdown": "So that's the reason... You did not downsample, but your session length is so short which equals to a downsampling effect from whatever total number to only 20.", "votes": null }, { "id": "2100558", "postDate": "01/15/2023 08:59:24", "content": "Thank you so much! I will fix that and then report how I improve (hopefully)
", "rawMarkdown": "Thank you so much! I will fix that and then report how I improve (hopefully)", "votes": null }, { "id": "2102120", "postDate": "01/16/2023 12:05:22", "content": "Unfortunately with 100 candidates CV score got worse and the issue didn't resolve (high score for order and carts.. )
", "rawMarkdown": "Unfortunately with 100 candidates CV score got worse and the issue didn't resolve (high score for order and carts.. )", "votes": null }, { "id": "2118596", "postDate": "01/28/2023 06:15:51", "content": "May I ask how do you fix this issue later? And how do you do neg sample?
", "rawMarkdown": "May I ask how do you fix this issue later? And how do you do neg sample?", "votes": null } ], "comments": [ { "id": 2097833, "author_name": "wuwenmin", "author_url": "", "post_date": "01/13/2023 01:04:41", "content": "My ndcg@20 for clicks and carts are near 0.86 using 20 trees, for orders it's 0.91. I haven't gotten chance to make a submission with reranking yet. Probably submit one today. Quite busy with the work this year…
", "votes": null, "replies": [] }, { "id": 2099887, "author_name": "buumoo", "author_url": "", "post_date": "01/14/2023 18:37:32", "content": "The ultra high ndcg comes from the negtive sample downsampling. Assuming your sampling ratio positive:negtive = 1:20, then recall at 20 means collect 20 samples from 21 samples for each session, therefore even a random guess can reach 90%+ hit rate. For the sake of a fair evalution, put the 'noise' (all negtive samples in your valid fold) back and inference. You will find your model cannot performs that good anymore under such large amount of nosie. You will get the true performance of your ranker model.
", "votes": null, "replies": [ { "id": 2099923, "author_name": "rayanaay", "author_url": "", "post_date": "01/14/2023 19:12:52", "content": "Insightful ! If I well understood you suggest to avoid downsampling negative sample in validation, in order to get a fair evaluation.
", "votes": null, "replies": [ { "id": 2099963, "author_name": "buumoo", "author_url": "", "post_date": "01/14/2023 19:47:25", "content": "Yes, especially if you use early stopping, ndcg@20 in sessions with only average 21 samples is not a good choice.
", "votes": null, "replies": [ { "id": 2100513, "author_name": "simonveitner", "author_url": "", "post_date": "01/15/2023 08:35:01", "content": "I also face the same problem (ncdg around .95 for carts and orders) even though I didn't downsample. Any ideas from where this comes from?
", "votes": null, "replies": [ { "id": 2100522, "author_name": "buumoo", "author_url": "", "post_date": "01/15/2023 08:39:31", "content": "How does your model generalize performance to test set, namely LB? If it is much lower than your validation, then one of some most posible reasons might be there is some target leakage during your feature generation.
", "votes": null, "replies": [ { "id": 2100525, "author_name": "simonveitner", "author_url": "", "post_date": "01/15/2023 08:41:26", "content": "My CV is 0.552. My LB is 0.568
", "votes": null, "replies": [ { "id": 2100528, "author_name": "buumoo", "author_url": "", "post_date": "01/15/2023 08:44:41", "content": "seems correct. Then how about your average session length? If your retrival items per session is very low, ndcg@20 would be high.
", "votes": null, "replies": [ { "id": 2100533, "author_name": "simonveitner", "author_url": "", "post_date": "01/15/2023 08:46:26", "content": "Currently I use 20 candidates per session. I will try more now. But in my Intuition that doesn't explain the extremly high scores for carts and orders.
", "votes": null, "replies": [ { "id": 2100541, "author_name": "buumoo", "author_url": "", "post_date": "01/15/2023 08:50:24", "content": "So that's the reason… You did not downsample, but your session length is so short which equals to a downsampling effect from whatever total number to only 20.
", "votes": null, "replies": [ { "id": 2100558, "author_name": "simonveitner", "author_url": "", "post_date": "01/15/2023 08:59:24", "content": "Thank you so much! I will fix that and then report how I improve (hopefully)
", "votes": null, "replies": [ { "id": 2102120, "author_name": "simonveitner", "author_url": "", "post_date": "01/16/2023 12:05:22", "content": "Unfortunately with 100 candidates CV score got worse and the issue didn't resolve (high score for order and carts.. )
", "votes": null, "replies": [ { "id": 2118596, "author_name": "yuzhang0422", "author_url": "", "post_date": "01/28/2023 06:15:51", "content": "May I ask how do you fix this issue later? And how do you do neg sample?
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2100530, "author_name": "simonveitner", "author_url": "", "post_date": "01/15/2023 08:44:59", "content": "The CV scores for clicks are comparable to my Co visitation matrix approach which scores 0.577. But the carts and orders recall much lower..
\nIf you could explain that to me I would be grateful..