{ "id": 368732, "title": "How do you train Ranking model?", "url": "/competitions/otto-recommender-system/discussion/368732", "author_name": "", "post_date": "2022-11-27T12:13:49.563888600Z", "votes": 14, "comment_count": 10, "views": 0, "content": "

I'm trying to build ranking model using following methods, but it's not working well.

\n
\n

Dataset & CV Strategy

\n \n

Candidate

\n \n

Features

\n \n

Model

\n \n
\n

How do you build models and datasets for training?
\nIn particular, I would like to know about the following two points

\n
    \n
  1. Which period of data is used for the training labels?
    2. \n
  2. Do you do any sampling for training?
    3. \n
\n

I'd be glad to know.

", "messages": [ { "id": "2045520", "postDate": "11/27/2022 12:13:49", "content": "

I'm trying to build ranking model using following methods, but it's not working well.

\n
\n

Dataset & CV Strategy

\n \n

Candidate

\n \n

Features

\n \n

Model

\n \n
\n

How do you build models and datasets for training?
\nIn particular, I would like to know about the following two points

\n
    \n
  1. Which period of data is used for the training labels?
    2. \n
  2. Do you do any sampling for training?
    3. \n
\n

I'd be glad to know.

", "rawMarkdown": "I'm trying to build ranking model using following methods, but it's not working well.\n\n> **Dataset & CV Strategy**\n- Use 2w and 3w labels as training data.\n(22-08-07 ~ 22-08-21)\n- Use 4w labels as validation data.\n(22-08-21 ~ 22-08-28)\n- Training with 10% of total session.\n- Create candidates and features by transaction data prior to the week of the label.\n\n> **Candidate**\n- Past event candidates\n- Co-visitation matrix candidates\n- popular item candidates\n\n> **Features**\n- aid_features, session_features, session×aid_features \nI refer to this [thread](https://www.kaggle.com/code/cdeotte/candidate-rerank-model-lb-0-575/comments#2030893). \n\n> **Model**\n- LGBM Classifier（training models for each event）\n\nHow do you build models and datasets for training?\nIn particular, I would like to know about the following two points\n\n1. Which period of data is used for the training labels?\n2. Do you do any sampling for training?\n\nI'd be glad to know.", "votes": null }, { "id": "2045622", "postDate": "11/27/2022 14:14:40", "content": "

How many candidates do you generate for each user? What are your hit rates for clicks,carts and orders. I think you maybe have a problem for candidate generation. I use word2vector,bpr,itemcf for candidate generation except past event candidates which all have low hit rates.

", "rawMarkdown": "How many candidates do you generate for each user? What are your hit rates for clicks,carts and orders. I think you maybe have a problem for candidate generation. I use word2vector,bpr,itemcf for candidate generation except past event candidates which all have low hit rates.", "votes": null }, { "id": "2045625", "postDate": "11/27/2022 14:19:05", "content": "

i use the approach shared by radek in his notebook.

", "rawMarkdown": "i use the approach shared by radek in his notebook.", "votes": null }, { "id": "2045960", "postDate": "11/27/2022 20:11:37", "content": "

Thanks. I'm sure there will be some useful information in comment thread. I will take a look.

", "rawMarkdown": "Thanks. I'm sure there will be some useful information in comment thread. I will take a look.", "votes": null }, { "id": "2045966", "postDate": "11/27/2022 20:23:40", "content": "

Also the associated threads point to useful techniques and Tricks.

", "rawMarkdown": "Also the associated threads point to useful techniques and Tricks.", "votes": null }, { "id": "2045968", "postDate": "11/27/2022 20:25:49", "content": "

Data for 4w (22-08-21 ~ 22-08-28) are as follows.

\n
\n

session count: 685.496（Randomly sampled 10%.）
\n candidates count mean: 96.6
\n click hit rate: 0.005
\n cart hit rate: 0.0015
\n order hit rate: 0.0012

\n
\n

Is it lower than your hit rate?

", "rawMarkdown": "Data for 4w (22-08-21 ~ 22-08-28) are as follows.\n\n> session count: 685.496（Randomly sampled 10%.）\ncandidates count mean: 96.6\nclick hit rate: 0.005\ncart hit rate: 0.0015\norder hit rate: 0.0012\n\nIs it lower than your hit rate?", "votes": null }, { "id": "2045983", "postDate": "11/27/2022 20:42:49", "content": "

Could you report your recall@20 per clicks/carts/orders

", "rawMarkdown": "Could you report your recall@20 per clicks/carts/orders", "votes": null }, { "id": "2045992", "postDate": "11/27/2022 21:01:42", "content": "

CV scores of 4w(22-08-21 ~ 22-08-28) are as follows.
\nHowever, LB score predicted by this model was 0.562

\n

The LB/CV correlation is not good, so I suspect there is a problem with the training data.

\n
\n

clicks: 0.5091
\ncarts: 0.4105
\norders: 0.6499

\n

Score: 0.5640

", "rawMarkdown": "CV scores of 4w(22-08-21 ~ 22-08-28) are as follows.\nHowever, LB score predicted by this model was 0.562\n\nThe LB/CV correlation is not good, so I suspect there is a problem with the training data.\n\n-------------------\nclicks: 0.5091\ncarts: 0.4105\norders: 0.6499\n\nScore: 0.5640", "votes": null }, { "id": "2046019", "postDate": "11/27/2022 21:17:49", "content": "

For now I have worse CV but I guess this is mostly due to the fact that I can just take top10 candiates from Co visitation matrix as I get memory error otherwise. May I ask how you incoporate canditates from Co visitation. For now I generate and than concat the resulting dataframe with canditates and afterwards count how many times each item was suggested by co visitation matrix. I guess this is not smart way and I could largely improve CV if I can take more canditates.. may I ask how you do that step? Maybe I should stop the counting of suggestion as that is where I get memory error. I will try.
\nIf I get Fürther improvement in CV I will report here. There is many features one can think of i didnt include so far, so there is long way to go.
\nMy current CV:
\nclicks recall = 0.5020045182833257
\ncarts recall = 0.4034401767965001
\norders recall = 0.6254552302403743

\n

Overall Recall = 0.5465056430115072

", "rawMarkdown": "For now I have worse CV but I guess this is mostly due to the fact that I can just take top10 candiates from Co visitation matrix as I get memory error otherwise. May I ask how you incoporate canditates from Co visitation. For now I generate and than concat the resulting dataframe with canditates and afterwards count how many times each item was suggested by co visitation matrix. I guess this is not smart way and I could largely improve CV if I can take more canditates.. may I ask how you do that step? Maybe I should stop the counting of suggestion as that is where I get memory error. I will try.\nIf I get Fürther improvement in CV I will report here. There is many features one can think of i didnt include so far, so there is long way to go.\nMy current CV:\nclicks recall = 0.5020045182833257\ncarts recall = 0.4034401767965001\norders recall = 0.6254552302403743\n\nOverall Recall = 0.5465056430115072", "votes": null }, { "id": "2046235", "postDate": "11/28/2022 03:23:56", "content": "

@simonveitner
\nI use each of the Top50 candidates from 4 different Co_visitation_matrixes as candidates.Since some of the matrix candidates are mixed with past aids, I remove duplicates at the end of the candidate generation process.
\nIn addition, sessions are sampled as a precautionary measure against memory errors.（10%）

\n

btw, what is the LB score for the model with CV 0.5465?

", "rawMarkdown": "simonveitner \nI use each of the Top50 candidates from 4 different Co_visitation_matrixes as candidates.Since some of the matrix candidates are mixed with past aids, I remove duplicates at the end of the candidate generation process.\nIn addition, sessions are sampled as a precautionary measure against memory errors.（10%）\n\nbtw, what is the LB score for the model with CV 0.5465?", "votes": null }, { "id": "2046396", "postDate": "11/28/2022 07:03:11", "content": "

I didn't submit this model so far to LB. Will do so later and report on leadeboard. Yea, the dublicates I also remove before giving to my model. For now I only use CLICKS matrix for the clicks ranker and only BUY matrix for the buy ranker. I need to include more and will try sampling session as an approach. Tnx! I will let you know abou4 New CV

", "rawMarkdown": "I didn't submit this model so far to LB. Will do so later and report on leadeboard. Yea, the dublicates I also remove before giving to my model. For now I only use CLICKS matrix for the clicks ranker and only BUY matrix for the buy ranker. I need to include more and will try sampling session as an approach. Tnx! I will let you know abou4 New CV", "votes": null } ], "comments": [ { "id": 2045622, "author_name": "jiahongxie", "author_url": "", "post_date": "11/27/2022 14:14:40", "content": "

How many candidates do you generate for each user? What are your hit rates for clicks,carts and orders. I think you maybe have a problem for candidate generation. I use word2vector,bpr,itemcf for candidate generation except past event candidates which all have low hit rates.

", "votes": null, "replies": [ { "id": 2045968, "author_name": "dehokanta", "author_url": "", "post_date": "11/27/2022 20:25:49", "content": "

Data for 4w (22-08-21 ~ 22-08-28) are as follows.

\n
\n

session count: 685.496（Randomly sampled 10%.）
\n candidates count mean: 96.6
\n click hit rate: 0.005
\n cart hit rate: 0.0015
\n order hit rate: 0.0012

\n
\n

Is it lower than your hit rate?

", "votes": null, "replies": [] }, { "id": 2045983, "author_name": "simonveitner", "author_url": "", "post_date": "11/27/2022 20:42:49", "content": "

Could you report your recall@20 per clicks/carts/orders

", "votes": null, "replies": [] }, { "id": 2045992, "author_name": "dehokanta", "author_url": "", "post_date": "11/27/2022 21:01:42", "content": "

CV scores of 4w(22-08-21 ~ 22-08-28) are as follows.
\nHowever, LB score predicted by this model was 0.562

\n

The LB/CV correlation is not good, so I suspect there is a problem with the training data.

\n
\n

clicks: 0.5091
\ncarts: 0.4105
\norders: 0.6499

\n

Score: 0.5640

", "votes": null, "replies": [] }, { "id": 2046019, "author_name": "simonveitner", "author_url": "", "post_date": "11/27/2022 21:17:49", "content": "

For now I have worse CV but I guess this is mostly due to the fact that I can just take top10 candiates from Co visitation matrix as I get memory error otherwise. May I ask how you incoporate canditates from Co visitation. For now I generate and than concat the resulting dataframe with canditates and afterwards count how many times each item was suggested by co visitation matrix. I guess this is not smart way and I could largely improve CV if I can take more canditates.. may I ask how you do that step? Maybe I should stop the counting of suggestion as that is where I get memory error. I will try.
\nIf I get Fürther improvement in CV I will report here. There is many features one can think of i didnt include so far, so there is long way to go.
\nMy current CV:
\nclicks recall = 0.5020045182833257
\ncarts recall = 0.4034401767965001
\norders recall = 0.6254552302403743

\n

Overall Recall = 0.5465056430115072

", "votes": null, "replies": [] }, { "id": 2046235, "author_name": "dehokanta", "author_url": "", "post_date": "11/28/2022 03:23:56", "content": "

@simonveitner
\nI use each of the Top50 candidates from 4 different Co_visitation_matrixes as candidates.Since some of the matrix candidates are mixed with past aids, I remove duplicates at the end of the candidate generation process.
\nIn addition, sessions are sampled as a precautionary measure against memory errors.（10%）

\n

btw, what is the LB score for the model with CV 0.5465?

", "votes": null, "replies": [] }, { "id": 2046396, "author_name": "simonveitner", "author_url": "", "post_date": "11/28/2022 07:03:11", "content": "

I didn't submit this model so far to LB. Will do so later and report on leadeboard. Yea, the dublicates I also remove before giving to my model. For now I only use CLICKS matrix for the clicks ranker and only BUY matrix for the buy ranker. I need to include more and will try sampling session as an approach. Tnx! I will let you know abou4 New CV

", "votes": null, "replies": [] } ] }, { "id": 2045625, "author_name": "simonveitner", "author_url": "", "post_date": "11/27/2022 14:19:05", "content": "

i use the approach shared by radek in his notebook.

", "votes": null, "replies": [ { "id": 2045960, "author_name": "dehokanta", "author_url": "", "post_date": "11/27/2022 20:11:37", "content": "

Thanks. I'm sure there will be some useful information in comment thread. I will take a look.

", "votes": null, "replies": [] }, { "id": 2045966, "author_name": "simonveitner", "author_url": "", "post_date": "11/27/2022 20:23:40", "content": "

Also the associated threads point to useful techniques and Tricks.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2045520": "I'm trying to build ranking model using following methods, but it's not working well.\n\n> **Dataset & CV Strategy**\n- Use 2w and 3w labels as training data.\n(22-08-07 ~ 22-08-21)\n- Use 4w labels as validation data.\n(22-08-21 ~ 22-08-28)\n- Training with 10% of total session.\n- Create candidates and features by transaction data prior to the week of the label.\n\n> **Candidate**\n- Past event candidates\n- Co-visitation matrix candidates\n- popular item candidates\n\n> **Features**\n- aid_features, session_features, session×aid_features \nI refer to this [thread](https://www.kaggle.com/code/cdeotte/candidate-rerank-model-lb-0-575/comments#2030893). \n\n> **Model**\n- LGBM Classifier（training models for each event）\n\nHow do you build models and datasets for training?\nIn particular, I would like to know about the following two points\n\n1. Which period of data is used for the training labels?\n2. Do you do any sampling for training?\n\nI'd be glad to know.", "2045622": "How many candidates do you generate for each user? What are your hit rates for clicks,carts and orders. I think you maybe have a problem for candidate generation. I use word2vector,bpr,itemcf for candidate generation except past event candidates which all have low hit rates.", "2045625": "i use the approach shared by radek in his notebook.", "2045960": "Thanks. I'm sure there will be some useful information in comment thread. I will take a look.", "2045966": "Also the associated threads point to useful techniques and Tricks.", "2045968": "Data for 4w (22-08-21 ~ 22-08-28) are as follows.\n\n> session count: 685.496（Randomly sampled 10%.）\ncandidates count mean: 96.6\nclick hit rate: 0.005\ncart hit rate: 0.0015\norder hit rate: 0.0012\n\nIs it lower than your hit rate?", "2045983": "Could you report your recall@20 per clicks/carts/orders", "2045992": "CV scores of 4w(22-08-21 ~ 22-08-28) are as follows.\nHowever, LB score predicted by this model was 0.562\n\nThe LB/CV correlation is not good, so I suspect there is a problem with the training data.\n\n-------------------\nclicks: 0.5091\ncarts: 0.4105\norders: 0.6499\n\nScore: 0.5640", "2046019": "For now I have worse CV but I guess this is mostly due to the fact that I can just take top10 candiates from Co visitation matrix as I get memory error otherwise. May I ask how you incoporate canditates from Co visitation. For now I generate and than concat the resulting dataframe with canditates and afterwards count how many times each item was suggested by co visitation matrix. I guess this is not smart way and I could largely improve CV if I can take more canditates.. may I ask how you do that step? Maybe I should stop the counting of suggestion as that is where I get memory error. I will try.\nIf I get Fürther improvement in CV I will report here. There is many features one can think of i didnt include so far, so there is long way to go.\nMy current CV:\nclicks recall = 0.5020045182833257\ncarts recall = 0.4034401767965001\norders recall = 0.6254552302403743\n\nOverall Recall = 0.5465056430115072", "2046235": "simonveitner \nI use each of the Top50 candidates from 4 different Co_visitation_matrixes as candidates.Since some of the matrix candidates are mixed with past aids, I remove duplicates at the end of the candidate generation process.\nIn addition, sessions are sampled as a precautionary measure against memory errors.（10%）\n\nbtw, what is the LB score for the model with CV 0.5465?", "2046396": "I didn't submit this model so far to LB. Will do so later and report on leadeboard. Yea, the dublicates I also remove before giving to my model. For now I only use CLICKS matrix for the clicks ranker and only BUY matrix for the buy ranker. I need to include more and will try sampling session as an approach. Tnx! I will let you know abou4 New CV" }, "source": "meta" }