{ "id": 368278, "title": "πŸ“‘ [Step-by-step guide] How I got to my current standing on the LB and how to improve going forward", "url": "/competitions/otto-recommender-system/discussion/368278", "author_name": "Radek Osmulski", "post_date": "2022-11-24T12:35:13.271000", "votes": 78, "comment_count": 20, "views": 0, "content": "

Hey!

\n

I got asked about this in the comments of one of my notebooks and thought I'd share it here as well πŸ™‚

\n

In fact, I wouldn't have been able to climb to where I am on the LB right now if it wasn't for the advice shared with me on the forums by @cdeotte πŸ™ so just passing it forward πŸ™‚

\n

What is most surprising to me is that I didn't do anything fancy and it is all based on the code I have already shared on Kaggle. Here is the recipe:

\n
    \n
  1. I process all the data like in the dataset I shared here: Otto Full Optimized Memory Footprint. I have been messing around on my computer with code from the organizer but to the best of my knowledge you should be able to get the same results if you were to use the code I shared in πŸ’‘A robust local validation framework πŸš€πŸš€πŸš€ line for line, AFAICT.
    2. \n
  2. THE DATA SPLIT LIKE THE ABOVE IS ESSENTIAL! You need something that resembles the test set to train your ranker on, and you of course need to have the labels.
    3. \n
  3. I then train a ranker model EXACTLY like I do in [polars] Proof of concept: LGBM RankerπŸ§ͺπŸ§ͺπŸ§ͺ. This competition is super fun but I am hard-pressed to find the time to do much beyond what I share here on the forums. I don't tweak the training of the ranker. In fact, in the \"observations\" section I share one important insight related to this and the single parameter that I alter.
    4. \n
  4. I didn't create that many new features. I won't share the code because I believe that would be crossing the line and would be unfair to people who have written feature generation code themselves, however here are the feature names along with individual importance:
    5. \n
\n
action_num_reverse_chrono_unique 0.2716890769703452\ncovisit_clicks_candidate_num 0.24848206202059703\naction_num_reverse_chrono 0.2417683829310662\nsec_to_session_end 0.19485467138278798\nsession_length_unique 0.022921177730713577\ncovisit_buys_candidate_num 0.009287900880354305\naid_interacted_with_count 0.004570131040186471\naid_clicked_count 0.0016809643174538038\ntype_weighted_log_recency_score 0.001080825477168101\nsec_since_session_start 0.0010450201166368961\nthis_aid_carted_count 0.0009320732094743725\nsession_length 0.000869834629795715\nlog_recency_score 0.0003542355935627293\naid_carted_count 0.00021963718332129753\nrelative_position_in_session 0.00021918611998695227\naid 2.4820396549324786e-05\nthis_aid_clicked_count 0.0\nthis_aid_ordered_count 0.0\npopularity_0 0.0\npopularity_1 0.0\npopularity_2 0.0\ntype 0.0\naid_ordered_count 0.0\n
\n

πŸ‘‰ !!! This is from the clicks ranker !!! πŸ‘ˆ Yes, that is a modification to the code I shared on Kaggle. Essentially, I am using the same features but am training a separate ranking model for each task!

\n

The names are self-explanatory. I show how to calculate the hardest one to code (type_weighted_log_recency_score) here: [polars] Proof of concept: LGBM RankerπŸ§ͺπŸ§ͺπŸ§ͺ

\n

Yes, the only additional thing I did is that I integrated candidates generated from the co-visitation matrix by @cdeotte.

\n

A single submission trained like this gets me to 0.576 on the LB. If I train more submissions and ensemble using vote ensembling, I get to 0.577, which is where I am right now.

\n

Observations

\n
    \n
  1. The LGBM ranker overfits! That's why training with dart is so useful. Plus, I got better results when I decreased the count of estimators to 10 from 20. Maybe you could squeeze even more performance using l1/l2 regularization, subsampling, etc.
    2. \n
  2. Ensembling should work even better if you train your other solutions using a different type of ranker (xgboost, catboost, etc).
    3. \n
  3. Follow the advice by @cdeotte πŸ™‚ Here he tells us exactly what features we can/should create. There is also another comment from him that I cannot find where Chris outlines how to create new candidates by creating new covisitation matrices. Maybe someone can find it and link below.
    4. \n
  4. My intuition is that the above solution is okay-ish on the feature side of things, but really struggles on the candidate generation side of things. That is probably where you can get the most mileage out of your time.
    5. \n
\n

Hopefully, this can clear up some confusion if you are new to these types of problems or can speed you along the way if you are an experienced Kaggler gunning for top spots πŸ™‚ Wishing you all the best in the competition!

\n

PS. My apologies some of the references in this posts to my notebooks/datasets are not links -- apparently, I have been referring to them too much and Kaggle is not letting me post them again, the platform is preventing me from publishing the post. Apologies for the inconvenience!

\n

Other resources you might find useful:

\n", "messages": [ { "id": 2042097, "postDate": "2022-11-24T12:35:13.270Z", "content": "

Hey!

\n

I got asked about this in the comments of one of my notebooks and thought I'd share it here as well πŸ™‚

\n

In fact, I wouldn't have been able to climb to where I am on the LB right now if it wasn't for the advice shared with me on the forums by @cdeotte πŸ™ so just passing it forward πŸ™‚

\n

What is most surprising to me is that I didn't do anything fancy and it is all based on the code I have already shared on Kaggle. Here is the recipe:

\n
    \n
  1. I process all the data like in the dataset I shared here: Otto Full Optimized Memory Footprint. I have been messing around on my computer with code from the organizer but to the best of my knowledge you should be able to get the same results if you were to use the code I shared in πŸ’‘A robust local validation framework πŸš€πŸš€πŸš€ line for line, AFAICT.
    2. \n
  2. THE DATA SPLIT LIKE THE ABOVE IS ESSENTIAL! You need something that resembles the test set to train your ranker on, and you of course need to have the labels.
    3. \n
  3. I then train a ranker model EXACTLY like I do in [polars] Proof of concept: LGBM RankerπŸ§ͺπŸ§ͺπŸ§ͺ. This competition is super fun but I am hard-pressed to find the time to do much beyond what I share here on the forums. I don't tweak the training of the ranker. In fact, in the \"observations\" section I share one important insight related to this and the single parameter that I alter.
    4. \n
  4. I didn't create that many new features. I won't share the code because I believe that would be crossing the line and would be unfair to people who have written feature generation code themselves, however here are the feature names along with individual importance:
    5. \n
\n
action_num_reverse_chrono_unique 0.2716890769703452\ncovisit_clicks_candidate_num 0.24848206202059703\naction_num_reverse_chrono 0.2417683829310662\nsec_to_session_end 0.19485467138278798\nsession_length_unique 0.022921177730713577\ncovisit_buys_candidate_num 0.009287900880354305\naid_interacted_with_count 0.004570131040186471\naid_clicked_count 0.0016809643174538038\ntype_weighted_log_recency_score 0.001080825477168101\nsec_since_session_start 0.0010450201166368961\nthis_aid_carted_count 0.0009320732094743725\nsession_length 0.000869834629795715\nlog_recency_score 0.0003542355935627293\naid_carted_count 0.00021963718332129753\nrelative_position_in_session 0.00021918611998695227\naid 2.4820396549324786e-05\nthis_aid_clicked_count 0.0\nthis_aid_ordered_count 0.0\npopularity_0 0.0\npopularity_1 0.0\npopularity_2 0.0\ntype 0.0\naid_ordered_count 0.0\n
\n

πŸ‘‰ !!! This is from the clicks ranker !!! πŸ‘ˆ Yes, that is a modification to the code I shared on Kaggle. Essentially, I am using the same features but am training a separate ranking model for each task!

\n

The names are self-explanatory. I show how to calculate the hardest one to code (type_weighted_log_recency_score) here: [polars] Proof of concept: LGBM RankerπŸ§ͺπŸ§ͺπŸ§ͺ

\n

Yes, the only additional thing I did is that I integrated candidates generated from the co-visitation matrix by @cdeotte.

\n

A single submission trained like this gets me to 0.576 on the LB. If I train more submissions and ensemble using vote ensembling, I get to 0.577, which is where I am right now.

\n

Observations

\n
    \n
  1. The LGBM ranker overfits! That's why training with dart is so useful. Plus, I got better results when I decreased the count of estimators to 10 from 20. Maybe you could squeeze even more performance using l1/l2 regularization, subsampling, etc.
    2. \n
  2. Ensembling should work even better if you train your other solutions using a different type of ranker (xgboost, catboost, etc).
    3. \n
  3. Follow the advice by @cdeotte πŸ™‚ Here he tells us exactly what features we can/should create. There is also another comment from him that I cannot find where Chris outlines how to create new candidates by creating new covisitation matrices. Maybe someone can find it and link below.
    4. \n
  4. My intuition is that the above solution is okay-ish on the feature side of things, but really struggles on the candidate generation side of things. That is probably where you can get the most mileage out of your time.
    5. \n
\n

Hopefully, this can clear up some confusion if you are new to these types of problems or can speed you along the way if you are an experienced Kaggler gunning for top spots πŸ™‚ Wishing you all the best in the competition!

\n

PS. My apologies some of the references in this posts to my notebooks/datasets are not links -- apparently, I have been referring to them too much and Kaggle is not letting me post them again, the platform is preventing me from publishing the post. Apologies for the inconvenience!

\n

Other resources you might find useful:

\n", "rawMarkdown": "Hey!\n\nI got asked about this in the comments of one of my notebooks and thought I'd share it here as well πŸ™‚\n\nIn fact, I wouldn't have been able to climb to where I am on the LB right now if it wasn't for the advice shared with me on the forums by @cdeotte πŸ™ so just passing it forward πŸ™‚\n\nWhat is most surprising to me is that I didn't do anything fancy and it is all based on the code I have already shared on Kaggle. Here is the recipe:\n\n1. I process all the data like in the dataset I shared here: `Otto Full Optimized Memory Footprint`. I have been messing around on my computer with [code from the organizer](https://github.com/otto-de/recsys-dataset) but to the best of my knowledge you should be able to get the same results if you were to use the code I shared in [πŸ’‘A robust local validation framework πŸš€πŸš€πŸš€] (https://www.kaggle.com/code/radek1/a-robust-local-validation-framework) line for line, AFAICT.\n2. THE DATA SPLIT LIKE THE ABOVE IS ESSENTIAL! You need something that resembles the test set to train your ranker on, and you of course need to have the labels.\n3. I then train a ranker model EXACTLY like I do in `[polars] Proof of concept: LGBM RankerπŸ§ͺπŸ§ͺπŸ§ͺ`. This competition is super fun but I am hard-pressed to find the time to do much beyond what I share here on the forums. I don't tweak the training of the ranker. In fact, in the \"observations\" section I share one important insight related to this and the single parameter that I alter.\n4. I didn't create that many new features. I won't share the code because I believe that would be crossing the line and would be unfair to people who have written feature generation code themselves, however here are the feature names along with individual importance:\n\n```\naction_num_reverse_chrono_unique 0.2716890769703452\ncovisit_clicks_candidate_num 0.24848206202059703\naction_num_reverse_chrono 0.2417683829310662\nsec_to_session_end 0.19485467138278798\nsession_length_unique 0.022921177730713577\ncovisit_buys_candidate_num 0.009287900880354305\naid_interacted_with_count 0.004570131040186471\naid_clicked_count 0.0016809643174538038\ntype_weighted_log_recency_score 0.001080825477168101\nsec_since_session_start 0.0010450201166368961\nthis_aid_carted_count 0.0009320732094743725\nsession_length 0.000869834629795715\nlog_recency_score 0.0003542355935627293\naid_carted_count 0.00021963718332129753\nrelative_position_in_session 0.00021918611998695227\naid 2.4820396549324786e-05\nthis_aid_clicked_count 0.0\nthis_aid_ordered_count 0.0\npopularity_0 0.0\npopularity_1 0.0\npopularity_2 0.0\ntype 0.0\naid_ordered_count 0.0\n```\n\nπŸ‘‰ !!! This is from the clicks ranker !!! πŸ‘ˆ Yes, that is a modification to the code I shared on Kaggle. Essentially, I am using the same features but am training a separate ranking model for each task!\n\nThe names are self-explanatory. I show how to calculate the hardest one to code (`type_weighted_log_recency_score`) here: `[polars] Proof of concept: LGBM RankerπŸ§ͺπŸ§ͺπŸ§ͺ`\n\nYes, the only additional thing I did is that I integrated candidates generated from `the co-visitation matrix` by @cdeotte.\n\nA single submission trained like this gets me to 0.576 on the LB. If I train more submissions and ensemble using vote ensembling, I get to 0.577, which is where I am right now.\n\n### Observations\n\n1. The LGBM ranker overfits! That's why training with `dart` is so useful. Plus, I got better results when I decreased the count of estimators to 10 from 20. Maybe you could squeeze even more performance using l1/l2 regularization, subsampling, etc.\n2. Ensembling should work even better if you train your other solutions using a different type of ranker (xgboost, catboost, etc).\n3. Follow the advice by @cdeotte πŸ™‚ [Here](https://www.kaggle.com/code/cdeotte/candidate-rerank-model-lb-0-575/comments#2030893) he tells us exactly what features we can/should create. There is also another comment from him that I cannot find where Chris outlines how to create new candidates by creating new covisitation matrices. Maybe someone can find it and link below.\n4. My intuition is that the above solution is okay-ish on the feature side of things, but really struggles on the candidate generation side of things. That is probably where you can get the most mileage out of your time.\n\nHopefully, this can clear up some confusion if you are new to these types of problems or can speed you along the way if you are an experienced Kaggler gunning for top spots πŸ™‚ Wishing you all the best in the competition!\n\nPS. My apologies some of the references in this posts to my notebooks/datasets are not links -- apparently, I have been referring to them too much and Kaggle is not letting me post them again, the platform is preventing me from publishing the post. Apologies for the inconvenience!\n\n### Other resources you might find useful:\n\n* [πŸ’‘ [2 methods] How-to ensemble predictions πŸ…πŸ…πŸ…](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [πŸ’‘ For my friends from Twitter and LinkedIn -- here is how to dive into this competition 🐳](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n* [co-visitation matrix - simplified, imprvd logic πŸ”₯](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [πŸ’‘ Word2Vec How-to [training and submission]πŸš€πŸš€πŸš€](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)\n", "votes": 75 }, { "id": 2082786, "postDate": "2023-01-01T23:37:37.453Z", "content": "

What do you mean by decreasing the count of estimators ?

", "rawMarkdown": "What do you mean by decreasing the count of estimators ? ", "votes": 1, "replies": [ { "id": 2082850, "postDate": "2023-01-02T02:55:52.623Z", "content": "

hey @rayanaay!

\n

Rankers are tree models so they are comprised of trees that operate on the data in sequence. Here by estimators I mean the trees -- instead of training 20 trees I train 10, which helps with the overfitting.

\n

Hope this helps! πŸ™‚

", "rawMarkdown": "hey @rayanaay!\n\nRankers are tree models so they are comprised of trees that operate on the data in sequence. Here by estimators I mean the trees -- instead of training 20 trees I train 10, which helps with the overfitting.\n\nHope this helps! πŸ™‚", "replies": [ { "id": 2083183, "postDate": "2023-01-02T10:14:42.400Z", "content": "

Yes I know this, I asked this question because it seems to me that the number of iterations / boosting round ( 20 in your case ) is too low, isn't ? Usually, we set it over 100.

", "rawMarkdown": "Yes I know this, I asked this question because it seems to me that the number of iterations / boosting round ( 20 in your case ) is too low, isn't ? Usually, we set it over 100. \n" } ] } ] }, { "id": 2056887, "postDate": "2022-12-06T14:23:17.983Z", "content": "

Hi @radek1, Thank you so much for sharing the feature names! I am ready to generate more features and I would like to replicate some of yours if possible.

\n

Would you mind to provide a brief description to on the features ( covisit_buys_candidate_num and covisit_clicks_candidate_num) you shared above? (not request for code nor detail description, of course).

\n

Also I wonder what are the differences between this_aid_carted_count and aid_carted_count? Thanks a lot! πŸ™

", "rawMarkdown": "Hi @radek1, Thank you so much for sharing the feature names! I am ready to generate more features and I would like to replicate some of yours if possible.\n\nWould you mind to provide a brief description to on the features ( `covisit_buys_candidate_num` and `covisit_clicks_candidate_num`) you shared above? (not request for code nor detail description, of course).\n\nAlso I wonder what are the differences between `this_aid_carted_count` and `aid_carted_count`? Thanks a lot! πŸ™", "votes": 1 }, { "id": 2048477, "postDate": "2022-11-29T13:17:23.340Z", "content": "

Hi @radek1 Thank you so much for this wonderful sharing!

\n

It's so amazing for you to share the feature names and individual importance. However, I don't understand what is the individual importance and why they are important. I don't remember you give any importance to the features in your LGBM Ranker notebook.

\n

Could you help me understand the 'individual importance' here? Thank you!

", "rawMarkdown": "Hi @radek1 Thank you so much for this wonderful sharing!\n\nIt's so amazing for you to share the feature names and individual importance. However, I don't understand what is the individual importance and why they are important. I don't remember you give any importance to the features in your LGBM Ranker notebook. \n\nCould you help me understand the 'individual importance' here? Thank you!", "votes": 1, "replies": [ { "id": 2048479, "postDate": "2022-11-29T13:20:19.937Z", "content": "

I calculate them locally on my computer. The ranker here is just an LGBMRanker and I use gain as the basis for these calculations.

\n

\"\"

", "rawMarkdown": "I calculate them locally on my computer. The ranker here is just an `LGBMRanker` and I use gain as the basis for these calculations.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F5550eb5de02fef3526f0d365902920c5%2Ffeature_importances.png?generation=1669727978627356&alt=media)", "votes": 1 }, { "id": 2048498, "postDate": "2022-11-29T13:47:46.300Z", "content": "

Thank you so much for this quick and helpful reply! This image tells so much and no more zero values which clears away some confusion I had. πŸ™

\n

So the individual importance for each feature is learnt by the LGBMRanker. Print them out can show us which features are important which are not, so I can focus on writing the most important features. Thank you!

", "rawMarkdown": "Thank you so much for this quick and helpful reply! This image tells so much and no more zero values which clears away some confusion I had. πŸ™\n\nSo the individual importance for each feature is learnt by the LGBMRanker. Print them out can show us which features are important which are not, so I can focus on writing the most important features. Thank you!" } ] }, { "id": 2044652, "postDate": "2022-11-26T17:23:36.827Z", "content": "

thanks for sharning

", "rawMarkdown": "thanks for sharning ", "votes": 1, "replies": [ { "id": 2044966, "postDate": "2022-11-26T21:54:29.177Z", "content": "

Pleasure, @ninjaac! πŸ™Œ Very glad you found this useful! πŸ™‚

", "rawMarkdown": "Pleasure, @ninjaac! πŸ™Œ Very glad you found this useful! πŸ™‚ " } ] }, { "id": 2042182, "postDate": "2022-11-24T13:46:56.447Z", "content": "

Hello Radek.
\nTo clarify:
\nBy \"training a separate ranking model for each task\" you mean training separate model for ranking clicks, carts and orders?
\nAnd for local validation you just predict on the test Set we always use for val and calculate the recall in usual way?
\nOne another point: in one of Diskussion before between you and Chris I noticed that you had something wrong in aggregating User history in the first version of ur notebook. Is this issue Fixed or can u point to where we need to for it ouself?
\nTnx in advance :)

", "rawMarkdown": "Hello Radek.\nTo clarify: \nBy \"training a separate ranking model for each task\" you mean training separate model for ranking clicks, carts and orders?\nAnd for local validation you just predict on the test Set we always use for val and calculate the recall in usual way? \nOne another point: in one of Diskussion before between you and Chris I noticed that you had something wrong in aggregating User history in the first version of ur notebook. Is this issue Fixed or can u point to where we need to for it ouself?\nTnx in advance :)", "votes": 1, "replies": [ { "id": 2042193, "postDate": "2022-11-24T13:58:00.120Z", "content": "

Maybe to clarify the last point more: can we take the notebook as it is and generate New features to get better score or is there something to be Fixed before in the Version publicly available

", "rawMarkdown": "Maybe to clarify the last point more: can we take the notebook as it is and generate New features to get better score or is there something to be Fixed before in the Version publicly available", "votes": 1 }, { "id": 2042718, "postDate": "2022-11-24T23:02:31.083Z", "content": "

Hey @simonveitner!

\n

Yes, I am training a separate model for each of the tasks, so I have one ranker model for clicks, another for carts and yet another for orders πŸ™‚

\n

And you are absolutely right, I now remember what this discussion with Chris was about!

\n

I only did any changes based on this when I was working on my local version with those additional features. All that it was about is that you need to have a single row for a single aid in your train/validation sets for the ranker. So if you are predicting on a session (or training or a session) where there are multiple entries with the same aid, you need to somehow aggregate them. Functions like cumsum and cumcount are your friends and also drop_duplicates last or first (depending on the situation, in polars the name of the method that does that is unique)

\n

Oh, and I speak a bit more to the datasets I shared on Kaggle that you can use here, maybe this can also be helpful.

\n

Best of luck in getting it all up and running! πŸ™‚

", "rawMarkdown": "Hey @simonveitner!\n\nYes, I am training a separate model for each of the tasks, so I have one ranker model for `clicks`, another for `carts` and yet another for `orders` πŸ™‚\n\nAnd you are absolutely right, I now remember what this discussion with Chris was about! \n\nI only did any changes based on this when I was working on my local version with those additional features. All that it was about is that you need to have a single row for a single aid in your train/validation sets for the ranker. So if you are predicting on a session (or training or a session) where there are multiple entries with the same `aid`, you need to somehow aggregate them. Functions like `cumsum` and `cumcount` are your friends and also `drop_duplicates` last or first (depending on the situation, in `polars` the name of the method that does that is `unique`)\n\nOh, and I speak [a bit more to the datasets I shared on Kaggle that you can use here](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368149#2042714), maybe this can also be helpful.\n\nBest of luck in getting it all up and running! πŸ™‚", "votes": 2 }, { "id": 2043037, "postDate": "2022-11-25T08:23:48.960Z", "content": "

Tnx radek! Now im Getting more familar with polars and i think I will manage to handle it! Tnx already for the help otherwise I would have problem to approach this prob.

", "rawMarkdown": "Tnx radek! Now im Getting more familar with polars and i think I will manage to handle it! Tnx already for the help otherwise I would have problem to approach this prob.", "votes": 1 }, { "id": 2043074, "postDate": "2022-11-25T09:19:20.733Z", "content": "

No problem, glad I could be of help πŸ™Œ best of luck in the competition! πŸ™‚

", "rawMarkdown": "No problem, glad I could be of help πŸ™Œ best of luck in the competition! πŸ™‚", "votes": 1 }, { "id": 2044353, "postDate": "2022-11-26T13:28:16.110Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 2043329, "postDate": "2022-11-25T15:13:36.963Z", "content": "

Thank you for your very illuminating posts and notebooks, @radek1 ! Learning a lot from you, @cdeotte , and other experienced kagglers who take the time time to share and explain their insights!

", "rawMarkdown": "Thank you for your very illuminating posts and notebooks, @radek1 ! Learning a lot from you, @cdeotte , and other experienced kagglers who take the time time to share and explain their insights!", "votes": 2, "replies": [ { "id": 2043739, "postDate": "2022-11-25T21:02:17.313Z", "content": "

Thank you very much, @thomasbrekkunnvik! Super glad to hear! πŸ™πŸ™‚

", "rawMarkdown": "Thank you very much, @thomasbrekkunnvik! Super glad to hear! πŸ™πŸ™‚ ", "votes": 1 } ] }, { "id": 2091097, "postDate": "2023-01-08T02:27:44.540Z", "content": "

Thanks for sharing, but got confused how you creating the ground truth for a LGBM ranker.

\n

First loading test data and test labels
\ntrain = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test.parquet')
\ntrain_labels = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test_labels.parquet')

\n

Seconed join this 2 dataframe together
\ntrain = train.join(train_labels, how='left', on=['session', 'type', 'aid']).with_column(pl.col('gt').fill_null(0))

\n

If the records can be joined then is going to be positive label otherwise its going to be a negative value. But I don't think this 2 dataframe is join-able. Test labels are unique records regarding session, aid, type and ts compared with test dataset. Second I think all the test data should be a positive label but can have different rank scoring due their transaction time or event type.

", "rawMarkdown": "Thanks for sharing, but got confused how you creating the ground truth for a LGBM ranker. \n\nFirst loading test data and test labels\n`train = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test.parquet')`\n`train_labels = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test_labels.parquet')`\n\nSeconed join this 2 dataframe together\n`train = train.join(train_labels, how='left', on=['session', 'type', 'aid']).with_column(pl.col('gt').fill_null(0))`\n\nIf the records can be joined then is going to be positive label otherwise its going to be a negative value. But I don't think this 2 dataframe is join-able. Test labels are unique records regarding session, aid, type and ts compared with test dataset. Second I think all the test data should be a positive label but can have different rank scoring due their transaction time or event type." }, { "id": 2079220, "postDate": "2022-12-29T06:16:22.637Z", "content": "

have you ever meet the question, the lgbrank perofromer well in train_data and vaild_data get auc 0.98 and ndcg@20 0.99 but in pd.concat([train_data ,train_data ]) it's auc only 0.5 and ndcg@20 1 why this happen, i use the model auc 0.98 and ndcg@20 0.99 but it doesn't perform well in LB, how to solve this

", "rawMarkdown": "have you ever meet the question, the lgbrank perofromer well in train_data and vaild_data get auc 0.98 and ndcg@20 0.99 but in pd.concat([train_data ,train_data ]) it's auc only 0.5 and ndcg@20 1 why this happen, i use the model auc 0.98 and ndcg@20 0.99 but it doesn't perform well in LB, how to solve this" }, { "id": 2060532, "postDate": "2022-12-10T04:54:35.867Z", "content": "

Hi,Radek,thank you for sharing,but I don't quite understand what action_num_reverse_chrono mean？

", "rawMarkdown": "Hi,Radek,thank you for sharing,but I don't quite understand what action_num_reverse_chrono mean？" } ], "comments": [ { "id": 2082786, "author_name": "Rayan-aay", "author_url": "", "post_date": "2023-01-01T23:37:37.453000", "content": "

What do you mean by decreasing the count of estimators ?

", "votes": 1, "replies": [ { "id": 2082850, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2023-01-02T02:55:52.623000", "content": "

hey @rayanaay!

\n

Rankers are tree models so they are comprised of trees that operate on the data in sequence. Here by estimators I mean the trees -- instead of training 20 trees I train 10, which helps with the overfitting.

\n

Hope this helps! πŸ™‚

", "votes": 0, "replies": [ { "id": 2083183, "author_name": "Rayan-aay", "author_url": "", "post_date": "2023-01-02T10:14:42.400000", "content": "

Yes I know this, I asked this question because it seems to me that the number of iterations / boosting round ( 20 in your case ) is too low, isn't ? Usually, we set it over 100.

", "votes": 0, "replies": [] } ] } ] }, { "id": 2056887, "author_name": "danielliao", "author_url": "", "post_date": "2022-12-06T14:23:17.983000", "content": "

Hi @radek1, Thank you so much for sharing the feature names! I am ready to generate more features and I would like to replicate some of yours if possible.

\n

Would you mind to provide a brief description to on the features ( covisit_buys_candidate_num and covisit_clicks_candidate_num) you shared above? (not request for code nor detail description, of course).

\n

Also I wonder what are the differences between this_aid_carted_count and aid_carted_count? Thanks a lot! πŸ™

", "votes": 1, "replies": [] }, { "id": 2048477, "author_name": "danielliao", "author_url": "", "post_date": "2022-11-29T13:17:23.340000", "content": "

Hi @radek1 Thank you so much for this wonderful sharing!

\n

It's so amazing for you to share the feature names and individual importance. However, I don't understand what is the individual importance and why they are important. I don't remember you give any importance to the features in your LGBM Ranker notebook.

\n

Could you help me understand the 'individual importance' here? Thank you!

", "votes": 1, "replies": [ { "id": 2048479, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-29T13:20:19.937000", "content": "

I calculate them locally on my computer. The ranker here is just an LGBMRanker and I use gain as the basis for these calculations.

\n

\"\"

", "votes": 1, "replies": [] }, { "id": 2048498, "author_name": "danielliao", "author_url": "", "post_date": "2022-11-29T13:47:46.300000", "content": "

Thank you so much for this quick and helpful reply! This image tells so much and no more zero values which clears away some confusion I had. πŸ™

\n

So the individual importance for each feature is learnt by the LGBMRanker. Print them out can show us which features are important which are not, so I can focus on writing the most important features. Thank you!

", "votes": 0, "replies": [] } ] }, { "id": 2044652, "author_name": "Pavithra Devi M", "author_url": "", "post_date": "2022-11-26T17:23:36.827000", "content": "

thanks for sharning

", "votes": 1, "replies": [ { "id": 2044966, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-26T21:54:29.177000", "content": "

Pleasure, @ninjaac! πŸ™Œ Very glad you found this useful! πŸ™‚

", "votes": 0, "replies": [] } ] }, { "id": 2042182, "author_name": "Simon Veitner", "author_url": "", "post_date": "2022-11-24T13:46:56.447000", "content": "

Hello Radek.
\nTo clarify:
\nBy \"training a separate ranking model for each task\" you mean training separate model for ranking clicks, carts and orders?
\nAnd for local validation you just predict on the test Set we always use for val and calculate the recall in usual way?
\nOne another point: in one of Diskussion before between you and Chris I noticed that you had something wrong in aggregating User history in the first version of ur notebook. Is this issue Fixed or can u point to where we need to for it ouself?
\nTnx in advance :)

", "votes": 1, "replies": [ { "id": 2042193, "author_name": "Simon Veitner", "author_url": "", "post_date": "2022-11-24T13:58:00.120000", "content": "

Maybe to clarify the last point more: can we take the notebook as it is and generate New features to get better score or is there something to be Fixed before in the Version publicly available

", "votes": 1, "replies": [] }, { "id": 2042718, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-24T23:02:31.083000", "content": "

Hey @simonveitner!

\n

Yes, I am training a separate model for each of the tasks, so I have one ranker model for clicks, another for carts and yet another for orders πŸ™‚

\n

And you are absolutely right, I now remember what this discussion with Chris was about!

\n

I only did any changes based on this when I was working on my local version with those additional features. All that it was about is that you need to have a single row for a single aid in your train/validation sets for the ranker. So if you are predicting on a session (or training or a session) where there are multiple entries with the same aid, you need to somehow aggregate them. Functions like cumsum and cumcount are your friends and also drop_duplicates last or first (depending on the situation, in polars the name of the method that does that is unique)

\n

Oh, and I speak a bit more to the datasets I shared on Kaggle that you can use here, maybe this can also be helpful.

\n

Best of luck in getting it all up and running! πŸ™‚

", "votes": 2, "replies": [] }, { "id": 2043037, "author_name": "Simon Veitner", "author_url": "", "post_date": "2022-11-25T08:23:48.960000", "content": "

Tnx radek! Now im Getting more familar with polars and i think I will manage to handle it! Tnx already for the help otherwise I would have problem to approach this prob.

", "votes": 1, "replies": [] }, { "id": 2043074, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-25T09:19:20.733000", "content": "

No problem, glad I could be of help πŸ™Œ best of luck in the competition! πŸ™‚

", "votes": 1, "replies": [] }, { "id": 2044353, "author_name": "", "author_url": "", "post_date": "2022-11-26T13:28:16.110000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 2043329, "author_name": "Thomas Brekk Unnvik", "author_url": "", "post_date": "2022-11-25T15:13:36.963000", "content": "

Thank you for your very illuminating posts and notebooks, @radek1 ! Learning a lot from you, @cdeotte , and other experienced kagglers who take the time time to share and explain their insights!

", "votes": 2, "replies": [ { "id": 2043739, "author_name": "Radek Osmulski", "author_url": "", "post_date": "2022-11-25T21:02:17.313000", "content": "

Thank you very much, @thomasbrekkunnvik! Super glad to hear! πŸ™πŸ™‚

", "votes": 1, "replies": [] } ] }, { "id": 2091097, "author_name": "Qucy Wei", "author_url": "", "post_date": "2023-01-08T02:27:44.540000", "content": "

Thanks for sharing, but got confused how you creating the ground truth for a LGBM ranker.

\n

First loading test data and test labels
\ntrain = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test.parquet')
\ntrain_labels = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test_labels.parquet')

\n

Seconed join this 2 dataframe together
\ntrain = train.join(train_labels, how='left', on=['session', 'type', 'aid']).with_column(pl.col('gt').fill_null(0))

\n

If the records can be joined then is going to be positive label otherwise its going to be a negative value. But I don't think this 2 dataframe is join-able. Test labels are unique records regarding session, aid, type and ts compared with test dataset. Second I think all the test data should be a positive label but can have different rank scoring due their transaction time or event type.

", "votes": 0, "replies": [] }, { "id": 2079220, "author_name": "wcq_glhf", "author_url": "", "post_date": "2022-12-29T06:16:22.637000", "content": "

have you ever meet the question, the lgbrank perofromer well in train_data and vaild_data get auc 0.98 and ndcg@20 0.99 but in pd.concat([train_data ,train_data ]) it's auc only 0.5 and ndcg@20 1 why this happen, i use the model auc 0.98 and ndcg@20 0.99 but it doesn't perform well in LB, how to solve this

", "votes": 0, "replies": [] }, { "id": 2060532, "author_name": "Jensen", "author_url": "", "post_date": "2022-12-10T04:54:35.867000", "content": "

Hi,Radek,thank you for sharing,but I don't quite understand what action_num_reverse_chrono mean？

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2042097": "Hey!\n\nI got asked about this in the comments of one of my notebooks and thought I'd share it here as well πŸ™‚\n\nIn fact, I wouldn't have been able to climb to where I am on the LB right now if it wasn't for the advice shared with me on the forums by @cdeotte πŸ™ so just passing it forward πŸ™‚\n\nWhat is most surprising to me is that I didn't do anything fancy and it is all based on the code I have already shared on Kaggle. Here is the recipe:\n\n1. I process all the data like in the dataset I shared here: `Otto Full Optimized Memory Footprint`. I have been messing around on my computer with [code from the organizer](https://github.com/otto-de/recsys-dataset) but to the best of my knowledge you should be able to get the same results if you were to use the code I shared in [πŸ’‘A robust local validation framework πŸš€πŸš€πŸš€] (https://www.kaggle.com/code/radek1/a-robust-local-validation-framework) line for line, AFAICT.\n2. THE DATA SPLIT LIKE THE ABOVE IS ESSENTIAL! You need something that resembles the test set to train your ranker on, and you of course need to have the labels.\n3. I then train a ranker model EXACTLY like I do in `[polars] Proof of concept: LGBM RankerπŸ§ͺπŸ§ͺπŸ§ͺ`. This competition is super fun but I am hard-pressed to find the time to do much beyond what I share here on the forums. I don't tweak the training of the ranker. In fact, in the \"observations\" section I share one important insight related to this and the single parameter that I alter.\n4. I didn't create that many new features. I won't share the code because I believe that would be crossing the line and would be unfair to people who have written feature generation code themselves, however here are the feature names along with individual importance:\n\n```\naction_num_reverse_chrono_unique 0.2716890769703452\ncovisit_clicks_candidate_num 0.24848206202059703\naction_num_reverse_chrono 0.2417683829310662\nsec_to_session_end 0.19485467138278798\nsession_length_unique 0.022921177730713577\ncovisit_buys_candidate_num 0.009287900880354305\naid_interacted_with_count 0.004570131040186471\naid_clicked_count 0.0016809643174538038\ntype_weighted_log_recency_score 0.001080825477168101\nsec_since_session_start 0.0010450201166368961\nthis_aid_carted_count 0.0009320732094743725\nsession_length 0.000869834629795715\nlog_recency_score 0.0003542355935627293\naid_carted_count 0.00021963718332129753\nrelative_position_in_session 0.00021918611998695227\naid 2.4820396549324786e-05\nthis_aid_clicked_count 0.0\nthis_aid_ordered_count 0.0\npopularity_0 0.0\npopularity_1 0.0\npopularity_2 0.0\ntype 0.0\naid_ordered_count 0.0\n```\n\nπŸ‘‰ !!! This is from the clicks ranker !!! πŸ‘ˆ Yes, that is a modification to the code I shared on Kaggle. Essentially, I am using the same features but am training a separate ranking model for each task!\n\nThe names are self-explanatory. I show how to calculate the hardest one to code (`type_weighted_log_recency_score`) here: `[polars] Proof of concept: LGBM RankerπŸ§ͺπŸ§ͺπŸ§ͺ`\n\nYes, the only additional thing I did is that I integrated candidates generated from `the co-visitation matrix` by @cdeotte.\n\nA single submission trained like this gets me to 0.576 on the LB. If I train more submissions and ensemble using vote ensembling, I get to 0.577, which is where I am right now.\n\n### Observations\n\n1. The LGBM ranker overfits! That's why training with `dart` is so useful. Plus, I got better results when I decreased the count of estimators to 10 from 20. Maybe you could squeeze even more performance using l1/l2 regularization, subsampling, etc.\n2. Ensembling should work even better if you train your other solutions using a different type of ranker (xgboost, catboost, etc).\n3. Follow the advice by @cdeotte πŸ™‚ [Here](https://www.kaggle.com/code/cdeotte/candidate-rerank-model-lb-0-575/comments#2030893) he tells us exactly what features we can/should create. There is also another comment from him that I cannot find where Chris outlines how to create new candidates by creating new covisitation matrices. Maybe someone can find it and link below.\n4. My intuition is that the above solution is okay-ish on the feature side of things, but really struggles on the candidate generation side of things. That is probably where you can get the most mileage out of your time.\n\nHopefully, this can clear up some confusion if you are new to these types of problems or can speed you along the way if you are an experienced Kaggler gunning for top spots πŸ™‚ Wishing you all the best in the competition!\n\nPS. My apologies some of the references in this posts to my notebooks/datasets are not links -- apparently, I have been referring to them too much and Kaggle is not letting me post them again, the platform is preventing me from publishing the post. Apologies for the inconvenience!\n\n### Other resources you might find useful:\n\n* [πŸ’‘ [2 methods] How-to ensemble predictions πŸ…πŸ…πŸ…](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [πŸ’‘ For my friends from Twitter and LinkedIn -- here is how to dive into this competition 🐳](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n* [co-visitation matrix - simplified, imprvd logic πŸ”₯](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [πŸ’‘ Word2Vec How-to [training and submission]πŸš€πŸš€πŸš€](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)\n", "2082786": "What do you mean by decreasing the count of estimators ? ", "2056887": "Hi @radek1, Thank you so much for sharing the feature names! I am ready to generate more features and I would like to replicate some of yours if possible.\n\nWould you mind to provide a brief description to on the features ( `covisit_buys_candidate_num` and `covisit_clicks_candidate_num`) you shared above? (not request for code nor detail description, of course).\n\nAlso I wonder what are the differences between `this_aid_carted_count` and `aid_carted_count`? Thanks a lot! πŸ™", "2048477": "Hi @radek1 Thank you so much for this wonderful sharing!\n\nIt's so amazing for you to share the feature names and individual importance. However, I don't understand what is the individual importance and why they are important. I don't remember you give any importance to the features in your LGBM Ranker notebook. \n\nCould you help me understand the 'individual importance' here? Thank you!", "2044652": "thanks for sharning ", "2042182": "Hello Radek.\nTo clarify: \nBy \"training a separate ranking model for each task\" you mean training separate model for ranking clicks, carts and orders?\nAnd for local validation you just predict on the test Set we always use for val and calculate the recall in usual way? \nOne another point: in one of Diskussion before between you and Chris I noticed that you had something wrong in aggregating User history in the first version of ur notebook. Is this issue Fixed or can u point to where we need to for it ouself?\nTnx in advance :)", "2043329": "Thank you for your very illuminating posts and notebooks, @radek1 ! Learning a lot from you, @cdeotte , and other experienced kagglers who take the time time to share and explain their insights!", "2091097": "Thanks for sharing, but got confused how you creating the ground truth for a LGBM ranker. \n\nFirst loading test data and test labels\n`train = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test.parquet')`\n`train_labels = pl.read_parquet('../input/otto-train-and-test-data-for-local-validation/test_labels.parquet')`\n\nSeconed join this 2 dataframe together\n`train = train.join(train_labels, how='left', on=['session', 'type', 'aid']).with_column(pl.col('gt').fill_null(0))`\n\nIf the records can be joined then is going to be positive label otherwise its going to be a negative value. But I don't think this 2 dataframe is join-able. Test labels are unique records regarding session, aid, type and ts compared with test dataset. Second I think all the test data should be a positive label but can have different rank scoring due their transaction time or event type.", "2079220": "have you ever meet the question, the lgbrank perofromer well in train_data and vaild_data get auc 0.98 and ndcg@20 0.99 but in pd.concat([train_data ,train_data ]) it's auc only 0.5 and ndcg@20 1 why this happen, i use the model auc 0.98 and ndcg@20 0.99 but it doesn't perform well in LB, how to solve this", "2060532": "Hi,Radek,thank you for sharing,but I don't quite understand what action_num_reverse_chrono mean？" } }