train.parquet and test.parquet, full's train.parquet and test.parquet separately.\n
![\"otto-dream-team-solution\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3563032%2F93c4b8a615882a249af1509539a7c374%2Fotto-dream-team-solution.png?generation=1675291808419108&alt=media\")
First, I would like to thank the organizers and those who shared knowledge. Especially we would like to thank @radek1 for sharing so many valuable notebooks and datasets, and @cdeotte for explaining many questions in detail.
\nWill Release Some Reproducible Code When The LB is Finalized
\nSince @radek1's CV setting tracks public LB perfectly, we directly use his datasets in this competition. The following graph shows how we leverage his datasets for feature engineering, local validation, and submission generation.
\ntrain.parquet and test.parquet, full's train.parquet and test.parquet separately.Our retrieving strategy is quite simple, we do some hyperparameters tuning on @cdeotte 's public notebook. Mainly the topN of each kind of co-visitation matrix and the N_REC for each session. The optimal topN for each co-visitation matrix is 100, and the optimal N_REC is 200. Then we use @tuongkhang 's public notebook to generate the recall200 candidates for both local validation and test submission. Then we were able to get recall@200 for each action type as follows:
What didn't work
\nWe kept monitoring the feature importance in Google Sheets, this allowed us to discuss and figure out new features efficiently
\n![\"feature-importance\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3563032%2Ff465e0967c097c07277c07f51d3080d8%2Ffeature-importance.png?generation=1675296313006485&alt=media\")
The 1st version of our feature set only contains 78 features, and we got 0.592 on LB and 0.5802 on CV. Finally added up to ~500 features, and we achieved 0.596 on LB with a single LGB model.
click2cart_or_order)n_covisit_{click, cart, order}, n_incovisit_{click, cart, order}, in_cosivist_{rank, avg_rank}{time, type, buy}_covisit_occurs_{timeweighted, typeweighted, recency weighted score} {time, type, buy}_covisit_rank_{timeweighted, typeweighted, recency weighted score}{click, cart, order}_word2vec_similarity_{min, max, mean, sum} between target aid and user interacted aidslambdarank objectiverank:pairwise objectiveWe searched the weights of each ranker's predicted scores based on the recall@20 of the local validation candidates using optuna, which means the recall@20 is calculated by weighted summing the scores of Tens of Millions of samples and calculating the recall@20. By leveraging the cudf and my implemented calc_recall_fast function, each trial only takes 700ms for carts (60M samples). We set the weight range to be (-1, 1) to get a better score:
() -> :\n weights = [ / (pred_cols)] * (pred_cols)\n i ((weights)):\n weights[i] = trial.suggest_float(, -, , step=)\n calc_recall_fast(\n model_preds,\n ground_truths,\n sess_len_cumsum,\n weights,\n gt_cnt\n )\nstudy = optuna.create_study(direction=, study_name=)\nstudy.optimize(objective, n_trials=, show_progress_bar=)\nFirst, I would like to thank the organizers and those who shared knowledge. Especially we would like to thank @radek1 for sharing so many valuable notebooks and datasets, and @cdeotte for explaining many questions in detail.
\nWill Release Some Reproducible Code When The LB is Finalized
\nSince @radek1's CV setting tracks public LB perfectly, we directly use his datasets in this competition. The following graph shows how we leverage his datasets for feature engineering, local validation, and submission generation.
\ntrain.parquet and test.parquet, full's train.parquet and test.parquet separately.Our retrieving strategy is quite simple, we do some hyperparameters tuning on @cdeotte 's public notebook. Mainly the topN of each kind of co-visitation matrix and the N_REC for each session. The optimal topN for each co-visitation matrix is 100, and the optimal N_REC is 200. Then we use @tuongkhang 's public notebook to generate the recall200 candidates for both local validation and test submission. Then we were able to get recall@200 for each action type as follows:
What didn't work
\nWe kept monitoring the feature importance in Google Sheets, this allowed us to discuss and figure out new features efficiently
\n
The 1st version of our feature set only contains 78 features, and we got 0.592 on LB and 0.5802 on CV. Finally added up to ~500 features, and we achieved 0.596 on LB with a single LGB model.
click2cart_or_order)n_covisit_{click, cart, order}, n_incovisit_{click, cart, order}, in_cosivist_{rank, avg_rank}{time, type, buy}_covisit_occurs_{timeweighted, typeweighted, recency weighted score} {time, type, buy}_covisit_rank_{timeweighted, typeweighted, recency weighted score}{click, cart, order}_word2vec_similarity_{min, max, mean, sum} between target aid and user interacted aidslambdarank objectiverank:pairwise objectiveWe searched the weights of each ranker's predicted scores based on the recall@20 of the local validation candidates using optuna, which means the recall@20 is calculated by weighted summing the scores of Tens of Millions of samples and calculating the recall@20. By leveraging the cudf and my implemented calc_recall_fast function, each trial only takes 700ms for carts (60M samples). We set the weight range to be (-1, 1) to get a better score:
() -> :\n weights = [ / (pred_cols)] * (pred_cols)\n i ((weights)):\n weights[i] = trial.suggest_float(, -, , step=)\n calc_recall_fast(\n model_preds,\n ground_truths,\n sess_len_cumsum,\n weights,\n gt_cnt\n )\nstudy = optuna.create_study(direction=, study_name=)\nstudy.optimize(objective, n_trials=, show_progress_bar=)\nCongrats. It's exactly same as our solution but I guess you have better features.
", "rawMarkdown": "Congrats. It's exactly same as our solution but I guess you have better features.", "votes": null }, { "id": "2125828", "postDate": "02/01/2023 23:26:02", "content": "Thanks, I'm updating the feature engineering and ranker parts. You can check later
", "rawMarkdown": "Thanks, I'm updating the feature engineering and ranker parts. You can check later", "votes": null }, { "id": "2125917", "postDate": "02/02/2023 01:55:48", "content": "Very clear pipeline plot! When you constructing the training samples for RANK model, do you use the train part (first three weeks) and random split it to input sequence and label?
", "rawMarkdown": "Very clear pipeline plot! When you constructing the training samples for RANK model, do you use the train part (first three weeks) and random split it to input sequence and label?", "votes": null }, { "id": "2125926", "postDate": "02/02/2023 02:17:40", "content": "Hi @evilpsycho42 , thanks! In short, (session, aid, label) is generated based on the 4th week's 1st split and 2nd split, whereas the features are generated based on 3 weeks + 4th week's 1st split.
(session, aid, label) in the training samples are based on radek's test.parquet and test_labels.parquet in the his local validation dataset, where the features are generated based on the train.parquet and test.parquet in this dataset.
Congrats! and thanks for sharing
", "rawMarkdown": "Congrats! and thanks for sharing", "votes": null }, { "id": "2126644", "postDate": "02/02/2023 11:01:01", "content": "How much did you benefit from your ensemble method?
", "rawMarkdown": "How much did you benefit from your ensemble method?", "votes": null }, { "id": "2126931", "postDate": "02/02/2023 15:14:22", "content": "Not that much, cv from 0.584491 to 0.585179
", "rawMarkdown": "Not that much, cv from 0.584491 to 0.585179", "votes": null }, { "id": "2132916", "postDate": "02/07/2023 05:41:02", "content": "When our team also used optuna to adjust my model, we always found that the parameters searched by optuna and the final model effect could not be reproduced. Has your team encountered this?
", "rawMarkdown": "When our team also used optuna to adjust my model, we always found that the parameters searched by optuna and the final model effect could not be reproduced. Has your team encountered this?", "votes": null }, { "id": "2133063", "postDate": "02/07/2023 08:28:51", "content": "\n\nthe parameters searched by optuna and the final model effect could not be reproduced
\n
Did you compare this to the local CV score? If so, given the weights, the local score is determined.
", "rawMarkdown": "> the parameters searched by optuna and the final model effect could not be reproduced\n\nDid you compare this to the local CV score? If so, given the weights, the local score is determined.", "votes": null }, { "id": "2133146", "postDate": "02/07/2023 09:15:33", "content": "No, I use optuna to optimize the parameters of my tree model and not the model fusion parameters. However, the parameter substitution model searched by optuna cannot reproduce the corresponding results.
", "rawMarkdown": "No, I use optuna to optimize the parameters of my tree model and not the model fusion parameters. However, the parameter substitution model searched by optuna cannot reproduce the corresponding results.", "votes": null }, { "id": "2133272", "postDate": "02/07/2023 10:38:31", "content": "For catboost and xgboost GPU implementation, there is some randomness during training, even you set random_state, the result cannot be 100% reproduced.
\nBut for lgbm, if you set random_state, the result can be reproduced to be exactly the same. (gradient boosting mode, cpu training)
I found that even after doing this; I still can't reproduce my results (I will substitute the model parameters in the picture); here is part of my code。
\n random\nrandom.seed()\n\n ():\n scores = \n reg_alpha = trial.suggest_float(, , ),\n reg_lambda = trial.suggest_float(, , ),\n num_leaves = trial.suggest_int(, , ),\n min_child_samples = trial.suggest_int(, , ),\n colsample_bytree = trial.suggest_float(, , ),\n learning_rate = trial.suggest_float(, , ),\n fold,(train_idx, valid_idx) (skf.split(df_feature_sample, df_feature_sample[], groups=df_feature_sample[] )):\n X_train = df_feature_sample.loc[train_idx, FEATURES]\n y_train = df_feature_sample.loc[train_idx, ]\n X_valid = df_feature_sample.loc[valid_idx, FEATURES]\n y_valid = df_feature_sample.loc[valid_idx, ]\n group_train=df_feature_sample.loc[train_idx].groupby().size()[df_feature_sample.loc[train_idx].session.drop_duplicates()]\n group_vali=df_feature_sample.loc[valid_idx].groupby().size()[df_feature_sample.loc[valid_idx].session.drop_duplicates()]\n ranker = lgb.LGBMRanker(\n objective=,\n metric=,\n boosting_type=,\n importance_type=,\n n_estimators=,\n eval_at= ,\n early_stopping_round=,\n n_jobs = -,\n random_state=,\n reg_alpha = reg_alpha,\n reg_lambda = reg_lambda,\n num_leaves = num_leaves,\n min_child_samples = min_child_samples,\n colsample_bytree = colsample_bytree,\n learning_rate=learning_rate,\n ) \n\n ranker.fit(X_train,\n y_train, \n eval_set=[(X_train, y_train), (X_valid, y_valid)],\n group=group_train,\n eval_group=[group_train,group_vali])\n score = gen_score(ranker)\n scores += score\n score<:\n * score\n scores\n\n\nsampler = TPESampler(seed=) \nstudy = optuna.create_study(\n direction=,\n sampler=sampler,\n study_name=)\nstudy.optimize(objective, n_trials=)\n![\"--\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3562400%2Fbc7e0335bcba2de98cda856220ad99ae%2F111111.jpg?generation=1675767526395608&alt=media\")
First, I guess dropout used in DART mode induces some extent of randomness, make dart mode difficult to be 100% deterministic. Just my speculation.
\nSecond, the native early stop API provided by lgbm should not be used together with DART mode. It won't work. If you want to use early stop in DART mode, you need to write your own early stop callback.
thanks, i think it's possible
", "rawMarkdown": "thanks, i think it's possible", "votes": null }, { "id": "2135849", "postDate": "02/08/2023 22:30:35", "content": "Both LGB and CatBoost's results are not reproducible. However, after many training experiments, I found that the gap between the tuning stage and the training stage can be decreased by increasing the early_stopping_round, I raised this param to 300 for this competition. The gap is relatively small.
\nIf you set a relatively small early_stopping_round e.g. 20 or 40. Then most likely the early stopping is triggered before your model actually converges. Because of the randomness (even with a fixed seed), it could be triggered earlier during training than tuning. Then you'll see a big gap between training and tuning regarding the model performance.
Congrats. It's exactly same as our solution but I guess you have better features.
", "votes": null, "replies": [ { "id": 2125828, "author_name": "wuwenmin", "author_url": "", "post_date": "02/01/2023 23:26:02", "content": "Thanks, I'm updating the feature engineering and ranker parts. You can check later
", "votes": null, "replies": [] } ] }, { "id": 2125917, "author_name": "evilpsycho42", "author_url": "", "post_date": "02/02/2023 01:55:48", "content": "Very clear pipeline plot! When you constructing the training samples for RANK model, do you use the train part (first three weeks) and random split it to input sequence and label?
", "votes": null, "replies": [ { "id": 2125926, "author_name": "wuwenmin", "author_url": "", "post_date": "02/02/2023 02:17:40", "content": "Hi @evilpsycho42 , thanks! In short, (session, aid, label) is generated based on the 4th week's 1st split and 2nd split, whereas the features are generated based on 3 weeks + 4th week's 1st split.
(session, aid, label) in the training samples are based on radek's test.parquet and test_labels.parquet in the his local validation dataset, where the features are generated based on the train.parquet and test.parquet in this dataset.
Congrats! and thanks for sharing
", "votes": null, "replies": [] }, { "id": 2126644, "author_name": "zachary666", "author_url": "", "post_date": "02/02/2023 11:01:01", "content": "How much did you benefit from your ensemble method?
", "votes": null, "replies": [ { "id": 2126931, "author_name": "wuwenmin", "author_url": "", "post_date": "02/02/2023 15:14:22", "content": "Not that much, cv from 0.584491 to 0.585179
", "votes": null, "replies": [] } ] }, { "id": 2132916, "author_name": "yasso1", "author_url": "", "post_date": "02/07/2023 05:41:02", "content": "When our team also used optuna to adjust my model, we always found that the parameters searched by optuna and the final model effect could not be reproduced. Has your team encountered this?
", "votes": null, "replies": [ { "id": 2133063, "author_name": "wuwenmin", "author_url": "", "post_date": "02/07/2023 08:28:51", "content": "\n\nthe parameters searched by optuna and the final model effect could not be reproduced
\n
Did you compare this to the local CV score? If so, given the weights, the local score is determined.
", "votes": null, "replies": [ { "id": 2133146, "author_name": "yasso1", "author_url": "", "post_date": "02/07/2023 09:15:33", "content": "No, I use optuna to optimize the parameters of my tree model and not the model fusion parameters. However, the parameter substitution model searched by optuna cannot reproduce the corresponding results.
", "votes": null, "replies": [ { "id": 2133272, "author_name": "buumoo", "author_url": "", "post_date": "02/07/2023 10:38:31", "content": "For catboost and xgboost GPU implementation, there is some randomness during training, even you set random_state, the result cannot be 100% reproduced.
\nBut for lgbm, if you set random_state, the result can be reproduced to be exactly the same. (gradient boosting mode, cpu training)
I found that even after doing this; I still can't reproduce my results (I will substitute the model parameters in the picture); here is part of my code。
\n random\nrandom.seed()\n\n ():\n scores = \n reg_alpha = trial.suggest_float(, , ),\n reg_lambda = trial.suggest_float(, , ),\n num_leaves = trial.suggest_int(, , ),\n min_child_samples = trial.suggest_int(, , ),\n colsample_bytree = trial.suggest_float(, , ),\n learning_rate = trial.suggest_float(, , ),\n fold,(train_idx, valid_idx) (skf.split(df_feature_sample, df_feature_sample[], groups=df_feature_sample[] )):\n X_train = df_feature_sample.loc[train_idx, FEATURES]\n y_train = df_feature_sample.loc[train_idx, ]\n X_valid = df_feature_sample.loc[valid_idx, FEATURES]\n y_valid = df_feature_sample.loc[valid_idx, ]\n group_train=df_feature_sample.loc[train_idx].groupby().size()[df_feature_sample.loc[train_idx].session.drop_duplicates()]\n group_vali=df_feature_sample.loc[valid_idx].groupby().size()[df_feature_sample.loc[valid_idx].session.drop_duplicates()]\n ranker = lgb.LGBMRanker(\n objective=,\n metric=,\n boosting_type=,\n importance_type=,\n n_estimators=,\n eval_at= ,\n early_stopping_round=,\n n_jobs = -,\n random_state=,\n reg_alpha = reg_alpha,\n reg_lambda = reg_lambda,\n num_leaves = num_leaves,\n min_child_samples = min_child_samples,\n colsample_bytree = colsample_bytree,\n learning_rate=learning_rate,\n ) \n\n ranker.fit(X_train,\n y_train, \n eval_set=[(X_train, y_train), (X_valid, y_valid)],\n group=group_train,\n eval_group=[group_train,group_vali])\n score = gen_score(ranker)\n scores += score\n score<:\n * score\n scores\n\n\nsampler = TPESampler(seed=) \nstudy = optuna.create_study(\n direction=,\n sampler=sampler,\n study_name=)\nstudy.optimize(objective, n_trials=)\n
First, I guess dropout used in DART mode induces some extent of randomness, make dart mode difficult to be 100% deterministic. Just my speculation.
\nSecond, the native early stop API provided by lgbm should not be used together with DART mode. It won't work. If you want to use early stop in DART mode, you need to write your own early stop callback.
thanks, i think it's possible
", "votes": null, "replies": [ { "id": 2135849, "author_name": "wuwenmin", "author_url": "", "post_date": "02/08/2023 22:30:35", "content": "Both LGB and CatBoost's results are not reproducible. However, after many training experiments, I found that the gap between the tuning stage and the training stage can be decreased by increasing the early_stopping_round, I raised this param to 300 for this competition. The gap is relatively small.
\nIf you set a relatively small early_stopping_round e.g. 20 or 40. Then most likely the early stopping is triggered before your model actually converges. Because of the randomness (even with a fixed seed), it could be triggered earlier during training than tuning. Then you'll see a big gap between training and tuning regarding the model performance.