![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F397595%2Fcff84b84bab47c015ef64ab84e20de5d%2Fmakotu_solution_v2.png?generation=1675314772195222&alt=media\")
PS: This solution is actually equivalent to 3rd place, but it was ban because a cheater was on the team. So please enjoy it as an \"imaginary\" 3rd place solution.
\nFirst of all, thank you for organizing a great competition. I would also like to thank everyone who participated in the competition with us and the four of us who worked together as a team.
\nAbout one week before the end of the competition, we formed a team. However, just on the next day, we were warned one of our teammate might have some problem(link). And since then, we other four were working hard together to achieve the best possible result.
\nWe can proudly tell you that this 3rd place score is definitely a result with no outside influence. (Of course, as we have already said, we leave it up to the kaggle team to decide how our results will ultimately be handled.)
\nIn this discussion, I would be happy to share with you the solutions of the four of us.
\nOf course we can share the code as well, but we hope we can open it after the rankings are finalized.
\n(The code is not yet public at this time, so it would be difficult for cheaters to imitate our accuracy.)
We write the solution each one below.
\nMy pipeline consists mainly of preprocess/make co-matrix/candidate / make feature / modeling parts.
\n
I was not involved in the stacking of results with other members, as all the stacking was carried out by shimacos with great skill. I hope you will wait for his addition.
\nhttps://github.com/makotu1208/Otto-kaggle-3rd-solution-makotupart
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F397595%2F9602e046e48f6d2028d0a05ccf898720%2Fsolution_shimacos.png?generation=1675249326142169&alt=media\"){kind=spacer}
1/rank_a + 1/rank_b + 1/rank_c and sort descending.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F397595%2Ffa7ab3c2005f4af8929e6ed5e7b6f42b%2Fsolution_sirius.png?generation=1675249468500843&alt=media\")
20*len(pos_samples)That’s @shimacos ’s credit. Very impressive work!
", "messages": [ { "id": "2124992", "postDate": "02/01/2023 11:07:27", "content": "PS: This solution is actually equivalent to 3rd place, but it was ban because a cheater was on the team. So please enjoy it as an \"imaginary\" 3rd place solution.
\nFirst of all, thank you for organizing a great competition. I would also like to thank everyone who participated in the competition with us and the four of us who worked together as a team.
\nAbout one week before the end of the competition, we formed a team. However, just on the next day, we were warned one of our teammate might have some problem(link). And since then, we other four were working hard together to achieve the best possible result.
\nWe can proudly tell you that this 3rd place score is definitely a result with no outside influence. (Of course, as we have already said, we leave it up to the kaggle team to decide how our results will ultimately be handled.)
\nIn this discussion, I would be happy to share with you the solutions of the four of us.
\nOf course we can share the code as well, but we hope we can open it after the rankings are finalized.
\n(The code is not yet public at this time, so it would be difficult for cheaters to imitate our accuracy.)
We write the solution each one below.
\nMy pipeline consists mainly of preprocess/make co-matrix/candidate / make feature / modeling parts.
\n
I was not involved in the stacking of results with other members, as all the stacking was carried out by shimacos with great skill. I hope you will wait for his addition.
\nhttps://github.com/makotu1208/Otto-kaggle-3rd-solution-makotupart
\n
1/rank_a + 1/rank_b + 1/rank_c and sort descending.
20*len(pos_samples)That’s @shimacos ’s credit. Very impressive work!
", "rawMarkdown": "**PS: This solution is actually equivalent to 3rd place, but it was ban because a cheater was on the team. So please enjoy it as an \"imaginary\" 3rd place solution.**\n\nFirst of all, thank you for organizing a great competition. I would also like to thank everyone who participated in the competition with us and the four of us who worked together as a team.\nAbout one week before the end of the competition, we formed a team. However, just on the next day, we were warned one of our teammate might have some problem([link](https://www.kaggle.com/competitions/otto-recommender-system/discussion/381321)). And since then, we other four were working hard together to achieve the best possible result. \nWe can proudly tell you that this 3rd place score is definitely a result with no outside influence. (Of course, as we have already said, we leave it up to the kaggle team to decide how our results will ultimately be handled.)\nIn this discussion, I would be happy to share with you the solutions of the four of us.\nOf course we can share the code as well, but we hope we can open it after the rankings are finalized.\n(The code is not yet public at this time, so it would be difficult for cheaters to imitate our accuracy.)\n\nWe write the solution each one below.\n\n-----------------------------------------------------------------------\n\n## Alvor part\n### Candidates selection:\n- all items from the session's history\n- items from co-visitation matrices. I did many experiments (window size, weights, action types etc.) and finally end up with 5 best matrices (in terms of maximum possible recall when using each matrix). Each of these 5 matrices I use both for candidates selection and for feature engineering (sum/avg/max candidate score, etc.)\n- I did not have fixed number of candidates per session\n- median number of candidates per session: 67\n- but there were sessions with very high number of candidates, so the average candidates number per session is 121.46\n- my recalls per target:\nclicks recall: 0.631531\ncarts recall: 0.540274\norders recall: 0.729626\nTotal: 0.6630\nWhat did not work for me:\n- add candidates from Word2Vec embeddings (K nearest neighbours)\n- add candidates from Matrix Factorization embeddings (K nearest neighbours)\n- simply increase number of candidates in my methods\n- all these attempts increased the maximum possible recall, but decreased the validation score\n### Feature engineering:\n- Sessions features\n- Items features\n- Items-2-Sessions interactions features\n- Co-Visitation matrices features (sum/avg/max of candidate score, etc.)\n- Word2Vec features: euclidean and angular distance between the candidate's embedding and session's last item's embedding\n- Matrix Factorization features: euclidean and angular distance between the candidate's embedding and last session's item's embedding\n- Features selection based on Feature importance of my experimental models\nSome of my best candidates:\n- index of last interaction with the candidate in the session\n- relative number of interactions with given candidate among all session's interactions\n- timestamp differences\n- angular distance between candidate's w2v embedding and last session's item w2v embedding\n- number of co-visitation matrices containing given item as a candidate\nWhat did not work for me:\n- Word2Vec and Matrix Factorization features: euclidean and angular distance between the candidate's embedding and session's penultimate item's embedding\n- Word2Vec and Matrix Factorization features: average/min euclidean and angular distance between the candidate's embedding and all session's items embeddings\n- Word2Vec and Matrix Factorization features: euclidean and angular distance between the candidate's embedding and session's last cart/order item's embedding (there was a minor improvement, but I abandoned these features)\n### Model\n- Group 3-Fold (by sessions) LightGBM Classifier (binary_logloss)\n- One model per one target (clicks/carts/orders)\n- No negative downsample (during almost the entire competition) so it required a lot of memory and time\n- 1/5 negative downsample for carts/orders and 1/15 for clicks (last week of competition)\n- But I use another trick to reduce memory/time usage: When I train a model for carts/orders, I use only sessions for which I have at least one ground-truth candidate among my candidates. Because the main goal of model is to distinguish between good candidates and bad candidates. So I think that sessions with only bad candidates don't help my model.\n (It could be by 2 reasons:\n 1) the session has not ground true cart/order at all\n 2) the session has ground-true cart/order, but I didn't select it among my candidates)\n- 'dart' boosting for carts nodel, 'gbdt' boosting for clicks and orders model.\n- add \"carts\" prediction as a feature for \"orders\" model (the best feature)\nWhat did not work for me:\n- Add second click to the positive samples as well as first click\n- add \"clicks\" prediction as a feature for \"orders\" model\n- multiple attempts to build a second model to re-arrange only top-X candidates from the first model\n\n-----------------------------------------------------------------------\n\n## Makotu part\nMy pipeline consists mainly of preprocess/make co-matrix/candidate / make feature / modeling parts.\n\n\n\n### Data\n- I used Radek's train/validation/test data\n### Preprocess\n- make aid vector using w2v\n- make BPR(Bayesian Personalized Ranking) feature (recommended Sirius)\n### Make-co-matrix and candidate\n- I made many pattern co-matrix. Basically, co-matrix is created in the form of probabilities. For example, the following pattern is used to quantify the relationship between aidA and aidB.\n - aid B count within same session / all aid A count\n - aid B count within same session (after aid A click) / all aid A count\n - aid B count within same session / all aid A count. (But if the same user has the same pair of idA and idB, make them unique and then aggregate)\n - aid B count within same session / all aid A count. (with time weighted)\n - aid B count within same session and within 1 hour / all aid A count.\n like this.\n- Select a candidate based on session last action aid / top action aid / with in 1 hour action aid / within 1 day action aid\n- candidate recall\n - order: 0.732 (@190 candidates)\n - cart: 0.546 (@190 candidates)\n - click: 0.682 (@140 candidates)\n### Make feature\n- User / aid / user&aid interaction features.\n- features that worked\n - last action aid\n - distance from last aid w2v to candidate aid w2v\n - BPR feature (recommened sirius)\n - co-matrix features (ex. probability of clicking on candidate aid after the last aid)\n - etc\n### Modeling\n- Catboost (loss: PairLogitPairwise)\n- Group Kfold(group: session)\n- Add oof features in addition to the above features\n - click model does not use oof.\n However, two patterns of click models were created. in addition to predicting the next clicked id, it also predicts all subsequent clicks of the id, and then uses as oof.\n - cart model add above 2 model oof.\n - order model add click 2 model oofs, cart model's oof\n- CV click: 0.5632 cart: 0.4429 order: 0.6699\n- LB: 0.602\n### Stacking\nI was not involved in the stacking of results with other members, as all the stacking was carried out by shimacos with great skill. I hope you will wait for his addition. \n### (add) code of my part\nhttps://github.com/makotu1208/Otto-kaggle-3rd-solution-makotupart\n\n-----------------------------------------------------------------------\n\n## Shimacos part\n\n\n### data\n- First, I used my own original truncate dataset.\n - train: 2022-08-14 ~ 2022-08-21 valid 2022-08-21 ~ 2022-08-28\n - train: 2022-08-21 ~ 2022-08-28 test: 2022-08-28 ~ 2022-09-04 (Training iteration is the calculated one in the previous step)\n- And Roughly 10 or so models were made and stacking. (Single 0.592 -> 0.595)\n- However since everyone except me was using the radek dataset, I shifted to using that one after team merge. This was to maximize ensemble results.\n### preprocess\n- made aid vector\n - w2v with gensim\n - n2v with pytorch_geometric\n- made session, aid vector\n - bpr with implicit\n### Candidates\n- As shown in the overview figure.\n- I created candidates and features by BigQuery. For example, the covisit candidates can be calculated in less than one minute for both train and test.\n- Recall\n - average candidate count: 188.69\n - click: 0.6799\n - cart: 0.5555\n - order: 0.7359\n - Overall: 0.6762\n### Feature\n- As shown in the overview figure.\n- I used the overall average for smoothing when calculating cvr.\n- High importance feature (CatBoost LossFunctionChange importance)\n - difference between the last time the aid was actioned across the overall data and the last time of the session.\n - candidate feature by next visit covisit\n - I think it was because in the end it was important to hit the next aid after the last action.\n - difference feature of w2v and bpr\n - covisit cvr feature\n### Model\n- LightGBM (lambdarank) and CatBoost (PairLogit)\n- I used CatBoost the day before the last day and found an improvement of about 0.002-0.003 over LightGBM. I should have used it earlier...\n - I got 0.596 and 0.598 at private LB finally.\n### Stacking\n- At the team merge, the scores were only 0.600 (makotu), 0.600 (sirius),0.598 (Alvor and Shimacos), but By rank blending each score as follows, we were able to produce a 0.603, which was the second at the time.\n - Use the top50 of each prediction.\n - And the each prediction was simply outer joined to create a pair of user items.\n - fillna with 1 / 50\n - rank blending with formula `1/rank_a + 1/rank_b + 1/rank_c ` and sort descending.\n- As the number of models increased, it became difficult to find the optimal weights, so stacking was performed.\n - Dataset is the same as above.\n - Feature: prediction score and rank\n- This resulted in a score of 0.604 by LightGBM. (CV: 0.59624)\n- The same was created in CatBoost (CV: 0.59592) and averaged.\n- Finally, I interpolated predictions by aid with the highest number of actions for each type in the test period.\n- Random Thought\n - Too many more models could lower the score. This is probably due to too many negative examples.\n - Stacking models created by different candidates tended to increase the overall score. \n\n----------------------------------------------------------------------- \n\n## Sirius part\n\n\n### Recalling\n- All the recalling methods are listed in the figure.\n- Avg candidates cnt for each user: 224.74\n- Overall recall: 0.677232,\n - Clicks recall:0.688493\n - Carts recall:0.557334\n - Orders recall:0.735304\n### Ranking\n#### Samples\n- Click model: the generated candidates which the user clicked are labeled with 1\n- Cart/Order model: the generated candidates which the user carted or ordered are labeled with 1. I merged cart and order labels because of their sharing of similar semantic\n- Downsampling of negtive samples is abopted. I just keep the negative samples with amount of `20*len(pos_samples)`\n\n#### Features\n- All the features are listed in the figure.\n- The feature with highest importance in my model is BPR①, implemented with Implicit package. ([Notebook](https://www.kaggle.com/code/sirius81/bpr-feature/notebook) will be public after everything is verified.)\n- Bi-gram② is a terminology of NLP, meaning the successive visiting of aid1, aid2 here. It is similar to co-visit matrix, but it only counts the next one action. And I normlize the count by the hotness of item then. This group of features gave the biggest boost among all my experiments, ~0.006. ([Notebook](https://www.kaggle.com/code/sirius81/otto-bigram-feature/notebook) will be public after everything is verified.)\n- OOF of other action③ help improve my score ~0.001.\n- Features ①+③+ shimacos’s w2v improved Makotu's CV ~0.001 and features ①+② improved Alvor's CV ~0.003.\n\n#### Models\n- Catboost is trained with Logloss. When training Cart/Order model, I set different weights to positive sample, 5 and 10 for is_carts and is_orders respectively. Along with the strategy of mergeing Cart labels and Order labels, it boosts CV ~0.002, compared with training Cart model and Order model separately.\n- MLP is trained with the same features as catboost, plus an standard scaler to normlize the features. But it is not accurate and diverse enough to improve the blending score. So in our final submission, it is not included. Beside MLP, I have tried to introduce GRU and pretrained item embedding into the NN model, but all failed.\n### Ensemble\nThat’s @shimacos ’s credit. Very impressive work!", "votes": null }, { "id": "2125148", "postDate": "02/01/2023 13:15:47", "content": "Thanks for the write up
\nAs the number of models increased, it became difficult to find the optimal weights, so stacking was performed.\nDataset is the same as above.\nFeature: prediction score and rank
Which dataset was used for stacking? Was there a OOF set?
", "rawMarkdown": "Thanks for the write up\n\n`As the number of models increased, it became difficult to find the optimal weights, so stacking was performed.\nDataset is the same as above.\nFeature: prediction score and rank`\n\nWhich dataset was used for stacking? Was there a OOF set?", "votes": null }, { "id": "2125159", "postDate": "02/01/2023 13:24:32", "content": "Yes, I used oof of radek dataset.
\nThis was possible because our team all used Redek's dataset.
Are you not using Radek's 4th week truncated dataset to train your individual reranker models? Makotu, Alvor and Sirius Model? On which dataset do you train these models and on which is the ensembled trained?
", "rawMarkdown": "Are you not using Radek's 4th week truncated dataset to train your individual reranker models? Makotu, Alvor and Sirius Model? On which dataset do you train these models and on which is the ensembled trained?", "votes": null }, { "id": "2125172", "postDate": "02/01/2023 13:52:59", "content": "We used Radek's 4th week truncated dataset to train by GroupKFold.
\nThen built stacking model by the oof prediction.
\nThe following is a section on how to create the dataset.
\n", "rawMarkdown": "We used Radek's 4th week truncated dataset to train by GroupKFold.\nThen built stacking model by the oof prediction.\nThe following is a section on how to create the dataset.\n> Use the top50 of each prediction.\nAnd the each prediction was simply outer joined to create a pair of user items.", "votes": null }, { "id": "2125188", "postDate": "02/01/2023 14:05:04", "content": "Use the top50 of each prediction.
\n
\n And the each prediction was simply outer joined to create a pair of user items.
Wow, so much content here. Great solution, congrats guys!
", "rawMarkdown": "Wow, so much content here. Great solution, congrats guys!", "votes": null }, { "id": "2125266", "postDate": "02/01/2023 14:56:05", "content": "No matter what the final result will be, cooperating with you guys is super pleasant. Thank you very much!
", "rawMarkdown": "No matter what the final result will be, cooperating with you guys is super pleasant. Thank you very much!", "votes": null }, { "id": "2125526", "postDate": "02/01/2023 17:34:52", "content": "Great explanation a lot to learn here .. specially use of angular distance (which we havent used and weren't aware of). Lessons to take in future comps . Also thanks @sirius81 for your tips from your previous H&M post helped us a lot for this competition :)
", "rawMarkdown": "Great explanation a lot to learn here .. specially use of angular distance (which we havent used and weren't aware of). Lessons to take in future comps . Also thanks @sirius81 for your tips from your previous H&M post helped us a lot for this competition :)", "votes": null }, { "id": "2125861", "postDate": "02/02/2023 00:13:53", "content": "Hi @gauravbrills , thanks for your comments. Very glad to hear that!
", "rawMarkdown": "Hi @gauravbrills , thanks for your comments. Very glad to hear that!", "votes": null }, { "id": "2125944", "postDate": "02/02/2023 02:42:55", "content": "I admire your team! Thank you for your hard work and nice sharing.
", "rawMarkdown": "I admire your team! Thank you for your hard work and nice sharing.", "votes": null }, { "id": "2125996", "postDate": "02/02/2023 03:39:59", "content": "Will you provide the code later? I would like to learn the details of the processing inside
", "rawMarkdown": "Will you provide the code later? I would like to learn the details of the processing inside", "votes": null }, { "id": "2126073", "postDate": "02/02/2023 04:42:24", "content": "Thanks for sharing and great team from 4 of you!! I hope Kaggle makes fair decision.
\n\n\nBi-gram② is a terminology of NLP, meaning the successive visiting of here. It is similar to co-visit matrix, but it only counts the next one action.
\n
Is it similar(or equal) to co-occurence i.e. count of (item, next_item)?
", "rawMarkdown": "Thanks for sharing and great team from 4 of you!! I hope Kaggle makes fair decision. \n>Bi-gram② is a terminology of NLP, meaning the successive visiting of here. It is similar to co-visit matrix, but it only counts the next one action.\n\n\nIs it similar(or equal) to co-occurence i.e. count of (item, next_item)?", "votes": null }, { "id": "2126081", "postDate": "02/02/2023 04:52:30", "content": "Yes, Exactly.
", "rawMarkdown": "Yes, Exactly.", "votes": null }, { "id": "2127822", "postDate": "02/03/2023 08:10:00", "content": "Thank you for sharing! Really impressed by stacking, bi-gram, and the speed of BigQuery.
\nBesides BigQuery, what computational resources did you use, and what were the main considerations?
Thanks for sharing. Quite Remarkable.
", "rawMarkdown": "Thanks for sharing. Quite Remarkable.", "votes": null }, { "id": "2127864", "postDate": "02/03/2023 09:26:03", "content": "Impressive !
", "rawMarkdown": "Impressive !", "votes": null }, { "id": "2129644", "postDate": "02/04/2023 19:12:40", "content": "congratulations @mhyodo !!
", "rawMarkdown": "congratulations @mhyodo !!", "votes": null }, { "id": "2131791", "postDate": "02/06/2023 11:24:09", "content": "https://www.kaggle.com/code/sirius81/otto-bigram-feature/notebook 404😂
", "rawMarkdown": "https://www.kaggle.com/code/sirius81/otto-bigram-feature/notebook 404😂", "votes": null }, { "id": "2131801", "postDate": "02/06/2023 11:42:52", "content": "\"Notebook will be public after everything is verified.\" But I have added you in the sharing list now.
", "rawMarkdown": "\"[Notebook](https://www.kaggle.com/code/sirius81/otto-bigram-feature/notebook) will be public after everything is verified.\" But I have added you in the sharing list now.", "votes": null }, { "id": "2132694", "postDate": "02/07/2023 02:13:18", "content": "Thanks, I learned a lot of new things from your code
", "rawMarkdown": "Thanks, I learned a lot of new things from your code", "votes": null }, { "id": "2143194", "postDate": "02/14/2023 05:55:12", "content": "excuse me, may i ask how to determine the w2v model trained well, i train the w2v model for recall and cv top 200 get round 0.600, and i use the w2v embedding for rerank but my rerank doesn't work, i have not find the reason,and i think my w2v embedding is not good, so how to consider it
", "rawMarkdown": "excuse me, may i ask how to determine the w2v model trained well, i train the w2v model for recall and cv top 200 get round 0.600, and i use the w2v embedding for rerank but my rerank doesn't work, i have not find the reason,and i think my w2v embedding is not good, so how to consider it", "votes": null } ], "comments": [ { "id": 2125148, "author_name": "benediktschifferer", "author_url": "", "post_date": "02/01/2023 13:15:47", "content": "Thanks for the write up
\nAs the number of models increased, it became difficult to find the optimal weights, so stacking was performed.\nDataset is the same as above.\nFeature: prediction score and rank
Which dataset was used for stacking? Was there a OOF set?
", "votes": null, "replies": [ { "id": 2125159, "author_name": "shimacos", "author_url": "", "post_date": "02/01/2023 13:24:32", "content": "Yes, I used oof of radek dataset.
\nThis was possible because our team all used Redek's dataset.
Are you not using Radek's 4th week truncated dataset to train your individual reranker models? Makotu, Alvor and Sirius Model? On which dataset do you train these models and on which is the ensembled trained?
", "votes": null, "replies": [ { "id": 2125172, "author_name": "shimacos", "author_url": "", "post_date": "02/01/2023 13:52:59", "content": "We used Radek's 4th week truncated dataset to train by GroupKFold.
\nThen built stacking model by the oof prediction.
\nThe following is a section on how to create the dataset.
\n", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2125188, "author_name": "michau96", "author_url": "", "post_date": "02/01/2023 14:05:04", "content": "Use the top50 of each prediction.
\n
\n And the each prediction was simply outer joined to create a pair of user items.
Wow, so much content here. Great solution, congrats guys!
", "votes": null, "replies": [] }, { "id": 2125266, "author_name": "sirius81", "author_url": "", "post_date": "02/01/2023 14:56:05", "content": "No matter what the final result will be, cooperating with you guys is super pleasant. Thank you very much!
", "votes": null, "replies": [] }, { "id": 2125526, "author_name": "gauravbrills", "author_url": "", "post_date": "02/01/2023 17:34:52", "content": "Great explanation a lot to learn here .. specially use of angular distance (which we havent used and weren't aware of). Lessons to take in future comps . Also thanks @sirius81 for your tips from your previous H&M post helped us a lot for this competition :)
", "votes": null, "replies": [ { "id": 2125861, "author_name": "sirius81", "author_url": "", "post_date": "02/02/2023 00:13:53", "content": "Hi @gauravbrills , thanks for your comments. Very glad to hear that!
", "votes": null, "replies": [] } ] }, { "id": 2125944, "author_name": "songwonho", "author_url": "", "post_date": "02/02/2023 02:42:55", "content": "I admire your team! Thank you for your hard work and nice sharing.
", "votes": null, "replies": [] }, { "id": 2125996, "author_name": "yasso1", "author_url": "", "post_date": "02/02/2023 03:39:59", "content": "Will you provide the code later? I would like to learn the details of the processing inside
", "votes": null, "replies": [] }, { "id": 2126073, "author_name": "bibek777", "author_url": "", "post_date": "02/02/2023 04:42:24", "content": "Thanks for sharing and great team from 4 of you!! I hope Kaggle makes fair decision.
\n\n\nBi-gram② is a terminology of NLP, meaning the successive visiting of here. It is similar to co-visit matrix, but it only counts the next one action.
\n
Is it similar(or equal) to co-occurence i.e. count of (item, next_item)?
", "votes": null, "replies": [ { "id": 2126081, "author_name": "sirius81", "author_url": "", "post_date": "02/02/2023 04:52:30", "content": "Yes, Exactly.
", "votes": null, "replies": [] } ] }, { "id": 2127822, "author_name": "sunoonlee", "author_url": "", "post_date": "02/03/2023 08:10:00", "content": "Thank you for sharing! Really impressed by stacking, bi-gram, and the speed of BigQuery.
\nBesides BigQuery, what computational resources did you use, and what were the main considerations?
Thanks for sharing. Quite Remarkable.
", "votes": null, "replies": [] }, { "id": 2127864, "author_name": "alexpierron", "author_url": "", "post_date": "02/03/2023 09:26:03", "content": "Impressive !
", "votes": null, "replies": [] }, { "id": 2129644, "author_name": "sohilsharma1996", "author_url": "", "post_date": "02/04/2023 19:12:40", "content": "congratulations @mhyodo !!
", "votes": null, "replies": [] }, { "id": 2131791, "author_name": "yasso1", "author_url": "", "post_date": "02/06/2023 11:24:09", "content": "https://www.kaggle.com/code/sirius81/otto-bigram-feature/notebook 404😂
", "votes": null, "replies": [ { "id": 2131801, "author_name": "sirius81", "author_url": "", "post_date": "02/06/2023 11:42:52", "content": "\"Notebook will be public after everything is verified.\" But I have added you in the sharing list now.
", "votes": null, "replies": [ { "id": 2132694, "author_name": "yasso1", "author_url": "", "post_date": "02/07/2023 02:13:18", "content": "Thanks, I learned a lot of new things from your code
", "votes": null, "replies": [] } ] } ] }, { "id": 2143194, "author_name": "wcqglhf", "author_url": "", "post_date": "02/14/2023 05:55:12", "content": "excuse me, may i ask how to determine the w2v model trained well, i train the w2v model for recall and cv top 200 get round 0.600, and i use the w2v embedding for rerank but my rerank doesn't work, i have not find the reason,and i think my w2v embedding is not good, so how to consider it
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2124992": "**PS: This solution is actually equivalent to 3rd place, but it was ban because a cheater was on the team. So please enjoy it as an \"imaginary\" 3rd place solution.**\n\nFirst of all, thank you for organizing a great competition. I would also like to thank everyone who participated in the competition with us and the four of us who worked together as a team.\nAbout one week before the end of the competition, we formed a team. However, just on the next day, we were warned one of our teammate might have some problem([link](https://www.kaggle.com/competitions/otto-recommender-system/discussion/381321)). And since then, we other four were working hard together to achieve the best possible result. \nWe can proudly tell you that this 3rd place score is definitely a result with no outside influence. (Of course, as we have already said, we leave it up to the kaggle team to decide how our results will ultimately be handled.)\nIn this discussion, I would be happy to share with you the solutions of the four of us.\nOf course we can share the code as well, but we hope we can open it after the rankings are finalized.\n(The code is not yet public at this time, so it would be difficult for cheaters to imitate our accuracy.)\n\nWe write the solution each one below.\n\n-----------------------------------------------------------------------\n\n## Alvor part\n### Candidates selection:\n- all items from the session's history\n- items from co-visitation matrices. I did many experiments (window size, weights, action types etc.) and finally end up with 5 best matrices (in terms of maximum possible recall when using each matrix). Each of these 5 matrices I use both for candidates selection and for feature engineering (sum/avg/max candidate score, etc.)\n- I did not have fixed number of candidates per session\n- median number of candidates per session: 67\n- but there were sessions with very high number of candidates, so the average candidates number per session is 121.46\n- my recalls per target:\nclicks recall: 0.631531\ncarts recall: 0.540274\norders recall: 0.729626\nTotal: 0.6630\nWhat did not work for me:\n- add candidates from Word2Vec embeddings (K nearest neighbours)\n- add candidates from Matrix Factorization embeddings (K nearest neighbours)\n- simply increase number of candidates in my methods\n- all these attempts increased the maximum possible recall, but decreased the validation score\n### Feature engineering:\n- Sessions features\n- Items features\n- Items-2-Sessions interactions features\n- Co-Visitation matrices features (sum/avg/max of candidate score, etc.)\n- Word2Vec features: euclidean and angular distance between the candidate's embedding and session's last item's embedding\n- Matrix Factorization features: euclidean and angular distance between the candidate's embedding and last session's item's embedding\n- Features selection based on Feature importance of my experimental models\nSome of my best candidates:\n- index of last interaction with the candidate in the session\n- relative number of interactions with given candidate among all session's interactions\n- timestamp differences\n- angular distance between candidate's w2v embedding and last session's item w2v embedding\n- number of co-visitation matrices containing given item as a candidate\nWhat did not work for me:\n- Word2Vec and Matrix Factorization features: euclidean and angular distance between the candidate's embedding and session's penultimate item's embedding\n- Word2Vec and Matrix Factorization features: average/min euclidean and angular distance between the candidate's embedding and all session's items embeddings\n- Word2Vec and Matrix Factorization features: euclidean and angular distance between the candidate's embedding and session's last cart/order item's embedding (there was a minor improvement, but I abandoned these features)\n### Model\n- Group 3-Fold (by sessions) LightGBM Classifier (binary_logloss)\n- One model per one target (clicks/carts/orders)\n- No negative downsample (during almost the entire competition) so it required a lot of memory and time\n- 1/5 negative downsample for carts/orders and 1/15 for clicks (last week of competition)\n- But I use another trick to reduce memory/time usage: When I train a model for carts/orders, I use only sessions for which I have at least one ground-truth candidate among my candidates. Because the main goal of model is to distinguish between good candidates and bad candidates. So I think that sessions with only bad candidates don't help my model.\n (It could be by 2 reasons:\n 1) the session has not ground true cart/order at all\n 2) the session has ground-true cart/order, but I didn't select it among my candidates)\n- 'dart' boosting for carts nodel, 'gbdt' boosting for clicks and orders model.\n- add \"carts\" prediction as a feature for \"orders\" model (the best feature)\nWhat did not work for me:\n- Add second click to the positive samples as well as first click\n- add \"clicks\" prediction as a feature for \"orders\" model\n- multiple attempts to build a second model to re-arrange only top-X candidates from the first model\n\n-----------------------------------------------------------------------\n\n## Makotu part\nMy pipeline consists mainly of preprocess/make co-matrix/candidate / make feature / modeling parts.\n\n\n\n### Data\n- I used Radek's train/validation/test data\n### Preprocess\n- make aid vector using w2v\n- make BPR(Bayesian Personalized Ranking) feature (recommended Sirius)\n### Make-co-matrix and candidate\n- I made many pattern co-matrix. Basically, co-matrix is created in the form of probabilities. For example, the following pattern is used to quantify the relationship between aidA and aidB.\n - aid B count within same session / all aid A count\n - aid B count within same session (after aid A click) / all aid A count\n - aid B count within same session / all aid A count. (But if the same user has the same pair of idA and idB, make them unique and then aggregate)\n - aid B count within same session / all aid A count. (with time weighted)\n - aid B count within same session and within 1 hour / all aid A count.\n like this.\n- Select a candidate based on session last action aid / top action aid / with in 1 hour action aid / within 1 day action aid\n- candidate recall\n - order: 0.732 (@190 candidates)\n - cart: 0.546 (@190 candidates)\n - click: 0.682 (@140 candidates)\n### Make feature\n- User / aid / user&aid interaction features.\n- features that worked\n - last action aid\n - distance from last aid w2v to candidate aid w2v\n - BPR feature (recommened sirius)\n - co-matrix features (ex. probability of clicking on candidate aid after the last aid)\n - etc\n### Modeling\n- Catboost (loss: PairLogitPairwise)\n- Group Kfold(group: session)\n- Add oof features in addition to the above features\n - click model does not use oof.\n However, two patterns of click models were created. in addition to predicting the next clicked id, it also predicts all subsequent clicks of the id, and then uses as oof.\n - cart model add above 2 model oof.\n - order model add click 2 model oofs, cart model's oof\n- CV click: 0.5632 cart: 0.4429 order: 0.6699\n- LB: 0.602\n### Stacking\nI was not involved in the stacking of results with other members, as all the stacking was carried out by shimacos with great skill. I hope you will wait for his addition. \n### (add) code of my part\nhttps://github.com/makotu1208/Otto-kaggle-3rd-solution-makotupart\n\n-----------------------------------------------------------------------\n\n## Shimacos part\n\n\n### data\n- First, I used my own original truncate dataset.\n - train: 2022-08-14 ~ 2022-08-21 valid 2022-08-21 ~ 2022-08-28\n - train: 2022-08-21 ~ 2022-08-28 test: 2022-08-28 ~ 2022-09-04 (Training iteration is the calculated one in the previous step)\n- And Roughly 10 or so models were made and stacking. (Single 0.592 -> 0.595)\n- However since everyone except me was using the radek dataset, I shifted to using that one after team merge. This was to maximize ensemble results.\n### preprocess\n- made aid vector\n - w2v with gensim\n - n2v with pytorch_geometric\n- made session, aid vector\n - bpr with implicit\n### Candidates\n- As shown in the overview figure.\n- I created candidates and features by BigQuery. For example, the covisit candidates can be calculated in less than one minute for both train and test.\n- Recall\n - average candidate count: 188.69\n - click: 0.6799\n - cart: 0.5555\n - order: 0.7359\n - Overall: 0.6762\n### Feature\n- As shown in the overview figure.\n- I used the overall average for smoothing when calculating cvr.\n- High importance feature (CatBoost LossFunctionChange importance)\n - difference between the last time the aid was actioned across the overall data and the last time of the session.\n - candidate feature by next visit covisit\n - I think it was because in the end it was important to hit the next aid after the last action.\n - difference feature of w2v and bpr\n - covisit cvr feature\n### Model\n- LightGBM (lambdarank) and CatBoost (PairLogit)\n- I used CatBoost the day before the last day and found an improvement of about 0.002-0.003 over LightGBM. I should have used it earlier...\n - I got 0.596 and 0.598 at private LB finally.\n### Stacking\n- At the team merge, the scores were only 0.600 (makotu), 0.600 (sirius),0.598 (Alvor and Shimacos), but By rank blending each score as follows, we were able to produce a 0.603, which was the second at the time.\n - Use the top50 of each prediction.\n - And the each prediction was simply outer joined to create a pair of user items.\n - fillna with 1 / 50\n - rank blending with formula `1/rank_a + 1/rank_b + 1/rank_c ` and sort descending.\n- As the number of models increased, it became difficult to find the optimal weights, so stacking was performed.\n - Dataset is the same as above.\n - Feature: prediction score and rank\n- This resulted in a score of 0.604 by LightGBM. (CV: 0.59624)\n- The same was created in CatBoost (CV: 0.59592) and averaged.\n- Finally, I interpolated predictions by aid with the highest number of actions for each type in the test period.\n- Random Thought\n - Too many more models could lower the score. This is probably due to too many negative examples.\n - Stacking models created by different candidates tended to increase the overall score. \n\n----------------------------------------------------------------------- \n\n## Sirius part\n\n\n### Recalling\n- All the recalling methods are listed in the figure.\n- Avg candidates cnt for each user: 224.74\n- Overall recall: 0.677232,\n - Clicks recall:0.688493\n - Carts recall:0.557334\n - Orders recall:0.735304\n### Ranking\n#### Samples\n- Click model: the generated candidates which the user clicked are labeled with 1\n- Cart/Order model: the generated candidates which the user carted or ordered are labeled with 1. I merged cart and order labels because of their sharing of similar semantic\n- Downsampling of negtive samples is abopted. I just keep the negative samples with amount of `20*len(pos_samples)`\n\n#### Features\n- All the features are listed in the figure.\n- The feature with highest importance in my model is BPR①, implemented with Implicit package. ([Notebook](https://www.kaggle.com/code/sirius81/bpr-feature/notebook) will be public after everything is verified.)\n- Bi-gram② is a terminology of NLP, meaning the successive visiting of aid1, aid2 here. It is similar to co-visit matrix, but it only counts the next one action. And I normlize the count by the hotness of item then. This group of features gave the biggest boost among all my experiments, ~0.006. ([Notebook](https://www.kaggle.com/code/sirius81/otto-bigram-feature/notebook) will be public after everything is verified.)\n- OOF of other action③ help improve my score ~0.001.\n- Features ①+③+ shimacos’s w2v improved Makotu's CV ~0.001 and features ①+② improved Alvor's CV ~0.003.\n\n#### Models\n- Catboost is trained with Logloss. When training Cart/Order model, I set different weights to positive sample, 5 and 10 for is_carts and is_orders respectively. Along with the strategy of mergeing Cart labels and Order labels, it boosts CV ~0.002, compared with training Cart model and Order model separately.\n- MLP is trained with the same features as catboost, plus an standard scaler to normlize the features. But it is not accurate and diverse enough to improve the blending score. So in our final submission, it is not included. Beside MLP, I have tried to introduce GRU and pretrained item embedding into the NN model, but all failed.\n### Ensemble\nThat’s @shimacos ’s credit. Very impressive work!", "2125148": "Thanks for the write up\n\n`As the number of models increased, it became difficult to find the optimal weights, so stacking was performed.\nDataset is the same as above.\nFeature: prediction score and rank`\n\nWhich dataset was used for stacking? Was there a OOF set?", "2125159": "Yes, I used oof of radek dataset.\nThis was possible because our team all used Redek's dataset.", "2125165": "Are you not using Radek's 4th week truncated dataset to train your individual reranker models? Makotu, Alvor and Sirius Model? On which dataset do you train these models and on which is the ensembled trained?", "2125172": "We used Radek's 4th week truncated dataset to train by GroupKFold.\nThen built stacking model by the oof prediction.\nThe following is a section on how to create the dataset.\n> Use the top50 of each prediction.\nAnd the each prediction was simply outer joined to create a pair of user items.", "2125188": "Wow, so much content here. Great solution, congrats guys!", "2125266": "No matter what the final result will be, cooperating with you guys is super pleasant. Thank you very much!", "2125526": "Great explanation a lot to learn here .. specially use of angular distance (which we havent used and weren't aware of). Lessons to take in future comps . Also thanks @sirius81 for your tips from your previous H&M post helped us a lot for this competition :)", "2125861": "Hi @gauravbrills , thanks for your comments. Very glad to hear that!", "2125944": "I admire your team! Thank you for your hard work and nice sharing.", "2125996": "Will you provide the code later? I would like to learn the details of the processing inside", "2126073": "Thanks for sharing and great team from 4 of you!! I hope Kaggle makes fair decision. \n>Bi-gram② is a terminology of NLP, meaning the successive visiting of here. It is similar to co-visit matrix, but it only counts the next one action.\n\n\nIs it similar(or equal) to co-occurence i.e. count of (item, next_item)?", "2126081": "Yes, Exactly.", "2127822": "Thank you for sharing! Really impressed by stacking, bi-gram, and the speed of BigQuery. \nBesides BigQuery, what computational resources did you use, and what were the main considerations?", "2127855": "Thanks for sharing. Quite Remarkable.", "2127864": "Impressive !", "2129644": "congratulations @mhyodo !!", "2131791": "https://www.kaggle.com/code/sirius81/otto-bigram-feature/notebook 404😂", "2131801": "\"[Notebook](https://www.kaggle.com/code/sirius81/otto-bigram-feature/notebook) will be public after everything is verified.\" But I have added you in the sharing list now.", "2132694": "Thanks, I learned a lot of new things from your code", "2143194": "excuse me, may i ask how to determine the w2v model trained well, i train the w2v model for recall and cv top 200 get round 0.600, and i use the w2v embedding for rerank but my rerank doesn't work, i have not find the reason,and i think my w2v embedding is not good, so how to consider it" }, "source": "meta" }