Hi,
\n\nFirst, thanks kaggle and avito for organizing such a worderful competition and congrats to everyone! And thanks to teams sharing fantastic solutions!
\n\nAlthough my whole solution is neither strong nor novel and not worth sharing, I want to share a script which generated a group of my strong features - target encoding (also called mean encoding).
\n\n\n\nI knew it long ago and found that the normal TE overfitting easily but this coursera course (week3) got me know that it will only work when you take some generalizing tricks. In this competition I took cross-validated (CV) & time-series split (TS) two kinds of generalization.
\n\nIdeas are simple:
\n\nThe experiment shows the lgb model using CV-only target encoding feature (and other common features) would lead to 0.0055(k=20, f=2) or 0.0063(k=50, f=5) gap between OOF valid score and LB, but with CV&TS target encoding it gives consistent gaps of 0.0037(k={20, 50, 100}; f={2, 5, 10}), which seems much better!
\n\nWill be happy if it helps~
\n\nNote1: The target of regression is tested in this competition but I didn't test the classification target, maybe there're some bugs :p
\n\nNote2: The code is arranged from this article (in Chinese). Credits to him~
", "messages": [ { "id": "349393", "postDate": "06/28/2018 04:13:25", "content": "Hi,
\n\nFirst, thanks kaggle and avito for organizing such a worderful competition and congrats to everyone! And thanks to teams sharing fantastic solutions!
\n\nAlthough my whole solution is neither strong nor novel and not worth sharing, I want to share a script which generated a group of my strong features - target encoding (also called mean encoding).
\n\n\n\nI knew it long ago and found that the normal TE overfitting easily but this coursera course (week3) got me know that it will only work when you take some generalizing tricks. In this competition I took cross-validated (CV) & time-series split (TS) two kinds of generalization.
\n\nIdeas are simple:
\n\nThe experiment shows the lgb model using CV-only target encoding feature (and other common features) would lead to 0.0055(k=20, f=2) or 0.0063(k=50, f=5) gap between OOF valid score and LB, but with CV&TS target encoding it gives consistent gaps of 0.0037(k={20, 50, 100}; f={2, 5, 10}), which seems much better!
\n\nWill be happy if it helps~
\n\nNote1: The target of regression is tested in this competition but I didn't test the classification target, maybe there're some bugs :p
\n\nNote2: The code is arranged from this article (in Chinese). Credits to him~
", "rawMarkdown": "Hi, \n\nFirst, thanks kaggle and avito for organizing such a worderful competition and congrats to everyone! And thanks to teams sharing fantastic solutions!\n\nAlthough my whole solution is neither strong nor novel and not worth sharing, I want to share a script which generated a group of my strong features - target encoding (also called mean encoding). \n\n[code here][1]\n\nI knew it long ago and found that the normal TE overfitting easily but [this coursera course (week3)][2] got me know that it will only work when you take some generalizing tricks. In this competition I took cross-validated (CV) & time-series split (TS) two kinds of generalization. \n\nIdeas are simple:\n\n- Only use data from other CV folds to do target encoding\n- Only use past data to do target encoding\n- And combine them\n\nThe experiment shows the lgb model using CV-only target encoding feature (and other common features) would lead to 0.0055(k=20, f=2) or 0.0063(k=50, f=5) gap between OOF valid score and LB, but with CV&TS target encoding it gives consistent gaps of 0.0037(k={20, 50, 100}; f={2, 5, 10}), which seems much better!\n\nWill be happy if it helps~\n\n\n\n-------\nNote1: The target of regression is tested in this competition but I didn't test the classification target, maybe there're some bugs :p\n\nNote2: The code is arranged from [this article][3] (in Chinese). Credits to him~\n\n\n [1]: https://www.kaggle.com/johnfarrell/adp-prepare-data-me-20-2-true\n [2]: https://www.coursera.org/learn/competitive-data-science\n [3]: https://zhuanlan.zhihu.com/p/26308272", "votes": null }, { "id": "349430", "postDate": "06/28/2018 05:08:06", "content": "Other script:
\n\nVery helpful. Congrats~~ and happy kaggling!
", "rawMarkdown": "Very helpful. Congrats~~ and happy kaggling!", "votes": null } ], "comments": [ { "id": 349430, "author_name": "johnfarrell", "author_url": "", "post_date": "06/28/2018 05:08:06", "content": "Other script:
\n\nVery helpful. Congrats~~ and happy kaggling!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "349393": "Hi, \n\nFirst, thanks kaggle and avito for organizing such a worderful competition and congrats to everyone! And thanks to teams sharing fantastic solutions!\n\nAlthough my whole solution is neither strong nor novel and not worth sharing, I want to share a script which generated a group of my strong features - target encoding (also called mean encoding). \n\n[code here][1]\n\nI knew it long ago and found that the normal TE overfitting easily but [this coursera course (week3)][2] got me know that it will only work when you take some generalizing tricks. In this competition I took cross-validated (CV) & time-series split (TS) two kinds of generalization. \n\nIdeas are simple:\n\n- Only use data from other CV folds to do target encoding\n- Only use past data to do target encoding\n- And combine them\n\nThe experiment shows the lgb model using CV-only target encoding feature (and other common features) would lead to 0.0055(k=20, f=2) or 0.0063(k=50, f=5) gap between OOF valid score and LB, but with CV&TS target encoding it gives consistent gaps of 0.0037(k={20, 50, 100}; f={2, 5, 10}), which seems much better!\n\nWill be happy if it helps~\n\n\n\n-------\nNote1: The target of regression is tested in this competition but I didn't test the classification target, maybe there're some bugs :p\n\nNote2: The code is arranged from [this article][3] (in Chinese). Credits to him~\n\n\n [1]: https://www.kaggle.com/johnfarrell/adp-prepare-data-me-20-2-true\n [2]: https://www.coursera.org/learn/competitive-data-science\n [3]: https://zhuanlan.zhihu.com/p/26308272", "349430": "Other script:\n\n- [Extract text metadata][1]\n- [Use pytorch to train FM and extract second last layer output as features available to other models like LGB][2]\n- [Use the same pytorch template to train textcnn and extract second last layer feature][3]\n- [Simple two-level stacking process][4]\n\n\n [1]: https://www.kaggle.com/johnfarrell/adp-prepare-text-metadata\n [2]: https://www.kaggle.com/johnfarrell/adp-pytorch-fm-pretrain-dev\n [3]: https://www.kaggle.com/johnfarrell/adp-pytorch-textcnn-context-only-dev\n [4]: https://www.kaggle.com/johnfarrell/adp-blend-final", "349660": "Very helpful. Congrats~~ and happy kaggling!" }, "source": "meta" }