![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F100236%2F64cc45bbe25144503bc93cf4b9e102f1%2Fmte.gif?generation=1594620515929361&alt=media\")
Congratulations to winners: Bowen-Yuan, Eureka, Daniel_NTU, nomo, hugepanda) and\nbrain-afk (Μαριος Μιχαηλιδης KazAnova, Faron, Darragh, Alexey Noskov). Also I want to thank my teammates for their great effort! Especially @Takuya, really brilliant work!
\n\nThis is not a summary of solution of our team, Three Data Points. The honor will be Takuya's. (CuteChibiko).
\n\nIn this post, I'll just introduce two tools I developed in the contest. light-ffm for my best single model, 0.693x lb and stack-nn, 0.695x lb. Both of them use less then 5 GB CPU memory and are faster than some popular alternatives, like ffm and xgb for the same feature set.
\n\nlight-ffm is based on libffm with the goal to accelerate out-of-core performance without sacrificing much performance. A submission can be generated in 2.5 hours.
\n\ncode: https://github.com/daxiongshu/light-ffm
\n\ndoc: https://github.com/daxiongshu/light-ffm/blob/master/light-ffm.pdf
\n\nMajor update of light-ffm is expected in near future: support both old and new format, support regression, multi-classification, ranking and so on. Codes also need to be cleaned in a great deal.
\n\nstack-nn is coded with tensorflow-gpu for 2nd level stacking (numerical features only) which can generate a reasonably good submission 0.695x in 20 mins with GPU. It scales to 200+ numerical features within 16 GB CPU memory but we didn't have that many useful features in the end :P
\n\ncode: https://github.com/daxiongshu/stack-nn-tensorflow
\n\nI don't have a doc yet but the basic idea is to 1) use 2 level batching to make sure data fit in CPU memory and load fast and 2) per-group softmax to optimize apk.
\n\nAny feedback are highly appreciated!
", "messages": [ { "id": "157042", "postDate": "01/19/2017 00:52:50", "content": "Congratulations to winners: Bowen-Yuan, Eureka, Daniel_NTU, nomo, hugepanda) and\nbrain-afk (Μαριος Μιχαηλιδης KazAnova, Faron, Darragh, Alexey Noskov). Also I want to thank my teammates for their great effort! Especially @Takuya, really brilliant work!
\n\nThis is not a summary of solution of our team, Three Data Points. The honor will be Takuya's. (CuteChibiko).
\n\nIn this post, I'll just introduce two tools I developed in the contest. light-ffm for my best single model, 0.693x lb and stack-nn, 0.695x lb. Both of them use less then 5 GB CPU memory and are faster than some popular alternatives, like ffm and xgb for the same feature set.
\n\nlight-ffm is based on libffm with the goal to accelerate out-of-core performance without sacrificing much performance. A submission can be generated in 2.5 hours.
\n\ncode: https://github.com/daxiongshu/light-ffm
\n\ndoc: https://github.com/daxiongshu/light-ffm/blob/master/light-ffm.pdf
\n\nMajor update of light-ffm is expected in near future: support both old and new format, support regression, multi-classification, ranking and so on. Codes also need to be cleaned in a great deal.
\n\nstack-nn is coded with tensorflow-gpu for 2nd level stacking (numerical features only) which can generate a reasonably good submission 0.695x in 20 mins with GPU. It scales to 200+ numerical features within 16 GB CPU memory but we didn't have that many useful features in the end :P
\n\ncode: https://github.com/daxiongshu/stack-nn-tensorflow
\n\nI don't have a doc yet but the basic idea is to 1) use 2 level batching to make sure data fit in CPU memory and load fast and 2) per-group softmax to optimize apk.
\n\nAny feedback are highly appreciated!
", "rawMarkdown": "Congratulations to winners: Bowen-Yuan, Eureka, Daniel_NTU, nomo, hugepanda) and\r\nbrain-afk (Μαριος Μιχαηλιδης KazAnova, Faron, Darragh, Alexey Noskov). Also I want to thank my teammates for their great effort! Especially @Takuya, really brilliant work!\r\n\r\n\r\n**This is not a summary of solution** of our team, Three Data Points. The honor will be **Takuya's. (CuteChibiko).** \r\n\r\nIn this post, I'll just introduce two tools I developed in the contest. light-ffm for my best single model, 0.693x lb and stack-nn, 0.695x lb. Both of them use less then 5 GB CPU memory and are faster than some popular alternatives, like ffm and xgb for the same feature set. \r\n\r\nlight-ffm is based on libffm with the goal to accelerate out-of-core performance without sacrificing much performance. A submission can be generated in 2.5 hours.\r\n\r\ncode: https://github.com/daxiongshu/light-ffm\r\n\r\ndoc: https://github.com/daxiongshu/light-ffm/blob/master/light-ffm.pdf\r\n\r\nMajor update of light-ffm is expected in near future: support both old and new format, support regression, multi-classification, ranking and so on. Codes also need to be cleaned in a great deal.\r\n\r\nstack-nn is coded with tensorflow-gpu for 2nd level stacking (numerical features only) which can generate a reasonably good submission 0.695x in 20 mins with GPU. It scales to 200+ numerical features within 16 GB CPU memory but we didn't have that many useful features in the end :P\r\n\r\ncode: https://github.com/daxiongshu/stack-nn-tensorflow\r\n\r\nI don't have a doc yet but the basic idea is to 1) use 2 level batching to make sure data fit in CPU memory and load fast and 2) per-group softmax to optimize apk.\r\n\r\nAny feedback are highly appreciated!", "votes": null }, { "id": "157048", "postDate": "01/19/2017 01:12:15", "content": "@rcarson - thank you for releasing this! This was my first time using FFMs. My computer wouldn't let me handle anything above 40 features total without automatically killing my job and training time was between 5-6 hours with the --on-disk option. I will certainly be using this code! Thanks again!
", "rawMarkdown": "rcarson - thank you for releasing this! This was my first time using FFMs. My computer wouldn't let me handle anything above 40 features total without automatically killing my job and training time was between 5-6 hours with the --on-disk option. I will certainly be using this code! Thanks again!", "votes": null }, { "id": "157049", "postDate": "01/19/2017 01:15:43", "content": "@RDizzl3, thank you!
\n\nI would suggest you wait a little while since the code for now requires a different data format and additional field map matrix which could be confusing at the moment.
\n\nI am gonna modify it so one can run this tool with original ffm data and parameters, without any changes
", "rawMarkdown": "RDizzl3, thank you! \r\n\r\nI would suggest you wait a little while since the code for now requires a different data format and additional field map matrix which could be confusing at the moment. \r\n\r\nI am gonna modify it so one can run this tool with original ffm data and parameters, without any changes", "votes": null }, { "id": "157056", "postDate": "01/19/2017 01:58:43", "content": "Thank you @rcarson for open sourcing this excellent code.
\n\nWill use this in the upcoming competitions :)
", "rawMarkdown": "Thank you @rcarson for open sourcing this excellent code. \r\n\r\nWill use this in the upcoming competitions :)", "votes": null }, { "id": "157057", "postDate": "01/19/2017 02:02:57", "content": "@SRK， thank you! actually we used earlier version of light-ffm in our 3rd finish of Grupo Bimbo but it was very rough at that time. :P
", "rawMarkdown": "SRK， thank you! actually we used earlier version of light-ffm in our 3rd finish of Grupo Bimbo but it was very rough at that time. :P", "votes": null }, { "id": "157059", "postDate": "01/19/2017 02:11:23", "content": "That is great. So this code has already given you two third place solutions ;)
", "rawMarkdown": "That is great. So this code has already given you two third place solutions ;)", "votes": null }, { "id": "157060", "postDate": "01/19/2017 02:13:37", "content": "As the first user (besides the author), I can confirm that the light-ffm is amazingly done. It saves lots of memory compared to libffm and is much faster. Highly recommended! :)
", "rawMarkdown": "As the first user (besides the author), I can confirm that the light-ffm is amazingly done. It saves lots of memory compared to libffm and is much faster. Highly recommended! :)", "votes": null }, { "id": "157063", "postDate": "01/19/2017 03:01:58", "content": "[quote=SRK;157059]
\n\nThat is great. So this code has already given you two third place solutions ;)
\n\n[/quote]
\n\nNot really. Winning kaggle is always about better features and massive ensemble. Thanks to Xiaozhou @bimbo and Takuya this time, and don't be mistaken, we do have gigantic machines with 200 GB memory. So in the end, it is still XGB and ffm that get the best score. Looking forward to top teams' solution sharing!
", "rawMarkdown": "[quote=SRK;157059]\r\n\r\nThat is great. So this code has already given you two third place solutions ;) \r\n\r\n[/quote]\r\n\r\nNot really. Winning kaggle is always about better features and massive ensemble. Thanks to Xiaozhou @bimbo and Takuya this time, and don't be mistaken, we do have gigantic machines with 200 GB memory. So in the end, it is still XGB and ffm that get the best score. Looking forward to top teams' solution sharing!", "votes": null }, { "id": "157088", "postDate": "01/19/2017 06:45:08", "content": "@rcarson: thanks for sharing and congrats. have you checked if lightffm generated better score than the standard libffm? or is the advantage only in resource usage and speed?
", "rawMarkdown": "rcarson: thanks for sharing and congrats. have you checked if lightffm generated better score than the standard libffm? or is the advantage only in resource usage and speed?", "votes": null }, { "id": "157096", "postDate": "01/19/2017 07:31:18", "content": "Can you push your feature engineering on this competition? I'm so interested in your features
", "rawMarkdown": "Can you push your feature engineering on this competition? I'm so interested in your features", "votes": null }, { "id": "157100", "postDate": "01/19/2017 07:43:03", "content": "@Sameh, our test shows something like apk:
\n\nlibffm with customized objective > light-ffm with customized objective >> original libffm with logloss objective.
\n\nSo yeah, you can expect light-ffm has some performance loss but it scale to more features with a limited machine.
", "rawMarkdown": "Sameh, our test shows something like apk: \r\n\r\nlibffm with customized objective > light-ffm with customized objective >> original libffm with logloss objective.\r\n\r\nSo yeah, you can expect light-ffm has some performance loss but it scale to more features with a limited machine.", "votes": null }, { "id": "157108", "postDate": "01/19/2017 08:31:47", "content": "Hi rcarson\nCongrats for the third finish.\nI got inspired by one of your forum comments that you are modifying libffm and I have done some simple changes in libffm code to cache the binary data. Another one was i thinking of ignoring some fields by giving options on command line. But as I started late I didn't get time to implement it. You other ideas of reducing memory footprint are simply great.
", "rawMarkdown": "Hi rcarson\r\nCongrats for the third finish.\r\nI got inspired by one of your forum comments that you are modifying libffm and I have done some simple changes in libffm code to cache the binary data. Another one was i thinking of ignoring some fields by giving options on command line. But as I started late I didn't get time to implement it. You other ideas of reducing memory footprint are simply great.", "votes": null }, { "id": "157112", "postDate": "01/19/2017 08:52:24", "content": "Congrats on third rcarson. Impressive refinement of libffm. :-)
", "rawMarkdown": "Congrats on third rcarson. Impressive refinement of libffm. :-)", "votes": null }, { "id": "157120", "postDate": "01/19/2017 09:43:59", "content": "Congratulations rcarson and team, and thanks for publishing light-ffm and the pypy code.
", "rawMarkdown": "Congratulations rcarson and team, and thanks for publishing light-ffm and the pypy code.", "votes": null }, { "id": "157126", "postDate": "01/19/2017 10:24:05", "content": "Thanks a lot for publishing code, rcarson!
\n\nAnd also for sharing idea of reimplementing FFM in forums early - it's really what inspired me in participating here. I also went this road, but took somewhat different implementation decisions, so hope will submit a one or two PRs based on my findings to help improve your tool ever further.
", "rawMarkdown": "Thanks a lot for publishing code, rcarson! \r\n\r\nAnd also for sharing idea of reimplementing FFM in forums early - it's really what inspired me in participating here. I also went this road, but took somewhat different implementation decisions, so hope will submit a one or two PRs based on my findings to help improve your tool ever further.", "votes": null }, { "id": "157208", "postDate": "01/19/2017 17:29:39", "content": "Thank you all for the reply. After merging with Takuya, I realized numerical feature could also be super useful and I should not throw them away. As long as the value is around 1, like the categorical feature, ffm should also pick it up. So I will make \"drop feature value\" as an option in case that there are no numerical features.\n
@rcarson - thank you for releasing this! This was my first time using FFMs. My computer wouldn't let me handle anything above 40 features total without automatically killing my job and training time was between 5-6 hours with the --on-disk option. I will certainly be using this code! Thanks again!
", "votes": null, "replies": [] }, { "id": 157049, "author_name": "jiweiliu", "author_url": "", "post_date": "01/19/2017 01:15:43", "content": "@RDizzl3, thank you!
\n\nI would suggest you wait a little while since the code for now requires a different data format and additional field map matrix which could be confusing at the moment.
\n\nI am gonna modify it so one can run this tool with original ffm data and parameters, without any changes
", "votes": null, "replies": [] }, { "id": 157056, "author_name": "sudalairajkumar", "author_url": "", "post_date": "01/19/2017 01:58:43", "content": "Thank you @rcarson for open sourcing this excellent code.
\n\nWill use this in the upcoming competitions :)
", "votes": null, "replies": [] }, { "id": 157057, "author_name": "jiweiliu", "author_url": "", "post_date": "01/19/2017 02:02:57", "content": "@SRK， thank you! actually we used earlier version of light-ffm in our 3rd finish of Grupo Bimbo but it was very rough at that time. :P
", "votes": null, "replies": [] }, { "id": 157059, "author_name": "sudalairajkumar", "author_url": "", "post_date": "01/19/2017 02:11:23", "content": "That is great. So this code has already given you two third place solutions ;)
", "votes": null, "replies": [] }, { "id": 157060, "author_name": "xiaozhouwang", "author_url": "", "post_date": "01/19/2017 02:13:37", "content": "As the first user (besides the author), I can confirm that the light-ffm is amazingly done. It saves lots of memory compared to libffm and is much faster. Highly recommended! :)
", "votes": null, "replies": [] }, { "id": 157063, "author_name": "jiweiliu", "author_url": "", "post_date": "01/19/2017 03:01:58", "content": "[quote=SRK;157059]
\n\nThat is great. So this code has already given you two third place solutions ;)
\n\n[/quote]
\n\nNot really. Winning kaggle is always about better features and massive ensemble. Thanks to Xiaozhou @bimbo and Takuya this time, and don't be mistaken, we do have gigantic machines with 200 GB memory. So in the end, it is still XGB and ffm that get the best score. Looking forward to top teams' solution sharing!
", "votes": null, "replies": [] }, { "id": 157088, "author_name": "samehif", "author_url": "", "post_date": "01/19/2017 06:45:08", "content": "@rcarson: thanks for sharing and congrats. have you checked if lightffm generated better score than the standard libffm? or is the advantage only in resource usage and speed?
", "votes": null, "replies": [] }, { "id": 157096, "author_name": "nhuantd", "author_url": "", "post_date": "01/19/2017 07:31:18", "content": "Can you push your feature engineering on this competition? I'm so interested in your features
", "votes": null, "replies": [] }, { "id": 157100, "author_name": "jiweiliu", "author_url": "", "post_date": "01/19/2017 07:43:03", "content": "@Sameh, our test shows something like apk:
\n\nlibffm with customized objective > light-ffm with customized objective >> original libffm with logloss objective.
\n\nSo yeah, you can expect light-ffm has some performance loss but it scale to more features with a limited machine.
", "votes": null, "replies": [] }, { "id": 157108, "author_name": "adityakumarsinha", "author_url": "", "post_date": "01/19/2017 08:31:47", "content": "Hi rcarson\nCongrats for the third finish.\nI got inspired by one of your forum comments that you are modifying libffm and I have done some simple changes in libffm code to cache the binary data. Another one was i thinking of ignoring some fields by giving options on command line. But as I started late I didn't get time to implement it. You other ideas of reducing memory footprint are simply great.
", "votes": null, "replies": [] }, { "id": 157112, "author_name": "frederik", "author_url": "", "post_date": "01/19/2017 08:52:24", "content": "Congrats on third rcarson. Impressive refinement of libffm. :-)
", "votes": null, "replies": [] }, { "id": 157120, "author_name": "darraghdog", "author_url": "", "post_date": "01/19/2017 09:43:59", "content": "Congratulations rcarson and team, and thanks for publishing light-ffm and the pypy code.
", "votes": null, "replies": [] }, { "id": 157126, "author_name": "alexeynoskov", "author_url": "", "post_date": "01/19/2017 10:24:05", "content": "Thanks a lot for publishing code, rcarson!
\n\nAnd also for sharing idea of reimplementing FFM in forums early - it's really what inspired me in participating here. I also went this road, but took somewhat different implementation decisions, so hope will submit a one or two PRs based on my findings to help improve your tool ever further.
", "votes": null, "replies": [] }, { "id": 157208, "author_name": "jiweiliu", "author_url": "", "post_date": "01/19/2017 17:29:39", "content": "Thank you all for the reply. After merging with Takuya, I realized numerical feature could also be super useful and I should not throw them away. As long as the value is around 1, like the categorical feature, ffm should also pick it up. So I will make \"drop feature value\" as an option in case that there are no numerical features.\n