Congratulations to code monkey, brain-afk and everyone!\nAnd I’d like to thank my teammates carl and Little Boat. I could learn lots of things from them.
\n\nAs Little Boat explained, our model is ffm models + xgboost models as first layer, and xgboost as second layer.
\n\nI would like to explain our first layer models here.
\n\nFollowing is the list of unobvious features we used.\nWe made lots of other features which turned out to be useless.\nBut analyzing large scale user behavior was very interesting to me.
\n\nIn addition to carl’s light-ffm, we used another customized libffm which has pair-wise rank and can take sample weight.\nBest public LB score is 0.6974.
\n\nXGB\n\nWe used xgboost with rank:pairwise objective.\nBest public LB score is 0.6885.
\n\nLeak Row Excluding\n\nWe trained ffm models without leak row by removing them from training data first, then blend/merge leak information later.\nThis improved single model score.
\n\nPresent/Future split\n\nWe split test data into ‘present’ data (in time samples) and ‘future’ data (out time samples), then built models for each data.\nHere, Features and hyper parameters for each model are optimized independently.\nFor example we used timestamp feature for ‘present’ model, but did not used it for ‘future’ model. And we used sample weight of training data for ‘future’ model to add higher weight to last day than first day.
\n\nPlease feel free to ask any questions.
", "messages": [ { "id": 157166, "postDate": "2017-01-19T14:18:11.927Z", "content": "Congratulations to code monkey, brain-afk and everyone!\nAnd I’d like to thank my teammates carl and Little Boat. I could learn lots of things from them.
\n\nAs Little Boat explained, our model is ffm models + xgboost models as first layer, and xgboost as second layer.
\n\nI would like to explain our first layer models here.
\n\nFollowing is the list of unobvious features we used.\nWe made lots of other features which turned out to be useless.\nBut analyzing large scale user behavior was very interesting to me.
\n\nIn addition to carl’s light-ffm, we used another customized libffm which has pair-wise rank and can take sample weight.\nBest public LB score is 0.6974.
\n\nXGB\n\nWe used xgboost with rank:pairwise objective.\nBest public LB score is 0.6885.
\n\nLeak Row Excluding\n\nWe trained ffm models without leak row by removing them from training data first, then blend/merge leak information later.\nThis improved single model score.
\n\nPresent/Future split\n\nWe split test data into ‘present’ data (in time samples) and ‘future’ data (out time samples), then built models for each data.\nHere, Features and hyper parameters for each model are optimized independently.\nFor example we used timestamp feature for ‘present’ model, but did not used it for ‘future’ model. And we used sample weight of training data for ‘future’ model to add higher weight to last day than first day.
\n\nPlease feel free to ask any questions.
", "rawMarkdown": "Congratulations to code monkey, brain-afk and everyone!\r\nAnd I’d like to thank my teammates carl and Little Boat. I could learn lots of things from them.\r\n\r\nAs [Little Boat explained](https://www.kaggle.com/c/outbrain-click-prediction/forums/t/27897/congrats-and-solution-sharing?forumMessageId=157062#post157062), our model is ffm models + xgboost models as first layer, and xgboost as second layer.\r\n\r\nI would like to explain our first layer models here.\r\n\r\n###**Feature extraction**\r\n\r\nFollowing is the list of unobvious features we used.\r\nWe made lots of other features which turned out to be useless.\r\nBut analyzing large scale user behavior was very interesting to me.\r\n\r\n - page view counts of each user.\r\n - page view counts of ad landing page.\r\n - impression counts for each ad_id, landing document_id, campaign_id and advertiser_id.\r\n - landing page confidence vector, where confidence vector is defined as a vector which element is composed of confidence level from documents_*.csv. This feature is used only for ffm as numeric data.\r\n - user confidence vector - this is the average of document confidence vector which is viewed by each user. This feature is used only for ffm as numeric data.\r\n - inner dot product of document confidence vectors of ad impression page and ad landing page .\r\n - inner dot product of user confidence vector and ad landing page document confidence vector\r\n - XGB leaf for ffm feature.\r\n - immediate document viewed after click event.\r\n\r\n###**Modeling and Training**\r\n\r\n####**FFM**\r\nIn addition to [carl’s light-ffm](https://www.kaggle.com/c/outbrain-click-prediction/forums/t/27892/introducing-light-ffm-and-stack-nn), we used another customized libffm which has pair-wise rank and can take sample weight.\r\nBest public LB score is 0.6974.\r\n\r\n####**XGB**\r\n We used xgboost with rank:pairwise objective.\r\nBest public LB score is 0.6885.\r\n\r\n####**Leak Row Excluding**\r\nWe trained ffm models without leak row by removing them from training data first, then blend/merge leak information later.\r\nThis improved single model score.\r\n\r\n####**Present/Future split**\r\nWe split test data into ‘present’ data (in time samples) and ‘future’ data (out time samples), then built models for each data.\r\nHere, Features and hyper parameters for each model are optimized independently.\r\nFor example we used timestamp feature for ‘present’ model, but did not used it for ‘future’ model. And we used sample weight of training data for ‘future’ model to add higher weight to last day than first day.\r\n\r\nPlease feel free to ask any questions.\r\n\r\n", "votes": 21 }, { "id": 158945, "postDate": "2017-01-30T22:13:13.557Z", "content": "Great @CuteChibiko ! Thanks for sharing!
", "rawMarkdown": "Great @CuteChibiko ! Thanks for sharing!", "votes": 1 }, { "id": 157578, "postDate": "2017-01-22T08:57:13.290Z", "content": "Thank you Yuriy Makarov,
\n\nYes, we tried CTR with some impression threshold. If impression threshold was low, CTR caused overfit. And if impression threshold was high, CTR did not contribute to score. So We did not use CTR in the end.
\n\n\"immediate document viewed after click event.\" means the first document that was seen by a user after the user clicked an ad.
", "rawMarkdown": "Thank you Yuriy Makarov,\r\n\r\nYes, we tried CTR with some impression threshold. If impression threshold was low, CTR caused overfit. And if impression threshold was high, CTR did not contribute to score. So We did not use CTR in the end.\r\n\r\n\"immediate document viewed after click event.\" means the first document that was seen by a user after the user clicked an ad. ", "votes": 1 }, { "id": 157274, "postDate": "2017-01-19T23:48:05.547Z", "content": "@Gabriel Moreira,
\n\nYour CV score drop, from CV=0.679 to 0.670, is very big, so I think something should be happening.\nI'm not sure why, but maybe you are right. We made XGB models for test and validation respectively, so this may cause the difference.
\n\nBTW, our best single score did not improved with GBDT leaves, but it worked for ensemble model.
\n\nOur best single CV score without GBDT leaves was 0.69586 and score with GBDT leaves was 0.69588 when we tested GBDT leaves. And weighted average of them scored 0.6971.So even single model score was not improved, GBDT leaves feature was very important.
", "rawMarkdown": "@Gabriel Moreira,\r\n\r\nYour CV score drop, from CV=0.679 to 0.670, is very big, so I think something should be happening.\r\nI'm not sure why, but maybe you are right. We made XGB models for test and validation respectively, so this may cause the difference.\r\n\r\nBTW, our best single score did not improved with GBDT leaves, but it worked for ensemble model.\r\n\r\nOur best single CV score without GBDT leaves was 0.69586 and score with GBDT leaves was 0.69588 when we tested GBDT leaves. And weighted average of them scored 0.6971.So even single model score was not improved, GBDT leaves feature was very important.", "votes": 1 }, { "id": 157225, "postDate": "2017-01-19T18:49:26.890Z", "content": "@CuteChibiko I also tried the 3 idiots approach to add GBDT leaves nodes as features to FFM, but did not succeeded improving the score using standard LibFFM, (actually they made my FFM CV score worse). I tried 30 trees with 64 and 128 leaves, using XGBoost and LightGBM.\nDo you think my GBDT model may not have being accurate enought (CV=0.670), while my FFM model was (0.679)?\nOther aspect is that I used a fixed validation set following test set days distribution. So, I trained GBDT on full train set and used the model to predict both the train set and validation set. Maybe that could cause some overfit on FFM model, even with using leaf nodes?
", "rawMarkdown": "@CuteChibiko I also tried the 3 idiots approach to add GBDT leaves nodes as features to FFM, but did not succeeded improving the score using standard LibFFM, (actually they made my FFM CV score worse). I tried 30 trees with 64 and 128 leaves, using XGBoost and LightGBM.\r\nDo you think my GBDT model may not have being accurate enought (CV=0.670), while my FFM model was (0.679)?\r\nOther aspect is that I used a fixed validation set following test set days distribution. So, I trained GBDT on full train set and used the model to predict both the train set and validation set. Maybe that could cause some overfit on FFM model, even with using leaf nodes?", "votes": 1 }, { "id": 157180, "postDate": "2017-01-19T15:18:38.207Z", "content": "\"impression counts for each ad_id, landing document_id, campaign_id and advertiser_id.\"
\n\nwhat do you mean by impression counts?
", "rawMarkdown": "\"impression counts for each ad_id, landing document_id, campaign_id and advertiser_id.\"\r\n\r\nwhat do you mean by impression counts?", "votes": 1 }, { "id": 157171, "postDate": "2017-01-19T14:40:51.577Z", "content": "I have 2 questions:
\n\nThanks and congrats!
", "rawMarkdown": "I have 2 questions: \r\n\r\n 1. Could you explain what XGB leaf feature is? \r\n 2. Do you normalize the confidence vectors for documents?\r\n\r\nThanks and congrats!", "votes": 1 }, { "id": 159127, "postDate": "2017-02-01T00:06:25.283Z", "content": "Good job @CuteChibiko - I took a look in your implementation and saw it really deserved a try. I've run today a benchmark using some of my datasets in FFM format comparing standard LibFFM and your fork with custom pairwise rank customization.
\nI just run the models with the same FFM hyperparameters (without tunning for MAP ranking) and weight 1 for all samples. The environment was a Google Compute Engine instance with 32 CPUs, 256 GB RAM and FFM data in a SSD disk.
1) Dataset with categorical fields and some binned numeric fields
\nParameters: -l 0.0001 -k 6 -t 20 -r 0.2 -s 28 --auto-stop
\nRounds: 20
LibFFM - Cross-validation MAP = 0.65911
\nLibFFM - Cross-validation MAP + leak = 0.67795
\nLibFFM - Training time = 291m
LibFFM pairwise - Cross-validation MAP = 0.66040
\nLibFFM pairwise - Cross-validation MAP + leak = 0.67792
\nLibFFM pairwise - Training time = 631m
2) Dataset with only categorical fields\nParameters: -l 0.00002 -k 6 -t 50 -r 0.1 -s 28 --auto-stop
\n\nLibFFM - Training rounds = 13 (auto-stop on validation logloss)
\nLibFFM - Cross-validation MAP = 0.66428
\nLibFFM - Cross-validation MAP + leak = 0.67859
\nLibFFM - Training time = 27m
LibFFM - Training rounds = 26 (auto-stop on validation MAP)
\nLibFFM pairwise - Cross-validation MAP = 0.66552
\nLibFFM pairwise - Cross-validation MAP + leak = 0.67973
\nLibFFM pairwise - Training time = 211m
I noticed a diff of around 0.001 in CV MAP by for two datasets. Was this the order of difference you could find in your experiments with your FFM with ranking optimization?
\nPs. I also noticed a great increase in training logloss (from 0.41009 to 0.56855), but this is expected as the optimization is now driven by MAP and not logloss), right?
I am worried only about accuracy at this time, but I could notice that training time of your implementation is very slow compared to standard FFM, and that during most of the processing it is using only 1 CPU. Is that because you had not time to parallelize MAP evaluation?
\n\nWhich sample weight strategy have you used? More weight on recent samples / clicked samples / display_ids with more ads?
\n\nThanks again!
", "rawMarkdown": "Good job @CuteChibiko - I took a look in your implementation and saw it really deserved a try. I've run today a benchmark using some of my datasets in FFM format comparing standard LibFFM and your fork with custom pairwise rank customization. \r\nI just run the models with the same FFM hyperparameters (without tunning for MAP ranking) and weight 1 for all samples. The environment was a Google Compute Engine instance with 32 CPUs, 256 GB RAM and FFM data in a SSD disk. \r\n\r\n1) Dataset with categorical fields and some binned numeric fields \r\nParameters: -l 0.0001 -k 6 -t 20 -r 0.2 -s 28 --auto-stop \r\nRounds: 20 \r\n \r\nLibFFM - Cross-validation MAP = 0.65911 \r\nLibFFM - Cross-validation MAP + leak = 0.67795 \r\nLibFFM - Training time = 291m \r\n \r\nLibFFM pairwise - Cross-validation MAP = 0.66040 \r\nLibFFM pairwise - Cross-validation MAP + leak = 0.67792 \r\nLibFFM pairwise - Training time = 631m \r\n\r\n----------------------------------------------------\r\n\r\n2) Dataset with only categorical fields\r\nParameters: -l 0.00002 -k 6 -t 50 -r 0.1 -s 28 --auto-stop\r\n\r\nLibFFM - Training rounds = 13 (auto-stop on validation logloss) \r\nLibFFM - Cross-validation MAP = 0.66428 \r\nLibFFM - Cross-validation MAP + leak = 0.67859 \r\nLibFFM - Training time = 27m\r\n\r\nLibFFM - Training rounds = 26 (auto-stop on validation MAP) \r\nLibFFM pairwise - Cross-validation MAP = 0.66552 \r\nLibFFM pairwise - Cross-validation MAP + leak = 0.67973 \r\nLibFFM pairwise - Training time = 211m\r\n\r\n-----------------------------------\r\n \r\nI noticed a diff of around 0.001 in CV MAP by for two datasets. Was this the order of difference you could find in your experiments with your FFM with ranking optimization? \r\nPs. I also noticed a great increase in training logloss (from 0.41009 to 0.56855), but this is expected as the optimization is now driven by MAP and not logloss), right? \r\n \r\nI am worried only about accuracy at this time, but I could notice that training time of your implementation is very slow compared to standard FFM, and that during most of the processing it is using only 1 CPU. Is that because you had not time to parallelize MAP evaluation? \r\n\r\nWhich sample weight strategy have you used? More weight on recent samples / clicked samples / display_ids with more ads? \r\n\r\nThanks again! ", "votes": 2 }, { "id": 158836, "postDate": "2017-01-30T11:06:15.573Z", "content": "@Gabriel Moreira,
\n\nI uploaded this source code. You can find pairwise rank customization around here.
\n\nif(param.use_default_train){\n # default libffm training\n} else {\n # pairwise rank training\n}\n@Sameh Faidi,
\n\nI meant impression count as how many times the ad is shown.\nExplanation of impression in the context of online advertising can be found wikipedia.
\n\nFor example, impression counts for advertiser_id means how many times ads from the advertiser is shown.
", "rawMarkdown": "@Sameh Faidi,\r\n\r\nI meant impression count as how many times the ad is shown.\r\nExplanation of impression in the context of online advertising can be found [wikipedia](https://en.wikipedia.org/wiki/Impression_(online_media)). \r\n\r\nFor example, impression counts for advertiser_id means how many times ads from the advertiser is shown.\r\n", "votes": 2 }, { "id": 157179, "postDate": "2017-01-19T15:10:08.563Z", "content": "Thank you, Sangzia,
\n\nThanks for your kindly share@CuteChibiko . It is really impressive and interesting. I have several questoins about gbdt leaf node as feature.
\n\n1). Does the gbdt feature contain label information? like click/ctr. Or if not contain, like impression/pageview count etc.
\n\n2). Does the gbdt training data have overlap with ffm training data, If yes, does it suffer from overfitting at first due to label twice used and do you use some method to avoid overfitting?
\n\n3). Your weighted average with gbdt leaf's gain is large. Have you compared with ffm containing the gbdt's count feature in some transformation like log without gbdt leafnode, or use gbdt and ffm pctr prediction score together feeded into stacking model. I'm curious that the gain is from gbdt leaf node transformation method or the count/statistics feature itself.
\n\n4). Ususallly gbdt training has several hundred or thousand trees, and the cost is large for ffm. How many trees you have used and do you have some hyper-paramter or feature tuning for the less tree numbers?
\n\nThanks a lot.
", "rawMarkdown": "Thanks for your kindly share@CuteChibiko . It is really impressive and interesting. I have several questoins about gbdt leaf node as feature.\n\n1). Does the gbdt feature contain label information? like click/ctr. Or if not contain, like impression/pageview count etc.\n\n2). Does the gbdt training data have overlap with ffm training data, If yes, does it suffer from overfitting at first due to label twice used and do you use some method to avoid overfitting?\n\n3). Your weighted average with gbdt leaf's gain is large. Have you compared with ffm containing the gbdt's count feature in some transformation like log without gbdt leafnode, or use gbdt and ffm pctr prediction score together feeded into stacking model. I'm curious that the gain is from gbdt leaf node transformation method or the count/statistics feature itself.\n\n4). Ususallly gbdt training has several hundred or thousand trees, and the cost is large for ffm. How many trees you have used and do you have some hyper-paramter or feature tuning for the less tree numbers?\n\n\nThanks a lot.\n", "replies": [ { "id": 159463, "postDate": "2017-02-02T11:35:06.360Z", "content": "@nomo,
\n\n1) The gbdt feature dose not contain label information but contains impression/pageview count. This is almost same as what I listed in the first post of this topic.
\n\n2) Yes, it dose. The gbdt training data was same as ffm training data. But it did not suffer from overfitting.
\n\n3) I'm not sure I could understand your question correctly, but we only tried xgb leafnode information itself as feature and did not compared with any variations.
\n\n4) We used ntrees=39 and maxDepth=15. Almost all features and hyper-paramters of this model are the same as values of best xgb single model. But only eta was tuned for this model to reduce ntrees.
", "rawMarkdown": "@nomo,\n\n1) The gbdt feature dose not contain label information but contains impression/pageview count. This is almost same as what I listed in the first post of this topic.\n\n2) Yes, it dose. The gbdt training data was same as ffm training data. But it did not suffer from overfitting.\n\n3) I'm not sure I could understand your question correctly, but we only tried xgb leafnode information itself as feature and did not compared with any variations.\n\n4) We used ntrees=39 and maxDepth=15. Almost all features and hyper-paramters of this model are the same as values of best xgb single model. But only eta was tuned for this model to reduce ntrees." }, { "id": 159490, "postDate": "2017-02-02T14:44:01.707Z", "content": "@CuteChibiko thank you very much for your detailed explain. It is interesting:-)
\n\nI have also tried split data as present/future for xgb stacking training, but the result is neutral to not split one, and am interested in the different and your success trying reason.
\n\n1). For a.single model ffm / b.single model xgb / c. stacking model, your present/future split has gains on all of these three models or some of these models?
\n\n2). How much gain do your solution has compared with not split modeling? Is it mainly from present part or future part?
\n\n3). The future model process is interesting. Does the sample weight use exponent decay, like DecayFactor ^ (LastDay - SampleDay) as sample weight?
\n\n4). I have also used timestamp feature for split modeling present part, but use same hyper-parameter for the present/future part, and the result is neutral. Do your different parameters for these two part influence a lot? Or some other feature or model method also make a difference, like weighted average of present/future split modeling and not split model.
\n\nthanks a lot!
", "rawMarkdown": "@CuteChibiko thank you very much for your detailed explain. It is interesting:-)\n\nI have also tried split data as present/future for xgb stacking training, but the result is neutral to not split one, and am interested in the different and your success trying reason.\n\n1). For a.single model ffm / b.single model xgb / c. stacking model, your present/future split has gains on all of these three models or some of these models?\n\n2). How much gain do your solution has compared with not split modeling? Is it mainly from present part or future part? \n\n3). The future model process is interesting. Does the sample weight use exponent decay, like DecayFactor ^ (LastDay - **SampleDay**) as sample weight?\n\n4). I have also used timestamp feature for split modeling present part, but use same hyper-parameter for the present/future part, and the result is neutral. Do your different parameters for these two part influence a lot? Or some other feature or model method also make a difference, like weighted average of present/future split modeling and not split model.\n\nthanks a lot!\n\n\n" }, { "id": 159552, "postDate": "2017-02-02T18:48:33.603Z", "content": "@nomo,
\n\n1) Splitting data worked for a and b. And It seemed to work for validation data set of stacking model, but LB score decreased. We did not have enough time to find the reason, so we did not use splitting for stacking model.
\n\n2) The single model gain is 0.001 for ffm future model with sample weight, 0.002 for ffm present model with timestamp feature and 0.002 for xgb present model with timestamp.
\n\n3) Exactly. We used this formula:
\n\nsample weight = 1 + 2 * 0.5 ^ ((max_timestamp - timestamp)/86400000.0)\n4) Timestamp feature for present data made big gain without any special method. We used almost same hyper-parameters for split model. Only tuned hyper-parameter is iterations/ntrees.
", "rawMarkdown": "@nomo,\n\n1) Splitting data worked for a and b. And It seemed to work for validation data set of stacking model, but LB score decreased. We did not have enough time to find the reason, so we did not use splitting for stacking model.\n\n2) The single model gain is 0.001 for ffm future model with sample weight, 0.002 for ffm present model with timestamp feature and 0.002 for xgb present model with timestamp.\n\n3) Exactly. We used this formula: \n\n sample weight = 1 + 2 * 0.5 ^ ((max_timestamp - timestamp)/86400000.0)\n\n4) Timestamp feature for present data made big gain without any special method. We used almost same hyper-parameters for split model. Only tuned hyper-parameter is iterations/ntrees.\n" } ] }, { "id": 159137, "postDate": "2017-02-01T01:45:28.363Z", "content": "@Gabriel Moreira, thank you for your feedback.
\n\nThe difference was 0.0015 ~ 0.002 for our models.\nRelatively bigger ETA for LibFFM pairwise made better results.
\n\nYou are right. It is using only 1 CPU to calculate validation score.\nI don't remember why I commented out here and did not use omp. :(
\n\nWe added higher weights to recent samples for ‘future’ model. This improved 'future' model score about 0.001.
", "rawMarkdown": "@Gabriel Moreira, thank you for your feedback.\r\n\r\nThe difference was 0.0015 ~ 0.002 for our models.\r\nRelatively bigger ETA for LibFFM pairwise made better results.\r\n\r\nYou are right. It is using only 1 CPU to calculate validation score.\r\nI don't remember why I commented out [here](https://github.com/CuteChibiko/Outbrain-Click-Prediction/blob/master/libffm_pairwise/ffm.cpp#L500) and did not use omp. :(\r\n\r\nWe added higher weights to recent samples for ‘future’ model. This improved 'future' model score about 0.001." }, { "id": 158778, "postDate": "2017-01-30T01:48:11.027Z", "content": "Thanks for sharing @CuteChibiko ! You've told you used a customized libffm supporting pairwise rank.\nI am really interested in how the FFM optimization works when using pairwise ranking for libffm.... could you share something about that?
", "rawMarkdown": "Thanks for sharing @CuteChibiko ! You've told you used a customized libffm supporting pairwise rank.\r\nI am really interested in how the FFM optimization works when using pairwise ranking for libffm.... could you share something about that?" }, { "id": 157575, "postDate": "2017-01-22T08:01:42.010Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 158945, "author_name": "Gabriel Moreira", "author_url": "", "post_date": "2017-01-30T22:13:13.557000", "content": "Great @CuteChibiko ! Thanks for sharing!
", "votes": 1, "replies": [] }, { "id": 157578, "author_name": "tito", "author_url": "", "post_date": "2017-01-22T08:57:13.290000", "content": "Thank you Yuriy Makarov,
\n\nYes, we tried CTR with some impression threshold. If impression threshold was low, CTR caused overfit. And if impression threshold was high, CTR did not contribute to score. So We did not use CTR in the end.
\n\n\"immediate document viewed after click event.\" means the first document that was seen by a user after the user clicked an ad.
", "votes": 1, "replies": [] }, { "id": 157274, "author_name": "tito", "author_url": "", "post_date": "2017-01-19T23:48:05.547000", "content": "@Gabriel Moreira,
\n\nYour CV score drop, from CV=0.679 to 0.670, is very big, so I think something should be happening.\nI'm not sure why, but maybe you are right. We made XGB models for test and validation respectively, so this may cause the difference.
\n\nBTW, our best single score did not improved with GBDT leaves, but it worked for ensemble model.
\n\nOur best single CV score without GBDT leaves was 0.69586 and score with GBDT leaves was 0.69588 when we tested GBDT leaves. And weighted average of them scored 0.6971.So even single model score was not improved, GBDT leaves feature was very important.
", "votes": 1, "replies": [] }, { "id": 157225, "author_name": "Gabriel Moreira", "author_url": "", "post_date": "2017-01-19T18:49:26.890000", "content": "@CuteChibiko I also tried the 3 idiots approach to add GBDT leaves nodes as features to FFM, but did not succeeded improving the score using standard LibFFM, (actually they made my FFM CV score worse). I tried 30 trees with 64 and 128 leaves, using XGBoost and LightGBM.\nDo you think my GBDT model may not have being accurate enought (CV=0.670), while my FFM model was (0.679)?\nOther aspect is that I used a fixed validation set following test set days distribution. So, I trained GBDT on full train set and used the model to predict both the train set and validation set. Maybe that could cause some overfit on FFM model, even with using leaf nodes?
", "votes": 1, "replies": [] }, { "id": 157180, "author_name": "Sameh Faidi", "author_url": "", "post_date": "2017-01-19T15:18:38.207000", "content": "\"impression counts for each ad_id, landing document_id, campaign_id and advertiser_id.\"
\n\nwhat do you mean by impression counts?
", "votes": 1, "replies": [] }, { "id": 157171, "author_name": "Sangxia", "author_url": "", "post_date": "2017-01-19T14:40:51.577000", "content": "I have 2 questions:
\n\nThanks and congrats!
", "votes": 1, "replies": [] }, { "id": 159127, "author_name": "Gabriel Moreira", "author_url": "", "post_date": "2017-02-01T00:06:25.283000", "content": "Good job @CuteChibiko - I took a look in your implementation and saw it really deserved a try. I've run today a benchmark using some of my datasets in FFM format comparing standard LibFFM and your fork with custom pairwise rank customization.
\nI just run the models with the same FFM hyperparameters (without tunning for MAP ranking) and weight 1 for all samples. The environment was a Google Compute Engine instance with 32 CPUs, 256 GB RAM and FFM data in a SSD disk.
1) Dataset with categorical fields and some binned numeric fields
\nParameters: -l 0.0001 -k 6 -t 20 -r 0.2 -s 28 --auto-stop
\nRounds: 20
LibFFM - Cross-validation MAP = 0.65911
\nLibFFM - Cross-validation MAP + leak = 0.67795
\nLibFFM - Training time = 291m
LibFFM pairwise - Cross-validation MAP = 0.66040
\nLibFFM pairwise - Cross-validation MAP + leak = 0.67792
\nLibFFM pairwise - Training time = 631m
2) Dataset with only categorical fields\nParameters: -l 0.00002 -k 6 -t 50 -r 0.1 -s 28 --auto-stop
\n\nLibFFM - Training rounds = 13 (auto-stop on validation logloss)
\nLibFFM - Cross-validation MAP = 0.66428
\nLibFFM - Cross-validation MAP + leak = 0.67859
\nLibFFM - Training time = 27m
LibFFM - Training rounds = 26 (auto-stop on validation MAP)
\nLibFFM pairwise - Cross-validation MAP = 0.66552
\nLibFFM pairwise - Cross-validation MAP + leak = 0.67973
\nLibFFM pairwise - Training time = 211m
I noticed a diff of around 0.001 in CV MAP by for two datasets. Was this the order of difference you could find in your experiments with your FFM with ranking optimization?
\nPs. I also noticed a great increase in training logloss (from 0.41009 to 0.56855), but this is expected as the optimization is now driven by MAP and not logloss), right?
I am worried only about accuracy at this time, but I could notice that training time of your implementation is very slow compared to standard FFM, and that during most of the processing it is using only 1 CPU. Is that because you had not time to parallelize MAP evaluation?
\n\nWhich sample weight strategy have you used? More weight on recent samples / clicked samples / display_ids with more ads?
\n\nThanks again!
", "votes": 2, "replies": [] }, { "id": 158836, "author_name": "tito", "author_url": "", "post_date": "2017-01-30T11:06:15.573000", "content": "@Gabriel Moreira,
\n\nI uploaded this source code. You can find pairwise rank customization around here.
\n\nif(param.use_default_train){\n # default libffm training\n} else {\n # pairwise rank training\n}\n@Sameh Faidi,
\n\nI meant impression count as how many times the ad is shown.\nExplanation of impression in the context of online advertising can be found wikipedia.
\n\nFor example, impression counts for advertiser_id means how many times ads from the advertiser is shown.
", "votes": 2, "replies": [] }, { "id": 157179, "author_name": "tito", "author_url": "", "post_date": "2017-01-19T15:10:08.563000", "content": "Thank you, Sangzia,
\n\nThanks for your kindly share@CuteChibiko . It is really impressive and interesting. I have several questoins about gbdt leaf node as feature.
\n\n1). Does the gbdt feature contain label information? like click/ctr. Or if not contain, like impression/pageview count etc.
\n\n2). Does the gbdt training data have overlap with ffm training data, If yes, does it suffer from overfitting at first due to label twice used and do you use some method to avoid overfitting?
\n\n3). Your weighted average with gbdt leaf's gain is large. Have you compared with ffm containing the gbdt's count feature in some transformation like log without gbdt leafnode, or use gbdt and ffm pctr prediction score together feeded into stacking model. I'm curious that the gain is from gbdt leaf node transformation method or the count/statistics feature itself.
\n\n4). Ususallly gbdt training has several hundred or thousand trees, and the cost is large for ffm. How many trees you have used and do you have some hyper-paramter or feature tuning for the less tree numbers?
\n\nThanks a lot.
", "votes": 0, "replies": [ { "id": 159463, "author_name": "tito", "author_url": "", "post_date": "2017-02-02T11:35:06.360000", "content": "@nomo,
\n\n1) The gbdt feature dose not contain label information but contains impression/pageview count. This is almost same as what I listed in the first post of this topic.
\n\n2) Yes, it dose. The gbdt training data was same as ffm training data. But it did not suffer from overfitting.
\n\n3) I'm not sure I could understand your question correctly, but we only tried xgb leafnode information itself as feature and did not compared with any variations.
\n\n4) We used ntrees=39 and maxDepth=15. Almost all features and hyper-paramters of this model are the same as values of best xgb single model. But only eta was tuned for this model to reduce ntrees.
", "votes": 0, "replies": [] }, { "id": 159490, "author_name": "nomo", "author_url": "", "post_date": "2017-02-02T14:44:01.707000", "content": "@CuteChibiko thank you very much for your detailed explain. It is interesting:-)
\n\nI have also tried split data as present/future for xgb stacking training, but the result is neutral to not split one, and am interested in the different and your success trying reason.
\n\n1). For a.single model ffm / b.single model xgb / c. stacking model, your present/future split has gains on all of these three models or some of these models?
\n\n2). How much gain do your solution has compared with not split modeling? Is it mainly from present part or future part?
\n\n3). The future model process is interesting. Does the sample weight use exponent decay, like DecayFactor ^ (LastDay - SampleDay) as sample weight?
\n\n4). I have also used timestamp feature for split modeling present part, but use same hyper-parameter for the present/future part, and the result is neutral. Do your different parameters for these two part influence a lot? Or some other feature or model method also make a difference, like weighted average of present/future split modeling and not split model.
\n\nthanks a lot!
", "votes": 0, "replies": [] }, { "id": 159552, "author_name": "tito", "author_url": "", "post_date": "2017-02-02T18:48:33.603000", "content": "@nomo,
\n\n1) Splitting data worked for a and b. And It seemed to work for validation data set of stacking model, but LB score decreased. We did not have enough time to find the reason, so we did not use splitting for stacking model.
\n\n2) The single model gain is 0.001 for ffm future model with sample weight, 0.002 for ffm present model with timestamp feature and 0.002 for xgb present model with timestamp.
\n\n3) Exactly. We used this formula:
\n\nsample weight = 1 + 2 * 0.5 ^ ((max_timestamp - timestamp)/86400000.0)\n4) Timestamp feature for present data made big gain without any special method. We used almost same hyper-parameters for split model. Only tuned hyper-parameter is iterations/ntrees.
", "votes": 0, "replies": [] } ] }, { "id": 159137, "author_name": "tito", "author_url": "", "post_date": "2017-02-01T01:45:28.363000", "content": "@Gabriel Moreira, thank you for your feedback.
\n\nThe difference was 0.0015 ~ 0.002 for our models.\nRelatively bigger ETA for LibFFM pairwise made better results.
\n\nYou are right. It is using only 1 CPU to calculate validation score.\nI don't remember why I commented out here and did not use omp. :(
\n\nWe added higher weights to recent samples for ‘future’ model. This improved 'future' model score about 0.001.
", "votes": 0, "replies": [] }, { "id": 158778, "author_name": "Gabriel Moreira", "author_url": "", "post_date": "2017-01-30T01:48:11.027000", "content": "Thanks for sharing @CuteChibiko ! You've told you used a customized libffm supporting pairwise rank.\nI am really interested in how the FFM optimization works when using pairwise ranking for libffm.... could you share something about that?
", "votes": 0, "replies": [] }, { "id": 157575, "author_name": "", "author_url": "", "post_date": "2017-01-22T08:01:42.010000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "157166": "Congratulations to code monkey, brain-afk and everyone!\r\nAnd I’d like to thank my teammates carl and Little Boat. I could learn lots of things from them.\r\n\r\nAs [Little Boat explained](https://www.kaggle.com/c/outbrain-click-prediction/forums/t/27897/congrats-and-solution-sharing?forumMessageId=157062#post157062), our model is ffm models + xgboost models as first layer, and xgboost as second layer.\r\n\r\nI would like to explain our first layer models here.\r\n\r\n###**Feature extraction**\r\n\r\nFollowing is the list of unobvious features we used.\r\nWe made lots of other features which turned out to be useless.\r\nBut analyzing large scale user behavior was very interesting to me.\r\n\r\n - page view counts of each user.\r\n - page view counts of ad landing page.\r\n - impression counts for each ad_id, landing document_id, campaign_id and advertiser_id.\r\n - landing page confidence vector, where confidence vector is defined as a vector which element is composed of confidence level from documents_*.csv. This feature is used only for ffm as numeric data.\r\n - user confidence vector - this is the average of document confidence vector which is viewed by each user. This feature is used only for ffm as numeric data.\r\n - inner dot product of document confidence vectors of ad impression page and ad landing page .\r\n - inner dot product of user confidence vector and ad landing page document confidence vector\r\n - XGB leaf for ffm feature.\r\n - immediate document viewed after click event.\r\n\r\n###**Modeling and Training**\r\n\r\n####**FFM**\r\nIn addition to [carl’s light-ffm](https://www.kaggle.com/c/outbrain-click-prediction/forums/t/27892/introducing-light-ffm-and-stack-nn), we used another customized libffm which has pair-wise rank and can take sample weight.\r\nBest public LB score is 0.6974.\r\n\r\n####**XGB**\r\n We used xgboost with rank:pairwise objective.\r\nBest public LB score is 0.6885.\r\n\r\n####**Leak Row Excluding**\r\nWe trained ffm models without leak row by removing them from training data first, then blend/merge leak information later.\r\nThis improved single model score.\r\n\r\n####**Present/Future split**\r\nWe split test data into ‘present’ data (in time samples) and ‘future’ data (out time samples), then built models for each data.\r\nHere, Features and hyper parameters for each model are optimized independently.\r\nFor example we used timestamp feature for ‘present’ model, but did not used it for ‘future’ model. And we used sample weight of training data for ‘future’ model to add higher weight to last day than first day.\r\n\r\nPlease feel free to ask any questions.\r\n\r\n", "158945": "Great @CuteChibiko ! Thanks for sharing!", "157578": "Thank you Yuriy Makarov,\r\n\r\nYes, we tried CTR with some impression threshold. If impression threshold was low, CTR caused overfit. And if impression threshold was high, CTR did not contribute to score. So We did not use CTR in the end.\r\n\r\n\"immediate document viewed after click event.\" means the first document that was seen by a user after the user clicked an ad. ", "157274": "@Gabriel Moreira,\r\n\r\nYour CV score drop, from CV=0.679 to 0.670, is very big, so I think something should be happening.\r\nI'm not sure why, but maybe you are right. We made XGB models for test and validation respectively, so this may cause the difference.\r\n\r\nBTW, our best single score did not improved with GBDT leaves, but it worked for ensemble model.\r\n\r\nOur best single CV score without GBDT leaves was 0.69586 and score with GBDT leaves was 0.69588 when we tested GBDT leaves. And weighted average of them scored 0.6971.So even single model score was not improved, GBDT leaves feature was very important.", "157225": "@CuteChibiko I also tried the 3 idiots approach to add GBDT leaves nodes as features to FFM, but did not succeeded improving the score using standard LibFFM, (actually they made my FFM CV score worse). I tried 30 trees with 64 and 128 leaves, using XGBoost and LightGBM.\r\nDo you think my GBDT model may not have being accurate enought (CV=0.670), while my FFM model was (0.679)?\r\nOther aspect is that I used a fixed validation set following test set days distribution. So, I trained GBDT on full train set and used the model to predict both the train set and validation set. Maybe that could cause some overfit on FFM model, even with using leaf nodes?", "157180": "\"impression counts for each ad_id, landing document_id, campaign_id and advertiser_id.\"\r\n\r\nwhat do you mean by impression counts?", "157171": "I have 2 questions: \r\n\r\n 1. Could you explain what XGB leaf feature is? \r\n 2. Do you normalize the confidence vectors for documents?\r\n\r\nThanks and congrats!", "159127": "Good job @CuteChibiko - I took a look in your implementation and saw it really deserved a try. I've run today a benchmark using some of my datasets in FFM format comparing standard LibFFM and your fork with custom pairwise rank customization. \r\nI just run the models with the same FFM hyperparameters (without tunning for MAP ranking) and weight 1 for all samples. The environment was a Google Compute Engine instance with 32 CPUs, 256 GB RAM and FFM data in a SSD disk. \r\n\r\n1) Dataset with categorical fields and some binned numeric fields \r\nParameters: -l 0.0001 -k 6 -t 20 -r 0.2 -s 28 --auto-stop \r\nRounds: 20 \r\n \r\nLibFFM - Cross-validation MAP = 0.65911 \r\nLibFFM - Cross-validation MAP + leak = 0.67795 \r\nLibFFM - Training time = 291m \r\n \r\nLibFFM pairwise - Cross-validation MAP = 0.66040 \r\nLibFFM pairwise - Cross-validation MAP + leak = 0.67792 \r\nLibFFM pairwise - Training time = 631m \r\n\r\n----------------------------------------------------\r\n\r\n2) Dataset with only categorical fields\r\nParameters: -l 0.00002 -k 6 -t 50 -r 0.1 -s 28 --auto-stop\r\n\r\nLibFFM - Training rounds = 13 (auto-stop on validation logloss) \r\nLibFFM - Cross-validation MAP = 0.66428 \r\nLibFFM - Cross-validation MAP + leak = 0.67859 \r\nLibFFM - Training time = 27m\r\n\r\nLibFFM - Training rounds = 26 (auto-stop on validation MAP) \r\nLibFFM pairwise - Cross-validation MAP = 0.66552 \r\nLibFFM pairwise - Cross-validation MAP + leak = 0.67973 \r\nLibFFM pairwise - Training time = 211m\r\n\r\n-----------------------------------\r\n \r\nI noticed a diff of around 0.001 in CV MAP by for two datasets. Was this the order of difference you could find in your experiments with your FFM with ranking optimization? \r\nPs. I also noticed a great increase in training logloss (from 0.41009 to 0.56855), but this is expected as the optimization is now driven by MAP and not logloss), right? \r\n \r\nI am worried only about accuracy at this time, but I could notice that training time of your implementation is very slow compared to standard FFM, and that during most of the processing it is using only 1 CPU. Is that because you had not time to parallelize MAP evaluation? \r\n\r\nWhich sample weight strategy have you used? More weight on recent samples / clicked samples / display_ids with more ads? \r\n\r\nThanks again! ", "158836": "@Gabriel Moreira,\r\n\r\nI uploaded this source code. You can find pairwise rank customization [around here](https://github.com/CuteChibiko/Outbrain-Click-Prediction/blob/master/libffm_pairwise/ffm.cpp#L443).\r\n\r\n if(param.use_default_train){\r\n # default libffm training\r\n } else {\r\n # pairwise rank training\r\n }", "157187": "@Sameh Faidi,\r\n\r\nI meant impression count as how many times the ad is shown.\r\nExplanation of impression in the context of online advertising can be found [wikipedia](https://en.wikipedia.org/wiki/Impression_(online_media)). \r\n\r\nFor example, impression counts for advertiser_id means how many times ads from the advertiser is shown.\r\n", "157179": "Thank you, Sangzia,\r\n\r\n 1. Page 9 of [this presentation by 3 idiots](http://www.csie.ntu.edu.tw/~r01922136/kaggle-2014-criteo.pdf) is very good explanation of XGB leaf feature. \r\n 2. Document confidence vectors are not normalized, but user confidence vectors are normalized.", "159411": "Thanks for your kindly share@CuteChibiko . It is really impressive and interesting. I have several questoins about gbdt leaf node as feature.\n\n1). Does the gbdt feature contain label information? like click/ctr. Or if not contain, like impression/pageview count etc.\n\n2). Does the gbdt training data have overlap with ffm training data, If yes, does it suffer from overfitting at first due to label twice used and do you use some method to avoid overfitting?\n\n3). Your weighted average with gbdt leaf's gain is large. Have you compared with ffm containing the gbdt's count feature in some transformation like log without gbdt leafnode, or use gbdt and ffm pctr prediction score together feeded into stacking model. I'm curious that the gain is from gbdt leaf node transformation method or the count/statistics feature itself.\n\n4). Ususallly gbdt training has several hundred or thousand trees, and the cost is large for ffm. How many trees you have used and do you have some hyper-paramter or feature tuning for the less tree numbers?\n\n\nThanks a lot.\n", "159137": "@Gabriel Moreira, thank you for your feedback.\r\n\r\nThe difference was 0.0015 ~ 0.002 for our models.\r\nRelatively bigger ETA for LibFFM pairwise made better results.\r\n\r\nYou are right. It is using only 1 CPU to calculate validation score.\r\nI don't remember why I commented out [here](https://github.com/CuteChibiko/Outbrain-Click-Prediction/blob/master/libffm_pairwise/ffm.cpp#L500) and did not use omp. :(\r\n\r\nWe added higher weights to recent samples for ‘future’ model. This improved 'future' model score about 0.001.", "158778": "Thanks for sharing @CuteChibiko ! You've told you used a customized libffm supporting pairwise rank.\r\nI am really interested in how the FFM optimization works when using pairwise ranking for libffm.... could you share something about that?", "157575": "" } }