XGBoost is the most used model in Classification problems on Kaggle nowadays. Can it be used for recommendation tasks? In recent paper XGBoost: A Scalable Tree Boosting System authors state that they have achieved comparable score to the benchmark paper published on Yahoo LTRC dataset. You can read it in Section 6.4 Learning to Rank of the paper.
\nThe demo of learning to rank given on the github repo is unclear.
\nCan anyone explain after setting objective rank:pairwise how it can be used for recommendations in this competition.
XGBoost is the most used model in Classification problems on Kaggle nowadays. Can it be used for recommendation tasks? In recent paper XGBoost: A Scalable Tree Boosting System authors state that they have achieved comparable score to the benchmark paper published on Yahoo LTRC dataset. You can read it in Section 6.4 Learning to Rank of the paper.
\nThe demo of learning to rank given on the github repo is unclear.
\nCan anyone explain after setting objective rank:pairwise how it can be used for recommendations in this competition.
What exactly is not clear?\nWhich part you have problems with?
", "rawMarkdown": "What exactly is not clear?\r\nWhich part you have problems with?", "votes": null }, { "id": "115409", "postDate": "04/18/2016 15:55:39", "content": "First , in the training input data we are giving the relevance score(0,1,2..) for each query in the beginning and then the features, but in the test data too we are giving the relevance score?? I mean if we are giving the relevance score of each query in test data also, then what is left to predict?
\nSecond, while running the demo example I increased the num_round to 100, many of the predictions changed to negative. Like for the first group of 8 queries prediction is
\n1.40337,\n-2.38498,\n0.817343,\n1.50825,\n0.483668,\n-0.0549217,\n-2.33494,\n-2.04612
\n What exactly is predicted here,please explain?
Third, I ran 4 different models,
\n1. No eval metric parameter
\n2. eval_metric="ndcg"
\n3. eval_metric="ndcg@2"
\n4. eval_metric="map@5"
\nEach one produced exactly the same pred.txt file, while it is written in official docs that " "ndcg@n","map@n": n can be assigned as an integer to cut off the top positions in the lists for evaluation.". Doesn't it optimize predictions according to eval metric chosen? And is it possible that ndcg@2 and map@5 produce exactly the same predictions??
To answer the third question. The objective function is used for optimization not the evaluation function. rank:pairwise and multi:softprob are objective functions, and will change the predictions depending on which one you use. Eval_metrics let you check for over fitting when used on a validation set and also allow for the application of early stopping.
Regarding the first point I think training and test, are like in cross validation training to get the parameter estimates and test is just to validate how good it is. If you look at the rundexp.sh likes 4-5
\n\n# split train and test\npython mknfold.py agaricus.txt 1\nso it splits the base dataset into train and test. \nThe same goes for mq2008.conf
\n\n# The path of validation data, used to monitor training process, here [test] sets name of the validation set\neval[test] = "mq2008.vali" \nI am talking about the testing file particularly, If you see the last lines of mq2008.conf
# The path of test data \ntest:data = "mq2008.test" \nIn the runexp.sh of rank folder, we have separate train,test and validate and all 3 of them are in same format.
\n\npython trans_data.py train.txt mq2008.train mq2008.train.group\n\npython trans_data.py test.txt mq2008.test mq2008.test.group\n\npython trans_data.py vali.txt mq2008.vali mq2008.vali.group\n[quote=Aditya;115409]
\n\nFirst , in the training input data we are giving the relevance score(0,1,2..) for each query in the beginning and then the features, but in the test data too we are giving the relevance score?? I mean if we are giving the relevance score of each query in test data also, then what is left to predict?
\nSecond, while running the demo example I increased the num_round to 100, many of the predictions changed to negative. Like for the first group of 8 queries prediction is
\n1.40337,\n-2.38498,\n0.817343,\n1.50825,\n0.483668,\n-0.0549217,\n-2.33494,\n-2.04612
\n What exactly is predicted here,please explain?
Third, I ran 4 different models,
\n1. No eval metric parameter
\n2. eval_metric="ndcg"
\n3. eval_metric="ndcg@2"
\n4. eval_metric="map@5"
\nEach one produced exactly the same pred.txt file, while it is written in official docs that " "ndcg@n","map@n": n can be assigned as an integer to cut off the top positions in the lists for evaluation.". Doesn't it optimize predictions according to eval metric chosen? And is it possible that ndcg@2 and map@5 produce exactly the same predictions??
[/quote]
\n\nFirst post here, hope that no rules will be broken by me.
\n\nActually, in Learning to Rank field, we are trying to predict the relative score for each document to a specific query. That is, this is not a regression problem or classification problem. Hence, if a document, attached to a query, gets a negative predict score, it means and only means that it's relatively less relative to the query, when comparing to other document(s), with positive scores.
\n\nThe eval_metric parameter is here to help you keeping track with the train set process. You can set a watch list, train set, validation set and their name, and pass the list to the train function, then the function will print the value of metric by each iteration. This is not a parameter that will effect the train process, and thus change this param will get the same predict scores (without random subsamplling).
\n\nL
", "rawMarkdown": "[quote=Aditya;115409]\r\n\r\nFirst , in the training input data we are giving the relevance score(0,1,2..) for each query in the beginning and then the features, but **in the test data too we are giving the relevance score??** I mean if we are giving the relevance score of each query in test data also, then what is left to predict?I have the same question with you. Whats the exactly prediction result mean……
", "rawMarkdown": "I have the same question with you. Whats the exactly prediction result mean……", "votes": null } ], "comments": [ { "id": 115401, "author_name": "mpekalski", "author_url": "", "post_date": "04/18/2016 14:43:12", "content": "What exactly is not clear?\nWhich part you have problems with?
", "votes": null, "replies": [] }, { "id": 115409, "author_name": "aditya23", "author_url": "", "post_date": "04/18/2016 15:55:39", "content": "First , in the training input data we are giving the relevance score(0,1,2..) for each query in the beginning and then the features, but in the test data too we are giving the relevance score?? I mean if we are giving the relevance score of each query in test data also, then what is left to predict?
\nSecond, while running the demo example I increased the num_round to 100, many of the predictions changed to negative. Like for the first group of 8 queries prediction is
\n1.40337,\n-2.38498,\n0.817343,\n1.50825,\n0.483668,\n-0.0549217,\n-2.33494,\n-2.04612
\n What exactly is predicted here,please explain?
Third, I ran 4 different models,
\n1. No eval metric parameter
\n2. eval_metric="ndcg"
\n3. eval_metric="ndcg@2"
\n4. eval_metric="map@5"
\nEach one produced exactly the same pred.txt file, while it is written in official docs that " "ndcg@n","map@n": n can be assigned as an integer to cut off the top positions in the lists for evaluation.". Doesn't it optimize predictions according to eval metric chosen? And is it possible that ndcg@2 and map@5 produce exactly the same predictions??
To answer the third question. The objective function is used for optimization not the evaluation function. rank:pairwise and multi:softprob are objective functions, and will change the predictions depending on which one you use. Eval_metrics let you check for over fitting when used on a validation set and also allow for the application of early stopping.
Regarding the first point I think training and test, are like in cross validation training to get the parameter estimates and test is just to validate how good it is. If you look at the rundexp.sh likes 4-5
\n\n# split train and test\npython mknfold.py agaricus.txt 1\nso it splits the base dataset into train and test. \nThe same goes for mq2008.conf
\n\n# The path of validation data, used to monitor training process, here [test] sets name of the validation set\neval[test] = "mq2008.vali" \nI am talking about the testing file particularly, If you see the last lines of mq2008.conf
# The path of test data \ntest:data = "mq2008.test" \nIn the runexp.sh of rank folder, we have separate train,test and validate and all 3 of them are in same format.
\n\npython trans_data.py train.txt mq2008.train mq2008.train.group\n\npython trans_data.py test.txt mq2008.test mq2008.test.group\n\npython trans_data.py vali.txt mq2008.vali mq2008.vali.group\n[quote=Aditya;115409]
\n\nFirst , in the training input data we are giving the relevance score(0,1,2..) for each query in the beginning and then the features, but in the test data too we are giving the relevance score?? I mean if we are giving the relevance score of each query in test data also, then what is left to predict?
\nSecond, while running the demo example I increased the num_round to 100, many of the predictions changed to negative. Like for the first group of 8 queries prediction is
\n1.40337,\n-2.38498,\n0.817343,\n1.50825,\n0.483668,\n-0.0549217,\n-2.33494,\n-2.04612
\n What exactly is predicted here,please explain?
Third, I ran 4 different models,
\n1. No eval metric parameter
\n2. eval_metric="ndcg"
\n3. eval_metric="ndcg@2"
\n4. eval_metric="map@5"
\nEach one produced exactly the same pred.txt file, while it is written in official docs that " "ndcg@n","map@n": n can be assigned as an integer to cut off the top positions in the lists for evaluation.". Doesn't it optimize predictions according to eval metric chosen? And is it possible that ndcg@2 and map@5 produce exactly the same predictions??
[/quote]
\n\nFirst post here, hope that no rules will be broken by me.
\n\nActually, in Learning to Rank field, we are trying to predict the relative score for each document to a specific query. That is, this is not a regression problem or classification problem. Hence, if a document, attached to a query, gets a negative predict score, it means and only means that it's relatively less relative to the query, when comparing to other document(s), with positive scores.
\n\nThe eval_metric parameter is here to help you keeping track with the train set process. You can set a watch list, train set, validation set and their name, and pass the list to the train function, then the function will print the value of metric by each iteration. This is not a parameter that will effect the train process, and thus change this param will get the same predict scores (without random subsamplling).
\n\nL
", "votes": null, "replies": [] }, { "id": 201011, "author_name": "sparkingarthur", "author_url": "", "post_date": "07/10/2017 07:14:00", "content": "I have the same question with you. Whats the exactly prediction result mean……
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "115398": "XGBoost is the most used model in Classification problems on Kaggle nowadays. Can it be used for recommendation tasks? In recent paper [XGBoost: A Scalable Tree Boosting System][1] authors state that they have achieved comparable score to the benchmark paper published on Yahoo LTRC dataset. You can read it in **Section 6.4 Learning to Rank** of the paper.