Hi, I've done ncdg5 before via a custom function in xgboost. I also see that there is an "map" and "map@n" eval_metric in xgboost, but simply it doesn't work for me. R crashes
\n\nI don't know if these is crossing the competition rules, but is there a hint on these?
\n\nThanks
", "messages": [ { "id": "117628", "postDate": "04/29/2016 21:55:59", "content": "Hi, I've done ncdg5 before via a custom function in xgboost. I also see that there is an "map" and "map@n" eval_metric in xgboost, but simply it doesn't work for me. R crashes
\n\nI don't know if these is crossing the competition rules, but is there a hint on these?
\n\nThanks
", "rawMarkdown": "Hi, I've done ncdg5 before via a custom function in xgboost. I also see that there is an \"map\" and \"map@n\" eval_metric in xgboost, but simply it doesn't work for me. R crashes\r\n\r\nI don't know if these is crossing the competition rules, but is there a hint on these?\r\n\r\nThanks", "votes": null }, { "id": "117630", "postDate": "04/29/2016 22:01:27", "content": "My understanding is that the ndcg@n and map@n eval functions built into xgboost only work when using "rank:pairwise" for your objective. If you want to use a different objective then you need to define a custom map@5 eval function for it to work.
\n\nFor reference, see this issue on xgboost's GitHub.
", "rawMarkdown": "My understanding is that the ndcg@n and map@n eval functions built into xgboost only work when using \"rank:pairwise\" for your objective. If you want to use a different objective then you need to define a custom map@5 eval function for it to work.\r\n\r\nFor reference, see this [issue][1] on xgboost's GitHub.\r\n\r\n\r\n [1]: https://github.com/dmlc/xgboost/issues/1143", "votes": null }, { "id": "117705", "postDate": "04/30/2016 13:01:40", "content": "[quote=Branden Murray;117630]
\n\nFor reference, see this issue on xgboost's GitHub.
\n\n[/quote]
\n\nThat's my issue ;). What I ended up doing is optimising for mlogloss instead, but if someone smarter than me can implement map@5 as a custom metric in xgboost that would be very useful!
\n\nI think xgboost might not be the way to go in this competition, as it takes so long to train (as it is 100-class problem, it trains 100 trees per boosting rounds) and this means it was taking me 12 hours on dual xeons to converge.
", "rawMarkdown": "[quote=Branden Murray;117630]\r\n\r\nFor reference, see this [issue][1] on xgboost's GitHub.\r\n\r\n\r\n [1]: https://github.com/dmlc/xgboost/issues/1143\r\n\r\n[/quote]\r\n\r\nThat's my issue ;). What I ended up doing is optimising for mlogloss instead, but if someone smarter than me can implement map@5 as a custom metric in xgboost that would be very useful!\r\n\r\nI think xgboost might not be the way to go in this competition, as it takes so long to train (as it is 100-class problem, it trains 100 trees per boosting rounds) and this means it was taking me 12 hours on dual xeons to converge.", "votes": null }, { "id": "117749", "postDate": "04/30/2016 17:20:28", "content": "In R,
\n\nlibrary(Metrics)\nmap5 <- function(preds, dtrain) {\n labels <- as.list(getinfo(dtrain,"label"))\n num.class = 100\n pred <- matrix(preds, nrow = num.class)\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\n top <- split(top, 1:NROW(top))\n\n map <- mapk(5, labels, top)\n return(list(metric = "map5", value = map))\n}\nThen set eval_metric=map5 in your params.
[edit] Realized the mapk function takes in lists, not vectors. This now gives the same output as SK's function below.
I believe in the classification case (and as long as you make sure to return 5 unique predictions per example) the metric degenerates to a simple:
\n\nmap5 = function(preds, dtrain) {\n labels = getinfo(dtrain, 'label')\n preds = t(matrix(preds, ncol = length(labels)))\n preds = t(apply(preds, 1, order, decreasing = T))[, 1:5] - 1\n succ = (preds == labels)\n w = 1 / (1:5)\n map5 = mean(succ %*% w)\n return (list(metric = 'map5', value = map5))\n}\nwhich can probably be optimized a bit more.
", "rawMarkdown": "I believe in the classification case (and as long as you make sure to return 5 unique predictions per example) the metric degenerates to a simple:\r\n\r\n map5 = function(preds, dtrain) {\r\n labels = getinfo(dtrain, 'label')\r\n preds = t(matrix(preds, ncol = length(labels)))\r\n preds = t(apply(preds, 1, order, decreasing = T))[, 1:5] - 1\r\n succ = (preds == labels)\r\n w = 1 / (1:5)\r\n map5 = mean(succ %*% w)\r\n return (list(metric = 'map5', value = map5))\r\n }\r\n\r\nwhich can probably be optimized a bit more.", "votes": null }, { "id": "117781", "postDate": "04/30/2016 21:31:58", "content": "[quote=Branden Murray;117749]
\n\nIn R,
\n\nlibrary(Metrics)\nmap5 <- function(preds, dtrain) {\n labels <- getinfo(dtrain,"label")\n num.class = 100\n pred <- matrix(preds, nrow = num.class)\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\n\n map <- mapk(5, labels, top)\n return(list(metric = "map5", value = map))\n}\nThen set eval_metric=map5 in your params.
[/quote]
\n\nThanks, I'm sure this will be useful for a lot of people. I'm a python guy myself, so I still need to find another way :)
", "rawMarkdown": "[quote=Branden Murray;117749]\r\n\r\nIn R, \r\n\r\n library(Metrics)\r\n map5 <- function(preds, dtrain) {\r\n labels <- getinfo(dtrain,\"label\")\r\n num.class = 100\r\n pred <- matrix(preds, nrow = num.class)\r\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\r\n \r\n map <- mapk(5, labels, top)\r\n return(list(metric = \"map5\", value = map))\r\n }\r\n\r\nThen set `eval_metric=map5` in your params.\r\n\r\n[/quote]\r\n\r\nThanks, I'm sure this will be useful for a lot of people. I'm a python guy myself, so I still need to find another way :)", "votes": null }, { "id": "117815", "postDate": "05/01/2016 03:15:33", "content": "@anokas, something like this should work here:
\n\ndef map5eval(preds, dtrain):\n actual = dtrain.get_label()\n predicted = preds.argsort(axis=1)[:,-np.arange(1,6)]\n metric = 0.\n for i in range(5):\n metric += np.sum(actual==predicted[:,i])/(i+1)\n metric /= actual.shape[0]\n return 'MAP@5', metric\nThen pass to xgb.train as feval = map5eval.
Edit: reverse the sign for use in early stopping.
\n\nEdit 2: fixed an error in array slicing. Thanks, @HN Musac.
", "rawMarkdown": "anokas, something like this should work here:\r\n\r\n def map5eval(preds, dtrain):\r\n actual = dtrain.get_label()\r\n predicted = preds.argsort(axis=1)[:,-np.arange(1,6)]\r\n metric = 0.\r\n for i in range(5):\r\n metric += np.sum(actual==predicted[:,i])/(i+1)\r\n metric /= actual.shape[0]\r\n return 'MAP@5', metric\r\nThen pass to `xgb.train` as `feval = map5eval`.\r\n\r\n*Edit: reverse the sign for use in early stopping.*\r\n\r\n*Edit 2: fixed an error in array slicing. Thanks, @HN Musac.*", "votes": null }, { "id": "118038", "postDate": "05/02/2016 17:26:03", "content": "Has anybody tried using any of these with xgboost yet? What is the difference between your local CV score and your LB score? On a random subset of 100,000 training rows I got 3-fold CV scores of ~0.52 (s.d. = 0.002) using my map5 function, but LB score was ~0.42. Using SK's function on the same subset I was getting CV scores of ~0.58.
", "rawMarkdown": "Has anybody tried using any of these with xgboost yet? What is the difference between your local CV score and your LB score? On a random subset of 100,000 training rows I got 3-fold CV scores of ~0.52 (s.d. = 0.002) using my map5 function, but LB score was ~0.42. Using SK's function on the same subset I was getting CV scores of ~0.58.", "votes": null }, { "id": "118103", "postDate": "05/02/2016 21:22:41", "content": "[quote=dune_dweller;117815]
\n\ndef map5eval(preds, dtrain):\n actual = dtrain.get_label()\n predicted = preds.argsort(axis=1)[:,-np.arange(5)]\n metric = 0.\n for i in range(5):\n metric += np.sum(actual==predicted[:,i])/(i+1)\n metric /= actual.shape[0]\n return 'MAP@5', metric\n[/quote]
\n\nHello Dune, i am a bit confused about the format of preds. Does preds contain the probabilities or the classes labels?
\n\n-np.arange(5) gives [0, -1, -2, -3, -4], , which slices the first, the last, 2nd to last, 3rd to last and 4th to last columns, correct?
", "rawMarkdown": "[quote=dune_dweller;117815]\r\n\r\n def map5eval(preds, dtrain):\r\n actual = dtrain.get_label()\r\n predicted = preds.argsort(axis=1)[:,-np.arange(5)]\r\n metric = 0.\r\n for i in range(5):\r\n metric += np.sum(actual==predicted[:,i])/(i+1)\r\n metric /= actual.shape[0]\r\n return 'MAP@5', metric\r\n\r\n[/quote]\r\n\r\nHello Dune, i am a bit confused about the format of preds. Does preds contain the probabilities or the classes labels?\r\n\r\n-np.arange(5) gives [0, -1, -2, -3, -4], , which slices the first, the last, 2nd to last, 3rd to last and 4th to last columns, correct?", "votes": null }, { "id": "118249", "postDate": "05/03/2016 02:45:55", "content": "preds.argsort(axis=1) returns an array of indices into each row from smallest element to largest. Although you made me realize that I'm slicing it wrong: need to replace -np.arange(5) with -np.arange(1,6), so that it takes the last element, then the second-last and so on. This way the most probable hotel cluster goes first.
Lol, this just made my NN model go from 0.17 to 0.27 in cv. Maybe it's not as hopeless as I thought =). I'll edit the original post to include this correction, thanks!
", "rawMarkdown": "`preds.`[`argsort`](http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.argsort.html)`(axis=1)` returns an array of indices into each row from smallest element to largest. Although you made me realize that I'm slicing it wrong: need to replace `-np.arange(5)` with `-np.arange(1,6)`, so that it takes the last element, then the second-last and so on. This way the most probable hotel cluster goes first.\r\n\r\nLol, this just made my NN model go from 0.17 to 0.27 in cv. Maybe it's not as hopeless as I thought =). I'll edit the original post to include this correction, thanks!", "votes": null }, { "id": "118493", "postDate": "05/03/2016 23:41:51", "content": "Another option is to use MAPK from Ben Hammers ml_metrics . He has both R and Python implementations.
", "rawMarkdown": "Another option is to use MAPK from Ben Hammers [ml_metrics](https://github.com/benhamner/Metrics) . He has both R and Python implementations.", "votes": null }, { "id": "118578", "postDate": "05/04/2016 11:02:17", "content": "Is there a way to compare the score on the leaderboard with the score of this map5 function? \nI got something like .24 in cross-validation, which does not seem much compared to the "naive" method of just going for the most popular hotels per destination.\nHowever, when I submit the results I got .04 - apparently there is still a bug in how I interpret the result matrix from the predict function.
", "rawMarkdown": "Is there a way to compare the score on the leaderboard with the score of this map5 function? \r\nI got something like .24 in cross-validation, which does not seem much compared to the \"naive\" method of just going for the most popular hotels per destination.\r\nHowever, when I submit the results I got .04 - apparently there is still a bug in how I interpret the result matrix from the predict function.", "votes": null }, { "id": "118583", "postDate": "05/04/2016 11:24:55", "content": "[quote=Branden Murray;118038]
\n\nHas anybody tried using any of these with xgboost yet? What is the difference between your local CV score and your LB score? On a random subset of 100,000 training rows I got 3-fold CV scores of ~0.52 (s.d. = 0.002) using my map5 function, but LB score was ~0.42. Using SK's function on the same subset I was getting CV scores of ~0.58.
\n\n[/quote]
\n\nDid you generate a time-based validation set? I found that with a small sample, ordering by time (which is what the LB uses) results in much worse performance than testing on a random subset. In a particular case I got validation score of 0.6 when test on a random subset of the train, and 0.37xx when the test set was generated by time.
", "rawMarkdown": "[quote=Branden Murray;118038]\r\n\r\nHas anybody tried using any of these with xgboost yet? What is the difference between your local CV score and your LB score? On a random subset of 100,000 training rows I got 3-fold CV scores of ~0.52 (s.d. = 0.002) using my map5 function, but LB score was ~0.42. Using SK's function on the same subset I was getting CV scores of ~0.58.\r\n\r\n[/quote]\r\n\r\nDid you generate a time-based validation set? I found that with a small sample, ordering by time (which is what the LB uses) results in much worse performance than testing on a random subset. In a particular case I got validation score of 0.6 when test on a random subset of the train, and 0.37xx when the test set was generated by time.", "votes": null }, { "id": "118606", "postDate": "05/04/2016 13:37:39", "content": "I trained a neural net model on the full train.csv by reading it in chunks. First I used each chunk to evaluate performance and then to train the model. By the end of training local score was around 0.29-0.30. And leaderboard score was about 0.277. This wasn't a time-based split, but not a small sample either :).
", "rawMarkdown": "I trained a neural net model on the full train.csv by reading it in chunks. First I used each chunk to evaluate performance and then to train the model. By the end of training local score was around 0.29-0.30. And leaderboard score was about 0.277. This wasn't a time-based split, but not a small sample either :).", "votes": null }, { "id": "118642", "postDate": "05/04/2016 15:26:02", "content": "Reply to self - fixed the submission issue, and difference between my CV score and LB score was .02, in other words very small. Alas, model is still not performing better than the "most popular".
\n\n[quote=mafux777;118578]
\n\nIs there a way to compare the score on the leaderboard with the score of this map5 function? \nI got something like .24 in cross-validation, which does not seem much compared to the "naive" method of just going for the most popular hotels per destination.\nHowever, when I submit the results I got .04 - apparently there is still a bug in how I interpret the result matrix from the predict function.
\n\n[/quote]
", "rawMarkdown": "Reply to self - fixed the submission issue, and difference between my CV score and LB score was .02, in other words very small. Alas, model is still not performing better than the \"most popular\".\r\n\r\n[quote=mafux777;118578]\r\n\r\nIs there a way to compare the score on the leaderboard with the score of this map5 function? \r\nI got something like .24 in cross-validation, which does not seem much compared to the \"naive\" method of just going for the most popular hotels per destination.\r\nHowever, when I submit the results I got .04 - apparently there is still a bug in how I interpret the result matrix from the predict function. \r\n\r\n[/quote]", "votes": null }, { "id": "120733", "postDate": "05/20/2016 06:19:59", "content": "[quote=Branden Murray;117749]
\n\nIn R,
\n\nlibrary(Metrics)\nmap5 <- function(preds, dtrain) {\n labels <- as.list(getinfo(dtrain,"label"))\n num.class = 100\n pred <- matrix(preds, nrow = num.class)\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\n top <- split(top, 1:NROW(top))\n\n map <- mapk(5, labels, top)\n return(list(metric = "map5", value = map))\n}\nThen set eval_metric=map5 in your params.
[edit] Realized the mapk function takes in lists, not vectors. This now gives the same output as SK's function below.
[/quote]
\n\nWhat does this code return? I am getting just a large numeric list, i thought multi:softprob was supposed to return a probability matrix for the classes?!
", "rawMarkdown": "[quote=Branden Murray;117749]\r\n\r\nIn R, \r\n\r\n library(Metrics)\r\n map5 <- function(preds, dtrain) {\r\n labels <- as.list(getinfo(dtrain,\"label\"))\r\n num.class = 100\r\n pred <- matrix(preds, nrow = num.class)\r\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\r\n top <- split(top, 1:NROW(top))\r\n \r\n map <- mapk(5, labels, top)\r\n return(list(metric = \"map5\", value = map))\r\n }\r\n\r\nThen set `eval_metric=map5` in your params.\r\n\r\n*[edit] Realized the `mapk` function takes in lists, not vectors. This now gives the same output as SK's function below.*\r\n\r\n[/quote]\r\n\r\nWhat does this code return? I am getting just a large numeric list, i thought multi:softprob was supposed to return a probability matrix for the classes?!", "votes": null }, { "id": "120734", "postDate": "05/20/2016 06:26:38", "content": "That's just an eval_function for xgboost, nothing to do with your issue. For multi-class problems xgboost returns a vector and you need to transform that vector into a matrix. You can do this with code below:
\n\ntestPreds <- predict(xgb_model, dtest)\ntestPreds <- t(matrix(testPreds, nrow=100))\nThe columns will be the classes in order, from 0 to 99.
", "rawMarkdown": "That's just an eval_function for xgboost, nothing to do with your issue. For multi-class problems xgboost returns a vector and you need to transform that vector into a matrix. You can do this with code below:\r\n\r\n testPreds <- predict(xgb_model, dtest)\r\n testPreds <- t(matrix(testPreds, nrow=100))\r\n\r\nThe columns will be the classes in order, from 0 to 99.", "votes": null }, { "id": "120735", "postDate": "05/20/2016 06:42:02", "content": "[quote=Branden Murray;120734]
\n\nThat's just an eval_function for xgboost, nothing to do with your issue. For multi-class problems xgboost returns a vector and you need to transform that vector into a matrix. You can do this with code below:
\n\ntestPreds <- predict(xgb_model, dtest)\ntestPreds <- t(matrix(testPreds, nrow=100))\nThe columns will be the classes in order, from 0 to 99.
\n\n[/quote]
\n\nohh! Just figured\nThanks for clearing the doubt!! :)
", "rawMarkdown": "[quote=Branden Murray;120734]\r\n\r\nThat's just an eval_function for xgboost, nothing to do with your issue. For multi-class problems xgboost returns a vector and you need to transform that vector into a matrix. You can do this with code below:\r\n\r\n testPreds <- predict(xgb_model, dtest)\r\n testPreds <- t(matrix(testPreds, nrow=100))\r\n\r\nThe columns will be the classes in order, from 0 to 99.\r\n\r\n[/quote]\r\n\r\nohh! Just figured\r\nThanks for clearing the doubt!! :)", "votes": null }, { "id": "120974", "postDate": "05/22/2016 09:49:49", "content": "do you get "xgboost.core.XGBoostError: label size predict size not match" error after using ml_metrics ? mapk5 function is i copied from that link and changed the name, and i run a cv validation, code is below, has anyone used that metric?:
\n\n\ndef apk(actual, predicted, k=5):\n if len(predicted)>k:\n predicted = predicted[:k]\n\n score = 0.0\n num_hits = 0.0\n\n for i,p in enumerate(predicted):\n if p in actual and p not in predicted[:i]:\n num_hits += 1.0\n score += num_hits / (i+1.0)\n\n if not actual:\n return 0.0\n\n return score / min(len(actual), k)\n\ndef mapk5(actual, predicted, k=5):\n return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])\n\nparams = {}\nparams["objective"] = "multi:softprob"\nparams["booster"] = "gbtree"\nparams["num_class"] = 100\nparams["eta"] = 0.01\nparams["subsample"] = 0.75\nparams["colsample_bytree"] = 0.75\nparams["max_depth"] = 6\nparams["min_child_weight"] = 3\nparams["silent"] = 1\nxgtrain = xgb.DMatrix(X, y)\nprint xgb.cv(params, xgtrain, 1000, nfold=5, metrics=['mapk5'], early_stopping_rounds=50)\n\r\ndef apk(actual, predicted, k=5):\r\n if len(predicted)>k:\r\n predicted = predicted[:k]\r\n\r\n score = 0.0\r\n num_hits = 0.0\r\n\r\n for i,p in enumerate(predicted):\r\n if p in actual and p not in predicted[:i]:\r\n num_hits += 1.0\r\n score += num_hits / (i+1.0)\r\n\r\n if not actual:\r\n return 0.0\r\n\r\n return score / min(len(actual), k)\r\n\r\ndef mapk5(actual, predicted, k=5):\r\n return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])\r\n\r\nparams = {}\r\nparams[\"objective\"] = \"multi:softprob\"\r\nparams[\"booster\"] = \"gbtree\"\r\nparams[\"num_class\"] = 100\r\nparams[\"eta\"] = 0.01\r\nparams[\"subsample\"] = 0.75\r\nparams[\"colsample_bytree\"] = 0.75\r\nparams[\"max_depth\"] = 6\r\nparams[\"min_child_weight\"] = 3\r\nparams[\"silent\"] = 1\r\nxgtrain = xgb.DMatrix(X, y)\r\nprint xgb.cv(params, xgtrain, 1000, nfold=5, metrics=['mapk5'], early_stopping_rounds=50)\r\nsorry for confusion, i've fixed that problem by passing feval= mapk5 instead of metrics param.
", "rawMarkdown": "sorry for confusion, i've fixed that problem by passing feval= mapk5 instead of metrics param.", "votes": null }, { "id": "121034", "postDate": "05/23/2016 04:37:41", "content": "@dune_dweller , which map@5 cv score do you get? i used your eval function and only get about 0.2, looks a bit lower. Maybe there're problems in my code.
", "rawMarkdown": "dune_dweller , which map@5 cv score do you get? i used your eval function and only get about 0.2, looks a bit lower. Maybe there're problems in my code.", "votes": null }, { "id": "122237", "postDate": "06/02/2016 08:41:46", "content": "@masterLiu, your code seems way too complicated to me.
\n\nA slightly simplified version of @dune_dweller map5 function runs quite fast:
\n\ndef map5eval(preds, dtrain, k=5):\n actual = dtrain.get_label()\n predicted = (-preds).argsort(axis=1)[:,:k]\n metric = 0.\n for i in range(5):\n metric += np.sum(actual==predicted[:,i])/(i+1)\n metric /= actual.shape[0]\n return 'MAP@5', metric\nAlso, do not forget to pass maximize=True as a parameter to train.
", "rawMarkdown": "masterLiu, your code seems way too complicated to me. \r\n\r\nA slightly simplified version of @dune_dweller map5 function runs quite fast:\r\n\r\n def map5eval(preds, dtrain, k=5):\r\n actual = dtrain.get_label()\r\n predicted = (-preds).argsort(axis=1)[:,:k]\r\n metric = 0.\r\n for i in range(5):\r\n metric += np.sum(actual==predicted[:,i])/(i+1)\r\n metric /= actual.shape[0]\r\n return 'MAP@5', metric\r\n\r\nAlso, do not forget to pass maximize=True as a parameter to train.", "votes": null }, { "id": "122558", "postDate": "06/05/2016 04:00:32", "content": "@Branden Murray I face the same problem with you. I use @CPMP evaluation metric and I could get very high validation performance. Is it caused by leak information or there's something wrong with the evaluation metric? I used 100000 rows with 33% as validation set.
", "rawMarkdown": "Branden Murray I face the same problem with you. I use @CPMP evaluation metric and I could get very high validation performance. Is it caused by leak information or there's something wrong with the evaluation metric? I used 100000 rows with 33% as validation set.", "votes": null }, { "id": "122568", "postDate": "06/05/2016 07:46:53", "content": "@Feng, If I recall correctly, my problems were the result of a leak from the way I was doing feature engineering and splitting my data. I stopped using map5 as my eval metric in xgboost because xgboost seemed to run faster using 'mlogloss' (I think just because mlogloss is built in and therefore calculates much faster)
", "rawMarkdown": "Feng, If I recall correctly, my problems were the result of a leak from the way I was doing feature engineering and splitting my data. I stopped using map5 as my eval metric in xgboost because xgboost seemed to run faster using 'mlogloss' (I think just because mlogloss is built in and therefore calculates much faster)", "votes": null }, { "id": "122607", "postDate": "06/05/2016 18:36:35", "content": "@Branden Murray Thank you. But for mlogloss, as your experience. What's the score corresponding to like map@5=0.3 or 0.35 . And when you change to mlogloss, does your code speed up significantly? Thank you.
", "rawMarkdown": "Branden Murray Thank you. But for mlogloss, as your experience. What's the score corresponding to like map@5=0.3 or 0.35 . And when you change to mlogloss, does your code speed up significantly? Thank you.", "votes": null }, { "id": "123000", "postDate": "06/09/2016 04:13:28", "content": "has anyone come across this error "IndexError: too many indices for array" on line 3 while executing the code by @CPMP.
", "rawMarkdown": "has anyone come across this error \"IndexError: too many indices for array\" on line 3 while executing the code by @CPMP.", "votes": null } ], "comments": [ { "id": 117630, "author_name": "brandenkmurray", "author_url": "", "post_date": "04/29/2016 22:01:27", "content": "My understanding is that the ndcg@n and map@n eval functions built into xgboost only work when using "rank:pairwise" for your objective. If you want to use a different objective then you need to define a custom map@5 eval function for it to work.
\n\nFor reference, see this issue on xgboost's GitHub.
", "votes": null, "replies": [] }, { "id": 117705, "author_name": "anokas", "author_url": "", "post_date": "04/30/2016 13:01:40", "content": "[quote=Branden Murray;117630]
\n\nFor reference, see this issue on xgboost's GitHub.
\n\n[/quote]
\n\nThat's my issue ;). What I ended up doing is optimising for mlogloss instead, but if someone smarter than me can implement map@5 as a custom metric in xgboost that would be very useful!
\n\nI think xgboost might not be the way to go in this competition, as it takes so long to train (as it is 100-class problem, it trains 100 trees per boosting rounds) and this means it was taking me 12 hours on dual xeons to converge.
", "votes": null, "replies": [] }, { "id": 117749, "author_name": "brandenkmurray", "author_url": "", "post_date": "04/30/2016 17:20:28", "content": "In R,
\n\nlibrary(Metrics)\nmap5 <- function(preds, dtrain) {\n labels <- as.list(getinfo(dtrain,"label"))\n num.class = 100\n pred <- matrix(preds, nrow = num.class)\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\n top <- split(top, 1:NROW(top))\n\n map <- mapk(5, labels, top)\n return(list(metric = "map5", value = map))\n}\nThen set eval_metric=map5 in your params.
[edit] Realized the mapk function takes in lists, not vectors. This now gives the same output as SK's function below.
I believe in the classification case (and as long as you make sure to return 5 unique predictions per example) the metric degenerates to a simple:
\n\nmap5 = function(preds, dtrain) {\n labels = getinfo(dtrain, 'label')\n preds = t(matrix(preds, ncol = length(labels)))\n preds = t(apply(preds, 1, order, decreasing = T))[, 1:5] - 1\n succ = (preds == labels)\n w = 1 / (1:5)\n map5 = mean(succ %*% w)\n return (list(metric = 'map5', value = map5))\n}\nwhich can probably be optimized a bit more.
", "votes": null, "replies": [] }, { "id": 117781, "author_name": "anokas", "author_url": "", "post_date": "04/30/2016 21:31:58", "content": "[quote=Branden Murray;117749]
\n\nIn R,
\n\nlibrary(Metrics)\nmap5 <- function(preds, dtrain) {\n labels <- getinfo(dtrain,"label")\n num.class = 100\n pred <- matrix(preds, nrow = num.class)\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\n\n map <- mapk(5, labels, top)\n return(list(metric = "map5", value = map))\n}\nThen set eval_metric=map5 in your params.
[/quote]
\n\nThanks, I'm sure this will be useful for a lot of people. I'm a python guy myself, so I still need to find another way :)
", "votes": null, "replies": [] }, { "id": 117815, "author_name": "dvasyukova", "author_url": "", "post_date": "05/01/2016 03:15:33", "content": "@anokas, something like this should work here:
\n\ndef map5eval(preds, dtrain):\n actual = dtrain.get_label()\n predicted = preds.argsort(axis=1)[:,-np.arange(1,6)]\n metric = 0.\n for i in range(5):\n metric += np.sum(actual==predicted[:,i])/(i+1)\n metric /= actual.shape[0]\n return 'MAP@5', metric\nThen pass to xgb.train as feval = map5eval.
Edit: reverse the sign for use in early stopping.
\n\nEdit 2: fixed an error in array slicing. Thanks, @HN Musac.
", "votes": null, "replies": [] }, { "id": 118038, "author_name": "brandenkmurray", "author_url": "", "post_date": "05/02/2016 17:26:03", "content": "Has anybody tried using any of these with xgboost yet? What is the difference between your local CV score and your LB score? On a random subset of 100,000 training rows I got 3-fold CV scores of ~0.52 (s.d. = 0.002) using my map5 function, but LB score was ~0.42. Using SK's function on the same subset I was getting CV scores of ~0.58.
", "votes": null, "replies": [] }, { "id": 118103, "author_name": "hitoshinagano", "author_url": "", "post_date": "05/02/2016 21:22:41", "content": "[quote=dune_dweller;117815]
\n\ndef map5eval(preds, dtrain):\n actual = dtrain.get_label()\n predicted = preds.argsort(axis=1)[:,-np.arange(5)]\n metric = 0.\n for i in range(5):\n metric += np.sum(actual==predicted[:,i])/(i+1)\n metric /= actual.shape[0]\n return 'MAP@5', metric\n[/quote]
\n\nHello Dune, i am a bit confused about the format of preds. Does preds contain the probabilities or the classes labels?
\n\n-np.arange(5) gives [0, -1, -2, -3, -4], , which slices the first, the last, 2nd to last, 3rd to last and 4th to last columns, correct?
", "votes": null, "replies": [] }, { "id": 118249, "author_name": "dvasyukova", "author_url": "", "post_date": "05/03/2016 02:45:55", "content": "preds.argsort(axis=1) returns an array of indices into each row from smallest element to largest. Although you made me realize that I'm slicing it wrong: need to replace -np.arange(5) with -np.arange(1,6), so that it takes the last element, then the second-last and so on. This way the most probable hotel cluster goes first.
Lol, this just made my NN model go from 0.17 to 0.27 in cv. Maybe it's not as hopeless as I thought =). I'll edit the original post to include this correction, thanks!
", "votes": null, "replies": [] }, { "id": 118493, "author_name": "kevinmcisaac", "author_url": "", "post_date": "05/03/2016 23:41:51", "content": "Another option is to use MAPK from Ben Hammers ml_metrics . He has both R and Python implementations.
", "votes": null, "replies": [] }, { "id": 118578, "author_name": "mafux777", "author_url": "", "post_date": "05/04/2016 11:02:17", "content": "Is there a way to compare the score on the leaderboard with the score of this map5 function? \nI got something like .24 in cross-validation, which does not seem much compared to the "naive" method of just going for the most popular hotels per destination.\nHowever, when I submit the results I got .04 - apparently there is still a bug in how I interpret the result matrix from the predict function.
", "votes": null, "replies": [] }, { "id": 118583, "author_name": "qwang88", "author_url": "", "post_date": "05/04/2016 11:24:55", "content": "[quote=Branden Murray;118038]
\n\nHas anybody tried using any of these with xgboost yet? What is the difference between your local CV score and your LB score? On a random subset of 100,000 training rows I got 3-fold CV scores of ~0.52 (s.d. = 0.002) using my map5 function, but LB score was ~0.42. Using SK's function on the same subset I was getting CV scores of ~0.58.
\n\n[/quote]
\n\nDid you generate a time-based validation set? I found that with a small sample, ordering by time (which is what the LB uses) results in much worse performance than testing on a random subset. In a particular case I got validation score of 0.6 when test on a random subset of the train, and 0.37xx when the test set was generated by time.
", "votes": null, "replies": [] }, { "id": 118606, "author_name": "dvasyukova", "author_url": "", "post_date": "05/04/2016 13:37:39", "content": "I trained a neural net model on the full train.csv by reading it in chunks. First I used each chunk to evaluate performance and then to train the model. By the end of training local score was around 0.29-0.30. And leaderboard score was about 0.277. This wasn't a time-based split, but not a small sample either :).
", "votes": null, "replies": [] }, { "id": 118642, "author_name": "mafux777", "author_url": "", "post_date": "05/04/2016 15:26:02", "content": "Reply to self - fixed the submission issue, and difference between my CV score and LB score was .02, in other words very small. Alas, model is still not performing better than the "most popular".
\n\n[quote=mafux777;118578]
\n\nIs there a way to compare the score on the leaderboard with the score of this map5 function? \nI got something like .24 in cross-validation, which does not seem much compared to the "naive" method of just going for the most popular hotels per destination.\nHowever, when I submit the results I got .04 - apparently there is still a bug in how I interpret the result matrix from the predict function.
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 120733, "author_name": "abhinavs01858", "author_url": "", "post_date": "05/20/2016 06:19:59", "content": "[quote=Branden Murray;117749]
\n\nIn R,
\n\nlibrary(Metrics)\nmap5 <- function(preds, dtrain) {\n labels <- as.list(getinfo(dtrain,"label"))\n num.class = 100\n pred <- matrix(preds, nrow = num.class)\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\n top <- split(top, 1:NROW(top))\n\n map <- mapk(5, labels, top)\n return(list(metric = "map5", value = map))\n}\nThen set eval_metric=map5 in your params.
[edit] Realized the mapk function takes in lists, not vectors. This now gives the same output as SK's function below.
[/quote]
\n\nWhat does this code return? I am getting just a large numeric list, i thought multi:softprob was supposed to return a probability matrix for the classes?!
", "votes": null, "replies": [] }, { "id": 120734, "author_name": "brandenkmurray", "author_url": "", "post_date": "05/20/2016 06:26:38", "content": "That's just an eval_function for xgboost, nothing to do with your issue. For multi-class problems xgboost returns a vector and you need to transform that vector into a matrix. You can do this with code below:
\n\ntestPreds <- predict(xgb_model, dtest)\ntestPreds <- t(matrix(testPreds, nrow=100))\nThe columns will be the classes in order, from 0 to 99.
", "votes": null, "replies": [] }, { "id": 120735, "author_name": "abhinavs01858", "author_url": "", "post_date": "05/20/2016 06:42:02", "content": "[quote=Branden Murray;120734]
\n\nThat's just an eval_function for xgboost, nothing to do with your issue. For multi-class problems xgboost returns a vector and you need to transform that vector into a matrix. You can do this with code below:
\n\ntestPreds <- predict(xgb_model, dtest)\ntestPreds <- t(matrix(testPreds, nrow=100))\nThe columns will be the classes in order, from 0 to 99.
\n\n[/quote]
\n\nohh! Just figured\nThanks for clearing the doubt!! :)
", "votes": null, "replies": [] }, { "id": 120974, "author_name": "masterliu", "author_url": "", "post_date": "05/22/2016 09:49:49", "content": "do you get "xgboost.core.XGBoostError: label size predict size not match" error after using ml_metrics ? mapk5 function is i copied from that link and changed the name, and i run a cv validation, code is below, has anyone used that metric?:
\n\n\ndef apk(actual, predicted, k=5):\n if len(predicted)>k:\n predicted = predicted[:k]\n\n score = 0.0\n num_hits = 0.0\n\n for i,p in enumerate(predicted):\n if p in actual and p not in predicted[:i]:\n num_hits += 1.0\n score += num_hits / (i+1.0)\n\n if not actual:\n return 0.0\n\n return score / min(len(actual), k)\n\ndef mapk5(actual, predicted, k=5):\n return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])\n\nparams = {}\nparams["objective"] = "multi:softprob"\nparams["booster"] = "gbtree"\nparams["num_class"] = 100\nparams["eta"] = 0.01\nparams["subsample"] = 0.75\nparams["colsample_bytree"] = 0.75\nparams["max_depth"] = 6\nparams["min_child_weight"] = 3\nparams["silent"] = 1\nxgtrain = xgb.DMatrix(X, y)\nprint xgb.cv(params, xgtrain, 1000, nfold=5, metrics=['mapk5'], early_stopping_rounds=50)\nsorry for confusion, i've fixed that problem by passing feval= mapk5 instead of metrics param.
", "votes": null, "replies": [] }, { "id": 121034, "author_name": "masterliu", "author_url": "", "post_date": "05/23/2016 04:37:41", "content": "@dune_dweller , which map@5 cv score do you get? i used your eval function and only get about 0.2, looks a bit lower. Maybe there're problems in my code.
", "votes": null, "replies": [] }, { "id": 122237, "author_name": "cpmpml", "author_url": "", "post_date": "06/02/2016 08:41:46", "content": "@masterLiu, your code seems way too complicated to me.
\n\nA slightly simplified version of @dune_dweller map5 function runs quite fast:
\n\ndef map5eval(preds, dtrain, k=5):\n actual = dtrain.get_label()\n predicted = (-preds).argsort(axis=1)[:,:k]\n metric = 0.\n for i in range(5):\n metric += np.sum(actual==predicted[:,i])/(i+1)\n metric /= actual.shape[0]\n return 'MAP@5', metric\nAlso, do not forget to pass maximize=True as a parameter to train.
", "votes": null, "replies": [] }, { "id": 122558, "author_name": "beedata", "author_url": "", "post_date": "06/05/2016 04:00:32", "content": "@Branden Murray I face the same problem with you. I use @CPMP evaluation metric and I could get very high validation performance. Is it caused by leak information or there's something wrong with the evaluation metric? I used 100000 rows with 33% as validation set.
", "votes": null, "replies": [] }, { "id": 122568, "author_name": "brandenkmurray", "author_url": "", "post_date": "06/05/2016 07:46:53", "content": "@Feng, If I recall correctly, my problems were the result of a leak from the way I was doing feature engineering and splitting my data. I stopped using map5 as my eval metric in xgboost because xgboost seemed to run faster using 'mlogloss' (I think just because mlogloss is built in and therefore calculates much faster)
", "votes": null, "replies": [] }, { "id": 122607, "author_name": "beedata", "author_url": "", "post_date": "06/05/2016 18:36:35", "content": "@Branden Murray Thank you. But for mlogloss, as your experience. What's the score corresponding to like map@5=0.3 or 0.35 . And when you change to mlogloss, does your code speed up significantly? Thank you.
", "votes": null, "replies": [] }, { "id": 123000, "author_name": "dilipkomar", "author_url": "", "post_date": "06/09/2016 04:13:28", "content": "has anyone come across this error "IndexError: too many indices for array" on line 3 while executing the code by @CPMP.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "117628": "Hi, I've done ncdg5 before via a custom function in xgboost. I also see that there is an \"map\" and \"map@n\" eval_metric in xgboost, but simply it doesn't work for me. R crashes\r\n\r\nI don't know if these is crossing the competition rules, but is there a hint on these?\r\n\r\nThanks", "117630": "My understanding is that the ndcg@n and map@n eval functions built into xgboost only work when using \"rank:pairwise\" for your objective. If you want to use a different objective then you need to define a custom map@5 eval function for it to work.\r\n\r\nFor reference, see this [issue][1] on xgboost's GitHub.\r\n\r\n\r\n [1]: https://github.com/dmlc/xgboost/issues/1143", "117705": "[quote=Branden Murray;117630]\r\n\r\nFor reference, see this [issue][1] on xgboost's GitHub.\r\n\r\n\r\n [1]: https://github.com/dmlc/xgboost/issues/1143\r\n\r\n[/quote]\r\n\r\nThat's my issue ;). What I ended up doing is optimising for mlogloss instead, but if someone smarter than me can implement map@5 as a custom metric in xgboost that would be very useful!\r\n\r\nI think xgboost might not be the way to go in this competition, as it takes so long to train (as it is 100-class problem, it trains 100 trees per boosting rounds) and this means it was taking me 12 hours on dual xeons to converge.", "117749": "In R, \r\n\r\n library(Metrics)\r\n map5 <- function(preds, dtrain) {\r\n labels <- as.list(getinfo(dtrain,\"label\"))\r\n num.class = 100\r\n pred <- matrix(preds, nrow = num.class)\r\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\r\n top <- split(top, 1:NROW(top))\r\n \r\n map <- mapk(5, labels, top)\r\n return(list(metric = \"map5\", value = map))\r\n }\r\n\r\nThen set `eval_metric=map5` in your params.\r\n\r\n*[edit] Realized the `mapk` function takes in lists, not vectors. This now gives the same output as SK's function below.*", "117762": "I believe in the classification case (and as long as you make sure to return 5 unique predictions per example) the metric degenerates to a simple:\r\n\r\n map5 = function(preds, dtrain) {\r\n labels = getinfo(dtrain, 'label')\r\n preds = t(matrix(preds, ncol = length(labels)))\r\n preds = t(apply(preds, 1, order, decreasing = T))[, 1:5] - 1\r\n succ = (preds == labels)\r\n w = 1 / (1:5)\r\n map5 = mean(succ %*% w)\r\n return (list(metric = 'map5', value = map5))\r\n }\r\n\r\nwhich can probably be optimized a bit more.", "117781": "[quote=Branden Murray;117749]\r\n\r\nIn R, \r\n\r\n library(Metrics)\r\n map5 <- function(preds, dtrain) {\r\n labels <- getinfo(dtrain,\"label\")\r\n num.class = 100\r\n pred <- matrix(preds, nrow = num.class)\r\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\r\n \r\n map <- mapk(5, labels, top)\r\n return(list(metric = \"map5\", value = map))\r\n }\r\n\r\nThen set `eval_metric=map5` in your params.\r\n\r\n[/quote]\r\n\r\nThanks, I'm sure this will be useful for a lot of people. I'm a python guy myself, so I still need to find another way :)", "117815": "anokas, something like this should work here:\r\n\r\n def map5eval(preds, dtrain):\r\n actual = dtrain.get_label()\r\n predicted = preds.argsort(axis=1)[:,-np.arange(1,6)]\r\n metric = 0.\r\n for i in range(5):\r\n metric += np.sum(actual==predicted[:,i])/(i+1)\r\n metric /= actual.shape[0]\r\n return 'MAP@5', metric\r\nThen pass to `xgb.train` as `feval = map5eval`.\r\n\r\n*Edit: reverse the sign for use in early stopping.*\r\n\r\n*Edit 2: fixed an error in array slicing. Thanks, @HN Musac.*", "118038": "Has anybody tried using any of these with xgboost yet? What is the difference between your local CV score and your LB score? On a random subset of 100,000 training rows I got 3-fold CV scores of ~0.52 (s.d. = 0.002) using my map5 function, but LB score was ~0.42. Using SK's function on the same subset I was getting CV scores of ~0.58.", "118103": "[quote=dune_dweller;117815]\r\n\r\n def map5eval(preds, dtrain):\r\n actual = dtrain.get_label()\r\n predicted = preds.argsort(axis=1)[:,-np.arange(5)]\r\n metric = 0.\r\n for i in range(5):\r\n metric += np.sum(actual==predicted[:,i])/(i+1)\r\n metric /= actual.shape[0]\r\n return 'MAP@5', metric\r\n\r\n[/quote]\r\n\r\nHello Dune, i am a bit confused about the format of preds. Does preds contain the probabilities or the classes labels?\r\n\r\n-np.arange(5) gives [0, -1, -2, -3, -4], , which slices the first, the last, 2nd to last, 3rd to last and 4th to last columns, correct?", "118249": "`preds.`[`argsort`](http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.argsort.html)`(axis=1)` returns an array of indices into each row from smallest element to largest. Although you made me realize that I'm slicing it wrong: need to replace `-np.arange(5)` with `-np.arange(1,6)`, so that it takes the last element, then the second-last and so on. This way the most probable hotel cluster goes first.\r\n\r\nLol, this just made my NN model go from 0.17 to 0.27 in cv. Maybe it's not as hopeless as I thought =). I'll edit the original post to include this correction, thanks!", "118493": "Another option is to use MAPK from Ben Hammers [ml_metrics](https://github.com/benhamner/Metrics) . He has both R and Python implementations.", "118578": "Is there a way to compare the score on the leaderboard with the score of this map5 function? \r\nI got something like .24 in cross-validation, which does not seem much compared to the \"naive\" method of just going for the most popular hotels per destination.\r\nHowever, when I submit the results I got .04 - apparently there is still a bug in how I interpret the result matrix from the predict function.", "118583": "[quote=Branden Murray;118038]\r\n\r\nHas anybody tried using any of these with xgboost yet? What is the difference between your local CV score and your LB score? On a random subset of 100,000 training rows I got 3-fold CV scores of ~0.52 (s.d. = 0.002) using my map5 function, but LB score was ~0.42. Using SK's function on the same subset I was getting CV scores of ~0.58.\r\n\r\n[/quote]\r\n\r\nDid you generate a time-based validation set? I found that with a small sample, ordering by time (which is what the LB uses) results in much worse performance than testing on a random subset. In a particular case I got validation score of 0.6 when test on a random subset of the train, and 0.37xx when the test set was generated by time.", "118606": "I trained a neural net model on the full train.csv by reading it in chunks. First I used each chunk to evaluate performance and then to train the model. By the end of training local score was around 0.29-0.30. And leaderboard score was about 0.277. This wasn't a time-based split, but not a small sample either :).", "118642": "Reply to self - fixed the submission issue, and difference between my CV score and LB score was .02, in other words very small. Alas, model is still not performing better than the \"most popular\".\r\n\r\n[quote=mafux777;118578]\r\n\r\nIs there a way to compare the score on the leaderboard with the score of this map5 function? \r\nI got something like .24 in cross-validation, which does not seem much compared to the \"naive\" method of just going for the most popular hotels per destination.\r\nHowever, when I submit the results I got .04 - apparently there is still a bug in how I interpret the result matrix from the predict function. \r\n\r\n[/quote]", "120733": "[quote=Branden Murray;117749]\r\n\r\nIn R, \r\n\r\n library(Metrics)\r\n map5 <- function(preds, dtrain) {\r\n labels <- as.list(getinfo(dtrain,\"label\"))\r\n num.class = 100\r\n pred <- matrix(preds, nrow = num.class)\r\n top <- t(apply(pred, 2, function(y) order(y)[num.class:(num.class-4)]-1))\r\n top <- split(top, 1:NROW(top))\r\n \r\n map <- mapk(5, labels, top)\r\n return(list(metric = \"map5\", value = map))\r\n }\r\n\r\nThen set `eval_metric=map5` in your params.\r\n\r\n*[edit] Realized the `mapk` function takes in lists, not vectors. This now gives the same output as SK's function below.*\r\n\r\n[/quote]\r\n\r\nWhat does this code return? I am getting just a large numeric list, i thought multi:softprob was supposed to return a probability matrix for the classes?!", "120734": "That's just an eval_function for xgboost, nothing to do with your issue. For multi-class problems xgboost returns a vector and you need to transform that vector into a matrix. You can do this with code below:\r\n\r\n testPreds <- predict(xgb_model, dtest)\r\n testPreds <- t(matrix(testPreds, nrow=100))\r\n\r\nThe columns will be the classes in order, from 0 to 99.", "120735": "[quote=Branden Murray;120734]\r\n\r\nThat's just an eval_function for xgboost, nothing to do with your issue. For multi-class problems xgboost returns a vector and you need to transform that vector into a matrix. You can do this with code below:\r\n\r\n testPreds <- predict(xgb_model, dtest)\r\n testPreds <- t(matrix(testPreds, nrow=100))\r\n\r\nThe columns will be the classes in order, from 0 to 99.\r\n\r\n[/quote]\r\n\r\nohh! Just figured\r\nThanks for clearing the doubt!! :)", "120974": "do you get \"xgboost.core.XGBoostError: label size predict size not match\" error after using [ml_metrics][1] ? mapk5 function is i copied from that link and changed the name, and i run a cv validation, code is below, has anyone used that metric?:\r\n\r\ndef apk(actual, predicted, k=5):\r\n if len(predicted)>k:\r\n predicted = predicted[:k]\r\n\r\n score = 0.0\r\n num_hits = 0.0\r\n\r\n for i,p in enumerate(predicted):\r\n if p in actual and p not in predicted[:i]:\r\n num_hits += 1.0\r\n score += num_hits / (i+1.0)\r\n\r\n if not actual:\r\n return 0.0\r\n\r\n return score / min(len(actual), k)\r\n\r\ndef mapk5(actual, predicted, k=5):\r\n return np.mean([apk(a,p,k) for a,p in zip(actual, predicted)])\r\n\r\nparams = {}\r\nparams[\"objective\"] = \"multi:softprob\"\r\nparams[\"booster\"] = \"gbtree\"\r\nparams[\"num_class\"] = 100\r\nparams[\"eta\"] = 0.01\r\nparams[\"subsample\"] = 0.75\r\nparams[\"colsample_bytree\"] = 0.75\r\nparams[\"max_depth\"] = 6\r\nparams[\"min_child_weight\"] = 3\r\nparams[\"silent\"] = 1\r\nxgtrain = xgb.DMatrix(X, y)\r\nprint xgb.cv(params, xgtrain, 1000, nfold=5, metrics=['mapk5'], early_stopping_rounds=50)\r\n