{
  "id": 21154,
  "title": "how to use custom eval function in GridSearchCV?",
  "url": "/competitions/expedia-hotel-recommendations/discussion/21154",
  "author_name": "",
  "post_date": "2016-05-23T12:59:06.693Z",
  "votes": null,
  "comment_count": 4,
  "views": 1313,
  "content": "<p>i tried to set scoring param of GridSearchCV with my custom map@5 eval function, it give s me error: TypeError: map5eval() takes exactly 2 arguments (3 given)</p>\n\n<p>I see GridSearchCV score definition is score = scorer(estimator, X_test, y_test), i think you should use GridSearchCV to choose params, how do you do that?</p>",
  "messages": [
    {
      "id": "121066",
      "postDate": "05/23/2016 12:59:06",
      "content": "<p>i tried to set scoring param of GridSearchCV with my custom map@5 eval function, it give s me error: TypeError: map5eval() takes exactly 2 arguments (3 given)</p>\n\n<p>I see GridSearchCV score definition is score = scorer(estimator, X_test, y_test), i think you should use GridSearchCV to choose params, how do you do that?</p>",
      "rawMarkdown": "i tried to set scoring param of GridSearchCV with my custom map@5 eval function, it give s me error: TypeError: map5eval() takes exactly 2 arguments (3 given)\r\n\r\nI see GridSearchCV score definition is score = scorer(estimator, X_test, y_test), i think you should use GridSearchCV to choose params, how do you do that?",
      "votes": null
    },
    {
      "id": "121394",
      "postDate": "05/26/2016 02:53:07",
      "content": "<p>What method did you use?  Xgboost?</p>",
      "rawMarkdown": "What method did you use?  Xgboost?",
      "votes": null
    },
    {
      "id": "121410",
      "postDate": "05/26/2016 06:25:16",
      "content": "<p>sklearn RandomForestClassifier</p>",
      "rawMarkdown": "sklearn RandomForestClassifier",
      "votes": null
    },
    {
      "id": "121654",
      "postDate": "05/28/2016 13:49:27",
      "content": "<p>You should use <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer\">make_scorer</a> function to create scorer from scoring function.</p>\n\n<p>For example if you have <code>map5</code> function, which calculates your score:</p>\n\n<pre><code>def map5(y_true, y_probs):\n    ...\n</code></pre>\n\n<p>Then you can create scorer from it:</p>\n\n<pre><code>from sklearn.metrics import make_scorer\n\nmap5_scorer = make_scorer(map5, needs_proba=True)\n</code></pre>\n\n<p>And pass it to <code>cross_val_score</code> or <code>GridSearchCV</code>:</p>\n\n<pre><code>cross_val_score(pipeline, X, y, scoring=map5_scorer)\n</code></pre>",
      "rawMarkdown": "You should use [make_scorer][1] function to create scorer from scoring function.\r\n\r\nFor example if you have `map5` function, which calculates your score:\r\n    \r\n    def map5(y_true, y_probs):\r\n        ...\r\n\r\nThen you can create scorer from it:\r\n\r\n    from sklearn.metrics import make_scorer\r\n\r\n    map5_scorer = make_scorer(map5, needs_proba=True)\r\n\r\nAnd pass it to `cross_val_score` or `GridSearchCV`:\r\n\r\n    cross_val_score(pipeline, X, y, scoring=map5_scorer)\r\n\r\n\r\n  [1]: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer",
      "votes": null
    },
    {
      "id": "121657",
      "postDate": "05/28/2016 14:19:04",
      "content": "<p>@alno, thanks, i'll have a try.</p>",
      "rawMarkdown": "alno, thanks, i'll have a try.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 121394,
      "author_name": "beedata",
      "author_url": "",
      "post_date": "05/26/2016 02:53:07",
      "content": "<p>What method did you use?  Xgboost?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121410,
      "author_name": "masterliu",
      "author_url": "",
      "post_date": "05/26/2016 06:25:16",
      "content": "<p>sklearn RandomForestClassifier</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121654,
      "author_name": "alexeynoskov",
      "author_url": "",
      "post_date": "05/28/2016 13:49:27",
      "content": "<p>You should use <a href=\"http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer\">make_scorer</a> function to create scorer from scoring function.</p>\n\n<p>For example if you have <code>map5</code> function, which calculates your score:</p>\n\n<pre><code>def map5(y_true, y_probs):\n    ...\n</code></pre>\n\n<p>Then you can create scorer from it:</p>\n\n<pre><code>from sklearn.metrics import make_scorer\n\nmap5_scorer = make_scorer(map5, needs_proba=True)\n</code></pre>\n\n<p>And pass it to <code>cross_val_score</code> or <code>GridSearchCV</code>:</p>\n\n<pre><code>cross_val_score(pipeline, X, y, scoring=map5_scorer)\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 121657,
      "author_name": "masterliu",
      "author_url": "",
      "post_date": "05/28/2016 14:19:04",
      "content": "<p>@alno, thanks, i'll have a try.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "121066": "i tried to set scoring param of GridSearchCV with my custom map@5 eval function, it give s me error: TypeError: map5eval() takes exactly 2 arguments (3 given)\r\n\r\nI see GridSearchCV score definition is score = scorer(estimator, X_test, y_test), i think you should use GridSearchCV to choose params, how do you do that?",
    "121394": "What method did you use?  Xgboost?",
    "121410": "sklearn RandomForestClassifier",
    "121654": "You should use [make_scorer][1] function to create scorer from scoring function.\r\n\r\nFor example if you have `map5` function, which calculates your score:\r\n    \r\n    def map5(y_true, y_probs):\r\n        ...\r\n\r\nThen you can create scorer from it:\r\n\r\n    from sklearn.metrics import make_scorer\r\n\r\n    map5_scorer = make_scorer(map5, needs_proba=True)\r\n\r\nAnd pass it to `cross_val_score` or `GridSearchCV`:\r\n\r\n    cross_val_score(pipeline, X, y, scoring=map5_scorer)\r\n\r\n\r\n  [1]: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html#sklearn.metrics.make_scorer",
    "121657": "alno, thanks, i'll have a try."
  },
  "source": "meta"
}