{
  "id": 52338,
  "title": "ROC function to find your model score",
  "url": "/competitions/talkingdata-adtracking-fraud-detection/discussion/52338",
  "author_name": "",
  "post_date": "2018-03-19T08:59:52.583687900Z",
  "votes": 6,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi everyone, \nTo test my model on the competition score <strong>I made 2 functions</strong> (depending on the using model) to find out what the model's score.\nThe first one auc() is if your model is using <strong>predict_proba</strong> as predict function and the second auc2() just the <strong>predict</strong> function.</p>\n\n<p>Functions parameters :</p>\n\n<ul>\n<li>m : your model</li>\n<li>x_train : your training features</li>\n<li>x_test : your testing features</li>\n<li>y_train : your training labels</li>\n<li>y_test : your testing labels</li>\n</ul>\n\n<p><code>\n    from sklearn import metrics</code></p>\n\n<pre><code>def auc(m, x_train, x_test, y_train, y_test):\n    return (metrics.roc_auc_score(y_train,m.predict_proba(x_train)[:,1]),\n                            metrics.roc_auc_score(y_test,m.predict_proba(x_test)[:,1]))\n\ndef auc2(m, x_train, x_test, y_train, y_test):\n    return (metrics.roc_auc_score(y_train,m.predict(x_train)),\n                            metrics.roc_auc_score(y_test,m.predict(x_test)))\n</code></pre>\n\n<p>I hope this will help you ! 👍</p>",
  "messages": [
    {
      "id": "298281",
      "postDate": "03/19/2018 08:59:52",
      "content": "<p>Hi everyone, \nTo test my model on the competition score <strong>I made 2 functions</strong> (depending on the using model) to find out what the model's score.\nThe first one auc() is if your model is using <strong>predict_proba</strong> as predict function and the second auc2() just the <strong>predict</strong> function.</p>\n\n<p>Functions parameters :</p>\n\n<ul>\n<li>m : your model</li>\n<li>x_train : your training features</li>\n<li>x_test : your testing features</li>\n<li>y_train : your training labels</li>\n<li>y_test : your testing labels</li>\n</ul>\n\n<p><code>\n    from sklearn import metrics</code></p>\n\n<pre><code>def auc(m, x_train, x_test, y_train, y_test):\n    return (metrics.roc_auc_score(y_train,m.predict_proba(x_train)[:,1]),\n                            metrics.roc_auc_score(y_test,m.predict_proba(x_test)[:,1]))\n\ndef auc2(m, x_train, x_test, y_train, y_test):\n    return (metrics.roc_auc_score(y_train,m.predict(x_train)),\n                            metrics.roc_auc_score(y_test,m.predict(x_test)))\n</code></pre>\n\n<p>I hope this will help you ! 👍</p>",
      "rawMarkdown": "Hi everyone, \nTo test my model on the competition score **I made 2 functions** (depending on the using model) to find out what the model's score.\nThe first one auc() is if your model is using **predict_proba** as predict function and the second auc2() just the **predict** function.\n\nFunctions parameters :\n\n - m : your model\n - x_train : your training features\n - x_test : your testing features\n - y_train : your training labels\n - y_test : your testing labels\n\n\n\n```\n    from sklearn import metrics```\n    \n    def auc(m, x_train, x_test, y_train, y_test):\n        return (metrics.roc_auc_score(y_train,m.predict_proba(x_train)[:,1]),\n                                metrics.roc_auc_score(y_test,m.predict_proba(x_test)[:,1]))\n    \n    def auc2(m, x_train, x_test, y_train, y_test):\n        return (metrics.roc_auc_score(y_train,m.predict(x_train)),\n                                metrics.roc_auc_score(y_test,m.predict(x_test)))\n\n\nI hope this will help you ! 👍",
      "votes": null
    },
    {
      "id": "301749",
      "postDate": "03/23/2018 07:29:50",
      "content": "<p>Repost it as a kernel so you enter the Kaggle tshirt competition.</p>",
      "rawMarkdown": "Repost it as a kernel so you enter the Kaggle tshirt competition.",
      "votes": null
    },
    {
      "id": "302111",
      "postDate": "03/23/2018 17:45:28",
      "content": "<p>Great idea I will do that soon ! </p>",
      "rawMarkdown": "Great idea I will do that soon !",
      "votes": null
    },
    {
      "id": "302259",
      "postDate": "03/23/2018 21:07:55",
      "content": "<p>Here's the R version to calculate and plot auc on validation data for LightGBM model:</p>\n\n<blockquote>\n  <p><strong>valid</strong> : validation data, \n  <strong>model</strong> : trained model, \n  <strong>package required</strong> : pROC</p>\n</blockquote>\n\n<pre><code>```{r fig.width= 5}\n\n# following function extracts validation score from model\nmax(unlist(model$record_evals[[\"validation\"]][[\"auc\"]][[\"eval\"]]))\n\n# Plot ROC\nval_preds = predict(model, \n                    data = as.matrix(valid[, colnames(valid) != \"is_attributed\"]), \n                    n = model$best_iter)\n\nauc.lgb = roc(valid$is_attributed, \n              val_preds, \n              levels=base::levels(as.factor(valid$is_attributed)), grid=TRUE,\n              plot = TRUE, col = \"steelblue\", lwd = 3)\n\nauc.lgb\n```\n</code></pre>",
      "rawMarkdown": "Here's the R version to calculate and plot auc on validation data for LightGBM model:\n\n&gt; **valid** : validation data, \n&gt; **model** : trained model, \n&gt; **package required** : pROC\n\n\n    ```{r fig.width= 5}\n\n    # following function extracts validation score from model\n    max(unlist(model$record_evals[[\"validation\"]][[\"auc\"]][[\"eval\"]]))\n\n    # Plot ROC\n    val_preds = predict(model, \n                        data = as.matrix(valid[, colnames(valid) != \"is_attributed\"]), \n                        n = model$best_iter)\n\n    auc.lgb = roc(valid$is_attributed, \n                  val_preds, \n                  levels=base::levels(as.factor(valid$is_attributed)), grid=TRUE,\n                  plot = TRUE, col = \"steelblue\", lwd = 3)\n\n    auc.lgb\n    ```",
      "votes": null
    },
    {
      "id": "302273",
      "postDate": "03/23/2018 21:13:00",
      "content": "<p>Thanks for sharing this !</p>",
      "rawMarkdown": "Thanks for sharing this !",
      "votes": null
    },
    {
      "id": "302275",
      "postDate": "03/23/2018 21:16:06",
      "content": "<p>My pleasure!</p>",
      "rawMarkdown": "My pleasure!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 301749,
      "author_name": "smcinerney",
      "author_url": "",
      "post_date": "03/23/2018 07:29:50",
      "content": "<p>Repost it as a kernel so you enter the Kaggle tshirt competition.</p>",
      "votes": null,
      "replies": [
        {
          "id": 302111,
          "author_name": "nathanlauga",
          "author_url": "",
          "post_date": "03/23/2018 17:45:28",
          "content": "<p>Great idea I will do that soon ! </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 302259,
      "author_name": "pranav84",
      "author_url": "",
      "post_date": "03/23/2018 21:07:55",
      "content": "<p>Here's the R version to calculate and plot auc on validation data for LightGBM model:</p>\n\n<blockquote>\n  <p><strong>valid</strong> : validation data, \n  <strong>model</strong> : trained model, \n  <strong>package required</strong> : pROC</p>\n</blockquote>\n\n<pre><code>```{r fig.width= 5}\n\n# following function extracts validation score from model\nmax(unlist(model$record_evals[[\"validation\"]][[\"auc\"]][[\"eval\"]]))\n\n# Plot ROC\nval_preds = predict(model, \n                    data = as.matrix(valid[, colnames(valid) != \"is_attributed\"]), \n                    n = model$best_iter)\n\nauc.lgb = roc(valid$is_attributed, \n              val_preds, \n              levels=base::levels(as.factor(valid$is_attributed)), grid=TRUE,\n              plot = TRUE, col = \"steelblue\", lwd = 3)\n\nauc.lgb\n```\n</code></pre>",
      "votes": null,
      "replies": [
        {
          "id": 302273,
          "author_name": "nathanlauga",
          "author_url": "",
          "post_date": "03/23/2018 21:13:00",
          "content": "<p>Thanks for sharing this !</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 302275,
          "author_name": "pranav84",
          "author_url": "",
          "post_date": "03/23/2018 21:16:06",
          "content": "<p>My pleasure!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "298281": "Hi everyone, \nTo test my model on the competition score **I made 2 functions** (depending on the using model) to find out what the model's score.\nThe first one auc() is if your model is using **predict_proba** as predict function and the second auc2() just the **predict** function.\n\nFunctions parameters :\n\n - m : your model\n - x_train : your training features\n - x_test : your testing features\n - y_train : your training labels\n - y_test : your testing labels\n\n\n\n```\n    from sklearn import metrics```\n    \n    def auc(m, x_train, x_test, y_train, y_test):\n        return (metrics.roc_auc_score(y_train,m.predict_proba(x_train)[:,1]),\n                                metrics.roc_auc_score(y_test,m.predict_proba(x_test)[:,1]))\n    \n    def auc2(m, x_train, x_test, y_train, y_test):\n        return (metrics.roc_auc_score(y_train,m.predict(x_train)),\n                                metrics.roc_auc_score(y_test,m.predict(x_test)))\n\n\nI hope this will help you ! 👍",
    "301749": "Repost it as a kernel so you enter the Kaggle tshirt competition.",
    "302111": "Great idea I will do that soon !",
    "302259": "Here's the R version to calculate and plot auc on validation data for LightGBM model:\n\n&gt; **valid** : validation data, \n&gt; **model** : trained model, \n&gt; **package required** : pROC\n\n\n    ```{r fig.width= 5}\n\n    # following function extracts validation score from model\n    max(unlist(model$record_evals[[\"validation\"]][[\"auc\"]][[\"eval\"]]))\n\n    # Plot ROC\n    val_preds = predict(model, \n                        data = as.matrix(valid[, colnames(valid) != \"is_attributed\"]), \n                        n = model$best_iter)\n\n    auc.lgb = roc(valid$is_attributed, \n                  val_preds, \n                  levels=base::levels(as.factor(valid$is_attributed)), grid=TRUE,\n                  plot = TRUE, col = \"steelblue\", lwd = 3)\n\n    auc.lgb\n    ```",
    "302273": "Thanks for sharing this !",
    "302275": "My pleasure!"
  },
  "source": "meta"
}