{
  "id": 202124,
  "title": "Do you guys get a better AUC on validation set than training set",
  "url": "/competitions/riiid-test-answer-prediction/discussion/202124",
  "author_name": "",
  "post_date": "2020-12-08T13:55:34.017055800Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p>I get better AUC on the training set than the validation set, even though the training log loss is lower.  Maybe, it's because the training set (90,000,000) has much more data than the validation set (2500,000). </p>\n<pre><code>Training until validation scores don't improve for 50 rounds\n[50]    training's auc: 0.751713    training's binary_logloss: 0.574392 valid_1's auc: 0.758776 valid_1's binary_logloss: 0.578673\n[100]    training's auc: 0.75325 training's binary_logloss: 0.554109 valid_1's auc: 0.760287 valid_1's binary_logloss: 0.557157\n</code></pre>",
  "messages": [
    {
      "id": "1106085",
      "postDate": "12/08/2020 13:55:34",
      "content": "<p>I get better AUC on the training set than the validation set, even though the training log loss is lower.  Maybe, it's because the training set (90,000,000) has much more data than the validation set (2500,000). </p>\n<pre><code>Training until validation scores don't improve for 50 rounds\n[50]    training's auc: 0.751713    training's binary_logloss: 0.574392 valid_1's auc: 0.758776 valid_1's binary_logloss: 0.578673\n[100]    training's auc: 0.75325 training's binary_logloss: 0.554109 valid_1's auc: 0.760287 valid_1's binary_logloss: 0.557157\n</code></pre>",
      "rawMarkdown": "I get better AUC on the training set than the validation set, even though the training log loss is lower.  Maybe, it's because the training set (90,000,000) has much more data than the validation set (2500,000). \n```\nTraining until validation scores don't improve for 50 rounds\n[50]\ttraining's auc: 0.751713\ttraining's binary_logloss: 0.574392\tvalid_1's auc: 0.758776\tvalid_1's binary_logloss: 0.578673\n[100]\ttraining's auc: 0.75325\ttraining's binary_logloss: 0.554109\tvalid_1's auc: 0.760287\tvalid_1's binary_logloss: 0.557157\n```",
      "votes": null
    },
    {
      "id": "1108428",
      "postDate": "12/10/2020 16:26:25",
      "content": "<p>This happens to me all the time … i am getting validation acc almost .82 but in leader board .74, <br>\nI think some kind of features is doing data leakage (aggregations made on target)</p>",
      "rawMarkdown": "This happens to me all the time ... i am getting validation acc almost .82 but in leader board .74, \nI think some kind of features is doing data leakage (aggregations made on target)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1108428,
      "author_name": "ptrikp",
      "author_url": "",
      "post_date": "12/10/2020 16:26:25",
      "content": "<p>This happens to me all the time … i am getting validation acc almost .82 but in leader board .74, <br>\nI think some kind of features is doing data leakage (aggregations made on target)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1106085": "I get better AUC on the training set than the validation set, even though the training log loss is lower.  Maybe, it's because the training set (90,000,000) has much more data than the validation set (2500,000). \n```\nTraining until validation scores don't improve for 50 rounds\n[50]\ttraining's auc: 0.751713\ttraining's binary_logloss: 0.574392\tvalid_1's auc: 0.758776\tvalid_1's binary_logloss: 0.578673\n[100]\ttraining's auc: 0.75325\ttraining's binary_logloss: 0.554109\tvalid_1's auc: 0.760287\tvalid_1's binary_logloss: 0.557157\n```",
    "1108428": "This happens to me all the time ... i am getting validation acc almost .82 but in leader board .74, \nI think some kind of features is doing data leakage (aggregations made on target)"
  },
  "source": "meta"
}