{
  "id": 77323,
  "title": "test set ratios differ from training set",
  "url": "/competitions/quora-insincere-questions-classification/discussion/77323",
  "author_name": "",
  "post_date": "2019-01-11T14:36:09.914173300Z",
  "votes": 3,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I was curious as to why I just couldn't replicate the LB results locally, so did a quick experiment to sanity check that the distributions of target values were the same between training and test - it turns out they're actually slightly different with the test set having a slightly lower positive value ratio</p>\n\n<p>It's hard to be exact since the LB score is to only 3dp, but it looks like the ratio of positive results in the training set is 0.06187, and around 0.059 for the test set</p>\n\n<p>I have no idea if there are similar differences between the private and public test sets</p>\n\n<p>I guess it's unrealistic for these to have exactly the same distributions, but it's worth mentioning none the less</p>\n\n<p><a href=\"https://www.kaggle.com/hamishdickson/submission-distributions\">https://www.kaggle.com/hamishdickson/submission-distributions</a></p>\n\n<p><strong>edit</strong>: I have no idea if someone else has pointed this out, apologies if so</p>",
  "messages": [
    {
      "id": "454369",
      "postDate": "01/11/2019 14:36:09",
      "content": "<p>I was curious as to why I just couldn't replicate the LB results locally, so did a quick experiment to sanity check that the distributions of target values were the same between training and test - it turns out they're actually slightly different with the test set having a slightly lower positive value ratio</p>\n\n<p>It's hard to be exact since the LB score is to only 3dp, but it looks like the ratio of positive results in the training set is 0.06187, and around 0.059 for the test set</p>\n\n<p>I have no idea if there are similar differences between the private and public test sets</p>\n\n<p>I guess it's unrealistic for these to have exactly the same distributions, but it's worth mentioning none the less</p>\n\n<p><a href=\"https://www.kaggle.com/hamishdickson/submission-distributions\">https://www.kaggle.com/hamishdickson/submission-distributions</a></p>\n\n<p><strong>edit</strong>: I have no idea if someone else has pointed this out, apologies if so</p>",
      "rawMarkdown": "I was curious as to why I just couldn't replicate the LB results locally, so did a quick experiment to sanity check that the distributions of target values were the same between training and test - it turns out they're actually slightly different with the test set having a slightly lower positive value ratio\n\nIt's hard to be exact since the LB score is to only 3dp, but it looks like the ratio of positive results in the training set is 0.06187, and around 0.059 for the test set\n\nI have no idea if there are similar differences between the private and public test sets\n\nI guess it's unrealistic for these to have exactly the same distributions, but it's worth mentioning none the less\n\nhttps://www.kaggle.com/hamishdickson/submission-distributions\n\n**edit**: I have no idea if someone else has pointed this out, apologies if so",
      "votes": null
    },
    {
      "id": "454436",
      "postDate": "01/11/2019 16:28:46",
      "content": "<p>Very nice way to check the mean target in test set. Anyway the difference between 0.06187 in train and 0.05988 in public part of test set looks statistically insignificant.\nAlso, if your score of \"all 1s\" was 0.11349 which was then rounded to 0.113 then we have mean test target:</p>\n\n<blockquote>\n  <p>0.11349/(2-0.11349)\n  0.06015870575825201\n  which differs less than 3% from train one.</p>\n</blockquote>",
      "rawMarkdown": "Very nice way to check the mean target in test set. Anyway the difference between 0.06187 in train and 0.05988 in public part of test set looks statistically insignificant.\nAlso, if your score of \"all 1s\" was 0.11349 which was then rounded to 0.113 then we have mean test target:\n&gt; 0.11349/(2-0.11349)\n0.06015870575825201\nwhich differs less than 3% from train one.",
      "votes": null
    },
    {
      "id": "454474",
      "postDate": "01/11/2019 17:32:24",
      "content": "<p>oh yeah I totally agree, it's well within the bounds of a reasonable error - certainly it's not going to put me on top of the leaderboard ;)</p>",
      "rawMarkdown": "oh yeah I totally agree, it's well within the bounds of a reasonable error - certainly it's not going to put me on top of the leaderboard ;)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 454436,
      "author_name": "akuropatwinski",
      "author_url": "",
      "post_date": "01/11/2019 16:28:46",
      "content": "<p>Very nice way to check the mean target in test set. Anyway the difference between 0.06187 in train and 0.05988 in public part of test set looks statistically insignificant.\nAlso, if your score of \"all 1s\" was 0.11349 which was then rounded to 0.113 then we have mean test target:</p>\n\n<blockquote>\n  <p>0.11349/(2-0.11349)\n  0.06015870575825201\n  which differs less than 3% from train one.</p>\n</blockquote>",
      "votes": null,
      "replies": [
        {
          "id": 454474,
          "author_name": "hamishdickson",
          "author_url": "",
          "post_date": "01/11/2019 17:32:24",
          "content": "<p>oh yeah I totally agree, it's well within the bounds of a reasonable error - certainly it's not going to put me on top of the leaderboard ;)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "454369": "I was curious as to why I just couldn't replicate the LB results locally, so did a quick experiment to sanity check that the distributions of target values were the same between training and test - it turns out they're actually slightly different with the test set having a slightly lower positive value ratio\n\nIt's hard to be exact since the LB score is to only 3dp, but it looks like the ratio of positive results in the training set is 0.06187, and around 0.059 for the test set\n\nI have no idea if there are similar differences between the private and public test sets\n\nI guess it's unrealistic for these to have exactly the same distributions, but it's worth mentioning none the less\n\nhttps://www.kaggle.com/hamishdickson/submission-distributions\n\n**edit**: I have no idea if someone else has pointed this out, apologies if so",
    "454436": "Very nice way to check the mean target in test set. Anyway the difference between 0.06187 in train and 0.05988 in public part of test set looks statistically insignificant.\nAlso, if your score of \"all 1s\" was 0.11349 which was then rounded to 0.113 then we have mean test target:\n&gt; 0.11349/(2-0.11349)\n0.06015870575825201\nwhich differs less than 3% from train one.",
    "454474": "oh yeah I totally agree, it's well within the bounds of a reasonable error - certainly it's not going to put me on top of the leaderboard ;)"
  },
  "source": "meta"
}