{
  "id": 198574,
  "title": "Very strange behaviour regarding train/test split",
  "url": "/competitions/riiid-test-answer-prediction/discussion/198574",
  "author_name": "",
  "post_date": "2020-11-21T21:58:56.261289800Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi everyone, I've noticed something very weird in my opinion : I'm sklearn using train_test_split to separate the competition datas, but the ratio behaves in a very strange way. It has to be 30% of the all dataset for testing. That's when I get the best results. But it's not just that. When I try 20% or 40%, my submission score ends up under 0.6, wheras with 30% I'm at 0.751.<br>\nDoes somebody find this weird too ? Have you also noticed this strange behaviour ?</p>",
  "messages": [
    {
      "id": "1086637",
      "postDate": "11/21/2020 21:58:56",
      "content": "<p>Hi everyone, I've noticed something very weird in my opinion : I'm sklearn using train_test_split to separate the competition datas, but the ratio behaves in a very strange way. It has to be 30% of the all dataset for testing. That's when I get the best results. But it's not just that. When I try 20% or 40%, my submission score ends up under 0.6, wheras with 30% I'm at 0.751.<br>\nDoes somebody find this weird too ? Have you also noticed this strange behaviour ?</p>",
      "rawMarkdown": "Hi everyone, I've noticed something very weird in my opinion : I'm sklearn using train_test_split to separate the competition datas, but the ratio behaves in a very strange way. It has to be 30% of the all dataset for testing. That's when I get the best results. But it's not just that. When I try 20% or 40%, my submission score ends up under 0.6, wheras with 30% I'm at 0.751.\nDoes somebody find this weird too ? Have you also noticed this strange behaviour ?",
      "votes": null
    },
    {
      "id": "1087009",
      "postDate": "11/22/2020 09:17:30",
      "content": "<p>If you only take such a small percentage for validation, there can be a lot of variation. I'd suggest using an even higher percentage (25% is the default but you can also choose bit lower too). Result should be more reliable then.</p>",
      "rawMarkdown": "If you only take such a small percentage for validation, there can be a lot of variation. I'd suggest using an even higher percentage (25% is the default but you can also choose bit lower too). Result should be more reliable then.",
      "votes": null
    },
    {
      "id": "1087108",
      "postDate": "11/22/2020 10:58:36",
      "content": "<p>Sorry, not 0.3% but 30% or 0.3/1, I've corrected it in my post, my bad.</p>",
      "rawMarkdown": "Sorry, not 0.3% but 30% or 0.3/1, I've corrected it in my post, my bad.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1087009,
      "author_name": "rafiko1",
      "author_url": "",
      "post_date": "11/22/2020 09:17:30",
      "content": "<p>If you only take such a small percentage for validation, there can be a lot of variation. I'd suggest using an even higher percentage (25% is the default but you can also choose bit lower too). Result should be more reliable then.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1087108,
          "author_name": "josephamigo",
          "author_url": "",
          "post_date": "11/22/2020 10:58:36",
          "content": "<p>Sorry, not 0.3% but 30% or 0.3/1, I've corrected it in my post, my bad.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1086637": "Hi everyone, I've noticed something very weird in my opinion : I'm sklearn using train_test_split to separate the competition datas, but the ratio behaves in a very strange way. It has to be 30% of the all dataset for testing. That's when I get the best results. But it's not just that. When I try 20% or 40%, my submission score ends up under 0.6, wheras with 30% I'm at 0.751.\nDoes somebody find this weird too ? Have you also noticed this strange behaviour ?",
    "1087009": "If you only take such a small percentage for validation, there can be a lot of variation. I'd suggest using an even higher percentage (25% is the default but you can also choose bit lower too). Result should be more reliable then.",
    "1087108": "Sorry, not 0.3% but 30% or 0.3/1, I've corrected it in my post, my bad."
  },
  "source": "meta"
}