{
  "id": 71350,
  "title": "Validation",
  "url": "/competitions/quora-insincere-questions-classification/discussion/71350",
  "author_name": "LDAR",
  "post_date": "2018-11-12T22:59:08.131000",
  "votes": 0,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hey guys, </p>\n\n<p>There is no target column in test set, how do you validate your result before submitting it? Create validation set from training set?</p>",
  "messages": [
    {
      "id": 420007,
      "postDate": "2018-11-12T23:08:40.323Z",
      "content": "<p>Yes you can create a validation set from the training set. For example the below using sklearn's train_test_split. </p>\n\n<p><code>train_df, val_df = train_test_split(train_df, test_size=0.1, random_state=42)</code></p>",
      "rawMarkdown": "Yes you can create a validation set from the training set. For example the below using sklearn's train_test_split. \n\n`train_df, val_df = train_test_split(train_df, test_size=0.1, random_state=42)`",
      "votes": 2,
      "replies": [
        {
          "id": 420293,
          "postDate": "2018-11-13T12:26:31.540Z",
          "content": "<p>Hey James, thanks I know. The question I am wondering is why the organizers provided the test set then if you can not validate upon it as there are not targets present there?</p>",
          "rawMarkdown": "Hey James, thanks I know. The question I am wondering is why the organizers provided the test set then if you can not validate upon it as there are not targets present there?"
        }
      ]
    },
    {
      "id": 420005,
      "postDate": "2018-11-12T22:59:08.133Z",
      "content": "<p>Hey guys, </p>\n\n<p>There is no target column in test set, how do you validate your result before submitting it? Create validation set from training set?</p>",
      "rawMarkdown": "Hey guys, \n\nThere is no target column in test set, how do you validate your result before submitting it? Create validation set from training set?"
    },
    {
      "id": 421893,
      "postDate": "2018-11-15T14:40:56.787Z",
      "rawMarkdown": "",
      "votes": 2,
      "isDeleted": true,
      "replies": [
        {
          "id": 423164,
          "postDate": "2018-11-17T16:50:19.017Z",
          "content": "<p>Hey Bastetien, thanks yes this is reasonable. You will have derive the val set with targets from train data. The only thong is to check if your val set is identical to train set, to make sure that your score will be more or less consistent between val and test sets </p>",
          "rawMarkdown": "Hey Bastetien, thanks yes this is reasonable. You will have derive the val set with targets from train data. The only thong is to check if your val set is identical to train set, to make sure that your score will be more or less consistent between val and test sets "
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 420007,
      "author_name": "James Requa",
      "author_url": "",
      "post_date": "2018-11-12T23:08:40.323000",
      "content": "<p>Yes you can create a validation set from the training set. For example the below using sklearn's train_test_split. </p>\n\n<p><code>train_df, val_df = train_test_split(train_df, test_size=0.1, random_state=42)</code></p>",
      "votes": 2,
      "replies": [
        {
          "id": 420293,
          "author_name": "LDAR",
          "author_url": "",
          "post_date": "2018-11-13T12:26:31.540000",
          "content": "<p>Hey James, thanks I know. The question I am wondering is why the organizers provided the test set then if you can not validate upon it as there are not targets present there?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 421893,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-11-15T14:40:56.787000",
      "content": "",
      "votes": 2,
      "replies": [
        {
          "id": 423164,
          "author_name": "LDAR",
          "author_url": "",
          "post_date": "2018-11-17T16:50:19.017000",
          "content": "<p>Hey Bastetien, thanks yes this is reasonable. You will have derive the val set with targets from train data. The only thong is to check if your val set is identical to train set, to make sure that your score will be more or less consistent between val and test sets </p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "420007": "Yes you can create a validation set from the training set. For example the below using sklearn's train_test_split. \n\n`train_df, val_df = train_test_split(train_df, test_size=0.1, random_state=42)`",
    "420005": "Hey guys, \n\nThere is no target column in test set, how do you validate your result before submitting it? Create validation set from training set?",
    "421893": ""
  }
}