{
  "id": 163148,
  "title": "How to split a tf.data.TFRecordDataset to train and val datasets?",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/163148",
  "author_name": "",
  "post_date": "2020-07-01T06:50:14.896099300Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hey,\nHow can we split tf.data.TFRecordDataset as validation_split in model.fit cannot be used? \nI have done this after shuffling the full dataset (from StackOverflow):</p>\n\n<p>training_dataset = full_dataset.take(0.9*Total_number_of_batches)\nval_dataset = full_dataset.skip(0.9*Total_number_of_batches)</p>\n\n<p>Is this correct? Does take and skip occur complementary to each other (the elements in training_dataset should not be repeated in val_dataset)? If not how to do the split? </p>",
  "messages": [
    {
      "id": "910460",
      "postDate": "07/01/2020 06:50:14",
      "content": "<p>Hey,\nHow can we split tf.data.TFRecordDataset as validation_split in model.fit cannot be used? \nI have done this after shuffling the full dataset (from StackOverflow):</p>\n\n<p>training_dataset = full_dataset.take(0.9*Total_number_of_batches)\nval_dataset = full_dataset.skip(0.9*Total_number_of_batches)</p>\n\n<p>Is this correct? Does take and skip occur complementary to each other (the elements in training_dataset should not be repeated in val_dataset)? If not how to do the split? </p>",
      "rawMarkdown": "Hey,\nHow can we split tf.data.TFRecordDataset as validation_split in model.fit cannot be used? \nI have done this after shuffling the full dataset (from StackOverflow):\n\ntraining_dataset = full_dataset.take(0.9*Total_number_of_batches)\nval_dataset = full_dataset.skip(0.9*Total_number_of_batches)\n\nIs this correct? Does take and skip occur complementary to each other (the elements in training_dataset should not be repeated in val_dataset)? If not how to do the split?",
      "votes": null
    },
    {
      "id": "910548",
      "postDate": "07/01/2020 07:53:01",
      "content": "<p>You might want to read this: <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/158395\">https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/158395</a></p>",
      "rawMarkdown": "You might want to read this: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/158395",
      "votes": null
    },
    {
      "id": "910584",
      "postDate": "07/01/2020 08:22:58",
      "content": "<p>Thanks :)</p>",
      "rawMarkdown": "Thanks :)",
      "votes": null
    },
    {
      "id": "912758",
      "postDate": "07/02/2020 17:52:37",
      "content": "<p>You can create two TFRecord datasets and split the files, for example:\n```\ntrain_files = load_all_train_files()\nkfold = KFold(5)\nfor train,validation in kfold.split(train_files):\n   fold_train_files = train_files[train]\n   fold_validation_files = train_files[validation]</p>\n\n<p>training_ds = tf.data.TFRecordDataset(fold_train_files)\n   validation_ds =  tf.data.TFRecordDataset(fold_validation_files)\n```</p>",
      "rawMarkdown": "You can create two TFRecord datasets and split the files, for example:\n```\ntrain_files = load_all_train_files()\nkfold = KFold(5)\nfor train,validation in kfold.split(train_files):\n   fold_train_files = train_files[train]\n   fold_validation_files = train_files[validation]\n\n   training_ds = tf.data.TFRecordDataset(fold_train_files)\n   validation_ds =  tf.data.TFRecordDataset(fold_validation_files)\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 910548,
      "author_name": "graf10a",
      "author_url": "",
      "post_date": "07/01/2020 07:53:01",
      "content": "<p>You might want to read this: <a href=\"https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/158395\">https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/158395</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 910584,
          "author_name": "josealways123",
          "author_url": "",
          "post_date": "07/01/2020 08:22:58",
          "content": "<p>Thanks :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 912758,
      "author_name": "treadon",
      "author_url": "",
      "post_date": "07/02/2020 17:52:37",
      "content": "<p>You can create two TFRecord datasets and split the files, for example:\n```\ntrain_files = load_all_train_files()\nkfold = KFold(5)\nfor train,validation in kfold.split(train_files):\n   fold_train_files = train_files[train]\n   fold_validation_files = train_files[validation]</p>\n\n<p>training_ds = tf.data.TFRecordDataset(fold_train_files)\n   validation_ds =  tf.data.TFRecordDataset(fold_validation_files)\n```</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "910460": "Hey,\nHow can we split tf.data.TFRecordDataset as validation_split in model.fit cannot be used? \nI have done this after shuffling the full dataset (from StackOverflow):\n\ntraining_dataset = full_dataset.take(0.9*Total_number_of_batches)\nval_dataset = full_dataset.skip(0.9*Total_number_of_batches)\n\nIs this correct? Does take and skip occur complementary to each other (the elements in training_dataset should not be repeated in val_dataset)? If not how to do the split?",
    "910548": "You might want to read this: https://www.kaggle.com/c/siim-isic-melanoma-classification/discussion/158395",
    "910584": "Thanks :)",
    "912758": "You can create two TFRecord datasets and split the files, for example:\n```\ntrain_files = load_all_train_files()\nkfold = KFold(5)\nfor train,validation in kfold.split(train_files):\n   fold_train_files = train_files[train]\n   fold_validation_files = train_files[validation]\n\n   training_ds = tf.data.TFRecordDataset(fold_train_files)\n   validation_ds =  tf.data.TFRecordDataset(fold_validation_files)\n```"
  },
  "source": "meta"
}