{
  "id": 201124,
  "title": "There are 2 validation splitting strategies",
  "url": "/competitions/hubmap-kidney-segmentation/discussion/201124",
  "author_name": "",
  "post_date": "2020-12-03T09:17:20.062944600Z",
  "votes": 4,
  "comment_count": 1,
  "views": 0,
  "content": "<p>How do you split up your validation? I think there are two types.<br>\nThe first way is to cut out more than 10,000 images from 8 images, as many people do, and then split them up.<br>\nThe other way is to split it by tiffs. For example, we have 8 images. 6 of them are treated as training and 2 as validation. This method lowers the apparent CV as the image of the validation will be completely unseen. And it is closer to LB. It also tends to result in an unbalanced data set because the number of images that are cropped per image is different.<br>\nIn my one experiment, both LBs were the almost same.<br>\nWhich strategy do you prefer? Or is there a proper validation method?<br>\nI support the second one, but I don't think it's also correct because the dataset is unbalanced.</p>",
  "messages": [
    {
      "id": "1100717",
      "postDate": "12/03/2020 09:17:20",
      "content": "<p>How do you split up your validation? I think there are two types.<br>\nThe first way is to cut out more than 10,000 images from 8 images, as many people do, and then split them up.<br>\nThe other way is to split it by tiffs. For example, we have 8 images. 6 of them are treated as training and 2 as validation. This method lowers the apparent CV as the image of the validation will be completely unseen. And it is closer to LB. It also tends to result in an unbalanced data set because the number of images that are cropped per image is different.<br>\nIn my one experiment, both LBs were the almost same.<br>\nWhich strategy do you prefer? Or is there a proper validation method?<br>\nI support the second one, but I don't think it's also correct because the dataset is unbalanced.</p>",
      "rawMarkdown": "How do you split up your validation? I think there are two types.\nThe first way is to cut out more than 10,000 images from 8 images, as many people do, and then split them up.\nThe other way is to split it by tiffs. For example, we have 8 images. 6 of them are treated as training and 2 as validation. This method lowers the apparent CV as the image of the validation will be completely unseen. And it is closer to LB. It also tends to result in an unbalanced data set because the number of images that are cropped per image is different.\nIn my one experiment, both LBs were the almost same.\nWhich strategy do you prefer? Or is there a proper validation method?\nI support the second one, but I don't think it's also correct because the dataset is unbalanced.",
      "votes": null
    },
    {
      "id": "1102186",
      "postDate": "12/04/2020 17:08:07",
      "content": "<p>On the basis of my personal experience, I would choose the second approach. Working with satellite imagery we split the dataset so that each whole image is assigned to a single set to make the evaluation fairer. Zones (tiles) of the same image cannot be assigned to different sets because it could give the wrong impression that the model generalizes better than it really does, e. g. if the model has memorized very similar zones (from the same image).</p>",
      "rawMarkdown": "On the basis of my personal experience, I would choose the second approach. Working with satellite imagery we split the dataset so that each whole image is assigned to a single set to make the evaluation fairer. Zones (tiles) of the same image cannot be assigned to different sets because it could give the wrong impression that the model generalizes better than it really does, e. g. if the model has memorized very similar zones (from the same image).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1102186,
      "author_name": "cayala",
      "author_url": "",
      "post_date": "12/04/2020 17:08:07",
      "content": "<p>On the basis of my personal experience, I would choose the second approach. Working with satellite imagery we split the dataset so that each whole image is assigned to a single set to make the evaluation fairer. Zones (tiles) of the same image cannot be assigned to different sets because it could give the wrong impression that the model generalizes better than it really does, e. g. if the model has memorized very similar zones (from the same image).</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1100717": "How do you split up your validation? I think there are two types.\nThe first way is to cut out more than 10,000 images from 8 images, as many people do, and then split them up.\nThe other way is to split it by tiffs. For example, we have 8 images. 6 of them are treated as training and 2 as validation. This method lowers the apparent CV as the image of the validation will be completely unseen. And it is closer to LB. It also tends to result in an unbalanced data set because the number of images that are cropped per image is different.\nIn my one experiment, both LBs were the almost same.\nWhich strategy do you prefer? Or is there a proper validation method?\nI support the second one, but I don't think it's also correct because the dataset is unbalanced.",
    "1102186": "On the basis of my personal experience, I would choose the second approach. Working with satellite imagery we split the dataset so that each whole image is assigned to a single set to make the evaluation fairer. Zones (tiles) of the same image cannot be assigned to different sets because it could give the wrong impression that the model generalizes better than it really does, e. g. if the model has memorized very similar zones (from the same image)."
  },
  "source": "meta"
}