{
  "id": 102801,
  "title": "How to do validation properly?",
  "url": "/competitions/aptos2019-blindness-detection/discussion/102801",
  "author_name": "",
  "post_date": "2019-08-05T06:02:19.972637500Z",
  "votes": 3,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi all!\nAs there is huge discrepancy in train, public test and previous competition data, how do you validate your models?</p>\n\n<p>I am now doing simple cross-validation with one out on training data only, however, I think it might lead to overfit 🙁 </p>",
  "messages": [
    {
      "id": "592293",
      "postDate": "08/05/2019 06:02:19",
      "content": "<p>Hi all!\nAs there is huge discrepancy in train, public test and previous competition data, how do you validate your models?</p>\n\n<p>I am now doing simple cross-validation with one out on training data only, however, I think it might lead to overfit 🙁 </p>",
      "rawMarkdown": "Hi all!\nAs there is huge discrepancy in train, public test and previous competition data, how do you validate your models?\n\nI am now doing simple cross-validation with one out on training data only, however, I think it might lead to overfit 🙁",
      "votes": null
    },
    {
      "id": "592411",
      "postDate": "08/05/2019 08:42:43",
      "content": "<p>If you are using just new competition data then try using 0.2 split and train till it does not overfit. </p>",
      "rawMarkdown": "If you are using just new competition data then try using 0.2 split and train till it does not overfit.",
      "votes": null
    },
    {
      "id": "592418",
      "postDate": "08/05/2019 08:49:41",
      "content": "<p>I am afraid that public LB data is very different from both train data and previous competition data. Now it's very differnt to guess what's in the private LB data. \nProbably, we need to aim for model that generalize <em>very</em> good (especially, considering the size of private dataset). </p>",
      "rawMarkdown": "I am afraid that public LB data is very different from both train data and previous competition data. Now it's very differnt to guess what's in the private LB data. \nProbably, we need to aim for model that generalize *very* good (especially, considering the size of private dataset).",
      "votes": null
    },
    {
      "id": "592420",
      "postDate": "08/05/2019 08:55:20",
      "content": "<p>Hmmm this is a tough one. Because validation qwk does not correlate well with public LB. But you still need validation data to make sure model doesn't overfit (Like when train loss is far below val loss, then it's an indication your model is overfitting). I guess you have no choice but to experiment (preprocessing, data augmentation, different models and such) and test your idea by submit to know what works.</p>",
      "rawMarkdown": "Hmmm this is a tough one. Because validation qwk does not correlate well with public LB. But you still need validation data to make sure model doesn't overfit (Like when train loss is far below val loss, then it's an indication your model is overfitting). I guess you have no choice but to experiment (preprocessing, data augmentation, different models and such) and test your idea by submit to know what works.",
      "votes": null
    },
    {
      "id": "593001",
      "postDate": "08/06/2019 05:12:03",
      "content": "<p>my best score is purely training one model, no modifications, on both datasets and no finetuning on the current dataset :)</p>",
      "rawMarkdown": "my best score is purely training one model, no modifications, on both datasets and no finetuning on the current dataset :)",
      "votes": null
    },
    {
      "id": "593123",
      "postDate": "08/06/2019 07:50:21",
      "content": "<p>By no modifications you mean no pre-processing?</p>",
      "rawMarkdown": "By no modifications you mean no pre-processing?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 592411,
      "author_name": "harshthaker",
      "author_url": "",
      "post_date": "08/05/2019 08:42:43",
      "content": "<p>If you are using just new competition data then try using 0.2 split and train till it does not overfit. </p>",
      "votes": null,
      "replies": [
        {
          "id": 592418,
          "author_name": "spsancti",
          "author_url": "",
          "post_date": "08/05/2019 08:49:41",
          "content": "<p>I am afraid that public LB data is very different from both train data and previous competition data. Now it's very differnt to guess what's in the private LB data. \nProbably, we need to aim for model that generalize <em>very</em> good (especially, considering the size of private dataset). </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 593001,
          "author_name": "sidhanthholalkere",
          "author_url": "",
          "post_date": "08/06/2019 05:12:03",
          "content": "<p>my best score is purely training one model, no modifications, on both datasets and no finetuning on the current dataset :)</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 593123,
          "author_name": "sam1320",
          "author_url": "",
          "post_date": "08/06/2019 07:50:21",
          "content": "<p>By no modifications you mean no pre-processing?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 592420,
      "author_name": "quandapro",
      "author_url": "",
      "post_date": "08/05/2019 08:55:20",
      "content": "<p>Hmmm this is a tough one. Because validation qwk does not correlate well with public LB. But you still need validation data to make sure model doesn't overfit (Like when train loss is far below val loss, then it's an indication your model is overfitting). I guess you have no choice but to experiment (preprocessing, data augmentation, different models and such) and test your idea by submit to know what works.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "592293": "Hi all!\nAs there is huge discrepancy in train, public test and previous competition data, how do you validate your models?\n\nI am now doing simple cross-validation with one out on training data only, however, I think it might lead to overfit 🙁",
    "592411": "If you are using just new competition data then try using 0.2 split and train till it does not overfit.",
    "592418": "I am afraid that public LB data is very different from both train data and previous competition data. Now it's very differnt to guess what's in the private LB data. \nProbably, we need to aim for model that generalize *very* good (especially, considering the size of private dataset).",
    "592420": "Hmmm this is a tough one. Because validation qwk does not correlate well with public LB. But you still need validation data to make sure model doesn't overfit (Like when train loss is far below val loss, then it's an indication your model is overfitting). I guess you have no choice but to experiment (preprocessing, data augmentation, different models and such) and test your idea by submit to know what works.",
    "593001": "my best score is purely training one model, no modifications, on both datasets and no finetuning on the current dataset :)",
    "593123": "By no modifications you mean no pre-processing?"
  },
  "source": "meta"
}