{
  "id": 44039,
  "title": "Data partitioning and local CV score",
  "url": "/competitions/tensorflow-speech-recognition-challenge/discussion/44039",
  "author_name": "",
  "post_date": "2017-11-22T16:55:03.690517300Z",
  "votes": 4,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi,</p>\n\n<p>I initially used a random partitioning where I had a 70-30 train/validation split and my model produced a local CV score of 0.81 but a Kaggle score of 0.49. I then thought that I should use the training/testing/validation split proposed by the text files included with the competition data and explained by the host. The new model had local CV of 0.80 and of 0.79 in the holdout set, yet on the LB it produced a 0.47. I wonder what I may be doing wrong.</p>\n\n<p>Are your local CV scores very far away from the LB scores in this competition?</p>\n\n<p>Also, I noticed that the _background_noise_ has very little files and in the train/testing/validation both testing and validation had none. What did you use to solve that problem?</p>\n\n<p>Thanks!\nMarco</p>",
  "messages": [
    {
      "id": "247251",
      "postDate": "11/22/2017 16:55:03",
      "content": "<p>Hi,</p>\n\n<p>I initially used a random partitioning where I had a 70-30 train/validation split and my model produced a local CV score of 0.81 but a Kaggle score of 0.49. I then thought that I should use the training/testing/validation split proposed by the text files included with the competition data and explained by the host. The new model had local CV of 0.80 and of 0.79 in the holdout set, yet on the LB it produced a 0.47. I wonder what I may be doing wrong.</p>\n\n<p>Are your local CV scores very far away from the LB scores in this competition?</p>\n\n<p>Also, I noticed that the _background_noise_ has very little files and in the train/testing/validation both testing and validation had none. What did you use to solve that problem?</p>\n\n<p>Thanks!\nMarco</p>",
      "rawMarkdown": "Hi,\n\nI initially used a random partitioning where I had a 70-30 train/validation split and my model produced a local CV score of 0.81 but a Kaggle score of 0.49. I then thought that I should use the training/testing/validation split proposed by the text files included with the competition data and explained by the host. The new model had local CV of 0.80 and of 0.79 in the holdout set, yet on the LB it produced a 0.47. I wonder what I may be doing wrong.\n\nAre your local CV scores very far away from the LB scores in this competition?\n\nAlso, I noticed that the _background_noise_ has very little files and in the train/testing/validation both testing and validation had none. What did you use to solve that problem?\n\nThanks!\nMarco",
      "votes": null
    },
    {
      "id": "247265",
      "postDate": "11/22/2017 17:29:23",
      "content": "<p>I have the same situation.  My local CV ~0.9 but LB ~0.7.  It maybe not related to CV but feature</p>",
      "rawMarkdown": "I have the same situation.  My local CV ~0.9 but LB ~0.7.  It maybe not related to CV but feature",
      "votes": null
    },
    {
      "id": "248568",
      "postDate": "11/26/2017 13:59:59",
      "content": "<p>I have about 7% difference between my local validation score and LB score. I believe this is largely related to the \"unknown\" class.</p>\n\n<p>Note that if you do your own train/validation split that you should keep in mind the ID of the speaker. The validation set should not have any speakers that are in the training set. I made this mistake and then my validation score was <em>way</em> better than the LB score, which makes sense because you're giving your validation set an unfair advantage.</p>",
      "rawMarkdown": "I have about 7% difference between my local validation score and LB score. I believe this is largely related to the \"unknown\" class.\n\nNote that if you do your own train/validation split that you should keep in mind the ID of the speaker. The validation set should not have any speakers that are in the training set. I made this mistake and then my validation score was *way* better than the LB score, which makes sense because you're giving your validation set an unfair advantage.",
      "votes": null
    },
    {
      "id": "262232",
      "postDate": "12/25/2017 22:23:56",
      "content": "<p>Same issue as @yuerlong. Local cv 0.9xx, LB 0.7x. How many folds are you guys using?</p>",
      "rawMarkdown": "Same issue as @yuerlong. Local cv 0.9xx, LB 0.7x. How many folds are you guys using?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 247265,
      "author_name": "yuerlong",
      "author_url": "",
      "post_date": "11/22/2017 17:29:23",
      "content": "<p>I have the same situation.  My local CV ~0.9 but LB ~0.7.  It maybe not related to CV but feature</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 248568,
      "author_name": "humananalog",
      "author_url": "",
      "post_date": "11/26/2017 13:59:59",
      "content": "<p>I have about 7% difference between my local validation score and LB score. I believe this is largely related to the \"unknown\" class.</p>\n\n<p>Note that if you do your own train/validation split that you should keep in mind the ID of the speaker. The validation set should not have any speakers that are in the training set. I made this mistake and then my validation score was <em>way</em> better than the LB score, which makes sense because you're giving your validation set an unfair advantage.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 262232,
      "author_name": "kirk86",
      "author_url": "",
      "post_date": "12/25/2017 22:23:56",
      "content": "<p>Same issue as @yuerlong. Local cv 0.9xx, LB 0.7x. How many folds are you guys using?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "247251": "Hi,\n\nI initially used a random partitioning where I had a 70-30 train/validation split and my model produced a local CV score of 0.81 but a Kaggle score of 0.49. I then thought that I should use the training/testing/validation split proposed by the text files included with the competition data and explained by the host. The new model had local CV of 0.80 and of 0.79 in the holdout set, yet on the LB it produced a 0.47. I wonder what I may be doing wrong.\n\nAre your local CV scores very far away from the LB scores in this competition?\n\nAlso, I noticed that the _background_noise_ has very little files and in the train/testing/validation both testing and validation had none. What did you use to solve that problem?\n\nThanks!\nMarco",
    "247265": "I have the same situation.  My local CV ~0.9 but LB ~0.7.  It maybe not related to CV but feature",
    "248568": "I have about 7% difference between my local validation score and LB score. I believe this is largely related to the \"unknown\" class.\n\nNote that if you do your own train/validation split that you should keep in mind the ID of the speaker. The validation set should not have any speakers that are in the training set. I made this mistake and then my validation score was *way* better than the LB score, which makes sense because you're giving your validation set an unfair advantage.",
    "262232": "Same issue as @yuerlong. Local cv 0.9xx, LB 0.7x. How many folds are you guys using?"
  },
  "source": "meta"
}