{
  "id": 56019,
  "title": "Using parameter tuning but score 0.006 drop. what's problem?",
  "url": "/competitions/talkingdata-adtracking-fraud-detection/discussion/56019",
  "author_name": "",
  "post_date": "2018-05-04T15:41:01.930063100Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello. i'm challenging first time to kaggle. </p>\n\n<p>I read the \"Talkingdata-added-new-features-in-lightgbm kernel (<a href=\"https://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm\">https://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm</a>) and take a baysian optimizaion parameter tuning like this(<a href=\"https://www.kaggle.com/chocozzz/lightgbm-parameter-tuning\">https://www.kaggle.com/chocozzz/lightgbm-parameter-tuning</a>). I get a not bad AUC.</p>\n\n<p>So I changed the hyperparameter and submitted it, but my score dropped.\nWhy?</p>",
  "messages": [
    {
      "id": "323198",
      "postDate": "05/04/2018 15:41:01",
      "content": "<p>Hello. i'm challenging first time to kaggle. </p>\n\n<p>I read the \"Talkingdata-added-new-features-in-lightgbm kernel (<a href=\"https://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm\">https://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm</a>) and take a baysian optimizaion parameter tuning like this(<a href=\"https://www.kaggle.com/chocozzz/lightgbm-parameter-tuning\">https://www.kaggle.com/chocozzz/lightgbm-parameter-tuning</a>). I get a not bad AUC.</p>\n\n<p>So I changed the hyperparameter and submitted it, but my score dropped.\nWhy?</p>",
      "rawMarkdown": "Hello. i'm challenging first time to kaggle. \n\nI read the \"Talkingdata-added-new-features-in-lightgbm kernel ([https://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm][1]) and take a baysian optimizaion parameter tuning like this([https://www.kaggle.com/chocozzz/lightgbm-parameter-tuning][2]). I get a not bad AUC.\n\nSo I changed the hyperparameter and submitted it, but my score dropped.\nWhy?\n\n\n  [1]: https://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm\n  [2]: https://www.kaggle.com/chocozzz/lightgbm-parameter-tuning",
      "votes": null
    },
    {
      "id": "323233",
      "postDate": "05/04/2018 16:54:56",
      "content": "<p>Assuming all the setup is correct, I think there's a good chance that your bayesian optimization is badly overfitting to the validation set you use. If you're using the 2.5 million tail as in the kernel,  that's a relatively small validation set in this context and comes from a time period that isn't necessarily representative of the public (or private) leaderboard.  </p>",
      "rawMarkdown": "Assuming all the setup is correct, I think there's a good chance that your bayesian optimization is badly overfitting to the validation set you use. If you're using the 2.5 million tail as in the kernel,  that's a relatively small validation set in this context and comes from a time period that isn't necessarily representative of the public (or private) leaderboard.",
      "votes": null
    },
    {
      "id": "323371",
      "postDate": "05/05/2018 00:20:46",
      "content": "<p>Thank you for your kind answer.!!! then i have to increase validation set.\nbut i using kaggle kernel that not stand more than 6 hours...  have any advice??\ni wait for your reply. thanks :D</p>",
      "rawMarkdown": "Thank you for your kind answer.!!! then i have to increase validation set.\nbut i using kaggle kernel that not stand more than 6 hours...  have any advice??\ni wait for your reply. thanks :D",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 323233,
      "author_name": "aquatic",
      "author_url": "",
      "post_date": "05/04/2018 16:54:56",
      "content": "<p>Assuming all the setup is correct, I think there's a good chance that your bayesian optimization is badly overfitting to the validation set you use. If you're using the 2.5 million tail as in the kernel,  that's a relatively small validation set in this context and comes from a time period that isn't necessarily representative of the public (or private) leaderboard.  </p>",
      "votes": null,
      "replies": [
        {
          "id": 323371,
          "author_name": "chocozzz",
          "author_url": "",
          "post_date": "05/05/2018 00:20:46",
          "content": "<p>Thank you for your kind answer.!!! then i have to increase validation set.\nbut i using kaggle kernel that not stand more than 6 hours...  have any advice??\ni wait for your reply. thanks :D</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "323198": "Hello. i'm challenging first time to kaggle. \n\nI read the \"Talkingdata-added-new-features-in-lightgbm kernel ([https://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm][1]) and take a baysian optimizaion parameter tuning like this([https://www.kaggle.com/chocozzz/lightgbm-parameter-tuning][2]). I get a not bad AUC.\n\nSo I changed the hyperparameter and submitted it, but my score dropped.\nWhy?\n\n\n  [1]: https://www.kaggle.com/asraful70/talkingdata-added-new-features-in-lightgbm\n  [2]: https://www.kaggle.com/chocozzz/lightgbm-parameter-tuning",
    "323233": "Assuming all the setup is correct, I think there's a good chance that your bayesian optimization is badly overfitting to the validation set you use. If you're using the 2.5 million tail as in the kernel,  that's a relatively small validation set in this context and comes from a time period that isn't necessarily representative of the public (or private) leaderboard.",
    "323371": "Thank you for your kind answer.!!! then i have to increase validation set.\nbut i using kaggle kernel that not stand more than 6 hours...  have any advice??\ni wait for your reply. thanks :D"
  },
  "source": "meta"
}