{
  "id": 92015,
  "title": "A problem from a beginner: How to turn hyperparameters since CV scores seem very different from LB scores?",
  "url": "/competitions/LANL-Earthquake-Prediction/discussion/92015",
  "author_name": "",
  "post_date": "2019-05-12T04:21:08.994594300Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Some methods to find better hyperparameters are repeated testing like [1], but in this competition CV scores seem not very matching LB scores. For example, my model using a method similar to [2] will get a better LB score before early stopping. So is there a better way to find hyperparameters than the method based on two LB scores per day?</p>\n\n<p>Thanks everybody!</p>\n\n<p>[1] <a href=\"https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt\">https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt</a>\n[2] <a href=\"https://www.kaggle.com/vettejeep/masters-final-project-model-lb-1-392\">https://www.kaggle.com/vettejeep/masters-final-project-model-lb-1-392</a></p>",
  "messages": [
    {
      "id": "530196",
      "postDate": "05/12/2019 04:21:08",
      "content": "<p>Some methods to find better hyperparameters are repeated testing like [1], but in this competition CV scores seem not very matching LB scores. For example, my model using a method similar to [2] will get a better LB score before early stopping. So is there a better way to find hyperparameters than the method based on two LB scores per day?</p>\n\n<p>Thanks everybody!</p>\n\n<p>[1] <a href=\"https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt\">https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt</a>\n[2] <a href=\"https://www.kaggle.com/vettejeep/masters-final-project-model-lb-1-392\">https://www.kaggle.com/vettejeep/masters-final-project-model-lb-1-392</a></p>",
      "rawMarkdown": "Some methods to find better hyperparameters are repeated testing like [1], but in this competition CV scores seem not very matching LB scores. For example, my model using a method similar to [2] will get a better LB score before early stopping. So is there a better way to find hyperparameters than the method based on two LB scores per day?\n\nThanks everybody!\n\n[1] https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt\n[2] https://www.kaggle.com/vettejeep/masters-final-project-model-lb-1-392",
      "votes": null
    },
    {
      "id": "530504",
      "postDate": "05/13/2019 03:22:02",
      "content": "<p>You need to first find a reliable CV strategy which is one of the key factors in this competition.\nIt has been discussed a lot in the discussions. a reliable CV in this competition, I believe, doesn't mean compatible LB-CV score but a consistent one. meaning that it doesn't matter if LB score is 1.45 when your CV score is 2.05; what matters is that if you managed to reduce your CV down to 2.00, you observe a reduction in the LB score as well.</p>\n\n<p>Only after you found a reliable CV strategy you can tune your hyperparameters. Otherwise you cannot measure whether your new parameters are working better or worse, which was your question. so, find a CV strategy that works.</p>",
      "rawMarkdown": "You need to first find a reliable CV strategy which is one of the key factors in this competition.\nIt has been discussed a lot in the discussions. a reliable CV in this competition, I believe, doesn't mean compatible LB-CV score but a consistent one. meaning that it doesn't matter if LB score is 1.45 when your CV score is 2.05; what matters is that if you managed to reduce your CV down to 2.00, you observe a reduction in the LB score as well.\n\nOnly after you found a reliable CV strategy you can tune your hyperparameters. Otherwise you cannot measure whether your new parameters are working better or worse, which was your question. so, find a CV strategy that works.",
      "votes": null
    },
    {
      "id": "530870",
      "postDate": "05/13/2019 21:10:55",
      "content": "<p>Assume you have a good CV scheme, you can try <a href=\"https://optuna.org/\">Optuna</a> and <a href=\"https://hyperopt.github.io/hyperopt/\">Hyperopt</a>. Please check the excellent post from bigironsphere:\n<a href=\"https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt\">https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt</a></p>",
      "rawMarkdown": "Assume you have a good CV scheme, you can try [Optuna](https://optuna.org/) and [Hyperopt](https://hyperopt.github.io/hyperopt/). Please check the excellent post from bigironsphere:\nhttps://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 530504,
      "author_name": "mhviraf",
      "author_url": "",
      "post_date": "05/13/2019 03:22:02",
      "content": "<p>You need to first find a reliable CV strategy which is one of the key factors in this competition.\nIt has been discussed a lot in the discussions. a reliable CV in this competition, I believe, doesn't mean compatible LB-CV score but a consistent one. meaning that it doesn't matter if LB score is 1.45 when your CV score is 2.05; what matters is that if you managed to reduce your CV down to 2.00, you observe a reduction in the LB score as well.</p>\n\n<p>Only after you found a reliable CV strategy you can tune your hyperparameters. Otherwise you cannot measure whether your new parameters are working better or worse, which was your question. so, find a CV strategy that works.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 530870,
      "author_name": "pukkinming",
      "author_url": "",
      "post_date": "05/13/2019 21:10:55",
      "content": "<p>Assume you have a good CV scheme, you can try <a href=\"https://optuna.org/\">Optuna</a> and <a href=\"https://hyperopt.github.io/hyperopt/\">Hyperopt</a>. Please check the excellent post from bigironsphere:\n<a href=\"https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt\">https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt</a></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "530196": "Some methods to find better hyperparameters are repeated testing like [1], but in this competition CV scores seem not very matching LB scores. For example, my model using a method similar to [2] will get a better LB score before early stopping. So is there a better way to find hyperparameters than the method based on two LB scores per day?\n\nThanks everybody!\n\n[1] https://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt\n[2] https://www.kaggle.com/vettejeep/masters-final-project-model-lb-1-392",
    "530504": "You need to first find a reliable CV strategy which is one of the key factors in this competition.\nIt has been discussed a lot in the discussions. a reliable CV in this competition, I believe, doesn't mean compatible LB-CV score but a consistent one. meaning that it doesn't matter if LB score is 1.45 when your CV score is 2.05; what matters is that if you managed to reduce your CV down to 2.00, you observe a reduction in the LB score as well.\n\nOnly after you found a reliable CV strategy you can tune your hyperparameters. Otherwise you cannot measure whether your new parameters are working better or worse, which was your question. so, find a CV strategy that works.",
    "530870": "Assume you have a good CV scheme, you can try [Optuna](https://optuna.org/) and [Hyperopt](https://hyperopt.github.io/hyperopt/). Please check the excellent post from bigironsphere:\nhttps://www.kaggle.com/bigironsphere/parameter-tuning-in-one-function-with-hyperopt"
  },
  "source": "meta"
}