{
  "id": 546038,
  "title": "Optimized Validation QWK vs Validation QWK",
  "url": "/competitions/child-mind-institute-problematic-internet-use/discussion/546038",
  "author_name": "",
  "post_date": "2024-11-13T14:12:47.861278200Z",
  "votes": 8,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I have applied multiobjective hyperparamter optimization with optuna. I have doen 250 trials with Bayesian Sampler. Below you find the plots of <code>Optimized Validation QWK</code> vs <code>TRain QWK</code>  and <code>Validation QWK</code> vs <code>TRain QWK</code> . </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2F59c03f783687ba23cd47b09ce120931f%2Fnewplot%20(3).png?generation=1731507094021131&amp;alt=media\" alt=\"\"></p>\n<p>💡 <strong>INSIGHTS</strong></p>\n<ul>\n<li>From the first plot one can conlcude that when <code>Train QWK &gt;= 0.6</code> the model starts overfitting.</li>\n<li>From the second plot one can conclude that when <code>Train QWK &gt;= 0.8</code> the model starts to overfit. But the variance of <code>Validation QWK</code> starts increaing from <code>0.6</code>.</li>\n</ul>\n<p>So can one conclude that the model starts overfitting from <code>Train QWK &gt;= 0.6</code>. Curious how you think? </p>\n<p>Check the <a href=\"https://www.kaggle.com/code/wti200/multiobjective-optimization\" target=\"_blank\">notebook</a> if you are interested in the specific details of the optimization process. </p>",
  "messages": [
    {
      "id": "3044559",
      "postDate": "11/13/2024 14:12:47",
      "content": "<p>I have applied multiobjective hyperparamter optimization with optuna. I have doen 250 trials with Bayesian Sampler. Below you find the plots of <code>Optimized Validation QWK</code> vs <code>TRain QWK</code>  and <code>Validation QWK</code> vs <code>TRain QWK</code> . </p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2F59c03f783687ba23cd47b09ce120931f%2Fnewplot%20(3).png?generation=1731507094021131&amp;alt=media\" alt=\"\"></p>\n<p>💡 <strong>INSIGHTS</strong></p>\n<ul>\n<li>From the first plot one can conlcude that when <code>Train QWK &gt;= 0.6</code> the model starts overfitting.</li>\n<li>From the second plot one can conclude that when <code>Train QWK &gt;= 0.8</code> the model starts to overfit. But the variance of <code>Validation QWK</code> starts increaing from <code>0.6</code>.</li>\n</ul>\n<p>So can one conclude that the model starts overfitting from <code>Train QWK &gt;= 0.6</code>. Curious how you think? </p>\n<p>Check the <a href=\"https://www.kaggle.com/code/wti200/multiobjective-optimization\" target=\"_blank\">notebook</a> if you are interested in the specific details of the optimization process. </p>",
      "rawMarkdown": "I have applied multiobjective hyperparamter optimization with optuna. I have doen 250 trials with Bayesian Sampler. Below you find the plots of `Optimized Validation QWK` vs `TRain QWK`  and `Validation QWK` vs `TRain QWK` . \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2F59c03f783687ba23cd47b09ce120931f%2Fnewplot%20(3).png?generation=1731507094021131&alt=media)\n\n💡 **INSIGHTS**\n- From the first plot one can conlcude that when `Train QWK >= 0.6` the model starts overfitting.\n- From the second plot one can conclude that when `Train QWK >= 0.8` the model starts to overfit. But the variance of `Validation QWK` starts increaing from `0.6`.\n\nSo can one conclude that the model starts overfitting from `Train QWK >= 0.6`. Curious how you think? \n\nCheck the [notebook](https://www.kaggle.com/code/wti200/multiobjective-optimization) if you are interested in the specific details of the optimization process.",
      "votes": null
    },
    {
      "id": "3044675",
      "postDate": "11/13/2024 17:28:37",
      "content": "<p>What could be the possible reasons for the sudden jump observed in the Pareto Front 1 chart?</p>",
      "rawMarkdown": "What could be the possible reasons for the sudden jump observed in the Pareto Front 1 chart?",
      "votes": null
    },
    {
      "id": "3045235",
      "postDate": "11/14/2024 09:54:52",
      "content": "<p><a href=\"https://www.kaggle.com/welcomeworld\" target=\"_blank\">@welcomeworld</a> I think that there are two possible explanations:</p>\n<ol>\n<li><em>Exploration and Exploitation</em>:  Bayesian optimization searchs mainly in the most promising regions but sometimes it can get stuck in a local optimum to mitigate this risk it tries to explore other regions as well. See the optimization history of the plot.</li>\n<li><em>KappaOptimizer</em>: I think that the threshold of KappaOptimizer is also a factor. If you take a close look at the scatter plot the most left trial have <code>Validation QWK = 0</code> but <code>Optimized Validation QWK ~ 0.36</code>. The same logic holds for some other trials that have a low <code>Validation QWK</code> but a high <code>Optimized Validation QWK</code>.</li>\n</ol>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2F9b75cc884de2c8b00f783f83cd8b6c5b%2Fnewplot%20(6).png?generation=1731577625421231&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "welcomeworld I think that there are two possible explanations:\n1. *Exploration and Exploitation*:  Bayesian optimization searchs mainly in the most promising regions but sometimes it can get stuck in a local optimum to mitigate this risk it tries to explore other regions as well. See the optimization history of the plot.\n2. *KappaOptimizer*: I think that the threshold of KappaOptimizer is also a factor. If you take a close look at the scatter plot the most left trial have `Validation QWK = 0` but `Optimized Validation QWK ~ 0.36`. The same logic holds for some other trials that have a low `Validation QWK` but a high `Optimized Validation QWK`.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2F9b75cc884de2c8b00f783f83cd8b6c5b%2Fnewplot%20(6).png?generation=1731577625421231&alt=media)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3044675,
      "author_name": "welcomeworld",
      "author_url": "",
      "post_date": "11/13/2024 17:28:37",
      "content": "<p>What could be the possible reasons for the sudden jump observed in the Pareto Front 1 chart?</p>",
      "votes": null,
      "replies": [
        {
          "id": 3045235,
          "author_name": "wti200",
          "author_url": "",
          "post_date": "11/14/2024 09:54:52",
          "content": "<p><a href=\"https://www.kaggle.com/welcomeworld\" target=\"_blank\">@welcomeworld</a> I think that there are two possible explanations:</p>\n<ol>\n<li><em>Exploration and Exploitation</em>:  Bayesian optimization searchs mainly in the most promising regions but sometimes it can get stuck in a local optimum to mitigate this risk it tries to explore other regions as well. See the optimization history of the plot.</li>\n<li><em>KappaOptimizer</em>: I think that the threshold of KappaOptimizer is also a factor. If you take a close look at the scatter plot the most left trial have <code>Validation QWK = 0</code> but <code>Optimized Validation QWK ~ 0.36</code>. The same logic holds for some other trials that have a low <code>Validation QWK</code> but a high <code>Optimized Validation QWK</code>.</li>\n</ol>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2F9b75cc884de2c8b00f783f83cd8b6c5b%2Fnewplot%20(6).png?generation=1731577625421231&amp;alt=media\" alt=\"\"></p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3044559": "I have applied multiobjective hyperparamter optimization with optuna. I have doen 250 trials with Bayesian Sampler. Below you find the plots of `Optimized Validation QWK` vs `TRain QWK`  and `Validation QWK` vs `TRain QWK` . \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2F59c03f783687ba23cd47b09ce120931f%2Fnewplot%20(3).png?generation=1731507094021131&alt=media)\n\n💡 **INSIGHTS**\n- From the first plot one can conlcude that when `Train QWK >= 0.6` the model starts overfitting.\n- From the second plot one can conclude that when `Train QWK >= 0.8` the model starts to overfit. But the variance of `Validation QWK` starts increaing from `0.6`.\n\nSo can one conclude that the model starts overfitting from `Train QWK >= 0.6`. Curious how you think? \n\nCheck the [notebook](https://www.kaggle.com/code/wti200/multiobjective-optimization) if you are interested in the specific details of the optimization process.",
    "3044675": "What could be the possible reasons for the sudden jump observed in the Pareto Front 1 chart?",
    "3045235": "welcomeworld I think that there are two possible explanations:\n1. *Exploration and Exploitation*:  Bayesian optimization searchs mainly in the most promising regions but sometimes it can get stuck in a local optimum to mitigate this risk it tries to explore other regions as well. See the optimization history of the plot.\n2. *KappaOptimizer*: I think that the threshold of KappaOptimizer is also a factor. If you take a close look at the scatter plot the most left trial have `Validation QWK = 0` but `Optimized Validation QWK ~ 0.36`. The same logic holds for some other trials that have a low `Validation QWK` but a high `Optimized Validation QWK`.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F225499%2F9b75cc884de2c8b00f783f83cd8b6c5b%2Fnewplot%20(6).png?generation=1731577625421231&alt=media)"
  },
  "source": "meta"
}