{
  "id": 552066,
  "title": "Impact of High Variance Between Trials on Overfitting Analysis",
  "url": "/competitions/child-mind-institute-problematic-internet-use/discussion/552066",
  "author_name": "",
  "post_date": "2024-12-17T13:22:25.540984600Z",
  "votes": 1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hello everyone,</p>\n<p>We have previously discussed the details of the dataset we are working with, so I will skip that part and go straight to the question. During 200 trials of tuning, I noticed that the variance between trials is quite large. For example: trial 1 = 0.360, trial 2 = 0.380, trial 3 = 0.420, etc. Is this something that should be concerning? If the variance were very low, could we say that the model is less prone to overfitting?</p>\n<p>Thank you.</p>",
  "messages": [
    {
      "id": "3074217",
      "postDate": "12/17/2024 13:22:25",
      "content": "<p>Hello everyone,</p>\n<p>We have previously discussed the details of the dataset we are working with, so I will skip that part and go straight to the question. During 200 trials of tuning, I noticed that the variance between trials is quite large. For example: trial 1 = 0.360, trial 2 = 0.380, trial 3 = 0.420, etc. Is this something that should be concerning? If the variance were very low, could we say that the model is less prone to overfitting?</p>\n<p>Thank you.</p>",
      "rawMarkdown": "Hello everyone,\n\nWe have previously discussed the details of the dataset we are working with, so I will skip that part and go straight to the question. During 200 trials of tuning, I noticed that the variance between trials is quite large. For example: trial 1 = 0.360, trial 2 = 0.380, trial 3 = 0.420, etc. Is this something that should be concerning? If the variance were very low, could we say that the model is less prone to overfitting?\n\nThank you.",
      "votes": null
    },
    {
      "id": "3074362",
      "postDate": "12/17/2024 15:39:56",
      "content": "<p>Do you think tuning will add any extra value here <a href=\"https://www.kaggle.com/trcnveli\" target=\"_blank\">@trcnveli</a>? I tried to tune models a while ago and found that my tuned models were more unstable than the untuned ones </p>",
      "rawMarkdown": "Do you think tuning will add any extra value here @trcnveli? I tried to tune models a while ago and found that my tuned models were more unstable than the untuned ones",
      "votes": null
    },
    {
      "id": "3074383",
      "postDate": "12/17/2024 15:53:53",
      "content": "<p>Yes, I think tuning leads to more overfitting, but I also believe that the variance between trials can provide us with valuable information about the model. However, I wanted to ask your opinion as well. Do you think making such an observation is reasonable? <a href=\"https://www.kaggle.com/ravi20076\" target=\"_blank\">@ravi20076</a> </p>",
      "rawMarkdown": "Yes, I think tuning leads to more overfitting, but I also believe that the variance between trials can provide us with valuable information about the model. However, I wanted to ask your opinion as well. Do you think making such an observation is reasonable? @ravi20076",
      "votes": null
    },
    {
      "id": "3074506",
      "postDate": "12/17/2024 18:27:32",
      "content": "<p>If you have such a variance, it should be because you have a lot of noise, and your model fit noise, and so you are overfitting noise. </p>\n<p>Reducing noise is important before tuning : by reducing noise you will reduce variance and overfitting, you will fit the signal. And then it will be interesting to finetune your model to optimize your score.</p>",
      "rawMarkdown": "If you have such a variance, it should be because you have a lot of noise, and your model fit noise, and so you are overfitting noise. \n\nReducing noise is important before tuning : by reducing noise you will reduce variance and overfitting, you will fit the signal. And then it will be interesting to finetune your model to optimize your score.",
      "votes": null
    },
    {
      "id": "3074629",
      "postDate": "12/17/2024 21:32:12",
      "content": "<p>Yes of course it is reasonable <a href=\"https://www.kaggle.com/trcnveli\" target=\"_blank\">@trcnveli</a> </p>",
      "rawMarkdown": "Yes of course it is reasonable @trcnveli",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3074362,
      "author_name": "ravi20076",
      "author_url": "",
      "post_date": "12/17/2024 15:39:56",
      "content": "<p>Do you think tuning will add any extra value here <a href=\"https://www.kaggle.com/trcnveli\" target=\"_blank\">@trcnveli</a>? I tried to tune models a while ago and found that my tuned models were more unstable than the untuned ones </p>",
      "votes": null,
      "replies": [
        {
          "id": 3074383,
          "author_name": "trcnveli",
          "author_url": "",
          "post_date": "12/17/2024 15:53:53",
          "content": "<p>Yes, I think tuning leads to more overfitting, but I also believe that the variance between trials can provide us with valuable information about the model. However, I wanted to ask your opinion as well. Do you think making such an observation is reasonable? <a href=\"https://www.kaggle.com/ravi20076\" target=\"_blank\">@ravi20076</a> </p>",
          "votes": null,
          "replies": [
            {
              "id": 3074629,
              "author_name": "ravi20076",
              "author_url": "",
              "post_date": "12/17/2024 21:32:12",
              "content": "<p>Yes of course it is reasonable <a href=\"https://www.kaggle.com/trcnveli\" target=\"_blank\">@trcnveli</a> </p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3074506,
      "author_name": "adaubas",
      "author_url": "",
      "post_date": "12/17/2024 18:27:32",
      "content": "<p>If you have such a variance, it should be because you have a lot of noise, and your model fit noise, and so you are overfitting noise. </p>\n<p>Reducing noise is important before tuning : by reducing noise you will reduce variance and overfitting, you will fit the signal. And then it will be interesting to finetune your model to optimize your score.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3074217": "Hello everyone,\n\nWe have previously discussed the details of the dataset we are working with, so I will skip that part and go straight to the question. During 200 trials of tuning, I noticed that the variance between trials is quite large. For example: trial 1 = 0.360, trial 2 = 0.380, trial 3 = 0.420, etc. Is this something that should be concerning? If the variance were very low, could we say that the model is less prone to overfitting?\n\nThank you.",
    "3074362": "Do you think tuning will add any extra value here @trcnveli? I tried to tune models a while ago and found that my tuned models were more unstable than the untuned ones",
    "3074383": "Yes, I think tuning leads to more overfitting, but I also believe that the variance between trials can provide us with valuable information about the model. However, I wanted to ask your opinion as well. Do you think making such an observation is reasonable? @ravi20076",
    "3074506": "If you have such a variance, it should be because you have a lot of noise, and your model fit noise, and so you are overfitting noise. \n\nReducing noise is important before tuning : by reducing noise you will reduce variance and overfitting, you will fit the signal. And then it will be interesting to finetune your model to optimize your score.",
    "3074629": "Yes of course it is reasonable @trcnveli"
  },
  "source": "meta"
}