{
  "id": 549818,
  "title": "Are Top Scores on Leaderboard by A Random Luck?",
  "url": "/competitions/child-mind-institute-problematic-internet-use/discussion/549818",
  "author_name": "Alperen Duru",
  "post_date": "2024-12-04T03:31:24.451000",
  "votes": 0,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi everyone, I am fairly new here and wanted to discuss the randomness in scores, especially in high places of the leaderboard. </p>\n<p>I am trying some type of ensembles method myself and it seems to be performing somewhat acceptable, however, I am not able to achieve above 0.49 with tuning, etc. Any time I try tuning, my score wanders around 0.465 to 0.48.</p>\n<p>I recently checked the published code in the \"Code\" section and found a code claiming to achieve 0.494 with including TabNet in ensembles with other boosting methods. So I copied and simply submitted the code and got 0.485.</p>\n<p>Ok, maybe a little variation here which can be acceptable. However, when I change the SEED value, I am getting something like 0.45! Now, I started losing my belief in top people actually getting those high numbers and not sure if their code would achieve similar values for changing SEED values. </p>\n<p>What is your opinion? Is this just my code or is this somewhat normal? If it is so sensitive to these seed values, then the final results for the competition will be like a lottery!</p>",
  "messages": [
    {
      "id": 3063117,
      "postDate": "2024-12-04T07:05:34.463Z",
      "content": "<p>If your predictions are less sensitive to random seeds on the public leaderboard compared to those produced by the best public notebooks, keep trusting your cross-validation !</p>",
      "rawMarkdown": "If your predictions are less sensitive to random seeds on the public leaderboard compared to those produced by the best public notebooks, keep trusting your cross-validation !",
      "votes": 1,
      "replies": [
        {
          "id": 3063740,
          "postDate": "2024-12-04T20:10:06.173Z",
          "content": "<p>I mean, I have methods that can wander around 0.45 to 0.46 ish reliably. However, I am not sure if someone can get a persistent score with changing SEED values, especially for scores over 0.48. </p>\n<p>I am simply curious, is there anyone getting persistency in their scores for anything over 0.48 with changing seed values?</p>",
          "rawMarkdown": "I mean, I have methods that can wander around 0.45 to 0.46 ish reliably. However, I am not sure if someone can get a persistent score with changing SEED values, especially for scores over 0.48. \n\nI am simply curious, is there anyone getting persistency in their scores for anything over 0.48 with changing seed values?"
        }
      ]
    },
    {
      "id": 3075370,
      "postDate": "2024-12-18T18:48:03.093Z",
      "content": "<p>I have faced the same issue with the same notebook you mentioned. However, it wasn't the case when running other top score notebooks. It’s completely normal for models, especially those involving randomness like ensemble methods, to show some variation when you change the seed. This may happen particularly when using models like TabNet, which rely on stochastic processes. </p>",
      "rawMarkdown": "I have faced the same issue with the same notebook you mentioned. However, it wasn't the case when running other top score notebooks. It’s completely normal for models, especially those involving randomness like ensemble methods, to show some variation when you change the seed. This may happen particularly when using models like TabNet, which rely on stochastic processes. "
    },
    {
      "id": 3062940,
      "postDate": "2024-12-04T03:31:24.450Z",
      "content": "<p>Hi everyone, I am fairly new here and wanted to discuss the randomness in scores, especially in high places of the leaderboard. </p>\n<p>I am trying some type of ensembles method myself and it seems to be performing somewhat acceptable, however, I am not able to achieve above 0.49 with tuning, etc. Any time I try tuning, my score wanders around 0.465 to 0.48.</p>\n<p>I recently checked the published code in the \"Code\" section and found a code claiming to achieve 0.494 with including TabNet in ensembles with other boosting methods. So I copied and simply submitted the code and got 0.485.</p>\n<p>Ok, maybe a little variation here which can be acceptable. However, when I change the SEED value, I am getting something like 0.45! Now, I started losing my belief in top people actually getting those high numbers and not sure if their code would achieve similar values for changing SEED values. </p>\n<p>What is your opinion? Is this just my code or is this somewhat normal? If it is so sensitive to these seed values, then the final results for the competition will be like a lottery!</p>",
      "rawMarkdown": "Hi everyone, I am fairly new here and wanted to discuss the randomness in scores, especially in high places of the leaderboard. \n\nI am trying some type of ensembles method myself and it seems to be performing somewhat acceptable, however, I am not able to achieve above 0.49 with tuning, etc. Any time I try tuning, my score wanders around 0.465 to 0.48.\n\nI recently checked the published code in the \"Code\" section and found a code claiming to achieve 0.494 with including TabNet in ensembles with other boosting methods. So I copied and simply submitted the code and got 0.485.\n\nOk, maybe a little variation here which can be acceptable. However, when I change the SEED value, I am getting something like 0.45! Now, I started losing my belief in top people actually getting those high numbers and not sure if their code would achieve similar values for changing SEED values. \n\nWhat is your opinion? Is this just my code or is this somewhat normal? If it is so sensitive to these seed values, then the final results for the competition will be like a lottery!\n"
    },
    {
      "id": 3063135,
      "postDate": "2024-12-04T07:22:07.363Z",
      "rawMarkdown": "",
      "votes": -1,
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 3063117,
      "author_name": "aldparis",
      "author_url": "",
      "post_date": "2024-12-04T07:05:34.463000",
      "content": "<p>If your predictions are less sensitive to random seeds on the public leaderboard compared to those produced by the best public notebooks, keep trusting your cross-validation !</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3063740,
          "author_name": "Alperen Duru",
          "author_url": "",
          "post_date": "2024-12-04T20:10:06.173000",
          "content": "<p>I mean, I have methods that can wander around 0.45 to 0.46 ish reliably. However, I am not sure if someone can get a persistent score with changing SEED values, especially for scores over 0.48. </p>\n<p>I am simply curious, is there anyone getting persistency in their scores for anything over 0.48 with changing seed values?</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3075370,
      "author_name": "Viktoria Melkumyan",
      "author_url": "",
      "post_date": "2024-12-18T18:48:03.093000",
      "content": "<p>I have faced the same issue with the same notebook you mentioned. However, it wasn't the case when running other top score notebooks. It’s completely normal for models, especially those involving randomness like ensemble methods, to show some variation when you change the seed. This may happen particularly when using models like TabNet, which rely on stochastic processes. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3063135,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-12-04T07:22:07.363000",
      "content": "",
      "votes": -1,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3063117": "If your predictions are less sensitive to random seeds on the public leaderboard compared to those produced by the best public notebooks, keep trusting your cross-validation !",
    "3075370": "I have faced the same issue with the same notebook you mentioned. However, it wasn't the case when running other top score notebooks. It’s completely normal for models, especially those involving randomness like ensemble methods, to show some variation when you change the seed. This may happen particularly when using models like TabNet, which rely on stochastic processes. ",
    "3062940": "Hi everyone, I am fairly new here and wanted to discuss the randomness in scores, especially in high places of the leaderboard. \n\nI am trying some type of ensembles method myself and it seems to be performing somewhat acceptable, however, I am not able to achieve above 0.49 with tuning, etc. Any time I try tuning, my score wanders around 0.465 to 0.48.\n\nI recently checked the published code in the \"Code\" section and found a code claiming to achieve 0.494 with including TabNet in ensembles with other boosting methods. So I copied and simply submitted the code and got 0.485.\n\nOk, maybe a little variation here which can be acceptable. However, when I change the SEED value, I am getting something like 0.45! Now, I started losing my belief in top people actually getting those high numbers and not sure if their code would achieve similar values for changing SEED values. \n\nWhat is your opinion? Is this just my code or is this somewhat normal? If it is so sensitive to these seed values, then the final results for the competition will be like a lottery!\n",
    "3063135": ""
  }
}