{
  "id": 552557,
  "title": "The shake-up was originally predictable and avoidable?",
  "url": "/competitions/child-mind-institute-problematic-internet-use/discussion/552557",
  "author_name": "",
  "post_date": "2024-12-20T09:15:31.906715500Z",
  "votes": 1,
  "comment_count": 3,
  "views": 0,
  "content": "<p>The huge shake-up performs like ICR competition, almost all high score on LB lost, but it was predictable.<br>\nThe public notebooks were all based on a ensemble version of three solutions, and the higher only changed few parameters. But if you submit the three solutions separately, you will find the data leakage source.</p>\n<table>\n<thead>\n<tr>\n<th>version</th>\n<th>mean-qwk</th>\n<th>optimized-qwk</th>\n<th>LB</th>\n<th>PB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>submission1</td>\n<td>0.4753</td>\n<td>0.535</td>\n<td>0.361</td>\n<td>0.266</td>\n</tr>\n<tr>\n<td>submission3</td>\n<td>0.3803</td>\n<td>0.450</td>\n<td>0.460</td>\n<td>0.445</td>\n</tr>\n</tbody>\n</table>\n<p>The first part of it had lowest score on LB but had highest score on CV, the data leakage is too obvious and it is not a wise choice to blend it. The biggest difference of it is use autoencoder to encode time-series data. However, the time-series data only has 996 rows and training set only has 3960 rows. Without enough data, autoencoder shows more randomness than correctness. <br>\nThe third part of public notebooks use the SimpleImputer to handle missing values. When training set is small, the traditional imputer seems more trustworthy. <br>\nIs my opinion correct?</p>",
  "messages": [
    {
      "id": "3076794",
      "postDate": "12/20/2024 09:15:31",
      "content": "<p>The huge shake-up performs like ICR competition, almost all high score on LB lost, but it was predictable.<br>\nThe public notebooks were all based on a ensemble version of three solutions, and the higher only changed few parameters. But if you submit the three solutions separately, you will find the data leakage source.</p>\n<table>\n<thead>\n<tr>\n<th>version</th>\n<th>mean-qwk</th>\n<th>optimized-qwk</th>\n<th>LB</th>\n<th>PB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>submission1</td>\n<td>0.4753</td>\n<td>0.535</td>\n<td>0.361</td>\n<td>0.266</td>\n</tr>\n<tr>\n<td>submission3</td>\n<td>0.3803</td>\n<td>0.450</td>\n<td>0.460</td>\n<td>0.445</td>\n</tr>\n</tbody>\n</table>\n<p>The first part of it had lowest score on LB but had highest score on CV, the data leakage is too obvious and it is not a wise choice to blend it. The biggest difference of it is use autoencoder to encode time-series data. However, the time-series data only has 996 rows and training set only has 3960 rows. Without enough data, autoencoder shows more randomness than correctness. <br>\nThe third part of public notebooks use the SimpleImputer to handle missing values. When training set is small, the traditional imputer seems more trustworthy. <br>\nIs my opinion correct?</p>",
      "rawMarkdown": "The huge shake-up performs like ICR competition, almost all high score on LB lost, but it was predictable.\nThe public notebooks were all based on a ensemble version of three solutions, and the higher only changed few parameters. But if you submit the three solutions separately, you will find the data leakage source.\n| version | mean-qwk | optimized-qwk | LB | PB |\n| --- | --- |\n| submission1 | 0.4753 | 0.535 | 0.361 | 0.266 |\n| submission3 | 0.3803 | 0.450 | 0.460 | 0.445 |\n\nThe first part of it had lowest score on LB but had highest score on CV, the data leakage is too obvious and it is not a wise choice to blend it. The biggest difference of it is use autoencoder to encode time-series data. However, the time-series data only has 996 rows and training set only has 3960 rows. Without enough data, autoencoder shows more randomness than correctness. \nThe third part of public notebooks use the SimpleImputer to handle missing values. When training set is small, the traditional imputer seems more trustworthy. \nIs my opinion correct?",
      "votes": null
    },
    {
      "id": "3077103",
      "postDate": "12/20/2024 15:10:42",
      "content": "<p>Predictable and unavoidable. The problem is not that 'most people used bad public notebook and if they were smart and not used it they would not shake' (there are a lot of very smart people on Kaggle, most of them would not submit random bad ensembles), but rather: 'a bad notebook got by pure luck to to top public LB, showing that the variance by luck is too high, hence unavoidable shake'. </p>",
      "rawMarkdown": "Predictable and unavoidable. The problem is not that 'most people used bad public notebook and if they were smart and not used it they would not shake' (there are a lot of very smart people on Kaggle, most of them would not submit random bad ensembles), but rather: 'a bad notebook got by pure luck to to top public LB, showing that the variance by luck is too high, hence unavoidable shake'.",
      "votes": null
    },
    {
      "id": "3077124",
      "postDate": "12/20/2024 15:32:08",
      "content": "<p>Shake up was quite predictable. Hence very few grandmasters in this competition. Most advanced kagglers tend to avoid those kind of lottery competitions. I spend a few days on this competition, but did not even submit, once I saw how folds are behaving compared to each other.</p>",
      "rawMarkdown": "Shake up was quite predictable. Hence very few grandmasters in this competition. Most advanced kagglers tend to avoid those kind of lottery competitions. I spend a few days on this competition, but did not even submit, once I saw how folds are behaving compared to each other.",
      "votes": null
    },
    {
      "id": "3077128",
      "postDate": "12/20/2024 15:36:32",
      "content": "<p>Wise decision. The providing data of this competition is too terrible and spending too much time on it is not that worthy.</p>",
      "rawMarkdown": "Wise decision. The providing data of this competition is too terrible and spending too much time on it is not that worthy.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3077103,
      "author_name": "shlomoron",
      "author_url": "",
      "post_date": "12/20/2024 15:10:42",
      "content": "<p>Predictable and unavoidable. The problem is not that 'most people used bad public notebook and if they were smart and not used it they would not shake' (there are a lot of very smart people on Kaggle, most of them would not submit random bad ensembles), but rather: 'a bad notebook got by pure luck to to top public LB, showing that the variance by luck is too high, hence unavoidable shake'. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3077124,
      "author_name": "christofhenkel",
      "author_url": "",
      "post_date": "12/20/2024 15:32:08",
      "content": "<p>Shake up was quite predictable. Hence very few grandmasters in this competition. Most advanced kagglers tend to avoid those kind of lottery competitions. I spend a few days on this competition, but did not even submit, once I saw how folds are behaving compared to each other.</p>",
      "votes": null,
      "replies": [
        {
          "id": 3077128,
          "author_name": "qufangcq",
          "author_url": "",
          "post_date": "12/20/2024 15:36:32",
          "content": "<p>Wise decision. The providing data of this competition is too terrible and spending too much time on it is not that worthy.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3076794": "The huge shake-up performs like ICR competition, almost all high score on LB lost, but it was predictable.\nThe public notebooks were all based on a ensemble version of three solutions, and the higher only changed few parameters. But if you submit the three solutions separately, you will find the data leakage source.\n| version | mean-qwk | optimized-qwk | LB | PB |\n| --- | --- |\n| submission1 | 0.4753 | 0.535 | 0.361 | 0.266 |\n| submission3 | 0.3803 | 0.450 | 0.460 | 0.445 |\n\nThe first part of it had lowest score on LB but had highest score on CV, the data leakage is too obvious and it is not a wise choice to blend it. The biggest difference of it is use autoencoder to encode time-series data. However, the time-series data only has 996 rows and training set only has 3960 rows. Without enough data, autoencoder shows more randomness than correctness. \nThe third part of public notebooks use the SimpleImputer to handle missing values. When training set is small, the traditional imputer seems more trustworthy. \nIs my opinion correct?",
    "3077103": "Predictable and unavoidable. The problem is not that 'most people used bad public notebook and if they were smart and not used it they would not shake' (there are a lot of very smart people on Kaggle, most of them would not submit random bad ensembles), but rather: 'a bad notebook got by pure luck to to top public LB, showing that the variance by luck is too high, hence unavoidable shake'.",
    "3077124": "Shake up was quite predictable. Hence very few grandmasters in this competition. Most advanced kagglers tend to avoid those kind of lottery competitions. I spend a few days on this competition, but did not even submit, once I saw how folds are behaving compared to each other.",
    "3077128": "Wise decision. The providing data of this competition is too terrible and spending too much time on it is not that worthy."
  },
  "source": "meta"
}