{
  "id": 499505,
  "title": "Test set has been changed?",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/499505",
  "author_name": "minhtu.mt.mt",
  "post_date": "2024-05-02T03:08:13.072000",
  "votes": 6,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hi everyone,<br>\nI noticed that some of my successful (and scored) submissions now threw exception when I re-run them. The notebooks only run about 3 minutes before throwing exception, which means they get error during the feature engineering process. The weird thing is that I tried to submit older submissions with less feature engineering and they scored successfully, only submissions with my latest engineerings raise error. I dont know exactly which feature is changed (or removed), but I'm sure that the test set (or at least the submission set-up) has been changed.<br>\nSo I think the host changed somethings in the test set silently, which can relate to some recent leakage reports?<br>\nIs there anyone having the same issue with me?</p>",
  "messages": [
    {
      "id": 2787955,
      "postDate": "2024-05-02T03:08:13.073Z",
      "content": "<p>Hi everyone,<br>\nI noticed that some of my successful (and scored) submissions now threw exception when I re-run them. The notebooks only run about 3 minutes before throwing exception, which means they get error during the feature engineering process. The weird thing is that I tried to submit older submissions with less feature engineering and they scored successfully, only submissions with my latest engineerings raise error. I dont know exactly which feature is changed (or removed), but I'm sure that the test set (or at least the submission set-up) has been changed.<br>\nSo I think the host changed somethings in the test set silently, which can relate to some recent leakage reports?<br>\nIs there anyone having the same issue with me?</p>",
      "rawMarkdown": "Hi everyone,\nI noticed that some of my successful (and scored) submissions now threw exception when I re-run them. The notebooks only run about 3 minutes before throwing exception, which means they get error during the feature engineering process. The weird thing is that I tried to submit older submissions with less feature engineering and they scored successfully, only submissions with my latest engineerings raise error. I dont know exactly which feature is changed (or removed), but I'm sure that the test set (or at least the submission set-up) has been changed.\nSo I think the host changed somethings in the test set silently, which can relate to some recent leakage reports?\nIs there anyone having the same issue with me?",
      "votes": 5
    },
    {
      "id": 2788317,
      "postDate": "2024-05-02T07:23:02.897Z",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/minhtu123\" target=\"_blank\">@minhtu123</a> ,<br>\nThere was no change in test set after the competition restart (beginning of March).</p>",
      "rawMarkdown": "Hi @minhtu123 ,\nThere was no change in test set after the competition restart (beginning of March).",
      "votes": 3,
      "replies": [
        {
          "id": 2788459,
          "postDate": "2024-05-02T08:45:40.667Z",
          "content": "<p>Thank for your reply.<br>\nIt's messy for me now :(</p>",
          "rawMarkdown": "Thank for your reply.\nIt's messy for me now :(",
          "replies": [
            {
              "id": 2788744,
              "postDate": "2024-05-02T11:59:39Z",
              "rawMarkdown": "",
              "votes": -4,
              "isDeleted": true
            },
            {
              "id": 2790646,
              "postDate": "2024-05-03T09:21:45.323Z",
              "content": "<p>The logs only show the success versions of the notebooks when running on the sample test dataset. If they show the information when running on full test data, people will hack it easily :D</p>",
              "rawMarkdown": "The logs only show the success versions of the notebooks when running on the sample test dataset. If they show the information when running on full test data, people will hack it easily :D",
              "votes": 1
            },
            {
              "id": 2791721,
              "postDate": "2024-05-03T20:15:18.233Z",
              "content": "<p>my bad, I ment logs for local runs, for submissions yeah, no logs :(</p>",
              "rawMarkdown": "my bad, I ment logs for local runs, for submissions yeah, no logs :("
            }
          ]
        }
      ]
    },
    {
      "id": 2792388,
      "postDate": "2024-05-04T07:43:18.373Z",
      "content": "<p>I can't submit either.<br>\nsave version, run commit shows success<br>\nUnable to submit</p>",
      "rawMarkdown": "I can't submit either.\nsave version, run commit shows success\nUnable to submit"
    },
    {
      "id": 2788312,
      "postDate": "2024-05-02T07:20:26.310Z",
      "content": "<p>I have no issue at all. My feature engineering has been relatively very similar from long ago.</p>",
      "rawMarkdown": "I have no issue at all. My feature engineering has been relatively very similar from long ago."
    }
  ],
  "comments": [
    {
      "id": 2788317,
      "author_name": "Tomas Jelinek",
      "author_url": "",
      "post_date": "2024-05-02T07:23:02.897000",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/minhtu123\" target=\"_blank\">@minhtu123</a> ,<br>\nThere was no change in test set after the competition restart (beginning of March).</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2788459,
          "author_name": "minhtu.mt.mt",
          "author_url": "",
          "post_date": "2024-05-02T08:45:40.667000",
          "content": "<p>Thank for your reply.<br>\nIt's messy for me now :(</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2788744,
              "author_name": "",
              "author_url": "",
              "post_date": "2024-05-02T11:59:39",
              "content": "",
              "votes": -4,
              "replies": []
            },
            {
              "id": 2790646,
              "author_name": "minhtu.mt.mt",
              "author_url": "",
              "post_date": "2024-05-03T09:21:45.323000",
              "content": "<p>The logs only show the success versions of the notebooks when running on the sample test dataset. If they show the information when running on full test data, people will hack it easily :D</p>",
              "votes": 1,
              "replies": []
            },
            {
              "id": 2791721,
              "author_name": "Andrey Chankin",
              "author_url": "",
              "post_date": "2024-05-03T20:15:18.233000",
              "content": "<p>my bad, I ment logs for local runs, for submissions yeah, no logs :(</p>",
              "votes": 0,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2792388,
      "author_name": "kawa develop",
      "author_url": "",
      "post_date": "2024-05-04T07:43:18.373000",
      "content": "<p>I can't submit either.<br>\nsave version, run commit shows success<br>\nUnable to submit</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2788312,
      "author_name": "Iqbal Syah Akbar",
      "author_url": "",
      "post_date": "2024-05-02T07:20:26.310000",
      "content": "<p>I have no issue at all. My feature engineering has been relatively very similar from long ago.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2787955": "Hi everyone,\nI noticed that some of my successful (and scored) submissions now threw exception when I re-run them. The notebooks only run about 3 minutes before throwing exception, which means they get error during the feature engineering process. The weird thing is that I tried to submit older submissions with less feature engineering and they scored successfully, only submissions with my latest engineerings raise error. I dont know exactly which feature is changed (or removed), but I'm sure that the test set (or at least the submission set-up) has been changed.\nSo I think the host changed somethings in the test set silently, which can relate to some recent leakage reports?\nIs there anyone having the same issue with me?",
    "2788317": "Hi @minhtu123 ,\nThere was no change in test set after the competition restart (beginning of March).",
    "2792388": "I can't submit either.\nsave version, run commit shows success\nUnable to submit",
    "2788312": "I have no issue at all. My feature engineering has been relatively very similar from long ago."
  }
}