{
  "id": 397414,
  "title": "Check New Hidden Test Leakage",
  "url": "/competitions/predict-student-performance-from-game-play/discussion/397414",
  "author_name": "",
  "post_date": "2023-03-25T13:42:03.026414700Z",
  "votes": 6,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>To check if there's data leakage in the hidden test set, I create and submit a simple notebook <a href=\"https://www.kaggle.com/code/abaojiang/check-new-hidden-test-leakage\" target=\"_blank\">here</a>. The underlying concept is summarized below:</p>\n<ol>\n<li>Check if there are multiple <code>level_group</code>s exist in one iteration.<ul>\n<li>If <code>True</code>, submit with <strong>all-one prediction</strong>.</li>\n<li>If <code>False</code>, submit with <strong>all-zero prediction</strong>.</li></ul></li>\n<li>Submit an <strong>all-zero prediction</strong> baseline.</li>\n</ol>\n<p>The first step is done in version 2 and the second one in version 3. Both of them obtain LB score 0.216. Hence, the assumption that each iteration only provides data of <strong>one <code>level_group</code> of one session</strong> is confirmed. However, multiple game plays still exist in some sessions, and more discussions can be found <a href=\"https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202\" target=\"_blank\">here</a>.</p>\n<p>If there's any mistake, please let me know. Thanks a lot!</p>",
  "messages": [
    {
      "id": "2196556",
      "postDate": "03/25/2023 13:42:03",
      "content": "<p>Hi everyone,</p>\n<p>To check if there's data leakage in the hidden test set, I create and submit a simple notebook <a href=\"https://www.kaggle.com/code/abaojiang/check-new-hidden-test-leakage\" target=\"_blank\">here</a>. The underlying concept is summarized below:</p>\n<ol>\n<li>Check if there are multiple <code>level_group</code>s exist in one iteration.<ul>\n<li>If <code>True</code>, submit with <strong>all-one prediction</strong>.</li>\n<li>If <code>False</code>, submit with <strong>all-zero prediction</strong>.</li></ul></li>\n<li>Submit an <strong>all-zero prediction</strong> baseline.</li>\n</ol>\n<p>The first step is done in version 2 and the second one in version 3. Both of them obtain LB score 0.216. Hence, the assumption that each iteration only provides data of <strong>one <code>level_group</code> of one session</strong> is confirmed. However, multiple game plays still exist in some sessions, and more discussions can be found <a href=\"https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202\" target=\"_blank\">here</a>.</p>\n<p>If there's any mistake, please let me know. Thanks a lot!</p>",
      "rawMarkdown": "Hi everyone,\n\nTo check if there's data leakage in the hidden test set, I create and submit a simple notebook [here](https://www.kaggle.com/code/abaojiang/check-new-hidden-test-leakage). The underlying concept is summarized below:\n1. Check if there are multiple `level_group`s exist in one iteration.\n    * If `True`, submit with **all-one prediction**.\n    * If `False`, submit with **all-zero prediction**.\n2. Submit an **all-zero prediction** baseline.\n\nThe first step is done in version 2 and the second one in version 3. Both of them obtain LB score 0.216. Hence, the assumption that each iteration only provides data of **one `level_group` of one session** is confirmed. However, multiple game plays still exist in some sessions, and more discussions can be found [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202).\n\nIf there's any mistake, please let me know. Thanks a lot!",
      "votes": null
    },
    {
      "id": "2197831",
      "postDate": "03/26/2023 13:22:49",
      "content": "<blockquote>\n  <p>multiple game plays still exist in some sessions</p>\n</blockquote>\n<p>Would you mind explaining what you mean by this?</p>",
      "rawMarkdown": ">multiple game plays still exist in some sessions\n\nWould you mind explaining what you mean by this?",
      "votes": null
    },
    {
      "id": "2197878",
      "postDate": "03/26/2023 14:12:00",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/hoangnguyen719\" target=\"_blank\">@hoangnguyen719</a>,</p>\n<p>If I don't miss anything, <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> is the first to discover this issue, as discussed in <a href=\"https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959\" target=\"_blank\">this thread</a>. This can be discovered in some sessions with <strong>reversed <code>level</code></strong> or <strong>reversed <code>level_group</code></strong> phenomenon. If your're interested in detailed implementation, you can refer to section 3 of my notebook <a href=\"https://www.kaggle.com/code/abaojiang/eda-on-game-progress\" target=\"_blank\">EDA on Game Progress\n</a>.</p>\n<p>Hope this helps, thanks for your reply!</p>",
      "rawMarkdown": "Hi @hoangnguyen719,\n\nIf I don't miss anything, @cdeotte is the first to discover this issue, as discussed in [this thread](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959). This can be discovered in some sessions with **reversed `level`** or **reversed `level_group`** phenomenon. If your're interested in detailed implementation, you can refer to section 3 of my notebook [EDA on Game Progress\n](https://www.kaggle.com/code/abaojiang/eda-on-game-progress).\n\nHope this helps, thanks for your reply!",
      "votes": null
    },
    {
      "id": "2201262",
      "postDate": "03/29/2023 07:11:19",
      "content": "<p>I checked and in the new test set, each batch seemed to have</p>\n<ul>\n<li>Only one <code>session_id</code></li>\n<li>Only one <code>level_group</code></li>\n<li>Only one <code>event_name=checkpoint</code></li>\n</ul>\n<p>Therefore I don't think the test set is having the same issue as train set, i.e. an user playing AND COMPLETING a chapter for more than once (but they can restart in the middle of a chapter - playing twice but <code>checkpoint</code> only once).</p>",
      "rawMarkdown": "I checked and in the new test set, each batch seemed to have\n- Only one `session_id`\n- Only one `level_group`\n- Only one `event_name=checkpoint`\n\nTherefore I don't think the test set is having the same issue as train set, i.e. an user playing AND COMPLETING a chapter for more than once (but they can restart in the middle of a chapter - playing twice but `checkpoint` only once).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2197831,
      "author_name": "hoangnguyen719",
      "author_url": "",
      "post_date": "03/26/2023 13:22:49",
      "content": "<blockquote>\n  <p>multiple game plays still exist in some sessions</p>\n</blockquote>\n<p>Would you mind explaining what you mean by this?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2197878,
          "author_name": "abaojiang",
          "author_url": "",
          "post_date": "03/26/2023 14:12:00",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/hoangnguyen719\" target=\"_blank\">@hoangnguyen719</a>,</p>\n<p>If I don't miss anything, <a href=\"https://www.kaggle.com/cdeotte\" target=\"_blank\">@cdeotte</a> is the first to discover this issue, as discussed in <a href=\"https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959\" target=\"_blank\">this thread</a>. This can be discovered in some sessions with <strong>reversed <code>level</code></strong> or <strong>reversed <code>level_group</code></strong> phenomenon. If your're interested in detailed implementation, you can refer to section 3 of my notebook <a href=\"https://www.kaggle.com/code/abaojiang/eda-on-game-progress\" target=\"_blank\">EDA on Game Progress\n</a>.</p>\n<p>Hope this helps, thanks for your reply!</p>",
          "votes": null,
          "replies": [
            {
              "id": 2201262,
              "author_name": "hoangnguyen719",
              "author_url": "",
              "post_date": "03/29/2023 07:11:19",
              "content": "<p>I checked and in the new test set, each batch seemed to have</p>\n<ul>\n<li>Only one <code>session_id</code></li>\n<li>Only one <code>level_group</code></li>\n<li>Only one <code>event_name=checkpoint</code></li>\n</ul>\n<p>Therefore I don't think the test set is having the same issue as train set, i.e. an user playing AND COMPLETING a chapter for more than once (but they can restart in the middle of a chapter - playing twice but <code>checkpoint</code> only once).</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2196556": "Hi everyone,\n\nTo check if there's data leakage in the hidden test set, I create and submit a simple notebook [here](https://www.kaggle.com/code/abaojiang/check-new-hidden-test-leakage). The underlying concept is summarized below:\n1. Check if there are multiple `level_group`s exist in one iteration.\n    * If `True`, submit with **all-one prediction**.\n    * If `False`, submit with **all-zero prediction**.\n2. Submit an **all-zero prediction** baseline.\n\nThe first step is done in version 2 and the second one in version 3. Both of them obtain LB score 0.216. Hence, the assumption that each iteration only provides data of **one `level_group` of one session** is confirmed. However, multiple game plays still exist in some sessions, and more discussions can be found [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202).\n\nIf there's any mistake, please let me know. Thanks a lot!",
    "2197831": ">multiple game plays still exist in some sessions\n\nWould you mind explaining what you mean by this?",
    "2197878": "Hi @hoangnguyen719,\n\nIf I don't miss anything, @cdeotte is the first to discover this issue, as discussed in [this thread](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959). This can be discovered in some sessions with **reversed `level`** or **reversed `level_group`** phenomenon. If your're interested in detailed implementation, you can refer to section 3 of my notebook [EDA on Game Progress\n](https://www.kaggle.com/code/abaojiang/eda-on-game-progress).\n\nHope this helps, thanks for your reply!",
    "2201262": "I checked and in the new test set, each batch seemed to have\n- Only one `session_id`\n- Only one `level_group`\n- Only one `event_name=checkpoint`\n\nTherefore I don't think the test set is having the same issue as train set, i.e. an user playing AND COMPLETING a chapter for more than once (but they can restart in the middle of a chapter - playing twice but `checkpoint` only once)."
  },
  "source": "meta"
}