{
  "id": 413847,
  "title": "Test Data Concatentation || Leak?",
  "url": "/competitions/predict-student-performance-from-game-play/discussion/413847",
  "author_name": "",
  "post_date": "2023-05-30T10:29:53.458831500Z",
  "votes": 6,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hello, I was wondering if it is legal to concatenate test to get info about previous sessions.</p>\n<p>I mean, I can do something like</p>\n<pre><code> jo_wilder_310\n pandas  pd\n\n\nenv = jo_wilder_310.make_env()\niter_test = env.iter_test()\n\nquestions_map = {\n    :   (, ),\n    :  (, ),\n    : (, ),\n}\nall_prev_lvls = {\n    :   (),\n    :  (, ),\n    : (, , ),\n}\n\nour_test = pd.DataFrame()\n\n idx, (test, sample_submission)  (iter_test):\n    our_test = pd.concat([our_test, test])\n    session_ids = test[].unique()\n    lvl_groups = test[].(all_prev_lvls).unique()\n\n    sample_submission = model.predict(\n        our_test[\n            (our_test[].isin(session_ids)) \\\n            &amp; (our_test[].isin(lvl_groups))\n        ]\n    )\n\n    env.predict(sample_submission)\n</code></pre>\n<p>and predict using more data. Is it possible and does it improve the scores?</p>",
  "messages": [
    {
      "id": "2280754",
      "postDate": "05/30/2023 10:29:53",
      "content": "<p>Hello, I was wondering if it is legal to concatenate test to get info about previous sessions.</p>\n<p>I mean, I can do something like</p>\n<pre><code> jo_wilder_310\n pandas  pd\n\n\nenv = jo_wilder_310.make_env()\niter_test = env.iter_test()\n\nquestions_map = {\n    :   (, ),\n    :  (, ),\n    : (, ),\n}\nall_prev_lvls = {\n    :   (),\n    :  (, ),\n    : (, , ),\n}\n\nour_test = pd.DataFrame()\n\n idx, (test, sample_submission)  (iter_test):\n    our_test = pd.concat([our_test, test])\n    session_ids = test[].unique()\n    lvl_groups = test[].(all_prev_lvls).unique()\n\n    sample_submission = model.predict(\n        our_test[\n            (our_test[].isin(session_ids)) \\\n            &amp; (our_test[].isin(lvl_groups))\n        ]\n    )\n\n    env.predict(sample_submission)\n</code></pre>\n<p>and predict using more data. Is it possible and does it improve the scores?</p>",
      "rawMarkdown": "Hello, I was wondering if it is legal to concatenate test to get info about previous sessions.\n\nI mean, I can do something like\n\n```python\nimport jo_wilder_310\nimport pandas as pd\n\n\nenv = jo_wilder_310.make_env()\niter_test = env.iter_test()\n\nquestions_map = {\n    '0-4':   (1, 4),\n    '5-12':  (4, 14),\n    '13-22': (14, 19),\n}\nall_prev_lvls = {\n    '0-4':   ('0-4'),\n    '5-12':  ('0-4', '5-12'),\n    '13-22': ('0-4', '5-12', '13-22'),\n}\n\nour_test = pd.DataFrame()\n\nfor idx, (test, sample_submission) in enumerate(iter_test):\n    our_test = pd.concat([our_test, test])\n    session_ids = test['session_id'].unique()\n    lvl_groups = test['level_group'].map(all_prev_lvls).unique()\n    \n    sample_submission = model.predict(\n        our_test[\n            (our_test['session_id'].isin(session_ids)) \\\n            & (our_test['level_group'].isin(lvl_groups))\n        ]\n    )\n\n    env.predict(sample_submission)\n```\n\nand predict using more data. Is it possible and does it improve the scores?",
      "votes": null
    },
    {
      "id": "2280955",
      "postDate": "05/30/2023 13:30:22",
      "content": "<p>It is legal. It depends on how you handle your whole model, I believe if you do it properly it will increase your performance since you are using the same model but with more data. </p>",
      "rawMarkdown": "It is legal. It depends on how you handle your whole model, I believe if you do it properly it will increase your performance since you are using the same model but with more data.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2280955,
      "author_name": "ataraxian",
      "author_url": "",
      "post_date": "05/30/2023 13:30:22",
      "content": "<p>It is legal. It depends on how you handle your whole model, I believe if you do it properly it will increase your performance since you are using the same model but with more data. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2280754": "Hello, I was wondering if it is legal to concatenate test to get info about previous sessions.\n\nI mean, I can do something like\n\n```python\nimport jo_wilder_310\nimport pandas as pd\n\n\nenv = jo_wilder_310.make_env()\niter_test = env.iter_test()\n\nquestions_map = {\n    '0-4':   (1, 4),\n    '5-12':  (4, 14),\n    '13-22': (14, 19),\n}\nall_prev_lvls = {\n    '0-4':   ('0-4'),\n    '5-12':  ('0-4', '5-12'),\n    '13-22': ('0-4', '5-12', '13-22'),\n}\n\nour_test = pd.DataFrame()\n\nfor idx, (test, sample_submission) in enumerate(iter_test):\n    our_test = pd.concat([our_test, test])\n    session_ids = test['session_id'].unique()\n    lvl_groups = test['level_group'].map(all_prev_lvls).unique()\n    \n    sample_submission = model.predict(\n        our_test[\n            (our_test['session_id'].isin(session_ids)) \\\n            & (our_test['level_group'].isin(lvl_groups))\n        ]\n    )\n\n    env.predict(sample_submission)\n```\n\nand predict using more data. Is it possible and does it improve the scores?",
    "2280955": "It is legal. It depends on how you handle your whole model, I believe if you do it properly it will increase your performance since you are using the same model but with more data."
  },
  "source": "meta"
}