{
  "id": 420190,
  "title": "10th place solution(tereka part)",
  "url": "/competitions/predict-student-performance-from-game-play/discussion/420190",
  "author_name": "",
  "post_date": "2023-06-29T14:59:15.827764300Z",
  "votes": 30,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Thank you for host and my team members <a href=\"https://www.kaggle.com/deepkun1995\" target=\"_blank\">@deepkun1995</a>, <a href=\"https://www.kaggle.com/ryotak12\" target=\"_blank\">@ryotak12</a>, <a href=\"https://www.kaggle.com/yurimaeda\" target=\"_blank\">@yurimaeda</a> <a href=\"https://www.kaggle.com/shu421\" target=\"_blank\">@shu421</a> <br>\nI appreciate for team merge and great work. it's a very great experience.</p>\n<p>our team solution is here(<a href=\"https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420132\" target=\"_blank\">https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420132</a>)</p>\n<p>this topic I share my work.</p>\n<h1>TL;DR</h1>\n<ol>\n<li>XGBoost(Public 0.701/Private 0.702/CV 0.703)</li>\n<li>use Feature Importance</li>\n<li>use before q answer probability</li>\n</ol>\n<h1>Solution</h1>\n<h2>Feature Engineering</h2>\n<p>it's the most important. <br>\nI use many combinations and bingo feature. I generate 100000 features<br>\nafter I filter features that have many pattern.</p>\n<pre><code>def (x, grp, use_extra, feature_suffix):\n    aggs = [\n        pl.().().(f),\n        *[pl.().((pl.().str.(c))).().(f) for c in\n          DIALOGS],\n        *[pl.().((pl.().str.(c))).().(f) for c in\n          DIALOGS],\n        *[pl.().((pl.().str.(c))).().(f) for c in\n          DIALOGS],\n        *[pl.().((pl.().str.(c))).().(f) for c in\n          DIALOGS],\n        *[pl.().((pl.().str.(c))).().(f) for c\n          in DIALOGS],\n\n        *[pl.().(pl.()==c).().(f) for c in\n          UNIQUE_TEXTS],\n\n\n        *[pl.(c).().().(f) for c in CATS],\n        *[pl.(c).().(f) for c in []],\n        *[pl.(c).().(f) for c in NUMS],\n        *[pl.(c).().(f) for c in NUMS],\n        *[pl.(c).().(f) for c in NUMS],\n        *[pl.(c).().(f) for c in NUMS],\n        *[pl.(c).().(f) for c in NUMS],\n        pl.().().(f),\n        *[pl.(c).().().(f) for c in CATS],\n\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n#         *[pl.().(pl.()==c).().(f) for c in LEVELS],\n        *[pl.().(pl.()==c).().(f) for c in room_fqids],\n#         *[pl.().(pl.()==c).().(f) for c in TEXT_FQIDS],\n\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature], \n\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n\n        *[pl.().(pl.()==c).(pl.()==c2).().(f) for c in name_feature for c2 in event_name_feature],\n        *[pl.().(pl.()==c).(pl.()==c2).().(f) for c in name_feature for c2 in event_name_feature],\n\n\n        *[pl.().(pl.()==c).().(f) for c in FQID_LIST],\n        *[pl.().(pl.()==c).().(f) for c in FQID_LIST],\n        *[pl.().(pl.()==c).().(f) for c in FQID_LIST],\n        *[pl.().(pl.()==c).().(f) for c in FQID_LIST],\n        pl.().(pl.() != None).(lambda x : .(x)).(f),\n    ]\n    if grp == :\n        grp_agg = [\n            *[pl.().(pl.()==c).().(f) for c in TEXT_FQIDS_GRP0],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n\n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP0 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP0 for ev_c in event_name_feature],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP0 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP0 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp0 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp0 for ev_c in event_name_feature], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0], \n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n\n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n\n             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp0], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp0], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP0], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP0],  \n        ]\n    elif grp==:\n        grp_agg = [\n            *[pl.().(pl.()==c).().(f) for c in TEXT_FQIDS_GRP1],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP1 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP1 for ev_c in event_name_feature],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP1 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP1 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp1 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp1 for ev_c in event_name_feature], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1], \n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n\n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1 for c2 in FQID_LIST], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1 for c2 in FQID_LIST], \n\n             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp1], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp1], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP1], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP1],  \n        ]\n    elif grp==:\n        grp_agg = [\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP2 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP2 for ev_c in event_name_feature],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP2 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP2 for ev_c in room_fqids],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp2 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp2 for ev_c in event_name_feature], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2], \n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n\n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2 for c2 in FQID_LIST], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2 for c2 in FQID_LIST], \n\n             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp2], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp2], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP2], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP2],\n        ]\n    aggs = aggs + grp_agg\n    df = x.([], maintain_order=True).(aggs).()\n\n    if use_extra:\n        if grp==:\n            aggs = [\n                pl.().((pl.()==)|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().((pl.()==)|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.()).(),\n            ]\n            tmp = x.([], maintain_order=True).(aggs).()\n            df = df.(tmp, on=, how=)\n\n        if grp==:\n            aggs = [\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.() if s.()&gt; else ).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.() if s.()&gt; else ).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.() if s.()&gt; else ).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.() if s.()&gt; else ).()\n            ]\n            tmp = x.([], maintain_order=True).(aggs).()\n            df = df.(tmp, on=, how=)\n    return df\n</code></pre>\n<h2>All features training</h2>\n<p>I train models to use all features. All model CV is about 0.695.</p>\n<h2>Training Filtered Features</h2>\n<p>I filter feature importance from all feature models. I get the top 300/600/900(each label group)<br>\nand difference q(e.g. q1 features q2 are different)<br>\nI train Filtered Features, it scored CV:0.701</p>\n<h2>Add before the correct probability</h2>\n<p>Finally, I add before-answer probability(before section), for example, in q2 prediction I use q1 probability. q3 is q1 and q2 probability.<br>\nIts strategy pushes up CV 0.701-&gt;0.7032.(Public0.701/Private 0.702)</p>",
  "messages": [
    {
      "id": "2322897",
      "postDate": "06/29/2023 14:59:15",
      "content": "<p>Thank you for host and my team members <a href=\"https://www.kaggle.com/deepkun1995\" target=\"_blank\">@deepkun1995</a>, <a href=\"https://www.kaggle.com/ryotak12\" target=\"_blank\">@ryotak12</a>, <a href=\"https://www.kaggle.com/yurimaeda\" target=\"_blank\">@yurimaeda</a> <a href=\"https://www.kaggle.com/shu421\" target=\"_blank\">@shu421</a> <br>\nI appreciate for team merge and great work. it's a very great experience.</p>\n<p>our team solution is here(<a href=\"https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420132\" target=\"_blank\">https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420132</a>)</p>\n<p>this topic I share my work.</p>\n<h1>TL;DR</h1>\n<ol>\n<li>XGBoost(Public 0.701/Private 0.702/CV 0.703)</li>\n<li>use Feature Importance</li>\n<li>use before q answer probability</li>\n</ol>\n<h1>Solution</h1>\n<h2>Feature Engineering</h2>\n<p>it's the most important. <br>\nI use many combinations and bingo feature. I generate 100000 features<br>\nafter I filter features that have many pattern.</p>\n<pre><code>def (x, grp, use_extra, feature_suffix):\n    aggs = [\n        pl.().().(f),\n        *[pl.().((pl.().str.(c))).().(f) for c in\n          DIALOGS],\n        *[pl.().((pl.().str.(c))).().(f) for c in\n          DIALOGS],\n        *[pl.().((pl.().str.(c))).().(f) for c in\n          DIALOGS],\n        *[pl.().((pl.().str.(c))).().(f) for c in\n          DIALOGS],\n        *[pl.().((pl.().str.(c))).().(f) for c\n          in DIALOGS],\n\n        *[pl.().(pl.()==c).().(f) for c in\n          UNIQUE_TEXTS],\n\n\n        *[pl.(c).().().(f) for c in CATS],\n        *[pl.(c).().(f) for c in []],\n        *[pl.(c).().(f) for c in NUMS],\n        *[pl.(c).().(f) for c in NUMS],\n        *[pl.(c).().(f) for c in NUMS],\n        *[pl.(c).().(f) for c in NUMS],\n        *[pl.(c).().(f) for c in NUMS],\n        pl.().().(f),\n        *[pl.(c).().().(f) for c in CATS],\n\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n#         *[pl.().(pl.()==c).().(f) for c in LEVELS],\n        *[pl.().(pl.()==c).().(f) for c in room_fqids],\n#         *[pl.().(pl.()==c).().(f) for c in TEXT_FQIDS],\n\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).(, ).(f) for c in event_name_feature], \n\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n        *[pl.().(pl.()==c).().(f) for c in event_name_feature],\n\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n        *[pl.().(pl.()==c).().(f) for c in name_feature],\n\n        *[pl.().(pl.()==c).(pl.()==c2).().(f) for c in name_feature for c2 in event_name_feature],\n        *[pl.().(pl.()==c).(pl.()==c2).().(f) for c in name_feature for c2 in event_name_feature],\n\n\n        *[pl.().(pl.()==c).().(f) for c in FQID_LIST],\n        *[pl.().(pl.()==c).().(f) for c in FQID_LIST],\n        *[pl.().(pl.()==c).().(f) for c in FQID_LIST],\n        *[pl.().(pl.()==c).().(f) for c in FQID_LIST],\n        pl.().(pl.() != None).(lambda x : .(x)).(f),\n    ]\n    if grp == :\n        grp_agg = [\n            *[pl.().(pl.()==c).().(f) for c in TEXT_FQIDS_GRP0],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0],\n\n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP0 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP0 for ev_c in event_name_feature],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP0 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP0 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp0 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp0 for ev_c in event_name_feature], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0], \n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp0], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP0],\n\n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n\n             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp0], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp0], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP0], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP0],  \n        ]\n    elif grp==:\n        grp_agg = [\n            *[pl.().(pl.()==c).().(f) for c in TEXT_FQIDS_GRP1],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP1 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP1 for ev_c in event_name_feature],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP1 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP1 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp1 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp1 for ev_c in event_name_feature], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1], \n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp1], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP1],\n\n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1 for c2 in FQID_LIST], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP1 for c2 in FQID_LIST], \n\n             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp1], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp1], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP1], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP1],  \n        ]\n    elif grp==:\n        grp_agg = [\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP2 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP2 for ev_c in event_name_feature],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP2 for ev_c in room_fqids], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in LEVEL_GRP2 for ev_c in room_fqids],\n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp2 for ev_c in event_name_feature], \n            *[pl.().(pl.()==c).(pl.()==ev_c).().(pl.Float32).(f) for c in room_fqids_grp2 for ev_c in event_name_feature], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2], \n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in room_fqids_grp2], \n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n            *[pl.().(pl.()==c).().(pl.Float32).(f) for c in LEVEL_GRP2],\n\n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2 for c2 in FQID_LIST], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in TEXT_FQIDS_GRP2 for c2 in FQID_LIST], \n\n             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp2], \n            *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c in FQID_LIST for c2 in room_fqids_grp2], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP2], \n#             *[pl.().(pl.()==c).(pl.()==c2).().(pl.Float32).(f) for c, c2 in FQID_TEXT_PAIR_GRP2],\n        ]\n    aggs = aggs + grp_agg\n    df = x.([], maintain_order=True).(aggs).()\n\n    if use_extra:\n        if grp==:\n            aggs = [\n                pl.().((pl.()==)|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().((pl.()==)|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.()).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.()).(),\n            ]\n            tmp = x.([], maintain_order=True).(aggs).()\n            df = df.(tmp, on=, how=)\n\n        if grp==:\n            aggs = [\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.() if s.()&gt; else ).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.() if s.()&gt; else ).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.() if s.()&gt; else ).(),\n                pl.().(((pl.()==)&amp;(pl.()==))|(pl.()==)).(lambda s: s.()-s.() if s.()&gt; else ).()\n            ]\n            tmp = x.([], maintain_order=True).(aggs).()\n            df = df.(tmp, on=, how=)\n    return df\n</code></pre>\n<h2>All features training</h2>\n<p>I train models to use all features. All model CV is about 0.695.</p>\n<h2>Training Filtered Features</h2>\n<p>I filter feature importance from all feature models. I get the top 300/600/900(each label group)<br>\nand difference q(e.g. q1 features q2 are different)<br>\nI train Filtered Features, it scored CV:0.701</p>\n<h2>Add before the correct probability</h2>\n<p>Finally, I add before-answer probability(before section), for example, in q2 prediction I use q1 probability. q3 is q1 and q2 probability.<br>\nIts strategy pushes up CV 0.701-&gt;0.7032.(Public0.701/Private 0.702)</p>",
      "rawMarkdown": "Thank you for host and my team members @deepkun1995, @ryotak12, @yurimaeda @shu421 \nI appreciate for team merge and great work. it's a very great experience.\n\nour team solution is here(https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420132)\n\nthis topic I share my work.\n\n# TL;DR\n1.  XGBoost(Public 0.701/Private 0.702/CV 0.703)\n2. use Feature Importance\n3. use before q answer probability\n\n# Solution\n## Feature Engineering\nit's the most important. \nI use many combinations and bingo feature. I generate 100000 features\nafter I filter features that have many pattern.\n\n```\ndef feature_engineer(x, grp, use_extra, feature_suffix):\n    aggs = [\n        pl.col(\"index\").count().alias(f\"session_number_{feature_suffix}\"),\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).mean().alias(f'word_mean_{c}_{feature_suffix}') for c in\n          DIALOGS],\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).std().alias(f'word_std_{c}_{feature_suffix}') for c in\n          DIALOGS],\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).max().alias(f'word_max_{c}_{feature_suffix}') for c in\n          DIALOGS],\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).sum().alias(f'word_sum_{c}_{feature_suffix}') for c in\n          DIALOGS],\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).median().alias(f'word_median_{c}_{feature_suffix}') for c\n          in DIALOGS],\n        \n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text\")==c).sum().alias(f'text_sum_{c}_{feature_suffix}') for c in\n          UNIQUE_TEXTS],\n        \n        \n        *[pl.col(c).drop_nulls().n_unique().alias(f\"{c}_unique_{feature_suffix}\") for c in CATS],\n        *[pl.col(c).sum().alias(f\"{c}_sum_{feature_suffix}\") for c in [\"elapsed_time_diff\"]],\n        *[pl.col(c).mean().alias(f\"{c}_mean_{feature_suffix}\") for c in NUMS],\n        *[pl.col(c).min().alias(f\"{c}_min_{feature_suffix}\") for c in NUMS],\n        *[pl.col(c).max().alias(f\"{c}_max_{feature_suffix}\") for c in NUMS],\n        *[pl.col(c).median().alias(f\"{c}_median_{feature_suffix}\") for c in NUMS],\n        *[pl.col(c).std().alias(f\"{c}_std_{feature_suffix}\") for c in NUMS],\n        pl.col(\"elapsed_time_diff\").sum().alias(f\"elapsed_time_diff_all_sum_{feature_suffix}\"),\n        *[pl.col(c).drop_nulls().count().alias(f\"{c}_cnt_{feature_suffix}\") for c in CATS],\n        \n        *[pl.col(\"index\").filter(pl.col(\"event_name\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in event_name_feature],\n#         *[pl.col(\"index\").filter(pl.col(\"level\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in LEVELS],\n        *[pl.col(\"index\").filter(pl.col(\"room_fqid\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in room_fqids],\n#         *[pl.col(\"index\").filter(pl.col(\"text\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in TEXT_FQIDS],\n\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).median().alias(f\"{c}_ET_median_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).mean().alias(f\"{c}_ET_mean_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).max().alias(f\"{c}_ET_max_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).min().alias(f\"{c}_ET_min_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).sum().alias(f\"{c}_ET_sum_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).std().alias(f\"{c}_ET_std_{feature_suffix}\") for c in event_name_feature],\n\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.1, \"nearest\").alias(f\"{c}_ET_quantile1_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.2, \"nearest\").alias(f\"{c}_ET_quantile2_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.4, \"nearest\").alias(f\"{c}_ET_quantile4_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.6, \"nearest\").alias(f\"{c}_ET_quantile6_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.8, \"nearest\").alias(f\"{c}_ET_quantile8_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.9, \"nearest\").alias(f\"{c}_ET_quantile9_{feature_suffix}\") for c in event_name_feature], \n        \n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).median().alias(f\"{c}_HD_median_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).mean().alias(f\"{c}_HD_mean_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).max().alias(f\"{c}_HD_max_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).min().alias(f\"{c}_HD_min_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).sum().alias(f\"{c}_HD_sum_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).std().alias(f\"{c}_HD_std_{feature_suffix}\") for c in event_name_feature],\n\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).median().alias(f\"{c}_ET_median_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).mean().alias(f\"{c}_ET_mean_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).max().alias(f\"{c}_ET_max_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).min().alias(f\"{c}_ET_min_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).std().alias(f\"{c}_ET_std_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).sum().alias(f\"{c}_ET_sum_{feature_suffix}\") for c in name_feature],\n\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).filter(pl.col(\"event_name\")==c2).mean().alias(f\"{c}_{c2}_ET_mean_{feature_suffix}\") for c in name_feature for c2 in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).filter(pl.col(\"event_name\")==c2).sum().alias(f\"{c}_{c2}_ET_sum_{feature_suffix}\") for c in name_feature for c2 in event_name_feature],\n\n        \n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).mean().alias(f\"{c}_ET_mean_{feature_suffix}\") for c in FQID_LIST],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).sum().alias(f\"{c}_ET_sum_{feature_suffix}\") for c in FQID_LIST],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).max().alias(f\"{c}_ET_max_{feature_suffix}\") for c in FQID_LIST],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).std().alias(f\"{c}_ET_std_{feature_suffix}\") for c in FQID_LIST],\n        pl.col(\"text\").filter(pl.col(\"text\") != None).apply(lambda x : \" \".join(x)).alias(f\"concat_text_{feature_suffix}\"),\n    ]\n    if grp == '0-4':\n        grp_agg = [\n            *[pl.col(\"index\").filter(pl.col(\"text\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_HD_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n\n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n            \n            *[pl.col(\"index\").filter(pl.col(\"level\")==c).count().cast(pl.Float32).alias(f\"{c}_cnt_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP0 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP0 for ev_c in event_name_feature],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP0 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP0 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp0 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp0 for ev_c in event_name_feature], \n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in room_fqids_grp0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in room_fqids_grp0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp0], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in room_fqids_grp0], \n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP0],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_HD_min_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in LEVEL_GRP0],\n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n            \n             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp0], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp0], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP0], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP0],  \n        ]\n    elif grp=='5-12':\n        grp_agg = [\n            *[pl.col(\"index\").filter(pl.col(\"text\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_HD_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            \n            *[pl.col(\"index\").filter(pl.col(\"level\")==c).count().cast(pl.Float32).alias(f\"{c}_cnt_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP1 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP1 for ev_c in event_name_feature],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP1 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP1 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp1 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp1 for ev_c in event_name_feature], \n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in room_fqids_grp1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in room_fqids_grp1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp1], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in room_fqids_grp1], \n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP1],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_HD_min_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in LEVEL_GRP1],\n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP1 for c2 in FQID_LIST], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP1 for c2 in FQID_LIST], \n\n             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp1], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp1], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP1], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP1],  \n        ]\n    elif grp=='13-22':\n        grp_agg = [\n            *[pl.col(\"index\").filter(pl.col(\"text\")==c).count().cast(pl.Float32).alias(f\"{c}_cnt_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_HD_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"index\").filter(pl.col(\"level\")==c).count().cast(pl.Float32).alias(f\"{c}_cnt_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP2 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP2 for ev_c in event_name_feature],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP2 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP2 for ev_c in room_fqids],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp2 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp2 for ev_c in event_name_feature], \n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in room_fqids_grp2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in room_fqids_grp2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp2], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in room_fqids_grp2], \n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP2],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_HD_min_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in LEVEL_GRP2],\n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP2 for c2 in FQID_LIST], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP2 for c2 in FQID_LIST], \n\n             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp2], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp2], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP2], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP2],\n        ]\n    aggs = aggs + grp_agg\n    df = x.groupby([\"session_id\"], maintain_order=True).agg(aggs).sort(\"session_id\")\n    \n    if use_extra:\n        if grp=='5-12':\n            aggs = [\n                pl.col(\"elapsed_time\").filter((pl.col(\"text\")==\"Here's the log book.\")|(pl.col(\"fqid\")=='logbook.page.bingo')).apply(lambda s: s.max()-s.min()).alias(\"logbook_bingo_duration\"),\n                pl.col(\"index\").filter((pl.col(\"text\")==\"Here's the log book.\")|(pl.col(\"fqid\")=='logbook.page.bingo')).apply(lambda s: s.max()-s.min()).alias(\"logbook_bingo_indexCount\"),\n                pl.col(\"elapsed_time\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='reader'))|(pl.col(\"fqid\")==\"reader.paper2.bingo\")).apply(lambda s: s.max()-s.min()).alias(\"reader_bingo_duration\"),\n                pl.col(\"index\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='reader'))|(pl.col(\"fqid\")==\"reader.paper2.bingo\")).apply(lambda s: s.max()-s.min()).alias(\"reader_bingo_indexCount\"),\n                pl.col(\"elapsed_time\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='journals'))|(pl.col(\"fqid\")==\"journals.pic_2.bingo\")).apply(lambda s: s.max()-s.min()).alias(\"journals_bingo_duration\"),\n                pl.col(\"index\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='journals'))|(pl.col(\"fqid\")==\"journals.pic_2.bingo\")).apply(lambda s: s.max()-s.min()).alias(\"journals_bingo_indexCount\"),\n            ]\n            tmp = x.groupby([\"session_id\"], maintain_order=True).agg(aggs).sort(\"session_id\")\n            df = df.join(tmp, on=\"session_id\", how='left')\n\n        if grp=='13-22':\n            aggs = [\n                pl.col(\"elapsed_time\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='reader_flag'))|(pl.col(\"fqid\")==\"tunic.library.microfiche.reader_flag.paper2.bingo\")).apply(lambda s: s.max()-s.min() if s.len()>0 else 0).alias(\"reader_flag_duration\"),\n                pl.col(\"index\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='reader_flag'))|(pl.col(\"fqid\")==\"tunic.library.microfiche.reader_flag.paper2.bingo\")).apply(lambda s: s.max()-s.min() if s.len()>0 else 0).alias(\"reader_flag_indexCount\"),\n                pl.col(\"elapsed_time\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='journals_flag'))|(pl.col(\"fqid\")==\"journals_flag.pic_0.bingo\")).apply(lambda s: s.max()-s.min() if s.len()>0 else 0).alias(\"journalsFlag_bingo_duration\"),\n                pl.col(\"index\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='journals_flag'))|(pl.col(\"fqid\")==\"journals_flag.pic_0.bingo\")).apply(lambda s: s.max()-s.min() if s.len()>0 else 0).alias(\"journalsFlag_bingo_indexCount\")\n            ]\n            tmp = x.groupby([\"session_id\"], maintain_order=True).agg(aggs).sort(\"session_id\")\n            df = df.join(tmp, on=\"session_id\", how='left')\n    return df\n```\n\n## All features training\nI train models to use all features. All model CV is about 0.695.\n\n## Training Filtered Features\nI filter feature importance from all feature models. I get the top 300/600/900(each label group)\nand difference q(e.g. q1 features q2 are different)\nI train Filtered Features, it scored CV:0.701\n\n## Add before the correct probability\nFinally, I add before-answer probability(before section), for example, in q2 prediction I use q1 probability. q3 is q1 and q2 probability.\nIts strategy pushes up CV 0.701->0.7032.(Public0.701/Private 0.702)",
      "votes": null
    },
    {
      "id": "2322909",
      "postDate": "06/29/2023 15:10:24",
      "content": "<p>Thanks for sharing. </p>\n<p>One question, before adding before the correct probability features, do you use 18 models for each question or use 3 models for each level_group?</p>",
      "rawMarkdown": "Thanks for sharing. \n\nOne question, before adding before the correct probability features, do you use 18 models for each question or use 3 models for each level_group?",
      "votes": null
    },
    {
      "id": "2322917",
      "postDate": "06/29/2023 15:17:34",
      "content": "<p>I use 18 models.<br>\neach question use one model(5folds).</p>\n<p>model -&gt; q0<br>\nmodel -&gt; q1<br>\nmodel -&gt; q2<br>\n….</p>\n<p>next stage<br>\nq1 = before filtered feature + q0 probability<br>\nq2 = before filtered feature + q0 probability + q1 probability<br>\nq3 = before filtered feature + q0 probability + q1 probability + q2 probability…..<br>\n….</p>",
      "rawMarkdown": "I use 18 models.\neach question use one model(5folds).\n\nmodel -> q0\nmodel -> q1\nmodel -> q2\n....\n\nnext stage\nq1 = before filtered feature + q0 probability\nq2 = before filtered feature + q0 probability + q1 probability\nq3 = before filtered feature + q0 probability + q1 probability + q2 probability.....\n....",
      "votes": null
    },
    {
      "id": "2325365",
      "postDate": "07/01/2023 09:24:14",
      "content": "<p>Thank you for sharing. I have one question.<br>\n\"difference q(e.g. q1 features q2 are different)\"<br>\nmeans that the q2 features are the original q2 features without the ones present in the q1 features?</p>",
      "rawMarkdown": "Thank you for sharing. I have one question.\n\"difference q(e.g. q1 features q2 are different)\"\nmeans that the q2 features are the original q2 features without the ones present in the q1 features?",
      "votes": null
    },
    {
      "id": "2328307",
      "postDate": "07/03/2023 13:49:25",
      "content": "<p>no.<br>\nwhen q1 prediction use feature1, feature2, feature3, feature10<br>\nwhen q2 prediction use feature2, feature3, feature5, feature8</p>",
      "rawMarkdown": "no.\nwhen q1 prediction use feature1, feature2, feature3, feature10\nwhen q2 prediction use feature2, feature3, feature5, feature8",
      "votes": null
    },
    {
      "id": "2564473",
      "postDate": "12/17/2023 06:23:37",
      "content": "<p><a href=\"https://www.kaggle.com/tereka\" target=\"_blank\">@tereka</a> Hi, I wish you could see my late question. Could you please tell me the methods you used for filtering features from 100k to less than 1k?</p>",
      "rawMarkdown": "tereka Hi, I wish you could see my late question. Could you please tell me the methods you used for filtering features from 100k to less than 1k?",
      "votes": null
    },
    {
      "id": "2565070",
      "postDate": "12/17/2023 15:55:27",
      "content": "<p>I used feature importance(model)</p>",
      "rawMarkdown": "I used feature importance(model)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2322909,
      "author_name": "hookman",
      "author_url": "",
      "post_date": "06/29/2023 15:10:24",
      "content": "<p>Thanks for sharing. </p>\n<p>One question, before adding before the correct probability features, do you use 18 models for each question or use 3 models for each level_group?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2322917,
          "author_name": "tereka",
          "author_url": "",
          "post_date": "06/29/2023 15:17:34",
          "content": "<p>I use 18 models.<br>\neach question use one model(5folds).</p>\n<p>model -&gt; q0<br>\nmodel -&gt; q1<br>\nmodel -&gt; q2<br>\n….</p>\n<p>next stage<br>\nq1 = before filtered feature + q0 probability<br>\nq2 = before filtered feature + q0 probability + q1 probability<br>\nq3 = before filtered feature + q0 probability + q1 probability + q2 probability…..<br>\n….</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2325365,
      "author_name": "tyabanoamami",
      "author_url": "",
      "post_date": "07/01/2023 09:24:14",
      "content": "<p>Thank you for sharing. I have one question.<br>\n\"difference q(e.g. q1 features q2 are different)\"<br>\nmeans that the q2 features are the original q2 features without the ones present in the q1 features?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2328307,
          "author_name": "tereka",
          "author_url": "",
          "post_date": "07/03/2023 13:49:25",
          "content": "<p>no.<br>\nwhen q1 prediction use feature1, feature2, feature3, feature10<br>\nwhen q2 prediction use feature2, feature3, feature5, feature8</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2564473,
      "author_name": "takanashihumbert",
      "author_url": "",
      "post_date": "12/17/2023 06:23:37",
      "content": "<p><a href=\"https://www.kaggle.com/tereka\" target=\"_blank\">@tereka</a> Hi, I wish you could see my late question. Could you please tell me the methods you used for filtering features from 100k to less than 1k?</p>",
      "votes": null,
      "replies": [
        {
          "id": 2565070,
          "author_name": "tereka",
          "author_url": "",
          "post_date": "12/17/2023 15:55:27",
          "content": "<p>I used feature importance(model)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2322897": "Thank you for host and my team members @deepkun1995, @ryotak12, @yurimaeda @shu421 \nI appreciate for team merge and great work. it's a very great experience.\n\nour team solution is here(https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/420132)\n\nthis topic I share my work.\n\n# TL;DR\n1.  XGBoost(Public 0.701/Private 0.702/CV 0.703)\n2. use Feature Importance\n3. use before q answer probability\n\n# Solution\n## Feature Engineering\nit's the most important. \nI use many combinations and bingo feature. I generate 100000 features\nafter I filter features that have many pattern.\n\n```\ndef feature_engineer(x, grp, use_extra, feature_suffix):\n    aggs = [\n        pl.col(\"index\").count().alias(f\"session_number_{feature_suffix}\"),\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).mean().alias(f'word_mean_{c}_{feature_suffix}') for c in\n          DIALOGS],\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).std().alias(f'word_std_{c}_{feature_suffix}') for c in\n          DIALOGS],\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).max().alias(f'word_max_{c}_{feature_suffix}') for c in\n          DIALOGS],\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).sum().alias(f'word_sum_{c}_{feature_suffix}') for c in\n          DIALOGS],\n        *[pl.col(\"elapsed_time_diff\").filter((pl.col('text').str.contains(c))).median().alias(f'word_median_{c}_{feature_suffix}') for c\n          in DIALOGS],\n        \n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text\")==c).sum().alias(f'text_sum_{c}_{feature_suffix}') for c in\n          UNIQUE_TEXTS],\n        \n        \n        *[pl.col(c).drop_nulls().n_unique().alias(f\"{c}_unique_{feature_suffix}\") for c in CATS],\n        *[pl.col(c).sum().alias(f\"{c}_sum_{feature_suffix}\") for c in [\"elapsed_time_diff\"]],\n        *[pl.col(c).mean().alias(f\"{c}_mean_{feature_suffix}\") for c in NUMS],\n        *[pl.col(c).min().alias(f\"{c}_min_{feature_suffix}\") for c in NUMS],\n        *[pl.col(c).max().alias(f\"{c}_max_{feature_suffix}\") for c in NUMS],\n        *[pl.col(c).median().alias(f\"{c}_median_{feature_suffix}\") for c in NUMS],\n        *[pl.col(c).std().alias(f\"{c}_std_{feature_suffix}\") for c in NUMS],\n        pl.col(\"elapsed_time_diff\").sum().alias(f\"elapsed_time_diff_all_sum_{feature_suffix}\"),\n        *[pl.col(c).drop_nulls().count().alias(f\"{c}_cnt_{feature_suffix}\") for c in CATS],\n        \n        *[pl.col(\"index\").filter(pl.col(\"event_name\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in event_name_feature],\n#         *[pl.col(\"index\").filter(pl.col(\"level\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in LEVELS],\n        *[pl.col(\"index\").filter(pl.col(\"room_fqid\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in room_fqids],\n#         *[pl.col(\"index\").filter(pl.col(\"text\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in TEXT_FQIDS],\n\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).median().alias(f\"{c}_ET_median_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).mean().alias(f\"{c}_ET_mean_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).max().alias(f\"{c}_ET_max_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).min().alias(f\"{c}_ET_min_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).sum().alias(f\"{c}_ET_sum_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).std().alias(f\"{c}_ET_std_{feature_suffix}\") for c in event_name_feature],\n\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.1, \"nearest\").alias(f\"{c}_ET_quantile1_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.2, \"nearest\").alias(f\"{c}_ET_quantile2_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.4, \"nearest\").alias(f\"{c}_ET_quantile4_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.6, \"nearest\").alias(f\"{c}_ET_quantile6_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.8, \"nearest\").alias(f\"{c}_ET_quantile8_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"event_name\")==c).quantile(0.9, \"nearest\").alias(f\"{c}_ET_quantile9_{feature_suffix}\") for c in event_name_feature], \n        \n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).median().alias(f\"{c}_HD_median_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).mean().alias(f\"{c}_HD_mean_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).max().alias(f\"{c}_HD_max_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).min().alias(f\"{c}_HD_min_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).sum().alias(f\"{c}_HD_sum_{feature_suffix}\") for c in event_name_feature],\n        *[pl.col(\"hover_duration\").filter(pl.col(\"event_name\")==c).std().alias(f\"{c}_HD_std_{feature_suffix}\") for c in event_name_feature],\n\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).median().alias(f\"{c}_ET_median_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).mean().alias(f\"{c}_ET_mean_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).max().alias(f\"{c}_ET_max_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).min().alias(f\"{c}_ET_min_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).std().alias(f\"{c}_ET_std_{feature_suffix}\") for c in name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).sum().alias(f\"{c}_ET_sum_{feature_suffix}\") for c in name_feature],\n\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).filter(pl.col(\"event_name\")==c2).mean().alias(f\"{c}_{c2}_ET_mean_{feature_suffix}\") for c in name_feature for c2 in event_name_feature],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"name\")==c).filter(pl.col(\"event_name\")==c2).sum().alias(f\"{c}_{c2}_ET_sum_{feature_suffix}\") for c in name_feature for c2 in event_name_feature],\n\n        \n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).mean().alias(f\"{c}_ET_mean_{feature_suffix}\") for c in FQID_LIST],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).sum().alias(f\"{c}_ET_sum_{feature_suffix}\") for c in FQID_LIST],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).max().alias(f\"{c}_ET_max_{feature_suffix}\") for c in FQID_LIST],\n        *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).std().alias(f\"{c}_ET_std_{feature_suffix}\") for c in FQID_LIST],\n        pl.col(\"text\").filter(pl.col(\"text\") != None).apply(lambda x : \" \".join(x)).alias(f\"concat_text_{feature_suffix}\"),\n    ]\n    if grp == '0-4':\n        grp_agg = [\n            *[pl.col(\"index\").filter(pl.col(\"text\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_HD_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP0],\n\n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n            \n            *[pl.col(\"index\").filter(pl.col(\"level\")==c).count().cast(pl.Float32).alias(f\"{c}_cnt_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP0 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP0 for ev_c in event_name_feature],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP0 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP0 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp0 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp0 for ev_c in event_name_feature], \n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in room_fqids_grp0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in room_fqids_grp0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp0], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in room_fqids_grp0], \n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP0],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_HD_min_{feature_suffix}\") for c in LEVEL_GRP0],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in LEVEL_GRP0],\n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP0 for c2 in FQID_LIST], \n            \n             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp0], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp0], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP0], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP0],  \n        ]\n    elif grp=='5-12':\n        grp_agg = [\n            *[pl.col(\"index\").filter(pl.col(\"text\")==c).count().alias(f\"{c}_cnt_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_HD_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP1],\n            \n            *[pl.col(\"index\").filter(pl.col(\"level\")==c).count().cast(pl.Float32).alias(f\"{c}_cnt_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP1 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP1 for ev_c in event_name_feature],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP1 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP1 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp1 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp1 for ev_c in event_name_feature], \n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in room_fqids_grp1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in room_fqids_grp1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp1], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in room_fqids_grp1], \n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP1],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_HD_min_{feature_suffix}\") for c in LEVEL_GRP1],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in LEVEL_GRP1],\n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP1 for c2 in FQID_LIST], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP1 for c2 in FQID_LIST], \n\n             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp1], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp1], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP1], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP1],  \n        ]\n    elif grp=='13-22':\n        grp_agg = [\n            *[pl.col(\"index\").filter(pl.col(\"text\")==c).count().cast(pl.Float32).alias(f\"{c}_cnt_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_HD_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"text_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in TEXT_FQIDS_GRP2],\n            *[pl.col(\"index\").filter(pl.col(\"level\")==c).count().cast(pl.Float32).alias(f\"{c}_cnt_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP2 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP2 for ev_c in event_name_feature],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP2 for ev_c in room_fqids], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).filter(pl.col(\"room_fqid\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP2 for ev_c in room_fqids],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).sum().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp2 for ev_c in event_name_feature], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).filter(pl.col(\"event_name\")==ev_c).mean().cast(pl.Float32).alias(f\"{ev_c}_{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp2 for ev_c in event_name_feature], \n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in room_fqids_grp2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in room_fqids_grp2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in room_fqids_grp2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in room_fqids_grp2], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"room_fqid\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in room_fqids_grp2], \n            \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_ET_mean_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_ET_max_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_ET_min_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_ET_std_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"level\")==c).sum().cast(pl.Float32).alias(f\"{c}_ET_sum_{feature_suffix}\") for c in LEVEL_GRP2],\n        \n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).mean().cast(pl.Float32).alias(f\"{c}_HD_mean_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).max().cast(pl.Float32).alias(f\"{c}_HD_max_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).min().cast(pl.Float32).alias(f\"{c}_HD_min_{feature_suffix}\") for c in LEVEL_GRP2],\n            *[pl.col(\"hover_duration\").filter(pl.col(\"level\")==c).std().cast(pl.Float32).alias(f\"{c}_HD_std_{feature_suffix}\") for c in LEVEL_GRP2],\n\n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in TEXT_FQIDS_GRP2 for c2 in FQID_LIST], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in TEXT_FQIDS_GRP2 for c2 in FQID_LIST], \n\n             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp2], \n            *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"fqid\")==c).filter(pl.col(\"room_fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c in FQID_LIST for c2 in room_fqids_grp2], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).sum().cast(pl.Float32).alias(f\"{c2}_{c}_ET_sum_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP2], \n#             *[pl.col(\"elapsed_time_diff\").filter(pl.col(\"text_fqid\")==c).filter(pl.col(\"fqid\")==c2).mean().cast(pl.Float32).alias(f\"{c2}_{c}_ET_mean_{feature_suffix}\") for c, c2 in FQID_TEXT_PAIR_GRP2],\n        ]\n    aggs = aggs + grp_agg\n    df = x.groupby([\"session_id\"], maintain_order=True).agg(aggs).sort(\"session_id\")\n    \n    if use_extra:\n        if grp=='5-12':\n            aggs = [\n                pl.col(\"elapsed_time\").filter((pl.col(\"text\")==\"Here's the log book.\")|(pl.col(\"fqid\")=='logbook.page.bingo')).apply(lambda s: s.max()-s.min()).alias(\"logbook_bingo_duration\"),\n                pl.col(\"index\").filter((pl.col(\"text\")==\"Here's the log book.\")|(pl.col(\"fqid\")=='logbook.page.bingo')).apply(lambda s: s.max()-s.min()).alias(\"logbook_bingo_indexCount\"),\n                pl.col(\"elapsed_time\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='reader'))|(pl.col(\"fqid\")==\"reader.paper2.bingo\")).apply(lambda s: s.max()-s.min()).alias(\"reader_bingo_duration\"),\n                pl.col(\"index\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='reader'))|(pl.col(\"fqid\")==\"reader.paper2.bingo\")).apply(lambda s: s.max()-s.min()).alias(\"reader_bingo_indexCount\"),\n                pl.col(\"elapsed_time\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='journals'))|(pl.col(\"fqid\")==\"journals.pic_2.bingo\")).apply(lambda s: s.max()-s.min()).alias(\"journals_bingo_duration\"),\n                pl.col(\"index\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='journals'))|(pl.col(\"fqid\")==\"journals.pic_2.bingo\")).apply(lambda s: s.max()-s.min()).alias(\"journals_bingo_indexCount\"),\n            ]\n            tmp = x.groupby([\"session_id\"], maintain_order=True).agg(aggs).sort(\"session_id\")\n            df = df.join(tmp, on=\"session_id\", how='left')\n\n        if grp=='13-22':\n            aggs = [\n                pl.col(\"elapsed_time\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='reader_flag'))|(pl.col(\"fqid\")==\"tunic.library.microfiche.reader_flag.paper2.bingo\")).apply(lambda s: s.max()-s.min() if s.len()>0 else 0).alias(\"reader_flag_duration\"),\n                pl.col(\"index\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='reader_flag'))|(pl.col(\"fqid\")==\"tunic.library.microfiche.reader_flag.paper2.bingo\")).apply(lambda s: s.max()-s.min() if s.len()>0 else 0).alias(\"reader_flag_indexCount\"),\n                pl.col(\"elapsed_time\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='journals_flag'))|(pl.col(\"fqid\")==\"journals_flag.pic_0.bingo\")).apply(lambda s: s.max()-s.min() if s.len()>0 else 0).alias(\"journalsFlag_bingo_duration\"),\n                pl.col(\"index\").filter(((pl.col(\"event_name\")=='navigate_click')&(pl.col(\"fqid\")=='journals_flag'))|(pl.col(\"fqid\")==\"journals_flag.pic_0.bingo\")).apply(lambda s: s.max()-s.min() if s.len()>0 else 0).alias(\"journalsFlag_bingo_indexCount\")\n            ]\n            tmp = x.groupby([\"session_id\"], maintain_order=True).agg(aggs).sort(\"session_id\")\n            df = df.join(tmp, on=\"session_id\", how='left')\n    return df\n```\n\n## All features training\nI train models to use all features. All model CV is about 0.695.\n\n## Training Filtered Features\nI filter feature importance from all feature models. I get the top 300/600/900(each label group)\nand difference q(e.g. q1 features q2 are different)\nI train Filtered Features, it scored CV:0.701\n\n## Add before the correct probability\nFinally, I add before-answer probability(before section), for example, in q2 prediction I use q1 probability. q3 is q1 and q2 probability.\nIts strategy pushes up CV 0.701->0.7032.(Public0.701/Private 0.702)",
    "2322909": "Thanks for sharing. \n\nOne question, before adding before the correct probability features, do you use 18 models for each question or use 3 models for each level_group?",
    "2322917": "I use 18 models.\neach question use one model(5folds).\n\nmodel -> q0\nmodel -> q1\nmodel -> q2\n....\n\nnext stage\nq1 = before filtered feature + q0 probability\nq2 = before filtered feature + q0 probability + q1 probability\nq3 = before filtered feature + q0 probability + q1 probability + q2 probability.....\n....",
    "2325365": "Thank you for sharing. I have one question.\n\"difference q(e.g. q1 features q2 are different)\"\nmeans that the q2 features are the original q2 features without the ones present in the q1 features?",
    "2328307": "no.\nwhen q1 prediction use feature1, feature2, feature3, feature10\nwhen q2 prediction use feature2, feature3, feature5, feature8",
    "2564473": "tereka Hi, I wish you could see my late question. Could you please tell me the methods you used for filtering features from 100k to less than 1k?",
    "2565070": "I used feature importance(model)"
  },
  "source": "meta"
}