{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Exolore the supplemental data\n\n- As mentioned in [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202), there are additional logs for this game. I tried parse it and check what they are.\n\n- In conclusion, we have to take care of this supplements because of \n    1. some json data has invalid forms\n    2. the data elements are different from what we know\n    3. supplemental data has label data in the logs","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport polars as pl\nfrom sklearn.metrics import f1_score\nimport re","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:13:03.123084Z","iopub.execute_input":"2023-03-24T14:13:03.123611Z","iopub.status.idle":"2023-03-24T14:13:03.131380Z","shell.execute_reply.started":"2023-03-24T14:13:03.123571Z","shell.execute_reply":"2023-03-24T14:13:03.129815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## What I tried first: json.loads()","metadata":{}},{"cell_type":"markdown","source":"Honestly I want to use json.loads parsing method. \nHowever I could not debug invalid json data...\n**So I directly extract information from raw texts stupidly....**\n\n**<font color=\"red\"> If you did parse in better ways, please let me know </font>**","metadata":{}},{"cell_type":"code","source":"# from HP, I downloaded event data from 20191101_to_20191130 \nsupplemental = pl.read_csv(\"/kaggle/input/jow-supplemental-data/JOWILDER_20191101_to_20191130_10bbaaf_events.tsv\",sep=\"\\t\")\n\nsupplemental.head()","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:13:03.134187Z","iopub.execute_input":"2023-03-24T14:13:03.135346Z","iopub.status.idle":"2023-03-24T14:13:04.063701Z","shell.execute_reply.started":"2023-03-24T14:13:03.135283Z","shell.execute_reply":"2023-03-24T14:13:04.062768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some important features are included in \"event_data\"","metadata":{}},{"cell_type":"code","source":"supplemental[\"event_data\"].head().to_list()[0]","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:13:04.064988Z","iopub.execute_input":"2023-03-24T14:13:04.069271Z","iopub.status.idle":"2023-03-24T14:13:04.076681Z","shell.execute_reply.started":"2023-03-24T14:13:04.069218Z","shell.execute_reply":"2023-03-24T14:13:04.075786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### The below code is my JUNKs \n\n- These are trying to parse with json module somehow.\n- the stringified json-formatted event data is sometimes bad\n    - e.g.: `\"{'text': 'how 'bout you?'}\"`","metadata":{}},{"cell_type":"code","source":"import ast\nimport re\nimport json\n# data = supplemental[\"event_data\"].iloc[1]\n\ndef load_json(new_str):\n    try:\n        return json.loads(new_str), True\n    except:\n        return new_str, False\n        \ndictionary = {}\nfor idx, event in enumerate(supplemental[\"event_data\"]):\n\n#     new_str = re.sub('([a-zA-Z])\"([a-zA-Z])', r\"\\1'\\2\", event.replace(\"'\", '\"'))\n#     new_str = re.sub(r'True', r'\"true\"', event)\n#     new_str = re.sub(r'False', r'\"false\"', new_str)\n#     new_str = re.sub(r'\"\"([a-zA-Z])', r'\"\\1', new_str)\n#     new_str = re.sub(r'([a-zA-Z]|\\.)\"\"', r'\\1\"', new_str)\n\n    try:\n        d = json.loads(event)\n#         print(d)\n        dictionary[idx] = d\n\n    except json.JSONDecodeError as e:\n        print(new_str)\n        raise e\n#         new_str, done = load_json(new_str)\n#         if not done:\n#             if '\"smells\"' in new_str:\n#                 new_str = re.sub(r'\"smells\"', r'SMELLS', new_str)\n#                 new_str, done = load_json(new_str)\n                \n#             new_str = new_str.replace(\"\\\\\", '').replace('\\\"', \"'\")\n#             new_str, done = load_json(new_str)\n#         if not done:\n#             new_str = new_str.replace('How \"bout', \"How 'bout\")\n#             new_str, done = load_json(new_str)\n#     except : \n#         new_str = new_str.replace('How \"bout', \"How 'bout\").replace(\"\\\\\", '').replace('\\\"', \"'\")\n#         d = ast.literal_eval(new_str)\n#         print(d)\n#     json.loads(new_str\n\n#                \n#                .replace('It\"s', \"It's\")\n#                .replace('women\\\"s', \"women\\'s\")\n#                .replace('Let\"s', \"Let's\")\n#                .replace('can\"t', \"can't\")\n#                .replace('I\"ll', \"I'll\").replace('I\"m', \"I'm\"), strict=False)\n    ","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:13:04.078741Z","iopub.execute_input":"2023-03-24T14:13:04.079860Z","iopub.status.idle":"2023-03-24T14:13:04.132175Z","shell.execute_reply.started":"2023-03-24T14:13:04.079822Z","shell.execute_reply":"2023-03-24T14:13:04.129771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As you know `json.loads` is Expecting property name enclosed in double quotes...\n\nBUT we have to take care of invalid jsons as follows.","metadata":{}},{"cell_type":"code","source":"for idx, event in enumerate(supplemental[\"event_data\"]):\n    \n    # after convert single quotes to double ones,\n    # try re-convert double quotes which is pinched by strings like I'll\n    new_str = re.sub('([a-zA-Z])\"([a-zA-Z])', r\"\\1'\\2\", event.replace(\"'\", '\"'))\n    new_str = re.sub(r'True', r'\"true\"', new_str)\n    new_str = re.sub(r'False', r'\"false\"', new_str)\n#     How \"bout\n    new_str = re.sub(r'How \"bout', 'How \\'bout', new_str)\n    # He\\\"s wrong \n    new_str = re.sub('He\\\"s wrong', 'He\\'s wrong', new_str)\n    try:\n        d = json.loads(new_str)\n#         print(d)\n        dictionary[idx] = d\n\n    except json.JSONDecodeError as e:\n        print(new_str)\n        raise e\n","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:13:08.740147Z","iopub.execute_input":"2023-03-24T14:13:08.740609Z","iopub.status.idle":"2023-03-24T14:13:08.800244Z","shell.execute_reply.started":"2023-03-24T14:13:08.740569Z","shell.execute_reply":"2023-03-24T14:13:08.798950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"not so good at regex handling... I could not treat with this problems...","metadata":{}},{"cell_type":"markdown","source":"## What I did \n\n### Using regex, I extracted the key information in a brute-force way.\n\n- each `event` stringified json has some key like 'persistent_session_id','level' and so on.\n- so I tried search the key first of all.\n- then, I extract the strings `from the key` - `to next\"'\" ` (`to next \":\"`  for 'text' key)\n\n\n....What a stupid way!!!","metadata":{}},{"cell_type":"code","source":"\n\n\nfrom dataclasses import dataclass, field\nfrom typing import List, Optional\n\n@dataclass\nclass Pattern:\n    key_name: str\n    key_patterns: Optional = field(default_factory=list)\n    exclude_patterns: Optional = field(default_factory=list)\n# example\np = Pattern(key_name=\"'event_custom'\", key_patterns=[\"([a-zA-Z._0-9 ]+)\"])\n\ndef get_values_by_pattern(p: Pattern, new_str:str, debug=False):\n    assert isinstance(p, Pattern)\n    g = re.search(pattern=p.key_name, string=new_str)\n    if g is None: return None\n    str_from_values = new_str[g.span()[1]+2:]\n    key_patterns = getattr(p, \"key_patterns\", [])\n    if key_patterns:\n        for _pattern in key_patterns:\n            res = re.search(pattern=_pattern, string=str_from_values)\n            str_from_values = res.group()\n            if debug: print(res.group())\n        return res.group()\n    exclude_patterns = getattr(p, \"exclude_patterns\", [])\n    # Naming weird\n    # 途中\n    if exclude_patterns:\n        for _pattern in exclude_patterns:\n            # import pdb; pdb.set_trace()\n            res = re.search(pattern=_pattern, string=str_from_values)\n            if debug: print(str_from_values[:res.span()[0]])\n        return str_from_values[:res.span()[0]]\n    \n    \n# get_values_by_pattern(Pattern(\"'room_coor'\", [\"([\\[\\]a-zA-Z,._0-9 -]+)\"]), new_str)\n\n# get_values_by_pattern(Pattern(\"'screen_coor'\", [\"([\\[\\]a-zA-Z,._0-9 -]+)\"]), new_str)\n\n# get_values_by_pattern(Pattern(\"'type'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]), new_str)\n\n# get_values_by_pattern(Pattern(\"'name'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]), new_str)\n\n# get_values_by_pattern(Pattern(\"'interacted_fqid'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]), new_str)\n\n# get_values_by_pattern(Pattern(\"'persistent_session_id'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]), new_str)\n\n# get_values_by_pattern(Pattern(\"'fqid'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]), new_str)\n\n# get_values_by_pattern(p, new_str)\n\n# elapsed_time\npattern_list = [\n    Pattern(\"'persistent_session_id'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    Pattern(\"'interacted_fqid'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    # Pattern(\"'screen_coor'\", [\"([\\[\\]a-zA-Z,._0-9 -]+)\"]),\n    Pattern(\"'type'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    Pattern(\"'subtype'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    Pattern(\"'room_fqid'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    Pattern(\"'text_fqid'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    Pattern(\"'name'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    Pattern(\"'screen_coor'\", [\"([\\[\\]a-zA-Z,._0-9 -]+)\"]),\n    Pattern(\"'room_coor'\", [\"([\\[\\]a-zA-Z,._0-9 -]+)\"]),\n    Pattern(\"'server_time'\", [\"([\\[\\]a-zA-Z,._0-9-:]+)\"]),\n    Pattern(\"'fqid'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    Pattern(\"'text'\", exclude_patterns=[\", '[a-zA-Z]+':\"]),\n    Pattern(\"'level'\", [\"([\\[\\]a-zA-Z,._0-9 ]+)\"]),\n    \n]\n# for p in pattern_list:\n#     print(\"====\", p.key_name, \"====\")\n#     get_values_by_pattern(p, new_str, debug=True)\n\n\n\ndictionary = {}\nimport time \nfor idx, event in enumerate(supplemental[\"event_data\"]):\n    dictionary[idx] = {}\n    # print(event)\n    for p in pattern_list:\n        # print(\"====\", p.key_name, \"====\")\n        dictionary[idx][p.key_name] = get_values_by_pattern(p, event)\n    # time.sleep(10)","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:14:01.535256Z","iopub.execute_input":"2023-03-24T14:14:01.535668Z","iopub.status.idle":"2023-03-24T14:14:15.600333Z","shell.execute_reply.started":"2023-03-24T14:14:01.535633Z","shell.execute_reply":"2023-03-24T14:14:15.598965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This way is so stupid, I know. But somehow it seems to work.","metadata":{}},{"cell_type":"code","source":"supplement_df = pd.DataFrame.from_records(dictionary).T\nsupplement_df.head()\n","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:15:30.792125Z","iopub.execute_input":"2023-03-24T14:15:30.792546Z","iopub.status.idle":"2023-03-24T14:15:41.426629Z","shell.execute_reply.started":"2023-03-24T14:15:30.792510Z","shell.execute_reply":"2023-03-24T14:15:41.425656Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"supplement_df.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:15:49.777750Z","iopub.execute_input":"2023-03-24T14:15:49.778371Z","iopub.status.idle":"2023-03-24T14:15:49.785138Z","shell.execute_reply.started":"2023-03-24T14:15:49.778333Z","shell.execute_reply":"2023-03-24T14:15:49.783847Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nsupplement_df.columns = [col[1:-1] for col in supplement_df.columns]\n\nsupplement_df[\"persistent_session_id\"].unique().shape\n\nimport matplotlib.pyplot as plt\n\ng = supplement_df.groupby(\"persistent_session_id\")[\"text\"].agg(\"nunique\").sort_values().plot(kind=\"bar\")\nplt.xticks([])\n\n(supplement_df.groupby(\"persistent_session_id\")[\"text\"].agg(\"nunique\") < 10).index\n\nsupplement_df[supplement_df[\"persistent_session_id\"]=='19080211455847756']\nsupplement_df[supplement_df[\"type\"] == \"endgame\"].room_fqid.unique()","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:17:27.531065Z","iopub.execute_input":"2023-03-24T14:17:27.531514Z","iopub.status.idle":"2023-03-24T14:17:29.073447Z","shell.execute_reply.started":"2023-03-24T14:17:27.531474Z","shell.execute_reply":"2023-03-24T14:17:29.072322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"supplement_df.groupby(\"persistent_session_id\")[\"level\"].max()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-03-24T14:17:29.558857Z","iopub.execute_input":"2023-03-24T14:17:29.559727Z","iopub.status.idle":"2023-03-24T14:17:29.731771Z","shell.execute_reply.started":"2023-03-24T14:17:29.559676Z","shell.execute_reply":"2023-03-24T14:17:29.730901Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Notes\n- Some sessions could not arrive at the end and drop out in the middle of the game.","metadata":{}},{"cell_type":"code","source":"supplement_df[\"level\"]= supplement_df[\"level\"].str.findall(\"\\d{1,2}\").map(lambda x: int(x[0]))","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-03-24T14:17:30.411843Z","iopub.execute_input":"2023-03-24T14:17:30.412701Z","iopub.status.idle":"2023-03-24T14:17:31.431664Z","shell.execute_reply.started":"2023-03-24T14:17:30.412660Z","shell.execute_reply":"2023-03-24T14:17:31.430289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_max_level = supplement_df.groupby(\"persistent_session_id\")[\"level\"].max()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-03-24T14:17:31.433805Z","iopub.execute_input":"2023-03-24T14:17:31.434188Z","iopub.status.idle":"2023-03-24T14:17:31.500535Z","shell.execute_reply.started":"2023-03-24T14:17:31.434154Z","shell.execute_reply":"2023-03-24T14:17:31.498962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_max_level.shape","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:17:31.504447Z","iopub.execute_input":"2023-03-24T14:17:31.504832Z","iopub.status.idle":"2023-03-24T14:17:31.510597Z","shell.execute_reply.started":"2023-03-24T14:17:31.504795Z","shell.execute_reply":"2023-03-24T14:17:31.509678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(session_max_level == 23).sum()","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:17:31.854340Z","iopub.execute_input":"2023-03-24T14:17:31.855018Z","iopub.status.idle":"2023-03-24T14:17:31.861421Z","shell.execute_reply.started":"2023-03-24T14:17:31.854979Z","shell.execute_reply":"2023-03-24T14:17:31.860400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_max_level.hist(bins=30)","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-03-24T14:17:32.363454Z","iopub.execute_input":"2023-03-24T14:17:32.364118Z","iopub.status.idle":"2023-03-24T14:17:32.613239Z","shell.execute_reply.started":"2023-03-24T14:17:32.364079Z","shell.execute_reply":"2023-03-24T14:17:32.612290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- we can see sessions which achieved level==23 are rare from the above graph.\n\n- among 368 sessions, only 56 sessions achieved the goal at the month","metadata":{}},{"cell_type":"code","source":"supplement_df['type'].value_counts()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-03-24T14:17:54.812796Z","iopub.execute_input":"2023-03-24T14:17:54.813181Z","iopub.status.idle":"2023-03-24T14:17:54.860049Z","shell.execute_reply.started":"2023-03-24T14:17:54.813146Z","shell.execute_reply":"2023-03-24T14:17:54.859022Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Note \n\n- `type` definition is not the same as data in this comeptition\n    - but we can infer what they are.","metadata":{}},{"cell_type":"code","source":"supplement_df[supplement_df[\"type\"] == \"checkpoint\"]\n\nsupplement_df[supplement_df['text'].str.contains(\"That doesn't prove your case.\").fillna(False)].head()","metadata":{"tags":[],"execution":{"iopub.status.busy":"2023-03-24T14:19:08.163280Z","iopub.execute_input":"2023-03-24T14:19:08.163718Z","iopub.status.idle":"2023-03-24T14:19:08.347439Z","shell.execute_reply.started":"2023-03-24T14:19:08.163679Z","shell.execute_reply":"2023-03-24T14:19:08.345650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Note \n\n- we can see `this does'nt prove your case` in text. This is displayed when user response was wrong for each question.\n","metadata":{}},{"cell_type":"markdown","source":"### What I guess \n\n- This dataset is \"raw\" data of dataset in this competition.\n\n- We can construct labels of our tasks from `this does'nt prove your case`, but it takes a bit time to do.\n","metadata":{}},{"cell_type":"code","source":"#  I did not check where hover_duration is.","metadata":{"execution":{"iopub.status.busy":"2023-03-24T14:13:04.150305Z","iopub.status.idle":"2023-03-24T14:13:04.150712Z","shell.execute_reply.started":"2023-03-24T14:13:04.150511Z","shell.execute_reply":"2023-03-24T14:13:04.150533Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# maybe it'll continue.","metadata":{},"execution_count":null,"outputs":[]}]}