{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# EDA on Game Progress\n\n## Problem Statement\nDuring the journey of EDA, I find out that each sample can't be **uniquely identified** by the pre-assumed composite key (`session_id`, `index`), which should indicate **the specific event in the specific session**. In this [forum](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/384342#2134312), the competition host states that there might be some data errors. Hence, I wonder if this issue can be fixed.<br>\nFurthermore, as the general game progresses, `index` should be **monotonically increasing**, which preserves the **ordering nature** of events. Nonetheless, this property doesn't always hold.<br>\nFinally, `checkpoint` is an important indicator of question prompts, which should be followed by a **level-up**. But again, this assumption seems not valid in some cases. \n\n## About this Notebook\nIn this kernel, properties of `index` columns are explored. Then, a simple workaround is implemented to fix the aforementioned issue, which could facilitate better interpretation about the **sequential characteristics** of the data. In addition, `checkpoint`-related problems are discussed, which could help us come up with better ways to clean the data.\n\n<a id=\"toc\"></a>\n## Table of Contents\n* [1. Duplicated (`session_id`, `index`) Pairs](#dup_pk)\n* [2. Reversed Index](#reverse_idx)\n* [3. Reversed Level](#reverse_lv)\n* [4. Jumped Index](#jump_idx)\n* [5. Checkpoint Exploration](#ckpt)\n* [6. Abnormal Game Plays](#abnormal_game)\n* [7. Conclusion](#conclusion)\n\n## Acknowledgements\nSpecial thanks to @cdeotte 's response [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388751#2150691). As for detailed discussion about time series API, please see [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479).\n\n## Note\nBecause the competition host has announced the data update policy [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/396202), I decide to re-run this notebook to make exploration results aligned with the newest data. What's more, the competitors are allowed to use supplemental data collected from [this site](https://fielddaylab.wisc.edu/opengamedata/), but I haven't tried it out. Hence, I only use the updated `train.csv` in this notebook without any other open source data.\n\n\n## Import Packages","metadata":{}},{"cell_type":"code","source":"import gc\nimport os\nimport warnings\nfrom collections import defaultdict\nfrom typing import Any, Dict, Optional, Union\nfrom tqdm import tqdm\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom matplotlib.axes import Axes\n\nwarnings.simplefilter(\"ignore\")\nsns.set_style(\"darkgrid\")\npd.options.display.max_rows = None\npd.options.display.max_columns = None\ncolors = sns.color_palette(\"Set2\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:01:41.237381Z","iopub.execute_input":"2023-03-21T08:01:41.237854Z","iopub.status.idle":"2023-03-21T08:01:41.910415Z","shell.execute_reply.started":"2023-03-21T08:01:41.237757Z","shell.execute_reply":"2023-03-21T08:01:41.909114Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"INPUT_PATH = \"/kaggle/input/predict-student-performance-from-game-play\"","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:01:43.953883Z","iopub.execute_input":"2023-03-21T08:01:43.954308Z","iopub.status.idle":"2023-03-21T08:01:43.960013Z","shell.execute_reply.started":"2023-03-21T08:01:43.954273Z","shell.execute_reply":"2023-03-21T08:01:43.959082Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load Data\nLet's load data first! Considering the RAM constraint, we can load the data with **downcasted data types**. For instance, loading `train.csv` with the downcastes dtypes can reduce memory usage from **2.0+ GB** to about **603.4 MB**.\n\n**Note:** The following code snippet comes from [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/384359). All the credits should go to @sakvaua. Furthermore, for the concern about memory reduction, please see [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/384475).","metadata":{}},{"cell_type":"code","source":"dtypes = {\n    \"session_id\": \"category\",\n    \"elapsed_time\": np.int32,\n    \"event_name\": \"category\",\n    \"level\": np.uint8,\n    \"text\": \"category\",\n    \"level_group\": \"category\",\n}","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:01:48.825210Z","iopub.execute_input":"2023-03-21T08:01:48.825653Z","iopub.status.idle":"2023-03-21T08:01:48.830883Z","shell.execute_reply.started":"2023-03-21T08:01:48.825617Z","shell.execute_reply":"2023-03-21T08:01:48.829670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To further constraint memory usage, I only load necessay columns to do the analysis.","metadata":{}},{"cell_type":"code","source":"%%time\n\ncols_to_use = [\"session_id\", \"index\", \"elapsed_time\", \"event_name\",\n               \"level\", \"text\", \"level_group\"]\ntrain = pd.read_csv(os.path.join(INPUT_PATH, \"train.csv\"), usecols=cols_to_use, dtype=dtypes)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:02:03.838890Z","iopub.execute_input":"2023-03-21T08:02:03.839296Z","iopub.status.idle":"2023-03-21T08:03:43.688105Z","shell.execute_reply.started":"2023-03-21T08:02:03.839263Z","shell.execute_reply":"2023-03-21T08:03:43.687121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Some definitions for global access\nN_SESS = train[\"session_id\"].nunique()\nLEVELS = range(23)\nLV_COLORS = plt.cm.get_cmap(\"hsv\", len(LEVELS))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:03:43.690047Z","iopub.execute_input":"2023-03-21T08:03:43.691001Z","iopub.status.idle":"2023-03-21T08:03:43.880848Z","shell.execute_reply.started":"2023-03-21T08:03:43.690957Z","shell.execute_reply":"2023-03-21T08:03:43.879844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"dup_pk\"></a>\n## 1. Duplicated (`session_id`, `index`) Pairs\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\nAt first, I assume each sample in `train.csv` can be uniquely identified by the composite key (`session_id`, `index`). However, we can find that there exist duplications.","metadata":{}},{"cell_type":"code","source":"pk_tmp = [\"session_id\", \"index\"]\nassert not train.duplicated(subset=pk_tmp).any(), \"There are some duplicated (`session_id`, `index`) pairs.\"","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:03:43.881979Z","iopub.execute_input":"2023-03-21T08:03:43.883011Z","iopub.status.idle":"2023-03-21T08:03:52.588631Z","shell.execute_reply.started":"2023-03-21T08:03:43.882974Z","shell.execute_reply":"2023-03-21T08:03:52.586849Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"There are 142 sessions with duplicated (`session_id`, `index`) pairs. Let's see what's going on.","metadata":{}},{"cell_type":"code","source":"sess_dup = train.loc[train.duplicated(subset=pk_tmp, keep=False)][\"session_id\"].unique().tolist()\ntrain_dup = train.loc[train[\"session_id\"].isin(sess_dup)].reset_index(drop=True)\nprint(f\"#Sessions with duplicated (`session_id`, `index`) pairs: {len(sess_dup)}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:03:54.286395Z","iopub.execute_input":"2023-03-21T08:03:54.286777Z","iopub.status.idle":"2023-03-21T08:04:04.201937Z","shell.execute_reply.started":"2023-03-21T08:03:54.286746Z","shell.execute_reply":"2023-03-21T08:04:04.200490Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As can be seen, `index` series in sessions with duplicated (`session_id`, `index`) pairs show **lightning shape**. Also, some sessions have no `index == 0` (*e.g.,* `session_id == \"20110508425704692\"`).\n\nTo see the plot, please unfold the cell.","metadata":{}},{"cell_type":"code","source":"n_rows, n_cols = 29, 5\nfig, axes = plt.subplots(nrows=n_rows, ncols=n_cols, figsize=(20, 80))\nfor i, (sess_id, gp) in enumerate(train_dup.groupby(\"session_id\", observed=True)):\n    gp = gp.reset_index(drop=True)\n    axes[i // n_cols, i % n_cols].plot(gp[\"index\"])\n    axes[i // n_cols, i % n_cols].set_title(f\"Event Index of Session {sess_id}\\n Min Index {gp['index'].min()}\")\n    axes[i // n_cols, i % n_cols].set_xlabel(f\"Event Ordering in DataFrame\")\n    axes[i // n_cols, i % n_cols].set_ylabel(f\"Event Index\")\nplt.tight_layout()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-21T08:04:59.722826Z","iopub.execute_input":"2023-03-21T08:04:59.724072Z","iopub.status.idle":"2023-03-21T08:05:32.519012Z","shell.execute_reply.started":"2023-03-21T08:04:59.723985Z","shell.execute_reply":"2023-03-21T08:05:32.518108Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"reverse_idx\"></a>\n## 2. Reversed Index\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\nAfter glimpsing through sessions with duplicated (`session_id`, `index`) pairs, let's try to explore all the **reversed index** phenomenon in `train.csv`.\n\nThere exist 258 sessions with this phenomenon. Combined with the analysis above, we can conclude that there are $258 - 142 = 116$ sessions with **reversed index** phenomenon, but without duplicated (`session_id`, `index`) pairs.","metadata":{}},{"cell_type":"code","source":"sess_with_reversed_index = []\nfor sess_id, gp in train.groupby(\"session_id\", observed=True):\n    if not gp[\"index\"].is_monotonic_increasing:\n        sess_with_reversed_index.append(sess_id)\n        \nprint(f\"There are {len(sess_with_reversed_index)} sessions with \\\"reversed index\\\" phenomenon.\")\nprint(f\"-> {len(sess_dup)} sessions with duplicated (`session_id`, `index`) pairs.\")\nprint(f\"-> {len(sess_with_reversed_index) - len(sess_dup)} sessions without duplicated (`session_id`, `index`) pairs.\")","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:05:46.110746Z","iopub.execute_input":"2023-03-21T08:05:46.111150Z","iopub.status.idle":"2023-03-21T08:05:53.084911Z","shell.execute_reply.started":"2023-03-21T08:05:46.111117Z","shell.execute_reply":"2023-03-21T08:05:53.083869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To check whether samples are sorted in align with the progress of the game, I simply verify if all the `text` entries before\n\n> Whatcha doing over there, Jo?\n\nare all `undefined`. And, it holds!!\n\nThen, we can move on to try to fix this **reversed index** phenomenon.","metadata":{}},{"cell_type":"code","source":"FIRST_DIALOG_PER_GRAMP = \"Whatcha doing over there, Jo?\"\n\nfor sess_id in sess_with_reversed_index:\n    df = train[train[\"session_id\"] == sess_id].reset_index(drop=True)\n    \n    text_prompts = defaultdict(int)\n    for text in df[\"text\"]:\n        if text == FIRST_DIALOG_PER_GRAMP:\n            assert len(text_prompts) == 1 and list(text_prompts.keys())[0] == \"undefined\" \n            break\n        text_prompts[text] += 1","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:06:30.281088Z","iopub.execute_input":"2023-03-21T08:06:30.281472Z","iopub.status.idle":"2023-03-21T08:06:35.261013Z","shell.execute_reply.started":"2023-03-21T08:06:30.281442Z","shell.execute_reply":"2023-03-21T08:06:35.259867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the following plot, original `index` series is the **blue** line and `index_fixed` is the **red** one.\n\nAfter fixing, the `index_fixed` column, which indicates the **ordering of events**, is monotonically increasing in all sessions. However, there still exist some issues to be better resolved, which are shown as follows:\n1. `Index` column has some **jumps**, meaning that the ordering of events doesn't always increase by 1.\n2.  In addition to the **reversed index** phenomenon, there is also **reversed level** phenomenon.\n\nTo see the plot, please unfold the cell.","metadata":{}},{"cell_type":"code","source":"train_with_reversed_index = train[train[\"session_id\"].isin(sess_with_reversed_index)].reset_index(drop=True)\n\nn_rows, n_cols = 52, 5\nfig, axes = plt.subplots(nrows=n_rows, ncols=n_cols, figsize=(20, 130))\nsess_with_reversed_lv = []\nfor i, (sess_id, gp) in enumerate(train_with_reversed_index.groupby(\"session_id\", observed=True)):\n    df = gp.reset_index(drop=True)\n    df[\"index_diff\"] = (df[\"index\"] - df[\"index\"].shift(1)).bfill()\n    reverse_pts = df[df[\"index_diff\"] < 0]\n    \n    df[\"index_fixed\"] = df[\"index\"] - df[\"index\"][0]\n    for idx, row in reverse_pts.iterrows():\n        index_diff_abs = abs(row[\"index_diff\"])\n        df.loc[df.index >= idx, \"index_fixed\"] = df.loc[df.index >= idx, \"index_fixed\"] + index_diff_abs\n    \n    axes[i // n_cols, i % n_cols].plot(df[\"index\"], \"b\")\n    axes[i // n_cols, i % n_cols].plot(df[\"index_fixed\"], \"r\")\n    axes[i // n_cols, i % n_cols].set_title(f\"Event Index of Session {sess_id}\\n\"\n                                            f\"Min Index {df['index'].min()} - \"\n                                            f\"Min Index Fixed {int(df['index_fixed'].min())}\")\n    axes[i // n_cols, i % n_cols].set_xlabel(f\"Event Ordering in DataFrame\")\n    axes[i // n_cols, i % n_cols].set_ylabel(f\"Event Index\")\n    \n    if not df[\"level\"].is_monotonic_increasing:\n        sess_with_reversed_lv.append(sess_id)\nplt.tight_layout()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-21T08:06:57.871390Z","iopub.execute_input":"2023-03-21T08:06:57.871768Z","iopub.status.idle":"2023-03-21T08:07:59.132591Z","shell.execute_reply.started":"2023-03-21T08:06:57.871738Z","shell.execute_reply":"2023-03-21T08:07:59.131157Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"There are {len(sess_with_reversed_lv)} sessions with \\\"reversed index\\\" having \\\"reversed level\\\" phenomenon.\\n\"\n      f\"-> Sessions: {sess_with_reversed_lv}\")","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:07:59.135233Z","iopub.execute_input":"2023-03-21T08:07:59.135812Z","iopub.status.idle":"2023-03-21T08:07:59.142385Z","shell.execute_reply.started":"2023-03-21T08:07:59.135776Z","shell.execute_reply":"2023-03-21T08:07:59.141514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"reverse_lv\"></a>\n## 3. Reversed Level\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\nIn this section, let's dive into the **reversed level** phenomenon, which indicates that `level` can go **from high to low** within the same session.\n\nFor those sessions with **reversed index** (258 sessions in total), there exist six sessions having level **goes from 22 (highest) to 0 (lowest)**, but `elapsed_time` continuously accumulated. we assume that it's because the corresponding student plays the game **twice in a row**. To prove whether the assumption holds, we go to check [this file](https://github.com/fielddaylab/jo_wilder/blob/5082da4057f30dd0917c97a65f2aa7be13469f79/src/utils/simplelog.js). The code snippet below looks like the way how `session_id` is generated.\n\n```javascript\nself.session_id = UUIDint();\nself.persistent_session_id = getCookie(\"persistent_session_id\");\nif(!self.persistent_session_id)\n{\n    self.persistent_session_id = self.session_id;\n    setCookie(\"persistent_session_id\",self.persistent_session_id,100);\n}\n```\n\nWith this evidence, we think that students can play the game **multiple times in a row** with the same `session_id` if `persistent_session_id` has already existed.","metadata":{}},{"cell_type":"code","source":"train_with_reversed_lv = train_with_reversed_index[train_with_reversed_index[\"session_id\"].isin(sess_with_reversed_lv)].reset_index(drop=True)\n\nfig, axes = plt.subplots(nrows=2, ncols=3, figsize=(20, 6))\nfor i, (sess_id, gp) in enumerate(train_with_reversed_lv.groupby(\"session_id\", observed=True)):\n    df = gp.reset_index(drop=True)\n    df[\"level_diff\"] = (df[\"level\"] - df[\"level\"].shift(1)).bfill()\n    reverse_pt = df[df[\"level_diff\"] < 0].index.values[0]\n    \n    ax = axes[i // 3, i % 3]\n    ax.plot(df[\"index\"])\n    ax.set_title(sess_id)\n    ax.set_xlabel(\"Event Ordering in DataFrame\")\n    ax.set_ylabel(\"Event Index\")\n    \n    ax2 = ax.twinx()\n    for i, lv in enumerate(LEVELS):\n        lv_seq = df[df[\"level\"] == lv][\"level\"]\n        ax2.scatter(lv_seq.index.values, lv_seq.values, 0.8, marker=\"_\", c=LV_COLORS(i), linewidths=2)\n    ax2.axvline(x=int(reverse_pt), color='r', label=\"Reversed Level Point\")\n    ax2.set_ylabel(\"Level\")\nplt.legend()\nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:08:47.102922Z","iopub.execute_input":"2023-03-21T08:08:47.103377Z","iopub.status.idle":"2023-03-21T08:08:50.314147Z","shell.execute_reply.started":"2023-03-21T08:08:47.103343Z","shell.execute_reply":"2023-03-21T08:08:50.313224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Among all of the game play sessions, there exist 486 sessions (about 2.06%) with **reversed level** phenomenon.","metadata":{}},{"cell_type":"code","source":"sess_with_reversed_lv = []\nfor sess_id, gp in train.groupby(\"session_id\"):\n    if not gp[\"level\"].is_monotonic_increasing:\n        sess_with_reversed_lv.append(sess_id)\n\nprint(f\"There are {len(sess_with_reversed_lv)} sessions (about {len(sess_with_reversed_lv) / N_SESS * 100:.2f}%) with \\\"reversed level\\\" phenomenon.\")","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:09:32.000479Z","iopub.execute_input":"2023-03-21T08:09:32.000913Z","iopub.status.idle":"2023-03-21T08:09:39.429571Z","shell.execute_reply.started":"2023-03-21T08:09:32.000877Z","shell.execute_reply":"2023-03-21T08:09:39.428220Z"},"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"jump_idx\"></a>\n## 4. Jumped Index\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\nAs pointed out in the [section 2](#reverse_idx), `index` doesn't always increase by 1. There are some jumping points within `index` sequence. Before exploring this phenomenon, let's first fix **reversed index** in `train` DataFrame, making sure that `index` sequences of all sessions are **monotonically increasing**. In addition to the following fixing method, **directly re-indexing** can be applied.","metadata":{}},{"cell_type":"code","source":"dump_fixed_train = False","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:10:01.983468Z","iopub.execute_input":"2023-03-21T08:10:01.983884Z","iopub.status.idle":"2023-03-21T08:10:01.989574Z","shell.execute_reply.started":"2023-03-21T08:10:01.983852Z","shell.execute_reply":"2023-03-21T08:10:01.988506Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for sess_id in tqdm(sess_with_reversed_index):\n    df = train[train[\"session_id\"] == sess_id].copy()\n    target_index = df.index\n    df.reset_index(drop=True, inplace=True)\n    df[\"index_diff\"] = (df[\"index\"] - df[\"index\"].shift(1)).bfill()\n    reverse_pts = df[df[\"index_diff\"] < 0]\n    \n    df[\"index_fixed\"] = df[\"index\"] - df[\"index\"][0]\n    for idx, row in reverse_pts.iterrows():\n        index_diff_abs = abs(row[\"index_diff\"])\n        df.loc[df.index >= idx, \"index_fixed\"] = df.loc[df.index >= idx, \"index_fixed\"] + index_diff_abs\n    \n    # Fix reversed index in original DataFrame\n    train.loc[target_index, \"index\"] = df[\"index_fixed\"].values\n\nassert train.groupby(\"session_id\")[\"index\"].is_monotonic_increasing.all()\nif dump_fixed_train:\n    train.to_csv(\"./train_fixed.csv\", index=False)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:10:04.570168Z","iopub.execute_input":"2023-03-21T08:10:04.570559Z","iopub.status.idle":"2023-03-21T08:11:07.589983Z","shell.execute_reply.started":"2023-03-21T08:10:04.570528Z","shell.execute_reply":"2023-03-21T08:11:07.588930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nearly all of the **jumped index** phenomena occur right after `checkpoint`. As we know that `index` indicates the ordering of events and **jumped index** also preserves the ordering nature, I think it's okay to ignore the jumps. But, the reason behind the scene remains unknown.","metadata":{}},{"cell_type":"code","source":"train[\"index_diff\"] = train.groupby(\"session_id\").apply(lambda x: (x[\"index\"] - x[\"index\"].shift(1)).bfill()).values\ntrain[\"jumped_index\"] = train[\"index_diff\"] > 1\nprint(f\"Number of jumped points: {train['jumped_index'].sum()}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:11:48.853360Z","iopub.execute_input":"2023-03-21T08:11:48.853762Z","iopub.status.idle":"2023-03-21T08:12:20.051519Z","shell.execute_reply.started":"2023-03-21T08:11:48.853729Z","shell.execute_reply":"2023-03-21T08:12:20.050567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_before_jumps = train.iloc[train[train[\"jumped_index\"]].index - 1]\nevent_cnt_before_jumps = train_before_jumps[\"event_name\"].value_counts()\npct = event_cnt_before_jumps / event_cnt_before_jumps.sum() * 100\nlabels = [f\"{sec} {ratio:.2f}%\" for sec, ratio in zip(event_cnt_before_jumps.index, pct)]\n\nfig, ax = plt.subplots(figsize=(10, 5))\npatches, texts = ax.pie(event_cnt_before_jumps.values, \n                        colors=sns.color_palette(\"pastel\"), \n                        shadow=True, \n                        startangle=90)\npatches, labels, dummy = zip(*sorted(zip(patches, labels, event_cnt_before_jumps.values),\n                                     key=lambda x: x[2],\n                                     reverse=True))\nax.legend(patches, labels, bbox_to_anchor=(-0.1, 1.), fontsize=8)\nax.set_title(\"Ratio of Events before Jumps\")\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:12:20.056202Z","iopub.execute_input":"2023-03-21T08:12:20.058924Z","iopub.status.idle":"2023-03-21T08:12:20.898958Z","shell.execute_reply.started":"2023-03-21T08:12:20.058878Z","shell.execute_reply":"2023-03-21T08:12:20.897894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After each `checkpoint`, it takes the student a period of time (*i.e.,* a larget gap of `elapsed time` difference) to answer the questions of the corresponding `level_group`, which is marked in green as follows. I take the first session as illustration.","metadata":{}},{"cell_type":"code","source":"train_demo = train[train[\"session_id\"] == train[\"session_id\"][0]].reset_index()\nckpts = train_demo[train_demo[\"event_name\"] == \"checkpoint\"].index\n\nfig, ax = plt.subplots(figsize=(10, 3))\n# Index\nl1 = ax.plot(train_demo[\"index\"], \"b\", label=\"Index\")\nax.set_title(f\"Index versus Elapsed Time Demo\")\nax.set_xlabel(f\"Event Order\")\nax.set_ylabel(f\"Index\")\n# Elapsed time\nax2 = ax.twinx()\nl2 = ax2.plot(train_demo[\"elapsed_time\"], \"r\", label=\"Elapsed Time\")\nax2.set_ylabel(f\"Elapsed Time\")\n# Checkpoints\nfor ckpt in ckpts:\n    ax.axvline(ckpt, linestyle=\"--\", linewidth=.5, color=\"g\")\nax.legend(l1+l2, [l.get_label() for l in l1+l2])\n\nax2.set_zorder(-1)\nax.patch.set_visible(False)\nax2.patch.set_visible(True)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:12:20.900152Z","iopub.execute_input":"2023-03-21T08:12:20.900474Z","iopub.status.idle":"2023-03-21T08:12:21.267636Z","shell.execute_reply.started":"2023-03-21T08:12:20.900437Z","shell.execute_reply":"2023-03-21T08:12:21.264826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"ckpt\"></a>\n## 5. Checkpoint Exploration\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\nAs the general game progresses, I think each session should have **three** `checkpoint`s in total, each of which occurs right before the questions prompted. However, there exist sessions having less and more than three `checkpoint`s. One related issue is discussed [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959).","metadata":{}},{"cell_type":"code","source":"n_ckpts_per_sess = train.groupby([\"session_id\", \"level_group\"]).apply(lambda x: (x[\"event_name\"] == \"checkpoint\").sum()).reset_index()\nn_ckpts_per_sess = n_ckpts_per_sess.pivot(index=\"session_id\", columns=\"level_group\", values=0)\n\nn_ckpts_per_sess[\"total\"] = n_ckpts_per_sess[\"0-4\"] + n_ckpts_per_sess[\"5-12\"] + n_ckpts_per_sess[\"13-22\"]\nn_ckpts_per_sess_val_cnt = n_ckpts_per_sess[\"total\"].value_counts()\n\nfig, ax = plt.subplots(figsize=(7, 4))\nsns.barplot(x=n_ckpts_per_sess_val_cnt.index, y=n_ckpts_per_sess_val_cnt.values, \n            palette=colors, ax=ax)\nfor container in ax.containers:\n    ax.bar_label(container)\nax.set_title(\"Number of Checkpoints Per Session\")\nax.set_xlabel(\"Number of Checkpoints\")\nax.set_ylabel(\"Session Count\")\nax.set_ylim([0, 100])\nax.text(0.98, 95, f\"{n_ckpts_per_sess_val_cnt[n_ckpts_per_sess_val_cnt.index == 3].values[0]}\\n≈\", horizontalalignment=\"center\")\nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:12:51.249857Z","iopub.execute_input":"2023-03-21T08:12:51.250297Z","iopub.status.idle":"2023-03-21T08:13:21.105150Z","shell.execute_reply.started":"2023-03-21T08:12:51.250253Z","shell.execute_reply":"2023-03-21T08:13:21.103401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"sess_with_two_ckpts\"></a>\nFor the only session `22090108192456930` which doesn't have a `checkpoint` when `level_group == \"0-4\"`, the game still progresses smoothly toward **level 5**.","metadata":{}},{"cell_type":"code","source":"sess_without_ckpt = n_ckpts_per_sess[(n_ckpts_per_sess == 0).any(axis=1)]\nprint(f\"The only session missing one `checkpoint`: {sess_without_ckpt.index[0]}\")\ntrain[train[\"session_id\"] == \"22090108192456930\"].reset_index(drop=True).iloc[281:283, :6]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:13:21.107150Z","iopub.execute_input":"2023-03-21T08:13:21.107506Z","iopub.status.idle":"2023-03-21T08:13:21.166670Z","shell.execute_reply.started":"2023-03-21T08:13:21.107475Z","shell.execute_reply":"2023-03-21T08:13:21.165525Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After `checkpoint`, the **level-up** should occur. However, this assumption doesn't always hold. There are three cases described as follows:\n* `level_group == \"0-4\"`: There are extra events occuring after `checkpoint` is triggered in level 4.\n* `level_group == \"5-12\"`: There are extra events occuring after `checkpoint` is triggered in level 12.\n* `level_group == \"13-22\"`: There are extra events occuring after `checkpoint` is triggered in level 22, which should be considered as **the end of the game**.","metadata":{}},{"cell_type":"code","source":"train[\"lv_up_after_ckpt\"] = train.groupby(\"session_id\").apply(lambda x: (x[\"level\"].shift(-1) - x[\"level\"]).fillna(-1)).values\nnot_lvup_mask = (train[\"event_name\"] == \"checkpoint\") & (train[\"level\"].isin([4, 12, 22])) & (train[\"lv_up_after_ckpt\"] == 0)\ntrain_ckpt_without_lvup = train[not_lvup_mask]\nckpt_without_lvup_per_lv =  train_ckpt_without_lvup[\"level\"].value_counts()\n\nfig, ax = plt.subplots(figsize=(7, 4))\nsns.barplot(x=ckpt_without_lvup_per_lv.index, y=ckpt_without_lvup_per_lv.values, \n            palette=colors, ax=ax)\nfor container in ax.containers:\n    ax.bar_label(container)\nax.set_title(\"Number of Checkpoints Without Level-Up Followed\")\nax.set_xlabel(\"Level\")\nax.set_ylabel(\"Checkpoint Count\")\nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:13:21.168281Z","iopub.execute_input":"2023-03-21T08:13:21.168730Z","iopub.status.idle":"2023-03-21T08:13:50.904845Z","shell.execute_reply.started":"2023-03-21T08:13:21.168696Z","shell.execute_reply":"2023-03-21T08:13:50.903633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's take session `20100012562027690` as an example. The student seems to double-check the clues after level goes up to 12, where the text prompts look different from those at level 11 (as shown below).\n\n[![dialog.png](https://i.postimg.cc/XJdpgJ9w/dialog.png)](https://postimg.cc/tZRqq9ST)\n\nIn the beginning of level 12, *Jo* should go back to the capitol and talk to *Mrs.M*. After clicking on *Mrs.M*, the dialogue begins with\n> Ooh, nice decorations!\n\nHowever, what makes me confused is that this event occurs **after** the `checkpoint`. So far, I've not figured out what's going on in these cases.\n\nMaybe, we can drop extra events **after the `checkpoint` and before level up**.","metadata":{}},{"cell_type":"code","source":"train_demo = train[train[\"session_id\"] == \"20100012562027690\"]\ntrain_demo[train_demo[\"level\"] == 12].tail()","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:13:50.907511Z","iopub.execute_input":"2023-03-21T08:13:50.907937Z","iopub.status.idle":"2023-03-21T08:13:50.949391Z","shell.execute_reply.started":"2023-03-21T08:13:50.907823Z","shell.execute_reply":"2023-03-21T08:13:50.948065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"abnormal_game\"></a>\n## 6. Abnormal Game Plays\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\nIn addition to the level-up problem mentioned above, let's combine `checkpoint` behaviour with issues like **reversed level** to gain deeper insights.\n\nFirst of all, there exist 168 sessions with less than or more than 3 checkpoints. As discussed [here](#sess_with_two_ckpts), session `22090108192456930` has only two checkpoints. Hence, there are 167 sessions with more than 3 checkpoints. Also, we can confirm that sessions with 3 checkpoints have **exactly 1 checkpoint for each `level_group`**, which is considered **valid** temporarily.","metadata":{}},{"cell_type":"code","source":"ckpt_valid_mask = (n_ckpts_per_sess[\"0-4\"] == 1) & (n_ckpts_per_sess[\"5-12\"] == 1) & (n_ckpts_per_sess[\"13-22\"] == 1)\nsess_with_invalid_ckpt = n_ckpts_per_sess.loc[~ckpt_valid_mask].index.astype(int).tolist()\nprint(f\"There are {len(sess_with_invalid_ckpt)} sessions with less than or more than 3 checkpoints.\")","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:14:34.698356Z","iopub.execute_input":"2023-03-21T08:14:34.698869Z","iopub.status.idle":"2023-03-21T08:14:34.715827Z","shell.execute_reply.started":"2023-03-21T08:14:34.698833Z","shell.execute_reply":"2023-03-21T08:14:34.714787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, let's see how `level` sequences perform in these so-called **invalid** sessions. Besides, **checkpoints** are marked with green line.","metadata":{}},{"cell_type":"code","source":"def plot_level_with_ckpt(sess_id: int, ax: Optional[Axes] = None) -> None:\n    \"\"\"Plot level sequence of the specified session.\n    \n    Parameters:\n        sess_id: session identifier\n    \n    Return:\n        None\n    \"\"\"\n    plot = False\n    sess = train[train[\"session_id\"] == sess_id].reset_index(drop=True)\n    ckpts = sess[sess[\"event_name\"] == \"checkpoint\"].index\n        \n    if ax is None:\n        fig, ax = plt.subplots(figsize=(8, 2))\n        plot = True\n    for i, lv in enumerate(LEVELS):\n        lv_seq = sess[sess[\"level\"] == lv][\"level\"]\n        ax.scatter(lv_seq.index.values, lv_seq.values, 0.8, marker=\"_\", c=LV_COLORS(i), linewidths=2)\n    for ckpt in ckpts:\n        ax.axvline(ckpt, linestyle=\"--\", linewidth=.7, color=\"g\")\n    ax.set_title(f\"Level Seq of {sess_id}\", fontsize=12)\n\n    if plot:\n        plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:15:15.642570Z","iopub.execute_input":"2023-03-21T08:15:15.642965Z","iopub.status.idle":"2023-03-21T08:15:15.653011Z","shell.execute_reply.started":"2023-03-21T08:15:15.642933Z","shell.execute_reply":"2023-03-21T08:15:15.652071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Session `22090108192456930` has no checkpoint for `level_group == \"0-4\"`. However, the level sequence looks reasonable.","metadata":{}},{"cell_type":"code","source":"train[\"session_id\"] = train[\"session_id\"].astype(int)\nplot_level_with_ckpt(22090108192456930)","metadata":{"execution":{"iopub.status.busy":"2023-03-21T08:15:16.758092Z","iopub.execute_input":"2023-03-21T08:15:16.758780Z","iopub.status.idle":"2023-03-21T08:15:17.273150Z","shell.execute_reply.started":"2023-03-21T08:15:16.758744Z","shell.execute_reply":"2023-03-21T08:15:17.272229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For the remaining 167 sessions with more than 3 checkpoints, it's obvious that multiple game plays exist in a single session. Nonetheless, the second game play in a single sessions doesn't always start from `level == 0` (*e.g.,* session `21100409270509124`). This phenomenon also reveals the truth that **reversed level** doesn't always imply **reversed level group**.\n\nTo see the plot, please unfold the cell.","metadata":{}},{"cell_type":"code","source":"sess_with_invalid_ckpt.remove(22090108192456930)\n\nfig, axes = plt.subplots(nrows=34, ncols=5, figsize=(20, 80))\nfor i, sess_id in enumerate(sess_with_invalid_ckpt):\n    plot_level_with_ckpt(sess_id, ax=axes[i // 5, i % 5])\nplt.tight_layout()","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-03-21T08:15:42.076209Z","iopub.execute_input":"2023-03-21T08:15:42.076688Z","iopub.status.idle":"2023-03-21T08:16:43.033981Z","shell.execute_reply.started":"2023-03-21T08:15:42.076649Z","shell.execute_reply":"2023-03-21T08:16:43.032681Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As for the potential data leakage reported by [@cdeotte](https://www.kaggle.com/cdeotte) [here](https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/388479#2149959), 442 out of 11779 sessions have the **reversed level group** phenomenon. Furthermore, there are 486 sessions with **reversed level** phenomenon. If **reversed level group** occurs, **reversed level** must hold, but not vice versa.  ","metadata":{}},{"cell_type":"code","source":"lvgp_order = {\"0-4\": 0, \"5-12\": 1, \"13-22\": 2}\ntrain[\"lvgp_order\"] = train[\"level_group\"].map(lvgp_order).astype(int)\n\nsess_with_reversed_lv = []\nsess_with_reversed_lvgp = []\nfor sess_id, gp in tqdm(train.groupby(\"session_id\")):\n    if not gp[\"level\"].is_monotonic_increasing:\n        sess_with_reversed_lv.append(sess_id)\n    if not gp[\"lvgp_order\"].is_monotonic_increasing:\n        sess_with_reversed_lvgp.append(sess_id)\nassert set(sess_with_reversed_lvgp).issubset(set(sess_with_reversed_lv))\n\nprint(f\"There are {len(sess_with_reversed_lv)} sessions with \\\"reversed level\\\" phenomenon,\\n\"\n      f\"and {len(sess_with_reversed_lvgp)} with \\\"reversed level group\\\" phenomenon.\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:16:43.036002Z","iopub.execute_input":"2023-03-21T08:16:43.036368Z","iopub.status.idle":"2023-03-21T08:16:55.622838Z","shell.execute_reply.started":"2023-03-21T08:16:43.036336Z","shell.execute_reply":"2023-03-21T08:16:55.621619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All of the 44 ($486 - 442$) sessions with only **reversed level** phenomenon have exactly 3 checkpoints. The level sequences show that after the last checkpoint (*i.e.,* the third one), `level` will fall back to some specific point within `level_group == \"13-22\"`.","metadata":{}},{"cell_type":"code","source":"sess_with_reversed_lv_only = set(sess_with_reversed_lv).difference(set(sess_with_reversed_lvgp))\n\nfig, axes = plt.subplots(nrows=9, ncols=5, figsize=(20, 25))\nfor i, sess_id in enumerate(sess_with_reversed_lv_only):\n    plot_level_with_ckpt(sess_id, ax=axes[i // 5, i % 5])\nplt.tight_layout()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-03-21T08:17:18.966145Z","iopub.execute_input":"2023-03-21T08:17:18.966546Z","iopub.status.idle":"2023-03-21T08:17:35.690427Z","shell.execute_reply.started":"2023-03-21T08:17:18.966515Z","shell.execute_reply":"2023-03-21T08:17:35.689251Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"conclusion\"></a>\n## 7. Conclusion\n[**<span style=\"color:#FEF1FE; background-color:#535d70;border-radius: 5px; padding: 2px\">Go to Table of Content</span>**](#toc)\n\nThrough exploring pre-assumed primary key (`session_id`, `index`) and properties related to `index` columns, we can find out that there exist some issues which can hinder us from interpreting the **sequential characteristics** of the data. With a simple fix, the **ordering nature** of events can be better represented.<br>\nAlso, the existence of `checkpoint`-related issues could lower data quality. What's worse, the problems like **reversed level** and **reversed level group** might lead to data leakage, which can facilitate LB boost to some extent. The quick exploration here might help us come up with better ways to clean the data in hand. Thanks!","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}