{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":" \n
\n
\n
\n
\n
Predict Student Performance from Game Play
\n
\n • EDA • Data Engineering • Baseline Models
\n\n\n\n\n \n
![](\"https://content.pbswisconsineducation.org/wp-content/uploads/2020/10/21193325/home-masthead-laptop-desktop-clipped-01.svg\")
\n
Created by: zhangyue325
\n
\n\nTABLE OF CONTENTS
\n\n---\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\n
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n1 INTRODUCTION & JUSTIFICATION ⤒
","metadata":{}},{"cell_type":"markdown","source":"
\n\n1.1 WHAT IS THIS?
\n\n---\n\n* This notebook will follow the authors learning path and highlight useful information and helper function about the competition\n* This notebook will conduct an Exploratory Data Analysis for the competition\n* This notebook will propose an open-source baseline solution with XGBoost and LightGBM","metadata":{}},{"cell_type":"markdown","source":"
\n\n1.2 WHY IS THIS?
\n\n---\n\n* Writing and sharing my learning path and the resulting exploratory data analysis can help improve my own understanding of the competition and the data.\n* Data engineering is crucial in this competition. Sharing my EDA might help others in the competition.\n* Writing and sharing my work is often a fun and rewarding experience! It now only allows me to explore and try different techniques, ideas and visualizations... but it also encourages and supports other learners and partipants.","metadata":{}},{"cell_type":"markdown","source":"
\n\n1.3 WHO IS THIS FOR?
\n\n---\n\nThe primary purpose of this notebook is to educate MYSELF, however, my review/learning might be beneficial to others:\n* Other Kagglers (aka. current and future competition participants)\n* Anyone interested in data exploration analysis and data visulazation\n* Anyone interested in XGBoost and LightGBM for classfication\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n2 COMPETITION INFORMATION ⤒
\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n2.1 PROBLEM STATMENT
\n\n---\n\n* The goal of this competition is to predict student performance during game-based learning in real-time. You'll develop a model trained on one of the largest open datasets of game logs.\n* Your work will help advance research into knowledge-tracing methods for game-based learning. You'll be supporting developers of educational games to create more effective learning experiences for students.\n* If successful, you'll enable game developers to improve educational games and further support the educators who use these games with dashboards and analytic tools. In turn, we might see broader support for game-based learning platforms.","metadata":{}},{"cell_type":"markdown","source":"
\n\n2.2 DATASET DESCRIPTION
\n\n---\n\nThe data include:\n\n* train dataset with game sessions data: `train.csv`\n* correct answers for all questions for each session: `train_labels.csv`\n* test dataset with game sessions data: `test.csv`\n* sample submission file: `sample_submission.csv`\n\nAs mentioned by the competition host, the raw data from the game will also be made available and can be found at this site, we can use as supplemental data for this competition.","metadata":{}},{"cell_type":"markdown","source":"
\n\n2.3 EVALUATION
\n\n---\n\nThis competition has two tracks: the first one focus on the accuracy of the model, and the second one focus on the efficiency of the model\n* First track: Accuracy\n * The submissions will be evaluated based on the F1 socre: $2 \\frac{precision*recall}{precision+recall}$\n* Second track: Efficiency\n * Must be among the submissions selected by a team for the Leaderboard Prize, or else among those submissions automatically selected under the conditions described in the My Submissions tab.\n * Must be ranked on the Private Leaderboard higher than the sample_submission.csv benchmark.\n * Must not have a GPU enabled. The Efficiency Prize is CPU Only.\n * The submissions will be evaluated based on the Efficiency: $\\frac{1}{Benchmark-maxF1} + \\frac{1}{32400}RuntimeSeconds $","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n3 IMPORTS ⤒
","metadata":{}},{"cell_type":"code","source":"!pip install mpl_scatter_density","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:14:30.339311Z","iopub.execute_input":"2023-04-21T05:14:30.340126Z","iopub.status.idle":"2023-04-21T05:14:45.313707Z","shell.execute_reply.started":"2023-04-21T05:14:30.340082Z","shell.execute_reply":"2023-04-21T05:14:45.312223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\n... IMPORTS STARTING ...\\n\")\n\n# Machine Learning and Data Science Imports (basics)\nimport pandas as pd; print(f\"\\t– PANDAS VERSION: {pd.__version__}\");\nimport numpy as np; print(f\"\\t– NUMPY VERSION: {np.__version__}\");\n\n# Built-In Imports (mostly don't worry about these)\nimport os\nimport time\nimport gc\n\n# Visualization Imports (overkill)\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom skimage import io \ntry: \n import mpl_scatter_density # for density scatter graph \nexcept:\n print(\"\\tPlease install mpl_scatter_density!\")\n \n# Other Imports\nfrom tqdm.notebook import tqdm # for progress bar\nimport jo_wilder # the API of this competition\n\nprint(\"\\n... IMPORTS COMPLETE ...\\n\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:14:50.269763Z","iopub.execute_input":"2023-04-21T05:14:50.270172Z","iopub.status.idle":"2023-04-21T05:14:51.781107Z","shell.execute_reply.started":"2023-04-21T05:14:50.270132Z","shell.execute_reply":"2023-04-21T05:14:51.779616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n4 SETUP AND HELPER FUNCTIONS ⤒
","metadata":{}},{"cell_type":"markdown","source":"
\n\n4.1 HELPER FUNCTIONS
\n\n---\n
\n\n","metadata":{}},{"cell_type":"code","source":"# for checking features properties during feature engineering\ndef check_features(features_df):\n fig, ax = plt.subplots(1, 2, figsize=(18,9))\n fig.tight_layout(pad=10.0)\n sns.boxplot(ax=ax[0], data=features_df, orient=\"h\")\n sns.violinplot(ax=ax[1], data=features_df, orient=\"h\")\n plt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:18:33.059136Z","iopub.execute_input":"2023-04-21T05:18:33.059569Z","iopub.status.idle":"2023-04-21T05:18:33.067368Z","shell.execute_reply.started":"2023-04-21T05:18:33.059530Z","shell.execute_reply":"2023-04-21T05:18:33.065775Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# convert features series into features datafame\ndef to_df(features_series): \n features_df = pd.concat(features_series,axis=1)\n features_df = features_df.reset_index()\n features_df = features_df.set_index('session_id')\n return features_df","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:18:34.279813Z","iopub.execute_input":"2023-04-21T05:18:34.280217Z","iopub.status.idle":"2023-04-21T05:18:34.287155Z","shell.execute_reply.started":"2023-04-21T05:18:34.280182Z","shell.execute_reply":"2023-04-21T05:18:34.285518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n4.2 LOAD DATA
\n\n---\n
\n","metadata":{}},{"cell_type":"code","source":"DATA_DIR = \"/kaggle/input/predict-student-performance-from-game-play\"\n\ndtypes = {'session_id': 'category',\n 'elapsed_time': np.int32,\n 'event_name': 'category',\n 'name': 'category',\n 'level': np.uint8,\n 'page': 'category',\n 'room_coor_x': np.float32,\n 'room_coor_y': np.float32,\n 'screen_coor_x': np.float32,\n 'screen_coor_y': np.float32,\n 'hover_duration': np.float32,\n 'text': 'category',\n 'fqid': 'category',\n 'room_fqid': 'category',\n 'text_fqid': 'category',\n 'fullscreen': np.int8,\n 'hq': np.int8,\n 'music': np.int8,\n 'level_group': 'category'}\nprint(\"\\n\\n... LOAD DATA FROM CSV FILE ...\")\ntrain = pd.read_csv(os.path.join(DATA_DIR, \"train.csv\"), dtype=dtypes)\ntest = pd.read_csv(os.path.join(DATA_DIR, \"test.csv\"), dtype=dtypes)\ntrain_labels = pd.read_csv(os.path.join(DATA_DIR, \"train_labels.csv\"))\nprint(f\"\\n\\n... LOAD DATA COMPLETE ...\\n\")\n","metadata":{"execution":{"iopub.status.busy":"2023-04-21T05:15:16.429570Z","iopub.execute_input":"2023-04-21T05:15:16.430546Z","iopub.status.idle":"2023-04-21T05:18:02.759666Z","shell.execute_reply.started":"2023-04-21T05:15:16.430494Z","shell.execute_reply":"2023-04-21T05:18:02.758031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Train data shape:', train.shape)\nprint(\"Sample of train data:\")\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-21T04:57:55.479036Z","iopub.execute_input":"2023-04-21T04:57:55.479443Z","iopub.status.idle":"2023-04-21T04:57:55.516251Z","shell.execute_reply.started":"2023-04-21T04:57:55.479407Z","shell.execute_reply":"2023-04-21T04:57:55.515352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Train labels data shape:', train_labels.shape)\nprint(\"Sample of train_labels data:\")\ntrain_labels.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-21T04:58:02.359193Z","iopub.execute_input":"2023-04-21T04:58:02.359945Z","iopub.status.idle":"2023-04-21T04:58:02.370610Z","shell.execute_reply.started":"2023-04-21T04:58:02.359905Z","shell.execute_reply":"2023-04-21T04:58:02.369626Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Test data shape:', test.shape)\nprint(\"Sample of test data:\")\ntest.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T04:58:08.119726Z","iopub.execute_input":"2023-04-21T04:58:08.120871Z","iopub.status.idle":"2023-04-21T04:58:08.154435Z","shell.execute_reply.started":"2023-04-21T04:58:08.120828Z","shell.execute_reply":"2023-04-21T04:58:08.153250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n5 EXPLORATORY DATA ANALYSIS ⤒
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n\n\n---\n\n* each game is a session defined by `session_id`\n* each row in the data is a game event, defined by `name`, type (`event_name`), unique ID (`fqid`), and game progress index (`index`)\n* the player progresses by moving from one game room (`room_fqid`) and level (`level`) to another\n* remaining columns refer to specific events:\n * for click events, the coordinates of the click are defined in `room_coor_x` and `room_coor_y` (in reference to the in-game room) or in `screen_coor_x` and `screen_coor_y` (in reference to the player’s screen).\n * for notebook-related events page identifies the page number;\n * for hover events hover_duration shows how long was the hover.\n\nYou can play the game here: https://pbswisconsineducation.org/jowilder/play-the-game/ \n
\nThe game code source is public: https://github.com/fielddaylab/jo_wilder\n","metadata":{}},{"cell_type":"markdown","source":"|No | Columns name | Meaning |\n|:---| :--- |:--- |\n| 1 | session_id | the ID of the session the event took place in |\n| 2 | index | game progress index |\n| 3 | elapsed_time | how much time has passed (in milliseconds) between the start of the session
and when the event was recorded |\n| 4 | event_name | the name of the event type |\n| 5 | name | the event name (e.g. identifies whether a notebook_click is is opening
or closing the notebook) |\n| 6 | level | what level of the game the event occurred in (0 ~ 22) |\n| 7 | page | the page number of the event (only for notebook-related events) |\n| 8 | room_coor_x | the coordinates of the click in reference to the in-game room (only for click events) |\n| 9 | room_coor_y | the coordinates of the click in reference to the in-game room (only for click events) |\n| 10 | screen_coor_x | the coordinates of the click in reference to the player’s screen (only for click events) |\n| 11 | screen_coor_y | the coordinates of the click in reference to the player’s screen (only for click events) |\n| 12 | hover_duration | how long (in milliseconds) the hover happened for (only for hover events) |\n| 13 | text | the text the player sees during this event |\n| 14 | fqid | the fully qualified ID of the event |\n| 15 | room_fqid | the fully qualified ID of the room the event took place in |\n| 16 | text_fqid | the fully qualified ID of the text the player sees |\n| 17 | fullscreen | whether the player is in fullscreen mode |\n| 18 | hq | whether the game is in high-quality |\n| 19 | music | whether the game music is on or off |\n| 20 | level_group | which group of levels - and group of questions - this row belongs to (0-4, 5-12, 13-22) |","metadata":{}},{"cell_type":"code","source":"train_missing = train.isna().sum() / len(train) *100\ntrain_missing_index = train_missing.index\ntrain_missing_values = train_missing.values\n\nfig, ax = plt.subplots(figsize=(18, 9))\nbarchart = sns.barplot(x = train_missing_index, y = train_missing_values, ax = ax)\nbarchart.axes.set_title(\"Share of missing values in train data\", fontsize=24, loc = 'center')\nbarchart.bar_label(barchart.containers[0], fmt=\"%.1f%%\", fontsize=18)\nbarchart.yaxis.set_tick_params(labelsize = 18)\nbarchart.xaxis.set_tick_params(rotation=40, labelsize = 16)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T04:58:42.117289Z","iopub.execute_input":"2023-04-21T04:58:42.117691Z","iopub.status.idle":"2023-04-21T04:58:43.428248Z","shell.execute_reply.started":"2023-04-21T04:58:42.117656Z","shell.execute_reply":"2023-04-21T04:58:43.427300Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(18, 9))\ng = sns.heatmap(train.corr(), annot=True, square=True, cmap='coolwarm', annot_kws={'size': 15},fmt='.2f')\ng.tick_params(axis='x', labelsize=15)\ng.tick_params(axis='y', labelsize=15)\ng.set_title('Correlations in train dataset', size=20, pad=15)\nplt.show()\n\ndel g","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T04:58:56.399329Z","iopub.execute_input":"2023-04-21T04:58:56.399733Z","iopub.status.idle":"2023-04-21T04:59:07.372834Z","shell.execute_reply.started":"2023-04-21T04:58:56.399698Z","shell.execute_reply":"2023-04-21T04:59:07.367433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n\n---\n* `session_id` is the ID of the session the event took place in. \n* There are 23562 unique sessions in the train dataset","metadata":{}},{"cell_type":"code","source":"train_events_counts = train['session_id'].value_counts()\nprint(f\"\\n The number of unique session in train set: {len(train_events_counts)}\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T04:59:39.030478Z","iopub.execute_input":"2023-04-21T04:59:39.033255Z","iopub.status.idle":"2023-04-21T04:59:39.221473Z","shell.execute_reply.started":"2023-04-21T04:59:39.033194Z","shell.execute_reply":"2023-04-21T04:59:39.220331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"events_counts = train_events_counts.values\n\nplt.figure(figsize=(18, 9))\nplt.hist(events_counts, edgecolor=\"black\", bins=300, color = \"#9FE2BF\")\nplt.xlim(xmin=0, xmax = 5000)\nplt.title(\"the number of events per session (column) for train dataset\", fontsize=24)\nplt.xlabel(\"the number of events per session (column)\", fontsize=18)\nplt.ylabel(\"count of session\", fontsize=18)\nplt.axvline(events_counts.mean(), color='r')\nplt.text(events_counts.mean(), 2500, f'average: {events_counts.mean():.1f}', fontsize=18)\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T04:59:41.310758Z","iopub.execute_input":"2023-04-21T04:59:41.311639Z","iopub.status.idle":"2023-04-21T04:59:42.298698Z","shell.execute_reply.started":"2023-04-21T04:59:41.311595Z","shell.execute_reply":"2023-04-21T04:59:42.297629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n---\n* `index` is the index of the event for each session\n* The index is not very clean for train dataset and test dataset\n * some sessions have duplicate index\n * some sessions have no `index == 0` (start of the game)\n * some sessions have reversed index\n * some sessions have jump index\n\nFor more information, you can refer to this notebook, it has a detailed exploration to the duplicate index, reversed index, jump index.","metadata":{}},{"cell_type":"code","source":"idx = train[['session_id', 'index']]\nidx_dupl = idx[idx.duplicated()]\nprint(f\"\\n{len(idx_dupl)}/{len(train)} {len(idx_dupl)/len(train) * 10000:.1f}%% of the `session_id` and `index` pairs are dulicapted\")\n\ndel idx, idx_dupl","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:01:50.752690Z","iopub.execute_input":"2023-04-21T05:01:50.753166Z","iopub.status.idle":"2023-04-21T05:02:01.425440Z","shell.execute_reply.started":"2023-04-21T05:01:50.753129Z","shell.execute_reply":"2023-04-21T05:02:01.424432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n\n\n---\n* `elapsed_time` is how much time has passed (in milliseconds) between the start of the session and when the event was recorded\n","metadata":{}},{"cell_type":"code","source":"elapsed_time_train = np.round((train['elapsed_time']).astype(np.float64)/60000.0, 1)\nelapsed_time_test = np.round((test['elapsed_time']).astype(np.float64)/60000.0, 1)\n\nstat = pd.DataFrame([elapsed_time_train.describe().index,\n np.round(elapsed_time_train.describe(), 2).values,\n np.round(elapsed_time_test.describe(), 2).values]).T\nstat.columns = [' ', 'train', 'test']\nstat[['train', 'test']] = stat[['train', 'test']].astype(np.float64)\nprint('The statistics of `elapsed_time` column:')\nstat[1:].style.hide_index().format(precision=1).background_gradient()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:03:07.642589Z","iopub.execute_input":"2023-04-21T05:03:07.643020Z","iopub.status.idle":"2023-04-21T05:03:09.769976Z","shell.execute_reply.started":"2023-04-21T05:03:07.642984Z","shell.execute_reply":"2023-04-21T05:03:09.768918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mean_elapsed_time_train = elapsed_time_train.mean()\n\nplt.figure(figsize=(18, 9))\nplt.hist(elapsed_time_train, edgecolor=\"black\", bins=range(0, 120, 1), color = \"#9FE2BF\")\nplt.xlim(0, 120)\nplt.title(\"elapsed time for each event in train dataset\", fontsize=24)\nplt.xlabel(\"elapsed time (mins)\", fontsize=18)\nplt.ylabel(\"count of event\", fontsize=18)\nplt.axvline(mean_elapsed_time_train.mean(), color='r')\nplt.text(mean_elapsed_time_train.mean(), 10**6, f'average: {mean_elapsed_time_train.mean():.1f} mins', fontsize=18)\nplt.show()\n\ndel mean_elapsed_time_train, elapsed_time_train, elapsed_time_test","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:03:13.622370Z","iopub.execute_input":"2023-04-21T05:03:13.622789Z","iopub.status.idle":"2023-04-21T05:03:14.947660Z","shell.execute_reply.started":"2023-04-21T05:03:13.622750Z","shell.execute_reply":"2023-04-21T05:03:14.946475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n---\n* `event_name` is the the name of the event type. \n * It has 11 unique values (including cutscene_click, person_click, navigate_click, navigate_click, notification_click, object_click, object_hover, map_hover, map_click, checkpoint, and notebook_click)\n * You can have a detailed udnderstanding of the meaning of those event_name in this notebook\n* `name` is the name of event\n * It has 6 unique values (including basic, underfined, close, open, prev, and next)\n \nFor more information, you can refer to this notebook, it has a detailed exploration on the meaning of each `event_name`.\n\n","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\n\nplt.subplot(1, 2, 1)\nevent_name = train[\"event_name\"].value_counts()\nplt.bar(event_name.index, event_name.values, edgecolor=\"black\", color = \"#6495ED\")\nplt.xticks(rotation = 35)\nplt.title(\"frequency of `event_name`\", fontsize = 20)\nplt.ylabel(\"count of event\", fontsize = 16)\n\nplt.subplot(1, 2, 2)\nname = train[\"name\"].value_counts()\nplt.bar(name.index, name.values, edgecolor=\"black\", color = \"#9FE2BF\")\nplt.title(\"frequency of `name`\", fontsize = 20)\nplt.ylabel(\"count of event\", fontsize = 16)\n\nplt.show()\n\ndel event_name, name","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:03:26.857331Z","iopub.execute_input":"2023-04-21T05:03:26.857746Z","iopub.status.idle":"2023-04-21T05:03:27.569314Z","shell.execute_reply.started":"2023-04-21T05:03:26.857711Z","shell.execute_reply":"2023-04-21T05:03:27.568454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pivot = train.pivot_table(index='name', columns='event_name', aggfunc='size')\npivot = (pivot.fillna(0) / 1000).round(decimals = 1)\nplt.figure(figsize=(18, 9))\nannotations = pivot.astype(str)\nannotations[pivot == 0] = \"0\"\ng = sns.heatmap(pivot, annot=annotations, fmt='', cmap='GnBu')\nplt.title(\"frequency of 'name' vs 'event_name' in train dataset (in thousands)\", fontsize=22)\nplt.xlabel(\"name\", fontsize = 15)\nplt.ylabel(\"event_name\", fontsize = 18)\nplt.show()\n\ndel pivot, g","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:03:31.319251Z","iopub.execute_input":"2023-04-21T05:03:31.320084Z","iopub.status.idle":"2023-04-21T05:03:32.800082Z","shell.execute_reply.started":"2023-04-21T05:03:31.320037Z","shell.execute_reply":"2023-04-21T05:03:32.799123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n\n\n\n\n---\n* `level` is what level event in the game (0 to 22).\n* `level_group` is which group of levels this level belongs to (0-4, 5-12, 13-22)","metadata":{}},{"cell_type":"code","source":"outer = train.groupby(by = 'level_group')['session_id'].count()\ninner = train.groupby(by = ['level_group', 'level'])['session_id'].count()\ninner = inner[inner != 0]\n# inner_labels = inner.index.get_level_values(1)\n\n\nplt.figure(figsize=(18, 12))\n\n# Make data\ngroup_names=outer.index\ngroup_size=outer.values\nsubgroup_names = inner.index.get_level_values(1)\nsubgroup_size=inner.values\n\n# Create colors\na, b, c=[plt.cm.Blues, plt.cm.Reds, plt.cm.Greens]\n \n# First Ring (outside)\nfig, ax = plt.subplots()\nax.axis('equal')\nmypie, _ = ax.pie(group_size, radius=1.8, labels=group_names, colors=[a(0.6), b(0.6), c(0.6)])\nplt.setp( mypie, width=0.6, edgecolor='white')\n \n# Second Ring (Inside)\nmypie2, _ = ax.pie(subgroup_size, radius=1.8-0.6, labels=subgroup_names, labeldistance=0.7, \n colors=[a(0.1), a(0.2), a(0.3), a(0.4), a(0.5),\n b(0.1), b(0.15), b(0.2), b(0.25), b(0.3), b(0.35), b(0.4), b(0.45), \n c(0.06 * 1), c(0.06 * 2), c(0.06 * 3), c(0.06 * 4), c(0.06 * 5), \n c(0.06 * 6), c(0.06 * 7), c(0.06 * 8), c(0.06 * 9), c(0.06 * 10), c(0.06 * 11), c(0.06 * 12) ])\nplt.setp( mypie2, width=0.6, edgecolor='white')\n\nplt.text(0.5, 1.28, \"the frequency of `level` and `level_group`\",\n horizontalalignment='center',\n fontsize=15,\n transform = ax.transAxes)\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:04:33.531127Z","iopub.execute_input":"2023-04-21T05:04:33.531614Z","iopub.status.idle":"2023-04-21T05:04:35.348520Z","shell.execute_reply.started":"2023-04-21T05:04:33.531575Z","shell.execute_reply":"2023-04-21T05:04:35.347438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"At levels 4, 12, 22 the game has checkpoint events when questions are being asked.\n\nThe number of questions in each level_group is fixed:\n\n\n|level_group|questions|number of questions|\n|:---| :--- |:--- |\n|0-4|q1 to q3|3|\n|5-12|q4 to q13|10\n|13-22|q14 to q18|5|\n\nThere are sessions in which one of the 22 levels is not present.","metadata":{}},{"cell_type":"markdown","source":"####
\n\n\n\n\n\n\n\n---\n* `page` is the number of the event \n* `page` is only applicable for notebook-related events","metadata":{}},{"cell_type":"code","source":"page = train[~train[\"page\"].isna()]\nprint(f\"\\nonly {len(page)} of `page` in train dataset is not NaN, it accounts for {len(page)/len(train)*100:.1f}% of the whole train dataset\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:04:53.202685Z","iopub.execute_input":"2023-04-21T05:04:53.203090Z","iopub.status.idle":"2023-04-21T05:04:53.406368Z","shell.execute_reply.started":"2023-04-21T05:04:53.203054Z","shell.execute_reply":"2023-04-21T05:04:53.405391Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"page_piv = page.groupby(\"page\")[\"session_id\"].count()\n\n\nplt.figure(figsize=(18, 9))\nplt.bar(page_piv.index, page_piv.values, edgecolor=\"black\", color = \"#9FE2BF\")\nplt.title(\"freqency of `page` in train dataset\", fontsize=22)\nplt.xlabel(\"`page`\", fontsize=18)\nplt.ylabel(\"count of event\", fontsize=18)\nplt.show()\n\ndel page, page_piv","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:04:56.622617Z","iopub.execute_input":"2023-04-21T05:04:56.623168Z","iopub.status.idle":"2023-04-21T05:04:56.869375Z","shell.execute_reply.started":"2023-04-21T05:04:56.623103Z","shell.execute_reply":"2023-04-21T05:04:56.868393Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"####
\n\n\n\n\n\n---\n\n* `room_coor_x` is the coordinates of the click in reference to the in-game room (only for click events)\n* `room_coor_y` is the coordinates of the click in reference to the in-game room (only for click events)\n* `screen_coor_x` is the coordinates of the click in reference to the player’s screen (only for click events)\n* `screen_coor_y` is the coordinates of the click in reference to the player’s screen (only for click events)\n* 92.1% of the geo-location data in train dataset is not NaN\n\nFor more information, you can refer to those notebooks below:\n* This notebook tells how deo-location data is stored when you play the game.\n* This notebook exlopred the relationship between geo-location data and useful clue in the game, it may be useful for feature engineering.\n* This notebook has an interactive view on clicks: it allows you to select a session and watch the session click events at each stage of the game.","metadata":{}},{"cell_type":"code","source":"coor = train[~train[\"room_coor_x\"].isna()]\nprint(f\"\\n{len(coor)} of the geo-location data in train dataset is not NaN, it accounts for {len(coor)/len(train)*100:.1f}% of the whole train dataset\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:07:01.882972Z","iopub.execute_input":"2023-04-21T05:07:01.883524Z","iopub.status.idle":"2023-04-21T05:07:03.796250Z","shell.execute_reply.started":"2023-04-21T05:07:01.883482Z","shell.execute_reply":"2023-04-21T05:07:03.795089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To make the geo-location data easily understood, we need click density for each specific room, it shows the locations of important objects or characters.","metadata":{}},{"cell_type":"code","source":"room_ids = train[\"room_fqid\"].unique().tolist()\n\nfig = plt.figure(figsize=(18, 9 * len(room_ids)))\n\nroom_id = \"tunic.historicalsociety.closet\"\ntemp = train[train[\"room_fqid\"] == room_id]\n\nfor i in range(len(room_ids)):\n room_id = room_ids[i]\n temp = train[train[\"room_fqid\"] == room_id]\n ax = fig.add_subplot(len(room_ids) + 1,1, i + 1, projection='scatter_density')\n image = io.imread(f\"https://raw.githubusercontent.com/zhangyue325/my_warehouse/main/{room_id}.png\")\n ax.imshow(image, \n extent=[temp[\"room_coor_x\"].min(), \n temp[\"room_coor_x\"].max(), \n temp[\"room_coor_y\"].min(),\n temp[\"room_coor_y\"].max()], \n alpha = 0.4)\n ax.title.set_text(f'the click density map in\\n {room_id}')\n ax.set_ylabel('room_coor_y')\n ax.set_xlabel('room_coor_x')\n x = temp[\"room_coor_x\"]\n y = temp[\"room_coor_y\"]\n ax.scatter_density(x, y, color='blue', downres_factor = 1)\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:09:10.489432Z","iopub.execute_input":"2023-04-21T05:09:10.489838Z","iopub.status.idle":"2023-04-21T05:09:37.734335Z","shell.execute_reply.started":"2023-04-21T05:09:10.489803Z","shell.execute_reply":"2023-04-21T05:09:37.733222Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"####
\n\n\n\n\n\n\n \n \n---\n* `hover_duration` is how long (in milliseconds) the hover happened for (only for hover events)\n* `hover_duration` is applicable when `event_name` is `object_hover` or `map_hover`","metadata":{}},{"cell_type":"code","source":"hover = train[~train[\"hover_duration\"].isna()]\nprint(f\"\\n{len(hover)} of the geo-location data in train dataset is not NaN, it accounts for {len(hover)/len(train)*100:.1f}% of the whole train dataset\")\nobject_hover = hover[hover[\"event_name\"] == \"object_hover\"]\nmap_hover = hover[hover[\"event_name\"] == \"map_hover\"]\n\nprint(f\"{len(object_hover)/len(hover)*100:.1f}% of the hover event is object_hover, and {100 - len(object_hover)/len(hover)*100:.1f}% of the hover event is map_hover\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:05.100961Z","iopub.execute_input":"2023-04-21T05:10:05.101608Z","iopub.status.idle":"2023-04-21T05:10:05.570340Z","shell.execute_reply.started":"2023-04-21T05:10:05.101551Z","shell.execute_reply":"2023-04-21T05:10:05.569004Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"object_hover = np.round((object_hover['hover_duration']).astype(np.float64)/1000.0, 1)\nmap_hover = np.round((map_hover['hover_duration']).astype(np.float64)/1000.0, 1)\n\nstat = pd.DataFrame([object_hover.describe().index,\n np.round(object_hover.describe(), 2).values,\n np.round(map_hover.describe(), 2).values]).T\nstat.columns = [' ', 'object hover', 'map hover']\nstat[['object hover', 'map hover']] = stat[['object hover', 'map hover']].astype(np.float64)\nprint('The statistics of `object_hover` and `map_hover` (in second):')\nstat[1:].style.hide_index().format(precision=1).background_gradient()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:09.933321Z","iopub.execute_input":"2023-04-21T05:10:09.933741Z","iopub.status.idle":"2023-04-21T05:10:10.135782Z","shell.execute_reply.started":"2023-04-21T05:10:09.933705Z","shell.execute_reply":"2023-04-21T05:10:10.134071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\n\nplt.subplot(1, 2, 1)\nplt.hist(object_hover, edgecolor=\"black\", bins=range(0, 15, 1), color = \"#6495ED\")\nplt.xlim(0, 20)\nplt.title(f\"object hover time for each event in train dataset \\n(hover time <= 15)\", fontsize=20)\nplt.xlabel(\"object hover time (seconds)\", fontsize=18)\nplt.ylabel(\"count of event\", fontsize=18)\nplt.axvline(object_hover.mean(), color='r')\nplt.text(object_hover.mean(), 5* 10**5, f'average: {object_hover.mean():.1f} mins', fontsize=18)\n\nplt.subplot(1, 2, 2)\nplt.hist(map_hover, edgecolor=\"black\", bins=range(0, 15, 1), color = \"#9FE2BF\")\nplt.xlim(0, 20)\nplt.title(f\"map hover time for each event in train dataset \\n(hover time <= 15)\", fontsize=20)\nplt.xlabel(\"map hover time (seconds)\", fontsize=18)\nplt.ylabel(\"count of event\", fontsize=18)\nplt.axvline(map_hover.mean(), color='r')\nplt.text(map_hover.mean(), 8* 10**5, f'average: {map_hover.mean():.1f} mins', fontsize=18)\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:14.938358Z","iopub.execute_input":"2023-04-21T05:10:14.938784Z","iopub.status.idle":"2023-04-21T05:10:15.482123Z","shell.execute_reply.started":"2023-04-21T05:10:14.938749Z","shell.execute_reply":"2023-04-21T05:10:15.481208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Some players may stop to play the game during object hover or map hover, so their hover time are extremely long.\")\nprint(f\"{len(object_hover[object_hover>60])/len(object_hover) * 10000:.2f}%% player's object hover time larger than 1 minute. {len(map_hover[map_hover>60])/len(map_hover) * 10000:.2f}%% player's map hover time larger than 1 minute. \")\nprint(f\"{len(object_hover[object_hover>60*10])/len(object_hover) * 10000:.2f}%% player's object hover time larger than 10 minutes. {len(map_hover[map_hover>60 * 10])/len(map_hover) * 10000:.2f}%% player's map hover time larger than 10 minutes.\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:20.806246Z","iopub.execute_input":"2023-04-21T05:10:20.806690Z","iopub.status.idle":"2023-04-21T05:10:20.822363Z","shell.execute_reply.started":"2023-04-21T05:10:20.806654Z","shell.execute_reply":"2023-04-21T05:10:20.821150Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"####
\n\n\n\n\n\n---\n* `text` is the the text the player sees during this event\n* `text_fqid` is the fully qualified ID of the text the player sees. If the `text_fqid` is not null, it is the concretration of `fqid` (person/object the player is interacting with), `room_fqid` (game room), and the id of text (for example: .intro_0_cs_0)","metadata":{}},{"cell_type":"code","source":"print(f\"\\nIn train daset, {(~train['text'].isna()).sum()} `text` of events are not NaN. It accounts for {100 - train['text'].isna().sum() / len(train) * 100:.1f}% of the whole train dataset.\")\nprint(f\"In train daset, {(~train['text_fqid'].isna()).sum()} `text_fqid` of events are not NaN. It accounts for {100 - train['text_fqid'].isna().sum() / len(train) * 100:.1f}% of the whole train dataset.\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:24.910543Z","iopub.execute_input":"2023-04-21T05:10:24.911070Z","iopub.status.idle":"2023-04-21T05:10:25.075744Z","shell.execute_reply.started":"2023-04-21T05:10:24.911031Z","shell.execute_reply":"2023-04-21T05:10:25.074538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\nAll the event whose `text` is not NaN, its `text_fqid` is also not NaN.\")\nprint(f'The number of events whose `text` is not NaN but `text_fqid` is NaN: {train[~(train[\"text\"].isna()) & (train[\"text_fqid\"].isna())][\"text\"].count()}.')\n\nprint(\"\\nThere are some events whose `text-fqid` is not NaN, but `text` is NaN.\")\nprint(f'The number of events whose `text` is not NaN but `test` is NaN: {train[~(train[\"text_fqid\"].isna()) & (train[\"text\"].isna())][\"text_fqid\"].count()}.')\n\nprint(f\"\\nThis is a sample of event whose `text` is not NaN but `text` is NaN.\")\npd.set_option('display.width', 1000)\ntrain[(train[\"text\"].isna()) & ~(train[\"text_fqid\"].isna())][[\"text\", \"text_fqid\"]]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:28.808149Z","iopub.execute_input":"2023-04-21T05:10:28.808626Z","iopub.status.idle":"2023-04-21T05:10:29.032656Z","shell.execute_reply.started":"2023-04-21T05:10:28.808585Z","shell.execute_reply":"2023-04-21T05:10:29.031734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'\\nIn train dataset, we have {train[\"text\"].nunique()} unique `text`.')\nprint(f'In train dataset, we have {train[\"text_fqid\"].nunique()} unique `text_fqid`.')\nprint(\"\\nThe relationship between `text` and `text_fqid` is N:N\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:34.436919Z","iopub.execute_input":"2023-04-21T05:10:34.437340Z","iopub.status.idle":"2023-04-21T05:10:34.713393Z","shell.execute_reply.started":"2023-04-21T05:10:34.437300Z","shell.execute_reply":"2023-04-21T05:10:34.712298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\nThe top 10 frequency `text` are:\")\nt = train.groupby(by = [\"text\"])[\"session_id\"].count()\nt = t.sort_values(ascending=False).head(10).to_frame().reset_index()\nt = t.rename(columns={\"session_id\": \"frequency of event\"})\nt.style.format(precision=0)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:41.210096Z","iopub.execute_input":"2023-04-21T05:10:41.210529Z","iopub.status.idle":"2023-04-21T05:10:41.492503Z","shell.execute_reply.started":"2023-04-21T05:10:41.210490Z","shell.execute_reply":"2023-04-21T05:10:41.491120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"\\nThe top 10 frequency `text_fqid` are:\")\nt = train.groupby(by = [\"text_fqid\"])[\"session_id\"].count()\nt = t.sort_values(ascending=False).head(10).to_frame().reset_index()\nt = t.rename(columns={\"session_id\": \"frequency of event\"})\nt.style.format(precision=0)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:44.000197Z","iopub.execute_input":"2023-04-21T05:10:44.000626Z","iopub.status.idle":"2023-04-21T05:10:44.263771Z","shell.execute_reply.started":"2023-04-21T05:10:44.000589Z","shell.execute_reply":"2023-04-21T05:10:44.262752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"####
\n\n\n\n\n\n\n\n---\n* `fullscreen` is whether the player is in fullscreen mode\n* `hq` is whether the game is in high-quality\n* `music` is whether the game music is on or off","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16, 6))\n\nplt.subplot(1, 3, 1)\nfullscreen = train[\"fullscreen\"].value_counts()\nplt.bar(fullscreen.index, fullscreen.values, edgecolor=\"black\", color = \"#6495ED\")\nplt.title(\"frequency of `fullscreen`\", fontsize = 20)\nplt.ylabel(\"count of event\", fontsize = 16)\nplt.xticks(ticks = [0, 1])\n\nplt.subplot(1, 3, 2)\nhq = train[\"hq\"].value_counts()\nplt.bar(hq.index, hq.values, edgecolor=\"black\", color = \"#9FE2BF\")\nplt.title(\"frequency of `hq`\", fontsize = 20)\nplt.ylabel(\"count of event\", fontsize = 16)\nplt.xticks(ticks = [0, 1])\n\nplt.subplot(1, 3, 3)\nmusic = train[\"music\"].value_counts()\nplt.bar(music.index, music.values, edgecolor=\"black\", color = \"#40E0D0\")\nplt.title(\"frequency of `music`\", fontsize = 20)\nplt.ylabel(\"count of event\", fontsize = 16)\nplt.xticks(ticks = [0, 1])\n\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:10:49.398320Z","iopub.execute_input":"2023-04-21T05:10:49.398760Z","iopub.status.idle":"2023-04-21T05:10:50.249988Z","shell.execute_reply.started":"2023-04-21T05:10:49.398711Z","shell.execute_reply":"2023-04-21T05:10:50.248881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Most of the players turned on music, and disabled fullscreen and high-quality as default.","metadata":{}},{"cell_type":"code","source":"for col in ['fullscreen', 'hq', \"music\"]:\n piv = train.groupby(['session_id', col])[\"index\"].count()\n piv = piv[piv != 0]\n session = piv.index.get_level_values(0)\n switched_music_amt = session.duplicated(keep='first').sum()\n print(f\"{switched_music_amt} player has switched {col} during playing the game\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:11:15.790511Z","iopub.execute_input":"2023-04-21T05:11:15.790936Z","iopub.status.idle":"2023-04-21T05:11:20.048430Z","shell.execute_reply.started":"2023-04-21T05:11:15.790900Z","shell.execute_reply":"2023-04-21T05:11:20.047405Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n6 FEATURE ENGINEERRING ⤒
","metadata":{}},{"cell_type":"markdown","source":"#### \n\n\n\n\n\n---\n\n* We will select and transform the raw train data into features that can be used in our supervised classfication model.\n* This notebook does some basic feature engineering, to make your model more accurate and robust, you may need to explore more meaningful features for your model.","metadata":{}},{"cell_type":"code","source":"def get_the_pct_of_event_for_each_event_name(df):\n dfs = []\n temp = df.groupby(by = [\"session_id\", \"event_name\"])[\"index\"].count()\n event_total = df.groupby(by = [\"session_id\"])[\"index\"].count()\n event_names = temp.index.get_level_values(1).unique()\n for event_name in event_names:\n temp_by_event_name = temp.loc[:, event_name] / event_total\n temp_by_event_name = temp_by_event_name.rename(f\"{event_name}_pct\")\n dfs.append(temp_by_event_name)\n return dfs\n\nprint(\"\\nThis function geneates the percentage for each `event_name` over all event\")\nprint(\"Let's take a look at those features:\")\ntrain1 = train[train[\"level_group\"] == \"0-4\"]\nfeatures_series = get_the_pct_of_event_for_each_event_name(train1)\nfeatures_df = to_df(features_series)\ncheck_features(features_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-21T05:19:32.879247Z","iopub.execute_input":"2023-04-21T05:19:32.879694Z","iopub.status.idle":"2023-04-21T05:19:35.762946Z","shell.execute_reply.started":"2023-04-21T05:19:32.879650Z","shell.execute_reply":"2023-04-21T05:19:35.761908Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_basic_aggregate_statistics_for_numeric_varaibles(df):\n dfs = []\n \n temp = df.groupby(by = [\"session_id\"])[\"elapsed_time\"].max() - df.groupby(by = [\"session_id\"])[\"elapsed_time\"].min()\n temp = temp.rename(\"elapesed_duration\")\n dfs.append(temp)\n \n temp = df.groupby(by = [\"session_id\"])[\"hover_duration\"].sum()\n temp = temp.rename(\"hover_duration_sum\")\n dfs.append(temp)\n \n temp = df.groupby(by = [\"session_id\"])[\"hover_duration\"].mean().fillna(0)\n temp = temp.rename(\"hover_duration_mean\")\n dfs.append(temp)\n \n temp = df.groupby(by = [\"session_id\"])[\"hover_duration\"].count()\n temp = temp.rename(\"hover_duration_count\")\n dfs.append(temp)\n \n return dfs\n\nprint(\"\\nThis function geneates some basic aggregate statistics for numeric varaibles\")\nprint(\"Let's take a look at those features:\")\ntrain1 = train[train[\"level_group\"] == \"0-4\"]\nfeatures_series = get_basic_aggregate_statistics_for_numeric_varaibles(train1)\nfeatures_df = to_df(features_series)\ncheck_features(features_df)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-04-21T05:21:00.127132Z","iopub.execute_input":"2023-04-21T05:21:00.127550Z","iopub.status.idle":"2023-04-21T05:21:01.430723Z","shell.execute_reply.started":"2023-04-21T05:21:00.127515Z","shell.execute_reply":"2023-04-21T05:21:01.429315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"categorical_variables = [\"page\", \"level\"] \ndef get_basic_aggregate_statistics_for_categorical_varaibles(df, categorical_variables):\n dfs = []\n for variable in categorical_variables:\n temp = df.groupby(by = [\"session_id\", variable])[\"index\"].count()\n eles = temp.index.get_level_values(1).unique()\n for ele in eles:\n temp_by_ele = temp.loc[:, ele]\n temp_by_ele = temp_by_ele.rename(f\"{variable}_{ele}_count\")\n dfs.append(temp_by_ele)\n return dfs\n\nprint(\"\\nThis function geneates some basic aggregate statistics for categorical varaibles: `page` and `level`\")\nprint(\"Let's take a look at those features:\")\ntrain1 = train[train[\"level_group\"] == \"0-4\"]\nfeatures_series = get_basic_aggregate_statistics_for_categorical_varaibles(train1, categorical_variables)\nfeatures_df = to_df(features_series)\ncheck_features(features_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-21T05:22:32.367141Z","iopub.execute_input":"2023-04-21T05:22:32.367534Z","iopub.status.idle":"2023-04-21T05:22:35.126796Z","shell.execute_reply.started":"2023-04-21T05:22:32.367500Z","shell.execute_reply":"2023-04-21T05:22:35.125447Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Let's use the function above to generate some aggregate statistics for other categorical varaibles: `event_name` and `name`\")\ntrain1 = train[train[\"level_group\"] == \"0-4\"]\ncategorical_variables = [\"event_name\", \"name\"] \nfeatures_series = get_basic_aggregate_statistics_for_categorical_varaibles(train1, categorical_variables)\nfeatures_df = to_df(features_series)\ncheck_features(features_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-21T05:24:37.961253Z","iopub.execute_input":"2023-04-21T05:24:37.961816Z","iopub.status.idle":"2023-04-21T05:24:42.158389Z","shell.execute_reply.started":"2023-04-21T05:24:37.961770Z","shell.execute_reply":"2023-04-21T05:24:42.156804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_game_config_data(df):\n dfs = []\n for game_config in [\"fullscreen\", \"hq\", \"music\"]:\n temp = df.groupby(by = [\"session_id\"])[game_config].first()\n temp.rename(game_config)\n dfs.append(temp)\n return dfs\n\nprint(\"Let's use the game config data (`fullscreen', `hq`, `music`) as features.\")\ntrain1 = train[train[\"level_group\"] == \"0-4\"]\ncategorical_variables = [\"event_name\", \"name\"] \nfeatures_series = get_game_config_data(train1)\nfeatures_df = to_df(features_series)\ncheck_features(features_df)","metadata":{"execution":{"iopub.status.busy":"2023-04-21T05:26:40.607288Z","iopub.execute_input":"2023-04-21T05:26:40.607871Z","iopub.status.idle":"2023-04-21T05:26:41.750117Z","shell.execute_reply.started":"2023-04-21T05:26:40.607829Z","shell.execute_reply":"2023-04-21T05:26:41.748110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### \n\n\n\n\n\n---\n\nWhen we submit our model, we need to use the competition API. The competition API presents the questions and data to us in order of `level_group` - level segments 0-4, 5-12, and 13-22 are each provided in sequence, and we will be predicting the correctness of each segment's questions as they are presented. Therefore, we also split train data into 3 batches by `level_group` - level segments 0-4, 5-12, and 13-22, and then generate the features table for each train data batch.","metadata":{}},{"cell_type":"code","source":"def concat_features(df):\n dfs1 = get_the_pct_of_event_for_each_event_name(df)\n dfs2 =get_basic_aggregate_statistics_for_numeric_varaibles(df)\n categorical_variables = [\"page\", \"level\", \"event_name\", \"name\"]\n dfs3 = get_basic_aggregate_statistics_for_categorical_varaibles(df, categorical_variables)\n dfs4 = get_game_config_data(df)\n\n dfs_total = dfs1 + dfs2 + dfs3 + dfs4\n indepdent_varaibles = pd.concat(dfs_total,axis=1)\n indepdent_varaibles = indepdent_varaibles.reset_index()\n indepdent_varaibles = indepdent_varaibles.set_index('session_id')\n return indepdent_varaibles\n\nprint(\"\\n...GENERATE FEATURES TABLES...\\n\")\n\ntrain1 = train[train[\"level_group\"] == \"0-4\"]\nFEATURES1 = concat_features(train1)\nprint(\"\\tFEATURES1 SUCCESSFULLY GENERATED\")\n\ntrain2 = train[train[\"level_group\"] == \"5-12\"]\nFEATURES2 = concat_features(train2)\nprint(\"\\tFEATURES2 SUCCESSFULLY GENERATED\")\n\ntrain3 = train[train[\"level_group\"] == \"13-22\"]\nFEATURES3 = concat_features(train3)\nprint(\"\\tFEATURES3 SUCCESSFULLY GENERATED\")\n\nprint(\"\\n...GENERATE FEATURES TABLES COMPLETE...\")\n\ndel train, train1, train2, train3","metadata":{"execution":{"iopub.status.busy":"2023-04-21T05:28:04.157360Z","iopub.execute_input":"2023-04-21T05:28:04.157763Z","iopub.status.idle":"2023-04-21T05:28:43.607433Z","shell.execute_reply.started":"2023-04-21T05:28:04.157728Z","shell.execute_reply":"2023-04-21T05:28:43.605690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f\"\\nThe shape of FEATURES1 is {FEATURES1.shape}\")\nprint(f\"The shape of FEATURES2 is {FEATURES2.shape}\")\nprint(f\"The shape of FEATURES3 is {FEATURES3.shape}\")","metadata":{"execution":{"iopub.status.busy":"2023-04-21T05:29:05.410879Z","iopub.execute_input":"2023-04-21T05:29:05.411426Z","iopub.status.idle":"2023-04-21T05:29:05.421194Z","shell.execute_reply.started":"2023-04-21T05:29:05.411382Z","shell.execute_reply":"2023-04-21T05:29:05.418610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### \n\n\n\n\n\n---\n* The target variable is whether the players answered questions correctly at first attempt.\n* We have 18 questions to be predicted.\n * question 1 ~ 3 will be predicted by the first batch of train data: Features1\n * question 4 ~ 13 will be predicted by second batch of train data: Features2\n * question 14 ~ 18 will be predicted by third batch of train data: Features3","metadata":{}},{"cell_type":"code","source":"def get_dependent_variable_matrix(df):\n df[\"question\"] = df[\"session_id\"].str.split(\"_\").str[1].str[1:].astype(int)\n df[\"session\"] = df[\"session_id\"].str.split(\"_\").str[0]\n return df\n\nTARGETS = get_dependent_variable_matrix(train_labels)\nprint(f\"The shape of TARGETS {TARGETS.shape}\")\nTARGETS","metadata":{"execution":{"iopub.status.busy":"2023-04-21T05:29:11.522081Z","iopub.execute_input":"2023-04-21T05:29:11.522551Z","iopub.status.idle":"2023-04-21T05:29:15.060147Z","shell.execute_reply.started":"2023-04-21T05:29:11.522510Z","shell.execute_reply":"2023-04-21T05:29:15.058607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n7 BASELINE MODELS ⤒
\n\n
","metadata":{}},{"cell_type":"markdown","source":"#### \n\n\n\n---\n* This notebook provides a baseline XGB model\n* This notebook provides a submission demo","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import GridSearchCV\nfrom sklearn.model_selection import StratifiedKFold\nimport xgboost as xgb\nimport warnings\n\nMODELS = {}\nF1 = {}\nparameters = {\n# 'max_depth': [2, 3, 4], \n# 'n_estimators': [20, 50, 100],\n# 'learning_rate': [0.01, 0.05, 0.1]\n 'max_depth': [1, 2]\n}\n\nfor question in range(1, 19):\n print(f'\\nTRAIN QUESTION {question} MODEL')\n \n if question <= 3:\n X = FEATURES1\n elif question <= 13:\n X = FEATURES2\n elif question <= 18:\n X = FEATURES3\n y = TARGETS[TARGETS[\"question\"] == question][\"correct\"]\n\n model_xgb = xgb.XGBClassifier(random_state = 1)\n model_xgb = GridSearchCV(\n model_xgb, \n parameters, \n cv=2,\n scoring='f1')\n model_xgb.fit(X, y)\n \n MODELS[f\"question {question} model\"] = model_xgb\n F1[f\"question {question} f1\"] = model_xgb.best_score_\n print(f\"\\tf1 score is {model_xgb.best_score_:.3f}\")\n print(f\"\\tbest params are {model_xgb.best_params_}\")\n print(f'QUESTION {question} MODEL COMPLETE')","metadata":{"execution":{"iopub.status.busy":"2023-04-20T11:04:37.378303Z","iopub.status.idle":"2023-04-20T11:04:37.379289Z","shell.execute_reply.started":"2023-04-20T11:04:37.379056Z","shell.execute_reply":"2023-04-20T11:04:37.379080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n8 SUBMISSION ⤒
\n\n
","metadata":{}},{"cell_type":"code","source":"# The iter_test can be called twice by the code below, thanks for @SERGEY BRYANSKY\n# https://www.kaggle.com/competitions/predict-student-performance-from-game-play/discussion/384814\ntry:\n env = jo_wilder.make_env()\n iter_test = env.iter_test()\n print(\"env made!\")\nexcept:\n jo_wilder.make_env.__called__ = False\n type(env)._state = type(type(env)._state).__dict__['INIT']\n env = jo_wilder.make_env()\n iter_test = env.iter_test()\n print(\"env re made!\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch = 1\nfor (test, sample_submission) in iter_test:\n X_test = concat_features(test)\n all_y_pred = []\n \n if batch%3 == 1:\n for question in [\"1\", \"2\", \"3\"]:\n y_pred = MODELS[f\"question {question} model\"].predict(X_test).tolist()\n all_y_pred.extend(y_pred)\n elif batch%3 == 2:\n for question in [\"4\", \"5\", \"6\", \"7\", \"8\", \"9\", \"10\", \"11\", \"12\", \"13\"]:\n y_pred = MODELS[f\"question {question} model\"].predict(X_test).tolist()\n all_y_pred.extend(y_pred)\n elif batch%3 == 0:\n for question in [\"14\", \"15\", \"16\", \"17\", \"18\"]:\n y_pred = MODELS[f\"question {question} model\"].predict(X_test).tolist()\n all_y_pred.extend(y_pred)\n \n sample_submission['correct'] = all_y_pred\n env.predict(sample_submission)\n \n batch += 1\n","metadata":{"execution":{"iopub.status.busy":"2023-04-20T11:04:37.384475Z","iopub.status.idle":"2023-04-20T11:04:37.385428Z","shell.execute_reply.started":"2023-04-20T11:04:37.385218Z","shell.execute_reply":"2023-04-20T11:04:37.385242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pd.read_csv(\"submission.csv\")\ndf","metadata":{"execution":{"iopub.status.busy":"2023-04-20T11:04:37.386395Z","iopub.status.idle":"2023-04-20T11:04:37.387393Z","shell.execute_reply.started":"2023-04-20T11:04:37.387179Z","shell.execute_reply":"2023-04-20T11:04:37.387205Z"},"trusted":true},"execution_count":null,"outputs":[]}]}