{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Predict Student Performance from Game Play - initial EDA\n\n![imagen.png](https://images.pexels.com/photos/735911/pexels-photo-735911.jpeg?auto=compress&cs=tinysrgb&w=1260&h=750&dpr=1)!\n\nThis notebook will be the first revision of the Kaggle competition with the same name, as the main objective is to predict student performance during game-based learning. \n\nThe related data was quite a challenge to understand, as this information contains the different behavior related to in-game mechanics such as clicks, problem-solving, etc. For this competition, the analyzed game was **Jo Wilder and the Capitol Case**.\n\nThen, this first notebook is structured in the following points:\n\n- Game understanding and objectives\n- Table analysis\n- Questions remarks\n- Final Insights and next steps\n- Bibliography","metadata":{}},{"cell_type":"markdown","source":"# 1. Game understanding and objectives\n\nBefore navigating through the data, I made an initial walkthrough to find information related to in-game mechanics and patterns that could affect the game's performance. First, this game is hosted on the Field Day Learning Games, where the main objective is to \"design games that bring contemporary research to the public\" alongside the usage of data to understand how people learn [1]\n\nFor this reason, Kaggle and the Field Day Lab hosted a competition where the aim was to predict the answer to 18 questions related to user interaction with the interface. The chosen game was Jo Wilder and the Capitol case, focused on grades 3-5 in the topic of history. \n\n![image.png](https://images.pexels.com/photos/5088021/pexels-photo-5088021.jpeg?auto=compress&cs=tinysrgb&w=1260&h=750&dpr=1)\n\n## 1.1. Context\n\nThis competition aims to identify ways for students to engage with educational content while tackling the problem of a limited number of datasets available to apply data science, especially in the field of educational games [2]. Then, when making a walkthrough the following elements should be considered when trying to understand the data:\n\n1. Characters: In this game, you have to interact with different characters and visit a wide variety of locations to get clues and solve mysteries related to Wisconsin history.\n2. Mouse and clicks: To move in the different landscapes you must use the mouse and click where you want to travel. This is also true with the interaction with the characters.\n3. Notebook: To make the information available, it was created a place where you can save the main findings of each of the clues you get through the game.\n4. Time: The time you spend in each of the places depends on how you understand the game platform, alongside the analysis and possible distractions through gameplay\n\nWith this in consideration, the delivered tables were analyzed for one user, as before making a high-scale data treatment is important to understand how is related to the user behavior and different levels:","metadata":{}},{"cell_type":"markdown","source":"# 2. Table analysis\n\nKaggle introduced the following data related to the csv and feature meaning, starting with the dataset [2]:\n\n- session_id - the ID of the session the event took place in - Related to the **user**\n- index - the index of the event for the session\n- elapsed_time - how much time has passed (in milliseconds) between the start of the session and when the event was recorded - Time series\n- event_name - the name of the event type\n- name - **the event name**\n- level - what level of the game the event occurred in \n- page - the page number of the event\n- room_coor_x - the coordinates of the **click** in reference to the in-game room\n- room_coor_y - the coordinates of the **click** in reference to the in-game room\n- screen_coor_x - the coordinates of the **click** in reference to the player’s screen\n- screen_coor_y - the coordinates of the **click** in reference to the player’s screen \n- hover_duration - how long (in milliseconds) the hover happened for\n- text - **the text the player sees during this event**\n- fqid - the fully-qualified ID of the event\n- room_fqid - the fully-qualified ID of the room the event took place in\n- text_fqid - the fully-qualified ID of the\n- fullscreen - whether the player is in fullscreen mode\n- hq - whether the game **is in high-quality**\n- music - whether the game music is on or off\n- level_group - which group of levels - and group of questions - this row belongs to (0-4, 5-12, 13-22)\n\nAs referred on the text, there are clue factors such as the user ID and the fact that the data is related to each of the events that happened through the game","metadata":{}},{"cell_type":"markdown","source":"## 2.1. Libraries\n\nFor this analysis, I used pandas, seaborn and matplotlib to identify the patterns of a single session.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport numpy as np","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-04-01T22:36:59.430826Z","iopub.execute_input":"2023-04-01T22:36:59.433633Z","iopub.status.idle":"2023-04-01T22:37:00.154119Z","shell.execute_reply.started":"2023-04-01T22:36:59.433563Z","shell.execute_reply":"2023-04-01T22:37:00.153077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, I loaded the data:","metadata":{}},{"cell_type":"code","source":"# https://www.kaggle.com/code/narendra/pspg-elapse-time-eda\ndf=pd.read_csv(\"/kaggle/input/predict-student-performance-from-game-play/train.csv\",\n                       usecols=['session_id', 'index', 'elapsed_time', 'event_name', 'name', 'level',\n       'room_coor_x', 'room_coor_y', 'screen_coor_x', 'screen_coor_y',\n       'room_fqid', 'level_group'],\n                       dtype={'index': np.int16, 'level': np.int8})\ndf.memory_usage()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:37:00.159038Z","iopub.execute_input":"2023-04-01T22:37:00.159421Z","iopub.status.idle":"2023-04-01T22:38:28.806233Z","shell.execute_reply.started":"2023-04-01T22:37:00.159388Z","shell.execute_reply":"2023-04-01T22:38:28.803873Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Afterward, the train_labels.csv compiles the answers for each of the 18 questions as 1 (correct) and 0 (incorrect). These were made to be able to merge with the user interaction data of the previous table:","metadata":{}},{"cell_type":"code","source":"scores=pd.read_csv(\"/kaggle/input/predict-student-performance-from-game-play/train_labels.csv\")\nscores.head(20)","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:28.807720Z","iopub.execute_input":"2023-04-01T22:38:28.808045Z","iopub.status.idle":"2023-04-01T22:38:29.180030Z","shell.execute_reply.started":"2023-04-01T22:38:28.808007Z","shell.execute_reply":"2023-04-01T22:38:29.179152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, the first session was identified and filtered on the original data, obtaining a reduced dataframe","metadata":{}},{"cell_type":"code","source":"#Check first participant\nfirst=[20090312431273200]\ntest_df=df[df[\"session_id\"] == first[0]]\ntest_df.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:29.182775Z","iopub.execute_input":"2023-04-01T22:38:29.183307Z","iopub.status.idle":"2023-04-01T22:38:29.253657Z","shell.execute_reply.started":"2023-04-01T22:38:29.183272Z","shell.execute_reply":"2023-04-01T22:38:29.252801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, a simple description table was built from pandas function `.describe()`. As it was seen, the categorical data of event_name contains 10 unique values while the room_fqid has 19. However, the most important clue is related to the level_group feature, where there are only three labels 0-4, 5-12, and 13-22 that compile the events of each of the steps and are related to the questions.","metadata":{}},{"cell_type":"code","source":"test_df.describe(include=\"all\")","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:29.254896Z","iopub.execute_input":"2023-04-01T22:38:29.255412Z","iopub.status.idle":"2023-04-01T22:38:29.305850Z","shell.execute_reply.started":"2023-04-01T22:38:29.255379Z","shell.execute_reply":"2023-04-01T22:38:29.304953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2. User trends by level\n\nThe following plot shows the time spend in each of the three stages, where it is identified a time gap between each of the stages. Especially, in the transition between the second and third stages the time gap is higher than in the first transition. This is related to the time used for answering the questions between each section.\n\nAt the same time, the seconds used for each of the stages increase as the difficulty also changes:","metadata":{}},{"cell_type":"code","source":"# Evolution of numeric variables\nsns.lineplot(x=test_df[\"index\"],y=test_df[\"elapsed_time\"],hue=test_df[\"level_group\"])  \nplt.grid()\nplt.xlabel(\"Event index\")\nplt.ylabel(\"Elapsed time on milliseconds\")\nplt.title(\"Time spend on each stage\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:29.307089Z","iopub.execute_input":"2023-04-01T22:38:29.307611Z","iopub.status.idle":"2023-04-01T22:38:29.580630Z","shell.execute_reply.started":"2023-04-01T22:38:29.307577Z","shell.execute_reply":"2023-04-01T22:38:29.579671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Similarly, the following trend shows the steps or stages navigated through the game. While in the first stage the number of events is the lowest, in the other two the totals are very similar:","metadata":{}},{"cell_type":"code","source":"# Level development\nsns.lineplot(x=test_df[\"index\"],y=test_df[\"level\"],hue=test_df[\"level_group\"])\nplt.grid()\nplt.xlabel(\"Event index\")\nplt.ylabel(\"Game Level\")\nplt.title(\"Number of events per stage\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:29.584084Z","iopub.execute_input":"2023-04-01T22:38:29.584411Z","iopub.status.idle":"2023-04-01T22:38:29.798840Z","shell.execute_reply.started":"2023-04-01T22:38:29.584379Z","shell.execute_reply":"2023-04-01T22:38:29.797872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Afterward, the following plot presents the location of the clicks on the room interface, as it shows that in the last stage the mouse is used to click on the lower part of the game while in the other stages the dots are more condensed","metadata":{}},{"cell_type":"code","source":"#Room total coordinates\nsns.scatterplot(data=test_df,x=\"room_coor_x\",y=\"room_coor_y\",hue=\"level_group\")\nplt.grid()\nplt.xlabel(\"x coordinate\")\nplt.ylabel(\"y coordinate\")\nplt.title(\"Click scatterplot on room interface\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:29.802442Z","iopub.execute_input":"2023-04-01T22:38:29.802862Z","iopub.status.idle":"2023-04-01T22:38:30.099637Z","shell.execute_reply.started":"2023-04-01T22:38:29.802828Z","shell.execute_reply":"2023-04-01T22:38:30.098745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is related to the screen coordinates, where there is no trend between each of the scenarios, with a small quantity of dots condensed on the lower right screen.","metadata":{}},{"cell_type":"code","source":"#Room total coordinates\nsns.scatterplot(data=test_df,x=\"screen_coor_x\",y=\"screen_coor_y\",hue=\"level_group\")\nplt.grid()\nplt.xlabel(\"x coordinate\")\nplt.ylabel(\"y coordinate\")\nplt.title(\"Click scatterplot on screen interface\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:30.100559Z","iopub.execute_input":"2023-04-01T22:38:30.100874Z","iopub.status.idle":"2023-04-01T22:38:30.399076Z","shell.execute_reply.started":"2023-04-01T22:38:30.100843Z","shell.execute_reply":"2023-04-01T22:38:30.398008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Later, on the click meaning, the following barplot shows the category of each of the stages. The navigate clicks have the highest value while the checkpoint is only one, as it corresponds to the question answering parallel to the game. \n\nThis is an important tool to understand the user navigation through the interface, for example on the cutscene click, where in stage 0-4 the total clicks are higher than in the following stage.","metadata":{}},{"cell_type":"code","source":"#Event name - Barplot\n#Groupby count\nplt1=test_df.groupby(by=[\"event_name\",\"level_group\"]).count()[\"index\"].sort_values(ascending=False).reset_index()\nsns.barplot(data=plt1,x=\"event_name\",y=\"index\",hue=\"level_group\")\nplt.xticks(rotation=90);\nplt.grid()\nplt.xlabel(\"event name\")\nplt.ylabel(\"total clicks\")\nplt.title(\"User click behavior\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:30.400556Z","iopub.execute_input":"2023-04-01T22:38:30.401386Z","iopub.status.idle":"2023-04-01T22:38:30.761399Z","shell.execute_reply.started":"2023-04-01T22:38:30.401348Z","shell.execute_reply":"2023-04-01T22:38:30.760518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Then, on the name general feature it is shown that the basic and undefined events have the highest frequency related to the close action. However, it is important to understand what is the meaning of this event and its importance on the user performance.","metadata":{}},{"cell_type":"code","source":"#Event name - Barplot\nplt1=test_df.groupby(by=[\"name\",\"level_group\"]).count()[\"index\"].sort_values(ascending=False).reset_index()\nsns.barplot(data=plt1,x=\"name\",y=\"index\",hue=\"level_group\")\nplt.xticks(rotation=90);\nplt.grid()\nplt.xlabel(\"name\")\nplt.ylabel(\"total events\")\nplt.title(\"Main events behavior\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:30.762723Z","iopub.execute_input":"2023-04-01T22:38:30.763720Z","iopub.status.idle":"2023-04-01T22:38:30.961072Z","shell.execute_reply.started":"2023-04-01T22:38:30.763680Z","shell.execute_reply":"2023-04-01T22:38:30.960184Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, in the following figure, it is shown that each of the stages has its locations where the events happen, for example the last stage, where the clicks focused on the wildlife center and historical society cage while on the first one (0-4) it is located on historical society closet and entry.","metadata":{}},{"cell_type":"code","source":"plt1=test_df.groupby(by=[\"room_fqid\",\"level_group\"]).count()[\"index\"].sort_values(ascending=False).reset_index()\nsns.barplot(data=plt1,x=\"room_fqid\",y=\"index\",hue=\"level_group\")\nplt.xticks(rotation=90);","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:30.962390Z","iopub.execute_input":"2023-04-01T22:38:30.963014Z","iopub.status.idle":"2023-04-01T22:38:31.754135Z","shell.execute_reply.started":"2023-04-01T22:38:30.962979Z","shell.execute_reply":"2023-04-01T22:38:31.753108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Question remarks\n\nTo identify the questions table structure, the following code extracts the values for the first user, creating an additional column to use only the answers referred to the main user:","metadata":{}},{"cell_type":"code","source":"scores[\"user_id\"] = scores[\"session_id\"].str.rpartition(\"_\")[0]\nscores.head()","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:31.757665Z","iopub.execute_input":"2023-04-01T22:38:31.758354Z","iopub.status.idle":"2023-04-01T22:38:32.402486Z","shell.execute_reply.started":"2023-04-01T22:38:31.758313Z","shell.execute_reply":"2023-04-01T22:38:32.401511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Process for numeric scale, as it is saved on string\nscores[\"user_id\"]=pd.to_numeric(scores[\"user_id\"])","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:32.404046Z","iopub.execute_input":"2023-04-01T22:38:32.404713Z","iopub.status.idle":"2023-04-01T22:38:32.671151Z","shell.execute_reply.started":"2023-04-01T22:38:32.404676Z","shell.execute_reply":"2023-04-01T22:38:32.670186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Answers\nscores[scores[\"user_id\"]==first[0]]","metadata":{"execution":{"iopub.status.busy":"2023-04-01T22:38:32.672772Z","iopub.execute_input":"2023-04-01T22:38:32.673460Z","iopub.status.idle":"2023-04-01T22:38:32.691572Z","shell.execute_reply.started":"2023-04-01T22:38:32.673422Z","shell.execute_reply":"2023-04-01T22:38:32.690482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is shown that this user only failed on questions 13 and 16. However, to be able to identify the possible correlation it is key to evaluate other users and then the overall data.","metadata":{}},{"cell_type":"markdown","source":"# Final Insights and next steps\n\nAfter this initial approach, the following insights were identified:\n\n1. Importance of understanding the game: Before constructing a Machine Learning Modeling it is important to understand how people can interact with the game, as to be able to do a proper feature engineering\n2. Relation with stage and answer: as an initial approach, relating the responses with the user interaction on each scenario and stage is a possible fact to implement in a prediction modeling\n3. Quantity of data: As these data recollects each user step is important to scale the information related to the memory limits with methods such as downcasting\n4. Quality of data: In this first approach, it is seen that most of the data is available or it is clear why it is missing.\n\n![image.png](https://images.pexels.com/photos/256502/pexels-photo-256502.jpeg?auto=compress&cs=tinysrgb&w=1260&h=750&dpr=1)\n\n\nThen, the following steps would be to build a simple prediction model with a sample of the population and relate the questions to each part of the game events.\n\n**Thanks for your attention.**","metadata":{}},{"cell_type":"markdown","source":"# References\n\n[1] https://fielddaylab.wisc.edu/about/\n\n[2] https://www.kaggle.com/competitions/predict-student-performance-from-game-play\n\n[3] https://fielddaylab.wisc.edu/play/jowilder/\n\nOriginal work and steps:\n\nhttps://github.com/ksmorgadog/kaggle_student_prediction","metadata":{}}]}