![](\"https://content.pbswisconsineducation.org/wp-content/uploads/2021/06/25221325/jw-facebook-playthegame.png\")
\n \n\nLearning is meant to be fun, which is where game-based learning comes in. This educational approach allows students to engage with educational content inside a game framework, making it enjoyable and dynamic. Although game-based learning is being used in a growing number of educational settings, there are still a limited number of open datasets available to apply data science and learning analytic principles to improve game-based learning.
\nMost game-based learning platforms do not sufficiently make use of knowledge tracing to support individual students. Knowledge tracing methods have been developed and studied in the context of online learning environments and intelligent tutoring systems. But there has been less focus on knowledge tracing in educational games.
\nCompetition host Field Day Lab is a publicly-funded research lab at the Wisconsin Center for Educational Research. They design games for many subjects and age groups that bring contemporary research to the public, making use of the game data to understand how people learn. Field Day Lab's commitment to accessibility ensures all of its games are free and available to anyone. The lab also partners with nonprofits like The Learning Agency Lab, which is focused on developing science of learning-based tools and programs for the social good.
\nIf successful, you'll enable game developers to improve educational games and further support the educators who use these games with dashboards and analytic tools. In turn, we might see broader support for game-based learning platforms. - Context\n
\n \n\nThe goal of this competition is to predict student performance during game-based learning in real-time. You'll develop a model trained on one of the largest open datasets of game logs.
\nYour work will help advance research into knowledge-tracing methods for game-based learning. You'll be supporting developers of educational games to create more effective learning experiences for students.\n - Goal of the Competition\n
\n Submissions will be evaluated based on their F1 score. - Evaluation
\n
\n\n F-score or F-measure is a measure of a test's accuracy. It is calculated from the precision and recall of the test, where the precision is the number of true positive results divided by the number of all positive results, including those not identified correctly, and the recall is the number of true positive results divided by the number of all samples that should have been identified as positive. Precision is also known as positive predictive value, and recall is also known as sensitivity in diagnostic binary classification.
\nThe F1 score is the harmonic mean of the precision and recall. The more generic $ F_{\\beta } $ score applies additional weights, valuing one of precision or recall more than the other.
\nThe highest possible value of an F-score is 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if either precision or recall are zero.\n - Wikipedia
\n
\n$ F_1 = \\frac{2}{recall^{-1} + precision^{-1}} = 2\\frac{precision \\cdot recall}{precision + recall} = \\frac{2tp}{2tp + fp + fn} $\n
","metadata":{}},{"cell_type":"markdown","source":"\nWe are hosting a second track that focuses on model efficiency, because highly accurate models are often computationally heavy. Such models have a stronger carbon footprint and frequently prove difficult to utilize in real-world educational contexts. We hope to use these models to help educational organizations, which have limited computational capabilities. - Efficiency Prize\n
\n\n We compute a submission's efficiency score by:\n
\n\n$ \\text{Efficiency} = \\frac{1}{ \\text{Benchmark} - \\max\\text{F1} }\\text{F1} + \\frac{1}{32400}\\text{RuntimeSeconds} $\n
\n\n where $ F1 $ is the submission's score on the main competition metric, $ Benchmark $ is the score of the benchmark sample_submission.csv, $ maxF1 $ is the maximum of all submissions on the Private Leaderboard, and $ RuntimeSeconds $ is the number of seconds it takes for the submission to be evaluated. The objective is to minimize the efficiency score. - Evaluation Metric\n
\n\nThis competition uses the Kaggle's time series API. Test data will be delivered in groupings that do not allow access to future data. The objective of this competition is to use time series data generated by an online educational game to determine whether players will answer questions correctly. There are three question checkpoints (level 4, level 12, and level 22), each with a number of questions. At each checkpoint, you will have access to all previous test data for that section. - Dataset Description
\nYou have access to the training data and labels. There are 18 questions for each sessions - you are not given the answers, but are simply told whether the user for a particular session answered each question correctly.
\nWhen you are ready to predict, use the sample notebook to iterate over the test data, which is split as described above and served up as Pandas dataframes. Make your predictions for each group of questions - at the end of this process a submission.csv file will have been created for you. Simply submit your notebook. - What files do I need?\n
\nThe training columns are as listed below. The label rows are identified with `session_id_question #`. Each session will have 18 rows, representing 18 questions. - What should I expect the data format to be?\n
\n train.csv - the training set\n
","metadata":{}},{"cell_type":"code","source":"train","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T20:04:58.898006Z","iopub.execute_input":"2023-02-09T20:04:58.898352Z","iopub.status.idle":"2023-02-09T20:04:58.927335Z","shell.execute_reply.started":"2023-02-09T20:04:58.898315Z","shell.execute_reply":"2023-02-09T20:04:58.926328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n train_labels.csv - correct value for all 18 questions for each session in the training set\n
","metadata":{}},{"cell_type":"code","source":"train_labels","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T20:04:58.930309Z","iopub.execute_input":"2023-02-09T20:04:58.931010Z","iopub.status.idle":"2023-02-09T20:04:58.944433Z","shell.execute_reply.started":"2023-02-09T20:04:58.930970Z","shell.execute_reply":"2023-02-09T20:04:58.943070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n test.csv - the test set\n
","metadata":{}},{"cell_type":"code","source":"test","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T20:04:58.946200Z","iopub.execute_input":"2023-02-09T20:04:58.946641Z","iopub.status.idle":"2023-02-09T20:04:58.976933Z","shell.execute_reply.started":"2023-02-09T20:04:58.946563Z","shell.execute_reply":"2023-02-09T20:04:58.975864Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n sample_submission.csv - a sample submission file in the correct format\n
","metadata":{}},{"cell_type":"code","source":"sample_submission","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T20:04:58.978490Z","iopub.execute_input":"2023-02-09T20:04:58.978854Z","iopub.status.idle":"2023-02-09T20:04:58.996826Z","shell.execute_reply.started":"2023-02-09T20:04:58.978818Z","shell.execute_reply":"2023-02-09T20:04:58.995963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nSometimes some data are duplicated and because of duplicates, the model becomes slightly overfitted, because, roughly speaking, it looks at the same samples during training, i.e. don't observe other \"cases\".\n
","metadata":{}},{"cell_type":"code","source":"exclude_columns = (\"index\")\ncolumns = [column for column in train.columns if column not in exclude_columns]\nduplicates = train[train.duplicated(subset=columns)]\n\nnum_samples = len(train)\nnum_duplicates = len(duplicates)\nprint(f\"Number of duplicates: {num_duplicates}\")\n\nduplicates_percentage = (num_duplicates / num_samples) * 100\nduplicates_percentage = round(duplicates_percentage, 2)\nprint(f\"Percentage of duplicates: {duplicates_percentage}%\")\nprint()\n\nduplicates","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T20:05:09.647690Z","iopub.execute_input":"2023-02-09T20:05:09.648467Z","iopub.status.idle":"2023-02-09T20:05:42.776111Z","shell.execute_reply.started":"2023-02-09T20:05:09.648428Z","shell.execute_reply":"2023-02-09T20:05:42.775084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n Missing data, or missing values, occur when no data value is stored for the variable in an observation. Missing data are a common occurrence and can have a significant effect on the conclusions that can be drawn from the data. - Wikipedia\n
\n\nThere are many techniques to produce the imputation/filling of missing values. The basic technique is to use mean or median for missed numerical values and mode (the most frequent value) for missed categorical values respectively. More powerful and accurate approaches are k Nearest Neighbors (KNN) or even training additional model(s), where the target(s) is (are) the missed column(s).\n
\n\nThe following papers provide more information about missing values and imputation techniques:\n
\n\nSome columns don't have any information, i.e the columns are fully missed, so let's drop them from the future analysis. \n
","metadata":{}},{"cell_type":"code","source":"missed_columns = [\"fullscreen\", \"hq\", \"music\", \"page\", \"hover_duration\"]\ntrain = train.drop(missed_columns, axis=1)\ntest = test.drop(missed_columns, axis=1)","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2023-02-09T20:05:42.791475Z","iopub.status.idle":"2023-02-09T20:05:42.792291Z","shell.execute_reply.started":"2023-02-09T20:05:42.792026Z","shell.execute_reply":"2023-02-09T20:05:42.792055Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n Optimization as well as other steps of project development is the necessary intermediate step. Due to proper optimization productivity rate is increased, hence allows making experiments much faster, especially in Data Science competitions.
\nThere are a lot of techniques to optimize data pre-processing (e.g Parallelling and reducing data types) and training models (e.g. Intel® Extension). Also, operations can be applied on GPU machines with help of the RAPIDS library, which is made by NVIDIA. \n
\nExploratory Data Analysis (EDA) is an integral and the most important stage during developing any Data Science projects. Exploratory Data Analysis can give important observations, relations and even ideas for further defining hypotheses and testing, which can be very beneficial for model development. From extracted observations, we can decide what metric, pre-processing, feature engineering, validation strategy, post-processing, model, etc, we should use, which potentially can improve model generalization, hence, improving of predictive ability. \n
","metadata":{}},{"cell_type":"markdown","source":"There is slight class imbalance. Because of imbalance model will be biased to more balanced classes, i.e will be overfitted and won’t generalize well for imbalance classes. There are tecniques to slightly smooth that problem: class-wise weights, oversampling or under-sampling, extracting external or generating synthetic data, and apply augmentations during training.
","metadata":{}},{"cell_type":"markdown","source":"\nWe are observing trend/correlation: more level => more time required to complete. \n
","metadata":{}},{"cell_type":"markdown","source":"\nLet's take a view on the one game session. \n
","metadata":{}},{"cell_type":"code","source":"def visualize_session(session, plot_directory=None, plot_filename_format=\"{session_id}_{index}.png\"):\n if plot_directory is None:\n plot_directory = str(session[\"session_id\"].values[0])\n if not os.path.exists(plot_directory):\n os.mkdir(plot_directory)\n \n # screen\n screen_x_min, screen_y_min = session[\"screen_coor_x\"].min(), session[\"screen_coor_y\"].min()\n screen_x_max, screen_y_max = session[\"screen_coor_x\"].max(), session[\"screen_coor_y\"].max()\n screen_width, screen_height = screen_x_max - screen_x_min, screen_y_max - screen_y_min\n screen_padding = 25\n \n # room\n room_x_min, room_y_min = session[\"room_coor_x\"].min(), session[\"room_coor_y\"].min()\n room_x_max, room_y_max = session[\"room_coor_x\"].max(), session[\"room_coor_y\"].max()\n room_width, room_height = room_x_max - room_x_min, room_y_max - room_y_min\n \n plot_pathes = []\n for index, sample in session.iterrows():\n figure = plt.figure(figsize=(12, 10))\n axis = figure.add_subplot()\n \n # environment\n screen_rectangle = mpl.patches.Rectangle(xy=(screen_x_min, screen_y_min), width=screen_width, height=screen_height, edgecolor=\"#fff\", label=\"Screen\", color=colors[-1])\n axis.add_patch(screen_rectangle)\n \n room_rectangle = mpl.patches.Rectangle(xy=(room_x_min, room_y_min), width=room_width, height=room_height, edgecolor=\"#fff\", label=\"Room\", color=colors[2])\n axis.add_patch(room_rectangle)\n \n axis.set_xlim(screen_x_min - screen_padding, screen_x_max + screen_padding)\n axis.set_ylim(screen_y_min - screen_padding, screen_y_max + screen_padding)\n axis.set(xticks=[], yticks=[])\n axis.legend()\n \n # events\n room_x, room_y = sample[\"room_coor_x\"], sample[\"room_coor_y\"]\n screeen_x, screeen_y = sample[\"screen_coor_x\"], sample[\"screen_coor_y\"]\n axis.plot(room_x, room_y, color=colors[0], marker=\"o\", markersize=5)\n axis.plot(screeen_x, screeen_y, color=colors[3], marker=\"o\", markersize=5)\n axis.set_title(\"FQID: {fqid}, Level: {level}, Index: {index}, Event name: {event_name}\".format(**sample.to_dict()), fontsize=20)\n \n # saving\n plot_filename = plot_filename_format.format(**sample.to_dict())\n plot_path = os.path.join(plot_directory, plot_filename)\n figure.savefig(plot_path)\n plt.close(figure) \n \n plot_pathes.append(plot_path)\n \n plt_ims = [plt.imread(plot_path) for plot_path in plot_pathes]\n return run_animation(plt_ims)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T20:05:42.819394Z","iopub.status.idle":"2023-02-09T20:05:42.820108Z","shell.execute_reply.started":"2023-02-09T20:05:42.819862Z","shell.execute_reply":"2023-02-09T20:05:42.819886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"session_id = 20110112402567744\nsession = train[train[\"session_id\"] == session_id].iloc[:50]\n\nvisualize_session(session)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-02-09T20:05:42.821363Z","iopub.status.idle":"2023-02-09T20:05:42.822084Z","shell.execute_reply.started":"2023-02-09T20:05:42.821832Z","shell.execute_reply":"2023-02-09T20:05:42.821858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n The game is mostly played at the beginning of the school year (August, September, October, November, and December months) and during the school day (from 8 a.m. to 15p.m., then increasingly are appeared in the evening (maybe children do homework). Each year, the mean gaming time is increasing linearly.\n
","metadata":{}}]}