\n
Table of content
\n \n[0. Understanding Our Data (Univariate)](#0)\n* [0.0 Check for missing values](#0.0)\n* [0.1 Timestamp](#0.1)\n* [0.2 User Events](#0.2)\n* [0.3 Content Id](#0.3)\n* [0.4 Content Type Id](#0.4)\n* [0.5 Task Container Id](#0.5)\n* [0.6 User Answers](#0.6)\n \n[1. Exploring Multivariate Relationships](#1)\n* [1.0 Prior question elapsed time against answered correctly](#1.0)\n* [1.1 Prior question had explanation against answered correctly](#1.1)"},{"metadata":{},"cell_type":"markdown","source":"First let's import the module and create an environment."},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\nimport pandas as pd\n\n# You can only call make_env() once, so don't lose it!\nenv = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training data is in the competition dataset as usual\nIt's larger than will fit in memory with default settings, so we'll specify more efficient datatypes and only load a subset of the data for now."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', low_memory=False, nrows=10**5, \n dtype={'row_id': 'int64', 'timestamp': 'int64', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n 'task_container_id': 'int16', 'user_answer': 'int8', 'answered_correctly': 'int8', 'prior_question_elapsed_time': 'float32', \n 'prior_question_had_explanation': 'boolean',\n }\n )\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n## Understanding Our Data (Univariate)"},{"metadata":{},"cell_type":"markdown","source":"
\n### Check for missing values:\n---\n\n
Go to TOC"},{"metadata":{"trusted":true},"cell_type":"code","source":"def display_missing(df): \n for col in df.columns.tolist(): \n print('{} column missing values: {}'.format(col, df[col].isnull().sum()))\n print('\\n')\n \n\ndisplay_missing(train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n### Timestamp\n---\n\n
Go to TOC"},{"metadata":{},"cell_type":"markdown","source":"Let's look at timestamp"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10,8))\nax1 = fig.add_subplot(211)\nax1.hist(train_df['timestamp'], bins = 40)\nax1.set_ylabel('Frequency')\nax1.set_xlabel(\"Timestamp\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Timestamp is left skewed, many of the attempts were completed early with a questions being answered later"},{"metadata":{},"cell_type":"markdown","source":"
\n\n### User Events\n---\n\n
Go to TOC"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10,8))\nax1 = fig.add_subplot(211)\nax1.hist(train_df.user_id.value_counts(), bins = 20)\nax1.set_ylabel('Frequency')\nax1.set_xlabel(\"Number of user events\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Number of user events by students are also left skewed"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.user_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 349 students in our sampled data"},{"metadata":{},"cell_type":"markdown","source":"
\n\n### Content Id\n---\n\n
Go to TOC"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.content_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 11532 different user interactions in our sampled data"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10,8))\nax1 = fig.add_subplot(211)\nax1.hist(train_df.content_id.value_counts(), bins = 20)\nax1.set_ylabel('Frequency')\nax1.set_xlabel(\"Counts of user interactions\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most user interactions happen infrequently"},{"metadata":{},"cell_type":"markdown","source":"
\n### Content Type Id\n---\n\n
Go to TOC"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nfig = plt.figure(figsize=(8,6))\nsns.countplot(x='content_type_id', data=train_df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.content_type_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the time, question is being posed to user. For other times, user was watching a lecture."},{"metadata":{},"cell_type":"markdown","source":"
\n### Task Container Id\n---\n\nId code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id. Monotonically increasing for each user.\n\n
Go to TOC"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.task_container_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10,8))\nax1 = fig.add_subplot(211)\nax1.hist(train_df.task_container_id.value_counts(), bins = 20)\nax1.set_ylabel('Frequency')\nax1.set_xlabel(\"Occurence of batch of questions\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most occurence for batch of questions happen infrequently"},{"metadata":{},"cell_type":"markdown","source":"
\n### User Answers\n---\n\n
Go to TOC"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.user_answer.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```user_answer``` is the user's answer to the question, -1 corresponds to no answer (user is watching lecture).
Note that user answers will actually be dependent on question posed "},{"metadata":{},"cell_type":"markdown","source":"
\n## Exploring Multivariate Relationships"},{"metadata":{},"cell_type":"markdown","source":"Let's check prior question elapsed time against answered correctly"},{"metadata":{},"cell_type":"markdown","source":"
\n### Prior question elapsed time against answered correctly\n---\n\n\n
Go to TOC"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(8,6))\nsns.catplot(x=\"answered_correctly\", y=\"prior_question_elapsed_time\",\n kind=\"violin\", data=train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Users who answered correctly have a lower median prior question elapsed time, but otherwise, we cannot easily seperate answers that are correct and wrong based on prior question elapsed time"},{"metadata":{},"cell_type":"markdown","source":"
\n### Prior question had explanation against answered correctly\n---\n\n\n
Go to TOC"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(8,6))\nsns.countplot(x=\"prior_question_had_explanation\", hue=\"answered_correctly\", data=train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interesting for prior questions that had explanations, they tend to have more correct answers (proportionally and countwise) than prior questions without explanations"},{"metadata":{},"cell_type":"markdown","source":"(To be continued)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}