{"cells":[{"metadata":{},"cell_type":"markdown","source":"![Riiid! App](https://venturebeat.com/wp-content/uploads/2020/07/1-1-e1595515659939.jpeg?w=1200&strip=all)\n\n\nThe goal of this notebook is to make an explanatory, exploratory and storytelling style walkthrough this data, I'll try to highlight all the relevant parts and update regularly to add important information. Reading this notebook from top to bottom should give you the big picture perspective on this data.\n\n### Overview of the data:\nThis data is simply records of users' interactions with an educational app, each user has his/her own unique id, these interactions are watching lectures or answering quesstions. You can think of the users as the data generating distribution since the users are given questions as input and expect answers as output. Or you can think of the app as the data generating distribution, by giving users and historic information about them as input you will expect their answers' correctness as output.\n\n### Goal of the modeling:\nOur goal in this competition is to make a model that will predict the users' answers correctness given the question and historic data about the users."},{"metadata":{},"cell_type":"markdown","source":"# 1. Importing Data:"},{"metadata":{},"cell_type":"markdown","source":"The \"train.csv\" dataframe has +100 million records, which makes it impossible to load it fully on a kaggle notebook with 16gb of ram (the size of the file is about 5gb on disk but pandas requires around x5 of ram). There are many approaches to solve this issue, the one I have chosen is @rohanrao's using a package called datatable (check out his notebook [here](https://www.kaggle.com/rohanrao/riiid-with-blazing-fast-rid))."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/python-datatable/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\n\nimport numpy as np\nimport pandas as pd\nimport datatable as dt\n\n\nimport matplotlib\nimport matplotlib.pyplot as plt\nfrom matplotlib.ticker import FuncFormatter\n\nfrom sklearn.metrics import roc_auc_score\n\nplt.style.use('ggplot')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = dt.fread('../input/riiid-test-answer-prediction/train.csv').to_pandas()\nquestions = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Train.csv:"},{"metadata":{},"cell_type":"markdown","source":"We start by describing and analyzing the train.csv columns one after the other:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.1 timestamp:"},{"metadata":{},"cell_type":"markdown","source":"This column tell us the duration between the user's first interaction and the current interaction in milliseconds.\nThis variable will give us an idea how active users were through time, let's see how this variable is distributed."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"timedelta = pd.to_timedelta(train['timestamp'], unit='ms')\nprint(f\"\\\nAverge: {timedelta.mean().floor('s')}\\n\\\nMedian: {timedelta.median().floor('s')}\\n\\\nMin:    {timedelta.min().floor('s')}\\n\\\nMax:    {timedelta.max().floor('s')}\")\ndel timedelta\n_ = gc.collect()\n\nfig = plt.figure(figsize=(15, 4))\nax = fig.add_subplot(111)\n\nax.hist((train['timestamp'] / (1000 * 60 * 60 * 24)), bins=100, rwidth=0.8)\n\nformat_yticks = matplotlib.ticker.FuncFormatter(lambda x, p: format(int(x), ','))\n\nax.set_xticks(range(0, int(train['timestamp'].max() / (1000 * 60 * 60 * 24)), 20))\nplt.xticks(rotation=70)\nax.set_xlabel('Days')\nax.yaxis.set_major_formatter(format_yticks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I ploted the variable in days, it would better to look more closely in a more meaningfull timespan..."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"hours = range(0, 1000 * 60 * 60 * 24, 1000 * 60 * 60 * 12)\ndays = range(1000 * 60 * 60 * 24, 1000 * 60 * 60 * 24 * 7, 1000 * 60 * 60 * 24)\nweeks = range(1000 * 60 * 60 * 24 * 7, 1000 * 60 * 60 * 24 * 7 * 4, 1000 * 60 * 60 * 24 * 7)\nmonths = range(1000 * 60 * 60 * 24 * 30, 1000 * 60 * 60 * 24 * 30 * 12, 1000 * 60 * 60 * 24 * 30 * 2)\nrest = [1000 * 60 * 60 * 24 * 30 * 12, train['timestamp'].max()]\n\nbins = list(hours) + list(days) + list(weeks) + list(months) + rest\n\nh,e = np.histogram(train['timestamp'], bins=bins)\n\nplt.figure(figsize=(15, 4))\n\nplt.xticks(np.arange(-0.5, len(h) + 0.5),\n           [\"0\", \"12 hours\", \"1 day\", \"2 days\", \"3 days\", \"4 days\", \"5 days\", \"6 days\", \"1 week\", \"2 weeks\", \"3 weeks\", \"1 month\", \" 3 months\", \"5 months\", \"7 months\", \"9 months\", \"11 monts\", \"1 year\", f\"max\"],\n           fontsize=14, rotation=60)\n\nformat_yticks = FuncFormatter(lambda x, p: format(int(x), ','))\n\nplt.bar(range(len(bins)-1), h, width=0.96,\n        color = [\"tab:blue\"] * len(hours) + [\"tab:orange\"] * len(days) + [\"tab:green\"] * len(weeks) + [\"tab:red\"] * len(months) + [\"tab:purple\"] * len(rest))\nplt.ylabel('Number of Interactions')\nplt.gcf().axes[0].yaxis.set_major_formatter(format_yticks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that many interactions happen in the first 12 hours, and also during the first and second month."},{"metadata":{},"cell_type":"markdown","source":"## 2.2 user_id:"},{"metadata":{},"cell_type":"markdown","source":"Each user has his own id, also in the hidden test set we will have more new user ids. Let's see how many users we have in this dataset:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f\"Total number of users: {train['user_id'].nunique():,}\\n\\\nNumber of interations / Number of users: {len(train['user_id']) / train['user_id'].nunique():.2f}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We plot the number of interactions per user variable, just to check if we have clusters of very active user and others with less activity:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"user_counts = train['user_id'].value_counts()\n\nprint(f\"\\\nAverge: {user_counts.mean():.02f}\\n\\\nMedian: {user_counts.median():>6}\\n\\\nMin:    {user_counts.min():>6}\\n\\\nMax:    {user_counts.max():,}\")\n\nfig = plt.figure(figsize=(15,4))\nax = fig.add_subplot(111)\n\nax.hist(user_counts, bins=100, rwidth=0.8)\n\nax.set_xlabel('Number of Interactions')\nax.set_ylabel('Users counts')\nax.yaxis.set_major_formatter(format_yticks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This distribution looks very skewed to the left, let re-plot this with a log scale:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"bins = np.logspace(0, 8, base=6, endpoint=user_counts.max(), num=50, dtype=np.int)\n\nh,e = np.histogram(user_counts, bins=bins)\n\nfig = plt.figure(figsize=(15, 4))\nax = fig.add_subplot(111)\n\nplt.sca(ax)\nplt.xticks(np.arange(-0.5, len(h) + 0.5), bins)\n\nax.bar(range(len(bins)-1), h, width=0.9)\n\nplt.xticks(rotation=70)\nax.set_xlabel('Number of Interactions')\nax.set_ylabel('Users counts')\nax.yaxis.set_major_formatter(format_yticks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It seems that we have a mixture of two distributions, the left one (the dominant one) are users with less interaction and the right one are users orders of magnitude more interactions."},{"metadata":{},"cell_type":"markdown","source":"## 2.3 content_id:"},{"metadata":{},"cell_type":"markdown","source":"This column represent the id of a questions or a lecture (the same id is also present in quesntions.csv and lecture.csv datasets). We can verify that this column is just ids from the other dataframes by using the code bellow:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"content_id_in_dfs = np.isin(train['content_id'].unique(), np.concatenate([questions.question_id, lectures.lecture_id]))\nprint(f\"Are all content ids in question and lecture dataframes?: {np.all(content_id_in_dfs)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"But does these ids represent ids uniquely for each question or lecture?..."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"ids_intersect = np.intersect1d(questions.question_id, lectures.lecture_id)\nprint(f\"Questions ids and lectures ids not intersecting?: {ids_intersect.size == 0}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is important, because to distinguish between questions and lectures we need to use content_type_id column and not only rely on content_id.\n\nWe need to answer another question now, are all lectures and questions present is our training dataset?"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"questions_not_in_train = np.isin(questions.question_id, train['content_id'][train['content_type_id'] == 0].unique(), invert=True)\nlectures_not_in_train = np.isin(lectures.lecture_id, train['content_id'][train['content_type_id'] == 1].unique(), invert=True)\nprint(f\"\\\nQuestions not in train.csv: {np.count_nonzero(questions_not_in_train)}\\n\\\nLectures not in train.csv:  {np.count_nonzero(lectures_not_in_train)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Only 3 lectures are not present is the training dataset.\n\nWe will analyze more the content_id in a later sections along the corresponding dataframes."},{"metadata":{},"cell_type":"markdown","source":"## 2.4 content_type_id:"},{"metadata":{},"cell_type":"markdown","source":"This is to indicate if the content is a lecture or a question, (0: question, 1: lecture)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f\"\\\nUnique values:            {train['content_type_id'].nunique()}\\n\\\nQuestions:       {np.count_nonzero(train['content_type_id'] == 0):,}\\n\\\nLectures:         {np.count_nonzero(train['content_type_id'] == 1):,}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.5 task_container_id:"},{"metadata":{},"cell_type":"markdown","source":"The questions or lectures are bundeled together and given to users as batches, this column tell us how different contents are grouped.\n\nLet's get and idea on how they are contained..."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"is_same_type_container = train['content_type_id'].groupby(train['task_container_id']).apply(lambda x: np.all(x == x.iloc[0]))\nprint(f\"\\\nTotal number of containers:                            {train['task_container_id'].nunique():,}\\n\\\nAny lectures and questions in the same container?:     {np.all(is_same_type_container) == False}\\n\\\nHow many containers with mixed content type:           {np.count_nonzero(is_same_type_container == False)}\\n\\\nHow many containers with one content type:             {np.count_nonzero(is_same_type_container)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.6 user_answer:"},{"metadata":{},"cell_type":"markdown","source":"The user's answer to the question, if any. -1 as null, for lectures."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f\"Unique values: {train['user_answer'].nunique()}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(5, 5))\nax = fig.add_subplot(111)\n\nax.hist(train['user_answer'])\n\nax.set_xticks(train['user_answer'].unique())\n\nax.set_xlabel('Answer Number')\nax.set_ylabel('Counts')\n\nax.yaxis.set_major_formatter(format_yticks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.7 answered_correctly:"},{"metadata":{},"cell_type":"markdown","source":"If the user answered correctly, -1 for lectures. This is the only feature that needs to predicted."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(5, 5))\nax = fig.add_subplot(111)\n\nax.hist(train['answered_correctly'])\n\nax.set_xticks(train['answered_correctly'].unique())\n\nax.set_xlabel('Answer Correctness')\nax.set_ylabel('Counts')\nax.yaxis.set_major_formatter(format_yticks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.8 prior_question_elapsed_time:"},{"metadata":{},"cell_type":"markdown","source":"The average time it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"total_na = train['prior_question_elapsed_time'].isna().sum()\ntimedelta = pd.to_timedelta(train['prior_question_elapsed_time'], unit='ms')\n\nprint(f\"\\\nTotal null values:      {total_na:,}\\n\\\nTotal non-null values:  {train.shape[0] - total_na:,}\\n\\\nAverge:                 {timedelta.mean().total_seconds():>12} s\\n\\\nMedian:                 {timedelta.median().total_seconds():>12} s\\n\\\nMin:                    {timedelta.min().total_seconds():>12} s\\n\\\nMax:                    {timedelta.max().total_seconds():>12} s\")\n\nfig = plt.figure(figsize=(15, 4))\nax = fig.add_subplot(111)\n\nax.hist(train['prior_question_elapsed_time'][train['prior_question_elapsed_time'].isna() == False] / 1000, bins = 50, rwidth=0.9)\n\nax.set_xlabel('Seconds')\nax.set_ylabel('User Count')\nax.yaxis.set_major_formatter(format_yticks)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.9 prior_question_had_explanation:"},{"metadata":{},"cell_type":"markdown","source":"Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle."},{"metadata":{"trusted":true},"cell_type":"code","source":"counts = train['prior_question_had_explanation'].value_counts()\nprint(f\"\\\nTrue:  {counts[0]:,}\\n\\\nFalse: {counts[1]:,}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. questions.csv:"},{"metadata":{},"cell_type":"markdown","source":"This dataframe gives us infromation about the questions. These questions will be the same for the hidden test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"tags = questions.tags.apply(lambda x: [int(t) if t.isdigit() else -1 for t in str(x).split(' ')])\ntags = np.concatenate(tags.tolist())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"\\\nTotal questions: {questions.question_id.nunique()}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. lectures.csv:"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Total number of lectures: {lectures.lecture_id.nunique()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Looking for patterns:\nOur goal in this competition as mentioned before is to predict the user's answer correctness, in this section we're going to try to find any patterns that describe our target using different predictors available in the dataset and statistics."},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Question difficulty:\nFirst let's see  if are difference in difficulty of anwering each question, we're gonna do that by calculating the ratio of correct answers for each question and plot it:"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"questions_ratios = train.query('content_type_id == 0').groupby('content_id')['answered_correctly'].mean()\nplt.figure(figsize=(15, 4))\nquestions_ratios.hist(bins=100, rwidth=0.8)\nplt.xlabel('Correctness Ratio')\n_ = plt.ylabel('Number of Questions')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here we have a normal distribution skewed to the right, but why we have this guaussian distribution rather than a uniform distribution? it could be that our questions follow a bernouli distribtion with different parameter each, that means each questions has it own probabily of being answered correctly.\n\nLet's try to use answer correctness mean as a prediction, and calculate the score:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"preds = questions_ratios[train.query('content_type_id == 0').content_id].values\nscore = roc_auc_score(train.query('content_type_id == 0').answered_correctly.values, preds)\nprint(f'Score: {score}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's a good score given that we used only a single feature, this show how import this feature is."},{"metadata":{},"cell_type":"markdown","source":"## 5.2 User's Performance:\nIt could be that users differ on how well they respond to questions, we will do the same as the previous section, group by user_id and aggregate the mean over the answer correctness column:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"users_ratios = train.query('content_type_id == 0').groupby('user_id')['answered_correctly'].mean()\nplt.figure(figsize=(15, 4))\nusers_ratios.hist(bins=100, rwidth=0.8)\nplt.xlabel('Correctness Ratio')\n_ = plt.ylabel('Number of Users')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's try this as a prediction..."},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = users_ratios[train.query('content_type_id == 0').user_id].values\nscore = roc_auc_score(train.query('content_type_id == 0').answered_correctly.values, preds)\nprint(f'Score: {score}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.3 Container difficulty:"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"containers_ratios = train.query('content_type_id == 0').groupby('task_container_id')['answered_correctly'].mean()\nplt.figure(figsize=(15, 4))\ncontainers_ratios.hist(bins=100, rwidth=0.8)\nplt.xlabel('Correctness Ratio')\n_ = plt.ylabel('Number of Containers')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This distribution has less variance than the other distributions, this suggests that containers don't differ much on difficulity. Container id won't be a good predictor for answer correctness."},{"metadata":{"trusted":true},"cell_type":"code","source":"preds = containers_ratios[train.query('content_type_id == 0').task_container_id].values\nscore = roc_auc_score(train.query('content_type_id == 0').answered_correctly.values, preds)\nprint(f'Score: {score}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As excpected it gave us a lower score than the other predictors.."},{"metadata":{},"cell_type":"markdown","source":"# 6. Submission process:\nSubmissions should be done through notebooks, using the time series API.\n\nThe submission process is an iterative process, your model will get data as batches. For each batch your model have to send predictions of the current to the API before receiving the next batch.\n\nAlong each batch you receive from the API, labels of the preceding are reveiled, so it would be useful for your model to learn and predict in the same time. \n\n[API Demonstration](https://www.kaggle.com/sohier/competition-api-detailed-introduction)."},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# 7. Private Dataset and Proportions:\nLastly, one important aspect in every competition, is how different the private is from the public test.\n\nOne thing worth highlighting is that the private dataset will have new users and the will have about 2.5 million questions."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}