{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"### Indroduction\nI have noticed that users are given same questions multiple times during the course of there learning. From my I analysis, I think that keeping the prior history of questions given to the users can be a powerfull feature during modelling. The following is the analysis i have carried out to reach this claim\n\n1. [Repeated Questions Detected](#2)\n2. [Statistical Analysis on Repeated Questions](#3)\n    1. [Probability Distribution of Fist Attempt Vs Repeated Attempts](#4)\n    2. [Hypothesis Testing](#5)\n        1. [Normality Test of Populations: First Attempt answer correctness](#6)\n        2. [Normality Test of Populations: Repeated Attempt answer correctness](#7)\n        3. [t-Test for comparing population mean](#8)\n3. [Conclusions](#9)\n    \n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport dask.dataframe as dd\nimport pandas as pd\nfrom time import time\nfrom contextlib import contextmanager\n\n@contextmanager\ndef timer(name):\n    t0 = time()\n    yield\n    print(f'[{name}] done in {time() - t0:.2f} s')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"with timer(\"Data Loading Time\"):\n    train = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',\n                        usecols = [0, 1, 2, 3, 4, 5, 7],\n                   dtype={'row_id': 'int32',\n                          'timestamp': 'int64',\n                          'user_id': 'int64',\n                          'content_id': 'int16',\n                          'content_type_id': 'int8',\n                          'task_container_id': 'int16',\n                          'user_answer': 'int8',\n                          'answered_correctly':'int8',\n                          'prior_question_elapsed_time': 'float32',\n                          'prior_question_had_explanation': 'boolean'}\n                   )\n    questions = pd.read_csv(\"/kaggle/input/riiid-test-answer-prediction/questions.csv\",\n                           dtype = {'question_id': 'int16', \n                                    'bundle_id': 'int16', \n                                    'correct_answer': 'int8',\n                                    'part': 'int8',\n                                   })\n    # lectures = pd.read_csv(\"/kaggle/input/riiid-test-answer-prediction/lectures.csv\",)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train = train[train.content_type_id == 0]\n#questions['content_type_id'] = 0\n#questions['content_type_id'] = questions['content_type_id'].astype('int8')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='2'></a>\n## Repeated Questions Detected\nI have noticed that same question(or bundle) is given to a user many time throughout the course of learning. The following analysis proves my point."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_user = train.loc[train.user_id == 801103753]\n\nassert train_user.content_type_id.nunique() == 1\n\ntrain_user = pd.merge(left=train_user, \n                       right=questions[['question_id', 'bundle_id']], \n                       left_on=[\"content_id\",],\n                       right_on=[\"question_id\",],\n                       how=\"left\",\n                      validate=\"m:1\")\n\n\ntrain_user = train_user[['user_id', 'question_id', 'bundle_id', 'timestamp', 'task_container_id', 'answered_correctly',]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"samples_1 = train_user[train_user.question_id == 853].reset_index(drop=True)\nsamples_1.loc[:, 'attempt_number'] = range(1, samples_1.shape[0]+1)\nsamples_1.loc[:, 'attempt_type'] = 'repeated_attempt'\nsamples_1.loc[0, 'attempt_type'] = 'first_attempt'\nsamples_1.style.background_gradient()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"samples_2 = train_user[train_user.question_id == 3348].reset_index(drop=True)\nsamples_2.loc[:, 'attempt_number'] = range(1, samples_2.shape[0]+1)\nsamples_2.loc[:, 'attempt_type'] = 'repeated_attempt'\nsamples_2.loc[0, 'attempt_type'] = 'first_attempt'\nsamples_2.style.background_gradient()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"samples_3 = train_user[train_user.question_id == 1754].reset_index(drop=True)\nsamples_3.loc[:, 'attempt_number'] = range(1, samples_3.shape[0]+1)\nsamples_3.loc[:, 'attempt_type'] = 'repeated_attempt'\nsamples_3.loc[0, 'attempt_type'] = 'first_attempt'\nsamples_3.style.background_gradient()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"3\"></a>\n### Statistical Analysis on Repeated Questions\nAfter the above analysis my hypothesis is that the probability of user answering a question correctly given that the question has already seen by the user before is higher than the probability of user answering the question correctly in the first shot.<br>\nI am trying to prove the above hypothesis through plotting the probability density of two populations(First attempt Vs Repeated Attempt) and also by doing a hypothesis on the two population means."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"random_user_ids = train.user_id.sample(n = 50000).unique()\n\ntrain = train[train.user_id.isin(random_user_ids)]\n\nprint(f\"number of users considered = {train.user_id.nunique()}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"master_data = train.merge(questions[['question_id', 'bundle_id',]], left_on=['content_id',],\n           right_on = ['question_id', ], how='left', validate=\"m:1\", copy=False)\n\ndel train\n\nmaster_data['question_count'] = master_data.groupby(['user_id', 'question_id']).row_id.transform('count')\nmaster_data['question_count'] = master_data['question_count'].astype('int8')\nmaster_data = master_data[master_data.timestamp != 0] # first few rows are part of onboarding process\nmaster_data = master_data[master_data.question_count >=2].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f\"number of row_ids in the analzying={master_data.shape[0]}\")\n\nprint(f\"maxismum number of times a question repeated in the sub set = {master_data.question_count.max()}\")\n\n# master_data.sort_values(['user_id', 'question_id', 'timestamp'], ascending=True, inplace=True)#\nprint(\"master data head\")\nmaster_data.head(10).style.background_gradient()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"non_repeated_questions = master_data.drop_duplicates(['user_id', 'question_id'], keep='first')\n\n# non_repeated_questions.head(10).style.background_gradient()\n\nassert non_repeated_questions.groupby(['user_id', 'question_id']).row_id.count().max() == 1\n\nrepeated_questions = master_data[~master_data.row_id.isin(non_repeated_questions.row_id)]\n\nassert repeated_questions.shape[0] + non_repeated_questions.shape[0] == master_data.shape[0]\n\n#repeated_questions.head(10).style.background_gradient()\n\nassert repeated_questions.groupby(['user_id', 'question_id']).row_id.count().max() >= 1\n\navg_correctness_non_repeat = non_repeated_questions.groupby(['user_id',], as_index=False).agg({'row_id': 'count',\n                                                                                    'timestamp':[min],\n                                                                                   'answered_correctly': 'mean'})\navg_correctness_non_repeat.columns = ['_'.join(col).strip() for col in avg_correctness_non_repeat.columns.values]\n\n\navg_correctness_repeat = repeated_questions.groupby(['user_id',], as_index=False).agg({'row_id': 'count', \n                                                                          'timestamp':[min, max],\n                                                                         'answered_correctly': 'mean'})\navg_correctness_repeat.columns = ['_'.join(col).strip() for col in avg_correctness_repeat.columns.values]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"Non-Repeating questions data saample\")\navg_correctness_non_repeat.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"number of samples of first attempted questions in population={avg_correctness_non_repeat.shape[0]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"Repeating questions data saample\")\navg_correctness_repeat.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"number of samples of repeated attempt questions population={avg_correctness_repeat.shape[0]}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='4'></a>\n### Probability Distribution of Fist Attempt Vs Repeated Attempts"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\nplt.rcParams['figure.figsize'] = [15,10]\nplt.rcParams['font.size'] = 14\nsns.kdeplot(avg_correctness_repeat.answered_correctly_mean, label=\"avg_correctness_first_attempt\", clip=[0,1])\nplt.axvline(avg_correctness_repeat.answered_correctly_mean.mean(), color='blue')\nsns.kdeplot(avg_correctness_non_repeat.answered_correctly_mean, label=\"avg_correctness_repeated_attempt\", clip=[0,1])\nplt.axvline(avg_correctness_non_repeat.answered_correctly_mean.mean(), color='orange')\n\n# add text \nplt.text(avg_correctness_non_repeat.answered_correctly_mean.mean()-.3, 3,\n         f\"first attempt mean={round(avg_correctness_non_repeat.answered_correctly_mean.mean(), 2)}\")\n\nplt.text(avg_correctness_repeat.answered_correctly_mean.mean()-.25, 2.5,\n         f\"repeated attempt mean={round(avg_correctness_repeat.answered_correctly_mean.mean(), 2)}\")\n\nplt.title(\"Probability Density of First attempt Vs repeated attempt answer correctness\")\nplt.xlabel(\"average answer correctness per user\")\nplt.ylabel(\"pdf\")\n\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='5'></a>\n## Hypothesis Testing"},{"metadata":{},"cell_type":"markdown","source":"<a id='6'></a>\n### Normality Test of Populations: First Attempt answer correctness\nNull Hypothesis H0: Sample comes from a Normal Distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy import stats\nnp.random.seed(12345678)\nk2, p = stats.normaltest(avg_correctness_non_repeat.answered_correctly_mean)\nalpha = 1e-3\nprint(\"p = {:g}\".format(p))\n\nif p < alpha:  # null hypothesis: x comes from a normal distribution\n    print(\"The null hypothesis can be rejected\")\nelse:\n    print(\"The null hypothesis cannot be rejected\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='7'></a>\n### Normality Test of Populations: Repeated Attempt answer correctness\nNull Hypothesis H0: Sample comes from a Normal Distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"k2, p = stats.normaltest(avg_correctness_repeat.answered_correctly_mean)\nalpha = 1e-3\nprint(\"p = {:g}\".format(p))\n\nif p < alpha:  # null hypothesis: x comes from a normal distribution\n    print(\"The null hypothesis can be rejected\")\nelse:\n    print(\"The null hypothesis cannot be rejected\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"k2, p = stats.normaltest(avg_correctness_repeat.answered_correctly_mean)\nalpha = 1e-3\nprint(\"p = {:g}\".format(p))\n\nif p < alpha:  # null hypothesis: x comes from a normal distribution\n    print(\"The null hypothesis can be rejected\")\nelse:\n    print(\"The null hypothesis cannot be rejected\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Note\nBoth The populations are not drawn from a Normal distribution. <br>\nBut since we are doing the test on the sample mean distribution, according to CLT the sample mean still be normally distributed if the sample size is large. This cross-validated post explain this well [here](https://stats.stackexchange.com/questions/9573/t-test-for-non-normal-when-n50)"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"<a id='8'></a>\n### t-test for comparing population mean<br>\n* H0: There are no significant difference in the mean answer correctness of two populations<br>\n* H1: There is a difference<br> \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"tstat, p = stats.ttest_ind(avg_correctness_repeat.answered_correctly_mean, \n                avg_correctness_non_repeat.answered_correctly_mean, equal_var = False)\nalpha = 1e-2 # 99% confidence \n\nif p < alpha:  # null hypothesis: x comes from a normal distribution\n    print(\"The null hypothesis can be rejected\")\nelse:\n    print(\"The null hypothesis cannot be rejected\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='9'></a>\n### Conclusion\n* The mean correctness of the users who have attempted a question for the first time is quite lower than the mean correctness when the user sees the question repeatedly\n* This finding can be used as a post processing techniques during modelling.\n* By keeping the log of already seen question/or some engineered featured may improve the quality of the model.\n"},{"metadata":{},"cell_type":"markdown","source":"### Please consider upvoting this kernel if you find ths informative"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}