{"cells":[{"metadata":{},"cell_type":"markdown","source":"1. Problem Identification\n\n2. Data Wrangling\n    * Data Collection: loading and joining\n    * Data Definition\n        * Column names\n        * Data types\n        * Count/Percent of unique values\n    * Data Cleaning\n        * NA or missing data\n        * Duplicates\n3. Exploratory Data Analysis\n4. Pre-processing,Training Data Development and Modeling\n5. Documentation"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n!pip install seaborn_qqplot\nfrom seaborn_qqplot import pplot\n\nimport gc\n\n# import riiideducation\n# env = riiideducation.make_env()\n\n# # datatable installation with internet\n!pip install datatable==0.11.0 > /dev/null\nimport datatable as dt\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**2.1. Data loading and joining**\n\nCombine training, lectures and questions datasets for more insight.\nThe training file is too large so datatable was used to read train.csv.\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%time\n# train_df = dt.fread('/kaggle/input/riiid-test-answer-prediction/train.csv')\n# train_df = train_df.to_pandas()\n# meta_lectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\n# meta_lectures = meta_lectures.to_pandas()\n# meta_questions = dt.fread('/kaggle/input/riiid-test-answer-prediction/questions.csv')\n# meta_questions = meta_questions.to_pandas()\n\n# train_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', \n#                       dtype=dtypes)\ntrain_df=pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',low_memory=False, nrows=10**6) #,dtype=dtypes)\nlectures=pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\nquestions=pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\n\nprint(\"Train size:\",train_df.shape)\nprint(\"Lectures size:\", lectures.shape)\nprint(\"Questions size:\", questions.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check if all records have metadata (lecture/question) associate with it\n# lectures:\nn_lec = train_df[train_df.content_type_id==1].content_id.isin(lectures.lecture_id.unique()).sum()\nn_ques = train_df[train_df.content_type_id==0].content_id.isin(questions.question_id.unique()).sum()\nprint(\"number of lectures that has a match: {}\".format(n_lec))\nprint(\"number of questions that has a match: {}\".format(n_ques))\nprint(\"total number equal to number of records: \", (n_lec+n_ques)==10**6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.memory_usage(deep=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# chunk_list = []  # append each chunk df here \n# ids = set()\n# # Each chunk is in df format\n# for chunk in df_chunk:  \n#     # perform data filtering \n#     chunk = chunk.drop_duplicates(['timestamp'])\n#     chunk = chunk[~chunk['timestamp'].isin(ids)]\n#     ids.update(chunk['timestamp'].values)\n#     # Once the data filtering is done, append the chunk to list\n#     chunk_list.append(chunk)\n    \n# # concat the list into dataframe \n# df_concat = pd.concat(chunk_list)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check for duplicates\nprint(train_df[train_df.duplicated()])\nprint(lectures[lectures.duplicated()])\nprint(questions[questions.duplicated()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_df.drop_duplicates(inplace=True)\n# lectures.drop_duplicates(inplace=True)\n# questions.drop_duplicates(inplace=True)\n# print(train_df.shape)\n# print(lectures.shape)\n# printquestions.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Got memory error when trying to apply drop_duplicates on train_df.\nFirst 10**6 records don't contain duplicates."},{"metadata":{"trusted":true},"cell_type":"code","source":"# combine train_df, lectures, questions\ntrain_lec = train_df[train_df.content_type_id==1][['user_id','timestamp','content_id','task_container_id']]\ntrain_lec_merge = train_lec.merge(lectures, how='left',left_on=['content_id'], right_on = ['lecture_id'])\\\n                    .drop(columns=['lecture_id','content_id','part','tag'])\ntrain_lec_merge = train_lec_merge.rename(columns={'type_of':'type_of_lec'})\n# train_lec_merge.head()\n# add a column to train_df: num_lec_watched: the number of lectures watched before doing the question\ntrain_df = train_df.sort_values(by=['user_id','timestamp'])\ntrain_df['num_lec_watched'] = train_df.groupby(['user_id']).content_type_id.cumsum()\n# (how to include lecture types???)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_q = train_df[train_df.content_type_id ==0]\ntrain_q = train_q.drop(columns=['content_type_id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # combine train_df, lectures, questions\n# lectures[\"content_type_id\"] = 1\n# questions[\"content_type_id\"] = 0\n# train_1 = train_df.merge(lectures, how='left',left_on=['content_id','content_type_id'], right_on = ['lecture_id','content_type_id'])\\\n#                     .drop(columns=['lecture_id'])\n# train = train_1.merge(questions,how='left', left_on=['content_id','content_type_id'], right_on = ['question_id','content_type_id'])\\\n#                     .drop(columns=['question_id','content_type_id'])\n# gc.collect()\n# train = train.rename(columns={'part_x':'part_lec', 'part_y':'part_ques', 'tag':'tag_lec', 'tags':'tag_ques'})\n# train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# columns with only one unique value\none_value = [col for col in train_q.columns if train_q[col].nunique() <= 1]\nprint(\"Number of columns with only one unique value: {}\".format(len(one_value)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**2.2. Data definition**"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Training set shape: {}\".format(train_q.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# data info\ntrain_q.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain_q['task_container_id'] = train_q['task_container_id'].astype('int16')\ntrain_q['user_answer'] = train_q['user_answer'].astype('int8')\ntrain_q['answered_correctly'] = train_q['answered_correctly'].astype('int8')\ntrain_q['prior_question_elapsed_time'] = train_q.prior_question_elapsed_time.fillna(0)\ntrain_q['prior_question_elapsed_time'] = train_q['prior_question_elapsed_time'].astype('float32')\n# train['prior_question_had_explanation'].replace({'True': True, 'False': False}, inplace=True)\ntrain_q['first_bundle'] =  np.where(train_q['prior_question_had_explanation'].isnull(), True, False)\n# train_q['prior_question_had_explanation'] = train_q.where(~train_q['prior_question_had_explanation'].isnull(),False)\ntrain_q['prior_question_had_explanation'].replace({np.NaN: False}, inplace=True)\ntrain_q['prior_question_had_explanation'] = train_q['prior_question_had_explanation'].astype('bool')\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_q.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# summary of missing values in each column\ntrain_q.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.options.display.float_format = \"{:.2f}\".format\ntrain_q[['timestamp','prior_question_elapsed_time']].describe().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique users: {}\".format(train_q.user_id.nunique()))\nprint(\"Average number of records per user: {:.2f}\".format(len(train_q.index)/train_q.user_id.nunique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"Number of lectures watched: {}\". format((~train.tag_lec.isnull()).sum()))\n# print(\"Number of questions asked: {}\". format((train.tag_lec.isnull()).sum()))\n# print(\"Percentage lectures in the samples: {:.2%}\".format((~train.tag_lec.isnull()).sum()/train.shape[0]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of correct answers: {}\".format((train_q.answered_correctly == 1).sum()))\nprint(\"Number of incorrect answers: {}\".format((train_q.answered_correctly == 0).sum()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Imbalanced dataset: incorrect records is half of correct records"},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(\"Number of unique content ids: {}\". format(train[[\"content_id\",\"tag_lec\"]].nunique()))\n# print(\"Number of unique lectures: {}\". format(train[train.tag_ques.isnull()].content_id.nunique()))\n# print(\"Number of unique questions: {}\". format(train[train.tag_lec.isnull()].content_id.nunique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique {}\". format(train_q[[\"content_id\"]].nunique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some lectures and questions share the same content ids."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique content ids in meta lectures: {}\".format(lectures.lecture_id.nunique()))\nprint(\"Number of unique content ids in meta questions: {}\".format(questions.question_id.nunique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Not all the lectures and questions in the metadata are included in the sampled dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of question/lecture bundles: {}\".format(train_q.task_container_id.nunique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Percentage of question that had explanation: {:.2%}\".format((train_q.prior_question_had_explanation == True).sum()/len(train_q.index)))\nprint(\"Percentage of question that were in first bundle: {:.2%}\".format((train_q.first_bundle==True).sum()/len(train_q.index)))\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Percentage of questions that's not in the first bundle and not explained \nprint(\"Percentage of questions that's not in the first bundle and not explained: {:.2%}\".format(train_q[train_q.prior_question_had_explanation!=~train_q.first_bundle].row_id.count()/len(train_q.index)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cor_rate_ques_explained = train_q[train_q.prior_question_had_explanation == True].answered_correctly.mean()\ncor_rate_ques_not_explained = train_q[train_q.prior_question_had_explanation == False].answered_correctly.mean()\nprint(\"Correctness rate with prior question explained: {:.2%}\".format(cor_rate_ques_explained))\nprint(\"Correctness rate with prior question not explained: {:.2%}\".format(cor_rate_ques_not_explained))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Whether the prior question is explained might have an effect on correctness, however, the percentage of question in the first bundle is small so can be ignored. Drop *first_bundle* column"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_q.drop(columns=['first_bundle'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Export processed data to new file**"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_q.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_q.to_csv(\"train_cleaned.csv\", index=False)\n# train.to_pickle(\"/kaggle/working/train_cleaned.pkl\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**3. Exploratory Data Analysis**"},{"metadata":{"trusted":true},"cell_type":"code","source":"dtypes = {'row_id': 'int64', \n         'timestamp': 'int64', \n         'user_id': 'int32', \n         'content_id': 'int32', \n         'task_container_id': 'int16', \n         'user_answer': 'int8', \n         'answered_correctly': 'int8', \n         'prior_question_elapsed_time': 'float32',\n         'prior_question_had_explanation': 'boolean',\n         'num_lec_watched': 'int16'}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.read_csv(\"./train_cleaned.csv\", dtype=dtypes)\n# df = pd.read_pickle(\"./train_cleaned.pkl\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# df[df.user_id==124]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Whether the question is answered correctly might be associated with:\n* Length of user interaction -- timestamp\n* Number of questions in a bundle -- task_container_id\n* Number of lectures in a bundle -- task_container_id\n* Question-lecture ratio in a bundle -- task_container_id\n* Length of time user took to answer the previous bundle -- prior_question_elapsed_time\n* Whether the explanation given for previous question bundle -- priot_question_had_explanation\n\nRegarding the questions file:\n* The number of correct answer -- correct_answer\n* The section of test -- part\n* Questions with certain tags -- tags\nRegarding the lectures file:\n* Type of lecture provided -- type_of\n* Certain lecture -- tag"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x=\"answered_correctly\", \n                data=df, kind=\"count\")\nplt.title('Correct/Incorrect answers')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Lectures**\n\nWhether having watched lectures increases correctness rate?"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(df.num_lec_watched,bins=20)\nplt.title(\"distribution plot of number of watched lectures\")\nplt.ylabel(\"probability\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The histogram is highly skewed, so we perform log transformation and plot again, separating correctly/incorrectly answered questions."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(np.log(df[(df[\"answered_correctly\"]==1) & (df['num_lec_watched']!=0)].num_lec_watched),bins=20)\nsns.distplot(np.log(df[(df[\"answered_correctly\"]==0) & (df['num_lec_watched']!=0)].num_lec_watched),bins=20)\nplt.legend(['Correct','Incorrect'])\nplt.title(\"log-histogram of number of watched lectures\")\nplt.ylabel(\"probability\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The log-transform of correct/incorrect questions are almost the same and not normal distribution."},{"metadata":{},"cell_type":"markdown","source":"**Timestamp**\n\nWhether timestamp is associated with correctness rate?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# sns.distplot(temp_[temp_[\"answered_correctly\"]==1].timestamp,\n#              hist=False,rug=True)\n# sns.distplot(temp_[temp_[\"answered_correctly\"]==0].timestamp,\n#              hist=False,rug=True)\n# sns.distplot(temp_[temp_[\"answered_correctly\"]==-1].timestamp,\n#              hist=False,rug=True)\n\nsns.distplot(df[df[\"answered_correctly\"]==1].timestamp,hist=False,rug=True)\nsns.distplot(df[df[\"answered_correctly\"]==0].timestamp,hist=False,rug=True)\nplt.legend(['Correct','Incorrect'])\nplt.title(\"rug plot of timestamp\")\nplt.ylabel(\"Probability\")\nplt.xlabel(\"timestamp\")\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Since the data was skewed, log transformation is taken\nsns.distplot(np.log(df[(df[\"answered_correctly\"]==1) & (df['timestamp']!=0)].timestamp))\nsns.distplot(np.log(df[(df[\"answered_correctly\"]==0) & (df['timestamp']!=0)].timestamp))\n\n# ax.set_xlim(0,4*10**10)\nplt.legend(['Correct','Incorrect'])\nplt.title(\"log-histogram of timestamp\")\nplt.ylabel(\"Probability\")\nplt.xlabel(\"timestamp after log transformation\")\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Incorrectly answered question has higher density at smaller timestamp number. No other obvious difference in these 3 categories associated with timestamp. The log timestamp does not follow a normal distribution."},{"metadata":{},"cell_type":"markdown","source":"**Previous elapsed time**\n\nWhether previous elapsed time is associated with correctness rate?"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(df[df[\"answered_correctly\"]==1].prior_question_elapsed_time,hist=False,rug=True)\nsns.distplot(df[df[\"answered_correctly\"]==0].prior_question_elapsed_time,hist=False,rug=True)\nplt.legend(['Correct','Incorrect'])\nplt.title(\"rug plot of prior_question_elapsed_time\")\nplt.ylabel(\"Probability\")\nplt.xlabel(\"prior_question_elapsed_time\")\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot relationship between task_container_id and correctness\n* Number of questions in a bundle -- task_container_id\n* Number of lectures in a bundle -- task_container_id\n* Question-lecture ratio in a bundle -- task_container_id"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(np.log(df[(df[\"answered_correctly\"]==1) & (df['prior_question_elapsed_time']!=0)].prior_question_elapsed_time))\nsns.distplot(np.log(df[(df[\"answered_correctly\"]==0) & (df['prior_question_elapsed_time']!=0)].prior_question_elapsed_time))\n\nplt.legend(['Correct','Incorrect','Lectures'])\nplt.title(\"log-histogram of prior_question_elapsed_time\")\nplt.ylabel(\"Probability\")\nplt.xlabel(\"prior_question_elapsed_time after log transformation\")\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Task containers**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# What are the frequencies of question bundles being visited? \nsns.distplot(df.groupby(\"task_container_id\").count())\nplt.title(\"histogram of task_container_id frequency\")\nplt.ylabel(\"Probability\")\nplt.xlabel(\"Number of times task_container_id frequency being visited\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(df.groupby([\"content_id\"]).size().nlargest(10))\n# print(df.groupby([\"content_id\"]).size().nsmallest(10))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# A bundle with multiple questions and same timestamp only counts as one visit  \ndf_unique_tasks = df.groupby(['user_id','timestamp']).first()\ndf_unique_tasks.head()\nprint(df_unique_tasks.groupby('task_container_id').size().nlargest(10))\nprint(df_unique_tasks.groupby('task_container_id').size().nsmallest(10))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The times of visit varies from 3824 to 1. The top 10 most visited questions id number are smaller than the least visited ones.\n\nWhether the times of visit and task container id are correlated?"},{"metadata":{"trusted":true},"cell_type":"code","source":"count_container= pd.DataFrame(df_unique_tasks.groupby('task_container_id').size())\ncount_container.reset_index(level=['task_container_id'],inplace=True)\ncount_container.rename(columns={0: \"freq\"}, inplace=True)\nsns.scatterplot(x='task_container_id',y=\"freq\",data=count_container,linewidth=0,s=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that the question bundles with higher id number are visited less."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(np.log(df_unique_tasks.groupby('task_container_id').size()))\nplt.title(\"log-histogram of task_container_id frequency\")\nplt.ylabel(\"Probability\")\nplt.xlabel(\"Number of times task_container_id frequency being visited(log)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most question bundles are only visited once. Could be because of the sampling."},{"metadata":{"trusted":true},"cell_type":"code","source":"# sns.scatterplot(df.task_container_id, df.groupby([\"task_container_id\"]).size(),size=1, linewidth=0)\n# plt.xlabel(\"task container id\")\n# plt.ylabel(\"number of times visited\")\n# plt.title(\"Scatter plot of task container id vs number of times visited\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_unique_tasks.reset_index(level=['timestamp'],inplace=True)\ndf_unique_tasks.reset_index(level=['user_id'],inplace=True)\np = sns.jointplot(x='task_container_id', y='timestamp', data=df_unique_tasks, s=1)\np.set_axis_labels(\"task container id\", \"timestamp\")\np.fig.suptitle(\"Scatter plot of task container id vs timestamp\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The tasks are visited less if the task container id number is larger, but the task container id and timestamp are not linearly correlated, which means learning longer time is not necessarily assigned question bundles with higher task container id number.\n\nHowever, short straight lines with different slopes are seen on the task_container_id vs timestamp plot. We can guess that the learning speed of each user is different. \n\nSo we derive questions_elapsed_time from previous_questions_elapsed_time and plot task_container_id vs questions_elapsed_time to verify our guess."},{"metadata":{"trusted":true},"cell_type":"code","source":"# df.sort_values(by=['user_id','timestamp'])\n# df[\"current_question_elapsed_time\"] = df.groupby(['user_id']).prior_question_elapsed_time.shift(-1)\n# df[df.user_id==124][[\"prior_question_elapsed_time\",\"current_question_elapsed_time\"]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"time_table = df_unique_tasks.sort_values(by=['user_id','timestamp'])[['user_id','prior_question_elapsed_time','task_container_id','timestamp']]\ntime_table[\"current_question_elapsed_time\"] = time_table.groupby(['user_id']).prior_question_elapsed_time.shift(-1)\n# fill the unknown current question elapsed time with mean\ntime_table.current_question_elapsed_time = time_table.groupby(['user_id']).current_question_elapsed_time.apply(lambda x: x.fillna(x.mean()))\ntime_table.drop(columns=['prior_question_elapsed_time'],inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.merge(time_table, how='left',on=['user_id','task_container_id','timestamp'])\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"p = sns.jointplot(x='task_container_id', y='current_question_elapsed_time', data=df, s=1)\np.set_axis_labels(\"task container id\", \"question_elapsed_time\")\np.fig.suptitle(\"Scatter plot of task container id vs question elapsed time\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is an edge at task container id 5600~. One guess is that not many students proceeded to questions bundles above that number."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of students finished bundles above 5500: {}\".format(df[df.task_container_id>=5500].user_id.nunique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of students finished one question using more that 295000ms: {}\"\\\n      .format(df[df.current_question_elapsed_time>=295000].user_id.nunique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Task container id**\n\nIs there a correctness rate difference in different question bundles?\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculate the correctness rate vs task_container_id\n# Exclude bundles number larger than 5500\nfig = plt.figure(figsize=(20,6))\nax1 = fig.add_subplot(121)\ncor_container_avg = df.groupby('task_container_id').answered_correctly.apply(lambda x: x.mean()).to_frame().reset_index()\ncor_container_avg_5k = cor_container_avg[cor_container_avg.task_container_id<=5500]\n# cor_container_std = df.groupby('task_container_id').answered_correctly.apply(lambda x: x.std())\nax1 = sns.regplot(x='task_container_id', y='answered_correctly',data=cor_container_avg_5k,line_kws={'color':'magenta'}, scatter_kws={'s':1},ci=99)\nplt.title(\"Task container id vs correctness rate\")\nplt.ylabel(\"Correctness rate\")\nplt.xlabel(\"Task container id\")\nax2 = fig.add_subplot(122)\nax2 = sns.distplot(cor_container_avg_5k.answered_correctly, bins=50)\nplt.title(\"Histgram\")\nplt.ylabel(\"Frequency\")\nplt.xlabel(\"Correctness rate\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The correctness rate varies from around 0.65(smaller ids) to 0.4-0.8(larger ids). For questions in bundle number over 5500, since they are visited only once, the correctness rate is either 0 or 1. There is no evidence that correctness is correlated with task container id.  "},{"metadata":{"trusted":true},"cell_type":"code","source":"# Whether questions in a bundle have higher correctness?\ndf_bundle = df.loc[(df.task_container_id.shift(-1)==df.task_container_id) | (df.task_container_id.shift(1)==df.task_container_id)]\nprint(\"The correctness of questions in bundle: {:.2%}\".format(df_bundle.answered_correctly.mean()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The correctness of questions in bundle is not higher than the average of all questions."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_bundle.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Content id**\n\nIs there a correctness rate difference in different questions?"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculate the correctness rate vs task_content_id\n# Exclude bundles number larger than 5500\nfig = plt.figure(figsize=(20,6))\nax1 = fig.add_subplot(121)\ncor_content_avg_5k = df[df.task_container_id<=5500].groupby('content_id').answered_correctly.apply(lambda x: x.mean()).to_frame().reset_index()\n# cor_container_std = df.groupby('task_container_id').answered_correctly.apply(lambda x: x.std())\nax1 = sns.regplot(x='content_id', y='answered_correctly',data=cor_content_avg_5k,line_kws={'color':'magenta'}, scatter_kws={'s':1},ci=99)\nplt.title(\"Content id vs correctness rate\")\nplt.ylabel(\"Correctness rate\")\nplt.xlabel(\"Content id\")\nax2 = fig.add_subplot(122)\nax2 = sns.distplot(cor_content_avg_5k.answered_correctly, bins=50)\nplt.title(\"Histgram\")\nplt.ylabel(\"Frequency\")\nplt.xlabel(\"Correctness rate\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is no pattern of correctness in content id number, but some questions have 100% correct rate while some have zero. We can investigate for how many records there are for those 0 or 100% correct questions. "},{"metadata":{"trusted":true},"cell_type":"code","source":"cor_100_id = cor_content_avg_5k[cor_content_avg_5k.answered_correctly==1].content_id\nprint(\"Number of 100% correct questions: {}\".format(len(cor_100_id)))\ncor_0_id = cor_content_avg_5k[cor_content_avg_5k.answered_correctly==0].content_id\nprint(\"Number of 0% correct questions: {}\".format(len(cor_0_id)))\n\nfig = plt.figure(figsize=(20,6))\n# plot the sample size of 100% correct questions\nax1 = fig.add_subplot(121)\ncor_100_num = df[df.content_id.isin(cor_100_id)].groupby('content_id').count()\nax1= sns.distplot(cor_100_num, bins=60)\nplt.title(\"Histgram\")\nplt.ylabel(\"Frequency\")\nplt.xlabel(\"Number of samples making up 100% correctness\")\n# plot the sample size of 0% correct questions\nax2 = fig.add_subplot(122)\nplt.title(\"Histgram\")\nplt.ylabel(\"Frequency\")\nplt.xlabel(\"Number of samples making up 0% correctness\")\ncor_0_num = df[df.content_id.isin(cor_0_id)].groupby('content_id').count()\nax2 = sns.distplot(cor_0_num)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the 0 or 100% correct questions only have 1 record, the prediction on those might not be accurate."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Plot a correlation matrix\ncorr_mat = df.corr().stack().reset_index(name=\"correlation\")\ng = sns.relplot(\n    data=corr_mat,\n    x=\"level_0\", y=\"level_1\", hue=\"correlation\", \n    palette=\"Blues\", edgecolor=\".7\", size=\"correlation\", height=10,\n    sizes=(100, 400), size_norm=(-.2, .8),\n)\nplt.xticks(rotation=90)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the correlation matrix we can see that answer_correctly is not significantly correlated with any other single variable."},{"metadata":{},"cell_type":"markdown","source":"**4. Preprocessing and Feature Engineering**\n\n* Create new features\n* Select specific features\n* Standardize numeric features\n* (Split into testing and training datasets)\n* Resampling training dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# \ndf.drop()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Oversampling: SMOTE**\n\nOne approach to deal with imbalanced datasets is to oversample the minority class, which is incorrectly answered questions in this case. A widely used approach is Synthetic Minority Oversampling Technique (SMOTE) for the minority class."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}