{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Contents\n\n\n1. Adding data set\n2. About train data \n    1. Data cleaning\n    2. Data exploring\n        1. Basic data exploration\n        2. Advance data exploration"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.style as style\nstyle.use('fivethirtyeight')\nimport seaborn as sns\nimport datetime","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1. Adding data set\n* riiid-test-answer-prediction\n* riiid-parquet-files\n\n>Since the train dataset is huge(5G), i added the riiid_parquet_files data [here](https://www.kaggle.com/ryati131457/riiid-parquet-files) in this kernel, and intent to use the train.parquet file to load the train dataset, and it deducted the loading time from 10 minutes to about 10 second. Thanks Ryati!"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain_original = pd.read_parquet('../input/riiid-parquet-files/train.parquet')\n\nprint('Trian size:', train_original.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# import datatable as dt\n\n# # reading data from csv using datatable and converting to pandas\n# train_data = dt.fread(\"../input/riiid-test-answer-prediction/train.csv\").to_pandas()\n\n# # writing dataset as pickle\n# train_data.to_pickle(\"riiid_train.pkl.gzip\")\n\n# # load pickled train data\n# train_data = pd.read_pickle(\"../input/riiid_train.pkl.gzip\")\n\n# print(\"Train size:\", data.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. About train data\n\n* row_id: (int64) ID code for the row.\n\n* timestamp: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n\n* user_id: (int32) ID code for the user.\n\n* content_id: (int16) ID code for the user interaction\n\n* content_type_id: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n* task_container_id: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n* user_answer: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\n* answered_correctly: (int8) if the user responded correctly. Read -1 as null, for lectures.\n\n* prior_question_elapsed_time: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n\n* prior_question_had_explanation: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.\n\n**Problems from exploring the train data file:**\n\n* data types are not approporate.\n* having missing values."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('original data information')\ntrain_original.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Data cleaning"},{"metadata":{},"cell_type":"markdown","source":"### 2.1.1 Convert data types \nAccroding to the description of the train data from the riiid education, we need to convert the dtypes to the corresponding data types. Since the parquet dataset has already converted most of the datatypes, here I only need to convert the content_type_id datatype from bool to int8."},{"metadata":{"trusted":true},"cell_type":"code","source":"# copy the original trian data for convienct exploration\ntrain = train_original.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# covert content_type_id data type\ntrain['content_type_id'] = train.content_type_id.astype('int8')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.1.2 Deal with missing data\nOnly below two columns have missing values\n* NaN values in prior_question_elapsed_time\n* NAN values in prior_question_had_explanation\n\nFor here, we will keep all the missing data so thatwe can easily explain the meaning of each column in the next simple EDA section.\n\nLater, we will drop all the nan values in the advanced EDA visualization section, since the first question in each bundle is a diagnostic question, and we focus on the question rather than the lecture."},{"metadata":{"trusted":true},"cell_type":"code","source":"# check all nan values\ntrain.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# count the num of values for lectures \ntrain[train['content_type_id'] == 1].count()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# count the num of values for lecture and first question bundles \ntrain[(train['prior_question_had_explanation'] != 1) & (train['prior_question_had_explanation'] != 0)].count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Findings:**\n\nAll 392506 Null values in prior_question_had_explanation are for user's first question bundles, which are all onboarding diagnostic tests where they did not get any feedbacks.\n\nOne part of null values(392506) in prior_question_elapsed_time are also for a user's first question bundle, and the other part of null values(2351538 - 392506 = 1959032) are for lectures.\n\nFor here, we will keep all the missing data so thatwe can easily explain the meaning of each column in the next simple EDA section.\n\nLater, we will drop all the nan values in the advanced EDA visualization section, since the first question in each bundle is a diagnostic question, and we focus on the question rather than the lecture."},{"metadata":{},"cell_type":"markdown","source":"## 2.2 Data exploring"},{"metadata":{},"cell_type":"markdown","source":"### 2.2.1 Basic data exploration\n* timestamp: the time(day) between this user interaction and the first event completion from that user\n* user_id: find the number of unique users.\n* content_id: find the total number of different content ids and the percentage of questions in the total content ids.\n* content_type_id: find number of question and lecture in the train data respectively. (0 represents a question, 1 represents a 1ecture)\n* task_container_id: find the number of unique id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n* user_answer: find user's answer to the question, -1 for lectures.\n* answered_correctly: find the correct responses, -1 for lectures."},{"metadata":{},"cell_type":"markdown","source":"timestamp: ms to minute"},{"metadata":{"trusted":true},"cell_type":"code","source":"# out of memory\n# 1 day = 86400000 ms\n# time_spent_dis = train.timestamp.apply(lambda x: x/86400000)\n# fig = plt.figure(figsize=(6,4))\n# time_spent_dis.plot.hist(bins=50)\n# plt.axvline(time_spent_dis.median(), color = 'r',linestyle = '--', linewidth=1)\n# plt.title('Histgram of Timestamp')\n# plt.xlabel('Days between this user interaction and the first event completion from that user')\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('After calculation, we can see that most user interactions were not active very long on the APP. \\\nThe median interactive time is about {:,} ms (about 31 days), the mean interactive timeis about {} ms (89 days),\\\nthe longest interactive time is about {:,} ms(1012 days). \\\nSince the median is small than mean, we can say that it is a right skewed distribution with some outliers.'\n      .format(round(train.timestamp.median()),\n              round(train.timestamp.median()), \n              round(train.timestamp.max())))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"user_id:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# user_id\nprint('Number of unique users in train dataset: {}'.format(train.user_id.nunique()))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"content_id:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# content_id\ntop_10_content_list = list(train.content_id.value_counts().sort_values(ascending=False)[:10])\nprint('There are {} unique content in the train set. The toal 10 most frequent used content ids are {}.'.format(\ntrain.content_id.nunique(), top_10_content_list))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"content_type_id"},{"metadata":{"trusted":true},"cell_type":"code","source":"# content_type_id\nquestion, lecture = train.content_type_id.value_counts()\nprint('There are {} questions and {} lectures in the trian dataset, and questions account for {}% of the total content.'.format(\nquestion, lecture, round(question/(question + lecture)*100,1)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"task_container_id: unique batches num"},{"metadata":{"trusted":true},"cell_type":"code","source":"# task_container_id: unique batches num\nprint('the number of unique batches for questions or lectures: {}'.format(\ntrain.task_container_id.nunique()))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"user_answer:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# user_answer\nprint('0-3 are the answers to questions, -1 is no-answer (lecture).')\ntrain.user_answer.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"answered_correctly: the correct response to questions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# answered_correctly: the correct response to questions\ncorrect_question, notcorrect_question, noanswer_lecture = train.answered_correctly.value_counts()\nprint('There are total {} answered questions in the train data. \\\n {} questions were answered correctly and {} were not answered correctly.\\\n The correct answered rate is about {}%.'.format(\n    correct_question + notcorrect_question, correct_question,\n    notcorrect_question, round(correct_question/(correct_question + notcorrect_question)*100,1)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# -1 for lecture, so only focus on question\nanswer_correct = train[train[\"answered_correctly\"] != -1].answered_correctly.value_counts(ascending= True)\nanswer_correct.plot.bar(tick_label = ['correct', 'not correct'])\nfor i,v in zip(answer_correct.index, answer_correct.values):\n    plt.text(i,v,'{:,}'.format(v))\n    \nplt.xticks([0,1],labels = ['not correct', 'correct'], rotation= 360)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 2.2.2 Advanced data exploration\n\n**Exploring possible features**\n1. total time(ms) spent on the APP\n2. Total number of questions answered in the Riiid APP\n3. the average rate of questions answered correctly for each user\n4. the average rate of whether a user saw an explanation after the prior question"},{"metadata":{},"cell_type":"markdown","source":"get all users' question data, and explore 4 possible features:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# drop all lectures and the first question of each bundle\ntrain = train.dropna()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get all users' question content type data \nusers = train.groupby('user_id')\n\n# get correct questions\nuser_answers = users['answered_correctly'].agg(correctly_answered_mean = 'mean', \n                correctly_answered = 'sum', total_answered = 'count')\nuser_answers['correctly_answered'] = user_answers.correctly_answered.astype('int64')\n\n# get the total spent time(ms)\nuser_time = users['timestamp'].agg(total_time_spent = 'max')\n\n# concate two dataframe\nuser_correct = pd.concat([user_answers,user_time], axis = 1)\n\n# get the average of prior_question_had_explanation\nuser_had_explanation = users['prior_question_had_explanation'].\\\n    agg(had_explanation_mean = 'mean')\n\n#concatenate user_answers and user_time two data frame to one\nuser_correct_info = pd.concat([user_correct, user_had_explanation], axis = 1)\nuser_correct_info.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# user_correct_info.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EDA visualization 1:\n\ntotal_time_spent\n\n1. Total time spent on the App"},{"metadata":{"trusted":true},"cell_type":"code","source":"# change ms to day, out of memory\n# 1 day = 86400000 ms\n#user_correct_info['total_time_spent'] = user_correct_info.total_time_spent.apply(lambda x: x/86400000)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# distribution of Timestamps (total time spent on App)\nfig = plt.figure(figsize=(8,4))\nuser_correct_info[\"total_time_spent\"].plot.hist(bins=50)\n\nplt.axvline(user_correct_info[\"total_time_spent\"].median(), color = 'r',linestyle = '--', linewidth=1)\nplt.axvline(user_correct_info[\"total_time_spent\"].mean(), color = 'y',linestyle = '--', linewidth=1)\nplt.title('Distribution of Total Time (ms) Spent on App')\nplt.xlabel('Total time (ms)')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"col = user_correct_info[\"total_time_spent\"]\nprint('From the distribution above, we can see that most user interactions were not active very long on the APP. \\\nThe median, mean, and longest interactive time between a user interaction and the first event completion from that user \\\nis about {}, {}, and {:,} days, respectively. \\\nFrom the graph above we can see it is a right skewed distribution with some outliers.'.format(\nround(col.median()/86400000), round(col.mean()/86400000), round(col.max()/86400000)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EDA visualization 2:\n\nprior_question_had_explanation\n1. Number of answers with a prior question had explanation\n2. Number of answers without a prior question had explanation\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# had explanation\nanswers_with_explanation = train[train[\"prior_question_had_explanation\"] == 1]\\\n                            .answered_correctly.value_counts(ascending=True)\ntitle1 = 'Answers with prior question had explanation'\n\n# no explanation\nanswer_without_explanation = train[train[\"prior_question_had_explanation\"] == 0]\\\n                        .answered_correctly.value_counts(ascending=True)\ntitle2 = 'Answers without prior question had explanation'\n\ndef plotbarh(data, title, ylabel='Answers', xlabel='Counts'):\n    plt.figure(figsize=(6,4)) \n    data.plot.barh()\n    \n    for i,v in zip(data.values, data.index):\n        plt.text(i,v,'{:,}'.format(i))\n    plt.yticks([0,1],labels = ['not correct', 'correct'], rotation= 360)\n    plt.title(title)\n    plt.ylabel(ylabel)\n    plt.xlabel(xlabel)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotbarh(answers_with_explanation, title1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plotbarh(answer_without_explanation, title2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **Findings:**\n\nIf a user saw an explanation and the correct response(s) after answering the previous question, the number of correct answers is more than twice the number of incorrect answers (60 million, 29 million).\n\nIf the user did not see the explanation and the correct answer after answering the previous question, there is no significant difference between the number of correct answers and the number of incorrect answers (4.6 million, 4.5 million)."},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}