{"cells":[{"metadata":{},"cell_type":"markdown","source":"Riiid Labs, an AI solutions provider delivering creative disruption to the education market, empowers global education players to rethink traditional ways of learning leveraging AI. With a strong belief in equal opportunity in education, Riiid launched an AI tutor based on deep-learning algorithms in 2017 that attracted more than one million South Korean students. This year, the company released EdNet, the world’s largest open database for AI education containing more than 100 million student interactions.\n\nIn this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions. You will pair your machine learning skills using Riiid’s EdNet data. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport os","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This part of the program is based on the notebook by Rohan Rao: https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets\n\n- The Riiid! Answer Correctness Prediction dataset has over 100 million rows and 10 columns.The usual  pd.read_csv will result in an out-of-memory error. \n\n- We also convert the dataset into another format which uses lesser disk space, is smaller in size and/or can be read faster for subsequent reads. "},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"Using Datable to read large databases. \nDocumentation: https://datatable.readthedocs.io/en/latest/index.html"},{"metadata":{},"cell_type":"markdown","source":"The training dataset has the following features:\n\n\n    - row_id: (int64) ID code for the row.\n    \n    - timestamp: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n    \n    - user_id: (int32) ID code for the user.\n    \n    - content_id: (int16) ID code for the user interaction\n    \n    - content_type_id: (bool) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n    \n    - task_container_id: (int16) ID code for the batch of questions or lectures. (eg. a user might see three questions in a row before seeing the explanations for any of them - those three would all share a task_container_id)\n    \n    - user_answer: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n    \n    - answered_correctly: (int8) if the user responded correctly. Read -1 as null, for lectures.\n    \n    - prior_question_elapsed_time: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between (is null for a user's first question bundle or lecture)\n    \n    - prior_question_had_explanation: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# Load the train data set\ntrain = pd.read_pickle('../input/riiid-train-data-multiple-formats/riiid_train.pkl.gzip')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### checking memory usage\ntrain.memory_usage(deep = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['prior_question_had_explanation'] = train['prior_question_had_explanation'].astype('bool')\n\ntrain.memory_usage(deep=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## what is the number of unique users in the dataset\nprint(f'We have {train.user_id.nunique()} unique user ids in the dataset')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train['content_type_id'].value_counts())\nprint('\\nIn percentages\\n',train['content_type_id'].value_counts(normalize = True))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### what is the number of interactions that are lectures in the dataset\ntemp = len(train.loc[train['content_type_id'] == True])\nprint(f' the number of interactions that are lectures are {temp} and the rest {(len(train) - temp)} are questions')\nprint(f' the proportion of interactions that are lectures are {lectures/len(train): 0.2f} and the proportion of questions are {(len(train) - temp)/len(train):0.2f}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## how many different types of contents are there in these interactions\nunique_contents = train['content_id'].nunique()\nprint(f'There are {unique_contents} unique contents in the entire dataset')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#within the dataset of only questions (leaving out lectures) how many contents are there\nunique_questions = train[train['content_type_id'] == False].content_id.nunique()\nprint(f'There are {unique_questions} unique contents in the dataset of just questions')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### showing the top most used content_ids\ncommon_ids = train['content_id'].value_counts()[0:50]\n\nfig = plt.figure(figsize=(12,6))\nax = common_ids.plot.bar()\nplt.title(\"Fifty most used content id's\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"task_container_id: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id."},{"metadata":{"trusted":true},"cell_type":"code","source":"## how many different types of task containers are there in these interactions\nunique_tasks = train['task_container_id'].nunique()\nprint(f'There are {unique_tasks} unique task containers in the entire dataset (this includes lectures and questions)')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### showing the top most used task containers\ncommon_tasks = train['task_container_id'].value_counts()[0:30]\n\nfig = plt.figure(figsize=(12,6))\nax = common_tasks.plot.bar()\nplt.title(\"Thirty most used task containers\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"User answer. questions are multiple choice (answers 0-3). As mentioned in the data description, -1 is actually no-answer (as the interaction was a lecture instead of a question). Remember that we already found that 0.019352 was lectures, so that informaiton should match up\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.concat([train.user_answer.value_counts(), \n                train.user_answer.value_counts(normalize=True).mul(100)],axis=1, keys=('counts','percentage'))\n\nprint(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### showing the top most used task containers\nuser_answers = train['user_answer'].value_counts()\n\nfig = plt.figure(figsize=(12,6))\nax = user_answers.plot.bar()\nplt.title(\"User Answers\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Examining the column 'answered correctly' '-1' stands for lectures so we can also find out what percentage of the answers are lectures"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.concat([train['answered_correctly'].value_counts(), \n                train['answered_correctly'].value_counts(normalize=True).mul(100)],axis=1, keys=('counts','percentage'))\n\nprint(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### for now deleting all the lecture interactions\n'''train = train[train['content_type_id'] == False]\ntrain['answered_correctly'].value_counts()'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## how many users are there after deleting lectures\nprint(f' There are {train.user_id.nunique()} users after deleting the lectures')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### About 1/3 answered correctly","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### showing correct answers\ncorrect_answers = train['answered_correctly'].value_counts()\n\nfig = plt.figure(figsize=(12,6))\nax = correct_answers.plot.bar()\nplt.title(\"Correct Answers\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### showing correct answers horizontally\ncorrect_answers = train['answered_correctly'].value_counts()\n\nfig = plt.figure(figsize=(12,6))\nax = correct_answers.plot.barh()\nplt.title(\"Answered Correctly\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### crosstab between answered correctly and user answer\npd.crosstab(train.user_answer, train.answered_correctly)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### crosstab between answered correctly and content type\npd.crosstab(train.content_type_id, train.answered_correctly)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Not sure what the connection is between answered correctly and user answer"},{"metadata":{},"cell_type":"markdown","source":"timestamp: (int64) the time in milliseconds between this user interaction and the first event completion from that user. As you can see, most interactions are from users that were not active very long on the platform yet."},{"metadata":{"trusted":true},"cell_type":"code","source":"## so all 393656 users should have a timestamp of zero\nzero_timestamps = train[train.timestamp == 0].user_id.nunique()\nprint(f' of all the {train.user_id.nunique()} users, The number of users with timestamp equal zero is {zero_timestamps}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Something is wrong here, there are some user_id with no entry for timestamp = 0"},{"metadata":{"trusted":true},"cell_type":"code","source":"#1 year = 31536000000 ms\nts = train['timestamp']/(31536000000/12)\nfig = plt.figure(figsize=(12,6))\nts.plot.hist(bins=100)\nplt.title(\"Histogram of timestamp\")\n#plt.xticks(rotation=0)\nplt.xlabel(\"Months between this user interaction and the first event completion from that user\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#1 year = 31536000000 ms\nts = train['timestamp']/(31536000000/365)\nfig = plt.figure(figsize=(12,6))\nts.plot.hist(bins=100)\nplt.title(\"Histogram of timestamp\")\n#plt.xticks(rotation=0)\nplt.xlabel(\"Days between this user interaction and the first event completion from that user\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%who\nimport gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Is there a relationship between time stamp and answered correctly"},{"metadata":{"trusted":true},"cell_type":"code","source":"#bin_labels_5 = ['Bin_1', 'Bin_2', 'Bin_3', 'Bin_4', 'Bin_5']\ntrain['ts_bin'] = pd.qcut(train['timestamp'], q=20, labels = np.arange(20))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(12,6))\n#ax = df.plot.bar()\nax = train.groupby('ts_bin')['answered_correctly'].mean().plot.bar()\nplt.title(\"Answered Correctly\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initial interactions got a lower percentage of the answers correct, after that interactions much further out, they seem to have the same propotion correct"},{"metadata":{},"cell_type":"markdown","source":"Is there a relationship between number of questions answered and the percentage of correct"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_percent = train[train.answered_correctly != -1].groupby('user_id')['answered_correctly'].agg(Mean = 'mean', Nquestions = 'count')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'The highest number of quesitons answered by a user was {user_percent.Nquestions.max()}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from scipy.stats import pearsonr\ncorr, _ = pearsonr(user_percent.Mean, user_percent.Nquestions)\nprint('Pearsons correlation: %.3f' % corr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample = user_percent[user_percent.Nquestions < 5000].sample(n=2000, random_state = 1)\nfig = plt.figure(figsize=(12,6))\nx = sample.Nquestions\ny = sample.Mean\nplt.scatter(x, y, marker='o')\nplt.title(\"Percent answered correctly versus number of questions answered\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Number of questions answered\")\nplt.ylabel(\"Percent answered correctly\")\nz = np.polyfit(x, y, 1)\np = np.poly1d(z)\nplt.plot(x,p(x),\"r--\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Does it help if the 'prior_question_had_explanation'? "},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.concat([train['prior_question_had_explanation'].value_counts(), \n                train['prior_question_had_explanation'].value_counts(normalize=True).mul(100)],axis=1, keys=('counts','percentage'))\n\nprint(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### showing prior question had explanation\ncorrect_answers = train['prior_question_had_explanation'].value_counts()\n\nfig = plt.figure(figsize=(12,6))\nax = correct_answers.plot.bar()\nplt.title(\"Prior Question had Explanation\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_answers = train[train.answered_correctly != -1].groupby('prior_question_had_explanation')['answered_correctly'].agg(Mean = 'mean')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nfig = plt.figure(figsize=(12,6))\nax = prior_answers.plot.bar()\nplt.title(\"Prior Question had Explanation\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## is there a relationshp between prior_question_elapsed_time and answered correctly","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_time = train[train.answered_correctly != -1].groupby('answered_correctly')['prior_question_elapsed_time'].agg(Mean = 'mean')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(12,6))\nax = prior_time.plot.bar()\nplt.title(\"Prior Question Time Taken\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Questions\n\nMetadata for the questions posed to users.\n\n    - question_id: foreign key for the train/test content_id column, when the content type is question (0).\n    - bundle_id: code for which questions are served together.\n    - correct_answer: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n    - part: the relevant section of the TOEIC test.\n    - tags: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\nquestions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Analysis with Tags"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions['tags'] = questions['tags'].astype(str)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tags = [x.split() for x in questions[questions.tags != \"nan\"].tags.values]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_list = []\nfor i in tags:\n    for j in i:\n        if j not in(tag_list):\n            tag_list.append(j)\n        \nprint(len(tag_list))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'There are {len(tag_list)} unique tags in the dataset')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tags_list = [x.split() for x in questions.tags.values]\nquestions['tags_list'] = tags_list\nquestions.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Find out which are the hardest tags and easiest tags"},{"metadata":{"trusted":true},"cell_type":"code","source":"## Also add information on lectures, there is a relationship between every watched a lecture and percent correct","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}