{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About the Riiid AIEd Challenge 2020\n\nRiiid Labs, an AI solutions provider delivering creative disruption to the education market, empowers global education players to rethink traditional ways of learning leveraging AI. With a strong belief in equal opportunity in education, Riiid launched an AI tutor based on deep-learning algorithms in 2017 that attracted more than one million South Korean students. This year, the company released EdNet, the world’s largest open database for AI education containing more than 100 million student interactions.\n\nIn this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions. You will pair your machine learning skills using Riiid’s EdNet data. \n\nSubmissions are evaluated on area under the ROC curve between the predicted probability and the observed target."},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents\n\n[**1. EDA**](#1.-EDA)\n\n[1.1 Exploring Train](#1.1-Exploring-Train)\n\n[1.2 Exploring Questions](#1.2-Exploring-Questions)\n\n[1.3 Exploring Lectures](#1.3-Exploring-Lectures)\n  \n[**2. Baseline model**](#2.-Baseline-model)"},{"metadata":{},"cell_type":"markdown","source":"# 1. EDA\n\nAltogether, we are given 7 files.\n\n>Tailoring education to a student's ability level is one of the many valuable things an AI tutor can do. Your challenge in this competition is a version of that overall task; you will predict whether students are able to answer their next questions correctly. You'll be provided with the same sorts of information a complete education app would have: that student's historic performance, the performance of other students on the same question, metadata about the question itself, and more.\n\n>This is a time-series code competition, you will receive test set data and make predictions with Kaggle's time-series API. Please be sure to review the Time-series API Details section closely.\n\nSo we should realize that example_test.csv really is just an example. The submission happens via the API."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.style as style\nstyle.use('fivethirtyeight')\nimport seaborn as sns\nimport os\nfrom matplotlib.ticker import FuncFormatter\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input/riiid-test-answer-prediction'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As the train dataset is huge, I am gladly using the pickle that Rohan Rao prepared in this kernel: https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets/ (Thanks Rohan!). I actually do this at work all the time, and in this case it reduces the time to load the dataset (with the data types specified in the file description) from close to 9 minutes to about 16 seconds.\n\nAs we can see, we have over 101 million rows the the train set.\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%time\n\ntrain = pd.read_pickle(\"../input/riiid-train-data-multiple-formats/riiid_train.pkl.gzip\")\n\nprint(\"Train size:\", train.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's start by checking how much memory this dataframe is using."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.memory_usage(deep=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hmm.....we can see that 'prior_question_had_explanation' is object and taking a lot of memory, while it is supposed to be boolean. Let's fix this before continuing."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['prior_question_had_explanation'] = train['prior_question_had_explanation'].astype('boolean')\n\ntrain.memory_usage(deep=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The other files don't take very long to load, and I am importing the CSVs directly."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%time\n\nquestions = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\nexample_test = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv')\nexample_sample_submission = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1.1 Exploring Train\n\nThe columns in the train file are described as:\n* row_id: (int64) ID code for the row.\n* timestamp: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n* user_id: (int32) ID code for the user.\n* content_id: (int16) ID code for the user interaction\n* content_type_id: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n* task_container_id: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n* user_answer: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n* answered_correctly: (int8) if the user responded correctly. Read -1 as null, for lectures.\n* prior_question_elapsed_time: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n* prior_question_had_explanation: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback\n\nThe train dataset is ordered by ascending user_id and ascending timestamp."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(10)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'We have {train.user_id.nunique()} unique users in our train set')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Content_type_id = False means that a question was asked. True means that the user was watching a lecture."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.content_type_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Content_id is a code for the user interaction. Basically, these are the questions if content_type is question (question_id: foreign key for the train/test content_id column, when the content type is question)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'We have {train.content_id.nunique()} content ids in our train set, of which {train[train.content_type_id == False].content_id.nunique()} are questions.')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"cids = train.content_id.value_counts()[:30]\n\nfig = plt.figure(figsize=(12,6))\nax = cids.plot.bar()\nplt.title(\"Thirty most used content id's\")\nplt.xticks(rotation=90)\nax.get_yaxis().set_major_formatter(FuncFormatter(lambda x, p: format(int(x), ','))) #add thousands separator\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"task_container_id: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'We have {train.task_container_id.nunique()} unique Batches of questions or lectures.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"User answer. Seems that the questions are multiple choice (answers 0-3). As mentioned in the data description, -1 is actually no-answer (as the interaction was a lecture instead of a question)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.user_answer.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"timestamp: (int64) the time in milliseconds between this user interaction and the first event completion from that user. As you can see, most interactions are from users that were not active very long on the platform yet."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#1 year = 31536000000 ms\nts = train['timestamp']/(31536000000/12)\nfig = plt.figure(figsize=(12,6))\nts.plot.hist(bins=100)\nplt.title(\"Histogram of timestamp\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Months between this user interaction and the first event completion from that user\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Do we have the full history of all user_id's? Yes, if we filter train on timestamp==0, we get a time 0 for all users."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'Of the {train.user_id.nunique()} users in train we have {train[train.timestamp == 0].user_id.nunique()} users with a timestamp zero row.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corr_matrix=train.corr()\ncorr_matrix['answered_correctly'].sort_values(ascending=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(13,10))\nsns.heatmap(corr_matrix,annot=True,\n           linewidths=5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The target: answered_correctly\nAnswered_correctly is our target, and we have to predict to probability for an answer to be correct. Without looking at the lecture interactions (-1), we see about 1/3 of the questions was answered incorrectly."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"correct = train[train.answered_correctly != -1].answered_correctly.value_counts(ascending=True)\n\nfig = plt.figure(figsize=(12,4))\ncorrect.plot.barh()\nfor i, v in zip(correct.index, correct.values):\n    plt.text(v, i, '{:,}'.format(v), color='white', fontweight='bold', fontsize=14, ha='right', va='center')\nplt.title(\"Questions answered correctly\")\nplt.xticks(rotation=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I also want to find out if there is a relationship between timestamp and answered_correctly. To find out I have made 5 bins of timestamp. As you can see, the only noticable thing is that users who have registered relatively recently perform a little worse than users who are active longer."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"bin_labels_5 = ['Bin_1', 'Bin_2', 'Bin_3', 'Bin_4', 'Bin_5']\ntrain['ts_bin'] = pd.qcut(train['timestamp'], q=5, labels=bin_labels_5)\n\n#make function that can also be used for other fields\ndef correct(field):\n    correct = train[train.answered_correctly != -1].groupby([field, 'answered_correctly'], as_index=False).size()\n    correct = correct.pivot(index= field, columns='answered_correctly', values='size')\n    correct['Percent_correct'] = round(correct.iloc[:,1]/(correct.iloc[:,0] + correct.iloc[:,1]),2)\n    correct = correct.sort_values(by = \"Percent_correct\", ascending = False)\n    correct = correct.iloc[:,2]\n    return(correct)\n\nbins_correct = correct(\"ts_bin\")\nbins_correct = bins_correct.sort_index()\n\nfig = plt.figure(figsize=(12,6))\nplt.bar(bins_correct.index, bins_correct.values)\nfor i, v in zip(bins_correct.index, bins_correct.values):\n    plt.text(i, v, v, color='white', fontweight='bold', fontsize=14, va='top', ha='center')\nplt.title(\"Percent answered_correctly for 5 bins of timestamp\")\nplt.xticks(rotation=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's also check out what the distribution of answered_correctly looks like if we groupby the (10,000 unique) task_container_id's."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"task_id_correct = correct(\"task_container_id\")\n\nfig = plt.figure(figsize=(12,6))\ntask_id_correct.plot.hist(bins=40)\nplt.title(\"Histogram of percent_correct grouped by task_container_id\")\nplt.xticks(rotation=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below I am plotting the number of answers per user_id against the percentage of questions answered correctly (sample of 200). As some users have answered huge amounts of questions, I have taken out the outliers (user_ids with 1000+ questions answered). As you can see, the trend is upward but there is also a lot of variation among users that have answered few questions."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"user_percent = train[train.answered_correctly != -1].groupby('user_id')['answered_correctly'].agg(Mean='mean', Answers='count')\nprint(f'the highest number of questions answered by a user is {user_percent.Answers.max()}')\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"user_percent = user_percent.query('Answers <= 1000').sample(n=200, random_state=1)\n\nfig = plt.figure(figsize=(12,6))\nx = user_percent.Answers\ny = user_percent.Mean\nplt.scatter(x, y, marker='o')\nplt.title(\"Percent answered correctly versus number of questions answered User\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Number of questions answered\")\nplt.ylabel(\"Percent answered correctly\")\nz = np.polyfit(x, y, 1)\np = np.poly1d(z)\nplt.plot(x,p(x),\"r--\")\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below, I am doing the same thing by content_id (is question_id for content_type is question). I am again taking a sample of 200, and have taken out the content_ids with more than 25,000 questions asked. As you can see there is a slight downward trend."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"content_percent = train[train.answered_correctly != -1].groupby('content_id')['answered_correctly'].agg(Mean='mean', Answers='count')\nprint(f'The highest number of questions asked by content_id is {content_percent.Answers.max()}.')\nprint(f'Of {len(content_percent)} content_ids, {len(content_percent[content_percent.Answers > 25000])} content_ids had more than 25,000 questions asked.')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"content_percent = content_percent.query('Answers <= 25000').sample(n=200, random_state=1)\n\nfig = plt.figure(figsize=(12,6))\nx = content_percent.Answers\ny = content_percent.Mean\nplt.scatter(x, y, marker='o')\nplt.title(\"Percent answered correctly versus number of questions answered Content_id\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Number of questions answered\")\nplt.ylabel(\"Percent answered correctly\")\nz = np.polyfit(x, y, 1)\np = np.poly1d(z)\nplt.plot(x,p(x),\"r--\")\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Does it help if the 'prior_question_had_explanation'? Yes, as you can see the percent answered correctly is about 17% higher when there was an explanation. Although it is probably better to treat not having an explanation as a disadvantage as there was an explanation before the vast majority of questions.\n\nIn addition, it is also interesting to see that the percent answered correctly for the missing values is closer to True than to False."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pq = train[train.answered_correctly != -1].groupby(['prior_question_had_explanation'], dropna=False).agg({'answered_correctly': ['mean', 'count']})\n#pq.index = pq.index.astype(str)\nprint(pq.iloc[:,1])\npq = pq.iloc[:,0]\n\nfig = plt.figure(figsize=(12,4))\npq.plot.barh()\n# for i, v in zip(pq.index, pq.values):\n#     plt.text(v, i, round(v,2), color='white', fontweight='bold', fontsize=14, ha='right', va='center')\nplt.title(\"Answered_correctly versus Prior Question had explanation\")\nplt.xlabel(\"Percent answered correctly\")\nplt.ylabel(\"Prior question had explanation\")\nplt.xticks(rotation=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"prior_question_elapsed_time: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n\nAt first glance, this does not seem very interesting regarding our target. For both wrong and correct answers, the mean is about 25 seconds."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"pq = train[train.answered_correctly != -1]\npq = pq[['prior_question_elapsed_time', 'answered_correctly']]\npq = pq.groupby(['answered_correctly']).agg({'answered_correctly': ['count'], 'prior_question_elapsed_time': ['mean']})\n\npq","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"However, as the feature works with regards to the CV (see Baseline model), I also wanted to find out if there is a trend. Below, I have taken a sample of 200 rows. As you can see, there is s slightly downward trend."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"#please be aware that there is an issues with train.prior_question_elapsed_time.mean()\n#see https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195032\nmean_pq = train.prior_question_elapsed_time.astype(\"float64\").mean()\n\ncondition = ((train.answered_correctly != -1) & (train.prior_question_elapsed_time.notna()))\npq = train[condition][['prior_question_elapsed_time', 'answered_correctly']].sample(n=200, random_state=1)\npq = pq.set_index('prior_question_elapsed_time').iloc[:,0]\n\nfig = plt.figure(figsize=(12,6))\nx = pq.index\ny = pq.values\nplt.scatter(x, y, marker='o')\nplt.title(\"Answered_correctly versus prior_question_elapsed_time\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Prior_question_elapsed_time\")\nplt.ylabel(\"Answered_correctly\")\nplt.vlines(mean_pq, ymin=-0.1, ymax=1.1)\nplt.text(x= 27000, y=0.4, s='mean')\nplt.text(x=80000, y=0.6, s='trend')\nz = np.polyfit(x, y, 1)\np = np.poly1d(z)\nplt.plot(x,p(x),\"r--\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1.2 Exploring Questions\n\nMetadata for the questions posed to users.\n\n* question_id: foreign key for the train/test content_id column, when the content type is question (0).\n* bundle_id: code for which questions are served together.\n* correct_answer: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n* part: the relevant section of the TOEIC test.\n* tags: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The tags seem valuable to me. First, let's check if there are any question_id's without tags. As you can see, there is exactly one question_id without at least one tag. Not a big deal, but we need to keep in mind that we have to impute something here if we make features based on tags."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"questions[questions.tags.isna()]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also.....when looking at train, we see that this question was just asked once ;-). "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.query('content_id == \"10033\" and answered_correctly != -1')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"questions['tags'] = questions['tags'].astype(str)\n\ntags = [x.split() for x in questions[questions.tags != \"nan\"].tags.values]\ntags = [item for elem in tags for item in elem]\ntags = set(tags)\ntags = list(tags)\nprint(f'There are {len(tags)} different tags')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's find out how many answers were Right and Wrong per question_id (so per content_id in train)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tags_list = [x.split() for x in questions.tags.values]\nquestions['tags'] = tags_list\nquestions.head()\n\ncorrect = train[train.answered_correctly != -1].groupby([\"content_id\", 'answered_correctly'], as_index=False).size()\ncorrect = correct.pivot(index= \"content_id\", columns='answered_correctly', values='size')\ncorrect.columns = ['Wrong', 'Right']\ncorrect = correct.fillna(0)\ncorrect[['Wrong', 'Right']] = correct[['Wrong', 'Right']].astype(int)\nquestions = questions.merge(correct, left_on = \"question_id\", right_on = \"content_id\", how = \"left\")\nquestions.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, I have also changed the tags column into lists of tags."},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.tags.values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, I can add up all Wrong and Right answers for all questions that are labeled with a particular tag and calculate the percent correct for each tag. Please note that there is \"double counting\" of questions; for instance if a question has 5 tags, its answers are aggregated in the totals of each of the 5 tags. "},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\ntags_df = pd.DataFrame()\nfor x in range(len(tags)):\n    df = questions[questions.tags.apply(lambda l: tags[x] in l)]\n    df1 = df.agg({'Wrong': ['sum'], 'Right': ['sum']})\n    df1['Total_questions'] = df1.Wrong + df1.Right\n    df1['Question_ids_with_tag'] = len(df)\n    df1['tag'] = tags[x]\n    df1 = df1.set_index('tag')\n    tags_df = tags_df.append(df1)\n\ntags_df[['Wrong', 'Right', 'Total_questions']] = tags_df[['Wrong', 'Right', 'Total_questions']].astype(int)\ntags_df['Percent_correct'] = tags_df.Right/tags_df.Total_questions\ntags_df = tags_df.sort_values(by = \"Percent_correct\")\n\ntags_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see, the differences are significant!"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"select_rows = list(range(0,10)) + list(range(178, len(tags_df)))\ntags_select = tags_df.iloc[select_rows,4]\n\nfig = plt.figure(figsize=(12,6))\nx = tags_select.index\ny = tags_select.values\nclrs = ['red' if y < 0.6 else 'green' for y in tags_select.values]\ntags_select.plot.bar(x, y, color=clrs)\nplt.title(\"Ten hardest and ten easiest tags\")\nplt.xlabel(\"Tag\")\nplt.ylabel(\"Percent answers correct of questions with the tag\")\nplt.xticks(rotation=90)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"However, we should also realize that the tag with the worst percent_correct only has about 250,000 answers. This a low number compared to the tags with most answers."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tags_select = tags_df.sort_values(by = \"Total_questions\", ascending = False).iloc[:30,:]\ntags_select = tags_select[\"Total_questions\"]\n\nfig = plt.figure(figsize=(12,6))\nax = tags_select.plot.bar()\nplt.title(\"Thirty tags with most questions answered\")\nplt.xticks(rotation=90)\nplt.ticklabel_format(style='plain', axis='y')\nax.get_yaxis().set_major_formatter(FuncFormatter(lambda x, p: format(int(x), ','))) #add thousands separator\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What are the so-called \"Parts\"? When following the link provided in the data description we find out that this relates to a test.\n\n> The TOEIC L&R uses an optically-scanned answer sheet. There are 200 questions to answer in two hours in Listening (approximately 45 minutes, 100 questions) and Reading (75 minutes, 100 questions). \n\nThe listening section consists of Part 1-4 (Listening Section (approx. 45 minutes, 100 questions)).\n\nThe reading section consists of Part 5-7 (Reading Section (75 minutes, 100 questions))."},{"metadata":{},"cell_type":"markdown","source":"# Below, I am displaying the count and percent correct by part. As you can see, Part 5 has a lot more question_id's and is also the most difficult."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = plt.figure(figsize=(12,8))\nax1 = fig.add_subplot(211)\nax1 = questions.groupby(\"part\").count()['question_id'].plot.bar()\nplt.title(\"Counts of part\")\nplt.xlabel(\"Part\")\nplt.xticks(rotation=0)\n\npart = questions.groupby('part').agg({'Wrong': ['sum'], 'Right': ['sum']})\npart['Percent_correct'] = part.Right/(part.Right + part.Wrong)\npart = part.iloc[:,2]\n\nax2 = fig.add_subplot(212)\nplt.bar(part.index, part.values)\nfor i, v in zip(part.index, part.values):\n    plt.text(i, v, round(v,2), color='white', fontweight='bold', fontsize=14, va='top', ha='center')\n\nplt.title(\"Percent_correct by part\")\nplt.xlabel(\"Part\")\nplt.xticks(rotation=0)\nplt.tight_layout(pad=2)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1.3 Exploring Lectures\n\nMetadata for the lectures watched by users as they progress in their education.\n* lecture_id: foreign key for the train/test content_id column, when the content type is lecture (1).\n* part: top level category code for the lecture.\n* tag: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n* type_of: brief description of the core purpose of the lecture\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'There are {lectures.shape[0]} lecture_ids.')\nlectures.shape\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](http://)Let's have a look at the type_of."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"lect_type_of = lectures.type_of.value_counts()\n\nfig = plt.figure(figsize=(12,6))\nplt.bar(lect_type_of.index, lect_type_of.values)\nfor i, v in zip(lect_type_of.index, lect_type_of.values):\n    plt.text(i, v, v, color='black', fontweight='bold', fontsize=14, va='bottom', ha='center')\nplt.title(\"Types of lectures\")\nplt.xlabel(\"type_of\")\nplt.ylabel(\"Count lecture_id\")\nplt.xticks(rotation=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since there are not that many lectures, I want to check if it helps if a user watches lectures at all. As you can see, it helps indeed!"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"user_lect = train.groupby([\"user_id\", \"answered_correctly\"]).size().unstack()\nuser_lect.columns = ['Lecture', 'Wrong', 'Right']\nuser_lect['Lecture'] = user_lect['Lecture'].fillna(0)\nuser_lect = user_lect.astype('Int64')\nuser_lect['Watches_lecture'] = np.where(user_lect.Lecture > 0, True, False)\n\nwatches_l = user_lect.groupby(\"Watches_lecture\").agg({'Wrong': ['sum'], 'Right': ['sum']})\nprint(user_lect.Watches_lecture.value_counts())\n\nwatches_l['Percent_correct'] = watches_l.Right/(watches_l.Right + watches_l.Wrong)\n\nwatches_l = watches_l.iloc[:,2]\n\nfig = plt.figure(figsize=(12,4))\nwatches_l.plot.barh()\nfor i, v in zip(watches_l.index, watches_l.values):\n    plt.text(v, i, round(v,2), color='white', fontweight='bold', fontsize=14, ha='right', va='center')\n\nplt.title(\"User watches lectures: Percent_correct\")\nplt.xlabel(\"Percent correct\")\nplt.ylabel(\"User watched at least one lecture\")\nplt.xticks(rotation=0)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Batches (task_container_id) may also contain lectures, and I want to find out if there are any batches with high numbers of lectures."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"batch_lect = train.groupby([\"task_container_id\", \"answered_correctly\"]).size().unstack()\nbatch_lect.columns = ['Lecture', 'Wrong', 'Right']\nbatch_lect['Lecture'] = batch_lect['Lecture'].fillna(0)\nbatch_lect = batch_lect.astype('Int64')\nbatch_lect['Percent_correct'] = batch_lect.Right/(batch_lect.Wrong + batch_lect.Right)\nbatch_lect['Percent_lecture'] = batch_lect.Lecture/(batch_lect.Lecture + batch_lect.Wrong + batch_lect.Right)\nbatch_lect = batch_lect.sort_values(by = \"Percent_lecture\", ascending = False)\n\nprint(f'The highest number of lectures watched within a single task_container_id is {batch_lect.Lecture.max()}.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see below (table sorted on descending Percent_lecture), the percent of lectures of the task_container_id's is never high. We can also see the highest percentages of lectures are around 2.8%, which means one lecture on about 36 questions."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"batch_lect.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Is there a correlation between the percent_lecture and the percent_correct? No, I don't really see it. If anything, the percent_correct actually seems to go down slightly."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"batch = batch_lect.iloc[:, 3:]\n\nfig = plt.figure(figsize=(12,6))\nx = batch.Percent_lecture\ny = batch.Percent_correct\nplt.scatter(x, y, marker='o')\nplt.title(\"Percent lectures in a task_container versus percent answered correctly\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Percent lectures\")\nplt.ylabel(\"Percent answered correctly\")\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The last thing that I want to check is if having a lecture in a batch helps. As you can see, it does not. Batches without lectures have about 8% more correct answers than batches with lectures."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"batch_lect['Has_lecture'] = np.where(batch_lect.Lecture == 0, False, True)\nprint(f'We have {batch_lect[batch_lect.Has_lecture == True].shape[0]} task_container_ids with lectures and {batch_lect[batch_lect.Has_lecture == False].shape[0]} task_container_ids without lectures.')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"batch_lect = batch_lect[['Wrong', 'Right', 'Has_lecture']]\nbatch_lect = batch_lect.groupby(\"Has_lecture\").sum()\nbatch_lect['Percent_correct'] = batch_lect.Right/(batch_lect.Wrong + batch_lect.Right)\nbatch_lect = batch_lect[['Percent_correct']]\nbatch_lect","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Example test\nThis file is a very small file, and only good to check what's in there.\n\nImportant: In the `Updates, corrections, and clarifications` topic is said that:\n* the hidden test set contains new users but not new questions\n* The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data. It remains possible that some questions weren't logged due to other issues that all datasets of mobile users are susceptible to,such as if a user lost their connection mid-question.\n* The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"batches_test = set(list(example_test.task_container_id.unique()))\nbatches_train = set(list(train.task_container_id.unique()))\nprint(f'All batches in example_test are also in train is {batches_test.issubset(batches_train)}.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Kaggle says that there are new users in the test set, but let's check this anyway with example_test. As we can see, there is a new user in example_test indeed."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"user_test = set(list(example_test.user_id.unique()))\nuser_train = set(list(train.user_id.unique()))\n\nprint(f'User_ids in example_test but not in train: {user_test - user_train}.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Baseline model"},{"metadata":{"trusted":true},"cell_type":"code","source":"#this clears everything loaded in RAM, including the libraries\n%reset -f","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\n\nimport numpy as np\nimport pandas as pd\nimport riiideducation\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.style as style\nstyle.use('fivethirtyeight')\nimport seaborn as sns\nimport os\nimport lightgbm as lgb\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.preprocessing import LabelEncoder\nimport gc\nimport sys\npd.set_option('display.max_rows', None)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"%%time\ncols_to_load = ['row_id', 'user_id', 'answered_correctly', 'content_id', 'prior_question_had_explanation', 'prior_question_elapsed_time']\ntrain = pd.read_pickle(\"../input/riiid-train-data-multiple-formats/riiid_train.pkl.gzip\")[cols_to_load]\ntrain['prior_question_had_explanation'] = train['prior_question_had_explanation'].astype('boolean')\n\nprint(\"Train size:\", train.shape)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"%%time\n\nquestions = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\nexample_test = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv')\nexample_sample_submission = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#adding user features\nuser_df = train[train.answered_correctly != -1].groupby('user_id').agg({'answered_correctly': ['count', 'mean']}).reset_index()\nuser_df.columns = ['user_id', 'user_questions', 'user_mean']\n\nuser_lect = train.groupby([\"user_id\", \"answered_correctly\"]).size().unstack()\nuser_lect.columns = ['Lecture', 'Wrong', 'Right']\nuser_lect['Lecture'] = user_lect['Lecture'].fillna(0)\nuser_lect = user_lect.astype('Int64')\nuser_lect['watches_lecture'] = np.where(user_lect.Lecture > 0, 1, 0)\nuser_lect = user_lect.reset_index()\nuser_lect = user_lect[['user_id', 'watches_lecture']]\n\nuser_df = user_df.merge(user_lect, on = \"user_id\", how = \"left\")\ndel user_lect\nuser_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#adding content features\ncontent_df = train[train.answered_correctly != -1].groupby('content_id').agg({'answered_correctly': ['count', 'mean']}).reset_index()\ncontent_df.columns = ['content_id', 'content_questions', 'content_mean']\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Tito rightfully argues that just taking the last couple of questions from each user as the validation set leads to much on \"light users\" in this kernel (Thanks Tito!): https://www.kaggle.com/its7171/cv-strategy"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n#using one of the validation sets composed by tito\ncv2_train = pd.read_pickle(\"../input/riiid-cross-validation-files/cv2_train.pickle\")['row_id']\ncv2_valid = pd.read_pickle(\"../input/riiid-cross-validation-files/cv2_valid.pickle\")['row_id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train[train.answered_correctly != -1]\n\n#save mean before splitting\n#please be aware that there is an issues with train.prior_question_elapsed_time.mean()\n#see https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/195032\nmean_prior = train.prior_question_elapsed_time.astype(\"float64\").mean()\n\nvalidation = train[train.row_id.isin(cv2_valid)]\ntrain = train[train.row_id.isin(cv2_train)]\n\nvalidation = validation.drop(columns = \"row_id\")\ntrain = train.drop(columns = \"row_id\")\n\ndel cv2_train, cv2_valid\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"In the previous version, I made a function for all the merges, fillna's and label encoding below. However, after adding a few features in this version, I ran into memory issues. I seemed as if a copy of train was kept in RAM at least temporarily, and I ran into an out-of memory error. Therefore, I unfortunately went back to an ugly version of code repetition for those steps (same code for validation and test_df)."},{"metadata":{"trusted":true},"cell_type":"code","source":"label_enc = LabelEncoder()\n\ntrain = train.merge(user_df, on = \"user_id\", how = \"left\")\ntrain = train.merge(content_df, on = \"content_id\", how = \"left\")\ntrain['content_questions'].fillna(0, inplace = True)\ntrain['content_mean'].fillna(0.5, inplace = True)\ntrain['watches_lecture'].fillna(0, inplace = True)\ntrain['user_questions'].fillna(0, inplace = True)\ntrain['user_mean'].fillna(0.5, inplace = True)\ntrain['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\ntrain['prior_question_had_explanation'].fillna(False, inplace = True)\ntrain['prior_question_had_explanation'] = label_enc.fit_transform(train['prior_question_had_explanation'])\ntrain[['content_questions', 'user_questions']] = train[['content_questions', 'user_questions']].astype(int)\ntrain.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validation = validation.merge(user_df, on = \"user_id\", how = \"left\")\nvalidation = validation.merge(content_df, on = \"content_id\", how = \"left\")\nvalidation['content_questions'].fillna(0, inplace = True)\nvalidation['content_mean'].fillna(0.5, inplace = True)\nvalidation['watches_lecture'].fillna(0, inplace = True)\nvalidation['user_questions'].fillna(0, inplace = True)\nvalidation['user_mean'].fillna(0.5, inplace = True)\nvalidation['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\nvalidation['prior_question_had_explanation'].fillna(False, inplace = True)\nvalidation['prior_question_had_explanation'] = label_enc.fit_transform(validation['prior_question_had_explanation'])\nvalidation[['content_questions', 'user_questions']] = validation[['content_questions', 'user_questions']].astype(int)\nvalidation.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see below, I am using only 5 features for this baseline. Initially, I also included \"watches_lecture\" and \"prior_question_had_explanation\", but these two features are very questionable. With those two features, CV barely goes up (less than 0.001), the public score goes down 0.001 and feature importance is very low for both features. Therefore, I believe that the simpler model is preferred. If you want still check what the numbers look like with those features, you can do that by simply hashing in and out the features line."},{"metadata":{"trusted":true},"cell_type":"code","source":"# features = ['user_questions', 'user_mean', 'content_questions', 'content_mean', 'watches_lecture',\n#             'prior_question_elapsed_time', 'prior_question_had_explanation']\n\nfeatures = ['user_questions', 'user_mean', 'content_questions', 'content_mean', 'prior_question_elapsed_time']\n\n\n#for now just taking 10.000.000 rows for training\ntrain = train.sample(n=10000000, random_state = 1)\n\ny_train = train['answered_correctly']\ntrain = train[features]\n\ny_val = validation['answered_correctly']\nvalidation = validation[features]\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {'objective': 'binary',\n          'metric': 'auc',\n          'seed': 2020,\n          'learning_rate': 0.1, #default\n          \"boosting_type\": \"gbdt\" #default\n         }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb_train = lgb.Dataset(train, y_train, categorical_feature = None)\nlgb_eval = lgb.Dataset(validation, y_val, categorical_feature = None)\ndel train, y_train, validation, y_val\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nmodel = lgb.train(\n    params, lgb_train,\n    valid_sets=[lgb_train, lgb_eval],\n    verbose_eval=50,\n    num_boost_round=10000,\n    early_stopping_rounds=8\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb.plot_importance(model)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    test_df = test_df.merge(user_df, on = \"user_id\", how = \"left\")\n    test_df = test_df.merge(content_df, on = \"content_id\", how = \"left\")\n    test_df['content_questions'].fillna(0, inplace = True)\n    test_df['content_mean'].fillna(0.5, inplace = True)\n    test_df['watches_lecture'].fillna(0, inplace = True)\n    test_df['user_questions'].fillna(0, inplace = True)\n    test_df['user_mean'].fillna(0.5, inplace = True)\n    test_df['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\n    test_df['prior_question_had_explanation'].fillna(False, inplace = True)\n    test_df['prior_question_had_explanation'] = label_enc.fit_transform(test_df['prior_question_had_explanation'])\n    test_df[['content_questions', 'user_questions']] = test_df[['content_questions', 'user_questions']].astype(int)\n    test_df['answered_correctly'] =  model.predict(test_df[features])\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.078276,"end_time":"2020-12-03T08:48:56.2356","exception":false,"start_time":"2020-12-03T08:48:56.157324","status":"completed"},"tags":[]},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}