{"cells":[{"metadata":{},"cell_type":"markdown","source":"# RIIID : Answer Correctness Prediction\n\n"},{"metadata":{},"cell_type":"markdown","source":"\nI came across this interesting competition on Kaggle about a month ago, and its been a great learning experience for me so far as I had to go through lot of challenges with the size of the data and full test dataset not being available to us. In the process of working through this competition, I got a chance to learn about GPUs and data processing and modelling libraries which use GPUs to process data faster. \n\nThrough some of the great notebooks written so far, I came across RAPIDS framework created by NVIDIA which allows for GPU based acceleration for analytics workflows. It also allowed me to use some of my weekly quota of GPUs for the first time on Kaggle. Please read through this kernel and provide your feedback in the comments. \n\n"},{"metadata":{},"cell_type":"markdown","source":"# Table of Contents - \n* [Overview of RAPIDS](#rapids)\n* [Importing raw datasets](#import)\n* [Exploratory Analysis](#eda)\n* [Feature Engineering ](#feature)\n* [Model Development & Feature Importance](#model)\n* [Predictions](#preds)\n\n\n\n### Questions answered in the EDA \n* How does the average correct answer rate vary across questions in the training dataset?\n* How many questions does an average user answer over his learning journey? What is the total span of data in number of days for an average user?\n* What is the distribution of number of appearances of a question on training dataset?\n* Does a student's likelihood to answer a question correctly improves over time?\n* What is the average size of a question bundle?\n* Does watching lectures improves the chances of users answering a question correctly?\n* What is the average time spent by users in reading the explanation for a prior question bundle?\n* How does the correct answer rate vary across different tags? Does the count of total tags in a question have an impact on the correct answer rate for a question?\n"},{"metadata":{},"cell_type":"markdown","source":"# Overview of RAPIDS \n\n[RAPIDS](https://rapids.ai/) is a suite of open-source software libraries and APIs for executing data science pipelines entirely on GPUs. Some of the advantages are - \n* **Faster Execution Time** - RAPIDS leverages NVIDIA CUDA® under the hood to accelerate your workflows by running the entire data science training pipeline on GPUs. This reduces training time and the frequency of model deployment from days to minutes.\n* **Use the Same Tools** - By hiding the complexities of working with the GPU and even the behind-the-scenes communication protocols within the data center architecture, RAPIDS creates a simple way to get data science done.\n\nI initially started working on this problem in Pandas, but my code was quite slow and I wasn't able to handle the datasets properly. Therefore I switched to RAPIDS and since then the overall runtime of my notebook has come down to ~2 mins(excluding load time for cudf packages)\n\n\nTo use RAPIDS in our notebook, we need to add RAPIDS package files to our notebook and then load the package. "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport plotly.express as px\nfrom plotly.subplots import make_subplots\nfrom matplotlib import pyplot\nimport plotly.graph_objects as go\nimport gc\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%time\n\nimport sys\n!cp ../input/rapids/rapids.0.15.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz > /dev/null\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7/site-packages\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + sys.path \n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import cudf #Cudf is a library for processing dataframes\nimport cupy # CuPy is an open-source array library accelerated with NVIDIA CUDA.\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Importing raw datasets \n\n\nIn this competition we've been given access to student data log from a learning app. Student activity includes watching lectures on different topics and then answering questions. Each question & lecture has some metadata associated to it, providing more details around the learning journey of a student. \n\nThere are three primary datasets that are available in this competition. Below is the description provided to us - \n\n**train.csv**\n\n* ```row_id```: (int64) ID code for the row.\n\n* ```timestamp```: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n\n* ```user_id```: (int32) ID code for the user.\n\n* ```content_id```: (int16) ID code for the user interaction\n\n* ```content_type_id```: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n* ```task_container_id```: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n* ```user_answer```: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\n* ```answered_correctly```: (int8) if the user responded correctly. Read -1 as null, for lectures.\n\n* ```prior_question_elapsed_time```: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n\n* ```prior_question_had_explanation```: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.\n\n**questions.csv**: metadata for the questions posed to users.\n\n* ```question_id```: foreign key for the train/test content_id column, when the content type is question (0).\n\n* ```bundle_id```: code for which questions are served together.\n\n* ```correct_answer```: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n* ```part```: the relevant section of the TOEIC test.\n\n* ```tags```: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together.\n\n**lectures.csv**: metadata for the lectures watched by users as they progress in their education.\n\n* ```lecture_id```: foreign key for the train/test content_id column, when the content type is lecture (1).\n\n* ```part```: top level category code for the lecture.\n\n* ```tag```: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n* ```type_of```: brief description of the core purpose of the lecture"},{"metadata":{},"cell_type":"markdown","source":"### Reading data using cudf\n\nSimilar to pandas, we use the ```read_csv``` function to in cudf package to read the input data provided to us. \n\nThis is the description for cudf on RAPIDS website - Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data. cuDF provides a pandas-like API, so users can use it to easily accelerate their workflows without going into the details of CUDA programming."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Importing raw datasets. We will only be importing 20M rows in training data. \ntrain=cudf.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',low_memory=False,nrows=2*(10**7), dtype={'row_id': 'int64', 'timestamp': 'int64', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n 'task_container_id': 'int16', 'user_answer': 'int8', 'answered_correctly': 'int8', 'prior_question_elapsed_time': 'float32', \n 'prior_question_had_explanation': 'boolean',\n }) \n\n\nlectures=cudf.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\nquestions=cudf.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some observations from having a peek at the data above - \n* timestamp values are represented in millisecond units and start at zero for every user\n* As expected, prior_question_had_explanation and prior_question_elapsed time values are zero for first question answered by a user\n* **content_id is the foreign key to be used to join train datasets with questions and lectures datasets**. Content id can represent a lecture or question\n* Lectures can be used for concepts or solving questions. Every lecture has a tag associated to it. \n* Single question can have multiple tags associated to it. Tags will help us understand topic associated with a lecture. However we don't have a mapping file to give us tag_id to tag_name mapping\n\nAlso taking a look at the number of records in each of the datasets here - \n* **The train dataset has more than 100 million records**, we've only imported around 20 million here\n* **There are about 418 different kinds of lectures**\n* **There are total of 13523 questions in our data**\n\nSize of the train data is much higher than our memory capacity in Kaggle notebook, therefore you will see me walking a tightrope with managing RAM resources here and frequently deleting datasets after their usage to free up memory. "},{"metadata":{},"cell_type":"markdown","source":" # Exploratory Analysis \n\nHaving taken a quick glance at the data, let's dive deep into datasets provided to us and try and gain a deeper understanding of the data. This will help us understand distribution of individual variables as well as relationships between independent and dependent variable. It will also allow us to create meaningful features which can then help us in creating a good model. We will be looking to answer the questions listed out at start of the notebook."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Count of records by distinct value of answered_correctly column\ntrain.groupby('answered_correctly').agg({'row_id': ['count']}).reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Count of records by content type\ntrain.groupby('content_type_id').agg({'row_id': ['count']}).reset_index()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* ```answered_correctly``` has three distinct values. Value -1 represents records where user didn't answer a question and watched a lecture instead\n* In our sample of training data imported, majority of data is for questions answered. Less than 2% of training data is for lectures watched by students. \n\n## Question and User Level summaries"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Merge the train and questions dataset\ntrain_questions=train.merge(questions,left_on='content_id',right_on='question_id',how='inner')\ntrain_questions_gp=train_questions.loc[train_questions['content_type_id']==0].groupby('question_id').agg({'answered_correctly':'sum','row_id':'count'}).reset_index()\ntrain_questions_gp['percent_correct']=train_questions_gp['answered_correctly']/train_questions_gp['row_id']\ntrain_questions_lim=train_questions_gp.loc[train_questions_gp['row_id']>=10]\ntrain_questions_lim_pandas=train_questions_lim.to_pandas()\n\nfig=px.histogram(train_questions_lim_pandas,x='percent_correct',title='Distribution of percentage correct answers across questions',template=\"simple_white\")\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\n\nquestion_cnt=train_questions.groupby('content_id').agg({'row_id':'count'}).reset_index()\nquestion_cnt_pd=cudf.DataFrame.to_pandas(question_cnt)\nfig1=px.histogram(question_cnt_pd,x='row_id',title='Count of Question Occurences',template=\"simple_white\")\nfig1.update_traces(marker=dict(color='cadetblue'))\nfig1.show()\n\ndel question_cnt_pd\n#del question_cnt\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#Group by training data on user_ids\ntrain_user_gp=train_questions.groupby('user_id').agg({'answered_correctly':'mean','row_id':'count'}).reset_index()\ntrain_user_gp.columns=['user_id','user_answer_mean','user_answer_count']\ntrain_user_gp_pandas=train_user_gp.to_pandas()\ntrain_user_gp_pandas=train_user_gp_pandas.loc[train_user_gp_pandas['user_answer_count']<1000]\nfig=px.histogram(train_user_gp_pandas,x='user_answer_count',title='Distribution of total Questions Answered by users',template=\"simple_white\")\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\n\n\ntrain_user_gp=train_questions.groupby('user_id').agg({'answered_correctly':'mean','row_id':'count'}).reset_index()\ntrain_user_gp_pandas=train_user_gp.to_pandas()\nfig1=px.violin(train_user_gp_pandas,y='answered_correctly',box=True,title='Distribution of Percentage of Answered Correctly across users',template=\"simple_white\")\nfig1.update_traces(marker=dict(color='cadetblue'))\nfig1.show()\n\nfig2=px.scatter(train_user_gp_pandas,x='row_id',y='answered_correctly')\nfig2.update_layout(title='Distribution of total Questions Answered by user',xaxis_title='Total Questions Answered',yaxis_title='Percentage Answered Correctly',template=\"simple_white\")\nfig2.update_traces(marker=dict(color='cadetblue'))\nfig2.show()\ndel train_user_gp_pandas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Few things are visible from the charts above - \n* The mean of correct answer rate across questions is quite high, **the distribution of percentage correct across questions has a peak between 0.7-0.8**\n* Majority of the questions in the dataset appear less than 5k times, however there are small number of questions which have very high unique occurences, sometimes higher than 10k occurences\n* When we look at distribution of percentage correct answers across users, we see that it has a skew towards lower values with median at 0.56. Q1 is at 0.42 and Q3 is at 0.66\n* **The total user scores are lower for users in initial stages, as the number of questions answered increases the average scores increase and show lower variance and vary between 0.6 - 0.8**\n* After 4k questions, there is no increase in average user scores\n* There is no linear relationship between total questions answered and average user scores.\n"},{"metadata":{},"cell_type":"markdown","source":"### Lectures"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Identifying users who have attended a lecture. As users who have attended a lecture have answered_correctly as -1 we can identify such users by looking for min value in answered_correctly as -1\ntrain_user_gp_new=train_questions.groupby('user_id').agg({'answered_correctly':'sum','row_id':'count'}).reset_index()\n\ntrain_user_gp_lec=train_questions.groupby('user_id').agg({'answered_correctly':'min'}).reset_index()\nlec_users=train_user_gp_lec.loc[train_user_gp_lec['answered_correctly']==-1]\nlec_users['lec_flag']=lec_users['answered_correctly']*-1\ndel lec_users['answered_correctly']\n#lec_users\n\ntrain_user_gp_lec=train_user_gp_new.merge(lec_users,left_on='user_id',right_on='user_id',how='left')\ntrain_user_gp_lec['lec_flag'].fillna(0,inplace=True)\ntrain_user_gp_lec_gp=train_user_gp_lec.groupby('lec_flag').agg({'answered_correctly':'sum','row_id':'mean','row_id':'sum'}).reset_index()\ntrain_user_gp_lec_gp['percent_correct']=train_user_gp_lec_gp['answered_correctly']/train_user_gp_lec_gp['row_id']\ntrain_user_gp_lec_gp_pd=train_user_gp_lec_gp.to_pandas()\n\nfig=px.bar(train_user_gp_lec_gp_pd,x='lec_flag',y='percent_correct',title='Student Performance variation with Lecture Viewing',template='simple_white')\nfig.update_xaxes(type='category')\nfig.update_traces(marker=dict(color='cadetblue'))\n\nfig.show()\n\n\n\n#Creating Flags for number of lectures & whether a student has watched a lecture or not\nlec_watchers=train_questions.loc[train_questions['answered_correctly']==-1]\n\ntrain_user_gp_lec=lec_watchers.groupby('user_id').agg({'answered_correctly':'sum'}).reset_index()\ntrain_user_gp_lec\ntrain_user_gp_lec['num_lec']=train_user_gp_lec['answered_correctly']*-1\ntrain_user_gp_lec['lec_flag']=1\ntrain_user_gp_lec\ndel train_user_gp_lec['answered_correctly']\n\ndel train_user_gp_lec_gp_pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_lec=train.merge(train_user_gp_lec,left_on='user_id',right_on='user_id',how='left')\ntrain_lec['num_lec'].fillna(0,inplace=True)\ntrain_lec['lec_flag'].fillna(0,inplace=True)\n\ntrain_lec_gp=train_lec.groupby('num_lec').agg({'answered_correctly':'mean'}).reset_index()\ndel train_lec\n\ntrain_lec_gp_pandas=train_lec_gp.to_pandas()\ntrain_lec_gp_pandas.answered_correctly.rolling(5).mean()\nfig=px.line(train_lec_gp_pandas,x='num_lec',y='answered_correctly',title='Variation in Correct Answer rate with number of lectures watched',template='simple_white')\n\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\ndel train_lec_gp_pandas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see above students who have seen lectures generally do better than students who haven't seen any lectures. When we dive deeper into the number of lectures vs correct answer rate we see that for number of lectures higher than 100 the correct answer rate is much higher than 70%. However such instances could be outliers and could be skewed by a few users due to low data volume. \n\n## Relationship between Task Containers, Bundle & Part and dependent variable"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Count the number of questions per task container\ntask_container=train.groupby('task_container_id').agg({'row_id': ['count']}).reset_index()\ntask_container.columns=['task_container_id','count']\ntask_container_pandas=task_container.to_pandas()\n\nfig = px.histogram(task_container_pandas, x=\"count\",nbins=50,labels={'x':'task_container_id', 'y':'count'},template='simple_white')\nfig.update_layout(title='Distribution of size of Task Containers',yaxis_title='Count of Task Containers')\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\ndel task_container_pandas","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# train_sample=train.sample(frac=0.25)\n# train_prior=train_sample.loc[~train_sample['prior_question_elapsed_time'].isna(),:]\n# #train_prior\n\n# train_prior_pandas=train_prior.to_pandas()\n# px.violin(train_prior_pandas,y='prior_question_elapsed_time',x='prior_question_had_explanation',box=True,title='Variation of Prior Question Elapsed time with Prior Question Explanation')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# # Deleting data frames to clear up some space\n# del train_prior\n# del train_prior_pandas\n# del train_sample","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_bundle_gp=train_questions.loc[train_questions['content_type_id']==0].groupby('bundle_id').agg({'question_id':'count'}).reset_index()\ntrain_bundle_gp_lim=train_bundle_gp.loc[train_bundle_gp['question_id']<=5000]\n\ntrain_bundle_gp_lim=train_bundle_gp_lim.to_pandas()\n\nfig=px.histogram(train_bundle_gp_lim,x='question_id',title='Distribution of question counts across bundle_ids',template='simple_white')\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\ndel train_bundle_gp_lim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_bundle_ans_gp=train_questions.loc[train_questions['content_type_id']==0].groupby('bundle_id').agg({'answered_correctly':'mean'}).reset_index()\n\ntrain_bundle_ans_gp_pandas=train_bundle_ans_gp.to_pandas()\nfig=px.histogram(train_bundle_ans_gp_pandas,x='answered_correctly',title='Distribution of Correct Answers by Question Bundle',template='simple_white')\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\ndel train_bundle_ans_gp_pandas\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Create flags for bundles based on percentage answered correct for each bundle in the training dataset\n#This will be used as a feature in the dataset\ntrain_bundle_ans_gp['bundle_flag']=0\n\ndef bundle_flag_fun(x):\n if (x>=0.75) :\n return 4\n if (x>=0.5) :\n return 3\n elif (x>=0.25) :\n return 2\n else :\n return 1\n \ntrain_bundle_ans_gp['bundle_flag']=train_bundle_ans_gp[\"answered_correctly\"].applymap(bundle_flag_fun)\n\ndel train_bundle_ans_gp['answered_correctly']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_gp=train_questions.groupby('part').agg({'answered_correctly':'mean','row_id':'count'}).reset_index()\ntrain_part_lim=train_part_gp.loc[train_part_gp['row_id']>=10]\n\ntrain_part_lim_pandas=train_part_lim.to_pandas()\n\nfig = make_subplots(specs=[[{\"secondary_y\": True}]])\n\n# Add traces\nfig.add_trace(\n go.Bar(x=train_part_lim_pandas['part'], y=train_part_lim_pandas['answered_correctly'], name=\"Percentage Answered Correctly\",marker=dict(color='cadetblue')),\n secondary_y=False,\n)\n\nfig.add_trace(\n go.Scatter(x=train_part_lim_pandas['part'], y=train_part_lim_pandas['row_id'], name=\"Count of Questions\",marker=dict(color='grey')),\n secondary_y=True,\n)\n\n\n# Add figure title\nfig.update_layout(\n title_text=\"Question Part - Percentage Answered Correctly & Count of Questions\"\n)\n#fig.update_traces(marker=dict(color='cadetblue'))\n\n# Set x-axis title\n#fig.update_xaxes(title_text=\"Number of Tags in a Question\")\nfig.show()\ndel train_part_gp['row_id']\ntrain_part_gp.columns=['part','part_percent_correct']\ndel train_part_lim_pandas\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next we look at three attributes associated to a question - Task Container, Bundle_id and part - \n* **Majority of task containers have a very small size**\n* Distribution of total question count across bundles has an uneven distribution with multiple peaks, biggest peaks are seen in the bins of 0-99,200-299 and 900-999\n* Distribution of correct answer across bundles looks very similar to distribution of correct answers across individual questions\n* We see that in our training dataset, **40% of questions belong to Part 5**. **Part 7 is the least common of all. The correct_answer rate across all parts ranges from 0.6-0.7**\n\n## Prior Question Elapsed Time and Prior Question had Explanation"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_priorquestions_gp=train_questions.groupby('prior_question_had_explanation').agg({'answered_correctly':'mean','row_id':'count'}).reset_index()\ntrain_priorquestions_gp_pandas=train_priorquestions_gp.to_pandas()\n\n\nfig = make_subplots(rows=1, cols=2)\n\n# Add traces\nfig.add_trace(\n go.Bar(x=train_priorquestions_gp_pandas['prior_question_had_explanation'], y=train_priorquestions_gp_pandas['answered_correctly'], name=\"Percentage Answered Correctly\",marker=dict(color='LightSkyBlue')),\n row=1, col=1\n)\n\nfig.add_trace(\n go.Bar(x=train_priorquestions_gp_pandas['prior_question_had_explanation'], y=train_priorquestions_gp_pandas['row_id'], name=\"Total Question Count\",marker=dict(color='lightseagreen')),\n row=1, col=2\n)\nfig.update_layout(title='Question Count & Percentage Correct Answers by Prior Question Had Explanation')\nfig.update_layout(template='simple_white')\n\nfig.show()\n\ndel train_priorquestions_gp_pandas","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"\ntrain_priorquestions_gp=train_questions.groupby('prior_question_had_explanation').agg({'prior_question_elapsed_time':'mean'}).reset_index()\ntrain_priorquestions_gp_pandas=train_priorquestions_gp.to_pandas()\n\nfig1=px.bar(train_priorquestions_gp_pandas,x='prior_question_had_explanation',y='prior_question_elapsed_time',title='Mean of prior_question_elapsed_time by prior_question_had_explanation',template='simple_white')\nfig1.update_traces(marker=dict(color='cadetblue'))\nfig1.show()\ndel train_priorquestions_gp_pandas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the data description above, ```prior_question_had_explanation``` is defined as - Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between.\n\nWe notice that whenever users saw explanation to prior question bundle, their correct answer rate improved substantially. Also for more than 90% of the question bundles, users referred to the explanation after answering the questions.\n\nAlso the ```prior_question_elapsed_time``` i.e. time spent by users on question bundle does not have any impact on likelihood of users reading explanation of bundle after answering it.\n\n## Question Tags\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_questions_nolec=train_questions_nolec.reset_index(drop=True)\nk=train_questions['tags']\ntrain_questions['tag_count']=k.str.count(' ')+1\ntrain_questions['tag1']= train_questions['tags'].str.split(' ')[0]\n\ntrain_tag_gp=train_questions.groupby('tag_count').agg({'row_id':'count','answered_correctly':'mean'}).reset_index()\ndel k\n\n#train_tag_gp\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create figure with secondary y-axis\ntrain_tag_gp_pandas=train_tag_gp.to_pandas()\nfig = make_subplots(specs=[[{\"secondary_y\": True}]])\n\n# Add traces\nfig.add_trace(\n go.Scatter(x=train_tag_gp_pandas['tag_count'], y=train_tag_gp_pandas['answered_correctly'], name=\"Percentage Answered Correctly\",marker=dict(color='LightSkyBlue')),\n secondary_y=False,\n)\n\nfig.add_trace(\n go.Scatter(x=train_tag_gp_pandas['tag_count'], y=train_tag_gp_pandas['row_id'], name=\"Count of Answers\",marker=dict(color='grey')),\n secondary_y=True,\n)\n\n# Add figure title\nfig.update_layout(\n title_text=\"Distribution of Correct Answers & Total Questions with number of Tags\"\n)\n\n# Set x-axis title\nfig.update_xaxes(title_text=\"Number of Tags in a Question\")\n\n# Set y-axes titles\nfig.update_yaxes(title_text=\"Percentage Questions Answered Correctly\", secondary_y=False)\nfig.update_yaxes(title_text=\"Total Questions Answered\", secondary_y=True)\nfig.show()\ndel train_tag_gp_pandas","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_gp=train_questions.loc[train_questions['content_type_id']==0].groupby('tag1').agg({'answered_correctly':'mean','row_id':'count'}).reset_index()\ntag_gp_pandas=tag_gp.to_pandas()\n\nfig = px.scatter(tag_gp_pandas, x=\"tag1\", y=\"answered_correctly\",\n size=\"row_id\",\n hover_name=\"tag1\", size_max=60)\nfig.update_layout(title='Correct Answer Rate & Question counts across values of Tag1',template='simple_white')\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\ndel tag_gp_pandas","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I also took a look at different tag values associated to different questions. In the above two visualizations -\n1. I extracted the count of tags associated to a question and then plotted correct answer rate next to it to understand if more number of tags have an impact on successful answer rate on a question\n2. We also extracted first value of a tag across each question and compared the total count of questions along with correct answer rate across each of the tags\n\nHere are some findings - \n* A very high percentage of questions only had a single tag associated to them\n* **Multi tag questions have a higher correct answer rate than single tag questions**\n* For each of the count of tags, number of questions associated with a tag count seems to be inversely proportional to correct answer rate for a tag count\n* **Most tags have a correct answer rate in the range of 0.4-0.8\n\n\nAs students when we practice over our learning material we tend to get better and better with time. We would check for the same in our dataset, we would look at whether a user's correct answer rate improve with two factors - **total number of questions answered and total timestamp**\n\n### User Activity"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Create flags for different tag values\ntag_gp['tag_flag']=0\n\ndef tag_flag_fun(x):\n if (x>=0.75) :\n return 4\n if (x>=0.5) :\n return 3\n elif (x>=0.25) :\n return 2\n else :\n return 1\n \ntag_gp['tag_flag']=tag_gp[\"answered_correctly\"].applymap(tag_flag_fun)\ndel tag_gp['answered_correctly']\ndel tag_gp['row_id']\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_user_gp_sum=train_questions.groupby('user_id').agg({'answered_correctly':'sum','row_id':'count'}).reset_index()\ntrain_user_gp_sum.columns=['user_id','user_answer_total','user_answer_count']\n\ntrain_user_gp_sum['user_answer_count_flag']=0\n\ndef user_answer_count_flag_fun(x):\n if (x<=100) :\n return 0\n if (x<=500) :\n return 1\n elif (x<=1000) :\n return 2\n elif (x<=2000) :\n return 3\n elif (x<=3000) :\n return 4\n else :\n return 5\n \n \ntrain_user_gp_sum['user_answer_count_flag']=train_user_gp_sum[\"user_answer_count\"].applymap(user_answer_count_flag_fun)\ngp=train_user_gp_sum.groupby('user_answer_count_flag').agg({'user_answer_total':'sum','user_answer_count':'sum'}).reset_index()\ngp['percent_answer_correct']=gp['user_answer_total']/gp['user_answer_count']\ngp=gp.to_pandas()\n\nfig = make_subplots(specs=[[{\"secondary_y\": True}]])\n\n# Add traces\nfig.add_trace(\n go.Bar(x=gp['user_answer_count_flag'], y=gp['user_answer_count'], name=\"Count of Questions\",marker=dict(color='LightSkyBlue')),\n secondary_y=False,\n)\n\nfig.add_trace(\n go.Scatter(x=gp['user_answer_count_flag'], y=gp['percent_answer_correct'], name=\"Percentage Answered Correctly\",marker=dict(color='grey')),\n secondary_y=True,\n)\n\n\n# Add figure title\nfig.update_layout(\n title_text=\"Improvement in Correct Answer Rate with total questions answered\"\n)\n\n# Set x-axis title\nfig.update_xaxes(title_text=\"Total Question Count Flag\")\nfig.show()\n\ndel gp","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#Converting timestamps to hours and days for readability\n\ntrain['timestamp_hours']=train['timestamp']/(3600*1000)\ntrain['timestamp_days']=train['timestamp_hours']/(24)\n\ntimestamp_series=cupy.asnumpy(train['timestamp_days'])\n\ntimestamp_series=np.random.choice(timestamp_series,size=10000)\ntimestamp_series\nfig = go.Figure(data=[go.Histogram(x=timestamp_series)])\nfig.update_layout(title='Total Activity across number of days',xaxis_title='number of days',yaxis_title='total questions answered')\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\ndel timestamp_series","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_sample=train.sample(n=10000, replace=True, random_state=1)\ntrain_sample['timestamp_flag']=0\n#train_sample=cudf.DataFrame.to_pandas(train_sample)\ntrain_sample=train_sample.reset_index(drop=True)\n\n#train_sample.loc[i,'timestamp_days']\ndef timestamp_flag(x):\n if (x<=50) :\n return 0\n elif (x<=100) :\n return 1\n elif (x<=365) :\n return 2\n else :\n return 3\n \n \ntrain_sample['timestamp_flag']=train_sample[\"timestamp_days\"].applymap(timestamp_flag)\n\ntrain_sp_gp=train_sample.groupby('timestamp_flag').agg({'answered_correctly':'mean'}).reset_index()\ntrain_sp_gp_pandas=train_sp_gp.to_pandas()\n\nfig=px.bar(train_sp_gp_pandas,x='timestamp_flag',y='answered_correctly',title='Variation in Percentage of Correct Answer Rate with time')\nfig.update_xaxes(type='category')\nfig.update_traces(marker=dict(color='cadetblue'))\nfig.show()\ndel train_sample\ndel train_sp_gp_pandas\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the cells above we created flag variables for total timestamp at which questions are answered as well as the total number of questions answered by users. We then plotted user's activity across these flag variables. \n\nWe see also see that majority of users have a learning journey of less than 100 days, a smaller portion of users are active after that period on the app. \n\nWe can also see that a **user's correct answer rate improves massively as they tend to answer more questions, however this is not reflected to a great extent in user's correct answer rate with total time**. As we can see that there is a slight improvement in correct answer rate with time but its not a substantial difference. This maybe due to the fact that users learning journey is not often proportional to time, there may be long gaps between periods when users are active. Therefore simple increase in timestamp does not justify increase in learning rate for users. \n\n\n## EDA Summary - Answers to Questions\nWe've been able to answer some of the questions above through the EDA analysis performed by us. Here are some of the key findings from the analysis -\n* There are more than 100M records available in the training dataset. \n* The mean of correct answer rate across questions is quite high, the distribution of percentage correct across questions has a peak between 0.7-0.8. The distribution for correct answer rate is quite spread out - indicating a big difference in correct answer rate across questions\n* Instances where students read the explanation for previous question bundle improves the correct answer rate of current bundle. \n* High percentage of questions belong to part 5\n* Students get better at answering questions through more and more practice. \n* High share of students have learning journey of less than 100 days\n* Students who watch lectures are more likely to answer questions correctly through their learning journey\n* Questions with higher number of tags have a higher likelihood of being answered correctly\n\nWe will now proceed to feature engineering and model development where we will be using this information obtained above.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Removing outliers for model development\nexclusion_questions=question_cnt.loc[question_cnt['row_id']<20,'content_id']\nexclusion_user=train_user_gp_sum.loc[train_user_gp_sum['user_answer_count']<20,'user_id']\nexclusion_user=exclusion_user.unique()\n\ndef outlier_analysis(df):\n df=df.loc[~df.user_id.isin(exclusion_user)]\n df=df.loc[~df.content_id.isin(exclusion_questions)]\n return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering\n\nAfter getting a good understanding of the data, our next step is to generate some features on the training data and create a model using them to generate predictions. Lot of our flag variables created above can directly be translated into features. In additon to that, here are some of the features i've come up with - \n\n**User Based Features** - \n* Percentage Answered Correctly\n* Total Questions Answered\n* Std Deviation Answered Correctly\n* Variance Answered Correctly\n* User min value answered_correctly\n* User max value answered_correctly\n\n**Question Based Features** - \n* Percentage Correct\n* Total count\n* Question answered_correctly Variance\n* Question answered_correctly Standard Deviation\n\n**Other Features** - \n* Bundle Flag\n* Tag Flag \n* Part Flag\n* Tag Count\n* Prior Question Had Explanation\n* Prior Question Elapsed Time\n* Lecture Flag (Based on whether a student has seen a lecture)\n* Number of lectures seen by a student\n\nThe feature generation process is generally iterative, we go with intuition and come up with an initial set of features and then remove the ones which don't improve model performance and then come up with newer features that might improve prediction accuracy. "},{"metadata":{"trusted":true},"cell_type":"code","source":"del train\ndel train_questions\n#del exclusion_questions\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true,"_kg_hide-input":true},"cell_type":"code","source":"import sys\ndef sizeof_fmt(num, suffix='B'):\n ''' by Fred Cirera, https://stackoverflow.com/a/1094933/1870254, modified'''\n for unit in ['','Ki','Mi','Gi','Ti','Pi','Ei','Zi']:\n if abs(num) < 1024.0:\n return \"%3.1f %s%s\" % (num, unit, suffix)\n num /= 1024.0\n return \"%.1f %s%s\" % (num, 'Yi', suffix)\n\nfor name, size in sorted(((name, sys.getsizeof(value)) for name, value in locals().items()),\n key= lambda x: -x[1])[:20]:\n print(\"{:>30}: {:>8}\".format(name, sizeof_fmt(size)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"del _5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Function for Feature Engineering\ndef feature_engg(df):\n\n #Timestamp\n # df['timestamp'].fillna(0,inplace=True)\n # df['timestamp_hours']=df['timestamp']/(3600*1000)\n # df['timestamp_days']=df['timestamp_hours']/(24)\n\n #MultiTag\n questions=cudf.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\n questions=questions[['question_id','part','tags','bundle_id']]\n \n df = df.merge(questions, how = 'left', left_on = 'content_id',right_on = 'question_id')\n del df['question_id']\n\n df['tags'].fillna('',inplace=True)\n df['tag_count']=df['tags'].str.count(' ')+1\n df['tag1']=df[\"tags\"].str.split(\" \", n = 1, expand = True)[[0]]\n df['tag_count'].fillna(1,inplace=True)\n del df['tags']\n\n df=df.merge(tag_gp,on='tag1',how='left')\n df['tag_flag'].fillna(2,inplace=True)\n\n #NA Values of Prior Question had Explanation with False\n df['prior_question_had_explanation'].fillna(False, inplace=True)\n \n #Prior Question Elapsed Time MVT\n df['prior_question_elapsed_time'].fillna(25302, inplace=True)\n \n df=df.merge(train_questions_gp,left_on='content_id',right_on='question_id',how='left')\n df['q_correct'].fillna(0.69,inplace=True)\n df['q_count'].fillna(74,inplace=True)\n df['q_var'].fillna(0.18,inplace=True)\n df['q_std'].fillna(0.41,inplace=True)\n \n #Part correct answer rate\n df['part'].fillna(5,inplace=True)\n df=df.merge(train_part_gp,on=['part'],how='left')\n df['part_percent_correct'].fillna(0.69,inplace=True)\n \n #Bundle Flagging\n df=df.merge(train_bundle_ans_gp,on=['bundle_id'],how='left')\n df['bundle_flag'].fillna(2,inplace=True)\n \n df=df.merge(user_flags,on=['user_id'],how='left')\n df['user_answer_mean'].fillna(0.53,inplace=True)\n df['user_answer_count'].fillna(286,inplace=True)\n df['user_min'].fillna(0,inplace=True)\n df['user_max'].fillna(1,inplace=True)\n df['user_std'].fillna(0.47,inplace=True)\n df['user_var'].fillna(0.22,inplace=True)\n \n #Lec Flag\n df=df.merge(train_user_gp_lec,left_on='user_id',right_on='user_id',how='left')\n df['num_lec'].fillna(0,inplace=True)\n df['lec_flag'].fillna(0,inplace=True)\n \n\n return df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_user_gp_lec=train_user_gp_lec.astype('int32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train=cudf.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',low_memory=False,nrows=4*(10**7),dtype={'row_id': 'int64', 'timestamp': 'int64', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n 'task_container_id': 'int16', 'user_answer': 'int8', 'answered_correctly': 'int8', 'prior_question_elapsed_time': 'float32', \n 'prior_question_had_explanation': 'boolean',\n }) \ndel train['timestamp'] \ndel train['row_id']\ndel train['task_container_id']\n\ntrain = train[train.content_type_id == False]\n\ndel train['content_type_id']\ngc.collect()\n\n#train.drop(['timestamp','content_type_id'], axis=1, inplace=True)\n\n#Creating User level flags -\nuser_flags=train.groupby('user_id').agg({'answered_correctly':['mean','count','min','max','std','var']}).reset_index()\nuser_flags.columns=['user_id','user_answer_mean','user_answer_count','user_min','user_max','user_std','user_var']\nuser_flags\n\n#Creating content level flags\ntrain_questions=train.merge(questions,left_on='content_id',right_on='question_id',how='left')\n\ntrain_questions_gp=train_questions.groupby('question_id').agg({'answered_correctly':['mean','count','var','std']}).reset_index()\ntrain_questions_gp.columns=['question_id','q_correct','q_count','q_var','q_std']\ndel train_questions\ngc.collect()\n\nprint(train.shape[0])\ntrain=outlier_analysis(train)\ntrain=feature_engg(train)\nprint(train.shape[0])\n\n#del train\ngc.collect()\n\ntrain=train[['answered_correctly',\n 'prior_question_elapsed_time', 'prior_question_had_explanation',\n 'tag_count','tag_flag',\n 'q_correct', 'q_count', 'q_var', 'q_std', 'part_percent_correct',\n 'bundle_flag', 'user_answer_mean', 'user_answer_count', 'user_min',\n 'user_max', 'user_std', 'user_var','num_lec','lec_flag']]\ntrain['prior_question_had_explanation'] =train['prior_question_had_explanation'].astype(int)\ntrain=train.astype('float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"train.isnull().any()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Random Forest & XGBoost Models"},{"metadata":{},"cell_type":"markdown","source":"This competition has been unique for me because we don't have visibility into the entire test dataset at once. We have to call an API and fetch batches of test data, iterate over test data in loops and make predictions for each of the batches. Because we don't have visibility into test data any errors in the code are visible only once we make a submission, which sometimes takes hours to run. Therefore running multiple iterations and experimenting has been a challenge in this competition.\n\n### Evaluation Metric : ROC\nFound this great [link](https://towardsdatascience.com/understanding-the-roc-and-auc-curves-a05b68550b69) here to read about the evaluation metric for this competition - ROC\n\n#### What is ROC - AUC Curve?\nAUC - ROC curve is a performance measurement for classification problem at various thresholds settings. ROC is a probability curve and AUC represents degree or measure of separability. It tells how much model is capable of distinguishing between classes. Higher the AUC, better the model is at predicting 0s as 0s and 1s as 1s. The ROC curve is plotted with True Positive Rate(TPR) against the (False Positive Rate)FPR where TPR is on y-axis and FPR is on the x-axis.\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":" ######cuML###### \n #Random Forest\nfrom sklearn.model_selection import train_test_split #Splitting data for model training\nfrom sklearn.metrics import roc_auc_score\nimport cupy as cp\n\n#from sklearn.ensemble import RandomForestClassifier\n\n#train_input=train\n#del model_data_input['answered_correctly']\n\n#X=model_data.loc[:, model_data.columns != 'answered_correctly']#Selecting feature variables\nX=train.loc[:,[ 'prior_question_elapsed_time',\n 'prior_question_had_explanation',\n 'tag_count','tag_flag', 'q_correct', 'q_count', 'q_var',\n 'q_std', 'part_percent_correct', 'bundle_flag', 'user_answer_mean',\n 'user_answer_count', 'user_min', 'user_max', 'user_std', 'user_var','num_lec','lec_flag']]#Selecting feature variables\n\nY=train['answered_correctly'] #Selecting the output columns\nfeature_list=X.columns\n\nX_train,X_test,Y_train,Y_test=train_test_split(X, Y,test_size=0.3,random_state=1)\nimport cuml\nfrom cuml import RandomForestClassifier as cuRF \n # cuml Random Forest params \ncu_rf_params = {'n_estimators': 100, \n 'max_depth': 8, \n 'max_features':7,\n 'min_rows_per_node':5,\n 'rows_sample':0.7\n } \n \ncu_rf = cuRF(**cu_rf_params) \ncu_rf.fit(X_train, Y_train) \npred=cu_rf.predict(X_test)\n#acc_score = cu_rf.score(pred, Y_test)\ncu_score = cuml.metrics.accuracy_score( Y_test, pred )\n#sk_score = accuracy_score( asnumpy( Y_test ), asnumpy( pred ) )\n\n#print( \" cuml accuracy: \", cu_score )\n\nprint('ROC Score for Random Forest')\nY_test=cp.asnumpy(Y_test)\npred=cp.asnumpy(pred)\nroc=roc_auc_score(Y_test, pred)\nprint(\" - ROC: {:.5}\".format(roc))\n#print()\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"import sys\ndef sizeof_fmt(num, suffix='B'):\n ''' by Fred Cirera, https://stackoverflow.com/a/1094933/1870254, modified'''\n for unit in ['','Ki','Mi','Gi','Ti','Pi','Ei','Zi']:\n if abs(num) < 1024.0:\n return \"%3.1f %s%s\" % (num, unit, suffix)\n num /= 1024.0\n return \"%.1f %s%s\" % (num, 'Yi', suffix)\n\nfor name, size in sorted(((name, sys.getsizeof(value)) for name, value in locals().items()),\n key= lambda x: -x[1])[:10]:\n print(\"{:>30}: {:>8}\".format(name, sizeof_fmt(size)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"del pred\ndel train\ndel Y\ndel lec_watchers\ndel X\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import xgboost\nfrom xgboost import XGBClassifier\nfrom xgboost import plot_importance\n\n#Parameters for XGBoost \nparams1 = {\n 'max_depth' : 7,\n # 'max_leaves' : 2**4,\n 'alpha':0.1, \n # 'lambda' : 0.2,\n 'min_child_weight ':2,\n 'subsample':0.7,\n 'tree_method' : 'gpu_hist',\n 'learning_rate': 0.1, #default = 0.3,\n 'colsample_bytree':0.7,\n 'eval_metric':'auc', \n 'objective' : 'binary:logistic',\n 'grow_policy' : 'lossguide',\n 'n_estimators':200\n}\n\ntrain_matrix = xgboost.DMatrix(data = X_train, label = Y_train)\ntest_matrix=xgboost.DMatrix(data = X_test)\nxgb = xgboost.train(params1, dtrain = train_matrix)\n\npredicts = xgb.predict(test_matrix)\nroc = roc_auc_score(Y_test.astype('int32'), predicts)\nprint('ROC for XGBoost model')\nprint(roc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"xgb.get_score(importance_type='gain')\nplot_importance(xgb)\npyplot.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Features related to mean & deviance of responses from users and question are most important for the model. Some of the flag features around tags, bundles and lectures don't do so well. Its likely that information from these features has already been captured in the features above. \n\nWe see that our XGBoost model performs much better than Random Forest on validation dataset. Therefore we would be using the XGBoost model for final predictions. "},{"metadata":{},"cell_type":"markdown","source":"# Final Predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"model=xgb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Importing the competition package\nimport riiideducation\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()\n#Read the test data in batches\nfor test_df, sample_prediction_df in iter_test:\n test_df = cudf.from_pandas(test_df)\n test_df=test_df.loc[test_df['content_type_id'] == 0]\n test_df = feature_engg(test_df)\n\n row_ids=test_df['row_id']\n \n test_df=test_df[['prior_question_elapsed_time',\n 'prior_question_had_explanation',\n 'tag_count','tag_flag', 'q_correct', 'q_count', 'q_var',\n 'q_std', 'part_percent_correct', 'bundle_flag', 'user_answer_mean',\n 'user_answer_count', 'user_min', 'user_max', 'user_std', 'user_var','num_lec','lec_flag']]\n \n test_df['prior_question_had_explanation'] =test_df['prior_question_had_explanation'].astype('float32')\n test_df=test_df.astype('float32')\n \n test_matrix=xgboost.DMatrix(data = test_df)\n# pred=model.predict_proba(test_df)[:,1]\n pred=model.predict(test_matrix)\n# rf_pred = model.predict_proba(test_df)[:,1]\n# lg_pred = lgb_model.predict(test_df)\n # row_ids=test_df['row_id']\n test_df['row_id']=row_ids\n test_df['answered_correctly'] =pred\n test_df = test_df.to_pandas()\n #print(test_df)\n env.predict(test_df[['row_id', 'answered_correctly']])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### References \nI got to learn about Rapids from this great notebook . Please take a look if you want to learn more about Rapids. - https://www.kaggle.com/andradaolteanu/answer-correctness-rapids-crazy-fast"},{"metadata":{},"cell_type":"markdown","source":"### Next Steps - \n* Using higher volume of training data\n* Continue working on Feature Engineering\n* Experiment with model ensembling and Hyperparameter Optimization for XGBoost\n* Create a running lookup table of user performance as we keep processing batches of user data on test dataset and use it to make predictions"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}