{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Riiid Challenge\n\n# Introduction\n\n![](https://media-exp1.licdn.com/dms/image/C511BAQFdNFjlEdfIVw/company-background_10000/0?e=2159024400&v=beta&t=1nlr0BJH9o8ihnnW7a5Gee0v3IM08hgUrLPNoUp0Ko8)\n\n### About Riiid:\nRiiid is global leading AI Tutor solution provider delivering creative disruption to the education market through its cutting-edge AI technology.\n\n### Problem:\n\nIn 2018, 260 million children weren't attending school. At the same time, more than half of these young students didn't meet minimum reading and math standards. Education was already in a tough place when COVID-19 forced most countries to temporarily close schools. This further delayed learning opportunities and intellectual development. The equity gaps in every country could grow wider. We need to re-think the current education system in terms of attendance, engagement, and individualized attention.\n\n### Solution:\n\nRiiid Labs, an AI solutions provider delivering creative disruption to the education market, empowers global education players to rethink traditional ways of learning leveraging AI. With a strong belief in equal opportunity in education, Riiid launched an AI tutor based on deep-learning algorithms in 2017 that attracted more than one million South Korean students.\n\n### Translating the Problem Into Machine Learning\n\n### Goals:\n\n- In this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions.\n\n#### *Therefore this is a classification problem.*\n\n### Limitations:\n\n- We've given pretty big database and here on Kaggle we have some computational limits like memory.\n- Loading and working on this data takes quite long time for bigger batches. So we should work in given run-time limits.\n- Traditional validation methods might be inefficient by the form of data given.\n\n### Performance Metric:\n\n- Submissions are evaluated on area under the ROC curve between the predicted probability and the observed target."},{"metadata":{},"cell_type":"markdown","source":"## Getting Things Ready\n\nHere we install and load our libraries and set some default settings for future use."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"!pip install ../input/python-datatable/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# importing basic libraries for eda\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n#\n\nimport datatable as dt\n\n#\n\nimport gc\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# styling settings\n\nplt.rcParams['figure.figsize'] = [18,10]\nplt.style.use('ggplot')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loading the Data\n\nThe data we given is huge, it's not easy to load them all into our RAM without getting out of memory. We need to set dtypes for each column to decrease memory usage. By default they are 32/64 for numeric columns, but can choose dtypes manually based on their max amount in the column. These dtype selections based on:\n\n>int8 / uint8 : consumes 1 byte of memory, range between -128/127 or 0/255,\n\n>bool : consumes 1 byte, true or false,\n\n>float16 / int16 / uint16: consumes 2 bytes of memory, range between -32768 and 32767 or 0/65535,\n\n>float32 / int32 / uint32 : consumes 4 bytes of memory, range between -2147483648 and 2147483647,\n\n>float64 / int64 / uint64: consumes 8 bytes of memory.\n\n[Source](https://medium.com/@vincentteyssier/optimizing-the-size-of-a-pandas-dataframe-for-low-memory-environment-5f07db3d72e)\n\nWe're also going to use **datatable** for faster loading. You can find deeper explanation [here](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets).\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# dict for dtypes\n\ndata_types = {\n    'row_id': 'int32',\n    'timestamp': 'int64',\n    'user_id': 'int64',\n    'content_id': 'int16',\n    'content_type_id': 'int8',\n    'task_container_id': 'int16',\n    'user_answer': 'int8',\n    'answered_correctly': 'int8',\n    'prior_question_elapsed_time': 'float32',\n    'prior_question_had_explanation': 'boolean'\n}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# loading data with datatable and converting it to pandas df.\n\ntrain_df = dt.fread('../input/riiid-test-answer-prediction/train.csv').to_pandas()\n\n# randomly selecting portion of the data for faster processing.\n\ntrain_df = train_df.sample(len(train_df)//5,random_state=42)\n\n\n# setting dtypes for each column\n\nfor column, d_type in data_types.items():\n    train_df[column] = train_df[column].astype(d_type) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading other data files\n\nquestions_df = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\nlectures_df = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv')\ntest_df = pd.read_csv('../input/riiid-test-answer-prediction/example_test.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Using handpicked dtypes for each column decreased their memory usage, which is a good sign..."},{"metadata":{"trusted":true},"cell_type":"code","source":"# dtypes and memory usage\n\ntrain_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Timestamp \n\n#### \"timestamp\": The time in milliseconds between this user interaction and the first event completion from that user.\n\n#### Here we can see earlier parts of the timeline is more active than longer sessions, which is expected. Also we can notice that there are some users spend quite a long time around!"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# plotting timestamp related graphs\n\nfig, ax = plt.subplots(ncols=2, nrows=1, figsize=(32,14))\n\nsns.distplot(train_df.timestamp, kde=False,hist_kws={\n                 'rwidth': 0.85,\n                 'edgecolor': 'black',\n                 'alpha': 0.8}, bins=100, ax=ax[0])\n\nax[0].set_xlabel('Time in Miliseconds')\nax[0].set_ylabel('Count')\nax[0].set_title('Timestamp Distribution', weight='bold')\n\n\nsns.distplot(train_df.groupby('user_id').agg({'timestamp': 'mean'}), kde=False, hist_kws={\n                 'rwidth': 0.85,\n                 'edgecolor': 'black',\n                 'alpha': 0.8}, bins=50,ax=ax[1])\n\nax[1].set_xlabel('Time in Miliseconds*')\nax[1].set_ylabel('Count')\nax[1].set_title('Mean Timestamp Per User Distribution', weight='bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.sort_values(\"timestamp\").head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df_serious_students.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[\"lecture\"] = (train_df[\"answered_correctly\"]==-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df.lecture]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting the performance of students based on counts of questions and lectures that they have seen\nmin_lectures = [i*5 for i in range(11)]\nfig, ax = plt.subplots(ncols=1, nrows=10, figsize=(32,140))\ntrain_df['counts']=1\ntrain_df[\"lecture\"] = (train_df[\"answered_correctly\"]==-1)\ntrain_df['lecture'] = train_df['lecture'].astype(np.int8)\ntrain_lectures = train_df.groupby('user_id').agg({'lecture' : 'sum'})\ntrain_df_serious_students = train_df[train_df.answered_correctly >= 0].groupby('user_id').agg({'counts': 'sum','answered_correctly': 'mean'})\ntrain_df_serious_students = train_df_serious_students.join(train_lectures, on = 'user_id', how ='left')\n\nfor i in range(10) : \n    plot_df = train_df_serious_students[(train_df_serious_students.counts >= 10) & (train_df_serious_students.lecture >=min_lectures[i])  & (train_df_serious_students.lecture <min_lectures[i+1])]\n    sns.distplot(plot_df['answered_correctly'], kde=True, hist_kws={\n                     'rwidth': 0.85,\n                     'edgecolor': 'black',\n                     'alpha': 0.8}, bins=50,ax=ax[i])\n    ax[i].set_xlabel('Average accuracy')\n    ax[i].set_ylabel('Counts')\n    ax[i].set_title('Average accuracy for students that have watched {} lectures \\n and answered at least 10 questions'.format(min_lectures[i]), weight='bold')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"47774654/(1000*3600)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Users and Contents\n\n##### Here we have unique ID for each user , if we count all the interections made by user we can see there are some pretty active users, out of ~300k unique users we see top 25 of them almost made more than 3k interactions.\n\n#### Content is pretty similar with user ID's. Here we can see most popular contents, seems  content #6116 is really favourite one  followed by #6173 and #4120 around 400k interactions.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting user and content related graphs\n\nfig, ax = plt.subplots(ncols=2, nrows=1, figsize=(32,14))\n\nsns.countplot(y='user_id', data=train_df, order=train_df.user_id.value_counts().index[:25], palette='autumn',ax = ax[0])\nax[0].set_title('Top 25 Active Users', weight='bold')\n\n\nsns.countplot(y='content_id', data=train_df, order=train_df.content_id.value_counts().index[:25], palette='autumn',ax = ax[1])\nax[1].set_title('Top 25 Content', weight='bold')\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Content Types\n\n#### \"content_type_id\": 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n#### Here we can see that 98% of our samples are questions and ~2% are lectures."},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting content types\n\ng=sns.countplot(train_df.content_type_id, palette='autumn')\n\n# adding percentages\n\ntotal = float(len(train_df['content_type_id']))\n\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2.,\n            height + 2,\n            '{:1.2f}%'.format((height / total) * 100),\n            ha='center')\n\nplt.ylabel('Count*10^7')    \nplt.title('Content Types - 0: Question, 1: Lecture', weight='bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Task Containers\n\n#### \"task_container_id\": Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n#### We can observe that tasks with smaller ID's are much more common than bigger numbers, meanwhile most popular one is #14"},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting task containers\n\nfig, ax = plt.subplots(ncols=2, nrows=1, figsize=(32,14))\n\nsns.distplot(train_df.task_container_id, kde=False,hist_kws={\n                 'rwidth': 0.85,\n                 'edgecolor': 'black',\n                 'alpha': 0.8}, ax=ax[0])\n\n\nax[0].set_ylabel('Frequency')\nax[0].set_title('Task Container ID Distribution', weight='bold')\n\n\nsns.countplot(y='task_container_id', data=train_df, order=train_df.task_container_id.value_counts().index[:25], palette='autumn', ax=ax[1])\nax[1].set_title('Top 25 Tasks', weight='bold')\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# User Answers\n\n#### \"user_answer\": The user's answer to the question, if any. Read -1 as null, for lectures.\n\n#### Here we can see answer option #2 is less common than rest of the three answers. Seems like users/instructors doesn't like option #2 that much :)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting user answers\n\ng=sns.countplot(train_df.user_answer, hue=train_df.answered_correctly, palette='autumn', order=train_df.user_answer.value_counts().index)\n\n# adding percentages\n\ntotal = float(len(train_df['user_answer']))\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2.,\n            height + 2,\n            '{:1.2f}%'.format((height / total) * 100),\n            ha='center')\n\nplt.title('False/Correct per User Answer  (-1 for Lectures)', weight='bold')\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prior Questions\n\n\n#### prior_question_elapsed_time:The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n\n#### prior_question_had_explanation: Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# plotting prior questions related stuff\n\nfig, ax = plt.subplots(ncols=2, nrows=1, figsize=(32,14))\n\n\nsns.distplot(train_df.prior_question_elapsed_time.dropna(), kde=False, hist_kws={\n                 'rwidth': 0.85,\n                 'edgecolor': 'black',\n                 'alpha': 0.8}, ax=ax[0])\n\nax[0].set_ylabel('Count')\nax[0].set_title('Prior Question Elapsed Time Distribution', weight='bold')\n\n\ng=sns.countplot(train_df.prior_question_had_explanation.dropna(), palette='autumn', ax=ax[1])\n\n# adding percentages to plot\n\ntotal = float(len(train_df.prior_question_had_explanation.dropna()))\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2.,\n            height + 2,\n            '{:1.2f}%'.format((height / total) * 100),\n            ha='center')\n\nax[1].set_title('Prior Question Elapsed had Explanation?', weight='bold')\n    \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Dropping lectures from the dataframe\n\ntrain_df = train_df.loc[train_df['answered_correctly'] != -1].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Questions\n\n#### Here we add extra data we have given. So we might find some insightful hints..."},{"metadata":{"trusted":true},"cell_type":"code","source":"# merging question data with train data\n\ntrain_df = pd.merge(train_df,questions_df[['question_id','part']], how='left', left_on='content_id', right_on='question_id').sort_values('row_id')\ntrain_df['part'] = train_df['part'].astype('int8')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Question Parts\n\n#### part: The relevant section of the TOEIC test.\n\n#### We've merged question parts based on their specific question ID's. These giving us relevant section of the TOEIC test explained here:\n\n![](https://i.imgur.com/2wqNAJ1.png)\n![](https://i.imgur.com/4B3AQyL.png)\n\n### Here we observe that Part 5 questions are the most popular one, where you have to complete the sentences by given four options."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"g=sns.countplot(train_df.part, hue=train_df.answered_correctly, palette='autumn')\n\n# adding percentages to plot\n\ntotal = float(len(train_df.part))\nfor p in g.patches:\n    height = p.get_height()\n    g.text(p.get_x() + p.get_width() / 2.,\n            height + 2,\n            '{:1.2f}%'.format((height / total) * 100),\n            ha='center')\n\nplt.title('False/Correct per Question Part', weight='bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# grouping by user id and getting mean, sum, counts\n\nusr_ans = train_df.groupby('user_id').agg({ 'answered_correctly': ['mean','sum', 'count']})\nusr_ans.columns = ['avg_correct_answer','num_of_correct', 'total_answers']\n\n# changing dtype for reducing memory (default = 64)\n\nusr_ans['num_of_correct'] = usr_ans['num_of_correct'].astype('int16')\nusr_ans['total_answers'] = usr_ans['total_answers'].astype('int16')\n\n\ntrain_df = pd.merge(train_df, usr_ans, how='left', on = 'user_id')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Correct Answer Accuracy\n\n#### We've filtered users who have answered more than 100 questions and sorted them based on their correct answer ratio, we got some verry accurate users!"},{"metadata":{"trusted":true},"cell_type":"code","source":"#  plotting top 25 Accurate Users\n\nsns.barplot(x='avg_correct_answer',y='user_id', orient='h', data=usr_ans[usr_ans['total_answers']>100].sort_values('avg_correct_answer', ascending=False).reset_index().iloc[:25],\n            palette='autumn', order=usr_ans[usr_ans['total_answers']>100].sort_values('avg_correct_answer', ascending=False).reset_index().user_id.iloc[:25])\nplt.title('Top 25 Accurate Users', weight='bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Answer Accuracy vs. Total Questions Answered\n\n#### Here we can observe increasing correct answer ratio by total number of questions answered by the user. Practice makes perfect!"},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting total answers vs avg accuracy\n\nsns.regplot(data=usr_ans[usr_ans['total_answers']> 100], y='avg_correct_answer', x='total_answers', ci=False, scatter_kws={'alpha':0.5}, line_kws={\"color\": \"orange\"})\nplt.axhline(train_df.avg_correct_answer.mean(), color='k', linestyle='dashed', linewidth=3)\nplt.axvline(train_df.total_answers.mean(), color='k', linestyle='dashed', linewidth=3)\n\nmin_ylim, max_ylim = plt.ylim()\nplt.text(train_df.total_answers.mean()+25, max_ylim*0.20, 'Average Questions Solved {:.2f}'.format(train_df.total_answers.mean()))\nplt.text(train_df.total_answers.mean()+2400, max_ylim*0.6, 'Average Correct Answer: {:.2f}'.format(train_df.avg_correct_answer.mean()))\n\nplt.title('Average Correct Answer Ratio vs. Total Questions Answered per User', weight='bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Answer Accuracy - Time Relations\n\n#### Here we took maximum time spent by the user and filtered ones out if they have less than one hour. We got slight increase on user accuracy by increasing time spent but it seems insignificant."},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting answer accuracy vs time\n\ntotal_time = train_df.groupby('user_id')[\"timestamp\"].max()\ntotal_time = pd.merge(total_time.reset_index(), usr_ans.reset_index(), how='left', on = 'user_id')\n\nsns.regplot(data=total_time[total_time['timestamp']> 3.6e+6], y='avg_correct_answer', x='timestamp', ci=False, scatter_kws={'alpha':0.5}, line_kws={\"color\": \"orange\"})\n\nplt.axvline(total_time.timestamp.mean(), color='k', linestyle='dashed', linewidth=3)\nplt.text(total_time.timestamp.mean()+total_time.timestamp.mean()*0.1, max_ylim*0.03, 'Average Time {:.2f}'.format(total_time.timestamp.mean()))\n\nplt.title('Average Correct Answer Ratio vs. Time Spent', weight='bold')\nplt.show()\n\n# deleting some variables to save memory:\n\ndel total_time\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Answer Accuracy - Content Relations\n\n#### When we take the answer accuracy by the content we can see that more popular/generic contents have lower correct answer ratio. Meanwhile less popular/specific questions have higher correct ratio."},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating new feature based on content id\n\ncnt_ans = train_df.groupby('content_id').agg({ 'answered_correctly': ['mean','sum', 'count']})\ncnt_ans.columns = ['avg_correct_answer_c','num_of_correct_c', 'total_answers_c']\n\n# changing dtype for reducing memory (default = 64)\n\ncnt_ans['num_of_correct_c'] = cnt_ans['num_of_correct_c'].astype('int32')\ncnt_ans['total_answers_c'] = cnt_ans['total_answers_c'].astype('int32')\n\ntrain_df = pd.merge(train_df, cnt_ans, how='left', on = 'content_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plotting contents vs. answer accuracies\n\nsns.regplot(data=cnt_ans[cnt_ans['total_answers_c']> 100], y='avg_correct_answer_c', x='total_answers_c', ci=False, scatter_kws={'alpha':0.5}, line_kws={\"color\": \"orange\"})\n\n# plotting mean lines\n\nplt.axhline(train_df.avg_correct_answer_c.mean(), color='k', linestyle='dashed', linewidth=3)\nplt.axvline(train_df.total_answers_c.mean(), color='k', linestyle='dashed', linewidth=3)\n\n\nmin_ylim, max_ylim = plt.ylim()\nplt.text(35000, max_ylim*0.65, 'Average Correct Answer: {:.2f}'.format(train_df.avg_correct_answer_c.mean()))\nplt.text(5500, max_ylim*0.10, 'Average Questions Solved per Content: {:.2f}'.format(train_df.total_answers_c.mean()))\n\nplt.title('Average Correct Answer Ratio vs. Total Questions Answered per Content', weight='bold')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Baseline Model\n\n#### Here's the part we gonna do some simple baseline model for benchmarking our future models. We start by filling some missing values in our data and then split it as X and y for modelling. We need to predict if the given user answered specific question correct or failed it."},{"metadata":{"trusted":true},"cell_type":"code","source":"# creating x variable for training\n\nX = train_df.copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# filling na values\n\nX['prior_question_elapsed_time'].fillna(0,  inplace=True)\nX['prior_question_had_explanation'] = X['prior_question_had_explanation'].fillna(value = False).astype(bool)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train_df\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# setting x and y for training\n\nX=X.sort_values(['user_id'])\ny = X[[\"answered_correctly\"]]\nX = X.drop([\"answered_correctly\"], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\nlb_make = LabelEncoder()\nX[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(X[\"prior_question_had_explanation\"])\nX['prior_question_had_explanation_enc'] = X['prior_question_had_explanation_enc'].astype('int8')\nX.head()\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# selecting feautres for training\n\nX = X[['avg_correct_answer','num_of_correct','total_answers', 'avg_correct_answer_c', 'prior_question_elapsed_time','prior_question_had_explanation_enc','part']] ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Here we have basic, default parameter Lightgbm classifier for training."},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading model(s) for testing\n\nfrom sklearn.model_selection import StratifiedKFold, cross_validate\nimport lightgbm as lgb\n\nlight = lgb.LGBMClassifier(\n)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Validation\n\n#### We will stratify and shuffle our target (which I'm not sure since there is some sort of timeline but we'll do this way for the baseline) and validate it using 3 folds."},{"metadata":{"trusted":true},"cell_type":"code","source":"# setting stratified kfold for validation\n\nkf = StratifiedKFold(3, shuffle=True, random_state=42)\nclassifiers = [light]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def model_check(X, y, classifiers, cv):\n    \n    ''' A function for testing multiple classifiers and return several metrics. '''\n    \n    model_table = pd.DataFrame()\n\n    row_index = 0\n    for cls in classifiers:\n\n        MLA_name = cls.__class__.__name__\n        model_table.loc[row_index, 'Model Name'] = MLA_name\n        \n        cv_results = cross_validate(\n            cls,\n            X,\n            y,\n            cv=cv,\n            scoring=('accuracy','f1','roc_auc'),\n            return_train_score=True,\n            n_jobs=-1\n        )\n        model_table.loc[row_index, 'Train Roc/AUC Mean'] = cv_results[\n            'train_roc_auc'].mean()\n        model_table.loc[row_index, 'Test Roc/AUC Mean'] = cv_results[\n            'test_roc_auc'].mean()\n        model_table.loc[row_index, 'Test Roc/AUC Std'] = cv_results['test_roc_auc'].std()\n\n        model_table.loc[row_index, 'Time'] = cv_results['fit_time'].mean()\n\n        row_index += 1        \n\n    model_table.sort_values(by=['Test Roc/AUC Mean'],\n                            ascending=False,\n                            inplace=True)\n\n    return model_table","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Results\n\n#### Alright. Looks like our baseline did OK! "},{"metadata":{"trusted":true},"cell_type":"code","source":"# displaying default model results\n\nraw_models = model_check(X, y, classifiers, kf)\ndisplay(raw_models)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Prediction\n\n#### Here we use competition specific prediction environment, we'll predict the test samples and submit them by using 'riiideducation' package."},{"metadata":{"trusted":true},"cell_type":"code","source":"# importing riid package\n\nimport riiideducation\n\nenv = riiideducation.make_env()\n\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# fitting the model\n\nlight.fit(X, y)\n\n# resetting indexes for pred merging\n\ncnt_ans=cnt_ans.reset_index()\nusr_ans=usr_ans.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    test_df = test_df.merge(usr_ans, how = 'left', on = 'user_id')\n    test_df = test_df.merge(cnt_ans, how = 'left', on = 'content_id')\n    test_df = pd.merge_ordered(test_df,questions_df[['question_id','part']], how='left', left_on='content_id', right_on='question_id', fill_method='ffill')\n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\n    test_df['prior_question_elapsed_time'].fillna(0,  inplace=True)\n    test_df['avg_correct_answer'].fillna(0.5, inplace=True)\n    test_df['avg_correct_answer_c'].fillna(0.5, inplace=True)\n    test_df.fillna(value = -1, inplace = True)\n    test_df[\"prior_question_had_explanation_enc\"] = lb_make.fit_transform(test_df[\"prior_question_had_explanation\"])\n    \n    \n    y_pred = light.predict_proba(test_df[['avg_correct_answer','num_of_correct','total_answers', 'avg_correct_answer_c', 'prior_question_elapsed_time','prior_question_had_explanation_enc','part']])[:,1]\n    test_df['answered_correctly'] = y_pred\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Last Words\n\n### Well, that's it then... I hope you enjoyed while reading this notebook and find it useful for you! Please don't forget to comment/upvote if you liked it!\n\n### This is early version of the notebook and I'll try to update it when I have time. Thanks for reading again, happy coding!\n\n## Work in Progress"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}