{"cells":[{"metadata":{},"cell_type":"markdown","source":"<img src=\"https://mma.prnewswire.com/media/1200045/Riiid_Labs.jpg?p=publish&w=950\" width=\"800\" height=\"400\">\n\n## <center>Riiid! Answer Correctness Prediction</center>\n### <center>🧠Track knowledge states of 1M+ students in the wild🧠</center>"},{"metadata":{},"cell_type":"markdown","source":"# Table of contents <a id='0.1'></a>\n\n* [Introduction](#1)\n* [Import Packages](#2)\n* [Utility](#3)\n* [Data Overview](#4)\n    * [Train Data](#4.1)\n    * [Questions Data](#4.2)\n    * [Lectures Data](#4.3)\n    * [Test Data](#4.4)\n* [Individual Features](#5)\n    * [Continous Feature Distribution](#5.1)\n        * [Train Data Feature Distribution](#5.1.1)\n        * [Lectures Data Feature Distribution](#5.1.2)\n        * [Test Data Feature Distribution](#5.1.3)\n    * [Categorical Feature Distribution](#5.2)\n        * [Train Data](#5.2.1)\n        * [Questions Data](#5.2.2)\n        * [Lectures Data](#5.2.3)\n* [Multiple Features](#6)\n    * [Train Data Features](#6.1)\n    * [Questions MetaData Features](#6.2)\n    * [Lectures MetaData Features](#6.3)\n    * [Feature Correlation](#6.4)\n* [Reference](#7)"},{"metadata":{},"cell_type":"markdown","source":"# 1. <a id='1'>Introduction📔</a>\n[Table of contents](#0.1)\n\nWelcome to this new competition hosted by [Riiid! Labs](https://www.riiid.co/en/main), leader in AI based education. Here are some of the [products](https://www.riiid.co/en/product) provided by [Riiid! Labs](https://www.riiid.co/en/main).\n\nIn this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions. You will pair your machine learning skills using Riiid’s EdNet data.\n\n## About Competition Data\n\nThe data is in tabular format. We have data regarding student's historic performance, the performance of other students on the same question, metadata about the question itself. \n\n**This is a time-series code competition, you will receive test set data and make predictions with Kaggle's time-series API. Please be sure to review the Time-series API Details section closely**.\n\nWe are provided with following **csv** files - \n\n* train.csv - Training features.\n* questions.csv - Metadata for the questions posed to users.\n* lectures.csv - Metadata for the lectures watched by users as they progress in their education.\n\nPlease check this starter kernels here to get more information.\n* [Competition API Detailed Introduction](https://www.kaggle.com/sohier/competition-api-detailed-introduction)\n* [Quick Sample Submission](https://www.kaggle.com/sohier/quick-sample-submission/)\n\n## What we are prediciting?\n\n You will predict whether students are able to answer their next questions correctly.\n \n## Evaluation Metric: Area Under ROC Curve\n\nSubmissions are evaluated on area under the ROC curve between the predicted probability and the observed target.\n\n<img src=\"https://i.ytimg.com/vi/J9l8J1MeCbY/hqdefault.jpg\" width=\"400\" height=\"400\" align='left'>"},{"metadata":{},"cell_type":"markdown","source":"# 2. <a id='2'>Import Packages📚</a>\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!pip install ../input/python-datatable/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl > /dev/null","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# import packages\nimport os, gc\nimport warnings\nimport numpy as np\nimport pandas as pd\nimport datatable as dt\n\n# visualization\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# riiideducation module\nimport riiideducation\n\n%matplotlib inline\nwarnings.filterwarnings('ignore')\n\n# directory\nprint('Competition Data/Files')\nos.listdir('../input/riiid-test-answer-prediction')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. <a id='3'>Utility</a>\n[Table of contents](#0.1)\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def countplot(column, plot_type='multiple', gridstyle='whitegrid', gs=None,\n              palette='Accent', xlab=None, ylab=None, title=None, fontsize=12):\n    \n    '''\n    Make countplots\n    -----------------\n    \n    Arguments:\n    column -- column with categorical values\n    plot_type -- multiple grid ('multiple/single')\n    gridstyle -- seaborn gridstyle\n    gs -- gridspec (if using subplots)\n    palette -- color palette\n    xlab -- x-axis label\n    ylab -- y-axis label\n    title -- plot title\n    fontsize -- fontsize\n    \n    Returns:\n    sns.countplot()\n    '''\n    if plot_type=='multiple':\n        with sns.axes_style(gridstyle):\n            ax = f.add_subplot(gs)\n            aa = sns.countplot(column, palette=palette)\n            for p in ax.patches:\n                height = p.get_height()\n                aa.text(p.get_x()+p.get_width()/2.,\n                        height,\n                        '{:1.2f}%'.format(height/len(column)*100),\n                        ha=\"center\", fontsize=fontsize)\n            plt.xlabel(xlab,fontsize=fontsize)\n            plt.ylabel(ylab,fontsize=fontsize)\n            plt.title(title)\n            \n    elif plot_type=='single':\n        with sns.axes_style(\"whitegrid\"):\n            aa = sns.countplot(column, palette=palette)\n            for p in aa.patches:\n                height = p.get_height()\n                aa.text(p.get_x()+p.get_width()/2.,\n                        height + 3,\n                        '{:1.2f}%'.format(height/len(column)*100),\n                        ha=\"center\", fontsize=fontsize)\n            plt.xlabel(xlab,fontsize=fontsize)\n            plt.ylabel(ylab,fontsize=fontsize)\n            plt.title(title)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. <a id='4'>Data Overview🔍</a>\n[Table of contents](#0.1)\n\nIn this section we will develop some intuition about the [competition data](https://www.kaggle.com/c/riiid-test-answer-prediction/data). The train.csv is huge around 5.45 GB we will use python **datatable** package to load this huge tabular data in our notebook. The **datatable** is adopted from the [R data.table](https://cran.r-project.org/web/packages/data.table/vignettes/datatable-intro.html) package for faster readability of tabular data. Thanks to [Rohan Rao](https://www.kaggle.com/rohanrao) for this [notebook](https://www.kaggle.com/rohanrao/riiid-with-blazing-fast-rid/notebook)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# root directory\nROOT = '../input/riiid-test-answer-prediction/'\n\n# files\ntrain = dt.fread(\"../input/riiid-test-answer-prediction/train.csv\").to_pandas()\n\ntrain = train.astype({\n    'row_id': 'int32',\n    'timestamp': 'int64',\n    'user_id': 'int64',\n    'content_id': 'int16',\n    'content_type_id': 'int8',\n    'task_container_id': 'int16',\n    'user_answer': 'int8',\n    'answered_correctly': 'int8',\n    'prior_question_elapsed_time': 'float32',\n    'prior_question_had_explanation': 'boolean'\n})\n\nquestions = pd.read_csv(f'{ROOT}questions.csv')\nlectures = pd.read_csv(f'{ROOT}lectures.csv')\nexample_test = pd.read_csv(f'{ROOT}example_test.csv')\nexample_sample_submission = pd.read_csv(f'{ROOT}example_sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.1 <a id='4.1'>Train Data</a>\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'We have {train.shape[0]} rows and {train.shape[1]} features in train.csv.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* row_id - ID code for the row.\n\n* timestamp - the time between this user interaction and the first event from that user.\n\n* user_id - ID code for the user.\n\n* content_id - ID code for the user interaction\n\n* content_type_id - 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n* task_container_id - Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id. Monotonically increasing for each user.\n\n* user_answer - the user's answer to the question, if any. Read -1 as null, for lectures.\n\n* answered_correctly - if the user responded correctly. Read -1 as null, for lectures.\n\n* prior_question_elapsed_time - How long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle.\n\n* prior_question_had_explanation - Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.\n\nLet's get somemore info about the training data."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Missing Values"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'Missing values in train.csv in each columns:\\n{train.isnull().sum()}')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'We have total of {train.isnull().values.sum()} missing values in train data.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n* We have **2744044** in total. **2351538** in column **prior_question_elapsed_time** and **392506** in **prior_question_had_explanation**."},{"metadata":{},"cell_type":"markdown","source":"### Unique Values"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Unique Values in each column of train.csv')\nprint('##########################################')\nfor col in train:\n    print(f'{col}: {train[col].nunique()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* We have **3,93,656 unique users**.\n* We have 10,000 unique batches of questions. \n* We have 4 categorical features **content_type_id, user_answer, answered_correctly, prior_question_had_explanation**."},{"metadata":{},"cell_type":"markdown","source":"## 4.2 <a id='4.2'>Questions Data (metadata)</a>\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"questions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'We have {questions.shape[0]} rows and {questions.shape[1]} features in questions.csv.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* question_id - foreign key for the train/test content_id column, when the content type is question (0).\n\n* bundle_id - code for which questions are served together.\n\n* correct_answer - the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n* part - top level category code for the question.\n\n* tags - one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together.\n\n### Missing Values in questions.csv"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'Missing values in questions.csv in each columns:\\n{questions.isnull().sum()}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'We have total of {questions.isnull().values.sum()} missing values in train data.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Unique Values"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Unique Values in each column of questions.csv')\nprint('##########################################')\nfor col in questions:\n    print(f'{col}: {questions[col].nunique()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## 4.3 <a id='4.3'>Lectures Data (metadata)</a>\n[Table of contents](#0.1)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"lectures.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'We have {lectures.shape[0]} rows and {lectures.shape[1]} features in lectures.csv.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* lecture_id - foreign key for the train/test content_id column, when the content type is lecture (1).\n\n* part - top level category code for the lecture.\n\n* tag - one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n* type_of - brief description of the core purpose of the lecture\n\n### Missing Values in lectures.csv"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'Missing values in lectures.csv in each columns:\\n{lectures.isnull().sum()}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(f'We have total of {lectures.isnull().values.sum()} missing values in lectures data.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Unique Values"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Unique Values in each column of lectures.csv')\nprint('##########################################')\nfor col in lectures:\n    print(f'{col}: {lectures[col].nunique()}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n## 4.4 <a id='4.4'>Test Data</a>\n[Table of contents](#0.1)\n\nIn this competition we have to predict which questions each student can answer correctly. You will loop through a series of batches of questions. Once you make that prediction, you can move on to the next batch.\n\nWe need to use **riiideducation** python module to work with our test data. For more detailed explanation please visit [here](https://www.kaggle.com/sohier/competition-api-detailed-introduction)."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# You can only call make_env() once, so don't lose it!\nenv = riiideducation.make_env()\n\n# You can only iterate through a result from `env.iter_test()` once\n# so be careful not to lose it once you start iterating.\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"count = 0\nfor (test_df, sample_prediction_df) in iter_test:\n    test_df['answered_correctly'] = 0.5\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n    count += len(test_df)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(f'We have {count} observations in total and {test_df.shape[1]} features in test.csv.')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"test_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\nWe can see test dataframe is same as train.csv except we have two new columns. \n\n* prior_group_responses - provides all of the user_answer entries for previous group in a string representation of a list in the first row of the group. All other rows in each group are null. If you are using Python, you will likely want to call eval on the non-null rows. Some rows may be null, or empty lists.\n\n* prior_group_answers_correct - provides all the answered_correctly field for previous group, with the same format and caveats as prior_group_responses. Some rows may be null, or empty lists."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"test_df.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. <a id='5'>Individual Features📊</a>\n[Table of contents](#0.1)\n\nNow we will visualize our data using information available to us in each of the csv file. "},{"metadata":{},"cell_type":"markdown","source":"## 5.1 <a id='5.1'>Continous Feature Distribution</a>\n### 5.1.1 <a id='5.1.1'>Train Data Feature Distribution</a>"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    train['timestamp'].hist(bins = 50,color='orange')\n    plt.title(\"Timestamp Distribution\")\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    train['user_id'].hist(bins = 50,color='red')\n    plt.title(\"User Id Distribution\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* **timestamp** represents user interaction upto first event completion. We see the graph is rightly skew. \n* **User Id** is unique id assigned to each user."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"mean = train['content_id'].mean()\nmedian = train['content_id'].median()\nmode = train['content_id'].mode()[0]\n\nmean_2 = train['task_container_id'].mean()\nmedian_2 = train['task_container_id'].median()\nmode_2 = train['task_container_id'].mode()[0]\n\nprint(f'Content Id (Mean): {mean}')\nprint(f'Content Id (Median): {median}')\nprint(f'Content Id (Mode): {mode}\\n')\nprint('######################################\\n')\nprint(f'Task Container Id (Mean): {mean_2}')\nprint(f'Task Container Id (Median): {median_2}')\nprint(f'Task Container Id (Mode): {mode_2}')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    sns.distplot(train['content_id'], color='green')\n    ax.axvline(int(mean), color='r', linestyle='--')\n    ax.axvline(int(median), color='y', linestyle='-')\n    ax.axvline(mode, color='b', linestyle='-')\n    plt.legend({'Mean':mean,'Median':median,'Mode':mode})\n    plt.title(\"Content Id Distribution\")\n    \nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    sns.distplot(train['task_container_id'], color='yellow')\n    ax.axvline(int(mean_2), color='r', linestyle='--')\n    ax.axvline(int(median_2), color='g', linestyle='-')\n    ax.axvline(mode_2, color='b', linestyle='-')\n    plt.legend({'Mean':mean_2,'Median':median_2,'Mode':mode_2})\n    plt.title(\"Task Container Id Distribution\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"mean_3= train['prior_question_elapsed_time'].mean()\nmedian_3 = train['prior_question_elapsed_time'].median()\nmode_3 = train['prior_question_elapsed_time'].mode()[0]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\n\nwith sns.axes_style(\"whitegrid\"):\n    sns.distplot(train['prior_question_elapsed_time'], color='olive')\n    plt.axvline(int(mean_3), color='c', linestyle='--')\n    plt.axvline(int(median_3), color='m', linestyle='-')\n    plt.axvline(mode_3, color='k', linestyle='-')\n    plt.legend({'Mean':mean_3,'Median':median_3,'Mode':mode_3})\n    plt.title(\"Prior Question Elapsed Time Distribution\")\n    \nprint(f'Prior Question Elapsed Time (Mean): {mean_3}')\nprint(f'Prior Question Elapsed Time (Median): {median_3}')\nprint(f'Prior Question Elapsed Time (Mode): {mode_3}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.1.2 <a id='5.1.2'>Lectures Data Feature Distribution</a>\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"mean_4 = lectures['tag'].mean()\nmedian_4 = lectures['tag'].median()\nmode_4 = lectures['tag'].mode()[0]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\n\nwith sns.axes_style(\"whitegrid\"):\n    sns.distplot(lectures['tag'], color='coral', bins=20)\n    plt.axvline(int(mean_4), color='r', linestyle='--')\n    plt.axvline(int(median_4), color='g', linestyle='-')\n    plt.axvline(mode_4, color='b', linestyle='-')\n    plt.legend({'Mean':mean_4,'Median':median_4,'Mode':mode_4})\n    plt.title(\"Lecture Tag Distribution\")\n    \nprint(f'Tag (Mean): {mean_4}')\nprint(f'Tag (Median): {median_4}')\nprint(f'Tag (Mode): {mode_4}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.1.3 <a id='5.1.3'>Test Data Feature Distribution</a>\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    test_df['timestamp'].hist(bins = 50,color='maroon')\n    plt.title(\"Timestamp Distribution in Test Data\")\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    test_df['user_id'].hist(bins = 50,color='gold')\n    plt.title(\"User Id Distribution in Test Data\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"mean_5 = test_df['content_id'].mean()\nmedian_5 = test_df['content_id'].median()\nmode_5 = test_df['content_id'].mode()[0]\n\nmean_6 = test_df['task_container_id'].mean()\nmedian_6 = test_df['task_container_id'].median()\nmode_6 = test_df['task_container_id'].mode()[0]\n\nprint(f'Content Id Test(Mean): {mean_5}')\nprint(f'Content Id Test(Median): {median_5}')\nprint(f'Content Id Test(Mode): {mode_5}\\n')\nprint('######################################\\n')\nprint(f'Task Container Id Test(Mean): {mean_6}')\nprint(f'Task Container Id Test(Median): {median_6}')\nprint(f'Task Container Id Test(Mode): {mode_6}')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    sns.distplot(test_df['content_id'], color='cyan')\n    ax.axvline(int(mean_5), color='r', linestyle='--')\n    ax.axvline(int(median_5), color='y', linestyle='-')\n    ax.axvline(mode_5, color='b', linestyle='-')\n    plt.legend({'Mean':mean_5,'Median':median_5,'Mode':mode_5})\n    plt.title(\"Content Id Distribution in Test Data\")\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    sns.distplot(test_df['task_container_id'], color='purple')\n    ax.axvline(int(mean_6), color='r', linestyle='--')\n    ax.axvline(int(median_6), color='y', linestyle='-')\n    ax.axvline(mode_6, color='b', linestyle='-')\n    plt.legend({'Mean':mean,'Median':median,'Mode':mode})\n    plt.title(\"Task Container Id Distribution in Test Data\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean_7 = test_df['prior_question_elapsed_time'].mean()\nmedian_7 = test_df['prior_question_elapsed_time'].median()\nmode_7 = test_df['prior_question_elapsed_time'].mode()[0]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\n\nwith sns.axes_style(\"whitegrid\"):\n    sns.distplot(test_df['prior_question_elapsed_time'], color='darkgoldenrod')\n    plt.axvline(int(mean_7), color='r', linestyle='--')\n    plt.axvline(int(median_7), color='g', linestyle='-')\n    plt.axvline(mode_7, color='b', linestyle='-')\n    plt.legend({'Mean':mean_7,'Median':median_7,'Mode':mode_7})\n    plt.title(\"Prior Question Elapsed Time Distribution\")\n    \nprint(f'Content Id Test(Mean): {mean_7}')\nprint(f'Content Id Test(Median): {median_7}')\nprint(f'Content Id Test(Mode): {mode_7}\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.2 <a id='5.2'>Categorical Feature Distribution</a>\n[Table of contents](#0.1)\n\n## 5.2.1 <a id='5.2.1'>Train Data</a>"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 3)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    ay = sns.countplot(y = train['user_id'], order=train.user_id.value_counts().index[:10], palette=\"ocean_r\")\n    plt.title(\"Top 10 Active Users\")\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    aa = sns.countplot(y = train['content_id'], order=train.content_id.value_counts().index[:10], palette=\"terrain\")\n    plt.title(\"Top 10 Popular Contents Ids\")\n    \nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 2])\n    aa = sns.countplot(y = train['task_container_id'], order=train.task_container_id.value_counts().index[:10], palette=\"OrRd_r\")\n    plt.title(\"Top 10 Tasks\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n* We see **Top 10 most active users in first plot**. User with Id **801103753** has most number of interactions around **17,917**.\n* **Content** with Id **6116** is most popular.  \n* **Task Id** is unique Id for batched of questions/lectures. **Task Id 14** is at the top followed by **15 and 4**. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    ay = sns.countplot(train['user_answer'], palette=\"Set3\")\n    for p in ax.patches:\n        height = p.get_height()\n        ay.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/101230332*100),\n                ha=\"center\", fontsize=12)\n    plt.xlabel('user answer',fontsize=12)\n    plt.ylabel('count',fontsize=12)\n    plt.title(\"User's Answer To Questions\")\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    aa = sns.countplot(train['answered_correctly'], palette=\"pastel\")\n    for p in ax.patches:\n        height = p.get_height()\n        aa.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/101230332*100),\n                ha=\"center\", fontsize=14)\n    plt.xlabel('answered correctly',fontsize=12)\n    plt.ylabel('count',fontsize=12)\n    plt.title(\"Correct Answers\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n* User's answers to MCQ type questions. We can see users **have 4 options to choose from**. **-1** means lecture videos.  \n* **answered_correctly** is our traget label. It is binary target variable **0 means False and 1 means True** ignoring -1 since it is label for lecture videos. It sepecify if the answer chose by users in graph one are correct or not. We can see **most the users tend to answer correctly**. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 10))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    ay = sns.countplot(train['prior_question_had_explanation'].dropna(), palette=\"Pastel1\")\n    for p in ax.patches:\n        height = p.get_height()\n        ay.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/101230332*100),\n                ha=\"center\", fontsize=12)\n    plt.xlabel('Prior question had explanation',fontsize=12)\n    plt.ylabel('count',fontsize=12)\n    plt.title(\"Users Saw Explanation\", fontsize=14)\n    \nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    aa = sns.countplot(train['content_type_id'], palette=\"twilight_r\")\n    for p in ax.patches:\n        height = p.get_height()\n        aa.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/101230332*100),\n                ha=\"center\", fontsize=12)\n    plt.xlabel('Content Type Id',fontsize=12)\n    plt.ylabel('count',fontsize=12)\n    plt.title(\"Posed Question/Watching Lecture\", fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* It seems that users saw explanation and correct response after answering the previous question bundle. We have boolean value **True** and **False** if the users saw explanation or not respectively. \n\n* Most of the events in second graph represents **questions posed to the users around 98%**. Very small percentage **(~2%) of the events are associated with users watching a lecture**.\n\n## 5.2.2 <a id='5.2.2'>Questions Data</a>\n[Table of contents](#0.1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    ay = sns.countplot(questions['correct_answer'], palette=\"hls\")\n    for p in ax.patches:\n        height = p.get_height()\n        ay.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/13523*100),\n                ha=\"center\", fontsize=14)\n    plt.title(\"User's Answer To Questions\")\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    aa = sns.countplot(questions['part'], palette=\"deep\")\n    for p in ax.patches:\n        height = p.get_height()\n        aa.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/13523*100),\n                ha=\"center\", fontsize=12)\n    plt.title(\"TOIEC English-language Assessment Section Number\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For more information visit [here](https://www.iibc-global.org/english/toeic/test/lr/about/format.html) regarding the part column in questions.csv.\n\n**📌 Points to note :**\n* We see user's answer to MCQ questions. The first graph is same as the graph just above it from the **train.csv's user_answer column** almost same distribution except we **don't have -1 label for lectures**.  \n* The secound bar graph has information related to the sections in TOIEC English-language Assessment. It has 7 parts as given [here](https://www.iibc-global.org/english/toeic/test/lr/about/format.html). Most of the questions appear from **part 5 (Incomplete Sentences)**. In TOIEC English-language Assessment we have 2 sections **Listening and Reading Section** where former section has **4 part** and later section has **3 parts**."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"questions['tag'] = questions['tags'].str.split(' ')\nquestions['tag_length'] = questions['tag'].str.len()\ntag_len = questions['tag_length'].dropna()\ntag_len = tag_len.astype({'tag_length': 'int8'})\n\ntop_tags = questions.tag.explode('tags').reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    aa = sns.countplot(tag_len, palette=\"coolwarm\")\n    for p in aa.patches:\n        height = p.get_height()\n        aa.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/13522*100),\n                ha=\"center\", fontsize=12)\n    plt.xlabel('number of tags',fontsize=14)\n    plt.ylabel('count',fontsize=14)\n    plt.title(\"Number Of Tags Per Questions\", fontsize=14)\n    \nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    sns.countplot(y = top_tags['tag'], order = top_tags.tag.value_counts().index[:10], palette=\"ocean_r\")\n    plt.xlabel('count',fontsize=14)\n    plt.ylabel('tag',fontsize=14)\n    plt.title(\"Top 10 Tags\",fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* Tags assign to each question. We can see most the questions have only **one tag (48%) followed by questions with three tags (29%)**. \n* **92 is the most used tags for questions**.\n\n\n## 5.2.3 <a id='5.2.3'>Lectures Data</a>\n[Table of contents](#0.1)\n\nThis tabular data contain metadata for the lectures watched by students. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    ay = sns.countplot(lectures['part'], palette='BuPu_r')\n    for p in ax.patches:\n        height = p.get_height()\n        ay.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/418*100),\n                ha=\"center\", fontsize=14)\n    plt.title(\"Category code for lecture\")\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    aa = sns.countplot(lectures['type_of'], palette=\"gist_stern_r\")\n    for p in ax.patches:\n        height = p.get_height()\n        aa.text(p.get_x()+p.get_width()/2.,\n                height + 3,\n                '{:1.2f}%'.format(height/418*100),\n                ha=\"center\", fontsize=12)\n    plt.title(\"Lecture Description\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* We can see **7 category codes** for lectures. **34%** lectures having **code 5**. Only **5%** lectures have **code 3**.\n* Most of the lectures seems to be describing **theoritical concepts(53%)** followed by **44%** lectures on **solving questions**. We can see significantly less percentage of lectures for **Intention and Starter** categories **2% and 1% respectively**. "},{"metadata":{},"cell_type":"markdown","source":"# 6. <a id='6'>Multiple Features📈</a>\n[Table of contents](#0.1)\n\n## 6.1 <a id='6.1'>Train Data Features</a>"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\ngs = f.add_gridspec(1, 2)\n\nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 0])\n    ay = sns.countplot(train['user_answer'], hue = train['prior_question_had_explanation'], palette=\"vlag\")\n    for p in ax.patches:\n        height = p.get_height()\n        ay.text(p.get_x()+p.get_width()/2.,\n                height + 2,\n                '{:1.2f}%'.format(height/101230332*100),\n                ha=\"center\", fontsize=12)\n    plt.xlabel('User Answer',fontsize=14)\n    plt.ylabel('count',fontsize=14)\n    plt.title(\"User's Answer With And Without Explanation\", fontsize=16)\n        \nwith sns.axes_style(\"whitegrid\"):\n    ax = f.add_subplot(gs[0, 1])\n    aa = sns.countplot(train['answered_correctly'], hue= train['prior_question_had_explanation'], palette=\"deep\")\n    for p in ax.patches:\n        height = p.get_height()\n        aa.text(p.get_x()+p.get_width()/2.,\n                height + 2,\n                '{:1.2f}%'.format(height/101230332*100),\n                ha=\"center\", fontsize=12)\n    plt.legend(loc='center upper')\n    plt.xlabel('Answered Correctly',fontsize=14)\n    plt.ylabel('count',fontsize=14)\n    plt.title(\"User's Saw explanation and Correct Answers\", fontsize=15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n* We can see that user most of the time user's saw explanation after giving answers. Also, **answers** ranges from **-1 to 3** where -1 indicate a lecture video. \n* User's tend to answer correctly often and see explanation after answering the previous question bundle. We can also see **class imbalance**."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\n\nwith sns.axes_style(\"whitegrid\"):\n    sns.countplot(train['user_answer'], hue = train['answered_correctly'], palette=\"husl\")\n    plt.title(\"User's Answer vs Answered Correctly\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* User's response to given questions (MCQ) and if the answer is correct or not. "},{"metadata":{},"cell_type":"markdown","source":"## 6.2 <a id='6.2'>Question MetaData Features</a>\n[Table of contents](#0.1)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\n\nwith sns.axes_style(\"whitegrid\"):\n    sns.countplot(questions['correct_answer'], hue = questions['part'], palette=\"Spectral\")\n    plt.title(\"User's Answer vs Answered Correctly\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* We can see correct responses by users for each of the 7 parts. There seems to be only three reponses for part 2."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\n\nwith sns.axes_style(\"white\"):\n    sns.catplot(x=\"part\", y=\"tag_length\", kind=\"box\",\n                col=\"correct_answer\", aspect=.7, data=questions)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* We observe for most of the users response **there are 3-4 tags associated per question**.\n* **Part 7** has more number of tags for all available correct answers. \n* For all correct answers **3 tag** are present in **Part 1, 3,and 4**.\n* For **correct answers (choices 0 and 4)** we see there are **questions with 4, 5 and 6 tags associated**.  "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"with sns.axes_style(\"white\"):\n    sns.pairplot(questions, hue=\"correct_answer\", palette=\"gnuplot_r\", diag_kind=\"kde\",\n                 height=3, corner=True, plot_kws=dict(linewidth=1, alpha=1))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.3 <a id='6.3'>Lectures MetaData Features</a>\n[Table of contents](#0.1)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\n\nwith sns.axes_style(\"whitegrid\"):\n    sns.countplot(lectures['part'], hue = lectures['type_of'], palette=\"Spectral\")\n    plt.title(\"Categories in Parts\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* **Concepts and Solving questions** categories are present in evey part. \n* **Intention and Starter** categories are almost missing in each part. There is only **one occurence of Intention category** in **part 2** and **starter category is present only in part 5 and 6**."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 8))\n\nwith sns.axes_style(\"white\"):\n    sns.catplot(x=\"part\", y=\"tag\", kind=\"box\",\n                col=\"type_of\", aspect=.7, data=lectures)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**📌 Points to note :**\n\n* In the above plot we have **added some more information to visualize more relationships i.e tag variable**. **Concepts and Solving questions** categories are present in evey part as we can see in the bar plots above this graph. \n\n* There are **151 unique tags for lectures**. Most of which are in two categories i.e **Concepts and Solving questions**.\n\n* **Parts 5 and 6** seems to have more tags mostly in three categories **Concepts, Solving questions and Starter**. \n\n* There is one **outlier in Part 7 for *Solving Question* category**. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"with sns.axes_style(\"white\"):\n    sns.pairplot(lectures, hue=\"type_of\", palette=\"copper\", height=3,\n                 corner=True, plot_kws=dict(linewidth=1, alpha=0.6))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.4 <a id='6.4'>Feature Correlation</a>\n[Table of contents](#0.1)\n\nLet's see some correlation using heatmap."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 10))\n\nmask = np.triu(np.ones_like(train.corr(), dtype=bool))\n\nwith sns.axes_style(\"white\"):\n    sns.heatmap(train.corr(), mask=mask, square=True, cmap = 'YlGnBu', annot=True);\n    plt.title(\"Train Data Feature Correlation\", fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 10))\n\nmask = np.triu(np.ones_like(questions.corr(), dtype=bool))\n\nwith sns.axes_style(\"white\"):\n    sns.heatmap(questions.corr(), mask=mask, square=True, cmap = 'YlOrBr', annot=True);\n    plt.title(\"Questions Data Feature Correlation\", fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"f = plt.figure(figsize=(16, 10))\n\nmask = np.triu(np.ones_like(lectures.corr(), dtype=bool))\n\nwith sns.axes_style(\"white\"):\n    sns.heatmap(lectures.corr(), mask=mask, square=True, cmap = 'icefire', annot=True);\n    plt.title(\"Lectures Data Feature Correlation\", fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 7. <a id='7'>Reference</a>\n[Table of contents](#0.1)\n* https://www.kaggle.com/sohier/competition-api-detailed-introduction\n* [Unique Values](https://stackoverflow.com/questions/27241253/print-the-unique-values-in-every-column-in-a-pandas-dataframe)\n* [Fast Tabular Data Read](https://www.kaggle.com/rohanrao/riiid-with-blazing-fast-rid/notebook)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}