{"cells":[{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"!pip install ../input/riiid-answer-correctness-prediction-utilities/site-packages/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\n\nfrom tqdm import tqdm\nfrom datetime import datetime\n\nimport numpy as np\nimport pandas as pd\npd.set_option('display.max_rows', 500)\npd.set_option('display.max_columns', 500)\npd.set_option('display.width', 1000)\nimport datatable as dt\n\nfrom scipy.stats import skew, kurtosis\n\nimport matplotlib as mpl\nmpl.rcParams['agg.path.chunksize'] = 10000\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom numba import jit\nfrom sklearn.metrics import roc_auc_score","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1. Introduction\n\nThis competition's objective is predicting whether the students are able to answer their next questions correctly or not. Dataset consist student's historic performance, the performance of other students on the same question, metadata about the question itself, and more. There are more than **100 million** rows in training set so it is not possible to read `train.csv` into memory without downcasting data types of the columns. It is better to use `numpy.ndarray` instead of `pandas.DataFrame` since memory consumption is 3 times larger in dataframes, but dataframes will be used in this notebook because they are very convenient."},{"metadata":{},"cell_type":"markdown","source":"### 1.1 Training Set\n\n`train.csv` is read into memory with `datatable.fread` for two reasons. First, `pandas.read_csv` takes more than 10 times to load the data compared to `datatable.fread`, and second, `pandas.read_csv` leaks memory. For large datasets like this, memory leak is not negligible. After `train.csv` is read with `datatable.fread`, it is converted to `pandas DataFrame`.\n\n* `row_id`: Unique ID of the row\n* `timestamp`: Time between users current interaction and the first event completion from that user\n* `user_id`: Unique ID of the user\n* `content_id`: Unique ID of the user interaction (question or lecture)\n* `content_type_id`: Type of the content (question = 0 and lecture = 1)\n* `task_container_id`: Unique ID of the batch of questions or lectures (monotonically increasing for each user)\n* `user_answer`: Users answer to the question (-1, 0, 1, 2, 3 ) \n* `answered_correctly`: User answered the question correctly or not (-1, 0, 1)\n* `prior_question_elapsed_time`: Average time a user spent to answer each question on previous question bundle (ignoring any lectures in between)\n* `prior_question_had_explanation`: Whether or not the user saw an explanation after answering the previous question bundle (ignoring any lectures in between)\n\nThere are **101,230,332** rows and **10** columns in training set. All of the columns had to be downcasted for using memory efficiently. `prior_question_had_explanation` is a binary feature, but it has missing values. Even though it has only 0 and 1 integer values, pandas forces an array of integers with any missing values to become floats. Using float type for a binary feature can be avoided by filling the missing values and casting it to an integer type. After that final touch, training set memory consumption is reduced to 2.7 GB."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"TRAIN_DTYPES = {\n    'row_id': np.uint32,\n    'timestamp': np.uint64,\n    'user_id': np.uint32,\n    'content_id': np.uint16,\n    'content_type_id': np.uint8,\n    'task_container_id': np.uint16,\n    'user_answer': np.int8,\n    'answered_correctly': np.int8,\n    'prior_question_elapsed_time': np.float32,\n    'prior_question_had_explanation': 'boolean'\n}\n\ndf_train = dt.fread('../input/riiid-test-answer-prediction/train.csv').to_pandas()\n\nfor column, dtype in TRAIN_DTYPES.items():\n    df_train[column] = df_train[column].astype(dtype) \n    \ndf_train['prior_question_had_explanation'] = df_train['prior_question_had_explanation'].astype(np.float16).fillna(-1).astype(np.int8)\n\nprint(f'Training Set Shape = {df_train.shape}')\nprint(f'Training Set Memory Usage = {df_train.memory_usage().sum() >> 20:.2f} MB')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.2 Questions\n\n`questions.csv` is the metadata of the questions that are asked to users. Unique identifier of the questions is `content_id` in training set, but it should be joined to questions (`content_type_id == 0`). Even though, questions metadata consume very few memory, it should be downcasted, because it will be joined to training set.\n\n* `question_id`: Unique ID of the question (0 - 13522)\n* `bundle_id`: Unique ID of the bundle (0 - 13522)\n* `correct_answer`: Correct answer of the question (0, 1, 2, 3)\n* `part`: Category code of the question (1, 2, 3, 4, 5, 6, 7)\n* `tags`: One or more detailed tag codes for the question\n\nThere are **13523** unique questions and **9765** unique bundles. Questions are multiple choice with 4 choices and they are asked in 7 different parts in tests."},{"metadata":{"trusted":true},"cell_type":"code","source":"QUESTIONS_DTYPES = {\n    'question_id': np.uint16,\n    'bundle_id': np.uint16,\n    'correct_answer': np.uint8,\n    'part': np.uint8,\n}\n\ndf_questions = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv', dtype=QUESTIONS_DTYPES)\n\nprint(f'Questions Shape = {df_questions.shape} - Unique Questions = {df_questions[\"question_id\"].nunique()} - Unique Bundles = {df_questions[\"bundle_id\"].nunique()}')\nprint(f'Questions Memory Usage = {df_questions.memory_usage().sum() >> 20:.2f} MB')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.3 Lectures\n\n`lectures.csv` is the metadata of the lectures watched by users as they progress in their education. Unique identifier of the lectures is `content_id` in training set, but it should be joined to lectures (`content_type_id == 1`). Even though, lectures metadata consume very few memory, it should be downcasted, because it will be joined to training set.\n\n* `lecture_id`: Unique ID of the question (89 - 32736)\n* `part`: Category code of the lecture (1, 2, 3, 4, 5, 6, 7)\n* `tag`: A single tag code of the lecture\n* `type_of`: Brief description of the core purpose of the lecture (concept, solving question, intention, starter)\n\nThere are **418** unique lectures and those lectures can be grouped into 4 major categories based on their types, and they are prepared for 7 parts of the tests."},{"metadata":{"trusted":true},"cell_type":"code","source":"LECTURES_DTYPES = {\n    'lecture_id': np.uint16,\n    'tag': np.uint16,\n    'part': np.uint8\n}\n\ndf_lectures = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv', dtype=LECTURES_DTYPES)\ndf_lectures['type_of'] = np.uint8(df_lectures['type_of'].map({'concept': 0, 'solving question': 1, 'intention': 2, 'starter': 3}))\n\nprint(f'Lectures Shape = {df_lectures.shape} - Unique Lectures = {df_lectures[\"lecture_id\"].nunique()}')\nprint(f'Lectures Memory Usage = {df_lectures.memory_usage().sum() >> 20:.2f} MB')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 1.4 Test Set and Submissions\n\nThis is a time-series code competition, so real test set data is recieved and predictions are made with Kaggle's time-series API. `example_test.csv` is a placeholder and the real test set is hidden. The purpose of a placeholder test set is, showing the structure of real test set and testing whether the submissions are in correct format or not. There will be roughly 2.5 million questions in the real test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_example_test = pd.read_csv('../input/riiid-test-answer-prediction/example_test.csv')\n\nprint(f'Example Test Set Shape = {df_example_test.shape}')\nprint(f'Example Test Set Memory Usage = {df_example_test.memory_usage().sum() >> 20:.2f} MB')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Entire hidden test set is not available in run-time as well. In hidden test set, there are information of previous groups, so in order to prevent data leak, submissions must be done with one group at a time in an order. This can only be done with Kaggle's time-series API.\n\nAPI can be started from `riiideducation` module in competition directory. After importing the module, environment can be created by calling `make_env` method. This loads methods to generate real test set, and it can be called once in a notebook session."},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\n\nenv = riiideducation.make_env()\nenv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`iter_test` method creates a generator which yields data of a single group in every iteration. Every group contains interactions from many different users but no more than one `task_container_id` of questions from any single user.\n\nThe API loads up to 1 GB of the test set data in memory after initialization so it is recommended to clean up the memory before making that call. The API also consumes roughly 15 minutes of runtime for loading and serving the data."},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()\niter_test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The API provides user interactions groups in the order in which they occurred when the generator is iterated. Each group will contain interactions from many different users, but no more than one `task_container_id` of questions from any single user, and each group has between **1** and **1000** users.\n\nIn every iteration, generator yields two dataframes. First one is the partial test set, and the second one is the submission dataframe that has row ids and target probabilities of the group. Both of those dataframes have `group_num` as index. This is the point when preprocessing should be done on partial test sets.\n\nThere are two additional columns in partial test sets. They are `prior_group_answers_correct` and `prior_group_responses`. Those features are only available on first rows of each group and they are missing in other rows. They contain target information of the previous group, so that's the reason why predictions must be made in an order via API."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_group, df_group_submission = next(iter_test)\n\nprint(f'Test Set Group\\n{\"-\" * 14}\\n\\n{df_group}\\n\\n')\nprint(f'Submission\\n{\"-\" * 10}\\n\\n{df_group_submission}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In order to prevent leakage, Kaggle API enforces to predict the current group before moving on to the next one. If the next group is tried to get generated, it will throw an error and generator doesn't return anything.\n\nAll of the questions (`content_type_id == 0`) must be predicted in the given submission format and passed to the API's `predict` method in order to move to the next group. That's how groups can be predicted while iterating over the generator. When the notebook is submitted, approximately 2.5 million questions are predicted and scored on public leaderboard.\n\nIn the cell below, every group's target probability is predicted 0.5. This will yield a 0.5 roc auc score on public and private leaderboard."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_group['answered_correctly'] = 0.5\nenv.predict(df_group.loc[df_group['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n\nfor df_group, df_group_submission in iter_test:\n    df_group['answered_correctly'] = 0.5\n    env.predict(df_group.loc[df_group['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Objective and Metric\n\nIt is a binary classification problem and predictions are evaluated with AUC (Area Under the ROC Curve) metric. It basically measures the area underneath the entire ROC curve. `sklearn.metrics.roc_auc_score` can be used for evaluation but it is very slow. A faster version of it, is shared [here](https://www.kaggle.com/c/microsoft-malware-prediction/discussion/76013) by [@CPMP](https://www.kaggle.com/cpmpml).\n\nFaster version is compiled with just in time (JIT) compiler. Instead of interpreting bytecode every time a method is invoked, bytecode is compiled into the machine code instructions of the running machine, and this compiled object code is invoked. For testing purposes, two different implementations calculated roc auc score on ground truth labels, and jit version is almost 5 times faster than sklearn version. "},{"metadata":{"trusted":true},"cell_type":"code","source":"@jit\ndef fast_roc_auc_score(y_true, y_prob):\n    \n    y_true = np.asarray(y_true)\n    y_true = y_true[np.argsort(y_prob)]\n    \n    n_false = 0\n    auc = 0\n    n = len(y_true)\n    \n    for i in range(n):\n        y_i = y_true[i]\n        n_false += (1 - y_i)\n        auc += y_i * n_false\n    auc /= (n_false * (n - n_false))\n    return auc\n\nquestions = df_train['answered_correctly'] != -1\n\nstart = datetime.now()\nfast_roc_auc_score = fast_roc_auc_score(df_train.loc[questions, 'answered_correctly'], df_train.loc[questions, 'answered_correctly'])\nfast_roc_auc_time = (datetime.now() - start).total_seconds()\nprint(f'Score: {fast_roc_auc_score:.6} - fast_roc_auc_score finished in {fast_roc_auc_time:.2} seconds.')\n\nstart = datetime.now()\nsklearn_roc_auc_score = roc_auc_score(df_train.loc[questions, 'answered_correctly'], df_train.loc[questions, 'answered_correctly'])\nroc_auc_score_time = (datetime.now() - start).total_seconds()\nprint(f'Score: {sklearn_roc_auc_score:.6} - sklearn.metrics.roc_auc_score finished in {roc_auc_score_time:.4} seconds.')\n\ndel questions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Target\n\nIn training set, target feature is named `answered_correctly` and it is binary (0 for incorrect answer and 1 for correct answer). It can be -1 as well but it's not another unique value. It means the target is null because the user is watching a lecture at that timestep, not answering any questions. It is evident that target is quite balanced and users are giving correct answers frequently."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(16, 4), dpi=100)\n    \nax = sns.barplot(x=df_train['answered_correctly'].value_counts().index, y=df_train['answered_correctly'].value_counts().values)\n\nax.set_xlabel('')\nax.set_ylabel('')\nax.set_xticklabels([f'{target} ({value_count:,})' for value_count, target in zip(sorted(df_train['answered_correctly'].value_counts().values), ['Not a Question', 'Incorrect', 'Correct'])])\nax.tick_params(axis='x', labelsize=15, pad=10)\nax.tick_params(axis='y', labelsize=15, pad=10)\nax.set_title(f'Target Distribution in Training Set', size=15, pad=15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Questions"},{"metadata":{},"cell_type":"markdown","source":"### 4.1. Questions, Bundles, and Correct Answers\n\nThere are **13523** unique questions and **9765** unique bundles. Number of unique questions in a bundle changes from 1 to 5, but the most common number is 1 (8151 out of 9765 bundles).\n\nCorrect answers of the questions are also given in `questions.csv`. Multiple choice question correct answers are usually located in the center (1 or 2) in most of the tests, but it is the opposite in this one. Majority of the correct answers are either 0 or 3, but bias isn't large enough for users to have psychological effects."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(16, 4), dpi=100)\n    \nax = sns.countplot(df_questions['correct_answer'])\n\nax.set_xlabel('')\nax.set_ylabel('')\nax.set_xticklabels([f'{value} ({count:,})' for value, count in df_questions['correct_answer'].value_counts().sort_index().to_dict().items()])\nax.tick_params(axis='x', labelsize=15, pad=10)\nax.tick_params(axis='y', labelsize=15, pad=10)\nax.set_title(f'Correct Answer Distribution in Questions', size=15, pad=15)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.2. Parts and Sections\n\nThere is a feature named `part` in `questions.csv`. It has values from 1 to 7 which corresponds to 7 parts in TOEIC test. Those parts can be seen from [here](https://www.iibc-global.org/english/toeic/test/lr/about/format.html). Parts from 1 to 4 are the listening section and parts from 5 to 7 are the reading section. Those sections are not given in the dataset so they have to be created manually. Label 1 is assigned to listening section and label 2 is assigned to reading section, and `section` feature is downcasted because it will be joined to training set.\n\nIt can be seen that 41.7% (5640) of the questions are from the listening section and 58.3% (7883) of the questions are from the reading section. Part distribution is somewhat balanced except part 5 which is defined as **\"Select the best answer of the four choices to complete the sentence, and mark your answer on the answer sheet.\"** on the given url above, and it has the most number of unique questions."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_questions.loc[df_questions['part'] <= 4, 'section'] = 1\ndf_questions.loc[df_questions['part'] > 4, 'section'] = 2\ndf_questions['section'] = df_questions['section'].astype(np.uint8)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(32, 8), ncols=2)\n    \nsns.countplot(df_questions['section'], ax=axes[0])\nsns.countplot(df_questions['part'], ax=axes[1])\n\nfor i in range(2):\n    axes[i].set_xlabel('')\n    axes[i].set_ylabel('')\n    axes[i].tick_params(axis='x', labelsize=22, pad=15)\n    axes[i].tick_params(axis='y', labelsize=22, pad=15)\n    \naxes[0].set_xticklabels([f'{value} ({count:,})' for value, count in df_questions['section'].value_counts().sort_index().to_dict().items()])\naxes[1].set_xticklabels([f'{value} ({count:,})' for value, count in df_questions['part'].value_counts().sort_index().to_dict().items()])\naxes[0].set_title(f'Section Distribution in Questions', size=25, pad=25)\naxes[1].set_title(f'Part Distribution in Questions', size=25, pad=25)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When correct answer counts in part groups are visualized, it can be seen that there is no single 3rd option correct answer in part 2. Part 2 and 5 are causing the correct answer imbalance. Correct answers are balanced in other parts except part 2 and 5. This suggests some questions in part 5 may have 3 multiple choices like questions in entire part 2. "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"correct_answers_by_parts = df_questions.groupby(['part', 'correct_answer'])['correct_answer'].count().rename('count').reset_index()\n\nfig = plt.figure(figsize=(32, 6), dpi=100)\n\nsns.barplot(x='part',\n            y='count',\n            hue='correct_answer',\n            data=correct_answers_by_parts)\n\nplt.xlabel('Part', size=25, labelpad=15)\nplt.ylabel('Count', size=25, labelpad=15)\nplt.tick_params(axis='x', labelsize=22)\nplt.tick_params(axis='y', labelsize=22)\nplt.legend(prop={'size': 20})\nplt.title('Correct Answer Distribution in Part Groups', size=25, pad=15)\n\nplt.show()\n\ndel correct_answers_by_parts\n\ncorrect_answers_by_sections = df_questions.groupby(['section', 'correct_answer'])['correct_answer'].count().rename('count').reset_index()\n\nfig = plt.figure(figsize=(32, 6), dpi=100)\n\nsns.barplot(x='section',\n            y='count',\n            hue='correct_answer',\n            data=correct_answers_by_sections)\n\nplt.xlabel('Section', size=25, labelpad=15)\nplt.ylabel('Count', size=25, labelpad=15)\nplt.tick_params(axis='x', labelsize=22)\nplt.tick_params(axis='y', labelsize=22)\nplt.legend(prop={'size': 20})\nplt.title('Correct Answer Distribution in Section Groups', size=25, pad=15)\n\nplt.show()\n\ndel correct_answers_by_sections","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 4.3. Tags\n\nOne last feature is left in `questions.csv` and that is `tags`. It is the tag codes of the questions delimited by spaces. This feature should not be joined to training set because data type of the column is object since there are spaces between integers. An object column of 101 million rows would consume too much memory. According to organizers, these codes are sufficient for clustering the questions together. Instead of the tag codes, clusters should be joined to training set."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_questions[['question_id', 'tags']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Lectures"},{"metadata":{},"cell_type":"markdown","source":"### 5.1. Lectures and Tag\n\nThere are **418** unique lectures users were watching before answering the questions. Unlike `questions.csv`, `tag` column is not an object type and it's not delimited by spaces in `lectures.csv`. Every lecture has only one tag which corresponds to same tags of the questions, so a lecture can have only one tag while a question can have multiple tags. There are 151 unique tags in both questions and lectures."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_lectures[['lecture_id', 'tag']].head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2. Types, Parts, and Sections\n\nThere is a feature named `type_of` in `lectures.csv` which is self-explanatory. 97% of the lectures are either 0 (concept) or 1 (solving question), and 3% of the lectures are either 2 (intention) or 3 (starter). Based on their labels, concept lectures are probably explaining the concepts behind the subject and question solving lectures are about solving sample questions. On the other hand, intention and starter lectures are introduction courses coming before concept and question solving lectures. Users should be able to answer questions correctly after watching concept and question solving lectures for an unseen subject.\n\nPart distribution is unbalanced compared to part distribution in questions. High number of part 5 lectures is the only thing matching between two distributions. Finally, since this file will be joined to training set, it should have `section` feature as well, just like questions."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_lectures.loc[df_lectures['part'] <= 4, 'section'] = 1\ndf_lectures.loc[df_lectures['part'] > 4, 'section'] = 2\ndf_lectures['section'] = df_lectures['section'].astype(np.uint8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, axes = plt.subplots(figsize=(32, 8), ncols=2)\n    \nsns.countplot(df_lectures['type_of'], ax=axes[0])\nsns.countplot(df_lectures['part'], ax=axes[1])\n\nfor i in range(2):\n    axes[i].set_xlabel('')\n    axes[i].set_ylabel('')\n    axes[i].tick_params(axis='x', labelsize=22, pad=15)\n    axes[i].tick_params(axis='y', labelsize=22, pad=15)\n    \naxes[0].set_xticklabels([f'{value} ({count:,})' for value, count in df_lectures['type_of'].value_counts().sort_index().to_dict().items()])\naxes[1].set_xticklabels([f'{value} ({count:,})' for value, count in df_lectures['part'].value_counts().sort_index().to_dict().items()])\naxes[0].set_title(f'Type Distribution in Lectures', size=25, pad=25)\naxes[1].set_title(f'Part Distribution in Lectures', size=25, pad=25)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Time Features\n\nThere are two time features, and their names are `timestamp` and `prior_question_elapsed_time`. Both of them are in milliseconds (0.001 seconds equals to 1 milliseconds). There is another feature named `task_container_id` and it monotonically increases for each user, so it should be analyzed together with previously mentioned time features. Those features can be confusing at first glance, but they are detailedly explained in this section."},{"metadata":{},"cell_type":"markdown","source":"### 6.1. Timestamp\n\n`timestamp` is the total milliseconds between user's current activity and first activity completed by that user. This value is entered when the current activity is **completed** by the user, so this feature is on user level. Every user have their own `timestamp` starting from 0 to their completed last activity. This means even if the users have spent any time on another activity after their last completed activity, that activity won't be entered if it is not completed.\n\nFor every user, `timestamp` starts from 0 and ends at their last completed activity. That's why 0 is the most common value in `timestamp` histogram. However, number of 0 values doesn't match the number of unique users, because some of the questions are given to users in a batch. In those cases, all questions in the same batch will share the same `timestamp` of the last completed activity. For instance, user 368624 had recieved 4 question batch in each activity, and `timestamp` value is shared on those activities. `task_container_id` column is there for distinguishing whether the consecutive questions are in a batch, or not. Questions that are given in a batch have the same `task_container_id`, and it can be seen from the sample below."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"first_timestamps = df_train.groupby('user_id')['timestamp'].first()\nlast_timestamps = df_train.groupby('user_id')['timestamp'].nth(-1)\n\nprint(f'There are {df_train[\"user_id\"].nunique():,} users in training set')\nprint(f'There are {df_train[df_train[\"timestamp\"] == 0].shape[0]:,} 0 timestamps in training set')\nprint(f'{first_timestamps[first_timestamps == 0].shape[0]:,} users first timestamp is 0')\nprint(f'{last_timestamps[last_timestamps == 0].shape[0]:,} users last timestamp is 0')\n\ndel first_timestamps, last_timestamps\n\ndf_train[df_train['user_id'] == 368624].head(12).style.background_gradient(subset=['task_container_id', 'timestamp'], cmap='viridis')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`timestamp` follows a skewed right distribution with lots of zeros because it is the only value that every user have in their first activities. Other nth activities of users can be any number because `timestamp` is in milliseconds, thus, cardinality is very high.\n\nStatistical measures of this variable are on a very large scale that they should be analyzed as days. Mean of `timestamp` is 89.2 days and median is 30.95 days. Difference between those two measures shows how big extreme values are. Some users waited very long time between their activities. The max value in `timestamp` is 7,703,643,654.326523 milliseconds which is equal to 1011.87 days and 2.77 years."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_mean = df_train['timestamp'].mean()\ntrain_median = df_train['timestamp'].median()\ntrain_std = df_train['timestamp'].std()\ntrain_min = df_train['timestamp'].min()\ntrain_max = df_train['timestamp'].max()\ntrain_skew = skew(df_train['timestamp'])\ntrain_kurt = kurtosis(df_train['timestamp'])\ntrain_var = np.var(df_train['timestamp'])\n\nprint(f'timestamp\\n{\"-\" * 9}')\nprint(f'Train Mean: {train_mean:,} - Median: {train_median:,} - Std: {train_std:,} - Var {train_var:,}')\nprint(f'Train Min: {train_min:,} - Max: {train_max:,} - Skew {train_skew:.4} - Kurt {train_kurt:.4} ')\n\nfigure = plt.figure(figsize=(18, 6))\n\nplt.hist(df_train['timestamp'], bins=200)\nplt.xlabel('timestamp', size=18, labelpad=15)\nplt.ylabel('')\nplt.tick_params(axis='x', labelsize=18, pad=15)\nplt.tick_params(axis='y', labelsize=18, pad=15)\nplt.title('timestamp Histogram in Training Set', size=20, pad=15)\n\nplt.show()\n\ndel train_mean, train_median, train_std, train_min, train_max, train_skew, train_kurt, train_var","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6.2. Prior Question Elapsed Time\n\n`prior_question_elapsed_time` is the average time in milliseconds a user spent to answer each question on the previous batch. Unlike `timestamp`, this value is the **average** time, so it doesn't monotonically increase and it has low cardinality compared to `timestamp` It is also entered when the current activity is completed by the user **if the completed activity is a question**. Lectures between questions are ignored. This feature is on user level as well, but there is one more important thing to consider.\n\nIn order to find the total amount of time spent on a batch, `prior_question_elapsed_time` values should be summed in `[user_id, task_container_id]` groups because it is the average time spent on **each** question from the previous batch. Based on the sample below, user 368624 spent 47250 milliseconds on each question in task container 0 and task container 1. Those batches have 4 questions in them so total amount of time spent on those batches are 189000 milliseconds."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"df_train[df_train['user_id'] == 368624].head(12).style.background_gradient(subset=['task_container_id', 'prior_question_elapsed_time'], cmap='viridis')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`prior_question_elapsed_time` follows a skewed right distribution just like `timestamp`, but without lots of zeros. Top 5 most common values in `prior_question_elapsed_time` are 15000, 16000, 17000, 18000, and 19000 with total 22234917 occurences. All of the values are multipliers of 10, which suggests, data is stored as seconds and converted to milliseconds afterwards. Cardinality is low because of this reason.\n\nStatistical measures of this variable are not on a very large scale thus they can be analyzed as seconds. Mean of `prior_question_elapsed_time` is 13 seconds and median is 21.2 seconds. Difference between those two measures can be explained with the long right tail. The max value in `prior_question_elapsed_time` is exactly 5 minutes which is probably the total amount of time given to users for a single question. The min value in `prior_question_elapsed_time` is 0.0 and that is probably users randomly selecting answers. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_mean = df_train['prior_question_elapsed_time'].mean()\ntrain_median = df_train['prior_question_elapsed_time'].median()\ntrain_std = df_train['prior_question_elapsed_time'].std()\ntrain_min = df_train['prior_question_elapsed_time'].min()\ntrain_max = df_train['prior_question_elapsed_time'].max()\ntrain_skew = skew(df_train['prior_question_elapsed_time'].dropna())\ntrain_kurt = kurtosis(df_train['prior_question_elapsed_time'].dropna())\ntrain_var = np.var(df_train['prior_question_elapsed_time'])\n\nprint(f'prior_question_elapsed_time\\n{\"-\" * 27}')\nprint(f'Train Mean: {train_mean:,} - Median: {train_median:,} - Std: {train_std:.4} - Var {train_var:.4}')\nprint(f'Train Min: {train_min:,} - Max: {train_max:,} - Skew {train_skew:.4} - Kurt {train_kurt:.4} ')\n\nfigure = plt.figure(figsize=(18, 6))\n\nplt.hist(df_train['prior_question_elapsed_time'], bins=200)\nplt.xlabel('prior_question_elapsed_time', size=18, labelpad=15)\nplt.ylabel('')\nplt.tick_params(axis='x', labelsize=18, pad=15)\nplt.tick_params(axis='y', labelsize=18, pad=15)\nplt.title('prior_question_elapsed_time Histogram in Training Set', size=20, pad=15)\n\nplt.show()\n\ndel train_mean, train_median, train_std, train_min, train_max, train_skew, train_kurt, train_var","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6.3. Task Container ID\n\n`task_container_id` is the unique identifier of the batch of questions or lectures. It starts from 0 and it monotonically increases for every user exactly like `timestamp`. It is added for identifying multiple questions or lectures in different batches. `content_id` of questions in a batch can change but they have same `task_container_id`. This can be seen from the sample below."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"first_task_container_ids = df_train.groupby('user_id')['task_container_id'].first()\nlast_task_container_ids = df_train.groupby('user_id')['task_container_id'].nth(-1)\n\nprint(f'There are {df_train[\"user_id\"].nunique():,} users in training set')\nprint(f'There are {df_train[df_train[\"timestamp\"] == 0].shape[0]:,} 0 task_container_id in training set')\nprint(f'{first_task_container_ids[first_task_container_ids == 0].shape[0]:,} users first task_container_id is 0')\nprint(f'{last_task_container_ids[last_task_container_ids == 0].shape[0]:,} users last task_container_id is 0')\n\ndel first_task_container_ids, last_task_container_ids\n\ndf_train[df_train['user_id'] == 368624].head(12).style.background_gradient(subset=['task_container_id', 'content_id'], cmap='viridis')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`task_container_id` has a skewed right distribution which is very similar to `timestamp` because both of those features start from 0 and monotonically increase for every user. Histograms of those two features are almost identical since both of them increase at the same timesteps.\n\nThe max value in `task_container_id` is 9999 and the min value in `task_container_id` is 0. Those 10,000 task containers could be size of the entire program in chronological order or they could be arbitrary numbers."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_mean = df_train['task_container_id'].mean()\ntrain_median = df_train['task_container_id'].median()\ntrain_std = df_train['task_container_id'].std()\ntrain_min = df_train['task_container_id'].min()\ntrain_max = df_train['task_container_id'].max()\ntrain_skew = skew(df_train['task_container_id'].dropna())\ntrain_kurt = kurtosis(df_train['task_container_id'].dropna())\ntrain_var = np.var(df_train['task_container_id'])\n\nprint(f'task_container_id\\n{\"-\" * 27}')\nprint(f'Train Mean: {train_mean:,} - Median: {train_median:,} - Std: {train_std:.4} - Var {train_var:.4}')\nprint(f'Train Min: {train_min:,} - Max: {train_max:,} - Skew {train_skew:.4} - Kurt {train_kurt:.4} ')\n\nfigure = plt.figure(figsize=(18, 6))\n\nplt.hist(df_train['task_container_id'], bins=200)\nplt.xlabel('task_container_id', size=18, labelpad=15)\nplt.ylabel('')\nplt.tick_params(axis='x', labelsize=18, pad=15)\nplt.tick_params(axis='y', labelsize=18, pad=15)\nplt.title('task_container_id Histogram in Training Set', size=20, pad=15)\n\nplt.show()\n\ndel train_mean, train_median, train_std, train_min, train_max, train_skew, train_kurt, train_var","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}