{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Insights from other Notebooks and Discussions:\n\nImportant links (References):\n\nNotebooks:\n1. Introduction to the API: https://www.kaggle.com/sohier/competition-api-detailed-introduction\n2. EDA + LGBM Starter: https://www.kaggle.com/isaienkov/riiid-answer-correctness-prediction-eda-modeling\n3. CV benchmark with LGBM: https://www.kaggle.com/sishihara/riiid-lgbm-5cv-benchmark\n4. Loading Large datasets: https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets\n\nDiscussion:\n1. Incremental/online learning relevance: https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190354\n2. Loading dataset faster with cuDF (requires GPU): https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190288\n3. Private dataset questions (unanswered as of now): https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190200\n4. Target Leakage (Past / Future must be kept seperate): https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189437\n5. Fancy Paper ideas (Advanced): https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189398\n\nMisc:\n- Load the data as pickle, feather, tfrec (or) use DataTable, cuDF, Dask, Bquery\n- The hidden test set contains *new users* but not new questions.\n- The test data follows chronologically after the train data. The test iterations give interactions of users chronologically.\n- Additional insights follow as I explore them.\n\n\n### Load necessary modules and data:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# Loading modules\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport riiideducation\nimport gc\nimport tqdm\n\n%matplotlib inline\n\n# Loading the API\nenv = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check the files we have been given\n!ls ../input/riiid-test-answer-prediction/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"Right now we are going to subsample the dataset instead of reading it all to the memory, let's try to understand what data we have been provided with before training on all the data given.\n\nWe need to change the dtypes of certain columns and read a subset of the entire data to be able to fit it all to the memory:"},{"metadata":{"trusted":true},"cell_type":"code","source":"dtypes = {\n    \"row_id\": \"int64\",\n    \"timestamp\": \"int64\",\n    \"user_id\": \"int32\",\n    \"content_id\": \"int16\",\n    \"content_type_id\": \"boolean\",\n    \"task_container_id\": \"int16\",\n    \"user_answer\": \"int8\",\n    \"answered_correctly\": \"int8\",\n    \"prior_question_elapsed_time\": \"float32\", \n    \"prior_question_had_explanation\": \"boolean\"\n}\n\ndata = pd.read_csv(\"../input/riiid-test-answer-prediction/train.csv\", dtype=dtypes, nrows=int(1e6))\n\nques = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv(\"../input/riiid-test-answer-prediction/lectures.csv\")\nex_sub = pd.read_csv('../input/riiid-test-answer-prediction/example_sample_submission.csv')\nex_test = pd.read_csv('../input/riiid-test-answer-prediction/example_test.csv')\n\ndata.shape, ques.shape, lectures.shape, ex_sub.shape, ex_test.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# returns total bytes consumed\ntemp = data.memory_usage(deep=True).sum()\n\n# bytes to MB\nprint (f\"{temp / (2**20):.2f}\", 'MB')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### EDA:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data.sample(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# row ID is basically redundant\n(data.row_id == data.index).all()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### From data description and discussions:\n\n`Time stamp`: The time in milliseconds between this user interaction (click n submit) and the first event COMPLETION from that user.\n\n`Prior_question_elapsed_time`: Is null for a user's first question bundle or lecture. It is the average time in a took to solve each question in the previous bundle. (NAN if curr ques belongs to first bundle or if curr ques is a lecture)\n\n`task_container_id`: Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n`prior_question_had_explanation`: Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user \nsees were part of an onboarding diagnostic test where they did not get any feedback.\n\n#### Let's check the data for any one random user to help us understand these features better:"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = data[data.user_id == np.random.choice(data.user_id.unique())].sort_values(\"timestamp\")\nprint (temp.shape)\ntemp.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Timestamp:\n- It's safe to say that timestamp = 0 is the first user interaction (COMPLETION)\n- TimeStamp will give us the order in which the student has interacted.\n\nPrior_ques_elapsed:\n- Time taken on avg to complete ques on prev bundle\n- Specfies how *quickly* the prev bundle questions were answered\n- DOES NOT specify the order of attempts.\n\nFrom [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189351):\n\n\"In rough terms you might want to look at the timestamp column to check whether a user worked on 10 questions a day or 10 questions per month and the `prior_question_elapsed_time` column to see if they need 1 second to respond or three minutes.\"\n\nFrom [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189465):\n\n\"`Task_container_id` captures the order that a user *first sees* tasks in, `timestamp` captures the order in which a user actually *completes* tasks. A user can start one task, start a second, and then finish the second before finishing the first -- this results in the later timestamp for the first task.\"\n\n#### Let's merge the questions and lectures together with our sample subset for better understanding of the whole:\n\nFrom [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189298) discussion, the `part` and `tags` codes are same for questions and lectures.csv. Though there is an apparant confusion since the tags don't seem to overlap between both the csv (as of now)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# combining questions and lectures together\nql = pd.concat([ques, lectures.rename({\"lecture_id\": \"question_id\"}, axis=1)], axis=0).reset_index(drop=True)\n\n# overlap the tags from both columns\nql.tags = ql.tags.fillna(ql.tag)\n\n# custom type of for questions\nql.type_of = ql.type_of.fillna(\"question\")\n\n# for distinguishing between lectures and questions\nql[\"content_type_id\"] = ql[\"type_of\"] != 'question'\n\n# bundle id and correct ans & tags is filled with -1\n# tag is missing for 1 row -> 10033\nql = ql.fillna(-1)\n\n# drop the unneeded tag feature\nql = ql.drop(\"tag\", 1)\n\n# rename the column for easy merge\nql = ql.rename({\"question_id\": \"content_id\"}, axis=1)\n\n# convert all the tags to list from string\nql.tags = ql.tags.apply(lambda x: [int(x)] if type(x) != str else list(map(int, x.split())))\n\n# note that missing values are filled with -1 though\nprint (\"Columns with missing values:\", ql.isna().sum().index[ql.isna().sum() != 0].tolist())\n\n# how does it look?\nql.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"What does the `part` column mean?\n\nIt refers to the relevant section of the [TOEIC](https://www.iibc-global.org/english/toeic/test/lr/about/format.html) test.\n\n1. Listening (Statements will not be printed):\n    - Part 1: Photographs: Four short statements regarding a photograph will be spoken only one time. The statements will not be printed. Of these four statements, select the one that best describes the photograph and mark your answer on the answer sheet.\n    \n    - Part 2: Question-Response: Three responses to one question or statement will be spoken only one time. They will not be printed. Select the best response for the question, and mark your answer on the answer sheet.\n    \n    - Part 3: Conversations: Conversations between two or three people will be spoken only one time. Listen to each conversation and read the questions printed in the test book (the questions will also be spoken). Some questions may require responses related to information found in diagrams, etc. printed on the test book as well as what you heard in the conversations. There are three questions for each conversation.\n    \n    - Part 4: Talks: Short talks such as announcements or narrations will be spoken only one time. Listen to each talk and read the questions printed in the test book (the questions will also be spoken), select the best response for the question, and mark your answer on the answer sheet. Some questions may require responses related to information found in diagrams, etc. printed on the test book as well as what you heard in the talks. There are three questions for each talk.\n\n2. Reading:\n    - Part 5: Incomplete Sentences: Select the best answer of the four choices to complete the sentence, and mark your answer on the answer sheet. \n\n    - Part 6: Text Completion: Select the best answer of the four choices (words, phrases, or a sentence) to complete the text, and mark your answer on the answer sheet. There are four questions for each text. \n\n    - Part 7: Passages: A range of different texts will be printed in the test book. Read the questions, select the best answer of the four choices, and mark your answer on the answer sheet. Some questions may require you to select the best place to insert a sentence within a text. There are multiple questions for each text. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# some might find reading section easier and listening tougher or vice versa\nql['reading_section'] = ql.part.isin([5, 6, 7]) # (5, 6, 7) => reading section (refer above)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# tags match for lectures and questions? (overlap)\n(set(ql[ql.content_type_id].tags.explode().unique()) \n == \n set(ql[~ql.content_type_id].tags.explode().unique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Return tags that exist only in one of the CSVs\nnp.setxor1d(\n    ql[ql.content_type_id].tags.explode().unique(),\n    ql[~ql.content_type_id].tags.explode().unique()\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# `part` overlaps?\nnp.intersect1d(lectures.part.unique(), ques.part.unique()).shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# content id overlap in lectures and questions\nnp.intersect1d(lectures.lecture_id.unique(), ques.question_id.unique()).shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# it would make much more sense if we merged lectures and questions together\n# we merge it on content_type_id since we need to differnetiate between\n# questions and lectures\ntemp = temp.merge(ql, on=['content_id', 'content_type_id'])\nprint(temp.shape)\ntemp.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that we have a overall understanding of what the features mean let's switch back to performing EDA on the entire dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# content ID denotes whether the interaction \n# is that of lecture or question\ndata.content_type_id.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print (len(data))\ndata.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Much less users compared to the number of questions (which is obvious)\n- Timestamps are a lot many more compared to prior_question_elapsed_time which is the case since it the average time taken to complete the previous bundle by a user, in other words it is the same for all questions for a particular user for a particular question bundle (see snippet below)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# prior_elapsed_time per bundle is always one (0 for nans)\n(data.groupby(['user_id', 'task_container_id'])\n ['prior_question_elapsed_time']\n .nunique().values <= 1).all()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# No overlap between the content id for varying task containers?\ndata.groupby(\"task_container_id\")['content_id'].nunique().sum() == data['content_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# is the timestamp increasing monotically for all users?\ndata.groupby(\"user_id\")['timestamp'].is_monotonic_increasing.all()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Describe function to see the overall stats:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Some visualizations to help us better:"},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = data.timestamp.plot(kind='hist', figsize=(10, 5), title='TimeStamp Distribution (log Scale)')\nax.set(yscale=\"log\", xlabel='Timestamp in milliseconds');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There's an exponentially decreasing trend. However we need per user stats to reliably tell anything about app popularity conclusively."},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = (data.groupby(\"user_id\")['timestamp'].max()\n      .plot(kind='hist', figsize=(10, 5),\n            title='User Retention (Since they first start using the app)'))\nax.set(yscale='log', xlabel='Milliseconds (max - min) timestamp');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The number of interactions with the app since their first interaction indeed decreases *exponentially* over time.\n\nLet's now visualize the avg time taken to solve questions (we can use prior_question_elapsed_time since it does the same thing we want):"},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = ((data.groupby(\"user_id\")['prior_question_elapsed_time'].mean() / 1000)\n      .plot(kind='hist', figsize=(10, 5), xticks=range(0, 101, 10)))\nax.set(title='Avg time taken to solve questions (User Avg)', xlabel='Time Elapsed in Seconds');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Any insights and corelations between timestamp, prior_question_elapsed_time and correct answer?"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = data.sample(frac=0.2).merge(ql, on=['content_id', 'content_type_id'], how='left')\n\n(temp.plot(kind='scatter', x='timestamp', y='prior_question_elapsed_time', \n           c='answered_correctly', cmap='jet', figsize=(10, 5), alpha=.8));","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There doesn't seem to be any. This somewhat makes sense since each user is unique and our predictions better be such that it is unique for each user. Each user may have a preference for `reading_section`, some for `writing_section` and so on. Let' try to visualise for the users individually:"},{"metadata":{"trusted":true},"cell_type":"code","source":"f, ax = plt.subplots(nrows=3, ncols=2, figsize=(20, 20))\nax = ax.ravel()\nfor i in range(6):\n    temp = data[data.user_id == np.random.choice(data.user_id.unique())].sort_values(\"timestamp\")\n    temp = temp.merge(ql, on=['content_id', 'content_type_id'])\n    sc = temp.plot(kind='scatter', x='timestamp', y='prior_question_elapsed_time', \n              ax=ax[i], s=temp.part*15, c='answered_correctly', cmap='jet', alpha=0.6,\n              title=f\"ID: {temp.user_id.values[0]}\\\n\\nScore avg: {temp.answered_correctly.mean():.2f} | Count: {len(temp)}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- There are some users who use the app spoardically in periodic intervals and there are users who use the app on a regular basis\n- Maybe we could group users together using an embedding or a simple kmeans clustering (for a later period)\n\n##### We will do much more EDA at a later time."},{"metadata":{"trusted":true},"cell_type":"markdown","source":"#### We now try to understand `ex_sub` & `ex_test` csv files:\n\nThese files are provided as sample for how the files produced by `env.itertest` would be. At one call, it would only give us a small batch. We need to make predictions with our models on this and submit with a `env.predict` before we can call the next batch. *This is done so as to mimic real life scenarios where the future data is not available for model training.*\n\nFrom data description:\n\n- `prior_group_responses` (string) provides all of the user_answer entries for previous group in a string representation of a list in the **first row** of the group. **All other rows in each group are null**. If you are using Python, you will likely want to call eval on the non-null rows. Some rows may be null, or empty lists.\n\n- `prior_group_answers_correct` (string) provides all the answered_correctly field for previous group, with the *same format and caveats as prior_group_responses*. Some rows may be null, or empty lists.\n\nA more thorough understanding can be obtained from this post [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190430) by Alex:\n\n     Once you submitted your predictions for a group, you get the next test_df off the iterator and that immediately tells you whether you were right or not. You can use this information to improve your model before continuing with going through the test set, or you can just ignore it.\n    As you can't submit predictions for the same group twice, you can't cheat with it. It's just meant to be used for improving your prediction algorithm as you get more information, as is typical for realtime applications."},{"metadata":{"trusted":true},"cell_type":"code","source":"# example input from the API looks like:\nex_test.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Only the first row of each group would contain the answers and scores. The rest of the rows are all null.\n- We are *NOT* provided with the `user_answer` during the predictions we are to make. We are only provided that information at the next batch along with whether the user's predictions were indeed correct. If this had not been the case, we could simply compare with the `questions.csv` and be able to perfectly predict if the users were correct ;)\n- During the prediction time we only have features such as the timestamp, question meta data and info regarding prior groups response."},{"metadata":{"trusted":true},"cell_type":"code","source":"# example prediction to the API must look like:\nex_sub.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- For submission, we only pass in row_id, predictions. Group_num although present here is not required for submission. (check 'Making Our Predictions' part)\n\nSome more insights about the time series API testframes:\n- All shapes for each batch aren't identical, each batch may have differing no of samples\n- From this discussion [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190698): Every test_iter will only have one group number -> The first row is a list in str format (could be an empty list too)."},{"metadata":{},"cell_type":"markdown","source":"### Data Cleaning / Pipeline\n\nAlso from [this](https://www.kaggle.com/dwit392/riiid-challenge-time-since-last-action-for-test) notebook it is said that time elapsed between the last interaction and current one is a good predictor of the answer correctness which would make sense since that time could be used by a student to prepare before taking up the next test. However with data scattered around and us loading only a tiny fraction of the actual dataset, creating this feature would prove really difficult (for a later time).\n\nLet's now write a function that when given testframe_1 and testframe_2 returns testframe_1 with `user_answer` and `answered_correctly` columns merged to it:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def post_process(fn0, fn1):\n    '''\n    fn0 is test dataframe at time t\n    fn1 is test dataframe at time t + 1\n    \n    If however no fn1 is provided, we need to simply assign all the \n    user_answer and answered_correctly as nans. For this purpose, \n    any dataframe with first row as nan will suffice.\n    '''\n    \n    fn_processed = pd.concat([\n        \n        # since group_num is set as index, we would lose it otherwise\n        fn0.iloc[:, :-2].reset_index(), \n        \n        # using eval to obtain the list values (including empty lists)\n        fn1.iloc[0, -2:].apply(lambda x: pd.Series(eval(x), dtype=np.int)).T\n        \n    ], axis=1)\n    \n    return fn_processed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Let's now load the data from the API for the purpose of understanding it, we will disable it when we wish to make a submission:\n\nIt will also help us check if our `post_process` function works good."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Donot set as False if you wish to submit\n# since iter_test can only be called once\nSUBMIT = True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not SUBMIT:\n    batches = []\n    prev_batch = next(iter_test)\n    while True:\n        try:\n            env.predict(prev_batch[1]) \n            batch = next(iter_test)\n            batches.append(post_process(prev_batch[0], batch[0]))\n            prev_batch = batch\n        except StopIteration:\n            batches.append(post_process(prev_batch[0]))\n            print (f\"All sample test batches exhausted after {len(batches)} iterations\")\n            break\n\n    print (\"Batch sizes for each test sample:\", list(map(lambda x: x.shape[0], batches)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Post_processed frame could then be used to help our model to learn about a students behaviour better in real time during submission. And since we are able to learn from the test data as well, it's best to create an *online / incremental model* for this competition.\n\n\n#### Evaluation metric:\n\nLet's understand the evaluation metric - `roc_auc_score`. It ensures that random predictions always yeild a score of 0.5. If however our score is less that 0.5 it means that we have made some mistake in our predictions (wrongly labeled the data, model does beter than random guess). A score of 1 (or 0) means that model is absolutely perfect and makes correct predictions 100% of the time. \n\nLet's verify this with some random guesses:"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Predicting constant values for the entire train dataset.\nfrom sklearn.metrics import roc_auc_score\ntemp = data.loc[~data.content_type_id, \"answered_correctly\"]\nfor value in [0, 1, 0.5, temp.mean()]:\n    print (\"At {:.2f} the score is: {:.3f}\".format(value, roc_auc_score(temp, np.full_like(temp, 0))))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A highly robust metric indeed. So we have to be a bit more smart in making predictions to beat this. What about random predictions *per question*?"},{"metadata":{"trusted":true},"cell_type":"code","source":"roc_auc_score(temp, np.random.rand(len(temp)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Does marginally (very marginally) better than previous predictions. Our next naive idea is to use per user mean accuracy as predictions but befor we do that we need to *something else*.\n\nLet's now write a function to split the data to train/val as reliably as possible to mimic the test case scenario. Further it should also function as a CV generator, given some train ids:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def generate_train_val(Train, train_users=None, tp=.70, vp=None, put=None): \n    '''\n    What we already know:\n    - Test dataset has new users but no new questions. \n    - Test follows chornologically after train\n    \n    * Parameters *  \n    train_ids -> The ID's completely used for training (If None, generate it using tp)\n    tp        -> train users percentage (completely used for training)\n    vp        -> Validation usage percentage completely used for val \n                 (used to make val users | partial users)\n                 If None, randomly chosen\n                 \n    put       -> timestamp threshold for partial users above which \n                 timetamp the row becomes validation dataset\n                 If None value chosen is same as `vp`\n    \n    * Output *\n    Returns a new dataframe with `train` column added for val/train split\n    \n    * Note *\n    1. Partial users are those users whose data is used for training and validation\n    2. Perentage of train data is always greater than tp\n    3. Exact percentage -> tp + \n    '''   \n    \n    # we will tinkering with it, best to copy it \n    # beware of passing in large sized DataFrames\n    train = Train.copy()\n    \n    total_users = train.user_id.unique()\n    \n    if train_users is None:\n        train_users = np.random.choice(total_users, int(len(total_users) * tp), replace=False)\n        \n    if vp is None: # validation percent \n        vp = np.random.choice(np.linspace(.20, .80, 13))\n    \n    if put is None: # threshold for timestamp cutoff\n        put = vp\n            \n    # partial users percentage\n    pp = 1 - vp\n    \n    remaining_users = np.setdiff1d(total_users, train_users)\n    val_users = np.random.choice(remaining_users, int(len(remaining_users) * vp), replace=False)\n    partial_users = np.setdiff1d(remaining_users, val_users)\n    \n    user_val_cutoff = (train[train.user_id.isin(partial_users)]\n                       .groupby(\"user_id\")['timestamp']\n                       .apply(lambda x: np.quantile(x, put).astype(int))\n                       .rename(\"val_cutoff\"))\n    \n    train = train.merge(user_val_cutoff, on='user_id', how='left')\n    train['val_cutoff'].fillna(0, inplace=True)\n    \n    train['train'] = (\n        train.user_id.isin(train_users) | \n        (train.user_id.isin(partial_users) & (train['timestamp'] < train['val_cutoff']))\n    )\n    \n    train = train.drop([\"val_cutoff\"], 1) \n    \n    return train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"How does the above function work? Lets see the split via a pie chart:"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = generate_train_val(data.copy())\nprint (data.shape, \"->\", temp.shape)\n(temp.train.astype(int).value_counts()\n .plot(kind='pie', autopct=lambda x: f\"{int(x)}%\", \n       title='Train/Val Split',\n       colors=['r', 'g'], explode=[.1, .15],\n       labels=[\"Train\", \"Validation\"]));","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Data Modelling:"},{"metadata":{},"cell_type":"markdown","source":"#### Submission Idea 1: \n\nWe create a dataframe containing all the users we see along with their average accuracy. We use this average accuracy as our predictions. "},{"metadata":{"trusted":true},"cell_type":"code","source":"folds = 5\ntotal_users = data.user_id.unique()\nnp.random.shuffle(total_users)\nval_size = len(total_users) // folds\nscores = {}\n\nfor i in range(folds):\n    train_users = np.setdiff1d(total_users, total_users[(i)*val_size:(i+1)*val_size])\n    temp = generate_train_val(data, train_users)\n    temp = temp[~temp.content_type_id]\n    \n    user_df = (temp[temp.train].drop(\"train\", 1)\n           .groupby(\"user_id\")['answered_correctly']\n           .agg(Mean='mean', Count='count', Sum='sum'))\n\n    pred = (temp.loc[(~temp.train)]\n     .merge(user_df, on='user_id', how='left'))\n\n    actual = pred['answered_correctly']\n    \n    scores['Off'] = scores.get(\"Off\", []) + [roc_auc_score(actual, pred['Mean'].fillna(0.5))]\n\n    pred = (\n        (pred.groupby(\"user_id\")['answered_correctly'].cumsum() + pred['Sum'].fillna(0)) / \n        (pred.groupby('user_id')['answered_correctly'].cumcount() + pred['Count'].fillna(0) + 1) \n    )\n    \n    scores['On'] = scores.get(\"On\", []) + [roc_auc_score(actual, pred)]\n\nprint (\"Scores after {} folds (User Mean Acc):\\n{}\\nWithout Online Learning: {:.2f} @ {:.2f} std\\\n\\nWith Online Learning: {:7.2f} @ {:.2f} std\"\n       .format(folds, \"=\"*40, np.mean(scores['Off']), np.std(scores['Off']), \n               np.mean(scores['On']), np.std(scores['On'])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above idea's score on public LB:\n1. Offline Naive Learning: `0.622` (ran for 30 min)\n2. Online Naive Learning: `0.634` (ran for 1 hr)\n\n#### Submission Idea 2: \n\nWe create a dataframe containing all the questions we see along with how accurate students usually answer them. We use this average accuracy as our predictions. \n\n`Note`: Even though it is said that the test set doesn't contain any new questions, it is still better to perform online learning since it may happen that eventually a tough question will get tackled head on and owing to increased students practice, cease to be a difficult question anymore."},{"metadata":{"trusted":true},"cell_type":"code","source":"folds = 5\ntotal_users = data.user_id.unique()\nnp.random.shuffle(total_users)\nval_size = len(total_users) // folds\nscores = {}\n\nfor i in range(folds):\n    train_users = np.setdiff1d(total_users, total_users[(i)*val_size:(i+1)*val_size])\n    temp = generate_train_val(data, train_users)\n    temp = temp[~temp.content_type_id]\n    \n    content_df = (temp[temp.train].drop(\"train\", 1)\n           .groupby(\"content_id\")['answered_correctly']\n           .agg(Mean='mean', Count='count', Sum='sum'))\n\n    pred = (temp.loc[(~temp.train)]\n     .merge(content_df, on='content_id', how='left'))\n\n    actual = pred['answered_correctly']\n    \n    scores['Off'] = scores.get(\"Off\", []) + [roc_auc_score(actual, pred['Mean'].fillna(0.5))]\n\n    pred = (\n        (pred.groupby(\"content_id\")['answered_correctly'].cumsum() + pred['Sum'].fillna(0)) / \n        (pred.groupby('content_id')['answered_correctly'].cumcount() + pred['Count'].fillna(0) + 1) \n    )\n    \n    scores['On'] = scores.get(\"On\", []) + [roc_auc_score(actual, pred)]\n\nprint (\"Scores after {} folds (Content Mean Acc):\\n{}\\nWithout Online Learning: {:.2f} @ {:.2f} std\\\n\\nWith Online Learning: {:7.2f} @ {:.2f} std\"\n       .format(folds, \"=\"*40, np.mean(scores['Off']), np.std(scores['Off']),\n               np.mean(scores['On']), np.std(scores['On'])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The above idea's scores on Public LB:\n1. Offline Naive Learning: `0.705` (ran for 20 min)\n2. Online Naive Learning: `0.705` (ran for 30 hr)"},{"metadata":{},"cell_type":"markdown","source":"### Making our predictions:\n\n1. For implementing Idea 1 simply change: `col = 'user_id'` in the snippet below.\n2. For implementing Idea 2 simply change: `col = 'content_id'` in the snippet below.\n\nWe use Idea 2 for now."},{"metadata":{"trusted":true},"cell_type":"code","source":"col = 'content_id'\n\ntemp = (pd.read_feather(\n    \"../input/riiid-train-data-multiple-formats/riiid_train.feather\", \n    columns=[col, 'content_type_id', 'answered_correctly']))\n\n# filter out only the questions\ntemp = temp[temp.content_type_id == 0]\ntemp = temp.drop(\"content_type_id\", axis=1)\n\ntemp = (temp.groupby(col)['answered_correctly'].agg(Mean='mean', Count='count', Sum='sum'))\n\nprint (f\"Memory Usage: {temp.memory_usage(deep=True).sum() / (2**20):.2f}\", 'MB')\nprint ( \"Length      :\", len(temp))\ntemp.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"prev_batch = next(iter_test)\ncalled, predicted = 1, 0\n\nwhile True:\n    try:\n        \n        ## =========================================================== ##\n        ## =================== Make the predictions ================== ##\n        ## =========================================================== ##\n        \n        q_mask = prev_batch[0]['content_type_id'] == 0\n        total = prev_batch[0].loc[q_mask, col].unique()\n        missing = np.setdiff1d(total, temp.index)\n        present = np.setdiff1d(total, missing)\n        \n        pred = prev_batch[0].loc[q_mask, ['row_id', col]].merge(\n            temp.loc[present, 'Mean'],            \n            on=col, how='left')\n        \n        pred['answered_correctly'] = pred['Mean'].fillna(0.5)\n        pred = pred[['row_id', 'answered_correctly']]\n        \n        ## =========================================================== ##\n        ## ================= API Calls (Donot Change) ================ ##\n        ## =========================================================== ##\n        \n        env.predict(pred) \n        predicted += 1\n        \n        batch = next(iter_test)\n        called += 1\n        \n        # Comment the line below for STATIC learning\n        processed_batch = post_process(prev_batch[0], batch[0])\n        \n        prev_batch = batch\n        \n        ## =========================================================== ##\n        ## ============= Update user_df (Online Learning) ============ ##\n        ##  Comment Code block below if you wish for Offline learning  ##\n        ## =========================================================== ##\n        \n        # We retain only those rows that are relevant\n        processed_batch = processed_batch[q_mask.reset_index(drop=True)]\n        \n        pb_grp = (processed_batch.groupby(col)['prior_group_answers_correct']\n                  .agg(Mean='mean', Count='count', Sum='sum'))\n\n        temp = temp.append(pb_grp.loc[missing])\n        temp.loc[present] = temp.loc[present] + pb_grp.loc[present]\n        temp.loc[present, 'Mean'] = temp.loc[present, 'Sum'] / temp.loc[present, 'Count']\n        \n        ## =========================================================== ##\n        \n    except StopIteration:\n        print (f\"All sample test batches exhausted after {predicted} iterations\")\n        break\n        \nassert predicted == called","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Offline learning scores lesser compared to online learning in our CVs. We observe the same trend in the LB as well (although much less compared to the difference observed in our CV). \n\nThe function to generate CV and the prediction Pipeline may be buggy (although executes without any error). If you did find any bugs kindly post them in the comments below. Thank you for reading :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}