{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Importing Libraries and data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nfrom collections import Counter\nfrom wordcloud import WordCloud\nfrom tqdm.notebook import tqdm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"os.listdir('../input/riiid-test-answer-prediction')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/riiid-test-answer-prediction/train.csv', nrows=1000000)\nlectures = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv')\nquestions = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\nexample_test = pd.read_csv('../input/riiid-test-answer-prediction/example_test.csv')\nexample_sample_submission = pd.read_csv('../input/riiid-test-answer-prediction/example_sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Before anything else let's see what is our target"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_sample_submission.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_test.head(2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Just in case you are wondering why a example_test file then the reason is that the actual test file could only be seen from the time series api of the riid. rest info is here - https://www.kaggle.com/sohier/competition-api-detailed-introduction "},{"metadata":{},"cell_type":"markdown","source":"#### OK!! So we basically have to predict a probability for \"answered_correctly\" for a given \"row_id\" and \"group_num\" With that out of the way let's look at our training data"},{"metadata":{},"cell_type":"markdown","source":"#### Also the evaluation metric is area under the ROC curve."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"if len(train_df) == len(train_df.row_id.unique()):\n    print('row_id column is the key')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('total train samples ', len(train_df))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Understanding The Problem"},{"metadata":{},"cell_type":"markdown","source":"#### So this is a binary classification and timeseries problem where you predict \"ansered_correctly\" by a student \"user_id\" over time \"timestamp\"."},{"metadata":{},"cell_type":"markdown","source":"we have been provided with `content_id` column which tells us user interestion with the content (not entirely sure wheather its like a book id fixed for that book for all user's or a id generated for each interaction but we will see it later on.), also we have a `content_type_id` column in which 0 means he answered a question and 1 means the content was a lecture and hence he watched a lecture. If he watched a lecture then there was no question hence we need not predict the `answered_correctly` and skip the row for prediction."},{"metadata":{},"cell_type":"markdown","source":"Next, we have `task_container_id` columns which means id's for set's of questions. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a `task_container_id`. ( A correlation/similarity is in a column `bundle_id` from `question.csv` file)"},{"metadata":{},"cell_type":"markdown","source":"Upnext, we have `user_answer` column where user gives a MCQ type answer say 1,2 or 3 etc. and whether that answer was correct or not is recorded in the next `answered_correctly` column which we need to predict."},{"metadata":{},"cell_type":"markdown","source":"Upnext, we have `prior_question_elapsed_time` column which tells how long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle. (so it depends on the number of questions that bundle had)."},{"metadata":{},"cell_type":"markdown","source":"At the last we have `prior_question_had_explanation` (bool) column which tells whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback. ( In case the questions are related then this can help much.)"},{"metadata":{},"cell_type":"markdown","source":"## Analizing the train.csv file"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"No of students = \", len(train_df['user_id'].unique()))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# idstribution of class labels\nsns.set()\nplt.figure(figsize=(10,6))\nsns.countplot(data=train_df, x='answered_correctly', hue='prior_question_had_explanation')\nplt.title('label distribution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, there is class imbalance. This is first time I am happy to see class Imbalance. This means that student do study and get more correct_answers. KEEP IT UP BOYS. Make it more imbalanced. <br>\nThe number of data samples is too much so class imbalance doesn't matter to that extent."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# distribution of number of samples per student\nsns.set()\nfig = plt.figure(figsize=(15,6))\nfig = sns.kdeplot(train_df.groupby(by='user_id').count()['row_id'], shade=True, gridsize=50, color='g', legend=False)\nfig.figure.suptitle(\"User_id distribution\", fontsize = 20)\nplt.xlabel('User_id counts', fontsize=16)\nplt.ylabel('Probability', fontsize=16);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, most of the students have < 2K samples."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# How many question does each student attempt\ndf = train_df[train_df['content_type_id'] == 0]\n\ndf = df.groupby(by='user_id').count()\n\nfig = plt.figure(figsize=(15,6))\nfig = sns.kdeplot(df['row_id'], shade=True, gridsize=50, color='r', legend=False)\nfig.figure.suptitle(\"User attempted questions distribution\", fontsize = 20)\nplt.xlabel('Questions counts', fontsize=16)\nplt.ylabel('Probability', fontsize=16)\nplt.legend(['Questions Attempted','Questions Correctly answered'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So they follow the same distribution. Majority of student have attempted < 2K questions."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# distribution of correct and incorrect and no answers\ndf = train_df[train_df['content_type_id'] == 0]\n\ndf2 = df[df['answered_correctly'] == 1]\ndf3 = df[df['answered_correctly'] == 0]\n\ndf2 = df2.groupby(by='user_id').count()\ndf3 = df3.groupby(by='user_id').count()\n\nfig = plt.figure(figsize=(15,6))\nfig = sns.kdeplot(df2['row_id'], shade=True, gridsize=50, color='b', legend=False)\nfig = sns.kdeplot(df3['row_id'], shade=True, gridsize=50, color='r', legend=False)\n\nfig.figure.suptitle(\"User attempted questions distribution\", fontsize = 20)\nplt.xlabel('Questions counts', fontsize=16)\nplt.ylabel('Probability', fontsize=16)\nplt.legend(['Correctly answered','Incorrectly answered'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we can see that the probability of ansering correctly is mostly 2.5 times that of answering incorrectly."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# What is the distribution of students correctly answering a question\nvalues = []\n\ndf = train_df[train_df['content_type_id'] == 0]\n\nfor group, frame in df.groupby(by='user_id'):\n    \n    value = len(frame[frame['answered_correctly'] == 1]) / len(frame)\n    values.append(value)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(15,6))\nfig = sns.kdeplot(values, shade=True, gridsize=50, color='c', legend=False)\nfig.figure.suptitle(\"User correctly answering distribution\", fontsize = 20)\nplt.xlabel('Percent Correct', fontsize=16)\n\nprint('MEAN: ', np.mean(values))\nprint(\"MAX: \", np.max(values))\nprint('MIN: ', np.min(values))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"the graph shows that the average accuracy of a student is near 54% in answering correctly. Some have scored perfect and some have scored zero. (We have outliars as getting all zeros is by probability as hard as getting all correct so we have to handle that)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# What precent of students see explanations\n\nvalues = []\n\ndf = train_df[train_df['content_type_id'] == 0]\n\nfor group, frame in df.groupby(by='user_id'):\n    \n    value = len(frame[frame['prior_question_had_explanation'] == True]) / len(frame)\n    values.append(value)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nsns.distplot(values, kde=False)\nplt.title('Distribution if students who see x percent of explanations')\nplt.xlabel('Percent explanation seen out of attempted questions')\nplt.ylabel('Counts')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is a considerable amount of students who never watched prior explanations and yet answered correctly. Any further than this we will need to use timestamps or other files."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print(\"Total task container id's having questions: \", len((train_df['task_container_id'][train_df['content_type_id'] == 0]).unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Before we move into the real EDA which is with respect to time_stamp let's have a look at questions and lectures files"},{"metadata":{},"cell_type":"markdown","source":"## Questions file"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`question_id`: foreign key for the train/test content_id column, when the content type is question (0).\n\n`bundle_id`: code for which questions are served together.\n\n`correct_answer`: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n`part`: top level category code for the question.\n\n`tags`: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.info()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Total number of questions: ', len(questions['question_id'].unique()))\nprint(\"Total number of unique bundles: \", len(questions['bundle_id'].unique()))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10,6))\nfig = sns.countplot(questions.groupby(by='bundle_id').count()['question_id'])\nplt.xlabel('bundle_id')\nplt.title('Question in bundles');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Well I think that the majority of the question sets in task_container_id is from bundle 1"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(10,6))\nfig = sns.countplot(questions['correct_answer'])\nplt.xlabel('Answers')\nplt.title('Correct Answers distribution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the answers almost have a uniform distribution."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Distribution of number of tags per question\n# I think of tags as subject includings like (maths, algebra, numbers. history etc. in encoded form)\n\nno_of_tags = []\nfor i in questions['tags']:\n    value = len(str(i).strip().split(' '))\n    no_of_tags.append(value)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nsns.countplot(no_of_tags)\nplt.xlabel('No of tags')\nplt.title('No of tags per question')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Quite Superising that the number of tags is not decreasing constantly but has a huge dip at 2."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# distribution of tags\n\ntotal = []\n\nfor i in questions['tags']:\n    for j in str(i).strip().split(' '):\n        total.append(j)\n        \nkeys = set(total)\nfinal = {}\nfor i in keys:\n    final[i] = total.count(i)\n    \nvalues = sorted(final.items(), key=lambda x: x[1], reverse=True)\nd = []\nfor i in values:\n    d.append(i[1])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\npx.line(d, title='Tags distribution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution of tags is very skewed. Only 40 tags occur almost > 80% of time. If we want to decrease the sparcity of our data we could use only the top 100 tags and it would be more than 95% of total tags with only 50% sparcity."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Most commmon tags\ntags = WordCloud().generate_from_frequencies(final)\npx.imshow(tags, title='Most frequent Tags')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## lectures file"},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"lectures.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`lecture_id`: foreign key for the train/test content_id column, when the content type is lecture (1).\n\n`part`: top level category code for the lecture. (This confuses me. If we have tag then why do we need part?)\n\n`tag`: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n`type_of`: brief description of the core purpose of the lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.info()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Total no. of lectures: ', len(lectures['lecture_id'].unique()))\nprint('Only one tag per row: ', )\nprint('Total no. of tags in lecture: ', len(lectures['tag'].unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now comes the troubling part. Tags in lectures has no relation with tags in questions and hence they could be different altogether."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# distribution of lecture tags\n\ntotal = []\n\nfor i in lectures['tag']:\n    for j in str(i).strip().split(' '):\n        total.append(j)\n        \nkeys = set(total)\nfinal = {}\nfor i in keys:\n    final[i] = total.count(i)\n    \nvalues = sorted(final.items(), key=lambda x: x[1], reverse=True)\nd = []\nfor i in values:\n    d.append(i[1])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\npx.line(d, title='Tags distribution')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now that is some amazing pattern. The first idea is that the tags could be like subject title. Like lectures with high importance comes from a important chapter and that chapter may have only certen types of tags. Say math chapter may have (Maths, numbers, Algebra) as tags(encoded). "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Most common tags\ntags = WordCloud().generate_from_frequencies(final)\npx.imshow(tags, title='Most frequent lecture Tags')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Looking at parts\nprint('Total type of parts: ', len(lectures.part.unique()))\nprint('Values of parts: ', lectures.part.unique())","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Counts of parts\nplt.figure(figsize=(10,6))\nsns.countplot(lectures['part'])\nplt.title('Counts of Parts in Lectures');","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# how many different unique tags does each part have or do they common tags as well ?\n\nno_unique_tags_l = []\nunique_tags_l = {}\ngroups = []\n\nfor group, frame in lectures.sort_values(by='part').groupby(by='part'):\n    \n    unique_tags = frame['tag'].unique()\n    no_unique_tags = len(unique_tags)\n    \n    unique_tags_l[group] = unique_tags\n    no_unique_tags_l.append(no_unique_tags)\n    groups.append(group)\n    \nno_unique_tags_l = pd.DataFrame(no_unique_tags_l, columns=['count'])\nno_unique_tags_l['group'] = groups","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Number of unique tags in each part ( here unique means internally part wise)\nplt.figure(figsize=(10,6))\nsns.barplot(x=no_unique_tags_l['group'], y=no_unique_tags_l['count'])\nplt.title('No. of unique tags in each part')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, it also follow the same distribution as above."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"final_unqiue = []\nparts = []\n\n\nfor part, array in unique_tags_l.items():\n    \n    unique_tags = []\n        \n    other_parts = list(unique_tags_l.keys())\n    final = set(other_parts)\n    final.remove(part)\n    \n    for j in final:\n        \n        for k in array:\n            \n            if k not in unique_tags_l[j]:\n                \n                unique_tags.append(k)\n    \n    final_unqiue.append(len(unique_tags))\n    parts.append(part)\n    \nfinal_unqiue = pd.DataFrame(final_unqiue, columns=['tags'])\nfinal_unqiue['part'] = parts","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# let's see how many tags are there in each part which are not in any other part\n\nplt.figure(figsize=(10,6))\nsns.barplot(x=no_unique_tags_l['group'], y=no_unique_tags_l['count'])\nplt.title('No. of unique tags in each part not in any other');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So, we could rest assured that one tag comes only in one part."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Finally let's look at type_of lectures\n\npx.bar(lectures, x='type_of', color=lectures['type_of'], labels={'value':'type_of'}, title='Type of lectures distribution Overall')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"px.bar(lectures, x='type_of', color=lectures['type_of'], labels={'value':'type_of'}, title='Type of lectures distribution based on each part', facet_col='part')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Suprisingly intention belongs to only one part and starter is only in 2 parts."},{"metadata":{},"cell_type":"markdown","source":"## Now it's time to use the time stamp and see a few students from train file"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# we will see first 8 students for trends\nno_students = 8\nscores = []\nuser_ids = []\nquestion_attempted_l = []\ncorrectly_answered_l = []\nprior_questions_explanations = []\n\nfor count, (group, frame) in enumerate(train_df.groupby(by='user_id')):\n    \n    if count == no_students:\n        break\n    \n    frame = frame.sort_values(by='timestamp')\n    \n    percentage = []\n    question_attempted = []\n    correctly_answered = []\n    explanations = []\n    attempted = 0\n    correct_answers = 0\n    explanation = 0\n    \n    df = frame[frame['content_type_id'] == 0]\n    \n    for answered_correctly, had_explanation in zip(df['answered_correctly'], df['prior_question_had_explanation']):\n        \n        attempted += 1\n        question_attempted.append(attempted)\n        \n        if answered_correctly == 1:\n            correct_answers += 1\n            \n        if had_explanation:\n            explanation += 1\n            \n        correctly_answered.append(correct_answers)\n            \n        percent = correct_answers / attempted * 100\n        percentage.append(percent)\n        explanations.append(explanation)\n        \n    \n    scores.append(percentage)\n    user_ids.append(group)\n    question_attempted_l.append(question_attempted)\n    correctly_answered_l.append(correctly_answered)\n    prior_questions_explanations.append(explanations)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Trend in attempted question and correctly answering\n\nplt.figure(figsize=(15,20))\n\nfor i in range(1,9):\n    plt.subplot(4,2,i)\n    plt.plot(question_attempted_l[i-1], question_attempted_l[i-1], label='Questions attempted')\n    plt.plot(question_attempted_l[i-1], correctly_answered_l[i-1], label='Questions correctly answered')\n    plt.plot(question_attempted_l[i-1], scores[i-1], label='Percentage correctly answered')\n    plt.plot(question_attempted_l[i-1], prior_questions_explanations[i-1], label='Prior_questions_explanations')\n    plt.legend()\n    plt.ylim(0,100)\n    plt.xlim(0,50)\n    plt.tight_layout(pad = 2)\n    plt.title(f'user_id: {user_ids[i-1]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So much to see. So much trends and patterns. Well those who had prior explanation had better results. So the trend has many types. sudden spikes(+ve, -ve), consistency, continuous increment, decrement.<br>\nBad Students: Almost no one started watching explanations until they started performing bad."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Does students time spend on answering prior questions\n\nno_students = 8\ntime_spend_l = []\n\nfor count, (group, frame) in enumerate(train_df.groupby(by='user_id')):\n    \n    if count == no_students:\n        break\n    \n    frame = frame.sort_values(by='timestamp')\n    total_time_spends = []\n    time_spends = 0\n    \n    for time_spend in frame['prior_question_elapsed_time'][frame['content_type_id'] == 0]:\n        \n        if time_spend > 0:\n            time_spends += time_spend\n            total_time_spends.append(time_spends)\n        \n    \n    time_spend_l.append(total_time_spends)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"time_spend_l = np.array(time_spend_l)\nfor index, value in enumerate(time_spend_l):\n    time_spend_l[index] = np.array(time_spend_l[index]) / 10000","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Trend in time spend with percentage\n\nplt.figure(figsize=(15,20))\n\nfor i in range(1,9):\n    plt.subplot(4,2,i)\n    plt.plot(question_attempted_l[i-1], correctly_answered_l[i-1], label='Questions correctly answered')\n    plt.plot(question_attempted_l[i-1][1:], time_spend_l[i-1], label='time spend in 10000')\n    plt.plot(question_attempted_l[i-1], scores[i-1], label='Percentage correctly answered')\n    plt.legend()\n    plt.ylim(0,100)\n    plt.xlim(0,50)\n    plt.tight_layout(pad = 2)\n    plt.title(f'user_id: {user_ids[i-1]}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is mostly a linear increase in prior question time elapsed."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"print('Correlation among variables of train file')\ntrain_df.corr().style.background_gradient(cmap='Blues')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Modelling the data (basic)"},{"metadata":{},"cell_type":"markdown","source":"Now we have to make models on our data, but we have far too much data. first let's free some memory."},{"metadata":{"trusted":true},"cell_type":"code","source":"for element in dir():\n    if element[0:2] != \"__\":\n        del globals()[element]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Steps to model the data <br>\n    * Integrate information from lectures and questions file to training file\n    * Select only those columns which are questions and make a column named prior_thing(bool)(0 = question 1 =lecture)\n    * Make a new column named prior_type(categorical)(if prior_thing is 0 then 'question' else the type_of values from lecture file\n    * Now we can drop columns which are not questions\n    * We add another columns which is correct_answer and it will also be available in the test data\n    * Use user_answer column as label column\n    * Select user_id's with more than 30 columns and make a batch of each user_id to feed for training"},{"metadata":{},"cell_type":"markdown","source":"For simplicity we will consider only the first tag."},{"metadata":{},"cell_type":"markdown","source":"## Importing Libraries and data again"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport pandas as pd\nimport numpy as np\nfrom tqdm.notebook import tqdm\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import preprocessing\nimport lightgbm as lgb\nfrom multiprocessing import Pool","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"lectures = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv')\nquestions = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv(\"../input/riiid-test-answer-prediction/train.csv\",\n                        chunksize=500000)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As lecture_id and question_id are foreign keys so the number of content_id should be equal to number of lecture_id + number of question_id"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Making Mapping dictionaries: \\n\")\n\nconversion1 = {}\nconversion2 = {}\nconversion3 = {}\nconversion4 = {}\n\nfor i in tqdm(lectures['lecture_id']):\n    conversion1[i] = lectures['tag'][lectures['lecture_id'] == i].values[0]\n    conversion2[i] = lectures['part'][lectures['lecture_id'] == i].values[0]\n    conversion3[i] = lectures['type_of'][lectures['lecture_id'] == i].values[0]\n    conversion4[i] = -1\n\nfor i in tqdm(questions['question_id']):\n    conversion1[i] = questions['tags'][questions['question_id'] == i].values[0]\n    conversion2[i] = questions['part'][questions['question_id'] == i].values[0]\n    conversion3[i] = 'question'\n    conversion4[i] = questions['correct_answer'][questions['question_id'] == i].values[0]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def preprocess_train(df, count):\n    \n    df['tags'] = df['content_id'].map(conversion1)\n    df['part'] = df['content_id'].map(conversion2)\n    \n    df = (df.assign(tags = df['tags'].str.strip().str.split(' '))\n         .explode('tags')\n         .reset_index(drop=True))\n    \n    df.fillna(value=-1, inplace=True)\n    \n    df['prior_thing'] = 0\n    \n    for group, frame in df.groupby(by='user_id'):\n\n        frame = frame.sort_values(by='timestamp')\n        frame['prior_thing'] = frame['content_type_id'].shift(1)\n\n    df['prior_type'] = 'question'\n    \n    df['prior_type'] = df['content_id'].map(conversion3)\n    \n    for group, frame in df.groupby(by='user_id'):\n\n        frame = frame.sort_values(by='timestamp')\n        frame['prior_type'] = frame['prior_type'].shift(1)\n    \n    # Now after extracting information we will drop all columns other than those with questions\n    df = df[df['content_type_id'] == 0]\n    \n    df['correct_answer'] = df['content_id'].map(conversion4)\n\n    df.drop(columns = ['row_id', 'content_type_id', 'user_answer', 'content_id'], inplace=True)\n    \n    df.fillna(value=-1, inplace=True)\n    \n    df['tags'] = df['tags'].astype(np.float32)\n\n    \n    le1 = preprocessing.LabelEncoder()\n    le2 = preprocessing.LabelEncoder()\n\n    df['prior_question_had_explanation'] = le1.fit_transform(df['prior_question_had_explanation'])\n    df['prior_type'] = le2.fit_transform(df.loc[:, 'prior_type'].values)\n    \n    return df, le1, le2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Making a model"},{"metadata":{},"cell_type":"markdown","source":"Though I will be making a simple train test split you should make validation by grouping the data by user_id and then sorting by timestamp and choosing the last index values in the validation."},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_features = ['user_id', 'task_container_id', 'prior_question_had_explanation', 'tags', 'part',\n                       'prior_thing', 'correct_answer']\n\nmodel = None\ncount = 0\n\nparams = {\n    'keep_training_booster' : True,\n    'objective': 'binary',\n    'verbose':100,\n    'learning_rate': 0.1,\n}\n\n# I we will run the model for 2 rounds, you could run it for more. (It takes time)\n\nfor df in train_df:\n    \n    if count == 2:\n        break\n        \n\n    \n    df, le1, le2 = preprocess_train(df, count)\n    \n    count += 1\n    \n  \n    xtrain, xvalid, ytrain, yvalid = train_test_split(df.drop(columns='answered_correctly'),\n                                                     df['answered_correctly'], test_size=0.2, random_state=1)\n    \n    lgb_train = lgb.Dataset(xtrain, ytrain, categorical_feature=categorical_features)\n    lgb_valid = lgb.Dataset(xvalid, yvalid, categorical_feature=categorical_features)\n    \n    model = lgb.train(params,\n            init_model=model,\n            train_set=lgb_train,\n            valid_sets=lgb_valid,\n            verbose_eval=10,\n            num_boost_round=100)\n    \n    print(\"ROC SCORE: \", roc_auc_score(yvalid, model.predict(xvalid)))\n    \n    del df, xtrain, ytrain, xvalid, yvalid, lgb_train, lgb_valid\n    gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### I NEED MORE, MUCH MORE RAM"},{"metadata":{},"cell_type":"markdown","source":"## KINDLY UPVOTE. Also try to use this for riiideducation prediction as exercise."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}