\n
Table of Contents
\n
About the Competition1 \n
Exploring the Data2\n
Explore Train data3\n
Explore Questions Data4\n
Explore Lectures Data5\n
Sample Test and Submission Data6 "},{"metadata":{},"cell_type":"markdown","source":"#
1. About the Competition"},{"metadata":{},"cell_type":"markdown","source":"Think back to your favorite teacher. They motivated and inspired you to learn. And they knew your strengths and weaknesses. The lessons they taught were based on your ability. For example, teachers would make sure you understood algebra before advancing to calculus. Yet, many students don’t have access to personalized learning. In a world full of information, data scientists like us can help. Machine learning can offer a path to success for young people around the world, and you are invited to be part of this mission.\n\nIn 2018, 260 million children weren't attending school. At the same time, more than half of these young students didn't meet minimum reading and math standards. Education was already in a tough place when COVID-19 forced most countries to temporarily close schools. This further delayed learning opportunities and intellectual development. The equity gaps in every country could grow wider. We need to re-think the current education system in terms of attendance, engagement, and individualized attention.\n\nIn this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions. You will pair your machine learning skills using Riiid’s EdNet data."},{"metadata":{},"cell_type":"markdown","source":"
Prize-winning teams will be invited to present their models at the AAAI-2021 Workshop on AI Education - Imagining Post-COVID Education with AI - in February 2021. All competition participants are welcome to submit their write-ups to the workshop.
"},{"metadata":{},"cell_type":"markdown","source":"# 1.1. Some Points to keep in mind\n\n- We have to use the official api to make our submissions.\n- Refer to the [starter notebook](https://www.kaggle.com/sohier/competition-api-detailed-introduction) for an example of how to complete a submission. \n- We should not try to submit anything for the rows that contain lectures.\n- The API provides user interactions groups in the order in which they occurred. \n- Each group will contain interactions from many different users, but no more than one task_container_id of questions from any single user. \n- Each group has between 1 and 1000 users.\n\n**Expect to see roughly 2.5 million questions in the hidden test set.**\n\n- The API will load up to 1 GB of the test set data in memory after initialization. The initialization step (env.iter_test()) will require meaningfully more memory than that; we recommend you do not load your model until after making that call. The API will also consume roughly 15 minutes of runtime for loading and serving the data. Hence, after making a submission, go and grab a cup of coffee.\n\n- The API loads the data using the types specified above (int32 for user_id, int8 for content_type_id, etc).\n\n- A baseline prediction using Keras and 5Fold Cross validation can be reffered [here](https://www.kaggle.com/mrutyunjaybiswal/riiid-neural-nets-starter-baseline-in-5f-cv)"},{"metadata":{},"cell_type":"markdown","source":"# 1.2. Updates, Corrections and Clarifications\n\n- Lecture tags will match the tags in `questions.csv`.\n- The hidden test set contains new users but not new questions.\n- The train/test data is complete, in the sense that there are no missing interactions in the union of train and test data. It remains possible that some questions weren't logged due to other issues that all datasets of mobile users are susceptible to, such as if a user lost their connection mid-question.\n- The test data follows chronologically after the train data. The test iterations give interactions of users chronologically."},{"metadata":{},"cell_type":"markdown","source":"> Evaluation Metrics: Area under the ROC Curve between the predicted probability and observed target\n\nAnd ofcoures yes, we gotta maximize our score. :)"},{"metadata":{},"cell_type":"markdown","source":"# 2.
Exploring the Data"},{"metadata":{},"cell_type":"markdown","source":"We have got really huge amount of data, and the hosts made it even interesting by testing our model using the kaggle time series API. More about the time series API can be found [here](https://www.kaggle.com/c/riiid-test-answer-prediction/data).\n\n**CSV Files**\n\n1. `train.csv`\n2. `questions.csv`: Meta data for the questions posted to users\n3. `lectures.csv`: metadata for the lectures watched by users as they progress in their education\n4. `example_test.csv`: Sample test data as it will be delivered by timeseries API. Almost same as `train.csv`\n5. `example_sample_submission.csv`"},{"metadata":{},"cell_type":"markdown","source":"Let's explore one by one."},{"metadata":{},"cell_type":"markdown","source":"Tip #1
\n \nLet's use datatable to load the complete data just for the shake of seeing the wall time. As we will proceed, we will use only $0.01\\%$ of whole data, i.e. around **1M data points**.\n\nHere is an awesome tutorial by World's third quadruple grandmaster Vopani:\nNotebook: https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets\n\nTo use datatable, you can do any of the following steps:\n1. Add datatable as a dataset to your kernel. Here is the dataset link by none other than Vopani: https://www.kaggle.com/rohanrao/python-datatable and then follow the steps in next code block and run it.\n2. Another way, if you fail to do the above step, then just fork this notebook and see how's that been performed ;)"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install ../input/python-datatable/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl > /dev/null 2>&1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport gc\nimport random\nimport numpy as np\nimport pandas as pd\nimport datatable as dt\nfrom collections import Counter, defaultdict\n\nimport plotly.express as px\nimport plotly.graph_objs as go\nfrom plotly.subplots import make_subplots\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nimport joblib\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n%matplotlib inline\n\ncolorMap = sns.light_palette(\"orange\", as_cmap=True)\n\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.preprocessing import LabelEncoder\n\n_ = np.seterr(divide='ignore', invalid='ignore')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# Helper functions\ndef distribution_plot(target_df, column, color, n=40):\n df = target_df[column].value_counts().reset_index()\n df.columns = [column,'count']\n df[column] = df[column].astype(str) + '-'\n df = df.sort_values(['count'],ascending=False)\n\n plt.figure(figsize=(15,10))\n plt.subplot(121)\n sns.distplot(df['count'],color=color)\n plt.title(f\"{column} distribution plot\")\n\n plt.subplot(122)\n sns.barplot(x='count',y=column,data=df[:n],orient='h')\n plt.title(f\"{column} count plot\")\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Explore Train data"},{"metadata":{},"cell_type":"markdown","source":"Here is the significance of each column.\n\n- `row_id`: (int64) ID code for the row.\n\n- `timestamp`: (int64) the time between this user interaction and the first event from that user.\n\n- `user_id`: (int32) ID code for the user.\n\n- `content_id`: (int16) ID code for the user interaction\n\n- `content_type_id`: (int8)(Binary category) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n- `task_container_id`: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id. Monotonically increasing for each user.\n\n- `user_answer`: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\n- `answered_correctly`: (int8) if the user responded correctly. Read -1 as null, for lectures.\n\n- `prior_question_elapsed_time`: (float32) How long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle.\n\n- `prior_question_had_explanation`: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{},"cell_type":"markdown","source":"Let's read the whole train dataframe using our weapon datatable. Before that, let's decide the dtypes to load in an optimized way. For better understanding, do refer to the organizer's notebooks."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"data_types_dict = {\n 'timestamp': 'int64',\n 'user_id': 'int32', \n 'content_id': 'int16', \n 'content_type_id':'int8', \n 'task_container_id': 'int16',\n 'user_answer': 'int8',\n 'answered_correctly': 'int8', \n 'prior_question_elapsed_time': 'float32', \n 'prior_question_had_explanation': 'bool'\n}\n\ntarget = 'answered_correctly'","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"%%time\n\ntrain_df = dt.fread('../input/riiid-test-answer-prediction/train.csv', columns=set(data_types_dict.keys())).to_pandas()\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train_df.nunique().to_frame().rename(columns={0:\"Unique_values\"}).style.background_gradient(cmap=colorMap)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
Takeaway #1
\n\n
Continuous Features
\n\n
\n - Time Features
\n \n - timestamp
Time between the user ineraction and the first event from that user.
\n - prior_questions_elapsed_time
Avg time a user took to solve each question in the previous bundle. For Ex: if there are three questions in a bundle, they share the same prior_question_elapsed_time and the value is the total time the user took to solve all three questions divided by three. Null for user's first questions bundle or lecture.
\n
\n - ID Features
\n \n - content_id
\n 13320 Unique values. ID code for the user interaction. Also refers to the question_id in the questions.csv file.
\n - task_container_id
7740 unique values. ID code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id. Monotonically increasing for each user.
\n
\n
\n\n
Categorical Features
\n\n
\n - content_type_id
\n \n - Binary Value
0 corresponds to content_id is the question_id of questions.csv and 1 corresponds to content_id also but it is the lecture_id of lectures.csv.
\n
\n - user_answer
\n \n - Contains 4 values
\n User's answer to the questions. Values are: 0, 1, 2, 3, and -1. -1 corresponds to Null value for lectures (not questions type content, i.e. valid for content_type_id == 1 only).
\n
\n - prior_question_had_explanation
\n \n - Boolean feature, contains True, False and NA.\n
Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between.
\n The value is shared across a single question bundle.
\n Null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.
\n
\n - user_id
\n What about user-id? low-cardinal feature? High-cardinal feature? Should we drop? How can we relate it analytically to our target?
\n
\n
Target
\n
answered_correctly\n
\n - Binary Value i.e 1 or 0.
\n - -1 as Null, and it implies the content is a lecture => content_type_id == 1.
\n - Ofc, Binary classification and Metrics is AUC_ROC.
\n
"},{"metadata":{},"cell_type":"markdown","source":"Let's understand the data more and try to find some interesting patterns.\n
If you find the kernel interesting, consider upvoting it and open a discussion thread in the comments section for further improvements. Thanks in advance :).
"},{"metadata":{},"cell_type":"markdown","source":"*Let's perform our analysis on 0.01 percent of whole data to save time as well as memory.*"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train = train_df.sample(frac=0.01)\nprint(\"Length of our sample dataset: \", len(train))\ndisplay(train)\n# delete original dataset\ndel train_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#
3.1. Missing/Nan Values
\nTarget Columns:
user_answer, prior_question_had_explanation, prior_question_elapsed_time, answered_correctly
"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train.isna().sum().to_frame().rename(columns={0:\"Nan_values\"}).style.background_gradient(cmap=colorMap)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As mentioned in the data description, Null/Nan values in `prior_question_elapsed_time` and `prior_question_had_explanation` corresponds to the user's first questions bundle or first lecture. For `user_answer` and `answered_correctly`, -1 represents NULL values where the content type is a lecture i.e `content_type_id == 1`. Let's check it out, how many questions and lectures contents are there in this 1M chunk."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"train['content_type_id'].value_counts().rename(index={0:\"Questions\", 1:\"Lectures\"}).to_frame().rename(columns={0:\"Counts\"}).style.background_gradient(cmap=colorMap)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that there are 19399 lecture contents in the train set and it has same numbers of Null values in `user_answer` and `answered_correctly` column. As these features/data points won't be there in the testing, we will drop that, i.e drop the rows having `content_type_id == 1`. Let's check the percentage of **to-be-dropped** rows in our dataset."},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"correct_counts = train['answered_correctly'].value_counts().rename(index={0:\"No\", 1:\"Yes\", -1:\"NaN\"}).reset_index()\ncorrect_counts.columns = ['answered_correctly', 'percentage']\ncorrect_counts.loc[:, \"percentage\"] /= len(train)\n\nuser_answer_counts = train['user_answer'].value_counts().rename(index={-1:\"NaN\"}).reset_index()\nuser_answer_counts.columns = ['User_answer', 'percentage']\nuser_answer_counts.loc[:, \"percentage\"] /= len(train)\n\ndef func(pct, allvalues):\n absolute = int(pct /100.*np.sum(allvalues))\n return \"{:.1f}%\".format(pct)\n pass\n\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 20))\nwegdes, texts, autotexts = ax1.pie(correct_counts['percentage'],\n autopct=lambda pct: func(pct, correct_counts['percentage']),\n explode=(0.05, 0.05, 0.05),\n labels=correct_counts['answered_correctly'],\n shadow=True,\n startangle=45,\n wedgeprops={\"linewidth\":1, \"edgecolor\":\"black\"},\n textprops= dict(color=\"black\"))\nax1.legend(wegdes, correct_counts['answered_correctly'],\n title=\"Answered Correctly\",\n loc=\"center\",\n bbox_to_anchor=(1, 0, 0, 0))\nplt.setp(autotexts, size=14, weight=\"bold\")\nax1.set_title(\"Answered Correctly Percentage\")\n\nwegdes, texts, autotexts = ax2.pie(user_answer_counts['percentage'],\n autopct=lambda pct: func(pct, user_answer_counts['percentage']),\n explode=(0.05, 0.05, 0.05, 0.05, 0.05),\n labels=user_answer_counts['User_answer'],\n shadow=True,\n startangle=45,\n wedgeprops={\"linewidth\":1, \"edgecolor\":\"black\"},\n textprops= dict(color=\"black\"))\nax2.legend(wegdes, user_answer_counts['User_answer'],\n title=\"User Answered\",\n loc=\"center\",\n bbox_to_anchor=(1, 0, 0, 0))\nplt.setp(autotexts, size=14, weight=\"bold\")\nax2.set_title(\"User Answers Percentage\")\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# free up spaces\ndel user_answer_counts, correct_counts\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"$2\\%$ is very insignificant, so let's drop the rows and check how many NaN values remaines in the columns of `prior_question_had_explanation` and `prior_question_elapsed_time`."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"drop_indexes = train[train['answered_correctly']==-1].index.to_list()\ndrop_data = train.loc[drop_indexes, :]\nretained_train = train.drop(index=drop_indexes).reset_index(drop=True)\n\nprint(\"Dropped datapoints (datapoints corresponding to lectures)\")\ndisplay(drop_data)\nprint(\"Retained datapoints (datapoints corresponding to questions)\")\ndisplay(retained_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check for remaining NaN values."},{"metadata":{"trusted":true},"cell_type":"code","source":"retained_train.isna().sum().to_frame().rename(columns={0:\"Nan_values\"}).style.background_gradient(cmap=colorMap)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, so the remaining NaN values are because of either first question bundle or lecture for that corresponding user. We will see how can impute these values later in our modelling notebook."},{"metadata":{},"cell_type":"markdown","source":"#
3.2. More on Continuous Features: Trends? exceptions? outliers?
\n\nTarget Columns:
timestamp, prior_question_elapsed_time, task_container_id, content_id, user_id, user_answer
"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(36, 20))\nsns.set_style('dark')\n\nchunk_data = retained_train.copy()\nchunk_data = chunk_data.sort_values(by=['timestamp'])\nchunk_data = chunk_data.drop_duplicates('timestamp')\n# we have 7,49,458 data points here, we will analyse into three pieces, having 2500 data points each from starting, middle and ending\n\nchunk1 = chunk_data.head(5000)\nplt.subplot(3, 1, 1)\nsns.pointplot(x=chunk1['timestamp'],\n y=chunk1['prior_question_had_explanation'],\n hue=chunk1['answered_correctly'],\n linestyle='--', color='yellow', markers='x')\nplt.xticks([])\nplt.yticks([0, 1])\n\nchunk2 = chunk_data[349500:350500]\nplt.subplot(3, 1, 2)\n\nsns.pointplot(x=chunk2['timestamp'],\n y=chunk2['prior_question_had_explanation'],\n hue=chunk2['answered_correctly'],\n linestyle='--', color='orange', markers='x')\nplt.xticks([])\nplt.yticks([0, 1])\n\nchunk3 = chunk_data.tail(1000)\nplt.subplot(3, 1, 3)\nsns.pointplot(x=chunk3['timestamp'],\n y=chunk3['prior_question_had_explanation'],\n hue=chunk3['answered_correctly'],\n linestyle='--', color='red', markers='x')\nplt.xticks([])\nplt.yticks([0, 1])\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Things to look upto:\n\n- With increasing timestamp i.e. time, the users become more efficient with answering correctly. This indicates a sequence model might give/beat state of the art/bench mark result ;)\n- In initial times, there are a very few explanations to the questions, yeah, its not a finding. It's a confirmation of the fact that user's first question bundle or lecture had no prior explanation."},{"metadata":{},"cell_type":"markdown","source":"#
3.3. Timestamps. What do you have pal?
"},{"metadata":{},"cell_type":"markdown","source":"- This is gonna play an important role in transformer models. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# let's explore more about timestamp\n# as it is known that, the timestamps are given in an incremental manner, i.e. non-decreasing, let's see user interaction\nplt.figure(figsize=(20, 8))\nsns.set_style(\"dark\")\nretained_train['timestamp'].hist(bins=50, color=\"blue\", legend=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"retained_train[retained_train['timestamp']==0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Well, well, I see something. There are 3824 unique user IDs even after removing lecture rows from the data. But here we can see that there are 3847 rows having `timestamp` value `0`. Let's find out the users having started an event or interaction at 0 ms."},{"metadata":{"_kg_hide-output":false,"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"init_timestamp_df = retained_train[retained_train['timestamp']==0]\nuser_names = init_timestamp_df['user_id'].to_list()\nuser_dictionary = Counter(user_names)\n\nmultiTaskers = []\nfor key, val in user_dictionary.items():\n if val >1 :\n multiTaskers.append(key)\nprint(f\"There are {len(multiTaskers)} multi-taskers in this 1M chunk.\") \n\nfor ui in multiTaskers:\n display(init_timestamp_df[init_timestamp_df['user_id']==ui])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# free up spaces\ndel init_timestamp_df\nretained_train.drop(columns=['content_type_id'], inplace=True)\ngc.collect()\ndisplay(retained_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Key findings:\n\n- There are 8 multitasking users given `timestamp==0`. Though it might differ as per the sampling, because the sampling is done randomly.\n- Each user has same `prior_question_elapsed_time` and `task_container_id`.\n- Each user has neighboring `content_id`, not necessarily incremental.\n- Same users also have instances of multitasking more than once."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def annotate(data, **kws):\n n = len(data)\n ax = plt.gca()\n ax.text(.7, .8, f\"N = {n}\", transform=ax.transAxes)\n pass\nsns.set_style('whitegrid')\ng = sns.FacetGrid(retained_train, col=\"user_answer\", height=4.5, aspect=0.8)\ncmap = sns.cubehelix_palette(rot=-.2, as_cmap=True)\n\ng.map(sns.scatterplot, 'timestamp', 'prior_question_elapsed_time',alpha=1, edgecolor='black',color='red')\ng.map_dataframe(annotate)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Well, the count shows users have answered `2` lesser than the rest."},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = make_subplots(rows=2, cols=2)\n\ntraces = [\n go.Bar(\n x=[-1, 0, 1], \n y=[\n len(retained_train[(retained_train['user_answer']==item) & (retained_train['answered_correctly'] == -1)]),\n len(retained_train[(retained_train['user_answer']==item) & (retained_train['answered_correctly'] == 0)]),\n len(retained_train[(retained_train['user_answer']==item) & (retained_train['answered_correctly'] == 1)])\n ], \n name='Option: ' + str(item),\n text = [\n str(round(100 * len(retained_train[(retained_train['user_answer']==item) & (retained_train['answered_correctly'] == -1)]) / len(retained_train[(retained_train['user_answer']==item)]), 2)) + '%',\n str(round(100 * len(retained_train[(retained_train['user_answer']==item) & (retained_train['answered_correctly'] == -0)]) / len(retained_train[(retained_train['user_answer']==item)]), 2)) + '%',\n str(round(100 * len(retained_train[(retained_train['user_answer']==item) & (retained_train['answered_correctly'] == 1)]) / len(retained_train[(retained_train['user_answer']==item)]), 2)) + '%',\n ],\n textposition='auto'\n ) for item in retained_train['user_answer'].unique().tolist()\n]\n\nfor i in range(len(traces)):\n fig.append_trace(traces[i], (i // 2) + 1, (i % 2) +1)\n\nfig.update_layout(\n title_text='Percent of correct answers for every option',\n height=900,\n width=800\n)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Although we observed some kind of imbalancement in terms of numbers, but we can see an almost balanced distribution of percentage over user answers. Let's look at the pair and correlation plot once before we wrap up for this session, sleepy :(."},{"metadata":{"trusted":true},"cell_type":"code","source":"retained_train[\"prior_question_had_explanation\"] = retained_train['prior_question_had_explanation'].replace({False:0, True:1})\nretained_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.pairplot(retained_train.drop(columns=[\"prior_question_had_explanation\"]), hue=\"answered_correctly\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corr = retained_train.corr()\nmask = np.zeros_like(corr)\nmask[np.triu_indices_from(mask)] = True\nwith sns.axes_style(\"white\"):\n f, ax = plt.subplots(figsize=(10, 10))\n ax = sns.heatmap(corr,mask=mask,square=True,linewidths=.8,cmap=\"viridis\",annot=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Okay, `task_container_id` is correlated with `timestamp`. Referring to our previous observations, users with mutlitasking instances i.e. having multiple interactions over single timestamp have same `task_container_id` as well as `prior_question_elapsed_time`. We will see more about this."},{"metadata":{"trusted":true},"cell_type":"code","source":"retained_train[[\"user_id\", \"timestamp\"]].groupby(\"user_id\").shift()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"retained_train[[\"user_id\", \"prior_question_elapsed_time\"]].groupby(\"user_id\").shift()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#
3.4. prior_question_had_explanation stats per high-cardinal features
"},{"metadata":{"trusted":true},"cell_type":"code","source":"context_explanation_agg = retained_train[[\"content_id\", \"prior_question_had_explanation\", target]].groupby([\"content_id\", \"prior_question_had_explanation\"]).agg(\"mean\")\ncontext_explanation_agg = context_explanation_agg.unstack().reset_index()\ncontext_explanation_agg.columns = [\"content_id\", \"false_mean_explanation\", \"true_mean_explanation\"]\n# change the dtypes to save more memory\ncontext_explanation_agg.content_id = context_explanation_agg.content_id.astype(\"int16\")\ncontext_explanation_agg.false_mean_explanation = context_explanation_agg.false_mean_explanation.astype(\"float16\")\ncontext_explanation_agg.true_mean_explanation = context_explanation_agg.content_id.astype(\"float16\")\n\ndisplay(context_explanation_agg)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#
3.5. Analysing (and creating) Attempts per categorical features
"},{"metadata":{"trusted":true},"cell_type":"code","source":"#TBD\n#retained_train[\"attempt_no\"] = 1\n#retained_train.groupby([\"user_id\"])[\"attempt_no\"].agg(\"sum\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4.
Explore Questions Data"},{"metadata":{},"cell_type":"markdown","source":"**Metadata for the questions posed to users.**\n\n- `question_id`: foreign key for the train/test content_id column, when the content type is question (0).\n\n- `bundle_id`: code for which questions are served together.\n\n- `correct_answer`: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n- `part`: the relevant section of the TOEIC test.\n\n- `tags`: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"questions = pd.read_csv(\"../input/riiid-test-answer-prediction/questions.csv\")\nquestions","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5.
Explore Lectures Data"},{"metadata":{},"cell_type":"markdown","source":"**Metadata for the lectures watched by users as they progress in their education.**\n\n- `lecture_id`: foreign key for the train/test content_id column, when the content type is lecture (1).\n\n- `part`: top level category code for the lecture.\n\n- `tag`: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n- `type_of`: brief description of the core purpose of the lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures = pd.read_csv(\"../input/riiid-test-answer-prediction/lectures.csv\")\nlectures","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As described earlier, `lecture_id` is the same as the `content_id` for `content_type_id==1`, let's have a look :)"},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train['content_type_id']==1].sort_values(['content_id'], ascending=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"OKay, we will merge the datasets and observe later. But for now, let's explore `tag`, `part` and `type_of`.\nAs far as `tag` is concerned, the description doesn't speaj much, but it has mentioned that we can cluster the lectures. Let's see."},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.nunique().to_frame().rename(columns={0:\"Value-counts\"}).style.background_gradient(cmap=colorMap)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So the `lectures.csv` contains 418 unique lecture IDs, 151 unique tags, 7 unique parts and 4 types of lectures. Let's see what's there in `tag`."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(30, 10))\nsns.barplot(lectures['tag'].value_counts().index, lectures['tag'].value_counts().values,alpha=0.8)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's analyse top-15 and buttom-15 tags by count and observe the impact and distribution of `part` and `type_of`."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"top_15_tags = lectures['tag'].value_counts()[:15].index.tolist()\nbtm_15_tags = lectures['tag'].value_counts()[-15:].index.tolist()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"distribution_plot(lectures, \"tag\", \"blue\", 15)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"distribution_plot(lectures, \"part\", \"green\", 15)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"distribution_plot(lectures, \"type_of\", \"red\", 15)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"topTags = lectures[lectures['tag'].isin(top_15_tags)]\nbtmTags = lectures[lectures['tag'].isin(btm_15_tags)]\n\ndistribution_plot(topTags, \"part\", \"blue\", 15)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"distribution_plot(topTags, \"type_of\", \"blue\", 15)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"distribution_plot(btmTags, \"part\", \"blue\", 15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"distribution_plot(btmTags, \"type_of\", \"blue\", 15)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Tags with lesser counts only have concept type. "},{"metadata":{},"cell_type":"markdown","source":"# 6.
Sample Test and Submission Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv(\"../input/riiid-test-answer-prediction/example_test.csv\")\ntest","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.read_csv(\"../input/riiid-test-answer-prediction/example_sample_submission.csv\")\nsub","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Work in Progress :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}
![](\"https://static1.squarespace.com/static/5eddaaf244c33b340e4f21bd/t/5f61a0d12e7ca571c592893e/1602469387294/?format=1500w\")