{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Import necessary libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom matplotlib import pyplot as plt\nimport seaborn as sns\nimport os","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Check data available"},{"metadata":{},"cell_type":"markdown","source":"We have 4 datasets at our disposal"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"os.listdir('../input/riiid-test-answer-prediction')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_csv = pd.read_csv(\"../input/riiid-test-answer-prediction/lectures.csv\")\nexample_test_csv = pd.read_csv(\"../input/riiid-test-answer-prediction/example_test.csv\")\ntrain_csv = pd.read_csv(\"../input/riiid-test-answer-prediction/train.csv\", low_memory=False, nrows=1000000)\nquestions_csv = pd.read_csv(\"../input/riiid-test-answer-prediction/questions.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's explore each of the datasets!"},{"metadata":{},"cell_type":"markdown","source":"## 1.1 ```'train.csv'```"},{"metadata":{},"cell_type":"markdown","source":"Let's take another look at our parameters:\n\n- ```row_id```: (int64) ID code for the row.\n\n- ```timestamp```: (int64) the time between this user interaction and the first event from that user.\n\n- ```user_id```: (int32) ID code for the user.\n\n- ```content_id```: (int16) ID code for the user interaction\n\n- ```content_type_id```: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n- ```task_container_id```: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id. Monotonically increasing for each user.\n\n- ```user_answer```: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\n- ```answered_correctly```: (int8) if the user responded correctly. Read -1 as null, for lectures.\n\n- ```prior_question_elapsed_time```: (float32) How long it took a user to answer their previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Note that the time is the total time a user took to solve all the questions in the previous bundle.\n\n- ```prior_question_had_explanation```: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can conclude that ```answered_correctly```, ```prior_question_had_explanation``` are a nominative features; ```user_answer``` is a rank variable; ```timestamp```, ```prior_question_elapsed_time```  are quantitative."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.describe()[['timestamp', 'user_answer', 'answered_correctly', 'prior_question_elapsed_time']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's check the Nan values"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The easiest way is to delete rows that contain nan values, but in this case we may lose important information. Alternatively, it can be replaced by the average value of the group, where the group is calculated taking into user's id and the content's id."},{"metadata":{"trusted":true},"cell_type":"code","source":"# train_csv[\"prior_question_elapsed_time\"] = train_csv.groupby([\"user_id\", \"content_id\"]).transform(lambda x: x.fillna(x.mean()))\n# train_csv[\"prior_question_had_explanation\"] = train_csv.groupby([\"user_id\", \"content_id\"]).transform(lambda x: x.fillna(x.mean()))\n\n\n# train_csv.dropna(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv['timestamp'].hist(bins = 50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can see that many users have a period of \"stagnation\" now."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.countplot(train_csv.groupby('user_id')['user_answer'].count().value_counts(), palette=\"hls\")\nplt.title(\"Count of answers per user\", fontsize=12)\nplt.xticks(rotation=90, fontsize=13)\nplt.ylabel('Number of answers')\nplt.xlabel('Count of users')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can single out one user who answered much more times than other students. Almost all users answered up to 30 times"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.countplot(train_csv.user_answer)\nplt.title(\"Distribution of Mean's answer per user\", fontsize=12)\nplt.xticks(rotation=90, fontsize=13)\nplt.ylabel('Frequency')\nplt.xlabel('Average answer')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see that users in principle equally likely to answer questions using answers 0,1,3. There are some -1 values\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.distplot(train_csv.groupby('user_id')['answered_correctly'].mean())\nplt.title(\"Distribution of correct's answer per user\", fontsize=12)\nplt.xticks(rotation=90, fontsize=13)\nplt.ylabel('Frequency')\nplt.xlabel('Average correct answer')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_csv.groupby('user_id')['answered_correctly'].mean().median()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It can be argued that users are more likely to respond **correctly**. Let's implement another plot to estimate this opinion"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.countplot(train_csv.answered_correctly)\nplt.title(\"Distribution of correct answer\", fontsize=12)\nplt.xticks(rotation=90, fontsize=13)\nplt.ylabel('Frequency')\nplt.xlabel('Answer')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Answer 1 is almost 2 times more often correct than 0"},{"metadata":{"trusted":true},"cell_type":"code","source":"s = train_csv.groupby('content_id')['user_answer'].count().sort_values(ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"s[:20]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"zz = train_csv.groupby('content_id')['user_answer'].count().sort_values(ascending=False)\nplt.figure(figsize=(15, 7))\nax = sns.lineplot(y=zz, x=range(0, len(zz)))\nplt.title(\"Count of answers per content_id\", fontsize=12)\nplt.locator_params(nbins=12)\nplt.ylabel('Number of answers')\nplt.xlabel('Number of content_id')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Approximately 2000 contents have more than 200 questions\n\nWe can find **most popular content**"},{"metadata":{"trusted":true},"cell_type":"code","source":"zz[:15]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.distplot(train_csv.groupby('user_id')['prior_question_elapsed_time'].mean())\nplt.title(\"Distribution of Mean's prior_question_elapsed_time per user\", fontsize=12)\nplt.xticks(rotation=90, fontsize=13)\nplt.ylabel('Frequency')\nplt.xlabel('Average prior_question_elapsed_time')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In avarage, each user needs 20000 to answer their previous question bundle, ignoring any lectures in between.\n\nNote: Note that the time is the total time a user took to solve all the questions in the previous bundle.\n"},{"metadata":{},"cell_type":"markdown","source":"## 1.2 ```'questions.csv'```"},{"metadata":{},"cell_type":"markdown","source":"Let's take another look at our parameters:\n\n- ```question_id```: foreign key for the train/test content_id column, when the content type is question (0).\n\n- ```bundle_id```: code for which questions are served together.\n\n- ```correct_answer```: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n- ```part```: top level category code for the question.\n\n- ```tags```: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_csv.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_csv.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_csv.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def split_tags(x):\n    try: return [int(i) for i in str(x).split()]\n    except: return [0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_csv.tags = questions_csv.tags.apply(lambda x: split_tags(x))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique, counts = np.unique(questions_csv.tags.sum(), return_counts=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.barplot(x=unique, y=counts)\nplt.title(\"Count of tag\", fontsize=12)\nplt.tick_params(axis='x',which='both', bottom=False, top=False, labelbottom=False)\nplt.ylabel('Count')\nplt.xlabel('Tag')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"idx = np.argsort(counts)[::-1]\nprint(f\"most frequently tags are: {unique[idx[:5]]}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.countplot(questions_csv['correct_answer'], palette=\"hls\")\nplt.title(\"Count of correct answer per each choice\", fontsize=12)\nplt.xticks(rotation=90, fontsize=12)\nplt.ylabel('Count')\nplt.xlabel('Correct answer')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.countplot(questions_csv.groupby('bundle_id').count()['question_id'], palette=\"hls\")\nplt.title(\"Count of questions per bundle_id\", fontsize=12)\nplt.xticks(rotation=90, fontsize=12)\nplt.ylabel('Number of bundle')\nplt.xlabel('Number of question')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most bundle have only 1 question"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.countplot(questions_csv['part'], palette=\"hls\")\nplt.title(\"Distribution of Path\", fontsize=12)\nplt.xticks(rotation=90, fontsize=12)\nplt.ylabel('Count')\nplt.xlabel('Path')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Type \"5\" is more frequent"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_csv.groupby(['part', 'correct_answer']).count()['question_id']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In each part approximately the same distribution of the answer variant"},{"metadata":{},"cell_type":"markdown","source":"## 1.3 ```'lectures.csv'```"},{"metadata":{},"cell_type":"markdown","source":"Let's take another look at our parameters:\n\n- ```lecture_id```: foreign key for the train/test content_id column, when the content type is lecture (1).\n\n- ```part```: top level category code for the lecture.\n\n- ```tag```: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n- ```type_of```: brief description of the core purpose of the lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_csv.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_csv.nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_csv.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.countplot(lectures_csv['part'], palette=\"hls\")\nplt.title(\"Distribution of Path\", fontsize=12)\nplt.xticks(rotation=90, fontsize=12)\nplt.ylabel('Count')\nplt.xlabel('Path')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(15, 7))\nax = sns.countplot(lectures_csv['type_of'], palette=\"hls\")\nplt.title(\"Distribution of Path\", fontsize=12)\nplt.xticks(rotation=90, fontsize=12)\nplt.ylabel('Count')\nplt.xlabel('Path')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can compare what is the significant difference in the number of different types of lectures"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}