{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis using Dask\n\nHi Kagglers, this is the first notebook I'm sharing on Kaggle. I've used dask dataframes to keep the memory usage low. This notebook is just for EDA. Let me know if it helped you in any way or if you have suggestions for better code, presentation, etc. Thanks!"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\nimport dask\nimport dask.dataframe as dd\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom itertools import chain","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data loading\nLoad a sample of the data using pandas"},{"metadata":{"trusted":true},"cell_type":"code","source":"path_append = '/kaggle/input/riiid-test-answer-prediction/'\ntrain_data = pd.read_csv(path_append + 'train.csv', nrows=1000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data loading using dask"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load data using dask\n\ntrain_data_dd = dd.read_csv(path_append + \"train.csv\", low_memory=False) # Lazy evaluation - doesn't actually load until .compute() is called","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Calling `dask.compute()` with multiple objects allows for shared computation steps, e.g., file loading, and reduces overall runtime"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calling dask.compute() with multiple objects allows for shared computation steps, e.g., file loading, and reduces overall runtime\nnum_content_ids, num_task_ids, num_content_types = dask.compute(train_data_dd['content_id'].nunique(),\\\n                                                               train_data_dd['task_container_id'].nunique(),\\\n                                                               train_data_dd['content_type_id'].nunique())\nprint(\"Unique content IDs: {}\".format(num_content_ids))\nprint(\"Unique task container IDs: {}\".format(num_task_ids))\nprint(\"Number of content types: {}\".format(num_content_types))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# How many users in total?\n\nprint(\"Total number of users: {}\".format(train_data_dd['user_id'].nunique().compute()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# What proportion of all answers are correct?\n\nprint(\"Overall answer correctness rate: {:.2f}%\".format(train_data_dd[train_data_dd['content_type_id']==0]['answered_correctly'].mean().compute() * 100))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analysis of questions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Aggregating at the content (question) level\ncontent_df = (train_data_dd.query(\"content_type_id==0\")\n              .groupby('content_id')\n              .agg({'user_id': 'count',\n                    'answered_correctly': 'mean'})\n              .compute())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load questions data"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df = pd.read_csv(path_append + 'questions.csv')\nquestions_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# How many unique 'parts'?\nprint(\"# unique parts: {}\".format(questions_df['part'].nunique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Merge the questions dataframe with the aggregated content dataframe \ncontent_df = (content_df.reset_index().rename(columns={\"content_id\": \"question_id\"})\n              .merge(questions_df, how='left', on=['question_id']))\ncontent_df['num_answered_correctly'] = (content_df['user_id'] * content_df['answered_correctly']).astype(int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Are all questions attempted equally often?"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(content_df['user_id'].sort_values(ascending=False).values)\nplt.yscale(\"log\")\nplt.title(\"Questions vs number of attempts\")\nplt.xlabel(\"Questions\")\nplt.ylabel(\"Number of attempts\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Are some questions harder than others?"},{"metadata":{"trusted":true},"cell_type":"code","source":"content_df[content_df['user_id']>100]['answered_correctly'].hist(bins=100)\nplt.title(\"Distribution of answer correctness rate by questions\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of time elapsed on previous question"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data_dd['prior_question_elapsed_time'].compute().hist(bins=100)\nplt.title(\"Distribution of time elapsed on previous question\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How well does time elapsed on previous question predict answer correctness?"},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_question_qcut = train_data_dd['prior_question_elapsed_time'].map_partitions(pd.qcut, 10, labels=False,\\\n                                                                                   meta=train_data_dd['prior_question_elapsed_time'])\npqet_df = (train_data_dd.groupby(prior_question_qcut).agg({'answered_correctly': 'mean'}).compute()\n           .reset_index().rename(columns={'prior_question_elapsed_time': 'prior_question_elapsed_time_decile'}))\nsns.barplot(x='prior_question_elapsed_time_decile', y='answered_correctly', data=pqet_df, hue=None)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Prior question had explanation - does this predict answer correctness?"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data_dd.groupby('prior_question_had_explanation').agg({'user_id': 'count', 'answered_correctly': 'mean'}).compute()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Does the section of the test (column: 'part') predict answer correctness?"},{"metadata":{"trusted":true},"cell_type":"code","source":"part_agg = content_df.groupby('part', as_index=False).agg({'user_id': 'sum', 'num_answered_correctly': 'sum'})\npart_agg['prop_correct'] = part_agg['num_answered_correctly'] / part_agg['user_id']\npart_agg","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.barplot(x='part', y='prop_correct', data=part_agg, hue=None)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Analysis of tags"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Convert the 'tags' string to a list\ncontent_df['tags'].fillna('', inplace=True)\ncontent_df['tags_list'] = content_df['tags'].apply(lambda x: [int(t) for t in x.split()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tags_df = content_df.apply(lambda x: [(t, x['user_id'], x['num_answered_correctly']) for t in x['tags_list']],axis=1).values\ntags_df = chain.from_iterable(tags_df)\ntags_df = pd.DataFrame(tags_df, columns=['tag', 'num_questions', 'num_answered_correctly'])\ntags_df = tags_df.groupby('tag', as_index=False).sum()\ntags_df['prop_correct'] = tags_df['num_answered_correctly'] / tags_df['num_questions']\ntags_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tags_df['prop_correct'].hist(bins=20)\nplt.title(\"Distribution of correctness rate by tag\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analysis of users"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_df = (train_data_dd.query(\"content_type_id==0\")\n           .groupby('user_id')\n           .agg({'user_answer': 'count', 'answered_correctly': 'mean', 'timestamp': 'max'})\n           .rename(columns={'user_answer': 'num_questions_answered', 'timestamp': 'total_time_spent'})).compute()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_df['total_time_spent_mins'] = user_df['total_time_spent'] / 60000.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of number of questions answered by each user"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_df[user_df['num_questions_answered'] < 2000]['num_questions_answered'].hist(bins=100)\nplt.title(\"Distribution of # questions answered by users\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of total time spent"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_df['total_time_spent_mins'].hist(bins=100)\nplt.title(\"Distribution of # questions answered by users\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Closer look at users with millions of minutes (long-time users)\n\noutlier_user_ids = train_data_dd[train_data_dd['timestamp'] > 1e6 * 60000][['user_id']].drop_duplicates()\noutlier_user_df = outlier_user_ids.merge(train_data_dd, how='inner', on='user_id').compute()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"outlier_user_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_user = outlier_user_df[outlier_user_df['user_id']==np.random.choice(outlier_user_df['user_id'].unique())].copy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_user['timestamp_mins'] = sample_user['timestamp'] / 60000\nsample_user['timestamp_hrs'] = sample_user['timestamp_mins'] / 60","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.plot(sample_user['timestamp_hrs'].values)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"sample_user.head(50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Distribution of user ability - are some users correct more often than others?"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_df[user_df['num_questions_answered']>10]['answered_correctly'].hist(bins=50)\nplt.title(\"Distribution of answer correctness by users\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Analysis of lectures"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df = pd.read_csv(path_append + 'lectures.csv')\nlectures_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df['type_of'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df['part'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To do: does viewing a lecture affect performance on the next set of questions? (feeling too lazy to do it now.....)"},{"metadata":{},"cell_type":"markdown","source":"# That's all, folks!"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}