{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **RIID Answer Correctness Prediction - EDA Summarized data Per User/Tag/Question/Tag Part/ Bundle**"},{"metadata":{},"cell_type":"markdown","source":"#### Ability to predict if a student will answer the question correctly depends primarily on the overall/topic_level ability of the student and also on how 'Simple/Difficult' an individual question, part,  topic/tag. \n\n #### In other words, The external influence for the performance of a student could depend on :\n* 'Simplicity Level' of a Tag . \n* 'Simplicity Level' of the Parts. \n* 'Simplicity Level' of the Bundle.\n* 'Simplicity Level' of the Question. \n\n#### The  internal indicators of the student could be :\n* Past performance of the student.\n* Familiarity/Expertise of the user wrt to the Tag. \n* Has the  user been reading the explanation / lectures. \n* Avg time spent by the user to answer a question. \n\n#### In this Notebook we will try to create and view these summarized data and calculate the correlation to the user performance,. \n"},{"metadata":{},"cell_type":"markdown","source":"*Note: This is memory sensitive notebook. Running all the cells together may result in \"out of memory\" or notebook restart. Run each cell at a time and after a gc operation wait for the memory to stabilize (arounf 5.7GB).*"},{"metadata":{},"cell_type":"markdown","source":"Acknowledgements:\n    \n    https://www.kaggle.com/erikbruin/riiid-comprehensive-eda-baseline\n    https://www.kaggle.com/ilialar/simple-eda-and-baseline\n        "},{"metadata":{},"cell_type":"markdown","source":"Version History\n\n* V1 - Initial Version\n* V2 - Converted 'Difficulty Level' to 'Simplicity Level'. \n* V3 - Added Visualizations, Made changes to the weighted levels "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt \nimport seaborn as sns\nfrom scipy.stats import pearsonr, spearmanr\nimport gc\nimport pyarrow.parquet as pq\nimport pyarrow as pa","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n##\n# Read the data from pickle format. Thanks to BIZEN(https://www.kaggle.com/hiro5299834)\n##\ntrain_df = pd.read_pickle(\"/kaggle/input/riid-pickle-file/train.pkl\")\nprint(\"Train size:\", train_df.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Check for missing/incorrect  data and impute\n##\nsummary = train_df.describe().transpose()\nsummary ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del summary\ngc.collect ()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# There are missing values in prior_question_elapsed_time. For now, replace with median. Also there seems to be outliers in the prior_question_elapsed_time.\n# This could be incorrect values. Replace this also by median. \n##\n\n#train_df['prior_question_elapsed_time'] = train_df['prior_question_elapsed_time'].fillna(train_df['prior_question_elapsed_time'].median)\nval_85_pc = train_df['prior_question_elapsed_time'].quantile(0.85)\nmedian_val = train_df['prior_question_elapsed_time'].median\n# Todo: Out of memory if this line is executed- Need to be looked at later. \n#train_df.loc[train_df['prior_question_elapsed_time'] > val_85_pc,'prior_question_elapsed_time'] = median_val","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del val_85_pc\ndel median_val\ngc.collect ()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## User Level Data Summary"},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Number of unique users in the train data \n##\nlen ( train_df['user_id'].unique() )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Number of unique questions/lectures in the train data \n##\nlen ( train_df['content_id'].unique() )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Mean/Min/Max/Median Number of questions per user. \n##\n\ntemp_df = train_df[train_df['content_type_id'] == 0].groupby(\"user_id\")['content_id'].count()\nprint( \" Mean:\", temp_df.mean() )\nprint( \" Median:\",temp_df.median() )\nprint( \" Min:\", temp_df.min() )\nprint( \" Max:\", temp_df.max() )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Seems like the same question can be posted to the user mutliple times "},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# An example of the same question being posted mutliple times \n##\nuser_id = temp_df.sort_values(ascending=False).index[0]\nuser_level_sorted_data = train_df[train_df['user_id'] == user_id].sort_values(by=\"content_id\")\nuser_level_sorted_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"content id '2' for the user '801103753' is being posted multiple times but with different container ids !!!"},{"metadata":{"trusted":true},"cell_type":"code","source":"del user_level_sorted_data\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Calculate the performance of the user based on the correctness of the answers and the time taken . The value of the elasped time is for the previous \n# question in the bundle. But since we are taking the median, we do not have to recalibrate this one for user summary. \n##\nuser_grouped_df = train_df[train_df['content_type_id']==0].groupby(\"user_id\")\nuser_info = user_grouped_df.agg({\"answered_correctly\":[\"sum\", \"count\"],\"prior_question_elapsed_time\": 'median' })\nuser_info.columns=['total_correct_ans', 'total_q_attempted', 'prior_question_elapsed_time']\nuser_info['performance'] = user_info['total_correct_ans'] / user_info['total_q_attempted']\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del user_grouped_df\ngc.collect ()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Add the number of lectures attended by the user\n##\nuser_lectures = train_df[train_df['content_type_id']==1].groupby(\"user_id\")['content_id'].count()\nuser_info = user_info.join(user_lectures)\nuser_info = user_info.rename (columns = { \"content_id\": \"lecture_count\" } )\nuser_info['lecture_count'] = user_info['lecture_count'].fillna(0).astype(int)\nuser_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# 'Performance' param gives undue prefernce for children who have attempted very few questions. \n# Hence, calculate weighted performance and find the best students \n# TODO: relook at the calculation of the weighted_performance\n##\ncorrect_answers_max = user_info['total_correct_ans'].max()\nuser_info['weighted_performance'] = user_info['performance']*0.5 +  (user_info['total_correct_ans']/correct_answers_max)*0.5\nuser_info_sorted = user_info.sort_values(by='weighted_performance', ascending=False)\nuser_info_sorted","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Remove outlier data \n##\nprint ( \"Users who have attempted less than 10 q:\", user_info[user_info['total_q_attempted'] < 10 ].shape[0])\nprint( \"Users who do have 0 questions answered correct:\", user_info[user_info['performance'] == 0.0 ].shape[0])\nuser_info = user_info[user_info['total_q_attempted'] > 10 ]\nuser_info = user_info[user_info['performance'] != 0.0 ]\nuser_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Is there a correlation between performace and the avg time spent on questions ?\n##\nax = sns.scatterplot ( x=user_info['weighted_performance'], y=user_info['prior_question_elapsed_time']  )\nfrom scipy.stats import pearsonr, spearmanr\nprint ( pearsonr(user_info['weighted_performance'], user_info['prior_question_elapsed_time']) )\nprint (  spearmanr(user_info['weighted_performance'], user_info['prior_question_elapsed_time']) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Is there a correlation between performace and the total questions attempted ?\n##\nax = sns.scatterplot ( x=user_info['performance'], y=user_info['total_q_attempted']  )\nprint ( pearsonr(user_info['performance'], user_info['total_q_attempted']) )\nprint ( spearmanr(user_info['performance'], user_info['total_q_attempted'])) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Is there a correlation between the lecture count and the weighted_performance \n##\n\nax = sns.scatterplot ( x=user_info['weighted_performance'], y=user_info['lecture_count'] )\nprint ( pearsonr(user_info['weighted_performance'], user_info['lecture_count']) )\nprint ( spearmanr(user_info['weighted_performance'], user_info['lecture_count'])) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*There seems to be a good correlation between the number of lectures attended and the performance*"},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Plot the top 10 users based on weighted_performance , Performance  \n##\n\nplt.figure(figsize = (20, 15))\nplt.subplot(1, 2, 1)\nplt.xticks(rotation=45)\nax=sns.barplot(x =user_info_sorted[0:10].index, y=user_info_sorted[0:10]['weighted_performance'], order=user_info_sorted[0:10].index )\n\nplt.subplot(1, 2, 2)\nplt.xticks(rotation=45)\nax=sns.barplot(x =user_info_sorted[0:10].index, y=user_info_sorted[0:10]['performance'], order=user_info_sorted[0:10].index )\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\n\ntable = pa.Table.from_pandas(user_info, preserve_index=True)\npq.write_table(table, '/kaggle/working/user_info.parquet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del user_info\ndel table\ndel temp_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Question Level Data Summary"},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Calculate the difficulty level. Note we are not considering the elapsed_time. Since the time represents the time for the previous row. \n# Hence this cant be consumed as is. \n##\nquestion_grouped_df = train_df[train_df['content_type_id']==0].groupby(\"content_id\")\nquestion_info = question_grouped_df.agg({\"answered_correctly\":[\"sum\", \"count\"]})\nquestion_info.columns=['total_users_correct', 'total_users_attempted']\nquestion_info['question_simplicity_level'] =  question_info['total_users_correct'] / question_info['total_users_attempted']\n\nquestion_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Some questions have no users. Hence calculate the weighted_difficulty_level. Multiply by a percentage of ppl attempted it. \n# TODO: relook at the calculation of the weighted_question_difficulty_level\nmax_q_user_correct = question_info['total_users_correct'].max()\nquestion_info['weighted_question_simplicity_level'] = question_info['question_simplicity_level'] *0.5 + (question_info['total_users_correct']/max_q_user_correct)*0.5\nquestion_info_sorted = question_info.sort_values(by='weighted_question_simplicity_level', ascending=False )\nquestion_info_sorted","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Plot the most difficult questions. i.e the questions that have been coorect for many users \n##\nplt.figure(figsize=(30,20))\nplt.subplot(1,2, 1)\nplt.xticks(rotation=45)\nax=sns.barplot(x =question_info_sorted[0:10].index, y=question_info_sorted[0:10]['weighted_question_simplicity_level'], order=question_info_sorted[0:10].index )\n\nplt.subplot(1,2, 2)\nplt.xticks(rotation=45)\nax=sns.barplot(x =question_info_sorted[0:10].index, y=question_info_sorted[0:10]['question_simplicity_level'], order=question_info_sorted[0:10].index )\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del question_grouped_df\ndel question_info_sorted\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Create columns for each tag and set if the question has the tag.\n##\n\nquestions_tag_info = pd.read_csv(\"/kaggle/input/riiid-test-answer-prediction/questions.csv\")\nquestions_tag_info['tags_new'] = questions_tag_info['tags'].apply( lambda x: str(x).split(\" \"))\n\nfor i in range( 0, 189) :\n    questions_tag_info[str(i)] = 0 \n\nfor idx, row in questions_tag_info.iterrows():\n    tags = row.tags_new\n    for sub_tag in tags:\n        if(sub_tag == 'nan'):\n            continue\n        questions_tag_info.loc[idx, str(sub_tag)]=1\nquestions_tag_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"question_info = pd.merge(question_info, questions_tag_info, how=\"outer\", left_on= \"content_id\", right_on = \"question_id\")\nquestion_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pyarrow.parquet as pq\nimport pyarrow as pa\n\ntable = pa.Table.from_pandas(question_info, preserve_index=True)\npq.write_table(table, '/kaggle/working/question_info.parquet')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del questions_tag_info \ndel table","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tag Summary Data"},{"metadata":{},"cell_type":"markdown","source":"Tags are \" one or more detailed tag codes for the question. The meaning of the tags are not  provided, but these codes are sufficient for clustering the questions together.\". Each tag represents a topic or a subject ."},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Calculate the number of questions per tag\n##\ntag_q_info = pd.DataFrame()\nfor i in  range(0, 188):\n    tag_q_info = tag_q_info.append(pd.DataFrame( {\"count\": question_info[question_info[str(i)]==1]['question_id'].count() }, index = [i] ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Calculate the number of questions answered correctly per tag \n# TODO: This is a time consuming operation - how can we simplify this ?\n##\n\n\nfrom tqdm import tqdm\ntag_info = pd.DataFrame()\n\n\nfor i in tqdm(range( 0, 187)) :\n    q_list_series = question_info[question_info[str(i)] == 1]['question_id']\n    q_list_series.astype(np.int32)\n    temp_df = train_df.join(q_list_series, how='inner', on=\"content_id\")['answered_correctly']\n    tag_info = tag_info.append ( pd.DataFrame({\"answered_correctly_per_tag\": temp_df [ temp_df == 1 ].shape[0],\n                            \"answered_incorrectly_per_tag\": temp_df [ temp_df == 0 ].shape[0],\n                            \"lectures_per_tag\": temp_df [ temp_df == -1 ].shape[0] },\n                           index = [i]) ) \ntag_info['q_count_per_tag'] = tag_q_info['count']\ntag_info['total_answers'] = tag_info['answered_correctly_per_tag'] + tag_info['answered_incorrectly_per_tag']\ntag_info['tag_simplicity_level'] = tag_info['answered_correctly_per_tag'] / tag_info['total_answers']\ntag_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max_tag_ans_corr = tag_info['answered_correctly_per_tag'].max()\ntag_info['weighted_tag_simplicity_level'] = tag_info['tag_simplicity_level'] *0.5 + (tag_info['answered_correctly_per_tag']/max_tag_ans_corr)*0.5\ntag_info_sorted = tag_info.sort_values(by=['weighted_tag_simplicity_level'], ascending = False )\ntag_info_sorted","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Plot the top 10 simple tags\n##\nplt.figure(figsize=(30,20))\n\nplt.subplot(1,2,1)\nplt.xticks(rotation=45)\nax=sns.barplot(x =tag_info_sorted[0:10].index, y=tag_info_sorted[0:10]['weighted_tag_simplicity_level'], order=tag_info_sorted[0:10].index )\n\nplt.subplot(1,2,2)\nplt.xticks(rotation=45)\nax=sns.barplot(x =tag_info_sorted[0:10].index, y=tag_info_sorted[0:10]['tag_simplicity_level'], order=tag_info_sorted[0:10].index )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Is there a correlation between the lecture count and the performance of a tag\n##\nax = sns.scatterplot ( x=tag_info_sorted['weighted_tag_simplicity_level'], y=tag_info_sorted['lectures_per_tag'] )\nprint ( pearsonr(tag_info_sorted['weighted_tag_simplicity_level'], tag_info_sorted['lectures_per_tag']) )\nprint ( spearmanr(tag_info_sorted['weighted_tag_simplicity_level'], tag_info_sorted['lectures_per_tag'])) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Is there a correlation between the q_count_per_tag and the performance of a tag\n##\nax = sns.scatterplot ( x=tag_info_sorted['tag_simplicity_level'], y=tag_info_sorted['q_count_per_tag'] )\nprint ( pearsonr(tag_info_sorted['tag_simplicity_level'], tag_info_sorted['q_count_per_tag']) )\nprint ( spearmanr(tag_info_sorted['tag_simplicity_level'], tag_info_sorted['q_count_per_tag'])) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pyarrow.parquet as pq\nimport pyarrow as pa\n\ntable = pa.Table.from_pandas(tag_info, preserve_index=True)\npq.write_table(table, '/kaggle/working/tag_info.parquet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del tag_q_info\ndel table\ndel temp_df\ndel q_list_series\ndel tag_info_sorted\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Bundle Summary Data "},{"metadata":{},"cell_type":"markdown","source":"#### Bundle represents a group of questions that are presented together. Lets check if they have consequtive timestamps. Lets check if they have similar tag id. "},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Read question and check the number of bundles. \n##\nquestions_df = pd.read_csv(\"/kaggle/input/riiid-test-answer-prediction/questions.csv\")\nquestions_df.set_index('question_id')\ntrain_q_df = train_df.join(questions_df, on=\"content_id\")\ntrain_q_df['bundle_id'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Calculate the number of bundles \n##\ntrain_q_df['bundle_id'].max()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Do the questions in same bundle have same tags ?\n##\nquestions_df.sort_values(['tags', 'bundle_id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Check how the bundles are presented to the user - Is it in consequtive timestamps ? Do they share the same container id ? \n##\ntemp_df = train_q_df[0:100000].groupby(['user_id','task_container_id']).agg({'bundle_id': [\"min\", 'max']} )\ntemp_df=temp_df.fillna(0)\ntemp_df_sorted = temp_df.sort_values('user_id')\ntemp_df_sorted","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[train_df['user_id'] == 115].sort_values(by=\"timestamp\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Question from the same bundle have similar tags. They have a similar timestamp and same 'task container id' per user. \nSince there are too many bundles and not many questions per bundle - this may not be very useful. So, not summarizing this data as of now.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"del questions_df\ndel train_q_df\ndel temp_df_sorted\ndel temp_df\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Part Summary Data "},{"metadata":{},"cell_type":"markdown","source":"#### Part represents the type of question e.g is it reading , listening. etc (https://www.iibc-global.org/english/toeic/test/lr/about/format.html). This is significant since some parts may be easier than the others. "},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Calculate the number of parts and the number questions for each 'part' \n##\nquestions_df = pd.read_csv(\"/kaggle/input/riiid-test-answer-prediction/questions.csv\")\nq_part_grouped = questions_df.groupby(\"part\").count().sort_values(by=\"part\")\nq_part_grouped","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Calculate the number of questions answered correctly per part.\n# Also calculate the number of questions per tag\n# \n##\nfrom tqdm import tqdm\n\n\npart_q_info = pd.DataFrame()\nfor i in  range(1, 8):\n    part_q_info = part_q_info.append(pd.DataFrame( {\"count\": questions_df[questions_df['part']==i]['question_id'].count() }, index = [i] ))\n    \npart_info = pd.DataFrame()\n\n\nfor i in tqdm(range( 1, 8)) :\n    q_list_series = questions_df[questions_df['part'] == i]['question_id']\n    q_list_series.astype(np.int32)\n    temp_df = train_df.join(q_list_series, how='inner', on=\"content_id\")['answered_correctly']\n    part_info = part_info.append ( pd.DataFrame({\"answered_correctly_per_part\": temp_df [ temp_df == 1 ].shape[0],\n                            \"answered_incorrectly_per_part\": temp_df [ temp_df == 0 ].shape[0],\n                            \"lectures_per_part\": temp_df [ temp_df == -1 ].shape[0] },\n                           index = [i]) ) \npart_info['q_count_per_part'] = part_q_info['count']\npart_info['total_answers_per_part'] = part_info['answered_correctly_per_part'] + part_info['answered_incorrectly_per_part']\npart_info['part_simplicity_level'] = part_info['answered_correctly_per_part'] / part_info['total_answers_per_part']\npart_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax=sns.barplot(x=part_info.index, y= part_info['q_count_per_part'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax=sns.barplot(x=part_info.index, y= part_info['part_simplicity_level'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Questions in Part 5 is more questions and have lesser percentage of people answering it correctly."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pyarrow.parquet as pq\nimport pyarrow as pa\n\ntable = pa.Table.from_pandas(part_info, preserve_index=True)\npq.write_table(table, '/kaggle/working/part_info.parquet')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del part_info\ndel part_q_info\ndel table\ndel q_part_grouped\ndel questions_df\ngc.collect () ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Other comparisons"},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Read the summary data \n##\ntable = pq.read_table(\"/kaggle/working/question_info.parquet\")\nquestion_info = table.to_pandas()\nquestion_info_sorted = question_info.sort_values(by=\"weighted_question_simplicity_level\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"table = pq.read_table(\"/kaggle/working/user_info.parquet\")\nuser_info = table.to_pandas()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"##\n# Check if the users who have  attempted the most easier/simpler questions are better off than the ones who have \n## \n\nq_list= question_info_sorted[0:200]['question_id']\nuser_list = train_df[train_df['content_id'].isin(q_list)][['user_id', 'content_id']]\nuser_list = user_list.groupby('user_id').count()\n\n# Get the list of users who have attempted less than 10 , 25, 50 , 75 easier questions \nuser_list_10 = user_list[user_list['content_id']<10].index\nuser_list_25 = user_list[(user_list['content_id']>10) & (user_list['content_id']<25) == True ].index\nuser_list_50 = user_list[(user_list['content_id']>25) & (user_list['content_id']<50) == True ].index\nuser_list_75 = user_list[(user_list['content_id']>50) & (user_list['content_id']<75) == True ].index\nuser_list_100 = user_list[(user_list['content_id']>75) & (user_list['content_id']<100) == True ].index\n\n# Get the mean performance of such users \nperf = np.zeros(5)\nperf[0] = user_info[user_info.index.isin(user_list_10)]['performance'].mean()\nperf[1] = user_info[user_info.index.isin(user_list_25)]['performance'].mean()\nperf[2] = user_info[user_info.index.isin(user_list_50)]['performance'].mean()\nperf[3] = user_info[user_info.index.isin(user_list_75)]['performance'].mean()\nperf[4] = user_info[user_info.index.isin(user_list_100)]['performance'].mean()\n\nx_axis_val = [10, 25, 50, 75, 100]\nax=sns.barplot (x =x_axis_val  , y = perf)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The graph shows that the more the user attempts the easier questions, the higher will be the number of correct answers "}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}