{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Cleaning users 💥 Features engineering\n\nHello Kagglers**** 🖐🖐 \n\nHere is my approach ⏬⏬⏬"},{"metadata":{},"cell_type":"markdown","source":"# 1. Import of training data and preliminary exploration"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nimport seaborn as sns\nimport matplotlib\nimport matplotlib.pyplot as plt\nimport squarify\nimport matplotlib.pyplot as plt\nplt.style.use(\"default\")\nfrom IPython.display import Image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def convert_from_ms(milliseconds):\n    seconds, milliseconds = divmod(milliseconds, 1000)\n    minutes, seconds = divmod(seconds, 60)\n    hours, minutes = divmod(minutes, 60)\n    days, hours = divmod(hours, 24)\n    seconds = seconds + milliseconds / 1000\n    return days, hours, minutes, round(seconds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I am using the dataset compressed in this [kernel](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets/)\n\nThanks to Rohan ! 😉"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\npath = \"../input/riiid-train-data-multiple-formats/riiid_train.pkl.gzip\"\ntrain = pd.read_pickle(path)\ntrain = train.astype({'row_id': 'int64',\n                      'timestamp': 'int64',\n                      'user_id': 'int32',\n                      'content_id': 'int16',\n                      'content_type_id': 'int8',\n                      'task_container_id': 'int16',\n                      'user_answer': 'int8',\n                      'answered_correctly': 'int8',\n                      'prior_question_elapsed_time': 'float32',\n                      'prior_question_had_explanation': 'boolean'})\n\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isna().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training data set is substantial; more than 100 million lines 😲😲 It is necessary to conduct a deep exploration and possibly a deep cleansing. I focus the cleaning on the users.\n\n💔 Do not remove interactions no matter how 💔"},{"metadata":{},"cell_type":"markdown","source":"# 2. Cleaning user\n## 2-1. Building the user dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_user = train.user_id.nunique()\nprint(f'There are {n_user} users')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I count the number of interactions for a user and i recover the number of days in the targeted interaction and the first. I calculate the score 💯 for informations"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nagg = {'row_id' : 'count',\n       'timestamp' : lambda x: convert_from_ms(x.max())[0],\n       'answered_correctly' : lambda x: round(x.mean() * 100)}\n\nuser_info = train.groupby('user_id').agg(agg)\nuser_info = user_info.rename(columns={'row_id' : 'nb_interactions',\n                                   'timestamp' : 'nb_jours',\n                                   'answered_correctly' : 'score'})\nuser_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_info.describe(percentiles=[.05, .25, .5, .75, 0.95])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfig = sns.pairplot(user_info, diag_kind=\"kde\", plot_kws={'alpha': 0.01})\nfig.savefig('./img_eda_user_pairplot.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(user_info.nb_interactions, hist=False)\nplt.xscale('log')\nplt.title('Nb interactions with log scale')\nplt.savefig('./img_eda_user_nb_interactions_log_scale.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(user_info.nb_jours, hist=False)\nplt.xscale('log')\nplt.title('Nb days with log scale')\nplt.savefig('./img_eda_user_nb_days_log_scale.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2-2. Number of interactions per user"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, (ax0, ax1) = plt.subplots(nrows=1,ncols=2, sharey=False, figsize=(7,4))\nsns.boxplot(y=user_info['nb_interactions'], ax=ax0)\nsns.boxplot(y=user_info['nb_interactions'], ax=ax1)\nax0.set(title=\"Number of interactions per user\")\nax1.set(ylim=(0,500), title=\"Focus on the box\", ylabel=\"\")\nfig.savefig('./img_eda_interaction_boxplot.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here I see big differences especially from the 95th percentile 💥💥 It is difficult for me to be able to generalize a model from these data, it must be excluded!"},{"metadata":{},"cell_type":"markdown","source":"## 2-3. Number of days between the first interaction and the last"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, (ax0, ax1) = plt.subplots(nrows=1,ncols=2, sharey=False, figsize=(7,4))\nsns.boxplot(y=user_info['nb_jours'], ax=ax0)\nsns.boxplot(y=user_info['nb_jours'], ax=ax1)\nax0.set(title=\"Number of days per user\")\nax1.set(ylim=(0,150), title=\"Focus on the box\", ylabel=\"\")\nfig.savefig('./img_eda_jour_boxplot.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Here, too, it is advisable to clean ✅"},{"metadata":{},"cell_type":"markdown","source":"## 2-4. Apply filters"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Régler ici les seuils de nettoyage\nQ1 = 0.05\nQ4 = 0.95\n\n# Filtre des interaction exceptionnelles\nq1_int = user_info.nb_interactions.quantile(Q1)\nq4_int = user_info.nb_interactions.quantile(Q4)\ncond_int = (user_info.nb_interactions > q1_int) & (user_info.nb_interactions < q4_int)\n\n# Filtre des délais exceptionnels\nq1_delay = user_info.nb_jours.quantile(Q1)\nq4_delay = user_info.nb_jours.quantile(Q4)\ncond_delay = (user_info.nb_jours > q1_delay) & (user_info.nb_jours < q4_delay)\n\nreduced_user_info = user_info[cond_int & cond_delay]\nreduced_user_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduced_n_user = len(reduced_user_info)\n\nreduced_user_rate = round((1 - reduced_n_user / n_user) * 100)\nprint(f'Users have been reduced by {reduced_user_rate} %.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfig = sns.pairplot(reduced_user_info,  diag_kind=\"kde\", plot_kws={'alpha': 0.01})\nfig.savefig('./img_eda_reduced_user_pairplot.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Distributions are less compressed ❗ I apply cleansing to the train data."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(reduced_user_info.nb_interactions, hist=False)\nplt.xscale('log')\nplt.title('Nb interactions with log scale')\nplt.savefig('./img_eda_reduced_user_nb_interactions_log_scale.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(reduced_user_info.nb_jours, hist=False)\nplt.xscale('log')\nplt.title('Nb days with log scale')\nplt.savefig('./img_eda_reduced_user_nb_days_log_scale.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduced_train = pd.merge(train, reduced_user_info, on='user_id')\n\ncol = train.columns\nreduced_train = reduced_train[col]\nreduced_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduced_train_rate = round((1 - len(reduced_train) / len(train)) * 100)\nprint(f'Interactions were reduced by {reduced_train_rate} %.')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"reduced_train.to_pickle(\"./reduced_riiid_train.pkl.gzip\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. Features engineering\n## 3-1. Help usage\n\nSome of the interactions are considered as conferences, no response is required from the user (answered_correctly = -1). I remove them from the data set.\n\nBefore, it is interesting to extract a feature on the level of use of the help."},{"metadata":{"trusted":true},"cell_type":"code","source":"help_usage = reduced_train[reduced_train.answered_correctly == -1].groupby('user_id')['content_id'].count()\nhelp_usage = help_usage.reset_index().rename(columns={'content_id': 'help_usage'})\nhelp_usage","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"help_usage.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.boxplot(y='help_usage', data=help_usage)\nplt.title('Help usage of users')\nplt.savefig('./img_eda_help_usage.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"help_usage['binned_help_usage'] = pd.cut(help_usage['help_usage'], bins=[-np.inf, 0, 1, 3, np.inf], labels=[0, 1, 2, 3])\nhelp_usage","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Export file for submissions\ncol = ['user_id', 'binned_help_usage']\nhelp_usage = help_usage[col]\nhelp_usage[col].to_csv('./help.csv', index=False)\n\n# Delete conferences\nreduced_train = reduced_train[reduced_train.content_type_id == 0]\nreduced_train.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3-2. Content difficult\n\n\nThe TOEIC exam is classified into 2 parts; listening and reading. Each part has difficulty groups, it is interesting to extract this information."},{"metadata":{"trusted":true},"cell_type":"code","source":"path = \"../input/riiid-test-answer-prediction/questions.csv\"\nquestions = pd.read_csv(path)\nquestions.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"col = ['question_id','part']\nquestions = questions[col]\nquestions = questions.rename(columns={'question_id': 'content_id'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Listening (0) or reading (1)\nquestions['L | R'] = pd.cut(questions['part'], bins=[-np.inf, 4, np.inf], labels=['Listening', 'Reading'])\nquestions.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions['Difficulty_level'] = 0\nquestions['Difficulty_level'][questions['L | R'] == 'Listening'] = questions['part'][questions['L | R'] == 'Listening']\nquestions['Difficulty_level'][questions['L | R'] == 'Reading'] = questions['part'][questions['L | R'] == 'Reading'] - 4\n\nquestions.sample(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.to_csv('./level_content.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3-3. New features exploration"},{"metadata":{"trusted":true},"cell_type":"code","source":"featured_train = pd.merge(reduced_train, help_usage, on='user_id', how='left')\nfeatured_train = pd.merge(featured_train, questions, on='content_id', how='left')\n# Users with no help usage equals 0\nfeatured_train.binned_help_usage = featured_train.binned_help_usage.fillna(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Parts"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\nagg = {'row_id' : 'count',\n       'answered_correctly' : lambda x: round(x.mean() * 100),\n       'prior_question_had_explanation' : lambda x: round(x.mean() * 100)}\n\npart_info = featured_train.groupby('part').agg(agg)\npart_info = part_info.rename(columns={'row_id' : 'nb_interactions',\n                                   'answered_correctly' : 'score'})\npart_info","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"norm = matplotlib.colors.Normalize(vmin=min(part_info.score), vmax=max(part_info.score))\ncolors = [matplotlib.cm.Blues(norm(value)) for value in part_info.score]\nsquarify.plot(sizes=part_info.nb_interactions, color=colors, label=part_info.index, alpha=0.8)\nplt.title(\"Part of TOEIC test\")\nplt.axis('off')\nplt.savefig('./img_part_toeic_test_repartitions.png', transparent=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The weaker the colors, the lower the score.\nPart 5 is the most worked on in the data, but is the one with the lowest score.\nThis is also where users require the most explanation."},{"metadata":{},"cell_type":"markdown","source":"### Content"},{"metadata":{"trusted":true},"cell_type":"code","source":"featured_train.groupby(\"L | R\")['row_id'].count().reset_index().plot.bar(x='L | R', y='row_id')\nplt.title('Interactions of Listening or Reading')\nplt.savefig('./img_part_LR_repartitions.png',\n            transparent=True,\n            bbox_inches=\"tight\")\n#sns.barplot(y=, data=featured_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Level usage"},{"metadata":{"trusted":true},"cell_type":"code","source":"featured_train.groupby('binned_help_usage')['row_id'].count().reset_index().plot.bar(x='binned_help_usage', y='row_id')\nplt.title('Users by help usage group ')\nplt.savefig('./img_help_usage_repartitions.png', transparent=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Reduced train dataset export"},{"metadata":{"trusted":true},"cell_type":"code","source":"col = ['timestamp', 'user_id', 'content_id',\n       'task_container_id', 'binned_help_usage', 'L | R', 'Difficulty_level', 'answered_correctly']\nfeatured_train = featured_train[col]\nfeatured_train.sample(20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"featured_train.to_pickle(\"./featured_riiid_train.pkl.gzip\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Baseline Coming soon"},{"metadata":{},"cell_type":"markdown","source":"If you made it to the point thank you for reading and 🆙 vote if it helped you. \nI will read your comments with pleasure.\n\nOooh ! And i'm soory for my perfectible english !\n\nThanks 😷 Stay at home 😷"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}