{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## train.csv\n### The variables : \n\n**row_id**: (int64) ID code for the row.\n\n**user_id**: (int32) ID code for the user.\n \n**timestamp**: (int64) the time in milliseconds between this user interaction and the first event completion from that user.
\n**Continuous variable**\n\n \n**user_answer**: (int8)
\n0, 1, 2, 3 if content_type_id == 0. -1 if content_type_id
\n**Categorical variable**\n\n\n**content_id**: (int16) ID code for the user interaction\n\n \n**content_type_id**: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.
\n**Categorical variable**\n\n**task_container_id**: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n**prior_question_elapsed_time**: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.
\n**Continuous variable**\n\n**prior_question_had_explanation**: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.
\n**Boolean variable**\n\n**TARGET**:
\n\n**answered_correctly**: (int8) if the user responded correctly. Read -1 as null, for lectures.
\n\n\n\n=> We want to predict if the user will answer correctly or not.
\nWe will exclure the value \"-1\" which corresponds to the lecture, not the answer.
\n**So we have a binary classification problem to solve.**","metadata":{}},{"cell_type":"markdown","source":"## Import librairies","metadata":{}},{"cell_type":"code","source":"import gc\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport os\nimport pandas as pd\nimport seaborn as sns\nimport time\nimport nltk\n\nfrom sklearn.metrics import roc_auc_score\nfrom tqdm.notebook import tqdm","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Functions","metadata":{}},{"cell_type":"code","source":"def print_pie(data_df, column, title_fig, title_legend) :\n \"\"\"\n Print a pie chart. \n \"\"\"\n fig, ax = plt.subplots(figsize=(6, 6))\n ttl = plt.title(title_fig, fontsize=15, weight=\"bold\")\n ttl.set_position([0, 1.05])\n data_df[column].value_counts(normalize=True).sort_index().plot(kind='pie',\n startangle=180, \n counterclock=False, \n autopct='%1.1f%%', \n fontsize = 14)\n plt.axis('equal')\n plt.ylabel('')\n plt.rcParams['legend.title_fontsize'] = 'large'\n ax.legend(title=title_legend, loc=\"center right\",\n bbox_to_anchor=(1, 0, 1, 1), fontsize='medium')\n plt.show()\n\n \ndef print_filling_rate (data_df, a_label_abscisse, color_graphe, color_threshold ) : \n \"\"\" \n Display the filling rate of the columns\n @donnees_df : dataframe qui contient les données\n @a_seuil : booléen égal à True si on souhaite afficher le seuil\n @a_label_abscisse : booléen égal à True si on souhaite afficher \n le nom des colonnes en abscisse\n @color_graphe\n @color_threshold \n \"\"\"\n if(color_graphe == ''):\n color_graphe = 'blue'\n data = (data_df.count() / len(data_df)).sort_values().values\n ind = np.arange(len(data))\n width = 0.5\n fig, axes = plt.subplots(1, 1, figsize=(6, 3), dpi=100)\n tr = axes.bar(ind, data, width, color=color_graphe)\n axes.set_ylabel('Filling rates');\n if(a_label_abscisse):\n axes.set_xticks(ind )\n axes.set_xticklabels((data_df.count() / len(data_df)).\\\n sort_values().index, fontsize=10, rotation=90)\n axes.legend([tr], ['Filling rates'])\n \ndef plot_distribution_of_number_of_tags(nb_tags, with_return):\n tags_most_common = tags_frequence.most_common(nb_tags)\n tags_df = pd.DataFrame(tags_most_common, columns=['tags' , 'nb_tags']) \n tags_df.columns\n tags_sorted_df = tags_df.sort_values(['nb_tags'], ascending=False)\n tags_counts = tags_sorted_df['nb_tags'].values\n plt.plot(tags_counts)\n plt.title(\"Distribution of number of tags\")\n plt.grid()\n plt.xlabel(\"Number of tags\")\n plt.ylabel(\"Number of occurences\")\n plt.show()\n if(with_return):\n return tags_df","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading data","metadata":{}},{"cell_type":"code","source":"%%time\n\ndtypes = {\n \"row_id\":\"int64\",\n \"timestamp\":\"int64\",\n \"user_id\":\"int32\",\n \"content_id\":\"int16\",\n \"content_type_id\":\"int8\",\n \"task_container_id\":\"int16\",\n \"user_answer\":\"int8\",\n \"answered_correctly\":\"int8\",\n \"prior_question_elapsed_time\":\"float32\", \n \"prior_question_had_explanation\":\"boolean\"\n}\n\ntrain_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', \n low_memory=False, \n nrows=10**6, \n dtype=dtypes\n )\nprint(\"Train size:\", train_df.shape)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.memory_usage(deep=True)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.describe(include='all')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_df.isnull().sum())\nprint(\"****************************************\")\nprint(train_df.isnull().sum() / len(train_df))\n\nprint_filling_rate (train_df, True, \"blue\", \"blue\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Distribution of questions and lectures content_type :","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(5, 5))\nplt.title('Percentage of content type : questions or lectures', fontsize=15, weight=\"bold\")\nlabels = ['Questions','Lectures']\ncolors = ['lightgray','peachpuff']\nexplode = (0.1,0.2 )\ntrain_df[\"content_type_id\"].value_counts(normalize=True).plot(kind='pie',\n labels=labels,\n colors=colors,\n explode=explode,\n startangle=50,\n autopct='%1.1f%%',\n fontsize=13)\nplt.axis('equal') \nplt.ylabel('')\nplt.show()\n\nprint(train_df['content_type_id'].value_counts().sort_index().to_frame())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. The majority of the users interactions are **questions** : 98% VS 2% for lectures.","metadata":{}},{"cell_type":"code","source":"%%time\n\nquestions_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Merge of train_df and questions_df","metadata":{}},{"cell_type":"code","source":"train_questions_df = pd.merge(train_df[train_df['content_type_id']==0],\n questions_df, \n how='left', \n left_on='content_id', \n right_on='question_id').sort_values('row_id')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_questions_df.columns","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_questions_df.describe()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"The dataset contains {} rows and {} columns. \\n\".\\\n format(train_questions_df.shape[0], train_questions_df.shape[0]))\nfor col in train_questions_df:\n print(\"The column {} has {} unique values.\".\\\n format(col, train_questions_df[col].nunique()))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Users\n\n3 824 unique users","metadata":{}},{"cell_type":"code","source":"nb_questions_per_user = train_questions_df[['user_id','row_id']].\\\n groupby(['user_id'],as_index=False).\\\n agg(['count']).reset_index()\n\nnb_questions_per_user.columns = [\"user_id\", \"nb_questions\"]\n\nprint(\"Mean : {}\".format(nb_questions_per_user[\"nb_questions\"].mean()))\nprint(\"Min : {}\".format(nb_questions_per_user[\"nb_questions\"].min()))\nprint(\"Max : {}\".format(nb_questions_per_user[\"nb_questions\"].max()))\nprint(\"Median : {}\".format(nb_questions_per_user[\"nb_questions\"].median()))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nMedian is 41. 50% of the users has answer at 41 or less questions.\n\nMax = 10797. Mean (=10797) is sensible at the outliers.\n\nLet have a look at the distribution of the number of question by user :","metadata":{}},{"cell_type":"code","source":"fig = plt.figure(figsize=(12,6))\nnb_questions_per_user['nb_questions'].plot.hist(bins=100)\nplt.title(\"Distribution of the number of questions by user\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Number of questions by user\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(12,6))\nnb_questions_per_user[nb_questions_per_user['nb_questions']<200]['nb_questions'].plot.hist(bins=100)\nplt.title(\"Distribution of the number of questions by user (max 1000 questions)\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Number of questions by user\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del nb_questions_per_user\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check if we have the same result if we calculate if the answer is correct and the target\nuser_accuracy_df = train_questions_df[['user_id', 'user_answer', 'correct_answer','answered_correctly']]\n\nuser_accuracy_df['answered_correctly_calc'] = user_accuracy_df['user_answer'] -\\\n user_accuracy_df['correct_answer']\n\nuser_accuracy_df['different_result'] = 1\nuser_accuracy_df.loc[user_accuracy_df['answered_correctly_calc'] != 0, 'different_result'] = 0\nuser_accuracy_df['is_different_result'] = user_accuracy_df['different_result'] -\\\n user_accuracy_df['answered_correctly']\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(user_accuracy_df['is_different_result'].sum())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\nThe columns 'user_answer' is just here for information, maybe we can find a pattern : when the user doesn't know the answer, he uses to choose the same answer. Or maybe the question has a subtlety and students with a lower level do the same error, that can help to detect a misunderstood and propose a solution to help students, ...","metadata":{}},{"cell_type":"code","source":"print_pie(user_accuracy_df, 'user_answer', \"Percentage of user answers:\", \"User answers: \")\nprint(user_accuracy_df.user_answer.value_counts())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The user answers are 0, 1, 2 or 3. But what means 0 : is it a possible answer, or does it mean \"no answer\" ? \nLet have a look at the possible answers (in questions_df) :","metadata":{}},{"cell_type":"code","source":"print_pie(questions_df, 'correct_answer', \n \"Percentage of possible answers:\", \n \"Possible answers: \")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Ok, 0 is a possible answer.","metadata":{}},{"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(5, 5))\nplt.title('Percentage of questions correctly answered:', fontsize=15, weight=\"bold\")\nlabels = ['Correct', 'Uncorrect']\ncolors = ['lightgray', 'peachpuff']\nexplode=(0.1,0)\nuser_accuracy_df[\"answered_correctly\"].\\\n value_counts(normalize=True).\\\n plot(kind='pie', labels=labels, colors=colors, explode=explode,\n startangle=50, autopct='%1.1f%%', fontsize = 13)\nplt.axis('equal') \nplt.ylabel('')\nplt.show()\n\nprint(user_accuracy_df['answered_correctly'].value_counts().sort_index().to_frame())","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### The part of questions : ","metadata":{}},{"cell_type":"code","source":"answer_part_df = train_questions_df[['part', 'answered_correctly']].\\\n groupby(['part'],as_index=False).\\\n agg(['mean']).reset_index()\nanswer_part_df.columns = [\"part\", 'answered_correctly_mean']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"answer_part_df","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Toeic has 2 sections : Listening (part 1 -> 4) and Reading (part 5 -> 7)\nThe section listening seems to be easy than the reading section.\nParts 1-> 3 are the easiest parts. Parts 4 and 5 the more difficult.","metadata":{}},{"cell_type":"code","source":"del answer_part_df\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"answer_nb_bundle_by_part_df = questions_df[['part', 'bundle_id']].\\\n groupby(['part'], as_index=False).\\\n agg(['count']).reset_index()\nanswer_nb_bundle_by_part_df.columns = [\"part\", 'nb_bundle']\nanswer_nb_bundle_by_part_df","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have between 992 and 5511 possible bundles of questions by part.\n","metadata":{}},{"cell_type":"code","source":"answer_nb_part_by_bundly_df = questions_df[['bundle_id', 'part']]\nanswer_nb_part_by_bundly_df.drop_duplicates(keep='first', inplace=True)\n\nresult_nb_part_by_bundle = answer_nb_part_by_bundly_df.\\\n groupby(['bundle_id'],as_index=False).agg(['count']).reset_index()\nresult_nb_part_by_bundle.columns = [\"bundle_id\", 'nb_part']\nresult_nb_part_by_bundle.describe()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"One part can have several bundles.\nBut one bundle belongs to only one part.","metadata":{}},{"cell_type":"code","source":"del answer_nb_part_by_bundly_df\ndel result_nb_part_by_bundle\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"answer_bundle_df = train_questions_df[['bundle_id', 'part', 'answered_correctly']].\\\n groupby(['bundle_id', 'part'],as_index=False).\\\n agg(['mean']).reset_index()\nanswer_bundle_df.columns = [\"bundle_id\", \"part\", 'answered_correctly_mean']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(12,6))\nanswer_bundle_df['answered_correctly_mean'].plot.hist(bins=100)\nplt.title(\"Distribution of mean of correct answers by bundle\")\nplt.xticks(rotation=0)\nplt.xlabel(\"mean of correct answer by bundle\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\n\nplt.hist(answer_bundle_df[answer_bundle_df['part']<4]['answered_correctly_mean'], \n bins=50, color=\"blue\", \n alpha=0.5, label='Bundle in part 1-3')\nplt.hist(answer_bundle_df[(answer_bundle_df['part']>=4) & \\\n (answer_bundle_df['part']<6)]['answered_correctly_mean'], \n bins=50, color=\"red\", alpha=0.25, \n label='Bundle in part 4-5')\nplt.legend(loc='upper left')\n\nplt.title('Distribution of mean of correctly answered questions by group of bundle')\nplt.grid()\n\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del answer_bundle_df\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## timestamp","metadata":{}},{"cell_type":"code","source":"user_timestamp_df = train_df[['user_id', 'timestamp']].\\\n groupby(['user_id'],as_index = False).\\\n agg(['max','min','mean']).reset_index()\nuser_timestamp_df.columns = [\"user_id\", 'max_timestamp', 'min_timestamp', 'mean_timestamp']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"user_timestamp_df.describe()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The min timestamp is 0. So we should have the history of interactions from the first interaction.\n\n'timestamp' is in milliseconds. It is difficult for human to have an idea about how long are this periods with this unity. We will convert it in month and observe if it easier to represent this periods.","metadata":{}},{"cell_type":"code","source":"train_df['timestamp_by_month'] = train_df['timestamp'] / (1000 * 60 * 60 * 24 * 365) * 12\nfig = plt.figure(figsize=(12,6))\ntrain_df['timestamp_by_month'].plot.hist(bins=100)\nplt.title(\"Histogram of timestamp converted in month\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Months between this user interaction and the first event completion from that user\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(12,6))\ntrain_df[(train_df['timestamp_by_month'] >= 1) &\\\n (train_df['timestamp_by_month'] < 6)]['timestamp_by_month'].plot.hist(bins=100)\nplt.title(\"Histogram of timestamp converted in month\\\n between 1 month and 6 months\")\nplt.xticks(rotation=0)\nplt.xlabel(\"Months between this user interaction and\\\n the first event completion from that user\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"user_timestamp_df['max_timestamp_in month'] = user_timestamp_df['max_timestamp'] / \\\n (1000 * 60 * 60 * 24 * 365) * 12","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\nfig = plt.figure(figsize=(12, 6))\n(user_timestamp_df[\"max_timestamp\"] / 1000 / 60 / 60 / 24 / 365 * 12).plot.hist(bins=100)\nplt.title(\"Histogram of max timestamp of users\")\nplt.xticks(rotation=0)\nplt.xlabel(\"max timestamp of users\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Questions","metadata":{}},{"cell_type":"code","source":"questions_df.info()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"questions_df.describe()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(questions_df.isnull().sum())\nprint(\"****************************************\")\nprint(questions_df.isnull().sum() / len(train_df))\n\nprint_filling_rate (questions_df, True, \"green\", \"green\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Only 1 row of \"tags\" has missing value.","metadata":{}},{"cell_type":"markdown","source":"## Tags","metadata":{}},{"cell_type":"code","source":"questions_df['tags'].fillna(\"\", inplace=True)\nquestions_df[\"nb_tags\"] = questions_df[\"tags\"].apply(lambda text: len(text.split()))\n\nquestions_df[\"nb_tags\"].describe()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.style as style\nstyle.use('seaborn-dark-palette')\nfig, ax = plt.subplots(figsize=(6, 4))\nax.set_title(\"Distribution of number of tags by question\", \n fontsize=15, weight=\"bold\");\n\nsns.countplot(questions_df['nb_tags'], palette=\"Set1\")\nax.set_ylabel(\"Frequency\", fontsize=14)\nax.set_xlabel(\"Number of tags by question\", fontsize=14);","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Create 2 lists :\n\nwith all tags of all questions (a same tag can appear several times in this list)\nwith unique tags. A same tag can appear only one time).","metadata":{}},{"cell_type":"code","source":"questions_df['tags_list'] = questions_df['tags'].apply(lambda x: x.split())\n\n# List with all tags \ntags_list = [item for sublist in questions_df['tags_list'].values for item in sublist]\nprint('{} tags are used in questions.'.format(len(tags_list)))\n\n# List of unique tags\ntags_unique_list = list(set(tags_list))\nprint('There are {} unique tags.'.format(len(tags_unique_list)))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tags_frequence= nltk.FreqDist(tags_list)\n\nplot_distribution_of_number_of_tags(len(tags_unique_list), False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_distribution_of_number_of_tags(25, False)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"words_most_common = tags_frequence.most_common(10)\nfq_words_df = pd.DataFrame(words_most_common, columns = ['tags' , 'nb_tags']) \nprint((fq_words_df['tags'][:10]).to_list())\n\ndel fq_words_df\ngc.collect()","metadata":{},"execution_count":null,"outputs":[]}]}