{"cells":[{"metadata":{},"cell_type":"markdown","source":"# load packages "},{"metadata":{"trusted":true},"cell_type":"code","source":"pip install seaborn==0.11.0","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#import packages\nimport pandas as pd\nimport seaborn as sns\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import cross_val_score, GridSearchCV, train_test_split\nfrom sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier\nfrom sklearn.metrics import confusion_matrix\nimport riiideducation","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pip list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# load data and prepare environment"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',nrows=10000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions = pd.read_csv(\"/kaggle/input/riiid-test-answer-prediction/questions.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data information copied from the Riiid! Case competition site at https://www.kaggle.com/c/riiid-test-answer-prediction/data\n\n### train Data\n\n<b>row_id: </b> (int64) ID code for the row.\n\n<b>timestamp: </b>(int64) the time in milliseconds between this user interaction and the first event completion from that user.\n\n<b>user_id: </b> (int32) ID code for the user.\n\n<b> content_id:  </b> (int16) ID code for the user interaction\n\n<b>content_type_id:</b>(int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n<b>task_container_id:</b> (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n<b>user_answer: </b>(int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\n<b>answered_correctly: </b> (int8) if the user responded correctly. Read -1 as null, for lectures.\n\n<b>prior_question_elapsed_time:  </b>(float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n\n<b>prior_question_had_explanation: </b> (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback.\n\n\n### Questions : \n\n<b> questions.csv:</b> metadata for the questions posed to users.\n\n<b> question_id: </b>foreign key for the train/test content_id column, when the content type is question (0).\n\n<b> bundle_id:</b> code for which questions are served together.\n\n<b> correct_answer:</b> the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n<b> part:</b> the relevant section of the TOEIC test.\n\n<b> tags:</b> one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together.\n\n### Lectures: \n\n<b>lecture_id: </b>foreign key for the train/test content_id column, when the content type is lecture (1).\n\n<b>part:</b> top level category code for the lecture.\n\n<b>tag:</b> one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n<b>type_of:</b> brief description of the core purpose of the lecture\n\n### Example test rows:\n\n<b>prior_group_responses: </b> (string) provides all of the user_answer entries for previous group in a string representation of a list in the first row of the group. All other rows in each group are null. If you are using Python, you will likely want to call eval on the non-null rows. Some rows may be null, or empty lists.\n\n<b>prior_group_answers_correct: </b> (string) provides all the answered_correctly field for previous group, with the same format and caveats as prior_group_responses. Some rows may be null, or empty lists."},{"metadata":{},"cell_type":"markdown","source":"# Preliminary data exploration:\n\nFor this preliminary data exploration we'll be looking at the distributions of the columns."},{"metadata":{},"cell_type":"markdown","source":"### train Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"<b>row_id: </b> (int64) ID code for the row."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train.shape)\nprint(len(train['row_id'].unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<b>timestamp: </b>(int64) the time in milliseconds between this user interaction and the first event completion from that user."},{"metadata":{"trusted":true},"cell_type":"code","source":"# we'll bin the timestamps into 20 buckets. \nprint(train['timestamp'].min())\nprint(train['timestamp'].max())\nstep = (train['timestamp'].max()/20)\nbins_list = [0]\nstep_added = 0\nfor i in range(0, 19):\n    step_added = step_added + step\n    bins_list.append(step_added)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['binned_timestamp'] = pd.cut(train['timestamp'], bins_list) #If 0 miliseconds have passed, the value of binned_timestamp is NA\ntrain['binned_timestamp'] = train['binned_timestamp'].astype(str)\ntrain = train.sort_values(['timestamp'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<b>content_type_id:</b>(int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n<b> content_id:  </b> (int16) ID code for the user interaction"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('content_type_id').count() #very few lectures","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<b>task_container_id:</b> (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id. This ID will make more sense from the questions point of view."},{"metadata":{},"cell_type":"markdown","source":"<b>user_answer: </b>(int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(\"user_answer\").count()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<b> comment </b> : from looking at this plot we can see that the correct answer is usually evenly distributed among the numberical options. However, students tend to choose to choose 2 the least. "},{"metadata":{},"cell_type":"markdown","source":"<b>prior_question_elapsed_time:  </b>(float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train[\"prior_question_elapsed_time\"].shape)\nprint(len(train['prior_question_elapsed_time'].unique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# we'll bin the timestamps into 20 buckets. \nprint(train['prior_question_elapsed_time'].min())\nprint(train['prior_question_elapsed_time'].max())\nstep = (train['prior_question_elapsed_time'].max()/20)\nbins_list = [0]\nstep_added = 0\nfor i in range(0, 19):\n    step_added = step_added + step\n    bins_list.append(step_added)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['binned_prior_question_elapsed_time'] = pd.cut(train['prior_question_elapsed_time'], bins_list) #If 0 miliseconds have passed, the value of binned_timestamp is NA\ntrain['binned_prior_question_elapsed_time'] = train['binned_prior_question_elapsed_time'].astype(str)\ntrain = train.sort_values(['prior_question_elapsed_time'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<b>prior_question_had_explanation: </b> (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"float_prior_question_had_explanation\"] = train[\"prior_question_had_explanation\"].astype(float)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<b> comment </b> : It seems that when the prior question has an explanation students tend not to get the question right"},{"metadata":{},"cell_type":"markdown","source":"-----"},{"metadata":{},"cell_type":"markdown","source":"### Questions : \n\n<b> questions.csv:</b> metadata for the questions posed to users.\n\n<b> question_id: </b>foreign key for the train/test content_id column, when the content type is question (0).\n\n<b> bundle_id:</b> code for which questions are served together.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(questions.shape)\nprint(len(questions['bundle_id'].unique()))\nprint(len(questions['question_id'].unique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_bundle = questions.groupby('bundle_id').count().reset_index()[['bundle_id', \"part\"]].rename(columns={\"part\":\"count_in_bundle\"})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df_bundle.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<b> comment </b> : the majority of hte questions are not bundled"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_questions = questions.merge(df_bundle, on = \"bundle_id\", how = \"left\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<b> correct_answer:</b> When we look at the number of asked questions and see how many fall into the category of \"count_in_bundle\" we see an increase in category 3, 4 and 5 (as we would have expected). "},{"metadata":{},"cell_type":"markdown","source":"<b> comment:</b> There's a fewer number of answers labled as option 2 that are correct. "},{"metadata":{},"cell_type":"markdown","source":"<b> part:</b> the relevant section of the TOEIC test."},{"metadata":{},"cell_type":"markdown","source":"<b> tags:</b> one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"#because there are so many tags and many tags combinations, I'll abstain from doing an very large number of feature by hotcoding the tags. \n#I'll look at the number of tags in the question instead under the hypothesis that the more tags the question has \n# the student has the opportunity to extrapolate the knowledge of the answer from more sources. \nprint(df_questions['tags'].shape)\nprint(len(df_questions['tags'].unique()))\ndf_questions['number_tags'] = df_questions['tags'].str.split(\" \").str.len()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\n### Lectures: \n\n<b>lecture_id: </b>foreign key for the train/test content_id column, when the content type is lecture (1).\n\n<b>part:</b> top level category code for the lecture.\n\n<b>tag:</b> one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n<b>type_of:</b> brief description of the core purpose of the lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(lectures['tag'].unique()))\nprint(len(lectures['type_of'].unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Joining the three datasets\nfor now, I'll abstain from making features related to previous lectures because I dont know how to identified whether the student saw the lecture before or after answering the question"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.rename(columns = {\"content_id\": \"question_id\"})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_joined = train[train[\"content_type_id\"]==0].merge(df_questions, on = ['question_id'], how = 'left')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature engineering"},{"metadata":{},"cell_type":"markdown","source":"feature 1: fea timestamp\n\nfeature 2: fea prior question elapsed time\n\nfeature 3: fea prior question had explanation\n\nfeature 4: fea part (question)\n\nfeature 5: fea_num_questions_bundle\n\nfeature 6: number of tags in question"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_joined = df_joined.rename(columns = {\n    \"timestamp\": \"fea_time_until_first_event_completion\",\n    'prior_question_elapsed_time' : 'fea_prior_question_elapsed_time',\n    'float_prior_question_had_explanation': 'fea_prior_question_had_explanation',\n    'part':'fea_question_part',\n    'count_in_bundle':\"fea_num_questions_bundle\",\n    'number_tags':'fea_number_of_tags_in_question'\n})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_master_table = df_joined[['row_id', 'answered_correctly']+[c for c in df_joined.columns if \"fea\" in c]].dropna()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_x = np.array(df_master_table[[c for c in df_joined.columns if \"fea\" in c]])\ndf_y = np.array(df_master_table[['answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Training Pipeline "},{"metadata":{},"cell_type":"markdown","source":"With the intention of keeping our notebook simple, we'll explore only three classification algorithms:\n    - Forest of Randomized Trees \n    - Gradient Boosting Classifier"},{"metadata":{"trusted":true},"cell_type":"code","source":"Dictionary_of_algorithms = {\n    \"GBM\" : GradientBoostingClassifier(random_state=5),\n    \"random_forest\": RandomForestClassifier(random_state=5),\n}\n#need to set the seed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"before doing gridsearch, we'll explore which algorithms seems to be fitting the data best"},{"metadata":{"trusted":true},"cell_type":"code","source":"report = pd.DataFrame(columns = [\"algorithm_name\", \"mean_squared_error\"])\nfor name, algo in Dictionary_of_algorithms.items():\n    scores = pd.DataFrame([cross_val_score(algo, df_x, df_y, cv=3, scoring='accuracy').mean()], columns = [\"mean_squared_error\"])\n    scores['algorithm_name'] = name\n    report = pd.concat([report,scores])\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"report #it seems to be that the GBM algorithm provides with a higher accuracy","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Gridsearch: \n\nbecause GBM offers a higher accuracy, we'll proceed to do gridsearch on the GBM model."},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train, X_test, y_train, y_test = train_test_split(df_x, df_y, test_size=0.33, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"parameters = {'random_state':[5], 'learning_rate':[0.05, 0.1, 0.15], \"min_samples_split\":[2,10,20]}\nclf = GridSearchCV(GradientBoostingClassifier(), parameters)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"gmb_gridsearched = clf.fit(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true},"cell_type":"code","source":"gmb_gridsearched.get_params()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_base = pd.DataFrame(y_test, columns =['Actual_y'])\ndf_base['predicted_y'] = gmb_gridsearched.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_test = pd.read_csv(\"/kaggle/input/riiid-test-answer-prediction/example_test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_test = example_test.rename(columns = {\"content_id\": \"question_id\"})\nexample_test = example_test[example_test[\"content_type_id\"]==0].merge(df_questions, on = ['question_id'], how = 'left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_test[\"float_prior_question_had_explanation\"] = example_test[\"prior_question_had_explanation\"].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_test = example_test.rename(columns = {\n    \"timestamp\": \"fea_time_until_first_event_completion\",\n    'prior_question_elapsed_time' : 'fea_prior_question_elapsed_time',\n    'float_prior_question_had_explanation': 'fea_prior_question_had_explanation',\n    'part':'fea_question_part',\n    'count_in_bundle':\"fea_num_questions_bundle\",\n    'number_tags':'fea_number_of_tags_in_question'\n})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_output = example_test['row_id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_test.dtypes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_output['predictions'] = gmb_gridsearched.predict(np.array(example_test[[c for c in example_test.columns if \"fea\" in c]].dropna()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_output.to_csv(\"submission.csv\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}