{"cells":[{"metadata":{},"cell_type":"markdown","source":"\n\n

A Deeper Dive Into Riiid!: EDA|Models|Blending

\n \n
\n

Problem Statement

\n \n
In this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions. You will pair your machine learning skills using Riiid’s EdNet data.
\n
\n

AIM:

\n
1) To create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time.
\n
2) To accurately predict how students will perform on future interactions.
\n
\n

Purpose:

\n
1) Help tackle global challenges in education.
\n
2) Using good ML model,it is possible that any student with an Internet connection can enjoy the benefits of a personalized learning experience.
\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"
\n

Evaluation Metrics: ROC-AUC Score

\n
Submissions are evaluated on area under the ROC curve between the predicted probability and the observed target.
\n

What is AUC-ROC Curve?

\n
The Receiver Operator Characteristic (ROC) curve is an evaluation metric for binary classification problems. It is a probability curve that plots the TPR against FPR at various threshold values and essentially separates the ‘signal’ from the ‘noise’. The Area Under the Curve (AUC) is the measure of the ability of a classifier to distinguish between classes and is used as a summary of the ROC curve.\nROC-AUC Score is the best measure for binary classification and imbalance class problem. This iste gives more information about AUC ROC curveAUC-ROC-Curve.
\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"
\n

Table Contents

\n \n
1. Exploratory Data Analysis
\n
2. Building Baseline Models
\n
3. Ensembling
\n
4. Conclusion
\n "},{"metadata":{},"cell_type":"markdown","source":"
\n

1. Exploratory Data Analysis

\n\n \n \n

Importing Necessary Libraries

\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np \nimport pandas as pd \nfrom collections import Counter\nimport pandas_profiling as pp\nfrom sklearn.model_selection import StratifiedKFold\nfrom lightgbm import LGBMClassifier\nfrom catboost import CatBoostClassifier\nfrom sklearn.metrics import roc_auc_score\nimport warnings\nwarnings.simplefilter('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\nenv = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Data Description

\n
\n
1. row_id:ID code for the row.
\n
2. timestamp: time between this user interaction and the first event.
\n
3. user_id: ID code for the user
\n
4. content_id:ID code for the user interaction
\n
5. content_type_id: 0 if question else 1 for Lecture.
\n
6. user_answer: the user's answer to the question, Read -1 as null, for lectures.
\n
7. user_answer: IF the user responded correctly. Read -1 as null, for lectures.
\n
8. prior_question_elapsed_time: How long it took a user to answer their previous question bundle, ignoring any lectures in between. It is the total time a user took to solve all questions in the previous bundle.
\n
9. prior_question_had_explanation: Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between.
\n\n"},{"metadata":{},"cell_type":"markdown","source":"

Loading Dataset

"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', low_memory=False,nrows=10**5, dtype={'row_id': 'int64',\n 'timestamp': 'int64','user_id': 'int32','content_id': 'int16', 'content_type_id': 'int8','task_container_id': 'int16',\n 'user_answer': 'int8','answered_correctly': 'int8','prior_question_elapsed_time': 'float32', 'prior_question_had_explanation': 'boolean'} ) \ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"

Data Cleaning

\n
Removing Unwanted columns
\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.drop('user_answer',axis=1)\ntrain = train.query('answered_correctly != -1').reset_index(drop=True)\ntrain.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
Missing Value Detection
\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def missing(df):\n total = df.isnull().sum().sort_values(ascending = False)\n total = total[total>0]\n percent = df.isnull().sum().sort_values(ascending = False)/len(df)*100\n percent = percent[percent>0]\n return pd.concat([total, percent], axis=1, keys=['Total','Percentage'])\nmissing(train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
Missing Value Treatment
\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['prior_question_had_explanation'] = train['prior_question_had_explanation'].fillna(train['prior_question_had_explanation'].mode()[0])\ntrain['prior_question_elapsed_time'] = train['prior_question_elapsed_time'].fillna(train['prior_question_elapsed_time'].mean())\nmissing(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Counter(train['answered_correctly'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n

2. Building Baseline Models

\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install -U pycaret","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['prior_question_had_explanation'] = train['prior_question_had_explanation'].astype(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from pycaret.classification import *\nclf1 = setup(train, target = 'answered_correctly',session_id = 786,use_gpu=True,silent = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"compare_models()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lightgbm = create_model('lightgbm')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# !pip install lightgbm ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Tune model using scikit-learn auto"},{"metadata":{"trusted":true},"cell_type":"code","source":"tuned_lightgbm = tune_model(lightgbm,optimize='AUC',n_iter=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Tune model using scikit-optimize"},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install scikit-optimize","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"tuned_lightgbm_auto = tune_model(lightgbm,optimize='AUC',n_iter=25,search_library=\"scikit-optimize\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Ensemble Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"bagged_lightgbm = ensemble_model(tuned_lightgbm_auto)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"boosted_lightgbm = ensemble_model(tuned_lightgbm_auto ,method = 'Boosting')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Blend Models"},{"metadata":{"trusted":true},"cell_type":"code","source":"blender = blend_models(estimator_list = [tuned_lightgbm,tuned_lightgbm_auto], method = 'soft')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Stack model "},{"metadata":{"trusted":true},"cell_type":"code","source":"# stacker = stack_models(estimator_list = [tuned_lightgbm,tuned_lightgbm_auto], meta_model=lightgbm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Analyze Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Best Model : tuned_lightgbm : AUC : 66\n\nplot_model(tuned_lightgbm)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n

Conclusion

\n\n
LGBMClassifier perform better compare to other model.
\n"},{"metadata":{},"cell_type":"markdown","source":"
\n

Things to be taken care before submission

\n
This competition is different from most of other Kaggle Competitions.You will loop through a series of batches of questions. Once you make that prediction, you can move on to the next batch, you will receive test set data and make predictions with Kaggle's time-series API. So it is good if you refer these kerenel before submission
\n
\n\n\n
"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}