{"cells":[{"metadata":{},"cell_type":"markdown","source":"#               Riiid -🔥🔥🔥Model Explainability🔥🔥🔥"},{"metadata":{},"cell_type":"markdown","source":"\n## <a name=\"Riiid! Answer Correctness Prediction\">About this Competition</a>\n\nRiiid Labs, an AI solutions provider delivering creative disruption to the education market, empowers global education players to rethink traditional ways of learning leveraging AI. With a strong belief in equal opportunity in education, Riiid launched an AI tutor based on deep-learning algorithms in 2017 that attracted more than one million South Korean students. This year, the company released EdNet, the world’s largest open database for AI education containing more than 100 million student interactions.\n\nIn this competition, your challenge is to create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions. You will pair your machine learning skills using Riiid’s EdNet data.\n\nYour innovative algorithms will help tackle global challenges in education. If successful, it’s possible that any student with an Internet connection can enjoy the benefits of a personalized learning experience, regardless of where they live. With your participation, we can build a better and more equitable model for education in a post-COVID-19 world."},{"metadata":{},"cell_type":"markdown","source":"## Contents\n- <a href='#1'>1. Introduction</a>  \n- <a href='#2'>2. Data Extraction </a>\n- <a href='#3'>3. Glimpse of Data</a>\n- <a href='#4'>4. Feature Engineering </a>\n- <a href='#5'>5. Baseline Model</a>\n- <a href='#9'>6. Model Explainability</a>"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThe challenge will rely on the world’s largest education dataset, EdNet, which consists of more than 130 million interactions coming from over 780,000 students. EdNet will be offered to top researchers and scientists.\n\nIn this competition, we will just try to create an algorithm for \"Knowledge Tracing,\" the modeling of student knowledge over time. our goal is to accurately predict how students will perform on future interactions. If successful, it’s possible that any student with an Internet connection can enjoy the benefits of a personalized learning experience, regardless of where they live.\n\nOur challenge in this competition is to predict whether students are able to answer their next questions correctly.\n\nThis competition is similar to Two Sigma competition, so we got test data using special API."},{"metadata":{"trusted":true},"cell_type":"code","source":"!pip install datatable","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport time \nimport matplotlib.pyplot as plt \nimport numpy as np \nimport pandas as pd \nimport seaborn as sns \nimport datatable as dt\nfrom sklearn import model_selection\n\n\nimport os \n\nimport plotly.express as px\nimport plotly.graph_objects as go \nfrom sklearn.model_selection import StratifiedKFold , KFold, RepeatedKFold,GroupKFold , GridSearchCV , train_test_split ,TimeSeriesSplit\nfrom sklearn.metrics import roc_auc_score\ncolor = sns.color_palette()\n\n\nimport multiprocessing as mp \n\nimport optuna # Hyperparameter tuning\n\nimport warnings \nwarnings.filterwarnings('ignore')\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## <a name=\"dataset_description\">Dataset Description</a>: \n\nThe data is images of wheat fields, with bounding boxes for each identified wheat head. Not all images include wheat heads / bounding boxes. The images were recorded in many locations around the world.\n\nThe CSV data is simple - the image ID matches up with the filename of a given image, and the width and height of the image are included, along with a bounding box (see below). There is a row in train.csv for each bounding box. Not all images have bounding boxes.\n\nMost of the test set images are hidden. A small subset of test images has been included for your use in writing code.\n\nWhat am I predicting?\nYou are attempting to predict bounding boxes around each wheat head in images that have them. If there are no wheat heads, you must predict no bounding boxes.\n\nFile details :- \n\n1.     train.csv - the training data\n2.     questions.csv - List question ID in the online education app\n3.     lectures.csv - Leture details coved in this file\n4.     example_sample_submission.csv - Sample Submission format \n\n\n\n### <a name=\"target_variable\">Target Variable</a>                                        \n* __Submission data__  \n    user_id & answered_correctly\n    \n    \n    \n"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"data_types = {\n    'timestamp': 'int64',\n    'user_id': 'int64',\n    'content_id': 'int16',\n    'content_type_id': 'int8',\n    'task_container_id': 'int16',\n    'answered_correctly': 'int8',\n    'prior_question_elapsed_time': 'float32',\n    'prior_question_had_explanation': 'boolean'\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain = dt.fread('/kaggle/input/riiid-test-answer-prediction/train.csv').to_pandas()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train[['timestamp', 'user_id', 'content_id', 'content_type_id','task_container_id', 'answered_correctly','prior_question_elapsed_time', 'prior_question_had_explanation']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for column, d_type in data_types.items():\n    train[column] = train[column].astype(d_type) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Take fraction of data for analysis while submission kindly comment this line "},{"metadata":{},"cell_type":"markdown","source":"### Split the training set by 5 parts for Quicker iterations "},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.sample(len(train)//5,random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#train = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', usecols=[1, 2, 3,4,5,7,8,9], dtype={'timestamp': 'int64', 'user_id': 'int32' ,'content_id': 'int16','content_type_id': 'int8','answered_correctly':'int8','prior_question_elapsed_time': 'float32','prior_question_had_explanation': 'boolean','task_container_id': 'int16'})\ntrain = train[train.content_type_id == False]\n#arrange by timestamp\n\ntrain = train.sort_values(['timestamp'], ascending=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"! head 10 /kaggle/input/riiid-test-answer-prediction/train.csv","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.answered_correctly.unique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Loading datasets having less memory "},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '/kaggle/input'\n\"\"\"\n\n\ntrain = pd.read_csv(f'{path}/riiid-test-answer-prediction/train.csv', low_memory=False, \n                       dtype={'row_id': 'int64',\n                              'timestamp': 'int64',\n                              'user_id': 'int32',\n                              'content_id': 'int16',\n                              'content_type_id': 'int8',\n                              'task_container_id': 'int16',\n                              'user_answer': 'int8',\n                              'answered_correctly': 'int8',\n                              'prior_question_elapsed_time': 'float32', \n                              'prior_question_had_explanation': 'boolean',\n                             }\n                      )\n\"\"\"\n\ntest = pd.read_csv(f'{path}/riiid-test-answer-prediction/example_test.csv')\nsubmit = pd.read_csv(f'{path}/riiid-test-answer-prediction/example_sample_submission.csv')\nlectures = pd.read_csv(f'{path}/riiid-test-answer-prediction/lectures.csv')\n\ntrain = train[train.content_type_id == False]\n\n\nprint('Train shapes: ', train.shape)\nprint('Test shapes: ', test.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering \n\n#### \"Applied Machine Learning is northing but feature engineering\" - Andrew Ng"},{"metadata":{"trusted":true},"cell_type":"code","source":"results_c = train[['content_id','answered_correctly']].groupby(['content_id']).agg(['mean'])\nresults_c.columns = [\"answered_correctly_content\"]\n\nresults_u = train[['user_id','answered_correctly']].groupby(['user_id']).agg(['mean', 'sum'])\nresults_u.columns = [\"answered_correctly_user\", 'sum']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv',\n                            usecols=[0,1, 3,4],\n                            dtype={'question_id': 'int16',\n                              'part': 'int8','bundle_id': 'int8','tags': 'str'}\n                          )\ntag = questions_df[\"tags\"].str.split(\" \", n = 10, expand = True) \ntag.columns = ['tags1','tags2','tags3','tags4','tags5','tags6']\n\nquestions_df =  pd.concat([questions_df,tag],axis=1)\nquestions_df['tags1'] = pd.to_numeric(questions_df['tags1'], errors='coerce')\nquestions_df['tags2'] = pd.to_numeric(questions_df['tags2'], errors='coerce')\nquestions_df['tags3'] = pd.to_numeric(questions_df['tags3'], errors='coerce')\nquestions_df['tags4'] = pd.to_numeric(questions_df['tags4'], errors='coerce')\nquestions_df['tags5'] = pd.to_numeric(questions_df['tags5'], errors='coerce')\nquestions_df['tags6'] = pd.to_numeric(questions_df['tags6'], errors='coerce')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Five Point Summary "},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"count of content_type_id is less , there are missing values in it , It requires treatment"},{"metadata":{},"cell_type":"markdown","source":"### --- Start of Sample data analysis ----"},{"metadata":{},"cell_type":"markdown","source":"### Let understand the data with sample "},{"metadata":{},"cell_type":"markdown","source":"To understand data ; Lets dive in.  For User ID - 115 there are multiple content Id\nContent type ID - 0 or 1 depend on weather the user watching the lecture or answering the question\n\nFor Content type ID - 0 ( User lectures ) , user_answer and answered_correctly data value should be -1 \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train[(train['user_id']==2746)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nquestions_df[questions_df['question_id']==128]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures[lectures['lecture_id']==128]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Total unique users :- ',train['user_id'].nunique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_correct_question_anys = train[train['content_type_id']==0].groupby(['user_id']).agg({'content_id':'count','answered_correctly':'sum'}).reset_index().rename(columns={'content_id':'Total_questions'})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_correct_question_anys['answered_wrongly']=df_correct_question_anys['Total_questions'] - df_correct_question_anys['answered_correctly']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_correct_question_anys['Pass_pct']= df_correct_question_anys['answered_correctly'] / df_correct_question_anys['Total_questions']\ndf_correct_question_anys['Fail_pct']= df_correct_question_anys['answered_wrongly'] / df_correct_question_anys['Total_questions']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### --- End of Sample data analysis ----"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.drop(['timestamp','content_type_id','task_container_id'], axis=1,   inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### We have divided the dataset into 5 parts , If we training with full dataset please uncomment below line \n#X = train.iloc[80000000:,:]\n\nX = train.copy()\nX['prior_question_had_explanation'].fillna(False, inplace=True)\n\nX['prior_question_had_explanation'] = X['prior_question_had_explanation'].fillna(X['prior_question_had_explanation'].mode()[0])\nX['prior_question_elapsed_time'] = X['prior_question_elapsed_time'].fillna(X['prior_question_elapsed_time'].mean())\n\nX = pd.merge(X, results_u, on=['user_id'], how=\"left\")\nX = pd.merge(X, results_c, on=['content_id'], how=\"left\")\nX = pd.merge(X, questions_df, left_on = 'content_id', right_on = 'question_id', how = 'left')\n\nX=X[X.answered_correctly!= -1 ]\nX=X.sort_values(['user_id'])\nY = X[[\"answered_correctly\"]]\nX = X.drop([\"answered_correctly\"], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\nlb_make = LabelEncoder()\nX['prior_question_had_explanation_enc'] = lb_make.fit_transform(X['prior_question_had_explanation'])\nX.head()\n\nX.fillna(0.5,inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X = X[['answered_correctly_user', 'answered_correctly_content', 'sum','bundle_id','part','prior_question_elapsed_time','prior_question_had_explanation_enc','tags1','tags2','tags3']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\nXt , Xv , Yt,Yv = train_test_split(X,Y,test_size=0.2,shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.ensemble import RandomForestClassifier","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Further trim the dataset for faster iteration"},{"metadata":{"trusted":true},"cell_type":"code","source":"Xt_trim = Xt[75000:].reset_index(drop=True)\nYt_trim = Yt[75000:].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Xv_trim = Xv[25000:].reset_index(drop=True)\nYv_trim = Yv[25000:].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model Interpretability "},{"metadata":{"trusted":true},"cell_type":"code","source":"import xgboost as xgb","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dmatrix = xgb.DMatrix(data=Xt_trim.values,label=Yt_trim.values)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {\"objective\":\"binary:logistic\",'colsample_bytree': 0.3,'learning_rate': 0.1,\n                'max_depth': 5, 'alpha': 10,\"eval_metric\": \"auc\"}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#clf = xgb.train(params=params, dtrain=data_dmatrix, num_boost_round=10) \nmodel = xgb.train(params, xgb.DMatrix(Xt_trim, label=Yt_trim),  num_boost_round=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# SHAP values"},{"metadata":{},"cell_type":"markdown","source":"SHAP values interpret the impact of having a certain value for a given feature in comparison to the prediction we'd make if that feature took some baseline value."},{"metadata":{"trusted":true},"cell_type":"code","source":"import shap \n\n# load JS visualization code to notebook\nshap.initjs()\n\nexplainer = shap.TreeExplainer(model)\nshap_values = explainer.shap_values(Xv_trim)\n\n#use matplotlib=True\n\n\n# visualize the first prediction's explanation (use matplotlib=True to avoid Javascript)\nshap.force_plot(explainer.expected_value, shap_values[0,:], Xv_trim.iloc[0,:])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### For first response in dataset have 12% chance of giving the correct answer \n\n### Model base value stands at 41.8% its need to be improved "},{"metadata":{"trusted":true},"cell_type":"code","source":"Xv_trim.iloc[0,:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train,X \nimport gc \ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Summarize the impact of all features over the entire dataset\nA SHAP value for a feature of a specific prediction represents how much the model prediction changes when we observe that feature. In the summary plot below we plot all the SHAP values for a single feature on a row, where the x-axis is the SHAP value (which for this model is in units of log odds of giving correct answer to Question). By doing this for all features, we see which features drive the model's prediction a lot (such as Correct Answers), and which only effect the prediction a little (such as kills). Note that when points don't fit together on the line they pile up vertically to show density. Each dot is also colored by the value of that feature from high to low."},{"metadata":{"trusted":true},"cell_type":"code","source":"shap_values = explainer.shap_values(Xv_trim[2:3])\nshap.force_plot(explainer.expected_value, shap_values, Xv_trim[2:3])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## How do you interpret this?\n\nWe predicted 0.16 , whereas the base_value is 0.418. Feature values causing increased predictions are in pink, and their visual size shows the magnitude of the feature's effect. Feature values decreasing the prediction are in blue. The biggest impact comes from tag 4.\n\nIf you subtract the length of the blue bars from the length of the pink bars, it equals the distance from the base value to the output."},{"metadata":{},"cell_type":"markdown","source":"# Examine how changes in a feature change the model's prediction\n\n## ( Feature Interaction with model output)\n\nThe XGBoost model we trained above is very complicated, but by plotting the SHAP value for a feature against the actual value of the feature for all players we can see how changes in the feature's value effect the model's output. Note that these plots are very similar to standard partial dependence plots, but they provide the added advantage of displaying how much context matters for a feature (or in other words how much interaction terms matter)"},{"metadata":{"trusted":true},"cell_type":"code","source":"# sort the features indexes by their importance in the model\n# (sum of SHAP value magnitudes over the validation dataset)\n\n\nexplainer = shap.TreeExplainer(model)\nshap_values = explainer.shap_values(Xv_trim)\n\n\ntop_inds = np.argsort(-np.sum(np.abs(shap_values), 0))\n\n# make SHAP plots of the three most important features\nfor i in range(len(top_inds)):\n    shap.dependence_plot(top_inds[i], shap_values, Xv_trim)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<font color=\"red\" size=3>Please upvote this kernel if you like it. It motivates me to create kernal with great content  :) </font>"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}