{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About competition\n\nThe challenge will rely on the world’s largest education dataset, EdNet, which consists of more than 130 million interactions coming from over 780,000 students. EdNet will be offered to top researchers and scientists. \n\nIn this competition, we will just try to create an algorithm for \"Knowledge Tracing,\" the modeling of student knowledge over time. our goal is to accurately predict how students will perform on future interactions. If successful, it’s possible that any student with an Internet connection can enjoy the benefits of a personalized learning experience, regardless of where they live. \n\nOur challenge in this competition is to predict whether students are able to answer their next questions correctly.\n\nThis competition is similar to Two Sigma competition, so we got test data using special API.\n\n<font size=3 color=\"red\">Please upvote this kernel if you like it. It motivates me to produce more quality content :)</font>\n\n![](https://ml8ygptwlcsq.i.optimole.com/fMKjlhs.f8AX~1c8f3/w:1200/h:678/q:auto/https://www.unite.ai/wp-content/uploads/2020/10/Screen-Shot-2020-10-06-at-7.38.57-PM.jpg)\n\n"},{"metadata":{},"cell_type":"markdown","source":"lets go by flow..."},{"metadata":{},"cell_type":"markdown","source":"# import all you need"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import time\nimport matplotlib.pyplot as plt \nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\ncolor = sns.color_palette()\nimport os\n       \nimport plotly.express as px \nimport plotly.graph_objects as go\nfrom sklearn.model_selection import StratifiedKFold, KFold, RepeatedKFold, GroupKFold, GridSearchCV, train_test_split, TimeSeriesSplit\n\n\nimport warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"as there are more than 100m rows in \"train.csv\", we can't read all the data in kaggle notebooks so , lets just take some "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"path = '/kaggle/input'\n\ntrain = pd.read_csv(f'{path}/riiid-test-answer-prediction/train.csv', low_memory=False, nrows=5 * (10**5), \n                       dtype={'row_id': 'int64',\n                              'timestamp': 'int64',\n                              'user_id': 'int32',\n                              'content_id': 'int16',\n                              'content_type_id': 'int8',\n                              'task_container_id': 'int16',\n                              'user_answer': 'int8',\n                              'answered_correctly': 'int8',\n                              'prior_question_elapsed_time': 'float32', \n                              'prior_question_had_explanation': 'boolean',\n                             }\n                      )\n\ntest = pd.read_csv(f'{path}/riiid-test-answer-prediction/example_test.csv')\nsubmit = pd.read_csv(f'{path}/riiid-test-answer-prediction/example_sample_submission.csv')\nquestions = pd.read_csv(f'{path}/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv(f'{path}/riiid-test-answer-prediction/lectures.csv')\nprint('Train shapes: ', train.shape)\nprint('Test shapes: ', test.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":" **answered_correctly** is our target! **-1** is a special value, we'll talk about it later"},{"metadata":{},"cell_type":"markdown","source":"## lets try to Exploring the features in train.csv"},{"metadata":{},"cell_type":"markdown","source":"### timestamp - \n\n> it is imprtant to remember that this is the time between this user interaction and the first event from that user. So starting time could be different for each user"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig,ax = plt.subplots(figsize=(12,8))\nplt.hist(train['timestamp'], bins=40);\nplt.xlabel('timestamp',fontsize=20)\nplt.ylabel('count',fontsize=20)\nax.tick_params(labelsize=20)\nplt.title('count of timestamp',fontsize=25)\nplt.grid()\nplt.ioff()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Top 40 users by number of actions"},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = train['user_id'].value_counts().reset_index()\nds.columns = ['user_id', 'count']\nds['user_id'] = ds['user_id'].astype(str) + '-'\nds = ds.sort_values(['count'])\ntop_40 = ds.tail(40)\n\n\nfig,ax = plt.subplots(figsize=(12,8))\nsns.barplot(top_40['count'],top_40['user_id'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## content_type_id \n\n0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.content_type_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f,ax=plt.subplots(1,2,figsize=(18,8))\ntrain['content_type_id'].value_counts().plot.pie(explode=[0,0.1],autopct='%1.1f%%',ax=ax[0],shadow=True)\nax[0].set_title('Percentage content_type_id Distribution')\nax[0].set_ylabel('Count')\nsns.countplot('content_type_id',data=train,ax=ax[1],order=train['content_type_id'].value_counts().index)\nax[1].set_title('Count of content_type_id')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"##  task_container_id\n\nId code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id. Monotonically increasing for each user."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.task_container_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## user_answer\n\nthe user's answer to the question, if any. Read -1 as null, for lectures."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.user_answer.value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"f,ax=plt.subplots(1,2,figsize=(18,8))\ntrain['user_answer'].value_counts().plot.pie(explode=[0,0.1,0.1,0.1,0.1],autopct='%1.1f%%',ax=ax[0],shadow=True)\nax[0].set_title('Percentage user_answer Distribution')\nax[0].set_ylabel('Count')\nsns.countplot('user_answer',data=train,ax=ax[1],order=train['user_answer'].value_counts().index)\nax[1].set_title('Count of user_answer')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## answered_correctly( pere ): \n\n(int8) if the user responded correctly. Read -1 as null, for lectures.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"f,ax=plt.subplots(1,2,figsize=(18,8))\ntrain['answered_correctly'].value_counts().plot.pie(explode=[0,0.1,0.1],autopct='%1.1f%%',ax=ax[0],shadow=True)\nax[0].set_title('Percentage Severity Distribution')\nax[0].set_ylabel('Count')\nsns.countplot('answered_correctly',data=train,ax=ax[1],order=train['answered_correctly'].value_counts().index)\nax[1].set_title('Count of answered_correctly')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## prior_question_had_explanation: \n\n(bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['prior_question_had_explanation'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### lets try to add some Features"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.loc[train['answered_correctly'] != -1].reset_index(drop=True)\ntrain['prior_question_had_explanation'] = train['prior_question_had_explanation'].fillna(value=False).astype(bool)\n\nuser_answers_ = train.groupby('user_id').agg({ 'answered_correctly': ['mean', 'count']}).copy()\nuser_answers_.columns = ['mean_user_accuracy', 'questions_answered']\n\ncontent_answers_ = train.groupby('content_id').agg({'answered_correctly': ['mean', 'count']}).copy()\ncontent_answers_.columns = ['mean_acc', 'questions_asked']\n\ntrain = train.merge(user_answers_, how='left', on = 'user_id')\ntrain = train.merge(content_answers_, how='left', on = 'content_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.merge(train,questions[['question_id','bundle_id','part']], left_on='user_id', right_on='question_id')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"grouped_df = train.groupby([\"questions_answered\"])[\"mean_user_accuracy\"].aggregate(\"count\").reset_index()\n\nplt.figure(figsize=(12,8))\nsns.pointplot(grouped_df['questions_answered'].values, grouped_df['mean_user_accuracy'].values, alpha=0.8, color=color[2])\nplt.ylabel('questions_answered', fontsize=12)\nplt.xlabel('mean_user_accuracy', fontsize=12)\nplt.title(\"mean_user_accuracy wise questions_answered\", fontsize=15)\nplt.xticks(rotation='vertical')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.jointplot(x=train.mean_acc.values,y=train.questions_asked.values,height=10)\nplt.ylabel('mean_acc', fontsize=12)\nplt.xlabel('questions_asked', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = ['timestamp', 'user_id', 'content_id', 'content_type_id','answered_correctly',\n       'task_container_id', 'prior_question_elapsed_time',\n       'prior_question_had_explanation', 'part', 'mean_user_accuracy', 'questions_answered','mean_acc', 'questions_asked']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = train[columns].copy()\ndf.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## well now, lets try to check for corrcoef for all the features with the target"},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = []\nvalues = []\nfor col in columns:\n    labels.append(col)\n    values.append(np.corrcoef(df[col].values, df.answered_correctly.values)[0,1])\ncorr_df = pd.DataFrame({'col_labels':labels, 'corr_values':values})\ncorr_df = corr_df.sort_values(by='corr_values')\n\nind = np.arange(len(labels))\nwidth = 0.9\nfig, ax = plt.subplots(figsize=(12,40))\nrects = ax.barh(ind, np.array(corr_df.corr_values.values), color='y')\nax.set_yticks(ind)\nax.set_yticklabels(corr_df.col_labels.values, rotation='horizontal')\nax.set_xlabel(\"Correlation coefficient\")\nax.set_title(\"Correlation coefficient of the variables\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## modelling "},{"metadata":{},"cell_type":"markdown","source":"### we can actually play with models but for now i will try some and plot the feature_importances_ for the features with target"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"columns_f = ['timestamp', 'user_id', 'content_id', 'content_type_id',\n       'task_container_id', 'prior_question_elapsed_time',\n       'prior_question_had_explanation', 'part', 'mean_user_accuracy', 'questions_answered','mean_acc', 'questions_asked']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.fillna(value = -1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# RandomForestClassifier"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_y = train['answered_correctly'].values\nnum_df = train[columns_f]\nfeat_name = num_df.columns.values\n\nfrom sklearn import ensemble \nmodel = ensemble.RandomForestClassifier(n_estimators=25,max_depth=30, n_jobs=-1, random_state=0) \nmodel.fit(num_df,train_y)\n\nimportances = model.feature_importances_\nstd = np.std([tree.feature_importances_ for tree in model.estimators_],axis=0)\nindi = np.argsort(importances)[::-1][:20]\n\nplt.figure(figsize=(12,12))\nplt.title(\"Feature importances\")\nplt.bar(range(len(indi)), importances[indi], color=color[4], yerr=std[indi], align=\"center\")\nplt.xticks(range(len(indi)), feat_name[indi], rotation='vertical')\nplt.xlim([-1, len(indi)])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Xgboost "},{"metadata":{"trusted":true},"cell_type":"code","source":"import xgboost as xgb \n\nxgb_prames = {\n    'eta': 0.05,\n    'max_depth': 8,\n    'subsample': 0.7,\n    'colsample_bytree': 0.7,\n    'objective': 'reg:linear',\n    'silent': 1,\n    'seed' : 0\n}\n\ndtrain = xgb.DMatrix(num_df, train_y, feature_names=num_df.columns.values)\n\nmodel = xgb.train(dict(xgb_prames, silent=0), dtrain, num_boost_round=50)\n\n\nfig, ax = plt.subplots(figsize=(12,18))\nxgb.plot_importance(model, max_num_features=50, height=0.8, ax=ax)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Lightgbm"},{"metadata":{},"cell_type":"markdown","source":"### helper functions \n\n![](http://)took from : https://www.kaggle.com/artgor/riiid-eda-feature-engineering-and-models/comments"},{"metadata":{"trusted":true},"cell_type":"code","source":"y = train['answered_correctly']\nX = train.drop(['answered_correctly', 'user_answer'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def fast_auc(y_true, y_prob):\n    \"\"\"\n    fast roc_auc computation: https://www.kaggle.com/c/microsoft-malware-prediction/discussion/76013\n    \"\"\"\n    y_true = np.asarray(y_true)\n    y_true = y_true[np.argsort(y_prob)]\n    nfalse = 0\n    auc = 0\n    n = len(y_true)\n    for i in range(n):\n        y_i = y_true[i]\n        nfalse += (1 - y_i)\n        auc += y_i * nfalse\n    auc /= (nfalse * (n - nfalse))\n    return auc\n\n\ndef eval_auc(y_true, y_pred):\n    \"\"\"\n    Fast auc eval function for lgb.\n    \"\"\"\n    return 'auc', fast_auc(y_true, y_pred), True","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import lightgbm as lgb\n\nscores = []\nfeature_importance = pd.DataFrame()\nmodels = []\n\nparams = {'num_leaves': 32,\n          'max_bin': 300,\n          'objective': 'binary',\n          'max_depth': 13,\n          'learning_rate': 0.03,\n          \"boosting_type\": \"gbdt\",\n          \"metric\": 'auc',\n         }\n\nfolds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)\nfor fold_n, (train_index, valid_index) in enumerate(folds.split(X,y)):\n    print(f'Fold {fold_n} started at {time.ctime()}')\n    X_train, X_valid = X[columns_f].iloc[train_index], X[columns_f].iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    model = lgb.LGBMClassifier(**params, n_estimators=700, n_jobs = 1)\n    model.fit(X_train, y_train, \n            eval_set=[(X_train, y_train), (X_valid, y_valid)], eval_metric=eval_auc,\n            verbose=1000, early_stopping_rounds=10)\n    score = max(model.evals_result_['valid_1']['auc'])\n    \n    models.append(model)\n    scores.append(score)\n\n    fold_importance = pd.DataFrame()\n    fold_importance[\"feature\"] = columns_f\n    fold_importance[\"importance\"] = model.feature_importances_\n    fold_importance[\"fold\"] = fold_n + 1\n    feature_importance = pd.concat([feature_importance, fold_importance], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_importance[\"importance\"] /= 5\ncols = feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n    by=\"importance\", ascending=False)[:50].index\n\nbest_features = feature_importance.loc[feature_importance.feature.isin(cols)]\n\nplt.figure(figsize=(16, 12));\nsns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False));\nplt.title('LGB Features (avg over folds)');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# making prediction \n\nyou can get from https://www.kaggle.com/sishihara/riiid-lgbm-5cv-benchmark"},{"metadata":{"trusted":true},"cell_type":"code","source":"import riiideducation\nenv = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    y_preds = []\n    test_df = test_df.merge(user_answers_, how = 'left', on = 'user_id')\n    test_df = test_df.merge(content_answers_, how = 'left', on = 'content_id')\n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\n    test_df = test_df.loc[test_df['content_type_id'] == 0].reset_index(drop=True)\n    test_df = pd.merge(test_df, questions[['question_id', 'bundle_id', 'part']], left_on='content_id', right_on='question_id', how='left')\n    test_df.fillna(value = -1, inplace = True)\n\n    for model in models:\n        y_pred = model.predict_proba(test_df[columns_f], num_iteration=model.best_iteration_)[:, 1]\n        y_preds.append(y_pred)\n\n    y_preds = sum(y_preds) / len(y_preds)\n    test_df['answered_correctly'] = y_preds\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To bo .... \n\nhave to do alots actually \n\n1. tunning modelling\n2. good feature engineering and selection \n3. pca (may be)\n\nwell update soon \n\n<font size=3 color=\"red\">Please upvote this kernel if you like it. It motivates me to produce more quality content :)</font>"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}