{"cells":[{"metadata":{},"cell_type":"markdown","source":"## General information\n\nIn this competition, we will create algorithms for \"Knowledge Tracing,\" the modeling of student knowledge over time. The goal is to accurately predict how students will perform on future interactions.\nIf successful, it’s possible that any student with an Internet connection can enjoy the benefits of a personalized learning experience, regardless of where they live. We can build a better and more equitable model for education in a post-COVID-19 world.\n\nOur challenge in this competition is to predict whether students are able to answer their next questions correctly.\n\nThis competition is similar to Two Sigma competition, where we got test data using special API.\n\n![](https://i.imgur.com/Ko5MxFQ.png)"},{"metadata":{},"cell_type":"markdown","source":"### importing libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# libraries\nimport riiideducation\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport seaborn as sns\nimport os\nfrom typing import List, Dict, Optional\nimport numpy as np\nfrom sklearn.model_selection import RepeatedKFold\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nimport math\nimport time\nimport random\nimport lightgbm as lgb\nimport gc\nimport os\nfrom sklearn.preprocessing import LabelEncoder\nfrom numba import jit\nfrom sklearn.model_selection import StratifiedKFold, KFold, RepeatedKFold, GroupKFold, GridSearchCV, train_test_split, TimeSeriesSplit\nfrom sklearn import metrics","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### helper functions"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"def reduce_mem_usage(df, verbose=True):\n    numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']\n    start_mem = df.memory_usage(deep=True).sum() / 1024**2\n    for col in df.columns:\n        col_type = df[col].dtypes\n        if col_type in numerics:\n            c_min = df[col].min()\n            c_max = df[col].max()\n            if str(col_type)[:3] == 'int':\n                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n                    df[col] = df[col].astype(np.int8)\n                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n                    df[col] = df[col].astype(np.int16)\n                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n                    df[col] = df[col].astype(np.int32)\n                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n                    df[col] = df[col].astype(np.int64)\n            else:\n                c_prec = df[col].apply(lambda x: np.finfo(x).precision).max()\n                if c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max and c_prec == np.finfo(np.float32).precision:\n                    df[col] = df[col].astype(np.float32)\n                else:\n                    df[col] = df[col].astype(np.float64)\n    end_mem = df.memory_usage(deep=True).sum() / 1024**2\n    if verbose: print('Mem. usage decreased to {:5.2f} Mb ({:.1f}% reduction)'.format(end_mem, 100 * (start_mem - end_mem) / start_mem))\n    return df\n    \n\n@jit\ndef fast_auc(y_true, y_prob):\n    \"\"\"\n    fast roc_auc computation: https://www.kaggle.com/c/microsoft-malware-prediction/discussion/76013\n    \"\"\"\n    y_true = np.asarray(y_true)\n    y_true = y_true[np.argsort(y_prob)]\n    nfalse = 0\n    auc = 0\n    n = len(y_true)\n    for i in range(n):\n        y_i = y_true[i]\n        nfalse += (1 - y_i)\n        auc += y_i * nfalse\n    auc /= (nfalse * (n - nfalse))\n    return auc\n\n\ndef eval_auc(y_true, y_pred):\n    \"\"\"\n    Fast auc eval function for lgb.\n    \"\"\"\n    return 'auc', fast_auc(y_true, y_pred), True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data overview"},{"metadata":{"trusted":true},"cell_type":"code","source":"!wc -l /kaggle/input/riiid-test-answer-prediction/train.csv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"More than 100m rows in train dataset!"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"path = '/kaggle/input'\n\ntrain = pd.read_csv(f'{path}/riiid-test-answer-prediction/train.csv',\n                    usecols=['timestamp', 'user_id', 'content_id', 'content_type_id', 'user_answer', 'answered_correctly',\n                             'prior_question_elapsed_time', 'prior_question_had_explanation'],\n                       dtype={'timestamp': 'int64',\n                              'user_id': 'int32',\n                              'content_id': 'int16',\n                              'content_type_id': 'int8',\n                              'user_answer': 'int8',\n                              'answered_correctly': 'int8',\n                              'prior_question_elapsed_time': 'float32', \n                              'prior_question_had_explanation': 'boolean',\n                             }\n                      )\ntrain = train.sort_values(['timestamp'], ascending=True)\nquestions = pd.read_csv(f'{path}/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv(f'{path}/riiid-test-answer-prediction/lectures.csv')\nprint('Train shapes: ', train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['answered_correctly'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`answered_correctly` is our target! `-1` is a special value, we'll talk about it later."},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Exploring the features"},{"metadata":{},"cell_type":"markdown","source":"### timestamp\n\nIt is imprtant to remember that this is the time between this **user** interaction and the first event from that **user**. So starting time could be different for each user"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(train['timestamp'], bins=40);","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby(['user_id'])['timestamp'].max().sort_values(ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some users have really huge activity time!"},{"metadata":{},"cell_type":"markdown","source":"### user_id\n\nObviously this is an unique user id."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['user_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 1749 unique user ids in out data. Some have little data, some have a lot of data. But please notice that I loaded only a small part of the data."},{"metadata":{},"cell_type":"markdown","source":"### content_type_id\n\n0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\nSo we will be mainly using `content_type_id` equal to `1` at first, but generating some features based on the lectures should be also useful."},{"metadata":{"trusted":true},"cell_type":"code","source":"train['content_type_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.loc[train['content_type_id'] == 1, 'user_id'].nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Hm, interesting. It seems that not all people watched lectures."},{"metadata":{},"cell_type":"markdown","source":"### content_id\n\nId of the content - question or lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['content_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Does low numbers mean that only one person answered this question? I wonder why there are such question."},{"metadata":{},"cell_type":"markdown","source":"Let's have a look at the most popular question"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.loc[train['content_id'] == 6116]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.loc[train['content_id'] == 6116, 'user_answer'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions.loc[questions['question_id'] == 6116]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that a lot of people made mistakes answering this question."},{"metadata":{},"cell_type":"markdown","source":"I'll continue EDA later."},{"metadata":{},"cell_type":"markdown","source":"## Feature engineering\n\nLet's generate more features.\n\nsome code is taken from https://www.kaggle.com/ilialar/simple-eda-and-baseline https://www.kaggle.com/lgreig/simple-lgbm-baseline"},{"metadata":{"trusted":true},"cell_type":"code","source":"# filter out lectures\ntrain = train.loc[train['answered_correctly'] != -1].reset_index(drop=True)\ntrain = train.drop(['timestamp','content_type_id'], axis=1)\ntrain['prior_question_had_explanation'] = train['prior_question_had_explanation'].fillna(value = False).astype(bool)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df = train.groupby('user_id').agg({'answered_correctly': ['mean', 'count']}).copy()\nuser_answers_df.columns = ['mean_user_accuracy', 'questions_answered']\n\ncontent_answers_df = train.groupby('content_id').agg({'answered_correctly': ['mean', 'count']}).copy()\ncontent_answers_df.columns = ['mean_accuracy', 'question_asked']\n\n# user_content_answers_df = train.groupby(['user_id', 'content_id']).agg({'answered_correctly': ['mean', 'count']}).copy()\n# user_content_answers_df.columns = ['mean_user_content_accuracy', 'content_questions_answered']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we will use only a part of data for training, to avoid leaks and memory error"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.iloc[90000000:,:]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.merge(user_answers_df, how = 'left', on = 'user_id')\ntrain = train.merge(content_answers_df, how = 'left', on = 'content_id')\n# train = train.merge(user_content_answers_df, how = 'left', on = ['user_id', 'content_id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.fillna(value = 0.5, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# train['mean_diff1'] = train['mean_user_accuracy'] - train['mean_user_content_accuracy']\n# train['mean_diff2'] = train['mean_accuracy'] - train['mean_user_content_accuracy']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"le = LabelEncoder()\ntrain[\"prior_question_had_explanation\"] = le.fit_transform(train[\"prior_question_had_explanation\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.sort_values(['user_id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y = train['answered_correctly']\n\ncolumns = ['mean_user_accuracy', 'questions_answered', 'mean_accuracy', 'question_asked',\n           'prior_question_had_explanation',# 'mean_diff1', 'mean_diff2', 'mean_user_content_accuracy'\n          ]\nX = train[columns]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"scores = []\nfeature_importance = pd.DataFrame()\nmodels = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"params = {'num_leaves': 32,\n          'max_bin': 300,\n          'objective': 'binary',\n          'max_depth': 13,\n          'learning_rate': 0.03,\n          \"boosting_type\": \"gbdt\",\n          \"metric\": 'auc',\n         }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"columns = ['mean_user_accuracy', 'questions_answered', 'mean_accuracy', 'question_asked',\n#            'prior_question_had_explanation', 'mean_diff1', 'mean_diff2'\n          ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"folds = StratifiedKFold(n_splits=5, shuffle=False)\nfor fold_n, (train_index, valid_index) in enumerate(folds.split(X, y)):\n    print(f'Fold {fold_n} started at {time.ctime()}')\n    X_train, X_valid = X[columns].iloc[train_index], X[columns].iloc[valid_index]\n    y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n    model = lgb.LGBMClassifier(**params, n_estimators=700, n_jobs = 1)\n    model.fit(X_train, y_train, \n            eval_set=[(X_train, y_train), (X_valid, y_valid)], eval_metric=eval_auc,\n            verbose=1000, early_stopping_rounds=10)\n    score = max(model.evals_result_['valid_1']['auc'])\n    \n    models.append(model)\n    scores.append(score)\n\n    fold_importance = pd.DataFrame()\n    fold_importance[\"feature\"] = columns\n    fold_importance[\"importance\"] = model.feature_importances_\n    fold_importance[\"fold\"] = fold_n + 1\n    feature_importance = pd.concat([feature_importance, fold_importance], axis=0)\n    break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('CV mean score: {0:.4f}, std: {1:.4f}.'.format(np.mean(scores), np.std(scores)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_importance[\"importance\"] /= 1\ncols = feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n    by=\"importance\", ascending=False)[:50].index\n\nbest_features = feature_importance.loc[feature_importance.feature.isin(cols)]\n\nplt.figure(figsize=(16, 12));\nsns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False));\nplt.title('LGB Features (avg over folds)');\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del X, y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Making predictions.\n\nCode is taken from https://www.kaggle.com/sishihara/riiid-lgbm-5cv-benchmark"},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    y_preds = []\n    test_df = test_df.merge(user_answers_df, how = 'left', on = 'user_id')\n    test_df = test_df.merge(content_answers_df, how = 'left', on = 'content_id')\n#     test_df = test_df.merge(user_content_answers_df, how = 'left', on = ['user_id', 'content_id'])\n#     test_df['mean_diff1'] = test_df['mean_user_accuracy'] - test_df['mean_user_content_accuracy']\n#     test_df['mean_diff2'] = test_df['mean_accuracy'] - test_df['mean_user_content_accuracy']\n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\n    test_df = test_df.loc[test_df['content_type_id'] == 0].reset_index(drop=True)\n    test_df.fillna(value = 0.5, inplace = True)\n    test_df[\"prior_question_had_explanation_enc\"] = le.fit_transform(test_df[\"prior_question_had_explanation\"])\n\n    for model in models:\n        y_pred = model.predict_proba(test_df[columns], num_iteration=model.best_iteration_)[:, 1]\n        y_preds.append(y_pred)\n\n    y_preds = sum(y_preds) / len(y_preds)\n    test_df['answered_correctly'] = y_preds\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}