{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import lightgbm as lgb\nimport numpy as np\nimport pandas as pd\nfrom sklearn import preprocessing\nfrom sklearn.model_selection import KFold\nfrom sklearn.metrics import roc_auc_score\nfrom sklearn.preprocessing import LabelEncoder\n\nimport riiideducation\n\npd.options.display.max_columns = None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('../input/riiid-test-answer-prediction/train.csv', low_memory=False, nrows=10**7,\n                      dtype={\n                          'row_id': 'int64', 'timestamp': 'int64', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n                              'task_container_id': 'int16', 'user_answer': 'int8', 'answered_correctly': 'int8', 'prior_question_elapsed_time': 'float32', \n                             'prior_question_had_explanation': 'boolean',\n                      })\n\n# train_df = train_df.query('answered_correctly != -1').reset_index(drop=True)\n# train_df['prior_question_had_explanation'] = train_df['prior_question_had_explanation'].astype(float) 後でやる","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"features_part_df と train_part_df に分ける。"},{"metadata":{"trusted":true},"cell_type":"code","source":"# 900万行・・・特徴量作成\nfeatures_part_df = train_df.iloc[:int( 9 / 10 * len(train_df) )]\n# 100万行・・・最新の100万件 \ntrain_part_df = train_df.iloc[int( 9 / 10 * len(train_df) ):]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"features_part_df で新しい特徴量を作成"},{"metadata":{"trusted":true},"cell_type":"code","source":"# 追加するデータ1 user_answers_df\ntrain_questions_only_df = features_part_df[features_part_df['answered_correctly'] != -1]\n\ngrouped_by_user_df = train_questions_only_df.groupby('user_id')\n\nuser_answers_df = grouped_by_user_df.agg({'answered_correctly': ['mean', 'count', 'std', 'median', 'skew'], 'prior_question_had_explanation': ['mean']}).copy()\n\n# ---- ここから新しく追加 ---- #\n\nuser_answers_df[('test', 'score')] = user_answers_df[('answered_correctly', 'mean')] * (user_answers_df[('answered_correctly', 'count')])\n\n# 平均\nmean = user_answers_df[('test', 'score')].mean()\n# 偏差　deviation\nuser_answers_df[('test', 'deviation')] = user_answers_df[('test', 'score')] - mean\n# 平方数　Square\nuser_answers_df[('test', 'square')] = user_answers_df[('test', 'deviation')] * user_answers_df[('test', 'deviation')]\n# 分散 全員の平方数の合計÷人数=分散\nvariance = user_answers_df[('test', 'square')].sum() / len(user_answers_df.index)\nstandard_deviation = np.sqrt(variance)\n\n# 偏差値　(平均との差×10÷標準偏差)+50=○○\nuser_answers_df[('test', 'Deviation_Value')] = (user_answers_df[('test', 'deviation')] * 10 / standard_deviation) + 50","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df[('test', 'smart_user')] = 0\n\n# 復習率が、80%以上のユーザー ＝　smart_user 1 にする\n# 偏差値が高い傾向にあるから\n\nfor index, row in user_answers_df.iterrows():\n    if row[('prior_question_had_explanation','mean')] >= 0.8:\n        user_answers_df[('test','smart_user')][index] = 1\n        \nuser_answers_df.columns = ['mean_user_accuracy', 'questions_answered', 'std_user_accuracy', 'median_user_accuracy', 'skew_user_accuracy', 'mean_prior_question_had_explanation', 'user_score', 'score_deviation', 'score_square', 'Deviation_Value', 'smart_user']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answers_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 追加するデータ2 questions_df\nquestions_df = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\n\ngrouped_by_content_df = train_questions_only_df.groupby('content_id')\n\ncontent_answers_df = grouped_by_content_df.agg({'answered_correctly': ['mean', 'count', 'std', 'median', 'skew'] }).copy()\ncontent_answers_df.columns = ['mean_accuracy', 'question_asked', 'std_accuracy', 'median_accuracy', 'skew_accuracy']\n\nquestions_df = questions_df.merge(content_answers_df, left_on = 'question_id', right_on = 'content_id', how = 'left')\n\nbundle_dict = questions_df['bundle_id'].value_counts().to_dict()\n\n# right_answers 正解数\nquestions_df['right_answers'] = questions_df['mean_accuracy'] * questions_df['question_asked']\n\nquestions_df['bundle_size'] = questions_df['bundle_id'].apply(lambda x: bundle_dict[x])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 追加するデータ3 bundle_answers_df\ngrouped_by_bundle_df = questions_df.groupby('bundle_id')\n\nbundle_answers_df = grouped_by_bundle_df.agg({'right_answers': 'sum', 'question_asked': 'sum'}).copy()\nbundle_answers_df.columns = ['bundle_right_answers', 'bundle_questions_asked']\n\nbundle_answers_df['bundle_accuracy'] = bundle_answers_df['bundle_right_answers'] / bundle_answers_df['bundle_questions_asked']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bundle_answers_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 追加するデータ4 part_answers_df\ngrouped_by_part_df = questions_df.groupby('part')\n\npart_answers_df = grouped_by_part_df.agg({'right_answers': 'sum', 'question_asked': 'sum'}).copy()\n\npart_answers_df.columns = ['part_right_answers', 'part_questions_asked']\npart_answers_df['part_accuracy'] = part_answers_df['part_right_answers'] / part_answers_df['part_questions_asked']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"part_answers_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 講義(-1)以外を抽出 train\ntrain_part_df = train_part_df[train_part_df['answered_correctly'] != -1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# user_answers_df\ntrain_part_df = train_part_df.merge(user_answers_df, how='left', on='user_id')\n\n# questions_df\ntrain_part_df = train_part_df.merge(questions_df, how='left', left_on='content_id', right_on='question_id')\n\n# bundle_answers_df\ntrain_part_df = train_part_df.merge(bundle_answers_df, how='left', on='bundle_id')\n\n# part_answers_df\ntrain_part_df = train_part_df.merge(part_answers_df, how='left', on='part')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ユーザーが質問に回答した後、説明と正しい回答を確認したかどうか 欠損値をFalseと置く、 astypeでデータ型の変換(キャスト)\ntrain_part_df['prior_question_had_explanation'] = train_part_df['prior_question_had_explanation'].fillna(value=False).astype(bool)\n\ntrain_part_df.fillna(value = -1, inplace = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ラベルエンコーディング\nle = LabelEncoder()\ntrain_part_df[\"prior_question_had_explanation\"] = le.fit_transform(train_part_df[\"prior_question_had_explanation\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_part_df.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# 旧\n# features = [\n#     'timestamp','mean_user_accuracy', 'questions_answered','mean_accuracy',\n#     'question_asked','prior_question_elapsed_time', 'prior_question_had_explanation',\n#     'bundle_size', 'bundle_accuracy','part_accuracy', 'right_answers'\n# ]\n\n\n# features = [\n#     'timestamp','prior_question_elapsed_time', 'prior_question_had_explanation',\n#     'mean_user_accuracy', 'questions_answered', 'std_user_accuracy',\n#     'median_user_accuracy', 'skew_user_accuracy','mean_accuracy',\n#     'question_asked', 'std_accuracy', 'median_accuracy', 'skew_accuracy',\n#     'bundle_size','bundle_accuracy', 'part_accuracy','user_score',\n#     'score_deviation', 'score_square', 'Deviation_Value',\n# ]\n\n# 新\nfeatures = [\n    'timestamp','prior_question_elapsed_time', 'prior_question_had_explanation',\n       'mean_user_accuracy', 'questions_answered', 'std_user_accuracy',\n       'median_user_accuracy', 'skew_user_accuracy',\n       'mean_prior_question_had_explanation','Deviation_Value', 'smart_user',\n       'mean_accuracy','question_asked', 'std_accuracy', 'median_accuracy', 'skew_accuracy',\n       'bundle_size','bundle_accuracy','part_accuracy'\n]\n\ntarget = 'answered_correctly'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train = train_part_df[features]\ny_train = train_part_df[target]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"X_train","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"models = []\noof_train = np.zeros(len(X_train),) ### array([0., 0., 0., ..., 0., 0., 0.])\ncategorical_features = ['prior_question_had_explanation']\n\nparams = {\n    'objective': 'binary',\n    'max_bin': 300,\n    'learning_rate': 0.05,\n    'num_leaves': 40\n}\n\nn_tr = round(981094 * 0.9)\n\nX_tr = X_train[:n_tr]\nX_val = X_train[n_tr:]\n\ny_tr = y_train[:n_tr]\ny_val = y_train[n_tr:]\n\nlgb_train = lgb.Dataset(X_tr, y_tr, categorical_feature=categorical_features)\nlgb_eval = lgb.Dataset(X_val, y_val, reference=lgb_train, categorical_feature=categorical_features)\n\nmodel = lgb.train(\n    params,\n    lgb_train,\n    valid_sets=[lgb_train, lgb_eval],\n    verbose_eval=10,\n    num_boost_round=1000,\n    early_stopping_rounds=100 # 30, 50 くらいで試す　\n)\n\noof_train = model.predict(X_val, num_iteration=model.best_iteration)\n\nmodels.append(model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# importanceを表示する\nimportance = pd.DataFrame(model.feature_importance(), index=X_train.columns, columns=['importance'])\nresult = importance.sort_values('importance', ascending=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ROC曲線のAUCスコア・・・曲線下の面積を意味するらしい。\nroc_auc_score(y_val, oof_train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"iter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n    y_preds = []\n    \n    test_df = test_df.merge(user_answers_df, how = 'left', on = 'user_id')\n    test_df = test_df.merge(questions_df, how = 'left', left_on = 'content_id', right_on = 'question_id')\n    test_df = test_df.merge(bundle_answers_df, how = 'left', on = 'bundle_id')\n    test_df = test_df.merge(part_answers_df, how = 'left', on = 'part')\n    \n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].fillna(value = False).astype(bool)\n    test_df.fillna(value = -1, inplace = True)\n    X_test = test_df[features]\n    \n    for model in models:\n        y_pred = model.predict(X_test, num_iteration=model.best_iteration)\n        y_preds.append(y_pred)\n        \n    y_preds = sum(y_preds) / len(y_preds)\n    test_df['answered_correctly'] = y_preds\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## discussion"},{"metadata":{},"cell_type":"markdown","source":"* Knowledge tracing (KT)について\n\nKnowledge tracing (KT) refers to the problem of predicting future learner performance given their past performance in educational applications\n\nEach student's knowledge is modeled by estimating the performance of the student on the learning activities.\n\nKnowledgeTracing、EdNet, Artificial Intelligence in Education (AIEd)に関する記事\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/188911\n\n* 時制に関して\n\n時制データには、ターゲットエンコーディングをすることが重要。(https://maxhalford.github.io/blog/pandas-tricks/#target-encoding-for-time-series)\n\nターゲットエンコーディングについて (https://maxhalford.github.io/blog/target-encoding/)\n\nThe one thing I do in every time series competition is target encoding. In short, the goal of target encoding is to replace a category by the average of the target values of the rows that belong to said category. Naturally, you’re not limited to using an average. You can also use Bayesian target encoding if some of your categories are rare.\n\nTarget encoding is very important for time series data. It has been used in almost every top model on Kaggle time series competitions, such as ASHRAE, Recruit Restaurants, and Wikipedia Web Traffic. When you do target encoding on temporal data, you need to be wary of not leaking current and future information into the aggregate of each row. Basically, you need to shift the target values within each group backwards. Indeed, for any given moment, we want our aggregate to pertain to past data only.\n\n参考ディスカション (https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189437)\n\n* Timestamp と prior_question_elaped_time について\n\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189351\n　"},{"metadata":{},"cell_type":"markdown","source":"* prior_question_elapsed_time\n\nCORRECT: Note that the time is the average time a user took to solve each question in the previous bundle\n\nThat is, if three questions are in a previous bundle, the questions in the following bundle share the same prior_question_elapsed_time and the value is the total time the user took to solve all three questions in the previous bundle, divided by three. We apologize for any inconvenience. The correction has been updated in the data description.\n\nSuppose a user spent 60 seconds answering their first question at 10:00. The timestamp for that row would read 10:00 (we'll skip the normalization for simplicity), and the prior_question_elapsed_time would be null since it's the first question. If they took 30 seconds to answer their next question at 11:00 the second row's timestamp would be 11:00 and the prior_question_elapsed_time would be 60 seconds."},{"metadata":{},"cell_type":"markdown","source":"データは、TOEICの内容らしい\nhttps://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190191"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}