{"cells":[{"metadata":{},"cell_type":"markdown","source":"# コンペ概要\n\nTOEICの学習アプリが題材．ユーザが出された問題に正解できる確率を，ユーザの今までの解答や勉強状況から予測する．注意点は，単純にある問題に対して回答できるかの予測ではなく、ユーザーの解いた問題数(時間経過)に伴って、ユーザーの正解率は変化するということ．間違えた問題も、2回目に出された場合は正解出来る確率が上がるはず、と考えられる．このような人の知識をモデリングする問題をKnowledge Trackingと言う．"},{"metadata":{},"cell_type":"markdown","source":"データ総数が1億件と大きすぎるため，pandasではデータ読み込みが遅すぎるので，datatableを使用する．"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"!pip install ../input/python-datatable/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl > /dev/null 2>&1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport random\nimport pandas as pd\nimport joblib\nfrom collections import defaultdict\nimport datatable as dt\nimport lightgbm as lgb\nfrom matplotlib import pyplot as plt\nimport riiideducation\nfrom sklearn.metrics import roc_auc_score\nimport gc\n\n_ = np.seterr(divide='ignore', invalid='ignore')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 前処理"},{"metadata":{},"cell_type":"markdown","source":"データタイプをあらかじめ指定してから読み込むと，データ読み込み速度が速くなる．データタイプをいちいち推測する必要がなくなるため．"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_types_dict = {\n    'timestamp': 'int64',\n    'user_id': 'int32', \n    'content_id': 'int16', \n    'content_type_id':'int8', \n    'task_container_id': 'int16',\n    #'user_answer': 'int8',\n    'answered_correctly': 'int8', \n    'prior_question_elapsed_time': 'float32', \n    'prior_question_had_explanation': 'bool'\n}\ntarget = 'answered_correctly'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n# train_df = dt.fread('../input/riiid-test-answer-prediction/train.csv', columns=set(data_types_dict.keys())).to_pandas()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# メモリオーバーするため，とりあえず100万件で．\n\ntrain_df = pd.read_csv('../input/riiid-test-answer-prediction/train.csv', nrows=1000000)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"カテゴリ変数をダミー変数（0と1）に変換"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df['type_of'] = lectures_df['type_of'].replace('solving question', 'solving_question')\n\nlectures_df = pd.get_dummies(lectures_df, columns=['part', 'type_of'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"訓練データと講義データをマージしたtrain_lecturesをつくる．問題を解いた後，一部は講義を見ているので，これを紐づける．"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_lectures = train_df[train_df.content_type_id == True].merge(lectures_df, left_on='content_id', right_on='lecture_id', how='left')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_lectures.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ユーザーごとに，受けた講義を合計する．たくさん講義を受けた人ほど問題の正解率が上がるはずだから．"},{"metadata":{"trusted":true},"cell_type":"code","source":"part_lectures_columns = [column for column in lectures_df.columns if column.startswith('part')]\ntypes_of_lectures_columns = [column for column in lectures_df.columns if column.startswith('type_of_')]\nuser_lecture_stats_part = train_lectures.groupby('user_id',as_index = False)[part_lectures_columns + types_of_lectures_columns].sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_lecture_stats_part.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# データ形式を整えておく．\n\nlecturedata_types_dict = {   \n    'user_id': 'int32', \n    'part_1': 'int8',\n    'part_2': 'int8',\n    'part_3': 'int8',\n    'part_4': 'int8',\n    'part_5': 'int8',\n    'part_6': 'int8',\n    'part_7': 'int8',\n    'type_of_concept': 'int8',\n    'type_of_intention': 'int8',\n    'type_of_solving_question': 'int8',\n    'type_of_starter': 'int8'\n}\nuser_lecture_stats_part = user_lecture_stats_part.astype(lecturedata_types_dict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"データが大きすぎてすぐにメモリオーバーしてしまうため，使わなくなったデータは消去し，メモリを解放しておく．"},{"metadata":{"trusted":true},"cell_type":"code","source":"#clearing memory\ndel(train_lectures)\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ユーザごとに，受けた講義数をsumで，解いた問題と受けた講義の合計をcountで取得する．"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_lecture_agg = train_df.groupby('user_id')['content_type_id'].agg(['sum', 'count'])\nuser_lecture_agg = user_lecture_agg.astype('int16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"user_lecture_agg.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ユーザごとに，受けた講義の累積和をcumsumで，解いた問題と受けた講義の累積和をcumcountで取得する．"},{"metadata":{"trusted":true},"cell_type":"code","source":"cum = train_df.groupby('user_id')['content_type_id'].agg(['cumsum', 'cumcount'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cum.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"作った特徴量を訓練データに加える"},{"metadata":{"trusted":true},"cell_type":"code","source":"cum['cumcount']=cum['cumcount']+1\n\n# ユーザーインタラクションの数（受けた講義の数＋解いた問題の数）\ntrain_df['user_interaction_count'] = cum['cumcount'] \n\n# インタラクション間の平均時間\ntrain_df['user_interaction_timestamp_mean'] = train_df['timestamp']/cum['cumcount'] \n\n# 受けた講義の累積和\ntrain_df['user_lecture_sum'] = cum['cumsum'] \n\n# 講義を受けるか，問題を解くかの割合\ntrain_df['user_lecture_lv'] = cum['cumsum'] / cum['cumcount']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# データタイプを揃える\ntrain_df.user_lecture_lv=train_df.user_lecture_lv.astype('float16')\ntrain_df.user_lecture_sum=train_df.user_lecture_sum.astype('int16')\ntrain_df.user_interaction_count=train_df.user_interaction_count.astype('int16')\ntrain_df['user_interaction_timestamp_mean']=train_df['user_interaction_timestamp_mean']/(1000*3600)\ntrain_df.user_interaction_timestamp_mean=train_df.user_interaction_timestamp_mean.astype('float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del cum\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"prior_question_had_explanation: 前回質問に回答した後に解説をちゃんと読んだかどうか．\n正解したか否かの情報がないデータは除いてしまう．欠損データは全体の1%に満たず，学習に影響がないため．"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['prior_question_had_explanation'].fillna(False, inplace=True)\ntrain_df = train_df.astype(data_types_dict)\ntrain_df = train_df[train_df[target] != -1].reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"前回問題の解説を見たか否かの平均正解率．見ている方が正解率は高いはずである．"},{"metadata":{"trusted":true},"cell_type":"code","source":"content_explation_agg = train_df[[\"content_id\",\"prior_question_had_explanation\",target]].groupby([\"content_id\",\"prior_question_had_explanation\"])[target].agg(['mean'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_explation_agg.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* unstack(): 行を列にピボットする．\n* reset_index(): インデックスをリセットする．"},{"metadata":{"trusted":true},"cell_type":"code","source":"content_explation_agg = content_explation_agg.unstack()\ncontent_explation_agg = content_explation_agg.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_explation_agg.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_explation_agg.columns = ['content_id', 'content_explation_false_mean','content_explation_true_mean']\n\n#データタイプをそろえる\ncontent_explation_agg.content_id=content_explation_agg.content_id.astype('int16')\ncontent_explation_agg.content_explation_false_mean=content_explation_agg.content_explation_false_mean.astype('float16')\ncontent_explation_agg.content_explation_true_mean=content_explation_agg.content_explation_true_mean.astype('float16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"content_explation_agg.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df[\"attempt_no\"] = 1\ntrain_df.attempt_no=train_df.attempt_no.astype('int8')\nattempt_no_agg=train_df.groupby([\"user_id\",\"content_id\"])[\"attempt_no\"].agg(['sum']).astype('int8')\nattempt_no_agg.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"ユーザーごとの問題解答数の累積和"},{"metadata":{"trusted":true},"cell_type":"code","source":"#attempt_no_agg=attempt_no_agg.astype('int8')\ntrain_df[\"attempt_no\"] = train_df[[\"user_id\",\"content_id\",'attempt_no']].groupby([\"user_id\",\"content_id\"])[\"attempt_no\"].cumsum()\n#attempt_no_agg=attempt_no_agg.reset_index()\nattempt_no_agg=attempt_no_agg[attempt_no_agg['sum'] >1]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## timestamp\n\nprior_question_elapsed_timeの欠損値を平均値で埋める．"},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_question_elapsed_time_mean=train_df['prior_question_elapsed_time'].mean()\ntrain_df['prior_question_elapsed_time'].fillna(prior_question_elapsed_time_mean, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"最初のイベントから最後のイベントまでの間"},{"metadata":{"trusted":true},"cell_type":"code","source":"max_timestamp_u = train_df[['user_id','timestamp']].groupby(['user_id']).agg(['max']).reset_index()\nmax_timestamp_u.columns = ['user_id', 'max_time_stamp']\nmax_timestamp_u.user_id=max_timestamp_u.user_id.astype('int32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"max_timestamp_u.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['lagtime'] = train_df.groupby('user_id')['timestamp'].shift()\n\nmax_timestamp_u2 = train_df[['user_id','lagtime']].groupby(['user_id']).agg(['max']).reset_index()\nmax_timestamp_u2.columns = ['user_id', 'max_time_stamp2']\nmax_timestamp_u2.user_id=max_timestamp_u2.user_id.astype('int32')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"次のインタラクションまでの時間．欠損値は平均値で埋める．"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['lagtime']=train_df['timestamp']-train_df['lagtime']\nlagtime_mean=train_df['lagtime'].mean()\ntrain_df['lagtime'].fillna(lagtime_mean, inplace=True)\ntrain_df['lagtime']=train_df['lagtime']/(1000*3600)\ntrain_df.lagtime=train_df.lagtime.astype('float32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# インタラクション間隔が2, 3のときも同様にする．\n\ntrain_df['lagtime2'] = train_df.groupby('user_id')['timestamp'].shift(2)\n\nmax_timestamp_u3 = train_df[['user_id','lagtime2']].groupby(['user_id']).agg(['max']).reset_index()\nmax_timestamp_u3.columns = ['user_id', 'max_time_stamp3']\nmax_timestamp_u3.user_id=max_timestamp_u3.user_id.astype('int32')\n\ntrain_df['lagtime2']=train_df['timestamp']-train_df['lagtime2']\nlagtime_mean2=train_df['lagtime2'].mean()\ntrain_df['lagtime2'].fillna(lagtime_mean2, inplace=True)\ntrain_df['lagtime2']=train_df['lagtime2']/(1000*3600)\ntrain_df.lagtime2=train_df.lagtime2.astype('float32')\n\ntrain_df['lagtime3'] = train_df.groupby('user_id')['timestamp'].shift(3)\n\ntrain_df['lagtime3']=train_df['timestamp']-train_df['lagtime3']\nlagtime_mean3=train_df['lagtime3'].mean()\ntrain_df['lagtime3'].fillna(lagtime_mean3, inplace=True)\ntrain_df['lagtime3']=train_df['lagtime3']/(1000*3600)\ntrain_df.lagtime3=train_df.lagtime3.astype('float32')\n\ntrain_df['timestamp']=train_df['timestamp']/(1000*3600)\ntrain_df.timestamp=train_df.timestamp.astype('float16')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"user_prior_question_elapsed_time: 最後から2番目の問題の回答にかかった平均時間"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_prior_question_elapsed_time = train_df[['user_id','prior_question_elapsed_time']].groupby(['user_id']).tail(1)\nuser_prior_question_elapsed_time.columns = ['user_id', 'prior_question_elapsed_time']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"delta_prior_question_elapsed_time：前回の問題と比べて回答時間がどれくらい減少しているか．"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['delta_prior_question_elapsed_time'] = train_df.groupby('user_id')['prior_question_elapsed_time'].shift()\ntrain_df['delta_prior_question_elapsed_time']=train_df['prior_question_elapsed_time']-train_df['delta_prior_question_elapsed_time']\ndelta_prior_question_elapsed_time_mean=train_df['delta_prior_question_elapsed_time'].mean()\ntrain_df['delta_prior_question_elapsed_time'].fillna(delta_prior_question_elapsed_time_mean, inplace=True)\ntrain_df.delta_prior_question_elapsed_time=train_df.delta_prior_question_elapsed_time.astype('int32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"lag: ユーザーの正解率の推移"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['lag'] = train_df.groupby('user_id')[target].shift()\n\ncum = train_df.groupby('user_id')['lag'].agg(['cumsum', 'cumcount'])\n##cum['cumcount']=cum['cumcount']+1\nuser_agg = train_df.groupby('user_id')['lag'].agg(['sum', 'count']).astype('int16')\ncum['cumsum'].fillna(0, inplace=True)\n\ntrain_df['user_correctness'] = cum['cumsum'] / cum['cumcount']\ntrain_df['user_correct_count'] = cum['cumsum']\ntrain_df['user_uncorrect_count'] = cum['cumcount']-cum['cumsum']\n#train_df['user_answer_count'] = cum['cumcount']\ntrain_df.drop(columns=['lag'], inplace=True)\ntrain_df['user_correctness'].fillna(0.67, inplace=True)\ntrain_df.user_correctness=train_df.user_correctness.astype('float16')\ntrain_df.user_correct_count=train_df.user_correct_count.astype('int16')\ntrain_df.user_uncorrect_count=train_df.user_uncorrect_count.astype('int16')\n#train_df.user_answer_count=train_df.user_answer_count.astype('int16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del cum\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.prior_question_had_explanation=train_df.prior_question_had_explanation.astype('int8')\nexplanation_agg = train_df.groupby('user_id')['prior_question_had_explanation'].agg(['sum', 'count'])\nexplanation_agg=explanation_agg.astype('int16')\n# explanation_agg.sum=explanation_agg.sum.astype('int16')\n# explanation_agg.count=explanation_agg.count.astype('int16')\n#explanation_agg.var=explanation_agg.var.astype('float16')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"どれくらい説明を見たか．"},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_df['lag'] = train_df.groupby('user_id')['prior_question_had_explanation'].shift()\n\ncum = train_df.groupby('user_id')['prior_question_had_explanation'].agg(['cumsum', 'cumcount'])\ncum['cumcount']=cum['cumcount']+1\ntrain_df['explanation_mean'] = cum['cumsum'] / cum['cumcount']\ntrain_df['explanation_true_count'] = cum['cumsum'] \ntrain_df['explanation_false_count'] =  cum['cumcount']-cum['cumsum']\n#train_df.drop(columns=['lag'], inplace=True)\n\ntrain_df.explanation_mean=train_df.explanation_mean.astype('float16')\ntrain_df.explanation_true_count=train_df.explanation_true_count.astype('int16')\ntrain_df.explanation_false_count=train_df.explanation_false_count.astype('int16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del cum\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"var: 分散\ncontent_agg: 問題ごとの正解が1として加算．sumは合計の正解数．countは解かれた数"},{"metadata":{"trusted":true},"cell_type":"code","source":"content_agg = train_df.groupby('content_id')[target].agg(['sum', 'count','var'])\ntask_container_agg = train_df.groupby('task_container_id')[target].agg(['sum', 'count','var'])\ncontent_agg=content_agg.astype('float32')\ntask_container_agg=task_container_agg.astype('float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"task_container_id: 質問または講義のバッチのIDコード．同じカテゴリの問題は同じIDを共有している．"},{"metadata":{"trusted":true},"cell_type":"code","source":"task_container_agg","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* task_container_uncor_count：task_container_idごとの誤答数\n* task_container_cor_count：正答数\n* task_container_std：標準偏差\n* task_container_correctness：正解率"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df['task_container_uncor_count'] = train_df['task_container_id'].map(task_container_agg['count']-task_container_agg['sum']).astype('int32')\ntrain_df['task_container_cor_count'] = train_df['task_container_id'].map(task_container_agg['sum']).astype('int32')\ntrain_df['task_container_std'] = train_df['task_container_id'].map(task_container_agg['var']).astype('float16')\ntrain_df['task_container_correctness'] = train_df['task_container_id'].map(task_container_agg['sum'] / task_container_agg['count'])\ntrain_df.task_container_correctness=train_df.task_container_correctness.astype('float16')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* content_elapsed_time_agg：前の質問バンドルの各質問に回答するのにかかった平均時間．問題ごと．\n* content_had_explanation_agg：問題ごと．前の問題の解説を見たか．"},{"metadata":{"trusted":true},"cell_type":"code","source":"content_elapsed_time_agg=train_df.groupby('content_id')['prior_question_elapsed_time'].agg(['mean'])\ncontent_had_explanation_agg=train_df.groupby('content_id')['prior_question_had_explanation'].agg(['mean'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# questions"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df = pd.read_csv(\n    '../input/riiid-test-answer-prediction/questions.csv', \n    usecols=[0, 1,3,4],\n    dtype={'question_id': 'int16','bundle_id': 'int16', 'part': 'int8','tags': 'str'}\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* tag: いくつかの質問をまとめるコード\n* part: toeicテストのセクション\n* bundle_id: 質問が一緒に提供されるコード"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"bundleが重複しているものはまとめる"},{"metadata":{"trusted":true},"cell_type":"code","source":"bundle_agg = questions_df.groupby('bundle_id')['question_id'].agg(['count'])\nquestions_df['content_sub_bundle'] = questions_df['bundle_id'].map(bundle_agg['count']).astype('int8')\nquestions_df['tags'].fillna('188', inplace=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"tag"},{"metadata":{"trusted":true},"cell_type":"code","source":"def gettags(tags,num):\n    tags_splits=tags.split(\" \")\n    result='' \n    for t in tags_splits:\n        x=int(t)\n        if(x<32*(num+1) and x>=32*num):#num \n            result=result+' '+t\n    return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\nfor num in range(0,6):\n    questions_df[\"tags\"+str(num)] = questions_df[\"tags\"].apply(lambda row: gettags(row,num))\n    le = LabelEncoder()\n    le.fit(np.unique(questions_df['tags'+str(num)].values))\n    #questions_df[['tags'+str(num)]=\n    questions_df['tags'+str(num)]=questions_df[['tags'+str(num)]].apply(le.transform)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df_dict = {   \n    'tags0': 'int8',\n    'tags1': 'int8',\n    'tags2': 'int8',\n    'tags3': 'int8',\n    'tags4': 'int8',\n    'tags5': 'int8',\n    #'tags6': 'int8',\n    #'tags7': 'int8'\n}\nquestions_df = questions_df.astype(questions_df_dict)\nquestions_df.drop(columns=['tags'], inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df['part_bundle_id']=questions_df['part']*100000+questions_df['bundle_id']\nquestions_df.part_bundle_id=questions_df.part_bundle_id.astype('int32')\n# tag = questions_df[\"tags\"].str.split(\" \", n = 10, expand = True)\n# tag.columns = ['tags1','tags2','tags3','tags4','tags5','tags6']\n# #\n\n# tag.fillna(0, inplace=True)\n# tag = tag.astype('int16')\n# questions_df =  pd.concat([questions_df,tag],axis=1).drop(['tags'],axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df.rename(columns={'question_id':'content_id'}, inplace=True)\nquestions_df = pd.merge(questions_df, content_explation_agg, on='content_id', how='left',right_index=True)#\n# questions_df.content_explation_false_mean=questions_df.content_explation_false_mean.astype('float16')\n# questions_df.content_explation_true_mean=questions_df.content_explation_true_mean.astype('float16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del content_explation_agg\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df['content_correctness'] = questions_df['content_id'].map(content_agg['sum'] / content_agg['count'])\nquestions_df.content_correctness=questions_df.content_correctness.astype('float16')\nquestions_df['content_correctness_std'] = questions_df['content_id'].map(content_agg['var'])\nquestions_df.content_correctness_std=questions_df.content_correctness_std.astype('float16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df['content_uncorrect_count'] = questions_df['content_id'].map(content_agg['count']-content_agg['sum'])\nquestions_df['content_uncorrect_count'].fillna(False, inplace=True)\nquestions_df['content_uncorrect_count'] = questions_df['content_uncorrect_count'].astype('int32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df['content_correct_count'] = questions_df['content_id'].map(content_agg['sum'])\nquestions_df['content_correct_count'].fillna(False, inplace=True)\nquestions_df['content_correct_count'] = questions_df['content_correct_count'].astype('int32').astype('int32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_df['content_elapsed_time_mean'] = questions_df['content_id'].map(content_elapsed_time_agg['mean'])\nquestions_df.content_elapsed_time_mean=questions_df.content_elapsed_time_mean.astype('float16')\nquestions_df['content_had_explanation_mean'] = questions_df['content_id'].map(content_had_explanation_agg['mean'])\nquestions_df.content_had_explanation_mean=questions_df.content_had_explanation_mean.astype('float16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del content_elapsed_time_agg\ndel content_had_explanation_agg\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"part_agg = questions_df.groupby('part')['content_correctness'].agg(['mean', 'var'])\nquestions_df['part_correctness_mean'] = questions_df['part'].map(part_agg['mean'])\nquestions_df['part_correctness_std'] = questions_df['part'].map(part_agg['var'])\nquestions_df.part_correctness_mean=questions_df.part_correctness_mean.astype('float16')\nquestions_df.part_correctness_std=questions_df.part_correctness_std.astype('float16')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"part_agg = questions_df.groupby('part')['content_uncorrect_count'].agg(['sum'])\nquestions_df['part_uncor_count'] = questions_df['part'].map(part_agg['sum']).astype('int32')\n#\npart_agg = questions_df.groupby('part')['content_correct_count'].agg(['sum'])\nquestions_df['part_cor_count'] = questions_df['part'].map(part_agg['sum']).astype('int32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bundle_agg = questions_df.groupby('bundle_id')['content_correctness'].agg(['mean'])\nquestions_df['bundle_correctness_mean'] = questions_df['bundle_id'].map(bundle_agg['mean'])\nquestions_df.bundle_correctness_mean=questions_df.bundle_correctness_mean.astype('float16')\n\n# bundle_agg = questions_df.groupby('bundle_id')['content_uncorrect_count'].agg(['sum'])\n# questions_df['bundle_uncor_count'] = questions_df['bundle_id'].map(bundle_agg['sum']).astype('int32')\n# #\n# bundle_agg = questions_df.groupby('bundle_id')['content_correct_count'].agg(['sum'])\n# questions_df['bundle_cor_count'] = questions_df['bundle_id'].map(bundle_agg['sum']).astype('int32')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del content_agg\ndel bundle_agg\ndel part_agg\n#del tags1_agg\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train"},{"metadata":{"trusted":true},"cell_type":"code","source":"features_dict = {\n    #'user_id',\n    'timestamp':'float16',#\n    'user_interaction_count':'int16',\n    'user_interaction_timestamp_mean':'float32',\n    'lagtime':'float32',#\n    'lagtime2':'float32',\n    'lagtime3':'float32',\n    #'lagtime_mean':'int32',\n    'content_id':'int16',\n    'task_container_id':'int16',\n    'user_lecture_sum':'int16',#\n    'user_lecture_lv':'float16',##\n    'prior_question_elapsed_time':'float32',#\n    'delta_prior_question_elapsed_time':'int32',#\n    'user_correctness':'float16',#\n    'user_uncorrect_count':'int16',#\n    'user_correct_count':'int16',#\n    #'content_correctness':'float16',\n    'content_correctness_std':'float16',\n    'content_correct_count':'int32',\n    'content_uncorrect_count':'int32',#\n    'content_elapsed_time_mean':'float16',\n    'content_had_explanation_mean':'float16',\n    'content_explation_false_mean':'float16',\n    'content_explation_true_mean':'float16',\n    'task_container_correctness':'float16',\n    'task_container_std':'float16',\n    'task_container_cor_count':'int32',#\n    'task_container_uncor_count':'int32',#\n    'attempt_no':'int8',#\n    'part':'int8',\n    'part_correctness_mean':'float16',\n    'part_correctness_std':'float16',\n    'part_uncor_count':'int32',\n    'part_cor_count':'int32',\n    'tags0': 'int8',\n    'tags1': 'int8',\n    'tags2': 'int8',\n    'tags3': 'int8',\n    'tags4': 'int8',\n    'tags5': 'int8',\n   # 'tags6': 'int8',\n   # 'tags7': 'int8',\n#     'tags0_correctness_mean':'float16',\n#     'tags1_correctness_mean':'float16',\n#     'tags2_correctness_mean':'float16',\n#     'tags4_correctness_mean':'float16',\n#     'bundle_id':'int16',\n#     'bundle_correctness_mean':'float16',\n#     'bundle_uncor_count':'int32',\n#     'bundle_cor_count':'int32',\n    'part_bundle_id':'int32',\n    'content_sub_bundle':'int8',\n    'prior_question_had_explanation':'int8',\n    'explanation_mean':'float16', #\n    #'explanation_var',#\n    'explanation_false_count':'int16',#\n    'explanation_true_count':'int16',#\n   # 'community':'int8',\n#     'part_1',\n#     'part_2',\n#     'part_3',\n#     'part_4',\n#     'part_5',\n#     'part_6',\n#     'part_7',\n#     'type_of_concept',\n#     'type_of_intention',\n#     'type_of_solving_question',\n#     'type_of_starter'\n}\ncategorical_columns= [\n    #'user_id',\n    'content_id',\n    'task_container_id',\n    'part',\n   # 'community',\n    'tags0',\n    'tags1',\n    'tags2',\n    'tags3',\n    'tags4',\n    'tags5',\n    #'tags6',\n    #'tags7',\n    #'bundle_id',\n    'part_bundle_id',\n    'content_sub_bundle',\n    'prior_question_had_explanation', \n#     'part_1',\n#     'part_2',\n#     'part_3',\n#     'part_4',\n#     'part_5',\n#     'part_6',\n#     'part_7',\n#     'type_of_concept',\n#     'type_of_intention',\n#     'type_of_solving_question',\n#     'type_of_starter'\n]\n\nfeatures=list(features_dict.keys())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"flag_lgbm=True\nclfs = list()\nparams = {\n'num_leaves': 200,\n'max_bin':450,\n# 'min_child_weight': 0.03454472573214212,\n'feature_fraction': 0.52,\n'bagging_fraction': 0.52,\n#'min_data_in_leaf': 106,\n# 'max_depth': -1,\n'objective': 'binary',\n'learning_rate': 0.05,\n\"boosting_type\": \"gbdt\",\n\"metric\": 'auc',\n# \"bagging_seed\": 11,\n# \"verbosity\": -1,\n# 'reg_alpha': 0.3899927210061127,\n# 'reg_lambda': 0.6485237330340494,\n# 'random_state': 47\n}\ntrains=list()\nvalids=list()\nnum=1\nfor i in range(0,num):\n    \n    #train_df=train_df.reset_index(drop=True)\n    #train_df_clf=train_df.sample(n=1200*10000)\n    \n#     train_df_clf=train_df[1200*10000:2*1200*10000]\n    train_df_clf=train_df\n    \n    print('sample end')\n    #train_df.drop(train_df_clf.index, inplace=True)\n    #print('train_df drop end')\n    \n   \n    #del train_df\n    \n    \n    \n    users=train_df_clf['user_id'].drop_duplicates()#\n    #\n    users=users.sample(frac=0.08)\n    users_df=pd.DataFrame()\n    users_df['user_id']=users.values\n   \n   \n    valid_df_newuser = pd.merge(train_df_clf, users_df, on=['user_id'], how='inner',right_index=True)\n    del users_df\n    del users\n    gc.collect()\n    #\n    train_df_clf.drop(valid_df_newuser.index, inplace=True)\n    print('pd.merge(train_df_clf, questions_df)')\n    #-----------\n    #train_df_clf=train_df_clf.sample(frac=0.2)\n    #train_df_clf.drop(valid_df_newuser.index, inplace=True)\n    train_df_clf = pd.merge(train_df_clf, questions_df, on='content_id', how='left',right_index=True)#\n    valid_df_newuser = pd.merge(valid_df_newuser, questions_df, on='content_id', how='left',right_index=True)#\n    \n#     train_df_clf = pd.merge(train_df_clf, user_lecture_stats_part, on='user_id', how=\"left\",right_index=True)\n#     valid_df_newuser = pd.merge(valid_df_newuser, user_lecture_stats_part, on='user_id', how=\"left\",right_index=True)\n    print('valid_df')\n    valid_df=train_df_clf.sample(frac=0.1)\n    train_df_clf.drop(valid_df.index, inplace=True)\n    \n#     test_df=train_df_clf.sample(n=100*10000)\n#     train_df_clf.drop(test_df.index, inplace=True)\n   \n    valid_df = valid_df.append(valid_df_newuser)\n    del valid_df_newuser\n    gc.collect()\n    #\n\n    trains.append(train_df_clf)\n    valids.append(valid_df)\n    print('train_df_clf length：',len(train_df_clf))\n    print('valid_df length：',len(valid_df))\n    #train_df=train_df.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#del train_df\ndel train_df_clf\ndel valid_df\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(0,num):\n      \n#     tr_data = lgb.Dataset(trains[i][features], label=trains[i][target])\n#     va_data = lgb.Dataset(valids[i][features], label=valids[i][target])\n\n    #Don't use DF to create lightgbm dataset, rather use np array:\n    X_train_np = trains[i][features].values.astype(np.float32)\n    X_valid_np = valids[i][features].values.astype(np.float32)\n    #features = train.columns\n    tr_data = lgb.Dataset(X_train_np, label=trains[i][target], feature_name=list(features))\n    va_data = lgb.Dataset(X_valid_np, label=valids[i][target], feature_name=list(features))\n    \n\n#     del train_df_clf\n#     del valid_df\n#     gc.collect()\n    del trains\n    del valids\n    del X_train_np\n    del X_valid_np\n    gc.collect()\n\n    model = lgb.train(\n        params, \n        tr_data,\n#         train_df[features],\n#         train_df[target],\n        num_boost_round=5000,\n        #valid_sets=[(train_df[features],train_df[target]), (valid_df[features],valid_df[target])], \n        valid_sets=[tr_data, va_data],\n        early_stopping_rounds=50,\n        feature_name=features,\n        categorical_feature=categorical_columns,\n        verbose_eval=50\n    )\n    clfs.append(model)\n    #print('test-auc:', roc_auc_score(test_df[target], model.predict(test_df[features])))\n    #model.save_model(f'model.txt')\n\n\n    fig,ax = plt.subplots(figsize=(15,15))\n    lgb.plot_importance(model, ax=ax,importance_type='gain',max_num_features=50)\n    plt.show()\n\n    del tr_data\n    del va_data\n    gc.collect()\n#    \n# del trains\n# del valids\n# gc.collect()\n# del test_df\n# gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}