{"cells":[{"metadata":{},"cell_type":"markdown","source":"# update\n\nThe comments section has a description of the features associated with tags and some thoughts and solutions to the overfitting problem.  \nPlease translate it by yourself for the time being. I will summarize those features soon.I'm so sorry. I'm so busy that I have to use Chinese for the time being.\n\n评论区有关于tags相关特征的说明，并且有一些关于过拟合问题的思考与解决办法。  \n麻烦各位朋友先自行翻译一下，我不久后会整理那些特征。太抱歉了，太忙了就只能暂时先用中文啦。  "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# introduction\n\nval:0.788 lb:No time to submit\n\nThanks to ragnar for this great script https://www.kaggle.com/ragnar123/riiid-model-lgbm  \nBased on the notebook mentioned above, I did the following:\n- Fixed some features that could cause overfitting, such as the \"time difference between the current timestamp and the last time you made a mistake\" feature.The original authors used answered_correctly under the same bundle for the frequency features, but in the test data, the answered_correctly under the current bundle is not known.\n- Fixed some statistical features made using \"prior_question_elapsed_time\" and \"prior_question_had_explanation\".These two features mean \"when was the last bundle\" and \"Whether was the last bundle solved\", but the original authors did not align them to the previous bundle.\n- The user attributes and title attributes are described from more angles, such as \"user-que\", \"user-part\", \"user-tags\", \"part\", \"tags\" (e.g., the highest correct rate of the tag under a question)...\n- The Gensim library is used to TSVD the Tags matrix(Direct use of SKLearn'S TSVD can cause memory leaks, while the GenSIM library can stream the TSVD decomposition model training in batches), indexing the highest numeric column in each row as the topic of tags.The number of topics tried 20, 50, 75.There was no significant improvement in the number of topics from 75 to 50.The number of topics 50 is 0.0005 higher than the number of topics 20, but in order to prevent overfitting, I finally chose 20 topics.  It is worth mentioning that after describing the tags and user-tags attributes, the tags can be improved to a certain extent (0.001). On this basis, TSVD can improve the tags to a certain extent (0.0015).\n\nThe following directions can be improved:\n- Adding a description of a \"user-lecture\" can be considered in the following directions, such as how many lectures the user has seen; when the user was working on a particular lecture, the current time difference between the last time a lecture with the same tag was viewed; and how often the lecture was viewed.I think these characteristics will lead to a very big improvement\n\nUnfortunately, due to the problem of laboratory work, I only participated in the competition for less than a week, so that I didn't even have time to move to the online.This plan after my correction, I think the probability has been high will not overfit, I hope to give you some inspiration!Once again, I wish you all good results.  \n***If this notebook is helpful to you, I hope you can do drop an upvote,thank you!***"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"介绍：\n很感谢这个作者的开源贡献：https://www.kaggle.com/ragnar123/riiid-model-lgbm  \n我在这个作者的工作上做了一些改进，具体如下：  \n- 修正了一些可能会造成过拟合的特征，例如“当前时间戳和上一次做错题目的时间差”特征。原作者在做频率特征时，使用到了同一个bundle下的answered_correctly，但测试数据中，当前bundle下的answered_correctly是不知道的。\n- 修正了使用“prior_question_elapsed_time”和“prior_question_had_explanation”所做的一些统计特征。这两个特征的意义是“上一个bundle所用时间”和“上一个bundle是否被解答”，但是原作者并没有将他们对齐到上一个bundle。\n- 从更多的角度来对用户属性和标题属性进行描述，如“user-que”、“user-part”、“user-tags”、“part”、“tags”(例如在一个问题下正确率最高的标签的正确率)……\n- 使用gensim库来对tags矩阵进行tsvd分解(直接使用sklearn的tsvd会造成内存泄漏，而gensim库可以流式的分批次进行tsvd分解模型的训练)，取每一行数值最高的那一列的索引来作为tags的主题。主题数尝试过20、50、75。主题数75较主题数50相比没有显著提升。主题数50较主题数20有0.0005的提升，但是为了防止过拟合，我最终选择了20个主题数。\n- 值得一提的是，对tags以及user-tags的属性进行描绘后能得到一定提升（0.001），在此基础上，对tags进行tsvd又能够获得一定的提升（0.0015）\n\n未来的一些可以改进的方向：\n- 加入对“user-lecture”的描述，可以从以下方向进行考虑，例如用户看过多少个lecture、用户在做某个题时、上一次观看具有相同tag的讲座的距离当前的时间差、观看讲座的频率。我认为这些特征会带来非常大的提升\n\n> ### 一些吐槽\n很遗憾的是因为实验室工作的问题（大家真的不要像我一样把比赛的事告诉导师o(╥﹏╥)o，估计也没人像我这么傻），riiid我只玩了不到一星期，一直是在线下做（因为线上太慢了）。鸽了好久，回来把我线下分数对照线上排行榜一看已经惨不忍睹，昨晚想把这个迁到线上去来着，结果又被叫去搬砖，心态炸了，索性不搞了。   \n这套方案经过我的修正后，我认为已经大概率不会过拟合了（也就是说估计如果能复现完，lb应该会比788要高），希望能给大家带来一些灵感！再次祝各位都能取得好成绩。（注：萌新第一次耍kaggle做开源，希望老哥们不要下次一定啦，给个赞吧(*^▽^*)，拜托了。有需要我实验具体数据的可以在评论区管我要，还有已经做好的tsvd分解表（可以直接merge到数据集上），还有已经做好的特征（存成了hdf5了）。小弟知无不言~）"},{"metadata":{},"cell_type":"markdown","source":"# ***Note that this code cannot be run directly and only provides the feature generation method and a list of model parameters***"},{"metadata":{},"cell_type":"markdown","source":"# Feature list"},{"metadata":{"trusted":true},"cell_type":"code","source":"['answered_correctly_u_avg',\n 'answered_correctly_u_count',\n 'answered_correctly_uq_count',\n 'elapsed_time_u_avg_xiuzheng',\n 'explanation_u_avg_xiuzheng',\n 'part_correctly_q_mean',###线下\n 'part_elapsed_time_mean',###线下\n 'part_had_explanation_mean',###线下\n 'part_user_count',\n 'part_user_mean',\n 'prior_question_elapsed_time',###原始\n 'prior_question_had_explanation',###原始\n 'question_correct_rate_last_20_mean',\n 'question_correctly_q_count',###线下\n 'question_correctly_q_mean',###线下\n 'question_elapsed_time_mean',###线下\n 'question_had_explanation_mean',###线下\n 'tag_acc_count',###线下\n 'tag_acc_max',###线下\n 'tag_acc_min',###线下\n 'tags_lsi',###线下\n 'task_container_id',###原始\n 'timestamp',###原始\n 'timestamp_u_correct_recency_1',\n 'timestamp_u_diff_1_2',\n 'timestamp_u_diff_2_3',\n 'timestamp_u_diff_3_end',\n 'timestamp_u_incorrect_recency_1',\n 'user_tag_acc_count',\n 'user_tag_acc_max',\n 'user_tag_acc_min']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Features you can do offline (upload to notebook, then merge)"},{"metadata":{"trusted":true},"cell_type":"code","source":"### question_elapsed_time_mean and question_had_explanation_mean\ntemp=all_data.groupby(\"question_id\")[[\"current_question_elapsed_time\",'current_question_had_explanation']].mean()\ntemp.columns=['question_elapsed_time_mean','question_had_explanation_mean']\nque_fea_group_1=temp\ndel temp\ngc.collect()\n\n###  question_correctly_q_count and question_correctly_q_mean\ntemp=all_data.groupby(\"question_id\")['answered_correctly'].agg([(\"question_correctly_q_count\",\"count\"),(\"question_correctly_q_mean\",\"mean\")])\nque_fea_group_2=temp\ndel temp\ngc.collect()\n\n### part_elapsed_time_mean、part_had_explanation_mean and part_correctly_q_mean\ntemp=all_data.groupby(\"part\")[[\"current_question_elapsed_time\",'current_question_had_explanation','answered_correctly']].mean()\ntemp.columns=['part_elapsed_time_mean','part_had_explanation_mean','part_correctly_q_mean']\npart_fea_group=temp\ndel temp\ngc.collect()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"### TSVD decomposition\nfrom time import time \nimport pandas as pd\nimport warnings\nfrom time import time \nimport warnings\nfrom gensim import corpora,similarities,models\nimport pandas as pd\n\nwarnings.filterwarnings(action='ignore',category=UserWarning,module='gensim')\ntime1=time()\nfrom gensim import corpora,similarities,models\n# test_data_all=pd.read_pickle(\"query_all.pickle\")\n\nall_data['tags'].fillna(\"-1\",inplace=True)\n\nclass mycorpus(object):\n    def __iter__(self):\n        for index,doc in enumerate(all_data['tags']):\n            if index%5000000==0:\n                print(index)\n            yield doc.split(' ')\ncorp = mycorpus()\n# dictionary.add_documents(corp)\ndictionary=corpora.Dictionary(corp)\ntime2=time()\nprint(time2-time1)\n\n\n\n# test_data_all=pd.read_pickle(\"/home/kesci/work/query_all.pickle\")\n\nprint(111523155656)\nclass MyCorpus(object):\n    def __init__(self):\n        print('MyCorpus')\n    def __iter__(self):\n        for index,doc in enumerate(all_data['tags']):\n            if index%5000000==0:\n                print(index)\n            yield dictionary.doc2bow(doc.split(' '))\ncorpus = MyCorpus()\n# corpus = [dictionary.doc2bow(text) for text in self.text]\ntfidf_model = models.TfidfModel(corpus, id2word=dictionary)\n# corpus_tfidf = tfidf_model[corpus]\n\ntfidf_model.save(\"model/tfidf.model\")\ndictionary.save(\"model/dictionary.model\")\n\n\nfrom time import time \nimport warnings\ntime1=time()\nwarnings.filterwarnings(action='ignore',category=UserWarning,module='gensim')\nfrom gensim import corpora,similarities,models\nimport pandas as pd\n# test_data_all=pd.read_pickle(\"/home/kesci/work/query_all.pickle\")\ncorpus_tfidf = tfidf_model[corpus]\nprint(111523155656)\nclass MyCorpus(object):\n    def __init__(self):\n        print('MyCorpus')\n    def __iter__(self):\n        for index,doc in enumerate(corpus_tfidf):\n            if index%1000000==0:\n                print(index)\n            yield doc\ncorpus3 = MyCorpus()\n# corpus = [dictionary.doc2bow(text) for text in self.text]\nlsi_model = models.LsiModel(corpus3, id2word=dictionary,chunksize=2500000,num_topics=50)\nlsi_model.save(\"lsi_model/lsi_all_query.lsi\")\ntime2=time()\nprint((time2-time1)/60)\n# corpus_tfidf = tfidf_model[corpus]\n\ndef get_arg_max(single_list):\n    max_index=0\n    max_num=single_list[0][1]\n    for index in range(len(single_list)-1):\n        if max_num<single_list[index+1][1]:\n            max_num=single_list[index+1][1]\n            max_index=index+1\n    return max_index\nlsi_model=models.LsiModel.load(\"model/lsi_model/lsi_all_query_20.lsi\")\ndictionary=corpora.Dictionary.load(\"model/dictionary.model\")\ntfidf_model=corpora.Dictionary.load(\"model/tfidf.model\")\nall_data_lsi=[]\nfor text in tqdm(question['tags']):\n    single_row_text=dictionary.doc2bow(text.split(' '))\n    single_row_lsi_list=list(lsi_model[tfidf_model[single_row_text]])\n    if len(single_row_lsi_list)==0:\n        all_data_lsi.append(0)\n        continue\n    single_row_lsi=get_arg_max(single_row_lsi_list)\n    all_data_lsi.append(single_row_lsi)\nquestion['tags_lsi']=all_data_lsi\nquestion.to_csv(\"temp/question_lsi20.csv\",index=None)\nquestion.head(50)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Features that need to be constantly updated"},{"metadata":{"trusted":true},"cell_type":"code","source":"#  'answered_correctly_u_avg',\n#  'answered_correctly_u_count',\n#  'answered_correctly_uq_count',\n\n#  'elapsed_time_u_avg_xiuzheng',\n#  'explanation_u_avg_xiuzheng',\n\n#  'part_user_count',\n#  'part_user_mean',\n\n#  'question_correct_rate_last_20_mean',\n\n#  'timestamp_u_correct_recency_1',\n#  'timestamp_u_incorrect_recency_1',\n\n#  'timestamp_u_diff_1_2',\n#  'timestamp_u_diff_2_3',\n#  'timestamp_u_diff_3_end',\n\n#  'user_tag_acc_count',\n#  'user_tag_acc_max',\n#  'user_tag_acc_min'\nanswered_correctly_u_count_dict = defaultdict(int)\nanswered_correctly_u_sum_dict = defaultdict(int)\nanswered_correctly_uq_dict = defaultdict(lambda: defaultdict(int))\n\nelapsed_time_u_sum_dict = defaultdict(int)\nexplanation_u_sum_dict = defaultdict(int)\nquestion_u_count_dict = defaultdict(int)\nquestion_u_last_bundle_count_dict = defaultdict(int)\n\npart_user_count_dict = defaultdict(lambda: defaultdict(int))\npart_user_sum_dict = defaultdict(lambda: defaultdict(int))\n\nquestion_correct_last_20_count_dict = defaultdict(int)\nquestion_correct_last_20_sum_dict = defaultdict(int)\nquestion_correct_last_20_all_dict = defaultdict(list)\n\ntimestamp_u_correct_dict = defaultdict(list)\ntimestamp_u_incorrect_dict = defaultdict(list)\n\ntimestamp_u_dict = defaultdict(list)\n\nuser_tag_acc_count_dict = defaultdict(lambda: defaultdict(int))\nuser_tag_acc_sum_dict = defaultdict(lambda: defaultdict(int))\n# -----------------------------------------------------------------------\ndef add_features(update=True):\n    # Client features\n    answered_correctly_u_avg = np.zeros(len(all_data), dtype = np.float32)\n    answered_correctly_u_count = np.zeros(len(all_data), dtype = np.float32)\n    answered_correctly_uq_count = np.zeros(len(all_data), dtype = np.int32)\n\n    elapsed_time_u_avg = np.zeros(len(all_data), dtype = np.float32)\n    explanation_u_avg = np.zeros(len(all_data), dtype = np.float32)\n    \n    part_user_count = np.zeros(len(all_data), dtype = np.float32)\n    part_user_mean = np.zeros(len(all_data), dtype = np.float32)\n    \n    question_correct_rate_last_20_sum = np.zeros(len(all_data), dtype = np.float32)\n    \n    timestamp_u_correct_recency_1 = np.zeros(len(all_data), dtype = np.float32)\n    timestamp_u_incorrect_recency_1 = np.zeros(len(all_data), dtype = np.float32)\n    \n    timestamp_u_diff_1 = np.zeros(len(all_data), dtype = np.float32)\n    timestamp_u_diff_2 = np.zeros(len(all_data), dtype = np.float32)\n    timestamp_u_diff_3 = np.zeros(len(all_data), dtype = np.float32)\n    \n    user_tag_acc_count = np.zeros(len(all_data), dtype = np.float32)\n    user_tag_acc_max = np.zeros(len(all_data), dtype = np.float32)\n    user_tag_acc_min = np.zeros(len(all_data), dtype = np.float32)\n    \n    list_last_user_task_table=[]####定义数组 用来保存旧组的信息\n    list_last_user_task_table_un_back=[]####定义数组 用来保存旧组的信息\n#     for num, row in enumerate(tqdm(all_data[['user_id', 'answered_correctly', 'content_id', 'prior_question_elapsed_time', 'prior_question_had_explanation', 'timestamp','task_container_id']].values)):\n    flag_current_task=0\n    all_data_temp=all_data[['user_id',\"task_container_id\", 'content_id', 'answered_correctly', 'prior_question_elapsed_time', 'prior_question_had_explanation', 'timestamp','part',\"tags\"]].values\n    for num in tqdm(range(len(all_data))):\n        row=all_data_temp[num]\n        if num+1!=len(all_data):\n            row2=all_data_temp[num+1]\n        else:\n            row2=[-100 for i in range(len(row))]\n        \n        \n        ####*********  elapsed_time_u_avg_xiuzheng和explanation_u_avg_xiuzheng\n        if row[6]!=0:##如果时间戳不是0的时候\n            if flag_current_task==0:\n                question_u_count_dict[row[0]]+=question_u_last_bundle_count_dict[row[0]]\n                elapsed_time_u_sum_dict[row[0]]+=row[4]*question_u_last_bundle_count_dict[row[0]]\n                explanation_u_sum_dict[row[0]]+=row[5]*question_u_last_bundle_count_dict[row[0]]\n            elapsed_time_u_avg[num]= elapsed_time_u_sum_dict[row[0]]/question_u_count_dict[row[0]]\n            explanation_u_avg[num] = explanation_u_sum_dict[row[0]]/question_u_count_dict[row[0]]\n            ###⑥只需要当前组的prior（也就是上一组的平均时间或者是否解答），就可以计算了\n        else:##时间戳为0的时候，肯定是不知道当前组的用时和解答情况的\n            elapsed_time_u_avg[num]=np.nan\n            explanation_u_avg[num] = np.nan\n        flag_current_task=1\n        \n        ###①求这个特征，需要不断的记录上一组一共有多少道题，到最后用    （不断累加（每组多少道题*每道题平均时间））/总做题次数\n        ###②需要把记录这组有多少道题放在后面计算，在前面计算平均时间并且填充到特征数组里\n        list_last_user_task_table_un_back.append([row[0]])###没换人换组的时候，先不断保存旧组的信息,并且在换人换组的时候也要保存，以防那次信息没被用到\n        if row[0]!=row2[0] or row[1]!=row2[1]:###换了一个task\n            flag_current_task=0\n            question_u_last_bundle_count_dict[row[0]]=len(list_last_user_task_table_un_back)\n            list_last_user_task_table_un_back=[]###在即将换task的时候，把旧组需要换成新组（更换成新组之前，需要先把旧组的信息在上面用完）\n            \n        ####*********\n        \n        ####*********   answered_correctly_u_avg、answered_correctly_u_count和answered_correctly_uq_count\n        if answered_correctly_u_count_dict[row[0]] != 0:\n            answered_correctly_u_avg[num] = answered_correctly_u_sum_dict[row[0]] / answered_correctly_u_count_dict[row[0]]\n            answered_correctly_u_count[num] = answered_correctly_u_count_dict[row[0]]\n        else:\n            answered_correctly_u_avg[num] = 0.67\n            answered_correctly_u_count[num] = 0\n\n        answered_correctly_uq_count[num] = answered_correctly_uq_dict[row[0]][row[2]]\n        ####*********\n        \n        ####*********   part_user_count和part_user_mean\n        if part_user_count_dict[row[0]][row[7]]==0:\n            part_user_count[num] = 0\n            part_user_mean[num] = 0.67\n        else:\n            part_user_count[num] = part_user_count_dict[row[0]][row[7]]\n            part_user_mean[num] = part_user_sum_dict[row[0]][row[7]]/part_user_count_dict[row[0]][row[7]]\n        ####*********\n        \n        ####*********   question_correct_rate_last_20_mean\n#         question_correct_rate_last_20_sum[num]=question_correct_last_20_sum_dict[row[0]]\n        ####*********\n        \n        \n        ####*********   timestamp_u_correct_recency_1，timestamp_u_incorrect_recency_1\n        if len(timestamp_u_correct_dict[row[0]]) == 0:\n            timestamp_u_correct_recency_1[num] = np.nan\n        elif len(timestamp_u_correct_dict[row[0]]) == 1:\n            timestamp_u_correct_recency_1[num] = row[6] - timestamp_u_correct_dict[row[0]][0]\n            \n        if len(timestamp_u_incorrect_dict[row[0]]) == 0:\n            timestamp_u_incorrect_recency_1[num] = np.nan\n        elif len(timestamp_u_incorrect_dict[row[0]]) == 1:\n            timestamp_u_incorrect_recency_1[num] = row[6] - timestamp_u_incorrect_dict[row[0]][0]\n        ####*********\n\n        ####*********   timestamp_u_diff_1_2，timestamp_u_diff_2_3，timestamp_u_diff_3_end\n        if len(timestamp_u_dict[row[0]]) == 0:\n            timestamp_u_diff_1[num] = np.nan\n            timestamp_u_diff_2[num] = np.nan\n            timestamp_u_diff_3[num] = np.nan\n        elif len(timestamp_u_dict[row[0]]) == 1:\n            timestamp_u_diff_1[num] = row[6] - timestamp_u_dict[row[0]][0]\n            timestamp_u_diff_2[num] = np.nan\n            timestamp_u_diff_3[num] = np.nan\n        elif len(timestamp_u_dict[row[0]]) == 2:\n            timestamp_u_diff_1[num] = row[6] - timestamp_u_dict[row[0]][1]\n            timestamp_u_diff_2[num] = timestamp_u_dict[row[0]][1] - timestamp_u_dict[row[0]][0]\n            timestamp_u_diff_3[num] = np.nan\n        elif len(timestamp_u_dict[row[0]]) == 3:\n            timestamp_u_diff_1[num] = row[6] - timestamp_u_dict[row[0]][2]\n            timestamp_u_diff_2[num] = timestamp_u_dict[row[0]][2] - timestamp_u_dict[row[0]][1]\n            timestamp_u_diff_3[num] = timestamp_u_dict[row[0]][1] - timestamp_u_dict[row[0]][0]\n\n        ####*********\n        \n        ####*********   user_tag_acc_count，user_tag_acc_max，user_tag_acc_min\n        if pd.isnull(row[8]):\n            user_tag_acc_count[num]=np.nan\n            user_tag_acc_max[num] = np.nan\n            user_tag_acc_min[num] = np.nan\n            continue\n        else:\n            tag_list_un_back=row[8].split()\n            row_all_tag_sum=0\n            row_all_tag_count=0\n            row_max_tag_mean=-1###尽量搞小\n            row_min_tag_mean=1000###尽量搞大\n\n            for single_tag in tag_list_un_back:\n                ###先做需要更新的###\n                single_tag_sum=user_tag_acc_sum_dict[row[0]][single_tag]\n                single_tag_count=user_tag_acc_count_dict[row[0]][single_tag]\n                row_all_tag_sum+=single_tag_sum\n                row_all_tag_count+=single_tag_count\n                if single_tag_count==0:\n                    single_tag_mean=0.67\n                else:\n                    single_tag_mean=single_tag_sum/single_tag_count\n                row_max_tag_mean=max(single_tag_mean,row_max_tag_mean)\n                row_min_tag_mean=min(single_tag_mean,row_min_tag_mean)\n            if row_all_tag_count==0:\n                user_tag_acc_count[num]=0\n                user_tag_acc_max[num] = 0.67\n                user_tag_acc_min[num] = 0.67\n            else:\n                user_tag_acc_count[num]=row_all_tag_count\n                user_tag_acc_max[num] = row_max_tag_mean\n                user_tag_acc_min[num] = row_min_tag_mean\n        ####*********\n        \n\n        \n\n\n        if update:\n            answered_correctly_u_count_dict[row[0]] += 1\n            answered_correctly_u_sum_dict[row[0]] += row[3]\n            answered_correctly_uq_dict[row[0]][row[2]] += 1\n            part_user_count_dict[row[0]][row[7]] += 1\n            part_user_sum_dict[row[0]][row[7]] += row[3]\n#             if question_correct_last_20_count_dict[row[0]]+1<=20:\n#                 question_correct_last_20_count_dict[row[0]]+=1\n#                 question_correct_last_20_sum_dict[row[0]]+=row[3]\n#                 question_correct_last_20_all_dict[row[0]].append(row[3])\n#             else:\n#                 question_correct_last_20_sum_dict[row[0]]+=row[3]\n#                 question_correct_last_20_sum_dict[row[0]]-=question_correct_last_20_all_dict[row[0]][-1]\n#                 question_correct_last_20_all_dict[row[0]].pop(0)\n#                 question_correct_last_20_all_dict[row[0]].append(row[3])\n            \n            tag_list=row[8].split()\n            for single_tag in tag_list:\n                ######更新一下 user-tag\n                user_tag_acc_count_dict[row[0]][single_tag] += 1\n                user_tag_acc_sum_dict[row[0]][single_tag] += row[3]\n            \n            #'user_id',\"task_container_id\", 'content_id', 'answered_correctly', 'prior_question_elapsed_time', 'prior_question_had_explanation', 'timestamp','part'\n            list_last_user_task_table.append([row[0],row[1],row[2],row[3],row[4],row[5],row[6],row[7]])###没换人换组的时候，先不断保存旧组的信息,并且在换人换组的时候也要保存，以防那次信息没被用到\n            if row[0]!=row2[0] or row[1]!=row2[1]:###换了一个task\n                \n                if len(timestamp_u_dict[row[0]]) == 3:\n                    timestamp_u_dict[row[0]].pop(0)\n                    timestamp_u_dict[row[0]].append(row[6])\n                else:\n                    timestamp_u_dict[row[0]].append(row[6])\n                \n                ####由于bundle下面包含很多question，每个question都有一个correct，所以需要用列表存储“旧的一整个组”的correct \n                for single_row_last_user_task_table in list_last_user_task_table:\n                    if single_row_last_user_task_table[3]==1:\n                        if len(timestamp_u_correct_dict[row[0]]) == 1:###这里，就使用row[0]就行，因为list_last_user_task_que_timestamp里全都是当前user-task的信息，而非下一个user-task的信息\n                            timestamp_u_correct_dict[row[0]].pop(0)\n                            timestamp_u_correct_dict[row[0]].append(single_row_last_user_task_table[6])\n                        else:\n                            timestamp_u_correct_dict[row[0]].append(single_row_last_user_task_table[6])\n                    else:\n                        if len(timestamp_u_incorrect_dict[row[0]]) == 1:###这里，就使用row[0]就行，因为list_last_user_task_que_timestamp里全都是当前user-task的信息，而非下一个user-task的信息\n                            timestamp_u_incorrect_dict[row[0]].pop(0)\n                            timestamp_u_incorrect_dict[row[0]].append(single_row_last_user_task_table[6])\n                        else:\n                            timestamp_u_incorrect_dict[row[0]].append(single_row_last_user_task_table[6])\n                list_last_user_task_table=[]###在即将换task的时候，把旧组需要换成新组（更换成新组之前，需要先把旧组的信息在上面用完）\n\n#     all_data['answered_correctly_u_avg']=answered_correctly_u_avg\n#     all_data['answered_correctly_u_count']=answered_correctly_u_count\n#     all_data['answered_correctly_uq_count']=answered_correctly_uq_count\n#     all_data['elapsed_time_u_avg_xiuzheng']=elapsed_time_u_avg\n#     all_data['explanation_u_avg_xiuzheng']=explanation_u_avg\n#     all_data['part_user_count']=part_user_count\n#     all_data['part_user_mean']=part_user_mean\n#     all_data['timestamp_u_correct_recency_1']=timestamp_u_correct_recency_1\n#     all_data['timestamp_u_incorrect_recency_1']=timestamp_u_incorrect_recency_1\n#     all_data['timestamp_u_diff_1_2']=timestamp_u_diff_1\n#     all_data['timestamp_u_diff_2_3']=timestamp_u_diff_2\n#     all_data['timestamp_u_diff_3_end']=timestamp_u_diff_3\n#     all_data['part_user_count']=part_user_count\n#     all_data['part_user_mean']=part_user_mean\n#     all_data['user_tag_acc_count']=user_tag_acc_count\n#     all_data['user_tag_acc_max']=user_tag_acc_max\n#     all_data['user_tag_acc_min']=user_tag_acc_min\n    \n       \nadd_features()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# model\nWhen num_leaves are increased and max_depth is controlled within a certain range, the training speed and the result accuracy can be improved under large data volume (it has been verified on this data set, the effect is improved by about 0.001, and the training time is shortened by 1/3)."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = lgb.LGBMClassifier(num_leaves=300,\n                        max_depth=15,\n                        learning_rate=0.1,\n                        subsample=0.8,\n                        feature_fraction=0.8,\n                        random_state=2020,\n                        n_estimators=200\n                        )\nlgb_model = model.fit(train_X[fea_list], \n                    train_Y,\n                    eval_names=['train', 'valid'],\n                    eval_set=[(train_X[fea_list], train_Y), (valid_X[fea_list], valid_Y)],\n                    verbose=10,\n                    eval_metric='auc',\n                    early_stopping_rounds=10,\n                     categorical_feature=['tags_lsi'])###                    categorical_feature=cate_feat\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_importance = lgb_model.feature_importances_\nfeature_importance = pd.DataFrame({'Features': fea_list, 'Importance': feature_importance}).sort_values('Importance', ascending = False)\n\nfig = plt.figure(figsize = (10, 10))\nfig.suptitle('Feature Importance', fontsize = 20)\nplt.tick_params(axis = 'x', labelsize = 12)\nplt.tick_params(axis = 'y', labelsize = 12)\nplt.xlabel('Importance', fontsize = 15)\nplt.ylabel('Features', fontsize = 15)\nsns.barplot(x = feature_importance['Importance'], y = feature_importance['Features'], orient = 'h')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![fea_importance](http://pic.downk.cc/item/5feae6fc3ffa7d37b359cf3b.jpg)"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}