{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **本Notebook仅用于提交LB，riiid_dataset 包含了已经训练好的模型和状态，本Notebook不包含训练过程！**\n\n\n* **这是我第一次参加Kaggle的竞赛，目前我在某非985院校大一（刚入校3个月）读非计算机专业，之前在高中也未参加过任何竞赛。大部分Python 和 是抽空在网上学的。故代码风格/模型描述十分混乱，还请多多多包涵。若您有任何建议或疑问，欢迎您在下方评论区指出。**\n\n* 本Notebook仅用于提交，训练Notebook代码由于太乱，需要整理，若近期有空，我可能会上传。\n\n\n* **关于riiid-dataset 数据集的说明**\n\n\n1. 模型（classifier.model）\n\nclassifier.model 为LightGBM 未经参数优化所训练得的模型，所有参数均为LightGBM默认(随机种子为2020)，且仅在前20M行数据上训练。\n若经过参数优化/在整个数据集训练，本模型可能尚有提高空间。\n\n2. Stage.Pickle\n\n鉴于特征工程是遍历DataFrame得到的，而这一过程耗时较长，为加快Notebook运行速度，我们将遍历完成后的最终状态保存在Pickle文件中。\n\n3. Lesson_Metadata 与 Question_Metadata\n\n在Lesson_Metadata 与 Question_Metadata 文件中，我们保存了预先计算的和试题有关的特征（Tag，平均正确率等）及和课程有关的特征（Tag）。\n\n\n* **特征说明**\n\nquestion_answered_num 用户回答问题的总数量\n\nquestion_answered_num_agg_field 用户总回答的问题(按TOEIC学科领域统计)\n\nquestion_answered_mean_accuracy 用户回答的问题的平均正确率\n\nquestion_answered_mean_accuracy_agg_field 用户总回答的问题的平均正确率 (按TOEIC学科领域统计)\n\nquestion_answered_mean_difficulty_weighted_accuracy = 0  用户总回答的问题的平均难度加权正确率\n\nquestion_answered_mean_difficulty_weighted_accuracy_agg_field 用户总回答的问题的平均难度加权正确率(按TOEIC学科领域统计)\n\nmax_solved_difficulty 用户解答的最难问题\n\nmax_solved_difficulty_agg_field  用户解答的最难问题(按TOEIC学科领域统计)\n\nmin_wrong_difficulty = 0   用户做错的最简单问题\n\nmin_wrong_difficulty_agg_field 用户做错的最简单问题(按TOEIC学科领域统计)\n\nlessons_overall 用户总共学了多少课\n\nlessons_overall_agg_field 用户总共学了多少课（按TOEIC学科领域统计）\n\nsession_time 用户本Session的分钟数 \n\nsince_last_session_time  距离上次Session的小时数\n\nuser_id 用户ID \n\ntag_1 问题的Tag1标签\n\ntag_2 问题的Tag1标签\n\ntag_encoded 基于Tag1，2，3创建的编码\n\nquestion_id 问题的content_id\n\nprevious_explained 之前问题是否得到解答\n\nquestion_seen 用户是否曾经见到过这一问题\n\nmean_question_accuracy 该题的中位正确率\n\nstd_question_accuracy 该题正确率的标准差\n\nmost_liked_guess_correct 假设你在考试中某道选择题不会做，那么你大概率会猜个C（我习惯猜选项较长的那一个）之类的。有鉴于此，我们在每一次用户做错时记录其最喜欢选的选项。之后，若下面提到的Trueskill模型认为用户该题做对的概率较小，我们便将用户最喜欢的选项与该题选项做对比，看看改为用户猜的对不对。\n\n\n\n*其余特征与Trueskill评分系统有关*\n\nTrueskill 是由微软开发的一套评分系统（https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/）\n\n若你玩过彩六，你或许接触过这套系统，在这个游戏中，你的排位就是由Trueskill系统决定的。\n\n不妨将每一个Question作为一个已经确定能力的用户（老玩家），而User作为刚开始玩的新手，之后，就可用Trueskill系统来决定其排位及其获胜概率。\n\n*注 我们之所以能够套用Trueskill系统，是由于问题难度的分布近似于正态分布（以（1-用户整体的中位正确率）决定）。\n\n\nmmr_overall 用户的MMR分数\n\nmmr_overall_agg_field 用户的MMR分数(按TOEIC学科领域统计)\n\nmmr_confidence 用户的MMR置信度\n\nmmr_overall_agg_field 用户的MMR置信度 (按TOEIC学科领域统计)\n\nmmr_win_prob 用户'战胜’该题的概率\n\nmmr_win_prob_agg_field  用户'战胜’该题的概率 (按TOEIC学科领域统计)\n\n\n\n\n\n\n* **版权声明**\n\nTrueskill(TM) 是微软公司的注册商标。\n\ntruskill 库 是由Heungsub Lee创作的，基于BSD开源的项目，项目地址：https://trueskill.org/"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport psutil","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"'''\nRiiid Competition Submission Ver 1.0.1 Alpha\n(C) Copyright By Author 2020 - Now\nAll rights reserved\n'''\nimport sys\nsys.path.append('/kaggle/input/riiid-dataset/')\n#路径\nquestion_metadata_dir = r'/kaggle/input/riiid-dataset/question_metadata.csv'\nlesson_metadata_dir = r'/kaggle/input/riiid-dataset/lesson_metadata.csv'\npickle_dir= r'/kaggle/input/riiid-dataset/stage.pickle'\nmodel_dir = r'/kaggle/input/riiid-dataset/classifier.model'\n\nimport datetime\nprint(\"{} 提交启动\".format(str(datetime.datetime.now())))\n# 加载库\ntry:\n    import pandas as pd\n    import pickle\n    import trueskill\n    import math\n    import lightgbm as lgb\n    import riiideducation\n    import time\n    from sklearn.metrics import roc_auc_score\n\n\nexcept ImportError as e:\n    print(\"{} 导入错误，错误信息：{}\".format(str(datetime.datetime.now()), e))\n\nprint(\"{} 包导入完成\".format(str(datetime.datetime.now())))\nenv = trueskill.TrueSkill(mu=0.3, sigma=0.164486, beta=0.05, tau=0.00164, draw_probability=0)\nenv.make_as_global()\n\n\ndef win_probability(team1, team2):\n    '''\n    根据两个TrueSkill对象，计算获胜概率\n    :param team1:用户TrueSkill对象\n    :param team2:问题Trueskill对象\n    :return: 获胜概率\n    '''\n    delta_mu = team1.mu - team2.mu\n    sum_sigma = sum([team1.sigma ** 2, team2.sigma ** 2])\n    size = 2\n    denom = math.sqrt(size * (0.05 * 0.05) + sum_sigma)\n    ts = trueskill.global_env()\n    return ts.cdf(delta_mu / denom)\n\n\nclass user:\n    '''\n    用户 类\n    '''\n\n    def __init__(self):\n        '''\n        初始化user类\n        :param None\n        :return: None\n        '''\n        # 直接可输出的特征\n\n        # 数量类\n        self.question_answered_num = 0  # 用户回答问题的总数量\n        self.question_answered_num_agg_field = [0] * 7  # 用户总回答的问题(按TOEIC学科领域统计)\n\n        # 正确率类\n        self.question_answered_mean_accuracy = 0  # 用户回答的问题的平均正确率\n        self.question_answered_mean_accuracy_agg_field = [0] * 7  # 用户总回答的问题的平均正确率\n        self.question_answered_mean_difficulty_weighted_accuracy = 0  # 用户总回答的问题的平均难度加权正确率\n        self.question_answered_mean_difficulty_weighted_accuracy_agg_field = [0] * 7  # 用户总回答的问题的平均难度加权正确率(按TOEIC学科领域统计)\n\n        # 极值类\n        self.max_solved_difficulty = 1  # 用户解答的最难问题\n        self.max_solved_difficulty_agg_field = [1] * 7  # 用户解答的最难问题(按TOEIC学科领域统计)\n        self.min_wrong_difficulty = 0  # 用户做错的最简单问题\n        self.min_wrong_difficulty_agg_field = [0] * 7  # 用户做错的最简单问题\n\n        # 课程学习类\n        self.lessons_overall = 0  # 用户总共学了多少课\n        self.lessons_overall_agg_field = [0] * 7  # 用户总共学了多少课（按TOEIC学科领域统计）\n\n        # 交互时间信息类\n        self.session_time = 0  # 用户本Session的分钟数\n        self.since_last_session_time = 0  # 距离上次Session的小时数\n\n        # 需要进一步处理的特征\n        self._mmr_object = trueskill.setup(mu=0.3, sigma=0.164486, beta=0.05, tau=0.00164,\n                                           draw_probability=0).Rating()  # MMR模块\n        self._mmr_object_agg_field = [trueskill.setup(mu=0.3, sigma=0.164486, beta=0.05, tau=0.00164,\n                                                      draw_probability=0).Rating()] * 7  # MMR模块（按TOEIC学科领域统计）\n        self._most_liked_guess = [0] * 4  # 用户做错时最喜欢的选项\n        self._last_session_start_time = 0  # 本Session开始的时间\n        self._first_action_time = 0  # 首次交互的时间\n        self._question_num_dict = {}  # 用户回答问题的记录\n        self._first_processed_flag = False  # 是否处理的表示\n\n    def update_user(self, data: pd.DataFrame):\n        '''\n        处理一帧测试集\n        :param data: pandas DataFrame\n        :return: None\n        '''\n        _temp = None\n\n        # 判断用户是否正在观看课程\n        if data['content_type_id'] == 0:\n            # Content Type 为 0，即用户正在回答问题\n\n            # 处理回答计数部分\n            self.question_answered_num = self.question_answered_num + 1\n            question_field = int(data['content_field'])\n            self.question_answered_num_agg_field[question_field - 1] = int(self.question_answered_num_agg_field[\n                                                                               question_field - 1]) + 1\n\n            # 处理正确率部分\n            if data['answered_correctly'] == 1:\n                self.question_answered_mean_accuracy = \\\n                    (self.question_answered_mean_accuracy * (\n                            self.question_answered_num - 1) + 1) / self.question_answered_num\n\n                self.question_answered_mean_accuracy_agg_field[question_field - 1] = \\\n                    (self.question_answered_mean_accuracy_agg_field[question_field - 1] * (\n                            self.question_answered_num_agg_field[question_field - 1] - 1) + 1) \\\n                    / self.question_answered_num_agg_field[question_field - 1]\n\n                self.question_answered_mean_difficulty_weighted_accuracy = \\\n                    (self.question_answered_mean_difficulty_weighted_accuracy * (self.question_answered_num - 1) + (\n                            1 - data['mean_question_accuracy']) * 3) \\\n                    / self.question_answered_num\n\n                self.question_answered_mean_difficulty_weighted_accuracy_agg_field[question_field - 1] = \\\n                    (self.question_answered_mean_difficulty_weighted_accuracy_agg_field[question_field - 1] * (\n                            self.question_answered_num_agg_field[question_field - 1] - 1) + (\n                             1 - data['mean_question_accuracy']) * 3) \\\n                    / self.question_answered_num_agg_field[question_field - 1]\n\n\n            else:\n                self.question_answered_mean_accuracy = \\\n                    (self.question_answered_mean_accuracy * (\n                            self.question_answered_num - 1)) / self.question_answered_num\n\n                self.question_answered_mean_accuracy_agg_field[question_field - 1] = \\\n                    (self.question_answered_mean_accuracy_agg_field[question_field - 1] * (\n                            self.question_answered_num_agg_field[question_field - 1] - 1)) / \\\n                    self.question_answered_num_agg_field[question_field - 1]\n\n                self.question_answered_mean_difficulty_weighted_accuracy = \\\n                    (self.question_answered_mean_difficulty_weighted_accuracy * (self.question_answered_num - 1)) \\\n                    / self.question_answered_num\n\n                self.question_answered_mean_difficulty_weighted_accuracy_agg_field[question_field - 1] = \\\n                    (self.question_answered_mean_difficulty_weighted_accuracy_agg_field[question_field - 1] * (\n                            self.question_answered_num_agg_field[question_field - 1] - 1)) \\\n                    / self.question_answered_num_agg_field[question_field - 1]\n\n            # 处理最大/最小正确率部分\n\n            if data['answered_correctly'] == 1:\n                if data['mean_question_accuracy'] < self.max_solved_difficulty:\n                    self.max_solved_difficulty = data['mean_question_accuracy']\n                if data['mean_question_accuracy'] < self.max_solved_difficulty_agg_field[question_field - 1]:\n                    self.max_solved_difficulty_agg_field[question_field - 1] = data['mean_question_accuracy']\n            else:\n                if data['mean_question_accuracy'] > self.min_wrong_difficulty:\n                    self.min_wrong_difficulty = data['mean_question_accuracy']\n                if data['mean_question_accuracy'] > self.min_wrong_difficulty_agg_field[question_field - 1]:\n                    self.min_wrong_difficulty_agg_field[question_field - 1] = data['mean_question_accuracy']\n\n            # 处理猜测部分\n            if data['answered_correctly'] == 0:\n                self._most_liked_guess[int(data['user_answer'])] = self._most_liked_guess[\n                                                                       int(data['user_answer'])] + 1\n\n            # 处理时间部分\n            if self._first_action_time == 0:\n                self._first_action_time = data['timestamp']\n                self._last_session_start_time = data['timestamp']\n            else:\n                if data['timestamp'] - self._last_session_start_time >= 7200 * 1000:\n                    self.since_last_session_time = (data[\n                                                        'timestamp'] - self._last_session_start_time) / 1000 / 3600\n                    self._last_session_start_time = data['timestamp']\n                    self.session_time = 0\n                else:\n                    self.session_time = (data['timestamp'] - self._last_session_start_time) / 1000 / 60\n\n            # 处理问题记录部分\n            if str(data['content_id']) in self._question_num_dict:\n                self._question_num_dict[str(data['content_id'])] = self._question_num_dict[str(data['content_id'])] + 1\n            else:\n                self._question_num_dict[str(data['content_id'])] = 1\n\n            # 处理TrueSkill部分\n            if data['answered_correctly'] == 1:\n                self._mmr_object, _temp = \\\n                    trueskill.rate_1vs1(self._mmr_object,\n                                        trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486,\n                                                        beta=0.05, tau=0.00164, draw_probability=0).Rating())\n                self._mmr_object_agg_field[question_field - 1], _temp = \\\n                    trueskill.rate_1vs1(self._mmr_object_agg_field[question_field - 1],\n                                        trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486,\n                                                        beta=0.05,\n                                                        tau=0.00164, draw_probability=0).Rating())\n            else:\n                _temp, self._mmr_object = \\\n                    trueskill.rate_1vs1(trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486,\n                                                        beta=0.05, tau=0.00164, draw_probability=0).Rating(),\n                                        self._mmr_object)\n\n                _temp, self._mmr_object_agg_field[question_field - 1] = \\\n                    trueskill.rate_1vs1(trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486,\n                                                        beta=0.05,\n                                                        tau=0.00164, draw_probability=0).Rating(),\n                                        self._mmr_object_agg_field[question_field - 1])\n\n\n\n        else:\n            # Content Type 不为 0 ，即用户在观看视频\n\n            self.lessons_overall = self.lessons_overall + 1\n            lesson_field = int(data['content_field'])\n            self.lessons_overall_agg_field[lesson_field - 1] = self.lessons_overall_agg_field[lesson_field - 1] + 1\n\n    def process_output(self, data):\n        '''\n        根据user现有属性设置输出训练数据\n        :param data: 本行数据集\n        :return: output_dict 训练数据\n        '''\n        output_dict = {}\n\n        # 回答数量类\n        output_dict['question_answered_num'] = self.question_answered_num\n        output_dict['question_answered_num_agg_field'] = self.question_answered_num_agg_field[\n            int(data['content_field']) - 1]\n\n        # 回答正确率类\n        output_dict['question_answered_mean_accuracy'] = self.question_answered_mean_accuracy\n\n        output_dict['question_answered_mean_accuracy_agg_field'] = self.question_answered_mean_accuracy_agg_field[\n            int(data['content_field']) - 1]\n        output_dict[\n            'question_answered_mean_difficulty_weighted_accuracy'] = self.question_answered_mean_difficulty_weighted_accuracy\n        output_dict['question_answered_mean_difficulty_weighted_accuracy_agg_field'] = \\\n            self.question_answered_mean_difficulty_weighted_accuracy_agg_field[int(data['content_field']) - 1]\n\n        # 极值类\n\n        output_dict['max_solved_difficulty'] = self.max_solved_difficulty\n        output_dict['max_solved_difficulty_agg_field'] = self.max_solved_difficulty_agg_field[\n            int(data['content_field']) - 1]\n        output_dict['min_wrong_difficulty'] = self.min_wrong_difficulty\n        output_dict['min_wrong_difficulty_agg_field'] = self.min_wrong_difficulty_agg_field[\n            int(data['content_field']) - 1]\n\n        # 课程学习类\n        output_dict['lessons_overall'] = self.lessons_overall\n        output_dict['lessons_overall_agg_field'] = self.lessons_overall_agg_field[int(data['content_field']) - 1]\n        if output_dict['lessons_overall_agg_field'] > 0:\n            output_dict['field_learnt'] = 1\n        else:\n            output_dict['field_learnt'] = 0\n        # 交互时间类\n        output_dict['session_time'] = self.session_time\n        output_dict['time_to_last_session'] = self.since_last_session_time\n\n        output_dict['task_id'] = data['task_container_id']\n        output_dict['prior_time'] = data['prior_question_elapsed_time']\n        # 问题信息类\n        output_dict['mean_question_accuracy'] = data['mean_question_accuracy']\n        output_dict['std_question_accuracy'] = data['std_accuracy']\n        output_dict['question_id'] = data['content_id']\n        # TrueSkill 信息类\n        output_dict['mmr_overall'] = self._mmr_object.mu\n        output_dict['mmr_overall_agg_field'] = self._mmr_object_agg_field[int(data['content_field']) - 1].mu\n        output_dict['mmr_confidence'] = self._mmr_object.sigma\n\n        output_dict['mmr_overall_agg_field'] = self._mmr_object_agg_field[int(data['content_field']) - 1].sigma\n        output_dict['mmr_win_prob'] = win_probability(self._mmr_object,\n                                                      trueskill.setup(mu=1 - data['mean_question_accuracy'],\n                                                                      sigma=0.164486,\n                                                                      beta=0.05, tau=0.00164,\n                                                                      draw_probability=0).Rating())\n        output_dict['mmr_win_prob_agg_field'] = win_probability(\n            self._mmr_object_agg_field[int(data['content_field']) - 1],\n            trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486, beta=0.05,\n                            tau=0.00164, draw_probability=0).Rating())\n        output_dict['user_id'] = data['user_id']\n        output_dict['tag_1'] = data['tag_1']\n        output_dict['tag_2'] = data['tag_2']\n\n        output_dict['tags_encoded'] = data['tags_encoded']\n        # 特殊特征类\n\n        if not pd.isna(['prior_question_had_explanation']):\n            output_dict['previous_explained'] = data['prior_question_had_explanation']\n        else:\n            output_dict['previous_explained'] = False\n\n        if str(data['content_id']) in self._question_num_dict:\n            output_dict['question_seen'] = 1\n        else:\n            output_dict['question_seen'] = 0\n\n        # 猜测类\n        max_choice = 0\n        max_choice_num = 0\n        i = 0\n        for item in self._most_liked_guess:\n            if item > max_choice_num:\n                max_choice_num = item\n                max_choice = i\n            i = i + 1\n\n        if output_dict['mmr_win_prob'] <= 0.4:\n            if max_choice == data['correct_answer']:\n                output_dict['most_liked_guess_correct'] = True\n            else:\n                output_dict['most_liked_guess_correct'] = False\n        else:\n            output_dict['most_liked_guess_correct'] = True\n\n        # 训练目标\n        #output_dict['answered_correctly'] = data['answered_correctly']\n\n        return output_dict\n\n# 导入Metadata\nquestion_metadata = pd.read_csv(question_metadata_dir)\nlesson_metadata = pd.read_csv(lesson_metadata_dir)\nprint(\"{} Metadata 文件导入完成\".format(str(datetime.datetime.now())))\n\n#设置Metadata索引\nquestion_metadata = question_metadata.set_index(keys=['content_id'])\nlesson_metadata = lesson_metadata.set_index(keys=['content_id'])\nprint(\"{} Metadata 索引设置完成\".format(str(datetime.datetime.now())))\n\n#导入Pickle状态\nwith open(pickle_dir, 'rb') as fo:\n    user_pickle = pickle.load(fo)\n\nprint(\"{} Pickle 导入完成\".format(str(datetime.datetime.now())))\n\n#重置Trueskill状态\nfor user_id,user_info in user_pickle.items():\n    user_pickle[user_id]._mmr_object = trueskill.setup(mu=user_pickle[user_id]._mmr_object[0],\n                                                       sigma=user_pickle[user_id]._mmr_object[1],\n                                                       beta=0.05, tau=0.00164,\n                                                       draw_probability=0).Rating()\n    for i in range(0, 7):\n        # 1+1\n        user_pickle[user_id]._mmr_object_agg_field[i] =  trueskill.setup(mu=user_pickle[user_id]._mmr_object_agg_field[i][0],\n                                                       sigma=user_pickle[user_id]._mmr_object_agg_field[i][1],\n                                                       beta=0.05, tau=0.00164,\n                                                       draw_probability=0).Rating()\n\nprint(\"{} Pickle Trueskill状态重置完成\".format(str(datetime.datetime.now())))\n\n#导入模型\nmodel = lgb.Booster(model_file=model_dir)\nprint(\"{} 模型导入完成\".format(str(datetime.datetime.now())))\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MAX_SEQ = 100\nimport torch,joblib\nimport psutil\nimport torch.nn as nn\nfrom torch.utils.data import Dataset, DataLoader\nclass FFN(nn.Module):\n    def __init__(self, state_size=200):\n        super(FFN, self).__init__()\n        self.state_size = state_size\n\n        self.lr1 = nn.Linear(state_size, state_size)\n        self.relu = nn.ReLU()\n        self.lr2 = nn.Linear(state_size, state_size)\n        self.dropout = nn.Dropout(0.2)\n    \n    def forward(self, x):\n        x = self.lr1(x)\n        x = self.relu(x)\n        x = self.lr2(x)\n        return self.dropout(x)\n\ndef future_mask(seq_length):\n    future_mask = np.triu(np.ones((seq_length, seq_length)), k=1).astype('bool')\n    return torch.from_numpy(future_mask)\n\n\nclass SAKTModel(nn.Module):\n    def __init__(self, n_skill, max_seq=MAX_SEQ, embed_dim=128):\n        super(SAKTModel, self).__init__()\n        self.n_skill = n_skill\n        self.embed_dim = embed_dim\n\n        self.embedding = nn.Embedding(2*n_skill+1, embed_dim)\n        self.pos_embedding = nn.Embedding(max_seq-1, embed_dim)\n        self.e_embedding = nn.Embedding(n_skill+1, embed_dim)\n\n        self.multi_att = nn.MultiheadAttention(embed_dim=embed_dim, num_heads=8, dropout=0.2)\n\n        self.dropout = nn.Dropout(0.2)\n        self.layer_normal = nn.LayerNorm(embed_dim) \n\n        self.ffn = FFN(embed_dim)\n        self.pred = nn.Linear(embed_dim, 1)\n    \n    def forward(self, x, question_ids):\n        device = x.device        \n        x = self.embedding(x)\n        pos_id = torch.arange(x.size(1)).unsqueeze(0).to(device)\n\n        pos_x = self.pos_embedding(pos_id)\n        x = x + pos_x\n\n        e = self.e_embedding(question_ids)\n\n        x = x.permute(1, 0, 2) # x: [bs, s_len, embed] => [s_len, bs, embed]\n        e = e.permute(1, 0, 2)\n        att_mask = future_mask(x.size(0)).to(device)\n        att_output, att_weight = self.multi_att(e, x, x, attn_mask=att_mask)\n        att_output = self.layer_normal(att_output + e)\n        att_output = att_output.permute(1, 0, 2) # att_output: [s_len, bs, embed] => [bs, s_len, embed]\n\n        x = self.ffn(att_output)\n        x = self.layer_normal(x + att_output)\n        x = self.pred(x)\n\n        return x.squeeze(-1), att_weight\nskills = joblib.load(\"/kaggle/input/riiid-sakt-model-dataset-public/skills.pkl.zip\")\nn_skill = len(skills)\ngroup = joblib.load(\"/kaggle/input/riiid-sakt-model-dataset-public/group.pkl.zip\")\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nmodel_SAKT = SAKTModel(n_skill, embed_dim=128)\ntry:\n    model_SAKT.load_state_dict(torch.load(\"/kaggle/input/riiid-sakt-model-dataset-public/sakt_model.pt\"))\nexcept:\n    model_SAKT.load_state_dict(torch.load(\"/kaggle/input/riiid-sakt-model-dataset-public/sakt_model.pt\", map_location='cpu'))\nmodel_SAKT.to(device)\nmodel_SAKT.eval()\nclass TestDataset(Dataset):\n    def __init__(self, samples, test_df, skills, max_seq=MAX_SEQ): \n        super(TestDataset, self).__init__()\n        self.samples = samples\n        self.user_ids = [x for x in test_df[\"user_id\"].unique()]\n        self.test_df = test_df\n        self.skills = skills\n        self.n_skill = len(skills)\n        self.max_seq = max_seq\n\n    def __len__(self):\n        return self.test_df.shape[0]\n\n    def __getitem__(self, index):\n        test_info = self.test_df.iloc[index]\n\n        user_id = test_info[\"user_id\"]\n        target_id = test_info[\"content_id\"]\n\n        q = np.zeros(self.max_seq, dtype=int)\n        qa = np.zeros(self.max_seq, dtype=int)\n\n        if user_id in self.samples.index:\n            q_, qa_ = self.samples[user_id]\n            \n            seq_len = len(q_)\n\n            if seq_len >= self.max_seq:\n                q = q_[-self.max_seq:]\n                qa = qa_[-self.max_seq:]\n            else:\n                q[-seq_len:] = q_\n                qa[-seq_len:] = qa_          \n        \n        x = np.zeros(self.max_seq-1, dtype=int)\n        x = q[1:].copy()\n        x += (qa[1:] == 1) * self.n_skill\n        \n        questions = np.append(q[2:], [target_id])\n        \n        return x, questions\nprev_test_df = None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# test_df['answered_correctly'] = 0\n# env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()\niter_test = env.iter_test()\nprev_test_df = None\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nprint(\"{} 比赛环境设置完成\".format(str(datetime.datetime.now())))\n\n#初始换变量\nrows_accum = 0 #行计数器\nfirst_submission = True #是否第一组标记\nmodel_prd = [0]\ntrue_value = []\nlast_df = pd.DataFrame()\nprint(\"{} 比赛变量设置完成\".format(str(datetime.datetime.now())))\n\nfor (test_df, sample_prediction_df) in iter_test:\n    \n    cur = (test_df, sample_prediction_df)\n    \n    if (prev_test_df is not None) & (psutil.virtual_memory().percent < 90):\n        prev_test_df['answered_correctly'] = eval(test_df['prior_group_answers_correct'].iloc[0])\n        prev_test_df = prev_test_df[prev_test_df.content_type_id == False]\n        \n        prev_group = prev_test_df[['user_id', 'content_id', 'answered_correctly']].groupby('user_id').apply(lambda r: (\n            r['content_id'].values,\n            r['answered_correctly'].values))\n        for prev_user_id in prev_group.index:\n            if prev_user_id in group.index:\n                group[prev_user_id] = (\n                    np.append(group[prev_user_id][0], prev_group[prev_user_id][0])[-MAX_SEQ:], \n                    np.append(group[prev_user_id][1], prev_group[prev_user_id][1])[-MAX_SEQ:]\n                )\n \n            else:\n                group[prev_user_id] = (\n                    prev_group[prev_user_id][0], \n                    prev_group[prev_user_id][1]\n                )\n\n    prev_test_df = test_df.copy()\n    \n    test_df = test_df[test_df.content_type_id == False]\n    TEST = test_df[test_df.content_type_id == False]\n    test_dataset = TestDataset(group, test_df, skills)\n    test_dataloader = DataLoader(test_dataset, batch_size=51200, shuffle=False)\n    \n    outs = []\n\n    for item in test_dataloader:\n        x = item[0].to(device).long()\n        target_id = item[1].to(device).long()\n\n        with torch.no_grad():\n            output, att_weight = model_SAKT(x, target_id)\n        outs.extend(torch.sigmoid(output)[:, -1].view(-1).data.cpu().numpy())\n    \n    \n    TEST['answered_correctly'] = outs\n    TEST = TEST.loc[TEST['content_type_id'] == 0, ['row_id', 'answered_correctly']]\n    \n    \n    \n    \n    (test_df, sample_prediction_df) = cur\n    \n    \n    if first_submission == False:\n        last_df['answered_correctly'] = eval(test_df.iloc[0]['prior_group_answers_correct'])\n        last_df['user_answer'] = eval(test_df.iloc[0]['prior_group_responses'])\n        true_value.extend(eval(test_df.iloc[0]['prior_group_answers_correct']))\n        for index,row in last_df.iterrows():\n            user_pickle[row['user_id']].update_user(row)\n    rows_accum = rows_accum + test_df.shape[0]\n    if first_submission == False:\n        1+1\n        #print(\"{} 当前正在处理第 {} 行 , 截至目前AUC为 {}\".format(str(datetime.datetime.now()),rows_accum,roc_auc_score(true_value,model_prd)))\n    test_df['answered_correctly'] = 0.6524\n    st = float(time.time())\n    # 完成Merge 与 Concat 工作\n    try:\n        sub_1 = test_df[test_df['content_type_id'] == False]\n        sub_2 = test_df[test_df['content_type_id'] == True]\n        del test_df\n        sub_1 = sub_1.merge(question_metadata, on=\"content_id\", how=\"left\")\n        sub_2 = sub_2.merge(lesson_metadata, on=\"content_id\", how=\"left\")\n        test_df = pd.DataFrame()\n        test_df = pd.concat([sub_1,sub_2])\n    except Exception:\n        pass\n\n    for index, row in test_df.iterrows():\n        #print(row.row_id)\n        try:\n            if row['user_id'] not in user_pickle:\n                user_pickle[row['user_id']] = user()\n            if row['content_type_id'] == 0:\n                predict_dict = user_pickle[row['user_id']].process_output(row)\n                l = []\n                for i,v in predict_dict.items():\n                    l.append(v)\n                prd_value = float(model.predict([l])[0])\n                test_df.loc[test_df.row_id == row.row_id, 'answered_correctly'] = 0.5*prd_value+float(TEST[TEST['row_id']==row.row_id]['answered_correctly'])*0.5\n                model_prd.append(prd_value)\n\n\n        except Exception as e:\n            print(e)\n            pass\n\n    time_taken = float(time.time()) - st\n    print(\"{} 基于当前速率，共需要 {} 分钟完成预测\".format(\n        str(datetime.datetime.now()),int(time_taken / test_df.shape[0] * 2500000 / 60)))\n    if first_submission == True:\n        first_submission = False\n    last_df = test_df\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"float(TEST[TEST['row_id']==row.row_id]['answered_correctly']*0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}