{"cells":[{"metadata":{},"cell_type":"markdown","source":"# **This notebook is only used to submit LB, riiid_dataset contains the trained model and final user state, this notebook does not include the training process!**\n\n\n* **This is my first time participating in a Kaggle Competition（Formal）. I am currently a freshman studying a non-computer major  (just 3 months after entering the colledge) and I have never participated in any competition in high school before. Most of my Python skills are learnt online. Therefore, the code style/model description may be very confusing, Sorry for the inconvenience. If you have any suggestions/questions, you are welcome to point out in the comment section below**\n\n* This notebook is for submission ONLY. The training code is quiet messy and needs to be sorted out. I may upload it if I have time in the near future.\n\n* This notebook is mostly translated by Google Translation,as English is not my native language.\n\n\n\n* **Notes on riiid-dataset dataset**\n\n\n1. Model（classifier.model）\n\nClassifier.model is a model trained by LightGBM without parameter optimization. All parameters are LightGBM default (random seed is 2020), and it is only trained on the first 20M rows of data.If parameters are optimized/trained on the entire data set, this model may be still  room for improvement.\n\n2. Stage.Pickle\n\nConsidering  the feature engineering is obtained by iterating the DataFrame, and this process takes a long time, in order to speed up the notebook ,we save the final state after the iteration and store it in the Pickle file.\n\n3. Lesson_Metadata and Question_Metadata\n\nIn the Lesson_Metadata and Question_Metadata files, we save the pre-calculated features related to the test questions (Tag, Mean Correct rate, etc.) and the features related to the course (Tag).\n\n\n* **Feature Description**\n\n\n1. question_answered_num \n\nThe total number of questions answered by users\n\n\n2. question_answered_num_agg_field \n\nThe total question answered by the user (according to TOEIC subject area statistics)\n\n\n3. question_answered_mean_accuracy \n\nThe average correct rate of questions answered by users\n\n\n4. question_answered_mean_accuracy_agg_field \n\nThe average correct rate of questions answered by users (according to TOEIC subject area statistics)\n\n\n5.question_answered_mean_difficulty_weighted_accuracy  \n\nThe average difficulty weighted correct rate of questions answered by users\n\n\n6. question_answered_mean_difficulty_weighted_accuracy_agg_field \n\nThe average difficulty weighted correct rate of the questions answered by users (according to TOEIC subject area statistics)\n\n\n7. max_solved_difficulty \n\nThe most difficult question answered by users\n\n\n\n8. max_solved_difficulty_agg_field \n\nThe most difficult question answered by users (according to TOEIC subject area statistics)\n\n\n\n9. min_wrong_difficulty \n\nThe simplest problem that users do wrong\n\n\n\n10. min_wrong_difficulty_agg_field \n\nThe simplest problem that users do wrong (according to TOEIC subject area statistics)\n\n\n\n11. lessons_overall\n\nHow many lessons the user learned in total\n\n\n\n12. lessons_overall_agg_field \n\nHow many lessons the user has learned in total (by TOEIC subject area statistics)\n\n\n\n13. session_time \n\nTime since starting of user's session (in minutes)\n\n\n\n14. since_last_session_time \n\nTime since the last session (in hours)\n\n\n\n15. user_id \n\nuser ID\n\n\n\n16. tag_1 \n\nQuestion Tag1 \n\n\n\n17. tag_2 \n\nQuestion Tag2 \n\n\n\n18. tag_encoded \n\nencoded number based on Tag1, 2, 3\n\n\n\n19. question_id \n\nThe content_id of the question\n\n\n\n20. previous_explained \n\nHas the previous question been answered?\n\n\n\n21. question_seen \n\nHas the user ever seen this question?\n\n\n\n22. mean_question_accuracy \n\nThe median accuracy of the question\n\n\n\n23. std_question_accuracy \n\nThe standard deviation of the correct rate of the question\n\n\n24. most_liked_guess_correct \n\nSuppose you can't do a certain multiple-choice question in the exam, then you will probably guess a C (I'm used to guessing the longer choice) or something. In view of this, we record the user’s favorite option every time a user makes a mistake. After that, if the Trueskill model mentioned below believes that the probability of the user doing the question correctly is low, we will compare the user's favorite option with the option of the question to see if the user guessed it correctly.\n\n\n\n*The remaining features are related to the Trueskill scoring system*\n\nTrueskill is a scoring system developed by Microsoft (https://www.microsoft.com/en-us/research/project/trueskill-ranking-system/)\n\nIf you have played Rainbow Six, you may have been exposed to this system. In this game, your ranking is determined by the Trueskill system.\n\nConsider regarding each Question as a user (experienced player) with determined abilities, and User as a novice who just started playing. After that, the Trueskill system can be used to determine its ranking and its winning probability.\n\n*Note The reason why we can apply the Trueskill system is because the distribution of the difficulty of the problem is close to a normal distribution [determined by (1-the median accuracy of the user group as a whole)].\n\n25. mmr_overall\n\nuser's MMR score\n\n26. mmr_overall_agg_field \n\nuser's MMR score (according to TOEIC subject area statistics)\n\n27. mmr_confidence \n\nUser's MMR confidence level\n\n28. mmr_overall_agg_field \n\nuser's MMR confidence level (according to TOEIC subject area statistics)\n\n29. mmr_win_prob \n\nThe probability of the user'beating' the question\n\n30. mmr_win_prob_agg_field \n\nThe probability of the user'beating' the question (based on TOEIC subject area statistics)\n\n\n\n\n* **Copyright Statements**\n\nTrueskill(TM) is a registered trademark of Microsoft Corporation.\n\nThe truskill library was created by Heungsub Lee, based on the BSD open source project, check out: https://trueskill.org/"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"'''\nRiiid Competition Submission Ver 1.0.1 Alpha\n(C) Copyright By Author 2020 - Now\nAll rights reserved\n'''\nimport sys\nsys.path.append('/kaggle/input/riiid-dataset/')\n#Dir\nquestion_metadata_dir = r'/kaggle/input/riiid-dataset/question_metadata.csv'\nlesson_metadata_dir = r'/kaggle/input/riiid-dataset/lesson_metadata.csv'\npickle_dir= r'/kaggle/input/riiid-dataset/stage.pickle'\nmodel_dir = r'/kaggle/input/riiid-dataset/classifier.model'\n\nimport datetime\nprint(\"{} Submission Started\".format(str(datetime.datetime.now())))\n# Import some Stuff\ntry:\n    import pandas as pd\n    import pickle\n    import trueskill\n    import math\n    import lightgbm as lgb\n    import riiideducation\n    import time\n    from sklearn.metrics import roc_auc_score\n\n\nexcept ImportError as e:\n    print(\"{} Import Processed finished with error：{}\".format(str(datetime.datetime.now()), e))\n\nprint(\"{} Import Completed\".format(str(datetime.datetime.now())))\nenv = trueskill.TrueSkill(mu=0.3, sigma=0.164486, beta=0.05, tau=0.00164, draw_probability=0)\nenv.make_as_global()\n\n\ndef win_probability(team1, team2):\n    '''\n    Calculate the win possibility based on two Trueskill objects\n    :param team1:User TrueSkill Object\n    :param team2:Question Trueskill Object\n    :return: Winning Prob\n    '''\n    delta_mu = team1.mu - team2.mu\n    sum_sigma = sum([team1.sigma ** 2, team2.sigma ** 2])\n    size = 2\n    denom = math.sqrt(size * (0.05 * 0.05) + sum_sigma)\n    ts = trueskill.global_env()\n    return ts.cdf(delta_mu / denom)\n\n\nclass user:\n    '''\n    User Class\n    '''\n\n    def __init__(self):\n        '''\n        Init User Class\n        :param None\n        :return: None\n        '''\n        # Please Refer to Documentation above for meaning of features.\n        # Directly output features\n\n        # Counting \n        self.question_answered_num = 0  \n        self.question_answered_num_agg_field = [0] * 7 \n\n        # Correct Rate\n        self.question_answered_mean_accuracy = 0  \n        self.question_answered_mean_accuracy_agg_field = [0] * 7  \n        self.question_answered_mean_difficulty_weighted_accuracy = 0  \n        self.question_answered_mean_difficulty_weighted_accuracy_agg_field = [0] * 7  \n        # Min/Max stuff\n        self.max_solved_difficulty = 1\n        self.max_solved_difficulty_agg_field = [1] * 7\n        self.min_wrong_difficulty = 0 \n        self.min_wrong_difficulty_agg_field = [0] * 7  \n\n        # Lessons stuff\n        self.lessons_overall = 0 \n        self.lessons_overall_agg_field = [0] * 7  \n\n        # Session timing \n        self.session_time = 0  \n        self.since_last_session_time = 0  \n\n        # Features need some processing\n        self._mmr_object = trueskill.setup(mu=0.3, sigma=0.164486, beta=0.05, tau=0.00164,\n                                           draw_probability=0).Rating() \n        self._mmr_object_agg_field = [trueskill.setup(mu=0.3, sigma=0.164486, beta=0.05, tau=0.00164,\n                                                      draw_probability=0).Rating()] * 7\n        self._most_liked_guess = [0] * 4 \n        self._last_session_start_time = 0  \n        self._first_action_time = 0  \n        self._question_num_dict = {}  \n        self._first_processed_flag = False  \n\n    def update_user(self, data: pd.DataFrame):\n        '''\n        Update user with one row of DataFrame\n        :param data: pandas DataFrame\n        :return: None\n        '''\n        _temp = None\n\n        # Judging whether user are watching courses\n        if data['content_type_id'] == 0:\n            # Content Type = 0,means User are answering Questions.\n\n            # Counting Part\n            self.question_answered_num = self.question_answered_num + 1\n            question_field = int(data['content_field'])\n            self.question_answered_num_agg_field[question_field - 1] = int(self.question_answered_num_agg_field[\n                                                                               question_field - 1]) + 1\n\n            # Average Correct Rate\n            if data['answered_correctly'] == 1:\n                self.question_answered_mean_accuracy = \\\n                    (self.question_answered_mean_accuracy * (\n                            self.question_answered_num - 1) + 1) / self.question_answered_num\n\n                self.question_answered_mean_accuracy_agg_field[question_field - 1] = \\\n                    (self.question_answered_mean_accuracy_agg_field[question_field - 1] * (\n                            self.question_answered_num_agg_field[question_field - 1] - 1) + 1) \\\n                    / self.question_answered_num_agg_field[question_field - 1]\n\n                self.question_answered_mean_difficulty_weighted_accuracy = \\\n                    (self.question_answered_mean_difficulty_weighted_accuracy * (self.question_answered_num - 1) + (\n                            1 - data['mean_question_accuracy']) * 3) \\\n                    / self.question_answered_num\n\n                self.question_answered_mean_difficulty_weighted_accuracy_agg_field[question_field - 1] = \\\n                    (self.question_answered_mean_difficulty_weighted_accuracy_agg_field[question_field - 1] * (\n                            self.question_answered_num_agg_field[question_field - 1] - 1) + (\n                             1 - data['mean_question_accuracy']) * 3) \\\n                    / self.question_answered_num_agg_field[question_field - 1]\n\n\n            else:\n                self.question_answered_mean_accuracy = \\\n                    (self.question_answered_mean_accuracy * (\n                            self.question_answered_num - 1)) / self.question_answered_num\n\n                self.question_answered_mean_accuracy_agg_field[question_field - 1] = \\\n                    (self.question_answered_mean_accuracy_agg_field[question_field - 1] * (\n                            self.question_answered_num_agg_field[question_field - 1] - 1)) / \\\n                    self.question_answered_num_agg_field[question_field - 1]\n\n                self.question_answered_mean_difficulty_weighted_accuracy = \\\n                    (self.question_answered_mean_difficulty_weighted_accuracy * (self.question_answered_num - 1)) \\\n                    / self.question_answered_num\n\n                self.question_answered_mean_difficulty_weighted_accuracy_agg_field[question_field - 1] = \\\n                    (self.question_answered_mean_difficulty_weighted_accuracy_agg_field[question_field - 1] * (\n                            self.question_answered_num_agg_field[question_field - 1] - 1)) \\\n                    / self.question_answered_num_agg_field[question_field - 1]\n\n            # Min/Max Part\n\n            if data['answered_correctly'] == 1:\n                if data['mean_question_accuracy'] < self.max_solved_difficulty:\n                    self.max_solved_difficulty = data['mean_question_accuracy']\n                if data['mean_question_accuracy'] < self.max_solved_difficulty_agg_field[question_field - 1]:\n                    self.max_solved_difficulty_agg_field[question_field - 1] = data['mean_question_accuracy']\n            else:\n                if data['mean_question_accuracy'] > self.min_wrong_difficulty:\n                    self.min_wrong_difficulty = data['mean_question_accuracy']\n                if data['mean_question_accuracy'] > self.min_wrong_difficulty_agg_field[question_field - 1]:\n                    self.min_wrong_difficulty_agg_field[question_field - 1] = data['mean_question_accuracy']\n\n            # Guessing Part\n            if data['answered_correctly'] == 0:\n                self._most_liked_guess[int(data['user_answer'])] = self._most_liked_guess[\n                                                                       int(data['user_answer'])] + 1\n\n            # Session Timing part\n            if self._first_action_time == 0:\n                self._first_action_time = data['timestamp']\n                self._last_session_start_time = data['timestamp']\n            else:\n                if data['timestamp'] - self._last_session_start_time >= 7200 * 1000:\n                    self.since_last_session_time = (data[\n                                                        'timestamp'] - self._last_session_start_time) / 1000 / 3600\n                    self._last_session_start_time = data['timestamp']\n                    self.session_time = 0\n                else:\n                    self.session_time = (data['timestamp'] - self._last_session_start_time) / 1000 / 60\n\n            # Answer history part\n            if str(data['content_id']) in self._question_num_dict:\n                self._question_num_dict[str(data['content_id'])] = self._question_num_dict[str(data['content_id'])] + 1\n            else:\n                self._question_num_dict[str(data['content_id'])] = 1\n\n            # Trueskill part\n            if data['answered_correctly'] == 1:\n                self._mmr_object, _temp = \\\n                    trueskill.rate_1vs1(self._mmr_object,\n                                        trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486,\n                                                        beta=0.05, tau=0.00164, draw_probability=0).Rating())\n                self._mmr_object_agg_field[question_field - 1], _temp = \\\n                    trueskill.rate_1vs1(self._mmr_object_agg_field[question_field - 1],\n                                        trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486,\n                                                        beta=0.05,\n                                                        tau=0.00164, draw_probability=0).Rating())\n            else:\n                _temp, self._mmr_object = \\\n                    trueskill.rate_1vs1(trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486,\n                                                        beta=0.05, tau=0.00164, draw_probability=0).Rating(),\n                                        self._mmr_object)\n\n                _temp, self._mmr_object_agg_field[question_field - 1] = \\\n                    trueskill.rate_1vs1(trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486,\n                                                        beta=0.05,\n                                                        tau=0.00164, draw_probability=0).Rating(),\n                                        self._mmr_object_agg_field[question_field - 1])\n\n\n\n        else:\n            # Content Type !=0,User are watching a  lecture\n\n            self.lessons_overall = self.lessons_overall + 1\n            lesson_field = int(data['content_field'])\n            self.lessons_overall_agg_field[lesson_field - 1] = self.lessons_overall_agg_field[lesson_field - 1] + 1\n\n    def process_output(self, data):\n        '''\n        \n         Output data according to user's existing attributes\n        :param data: One row of dataset\n        :return: output_dict dict data for training/predicting\n        '''\n        output_dict = {}\n\n        # Counting Part\n        output_dict['question_answered_num'] = self.question_answered_num\n        output_dict['question_answered_num_agg_field'] = self.question_answered_num_agg_field[\n            int(data['content_field']) - 1]\n\n        # Average Correct Rate\n        output_dict['question_answered_mean_accuracy'] = self.question_answered_mean_accuracy\n\n        output_dict['question_answered_mean_accuracy_agg_field'] = self.question_answered_mean_accuracy_agg_field[\n            int(data['content_field']) - 1]\n        output_dict[\n            'question_answered_mean_difficulty_weighted_accuracy'] = self.question_answered_mean_difficulty_weighted_accuracy\n        output_dict['question_answered_mean_difficulty_weighted_accuracy_agg_field'] = \\\n            self.question_answered_mean_difficulty_weighted_accuracy_agg_field[int(data['content_field']) - 1]\n\n        #  Min/Max Part\n\n        output_dict['max_solved_difficulty'] = self.max_solved_difficulty\n        output_dict['max_solved_difficulty_agg_field'] = self.max_solved_difficulty_agg_field[\n            int(data['content_field']) - 1]\n        output_dict['min_wrong_difficulty'] = self.min_wrong_difficulty\n        output_dict['min_wrong_difficulty_agg_field'] = self.min_wrong_difficulty_agg_field[\n            int(data['content_field']) - 1]\n\n        # Lesson Learning part\n        output_dict['lessons_overall'] = self.lessons_overall\n        output_dict['lessons_overall_agg_field'] = self.lessons_overall_agg_field[int(data['content_field']) - 1]\n        if output_dict['lessons_overall_agg_field'] > 0:\n            output_dict['field_learnt'] = 1\n        else:\n            output_dict['field_learnt'] = 0\n        # Session Timing part\n        output_dict['session_time'] = self.session_time\n        output_dict['time_to_last_session'] = self.since_last_session_time\n\n        output_dict['task_id'] = data['task_container_id']\n        output_dict['prior_time'] = data['prior_question_elapsed_time']\n        # Question Statics part\n        output_dict['mean_question_accuracy'] = data['mean_question_accuracy']\n        output_dict['std_question_accuracy'] = data['std_accuracy']\n        output_dict['question_id'] = data['content_id']\n        # TrueSkill part\n        output_dict['mmr_overall'] = self._mmr_object.mu\n        output_dict['mmr_overall_agg_field'] = self._mmr_object_agg_field[int(data['content_field']) - 1].mu\n        output_dict['mmr_confidence'] = self._mmr_object.sigma\n\n        output_dict['mmr_overall_agg_field'] = self._mmr_object_agg_field[int(data['content_field']) - 1].sigma\n        output_dict['mmr_win_prob'] = win_probability(self._mmr_object,\n                                                      trueskill.setup(mu=1 - data['mean_question_accuracy'],\n                                                                      sigma=0.164486,\n                                                                      beta=0.05, tau=0.00164,\n                                                                      draw_probability=0).Rating())\n        output_dict['mmr_win_prob_agg_field'] = win_probability(\n            self._mmr_object_agg_field[int(data['content_field']) - 1],\n            trueskill.setup(mu=1 - data['mean_question_accuracy'], sigma=0.164486, beta=0.05,\n                            tau=0.00164, draw_probability=0).Rating())\n        output_dict['user_id'] = data['user_id']\n        output_dict['tag_1'] = data['tag_1']\n        output_dict['tag_2'] = data['tag_2']\n\n        output_dict['tags_encoded'] = data['tags_encoded']\n        # Other features\n\n        if not pd.isna(['prior_question_had_explanation']):\n            output_dict['previous_explained'] = data['prior_question_had_explanation']\n        else:\n            output_dict['previous_explained'] = False\n\n        if str(data['content_id']) in self._question_num_dict:\n            output_dict['question_seen'] = 1\n        else:\n            output_dict['question_seen'] = 0\n\n        # Guessing part\n        max_choice = 0\n        max_choice_num = 0\n        i = 0\n        for item in self._most_liked_guess:\n            if item > max_choice_num:\n                max_choice_num = item\n                max_choice = i\n            i = i + 1\n\n        if output_dict['mmr_win_prob'] <= 0.4:\n            if max_choice == data['correct_answer']:\n                output_dict['most_liked_guess_correct'] = True\n            else:\n                output_dict['most_liked_guess_correct'] = False\n        else:\n            output_dict['most_liked_guess_correct'] = True\n\n        # Target\n        #output_dict['answered_correctly'] = data['answered_correctly']\n\n        return output_dict\n\n# Import Metadata\nquestion_metadata = pd.read_csv(question_metadata_dir)\nlesson_metadata = pd.read_csv(lesson_metadata_dir)\nprint(\"{} Metadata Imported\".format(str(datetime.datetime.now())))\n\n#Indexing Metadata\nquestion_metadata = question_metadata.set_index(keys=['content_id'])\nlesson_metadata = lesson_metadata.set_index(keys=['content_id'])\nprint(\"{} Metadata Indexed\".format(str(datetime.datetime.now())))\n\n#Import pickle Object\nwith open(pickle_dir, 'rb') as fo:\n    user_pickle = pickle.load(fo)\n\nprint(\"{} Pickle Object Imported\".format(str(datetime.datetime.now())))\n\n#Rebuilding Trueskill \nfor user_id,user_info in user_pickle.items():\n    user_pickle[user_id]._mmr_object = trueskill.setup(mu=user_pickle[user_id]._mmr_object[0],\n                                                       sigma=user_pickle[user_id]._mmr_object[1],\n                                                       beta=0.05, tau=0.00164,\n                                                       draw_probability=0).Rating()\n    for i in range(0, 7):\n        # 1+1\n        user_pickle[user_id]._mmr_object_agg_field[i] =  trueskill.setup(mu=user_pickle[user_id]._mmr_object_agg_field[i][0],\n                                                       sigma=user_pickle[user_id]._mmr_object_agg_field[i][1],\n                                                       beta=0.05, tau=0.00164,\n                                                       draw_probability=0).Rating()\n\nprint(\"{} Pickle Trueskill Rebuilt\".format(str(datetime.datetime.now())))\n\n#Import Model\nmodel = lgb.Booster(model_file=model_dir)\nprint(\"{} Model Imported\".format(str(datetime.datetime.now())))\n\n#Init Environment\nenv = riiideducation.make_env()\niter_test = env.iter_test()\nprint(\"{} Init Competiton Environment\".format(str(datetime.datetime.now())))\n\n#Init Others\nrows_accum = 0 #Row Counter\nfirst_submission = True \nmodel_prd = [0]\ntrue_value = []\nlast_df = pd.DataFrame()\nprint(\"{} Init Done!\".format(str(datetime.datetime.now())))\n\nfor (test_df, sample_prediction_df) in iter_test:\n    if first_submission == False:\n        last_df['answered_correctly'] = eval(test_df.iloc[0]['prior_group_answers_correct'])\n        last_df['user_answer'] = eval(test_df.iloc[0]['prior_group_responses'])\n        true_value.extend(eval(test_df.iloc[0]['prior_group_answers_correct']))\n        for index,row in last_df.iterrows():\n            user_pickle[row['user_id']].update_user(row)\n    rows_accum = rows_accum + test_df.shape[0]\n    if first_submission == False:\n        1+1\n        #print(\"{} Processing Row {}  ,AUC is {}\".format(str(datetime.datetime.now()),rows_accum,roc_auc_score(true_value,model_prd)))\n    test_df['answered_correctly'] = 0.6524\n    st = float(time.time())\n    # Merging and Concating\n    try:\n        sub_1 = test_df[test_df['content_type_id'] == False]\n        sub_2 = test_df[test_df['content_type_id'] == True]\n        del test_df\n        sub_1 = sub_1.merge(question_metadata, on=\"content_id\", how=\"left\")\n        sub_2 = sub_2.merge(lesson_metadata, on=\"content_id\", how=\"left\")\n        test_df = pd.DataFrame()\n        test_df = pd.concat([sub_1,sub_2])\n    except Exception:\n        pass\n\n    for index, row in test_df.iterrows():\n        try:\n            if row['user_id'] not in user_pickle:\n                user_pickle[row['user_id']] = user()\n            if row['content_type_id'] == 0:\n                predict_dict = user_pickle[row['user_id']].process_output(row)\n                l = []\n                for i,v in predict_dict.items():\n                    l.append(v)\n                prd_value = float(model.predict([l])[0])\n                test_df.loc[test_df.row_id == row.row_id, 'answered_correctly'] = prd_value\n                model_prd.append(prd_value)\n\n\n        except Exception as e:\n            print(e)\n            pass\n\n    time_taken = float(time.time()) - st\n    print(\"{} Based on current speed，it would take {} minutes to complete\".format(\n        str(datetime.datetime.now()),int(time_taken / test_df.shape[0] * 2500000 / 60)))\n    if first_submission == True:\n        first_submission = False\n    last_df = test_df\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}