{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this notebook\n\nASSUME when we submit a notebook for this competition, it runs on the whole private test dataset (and 20% of them are used to calculate the public LB score).\n\nIn this notebook, we try to answer the following questions:\n\nWhen we commit vs. when we submit a notebook\n\n* if the `questions.csv` contains the same `question_id`.\n* if the `lectures.csv` contains the same `question_id`.\n* if all the question ids in the private test dataset are seen in `train.csv` and `questions.csv`.\n* if all the lecture ids in the private test dataset are seen in `train.csv` and `lectures.csv`.\n* if a batch from the private test dataset has timestamps larger (or at least equal) than the timestamps of the corresponding users in `train.csv`. \n* if the batches from the private test datasets have monotonically increasing (actually, non-decreasing) timestamps.\n* if there is only one (if any) question bundle for a sinlge user in a single test batch (this is true as mentioned in the competition Data page).\n* if each question bundle is in a consecutive block in a test batch (despite the row ids may jumps).\n* if the question bundle for a single user in a test batch will always be at the end of this user's sequence.\n\nThe code could be used to build a full user history (training time + the previous batches in test time) for prediction. However, it is not optimized, since the current version is only for verifying some assumptions."},{"metadata":{"trusted":true},"cell_type":"code","source":"# installation with internet\n# !pip install datatable==0.11.0\n\n# installation without internet\n!pip install ../input/python-datatable/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport datatable as dt\nimport json\nfrom collections import defaultdict","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.076957,"end_time":"2020-10-08T17:19:41.189182","exception":false,"start_time":"2020-10-08T17:19:41.112225","status":"completed"},"tags":[]},"cell_type":"markdown","source":"## Reading data in jay format\nThe dataset can be saved in binary format and read back using datatable in less than a second!\n\nSee original notebook [RIIID with blazing fast RID](https://www.kaggle.com/rohanrao/riiid-with-blazing-fast-rid)"},{"metadata":{"trusted":true},"cell_type":"code","source":"dt.fread(\"/kaggle/input/riiid-test-answer-prediction/train.csv\").to_jay(\"train.jay\")\ntrain_dt = dt.fread('train.jay')\n\n# If `train.jay`, we don't need the above conversion - saving time / memory\n# train_dt = dt.fread('/kaggle/input/r3id-trainjay/train.jay')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Get unique questions and lectrues from training data"},{"metadata":{"trusted":true},"cell_type":"code","source":"unique_question_id_train = dt.unique(train_dt[dt.f.content_type_id == 0, 'content_id']).to_list()[0]\nwith open('unique_question_id_train.json', 'w', encoding='UTF-8') as fp:\n    json.dump(unique_question_id_train, fp, ensure_ascii=False)\nunique_question_id_train = set(unique_question_id_train)\n\nunique_lecture_id_train = dt.unique(train_dt[dt.f.content_type_id == 1, 'content_id']).to_list()[0]\nwith open('unique_lecture_id_train.json', 'w', encoding='UTF-8') as fp:\n    json.dump(unique_lecture_id_train, fp, ensure_ascii=False)\nunique_lecture_id_train = set(unique_lecture_id_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Collect questions / lectures from csv files"},{"metadata":{"trusted":true},"cell_type":"code","source":"question_df_from_public = pd.read_csv('/kaggle/input/r3id-info/questions.csv')\nlecture_df_from_public = pd.read_csv('/kaggle/input/r3id-info/lectures.csv')\n\nquestion_df_from_private = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\nlecture_df_from_private = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\n\nquestions_from_public = set(question_df_from_public['question_id'].values.tolist())\nlectures_from_public = set(lecture_df_from_public['lecture_id'].values.tolist())\n\nquestions_from_private = set(question_df_from_private['question_id'].values.tolist())\nlectures_from_private = set(lecture_df_from_private['lecture_id'].values.tolist())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(unique_question_id_train))\nprint(len(questions_from_public))\nprint(len(questions_from_private))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(unique_lecture_id_train))\nprint(len(lectures_from_public))\nprint(len(lectures_from_private))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert_ok = True\n\ntry:\n    \n    assert questions_from_private == questions_from_public\n    assert unique_question_id_train == questions_from_public\n    \n    assert lectures_from_private == lectures_from_public\n    assert unique_lecture_id_train.issubset(lectures_from_public)   \n            \nexcept AssertionError:\n        \n    assert_ok = False","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read precomputed user_id <-> indices map\n\nThis is a precomputed dictionary to record the starting and ending indices in `train.csv` for each user in it. The format is like\n\n\n    {\n        \"115\": [\n            0,\n            45\n        ],\n        \"124\": [\n            46,\n            75\n        ],\n        \"2746\": [\n            76,\n            95\n        ],\n        ...\n    }\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"with open('/kaggle/input/r3id-info/user_id_to_row_id_train.json', 'r', encoding='UTF-8') as fp:\n    _user_id_to_row_id_train = json.load(fp)\n    \n# Don't forget to change the key to type int!\nuser_id_to_row_id_train = {int(k): v for k, v in _user_id_to_row_id_train.items()}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_user_dt(user_id, _dt, user_id_to_row_id):\n    \"\"\"Get the partial `datatable.Frame` in `_dt` containing only `user_id`.\n    \n    Args:\n        user_id: `int`. It must in `user_id_to_row_id`, which should be precomputed from `_dt`.\n        _dt: `datatable.Frame`.\n        user_id_to_row_id: `dict`. See the above markdown cell for the format. \n    \"\"\"\n\n    assert user_id in user_id_to_row_id\n    \n    (start, end) = user_id_to_row_id[user_id]\n    user_dt = _dt[start:end + 1, :]\n\n    return user_dt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"attrs = [\n    'user_id',\n    'row_id',\n    'timestamp',\n    'content_id',\n    'content_type_id',\n    'task_container_id',\n    'user_answer',\n    'answered_correctly',\n    'prior_question_elapsed_time',\n    'prior_question_had_explanation'\n]\n\nclass User_Record:\n    \"\"\"\n    Sequences (per attribute) of records for a single user.\n    \"\"\"\n\n    def __init__(self, user_dt=None, user_id=None):\n        \"\"\"Exactly one argument should be `None`.\n        \n        Args:\n            user_dt: A `datatable.Frame` object for a single user.\n            user_id: int.\n        \"\"\"\n\n        assert (user_dt is not None and user_id is None) or (user_id is not None and user_dt is None)\n        \n        user_ids = user_dt[:, 'user_id'].to_list()[0] if user_dt is not None else []\n        \n        # single value - Each record is for a single user.\n        if user_dt is not None:\n            assert len(set(user_ids)) == 1\n        \n        self.user_id = user_ids[0] if user_dt is not None else user_id\n    \n        self.row_id = user_dt[:, 'row_id'].to_list()[0] if user_dt is not None else []\n        self.timestamp = user_dt[:, 'timestamp'].to_list()[0] if user_dt is not None else []\n        self.content_id = user_dt[:, 'content_id'].to_list()[0] if user_dt is not None else []\n        self.content_type_id = user_dt[:, 'content_type_id'].to_list()[0] if user_dt is not None else []\n        self.task_container_id = user_dt[:, 'task_container_id'].to_list()[0] if user_dt is not None else []\n        self.user_answer = user_dt[:, 'user_answer'].to_list()[0] if user_dt is not None else []\n        self.answered_correctly = user_dt[:, 'answered_correctly'].to_list()[0] if user_dt is not None else []\n        self.prior_question_elapsed_time = user_dt[:, 'prior_question_elapsed_time'].to_list()[0] if user_dt is not None else []\n        self.prior_question_had_explanation = user_dt[:, 'prior_question_had_explanation'].to_list()[0] if user_dt is not None else []\n\n        # make sure the timestamp is always in order.\n        assert self.timestamp == sorted(self.timestamp)\n            \n    def extend(self, other):\n        \"\"\"\n        Add the content of another recocrd `other` to the record `self`.\n        \"\"\"        \n        \n        assert (self.user_id == other.user_id)\n        \n        # The `timestamp` should be in order while adding new entries to existing record.\n        if len(self.timestamp) > 0:\n            assert self.timestamp[-1] <= other.timestamp[0]\n                \n        for k in attrs:\n            if k != 'user_id':\n                getattr(self, k).extend(getattr(other, k))\n\n    def update_answer_results(self, prior_correctnesses, prior_answers):\n        \"\"\"\n        Update the answers and their correctnesses in a record which was previously unknown in the last test batch.\n        \"\"\"\n        \n        # sanity check\n        assert len(prior_correctnesses) == len(prior_answers)\n        \n        assert len(self.answered_correctly) >= len(prior_correctnesses)\n        assert len(self.user_answer) >= len(prior_answers)\n        \n        # the places to be updated should contain only -1 (i.e. unknown results)\n        assert set(self.answered_correctly[-len(prior_correctnesses):]) == {-1}\n        assert set(self.user_answer[-len(prior_answers):]) == {-1}\n                \n        self.answered_correctly = self.answered_correctly[:-len(prior_correctnesses)] + prior_correctnesses\n        self.user_answer = self.user_answer[:-len(prior_answers)] + prior_answers\n\n    def toJSON(self):\n        \n        return json.dumps(self, default=lambda o: o.__dict__, sort_keys=False, indent=4)\n    \n    def __str__(self):\n        \n        return self.toJSON()   \n\nclass Record_Buffer:\n    \"\"\"\n    A dictionary like buffer to store and manage (i.e. updating) records.\n    \"\"\"\n    \n    def __init__(self, record_dict=None):\n        \"\"\"\n        `record_dict`: A `dict` mapping user ids (`str`) to their records (`User_Record`).\n        \"\"\"\n        \n        if record_dict is None:\n            self.buffer = {}\n        else:\n            self.buffer = record_dict\n           \n    def __contains__(self, x):\n        \n        return x in self.buffer\n        \n    def __getitem__(self, x):\n        \n        if x not in self.buffer:\n            raise KeyError(str(x))\n        \n        return self.buffer[x]\n    \n    def __len__(self):\n        \n        return len(self.buffer)\n    \n    def __del__(self):\n        \n        del self.buffer\n                \n    def update(self, record):\n        \"\"\"\n        Add a record to the buffer. If its user_id already exists, find and update the existing record.\n        \"\"\"\n\n        if record.user_id not in self.buffer:\n            self.buffer[record.user_id] = User_Record(user_id=record.user_id)\n            \n        self.buffer[record.user_id].extend(record)\n\n    def update_answer_results(self, user_id, prior_correctnesses, prior_answers):\n        \"\"\"\n        Update the answers and their correctnesses for a single user which was previously unknown in the last test batch.\n        \"\"\"\n\n        assert user_id in self.buffer        \n        record = self.buffer[user_id]\n\n        record.update_answer_results(prior_correctnesses, prior_answers)\n                \n    def toJSON(self):\n        \n        return json.dumps(self, default=lambda o: o.__dict__, sort_keys=False, indent=4)\n    \n    def __str__(self):\n        \n        return self.toJSON()\n        \n        \ndef convert_dt(_dt, test=False):\n    \"\"\"\n    Change column type, deal with NaN value. If it is a `datatable.Frame` from the test dataset,\n    change it to a format suitable for prediction.\n    \n    Args:\n        _dt: `datatable.Frame`, representing a block of the training dataset or a test batch given by `env.iter_test`.\n    \"\"\"\n\n    _dt[dt.f.prior_question_elapsed_time] = dt.float32\n    _dt[dt.f.prior_question_elapsed_time == None, 'prior_question_elapsed_time'] = -1.0\n    _dt[dt.f.prior_question_had_explanation] = dt.int8\n    _dt[dt.f.prior_question_had_explanation == None, 'prior_question_had_explanation'] = -1\n    _dt[dt.f.content_type_id] = dt.int8\n\n    if test:\n        \n        _dt['answered_correctly'] = -1\n        _dt['user_answer'] = -1\n\n        del _dt['prior_group_answers_correct']\n        del _dt['prior_group_responses']\n        # del _dt['group_num']\n        \n        user_ids = _dt['user_id'].to_list()[0]\n                \n        # All test questions must have been seen in training time.\n        assert set(_dt[dt.f.content_type_id == 0, 'content_id'].to_list()[0]).issubset(unique_question_id_train)\n\n        # All test lectures must have been seen in training time.\n        assert set(_dt[dt.f.content_type_id == 1, 'content_id'].to_list()[0]).issubset(unique_lecture_id_train)\n        \ndef convert_df_to_dt(df, test=False):\n    \"\"\"Convert a `pandas.DataFrame` to `datatable.Frame` with some extra processing.\n    \n    Args:\n        df: `pandas.DataFrame`, representing a block of the training dataset or a test batch given by `env.iter_test`.    \n    \"\"\"\n    \n    _dt = dt.Frame(df.astype({\"prior_question_had_explanation\": float}))\n\n    if test:\n                \n        prior_group_answers_correct = eval(_dt[0, 'prior_group_answers_correct'])\n        prior_group_responses = eval(_dt[0, 'prior_group_responses'])\n        \n        if prior_group_answers_correct is None:\n            prior_group_answers_correct = [-1] * _dt.nrows\n        if prior_group_responses is None:\n            prior_group_responses = [-1] * _dt.nrows\n\n        assert type(prior_group_answers_correct) == list\n        assert type(prior_group_responses) == list\n\n    convert_dt(_dt, test=test)\n    \n    if test:\n        return _dt, prior_group_answers_correct, prior_group_responses\n    else:\n        return _dt\n\n    \ndef combine_user_record(record_1, record_2):\n    \"\"\"\n    Returns:\n        A new record that combines the two `User_Record` objects `record_1` and  `record_2`.\n        The arguments are not modified.\n    \"\"\"\n\n    assert record_1.user_id == record_2.user_id\n\n    record = User_Record(user_id=record_1.user_id)\n\n    record.extend(record_1)\n    record.extend(record_2)\n\n    return record    \n    \n\nclass Prediction_Manager:\n    \"\"\"\n    See `update()` for the description.\n    \"\"\"\n    \n    def __init__(self, train_dt, user_id_to_row_id_train, max_train_buffer_size=30000):\n        \n        self.train_dt = train_dt\n        self.user_id_to_row_id_train = user_id_to_row_id_train\n        \n        self.train_record_buffer = Record_Buffer()\n        self.test_record_buffer = Record_Buffer()\n        \n        self.current_batch_row_ids = None        \n        self.current_batch_users = None\n        \n        # Used to avoid memory error - not sure if it is necessary.\n        self.max_train_buffer_size = max_train_buffer_size\n        \n    def reset_train_record_buffer(self):\n        \"\"\"\n        \"\"\"\n        \n        del self.train_record_buffer\n        self.train_record_buffer = Record_Buffer()\n        \n    def update_batch_users(self, user_ids):\n        \"\"\"\n        Store the user ids in the current test batch.\n        \"\"\"\n        \n        self.current_batch_users = user_ids\n\n    def update_answer_results(self, prev_batch_users, prior_group_answers_correct, prior_group_responses):\n        \"\"\"\n        When we get a new test batch, we also get the `prior_group_answers_correct` and `prior_group_responses`.\n        We use these information to update the `answered_correctly` and `user_anser` fields of the records of\n        the users in the previous test batch.\n        \"\"\"\n        \n        # sanity check: try to make sure the answer results are for `prev_batch_users` by verifying their lengths.\n        assert len(prev_batch_users) == len(prior_group_answers_correct)\n        assert len(prior_group_answers_correct) == len(prior_group_responses)\n        \n        d1 = defaultdict(list)\n        d2 = defaultdict(list)\n        for user_id, prior_ans_correct, prior_ans in zip(prev_batch_users, prior_group_answers_correct, prior_group_responses):\n            d1[user_id].append(prior_ans_correct)\n            d2[user_id].append(prior_ans)\n            \n        for user_id in d1:\n            self.test_record_buffer.update_answer_results(user_id, d1[user_id], d2[user_id])\n\n    def update(self, test_df):\n        \"\"\"\n        For a test batch `test_df` (`pandas.DataFrame`) given by `env.iter_test`, this method performs:\n            1. update the `answered_correctly` and `user_anser` information in the previous test batch\n            2. update the user records (only in the test time) by appending the information in the current batch\n            3. get the user records in the training time\n            4. combine the user records in the training time and test time - so we have a full history and the current batch to predict\n        \"\"\"\n        \n        # conversion\n        test_dt, prior_group_answers_correct, prior_group_responses = convert_df_to_dt(test_df, test=True)\n        \n        user_ids = test_dt['user_id'].to_list()[0]\n        prev_batch_users = self.current_batch_users\n        if prev_batch_users is not None:\n            self.update_answer_results(prev_batch_users, prior_group_answers_correct, prior_group_responses)\n        self.update_batch_users(user_ids)\n\n        row_ids = test_dt['row_id'].to_list()[0]\n        prev_batch_row_ids = self.current_batch_row_ids\n        \n        debug_test_batch_row_ids(row_ids, prev_batch_row_ids)\n        \n        self.current_batch_row_ids = row_ids\n        \n        # To `User_Record`. Here, each record contains exactly one user interaction.\n        # But the same user might appear several times in the list.\n        record_batch = [User_Record(user_dt=test_dt[idx, :]) for idx in range(test_dt.nrows)]\n        debug_record_batch(record_batch)\n                        \n        # update test buffer - add info about the new test batch\n        for record in record_batch:\n            self.test_record_buffer.update(record)\n\n        # get the updated record from test buffer for each user in `user_ids`.\n        # The same user might appear several times in the list, however, they get the same user history (test time) sequence.\n        test_record_batch = [self.test_record_buffer[x] for x in user_ids]\n        \n        # get the record from train_dt or train record buffer for each user in `user_ids`\n        training_record_batch = [self.get_training_record(x) for x in user_ids]\n        \n        # obtain the full history (in training time + the previous batches in test time) and the current batch to predict\n        predict_record_batch = [combine_user_record(x, y) for x, y in zip(training_record_batch, test_record_batch)]\n    \n        if len(self.train_record_buffer) >= self.max_train_buffer_size:\n            self.reset_train_record_buffer()\n\n    def get_training_record(self, user_id):\n        \"\"\"\n        Get the training time history of the user with `user_id`.\n        \"\"\"\n                \n        if user_id in self.train_record_buffer:\n            return self.train_record_buffer[user_id]\n\n        if user_id in self.user_id_to_row_id_train:\n            \n            user_dt = get_user_dt(user_id, self.train_dt, self.user_id_to_row_id_train)\n                    \n            assert len(user_dt) > 0\n            \n            convert_dt(user_dt, test=False)\n            \n            user_record = User_Record(user_dt=user_dt)\n            self.train_record_buffer.update(user_record)\n            \n            return self.train_record_buffer[user_id]\n\n        else:\n\n            user_record = User_Record(user_id=user_id)\n            self.train_record_buffer.update(user_record)\n            \n            return self.train_record_buffer[user_id]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"pm = Prediction_Manager(train_dt=train_dt, user_id_to_row_id_train=user_id_to_row_id_train)\npm","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def debug_test_batch_row_ids(row_ids, prev_batch_row_ids):\n    \"\"\"Verify the properties of row ids in a single and across test batch(es).\n    \n    Args:\n        row_ids: The row ids in a batch during the test time given by `env.iter_test()`\n        prev_batch_row_ids: The row ids in the batch just before the batch of `row_ids` during the test time given by `env.iter_test()`\n    \"\"\"\n\n    # sanity check\n    # row ids must be distinct\n    assert len(set(row_ids)) == len(row_ids)\n    \n    # row ids must be in sorted order in a batch\n    assert row_ids == sorted(row_ids)\n    \n    # row ids must be in sorted order across all batch during the test time.\n    if prev_batch_row_ids is not None:\n        assert row_ids[0] > prev_batch_row_ids[-1]\n\n\ndef debug_record_batch(record_batch):\n    \"\"\"Verify the properties of the question bundle for a single user in a test batch.\n    \n    Args:\n        record_batch: A list. Each element (`User_Record`) should contain only one record in a single timestamp.\n    \"\"\"\n\n    _tmp = defaultdict(list)\n    for record in record_batch:\n        # each record here contains only one entry.\n        assert len(record.row_id) == 1\n        _tmp[record.user_id].append(record)\n\n    for user_id, records in _tmp.items():\n\n        task_container_ids_for_questions = [x.task_container_id[0] for x in records if x.content_type_id[0] == 0]\n\n        # If there is any question for a user\n        if len(task_container_ids_for_questions) > 0:\n\n            # There must be exactly one question bundle.\n            # This is `True`.\n            assert len(set(task_container_ids_for_questions)) == 1\n\n            row_ids_for_questions = [x.row_id[0] for x in records if x.content_type_id[0] == 0]\n            \n            # This is `False`: the question bundle must be in a consecutive block with continuous row ids.\n            # assert row_ids_for_questions == list(range(row_ids_for_questions[0], row_ids_for_questions[-1] + 1))\n\n            records_in_between = [x for x in records if row_ids_for_questions[0] <= x.row_id[0] <= row_ids_for_questions[-1]]\n            row_ids_in_between = [x.row_id[0] for x in records_in_between]\n            \n            # This is `True`: the question bundle must be in a consecutive block (but the row ids may jump).\n            assert row_ids_for_questions == row_ids_in_between\n\n            # The question bundle must be at the end of the sequence (for a single user) in a test batch.\n            assert row_ids_for_questions == [x.row_id[0] for x in records][-len(row_ids_for_questions):]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"assert_ok","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# if some assertion fails, no submission.csv is generated, and we get submission scoring error.\nif assert_ok:\n\n    import riiideducation\n    env = riiideducation.make_env()\n    iter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_history = []\n\nn_test_batch = 0\nfor test_df, _ in iter_test:\n    \n    n_test_batch += 1\n        \n    try:\n        \n        # If some assertion fails, no submission.csv is generated, and we get submission scoring error.\n        # assert 1 == 0\n        \n        _question_ids = set(test_df[test_df['content_type_id'] == 0]['content_id'].values.tolist())\n        _lecture_ids = set(test_df[test_df['content_type_id'] == 1]['content_id'].values.tolist())        \n        \n        # All test questions must have been seen in training time.\n        assert _question_ids.issubset(unique_question_id_train)\n        \n        # All test lectures must have been seen in training time.\n        assert _lecture_ids.issubset(unique_lecture_id_train)\n        \n        pm.update(test_df)\n        \n        # Save a few buffer status to check things are expected.\n        if n_test_batch <= 4:\n            test_history.append(str(pm.test_record_buffer))\n        \n    except AssertionError as e:\n        print('some assertions are wrong, breaking the loop')\n        break\n    \n    test_df['answered_correctly'] = 0.5\n    env.predict(test_df.loc[test_df['content_type_id'] == 0, ['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Save some status to outputs - so you can verify"},{"metadata":{"trusted":true},"cell_type":"code","source":"for idx in range(4):\n    with open(f'test_batch_{idx}.json', 'w', encoding='UTF-8') as fp:\n        fp.write(test_history[idx])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if n_test_batch <= 4:\n    with open('train_record_buffer.json', 'w', encoding='UTF-8') as fp:\n        fp.write(str(pm.train_record_buffer))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\n\n* It seems that the answers to the questions mentioned in the top cell of this notebook are all `Yes`.\n* I hope there is no missing part or logical error in this notebook.\n* I encourage you to verify by yourself.\n* Any feedback is appreciated.\n* I don't take any responsibility for any error in this notebook."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}