{"cells":[{"metadata":{},"cell_type":"markdown","source":"As the train data is complete, and interactions of users are arranged chronologically. Unlike totally shuffle, I thought it would be one way to split train/valid data userwise as follows:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load train.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dtypes_dict = {\n    \"row_id\": \"int64\",\n    \"timestamp\": \"int64\",\n    \"user_id\": \"int32\",\n    \"content_id\": \"int16\",\n    \"content_type_id\": \"int8\",\n    #\"task_container_id\": \"int16\",\n    #\"user_answer\": \"int8\",\n    \"answered_correctly\": \"int8\",\n    #\"prior_question_elapsed_time\": \"float32\", \n    #\"prior_question_had_explanation\": \"boolean\"\n}\n\ndata = pd.read_csv(\"../input/riiid-test-answer-prediction/train.csv\",\n                         nrows=10**5, #how many rows to read\n                         usecols = train_dtypes_dict.keys(),\n                         dtype=train_dtypes_dict,\n                        )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Choose question type interaction only if you want."},{"metadata":{"trusted":true},"cell_type":"code","source":"data_q = data[data.content_type_id == 0]\ndata_q.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Get users and their interaction counts"},{"metadata":{"trusted":true},"cell_type":"code","source":"users_i_counts = data_q.user_id.value_counts()\nuser_list = users_i_counts.keys()\nprint(user_list[:5])\nprint(users_i_counts.head())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Choose how many users you want to train depending on their interaction counts. For example, if you want interaction counts greater than 6000, you can choose the first two users in user_list. And valid_num below controls how many valid data you want."},{"metadata":{"trusted":true},"cell_type":"code","source":"user_num = 2 #control how many user to train\nvalid_num = 2 #control how many valid data to keep\ntrain_data = pd.concat([data_q[data_q.user_id==u][:-valid_num] for u in user_list[:user_num]], axis=0)\nvalid_data = pd.concat([data_q[data_q.user_id==u][-valid_num:] for u in user_list[:user_num]], axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"valid_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Check where different users separate in train_data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data[7472:7476]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So as you can see, valid data always come after train data. It's a time-aware userwise train-valid-split method."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}