Creating a good test set for this competition is challenging especially if you want to use target encoding. You can see this discussion and the referenced blog for more details.
\nIn this notebook I will start explaining the strategy that I used to create a test dataset. Then I will create a generator class that will return batches of test data in a similar way than the official riiideducation package. This class can be used to simulate a real submission that could include incremental learning.
To have a test dataset can help us to:
\nI start by creating a toy dataset with just three columns: user_id, timestamp, answered_correctly and user_answer"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.DataFrame(\n [[0, 0, 1, 0], \n [0, 2, 1, 1], \n [0, 3, 0, 2], \n [0, 40, 0, 0], \n [0, 45, 0, 3], \n [1, 0, 1, 1], \n [1, 8, 0, 0], \n [1, 9, 1, 1], \n [1, 10, 1, 3], \n [2, 0, 1, 2]],\n columns=['user_id', 'timestamp', 'answered_correctly', 'user_answer'])\ndf","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is a visual representation of the data:\n\nThe problem is that the timestamp starts by 0 or every user_id. We don't have any idea on how those timestamps for different users are related.\n
If we create a test set just taking the last interactions for each users, the users with only one interaction will always be in the test dataset and this is not good. If we take the last interactions globally, the users with few interactions will always be in the train dataset and this is not good either.
\nJust taking random interaction for the test set is also a bad idea especially if we are using target encoding.
\nTo avoid those concerns, I will create a random starting timestamp for each user_id that is somewhere between 0 and max(timestamp)-max(timestamp_for_the_user). Then I take the last interactions to the test dataset and the first ones for training. With the same function we can also sepparate the data in several folds if this is what we want.
train_idx and test_idx are the indexes of the provided dataset ordered by the new timestamp. It is very important to keep the test indexes for the next step where we will generate batches of test data based on those indexes."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_idx,test_idx)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = df.iloc[train_idx]\ntrain_df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = df.iloc[test_idx]\ntest_df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is how the dataframe looks like after randomizing the start timestamp:\n\nIf you are interested in split the data in several folds, this can be done with the same function like that:"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(49)\ndataset_split(df, q=[0.333,0.667], print_timestamps=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2- Test Generator\n
Once we have split the data in train/test sets, we can train a model using the train set and evaluate it using the test set.
\nTo evaluate the model, we must use a generator that will yield batches of data to the model. We can retrain the model (or a part of it) after getting each batch. This is not straightforward as we need to assign the data in prior_group_answers_correct and prior_group_responses to the correct rows. The class TestGenerator does this job.
We create the generator with two parameters, the dataframe and an array with the size of the batches. There is nothing (or I didn't find anything) in the documentation explaining what is the distribution of those batches, so the function will receive the mean size of the batches and the standard distribution. By default I set that to [1000, 100], which means that almost all the batches will have between 700 and 1300 instances.
After the creation, we can use iter_test to get a batch of test data and predict to append a new prediction. This is similar to what the riiideducation package does. When all the instances of the dataframe have been consumed, the predict function prints the score
I will apply the previous functions to the real train dataset to create a train/test split that I will save to disk in feather format. Other notebooks will be able to use those datasets.
"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain = pd.read_feather('../input/riiid-feather-dataset/train.feather')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(49)\ntrain_idx, test_idx = dataset_split(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = train.iloc[test_idx].reset_index(drop=True)\ntest","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train = train.iloc[train_idx].sort_index().reset_index(drop=True)\ntrain","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I save the outputs in feather format so they will be much faster to read (than csv)."},{"metadata":{"trusted":true},"cell_type":"code","source":"train.to_feather(\"train_df.feather\")\ntest.to_feather(\"test_df.feather\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You can read those datasets with:
\n\ntrain_df = pd.read_feather('../input/riiid-creating-a-test-dataset/train_df.feather')\ntest_df = pd.read_feather('../input/riiid-creating-a-test-dataset/test_df.feather')\n
In this notebook there is a real example on how to use the DataGenerator class
"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}