{"cells":[{"metadata":{},"cell_type":"markdown","source":"Riid AIEd Challenge 2020 - Part II new
\n\nDue to memory/time restrictions in this competition, work is divided into several parts (kernels):\n\n - Part I - Memory optimization
\n - Part II - Splitting data
\n - Part III - Feature engineering
\n - Part IV - Training and validation
\n - Part V - Prediction and submission
\n
\n\nThis is Part II. In this part I'll \n\n - Divide the competition data into two parts. The first part, which I'll call
past_data will be used to create features for training. The second part will be the test dataset, and will be designed to be similar to the competition test set. \n - A small part of
past_data will be used to create folds of training/validation data. \n - Save everything in pickle format to be used by the next phase.
\n
"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Imports\n\nimport os\nimport pandas as pd\nimport numpy as np\nimport pickle\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define directories used\n\nDATA_DIR = '/kaggle/input/riiid-test-answer-prediction'\nPART_I_OUTPUT_DIR = '/kaggle/input/riiid-aied-part-i'\nWORKING_DIR = '/kaggle/working'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Read competition data\ncompetition_data = pd.read_pickle(os.path.join(PART_I_OUTPUT_DIR, 'competition_data.pkl'))\ncompetition_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"competition_data.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sort competition data by date
\n\nFor this, I use this notebook by tito."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Create a dataframe with max timestamp per user\ntimestamps_df = competition_data.groupby('user_id')['timestamp'].max().reset_index()\ntimestamps_df.columns = ['user_id', 'max_timestamp']\n\n# Calculate maximum timestamp of all users\nMAX_TIMESTAMP = timestamps_df['max_timestamp'].max()\n\nprint(f'max timestamp = {MAX_TIMESTAMP}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Set start of each user's interactions at some random point between 0 and MAX_TIMESTAMP - user's max timestamp\n\ndef random_start(max_timestamp):\n return np.random.randint(0, high=MAX_TIMESTAMP - max_timestamp + 1)\n\ntimestamps_df['random_start'] = timestamps_df.max_timestamp.apply(random_start)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Join competition data with this new information about users\n\ncompetition_data = competition_data.merge(timestamps_df, on='user_id', how='left')\n\ndel timestamps_df\n_ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Calculate the virtual timestamp of every interaction\n\ncompetition_data['virtual_timestamp'] = competition_data['random_start'] + competition_data['timestamp']\n\n# Free memory\ncompetition_data.drop(columns=['max_timestamp', 'random_start'], inplace=True)\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Sort the competition_data by virtual_timestamp\n\ncompetition_data = competition_data.sort_values(by='virtual_timestamp', ascending=True).reset_index(drop=True)\n\n_ = gc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Split data into past data and test data
"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create test set as the last 100K rows\ntest = competition_data.iloc[-100000:].copy()\ntest.reset_index()\n\n# Save test data\ntest.to_pickle(os.path.join(WORKING_DIR, 'test.pkl'))\n\n# Create past_data and save it (needed for feature creation)\npast_data = competition_data.drop(index=test.index).reset_index(drop=True)\npast_data.to_pickle(os.path.join(WORKING_DIR, 'past_data.pkl'))\n\n# Create train/validation data as the last 15M rows of past_data\ntrain_val = past_data.iloc[-10000000:].copy()\ntrain_val.reset_index()\n\ndel past_data\ndel test\ndel competition_data\n\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Create train/validation folds
"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Split users in train_val into 4 groups, each one of which will be used to create a cross-validation fold.\n\nNUM_FOLDS = 4\n\nnp.random.seed(42)\n\nuser_ids = train_val.user_id.unique()\nnp.random.shuffle(user_ids)\nuser_groups = np.array_split(user_ids, NUM_FOLDS)\n\nfor i, group in enumerate(user_groups):\n train_val_fold = train_val.loc[train_val.user_id.isin(user_groups[i])]\n \n # The last 500K rows are for the validation set and the rest for the training set\n train = train_val_fold.iloc[:-500000].reset_index(drop=True)\n val = train_val_fold.iloc[-500000:].reset_index(drop=True)\n \n print(f'train_{i}.shape={train.shape}, val_{i}.shape={val.shape}')\n \n # Save everything\n train.to_pickle(os.path.join(WORKING_DIR, f'train_{i}.pkl'))\n val.to_pickle(os.path.join(WORKING_DIR, f'val_{i}.pkl'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's all folks"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}