{"cells":[{"metadata":{"_uuid":"416f6fcffa597fc1e7522af57f2a67dc0061496a"},"cell_type":"markdown","source":"Template to generate sets of training data (\"<code>training</code>\") and validation data (\"<code>validation</code>\") that parallel the contents of the full training set and the test set, shifted by 24 hours.  Based on [this kernel](https://www.kaggle.com/konradb/validation-set) from Konrad Banachewicz, but taking into account what we learned in [this thread](https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51877) from Alexander Firsov (including comments by James Trotman and me)."},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true,"collapsed":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd","execution_count":1,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":true},"cell_type":"code","source":"columns = ['ip', 'app', 'device', 'os', 'channel', 'click_time', 'is_attributed']\ndtypes = {\n        'ip'            : 'uint32',\n        'app'           : 'uint16',\n        'device'        : 'uint16',\n        'os'            : 'uint16',\n        'channel'       : 'uint16',\n        'is_attributed' : 'uint8',\n        }","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"bbbed4ad3b3086e93fedd143c7d535f563db719f"},"cell_type":"markdown","source":"## Training Data"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"31ca3f78b465f6234cfbc5f7c096b97719a11b47"},"cell_type":"code","source":"training = pd.read_csv( \"../input/train.csv\", \n                        nrows=122071523, \n                        usecols=columns, \n                        dtype=dtypes)","execution_count":10,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5dba036deeddd7fd725684eb61922528ce6ccb14"},"cell_type":"code","source":"training.tail()","execution_count":11,"outputs":[]},{"metadata":{"_uuid":"b918a1ea79993e20f7a18ef0d1a501ac25e7665b"},"cell_type":"markdown","source":"Note that the full training set ends at 16:00:00 on 2017-11-09, so to parallel what would be available to predict the test set, I truncate the training data one day earlier."},{"metadata":{"_uuid":"8264c91845564c66bd3f9dc5342592fd6be5473d"},"cell_type":"markdown","source":"## Validation Data"},{"metadata":{"_uuid":"925c6ef1553de6d4dc846e0fe6e399b4b589c8ac"},"cell_type":"markdown","source":"There are 3 separate chunks of test data, each extending for 2 hours and 1 second.  Here I read in each analogous chunk separately and concatenate them."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"208b4ab188af0f018b94de9399fa1d34795ceee3"},"cell_type":"code","source":"valid1 = pd.read_csv( \"../input/train.csv\", \n                      skiprows=range(1,144708153), \n                      nrows=7705357, \n                      usecols=columns, \n                      dtype=dtypes)","execution_count":12,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cbcedf4199b4717dce5a7f6dcadde1bfa9edc409"},"cell_type":"code","source":"valid1.head()","execution_count":13,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1cc1719f949be349685e53b8d094f6f11af875ff"},"cell_type":"code","source":"valid1.tail()","execution_count":14,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"46d0a43737cd4c90fa401865e10fd2bbe1d98b19"},"cell_type":"code","source":"valid2 = pd.read_csv( \"../input/train.csv\", \n                      skiprows=range(1,161974466), \n                      nrows=6291379, \n                      usecols=columns, \n                      dtype=dtypes)","execution_count":18,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"962c3d5b1444cd98897e2600e94eece017a927c6"},"cell_type":"code","source":"valid2.head()","execution_count":19,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5d22d97f32674fa910b65eb210391fefae1eaecd"},"cell_type":"code","source":"valid2.tail()","execution_count":20,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"344edd67a043d89b3c9d16d4bf42871e7281ca2a"},"cell_type":"code","source":"valid2 = pd.concat([valid1, valid2])","execution_count":21,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e5ce904d42576c57deb630cd30c96b15288926b0"},"cell_type":"code","source":"valid2.head()","execution_count":22,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2d5891c81a98a5531c28092610d0407eb70f675b"},"cell_type":"code","source":"valid2.tail()","execution_count":23,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7989716358fc180198a2f3310da8419dc9255102"},"cell_type":"code","source":"del valid1\nimport gc\ngc.collect()","execution_count":24,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"a0e5e814922df88f0739fd4cedc4d4baff0beb1f"},"cell_type":"code","source":"valid3 = pd.read_csv( \"../input/train.csv\", \n                      skiprows=range(1,174976527), \n                      nrows=6901686, \n                      usecols=columns, \n                      dtype=dtypes)","execution_count":25,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19c5111e063dbd0fc7d92378730f7e78d4340b19"},"cell_type":"code","source":"valid3.head()","execution_count":26,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b523aea43a0efed4478f8d2c61cbcb5d82d07c10"},"cell_type":"code","source":"valid3.tail()","execution_count":27,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"9d85ae357c33b5083c00bc43337ec931fdecf9be"},"cell_type":"code","source":"valid3 = pd.concat([valid2,valid3])","execution_count":28,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8f8147114504057e57cdda9fd4e0b00ab27dbdf9"},"cell_type":"code","source":"valid3.head()","execution_count":30,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04ab27c8953a9a6aa7e17a084e149db4133163aa"},"cell_type":"code","source":"valid3.tail()","execution_count":31,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fd693b921a74f9cf7b7bde4be8e2c660962795d6"},"cell_type":"code","source":"del valid2\ngc.collect()","execution_count":29,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a5ec7b54b7fc9e4b7274f6956c9895b509544b33"},"cell_type":"code","source":"validation = valid3\ndel valid3\ngc.collect()\nvalidation.head()","execution_count":32,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6aeb742bb46e87da4956e5d4618f971ee265e775"},"cell_type":"code","source":"validation.tail()","execution_count":33,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"2e03631a3ae3bf0ba439bf8a30e38d5785a03729"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}