{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"#### In this notebook, I try to answer two important questions: \n- Is the dataset properly sorted? \n- How are training data and test data overlapped?\n\nGiven the data size, it is challenging to even read the entire test data. Using RAPIDS cudf, we can process the data and get our answer in minutes. If you check the notebook log, you can find the entire notebook running time. However on a computer with faster disk, it can be done within 10 mins!","metadata":{}},{"cell_type":"markdown","source":"#### TLDR\n- Yes, the dataset is properly sorted by `customer_ID` then `S_2` (timestamp) so the records of customer is continuous in the csv in the order of timing.\n- No, there are no overlap between training and test data in terms of `customer_ID` and `timestamps`. Specifically, no `customer_ID` are found in common. training data is from `2017-03-01` to `2018-03-01`. test data is from `2018-04-01` to `2019-10-01` ","metadata":{}},{"cell_type":"code","source":"import cudf\nimport cuml\nimport cupy\nfrom tqdm import tqdm","metadata":{"execution":{"iopub.status.busy":"2022-05-26T03:38:06.390302Z","iopub.execute_input":"2022-05-26T03:38:06.390787Z","iopub.status.idle":"2022-05-26T03:38:11.705046Z","shell.execute_reply.started":"2022-05-26T03:38:06.390738Z","shell.execute_reply":"2022-05-26T03:38:11.703892Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\npath = '../input/amex-default-prediction'\nos.listdir(path)","metadata":{"execution":{"iopub.status.busy":"2022-05-26T03:38:37.799293Z","iopub.execute_input":"2022-05-26T03:38:37.799695Z","iopub.status.idle":"2022-05-26T03:38:37.809247Z","shell.execute_reply.started":"2022-05-26T03:38:37.799662Z","shell.execute_reply":"2022-05-26T03:38:37.807194Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntest = cudf.read_csv(f'{path}/test_data.csv', nrows=10)\ncols = test.columns\ntest.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Read the file in chunks using cudf","metadata":{}},{"cell_type":"markdown","source":"You might wonder why so much fuss reading the file. It turns out that `read_csv` is less than ideal due to the memory overhead. The peak memory usage is so high that we have to read in chunks even when we only use two columns in this case.","metadata":{}},{"cell_type":"code","source":"%%time\n\ndef read_csv_iter(path, total_rows, chunk_rows, all_cols, use_cols):\n    sofar = 1\n    ts = []\n    for i in tqdm(range(total_rows//chunk_rows+1)):\n        nr = total_rows - sofar\n        nr = min(chunk_rows, nr)\n        t = cudf.read_csv(path, header=None, names=cols, nrows = nr, skiprows=sofar, usecols=use_cols)\n        sofar += nr\n        ts.append(t)\n    return cudf.concat(ts)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\ntest_rows = 11363762\n\n# please note that the total_rows here includes the header row of the csv\ntest = read_csv_iter(f'{path}/test_data.csv', total_rows=test_rows+1, \n                     chunk_rows=2_000_000, all_cols=cols, \n                     use_cols=['customer_ID','S_2'])\nprint(test.shape)\ntest.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\ntrain_rows = 5531451\n\n# please note that the total_rows here includes the header row of the csv\ntrain = read_csv_iter(f'{path}/train_data.csv', total_rows=train_rows+1, \n                     chunk_rows=2_000_000, all_cols=cols, \n                     use_cols=['customer_ID','S_2'])\nprint(train.shape)\ntrain.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Check if the data is sorted","metadata":{}},{"cell_type":"markdown","source":"Why do we care about if the data is sorted?\n\nDue to the datasize, it is very likely that the data and model can't fit in the system or GPU memory. Hence it is desirable to process and train the data in a streaming/batching fashion. If the data is properly sorted, the records of one customer are in continuous rows of the csv file, which makes batching much easier. ","metadata":{}},{"cell_type":"markdown","source":"So how do we check? We simply sort the dataframe with the desired way and compare the row index. if they are equal, it proves that the data is already sorted in that way.","metadata":{}},{"cell_type":"code","source":"def check_sorted(df):    \n    df['row_id'] = cupy.arange(df.shape[0])\n    df['cid'],_ = df.customer_ID.factorize()\n    df_sort = df.sort_values(['cid','S_2'])\n    return (df['row_id'] == df_sort['row_id']).all()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\ncheck_sorted(test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\ncheck_sorted(train)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Great! Both train and test data are properly sorted.","metadata":{}},{"cell_type":"markdown","source":"#### Check if train and test are overlapped\n\nTo setup a robust validation scheme, we need to understand how the train and test are split. Specifically we want to check `customer_ID` and `S_2` (timestamp).","metadata":{}},{"cell_type":"markdown","source":"First, let's check if the timestamps are overlapped between train and test","metadata":{}},{"cell_type":"code","source":"%%time\ntrain['S_2'] = cudf.to_datetime(train['S_2'], format='%Y-%m-%d')\ntest['S_2'] = cudf.to_datetime(test['S_2'], format='%Y-%m-%d')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntrain['S_2'].min(), train['S_2'].max()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\ntest['S_2'].min(), test['S_2'].max()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Nope, test is the future and train is the past. So we have a forecast problem","metadata":{}},{"cell_type":"code","source":"%%time\ntrain_ids = train.customer_ID.unique()\ntest_ids = test.customer_ID.unique()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mask = train_ids.isin(test_ids)\nmask.sum()/train_ids.shape[0]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mask = test_ids.isin(train_ids)\nmask.sum()/test_ids.shape[0]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"No shared customer IDs. Very challenging!","metadata":{}}]}