{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sampling the training dataset based on Customer_ID\n\nThe training dataset is really big, too big to fit into memory. \n\nWe want to be able to create a smaller sample dataset that's easier to work with and do EDA on.\n\nHowever, it's not a good idea to just randomly sample records since the dataset contains timeseries data with multiple rows for each customer.\n\nInstead, we'll randomly sample a set of customer_ID's, then create a sample dataset that contains all the rows for each customer_ID.\nWe'll use the `chunk_size` parameter of the `read_csv` function so that we don't run out of memory.\n\nIn this Notebook I'm only going to create sample training data, but the same approach can be used for the test data.","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-07T22:47:03.821696Z","iopub.execute_input":"2022-07-07T22:47:03.822179Z","iopub.status.idle":"2022-07-07T22:47:03.829151Z","shell.execute_reply.started":"2022-07-07T22:47:03.822130Z","shell.execute_reply":"2022-07-07T22:47:03.827159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Important defintions\n\nHere we set how many customer_ID's to sample and the size of the chunks we'll read from the training data.","metadata":{}},{"cell_type":"code","source":"SAMPLE_CUSTOMER_COUNT = 10000\nCHUNK_SIZE = 1000000 #what size chunks to read the data from disk","metadata":{"execution":{"iopub.status.busy":"2022-07-07T22:27:38.963340Z","iopub.execute_input":"2022-07-07T22:27:38.963681Z","iopub.status.idle":"2022-07-07T22:27:38.976812Z","shell.execute_reply.started":"2022-07-07T22:27:38.963653Z","shell.execute_reply":"2022-07-07T22:27:38.975501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Set paths \nINPUT_DIR = '/kaggle/input/amex-default-prediction'\nWORKING_DIR = '/kaggle/working'\nTRAIN_DATA = f'{INPUT_DIR}/train_data.csv'\nSAMPLE_CUSTOMERS =  f'{WORKING_DIR}/sample_customers.csv'\nSAMPLE_TRAIN_DATA = f'{WORKING_DIR}/sample_train_data.csv'","metadata":{"execution":{"iopub.status.busy":"2022-07-07T22:27:38.978576Z","iopub.execute_input":"2022-07-07T22:27:38.978929Z","iopub.status.idle":"2022-07-07T22:27:38.989259Z","shell.execute_reply.started":"2022-07-07T22:27:38.978896Z","shell.execute_reply":"2022-07-07T22:27:38.988270Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Grab a random sample of customers\n\nThe `shuf` shell command is good enough to do this. The number of customers is defined by `SAMPLE_CUSTOMER_COUNT`","metadata":{}},{"cell_type":"code","source":"!shuf -n $SAMPLE_CUSTOMER_COUNT $TRAIN_DATA | cut -d ',' -f1 | uniq > $SAMPLE_CUSTOMERS","metadata":{"execution":{"iopub.status.busy":"2022-07-07T22:27:38.990695Z","iopub.execute_input":"2022-07-07T22:27:38.991459Z","iopub.status.idle":"2022-07-07T22:30:06.983900Z","shell.execute_reply.started":"2022-07-07T22:27:38.991420Z","shell.execute_reply":"2022-07-07T22:30:06.981202Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Read the sample customer_ID's for looking up ","metadata":{}},{"cell_type":"code","source":"df_customer_ids = pd.read_csv(SAMPLE_CUSTOMERS, header=None, names=['customer_ID'])","metadata":{"execution":{"iopub.status.busy":"2022-07-07T22:30:06.987493Z","iopub.execute_input":"2022-07-07T22:30:06.988003Z","iopub.status.idle":"2022-07-07T22:30:07.032151Z","shell.execute_reply.started":"2022-07-07T22:30:06.987950Z","shell.execute_reply":"2022-07-07T22:30:07.031109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create the sample data\n\nWe read the data into memory in chunks. For each chunk we join in our sample customer_ID's to only grab rows associated with our sample customers, which we can append to our output file.","metadata":{}},{"cell_type":"code","source":"#if the sample output file already exists, delete it since we're creating a new one\nif os.path.exists(SAMPLE_TRAIN_DATA):\n    os.remove(SAMPLE_TRAIN_DATA)\n    \ndf_chunks = pd.read_csv(TRAIN_DATA, chunksize=CHUNK_SIZE)\n\nheader = True #write header on first chunk\n\nfor idx, df_chunk in enumerate(df_chunks):\n    merged = pd.merge(df_chunk, df_customer_ids, on='customer_ID', how='inner')\n    merged.to_csv(SAMPLE_TRAIN_DATA, header=header, index=False, mode='a')\n    header = False\n    print(f'Wrote out matches in chunk {idx}')","metadata":{"execution":{"iopub.status.busy":"2022-07-07T22:30:07.034020Z","iopub.execute_input":"2022-07-07T22:30:07.034407Z","iopub.status.idle":"2022-07-07T22:37:20.132410Z","shell.execute_reply.started":"2022-07-07T22:30:07.034370Z","shell.execute_reply":"2022-07-07T22:37:20.131088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Check our sample data\n\nNow we have a smaller dataset that can be read into memory","metadata":{}},{"cell_type":"code","source":"df_sample = pd.read_csv(SAMPLE_TRAIN_DATA)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T22:44:03.262151Z","iopub.execute_input":"2022-07-07T22:44:03.262652Z","iopub.status.idle":"2022-07-07T22:44:08.811886Z","shell.execute_reply.started":"2022-07-07T22:44:03.262605Z","shell.execute_reply":"2022-07-07T22:44:08.809329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(df_sample)","metadata":{"execution":{"iopub.status.busy":"2022-07-07T22:44:50.597075Z","iopub.execute_input":"2022-07-07T22:44:50.597502Z","iopub.status.idle":"2022-07-07T22:44:50.607750Z","shell.execute_reply.started":"2022-07-07T22:44:50.597468Z","shell.execute_reply":"2022-07-07T22:44:50.606655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_sample.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-07T22:45:00.573128Z","iopub.execute_input":"2022-07-07T22:45:00.573985Z","iopub.status.idle":"2022-07-07T22:45:00.622699Z","shell.execute_reply.started":"2022-07-07T22:45:00.573946Z","shell.execute_reply":"2022-07-07T22:45:00.621759Z"},"trusted":true},"execution_count":null,"outputs":[]}]}