{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Time Series GRU TensorFlow Starter Notebook\nIn this notebook we present starter code for a time series GRU model and starter code for processing Kaggle's 50GB CSV files into multiple saved NumPy files. Using a time series GRU allows us to use all the provided customer data and not just the customer's last data point. We published plots of time series data [here][1]. In this notebook we\n* Processes the train data from dataframes into 3D NumPy array of dimensions `num_of_customers x 13 x 188`\n* Save processed arrays as multiple NumPy files on disk\n* Next we build and train a GRU from the multiple files on disk\n* We compute validation score and achieve 0.787\n* Finally we process and save test data, infer test, and create a submission\n\nIt is important to note that you **do not need** to process the train and test files every time you run this notebook. Only process the data again when you engineer new features. Otherwise, upload your saved NumPy arrays to a Kaggle dataset (or use my Kaggle dataset [here][2]). Then as you customize and improve your GRU model, set the variable `PROCESS_DATA = False` and `PATH_TO_DATA = [the path to your kaggle dataset]`.\n\nTo view time series EDA which can help give you intuition about feature engineering and improving model architecture, see my other notebook [here][1]. Note in the code below, we partition the GPU into 8GB for RAPIDS (feature engineering) and 8GB for TensorFlow (model build and train).\n\n[1]: https://www.kaggle.com/cdeotte/time-series-eda\n[2]: https://www.kaggle.com/datasets/cdeotte/amex-data-for-transformers-and-rnns","metadata":{}},{"cell_type":"code","source":"import os\nos.environ[\"CUDA_VISIBLE_DEVICES\"]=\"0\"\n\nimport tensorflow as tf\nimport tensorflow.keras.backend as K\nprint('Using TensorFlow version',tf.__version__)\n\n# RESTRICT TENSORFLOW TO 8GB OF GPU RAM\n# SO THAT WE HAVE 8GB RAM FOR RAPIDS\nLIMIT = 8\ngpus = tf.config.experimental.list_physical_devices('GPU')\nif gpus:\n  try:\n    tf.config.experimental.set_virtual_device_configuration(\n        gpus[0],\n        [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024*LIMIT)])\n    logical_gpus = tf.config.experimental.list_logical_devices('GPU')\n  except RuntimeError as e:\n    print(e)\nprint('We will restrict TensorFlow to max %iGB GPU RAM'%LIMIT)\nprint('then RAPIDS can use %iGB GPU RAM'%(16-LIMIT))","metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-05-30T17:40:14.013912Z","iopub.execute_input":"2022-05-30T17:40:14.014267Z","iopub.status.idle":"2022-05-30T17:40:14.022693Z","shell.execute_reply.started":"2022-05-30T17:40:14.014238Z","shell.execute_reply":"2022-05-30T17:40:14.021923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Process Train Data\nWe process both train and test data in chunks. We split train data into 10 parts and process each part separately and save to disk. We split test into 20 parts. This allows us to avoid memory errors during processing. We can also perform processing on GPU which is faster than CPU. Discussions about data preprocessing are [here][1] and [here][2]\n\n[1]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/327828\n[2]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/328054","metadata":{}},{"cell_type":"code","source":"# LOADING JUST FIRST COLUMN OF TRAIN OR TEST IS SLOW\n# INSTEAD YOU CAN LOAD FIRST COLUMN FROM MY DATASET\n# OTHERWISE SET VARIABLE TO NONE TO LOAD FROM KAGGLE'S ORIGINAL DATAFRAME\nPATH_TO_CUSTOMER_HASHES = '../input/amex-data-files/'\n\n# AFTER PROCESSING DATA ONCE, UPLOAD TO KAGGLE DATASET\n# THEN SET VARIABLE BELOW TO FALSE\n# AND ATTACH DATASET TO NOTEBOOK AND PUT PATH TO DATASET BELOW\nPROCESS_DATA = True\nPATH_TO_DATA = './data/'\n#PATH_TO_DATA = '../input/amex-data-for-transformers-and-rnns/data/'\n\n# AFTER TRAINING MODEL, UPLOAD TO KAGGLE DATASET\n# THEN SET VARIABLE BELOW TO FALSE\n# AND ATTACH DATASET TO NOTEBOOK AND PUT PATH TO DATASET BELOW\nTRAIN_MODEL = True\nPATH_TO_MODEL = './model/'\n#PATH_TO_MODEL = '../input/amex-data-for-transformers-and-rnns/model/'\n\nINFER_TEST = True","metadata":{"execution":{"iopub.status.busy":"2022-05-30T17:40:14.024539Z","iopub.execute_input":"2022-05-30T17:40:14.025089Z","iopub.status.idle":"2022-05-30T17:40:14.036773Z","shell.execute_reply.started":"2022-05-30T17:40:14.025050Z","shell.execute_reply":"2022-05-30T17:40:14.035875Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import cupy, cudf # GPU LIBRARIES\nimport numpy as np, pandas as pd # CPU LIBRARIES\nimport matplotlib.pyplot as plt, gc\n\nif PROCESS_DATA:\n    # LOAD TARGETS\n    targets = cudf.read_csv('../input/amex-default-prediction/train_labels.csv')\n    targets['customer_ID'] = targets['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\n    print(f'There are {targets.shape[0]} train targets')\n    \n    # GET TRAIN COLUMN NAMES\n    train = cudf.read_csv('../input/amex-default-prediction/train_data.csv', nrows=1)\n    T_COLS = train.columns\n    print(f'There are {len(T_COLS)} train dataframe columns')\n    \n    # GET TRAIN CUSTOMER NAMES (use pandas to avoid memory error)\n    if PATH_TO_CUSTOMER_HASHES:\n        train = cudf.read_parquet(f'{PATH_TO_CUSTOMER_HASHES}train_customer_hashes.pqt')\n    else:\n        train = pd.read_csv('/raid/Kaggle/amex/train_data.csv', usecols=['customer_ID'])\n        train['customer_ID'] = train['customer_ID'].apply(lambda x: int(x[-16:],16) ).astype('int64')\n    customers = train.drop_duplicates().sort_index().values.flatten()\n    print(f'There are {len(customers)} unique customers in train.')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-30T17:40:14.038178Z","iopub.execute_input":"2022-05-30T17:40:14.038825Z","iopub.status.idle":"2022-05-30T17:40:16.022228Z","shell.execute_reply.started":"2022-05-30T17:40:14.038712Z","shell.execute_reply":"2022-05-30T17:40:16.021339Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CALCULATE SIZE OF EACH SEPARATE FILE\ndef get_rows(customers, train, NUM_FILES = 10, verbose = ''):\n    chunk = len(customers)//NUM_FILES\n    if verbose != '':\n        print(f'We will split {verbose} data into {NUM_FILES} separate files.')\n        print(f'There will be {chunk} customers in each file (except the last file).')\n        print('Below are number of rows in each file:')\n    rows = []\n\n    for k in range(NUM_FILES):\n        if k==NUM_FILES-1: cc = customers[k*chunk:]\n        else: cc = customers[k*chunk:(k+1)*chunk]\n        s = train.loc[train.customer_ID.isin(cc)].shape[0]\n        rows.append(s)\n    if verbose != '': print( rows )\n    return rows\n\nif PROCESS_DATA:\n    NUM_FILES = 10\n    rows = get_rows(customers, train, NUM_FILES = NUM_FILES, verbose = 'train')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-30T17:40:16.024041Z","iopub.execute_input":"2022-05-30T17:40:16.024916Z","iopub.status.idle":"2022-05-30T17:40:16.581835Z","shell.execute_reply.started":"2022-05-30T17:40:16.024876Z","shell.execute_reply":"2022-05-30T17:40:16.580990Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocess and Feature Engineering\nThe function below processes the data. Discussions describing the process are [here][1] and [here][2]. Currently the code below uses [RAPIDS][3] and GPU to\n* Reduces memory usage of customer_ID column by converting to int64\n* Reduces memory usage of date time column (then deletes the column).\n* We fill NANs\n* Label encodes the categorical columns\n* We reduce memory usage dtypes of columns\n* Converts every customer into a 3D array with sequence length 13 and feature length 188\n\nTo improve this model, try adding new feautures. The columns have been rearanged to have the 11 categorical features first. This makes building the TensorFlow model later easier. We can also try adding Standard Scaler. Currently the data is being used without scaling from the original Kaggle train data. \n\n[1]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/327828\n[2]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/328054\n[3]: https://rapids.ai/","metadata":{}},{"cell_type":"code","source":"def feature_engineer(train, PAD_CUSTOMER_TO_13_ROWS = True, targets = None):\n        \n    # REDUCE STRING COLUMNS \n    # from 64 bytes to 8 bytes, and 10 bytes to 3 bytes respectively\n    train['customer_ID'] = train['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\n    train.S_2 = cudf.to_datetime( train.S_2 )\n    train['year'] = (train.S_2.dt.year-2000).astype('int8')\n    train['month'] = (train.S_2.dt.month).astype('int8')\n    train['day'] = (train.S_2.dt.day).astype('int8')\n    del train['S_2']\n        \n    # LABEL ENCODE CAT COLUMNS (and reduce to 1 byte)\n    # with 0: padding, 1: nan, 2,3,4,etc: values\n    d_63_map = {'CL':2, 'CO':3, 'CR':4, 'XL':5, 'XM':6, 'XZ':7}\n    train['D_63'] = train.D_63.map(d_63_map).fillna(1).astype('int8')\n\n    d_64_map = {'-1':2,'O':3, 'R':4, 'U':5}\n    train['D_64'] = train.D_64.map(d_64_map).fillna(1).astype('int8')\n    \n    CATS = ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_66', 'D_68']\n    OFFSETS = [2,1,2,2,3,2,3,2,2] #2 minus minimal value in full train csv\n    # then 0 will be padding, 1 will be NAN, 2,3,4,etc will be values\n    for c,s in zip(CATS,OFFSETS):\n        train[c] = train[c] + s\n        train[c] = train[c].fillna(1).astype('int8')\n    CATS += ['D_63','D_64']\n    \n    # ADD NEW FEATURES HERE\n    # EXAMPLE: train['feature_189'] = etc etc etc\n    # EXAMPLE: train['feature_190'] = etc etc etc\n    # IF CATEGORICAL, THEN ADD TO CATS WITH: CATS += ['feaure_190'] etc etc etc\n    \n    # REDUCE MEMORY DTYPE\n    SKIP = ['customer_ID','year','month','day']\n    for c in train.columns:\n        if c in SKIP: continue\n        if str( train[c].dtype )=='int64':\n            train[c] = train[c].astype('int32')\n        if str( train[c].dtype )=='float64':\n            train[c] = train[c].astype('float32')\n            \n    # PAD ROWS SO EACH CUSTOMER HAS 13 ROWS\n    if PAD_CUSTOMER_TO_13_ROWS:\n        tmp = train[['customer_ID']].groupby('customer_ID').customer_ID.agg('count')\n        more = cupy.array([],dtype='int64') \n        for j in range(1,13):\n            i = tmp.loc[tmp==j].index.values\n            more = cupy.concatenate([more,cupy.repeat(i,13-j)])\n        df = train.iloc[:len(more)].copy().fillna(0)\n        df = df * 0 - 1 #pad numerical columns with -1\n        df[CATS] = (df[CATS] * 0).astype('int8') #pad categorical columns with 0\n        df['customer_ID'] = more\n        train = cudf.concat([train,df],axis=0,ignore_index=True)\n        \n    # ADD TARGETS (and reduce to 1 byte)\n    if targets is not None:\n        train = train.merge(targets,on='customer_ID',how='left')\n        train.target = train.target.astype('int8')\n        \n    # FILL NAN\n    train = train.fillna(-0.5) #this applies to numerical columns\n    \n    # SORT BY CUSTOMER THEN DATE\n    train = train.sort_values(['customer_ID','year','month','day']).reset_index(drop=True)\n    train = train.drop(['year','month','day'],axis=1)\n    \n    # REARRANGE COLUMNS WITH 11 CATS FIRST\n    COLS = list(train.columns[1:])\n    COLS = ['customer_ID'] + CATS + [c for c in COLS if c not in CATS]\n    train = train[COLS]\n    \n    return train","metadata":{"execution":{"iopub.status.busy":"2022-05-30T17:40:16.582958Z","iopub.execute_input":"2022-05-30T17:40:16.583956Z","iopub.status.idle":"2022-05-30T17:40:16.605847Z","shell.execute_reply.started":"2022-05-30T17:40:16.583918Z","shell.execute_reply":"2022-05-30T17:40:16.605059Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if PROCESS_DATA:\n    # CREATE PROCESSED TRAIN FILES AND SAVE TO DISK        \n    for k in range(NUM_FILES):\n\n        # READ CHUNK OF TRAIN CSV FILE\n        skip = int(np.sum( rows[:k] ) + 1) #the plus one is for skipping header\n        train = cudf.read_csv('../input/amex-default-prediction/train_data.csv', nrows=rows[k], \n                              skiprows=skip, header=None, names=T_COLS)\n\n        # FEATURE ENGINEER DATAFRAME\n        train = feature_engineer(train, targets = targets)\n\n        # SAVE FILES\n        print(f'Train_File_{k+1} has {train.customer_ID.nunique()} customers and shape',train.shape)\n        tar = train[['customer_ID','target']].drop_duplicates().sort_index()\n        if not os.path.exists(PATH_TO_DATA): os.makedirs(PATH_TO_DATA)\n        tar.to_parquet(f'{PATH_TO_DATA}targets_{k+1}.pqt',index=False)\n        data = train.iloc[:,1:-1].values.reshape((-1,13,188))\n        cupy.save(f'{PATH_TO_DATA}data_{k+1}',data.astype('float32'))\n\n    # CLEAN MEMORY\n    del train, tar, data\n    del targets\n    gc.collect()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-30T17:40:16.606884Z","iopub.execute_input":"2022-05-30T17:40:16.607143Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build Model\nWe will just input the sequence data into a basic GRU. We will follow that we two dense layers and finally a sigmoid output to predict default. Try improving the model architecture.","metadata":{}},{"cell_type":"code","source":"# SIMPLE GRU MODEL\ndef build_model():\n    \n    # INPUT - FIRST 11 COLUMNS ARE CAT, NEXT 177 ARE NUMERIC\n    inp = tf.keras.Input(shape=(13,188))\n    embeddings = []\n    for k in range(11):\n        emb = tf.keras.layers.Embedding(10,4)\n        embeddings.append( emb(inp[:,:,k]) )\n    x = tf.keras.layers.Concatenate()([inp[:,:,11:]]+embeddings)\n    \n    # SIMPLE RNN BACKBONE\n    x = tf.keras.layers.GRU(units=128, return_sequences=False)(x)\n    x = tf.keras.layers.Dense(64,activation='relu')(x)\n    x = tf.keras.layers.Dense(32,activation='relu')(x)\n    \n    # OUTPUT\n    x = tf.keras.layers.Dense(1,activation='sigmoid')(x)\n    \n    # COMPILE MODEL\n    model = tf.keras.Model(inputs=inp, outputs=x)\n    opt = tf.keras.optimizers.Adam(learning_rate=0.001)\n    loss = tf.keras.losses.BinaryCrossentropy()\n    model.compile(loss=loss, optimizer = opt)\n    \n    return model","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CUSTOM LEARNING SCHEUDLE\ndef lrfn(epoch):\n    lr = [1e-3]*5 + [1e-4]*2 + [1e-5]*1\n    return lr[epoch]\nLR = tf.keras.callbacks.LearningRateScheduler(lrfn, verbose = False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Competition Metric Code\nThe code below is from Konstantin Yakovlev's discussion post [here][1]\n\n[1]: https://www.kaggle.com/competitions/amex-default-prediction/discussion/327534","metadata":{}},{"cell_type":"code","source":"# COMPETITION METRIC FROM Konstantin Yakovlev\n# https://www.kaggle.com/kyakovlev\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327534\ndef amex_metric_mod(y_true, y_pred):\n\n    labels     = np.transpose(np.array([y_true, y_pred]))\n    labels     = labels[labels[:, 1].argsort()[::-1]]\n    weights    = np.where(labels[:,0]==0, 20, 1)\n    cut_vals   = labels[np.cumsum(weights) <= int(0.04 * np.sum(weights))]\n    top_four   = np.sum(cut_vals[:,0]) / np.sum(labels[:,0])\n\n    gini = [0,0]\n    for i in [1,0]:\n        labels         = np.transpose(np.array([y_true, y_pred]))\n        labels         = labels[labels[:, i].argsort()[::-1]]\n        weight         = np.where(labels[:,0]==0, 20, 1)\n        weight_random  = np.cumsum(weight / np.sum(weight))\n        total_pos      = np.sum(labels[:, 0] *  weight)\n        cum_pos_found  = np.cumsum(labels[:, 0] * weight)\n        lorentz        = cum_pos_found / total_pos\n        gini[i]        = np.sum((lorentz - weight_random) * weight)\n\n    return 0.5 * (gini[1]/gini[0] + top_four)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train Model\nWe train 5 folds for 8 epochs each. We save the 5 fold models for test inference later. If you only want to infer without training, then set variable `TRAIN_MODEL = False` in the beginning of this notebook.","metadata":{}},{"cell_type":"code","source":"if TRAIN_MODEL:\n    # SAVE TRUE AND OOF\n    true = np.array([])\n    oof = np.array([])\n    VERBOSE = 2 # use 1 for interactive \n\n    for fold in range(5):\n\n        # INDICES OF TRAIN AND VALID FOLDS\n        valid_idx = [2*fold+1, 2*fold+2]\n        train_idx = [x for x in [1,2,3,4,5,6,7,8,9,10] if x not in valid_idx]\n\n        print('#'*25)\n        print(f'### Fold {fold+1} with valid files', valid_idx)\n\n        # READ TRAIN DATA FROM DISK\n        X_train = []; y_train = []\n        for k in train_idx:\n            X_train.append( np.load(f'{PATH_TO_DATA}data_{k}.npy'))\n            y_train.append( pd.read_parquet(f'{PATH_TO_DATA}targets_{k}.pqt') )\n        X_train = np.concatenate(X_train,axis=0)\n        y_train = pd.concat(y_train).target.values\n        print('### Training data shapes', X_train.shape, y_train.shape)\n\n        # READ VALID DATA FROM DISK\n        X_valid = []; y_valid = []\n        for k in valid_idx:\n            X_valid.append( np.load(f'{PATH_TO_DATA}data_{k}.npy'))\n            y_valid.append( pd.read_parquet(f'{PATH_TO_DATA}targets_{k}.pqt') )\n        X_valid = np.concatenate(X_valid,axis=0)\n        y_valid = pd.concat(y_valid).target.values\n        print('### Validation data shapes', X_valid.shape, y_valid.shape)\n        print('#'*25)\n\n        # BUILD AND TRAIN MODEL\n        K.clear_session()\n        model = build_model()\n        h = model.fit(X_train,y_train, \n                      validation_data = (X_valid,y_valid),\n                      batch_size=512, epochs=8, verbose=VERBOSE,\n                      callbacks = [LR])\n        if not os.path.exists(PATH_TO_MODEL): os.makedirs(PATH_TO_MODEL)\n        model.save_weights(f'{PATH_TO_MODEL}gru_fold_{fold+1}.h5')\n\n        # INFER VALID DATA\n        print('Inferring validation data...')\n        p = model.predict(X_valid, batch_size=512, verbose=VERBOSE).flatten()\n\n        print()\n        print(f'Fold {fold+1} CV=', amex_metric_mod(y_valid, p) )\n        print()\n        true = np.concatenate([true, y_valid])\n        oof = np.concatenate([oof, p])\n        \n        # CLEAN MEMORY\n        del model, X_train, y_train, X_valid, y_valid, p\n        gc.collect()\n\n    # PRINT OVERALL RESULTS\n    print('#'*25)\n    print(f'Overall CV =', amex_metric_mod(true, oof) )\n    K.clear_session()","metadata":{"scrolled":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Process Test Data\nWe process the test data in the same way as train data.","metadata":{}},{"cell_type":"code","source":"if PROCESS_DATA:\n    # GET TEST COLUMN NAMES\n    test = cudf.read_csv('../input/amex-default-prediction/test_data.csv', nrows=1)\n    T_COLS = test.columns\n    print(f'There are {len(T_COLS)} test dataframe columns')\n    \n    # GET TEST CUSTOMER NAMES (use pandas to avoid memory error)\n    if PATH_TO_CUSTOMER_HASHES:\n        test = cudf.read_parquet(f'{PATH_TO_CUSTOMER_HASHES}test_customer_hashes.pqt')\n    else:\n        test = pd.read_csv('/raid/Kaggle/amex/test_data.csv', usecols=['customer_ID'])\n        test['customer_ID'] = test['customer_ID'].apply(lambda x: int(x[-16:],16) ).astype('int64')\n    customers = test.drop_duplicates().sort_index().values.flatten()\n    print(f'There are {len(customers)} unique customers in test.')","metadata":{"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_FILES = 20\nif PROCESS_DATA:\n    # CALCULATE SIZE OF EACH SEPARATE FILE\n    rows = get_rows(customers, test, NUM_FILES = NUM_FILES, verbose = 'test')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if PROCESS_DATA:\n    # SAVE TEST CUSTOMERS INDEX\n    test_customer_hashes = cupy.array([],dtype='int64')\n    \n    # CREATE PROCESSED TEST FILES AND SAVE TO DISK\n    for k in range(NUM_FILES):\n\n        # READ CHUNK OF TEST CSV FILE\n        skip = int(np.sum( rows[:k] ) + 1) #the plus one is for skipping header\n        test = cudf.read_csv('../input/amex-default-prediction/test_data.csv', nrows=rows[k], \n                              skiprows=skip, header=None, names=T_COLS)\n\n        # FEATURE ENGINEER DATAFRAME\n        test = feature_engineer(test, targets = None)\n        \n        # SAVE TEST CUSTOMERS INDEX\n        cust = test[['customer_ID']].drop_duplicates().sort_index().values.flatten()\n        test_customer_hashes = cupy.concatenate([test_customer_hashes,cust])\n\n        # SAVE FILES\n        print(f'Test_File_{k+1} has {test.customer_ID.nunique()} customers and shape',test.shape)\n        data = test.iloc[:,1:].values.reshape((-1,13,188))\n        cupy.save(f'{PATH_TO_DATA}test_data_{k+1}',data.astype('float32'))\n        \n    # SAVE CUSTOMER INDEX OF ALL TEST FILES\n    cupy.save(f'{PATH_TO_DATA}test_hashes_data', test_customer_hashes)\n\n    # CLEAN MEMORY\n    del test, data\n    gc.collect()","metadata":{"scrolled":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Infer Test Data\nWe infer the test data from our saved fold models. If you don't wish to infer test but you only want your notebook to compute a validation score to evaluate model changes, then set variable `INFER_TEST = False` in the beginning of this notebook. Also if you wish to infer from previously trained models, then add the path to the Kaggle dataset in the variable `PATH_TO_MODEL` in the beginning of this notebook.","metadata":{}},{"cell_type":"code","source":"if INFER_TEST:\n    # INFER TEST DATA\n    start = 0; end = 0\n    sub = cudf.read_csv('../input/amex-default-prediction/sample_submission.csv')\n    \n    # REARANGE SUB ROWS TO MATCH PROCESSED TEST FILES\n    sub['hash'] = sub['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\n    test_hash_index = cupy.load(f'{PATH_TO_DATA}test_hashes_data.npy')\n    sub = sub.set_index('hash').loc[test_hash_index].reset_index(drop=True)\n    \n    for k in range(NUM_FILES):\n        # BUILD MODEL\n        K.clear_session()\n        model = build_model()\n        \n        # LOAD TEST DATA\n        print(f'Inferring Test_File_{k+1}')\n        X_test = np.load(f'{PATH_TO_DATA}test_data_{k+1}.npy')\n        end = start + X_test.shape[0]\n\n        # INFER 5 FOLD MODELS\n        model.load_weights(f'{PATH_TO_MODEL}gru_fold_1.h5')\n        p = model.predict(X_test, batch_size=512, verbose=0).flatten() \n        for j in range(1,5):\n            model.load_weights(f'{PATH_TO_MODEL}gru_fold_{j+1}.h5')\n            p += model.predict(X_test, batch_size=512, verbose=0).flatten()\n        p /= 5.0\n\n        # SAVE TEST PREDICTIONS\n        sub.loc[start:end-1,'prediction'] = p\n        start = end\n        \n        # CLEAN MEMORY\n        del model, X_test, p\n        gc.collect()","metadata":{"scrolled":true,"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create Submission","metadata":{}},{"cell_type":"code","source":"if INFER_TEST:\n    sub.to_csv('submission.csv',index=False)\n    print('Submission file shape is', sub.shape )\n    display( sub.head() )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if INFER_TEST:\n    # DISPLAY SUBMISSION PREDICTIONS\n    plt.hist(sub.to_pandas().prediction, bins=100)\n    plt.title('Test Predictions')\n    plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}