{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"

American Express - Default Prediction

\n
CREATED BY: Josua Naiborhu
\n\n
\n\n\n
\n 🛑   WARNING:

THIS IS A WORK IN PROGRESS
\n
\n\n---\n
\n đź‘Ź   IF YOU FORK THIS OR FIND THIS HELPFUL   đź‘Ź

PLEASE UPVOTE!

\n
\n\n\n","metadata":{"execution":{"iopub.status.busy":"2022-06-15T12:14:59.109211Z","iopub.execute_input":"2022-06-15T12:14:59.109858Z","iopub.status.idle":"2022-06-15T12:14:59.142734Z","shell.execute_reply.started":"2022-06-15T12:14:59.109755Z","shell.execute_reply":"2022-06-15T12:14:59.141961Z"}}},{"cell_type":"markdown","source":"

\n\n

\n\n

TABLE OF CONTENTS

\n\n---\n\n

0    IMPORTS

\n\n---\n\n

1    BACKGROUND INFORMATION

\n\n---\n\n

2    SETUP

\n\n---\n\n

4    DATASET EXPLORATION

\n\n---\n\n

5    MODELLING

\n\n---\n\n\n

5    EVALUATION

","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

0  IMPORTS    ⤒

","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport dask.dataframe as dd\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfrom matplotlib import gridspec\nimport cupy ,cudf # GPU libraries so you have to activate your GPU if you use kaggle kernels\nimport matplotlib.pyplot as plt, gc, os\n\n#for cleaning data after using GPU\nfrom numba import cuda\ncuda.select_device(0)\ncuda.close()\ncuda.select_device(0)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:46:07.028877Z","iopub.execute_input":"2022-06-19T05:46:07.029339Z","iopub.status.idle":"2022-06-19T05:46:12.689305Z","shell.execute_reply.started":"2022-06-19T05:46:07.029279Z","shell.execute_reply":"2022-06-19T05:46:12.687441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

1  BACKGROUND INFORMATION    ⤒

\n","metadata":{}},{"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo('hPPNJ0_Dyhk', width=800, height=500)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:46:12.691581Z","iopub.execute_input":"2022-06-19T05:46:12.692193Z","iopub.status.idle":"2022-06-19T05:46:32.738889Z","shell.execute_reply.started":"2022-06-19T05:46:12.692148Z","shell.execute_reply":"2022-06-19T05:46:32.73774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

1.1 BASIC COMPETITION INFORMATION

\n\n---\n\n
PRIMARY TASK DESCRIPTION\n\nIn this competition, you’ll apply your machine learning skills to predict credit default. Specifically, you will leverage an industrial scale data set to build a machine learning model that challenges the current model in production. Training, validation, and testing datasets include time-series behavioral data and anonymized customer profile information. You're free to explore any technique to create the most powerful model, from creating features to using the data in a more organic way within a model..\n\n
BASIC BACKGROUND INFORMATION\n\nWhether out at a restaurant or buying tickets to a concert, modern life counts on the convenience of a credit card to make daily purchases. It saves us from carrying large amounts of cash and also can advance a full purchase that can be paid over time. How do card issuers know we’ll pay back what we charge? That’s a complex problem with many existing solutions—and even more potential improvements, to be explored in this competition.\n\nCredit default prediction is central to managing risk in a consumer lending business. Credit default prediction allows lenders to optimize lending decisions, which leads to a better customer experience and sound business economics. Current models exist to help manage risk. But it's possible to create better models that can outperform those currently in use.\n\nAmerican Express is a globally integrated payments company. The largest payment card issuer in the world, they provide customers with access to products, insights, and experiences that enrich lives and build business success.If successful, you'll help create a better customer experience for cardholders by making it easier to be approved for a credit card. Top solutions could challenge the credit default prediction model used by the world's largest payment card issuer—earning you cash prizes, the opportunity to interview with American Express, and potentially a rewarding new career.\n\n
COMPETITION HOST INFORMATION\n\nAmerican Express is a leading issuer of personal, small business, and corporate credit cards. The company's travel-related offerings include traveler's checks, credit cards, corporate and personal travel planning services, tour packages, and agencies for hotel and car-rental reservations. By the early 21st century, American Express operated in more than 40 countries. The company also had a publishing division, which produced such magazines as Travel & Leisure and Food & Wine. However, it was sold to Time Inc. in 2013.\n\n
COMPETITION IMPACT STATEMENT\n\nYou'll help create a better customer experience for cardholders by making it easier to be approved for a credit card that can optimize lending decision which leads to a better customer experience and sound business economics. ","metadata":{}},{"cell_type":"markdown","source":"

1.2 COMPETITION EVALUATION

\n\n---\n\n
GENERAL EVALUATION INFORMATION\n\nThe objective of this competition is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile. The target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\n\nThe dataset contains aggregated profile features for each customer at each statement date. Features are anonymized and normalized, and fall into the following general categories:\n\n- D_* = Delinquency variables\n- S_* = Spend variables\n- P_* = Payment variables\n- B_* = Balance variables\n- R_* = Risk variables\n\nwith the following features being categorical:\n\n- ['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\nYour task is to predict, for each customer_ID, the probability of a future payment default (target = 1).\n\nNote that the negative class has been subsampled for this dataset at 5%, and thus receives a 20x weighting in the scoring metric which means\n\n\"Of the 458913 customer_IDs in the training data, 340000 (74 %) have a label of 0 (good customer, no default) and 119000 (26 %) have a label of 1 (bad customer, default).\n\nIn reality, however, American Express has twenty times more good customers (they have 6.8 million non-defaulting customers). 98 % (6.8 million of 6.9 million) of the customers are good; 2 % (0.1 million of 6.9 million) are bad. Why did they subsample the non-defaulting customers? They wanted to be nice and give us a dataset which fits into memory. If the dataset contained 6.9 million customers, every notebook would crash when you try to load the dataset, and almost nobody would participate in the competition.\" Taken from [Dataset explanations Discussions](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327234#1802056)\n\n\n
ADDITIONAL INFO\n\nThe competition metric has two components: the normalized Gini coefficient and the default rate captured at 4 %:\nThe normalized Gini coefficient is simply a stretched AUC: AUC is the light red area under the curve, which has a value between 0 and 1. The normalized Gini coefficient is equal to 2*AUC-1 and is between -1 and 1. The larger the red area, the better is the score. The default rate captured at 4 % is the true positive rate (recall) for a threshold set at 4 % of the total (weighted) sample count. It corresponds to the y coordinate of the intersection between the green line and the red roc curve (marked with a green dot) and is always between 0 and 1. The higher the intersection point, the better is the score. Taken from [Graphical explanation of the competition metric](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327464#1822004).\n\n\n
\n\n
SUBMISSION FILE INFORMATION\n\nFor each customer_ID in the test set, you must predict a probability for the target variable.\n
\n\nThe file should contain a header and have the following format:\n\n```\ncustomer_ID,prediction\n00000469ba...,0.01\n00001bf2e7...,0.22\n0000210045...,0.98\netc.\n```\n","metadata":{}},{"cell_type":"markdown","source":"
FILE INFORMATION\n\n**`train.csv`** \n- Customer_ID, Features columns for all customers in S_,D_,B_,R_,P_ and target features \n- **Columns**\n * **`Customer_ID`**\n * unique identifier for each customer\n * **`S_*`**\n * Delinquency variables \n * **`S_*`**\n * Spend variables\n * **`B_**`**\n * Balance variables\n * **`R_*`**\n * Risk variables\n
\n\n**`sample_submission.csv`**\n- A sample submission file in the correct format\n\n
\n\n**`test/`**\n- a file of customer_ID where we predict the probability of default(1).","metadata":{}},{"cell_type":"code","source":"for dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n file_size = round(os.path.getsize(os.path.join(dirname, filename)) / (1e9), 2)\n print('*' * 70)\n print(f\"Filename : {filename} \\t File Size : {file_size} GB\")\n print('*' * 70)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:46:32.741475Z","iopub.execute_input":"2022-06-19T05:46:32.742065Z","iopub.status.idle":"2022-06-19T05:46:32.768052Z","shell.execute_reply.started":"2022-06-19T05:46:32.742017Z","shell.execute_reply":"2022-06-19T05:46:32.767016Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can see that the data set consist of 16.39 GB for training data and 33.89 GB for test data. Maybe you are wondering why the test dataset is twice bigger than train dataset. This is what i am also looking for and thanks to @ambrosm through discussion forum who explained it very well [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/329088). For the experimentation for loading the data and experiment with ML Algorithms, we will use compressed dataset provided by @raddar to load the data with smaller size where the train size is 1.64 GB and test size is 3.3 GB and @cdotte for time series data to look at trend among customers. Maybe we can use this feature for our XGBoost model. I am still thinking this as new feature for next development of myself to compete for this competition. Thanks to @cdotte for amazing tutorial of implementing XGBoost for this competition. You can take a look at how he convert the data into parquet and change the precision size of each columns so that the data size become so much easier to load using pandas/dask/cudf. You can check the discussions [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514). I really love the discussion during that add my knowledge to load REALLY BIG DATA. This is really different size provided by the organizer compared to other competitions and make us load data faster to do EDA or even do training and inference to predict the probability of default customers.","metadata":{}},{"cell_type":"code","source":"train_df = pd.read_parquet('../input/amex-data-integer-dtypes-parquet-format/train.parquet')\ntrain_df.head(14)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:46:32.770352Z","iopub.execute_input":"2022-06-19T05:46:32.771188Z","iopub.status.idle":"2022-06-19T05:46:55.751372Z","shell.execute_reply.started":"2022-06-19T05:46:32.77114Z","shell.execute_reply":"2022-06-19T05:46:55.750275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Check Missing Values ","metadata":{}},{"cell_type":"code","source":"train_df.info(max_cols=200, show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:49:35.735708Z","iopub.execute_input":"2022-06-19T05:49:35.736727Z","iopub.status.idle":"2022-06-19T05:49:38.825644Z","shell.execute_reply.started":"2022-06-19T05:49:35.736682Z","shell.execute_reply":"2022-06-19T05:49:38.824558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_df.isnull().sum()/len(train_df)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:49:58.363914Z","iopub.execute_input":"2022-06-19T05:49:58.364335Z","iopub.status.idle":"2022-06-19T05:50:01.004542Z","shell.execute_reply.started":"2022-06-19T05:49:58.364285Z","shell.execute_reply":"2022-06-19T05:50:01.003404Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see most of the features have missing values. We can choose Decision tree based Algorithm like XGBoost, Light GBM that can handle missing values so that we do not need to impute them. However, When using Neural Network, we have to handle missing values by various methodologies that you can check [here](https://www.kaggle.com/code/parulpandey/a-guide-to-handling-missing-values-in-python).","metadata":{}},{"cell_type":"markdown","source":"# Check counts statements per Customers","metadata":{}},{"cell_type":"code","source":"targets =pd.read_csv(\"../input/amex-default-prediction/train_labels.csv\")\ntargets.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:50:02.255198Z","iopub.execute_input":"2022-06-19T05:50:02.255707Z","iopub.status.idle":"2022-06-19T05:50:03.197072Z","shell.execute_reply.started":"2022-06-19T05:50:02.255654Z","shell.execute_reply":"2022-06-19T05:50:03.196186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns \nimport pandas as pd\nimport numpy as np \nplt.style.use('seaborn')\ncolor_pal = sns.color_palette(\"Set2\")\ntrain_label = pd.read_csv(\"../input/amex-default-prediction/train_labels.csv\")\npct_default = np.mean(train_label['target'])\nprint(f'{(pct_default *100): 0.2f}% of the Training Data Defaults')\n(train_label['target'].value_counts()\n .plot(kind='bar',\n title='Distribution of Target',\n color=color_pal[2]))","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:50:03.199042Z","iopub.execute_input":"2022-06-19T05:50:03.199789Z","iopub.status.idle":"2022-06-19T05:50:04.043468Z","shell.execute_reply.started":"2022-06-19T05:50:03.199748Z","shell.execute_reply":"2022-06-19T05:50:04.042306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It is obvious that the target column is imbalanced. Hence,we do not use Accuracy as our metric. The Organizer use Gini Coefficient(like ROC Curve) and the default rate captured at 4%. This prompts the nature of credit default which is less than non default. You can also check this discussion [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/327234#1802056). Before implementing Machine Learning Algorithms, we have to pay attention to imbalanced target/label. We have to handle this one by using various method like SMOTE, downsampling, upsampling or the easiest way through implementing StratifieldKFold provided by sklearn to ascertain the proportion of the label to be balanced. You can check the guideline [here](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html)","metadata":{}},{"cell_type":"markdown","source":"## Check number of features for various type Features\n","metadata":{}},{"cell_type":"code","source":"d_feats = [c for c in train_df.columns if c.startswith('D_')]\ns_feats = [c for c in train_df.columns if c.startswith('S_')]\np_feats = [c for c in train_df.columns if c.startswith('P_')]\nb_feats = [c for c in train_df.columns if c.startswith('B_')]\nr_feats = [c for c in train_df.columns if c.startswith('R_')]","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:50:05.981855Z","iopub.execute_input":"2022-06-19T05:50:05.982386Z","iopub.status.idle":"2022-06-19T05:50:05.990422Z","shell.execute_reply.started":"2022-06-19T05:50:05.98234Z","shell.execute_reply":"2022-06-19T05:50:05.989113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Number of Delinquency variables: {len(d_feats)}')\nprint(f'Number of Spend variables: {len(s_feats)}')\nprint(f'Number of Payment variables: {len(p_feats)}')\nprint(f'Number of Balance variables: {len(b_feats)}')\nprint(f'Number of Risk variables: {len(r_feats)}')","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:50:06.648307Z","iopub.execute_input":"2022-06-19T05:50:06.648698Z","iopub.status.idle":"2022-06-19T05:50:06.655353Z","shell.execute_reply.started":"2022-06-19T05:50:06.648665Z","shell.execute_reply":"2022-06-19T05:50:06.654285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Delinquency has the most variable in the train dataset.","metadata":{}},{"cell_type":"code","source":"#https://www.kaggle.com/competitions/amex-default-prediction/discussion/327651\nimport pandas as pd \ncol_B2 = pd.read_csv('../input/amex-default-prediction/train_data.csv', usecols=['B_2'])","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:50:07.839012Z","iopub.execute_input":"2022-06-19T05:50:07.839457Z","iopub.status.idle":"2022-06-19T05:54:07.582957Z","shell.execute_reply.started":"2022-06-19T05:50:07.839421Z","shell.execute_reply":"2022-06-19T05:54:07.581863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"col_B2.hist()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:54:07.585133Z","iopub.execute_input":"2022-06-19T05:54:07.585843Z","iopub.status.idle":"2022-06-19T05:54:08.07898Z","shell.execute_reply.started":"2022-06-19T05:54:07.585801Z","shell.execute_reply":"2022-06-19T05:54:08.078153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"it appears that column B_8 is a binary feature that has values 0 and 1.","metadata":{}},{"cell_type":"code","source":"col_B2.loc[col_B2.B_2>0.99,'B_2'].hist()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:54:08.080648Z","iopub.execute_input":"2022-06-19T05:54:08.08129Z","iopub.status.idle":"2022-06-19T05:54:08.382106Z","shell.execute_reply.started":"2022-06-19T05:54:08.081249Z","shell.execute_reply":"2022-06-19T05:54:08.381214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(col_B2.loc[(col_B2.B_2>0.8) & (col_B2.B_2<0.83),'B_2']).hist()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:54:08.384447Z","iopub.execute_input":"2022-06-19T05:54:08.384871Z","iopub.status.idle":"2022-06-19T05:54:08.657135Z","shell.execute_reply.started":"2022-06-19T05:54:08.38483Z","shell.execute_reply":"2022-06-19T05:54:08.656274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"col_B2.loc[col_B2.B_2<0.1,'B_2'].hist()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:54:08.658761Z","iopub.execute_input":"2022-06-19T05:54:08.659169Z","iopub.status.idle":"2022-06-19T05:54:08.934198Z","shell.execute_reply.started":"2022-06-19T05:54:08.659128Z","shell.execute_reply":"2022-06-19T05:54:08.933384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(col_B2['B_2'].unique())/len(col_B2['B_2'])","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:54:08.935438Z","iopub.execute_input":"2022-06-19T05:54:08.936245Z","iopub.status.idle":"2022-06-19T05:54:09.795725Z","shell.execute_reply.started":"2022-06-19T05:54:08.936186Z","shell.execute_reply":"2022-06-19T05:54:09.794737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd \ncol_R26 = pd.read_csv('../input/amex-default-prediction/train_data.csv', usecols=['R_26'])","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:54:09.797341Z","iopub.execute_input":"2022-06-19T05:54:09.797742Z","iopub.status.idle":"2022-06-19T05:58:09.147202Z","shell.execute_reply.started":"2022-06-19T05:54:09.797703Z","shell.execute_reply":"2022-06-19T05:58:09.146224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"col_R26.hist()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:09.148741Z","iopub.execute_input":"2022-06-19T05:58:09.149121Z","iopub.status.idle":"2022-06-19T05:58:09.472124Z","shell.execute_reply.started":"2022-06-19T05:58:09.149082Z","shell.execute_reply":"2022-06-19T05:58:09.470654Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"col_R26.loc[col_R26.R_26<0.2,'R_26'].hist()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:09.478995Z","iopub.execute_input":"2022-06-19T05:58:09.48325Z","iopub.status.idle":"2022-06-19T05:58:09.860756Z","shell.execute_reply.started":"2022-06-19T05:58:09.4832Z","shell.execute_reply":"2022-06-19T05:58:09.859961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here is another example of random uniform noise. The variable R_26 has 548160 unique values below 0.2 and is provided as float64 (8 bytes), but really from the plot we see it is only 6 discrete values with (random uniform) noise added and can be converted to int8 (1 byte) - without information loss.","metadata":{}},{"cell_type":"code","source":"# LOAD TRAIN DATA AND MERGE TARGETS ONTO FEATURES\ndf = pd.read_csv('../input/amex-default-prediction/train_data.csv', nrows=100_000)\ndf.S_2 = pd.to_datetime(df.S_2)\ndf2 = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\ndf = df.merge(df2,on='customer_ID',how='left')","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:09.867226Z","iopub.execute_input":"2022-06-19T05:58:09.868213Z","iopub.status.idle":"2022-06-19T05:58:17.526115Z","shell.execute_reply.started":"2022-06-19T05:58:09.868166Z","shell.execute_reply":"2022-06-19T05:58:17.525157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_time_series(prefix='D', cols=None, display_ct=32):\n \n # DETERMINE WHICH COLUMNS TO PLOT\n if cols is not None and len(cols)==0: cols = None\n if cols is None:\n COLS = df.columns[2:-1]\n COLS = np.sort( [int(x[2:]) for x in COLS if x[0]==prefix] )\n COLS = [f'{prefix}_{x}' for x in COLS]\n print('#'*25)\n print(f'Plotting all {len(COLS)} columns with prefix {prefix}')\n print('#'*25)\n else:\n COLS = [f'{prefix}_{x}' for x in cols]\n print('#'*25)\n print(f'Plotting {len(COLS)} columns with prefix {prefix}')\n print('#'*25)\n\n # ITERATE COLUMNS\n for c in COLS:\n\n # CONVERT DATAFRAME INTO SERIES WITH COLUMN\n tmp = df[['customer_ID','S_2',c,'target']].copy()\n tmp2 = tmp.groupby(['customer_ID','target'])[['S_2',c]].agg(list).reset_index()\n tmp3 = tmp2.loc[tmp2.target==1]\n tmp4 = tmp2.loc[tmp2.target==0]\n\n # FORMAT PLOT\n spec = gridspec.GridSpec(ncols=2, nrows=1,\n width_ratios=[3, 1], wspace=0.1,\n hspace=0.5, height_ratios=[1])\n fig = plt.figure(figsize=(20,10))\n ax0 = fig.add_subplot(spec[0])\n\n # PLOT 32 DEFAULT CUSTOMERS AND 32 NON-DEFAULT CUSTOMERS\n t0 = []; t1 = []\n for k in range(display_ct):\n try:\n # PLOT DEFAULTING CUSTOMERS\n row = tmp3.iloc[k]\n ax0.plot(row.S_2,row[c],'-o',color='blue')\n t1 += row[c]\n # PLOT NON-DEFAULT CUSTOMERS\n row = tmp4.iloc[k]\n ax0.plot(row.S_2,row[c],'-o',color='orange')\n t0 += row[c]\n except:\n pass\n plt.title(f'Feature {c} (Key: BLUE=DEFAULT, orange=no default)',size=18)\n\n # PLOT HISTOGRAMS\n ax1 = fig.add_subplot(spec[1])\n try:\n # COMPUTE BINS\n t = t0+t1; mn = np.nanmin(t); mx = np.nanmax(t)\n if mx==mn:\n mx += 0.01; mn -= 0.01\n bins = np.arange(mn,mx+(mx-mn)/20,(mx-mn)/20 )\n # PLOT HISTOGRAMS\n if np.sum(np.isnan(t1))!=len(t1):\n ax1.hist(t1,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='blue')\n if np.sum(np.isnan(t0))!=len(t0):\n ax1.hist(t0,bins=bins,orientation=\"horizontal\",alpha = 0.8,color='orange')\n except:\n pass\n plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:17.527943Z","iopub.execute_input":"2022-06-19T05:58:17.528455Z","iopub.status.idle":"2022-06-19T05:58:17.547597Z","shell.execute_reply.started":"2022-06-19T05:58:17.528415Z","shell.execute_reply":"2022-06-19T05:58:17.546616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A very good explanation of how to take benefit from time feature provided by @cdeotte. This is very well explained by him.","metadata":{}},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('D',[39,41,47,45,46,48,54,59,61,62,75,96,105,112,124])","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:17.54942Z","iopub.execute_input":"2022-06-19T05:58:17.549841Z","iopub.status.idle":"2022-06-19T05:58:39.163096Z","shell.execute_reply.started":"2022-06-19T05:58:17.549791Z","shell.execute_reply":"2022-06-19T05:58:39.162216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('S',[3,7,19,23,26])","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:39.16451Z","iopub.execute_input":"2022-06-19T05:58:39.16547Z","iopub.status.idle":"2022-06-19T05:58:46.242091Z","shell.execute_reply.started":"2022-06-19T05:58:39.165428Z","shell.execute_reply":"2022-06-19T05:58:46.241209Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('P',[2,3])","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:46.243453Z","iopub.execute_input":"2022-06-19T05:58:46.244372Z","iopub.status.idle":"2022-06-19T05:58:48.980028Z","shell.execute_reply.started":"2022-06-19T05:58:46.244328Z","shell.execute_reply":"2022-06-19T05:58:48.979215Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('B',[2,3,4,5,7,9,20])","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:48.981343Z","iopub.execute_input":"2022-06-19T05:58:48.98208Z","iopub.status.idle":"2022-06-19T05:58:59.100369Z","shell.execute_reply.started":"2022-06-19T05:58:48.982037Z","shell.execute_reply":"2022-06-19T05:58:59.099445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LEAVE LIST BLANK TO PLOT ALL\nplot_time_series('R',[1,3,13,18])","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:58:59.101956Z","iopub.execute_input":"2022-06-19T05:58:59.102622Z","iopub.status.idle":"2022-06-19T05:59:04.6633Z","shell.execute_reply.started":"2022-06-19T05:58:59.102581Z","shell.execute_reply":"2022-06-19T05:59:04.66149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LOAD LIBRARIES\nprint('RAPIDS version',cudf.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:04.666254Z","iopub.execute_input":"2022-06-19T05:59:04.667976Z","iopub.status.idle":"2022-06-19T05:59:04.679031Z","shell.execute_reply.started":"2022-06-19T05:59:04.667903Z","shell.execute_reply":"2022-06-19T05:59:04.67746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# VERSION NAME FOR SAVED MODEL FILES\nVER = 1\n\n# TRAIN RANDOM SEED\nSEED = 42\n\n# FILL NAN VALUE\nNAN_VALUE = -127 # will fit in int8\n\n# FOLDS PER MODEL\nFOLDS = 5","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:04.681099Z","iopub.execute_input":"2022-06-19T05:59:04.681636Z","iopub.status.idle":"2022-06-19T05:59:04.692732Z","shell.execute_reply.started":"2022-06-19T05:59:04.681596Z","shell.execute_reply":"2022-06-19T05:59:04.691188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_file(path = '', usecols = None):\n # LOAD DATAFRAME\n if usecols is not None: df = cudf.read_parquet(path, columns=usecols)\n else: df = cudf.read_parquet(path)\n # REDUCE DTYPE FOR CUSTOMER AND DATE\n df['customer_ID'] = df['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\n df.S_2 = cudf.to_datetime( df.S_2 )\n # SORT BY CUSTOMER AND DATE (so agg('last') works correctly)\n #df = df.sort_values(['customer_ID','S_2'])\n #df = df.reset_index(drop=True)\n # FILL NAN\n df = df.fillna(NAN_VALUE) \n print('shape of data:', df.shape)\n \n return df\n\nprint('Reading train data...')\nTRAIN_PATH = '../input/amex-data-integer-dtypes-parquet-format/train.parquet'\ntrain = read_file(path = TRAIN_PATH)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:04.69474Z","iopub.execute_input":"2022-06-19T05:59:04.695289Z","iopub.status.idle":"2022-06-19T05:59:25.605576Z","shell.execute_reply.started":"2022-06-19T05:59:04.69524Z","shell.execute_reply":"2022-06-19T05:59:25.604503Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def process_and_feature_engineer(df):\n # FEATURE ENGINEERING FROM \n # https://www.kaggle.com/code/huseyincot/amex-agg-data-how-it-created\n all_cols = [c for c in list(df.columns) if c not in ['customer_ID','S_2']]\n cat_features = [\"B_30\",\"B_38\",\"D_114\",\"D_116\",\"D_117\",\"D_120\",\"D_126\",\"D_63\",\"D_64\",\"D_66\",\"D_68\"]\n num_features = [col for col in all_cols if col not in cat_features]\n\n test_num_agg = df.groupby(\"customer_ID\")[num_features].agg(['mean', 'std', 'min', 'max', 'last'])\n test_num_agg.columns = ['_'.join(x) for x in test_num_agg.columns]\n\n test_cat_agg = df.groupby(\"customer_ID\")[cat_features].agg(['count', 'last', 'nunique'])\n test_cat_agg.columns = ['_'.join(x) for x in test_cat_agg.columns]\n\n df = cudf.concat([test_num_agg, test_cat_agg], axis=1)\n del test_num_agg, test_cat_agg\n print('shape after engineering', df.shape )\n \n return df\n\ntrain = process_and_feature_engineer(train)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:25.607489Z","iopub.execute_input":"2022-06-19T05:59:25.607902Z","iopub.status.idle":"2022-06-19T05:59:27.044964Z","shell.execute_reply.started":"2022-06-19T05:59:25.607861Z","shell.execute_reply":"2022-06-19T05:59:27.044029Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:27.046394Z","iopub.execute_input":"2022-06-19T05:59:27.047242Z","iopub.status.idle":"2022-06-19T05:59:27.985096Z","shell.execute_reply.started":"2022-06-19T05:59:27.0472Z","shell.execute_reply":"2022-06-19T05:59:27.984214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ADD TARGETS\ntargets = cudf.read_csv('../input/amex-default-prediction/train_labels.csv')\ntargets['customer_ID'] = targets['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\ntargets = targets.set_index('customer_ID')\ntrain = train.merge(targets, left_index=True, right_index=True, how='left')\ntrain.target = train.target.astype('int8')\ndel targets\n\n# NEEDED TO MAKE CV DETERMINISTIC (cudf merge above randomly shuffles rows)\ntrain = train.sort_index().reset_index()\n\n# FEATURES\nFEATURES = train.columns[1:-1]\nprint(f'There are {len(FEATURES)} features!')","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:27.986551Z","iopub.execute_input":"2022-06-19T05:59:27.987551Z","iopub.status.idle":"2022-06-19T05:59:28.915414Z","shell.execute_reply.started":"2022-06-19T05:59:27.987507Z","shell.execute_reply":"2022-06-19T05:59:28.913251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# LOAD XGB LIBRARY\nfrom sklearn.model_selection import StratifiedKFold\nimport xgboost as xgb\nprint('XGB Version',xgb.__version__)\n\n# XGB MODEL PARAMETERS\nxgb_parms = { \n 'max_depth':5, \n 'learning_rate':0.05, \n 'subsample':0.8,\n 'colsample_bytree':0.6, \n 'eval_metric':'logloss',\n 'objective':'binary:logistic',\n 'tree_method':'gpu_hist',\n 'predictor':'gpu_predictor',\n 'random_state':SEED\n}","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:28.916724Z","iopub.execute_input":"2022-06-19T05:59:28.917735Z","iopub.status.idle":"2022-06-19T05:59:29.072019Z","shell.execute_reply.started":"2022-06-19T05:59:28.917691Z","shell.execute_reply":"2022-06-19T05:59:29.070929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# NEEDED WITH DeviceQuantileDMatrix BELOW\nclass IterLoadForDMatrix(xgb.core.DataIter):\n def __init__(self, df=None, features=None, target=None, batch_size=256*1024):\n self.features = features\n self.target = target\n self.df = df\n self.it = 0 # set iterator to 0\n self.batch_size = batch_size\n self.batches = int( np.ceil( len(df) / self.batch_size ) )\n super().__init__()\n\n def reset(self):\n '''Reset the iterator'''\n self.it = 0\n\n def next(self, input_data):\n '''Yield next batch of data.'''\n if self.it == self.batches:\n return 0 # Return 0 when there's no more batch.\n \n a = self.it * self.batch_size\n b = min( (self.it + 1) * self.batch_size, len(self.df) )\n dt = cudf.DataFrame(self.df.iloc[a:b])\n input_data(data=dt[self.features], label=dt[self.target]) #, weight=dt['weight'])\n self.it += 1\n return 1","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:29.073571Z","iopub.execute_input":"2022-06-19T05:59:29.074428Z","iopub.status.idle":"2022-06-19T05:59:29.086243Z","shell.execute_reply.started":"2022-06-19T05:59:29.074385Z","shell.execute_reply":"2022-06-19T05:59:29.085244Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# https://www.kaggle.com/kyakovlev\n# https://www.kaggle.com/competitions/amex-default-prediction/discussion/327534\ndef amex_metric_mod(y_true, y_pred):\n\n labels = np.transpose(np.array([y_true, y_pred]))\n labels = labels[labels[:, 1].argsort()[::-1]]\n weights = np.where(labels[:,0]==0, 20, 1)\n cut_vals = labels[np.cumsum(weights) <= int(0.04 * np.sum(weights))]\n top_four = np.sum(cut_vals[:,0]) / np.sum(labels[:,0])\n\n gini = [0,0]\n for i in [1,0]:\n labels = np.transpose(np.array([y_true, y_pred]))\n labels = labels[labels[:, i].argsort()[::-1]]\n weight = np.where(labels[:,0]==0, 20, 1)\n weight_random = np.cumsum(weight / np.sum(weight))\n total_pos = np.sum(labels[:, 0] * weight)\n cum_pos_found = np.cumsum(labels[:, 0] * weight)\n lorentz = cum_pos_found / total_pos\n gini[i] = np.sum((lorentz - weight_random) * weight)\n\n return 0.5 * (gini[1]/gini[0] + top_four)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:29.087653Z","iopub.execute_input":"2022-06-19T05:59:29.088123Z","iopub.status.idle":"2022-06-19T05:59:29.104119Z","shell.execute_reply.started":"2022-06-19T05:59:29.088085Z","shell.execute_reply":"2022-06-19T05:59:29.103125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"importances = []\noof = []\ntrain = train.to_pandas() # free GPU memory\nTRAIN_SUBSAMPLE = 1.0\ngc.collect()\n\nskf = StratifiedKFold(n_splits=FOLDS, shuffle=True, random_state=SEED)\nfor fold,(train_idx, valid_idx) in enumerate(skf.split(\n train, train.target )):\n \n # TRAIN WITH SUBSAMPLE OF TRAIN FOLD DATA\n if TRAIN_SUBSAMPLE<1.0:\n np.random.seed(SEED)\n train_idx = np.random.choice(train_idx, \n int(len(train_idx)*TRAIN_SUBSAMPLE), replace=False)\n np.random.seed(None)\n \n print('#'*25)\n print('### Fold',fold+1)\n print('### Train size',len(train_idx),'Valid size',len(valid_idx))\n print(f'### Training with {int(TRAIN_SUBSAMPLE*100)}% fold data...')\n print('#'*25)\n \n # TRAIN, VALID, TEST FOR FOLD K\n Xy_train = IterLoadForDMatrix(train.loc[train_idx], FEATURES, 'target')\n X_valid = train.loc[valid_idx, FEATURES]\n y_valid = train.loc[valid_idx, 'target']\n \n dtrain = xgb.DeviceQuantileDMatrix(Xy_train, max_bin=256)\n dvalid = xgb.DMatrix(data=X_valid, label=y_valid)\n \n # TRAIN MODEL FOLD K\n model = xgb.train(xgb_parms, \n dtrain=dtrain,\n evals=[(dtrain,'train'),(dvalid,'valid')],\n num_boost_round=9999,\n early_stopping_rounds=100,\n verbose_eval=100) \n model.save_model(f'XGB_v{VER}_fold{fold}.xgb')\n \n # GET FEATURE IMPORTANCE FOR FOLD K\n dd = model.get_score(importance_type='weight')\n df = pd.DataFrame({'feature':dd.keys(),f'importance_{fold}':dd.values()})\n importances.append(df)\n \n # INFER OOF FOLD K\n oof_preds = model.predict(dvalid)\n acc = amex_metric_mod(y_valid.values, oof_preds)\n print('Kaggle Metric =',acc,'\\n')\n \n # SAVE OOF\n df = train.loc[valid_idx, ['customer_ID','target'] ].copy()\n df['oof_pred'] = oof_preds\n oof.append( df )\n \n del dtrain, Xy_train, dd, df\n del X_valid, y_valid, dvalid, model\n _ = gc.collect()\n \nprint('#'*25)\noof = pd.concat(oof,axis=0,ignore_index=True).set_index('customer_ID')\nacc = amex_metric_mod(oof.target.values, oof.oof_pred.values)\nprint('OVERALL CV Kaggle Metric =',acc)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T05:59:29.10564Z","iopub.execute_input":"2022-06-19T05:59:29.106086Z","iopub.status.idle":"2022-06-19T06:07:54.354764Z","shell.execute_reply.started":"2022-06-19T05:59:29.106027Z","shell.execute_reply":"2022-06-19T06:07:54.353827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CLEAN RAM\ndel train\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:07:54.356072Z","iopub.execute_input":"2022-06-19T06:07:54.357608Z","iopub.status.idle":"2022-06-19T06:07:54.59521Z","shell.execute_reply.started":"2022-06-19T06:07:54.357564Z","shell.execute_reply":"2022-06-19T06:07:54.594185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"oof_xgb = pd.read_parquet(TRAIN_PATH, columns=['customer_ID']).drop_duplicates()\noof_xgb['customer_ID_hash'] = oof_xgb['customer_ID'].apply(lambda x: int(x[-16:],16) ).astype('int64')\noof_xgb = oof_xgb.set_index('customer_ID_hash')\noof_xgb = oof_xgb.merge(oof, left_index=True, right_index=True)\noof_xgb = oof_xgb.sort_index().reset_index(drop=True)\noof_xgb.to_csv(f'oof_xgb_v{VER}.csv',index=False)\noof_xgb.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:07:54.599575Z","iopub.execute_input":"2022-06-19T06:07:54.599936Z","iopub.status.idle":"2022-06-19T06:07:59.5496Z","shell.execute_reply.started":"2022-06-19T06:07:54.599903Z","shell.execute_reply":"2022-06-19T06:07:59.548551Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PLOT OOF PREDICTIONS\nplt.hist(oof_xgb.oof_pred.values, bins=100)\nplt.title('OOF Predictions')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:07:59.551169Z","iopub.execute_input":"2022-06-19T06:07:59.551573Z","iopub.status.idle":"2022-06-19T06:07:59.908004Z","shell.execute_reply.started":"2022-06-19T06:07:59.551535Z","shell.execute_reply":"2022-06-19T06:07:59.907177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CLEAR VRAM, RAM FOR INFERENCE BELOW\ndel oof_xgb, oof\n_ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:07:59.909558Z","iopub.execute_input":"2022-06-19T06:07:59.909964Z","iopub.status.idle":"2022-06-19T06:08:00.078129Z","shell.execute_reply.started":"2022-06-19T06:07:59.909924Z","shell.execute_reply":"2022-06-19T06:08:00.076923Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\ndf = importances[0].copy()\nfor k in range(1,FOLDS): \n df = df.merge(importances[k], on='feature', how='left')\ndf['importance'] = df.iloc[:,1:].mean(axis=1)\ndf = df.sort_values('importance',ascending=False)\ndf.to_csv(f'xgb_feature_importance_v{VER}.csv',index=False)","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:08:00.080084Z","iopub.execute_input":"2022-06-19T06:08:00.080539Z","iopub.status.idle":"2022-06-19T06:08:00.113874Z","shell.execute_reply.started":"2022-06-19T06:08:00.080488Z","shell.execute_reply":"2022-06-19T06:08:00.113026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"NUM_FEATURES = 20\nplt.figure(figsize=(10,5*NUM_FEATURES//10))\nplt.barh(np.arange(NUM_FEATURES,0,-1), df.importance.values[:NUM_FEATURES])\nplt.yticks(np.arange(NUM_FEATURES,0,-1), df.feature.values[:NUM_FEATURES])\nplt.title(f'XGB Feature Importance - Top {NUM_FEATURES}')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:08:00.115673Z","iopub.execute_input":"2022-06-19T06:08:00.11649Z","iopub.status.idle":"2022-06-19T06:08:00.393026Z","shell.execute_reply.started":"2022-06-19T06:08:00.116444Z","shell.execute_reply":"2022-06-19T06:08:00.392193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# CALCULATE SIZE OF EACH SEPARATE TEST PART\ndef get_rows(customers, test, NUM_PARTS = 8, verbose = ''):\n chunk = len(customers)//NUM_PARTS\n if verbose != '':\n print(f'We will process {verbose} data as {NUM_PARTS} separate parts.')\n print(f'There will be {chunk} customers in each part (except the last part).')\n print('Below are number of rows in each part:')\n rows = []\n\n for k in range(NUM_PARTS):\n if k==NUM_PARTS-1: cc = customers[k*chunk:]\n else: cc = customers[k*chunk:(k+1)*chunk]\n s = test.loc[test.customer_ID.isin(cc)].shape[0]\n rows.append(s)\n if verbose != '': print( rows )\n return rows,chunk\n\n# COMPUTE SIZE OF 4 PARTS FOR TEST DATA\nNUM_PARTS = 16\nTEST_PATH = '../input/amex-data-integer-dtypes-parquet-format/test.parquet'\n\nprint(f'Reading test data...')\ntest = read_file(path = TEST_PATH, usecols = ['customer_ID','S_2'])\ncustomers = test[['customer_ID']].drop_duplicates().sort_index().values.flatten()\nrows,num_cust = get_rows(customers, test[['customer_ID']], NUM_PARTS = NUM_PARTS, verbose = 'test')","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:08:00.394609Z","iopub.execute_input":"2022-06-19T06:08:00.395244Z","iopub.status.idle":"2022-06-19T06:08:04.30882Z","shell.execute_reply.started":"2022-06-19T06:08:00.395205Z","shell.execute_reply":"2022-06-19T06:08:04.307871Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# INFER TEST DATA IN PARTS\nskip_rows = 0\nskip_cust = 0\ntest_preds = []\n\nfor k in range(NUM_PARTS):\n \n # READ PART OF TEST DATA\n print(f'\\nReading test data...')\n test = read_file(path = TEST_PATH)\n test = test.iloc[skip_rows:skip_rows+rows[k]]\n skip_rows += rows[k]\n print(f'=> Test part {k+1} has shape', test.shape )\n \n # PROCESS AND FEATURE ENGINEER PART OF TEST DATA\n test = process_and_feature_engineer(test)\n if k==NUM_PARTS-1: test = test.loc[customers[skip_cust:]]\n else: test = test.loc[customers[skip_cust:skip_cust+num_cust]]\n skip_cust += num_cust\n \n # TEST DATA FOR XGB\n X_test = test[FEATURES]\n dtest = xgb.DMatrix(data=X_test)\n test = test[['P_2_mean']] # reduce memory\n del X_test\n gc.collect()\n\n # INFER XGB MODELS ON TEST DATA\n model = xgb.Booster()\n model.load_model(f'XGB_v{VER}_fold0.xgb')\n preds = model.predict(dtest)\n for f in range(1,FOLDS):\n model.load_model(f'XGB_v{VER}_fold{f}.xgb')\n preds += model.predict(dtest)\n preds /= FOLDS\n test_preds.append(preds)\n\n # CLEAN MEMORY\n del dtest, model\n _ = gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:08:04.310355Z","iopub.execute_input":"2022-06-19T06:08:04.310929Z","iopub.status.idle":"2022-06-19T06:11:38.840996Z","shell.execute_reply.started":"2022-06-19T06:08:04.310886Z","shell.execute_reply":"2022-06-19T06:11:38.840153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# WRITE SUBMISSION FILE\ntest_preds = np.concatenate(test_preds)\ntest = cudf.DataFrame(index=customers,data={'prediction':test_preds})\nsub = cudf.read_csv('../input/amex-default-prediction/sample_submission.csv')[['customer_ID']]\nsub['customer_ID_hash'] = sub['customer_ID'].str[-16:].str.hex_to_int().astype('int64')\nsub = sub.set_index('customer_ID_hash')\nsub = sub.merge(test[['prediction']], left_index=True, right_index=True, how='left')\nsub = sub.reset_index(drop=True)\n\n# DISPLAY PREDICTIONS\nsub.to_csv(f'submission_xgb_v{VER}.csv',index=False)\nprint('Submission file shape is', sub.shape )\nsub.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:11:38.842831Z","iopub.execute_input":"2022-06-19T06:11:38.843208Z","iopub.status.idle":"2022-06-19T06:11:40.026776Z","shell.execute_reply.started":"2022-06-19T06:11:38.843178Z","shell.execute_reply":"2022-06-19T06:11:40.025905Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# PLOT PREDICTIONS\nplt.hist(sub.to_pandas().prediction, bins=100)\nplt.title('Test Predictions')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-19T06:11:40.028176Z","iopub.execute_input":"2022-06-19T06:11:40.0296Z","iopub.status.idle":"2022-06-19T06:11:40.765034Z","shell.execute_reply.started":"2022-06-19T06:11:40.029559Z","shell.execute_reply":"2022-06-19T06:11:40.764186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Tips and Tricks for increasing Score in Leaderboard\nIncreasing score sometimes tricky. However, during my experimentations to compete in a few competitions, there are a few tips and tricks that you can follow even sometimes a little bit increase of the score. These are all the methods i found during my participations in a a few competitions. \n\n1. Change the variety of SEED. You can take a look at the paper [here](https://arxiv.org/pdf/2109.08203.pdf) why changing SEED can impact the score\n2. Update HyperParameter tuning\n3. Check the target value whether it is imbalanced or not. if it is, you can use StratifiedKFold for balancing the proportion between the labels.\n4. ENSEMBLING. I found Ensembling method tends to increase score. It depends on the model that would like to join. You have to consider to choose the uncorrelated model in order to get higher score. If it is correlated model, the model will learn the same pattern from the data that can not detect the advantages and the weaknesses. Hence, it wont make average score better. You can check this great [blog post](https://web.archive.org/web/20160304031055/http://mlwave.com/kaggle-ensembling-guide/) for implementing it.\n5. Will continue to add","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}