{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"from IPython.core.display import HTML\nHTML(\"\"\"\n\n\n
Pop–Art Diamond Bank Card Design
\n\"\"\")","metadata":{"execution":{"iopub.status.busy":"2022-10-04T08:55:22.597087Z","iopub.execute_input":"2022-10-04T08:55:22.597656Z","iopub.status.idle":"2022-10-04T08:55:22.630591Z","shell.execute_reply.started":"2022-10-04T08:55:22.597556Z","shell.execute_reply":"2022-10-04T08:55:22.629399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"## Table of Contents:**","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> **Table of Contents:**\n> * [🚫 Introduction on Payment Default](#1)\n> * [πŸ™ Credits](#2)\n> * [πŸ›  Feature Engineering](#3)\n> * [πŸ₯’ Feature Selection](#4)\n> * [πŸ€– Model Training](#5)\n> * [πŸ… Feature Importance](#6)\n> * [πŸ“ˆ Cross-Val Roc Curves](#7)\n> * [🀞 Predict Test data & Submission](#8)\n> ---","metadata":{}},{"cell_type":"markdown","source":" \n#

🚫 Introduction on Payment Default

\nEven though credit cards presents many advantages such us avoiding carrying a bulky wallet in your pocket, tracking the spending behaviour, fraud detection... On the other hand the major downside for credit card usage is the increasing tendency to default on their payments. Aggressive marketing strategies can encourage credit card use beyond payment capacity, thus increasing the bearer’s credit risk and resulting in defaults and losses that might have not been properly anticipated\n\n \n#

πŸ™ Credits

\n In this notebook we build and train a LightGBM model using [@raddar'dataset](https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format) (discussion [here](https://www.kaggle.com/competitions/amex-default-prediction/discussion/328514) and my notebook explanation [here](https://www.kaggle.com/code/schopenhacker75/data-deanonymization) )\n\nThe feature engineering part is very inspired from [this notbook](https://www.kaggle.com/code/huseyincot/amex-agg-data-how-it-created) and the design aspect s from [my previous notebook](https://www.kaggle.com/code/schopenhacker75/fancy-complete-eda/notebook) whose it self ispired from [this notebook](https://www.kaggle.com/code/kellibelcher/amex-default-prediction-eda-lgbm-baseline/notebook). The LightGBM wrapper implementation is inspired from [jayjay's notebook](https://www.kaggle.com/code/jayjay75/wids2020-lgb-starter-adversarial-validation)\n","metadata":{}},{"cell_type":"code","source":"## ESSENTIALS\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nimport gc\nimport sys\nimport pickle\nimport glob\nimport pandas as pd\nfrom sklearn.preprocessing import LabelEncoder\n\npd.set_option('display.max_columns', None)\nimport random\nrandom.seed(75)\nfrom tqdm.notebook import tqdm_notebook\n\nfrom functools import partial, reduce\nimport datetime\nimport time\n\n\n### warnings setting\nimport sys\nimport warnings\nif not sys.warnoptions:\n warnings.simplefilter(\"ignore\")\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning)\n\n#### model\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.metrics import roc_auc_score, roc_curve, auc\nimport catboost\nfrom catboost import Pool, CatBoostClassifier \nimport lightgbm as lgb\nimport joblib\nimport pickle\nfrom tqdm.notebook import tqdm_notebook \nimport uuid\n\n##### LOGGING Stettings #####\nimport logging\n# Create logger\nlogger = logging.getLogger()\nlogger.setLevel(logging.INFO)\n# Create STDERR handler\nhandler = logging.StreamHandler(sys.stderr)\n# Create formatter and add it to the handler\nformatter = logging.Formatter('%(asctime)s [%(levelname)s] %(name)s - %(message)s', datefmt='%Y-%m-%d %H:%M:%S',)\nhandler.setFormatter(formatter)\n# Set STDERR handler as the only handler \nlogger.handlers = [handler]\n\n#### plots\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport matplotlib.colors\n\nsns.set(rc={'axes.facecolor':'#f9ecec', 'figure.facecolor':'#f9ecec'})\n\nimport plotly.express as px\nimport plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\nfrom plotly.offline import init_notebook_mode\n\n### Plotly settings\ntheme_palette={\n 'base': '#a3d3eb',\n 'complementary':'#ebbba3',\n 'triadic' : '#eba3d3', \n 'backgound' : '#f6fbfd' \n}\n\ntemp=dict(layout=go.Layout(font=dict(family=\"Ubuntu\", size=14), \n height=600, \n legend=dict(#traceorder='reversed',\n orientation=\"v\",\n y=1.15,\n x=0.9),\n plot_bgcolor = theme_palette['backgound'],\n paper_bgcolor = theme_palette['backgound']))\n\n\nSAVED=True\nINTERIM_DATA = \"../input/interim-data-v2\"\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-09-30T14:56:50.081903Z","iopub.execute_input":"2022-09-30T14:56:50.082431Z","iopub.status.idle":"2022-09-30T14:56:52.577059Z","shell.execute_reply.started":"2022-09-30T14:56:50.082331Z","shell.execute_reply":"2022-09-30T14:56:52.575920Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n#

πŸ›  Feature Engineering

\n\nTo Save the exectution time all the Data Processing part has been caaried out upstream, and stored [here](https://www.kaggle.com/datasets/schopenhacker75/interim-data-v2): we will get directly the prepared [train](https://www.kaggle.com/datasets/schopenhacker75/interim-data-v2?select=train_prepared.ftr) and [test](https://www.kaggle.com/datasets/schopenhacker75/interim-data-v2?select=test_prepared.ftr) datasets ","metadata":{}},{"cell_type":"markdown","source":"

πŸ‘‰ -- Get Train Data --

","metadata":{}},{"cell_type":"code","source":"if not(SAVED):\n train = pd.read_parquet('/kaggle/input/amex-data-integer-dtypes-parquet-format/train.parquet')\n print(f\"Shape = {train.shape}, number of customers = {train['customer_ID'].nunique()}\")\n gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:10:45.595986Z","iopub.execute_input":"2022-09-30T04:10:45.597924Z","iopub.status.idle":"2022-09-30T04:10:45.604959Z","shell.execute_reply.started":"2022-09-30T04:10:45.597853Z","shell.execute_reply":"2022-09-30T04:10:45.603341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

πŸ‘‰ -- Train Data Feature Engineering --

","metadata":{}},{"cell_type":"markdown","source":"I chose to refer to **[Martin's](https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977/notebook) excellent work**. Because it is simple to implement and efficient\n\nThe idea is to aggreagate data by `customer_ID` and apply some statistical functions\n\n

🫐 a - Numerical features aggregation

\n\nFor each customer ID numerical data we retain:\n* `mean`: the mean of numerical features\n* `std` : standard deviation of each numerical features\n* `min` and `max`: the minimal and maximal values of each numerical feature\n* `last` : the last values\n\n\n

🫐 b - Categorical features aggregation

\n\nFor each customer ID categorical data we retain:\n* `count` the count of transactions\n* `last` : the last values\n* `number` :number of unique values \n\n

🫐 c - Difference based features

\n\nIt [has been shown](http://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977/notebook) that these features carry a powerful predictive signal:\n* the difference between the last transaction and the second of last\n* the difference between the last transaction and the mean","metadata":{}},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:10:45.607054Z","iopub.execute_input":"2022-09-30T04:10:45.607654Z","iopub.status.idle":"2022-09-30T04:10:45.838775Z","shell.execute_reply.started":"2022-09-30T04:10:45.607598Z","shell.execute_reply":"2022-09-30T04:10:45.837287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977\nif not SAVED:\n features = train.drop(['customer_ID', 'S_2'], axis = 1).columns.to_list()\n\n cat_features = [\n \"B_30\",\n \"B_38\",\n \"D_114\",\n \"D_116\",\n \"D_117\",\n \"D_120\",\n \"D_126\",\n \"D_63\",\n \"D_64\",\n \"D_66\",\n \"D_68\",\n ]\n num_features = [col for col in features if col not in cat_features]\n\n with open('features.pkl', 'wb') as f:\n pickle.dump(features, f)\n\n with open('cat_cols.pkl', 'wb') as f:\n pickle.dump(cat_features, f)\n\n with open('num_cols.pkl', 'wb') as f:\n pickle.dump(num_features, f)\nelse:\n with open(os.path.join(INTERIM_DATA, 'features.pkl'), 'rb') as f:\n features = pickle.load(f)\n\n with open(os.path.join(INTERIM_DATA, 'cat_cols.pkl'), 'rb') as f:\n cat_features = pickle.load(f)\n\n with open(os.path.join(INTERIM_DATA, 'num_cols.pkl'), 'rb') as f:\n num_features = pickle.load(f)\n \n#","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:10:45.841591Z","iopub.execute_input":"2022-09-30T04:10:45.842064Z","iopub.status.idle":"2022-09-30T04:10:45.874822Z","shell.execute_reply.started":"2022-09-30T04:10:45.842012Z","shell.execute_reply":"2022-09-30T04:10:45.873102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To address the OOM issues, I inplemented **a batch generator** to apply the data processing batch by batch then cancatenate it all. Please note that most of the used aggregation functions are grouped by customer_ID, **so all the statements of a given client must be grouped on a single batch.**","metadata":{}},{"cell_type":"code","source":"\nclass BatchGenerator:\n def __init__(self, df, batch_feature='customer_ID', keep_features=features, n_batchs=750):\n self.df = df\n self.batch_feature = batch_feature\n self.keep_feature = list(set([batch_feature]+keep_features))\n self.n_batchs = n_batchs\n \n def __iter__(self):\n unique_vals = self.df[self.batch_feature].unique()\n batch_size = int(np.ceil(len(unique_vals) / self.n_batchs))\n groups = self.df.groupby(self.batch_feature).groups\n n_batchs= min(self.n_batchs, int(np.ceil(len(unique_vals) / batch_size)))\n for i in range(n_batchs):\n keys = unique_vals[i*batch_size:(i+1)*batch_size]\n idx=[i for s in keys for i in groups[s] ]\n if i == n_batchs-1:\n keys = unique_vals[(i+1)*batch_size:]\n idx = idx + [i for s in keys for i in groups[s] ]\n yield self.df.loc[idx, self.keep_feature]\n \n \n# from https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977\ndef get_difference(data, num_features):\n df1 = []\n customer_ids = []\n for customer_id, df in tqdm_notebook(data.groupby(['customer_ID'])):\n # Get the differences\n diff_df1 = df[num_features].diff(1).iloc[[-1]].values.astype(np.float32)\n # Append to lists\n df1.append(diff_df1)\n customer_ids.append(customer_id)\n # Concatenate\n df1 = np.concatenate(df1, axis = 0)\n # Transform to dataframe\n df1 = pd.DataFrame(df1, columns = [col + '_diff1' for col in df[num_features].columns])\n # Add customer id\n df1['customer_ID'] = customer_ids\n return df1\n\ndef feature_engineering(df, data_type='train'):\n # num features\n print(\"num features aggregation\")\n df_num_agg = df.groupby(\"customer_ID\")[num_features].agg(['mean', 'std', 'min', 'max', 'last'])\n df_num_agg.columns = ['_'.join(x) for x in df_num_agg.columns]\n df_num_agg = df_num_agg.reset_index()\n # cat features\n print(\"cat features aggregation\")\n df_cat_agg = df.groupby(\"customer_ID\")[cat_features].agg(['count', 'last', 'nunique'])\n df_cat_agg.columns = ['_'.join(x) for x in df_cat_agg.columns]\n df_cat_agg = df_cat_agg.reset_index()\n \n # Transform float64 columns to float32\n print(\"reduce float data size\")\n cols = list(df_num_agg.dtypes[df_num_agg.dtypes == 'float64'].index)\n for col in tqdm_notebook(cols):\n df_num_agg[col] = df_num_agg[col].astype(np.float32)\n # Transform int64 columns to int32\n print(\"reduce cat data size\")\n cols = list(df_cat_agg.dtypes[df_cat_agg.dtypes == 'int64'].index)\n for col in tqdm_notebook(cols):\n df_cat_agg[col] = df_cat_agg[col].astype(np.int32)\n # Get the difference\n print(\"get diff between last and second of last transaction\")\n df_diff = get_difference(df, num_features)\n # merge all\n print(\"merge all\")\n df = df_num_agg.merge(df_cat_agg, how = 'inner', on = 'customer_ID').merge(df_diff, how = 'inner', on = 'customer_ID')\n if data_type=='train':\n print(\"get labels\")\n train_labels = pd.read_csv('../input/amex-default-prediction/train_labels.csv')\n df = df.merge(train_labels, how = 'inner', on = 'customer_ID')\n del train_labels\n del df_num_agg, df_cat_agg, df_diff\n \n # Round last float features to 2 decimal place\n num_cols = list(df.dtypes[(df.dtypes == 'float32') | (df.dtypes == 'float64')].index)\n num_cols = [col for col in num_cols if 'last' in col]\n print(\"Round last float features to 2 decimal place\")\n for col in num_cols:\n df[col + '_round2'] = df[col].round(2)\n # Get the difference between last and mean\n num_cols = list(filter(lambda x: 'last' in x, df))\n num_cols = [col[:-5] for col in num_cols if 'round' not in col]\n print(\"diff between last and mean transaction\")\n for col in tqdm_notebook(num_cols):\n try:\n df[f'{col}_last_mean_diff'] = df[f'{col}_last'] - df[f'{col}_mean']\n except:\n pass\n # Transform float64 and float32 to float16\n num_cols = list(df.dtypes[(df.dtypes == 'float32') | (df.dtypes == 'float64')].index)\n print(\"reduce float data size\")\n for col in tqdm_notebook(num_cols):\n df[col] = df[col].astype(np.float16)\n df.to_feather(f\"{data_type}_{str(uuid.uuid4())}.ftr\")\n return df['customer_ID'].nunique()\n\n# label encoding dor categorical features\ndef label_encoding(df, train=True):\n cat_lasts = [f\"{cf}_last\" for cf in cat_features]\n for cat_col in tqdm_notebook(cat_lasts):\n print(f\"{cat_col} label encoding\")\n if train:\n le = LabelEncoder()\n df[cat_col] = le.fit_transform(df[cat_col])\n joblib.dump(le, f'{cat_col}_encoder.pkl')\n else:\n le = joblib.load(os.path.join('../input/interim-data-v2', f'{cat_col}_encoder.pkl'))\n df[cat_col] = le.transform(test[cat_col])\n return df\n","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:10:46.428089Z","iopub.execute_input":"2022-09-30T04:10:46.428871Z","iopub.status.idle":"2022-09-30T04:10:46.462554Z","shell.execute_reply.started":"2022-09-30T04:10:46.428790Z","shell.execute_reply":"2022-09-30T04:10:46.461110Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:10:46.990824Z","iopub.execute_input":"2022-09-30T04:10:46.991323Z","iopub.status.idle":"2022-09-30T04:10:47.214496Z","shell.execute_reply.started":"2022-09-30T04:10:46.991285Z","shell.execute_reply":"2022-09-30T04:10:47.212955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if not SAVED:\n N_CID = train['customer_ID'].nunique()\n samples_df = BatchGenerator(train, feature='customer_ID', n_batchs=50)\n processed_elements = sum(map(partial(feature_engineering, data_type='train'), tqdm_notebook(samples_df, total=50)))\n assert processed_elements==N_CID\n del train\n gc.collect()\n train = pd.DataFrame()\n for path in tqdm_notebook(glob.glob('train_*.ftr')):\n train = pd.concat([train, pd.read_feather(path)])\n os.remove(path)\n\n\n train = label_encoding(train, train=True)\n train.reset_index(drop=True).to_feather(\"train_prepared.ftr\")\n\nelse: \n train = pd.read_feather(os.path.join(INTERIM_DATA, \"train_prepared.ftr\"))","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:10:49.554787Z","iopub.execute_input":"2022-09-30T04:10:49.555266Z","iopub.status.idle":"2022-09-30T04:10:57.991133Z","shell.execute_reply.started":"2022-09-30T04:10:49.555230Z","shell.execute_reply":"2022-09-30T04:10:57.989821Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-30T14:57:06.186055Z","iopub.execute_input":"2022-09-30T14:57:06.186477Z","iopub.status.idle":"2022-09-30T14:57:06.351691Z","shell.execute_reply.started":"2022-09-30T14:57:06.186439Z","shell.execute_reply":"2022-09-30T14:57:06.350163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

πŸ‘‰ -- Test Data Feature Engineering --

\n\nApply the same pipeline to the test dataset with some differences:\n* The label merging part is omitted (set `data_type` parameter of the `feature_engineering` function to 'test')\n* Apply the pre-fitted label encoders on the training dataset (set `train` parameter of the `label_encoding` function to **False**)","metadata":{}},{"cell_type":"code","source":"INFER = False\nif not SAVED:\n if INFER:\n test = pd.read_parquet(\"../input/amex-data-integer-dtypes-parquet-format/test.parquet\")\n N_CID = test['customer_ID'].nunique()\n\n samples_df = BatchGenerator(test, batch_feature='customer_ID', n_batchs=200)\n processed_elements = sum(map(partial(feature_engineering, data_type='test'), tqdm_notebook(samples_df, total=200)))\n assert processed_elements==N_CID\n \n del test\n gc.collect()\n\n\n test = pd.DataFrame()\n for path in tqdm_notebook(glob.glob('test_*.ftr')):\n test = pd.concat([test, pd.read_feather(path)])\n os.remove(path)\n\n test = label_encoding(test, train=False)\n test.reset_index(drop=True).to_feather(\"test_prepared.ftr\")\n\n else:\n test = pd.read_feather(os.path.join(INTERIM_DATA, \"test_prepared.ftr\"))\n","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:02.018633Z","iopub.execute_input":"2022-09-30T04:11:02.019121Z","iopub.status.idle":"2022-09-30T04:11:02.030601Z","shell.execute_reply.started":"2022-09-30T04:11:02.019085Z","shell.execute_reply":"2022-09-30T04:11:02.028999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n#

πŸ₯’ Feature Selection

\nFor the feature selection part I used some **filter based techniques** because these methods are faster and less computationally expensive than other feature selection methods such as wrapper methods. Basically:\n* Drop Features with high missing values rate\n* Keep the TOP target correlated features\n\n

πŸ‘‰ -- Drop Features with high missing rate --

\n\nAll features having **more than 75% of missing values are ommited**","metadata":{}},{"cell_type":"code","source":"def missing_values_table(df):\n # Total missing values by column\n mis_val = df.isnull().sum()\n\n # Percentage of missing values by column\n mis_val_percent = 100 * df.isnull().sum() / len(df)\n\n # build a table with the thw columns\n mis_val_table = pd.concat([mis_val, mis_val_percent], axis=1)\n\n # Rename the columns\n mis_val_table_ren_columns = mis_val_table.rename(\n columns = {0 : 'Missing Values', 1 : '% of Total Values'})\n\n # Sort the table by percentage of missing descending\n mis_val_table_ren_columns = mis_val_table_ren_columns[\n mis_val_table_ren_columns.iloc[:,1] != 0].sort_values(\n '% of Total Values', ascending=False).round(1)\n\n # Print some summary information\n print (\"Your selected dataframe has \" + str(df.shape[1]) + \" columns.\\n\" \n \"There are \" + str(mis_val_table_ren_columns.shape[0]) +\n \" columns that have missing values.\")\n\n # Return the dataframe with missing information\n return mis_val_table_ren_columns\n\n# Missing values for training data\nmissing_values_train = missing_values_table(train)\n#cm = sns.color_palette('Set2', as_cmap=True)\n#missing_values_train[:20]#.style.background_gradient(cmap=cm)\nTHRESHOLD = 75\ndrop_cols = missing_values_train[missing_values_train[ '% of Total Values']>THRESHOLD].index.to_list()\nprint(f\"Drop {len(drop_cols)} features with more than {THRESHOLD}% of missing values\")\ntrain = train.drop(drop_cols, axis=1)\nprint(\"Training data shape after dropping highly missing values columns\", train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:03.871778Z","iopub.execute_input":"2022-09-30T04:11:03.872321Z","iopub.status.idle":"2022-09-30T04:11:12.615789Z","shell.execute_reply.started":"2022-09-30T04:11:03.872277Z","shell.execute_reply":"2022-09-30T04:11:12.614521Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(drop_cols)","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:12.617943Z","iopub.execute_input":"2022-09-30T04:11:12.618679Z","iopub.status.idle":"2022-09-30T04:11:12.626026Z","shell.execute_reply.started":"2022-09-30T04:11:12.618633Z","shell.execute_reply":"2022-09-30T04:11:12.624571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n

πŸ‘‰ -- Select top correlated features with the target--

\n\nWith this method we assume that high predictive features are **highly correlated with the target**\n\n","metadata":{}},{"cell_type":"code","source":"corr = train.corrwith(train['target'], axis=0)\ncorr = corr[corr.notna()].sort_values(key=abs, ascending=False)\nTHRESHOLD = 0.15\nCORR_SELECTION=False\nif CORR_SELECTION:\n selected_feats = corr[corr.abs()>THRESHOLD].index\n train = train[list(selected_feats)]\n print(f\"Training data shape after dropping uncorrelated features\"\n f\"(threshold Pearson correlation = {THRESHOLD})\", \n train.shape)\n\n \n","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:14.652275Z","iopub.execute_input":"2022-09-30T04:11:14.652798Z","iopub.status.idle":"2022-09-30T04:11:30.979403Z","shell.execute_reply.started":"2022-09-30T04:11:14.652758Z","shell.execute_reply":"2022-09-30T04:11:30.978034Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:30.982018Z","iopub.execute_input":"2022-09-30T04:11:30.982615Z","iopub.status.idle":"2022-09-30T04:11:31.205280Z","shell.execute_reply.started":"2022-09-30T04:11:30.982548Z","shell.execute_reply":"2022-09-30T04:11:31.203470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's display the top correlated features to the target ","metadata":{}},{"cell_type":"code","source":"sorted_corr = corr.sort_values(key=abs, ascending=False)[:11] # top but we have to drop corr=1\n\n\npos = sorted_corr[(sorted_corr>0) & (sorted_corr<1)]\nneg = sorted_corr[(sorted_corr<0)].sort_values(ascending=False)\n\nfig = go.Figure()\nfig.add_trace(go.Bar(x=pos.index, y= pos.values,\n orientation='v',\n name='Positive',\n marker=dict(color=theme_palette['base'],line=dict(color=theme_palette['complementary'],width=0)),\n text = [\"%.2f\" %(round(v ,2) *100) + '%' for v in pos.values],\n textposition = 'outside',\n textfont_color = '#212a2f'))\n\nfig.add_trace(go.Bar(x=neg.index, y= neg.values,\n orientation='v',\n name='Negative',\n marker=dict(color=theme_palette['complementary'],line=dict(color=theme_palette['base'],width=0)),\n text = [\"%.2f\" %(round(v ,2) *100) + '%' for v in neg.values],\n textposition = 'outside',\n textfont_color = '#212a2f'))\n\n#theme_palette['2']\nfig.update_layout(template = temp,\n title={\n \"text\": \"Top-10 Correlated Features with the Payment Default Feature
Pearson Values > 0.5

\",\n \"x\":0.035,\n \"font_size\": 18,\n \n },\n plot_bgcolor = theme_palette['backgound'],\n paper_bgcolor = theme_palette['backgound'],\n legend=dict(\n y=1.15,\n x=0.88))","metadata":{"execution":{"iopub.status.busy":"2022-09-28T10:17:08.694985Z","iopub.execute_input":"2022-09-28T10:17:08.695315Z","iopub.status.idle":"2022-09-28T10:17:08.850775Z","shell.execute_reply.started":"2022-09-28T10:17:08.695289Z","shell.execute_reply":"2022-09-28T10:17:08.849637Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

πŸ‘‰ -- Store Interim Data--

\n\nDue to the redundant crashs we ought to store the used columns to use it later (it has been stored upstream in [this dataset](https://www.kaggle.com/datasets/schopenhacker75/interim-data-v2) ) ","metadata":{}},{"cell_type":"code","source":"if True:#not(SAVED):\n train=train.set_index('customer_ID')\n types = train.dtypes\n target_col = 'target'\n\n cat_cols = list(types[types.apply(lambda x:not(str(x).startswith('float')))].index)\n cat_cols = list(filter(lambda x:x!=target_col, cat_cols))\n features = list(train.drop(target_col, axis=1).columns)\n gc.collect()\n print('len cat_col', len(cat_cols))\n print('len features', len(features))\n \n with open('final_features_drop_na_cols.pkl', 'wb') as f:\n pickle.dump(features, f)\n\n with open('final_cat_cols_drop_na_cols.pkl', 'wb') as f:\n pickle.dump(cat_cols, f)\n \nelse:\n with open(os.path.join(INTERIM_DATA, 'final_features.pkl'), 'rb') as f:\n features = pickle.load(f)\n with open(os.path.join(INTERIM_DATA, 'final_cat_cols.pkl'), 'rb') as f:\n cat_cols = pickle.load(f)\n","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:31.207158Z","iopub.execute_input":"2022-09-30T04:11:31.207575Z","iopub.status.idle":"2022-09-30T04:11:36.311880Z","shell.execute_reply.started":"2022-09-30T04:11:31.207532Z","shell.execute_reply":"2022-09-30T04:11:36.310312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:36.317370Z","iopub.execute_input":"2022-09-30T04:11:36.318594Z","iopub.status.idle":"2022-09-30T04:11:36.543892Z","shell.execute_reply.started":"2022-09-30T04:11:36.318521Z","shell.execute_reply":"2022-09-30T04:11:36.542423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n#

πŸ€– Model Training

\n\n\nI implemented a gradient boosting model Wrapper **BaseModel** (inspired from [jayjay's notebook](https://www.kaggle.com/code/jayjay75/wids2020-lgb-starter-adversarial-validation)) in which I defined most of in common methods and attributes of gradient boosting based models, such as _cv training, feature importances, prediction_ ...\n**LgbModel** and **CatBoost** would inherit from the BaseModel, Later I can add XGboost, hence we get the overall Gradient Boosting models comparison.\n\n

πŸ‘‰ -- Competition Metric --

","metadata":{}},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:36.546215Z","iopub.execute_input":"2022-09-30T04:11:36.546718Z","iopub.status.idle":"2022-09-30T04:11:36.778211Z","shell.execute_reply.started":"2022-09-30T04:11:36.546677Z","shell.execute_reply":"2022-09-30T04:11:36.776833Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def amex_metric(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n #https://www.kaggle.com/code/inversion/amex-competition-metric-python\n def top_four_percent_captured(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n df = (pd.concat([y_true, y_pred], axis='columns')\n .sort_values('prediction', ascending=False))\n df['weight'] = df['target'].apply(lambda x: 20 if x == 0 else 1)\n four_pct_cutoff = int(0.04 * df['weight'].sum())\n df['weight_cumsum'] = df['weight'].cumsum()\n df_cutoff = df.loc[df['weight_cumsum'] <= four_pct_cutoff]\n return (df_cutoff['target'] == 1).sum() / (df['target'] == 1).sum()\n\n def weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n df = (pd.concat([y_true, y_pred], axis='columns')\n .sort_values('prediction', ascending=False))\n df['weight'] = df['target'].apply(lambda x: 20 if x == 0 else 1)\n df['random'] = (df['weight'] / df['weight'].sum()).cumsum()\n total_pos = (df['target'] * df['weight']).sum()\n df['cum_pos_found'] = (df['target'] * df['weight']).cumsum()\n df['lorentz'] = df['cum_pos_found'] / total_pos\n df['gini'] = (df['lorentz'] - df['random']) * df['weight']\n return df['gini'].sum()\n\n def normalized_weighted_gini(y_true: pd.DataFrame, y_pred: pd.DataFrame) -> float:\n y_true_pred = y_true.rename(columns={'target': 'prediction'})\n return weighted_gini(y_true, y_pred) / weighted_gini(y_true, y_true_pred)\n \n y_pred=pd.DataFrame(data={'prediction':y_pred})\n y_true=pd.DataFrame(data={'target':y_true.reset_index(drop=True)})\n g = normalized_weighted_gini(y_true, y_pred)\n d = top_four_percent_captured(y_true, y_pred)\n\n return 0.5 * (g + d)\n","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:11:38.554918Z","iopub.execute_input":"2022-09-30T04:11:38.555445Z","iopub.status.idle":"2022-09-30T04:11:38.572444Z","shell.execute_reply.started":"2022-09-30T04:11:38.555402Z","shell.execute_reply":"2022-09-30T04:11:38.570690Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# a wrapper class that we can have the same ouput whatever the model we choose\ndef get_partial_pred_df(fold, y_val, pred):\n df = pd.DataFrame()\n df['y_true'] = y_val\n df['y_pred'] = pred\n df['fold'] = fold\n return df[['fold','y_true', 'y_pred']]\n\nMODELS_PATH = \"../input/cv-models\"\nclass BaseModel:\n RAND_SEED = 75\n # TODO make a longer list of colors\n COLORS =['#a3d3eb', '#dfa3eb', '#ebbba3', '#afeba3', '#ebe6a3']\n def __init__(self, name, features, categoricals=[], n_splits=5, verbose=True, ps=None, target_col='target'):\n self.name = name\n self.features = features\n self.n_splits = n_splits\n self.categoricals = categoricals\n self.target = target_col\n self.cv_models = []\n self.verbose = verbose\n self.model=None\n if not ps:\n self.params = self.get_default_params()\n else:\n self.params = ps\n \n \n def train_model(self, train_set, val_set, eval_metric=[]):\n raise NotImplementedError\n \n def get_cv(self, train_df):\n cv = StratifiedKFold(n_splits=self.n_splits, shuffle=True, random_state=self.RAND_SEED)\n return cv.split(train_df, train_df[self.target])\n \n def get_default_params(self):\n raise NotImplementedError\n \n def convert_dataset(self, x_train, y_train):\n raise NotImplementedError\n \n def convert_x(self, x):\n return x\n \n def fit_cv(self, train_df, save_cv=False):\n self.oof_pred = np.zeros((len(train_df), ))\n \n cv = self.get_cv(train_df)\n self.partial_oof_scores_=[]\n self.cv_df=pd.DataFrame()\n \n for fold, (train_idx, val_idx) in enumerate(cv): \n print(\"*-\"*100)\n print(f\"*** FOLD == {fold} **\")\n print(\"*-\"*100)\n x_train, x_val = train_df[self.features].iloc[train_idx], train_df[self.features].iloc[val_idx]\n y_train, y_val = train_df[self.target][train_idx], train_df[self.target][val_idx]\n train_set = self.convert_dataset(x_train, y_train)\n val_set = self.convert_dataset(x_val, y_val)\n conv_x_val = self.convert_x(x_val)\n # OOM\n if os.path.isfile(os.path.join(MODELS_PATH, f'{self.name}_{fold}.pkl')):\n print(\"load\", f'{self.name}_{fold}.pkl')\n model = joblib.load(os.path.join(MODELS_PATH, f'{self.name}_{fold}.pkl'))\n self.cv_models.append(model)\n else: \n model = self.train_model(train_set, val_set)\n self.cv_models.append(model)\n\n self.oof_pred[val_idx] = model.predict(conv_x_val).reshape(self.oof_pred[val_idx].shape)\n \n partial_oof_score = amex_metric(y_val, self.oof_pred[val_idx])\n self.partial_oof_scores_.append(partial_oof_score)\n print('Partial score of fold {} is: {}'.format(fold, partial_oof_score))\n if save_cv: \n # save model\n joblib.dump(model, f'{self.name}_{fold}.pkl')\n \n partial_pred_df = get_partial_pred_df(fold, y_val, self.oof_pred[val_idx])\n self.cv_df = pd.concat([self.cv_df, partial_pred_df])\n\n self.oof_score_ = amex_metric(train_df[self.target], self.oof_pred) \n if self.verbose:\n print('Our oof score is: ', self.oof_score_)\n \n def fit(self, train):\n x_train = train[self.features]\n y_train = train[self.target]\n train_set = self.convert_dataset(x_train, y_train)\n self.model = self.train_model(train_set)\n print(\"model trained in all training dataset\")\n \n def predict_cv(self, test_df):\n y_pred = np.zeros((len(test_df), ))\n x_test = self.convert_x(test_df[self.features])\n for model in self.cv_models:\n y_pred += model.predict(x_test).reshape(y_pred.shape) / self.n_splits\n return y_pred\n \n def predict(self, test_df):\n x_test = self.convert_x(test_df[self.features])\n y_pred = self.model.predict(x_test).reshape((len(test_df), ))\n return y_pred\n \n def get_cv_feature_importance(self):\n raise NotImplementedError\n \n def plot_cv_roc(self): \n fig=go.Figure()\n fig.add_trace(go.Scatter(x=np.linspace(0,1,11), y=np.linspace(0,1,11), \n name='Random Chance',mode='lines', showlegend=False,\n line=dict(color=\"Black\", width=1, dash=\"dot\")))\n for fold, sample_df in self.cv_df.groupby('fold'):\n fpr, tpr, _ = roc_curve(sample_df.y_true, sample_df.y_pred)\n roc_auc = auc(fpr,tpr)\n fig.add_trace(go.Scatter(x=fpr, y=tpr, line=dict(color=self.COLORS[fold], width=3), \n hovertemplate = 'True positive rate = %{y:.3f}
False positive rate = %{x:.3f}',\n name='Fold {}: AUC = {:.3f}'.format(fold+1, roc_auc)))\n\n fig.update_layout(template = temp,\n yaxis_automargin=True,\n height = 800,\n plot_bgcolor = theme_palette['backgound'],\n paper_bgcolor = theme_palette['backgound'],\n title={\n \"text\": \"Cross-Validation ROC Curves \",\n \"x\":0.045,\n \"font_size\": 20, \n },\n margin={'pad':5},\n xaxis_title='False Positive Rate (1 - Specificity)',\n yaxis_title='True Positive Rate (Sensitivity)',\n legend=dict(orientation='v', y=.07, x=1, xanchor=\"right\",\n bordercolor=\"black\", borderwidth=.5)\n )\n\n return fig\n \n def plot_cv_feature_importance(self, top=20):\n feat_imp = self.get_cv_feature_importance()\n data = feat_imp[:top]\n threshold = data.iloc[5]['importance']\n fig = go.Figure()\n fig.add_trace(go.Bar(y=data.index, x= data['importance'],\n orientation='h',\n width=[0.6]*len(data),\n marker=dict(color=(data['importance'] < threshold).astype('int'),\n colorscale=[[0, theme_palette['complementary']], [1, theme_palette['base']]], ),\n\n text = [\"%.2f\"%(round(v ,2))+'' for v in data['importance']],\n textposition = 'inside',\n textfont_color = theme_palette['backgound']))\n\n fig.update_layout(template = temp,\n yaxis_automargin=True,\n height = 800,\n plot_bgcolor = theme_palette['backgound'],\n paper_bgcolor = theme_palette['backgound'],\n yaxis=dict(autorange=\"reversed\"),\n title={\n \"text\": \"Features Importances \",#\"β€” Gain - Threshold = 5.2e+04
the last payment 2 is farway in the top of list

\",\n \"x\":0.045,\n \"font_size\": 20, \n },\n margin={'pad':5},\n )\n return fig\n\n","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:12:04.296371Z","iopub.execute_input":"2022-09-30T04:12:04.296983Z","iopub.status.idle":"2022-09-30T04:12:04.337790Z","shell.execute_reply.started":"2022-09-30T04:12:04.296942Z","shell.execute_reply":"2022-09-30T04:12:04.336219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#we choose to try a LightGbM using the Base_Model class\nclass LgbModel(BaseModel):\n \n def train_model(self, train_set, val_set=None):\n verbosity = 100 if self.verbose else 0\n valid_sets=[train_set]\n if val_set:\n valid_sets = [train_set, val_set]\n return lgb.train(self.params, train_set, \n valid_sets=valid_sets, \n num_boost_round = 10500,\n early_stopping_rounds = 1500,\n verbose_eval=verbosity)\n \n def convert_dataset(self, x_train, y_train):\n train_set = lgb.Dataset(x_train, y_train, categorical_feature=self.categoricals)\n return train_set\n \n \n def get_default_params(self):\n params = {'n_estimators':50000,\n 'boosting_type': 'gbdt',\n 'objective': 'binary',\n 'metric': 'auc',\n 'subsample': 0.75,\n 'subsample_freq': 1,\n 'learning_rate': 0.1,\n 'feature_fraction': 0.9,\n 'max_depth': 15,\n 'lambda_l1': 1, \n 'lambda_l2': 1,\n 'early_stopping_rounds': 100,\n #'is_unbalance' : True ,\n 'scale_pos_weight' : 3\n \n }\n return params\n \n def get_cv_feature_importance(self):\n imp_df = pd.DataFrame(index=self.features)\n imp_df['importance'] = 0\n for model in self.cv_models: \n imp_df['importance'] = (imp_df['importance'] + pd.Series(model.feature_importance(), index=model.feature_name()))/self.n_splits\n return imp_df.sort_values('importance', ascending=False)\n \n \n\n#we choose to try a LightGbM using the Base_Model class\nclass CatBoost(BaseModel):\n def train_model(self, train_set, val_set=None):\n# eval_set=[val_set] if val_set else None \n verbosity = 100 if self.verbose else 0 \n return catboost.train(pool=train_set, params=self.params, \n eval_set=val_set, verbose_eval=verbosity) \n \n def convert_dataset(self, x_train, y_train):\n train_set = Pool(data=x_train, label=y_train, \n cat_features=self.categoricals)\n return train_set\n \n \n def get_default_params(self):\n params = {'iterations' : 5000,\n 'random_seed':self.RAND_SEED\n# 'metric': ['auc','binary_logloss'],\n }\n return params\n \n def get_cv_feature_importance(self):\n imp_df = pd.DataFrame(index=self.features)\n imp_df['importance'] = 0\n for model in self.cv_models: \n imp_df['importance'] = (imp_df['importance'] + pd.Series(model.feature_importances_, index=model.feature_names_))/self.n_splits\n return imp_df.sort_values('importance', ascending=False)\n \n ","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:12:05.168035Z","iopub.execute_input":"2022-09-30T04:12:05.168585Z","iopub.status.idle":"2022-09-30T04:12:05.187146Z","shell.execute_reply.started":"2022-09-30T04:12:05.168537Z","shell.execute_reply":"2022-09-30T04:12:05.186138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train['target'] = train[\"target\"].astype('int')\nprint('Transform all String features to category.\\n')\nfor usecol in tqdm_notebook(cat_cols):\n train[usecol] = train[usecol].replace(np.nan, 0).astype('int').astype('category')\n","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:12:06.850530Z","iopub.execute_input":"2022-09-30T04:12:06.851401Z","iopub.status.idle":"2022-09-30T04:12:16.000608Z","shell.execute_reply.started":"2022-09-30T04:12:06.851337Z","shell.execute_reply":"2022-09-30T04:12:15.999290Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

πŸ‘‰ -- LightGBM Training--

\n","metadata":{}},{"cell_type":"markdown","source":"\n","metadata":{}},{"cell_type":"code","source":"#https://www.kaggle.com/code/ragnar123/amex-lgbm-dart-cv-0-7977#kln-129\nparams = {\n 'objective': 'binary',\n 'metric': 'binary_logloss',\n 'boosting': 'dart',\n 'seed': 75,\n 'num_leaves': 100,\n 'learning_rate': 0.01,\n 'feature_fraction': 0.20,\n 'bagging_freq': 10,\n 'bagging_fraction': 0.50,\n 'n_jobs': -1,\n 'lambda_l2': 2,\n 'min_data_in_leaf': 40,\n }\n\n\nstart_time = time.time()\n\nlgb_model = LgbModel(name='lgb', features=features, n_splits=5, categoricals=cat_cols, ps=params, verbose=None)\nlgb_model.fit_cv(train, save_cv=True)\n\nprint(\"LGB TRAINING TIME\")\nprint(\"--- %s seconds ---\" % (str(datetime.timedelta(seconds=time.time() - start_time))))\n\njoblib.dump(lgb_model, \"lgb_model.sav\")","metadata":{"execution":{"iopub.status.busy":"2022-09-30T04:12:51.819139Z","iopub.execute_input":"2022-09-30T04:12:51.819671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"lgb_model.oof_score_","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

πŸ‘‰ -- Catboost Training--

\n","metadata":{}},{"cell_type":"code","source":"# very heavy with all feature => we will select more \nprint(THRESHOLD)\ndeeper_feat_sel = True\nif deeper_feat_sel:\n corr = train.corrwith(train['target'], axis=0)\n corr = corr[corr.notna()].sort_values(key=abs, ascending=False)\n THRESHOLD = 0.15\n CORR_SELECTION=True\n selected_feats = corr[corr.abs()>THRESHOLD].index\n train = train[list(selected_feats)]\n\n\n types = train.dtypes\n target_col = 'target'\n\n selected_cat_cols = list(types[types.apply(lambda x:not(str(x).startswith('float')))].index)\n cb_cat_cols = list(filter(lambda x:x!=target_col, selected_cat_cols))\n cb_features = list(train.drop(target_col, axis=1).columns)","metadata":{"execution":{"iopub.status.busy":"2022-09-21T20:20:05.061629Z","iopub.execute_input":"2022-09-21T20:20:05.062088Z","iopub.status.idle":"2022-09-21T20:20:17.263483Z","shell.execute_reply.started":"2022-09-21T20:20:05.062054Z","shell.execute_reply":"2022-09-21T20:20:17.262206Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"start_time = time.time()\n\ncb_model = CatBoost(name='cb', features=cb_features, categoricals=cb_cat_cols, n_splits=5)\ncb_model.fit_cv(train, save_cv=True)\n\nprint(\"CB TRAINING TIME\")\nprint(\"--- %s seconds ---\" % (str(datetime.timedelta(seconds=time.time() - start_time))))\n\njoblib.dump(cb_model, \"cb_model.sav\")","metadata":{"execution":{"iopub.status.busy":"2022-09-21T20:20:24.085280Z","iopub.execute_input":"2022-09-21T20:20:24.085717Z","iopub.status.idle":"2022-09-21T22:15:05.959678Z","shell.execute_reply.started":"2022-09-21T20:20:24.085681Z","shell.execute_reply":"2022-09-21T22:15:05.958223Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" \n#

πŸ… Feature Importance

\n\nLet's explore the feature importance of each model. One of the advantages of gradient boosting based models is that the feature importances can be directly extracted from the splitting gain of tree algorithm\n\n\n

🫐 LightGBM Feature Importance

\n","metadata":{}},{"cell_type":"code","source":"fig = lgb_model.plot_cv_feature_importance()\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

🫐 CatBoost Feature Importance

","metadata":{}},{"cell_type":"code","source":"fig = cb_model.plot_cv_feature_importance()\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-21T22:15:06.045908Z","iopub.execute_input":"2022-09-21T22:15:06.046319Z","iopub.status.idle":"2022-09-21T22:15:06.075587Z","shell.execute_reply.started":"2022-09-21T22:15:06.046284Z","shell.execute_reply":"2022-09-21T22:15:06.074418Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#

πŸ“ˆ CV Roc Curves

\nLets display Cross validation Roc curves for each model\n\n\n

🫐 LightGBM CV roc curves

\n","metadata":{}},{"cell_type":"code","source":"fig = lgb_model.plot_cv_roc()\n#fig = plot_roc(lgb_model)\nfig.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

🫐 CatBoost CV roc curves

\n","metadata":{}},{"cell_type":"code","source":"fig = cb_model.plot_cv_roc()\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-21T22:15:06.480858Z","iopub.execute_input":"2022-09-21T22:15:06.481172Z","iopub.status.idle":"2022-09-21T22:15:06.821221Z","shell.execute_reply.started":"2022-09-21T22:15:06.481144Z","shell.execute_reply":"2022-09-21T22:15:06.818607Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n#

πŸ‘‰ Infer (or Predict Test data πŸ˜…)

\n\nGenerate the test predictions ","metadata":{}},{"cell_type":"code","source":"del train\ngc.collect()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = pd.read_feather(\"../input/interim-data-v2/test_prepared.ftr\").set_index('customer_ID')\nprint('Transform all String features to category.\\n')\nfor usecol in tqdm_notebook(cat_cols):\n test[usecol] = test[usecol].replace(np.nan, 0).astype('int').astype('category')\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### Later\n#lgb_model = joblib.load('../input/interim-data/lgb_model.sav')\n#cb_model = joblib.load('../input/interim-data/cb_model.sav')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from functools import partial, reduce\n\ndef get_batchs(df, batch_size, keep_features):\n n_batchs = int(len(df)/batch_size)\n cid = list(df.index)\n for i in range(n_batchs):\n idx = cid[i*batch_size:(i+1)*batch_size]\n if i == n_batchs-1:\n idx = idx + cid[(i+1)*batch_size:]\n yield df.loc[idx, keep_features]\n \ndef predict(sample_df, model):\n y_pred = model.predict_cv(sample_df)\n sub = pd.Series(y_pred, index=sample_df.index, name='prediction')\n# sub.to_frame().to_csv(outfile)\n return sub","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

🫐 LightGBM submission

","metadata":{}},{"cell_type":"code","source":"n_batchs=15000\nsamples_df = get_batchs(test,batch_size=n_batchs, keep_features=lgb_model.features)\nsub_lg = pd.concat(map(partial(predict, model=lgb_model), tqdm_notebook(samples_df)))\nprint(len(sub_lg))\nprint(len(test))\n\nsub_lg.to_frame().to_csv('submission_lgb.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

🫐 CatBoost Submission

","metadata":{}},{"cell_type":"code","source":"samples_df = get_batchs(test, batch_size=n_batchs, keep_features=cb_model.features)\nsub_cb = pd.concat(map(partial(predict, model=cb_model), tqdm_notebook(samples_df)))\nprint(len(sub_cb))\nprint(len(test))\nsub_cb.to_frame().to_csv('submission_cb.csv')","metadata":{"execution":{"iopub.status.busy":"2022-09-21T22:28:11.828327Z","iopub.execute_input":"2022-09-21T22:28:11.828796Z","iopub.status.idle":"2022-09-21T22:34:26.351652Z","shell.execute_reply.started":"2022-09-21T22:28:11.828750Z","shell.execute_reply":"2022-09-21T22:34:26.350507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

🫐 Weighted submission

","metadata":{}},{"cell_type":"code","source":"# weighted submission\nweighted_sub = (sub_lg*lgb_model.oof_score_ + sub_cb*cb_model.oof_score_)/(lgb_model.oof_score_+cb_model.oof_score_)\nweighted_sub.to_frame().to_csv('weighted_sub.csv')","metadata":{"execution":{"iopub.status.busy":"2022-09-21T22:34:26.353195Z","iopub.execute_input":"2022-09-21T22:34:26.353663Z","iopub.status.idle":"2022-09-21T22:34:29.294978Z","shell.execute_reply.started":"2022-09-21T22:34:26.353629Z","shell.execute_reply":"2022-09-21T22:34:29.293678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#

πŸ“Š Predictions Distribution

","metadata":{}},{"cell_type":"code","source":"def get_dirtibution(sub,threshold=0.5, model_name='lgb'):\n target = (sub>threshold).astype(int)\n target = target.value_counts(normalize=True)\n target.rename(index={1:'Default',0:'Paid'},inplace=True)\n fig=go.Figure()\n fig.add_trace(go.Pie(labels=target.index, values=target*100,# hole=.45, \n showlegend=True,#sort=True, \n marker=dict(colors=list(theme_palette.values())),\n # marker=dict(colors=color,line=dict(color=pal,width=2.5)),\n hovertemplate = \"%{label} Amex Acoounts: %{value:.2f}%\"))\n fig.update_layout(template=temp, \n title={\n \"text\":f'Default VS Paid Transaction
{model_name} predictions',\n \"x\":0.035,\n \"font_size\": 20,\n\n },\n\n uniformtext_minsize=15,# width=700,\n # height=800,\n margin={'t':150, 'l':5})\n return fig","metadata":{"execution":{"iopub.status.busy":"2022-09-21T22:36:53.527155Z","iopub.execute_input":"2022-09-21T22:36:53.528409Z","iopub.status.idle":"2022-09-21T22:36:53.537810Z","shell.execute_reply.started":"2022-09-21T22:36:53.528358Z","shell.execute_reply":"2022-09-21T22:36:53.536529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = get_dirtibution(sub_lg, model_name='LightGBM')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-21T22:36:54.228325Z","iopub.execute_input":"2022-09-21T22:36:54.228899Z","iopub.status.idle":"2022-09-21T22:36:54.265419Z","shell.execute_reply.started":"2022-09-21T22:36:54.228854Z","shell.execute_reply":"2022-09-21T22:36:54.263689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = get_dirtibution(sub_cb, model_name='CatBoost')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-21T22:37:09.797089Z","iopub.execute_input":"2022-09-21T22:37:09.797634Z","iopub.status.idle":"2022-09-21T22:37:09.829330Z","shell.execute_reply.started":"2022-09-21T22:37:09.797593Z","shell.execute_reply":"2022-09-21T22:37:09.828112Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}