{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 1. About AMEX\n\nAmerican Express Company (called also Amex) is a multinational corporation with a sepcialisation in payment cards. It is present on the New York Stock Exchange (NYSE) with a ticker AXP. New York is also a city where they have the headquarter. They employ over 63k employees worldwide and hold about 23% payment card market in US.\n\nAs for many banks the core of its operation is crediting, it's crucial to manage it efficiently. Especially banks want to lend money only to people who can pay the loan. That's why in this chalange we are asked to find out which customers are not able to do it. Not paying the money back is called default.","metadata":{}},{"cell_type":"markdown","source":"# 2. Introduction to the competition\n\n### Competition Objective\nThe objective of this competition is to predict the probability that a customer does not pay back their credit card balance amount in the future based on their monthly customer profile. \n\nThe target binary variable is calculated by observing 18 months performance window after the latest credit card statement, and if the customer does not pay due amount in 120 days after their latest statement date it is considered a default event.\n\n### Little bit about the data\nThe dataset contains aggregated profile features for each customer at each statement date. Features are anonymized and normalized, and fall into the following general categories:\n\nD_* = Delinquency variables,\nS_* = Spend variables,\nP_* = Payment variables,\nB_* = Balance variables,\nR_* = Risk variables\nwith the following features being categorical:\n\n['B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68']\n\nOur task is to predict, for each customer_ID, the probability of a future payment default (target = 1).\n\nNote that the negative class has been subsampled for this dataset at 5%, and thus receives a 20x weighting in the scoring metric.\n\n### Evaluation Metric\n\nIn this competition the evaluation metric is referenced in [this](https://www.kaggle.com/code/inversion/amex-competition-metric-python) notebook. It uses two submetrics where one is a normalized Gini coefficient, a good and clean explanation of this metrics can be found [here](https://theblog.github.io/post/gini-coefficient-intuitive-explanation/).","metadata":{}},{"cell_type":"markdown","source":"# 3. Data Overview\n- train_data.csv - training data with multiple statement dates per customer_ID\n- train_labels.csv - target label for each customer_ID\n- test_data.csv - corresponding test data; your objective is to predict the target label for each -customer_ID\n-sample_submission.csv - a sample submission file in the correct format","metadata":{}},{"cell_type":"markdown","source":"# 4. Import necessary libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport gc\nimport warnings\nwarnings.filterwarnings('ignore')\n\n# General Settings\n\n# display all the columns in the dataset\npd.pandas.set_option('display.max_columns', None)\n\n# Setting color palette.\npurple_black = [\n\"#9b59b6\", \"#3498db\", \"#95a5a6\", \"#e74c3c\", \"#34495e\", \"#2ecc71\"\n]\n\n# Setting plot styling.\n#plt.style.use('ggplot')\nplt.style.use('fivethirtyeight')","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:41:53.397277Z","iopub.execute_input":"2022-07-02T05:41:53.397711Z","iopub.status.idle":"2022-07-02T05:41:53.40568Z","shell.execute_reply.started":"2022-07-02T05:41:53.397663Z","shell.execute_reply":"2022-07-02T05:41:53.402961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Check class distribution","metadata":{}},{"cell_type":"code","source":"labels = pd.read_csv(\"../input/amex-default-prediction/train_labels.csv\")\nlabels.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:41:53.452305Z","iopub.execute_input":"2022-07-02T05:41:53.452552Z","iopub.status.idle":"2022-07-02T05:41:53.915255Z","shell.execute_reply.started":"2022-07-02T05:41:53.452529Z","shell.execute_reply":"2022-07-02T05:41:53.914343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- There are 458913 customers in the given data, lets check for duplicates and missing values","metadata":{}},{"cell_type":"code","source":"print(\"No. of unique customers {}\".format(labels.customer_ID.nunique()))\nprint(\"\\nMissing Values:\\n{}\".format(labels.isnull().sum()))","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:41:53.916939Z","iopub.execute_input":"2022-07-02T05:41:53.917363Z","iopub.status.idle":"2022-07-02T05:41:54.144844Z","shell.execute_reply.started":"2022-07-02T05:41:53.917328Z","shell.execute_reply":"2022-07-02T05:41:54.143433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- There are no duplicates in our data, and no null values","metadata":{}},{"cell_type":"code","source":"# lets check class distribution\nsns.countplot(labels.target)\nplt.xlabel(\"Class\")\nplt.ylabel(\"Count\")\nplt.title(\"Class Distribution\")\nplt.show()\nplt.figure().clear()\nplt.close","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:41:54.146308Z","iopub.execute_input":"2022-07-02T05:41:54.146683Z","iopub.status.idle":"2022-07-02T05:41:54.327376Z","shell.execute_reply.started":"2022-07-02T05:41:54.146647Z","shell.execute_reply":"2022-07-02T05:41:54.326526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"labels_stats_df = pd.DataFrame(columns = ['absolute','percentage'])\nlabels_stats_df['absolute'] = labels.target.value_counts()\nlabels_stats_df['percentage'] = (labels.target.value_counts()/labels.shape[0]) * 100\nlabels_stats_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:41:54.330073Z","iopub.execute_input":"2022-07-02T05:41:54.330438Z","iopub.status.idle":"2022-07-02T05:41:54.352442Z","shell.execute_reply.started":"2022-07-02T05:41:54.330403Z","shell.execute_reply":"2022-07-02T05:41:54.351755Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- We have data for around 300k good customers and 100k bad customers\n- Dataset is imbalanced as we 74% of the data is for good customers\n- Since the dataset is imbalanced, we can not rely on accuracy, we will have to look at other measures like precision ,recall, ROC & AUC","metadata":{}},{"cell_type":"code","source":"del labels,labels_stats_df\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:41:54.353557Z","iopub.execute_input":"2022-07-02T05:41:54.354039Z","iopub.status.idle":"2022-07-02T05:41:54.590799Z","shell.execute_reply.started":"2022-07-02T05:41:54.354003Z","shell.execute_reply":"2022-07-02T05:41:54.589777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6. Let's have a look at the data","metadata":{}},{"cell_type":"markdown","source":"The dataset of this competition is of huge size:\n\ntraining data ~ 16GB\ntest data ~ 34GB\n\nWe won't be able to load these csv files, we would run into out of memory issues.\n\nThat's why we would read the data from @munumbutt's AMEX-Feather-Dataset. \n\nIn this Feather file, the floating point precision has been reduced from 64 bit to 16 bit. \n\nAlso, reading a Feather file is faster than reading a csv file because the Feather file format is binary.","metadata":{"execution":{"iopub.status.busy":"2022-06-22T04:15:09.288429Z","iopub.execute_input":"2022-06-22T04:15:09.288919Z","iopub.status.idle":"2022-06-22T04:15:09.295893Z","shell.execute_reply.started":"2022-06-22T04:15:09.288887Z","shell.execute_reply":"2022-06-22T04:15:09.294859Z"}}},{"cell_type":"code","source":"# load the dataset\ntrain = pd.read_feather('../input/amexfeather/train_data.ftr')\n#test = pd.read_feather('../input/amexfeather/test_data.ftr')\ntrain.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:41:54.592545Z","iopub.execute_input":"2022-07-02T05:41:54.593027Z","iopub.status.idle":"2022-07-02T05:42:00.959993Z","shell.execute_reply.started":"2022-07-02T05:41:54.592989Z","shell.execute_reply":"2022-07-02T05:42:00.95911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We have 5.5 million rows for training and 11 million rows in the test data set.","metadata":{}},{"cell_type":"code","source":"# display few rows of the training data\ntrain.head(3)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:00.961264Z","iopub.execute_input":"2022-07-02T05:42:00.961854Z","iopub.status.idle":"2022-07-02T05:42:01.115137Z","shell.execute_reply.started":"2022-07-02T05:42:00.961819Z","shell.execute_reply":"2022-07-02T05:42:01.114437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.info(max_cols=191, show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:01.116463Z","iopub.execute_input":"2022-07-02T05:42:01.117034Z","iopub.status.idle":"2022-07-02T05:42:06.912294Z","shell.execute_reply.started":"2022-07-02T05:42:01.116998Z","shell.execute_reply":"2022-07-02T05:42:06.911316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As I mentioned earlier, we have following type of variables in our dataset:\n\n- D_* = Delinquency variables\n- S_* = Spend variables,\n- P_* = Payment variables, \n- B_* = Balance variables, \n- R_* = Risk variables\n\nLets start analysing them one by one","metadata":{}},{"cell_type":"markdown","source":"# 7. Check Null Values","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16,8))\ncount= train.isnull().sum().sort_values(ascending=False)[:50]\nsns.barplot(count.index,count.values)\nplt.xlabel(\"Column\")\nplt.ylabel(\"Count\")\nplt.title(\"Null Values - Top 50\")\nplt.xticks(rotation=90)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:06.913719Z","iopub.execute_input":"2022-07-02T05:42:06.914084Z","iopub.status.idle":"2022-07-02T05:42:13.042202Z","shell.execute_reply.started":"2022-07-02T05:42:06.914049Z","shell.execute_reply":"2022-07-02T05:42:13.041478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":" - Null values are quite big in numbers, that too for so many columns","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(16,8))\ncount= train.isnull().sum().sort_values(ascending=False)[:50]\nsns.barplot(count.index,((count.values)/train.shape[0]) * 100)\nplt.xlabel(\"Column\")\nplt.ylabel(\"Percentage of Null Values\")\nplt.title(\"Null Values in % - Top 50\")\nplt.xticks(rotation=90)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:13.045405Z","iopub.execute_input":"2022-07-02T05:42:13.045987Z","iopub.status.idle":"2022-07-02T05:42:18.435186Z","shell.execute_reply.started":"2022-07-02T05:42:13.045949Z","shell.execute_reply":"2022-07-02T05:42:18.434464Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Percentage wise distribution of null values gives a better picture, here we can see we have columns with close to 100% null values","metadata":{}},{"cell_type":"markdown","source":"# 8. Analyse Numeric features","metadata":{}},{"cell_type":"code","source":"# Let's create a seperate list for Numeric variables\n\nnumeric = []\n\nfor col in train.columns:\n    if train[col].dtypes == \"float16\" or train[col].dtypes ==\"int64\":\n        numeric.append(col)        \n        \nprint(\"We have {} numeric variables\".format(len(numeric)))","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.436582Z","iopub.execute_input":"2022-07-02T05:42:18.437148Z","iopub.status.idle":"2022-07-02T05:42:18.445599Z","shell.execute_reply.started":"2022-07-02T05:42:18.437111Z","shell.execute_reply":"2022-07-02T05:42:18.44488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lets Analyse numeric fields first\nncols = 4\nfor idx, feat in enumerate(numeric):\n    if idx % ncols == 0: \n        if idx > 0: \n            plt.show()\n        plt.figure(figsize=(16, 3))\n        if idx == 0: plt.suptitle('Numeric features', fontsize=20, y=1.02)\n    plt.subplot(1, ncols, idx % ncols + 1)\n    sns.kdeplot(x=feat,hue='target',data=train)\n    plt.xlabel(feat)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:49:35.711779Z","iopub.execute_input":"2022-07-02T05:49:35.712119Z","iopub.status.idle":"2022-07-02T05:54:26.028611Z","shell.execute_reply.started":"2022-07-02T05:49:35.712092Z","shell.execute_reply":"2022-07-02T05:54:26.027267Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create box plots to check outliers in the dataset\nncols = 4\nfor idx, feat in enumerate(numeric):\n    if idx % ncols == 0: \n        if idx > 0: \n            plt.show()\n        plt.figure(figsize=(16, 3))\n        if idx == 0: plt.suptitle('Numeric features', fontsize=20, y=1.02)\n    plt.subplot(1, ncols, idx % ncols + 1)\n    sns.boxplot(train[feat])\n    plt.xlabel(feat)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.629325Z","iopub.status.idle":"2022-07-02T05:42:18.629663Z","shell.execute_reply.started":"2022-07-02T05:42:18.629491Z","shell.execute_reply":"2022-07-02T05:42:18.629505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 9. Analyse Date Feature","metadata":{}},{"cell_type":"code","source":"# Now lets check date varaibele\n\ndates = []\n\nfor col in train.columns:\n    if train[col].dtypes == 'datetime64[ns]':\n        dates.append(col)        \n    \nprint(\"There is {} date variable {} in our dataset\".format(len(dates),dates))\n\nprint(\"oldest statement date {}, latest statement date {}\".format(min(train.S_2), max(train.S_2)))\n\nprint(\"data present for {}\".format(max(train.S_2)-min(train.S_2)))","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.632874Z","iopub.status.idle":"2022-07-02T05:42:18.633468Z","shell.execute_reply.started":"2022-07-02T05:42:18.633234Z","shell.execute_reply":"2022-07-02T05:42:18.633257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- We have data for more than a year, 13 months to be more precise","metadata":{}},{"cell_type":"code","source":"# keep cleaning the memory to avoid out of memory issues\ndel numeric,dates\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.634558Z","iopub.status.idle":"2022-07-02T05:42:18.635156Z","shell.execute_reply.started":"2022-07-02T05:42:18.634928Z","shell.execute_reply":"2022-07-02T05:42:18.63495Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check customer wise statement distribution\nplt.figure(figsize=(16, 4))\nsns.countplot(train.groupby('customer_ID')['S_2'].count(),color='c')\nplt.xlabel('S_2')\nplt.ylabel('Count')\nplt.title(\"Customer wise distribution\")\nplt.show()\nplt.figure().clear()\nplt.close()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.636223Z","iopub.status.idle":"2022-07-02T05:42:18.636831Z","shell.execute_reply.started":"2022-07-02T05:42:18.636582Z","shell.execute_reply":"2022-07-02T05:42:18.636605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- As it is very much clear from the above plot, most of the customers have recieved statements 13 times,once per month","metadata":{}},{"cell_type":"code","source":"# lets check different statement dates and their counts\ntrain.S_2.value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.63789Z","iopub.status.idle":"2022-07-02T05:42:18.63848Z","shell.execute_reply.started":"2022-07-02T05:42:18.638252Z","shell.execute_reply":"2022-07-02T05:42:18.638274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- From above stats we can say that different customers are having different billing cycles","metadata":{}},{"cell_type":"code","source":"# Now lets check all the dates in the last month and see the count of customers issued statements on those dates\n\nplt.figure(figsize=(16, 4))\nplt.hist(train.groupby('customer_ID')['S_2'].max(),bins=pd.date_range(\"2018-03-01\", \"2018-04-01\", freq=\"d\"),rwidth=0.8,color='c')\nplt.title(\"Customer's last statements date\", fontsize=20)\nplt.xlabel('Date')\nplt.ylabel('Count')\nplt.show()\nplt.figure().clear()\nplt.close()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.63956Z","iopub.status.idle":"2022-07-02T05:42:18.640321Z","shell.execute_reply.started":"2022-07-02T05:42:18.640013Z","shell.execute_reply":"2022-07-02T05:42:18.640041Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check customer wise statement distribution\ndiff = train.groupby('customer_ID')['S_2'].agg(['min','max'])\n\nplt.figure(figsize=(16, 3))\nplt.hist((diff['max'] - diff['min']).dt.days, bins = 10,color='c') \nplt.xlabel('days')\nplt.ylabel('count')\nplt.title('Number of days between first and last statement of customer', fontsize=20)\nplt.show()\nplt.figure().clear()\nplt.close()\n\ndel diff\nprint(\"collecting garbage:\",gc.collect())","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.641513Z","iopub.status.idle":"2022-07-02T05:42:18.642101Z","shell.execute_reply.started":"2022-07-02T05:42:18.641875Z","shell.execute_reply":"2022-07-02T05:42:18.641899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Most of the customers got their first statement almost a year back (between 320 to 390 days)","metadata":{}},{"cell_type":"code","source":"# Let's analyse the train & test datasets over time\n# load test dataset\n#test = pd.read_feather('../input/amexfeather/test_data.ftr')\n\n# concat train and test datasets\n#required_cols = ['customer_ID', 'S_2']\n#master = pd.concat([train[required_cols],test[required_cols]],axis=0)\n#print(\"size of master dataset {}\".format(master.size))","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.64319Z","iopub.status.idle":"2022-07-02T05:42:18.643784Z","shell.execute_reply.started":"2022-07-02T05:42:18.643545Z","shell.execute_reply":"2022-07-02T05:42:18.643567Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#train['last_month'] = train.groupby('customer_ID').S_2.transform('max')\n#train.last_month = train.last_month.dt.month\n#last_month = train.last_month.values","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.644828Z","iopub.status.idle":"2022-07-02T05:42:18.645408Z","shell.execute_reply.started":"2022-07-02T05:42:18.645167Z","shell.execute_reply":"2022-07-02T05:42:18.645188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#test = pd.read_feather('../input/amexfeather/test_data.ftr')\n#test['last_month'] = test.groupby('customer_ID').S_2.transform('max')\n#test.last_month = test.last_month.dt.month\n#last_month = test.last_month.values","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.646437Z","iopub.status.idle":"2022-07-02T05:42:18.647019Z","shell.execute_reply.started":"2022-07-02T05:42:18.646792Z","shell.execute_reply":"2022-07-02T05:42:18.646814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 10. Categorical features\n\nAccording to the data description, there are eleven categorical features. \n\n'B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68'\n\nLet's try analyzing them","metadata":{"execution":{"iopub.status.busy":"2022-06-27T04:21:12.530471Z","iopub.execute_input":"2022-06-27T04:21:12.530851Z","iopub.status.idle":"2022-06-27T04:21:12.899604Z","shell.execute_reply.started":"2022-06-27T04:21:12.530818Z","shell.execute_reply":"2022-06-27T04:21:12.898605Z"}}},{"cell_type":"code","source":"categorical = []\n\nfor col in train.columns:\n    if train[col].dtypes == \"category\":\n        categorical.append(col)\n        \nprint(\"We have {} categorical variables\".format(len(categorical)))","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.648071Z","iopub.status.idle":"2022-07-02T05:42:18.64866Z","shell.execute_reply.started":"2022-07-02T05:42:18.648419Z","shell.execute_reply":"2022-07-02T05:42:18.648441Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (15,30))\nfor idx,col in enumerate(categorical):\n    plt.subplot(6,2,idx+1)\n    sns.countplot(train[col],hue=train.target)\nplt.suptitle('Categorical features', fontsize=20, y=0.93)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.649727Z","iopub.status.idle":"2022-07-02T05:42:18.650454Z","shell.execute_reply.started":"2022-07-02T05:42:18.650162Z","shell.execute_reply":"2022-07-02T05:42:18.65019Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Data for target = 0 and 1 is imbalanced, we have more data for target = 0 for all the categorical variables\n- For B_38, we have most variance, it has 7 different values\n- All other features have less than 7 different values","metadata":{}},{"cell_type":"code","source":"del categorical\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.651587Z","iopub.status.idle":"2022-07-02T05:42:18.652162Z","shell.execute_reply.started":"2022-07-02T05:42:18.65194Z","shell.execute_reply":"2022-07-02T05:42:18.651962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 11. Correlation\n\nLet's now check the correlation between features.\n\nSince we have lots of data, we will check correlation only on a sample data.\n\nWe would also drop columns having more than 25% null values","metadata":{}},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.653308Z","iopub.status.idle":"2022-07-02T05:42:18.654048Z","shell.execute_reply.started":"2022-07-02T05:42:18.653763Z","shell.execute_reply":"2022-07-02T05:42:18.653792Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"((train.isnull().sum()/train.shape[0])*100).sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.655159Z","iopub.status.idle":"2022-07-02T05:42:18.655759Z","shell.execute_reply.started":"2022-07-02T05:42:18.655513Z","shell.execute_reply":"2022-07-02T05:42:18.655535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# drop columns with more than 25% null values\nthresh = len(train) * .75\ntrain.dropna(thresh = thresh, axis = 1, inplace = True)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.656794Z","iopub.status.idle":"2022-07-02T05:42:18.657363Z","shell.execute_reply.started":"2022-07-02T05:42:18.657131Z","shell.execute_reply":"2022-07-02T05:42:18.657153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"((train.isnull().sum()/train.shape[0])*100).sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.658394Z","iopub.status.idle":"2022-07-02T05:42:18.658986Z","shell.execute_reply.started":"2022-07-02T05:42:18.658758Z","shell.execute_reply":"2022-07-02T05:42:18.658781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.shape","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.660128Z","iopub.status.idle":"2022-07-02T05:42:18.660884Z","shell.execute_reply.started":"2022-07-02T05:42:18.660572Z","shell.execute_reply":"2022-07-02T05:42:18.660601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- We have dropped 23 columns in the exercise donew above","metadata":{}},{"cell_type":"code","source":"train_sample = train.sample(10000)\ndel train\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.662106Z","iopub.status.idle":"2022-07-02T05:42:18.662697Z","shell.execute_reply.started":"2022-07-02T05:42:18.662453Z","shell.execute_reply":"2022-07-02T05:42:18.662475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"correlations = train_sample.corr().abs()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.663735Z","iopub.status.idle":"2022-07-02T05:42:18.664314Z","shell.execute_reply.started":"2022-07-02T05:42:18.664078Z","shell.execute_reply":"2022-07-02T05:42:18.6641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(16,8))\nsns.heatmap(correlations)\nplt.title(\"Correlation between features\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.665353Z","iopub.status.idle":"2022-07-02T05:42:18.665941Z","shell.execute_reply.started":"2022-07-02T05:42:18.665711Z","shell.execute_reply":"2022-07-02T05:42:18.665733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Most of the features do not seem to have any correlation with each other, but there are few which are highly correlated. Lets look further","metadata":{}},{"cell_type":"code","source":"# Credits : Code in the following few cells is taken from https://www.kaggle.com/code/datark1/american-express-eda\nunstacked = correlations.unstack()\nunstacked = unstacked.sort_values(ascending=False, kind=\"quicksort\").drop_duplicates().head(25)\nunstacked","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.66698Z","iopub.status.idle":"2022-07-02T05:42:18.667555Z","shell.execute_reply.started":"2022-07-02T05:42:18.667327Z","shell.execute_reply":"2022-07-02T05:42:18.667349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Above is the list of top 25 features which are highly correlated with each other, this would be an important factor when we will build our model","metadata":{}},{"cell_type":"code","source":"x1, y1 = unstacked.index[1]\nx2, y2 = unstacked.index[2]\nx3, y3 = unstacked.index[3]\n\nfig, ax = plt.subplots(1,3, figsize=(15,5))\nsns.scatterplot(x=x1, y=y1, data=train_sample, hue='target', ax=ax[0])\nsns.scatterplot(x=x2, y=y2, data=train_sample, hue='target', ax=ax[1])\nsns.scatterplot(x=x3, y=y3, data=train_sample, hue='target', ax=ax[2])\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.668607Z","iopub.status.idle":"2022-07-02T05:42:18.66919Z","shell.execute_reply.started":"2022-07-02T05:42:18.668962Z","shell.execute_reply":"2022-07-02T05:42:18.668983Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Nothing intuitive from the above plots. Lets'create smaller heat maps","metadata":{}},{"cell_type":"code","source":"# Heat map for risk variables\nrisk_variables = []\nfor col in train_sample.columns:\n    if col.startswith('R'):\n        risk_variables.append(col)\n        \ncorrelations = train_sample[risk_variables].corr().abs()\n\nplt.figure(figsize=(30,30))\nsns.heatmap(correlations, annot=True)\nplt.title(\"Correlation between Risk Variables\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.670413Z","iopub.status.idle":"2022-07-02T05:42:18.671117Z","shell.execute_reply.started":"2022-07-02T05:42:18.670865Z","shell.execute_reply":"2022-07-02T05:42:18.670904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del risk_variables\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.672145Z","iopub.status.idle":"2022-07-02T05:42:18.672728Z","shell.execute_reply.started":"2022-07-02T05:42:18.672487Z","shell.execute_reply":"2022-07-02T05:42:18.67251Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"spend_variables = []\nfor col in train_sample.columns:\n    if col.startswith('S'):\n        spend_variables.append(col)\n        \ncorrelations = train_sample[spend_variables].corr().abs()\n\nplt.figure(figsize=(30,30))\nsns.heatmap(correlations, annot=True)\nplt.title(\"Correlation between Spend Variables\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.673928Z","iopub.status.idle":"2022-07-02T05:42:18.674508Z","shell.execute_reply.started":"2022-07-02T05:42:18.674279Z","shell.execute_reply":"2022-07-02T05:42:18.674301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"payment_variables = []\nfor col in train_sample.columns:\n    if col.startswith('P'):\n        payment_variables.append(col)\n        \ncorrelations = train_sample[payment_variables].corr().abs()\n\nplt.figure(figsize=(10,8))\nsns.heatmap(correlations, annot=True)\nplt.title(\"Correlation between Payment Variables\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-02T05:42:18.675826Z","iopub.status.idle":"2022-07-02T05:42:18.676413Z","shell.execute_reply.started":"2022-07-02T05:42:18.676175Z","shell.execute_reply":"2022-07-02T05:42:18.676197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Artifical Noise\n\nAny discussion on this competition would be incomplete without discussing the artifical noise injected by Amex in the data.\n\nThere are already lots of notebooks & discussions explaining this in great detail.\n\nBelow is the link to few of the most useful resources to learn about artifical noise in this competition, its worth reading and spending some time on it.\n\n### Discussions\n\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/327649\n\nhttps://www.kaggle.com/competitions/amex-default-prediction/discussion/328514\n\n### Notebooks\nhttps://www.kaggle.com/code/icecreamtea/noise-analisys","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}