{"cells":[{"metadata":{},"cell_type":"markdown","source":"
"},{"metadata":{},"cell_type":"markdown","source":"In this kernel i have dedscribed how one can do data cleaning of the data set and perform some univariate and bivariate analysis on this dataset.I will be updating the content of the kernel in future to make it more insightful"},{"metadata":{},"cell_type":"markdown","source":"Please upvote this kernel if you like it. It motivates me to improve and produce more quality content :)"},{"metadata":{},"cell_type":"markdown","source":"# Contents\n\n* [Feature analysis](#1)\n\n * [Data cleaning](#1.1)\n * [Univariate Analysis](#1.2)\n * [Bivariate Analysis](#1.3)\n \n \n* [Modeling](#2)\n\n * [TSNE-PCA visulization](#2.1)\n * [Model Training with no param tuning](#2.2)\n * [Param tuning](#2.3)\n * [Learning Curve and Feature Importance](#2.4)\n * [Stacked Classifier Glimpse](#2.5)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nprint('numpy version\\t:',np.__version__)\nimport pandas as pd\nprint('pandas version\\t:',pd.__version__)\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom scipy import stats\n\n# Regular expressions\nimport re\n\n# seaborn : advanced visualization\nimport seaborn as sns\nprint('seaborn version\\t:',sns.__version__)\n\npd.options.mode.chained_assignment = None #set it to None to remove SettingWithCopyWarning\npd.options.display.float_format = '{:.4f}'.format #set it to convert scientific noations such as 4.225108e+11 to 422510842796.00\npd.set_option('display.max_columns', 100) # to display all the columns\n\nnp.set_printoptions(suppress=True,formatter={'float_kind':'{:f}'.format})\n\nimport os\n\nimport warnings\nwarnings.filterwarnings('ignore') # if there are any warning due to version mismatch, it will be ignored","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loan=pd.read_excel('/kaggle/input/task1/loan.xlsx')\nloan.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dict = pd.read_excel('/kaggle/input/task1/Data_Dictionary.xlsx')\ndata = data_dict.dropna()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from IPython.display import display, HTML\n\ndisplay(HTML(data.to_html()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Feature analysis\n## Data cleaning "},{"metadata":{},"cell_type":"markdown","source":"### Though the first step is to describe what our data signifies but first we will remove the columns that are around 30% null"},{"metadata":{},"cell_type":"markdown","source":"SO WE WILL START WITH THE EDA FOR LOAN ANALYSIS"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('What is the shape of the dataset:',loan.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.graph_objects as go\nNA_col = loan.isnull().sum()\nNA_col = NA_col[NA_col.values >(0.3*len(loan))]\nd = pd.Series((NA_col.values/loan.shape[0])*100)\n\nfig = go.Figure()\nfig.add_trace(go.Bar(\n x=NA_col.index,\n y=NA_col.values,\n name='Null points',\n marker_color='indianred',\n text = d\n))\n\n# Here we modify the tickangle of the xaxis, resulting in rotated labels.\nfig.update_layout(barmode='group', xaxis_tickangle=-90)\nfig.update_traces(texttemplate='%{text:.2s}', textposition='inside')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def removeNulls(dataframe, axis =1, percent=0.7):\n '''\n * removeNull function will remove the rows and columns based on parameters provided.\n * dataframe : Name of the dataframe \n * axis : axis = 0 defines drop rows, axis =1(default) defines drop columns \n * percent : percent of data where column/rows values are null,default is 0.3(30%)\n \n '''\n df = dataframe.copy()\n ishape = df.shape\n if axis == 0:\n rownames = df.transpose().isnull().sum()\n rownames = list(rownames[rownames.values > percent*len(df)].index)\n df.drop(df.index[rownames],inplace=True) \n print(\"\\nNumber of Rows dropped\\t: \",len(rownames))\n else:\n colnames = (df.isnull().sum()/len(df))\n colnames = list(colnames[colnames.values>=percent].index)\n df.drop(labels = colnames,axis =1,inplace=True) \n print(\"Number of Columns dropped\\t: \",len(colnames))\n \n print(\"\\nOld dataset rows,columns\",ishape,\"\\nNew dataset rows,columns\",df.shape)\n\n return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"THIS IS DONE TO REMOVE COLUMNS THAT CONTAIN MORE THAN 30% DATA AS NULLs"},{"metadata":{"trusted":true},"cell_type":"code","source":"loan = removeNulls(loan, axis =1,percent = 0.3)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"SO NOW WHAT WE CAN SEE THAT 56 COLUMNS ARE THE ONES WHICH ARE 100% NULL OR MORE THAN 92% SO THEY SHOULD BE REMOVED"},{"metadata":{},"cell_type":"markdown","source":"SOME MORE COLUMNS ARE ELIMINATED BECAUSE OF NOT UNIQUE VALUES"},{"metadata":{"trusted":true},"cell_type":"code","source":"loan['issue_d_Year'] = loan['issue_d'].dt.year\nloan['issue_d_Month'] = loan['issue_d'].dt.month\nloan['issue_d_Day'] = loan['issue_d'].dt.day\nloan['earliest_cr_line_Year'] = loan['earliest_cr_line'].dt.year\nloan['earliest_cr_line_Month'] = loan['earliest_cr_line'].dt.month\nloan['earliest_cr_line_Day'] = loan['earliest_cr_line'].dt.day\nloan['last_credit_pull_d_Year'] = loan['last_credit_pull_d'].dt.year\nloan['last_credit_pull_d_Month'] = loan['last_credit_pull_d'].dt.month\nloan['last_credit_pull_d_Day'] = loan['last_credit_pull_d'].dt.day\nloan['last_pymnt_d_Year'] = loan['last_pymnt_d'].dt.year\nloan['last_pymnt_d_Month'] = loan['last_pymnt_d'].dt.month\nloan['last_pymnt_d_Day'] = loan['last_pymnt_d'].dt.day\nloan = loan.drop(columns=['issue_d','last_pymnt_d','last_credit_pull_d','earliest_cr_line'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique = loan.nunique()\nunique = unique[unique.values == 1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loan.drop(labels = list(unique.index), axis =1, inplace=True)\nprint(\"So now we are left with\",loan.shape ,\"rows & columns.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(loan.emp_length.unique())\nloan.emp_length.fillna('0',inplace=True)\nloan.emp_length.replace(['n/a'],'Self-Employed',inplace=True)\nprint(loan.emp_length.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"not_required_columns = [\"id\",\"member_id\",\"url\",\"zip_code\"]\nloan.drop(labels = not_required_columns, axis =1, inplace=True)\nprint(\"So now we are left with\",loan.shape ,\"rows & columns.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loan","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plotting_3_chart(df, feature):\n ## Importing seaborn, matplotlab and scipy modules. \n import seaborn as sns\n import matplotlib.pyplot as plt\n import matplotlib.gridspec as gridspec\n from scipy import stats\n import matplotlib.style as style\n style.use('fivethirtyeight')\n\n ## Creating a customized chart. and giving in figsize and everything. \n fig = plt.figure(constrained_layout=True, figsize=(7,7))\n ## creating a grid of 3 cols and 3 rows. \n grid = gridspec.GridSpec(ncols=3, nrows=3, figure=fig)\n #gs = fig3.add_gridspec(3, 3)\n\n ## Customizing the histogram grid. \n ax1 = fig.add_subplot(grid[0, :2])\n ## Set the title. \n ax1.set_title(feature + ' Histogram')\n ## plot the histogram. \n sns.distplot(df.loc[:,feature], fit = stats.norm,norm_hist=True, ax = ax1)\n\n # customizing the QQ_plot. \n ax2 = fig.add_subplot(grid[1, :2])\n ## Set the title. \n ax2.set_title(feature + ' QQ_plot')\n ## Plotting the QQ_Plot. \n stats.probplot(df.loc[:,feature], plot = ax2)\n\n ## Customizing the Box Plot. \n ax3 = fig.add_subplot(grid[:, 2])\n ## Set title. \n ax3.set_title(feature + ' Box Plot')\n ## Plotting the box plot. \n sns.boxplot(df.loc[:,feature], orient='v', ax = ax3 )\n k2, p = stats.normaltest(df[feature].values)\n skewness = df[feature].skew()\n kurtosis = df[feature].kurtosis()\n print('\\n kurtosis = %.3f,skew = %.3f ,p-value = %.3f' % (kurtosis,skewness,p))\n\n alpha = 0.05\n if p > alpha:\n print('\\nThe transformed data is Gaussian (fails to reject the null hypothesis)')\n else:\n print('\\nThe transformed data does not look Gaussian (reject the null hypothesis)')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(loan.purpose.value_counts()*100)/len(loan)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"numeric_columns = ['loan_amnt','funded_amnt','funded_amnt_inv','installment','int_rate','annual_inc','dti']\n\nloan[numeric_columns] = loan[numeric_columns].apply(pd.to_numeric)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del_loan_purpose = (loan.purpose.value_counts()*100)/len(loan)\ndel_loan_purpose = del_loan_purpose[(del_loan_purpose < 0.75) | (del_loan_purpose.index == 'other')]\nloan = loan.drop(columns=['emp_title'])\nloan.drop(labels = loan[loan.purpose.isin(del_loan_purpose.index)].index, inplace=True)\nprint(\"So now we are left with\",loan.shape ,\"rows & columns.\")\n\nprint(loan.purpose.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"(loan.loan_status.value_counts()*100)/len(loan)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del_loan_status = (loan.loan_status.value_counts()*100)/len(loan)\ndel_loan_status = del_loan_status[(del_loan_status < 1.5)]\n\nloan.drop(labels = loan[loan.loan_status.isin(del_loan_status.index)].index, inplace=True)\nprint(\"So now we are left with\",loan.shape ,\"rows & columns.\")\n\nprint(loan.loan_status.unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loan['loan_income_ratio']= loan['loan_amnt']/loan['annual_inc']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bins = [0, 5000, 10000, 15000, 20000, 25000,40000]\nslot = ['0-5000', '5000-10000', '10000-15000', '15000-20000', '20000-25000','25000 and above']\nloan['loan_amnt_range'] = pd.cut(loan['loan_amnt'], bins, labels=slot)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bins = [0, 25000, 50000, 75000, 100000,6000000]\nslot = ['0-25000', '25000-50000', '50000-75000', '75000-100000', '100000 and above']\nloan['annual_inc_range'] = pd.cut(loan['annual_inc'], bins, labels=slot)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"bins = [0, .075, .10, .125, .15,.20,.25]\nslot = ['0-7.5', '7.5-10', '10-12.5', '12.5-15', '15 -20','20-25']\nloan['int_rate_range'] = pd.cut(loan['int_rate'], bins, labels=slot)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"CREATING BINS FOR EASE OF PLOTTING"},{"metadata":{"trusted":true},"cell_type":"code","source":"quantitative = [f for f in loan.columns if loan.dtypes[f] != 'object']\nqualitative = [f for f in loan.columns if loan.dtypes[f] == 'object']\nqualitative.remove('loan_status')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## UNIVARIATE ANALYSIS "},{"metadata":{},"cell_type":"markdown","source":"STARTING WITH UNIVARIATE ANALYSIS"},{"metadata":{},"cell_type":"markdown","source":"\nContinuous Variables\n\nIn case of continuous variables, we need to understand the CENTRAL TENDENCY and SPREAD of the variable.\nThese are measured using various statistical metrics visualization methods such as \nBOXPLOT,HISTOGRAM/DISTRIBUTION PLOT, VIOLIN PLOT etc.\n
\n
\nCategorical Variables\n\n
\nFor categorical variables, we’ll use frequency table to understand distribution of each category. \nIt can be be measured using two metrics, COUNT and COUNT% against each category.\nCOUNTBARPLOT or BAR CHART can be used as visualization.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def univariate(df,col,vartype,hue =None):\n \n '''\n Univariate function will plot the graphs based on the parameters.\n df : dataframe name\n col : Column name\n vartype : variable type : continuos or categorical\n Continuos(0) : Distribution, Violin & Boxplot will be plotted.\n Categorical(1) : Countplot will be plotted.\n hue : It's only applicable for categorical analysis.\n \n '''\n sns.set(style=\"darkgrid\")\n \n if vartype == 0:\n plotting_3_chart(df,col)\n \n if vartype == 1:\n temp = pd.Series(data = hue)\n fig, ax = plt.subplots()\n width = len(df[col].unique()) + 6 + 4*len(temp.unique())\n fig.set_size_inches(width , 7)\n ax = sns.countplot(data = df, x= col, order=df[col].value_counts().index,hue = hue) \n if len(temp.unique()) > 0:\n for p in ax.patches:\n ax.annotate('{:1.1f}%'.format((p.get_height()*100)/float(len(loan))), (p.get_x()+0.05, p.get_height()+20)) \n else:\n for p in ax.patches:\n ax.annotate(p.get_height(), (p.get_x()+0.32, p.get_height()+20)) \n del temp\n else:\n exit\n \n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='loan_amnt',vartype=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inference: Most of the loan amounts are distributed between 6000 to 15000"},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='int_rate',vartype=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inference: Most of the loans interest rates are distributed between 9.5% to 14.5%."},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='annual_inc',vartype=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the applicants earns beteen 48000 to 85000 annually."},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='installment',vartype=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='open_acc',vartype=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='loan_amnt_range',vartype=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='loan_status',vartype=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"15% of the applicants Charged off"},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='verification_status',vartype=1,hue='loan_status')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='term',vartype=1,hue='loan_status')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loan.home_ownership.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"rem = ['OTHER', 'NONE', 'ANY']\nloan.drop(loan[loan['home_ownership'].isin(rem)].index,inplace=True)\nloan.home_ownership.unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"univariate(df=loan,col='home_ownership',vartype=1,hue='loan_status')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"47% of applicants are living in rented home whereas 45.3% applicants were mortagaged their home \nand 7.5% in their own"},{"metadata":{},"cell_type":"markdown","source":"## Bivariate Analysis "},{"metadata":{},"cell_type":"markdown","source":"Bivariate/Multivariate Analysis\n\nBivariate/Multivariate Analysis finds out the relationship between two/two or more variables.\nWe can perform Bivariate/Multivariate analysis for any combination of categorical and continuous variables. \nThe combination can be: Categorical & Categorical, Categorical & Continuous and Continuous & Continuous.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16,12))\nsns.boxplot(data =loan, x='purpose', y='loan_amnt', hue ='loan_status')\nplt.title('Purpose of Loan vs Loan Amount')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.graph_objects as go\n\nfig = go.Figure(data=go.Heatmap(\n z=loan.corr(),x=loan.corr().columns\n,y=loan.corr().columns))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is clear from the Heatmap that how 'loan_amnt','funded_amnt' & 'funded_amnt_inv' are closely interrelated.\nSo we can take any one column out of them for our analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"loanstatus=loan.pivot_table(index=['loan_status','purpose','emp_length'],values='loan_amnt',aggfunc=('count')).reset_index()\nloanstatus=loan.loc[loan['loan_status']=='Charged Off']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = plt.figure(figsize=(30, 18))\nax = sns.boxplot(x='emp_length',y='loan_amnt',hue='purpose',data=loanstatus)\nax.set_title('Employment Length vs Loan Amount for different pupose of Loan',fontsize=22,weight=\"bold\")\nax.set_xlabel('Employment Length',fontsize=16)\nax.set_ylabel('Loan Amount',color = 'b',fontsize=16)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def crosstab(df,col):\n '''\n df : Dataframe\n col: Column Name\n '''\n crosstab = pd.crosstab(df[col], df['loan_status'],margins=True)\n crosstab['Probability_Charged Off'] = round((crosstab['Charged Off']/crosstab['All']),3)\n crosstab = crosstab[0:-1]\n return crosstab","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Bivariate/Multivariate Analysis with Probability of Charge off\nCategorical Variables vs Probability of Charged Off\n\nThe main motive of this use case to find what parameters are impacting the most on Loan Status \nthat is if a applicant will successfully complete the loan term or will charge off.\n\nSo we will be using a new term now Probability of Charged Off that will be equal to :\n\nProbabilityofChargedOff=Number of Applicants who charged off / TotalNo.ofApplicants\n\nWe will calculate this probability w.r.t each column in bivariate analysis & will see how the \nProbability of Charged Off changes with these columns.\n\nWe will create a user defined function for this"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Probability of charge off\ndef bivariate_prob(df,col,stacked= False):\n '''\n df : Dataframe\n col : Column Name\n stacked : True(default) for Stacked Bar\n '''\n # get dataframe from crosstab function\n plotCrosstab = crosstab(df,col)\n \n linePlot = plotCrosstab[['Probability_Charged Off']] \n barPlot = plotCrosstab.iloc[:,0:2]\n ax = linePlot.plot(figsize=(20,8), marker='o',color = 'b')\n ax2 = barPlot.plot(kind='bar',ax = ax,rot=1,secondary_y=True,stacked=stacked)\n ax.set_title(df[col].name.title()+' vs Probability Charge Off',fontsize=20,weight=\"bold\")\n ax.set_xlabel(df[col].name.title(),fontsize=14)\n ax.set_ylabel('Probability of Charged off',color = 'r',fontsize=14)\n ax2.set_ylabel('Number of Applicants',color = 'g',fontsize=14)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"filter_states = loan.addr_state.value_counts()\nfilter_states = filter_states[(filter_states < 10)]\n\nloan_filter_states = loan.drop(labels = loan[loan.addr_state.isin(filter_states.index)].index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"states = crosstab(loan_filter_states,'addr_state')\ndisplay(states.tail(20))\n\nbivariate_prob(df =loan_filter_states,col ='addr_state')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are multiple States/Provinces with high probability of charge,highest being 'NV' at 20%"},{"metadata":{"trusted":true},"cell_type":"code","source":"purpose = crosstab(loan,'purpose')\ndisplay(purpose)\n\nbivariate_prob(df =loan,col ='purpose',stacked=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Applicants who has taken the Loan for 'small business' has the highest probabilty of charge off of 26%.\nSo bank should take extra caution like take some asset or guarentee \nwhile approving the loan for purpose of 'small business'"},{"metadata":{"trusted":true},"cell_type":"code","source":"grade = crosstab(loan,'grade')\ndisplay(grade)\n\nbivariate_prob(df =loan,col ='grade',stacked=False)\nbivariate_prob(df =loan,col ='sub_grade')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we move from Grade A to G, probability that person will charged off is increasing."},{"metadata":{"trusted":true},"cell_type":"code","source":"annual_inc_range = crosstab(loan,'annual_inc_range')\ndisplay(annual_inc_range)\n\nbivariate_prob(df =loan,col ='annual_inc_range')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As the annual income is decreasing the probability \nthat person will default is increasing with highest\nof 18% at (0 to 25000) salary bracket."},{"metadata":{"trusted":true},"cell_type":"code","source":"emp_length = crosstab(loan,'emp_length')\ndisplay(emp_length)\n\nbivariate_prob(df =loan,col ='emp_length')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Applicants who are self employed & less than 1 year of experience are more probable of charged off.. "},{"metadata":{"trusted":true},"cell_type":"code","source":"int_rate_range = crosstab(loan,'int_rate_range')\ndisplay(int_rate_range)\n\nbivariate_prob(df =loan,col ='int_rate_range')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"interest rate of 15% or above shows around 25% chance of charged off \nwith 22.5% chances with interest rate of 10-12.5%"},{"metadata":{"trusted":true},"cell_type":"code","source":"corr_matrix = loan.corr().abs()\nupper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))\nto_drop = [column for column in upper.columns if any(upper[column] > 0.6)]\nloan.drop(to_drop, axis=1, inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loan['loan_amnt_range'] = loan['loan_amnt_range'].cat.codes\nloan['annual_inc_range'] =loan['annual_inc_range'].cat.codes\nloan['int_rate_range'] = loan['int_rate_range'].cat.codes\nfor i in qualitative:\n loan[i] = loan[i].astype('category').cat.codes\nloan['loan_status'] = loan['loan_status'].astype('category').cat.codes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loan","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"quantitative = [f for f in loan.columns if loan.dtypes[f] != 'object' or loan.dtypes[f] != 'int8']\nquantitative.remove('issue_d_Year')\nquantitative.remove('issue_d_Month')\nquantitative.remove('earliest_cr_line_Year')\nquantitative.remove('earliest_cr_line_Month')\nquantitative.remove('last_credit_pull_d_Year')\nquantitative.remove('last_credit_pull_d_Month')\nquantitative.remove('last_pymnt_d_Month')\nquantitative.remove('loan_status')\na = set(quantitative)\nb = set(qualitative)\na = a - a.intersection(b)\nquantitative = a\nquantitative = list(quantitative)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import Counter\ndef detect_outliers(df,n,features):\n \"\"\"\n Takes a dataframe df of features and returns a list of the indices\n corresponding to the observations containing more than n outliers according\n to the Tukey method.\n \"\"\"\n outlier_indices = []\n for col in features:\n # 1st quartile (25%)\n Q1 = np.percentile(df[col], 25)\n # 3rd quartile (75%)\n Q3 = np.percentile(df[col],75)\n # Interquartile range (IQR)\n IQR = Q3 - Q1\n \n # outlier step\n outlier_step = 1.5 * IQR\n \n # Determine a list of indices of outliers for feature col\n outlier_list_col = df[(df[col] < Q1 - outlier_step) | (df[col] > Q3 + outlier_step )].index\n \n # append the found outlier indices for col to the list of outlier indices \n outlier_indices.extend(outlier_list_col)\n \n # select observations containing more than 2 outliers\n outlier_indices = Counter(outlier_indices) \n multiple_outliers = list( k for k, v in outlier_indices.items() if v > n )\n \n return multiple_outliers \n\n#Outliers_to_drop = detect_outliers(df,2,df.columns.values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pca-TSNE plots + Naive kmeans "},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.manifold import TSNE\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn.preprocessing import *\nmodel = TSNE(n_components=3, random_state=0, perplexity=50)\ny = loan.loan_status.values\nX = loan.drop(columns=['loan_status'])\nX = X.fillna(0.)\nstd = StandardScaler()\nX = std.fit_transform(X)\nprint('<-----------------------------TSNE + PCA on data--------------------------->')\npca = PCA(n_components=30)\npca.fit(X)\npc = pca.transform(X)\nkmeans = KMeans(n_clusters=3)\nkmeans.fit(pc)\ny_res = kmeans.predict(pc) \ntsne1 = model.fit_transform(pc)\nfr1 = pd.DataFrame({'tsne1': tsne1[:,0], 'tsne2': tsne1[:, 1], 'tsne3':tsne1[:,2],'cluster': y})\nfr = pd.DataFrame({'tsne1': tsne1[:,0], 'tsne2': tsne1[:, 1], 'tsne3':tsne1[:,2],'cluster': y_res})\nprint('explained varience : with 30 components' , np.sum(pca.explained_variance_ratio_))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.express as px\nfig = px.scatter_3d(fr1, x='tsne1', y='tsne2', z='tsne3',\n color='cluster',opacity=0.6)\nfig.show()\nfig = px.scatter_3d(fr, x='tsne1', y='tsne2', z='tsne3',\n color='cluster',opacity=0.6)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### upper one is plot of TSNE + PCA labelled with true Classes\n#### lower one is same as upper one difference is of the points are labelled with kmeans prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"print('f1_Score of kmeans:', f1_score(y,y_res,average='weighted'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training predefined params "},{"metadata":{},"cell_type":"markdown","source":"#### results of training on pca data"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.linear_model import *\nfrom sklearn.metrics import f1_score\nfrom sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, GradientBoostingClassifier, ExtraTreesClassifier, VotingClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.neighbors import KNeighborsClassifier\nfrom sklearn.tree import DecisionTreeClassifier\nfrom sklearn.neural_network import MLPClassifier\nfrom sklearn.svm import SVC\nfrom sklearn.model_selection import GridSearchCV, cross_val_score, StratifiedKFold, learning_curve\nfrom sklearn.pipeline import make_pipeline\nfrom mlxtend.classifier import StackingCVClassifier\nfrom xgboost import XGBClassifier\nfrom lightgbm import LGBMClassifier\nfrom sklearn.discriminant_analysis import LinearDiscriminantAnalysis\n\nkfold = StratifiedKFold(n_splits=10)\n# Modeling step Test differents algorithms \nrandom_state = 2\nclassifiers = []\nclassifiers.append(SVC(random_state=random_state))\nclassifiers.append(DecisionTreeClassifier(random_state=random_state))\nclassifiers.append(AdaBoostClassifier(DecisionTreeClassifier(random_state=random_state),random_state=random_state,learning_rate=0.01))\nclassifiers.append(RandomForestClassifier(random_state=random_state))\nclassifiers.append(ExtraTreesClassifier(random_state=random_state))\nclassifiers.append(GradientBoostingClassifier(random_state=random_state))\nclassifiers.append(MLPClassifier(random_state=random_state))\nclassifiers.append(KNeighborsClassifier())\nclassifiers.append(LogisticRegression(random_state = random_state))\nclassifiers.append(LinearDiscriminantAnalysis())\nclassifiers.append(XGBClassifier(random_state = random_state))\nclassifiers.append(LGBMClassifier(boosting_type = 'gbdt',objective= 'multiclass',num_class=3,random_state = random_state))\n\ncv_results = []\nfor classifier in classifiers :\n print(classifier)\n cv_results.append(cross_val_score(classifier, pc, y = y, scoring = \"f1_weighted\", cv = kfold, n_jobs=6))\n\ncv_means = []\ncv_std = []\nfor cv_result in cv_results:\n cv_means.append(cv_result.mean())\n cv_std.append(cv_result.std())\n\ncv_res = pd.DataFrame({\"CrossValMeans\":cv_means,\"CrossValerrors\": cv_std,\"Algorithm\":[\"SVC\",\"DecisionTree\",\"AdaBoost\",\n\"RandomForest\",\"ExtraTrees\",\"GradientBoosting\",\"MultipleLayerPerceptron\",\"KNeighboors\",\"LogisticRegression\",\n \"LinearDiscriminantAnalysis\",\"XGBClassifier\",\"LGBMClassifier\"]})\n\ng = sns.barplot(\"CrossValMeans\",\"Algorithm\",data = cv_res, palette=\"Set3\",orient = \"h\",**{'xerr':cv_std})\ng.set_xlabel(\"Mean f1_weighted\")\ng = g.set_title(\"Cross validation scores\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import plotly.graph_objects as go\n\nmonths = cv_res.Algorithm.values\n\nfig = go.Figure()\nfig.add_trace(go.Bar(\n x=months,\n y=cv_res.CrossValerrors.values,\n name='CrossValerrors',\n marker_color='Blue'\n))\nfig.add_trace(go.Bar(\n x=months,\n y=cv_res.CrossValMeans.values,\n name='CrossValMeans',\n marker_color='lightsalmon'\n))\n\n# Here we modify the tickangle of the xaxis, resulting in rotated labels.\nfig.update_layout(barmode='group', xaxis_tickangle=-45)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### result on regular data"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = loan.drop(columns=['loan_status'])\nX = X.fillna(0.)\nX = RobustScaler().fit_transform(X)\ncv_results = []\nfor classifier in classifiers :\n print(classifier)\n cv_results.append(cross_val_score(classifier, X, y = y, scoring = \"f1_weighted\", cv = kfold, n_jobs=6))\n\ncv_means = []\ncv_std = []\nfor cv_result in cv_results:\n cv_means.append(cv_result.mean())\n cv_std.append(cv_result.std())\n\ncv_res = pd.DataFrame({\"CrossValMeans\":cv_means,\"CrossValerrors\": cv_std,\"Algorithm\":[\"SVC\",\"DecisionTree\",\"AdaBoost\",\n\"RandomForest\",\"ExtraTrees\",\"GradientBoosting\",\"MultipleLayerPerceptron\",\"KNeighboors\",\"LogisticRegression\",\n \"LinearDiscriminantAnalysis\",\"XGBClassifier\",\"LGBMClassifier\"]})\n\ng = sns.barplot(\"CrossValMeans\",\"Algorithm\",data = cv_res, palette=\"Set3\",orient = \"h\",**{'xerr':cv_std})\ng.set_xlabel(\"Mean f1_weighted\")\ng = g.set_title(\"Cross validation scores\")\n\nmonths = cv_res.Algorithm.values\n\nfig = go.Figure()\nfig.add_trace(go.Bar(\n x=months,\n y=cv_res.CrossValerrors.values,\n name='CrossValerrors',\n marker_color='Blue'\n))\nfig.add_trace(go.Bar(\n x=months,\n y=cv_res.CrossValMeans.values,\n name='CrossValMeans',\n marker_color='lightsalmon'\n))\n\n# Here we modify the tickangle of the xaxis, resulting in rotated labels.\nfig.update_layout(barmode='group', xaxis_tickangle=-45)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### H-Param tuning "},{"metadata":{"trusted":true},"cell_type":"code","source":"MLP = MLPClassifier((34,10,3),learning_rate_init=0.01,alpha=1e-5,validation_fraction=.1)\n\nmlp_param_grid = {'solver': ['sgd','adam'], \n 'max_iter': [1100,800],\n 'random_state':[2],\n 'learning_rate':['constant', 'adaptive']\n }\n\ngsMlpC = GridSearchCV(MLP,param_grid = mlp_param_grid, cv=kfold, scoring=\"f1_weighted\", n_jobs= -1, verbose = 1)\n\ngsMlpC.fit(X,y)\n\nMLP_best = gsMlpC.best_estimator_\n\n# Best score\ngsMlpC.best_score_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Adaboost\nDTC = DecisionTreeClassifier()\n\nadaDTC = AdaBoostClassifier(DTC, random_state=2)\n\nada_param_grid = {\"base_estimator__criterion\" : [\"entropy\"],\n \"n_estimators\" :[1,2],\n \"learning_rate\": [0.001, 0.01, 0.1]}\n\ngsadaDTC = GridSearchCV(adaDTC,param_grid = ada_param_grid, cv=kfold, scoring=\"f1_weighted\", n_jobs= -1, verbose = 1)\n\ngsadaDTC.fit(X,y)\n\nada_best = gsadaDTC.best_estimator_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#ExtraTrees \nExtC = ExtraTreesClassifier()\n\n\n## Search grid for optimal parameters\nex_param_grid = {\"max_depth\": [None],\n \"min_samples_split\": [2, 10],\n \"min_samples_leaf\": [1,3],\n \"bootstrap\": [False],\n \"n_estimators\" :[100,300],\n \"criterion\": [\"entropy\"]}\n\n\ngsExtC = GridSearchCV(ExtC,param_grid = ex_param_grid, cv=kfold, scoring=\"f1_weighted\", n_jobs= -1, verbose = 1)\n\ngsExtC.fit(X,y)\n\nExtC_best = gsExtC.best_estimator_\n\n# Best score\ngsExtC.best_score_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# RFC Parameters tunning \nRFC = RandomForestClassifier()\n\n\n## Search grid for optimal parameters\nrf_param_grid = {\"max_depth\": [4,8],\n \"min_samples_split\": [2, 10],\n \"min_samples_leaf\": [3, 10],\n \"n_estimators\" :[150]}\n\ngsRFC = GridSearchCV(RFC,param_grid = rf_param_grid, cv=kfold, scoring=\"f1_weighted\", n_jobs= -1, verbose = 1)\n\ngsRFC.fit(X,y)\n\nRFC_best = gsRFC.best_estimator_\n\n# Best score\ngsRFC.best_score_","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Gradient boosting tunning\n\nGBC = GradientBoostingClassifier()\ngb_param_grid = {'loss' : [\"deviance\"],\n 'n_estimators' : [100,300],\n 'learning_rate': [0.001, 0.01],\n 'max_depth': [4, 8],\n }\n\ngsGBC = GridSearchCV(GBC,param_grid = gb_param_grid, cv=kfold, scoring=\"f1_weighted\", n_jobs= -1, verbose = 1)\n\ngsGBC.fit(X,y)\n\nGBC_best = gsGBC.best_estimator_\n\n# Best score\ngsGBC.best_score_","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Learning curve and Feature Importance "},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_learning_curve(estimator, title, X, y, ylim=None, cv=None,\n n_jobs=-1, train_sizes=np.linspace(.1, 1.0, 5)):\n \"\"\"Generate a simple plot of the test and training learning curve\"\"\"\n plt.figure()\n plt.title(title)\n if ylim is not None:\n plt.ylim(*ylim)\n plt.xlabel(\"Training examples\")\n plt.ylabel(\"Score\")\n print(estimator)\n train_sizes, train_scores, test_scores = learning_curve(\n estimator, X, y, cv=cv, n_jobs=n_jobs, train_sizes=train_sizes)\n train_scores_mean = np.mean(train_scores, axis=1)\n train_scores_std = np.std(train_scores, axis=1)\n test_scores_mean = np.mean(test_scores, axis=1)\n test_scores_std = np.std(test_scores, axis=1)\n plt.grid()\n\n plt.fill_between(train_sizes, train_scores_mean - train_scores_std,\n train_scores_mean + train_scores_std, alpha=0.1,\n color=\"r\")\n plt.fill_between(train_sizes, test_scores_mean - test_scores_std,\n test_scores_mean + test_scores_std, alpha=0.1, color=\"g\")\n plt.plot(train_sizes, train_scores_mean, 'o-', color=\"r\",\n label=\"Training score\")\n plt.plot(train_sizes, test_scores_mean, 'o-', color=\"g\",\n label=\"Cross-validation score\")\n\n plt.legend(loc=\"best\")\n return plt\nX_train = X\nY_train = y\ng = plot_learning_curve(gsRFC.best_estimator_,\"RF mearning curves\",X_train,Y_train,cv=kfold)\ng = plot_learning_curve(gsExtC.best_estimator_,\"ExtraTrees learning curves\",X_train,Y_train,cv=kfold)\ng = plot_learning_curve(gsadaDTC.best_estimator_,\"AdaBoost learning curves\",X_train,Y_train,cv=kfold)\ng = plot_learning_curve(gsMlpC.best_estimator_,\"Multilayer Perceptron\",X_train,Y_train,cv=kfold)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"nrows = ncols = 2\nX_train = X\nfig, axes = plt.subplots(nrows = nrows, ncols = ncols, sharex=\"all\", figsize=(15,15))\nloans = loan1.drop(columns=['loan_status'])\nX_train = pd.DataFrame(X)\nX_train.columns = loans.columns\nnames_classifiers = [(\"AdaBoosting\", ada_best),(\"ExtraTrees\",ExtC_best),\n (\"RandomForest\",RFC_best),(\"GradientBoosting\",GBC_best)]\n\nnclassifier = 0\nfor row in range(nrows):\n for col in range(ncols):\n name = names_classifiers[nclassifier][0]\n classifier = names_classifiers[nclassifier][1]\n indices = np.argsort(classifier.feature_importances_)[::-1]\n g = sns.barplot(y=X_train.columns[indices],x = classifier.feature_importances_[indices] , orient='h',ax=axes[row][col])\n g.set_xlabel(\"Relative importance\",fontsize=12)\n g.set_ylabel(\"Features\",fontsize=12)\n g.tick_params(labelsize=9)\n g.set_title(name + \" feature importance\")\n nclassifier += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Glimpse of stacked classifier "},{"metadata":{"trusted":true},"cell_type":"code","source":"LGBC = LGBMClassifier(random_state = 2)\nsclf = StackingCVClassifier(classifiers=classifiers[:-1],\n meta_classifier=LGBC,\n random_state=2)\n\nscores = cross_val_score(sclf, X, y, cv=3, scoring='f1_weighted')\nprint(\"f1_weighted : %0.2f (+/- %0.2f) [%s]\" \n % (scores.mean(), scores.std(), 'stacked_classifier'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please upvote this kernel if you like it. It motivates me to improve and produce more quality content :)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":4}