{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# This notebook explore different types of feature selection techniques \nBased on article share by the following link\nhttps://medium.com/analytics-vidhya/feature-selection-techniques-2614b3b7efcd\n\nFeature selection can be done in 3 categories:\n1. [Filter Method](#section-one)\n - [Pearson Correlation](#subsection-one)\n - [Variance Inflation Factor (VIF)](#subsection-two)\n1. [Wrapper Method](#section-two)\n - [Step Forward Selection](#subsection-three)\n - [Backward elimination](#subsection-four)\n - [Recursive Feature elimination](#subsection-five)\n1. [Embedded Method](#section-three)\n1. [Summary](#section-four)","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport catboost as cat\nfrom catboost import CatBoostClassifier\nfrom catboost import CatBoostRegressor\nfrom sklearn.ensemble import RandomForestRegressor\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score,roc_curve,log_loss\nfrom sklearn.metrics import mean_squared_error\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n# for filename in filenames:\n# print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-08-25T14:07:21.788510Z","iopub.execute_input":"2023-08-25T14:07:21.789395Z","iopub.status.idle":"2023-08-25T14:07:23.924573Z","shell.execute_reply.started":"2023-08-25T14:07:21.789268Z","shell.execute_reply":"2023-08-25T14:07:23.922869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainData = pd.read_csv('/kaggle/input/playground-series-s3e21/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:23.927878Z","iopub.execute_input":"2023-08-25T14:07:23.929869Z","iopub.status.idle":"2023-08-25T14:07:23.985102Z","shell.execute_reply.started":"2023-08-25T14:07:23.929813Z","shell.execute_reply":"2023-08-25T14:07:23.983945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainData","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:23.986860Z","iopub.execute_input":"2023-08-25T14:07:23.987560Z","iopub.status.idle":"2023-08-25T14:07:24.052530Z","shell.execute_reply.started":"2023-08-25T14:07:23.987522Z","shell.execute_reply":"2023-08-25T14:07:24.051121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Process trainData to fill up null, drop unused columns that are not present in test dataset","metadata":{}},{"cell_type":"code","source":"def generateXY(trainData):\n x = trainData.drop(columns=['id','target'])\n x = x.fillna(method='ffill')\n x = x.fillna(0)\n y = trainData.target\n return x,y","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:24.056283Z","iopub.execute_input":"2023-08-25T14:07:24.056837Z","iopub.status.idle":"2023-08-25T14:07:24.063947Z","shell.execute_reply.started":"2023-08-25T14:07:24.056788Z","shell.execute_reply":"2023-08-25T14:07:24.062529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Calculate RMSE based on selected features","metadata":{}},{"cell_type":"code","source":"def calculateRMSE(x,y,features,model):\n for i,val in enumerate(x.columns):\n if val not in features:\n x = x.drop(columns=val)\n X_train, X_val, y_train, y_val = train_test_split(x,y,test_size=0.2, random_state=42)\n clf = RandomForestRegressor(\n n_estimators=1000,\n max_depth=7,\n n_jobs=-1,\n random_state=42)\n clf.fit(X_train,y_train)\n prediction = clf.predict(X_val)\n loss = mean_squared_error(y_val,prediction,squared=False)\n print(f\"{model} has RMSE = {loss}\")\n return clf","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:24.066056Z","iopub.execute_input":"2023-08-25T14:07:24.066888Z","iopub.status.idle":"2023-08-25T14:07:24.080970Z","shell.execute_reply.started":"2023-08-25T14:07:24.066840Z","shell.execute_reply":"2023-08-25T14:07:24.079116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# 1. Filter method\nFilter and take only the subset of the relevant features, most commonly done using Pearson correlation and VIF\n\n\n# A] Pearson Correlation\n\nThe Pearson correlation measures the strength of the linear relationship between two variables. It has a value between -1 to 1, with a value of -1 meaning a total negative linear correlation, 0 being no correlation, and + 1 meaning a total positive correlation.","metadata":{}},{"cell_type":"code","source":"trainCorr = trainData.corr()\nplt.figure(figsize=(30,15))\nsns.heatmap(trainCorr, annot=True)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:24.083438Z","iopub.execute_input":"2023-08-25T14:07:24.084314Z","iopub.status.idle":"2023-08-25T14:07:32.064261Z","shell.execute_reply.started":"2023-08-25T14:07:24.084263Z","shell.execute_reply":"2023-08-25T14:07:32.063328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Setting threshold to remove features with low correlation with target label (emission)","metadata":{}},{"cell_type":"code","source":"threshold = 0.05\ncorr=abs(trainCorr['target'])\nresult = corr[corr>threshold]\nresult.sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:32.065571Z","iopub.execute_input":"2023-08-25T14:07:32.066199Z","iopub.status.idle":"2023-08-25T14:07:32.078964Z","shell.execute_reply.started":"2023-08-25T14:07:32.066163Z","shell.execute_reply":"2023-08-25T14:07:32.078040Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Discover features with high correlation, and keep and drop the other. Same process is followed until last variable.","metadata":{}},{"cell_type":"code","source":"columns = trainData.columns.tolist()\nhighCorrFeature = []\nfor i in range(5,len(columns)):\n if columns[i] in highCorrFeature:\n continue\n for j in range(i,len(columns)):\n if columns[j] in highCorrFeature:\n continue\n if i != j:\n c = trainData[[columns[i],columns[j]]].corr()\n val = c[columns[j]][0]\n if val > 0.5:\n print(f\"{columns[i]} and {columns[j]} has high correlation {val}\")\n highCorrFeature.append(columns[j])\n \nprint(f\"\\nSize of original data {trainData.shape[1]}, \\\nnumber of high correlation features {len(highCorrFeature)}\")","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:32.080105Z","iopub.execute_input":"2023-08-25T14:07:32.081098Z","iopub.status.idle":"2023-08-25T14:07:32.511742Z","shell.execute_reply.started":"2023-08-25T14:07:32.081062Z","shell.execute_reply":"2023-08-25T14:07:32.510514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x = trainData.drop(columns=highCorrFeature)\nfeaturePC = x.columns.tolist()\nprint(\"Selected features: \",featurePC)\nx,y = generateXY(trainData)\ncalculateRMSE(x,y,featurePC,'Pearson Correlation')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:32.513567Z","iopub.execute_input":"2023-08-25T14:07:32.514426Z","iopub.status.idle":"2023-08-25T14:07:39.279744Z","shell.execute_reply.started":"2023-08-25T14:07:32.514375Z","shell.execute_reply":"2023-08-25T14:07:39.278470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# B] Variance Inflation Factor (VIF)\n\nVariance inflation factor (VIF) is a measure of the amount of multicollinearity in a set of multiple regression variables. \n\nTo detect collinearity among variables, simply create a correlation matrix and find variables with large absolute values.","metadata":{}},{"cell_type":"code","source":"from statsmodels.stats.outliers_influence import variance_inflation_factor\ndef cal_vif(x):\n thresh = 2\n output = pd.DataFrame()\n k = x.shape[1]\n vif = [variance_inflation_factor(x.values,i) for i in range(x.shape[1])]\n for i in range(1,k):\n a = np.argmax(vif)\n print('Max vif is for variable no:',a)\n if (vif[a]<=thresh):\n break\n if (i==1):\n output = x.drop(x.columns[a],axis=1)\n vif = [variance_inflation_factor(output.values,j) for j in range(output.shape[1])]\n elif (i>1):\n output = output.drop(output.columns[a],axis=1)\n vif = [variance_inflation_factor(output.values,j) for j in range(output.shape[1])]\n return output\n\nx,y = generateXY(trainData)\nx = cal_vif(x)\nfeatureVIF = x.columns.tolist()\nprint(\"Selected_features: \",featureVIF)\ncalculateRMSE(x,y,featureVIF,'Variance Inflation Factor')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:39.281597Z","iopub.execute_input":"2023-08-25T14:07:39.282978Z","iopub.status.idle":"2023-08-25T14:07:50.082293Z","shell.execute_reply.started":"2023-08-25T14:07:39.282913Z","shell.execute_reply":"2023-08-25T14:07:50.079940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# 2. Wrapper Method\n\n\n# A] Step Forward Selection\n\nStep forward selection starts with the evaluation of each individual feature, and selects that which results in the best performing selected algorithm model. \n\nWe start with having no feature in the model. In each iteration, we keep adding the feature which best improves our model till an addition of a new variable does not improve the performance of the model.","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import accuracy_score as acc\nfrom mlxtend.feature_selection import SequentialFeatureSelector as sfs\nimport warnings\nwarnings.filterwarnings('ignore')\nx,y = generateXY(trainData)\nX_train, X_val, y_train, y_val = train_test_split(x,y,test_size=0.2, random_state=42)\nclf = CatBoostRegressor(iterations=10,metric_period=500,verbose=False)\nsfs1 = sfs(clf,k_features=5,forward=True,floating=False, verbose=1, cv=3)\nsfs1 = sfs1.fit(X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:07:50.087998Z","iopub.execute_input":"2023-08-25T14:07:50.088391Z","iopub.status.idle":"2023-08-25T14:08:15.143576Z","shell.execute_reply.started":"2023-08-25T14:07:50.088357Z","shell.execute_reply":"2023-08-25T14:08:15.142328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"feat_cols = list(sfs1.k_feature_idx_)\nfeatureSFS = x.columns[feat_cols].tolist()\nprint(\"Selected features:\", featureSFS)\ncalculateRMSE(x,y,featureSFS,'Step Forward Selection')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:15.144957Z","iopub.execute_input":"2023-08-25T14:08:15.145317Z","iopub.status.idle":"2023-08-25T14:08:18.920181Z","shell.execute_reply.started":"2023-08-25T14:08:15.145284Z","shell.execute_reply":"2023-08-25T14:08:18.918944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# B] Backward Elimination\n\nIn backward elimination, we start with all the features and removes the least significant feature at each iteration which improves the performance of the model. We repeat this until no improvement is observed on removal of features.","metadata":{}},{"cell_type":"code","source":"x,y = generateXY(trainData)\nimport statsmodels.api as sm\ncols = list(x.columns)\npmax = 1\nwhile (len(cols)>0):\n p = []\n x_1 = x[cols]\n x_1 = sm.add_constant(x_1)\n model = sm.OLS(y,x_1).fit()\n p = pd.Series(model.pvalues.values[1:],index=cols)\n pmax = max(p)\n features_with_p_max = p.idxmax()\n if (pmax>0.05):\n cols.remove(features_with_p_max)\n else:\n break\nfeatureBE=cols\nprint(\"Selected features: \",featureBE)\ncalculateRMSE(x,y,featureBE,'Backward Elimination')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:18.921991Z","iopub.execute_input":"2023-08-25T14:08:18.922506Z","iopub.status.idle":"2023-08-25T14:08:24.437892Z","shell.execute_reply.started":"2023-08-25T14:08:18.922455Z","shell.execute_reply":"2023-08-25T14:08:24.436534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# C] Recursive Feature elimination\n\nRecursive feature elimination (RFE) is a feature selection method that fits a model and removes the weakest feature (or features) until the specified number of features is reached.","metadata":{}},{"cell_type":"code","source":"from sklearn.feature_selection import RFE\nx,y = generateXY(trainData)\nclf = CatBoostRegressor(iterations=100,metric_period=500,verbose=False)\nrfe = RFE(clf)\nX_rfe = rfe.fit_transform(x,y)\nclf.fit(X_rfe,y)\ncols = list(x.columns)\ntemp = pd.Series(rfe.support_,index = cols)\nfeatureRFE = temp[temp==True].index.tolist()\nprint(\"Selected features: \",featureRFE)\ncalculateRMSE(x,y,featureRFE,'Recursive Feature Elimination')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:24.439664Z","iopub.execute_input":"2023-08-25T14:08:24.440182Z","iopub.status.idle":"2023-08-25T14:08:36.876706Z","shell.execute_reply.started":"2023-08-25T14:08:24.440135Z","shell.execute_reply":"2023-08-25T14:08:36.875501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# 3. Embedded Method\n\nEmbedded methods are iterative process to extract those features which contribute the most to the training for a particular iteration. Regularization methods, such as Lasso regularization, is the most commonly used embedded methods which penalize a feature given a coefficient threshold. If the feature is irrelevant, lasso penalizes its coefficient and make it 0. Hence the features with coefficient = 0 are removed and the rest are taken.","metadata":{}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\nfrom sklearn.linear_model import LassoCV\nx,y = generateXY(trainData)\nreg = LassoCV(cv=5,n_alphas=200,tol=1e-4,fit_intercept=False)\nreg.fit(x,y)\nprint(\"Best score: \", reg.score(x,y))\ncoef = pd.Series(reg.coef_, index=x.columns)\nprint(\"Lasso picked \" + str(sum(coef != 0))+ \" variables and eliminated the other \" + str(sum(coef ==0)) + \" variables\")\nimp_coef = coef.sort_values()\nfeatureEM = imp_coef[imp_coef>0].index.tolist()\nprint(\"Selected features: \",featureEM)\ncalculateRMSE(x,y,featureEM,'Embedded Method')","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:36.878415Z","iopub.execute_input":"2023-08-25T14:08:36.879172Z","iopub.status.idle":"2023-08-25T14:08:42.233745Z","shell.execute_reply.started":"2023-08-25T14:08:36.879121Z","shell.execute_reply":"2023-08-25T14:08:42.232828Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Summary\n\nFinal model is contructed based on the list of selected feature based on 6 different techniques:\n* Pearson Correlation\n* Variance Inflation Factor\n* Step Forward Selection\n* Backward elimination\n* Recursive Feature elimination\n* Embedded Method\n\n","metadata":{}},{"cell_type":"code","source":"allfeature = featurePC + featureVIF + featureSFS + featureBE + featureRFE + featureEM\nfeature, count = np.unique(allfeature, return_counts=True)\nfeatureCount = pd.DataFrame(count,index=feature, columns=['Count'])\nfeatureCount = featureCount.sort_values(by=['Count'])\nfeatureCount.plot(kind = \"barh\")\nplt.rcParams[\"figure.figsize\"] = (10,30)\nplt.title(\"Feature importance based on all selection techniques\")","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:42.235131Z","iopub.execute_input":"2023-08-25T14:08:42.235522Z","iopub.status.idle":"2023-08-25T14:08:42.901857Z","shell.execute_reply.started":"2023-08-25T14:08:42.235487Z","shell.execute_reply":"2023-08-25T14:08:42.900616Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"selected_features = []\nfor i, val in enumerate(feature):\n if count[i] > 4:\n selected_features.append(val)\nprint(\"Final selected features: \", selected_features)\ntrainData = pd.read_csv('/kaggle/input/playground-series-s3e21/sample_submission.csv')\nx,y = generateXY(trainData)\nclf = calculateRMSE(x,y,selected_features,\"Final model\")","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:42.903960Z","iopub.execute_input":"2023-08-25T14:08:42.904473Z","iopub.status.idle":"2023-08-25T14:08:46.528157Z","shell.execute_reply.started":"2023-08-25T14:08:42.904425Z","shell.execute_reply":"2023-08-25T14:08:46.526684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Replace dropped features with zeros","metadata":{}},{"cell_type":"code","source":"trainData = pd.read_csv('/kaggle/input/playground-series-s3e21/sample_submission.csv')\nfor i in trainData.columns:\n if i not in selected_features and i != \"target\" and i != \"id\":\n trainData[i] = 0\n \nx,y = generateXY(trainData)\nclf = calculateRMSE(x,y,selected_features,\"Final model\")","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:46.529875Z","iopub.execute_input":"2023-08-25T14:08:46.530749Z","iopub.status.idle":"2023-08-25T14:08:50.144199Z","shell.execute_reply.started":"2023-08-25T14:08:46.530699Z","shell.execute_reply":"2023-08-25T14:08:50.142961Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Final Submission","metadata":{}},{"cell_type":"code","source":"trainData.to_csv('sample_submission.csv',index=False)\nprint(\"Your submission was successfully saved!\")","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:50.145569Z","iopub.execute_input":"2023-08-25T14:08:50.145932Z","iopub.status.idle":"2023-08-25T14:08:50.202785Z","shell.execute_reply.started":"2023-08-25T14:08:50.145898Z","shell.execute_reply":"2023-08-25T14:08:50.201537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"trainData","metadata":{"execution":{"iopub.status.busy":"2023-08-25T14:08:50.204367Z","iopub.execute_input":"2023-08-25T14:08:50.204736Z","iopub.status.idle":"2023-08-25T14:08:50.237864Z","shell.execute_reply.started":"2023-08-25T14:08:50.204703Z","shell.execute_reply":"2023-08-25T14:08:50.236815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"Please leave a comment, if you have any suggestion on other feature selection techniques that are beneficial. I will continue to improve this notebook for knowledge sharing.","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}