{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"The competition's aim is to predict how many people will die from Covid-19 in the next week, for a particular location.\nBefore doing any feature engineering, it is good to know which features are important for the prediction and which ones are not. This notebook will show how to do this with **XGBoost**. We will then compare the results with a **correlation analysis**.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-07-27T10:07:28.529651Z","iopub.execute_input":"2022-07-27T10:07:28.530433Z","iopub.status.idle":"2022-07-27T10:07:30.058919Z","shell.execute_reply.started":"2022-07-27T10:07:28.530307Z","shell.execute_reply":"2022-07-27T10:07:30.057408Z"}}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport xgboost as xgb\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.model_selection import cross_val_score, KFold\nfrom sklearn.metrics import mean_squared_error\nfrom sklearn import preprocessing\nimport matplotlib.pyplot as plt","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:56:56.418852Z","iopub.execute_input":"2022-07-27T12:56:56.419255Z","iopub.status.idle":"2022-07-27T12:56:56.426295Z","shell.execute_reply.started":"2022-07-27T12:56:56.419222Z","shell.execute_reply":"2022-07-27T12:56:56.424729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  <div style=\"padding:20px;color:white;margin:0;font-size:175%;text-align:center;display:fill;border-radius:5px;background-color:#016CC9;overflow:hidden;font-weight:500\">1. LOAD TRAIN</div>","metadata":{}},{"cell_type":"code","source":"train=pd.read_csv('../input/Covid19-Death-Predictions/train.csv')","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:56:56.431192Z","iopub.execute_input":"2022-07-27T12:56:56.431962Z","iopub.status.idle":"2022-07-27T12:56:56.710278Z","shell.execute_reply.started":"2022-07-27T12:56:56.431928Z","shell.execute_reply":"2022-07-27T12:56:56.709053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:56:56.712564Z","iopub.execute_input":"2022-07-27T12:56:56.712941Z","iopub.status.idle":"2022-07-27T12:56:56.722184Z","shell.execute_reply.started":"2022-07-27T12:56:56.712900Z","shell.execute_reply":"2022-07-27T12:56:56.720912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Location** is the only categorical column. Because all features should be numerical, let's Label Encode it.","metadata":{}},{"cell_type":"code","source":"le = preprocessing.LabelEncoder()\ntrain['Location'] = le.fit_transform(train['Location'])\nfeatures = [c for c in train.columns if (c != 'Id' and c != \"Next Week's Deaths\")]","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:56:56.723731Z","iopub.execute_input":"2022-07-27T12:56:56.724235Z","iopub.status.idle":"2022-07-27T12:56:56.773379Z","shell.execute_reply.started":"2022-07-27T12:56:56.724185Z","shell.execute_reply":"2022-07-27T12:56:56.772178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#  <div style=\"padding:20px;color:white;margin:0;font-size:175%;text-align:center;display:fill;border-radius:5px;background-color:#016CC9;overflow:hidden;font-weight:500\">2. TRAIN XGBOOST</div>","metadata":{}},{"cell_type":"code","source":"params = {\n            'objective': 'reg:squarederror',\n            'tree_method': 'hist',\n            'max_depth': 7,\n            'subsample':0.88,\n            'colsample_bytree': 0.5,\n            'gamma':1.5,\n            'min_child_weight':8,\n            'lambda':70,\n            'eta':0.08,\n    }\ndtrain = xgb.DMatrix(train[features], train[\"Next Week's Deaths\"])\nmodel = xgb.train(dtrain=dtrain,params=params,num_boost_round=2000) ","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:56:56.774612Z","iopub.execute_input":"2022-07-27T12:56:56.774970Z","iopub.status.idle":"2022-07-27T12:57:19.259529Z","shell.execute_reply.started":"2022-07-27T12:56:56.774938Z","shell.execute_reply":"2022-07-27T12:57:19.258558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#   <div style=\"padding:20px;color:white;margin:0;font-size:175%;text-align:center;display:fill;border-radius:5px;background-color:#016CC9;overflow:hidden;font-weight:500\">3. FEATURE IMPORTANCE</div>","metadata":{}},{"cell_type":"code","source":"# GET FEATURE IMPORTANCE \ndd = model.get_score(importance_type='gain')\nimportance = pd.DataFrame({'feature':dd.keys(),'importance':dd.values()}).sort_values('importance',ascending=False) ","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:57:19.263029Z","iopub.execute_input":"2022-07-27T12:57:19.263350Z","iopub.status.idle":"2022-07-27T12:57:19.277092Z","shell.execute_reply.started":"2022-07-27T12:57:19.263314Z","shell.execute_reply":"2022-07-27T12:57:19.275896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nb_of_features=len(importance)\nplt.figure(figsize=(10,10))\nplt.barh(np.arange(nb_of_features,0,-1), importance['importance'])\nplt.yticks(np.arange(nb_of_features,0,-1), importance['feature'])\nplt.title('Feature Importance')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-27T12:57:19.278842Z","iopub.execute_input":"2022-07-27T12:57:19.279284Z","iopub.status.idle":"2022-07-27T12:57:19.676607Z","shell.execute_reply.started":"2022-07-27T12:57:19.279243Z","shell.execute_reply":"2022-07-27T12:57:19.675659Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We see that the best indicators for next week's deaths are **Weekly Deaths** and **Weekly Cases.**","metadata":{}},{"cell_type":"markdown","source":"#   <div style=\"padding:20px;color:white;margin:0;font-size:175%;text-align:center;display:fill;border-radius:5px;background-color:#016CC9;overflow:hidden;font-weight:500\">4. FEATURES/TARGET CORRELATION</div>","metadata":{}},{"cell_type":"markdown","source":"Let's see if this is consistent with the features/target correlations. ","metadata":{}},{"cell_type":"code","source":"train=train.drop('Id',axis=1)\ncorrelations = train.corr().unstack().sort_values(ascending=False) # Build correlation matrix\ncorrelations = pd.DataFrame(correlations).reset_index() # Convert to dataframe\ncorrelations.columns = ['col1', 'col2', 'correlation'] # Label it\ncorrelations=correlations[correlations['col1']==\"Next Week's Deaths\"].sort_values('correlation',ascending=False)\ncorrelations=correlations[correlations['col2']!=\"Next Week's Deaths\"]","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:57:19.678213Z","iopub.execute_input":"2022-07-27T12:57:19.678656Z","iopub.status.idle":"2022-07-27T12:57:19.797340Z","shell.execute_reply.started":"2022-07-27T12:57:19.678612Z","shell.execute_reply":"2022-07-27T12:57:19.796262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nb_of_features=len(correlations)\nplt.figure(figsize=(10,10))\nplt.barh(np.arange(nb_of_features,0,-1), correlations['correlation'])\nplt.yticks(np.arange(nb_of_features,0,-1), correlations['col2'])\nplt.title('Feature/Target correlation')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-27T12:57:19.798841Z","iopub.execute_input":"2022-07-27T12:57:19.799510Z","iopub.status.idle":"2022-07-27T12:57:20.096931Z","shell.execute_reply.started":"2022-07-27T12:57:19.799464Z","shell.execute_reply":"2022-07-27T12:57:20.095788Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"A strongly negatively correlated feature is actually a good predictor, so we need to look at the correlations in absolute value.","metadata":{}},{"cell_type":"code","source":"correlations['abs_correlation'] = correlations['correlation'].abs()\ncorrelations=correlations.sort_values('abs_correlation',ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:57:20.099132Z","iopub.execute_input":"2022-07-27T12:57:20.099817Z","iopub.status.idle":"2022-07-27T12:57:20.107051Z","shell.execute_reply.started":"2022-07-27T12:57:20.099772Z","shell.execute_reply":"2022-07-27T12:57:20.105926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(10,10))\nplt.barh(np.arange(nb_of_features,0,-1), correlations['abs_correlation'])\nplt.yticks(np.arange(nb_of_features,0,-1), correlations['col2'])\nplt.title('Feature/Target correlation (absolute value)')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-27T12:57:20.108324Z","iopub.execute_input":"2022-07-27T12:57:20.108646Z","iopub.status.idle":"2022-07-27T12:57:20.403326Z","shell.execute_reply.started":"2022-07-27T12:57:20.108619Z","shell.execute_reply":"2022-07-27T12:57:20.402035Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The most correlated features to the next week's deaths are again **Weekly Deaths** and **Weekly Cases.** This is consistent with XGBoost feature importance.","metadata":{}},{"cell_type":"markdown","source":"#   <div style=\"padding:20px;color:white;margin:0;font-size:175%;text-align:center;display:fill;border-radius:5px;background-color:#016CC9;overflow:hidden;font-weight:500\">5. COMPARE FEATURE IMPORTANCE AND FEATURE/TARGET CORRELATION</div>","metadata":{}},{"cell_type":"markdown","source":"To better compare the feature importance and the feature/target correlations, let's put both graphs together.","metadata":{}},{"cell_type":"code","source":"fig, (ax1, ax2) = plt.subplots(2,1, figsize=(10,10))\nax1.barh(np.arange(nb_of_features,0,-1), importance['importance'])\nax1.set_yticks(np.arange(nb_of_features,0,-1), importance['feature'])\nax1.set_title('Feature Importance')\n\nax2.barh(np.arange(nb_of_features,0,-1), correlations['abs_correlation'])\nax2.set_yticks(np.arange(nb_of_features,0,-1), correlations['col2'])\nax2.set_title('Feature/Target correlation (absolute value)')\nplt.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-07-27T12:57:20.404918Z","iopub.execute_input":"2022-07-27T12:57:20.405345Z","iopub.status.idle":"2022-07-27T12:57:20.907142Z","shell.execute_reply.started":"2022-07-27T12:57:20.405301Z","shell.execute_reply":"2022-07-27T12:57:20.905946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#   <div style=\"padding:20px;color:white;margin:0;font-size:175%;text-align:center;display:fill;border-radius:5px;background-color:#016CC9;overflow:hidden;font-weight:500\">6. CONCLUSION</div>","metadata":{}},{"cell_type":"markdown","source":"Both methods give a somehow similar sorting. The top 2 features are identical. However there are some small differences in the ranking further down. There is actually a third method which is based on **random permutation**. This method will randomly shuffle the features one by one and compute the change in the model’s performance. The performance impact indicates the feature importance.","metadata":{}}]}