{"cells":[{"metadata":{"_uuid":"bd247fe1a341c0fea766374cdd9eae4390efc0bf"},"cell_type":"markdown","source":"### **Purpose of this notebook** : To gain model insights\n---\n### **Methods used to serve the purpose** : Permutation Importance, Partial Plots and SHAP values\n---\n### **Prerequisites** : Since a model is necessary for doing the above, we'll quickly do feature engineering and train a model based on [this](https://www.kaggle.com/harmeggels/random-forest-feature-importances/notebook) popular notebook. Credits to this notebook for useful information.\n----"},{"metadata":{"_uuid":"396f8b4961248ac9c84a9a84990a7cac5790808d"},"cell_type":"markdown","source":"### **Notebook contents** \n\n* [Necessary Imports](#0)\n* [Load Data and Feature Engineering](#1)\n* [Train Model](#2)\n* [**Derive Model Insights**](#3)\n    * [**Permutation Importance**](#3.1)\n    * [**Partial Plots**](#3.2)\n    * [**SHAP Values**](#3.3)"},{"metadata":{"_uuid":"10ab66fde54edac9336c8d558759a394c2c6b3c3"},"cell_type":"markdown","source":"### **Please upvote if you find this kernel useful**. "},{"metadata":{"_uuid":"524a7444ca7f180d08177042191775c55c0a8f1f"},"cell_type":"markdown","source":"<a id=\"0\"></a> <br>\n## **Necessary Imports**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"%matplotlib inline\n\nimport pandas as pd\nfrom sklearn.ensemble import RandomForestClassifier, forest\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score\nimport matplotlib.pyplot as plt\nfrom IPython.display import display\nimport numpy as np\nimport scipy\nimport re\n\n# Permutation Importance\nimport eli5\nfrom eli5.sklearn import PermutationImportance\n\n# Partial Plots\nfrom pdpbox import pdp, get_dataset, info_plots\n\n# Package used to calculate SHAP Values\nimport shap","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4dd5836df69670ebf12043f0d8a396974302eea9"},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n## **Load Data and Feature Engineering**"},{"metadata":{"_uuid":"b0984f37cc521db229fcbe415703ab4cf8968591"},"cell_type":"markdown","source":"As explained in [this](https://www.kaggle.com/theoviel/load-the-totality-of-the-data) notebook, we set the data types(example : switching from float64 to float32) of variables/features to reduce memory usage"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"dtypes = {\n        'MachineIdentifier':                                    'category',\n        'ProductName':                                          'category',\n        'EngineVersion':                                        'category',\n        'AppVersion':                                           'category',\n        'AvSigVersion':                                         'category',\n        'IsBeta':                                               'int8',\n        'RtpStateBitfield':                                     'float16',\n        'IsSxsPassiveMode':                                     'int8',\n        'DefaultBrowsersIdentifier':                            'float16',\n        'AVProductStatesIdentifier':                            'float32',\n        'AVProductsInstalled':                                  'float16',\n        'AVProductsEnabled':                                    'float16',\n        'HasTpm':                                               'int8',\n        'CountryIdentifier':                                    'int16',\n        'CityIdentifier':                                       'float32',\n        'OrganizationIdentifier':                               'float16',\n        'GeoNameIdentifier':                                    'float16',\n        'LocaleEnglishNameIdentifier':                          'int8',\n        'Platform':                                             'category',\n        'Processor':                                            'category',\n        'OsVer':                                                'category',\n        'OsBuild':                                              'int16',\n        'OsSuite':                                              'int16',\n        'OsPlatformSubRelease':                                 'category',\n        'OsBuildLab':                                           'category',\n        'SkuEdition':                                           'category',\n        'IsProtected':                                          'float16',\n        'AutoSampleOptIn':                                      'int8',\n        'PuaMode':                                              'category',\n        'SMode':                                                'float16',\n        'IeVerIdentifier':                                      'float16',\n        'SmartScreen':                                          'category',\n        'Firewall':                                             'float16',\n        'UacLuaenable':                                         'float32',\n        'Census_MDC2FormFactor':                                'category',\n        'Census_DeviceFamily':                                  'category',\n        'Census_OEMNameIdentifier':                             'float16',\n        'Census_OEMModelIdentifier':                            'float32',\n        'Census_ProcessorCoreCount':                            'float16',\n        'Census_ProcessorManufacturerIdentifier':               'float16',\n        'Census_ProcessorModelIdentifier':                      'float16',\n        'Census_ProcessorClass':                                'category',\n        'Census_PrimaryDiskTotalCapacity':                      'float32',\n        'Census_PrimaryDiskTypeName':                           'category',\n        'Census_SystemVolumeTotalCapacity':                     'float32',\n        'Census_HasOpticalDiskDrive':                           'int8',\n        'Census_TotalPhysicalRAM':                              'float32',\n        'Census_ChassisTypeName':                               'category',\n        'Census_InternalPrimaryDiagonalDisplaySizeInInches':    'float16',\n        'Census_InternalPrimaryDisplayResolutionHorizontal':    'float16',\n        'Census_InternalPrimaryDisplayResolutionVertical':      'float16',\n        'Census_PowerPlatformRoleName':                         'category',\n        'Census_InternalBatteryType':                           'category',\n        'Census_InternalBatteryNumberOfCharges':                'float32',\n        'Census_OSVersion':                                     'category',\n        'Census_OSArchitecture':                                'category',\n        'Census_OSBranch':                                      'category',\n        'Census_OSBuildNumber':                                 'int16',\n        'Census_OSBuildRevision':                               'int32',\n        'Census_OSEdition':                                     'category',\n        'Census_OSSkuName':                                     'category',\n        'Census_OSInstallTypeName':                             'category',\n        'Census_OSInstallLanguageIdentifier':                   'float16',\n        'Census_OSUILocaleIdentifier':                          'int16',\n        'Census_OSWUAutoUpdateOptionsName':                     'category',\n        'Census_IsPortableOperatingSystem':                     'int8',\n        'Census_GenuineStateName':                              'category',\n        'Census_ActivationChannel':                             'category',\n        'Census_IsFlightingInternal':                           'float16',\n        'Census_IsFlightsDisabled':                             'float16',\n        'Census_FlightRing':                                    'category',\n        'Census_ThresholdOptIn':                                'float16',\n        'Census_FirmwareManufacturerIdentifier':                'float16',\n        'Census_FirmwareVersionIdentifier':                     'float32',\n        'Census_IsSecureBootEnabled':                           'int8',\n        'Census_IsWIMBootEnabled':                              'float16',\n        'Census_IsVirtualDevice':                               'float16',\n        'Census_IsTouchEnabled':                                'int8',\n        'Census_IsPenCapable':                                  'int8',\n        'Census_IsAlwaysOnAlwaysConnectedCapable':              'float16',\n        'Wdft_IsGamer':                                         'float16',\n        'Wdft_RegionIdentifier':                                'float16',\n        'HasDetections':                                        'int8'\n        }\n\n%time train = pd.read_csv('../input/train.csv', nrows=1000000, usecols=dtypes.keys(), low_memory=False)\n\n#display(train.describe(include='all').T)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"baef6811779d28bd30318d071f9d13527de244fb","_kg_hide-input":true,"_kg_hide-output":false},"cell_type":"code","source":"col = ['EngineVersion', 'AppVersion', 'AvSigVersion', 'OsBuildLab', 'Census_OSVersion']\nfor c in col:\n    for i in range(6):\n        train[c + str(i)] = train[c].map(lambda x: re.split('\\.|-', str(x))[i] if len(re.split('\\.|-', str(x))) > i else -1)\n        try:\n            train[c + str(i)] = pd.to_numeric(train[c + str(i)])\n        except:\n            #print(f'{c + str(i)} cannot be casted to number')\n            pass\n            \ntrain['HasExistsNotSet'] = train['SmartScreen'] == 'ExistsNotSet'\n#In the competition details, a strong time component was indicated. \n#At this point, I am not aware of any columns which show this time component, so lets for now split our validation set based on the index\ndef split_train_val_set(X, Y, n):\n    if n < 1: n=int(len(X.index) * n)\n    return X.iloc[:n], X.iloc[n:], Y.iloc[:n], Y.iloc[n:]\n\n#We prepare the training data by replacing the category variables with the category codes \n#and replacing the nan values in the numerical columns with the median\nfor col, val in train.items():\n    if pd.api.types.is_string_dtype(val): \n        train[col] = val.astype('category').cat.as_ordered()\n        train[col] = train[col].cat.codes\n    elif pd.api.types.is_numeric_dtype(val) and val.isnull().sum() > 0:\n        train[col] = val.fillna(val.median())\n\nX, Y = train.drop('HasDetections', axis=1), train['HasDetections']\nX_train, X_val, Y_train, Y_val = train_test_split(X, Y, test_size=0.2, random_state=42)\n#X_train, X_val, Y_train, Y_val = split_train_val_set(X, Y, n=0.1)\nX_train.head(5)\n\n#To be able to test the models rapidly, we create a function to print the scores of the model.\ndef print_score(m):\n    res = [roc_auc_score(m.predict(X_train), Y_train), roc_auc_score(m.predict(X_val), Y_val), \n           m.score(X_train, Y_train), m.score(X_val, Y_val)\n          ]\n    if hasattr(m, 'oob_score_'): res.append(m.oob_score_)\n    print(res)\n    \n#As in the fastai course, the rf_samples can be reduced to allow for faster repetition cycles. \n#We also immediately create a reset function to check the model performance on the entire dataset.\ndef set_rf_samples(n):\n    \"\"\" Changes Scikit learn's random forests to give each tree a random sample of\n    n random rows.\n    \"\"\"\n    forest._generate_sample_indices = (lambda rs, n_samples: forest.check_random_state(rs).randint(0, n_samples, n))\n    \ndef reset_rf_samples():\n    \"\"\" Undoes the changes produced by set_rf_samples.\n    \"\"\"\n    forest._generate_sample_indices = (lambda rs, n_samples:\n        forest.check_random_state(rs).randint(0, n_samples, n_samples))\n    \nset_rf_samples(50000)\n","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":false,"trusted":true,"_uuid":"432b8eabca2d67fdac8cb61982392a12fc97e4e2"},"cell_type":"code","source":"train[:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4f91938b0119ec0a63b374affff55a6a734e1fb8"},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8dc77cc160a14856154931ed983b1024a89ebae"},"cell_type":"markdown","source":"<a id=\"2\"></a> <br>\n## **Train Model**"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true,"_uuid":"fade4488aa6ba08ea7b8838a6217286c6d836438"},"cell_type":"code","source":"model = RandomForestClassifier(n_estimators=100, min_samples_leaf=50, max_features=0.5, n_jobs=-1, oob_score=False)\n%time model.fit(X_train, Y_train)\n\nprint_score(model)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b21b829e7e4c4d64fd6ec3383a0b301f36fe66ab"},"cell_type":"markdown","source":"<a id=\"3\"></a> <br>\n## **Derive Model insights**"},{"metadata":{"_uuid":"e7dcd5c251c94f33d1ab235bcc3918a649b3d25a"},"cell_type":"markdown","source":"* Q) How in general model insights help?\n* *A) Debugging(ex: leakage problems), better feature engineering , direct future data collection and inform human decision making.*\n---\n\n* Q) How in this Microsoft malware prediction competition does model insights help?\n* *A) Better feature engineering*"},{"metadata":{"_uuid":"d7d6586c40d3fb5aaad89b338666848f2e66b2bc"},"cell_type":"markdown","source":"<a id=\"3.1\"></a> <br>\n### **Permutation Importance**\nPermutation Importance help us understand what features have the biggest impact on predictions"},{"metadata":{"trusted":true,"_uuid":"6cae5e9733118b457274fd580b75be972a493c12"},"cell_type":"code","source":"perm = PermutationImportance(model, random_state=1).fit(X_val, Y_val)\neli5.show_weights(perm, feature_names = X_val.columns.tolist())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9698fdad201792439152a9f589e0692c79700225"},"cell_type":"markdown","source":"For now, let us dig more into the top 3 original features(AVProductStatesIdentifier, AVProductsInstalled, AvSigVersion) as indicated by Permutation Importance."},{"metadata":{"_uuid":"c92d21197c707915e56d736c03bbf6846feba431"},"cell_type":"markdown","source":"<a id=\"3.2\"></a> <br>\n### **Partial Plots**\nPartial plots help us understand how a feature affects predictions."},{"metadata":{"trusted":true,"_uuid":"17770b3ea23c66f8f1024a4c9f7d3061a217a961"},"cell_type":"code","source":"feat_names = ['AVProductStatesIdentifier', 'AVProductsInstalled', 'AvSigVersion']\n\nfor feat_name in feat_names:\n    pdp_dist = pdp.pdp_isolate(model=model, dataset=X_val, model_features=X_val.columns.tolist(), feature=feat_name)\n    pdp.pdp_plot(pdp_dist, feat_name)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fe2c8193194d7d2c1fb7f8d75a90b7f7c7542168"},"cell_type":"markdown","source":"* **AVProductStatesIdentifier** -  ID for the specific configuration of a user's antivirus software \n    * Chances of malware detection considerably increase from a value of ~23000 till ~48000. Thereafter, though chances decrease, these machine are still not as protected as the ones with a value < 23000. Having a value < 23000 for this feature seems ideal for a machine not to be affected by malware\n    * Additional stats : As can be seen in train.describe() above, min value = 6 and max value = 70492. \n    \n* **AVProductsInstalled** - NA in description\n    * Having a value >= 2 lowers prediction of malware detection.\n\n* **AvSigVersion** - Defender state information e.g. 1.217.1014.0 \n    * To be filled in"},{"metadata":{"_uuid":"563662cb7e40f7de76ac8420c4c6c2966443ae2d"},"cell_type":"markdown","source":"Let's also check the interaction between the top 2 features. We'll make use of 2D interactive plots for this."},{"metadata":{"trusted":true,"_uuid":"dd36d2f44ee19ef16c0cadaa44dd9394aa893c6c"},"cell_type":"code","source":"inter1  =  pdp.pdp_interact(model=model, dataset=X_val, model_features=X_val.columns.tolist(), features=['AVProductStatesIdentifier', 'AVProductsInstalled'])\n\npdp.pdp_interact_plot(pdp_interact_out=inter1, feature_names=['AVProductStatesIdentifier', 'AVProductsInstalled'], plot_type='contour')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42efe2e05866774e1762afd2435d65d1d753741a"},"cell_type":"markdown","source":"This graph shows predictions for any combination of Goals Scored and Distance covered.\n\nWhen AVProductsInstalled >= 2, the impact of AVProductStatesIdentifier on the prediction is not as much as it would have had when AVProductsInstalled < 2"},{"metadata":{"_uuid":"b789956859beb657ca35d3217bbb63ab0f141750"},"cell_type":"markdown","source":"<a id=\"3.3\"></a> <br>\n### **SHAP Values**\n SHAP values break down a single prediction to show the impact of each feature"},{"metadata":{"trusted":true,"_uuid":"777b3ad0e5c2435e587de6b35199361e1f6819ee"},"cell_type":"code","source":"row_to_show = 17\ndata_for_prediction = X_val.iloc[row_to_show]  # use 1 row of data here. Could use multiple rows if desired\ndata_for_prediction_array = data_for_prediction.values.reshape(1, -1)\n\nmodel.predict_proba(data_for_prediction_array)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"92a30ca2a755fbb7261ce9535b69cf83ae55277c"},"cell_type":"markdown","source":"The machine is **83.38%** likely to get infected by malware"},{"metadata":{"trusted":true,"_uuid":"e61bc0372198f44bad148280909a1db8ae9b9d3d"},"cell_type":"code","source":"# Create object that can calculate shap values\nexplainer = shap.TreeExplainer(model)\n\n# Calculate Shap values\nshap_values = explainer.shap_values(data_for_prediction)\n\nshap.initjs()\nshap.force_plot(explainer.expected_value[1], shap_values[1], data_for_prediction)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3e2d2eb0d5298d07976c1b00e7ca614da277aa2a"},"cell_type":"markdown","source":"**Feature values causing increased predictions are in pink, and their visual size shows the magnitude of the feature's effect. Feature values decreasing the prediction are in blue**"},{"metadata":{"_uuid":"886d96fc85fd509690be13717acff18cc9326088"},"cell_type":"markdown","source":"**SHAP Summary Plot:** SHAP values all of validation data samples and not just a single row"},{"metadata":{"trusted":true,"_uuid":"8e727d20998872dadae54c580f31d83d2ecc7238"},"cell_type":"code","source":"# Calculate shap_values for all of val_X rather than a single row, to have more data for plot.\nshap_values = explainer.shap_values(X_val)\n\n# Make plot. Index of [1] is explained in text below.\nshap.summary_plot(shap_values[1], X_val)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f40a7f8a9fb04ecdb7f1489b81e0ef0dc0311b6"},"cell_type":"markdown","source":"**More to come. Stay tuned!**"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}